Skip to content

PropFirmVSBot

PropFirmVSBot keeps the comparison on PropFirmVS current. It reads the public pages where prop firms publish their rules, prices and promotions, so a change on a firm’s site shows up here within a day, with the date and the source, when the firm’s website allows our checker.

What it reads

  • Public pages only: a firm’s homepage, its rules, FAQ, payout and scaling pages, and the pages we cite as sources.
  • It never logs in, never fills in or submits forms, never starts a checkout and never follows affiliate links.
  • For firms we do not compare yet, it only checks that the website is still online.

How often

  • Once a night (from about 00:00 UTC): a firm’s main pages and up to three rules pages.
  • Once around midday (about 12:00 UTC): only the homepage, for discounts and promo codes.
  • Pages that could not be read are tried once more in the early morning.
  • One page at a time per website, with a pause after each page. A typical firm sees a few dozen requests a day at most.

How it identifies itself

Plain requests use this user agent:

Mozilla/5.0 (compatible; PropFirmVSBot/1.0; +https://propfirmvs.com/bot)

Some sites only show their content after JavaScript has run, so most pages are read in a real browser. That browser keeps a normal Chrome identity with our name added:

Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0.0.0 Safari/537.36 (compatible; PropFirmVSBot/1.0; +https://propfirmvs.com/bot)

When a site's firewall refuses both identities above on every route we have, the checker makes one last attempt with a standard Chrome identity, without our name, from a separate address of our own server. That attempt follows the same rules as every other visit: robots.txt is checked first (a page it disallows for PropFirmVSBot is never read), it only reads public pages, and it never logs in, submits forms or follows affiliate links. Firms that do not want to be read at all can disallow PropFirmVSBot in robots.txt or email us, and we stop checking their site.

robots.txt

PropFirmVSBot follows robots.txt. It obeys rules for the token PropFirmVSBot, and otherwise the rules for *. A page it may not read is skipped, and the firm is not automatically checked that night; when no other route has read its pages either, the firm page says that the firm’s robots.txt does not allow our checker, next to the date of the last successful check. To keep it away from part of your site:

User-agent: PropFirmVSBot
Disallow: /members/

To allow it explicitly (for example when your robots.txt disallows unknown bots):

User-agent: PropFirmVSBot
Allow: /

If your firewall or bot protection blocks it, you can allow the two user agents above, so the checker never needs the last-resort attempt. On Cloudflare, for example, a WAF custom rule with the expression (http.user_agent contains "PropFirmVSBot") and the action Skip does this. Please do not block the pages where you publish your rules: without them we cannot keep your listing correct.

When it cannot read a listed firm

Our crawler desk, an AI assistant that writes from abbott@propfirmvs.com, emails the firm and says what blocks the checker (robots.txt or a firewall) and how to allow it. Meanwhile the firm page shows the date of the last successful check, what blocks the checker and, once the mail has gone out, that we have asked the firm to allow it. Every such mail says it was written by an AI assistant. A firm we have an affiliate relationship with gets this mail at its affiliate contact, with the details its tech team needs to allow the checker (user agent, this page, our server’s IP addresses and the pages we read), and we offer to use a partner feed or API instead if the firm has one. If nothing changes, a short reminder follows 7 days after the mail (4 days for affiliate partners). If the pages still cannot be read 14 days after that mail, the firm is moved out of the comparison to our list of firms we cannot check automatically, because we can no longer keep its data correct. It returns by itself as soon as the checker can read its pages again. The same parking rules apply to every firm, including firms we have an affiliate relationship with.

Questions or problems

Email info@propfirmvs.com about crawling, a wrong figure or a page we should read instead. Corrections are handled as described in our editorial policy.

Compare (0/3)