AppFinderBot

Every request sends this user agent:

AppFinderBot/1.0 (+https://appfinder.tech/trust/crawler)

In robots.txt, address it as AppFinderBot. It doesn’t run JavaScript, fill in forms, log in or follow links it finds on your pages.

Only for apps with a published listing, and only on the domain of the listed website:

• / (the listed homepage)
• /pricing
• /security
• /trust
• /features
• /integrations

• Over HTTPS, from public internet addresses only. Redirects must stay on the same domain.
• HTML only, and at most the first 1.5 MB of a page. Images, scripts, PDFs and files are never downloaded.
• We keep the visible text of the newest 3 versions of each page to notice changes. It’s used internally and never republished. Fact quotes we keep are 300 characters or less.

AppFinderBot also reads a site once when someone listing an app asks us to fill in the form from its website: the homepage, and the pricing, features, integrations and download pages it links to (at most six requests), with the same rules and the same robots.txt. Nothing from that read is kept; the person checks what we found and submits it for review.

Each page

At most once every 72 hours, unless our staff ask for a fresh check.

Each host

At least 10 seconds between requests, across all of our servers. A longer Crawl-delay in your robots.txt is respected (up to 2 minutes).

robots.txt

Read before crawling and cached for 24 hours.

Errors

Pages that fail are retried later with growing gaps, up to two weeks. Pages that return 404 or 410 are checked again after a month.

AppFinderBot follows the robots.txt standard (RFC 9309). It uses the group for AppFinderBot when there is one, otherwise the * group. If your robots.txt is missing (a 4xx response) we may crawl; if it can’t be reached (a 5xx response or a timeout) we treat everything as disallowed and try again later.

To stop AppFinderBot completely:

User-agent: AppFinderBot
Disallow: /

To keep it away from one page, such as your pricing page:

User-agent: AppFinderBot
Disallow: /pricing

Opting out never removes or demotes a listing. The facts already on the profile stay, with their sources, and the app’s owner can keep them current from the dashboard.

• The crawler only proposes facts, each with the page it came from and a short quote.
• The app’s verified owner or AppFinder staff accept or reject every proposal. Nothing is published automatically.
• Published facts show their source on the profile, for example “Found on the pricing page · Oct 3, 2026”.
• Facts never depend on what a company pays us, and paid plans don’t change what the crawler reads.

How ranking works

If AppFinderBot causes a problem on your site, or you want to ask about it, write to trust@appfinder.tech. Include your domain and, if you can, the time of the requests.