What the bot fetches, when it runs, and how site operators can identify or block it
The Profound Knowledge Base Fetcher is an HTTP agent that reads a website’s sitemap when you import that site into a Knowledge Base. It also re-downloads PDF files already in a Knowledge Base to check whether they have changed.
User-initiated sitemap imports, and auto-sync when enabled for an imported folder or page
Schedule
Only on the auto-sync cadence you choose for an imported folder or page: daily, weekly, or monthly. Off by default
Resources fetched
Sitemap files and child sitemaps they reference, and PDF files already imported into a Knowledge Base
Network origin
AWS, no fixed IP allow-list available
What it does
When you import a website with the Sitemap tab of Crawl Website, the fetcher requests the sitemap URL you enter and any child sitemaps listed in a sitemap index. Profound queues the page URLs it finds for import.The fetcher only reads sitemap files, so it runs when the Sitemap URL ends in .xml or .xml.gz. If you enter a plain domain instead, Profound finds the site’s pages another way and this fetcher isn’t involved.You can leave out https://, and the fetcher adds it. If the first attempt finds no pages and the address has no www., the fetcher tries again with www. added.With auto-sync on, the fetcher reads the sitemap again on every sync run to pick up new or removed pages. It also downloads each PDF that was imported from the web on that cadence and compares a fingerprint of its bytes with the previous version. Profound processes only the files that changed.
What it doesn’t do
Fetch HTML pages, images, scripts, or stylesheets. Page content is fetched separately, by ProfoundBot for domains enrolled in organization Settings > Web Scrape
Read robots.txt
Submit forms, follow login flows, or access authenticated areas
Run JavaScript or use a headless browser
Reach private or internal network targets
Request behavior
Per request timeout
20 seconds
Redirects
Followed manually, up to 5 hops
Concurrency
Sitemap files are fetched one at a time. During auto-sync, up to 10 documents sync at once, so up to 10 PDF downloads can run in parallel
Sitemap limits
Up to 50 sitemap files and 3 levels of nesting per import, 20 MB per file
All Profound bots use User-Agent strings that start with Profound, so a prefix match on Profound is a good way to identify all Profound traffic, current and future.
How to block
Block the fetcher using any of the following methods:
A web application firewall (WAF) or content delivery network (CDN) rule matching User-Agent containing ProfoundKnowledgeBase
A WAF or CDN rule matching User-Agent containing the prefix Profound
Reporting abuse
If you observe behavior that doesn’t match this documentation, such as requests for HTML pages, request rates that look like a crawl, or activity from an unrecognized Profound* User-Agent, report it to security@tryprofound.com with example log lines.Profound treats these reports as security issues and responds to all of them.
Was this page helpful?
Assistant
Responses are generated using AI and may contain mistakes.