Summary
What it does
When you supply a public webpage URL to read the content from, ProfoundBot makes a bounded set of HTTP requests to that URL’s origin to fetch the page, then returns the content as Markdown, HTML or both. When a page is rendered with a headless browser, that page’s own subresources (scripts, stylesheets, images) load as they would in any browser. ProfoundBot doesn’t follow links to other pages. For Knowledge Bases, ProfoundBot fetches each page added with Scrape Page and each page URL listed in a sitemap imported with Crawl Website > Sitemap, as long as the page’s domain is enrolled in organization Settings > Web Scrape (see Network origin and dedicated IPs). Scheduled syncs of those documents fetch the pages the same way.What it doesn’t do
- Crawl. Each customer action fetches exactly one URL. The bot does not recursively follow links or build an index of your site.
- Submit forms, follow login flows, or access authenticated areas.
- Persist a long-running index of your site. Fetched content is delivered to you and is not republished.
- Reach private or internal network targets.
Request behavior
Each user action triggers a small, bounded interaction scoped to the single requested URL.Network origin and dedicated IPs
By default, ProfoundBot egresses from a dynamic IP range. Identify it by its User-Agent. You can enroll a domain (a per-domain setting, off by default) so that ProfoundBot fetches it from a dedicated egress IP range that Profound owns. This is useful if you prefer to allowlist Profound by IP rather than by User-Agent. When a domain is enrolled, requests egress from this range:134.49.168.0/24
54.71.251.6054.185.59.110100.22.234.65
If some of your domains still use the previous IPs, Settings → Web Scrape lists them with a Move to the new IP range button. Add
134.49.168.0/24 to your allowlist before you move them, because after the move ProfoundBot fetches those domains only from the new range.robots.txt handling
ProfoundBot is user-initiated and fetches a single page per user action rather than crawling. Support for honoring Disallow directives in robots.txt is planned for a future release.
To prevent ProfoundBot from accessing your site in the meantime, see How to block section.
How to identify the bot
Use the exact User-Agent header:How to block
Block ProfoundBot using any of the following methods:- A WAF or CDN rule matching User-Agent containing
ProfoundBot - A WAF or CDN rule matching User-Agent containing the prefix
Profound - For enrolled domains, a firewall or WAF rule blocking the dedicated egress range (
134.49.168.0/24) and, for domains that still use them, the previous dedicated IPs (54.71.251.60,54.185.59.110,100.22.234.65)
Reporting abuse
If you observe behavior that doesn’t match this documentation, such as recursive crawling, request rates inconsistent with user-initiated single-page fetches, or activity from an unrecognizedProfound* User-Agent, report it to security@tryprofound.com with example log lines.
Profound treats these reports as security issues and responds to all of them.