Skip to main content
ProfoundBot is a user-initiated HTTP agent. When you supply a URL through the Profound platform or API, it fetches that public web page and returns its content as Markdown and/or HTML. For domains enrolled in organization Settings > Web Scrape, it also fetches the pages you import into a Knowledge Base with Scrape Page or Crawl Website > Sitemap.

Summary

What it does

When you supply a public webpage URL to read the content from, ProfoundBot makes a bounded set of HTTP requests to that URL’s origin to fetch the page, then returns the content as Markdown, HTML or both. When a page is rendered with a headless browser, that page’s own subresources (scripts, stylesheets, images) load as they would in any browser. ProfoundBot doesn’t follow links to other pages. For Knowledge Bases, ProfoundBot fetches each page added with Scrape Page and each page URL listed in a sitemap imported with Crawl Website > Sitemap, as long as the page’s domain is enrolled in organization Settings > Web Scrape (see Network origin and dedicated IPs). Scheduled syncs of those documents fetch the pages the same way.

What it doesn’t do

  • Crawl. Each customer action fetches exactly one URL. The bot does not recursively follow links or build an index of your site.
  • Submit forms, follow login flows, or access authenticated areas.
  • Persist a long-running index of your site. Fetched content is delivered to you and is not republished.
  • Reach private or internal network targets.

Request behavior

Each user action triggers a small, bounded interaction scoped to the single requested URL.

Network origin and dedicated IPs

By default, ProfoundBot egresses from a dynamic IP range. Identify it by its User-Agent. You can enroll a domain (a per-domain setting, off by default) so that ProfoundBot fetches it from a dedicated egress IP range that Profound owns. This is useful if you prefer to allowlist Profound by IP rather than by User-Agent. When a domain is enrolled, requests egress from this range:
  • 134.49.168.0/24
Domains enrolled before Profound moved to this range keep egressing from the previous dedicated IPs until you move them:
  • 54.71.251.60
  • 54.185.59.110
  • 100.22.234.65
These dedicated IPs are also available in a machine-readable JSON format.
To enroll a domain for dedicated-IP fetching, go to your Profound account Settings → Web Scrape and toggle the Primary Domain Scraping via Static IPs setting to On.
If some of your domains still use the previous IPs, Settings → Web Scrape lists them with a Move to the new IP range button. Add 134.49.168.0/24 to your allowlist before you move them, because after the move ProfoundBot fetches those domains only from the new range.

robots.txt handling

ProfoundBot is user-initiated and fetches a single page per user action rather than crawling. Support for honoring Disallow directives in robots.txt is planned for a future release. To prevent ProfoundBot from accessing your site in the meantime, see How to block section.

How to identify the bot

Use the exact User-Agent header:
All Profound bots use User-Agent strings that start with Profound, so a prefix match is a good way to identify all Profound traffic, current and future. For enrolled domains, you can also match on the dedicated egress IPs listed in the Network origin and dedicated IPs section.

How to block

Block ProfoundBot using any of the following methods:
  • A WAF or CDN rule matching User-Agent containing ProfoundBot
  • A WAF or CDN rule matching User-Agent containing the prefix Profound
  • For enrolled domains, a firewall or WAF rule blocking the dedicated egress range (134.49.168.0/24) and, for domains that still use them, the previous dedicated IPs (54.71.251.60, 54.185.59.110, 100.22.234.65)

Reporting abuse

If you observe behavior that doesn’t match this documentation, such as recursive crawling, request rates inconsistent with user-initiated single-page fetches, or activity from an unrecognized Profound* User-Agent, report it to security@tryprofound.com with example log lines. Profound treats these reports as security issues and responds to all of them.