About the DisclosureFeed crawler

DisclosureFeed operates an automated source-fetching bot that retrieves formally-filed breach disclosures from public regulatory portals.

Identification

Our bot identifies itself in the HTTP User-Agent header as:

DisclosureFeed bot (https://disclosurefeed.com/about-the-bot)

SEC EDGAR requests additionally carry a contact email per the SEC user-agent policy.

Politeness policy

• SEC EDGAR: max 8 requests/sec (within SEC's 10/sec documented cap)
• HHS OCR: 1 request/sec, sequential
• State AG portals: 1 request per 3 seconds, jittered
• UK ICO: 0.5 request/sec
• OAIC, CNIL, BfDI: 0.5 request/sec

We honor robots.txt directives and Crawl-delay headers. We back off exponentially on HTTP 429 and 5xx responses.

What we collect

Only material that is already public on the source regulator's portal. We archive raw filings to immutable storage for evidentiary purposes (audit replay, schema migration), but we do not redistribute raw bodies on our public surfaces — links go to the originating regulator URL.

Block requests

If you operate a regulator portal and need our bot to slow down or stop, email abuse@disclosurefeed.com. We respond within one business day and apply the requested rate limit immediately.