On October 15, 2026, website crawls for knowledge bases did not run because of a usage limit on our web crawling provider. During that time, the affected sources showed as processing instead of reporting an error.
At about 5:40 PM ET the same day, we raised the limit, and crawls started working again.
To prevent this from happening again:
We now apply weekly crawl limits to each knowledge source, so no single source can use up the shared allowance.
We are adding alerts that warn us before we approach the provider's usage limit.
We are adding a clear error in the app when the provider refuses a crawl, so a source no longer stays in processing.
No components marked as affected
Resolved
On October 15, 2026, website crawls for knowledge bases did not run because of a usage limit on our web crawling provider. During that time, the affected sources showed as processing instead of reporting an error.
At about 5:40 PM ET the same day, we raised the limit, and crawls started working again.
To prevent this from happening again:
We now apply weekly crawl limits to each knowledge source, so no single source can use up the shared allowance.
We are adding alerts that warn us before we approach the provider's usage limit.
We are adding a clear error in the app when the provider refuses a crawl, so a source no longer stays in processing.