Root Cause Analysis
During an infrastructure update, a newly introduced network component incorrectly changed protocol information on inbound webhook requests. Because request authentication depends on reconstructing the original request details, this mismatch caused valid requests to fail signature verification and be rejected.
Resolution
Engineers corrected the network configuration so inbound voice traffic retained the expected protocol information. Signature validation resumed immediately, and normal call processing was fully restored. The incident affected a single region for approximately 50 minutes.
Remediation and Preventive Actions
Enhanced observability and alerting
We are deploying monitoring that alerts on signature-validation failures and unexpected drops in successful call volume. This will help detect authentication issues independently of server error rates. During the incident, the service remained available and returned timely responses while rejecting affected requests, so traditional error-rate monitoring did not detect the issue.
Post-deployment production verification
The issue occurred on a network path that is not fully represented in pre-production environments. We are adding an automated, signed validation request that will run immediately after changes to voice traffic routing.
Fallback hardening
During the incident, we corrected fallback configuration for the affected production environment. We are also conducting a broader audit of fallback routing and adding validation to prevent incomplete configurations from being saved.
