Summary
On March 16, a service configuration issue caused all inbound voice calls to fail. Callers experienced a busy tone rather than reaching their intended destination. Once identified, our engineering team resolved the issue and restored full service. We recognize the impact this had and are taking concrete steps to prevent recurrence.
Root cause
A routine infrastructure update introduced a misconfiguration in how our voice services route traffic. As a result, the service was unable to identify available routing targets and failed to handle incoming calls.
Separately, two independent gaps in our monitoring contributed to a delayed response. Our primary alerting mechanism relies on signals from our telephony provider — because the misconfiguration prevented those signals from being generated, no alert was triggered. A secondary, provider-independent monitoring system that should have caught the outage was not operating as expected, and this gap had not been detected prior to the incident.
Resolution
Our engineering team corrected the service configuration, restoring normal traffic routing and returning the voice calling service to full operation.
Preventive actions
We are prioritizing the following improvements to reduce both the likelihood and the duration of similar incidents:
Independent monitoring coverage. Adding redundant health checks that operate independently of provider-generated signals, so a failure at the service level does not also suppress the alert.
Continuous validation of external health checks. Ensuring that secondary monitoring systems are actively verified on an ongoing basis, not assumed to be running.
Faster detection targets. Setting explicit time-to-detect goals for voice service availability and aligning our monitoring stack to meet them.
