Root Cause Analysis

This incident was caused by an authentication failure following a routine application secret rotation. One service (Agent Designer) was still using an outdated credential due to an incomplete prior rotation. Because the system temporarily allows multiple valid credentials, the mismatch was not detected earlier. When a subsequent rotation invalidated the old credential, it resulted in service errors.

Resolution

We resolved the issue by updating Agent Designer to use the latest valid credential and restarting the service. This restored normal functionality.

Remediation Items

• Improve validation during secret rotations to ensure all dependent services are updated correctly

• Reduce ambiguity in application ownership and configuration to prevent credential mismatches

• Avoid sharing credentials across multiple services where possible