A portion of queries experienced increased latency and elevated error rates which was caused by an interaction between our service mesh and internal cache services. Following a deployment, some query workers continued attempting to connect to outdated cache instance addresses, even after the replacement instances had become healthy. This resulted in query timeouts and failures.
We identified the underlying issue and mitigated it by adjusting the networking configuration of our query workers. Normal operation was restored by 12:07 UTC, and query error rates have returned to their expected levels.
We are reviewing the incident and implementing measures to prevent similar issues in the future.
Resolved
A portion of queries experienced increased latency and elevated error rates which was caused by an interaction between our service mesh and internal cache services. Following a deployment, some query workers continued attempting to connect to outdated cache instance addresses, even after the replacement instances had become healthy. This resulted in query timeouts and failures.
We identified the underlying issue and mitigated it by adjusting the networking configuration of our query workers. Normal operation was restored by 12:07 UTC, and query error rates have returned to their expected levels.
We are reviewing the incident and implementing measures to prevent similar issues in the future.
Investigating
We are investigating elevated query error rates and increased latency affecting some customers. Our engineering team is actively working to identify the cause and restore normal operation.
We will provide further updates as more information becomes available.
