We have monitored the systems since the fix to the query pipeline was deployed and not observed any signs of memory corruption since then. As noted previously, the problem was in our query infrastructure and the integrity of ingested data was never compromised.
Resolved
We have monitored the systems since the fix to the query pipeline was deployed and not observed any signs of memory corruption since then. As noted previously, the problem was in our query infrastructure and the integrity of ingested data was never compromised.
Monitoring
We've been observing the service closely over the past 24 hours and have deployed tooling to detect any more memory corruption. We've also released a fix for the query pipeline and will continue to monitor until confident the issue has been resolved.
Monitoring
We have deployed a fix which should alleviate the excessive / misbehaving email notifications from Monitors. Monitors should not misfire again due to this issue. Rest assured the issue lies with our query infrastructure and therefore the integrity of ingested data is 100%.
Monitoring
The issue has been resolved by restarting the processes which caused the malformed data to be returned. We're currently investigating the root cause which is likely originating from an external library corrupting shared memory.
Monitoring
We've run mitigations and queries work correctly again, but we continue investigation for root cause.
Investigating
We see another instance of monitors misfiring due to corrupted results of queries. The team is investigating the root cause of this recurring problem.
