Gladstone Cloud | Connection Issues

Incident Report for Gladstone Software

Postmortem

Summary

On July 9th, a small number of our customers experienced service degradation due to an issue within our cloud platform's database layer. A background process led to an unexpected spike in temporary storage usage which caused one of our database clusters to stop processing requests for a short period. Services were fully restored within 30 minutes.

What Happened

A scheduled background process consumed temporary database storage at a rate that exceeded expected usage patterns. As a result, the TempDB volume reached full capacity, which prevented the database cluster from responding to incoming queries. This impacted platform responsiveness for affected customers during the incident window. Our internal monitoring systems had already flagged elevated resource usage, and an investigation was underway at the time. However, the issue escalated faster than anticipated, requiring manual intervention to clear the affected process and restore normal operations.

What We’ve Done

To prevent this kind of issue from recurring, we have taken the following actions:

  • We’ve expanded our observability systems to track TempDB usage trends, enabling earlier detection and alerting based on growth rates, not just usage thresholds.
  • Our internal database engineering team is implementing automated safeguards to detect and limit processes that generate unusually high temporary load.

We sincerely apologise for the inconvenience caused. Platform reliability is one of our highest priorities, and we are continuously working to strengthen the resilience of our systems. This incident has highlighted opportunities to improve early warning and automated protection, and we’re acting decisively to address them.

If you have any questions or need further clarification, please don’t hesitate to reach out to your account representative or our support team.

Posted Jul 18, 2025 - 11:07 UTC

Resolved

Incident Date: 9th July 2025
Impact Duration: Up to 30 minutes
Affected Service: Cloud platform – database availability for a subset of customers
Posted Jul 09, 2025 - 18:45 UTC