Infrastructure metrics have now returned to normal levels, with activity increasing again while latency remains within the expected range.

We will continue to closely monitor the environment to ensure the infrastructure remains stable and that no residual impact from this incident is observed.

Our monitoring and security mechanisms remain active at all times, allowing us to quickly identify abnormal behaviour and take immediate mitigation measures should this or any similar issue occur again.

At this stage, service performance is considered fully restored and operating normally.

Mitigation measures have now been applied, and we are currently seeing a progressive improvement in infrastructure performance.

Our initial suspicion has been confirmed. The primary cause of the increased latency was a configuration issue introduced within a client-managed application. This resulted in thousands of transactions remaining active simultaneously, placing significant pressure on the environment and increasing latency across other services hosted within the same infrastructure.

The client has since corrected the configuration, and the updated version has been successfully deployed to their environment.

During the incident, the underlying Cloud infrastructure also automatically applied a containment measure in response to the abnormal behaviour detected. While intended to protect the environment, this measure further affected the performance of the infrastructure and contributed to the degradation experienced across our services.

Following the correction and the mitigation actions already taken, infrastructure performance is now improving significantly. We are continuing to review all available telemetry, logs and transaction data to validate the recovery, assess the full impact of the incident and ensure there are no remaining effects.

The incident remains under active monitoring while we confirm that performance has fully returned to normal levels.

A significant increase in latency across our infrastructure was identified at 16:55 (Lisbon time), affecting general operations and the deployment and management of both our own services and managed client applications.

As part of the ongoing investigation, we reviewed the telemetry, access records and logs currently available to us. This analysis showed a substantial increase in external traffic associated with one specific client application. Our security layer, provided by Cloudflare, recorded more than 650,000 requests in just over 10 minutes, representing a considerable and highly unusual increase in traffic to that service.

During the same investigation, we also identified a configuration issue within the affected client service. This configuration is outside of our management and may have contributed to the excessive load and latency experienced across the infrastructure.

In parallel, we detected several errors within the underlying Cloud infrastructure used to host and manage our services. Based on the information currently available, these errors began appearing at approximately 15:30 (Lisbon time), before the more noticeable increase in latency was identified. We are continuing to investigate whether these infrastructure-level issues contributed to the incident or amplified its overall impact.

As an immediate mitigation measure, the affected client application was fully isolated from the remaining infrastructure. Following this action, we observed a significant improvement in latency across the rest of the environment, including our own applications and other managed client services.

The incident remains ongoing. While the mitigation measures already applied have significantly reduced the impact, we do not yet consider the infrastructure fully stabilised and the root cause has not been conclusively identified.

Our team remains actively engaged in the investigation, analysing all available information to understand how the identified factors are related, determine the primary cause and restore normal infrastructure performance as quickly as possible.

Further updates will be provided as the investigation progresses.

We are currently experiencing elevated latency across the infrastructure hosting both our own services and managed client services.

The issue is affecting general infrastructure operations and system updates, including the deployment of new application versions to managed client environments and our own applications.

Our team is actively investigating the cause and working to restore normal performance as quickly as possible. We will provide further updates as more information becomes available.