Degraded Data Ingestion [NA-01, EU-01]

Incident Report for SolarWinds Observability

Postmortem

2025-11-05 (Final Update) It was identified today that one of our cloud providers, had a bad network link affecting the path used to send data from the legacy Papertrail infrastructure to the Solarwinds Observability infrastructure. The affected link has since been disabled, and we have confirmed that all impact on our side has been fully resolved. Our cloud provider will perform the necessary repairs with no expected impact to our services.

2025-11-04
During the incident, some customers experienced delays in log ingestion due to an issue affecting our legacy Papertrail infrastructure. This resulted in higher-than-normal log processing latency for a subset of migrated customers. The issue has since resolved, and log ingestion has returned to normal performance.

We believe the incident was related to a network issue within our cloud provider’s environment.

Our team will continue to work with the provider to confirm the root cause and will publish a post-incident review once the investigation is complete.

Posted Nov 04, 2025 - 20:20 UTC

Resolved

This incident has been resolved.
Posted Nov 04, 2025 - 19:10 UTC

Update

We’re currently investigating delay in Logs data ingestion. As a result, New data may not appear in the UI, alerts may be delayed, and some dashboards could show missing or outdated values. We’ll provide another update within the hour.
Posted Nov 04, 2025 - 18:12 UTC

Update

We’re currently investigating delay in Logs data ingestion. As a result, New data may not appear in the UI, alerts may be delayed, and some dashboards could show missing or outdated values. We’ll provide another update within the hour.
Posted Nov 04, 2025 - 17:14 UTC

Update

We’re currently investigating delay in Logs data ingestion. As a result, New data may not appear in the UI, alerts may be delayed, and some dashboards could show missing or outdated values. We’ll provide another update within the hour.
Posted Nov 04, 2025 - 16:12 UTC

Update

We’re currently investigating delay in Logs data ingestion. As a result, New data may not appear in the UI, alerts may be delayed, and some dashboards could show missing or outdated values. We’ll provide another update within the hour.
Posted Nov 04, 2025 - 15:09 UTC

Update

We’re currently investigating delay in Logs data ingestion. As a result, New data may not appear in the UI, alerts may be delayed, and some dashboards could show missing or outdated values. We’ll provide another update within the hour.
Posted Nov 04, 2025 - 14:09 UTC

Update

We’re currently investigating delay in Logs data ingestion. As a result, New data may not appear in the UI, alerts may be delayed, and some dashboards could show missing or outdated values. We’ll provide another update within the hour.
Posted Nov 04, 2025 - 13:11 UTC

Update

We’re currently investigating delay in Logs data ingestion. As a result, New data may not appear in the UI, alerts may be delayed, and some dashboards could show missing or outdated values. We’ll provide another update within the hour.
Posted Nov 04, 2025 - 12:11 UTC

Update

We’re currently investigating delay in Logs data ingestion. As a result, New data may not appear in the UI, alerts may be delayed, and some dashboards could show missing or outdated values. We’ll provide another update within the hour.
Posted Nov 04, 2025 - 11:04 UTC

Update

We’re currently investigating delay in Logs data ingestion. As a result, New data may not appear in the UI, alerts may be delayed, and some dashboards could show missing or outdated values. We’ll provide another update within the hour.
Posted Nov 04, 2025 - 10:03 UTC

Update

We’re currently investigating delay in Logs data ingestion. As a result, New data may not appear in the UI, alerts may be delayed, and some dashboards could show missing or outdated values. We’ll provide another update within the hour.
Posted Nov 04, 2025 - 09:05 UTC

Update

We’re currently investigating delay in Logs data ingestion. As a result, New data may not appear in the UI, alerts may be delayed, and some dashboards could show missing or outdated values. We’ll provide another update within the hour.
Posted Nov 04, 2025 - 08:04 UTC

Update

We’re currently investigating delay in Logs data ingestion. As a result, New data may not appear in the UI, alerts may be delayed, and some dashboards could show missing or outdated values. We’ll provide another update within the hour.
Posted Nov 04, 2025 - 07:01 UTC

Update

We’re currently investigating delay in Logs data ingestion. As a result, New data may not appear in the UI, alerts may be delayed, and some dashboards could show missing or outdated values. We’ll provide another update within the hour.
Posted Nov 04, 2025 - 06:02 UTC

Update

We’re currently investigating delay in Logs data ingestion. As a result, New data may not appear in the UI, alerts may be delayed, and some dashboards could show missing or outdated values. We’ll provide another update within the hour.
Posted Nov 04, 2025 - 05:02 UTC

Update

We’re currently investigating delay in Logs data ingestion. As a result, New data may not appear in the UI, alerts may be delayed, and some dashboards could show missing or outdated values. We’ll provide another update within the hour.
Posted Nov 04, 2025 - 04:00 UTC

Update

We’re currently investigating delay in Logs data ingestion. As a result, New data may not appear in the UI, alerts may be delayed, and some dashboards could show missing or outdated values. We’ll provide another update within the hour.
Posted Nov 04, 2025 - 02:59 UTC

Update

We’re currently investigating delay in Logs data ingestion. As a result, New data may not appear in the UI, alerts may be delayed, and some dashboards could show missing or outdated values. We’ll provide another update within the hour.
Posted Nov 04, 2025 - 02:01 UTC

Update

We’re currently investigating delay in Logs data ingestion. As a result, New data may not appear in the UI, alerts may be delayed, and some dashboards could show missing or outdated values. We’ll provide another update within the hour.
Posted Nov 04, 2025 - 01:06 UTC

Update

We’re currently investigating delay in Logs data ingestion. As a result, New data may not appear in the UI, alerts may be delayed, and some dashboards could show missing or outdated values. We’ll provide another update within the hour.
Posted Nov 03, 2025 - 23:59 UTC

Update

We’re currently investigating delay in Logs data ingestion. As a result, New data may not appear in the UI, alerts may be delayed, and some dashboards could show missing or outdated values. We’ll provide another update within the hour.
Posted Nov 03, 2025 - 22:59 UTC

Update

We’re currently investigating delay in Logs data ingestion. As a result, New data may not appear in the UI, alerts may be delayed, and some dashboards could show missing or outdated values. We’ll provide another update within the hour.
Posted Nov 03, 2025 - 21:59 UTC

Update

We’re currently investigating delay in Logs data ingestion. As a result, New data may not appear in the UI, alerts may be delayed, and some dashboards could show missing or outdated values. We’ll provide another update within the hour.
Posted Nov 03, 2025 - 21:00 UTC

Update

We’re currently investigating delay in Logs data ingestion. As a result, New data may not appear in the UI, alerts may be delayed, and some dashboards could show missing or outdated values. We’ll provide another update within the hour.
Posted Nov 03, 2025 - 20:00 UTC

Update

We’re currently investigating delay in Logs data ingestion. As a result, New data may not appear in the UI, alerts may be delayed, and some dashboards could show missing or outdated values. We’ll provide another update within the hour.
Posted Nov 03, 2025 - 19:00 UTC

Update

We are continuing to investigate this issue.
Posted Nov 03, 2025 - 17:59 UTC

Update

We are continuing to investigate this issue.
Posted Nov 03, 2025 - 16:58 UTC

Update

We are continuing to investigate this issue.
Posted Nov 03, 2025 - 16:02 UTC

Update

We are continuing to investigate this issue.
Posted Nov 03, 2025 - 15:04 UTC

Investigating

We are currently investigating an issue with Logs data ingestion. As a result of this issue, submitting new data through the API might be delayed. We’ll provide an update within the hour.
Posted Nov 03, 2025 - 14:04 UTC
This incident affected: Alerting and Logs.