Netdata status is Minor Service Outage

Wed 26
Thu 27
Fri 28
Sat 29
Sun 30
Mon 31
Tue 1
now

Netdata Agent (all platforms)

Wed 26
Thu 27
Fri 28
Sat 29
Sun 30
Mon 31
Tue 1
now
Last updated 1 minute ago from official status page. Learn more
Stay ahead of Netdata outages
Sign up to create a custom dashboard to monitor the services you rely on. 3,000+ services supported.

Active Incidents

Nightly static builds overwrite node and metric data upon install
Started 1 Apr 2025 09:00:06 (23 hours ago), still ongoing
Major Incident
Investigating
Agent (all platforms)

We have found that a recent change in the nightly static builds of Netdata Agent causes meta data on the Agent to be overwritten. Specifically the sqlite3 database that keeps meta data on which timeseries stored in dbengine correspond with which metrics, and the information on the Agent's "machine GUID" are overwritten with the same copy in the build package.

Not affected are:

  • All stable releases
  • Native packages (.deb and .rpm)

Affected are all nightly static builds with the following version numbers:

  • 2.3.0-50-nightly
  • 2.3.0-60-nightly
  • 2.3.0-72-nightly
  • 2.3.0-78-nightly
  • 2.3.0-87-nightly

The initial impact is that all affected Agent installs, even though they still have the timeseries data stored on disk, have lost all meta data associated with it, so these timeseries become inaccessable. This is unrecoverable. Additionally, the main form of identification is overwritten, too.

We are assessing what the impact is for users of Netdata Cloud, and will update this incident with more information when the investigation is completed.

The bug itself has been fixed and merged. We will issue a new nightly build shortly.

Recently Resolved Incidents

No recent incidents

Netdata Outage Survival Guide

A step-by-step guide to help you survive a Netdata outage
NaN%

    Netdata Components

    Wed 26
    Thu 27
    Fri 28
    Sat 29
    Sun 30
    Mon 31
    Tue 1
    now

    Netdata Cloud Web UI

    Wed 26
    Thu 27
    Fri 28
    Sat 29
    Sun 30
    Mon 31
    Tue 1
    now

    Netdata Cloud Alerts

    Wed 26
    Thu 27
    Fri 28
    Sat 29
    Sun 30
    Mon 31
    Tue 1
    now

    Netdata Cloud Charts

    Wed 26
    Thu 27
    Fri 28
    Sat 29
    Sun 30
    Mon 31
    Tue 1
    now

    Netdata Cloud Event Feed

    Wed 26
    Thu 27
    Fri 28
    Sat 29
    Sun 30
    Mon 31
    Tue 1
    now

    Netdata Cloud Payment

    Wed 26
    Thu 27
    Fri 28
    Sat 29
    Sun 30
    Mon 31
    Tue 1
    now

    Netdata Agent - Cloud Connection (ACLK)

    Wed 26
    Thu 27
    Fri 28
    Sat 29
    Sun 30
    Mon 31
    Tue 1
    now

    Netdata Agent (all platforms)

    Wed 26
    Thu 27
    Fri 28
    Sat 29
    Sun 30
    Mon 31
    Tue 1
    now
    Nightly static builds overwrite node and metric data upon install
    Started 1 Apr 2025 09:00:06 (23 hours ago), still ongoing
    Major Incident
    Investigating
    Agent (all platforms)

    We have found that a recent change in the nightly static builds of Netdata Agent causes meta data on the Agent to be overwritten. Specifically the sqlite3 database that keeps meta data on which timeseries stored in dbengine correspond with which metrics, and the information on the Agent's "machine GUID" are overwritten with the same copy in the build package.

    Not affected are:

    • All stable releases
    • Native packages (.deb and .rpm)

    Affected are all nightly static builds with the following version numbers:

    • 2.3.0-50-nightly
    • 2.3.0-60-nightly
    • 2.3.0-72-nightly
    • 2.3.0-78-nightly
    • 2.3.0-87-nightly

    The initial impact is that all affected Agent installs, even though they still have the timeseries data stored on disk, have lost all meta data associated with it, so these timeseries become inaccessable. This is unrecoverable. Additionally, the main form of identification is overwritten, too.

    We are assessing what the impact is for users of Netdata Cloud, and will update this incident with more information when the investigation is completed.

    The bug itself has been fixed and merged. We will issue a new nightly build shortly.

    Netdata Agent Repositories

    Wed 26
    Thu 27
    Fri 28
    Sat 29
    Sun 30
    Mon 31
    Tue 1
    now