Buildkite outage history and live incidents

Status: resolved Between 23:16 and 23:27 we saw elevated error rates across our platform due to a scheduled database upgrade. That upgrade was not intended to cause any impact, but due to an unforeseen dependency on this database customers experienced error rates averages 2.5% during this period.

Status: investigating We've received reports that Hosted Agents jobs are delayed in starting and are investigating.

Status: investigating The private keys for eight certificates issued by eMudhra, an external certificate authority, have been compromised. Software that still trusts these certificates may accept fraudulent HTTPS certificates. We’re updating the services that power Buildkite and preparing new CLI and Agent releases, which we expect to ship shortly. We’re also tracking fixes to the libraries and trust stores we depend on and will apply them as they become available. We’ll update this page with the new versions and any action customers need to take. Mozilla has published background on the compromise.

Status: investigating We are investigating delays to build and job notifications for a subset of customers.

Status: resolved We received reports from a few customers that Hosted Agent jobs were delayed in starting. We investigated and found this related to a GitHub incident with webhooks that caused throttling in Hosted Agent job creation once webhooks were once again being ingested. Any affected Hosted Agents customers would have seen delays in instance creation between 15:45-17:12UTC.