SysEngineering

outage-slack-ping

Watches the public status feeds of our core SaaS vendors and posts to a Slack channel when an incident opens and again when it resolves — a clean outage timeline, not a firehose. Between those two edges it stays silent.

Vendors watched

Vendor Status feed Kind
Slack status.slack.com/api/v2.0.0/current Slack-native API
Zoom status.zoom.us/api/v2/incidents/unresolved.json Atlassian Statuspage
Splunk status.splunkcloud.com/api/v2/incidents/unresolved.json Atlassian Statuspage
NetSuite status.netsuite.com/api/v2/incidents/unresolved.json Atlassian Statuspage
Google Workspace google.com/appsstatus/dashboard/incidents.json Google dashboard feed
Concur (SAP) open.concur.com/api/open/incidents Concur Open (undocumented JSON API)
GitHub githubstatus.com/api/v2/incidents/unresolved.json Atlassian Statuspage
DocuSign status.docusign.com/api/v2/... Atlassian Statuspage
Perimeter 81 status.perimeter81.com/api/v2/... Atlassian Statuspage
Cisco Meraki status.meraki.com/api/v2/... (→ meraki.net) Atlassian Statuspage
Palo Alto Networks status.paloaltonetworks.com/api/v2/... Atlassian Statuspage
Smartsheet status.smartsheet.com/api/v2/... Atlassian Statuspage
Claude (Anthropic) status.claude.com/api/v2/... Atlassian Statuspage
ChatGPT (OpenAI) status.openai.com/api/v2/summary.json Atlassian Statuspage (summary.json only)
ShareFile (Citrix) status.sharefile.com/api/v2/... Atlassian Statuspage
Sophos sophoscentral.status.page/summary.json StatusCast

All feeds are public — no API keys or auth needed for any vendor.

Not watched: Okta

Okta has no public status feed. status.okta.com sits behind a Salesforce login (its API paths and RSS both 401 → login redirect), and trust.okta.com only hosts security advisories, not live incident data. So it can’t be polled the way the vendors above are. If we want Okta coverage: a paid third-party status aggregator with an API (e.g. StatusGator / IsDown), or piping Okta’s own email/SMS Trust notifications into Slack separately.

How it works

  1. Poll every vendor’s feed and normalize each active incident into a common shape (vendor, affected services, severity, status, link).
  2. Keep only major incidents by default (real outages/disruptions). Set INCLUDE_MINOR=true to also alert on minor degradations and notices.
  3. Diff the current active set against state.json (what we saw last run):
    • keys that are new → post a 🚨 outage alert
    • keys that disappeared → post a 🟢 resolved alert
    • unchanged → nothing
  4. Write the new active set back to state.json.

In CI, the workflow only commits state.json to the isolated automation/outage-state branch when the active set actually changed. That branch is never merged back, which keeps runtime writes off main while its history remains an outage log.

Feed-down safety: if a vendor’s feed errors on a run, that vendor’s known incidents are held in state (not falsely reported as resolved), and no false “recovered” message is sent.

Severity mapping

Vendor → Major → Minor
Statuspage (Zoom/Splunk/NetSuite) impact = major / critical impact = minor
Slack type = outage / incident notice / maintenance
Google Workspace SERVICE_DISRUPTION SERVICE_INFORMATION
Concur severity = disruption / degradation performance / partial
Sophos (StatusCast) any unresolved incident — (maintenance is skipped entirely)

Slack setup (one-time)

The code posts via a Slack incoming webhook — the same mechanism as okta-api-ping. To wire it up:

  1. Create (or reuse) a Slack app at https://api.slack.com/apps.
  2. Incoming Webhooks → enable → Add New Webhook to Workspace → pick the outage channel. Copy the webhook URL.
  3. Add it as a GitHub Actions repo secret named OUTAGE_SLACK_WEBHOOK_URL (Settings → Secrets and variables → Actions).

Until that secret exists, the workflow runs fine and logs what it would post (dry run) — so you can watch it work before turning on real alerts.

The messages use Block Kit: a header, a section with the incident title, fields for vendor / severity / affected services / status, and a View status page button linking straight to the vendor’s incident.

Local dev / testing

cd integrations/outage_slack_ping
pip install -r requirements.txt
cp .env.example .env            # optional: add SLACK_WEBHOOK_URL to test real posts

python status_ping.py --summary   # list everything currently active (all severities)
python status_ping.py --dry-run   # full run, no Slack, no state write
python status_ping.py --no-slack  # full run, writes state, no Slack
python status_ping.py             # real run (posts if SLACK_WEBHOOK_URL set)

Handy env vars: INCLUDE_MINOR=true, ONLY_PROVIDERS=Slack,Zoom, STATE_PATH=/tmp/state.json (test without touching the committed state).

Schedule

GitHub Actions cron */5 * * * * (every 5 minutes — the platform minimum). Also runnable on demand via workflow_dispatch.

Adding / removing a vendor

Edit the PROVIDERS registry near the bottom of the provider section in status_ping.py. Anything on Atlassian Statuspage is a one-liner:

"Okta": lambda s: fetch_statuspage(s, "Okta", "https://status.okta.com"),

Vendors with a custom feed (like Slack, Google, Concur) need a small dedicated fetch_* function that returns a list of normalized Incidents.