← Back to projects

July 2026

Complete

Linux SRE Health Monitor

A production-style Linux monitoring solution built with Bash, Cron, Linux utilities, Slack Webhooks, and HTML reporting. The project continuously monitors system health, generates daily reports, logs metrics, and sends alerts when configured thresholds are breached.

LinuxUbuntuBashCronSlackGit
Linux SRE Health Monitor cover

In production Linux environments, critical service degradations often stem from gradual, unchecked resource leaks: uncollected zombie processes, mounting temporary disk usage, or silent runaway worker threads.

The Linux SRE Health Monitor is a lightweight, zero-dependency monitoring utility written in POSIX-compliant Bash. Designed to operate efficiently even on minimal compute nodes where running a heavy agent is impractical, it monitors system vitals and proactively dispatches threshold-based alerts to Slack.


Core Capabilities & Inspection Routines

The script inspects six vital system subsystems on every execution cycle:

  1. CPU & Load Averages: Evaluates 1-minute, 5-minute, and 15-minute load averages against total physical CPU core counts using /proc/loadavg and nproc.
  2. Memory Distribution: Extracts active, buffered, and cached memory statistics (free -m), detecting kernel memory pressure before the Out-Of-Memory (OOM) killer is invoked.
  3. Disk Partition Saturation: Monitors mounted filesystems via df -Ph, flagging any partition exceeding defined saturation thresholds (typically 80% warning, 90% critical).
  4. Process Health & Rogue States: Scans the process table (ps aux) to identify zombie (Z), uninterruptible sleep (D), or orphan processes, alongside the top 5 CPU and memory-consuming processes.
  5. Network Socket States: Audits TCP socket states using ss -s / netstat, tracking open file descriptors against system limits (ulimit -n).
  6. Automated Incident Triage Reporting: Assembles a formatted incident payload and transmits it via incoming Slack Webhooks with color-coded severity tags.

Architecture & Automation Workflow

terminal / snippet
+------------------------------------------------------------+
| Linux System / Kernel (/proc, /sys)                        |
+------------------------------+-----------------------------+
                               |
                               v
                     +-------------------+
                     | Cron Daemon       |  Runs every 5 mins
                     +---------+---------+
                               |
                               v
                     +-------------------+
                     | sre_monitor.sh    |  Bash Engine
                     +---------+---------+
                               |
        +----------------------+----------------------+
        |                                             |
  [Normal Bounds]                             [Threshold Breached]
        |                                             |
        v                                             v
+---------------+                             +---------------+
| Log to Local  |                             | Slack Webhook |
| Health Audit  |                             | Alert Payload |
+---------------+                             +---------------+

Production Reliability Practices

  • Zero-Dependency Architecture: Leverages standard Linux utilities available on virtually any distribution (Ubuntu, Debian, RHEL, Alpine).
  • Intelligent Cooldown & De-duplication: Implemented state-file tracking to prevent notification storms when a threshold remains in an alarm state across consecutive runs.
  • Fail-Safe Scripting: Built with strict error handling (set -euo pipefail), input sanitization, and structured trap routines for clean exit status.
  • Actionable Diagnostics: Alerts don't just state "Memory High" — they include the process ID, owner, and invocation arguments of the top memory-consuming process to enable immediate remediation.