
July 20, 2026
From Terminal Basics to CloudOps: My Journey in Linux, SRE & Observability
Chandrashekhar H S
How diving deep into Linux OS internals, Prometheus metrics, and deliberate failure testing shaped the way I build reliable systems.
When I first opened a Linux terminal, it felt stark and unforgiving. There were no friendly graphical buttons or wizard dialogs to guide me — just a blinking cursor waiting on stdin.
At first, like many beginners, I learned commands by rote: cd, ls -la, grep, systemctl. But as I started studying Computer Science and Engineering at Sahyadri College, something clicked: software doesn't run in a vacuum. Underneath every framework and API is a kernel managing memory, scheduling CPU cycles, handling file descriptors, and passing network packets through TCP sockets.
That realization changed everything about how I approached engineering.
Why Observability Matters More Than Uptime Guarantees
Anyone can deploy a container and call it "live." But what happens when the latency spikes to 3 seconds during peak hours? What happens when a memory leak slowly exhausts system RAM until the Linux Out-Of-Memory (OOM) killer drops a mission-critical process?
If you don't have observability, you're debugging in the dark. You are guessing.
When I started designing my CloudOps Observability Platform, my goal wasn't just to install Prometheus and Grafana. It was to build a system that tells the complete story of a cluster:
- Metrics (Prometheus & Node Exporter): Providing quantitative telemetry — CPU percentages, disk I/O, network throughput, and custom HTTP latency histograms.
- Logs (Loki & Promtail): Centralizing container and system logs so that when an anomaly occurs, I can jump directly from a spike on a dashboard to the exact log lines emitted at that second.
- Alerting (Alertmanager): Configuring severity-based routing so that critical issues notify engineering channels immediately while warning conditions are batched intelligently.
# Deliberate failure testing: verifying alert firing
- alert: HighMemoryUsage
expr: (node_memory_MemTotal_bytes - node_memory_MemAvailable_bytes) / node_memory_MemTotal_bytes * 100 > 80
for: 2m
labels:
severity: warning
annotations:
summary: "Host RAM utilization exceeded 80%"
Deliberate Failure Testing: Breaking Things on Purpose
One of the biggest lessons I learned building these systems is that an untested alert is worse than no alert at all. If you've never verified that Alertmanager actually routes a notification when a node crashes, you're hoping for luck in production.
I started injecting artificial bottlenecks:
- Filling up temporary disk partitions to test filesystem thresholds.
- Simulating network packet latency with
tc(traffic control). - Spawning dummy memory hogs to ensure our monitoring and Slack webhooks fired within seconds.
Seeing a Slack webhook light up with an accurate incident diagnosis before a system suffered an outage was the moment I fell in love with Site Reliability Engineering (SRE).
The Philosophy of Reliability
To me, SRE and Cloud Operations are not about eliminating failure completely — hardware will fail, networks will drop packets, and software will have bugs.
Reliability is about transparency, automation, and quick recovery:
- Automate operational toil: If you run a command more than twice a week, write a Bash script and automate it with Cron or GitHub Actions.
- Treat infrastructure as code: Everything in version control, containerized with Docker, reproducible with Docker Compose.
- Dig for the root cause: Don't just reboot a frozen server; check
dmesg, analyze process states (zombies and orphans), inspectlsof, and fix the leak.
I'm still learning every day, experimenting with new cloud architectures, and refining my tools. But one thing is certain: building reliable infrastructure is where my passion lies.
Thank you for reading!
— Chandrashekhar H S