I'm rejecting the next architecture PR that uses a Service Mesh for a team of 4 developers. We are gaslighting ourselves.

I’ve been lurking here for years, and after reading some recent posts, I need to say something that might make me unpopular with the "CV-Driven Development" crowd.

We are engineering our own burnout.

I've sat on hiring panels for the last 6 months, and the state of "Senior" DevOps is terrifying. I’m seeing a generation of engineers who can write complex Helm charts but can’t explain how DNS propagation works or debugging a TCP handshake.

Here is my analysis of why our industry is currently broken:

1. The Abstraction Addiction We are solving problems we don't have. I saw a candidate last week propose a multi-cluster Kubernetes setup with Istio for a simple internal CRUD app. When I asked why not just use a boring EC2 instance or ECS task, they looked at me like I suggested using FTP. We are choosing tools not because they solve a business problem, but because we want to put them on our LinkedIn. We are voluntarily taking on the operational overhead of Netflix without having their scale or their headcount.

2. The Death of Debugging To the user who posted "New DevOps please learn networking": Thank you. We are abstracting away the underlying systems so heavily that we are creating engineers who can "configure" but cannot "fix." When the abstraction leaks (and it always does, usually at 3 AM), these "YAML Engineers" are helpless because they don't understand the Linux primitives underneath.

3. Hiring is a Carnival Game We ask for 8 rounds of interviews to test for trivia on 15 different tools, but we don't test for systems thinking. Real seniority isn't knowing the flags for every CLI tool; it's knowing when not to use a tool. It's about telling management, "No, we don't need to migrate to that shiny new thing."

4. Complexity = Job Security (False) We tell ourselves that building complex systems makes us valuable. It doesn't. It makes us pagers. The best infrared engineers I know build systems so boring that they sleep through the night. If you are currently building a resume-padder architecture: Stop.



If you are a Junior: Stop trying to learn the entire CNCF landscape. Learn Linux. Learn Networking. Learn a noscripting language deeply. If you are a Senior: Stop checking boxes. Start deleting code.


The most senior thing you can do is build something so simple it looks like a junior did it, but it never goes down.

/endrant

https://redd.it/1pzkibf
@r_devops

Release management nightmare - how do you track what's actually going out?

Just had our third surprise production issue this month bc nobody knew which features were bundled in our release. Engineering says feature X is ready, QA cleared it last week, but somehow it wasn't in the build that went out Friday.

We have relied on Slack threads and manual Git tag checking, they have served us fine for a while but I think we've reached a breaking point. How does this roll up to leadership when they ask what shipped this sprint? Like, what are you using for release management to ensure everything falls into place?

https://redd.it/1pzi7l9
@r_devops

ai generated k8s configs saved me time then broke prod in the weirdest way

context: migrating from docker swarm to k8s. small team, needed to move fast. i had some k8s experience but never owned a prod cluster

used cursor to generate configs for our 12 services. honestly saved my ass, would have taken days otherwise. got deployments, services, ingress done in maybe an hour. ran in staging for a few days, did some basic load testing on the api endpoints, looked solid

deployed tuesday afternoon during low traffic window. everything fine for about 6 hours. then around 9pm our monitoring started showing weird patterns - some requests fast, some timing out, no clear pattern

spent the next few hours debugging the most confusing issue. turns out multiple things were breaking simultaneously:

our main api was crashlooping but only 3 out of 8 pods. took forever to realize the ai set liveness probe initialDelaySeconds to 5s. works fine in staging where we have tiny test data. prod loads way more reference data on startup, usually takes 8-10 seconds but varies by node. so some pods would start fast enough, others kept getting killed mid-initialization. probably network latency or node performance differences, never figured out exactly why

while fixing that, noticed our batch processor was getting cpu throttled hard. ai had set pretty conservative limits - 500m cpu for most services. batch job spikes to like 2 cores during processing. didnt catch it in staging because we never run the full batch there, just tested the api layer

then our cache service started oom killing. 256Mi limit looked reasonable in the configs but under real load it needs closer to 1Gi. staging cache is basically empty so never saw this coming

the configs themselves were fine, just completely generic. real problem was my staging environment told me nothing useful:

test dataset is 1% of prod size
never run batch jobs in staging
no real traffic patterns
didnt know startup probes were even a thing
zero baseline metrics for what "normal" looks like

basically ai let me move fast but i had no idea what i didnt know. thought i was ready because the yaml looked correct and staging tests passed

took about 2 weeks to get everything stable:

added startup probes (game changer for slow-starting services)
actually load tested batch scenarios
set up prometheus properly, now i have real data
resource limits based on actual usage not guesses
tried a few different tools for generating configs after this mess. cursor is fast but pretty generic. copilot similar. someone mentioned verdent which seems to pick up more context from existing services, but honestly at this point i just validate everything manually regardless of what generates it

costs are down about 25% vs swarm which is nice. still probably over-provisioned in places but at least its stable

lesson learned: ai tools are incredible for velocity but they dont teach you what questions to ask. its like having an intern who codes really fast but never tells you when something might be a bad idea

https://redd.it/1pzn5f9
@r_devops

How would you define proactive AWS Hygiene and Ownership process

We currently lack a standardized way to track ownership, lifespan, and relevance of AWS resources, especially in non-prod accounts. This leads to unused resources, unnecessary cost, and ambiguity during alerts or incidents. We need a proactive process to keep AWS environments clean and accountable.

While I will give some thoughts about this. I want to ask to fellow people, how would you define a process? What steps should be good here? What requirements do you feel we as DevOps need here?

https://redd.it/1pzlj8c
@r_devops

Holiday hack: EKS with your own machines

Hey folks, I’m hacking on a side project over the holidays and would love a sanity check from folks running EKS at scale.

Problem: EKS/EC2 is still a big chunk of my AWS bills even after the “usual” optimizations. I’m exploring a way to reduce EKS costs even further without rewriting everything from scratch without EKS.

Most advice (and what I’ve done before) clusters around:

- Spot + smart autoscaling (Karpenter, consolidation, mixed instance types)
- Rightsizing requests/limits, bin packing, node shapes, and deleting idle workloads
- Graviton/ARM where possible
- Reduce cross-AZ spend (or even go single AZ if you can)
- FinOps visibility (Kubecost, etc.) to find the real culprits (eg, unallocated requests)
- “Kubernetes tax” avoidance: move some workloads to ECS/Fargate when you can

But even after doing all this, EC2 is just… Expensive.

So I'm playing around with a hybrid EKS cluster:

- Keep the managed EKS control plane in AWS
- Run worker nodes on much cheaper compute outside AWS (e.g. bare metal servers on Hetzner)
- Burst to EC2 for spikes using labels/taints + Karpenter on the AWS node pools

AWS now offers “EKS Hybrid Nodes” for this, but the pricing is even more expensive than EC2 itself (why?), so I’m experimenting with a hybrid setup without that managed layer.

Questions for the crowd:

- Would you ever run production workloads on off-AWS worker nodes while keeping EKS control plane in AWS? Why/why not?
- What’s the biggest deal-breaker: networking latency, security boundaries, ops overhead, supportability, something else?

If this resonates, I’m happy to share more details (or a small writeup) once I’ve cleaned it up a bit.

https://redd.it/1pzom7p
@r_devops

I made a CLI game to learn Kubernetes by fixing broken clusters (50 levels, runs locally on kind)

Hey ,


I built this thing called K8sQuest because I was tired of paying for cloud sandboxes and wanted to practice debugging broken clusters.


## What it is


It's basically a game that intentionally breaks things in your local kind cluster and makes you fix them. 50 levels total, going from "why is this pod crashing" to "here's 9 broken things in a production scenario, good luck."


Runs entirely on Docker Desktop with kind. No cloud costs.


## How it works


1. Run ./play.sh - game starts, breaks something in k8s
2. Open another terminal and debug with kubectl
3. Fix it however you want
4. Run validate in the game to check
5. Get a debrief explaining what was wrong and why


The game Has hints, progress tracking, and step-by-step guides if you get stuck.


## What you'll debug


- World 1: CrashLoopBackOff, ImagePullBackOff, pending pods, labels, ports
- World 2: Deployments, HPA, liveness/readiness probes, rollbacks
- World 3: Services, DNS, Ingress, NetworkPolicies
- World 4: PVs, PVCs, StatefulSets, ConfigMaps, Secrets
- World 5: RBAC, SecurityContext, node scheduling, resource quotas


Level 50 is intentionally chaotic - multiple failures at once.


## Install

    git clone https://github.com/Aryan4266/k8squest.git
    cd k8squest
    ./install.sh
    ./play.sh
    

Needs: Docker Desktop, kubectl, kind, python3


## Why I made this


Reading docs didn't really stick for me. I learn better when things are broken and I have to figure out why. This simulates the actual debugging you do in prod, but locally and with hints.


Also has safety guards so you can't accidentally nuke your whole cluster (learned that the hard way).


Feedback welcome. If it helps you learn, cool. If you find bugs or have ideas for more levels, let me know.


GitHub: https://github.com/Aryan4266/k8squest

https://redd.it/1pzr4jh
@r_devops

Docker's hardened images, just Bitnami panic marketing or useful?

Our team's been burned by vendor rug pulls before. Docker drops these hardened images right after Bitnami licensing drama. Feels suspicious.

Limited to Alpine/Debian only, CVE scanning still inconsistent between tools, and suppressed vulns worry me.

Anyone moving prod workloads to these? What's your take?

https://redd.it/1pzrz1p
@r_devops

How do you integrate identity verification into CI/CD without slowing pipelines?

Hey folks, DevOps teams always need identity verification that plugs straight into pipelines without blocking deployments or creating security gaps since most solutions either slow everything down or leave staging environments exposed and we're looking for clean API handoffs delivering reliable signals at real scale.

Does anyone know of what works seamlessly for CI/CD flows?

https://redd.it/1pzuoy1
@r_devops

I got tired of the GitHub runner scare, so I moved my CI/CD to a self-hosted Gitea runner.

With the recent uncertainty around GitHub runner pricing and data privacy, I finally moved my personal projects to a self-hosted Gitea instance running on Docker.

The biggest finding: Gitea Actions is compatible with existing GitHub Actions .yaml files. I didn't have to rewrite my pipelines; I just spun up a local runner container, pointed it to my Gitea instance, and the existing noscripts worked immediately.

It’s now running on my home server (Portainer) with $0 cost, zero cold-starts, and total data privacy.

Full walkthrough of the docker-compose setup and runner registration:https://youtu.be/-tCRlfaOMjM

Is anyone else running Gitea Actions for actual production workloads yet? Curious how it scales.

https://redd.it/1pzvjv0
@r_devops

How do u know a CloudFormation CHANGE won’t break something subtle?

You change one resource.
The stack deploys successfully.
Nothing errors.

But something downstream breaks.

How do you catch that before deploy?
Or do you just accept the risk?

Curious how people think about this in practice.


https://redd.it/1pzu7dl
@r_devops

Does anyone here use rapidapi? Having issues making a payment

I'm trying to add my card to purchase a subnoscription yet my card keeps declining. So then I decide to use klarna as a loan payback option and it gets declined. Then I use affirm for loan payback and the loan was charged but the payment was blocked by rapidapi. The only possible conclusion why this happened is I was making api calls from my laptop while using hotspot so I don't know if rapidapi considered this a proxy and decided to block me from making payments?

https://redd.it/1q00zkz
@r_devops

I built a browser extension for managing multiple AWS accounts

I wanted to share this browser extension I built a few days ago. I built it to solve my own problem while working with different clients’ AWS environments. My password manager was not very helpful, as it struggled to keep credentials organized in one place and quickly became messy.

So I decided to build a solution for myself, and I thought I would share it here in case others are dealing with a similar issue.

The extension is very simple and does the following:

Stores AWS accounts with nicknames and color coding
Displays a colored banner in the AWS console to identify the current account
Supports one click account switching
Provides keyboard shortcuts (Cmd or Ctrl + Shift + 1 to 5) for frequently used accounts
Allows importing accounts from CSV or `~/.aws/config`
Groups accounts by project or client

I have currently published it on the Firefox Store:
https://addons.mozilla.org/en-US/firefox/addon/aws-omniconsole/

The source code is also available on GitHub:
https://github.com/mraza007/aws-omni

https://redd.it/1q02rc4
@r_devops

Is it just me or are some KodKloud course materials AI-generated?

Been using KodeKloud for a while now — love the hands-on labs and sandbox environments, they're genuinely useful for practical learning.

But I've started noticing some of the written course content has all the hallmarks of AI-generated text:

Forced analogies every other paragraph ("think of it like a VIP list...")
Formulaic transitions ("First things first," "Next up," "Time for a test run")
Repeated phrases/typos that suggest no human reviewed it ("violations and violations," "real-world world scenario")
Generic safety disclaimers at the end

Combined with other production issues I've noticed — choppy video edits, inconsistent audio quality, pixelated graphics, cropped screenshots cutting off text — it feels like they're prioritizing quantity over quality.

Anyone else noticing this? For what we pay, I'd expect better QA on the content. The practical stuff is solid but the courseware itself feels rushed.

EDIT: Typo in the noscript, oops, KodeKloud.

https://redd.it/1q04riy
@r_devops

Artifactory nginx replacement

I am hosting Artifactory on EKS with nginx ingress controller for url rewrite. Since nginx ingress controller will be retired, what to use instead? First though is to use ALB because it now supports url rewrite. Any other options?

Please let me know your opinions and experience.

Thank you.

https://redd.it/1q071hx
@r_devops

Stuck on the Java 8 / Spring Boot 2 upgrade. Do you need a "Map" or a "Driver"?

We are currently debating how to handle a massive legacy migration (Java/Spring) that has been postponing for years. The team is paralyzed because nobody knows the blast radius or the exact effort involved.

We are trying to validate what would actually unblock teams in this situation.

The Hypothetical Solution:
Imagine a "Risk Intelligence Service" where you grant read-access to the repo, and you get back a comprehensive Upgrade Strategy Report.
It identifies exactly what breaks, where the test gaps are, and provides a step-by-step migration plan (e.g., "Fix these 3 libs first, then upgrade module X").

My question to Engineering Managers / Tech Leads:
If you had budget ($3k-$10k range) to solve this headache, which option would you actually buy?
- Option A (The Map): "Just give us the deep-dive analysis and the plan. We have the devs, we just need to know exactly what to do so we don't waste weeks on research."
- Option B (The Driver): "I don't want a report. I want you to come in, do the grunt work (refactoring/upgrading), and hand me a clean PR."
- Option C (Status Quo): "We wouldn't pay for either. We just accept the pain and do it manually in-house."

Trying to figure out if the bottleneck is knowledge (risk assessment) or capacity (doing the work).

https://redd.it/1q07zt7
@r_devops

The cognitive overhead of cloud infra choices feels under-discussed


Curious how people here think about this from an ops perspective.

We started on AWS (like most teams), and functionally it does everything we need. That said, once you move past basic usage, the combination of IAM complexity, cost attribution, and compliance-related questions adds a non-trivial amount of cognitive overhead. For context, our requirements are fairly standard: VMs, networking, backups, and some basic automation,,, nothing particularly exotic.

Because we’re EU-focused, I’ve been benchmarking a few non-hyperscaler setups in parallel, mostly as a sanity check to understand tradeoffs rather than as a migration plan. One of the environments I tested was a Swiss-based IaaS (Xelon), primarily to look at API completeness, snapshot semantics, and what day-2 operations actually feel like compared to AWS.

The experience was mixed in predictable ways: fewer abstractions and less surface area, but also a smaller ecosystem and less polish overall. It did, however, make it easier to reason about certain operational behaviors.

Idk what the “right” long-term answer is, but I’m interested in how others approach this in practice: Do you default to hyperscalers until scale demands otherwise, or do you intentionally optimize for simplicity earlier on?


https://redd.it/1q07j39
@r_devops

How did you get into DevOps and what actually mattered early on?

I’m learning DevOps right now and trying to be smart about where I spend my time.

For people already working in DevOps:

- What actually helped you get your first role?

- What did you stress about early on that didn’t really matter later?

- When did you personally feel “ready” for a job versus just learning tools?

One thing I keep thinking about is commands. I understand concepts pretty well, but I don’t always remember exact syntax. In real work, do you mostly rely on memory, or is it normal to lean on docs, old noscripts, and Google as long as you understand what you’re doing?
I’m more interested in real experiences than generic advice. Would love to hear how it was for you.

https://redd.it/1q09vxa
@r_devops