Shai Hulud Launches Second Supply-Chain Attack (2025-11-24)

Came across this (quite frightening) information. Some infected npm packages are executing malicious code to steal credentials and other secrets on developer machines, then publish them publicly on Github. Right now, thousands of new repo are being created to leak secrets. If you're using node in your pipeline, you should have a look in this.

Link to the article: https://www.aikido.dev/blog/shai-hulud-strikes-again-hitting-zapier-ensdomains (not affiliated in any way with them)

https://redd.it/1p5ih1j
@r_devops

We surveyed 200 Platform Engineers at KubeCon


Disclaimer: I’m the ceo of Port (no promotional stuff)

During KubeCon Atlanta a few weeks ago, we ran a small survey at our booth (~200 responses) to get a pulse on what Platform Engineering teams are actually dealing with day-to-day. Figured this subreddit might find some of the patterns interesting.

https://info.getport.io/hubfs/State%20of%20KubeCon%20Atlanta%202025.pdf?hstc=17958374.820a64313bb6ed5fb70cd5e6b36d95ac.1760895304604.1763984449811.1763987990522.6&hssc=17958374.17.1763987990522&hsfp=189584027

https://redd.it/1p5i5zz
@r_devops

My laptop died and locked me out of my homelab. It was the best thing that ever happened to my project.

Hello r/devops,

This is my second time posting on this sub after this post (link) where I shared my project for automating an RKE2 cluster on Proxmox with Terraform and Ansible. I got some great feedback, and since then, I've integrated HashiCorp Vault. It's been a journey, and I wanted to share what I learned.

Initially, I just thought having an automated K8s cluster was cool. But I soon realized I needed different environments (dev, staging, prod) for testing, verification, and learning. This forced me into a bad habit: copying .env files, pasting them into temp folders, and managing a mess of variables. After a while, I got it working but was tired of it. The whole idea was automation, and the manual steps to set up the automation were defeating the purpose.

Then, my laptop died a week ago (don't ask my why, it just didn't boot anymore, something related to TPM hardware changes)

And with it, I lost everything: all my environment variables, the only SSH key I'd authorized on my VMs, and my kubeconfig file. I was completely locked out of my own cluster. I had to manually regenerate the cloud-init files, swap the SSH keys on the VM disks, and fetch all the configs again.

This was the breaking point. I decided to build something more robust that would solve both the "dead laptop" problem and the manual copy/paste problem.

My solution was HashiCorp Vault + GitHub Actions.

At first, I was just using Vault as a glorified password manager, a central place to store secrets. I was still manually copying from Vault and pasting into .env files. I realized I was being "kinda dumb" until I found the Vault CLI and learned what it could really do. That's when I got the idea: run the entire Terraform+Ansible workflow in GitHub Actions.

This opened a huge rabbit hole, and I learned a ton about JWT/OIDC authentication. Here's what my new pipeline looks like:

1. GitHub Actions Auth: I started by (badly) using the Vault root token. I quickly learned I could have GHA authenticate to Vault using OIDC. The runner gets a short-lived JWT from GitHub, presents it to Vault, and Vault verifies it. No static Vault tokens in my GHA repo. I just need a separate, one-time Terraform project to configure Vault to trust GitHub's OIDC provider.
2. Dynamic SSH Keys: Instead of baking my static admin SSH key into cloud-init, I now configure my VMs to trust my Vault's SSH CA public key. When a GHA job runs, it:
Generates a brand new, fresh SSH keypair for that job.
Asks Vault (using its OIDC token) to sign the new public key.
Receives a short-lived SSH certificate back.
Uses that certificate to run Ansible. When the job is done, the key and cert are destroyed and are useless.
3. kubectl Auth: I applied the same logic to kubectl. I found out Vault can also be an OIDC provider. I no longer have to ssh into the control plane to fetch the admin config. I just use the kubelogin plugin. It pops open a browser, I log into Vault, and kubectl gets a short-lived OIDC token. My K8s API server (which I configured to trust Vault) maps that token to an RBAC role (admin, developer, or viewer) and grants me the right permissions.
4. In-Cluster Secrets: Finally, external-secrets-operator. It authenticates to Vault using its own K8s ServiceAccount JWT (just like the GHA runner), pulls secrets, and creates/syncs native K8s Secret objects. My pods don't even know Vault exists.

With all of that, now if I want to add a node, I just change a JSON file that defines my VMs, commit it, and open a PR. GitHub Actions runs terraform plan and posts the output as a comment. If I like it, I merge.

A new pipeline kicks off, fetches all secrets from Vault, applies the Terraform changes, and then runs Ansible (using a dynamic SSH cert) to bootstrap K8s. The cluster is fully configured with all my

apps, RBAC, and OIDC auth, all from a single git push.

Here's the project if you want to see the code: https://github.com/phuchoang2603/kubernetes-proxmox

https://redd.it/1p5kq4y
@r_devops

Domain monitoring tool - looking for feedback/advice!

Hi guys!

For the past few months now I've been working on a little tool that routinely monitors the WHOIS/RDAP data, DNS records and the SSL status of domains. If any of this changes, you'll get a little email immediately letting you know.

I would really appreciate feedback on any aspect of the project, whether that's the landing page, something inside the app itself and such.

It doesn't have any ghastly AI features (nor does it need it!) and has only been worked on by myself so I'm pretty eager for feedback.

You can find the project here: https://domainwarden.app

Thank you so much for any feedback! I do appreciate it. :)

https://redd.it/1p5q7y3
@r_devops

Small but useful DevOps project: CPU usage monitor in Bash (alerts + logs)

Exploring small automation ideas. Built a Bash-based CPU monitor with thresholds + logging.

Tutorial: https://youtu.be/nVU1JIWGnmI

source code : https://github.com/Abhilashchauhan1994/bash\_noscripts/blob/main/cpu\_usage.sh

Please review this and provide me any suggestion that will make this better.

https://redd.it/1p5jrkl
@r_devops

Upcoming interview, what to expect?

First ever interview for a DevOps (Associate) role, want to transition from SQA/automation.

What to expect in this weird time we are living?





https://redd.it/1p60srm
@r_devops

Which metrics are most reliable?

Recently i noticed there is always a difference between ec2 instance utilization( cpu,memory) metrics and th e one provided by new relic agent.

I want to keep only one of them in new relic and make alerts, decisions based on that only.

Any insights on which are more reliable?

https://redd.it/1p656av
@r_devops

Words of new CEO - „Why hire seniors when single junior with AI can do work of seniors”

Its silly how the wave has turned in IT because of AI.

Beside offshoring to cheaper countries, AI seems to be the new way to push ppl to do more and more with less staff on the board.

CEO said he literally sees zero reasons to hire for senior roles now. GPT seems to be on a level good enough to replace all of them. AI agents replaced all of our less senior testers, support call centre is replaced by AI call center, senior devs fired and replaced with 1/10 of juniors with AI at hand.

Funny thing is company did not slow down, rather got faster releases, # of issues decreased and overall customer satisfaction went up.

Sad days to be someone continuing IT journey without AI :/

On the other hand - amazing news for Senior ppl in less expensive countries.

“This looks like the times when whole floors of switchboard operators were replaced by a few technicians maintaining automated systems.”

https://redd.it/1p669oq
@r_devops

Need realtime ci cd issues

Hi, i know ci cd pipelines and how to set it up, but i need to know what kind of realtime issues companies go through in the ci cd implementation. It can be caching issue or long running pipelines or any thing. I need someone to explain it very well so i can replicate the same thing in my homelab and explore it more.

I would request people to throw their insights over this one.

https://redd.it/1p65gf3
@r_devops

Why do project-management refugees think a weekend AWS course makes them engineers?

Project-management refugees wandering into tech like they can just cosplay engineering for a weekend is beyond insulting. Years grinding through real systems, debugging at 3 a.m., tearing down and rebuilding your own understanding of how machines behave – all of that gets flattened by someone who thinks an AWS bootcamp slapped on top of zero technical substrate makes them your peer. They drain the fun out of the craft, flatten the discipline, and then act confused when they faceplant the moment anything non-clickops appears. The arrogance isn’t just annoying; it’s a contamination of the field by people who never respected it in the first place.

https://redd.it/1p68qzc
@r_devops

anyone else feel like ai tools are either quiet helpers or complete chaos?

​

i’ve been messing around with a bunch of these ai coding tools lately, and honestly some of them feel like they’re trying way too hard. a few of the agent-style ones start touching files i didn’t even bring up. cool demos, scary in real projects.

the ones that actually stick for me are the calmer ones that stay in lane like aider when i need clean multi-file edits, windsurf or cursor when i want a simple plan instead of a magic trick, and cosine whenever i’m lost in a big repo and need to follow the logic across a bunch of files. i’ve tried tabnine and continue dev too, but they’re hit or miss depending on the day.

curious if anyone else is going through this, what tools ended up becoming part of your routine, and which ones did you quietly uninstall because they made more mess than progress?

https://redd.it/1p69elx
@r_devops

testing platforms with actual AI (not just marketing fluff) do they exist?

Every vendor pitch i sit through now mentions "AI powered" something but when you dig into it, it's just basic automation with maybe a chatgpt integration slapped on top.

I'm looking for a test automation platform that actually uses AI in meaningful ways, like understanding user intent, adapting to ui changes without breaking, generating test scenarios from app exploration, that kind of stuff. Not just keyword matching or basic ml.

We're running a pretty standard ci/cd pipeline with github actions, about 300 tests across ui and api. Current setup is playwright which works fine but maintenance is brutal. Every release we spend half a day fixing tests that broke due to ui changes.

Has anyone actually used an ai test automation platform that delivered on the promises? Or is this all just next gen marketing speak for the same old stuff?

Genuinely curious because if the tech is there i want to try it, but i'm not interested in another "revolutionary" tool that's just selenium with extra steps.

https://redd.it/1p6a3zp
@r_devops

Failing Every Devops Interview need help

Hey everyone, I’m going through a tough phase and could really use some advice from this community.

I was laid off on 10th October 2025, and since then I’ve been actively interviewing for DevOps roles. It’s been a little over 2 months now, but I keep failing interviews. Some rounds feel like they go well, yet I still end up rejected, and I’m honestly not sure where I’m falling short.

I’ve been practicing Jenkins, Git, Linux, AWS basics, Terraform, CI/CD pipelines, and doing hands-on labs, but I feel like something is still missing, either in my preparation or in the way I communicate during interviews.

If anyone here has been through something similar or is currently working in DevOps, I’d really appreciate any guidance. What should I focus on the most?

How do you approach DevOps interviews?

Any good resources/labs/mock interview groups to improve?

What helped you break into your first DevOps job?

Any help or honest feedback would mean a lot. Thanks in advance.

https://redd.it/1p6bwsk
@r_devops

AI-Powered Attack Automation: When Machine Learning Writes the Exploit Code 🤖

https://instatunnel.my/blog/ai-powered-attack-automation-when-machine-learning-writes-the-exploit-code

https://redd.it/1p6b944
@r_devops

Are Azure DevOps pipelines hard to use or is it just me?

Hello all. This one is a bit of a discussion/rant but I wanted to get some opinions on the state of Azure DevOps Pipelines versus the competitors. Have been banging my head against it just trying to do simple stuff such as having it work with combinations of static and dynamic inputs and I feel like I'm finding 1,000 ways to do it wrong and zero ways to get it working.

I think I understand the difference between compile-time and runtime parameters, but it seems incredibly difficult to find the right magic incantation to get runtime parameters to evaluate correctly, especially when using lots and lots of templates (I'm currently working at a place with an existing pipeline setup that I'm trying to amend and there are several layers of nested templates to deal with).

I've been working either directly in DevOps teams or adjacent to them for well over a decade now and have worked with TeamCity, Octopus, Jenkins and GitLab pipelines and I have never had so many headaches as I've had with Azure DevOps pipelines. Is this a common experience?

If it's not, and it's actually just down to my own lack of understanding (very possible) then can anyone recommend some good training resources?

https://redd.it/1p6e02r
@r_devops

Tools like Graphite and Coderabbit any good?

I’ve been seeing people talk about Graphite and CodeRabbit on twitter and in some YT breakdowns, but it’s hard to tell what’s hype and what’s actually useful when you’re still new to the skill. 

I’m a junior backend dev and my biggest struggle is keeping PRs readable and making sure I’m not missing stuff when reviewing others’ work.

Looking for tool recommendations pls 🙏

https://redd.it/1p6ebdi
@r_devops

Aws lambda deployments. Sam vs aws deploy

In production what should be used

Sam or aws deploy noscripts ?

Since Sam is doing lot of management. For startups is it OK to use Sam in the ci cd ?


https://redd.it/1p6fk6z
@r_devops

Asked a fresher to shut down an EC2 server… he shut down his own laptop instead

So this happened at work and I’m still laughing about it.

I told a fresher on our team to shut down an EC2 instance before he left for the day so we could save on AWS costs.

Next morning, I log in and see the server is still running.
I ask him, “Hey, did you actually shut it down?”
He nods confidently, “Yes sir, I did. I ran the shutdown command in the terminal.”

Now I’m confused, so I ask him to show me what he did.

He opens his laptop, types the shutdown command in his local terminal, hits enter… and his laptop instantly goes black. Just shuts off.
He looks at me like, “See? It works.”

https://redd.it/1p6jlfv
@r_devops

I Need Scaling YOLOv11/OpenCV warehouse analytics to ~1000 sites – edge vs centralized?

I am currently working on a computer vision analytics project. Now its the time for deployment.

This project is used fro operational analytics inside the warehouse.

The stacks i am used are opencv and yolo v11

Each warehouse gonna have minimum of 3 cctv camera.


I want to know:
should i consider the centralised server to process images realtime or edge computing.


what is your opinon and suggestion?
if anybody worked on this similar could you pls help me how you actually did it.

Thanks in advance

https://redd.it/1p6k2b7
@r_devops

Production and at scale Kubernetes learning advice

Currently, manage 2 non prod clusters and 1 prod cluster that are dedicated to my team only. Have pretty decent setup, push based gitops, helm, cluster autoscalers, HPA, fluentd logging to cloudwatch, prometheus/grafana/thanos stack for observability.

Looking for some jobs that require K8s in production and at scale.

What do I have to learn and do to be in that level?



https://redd.it/1p6izwe
@r_devops