I built a bash noscript that finds K8s resource waste locally because installing Kubecost/CastAI agents triggered a 3-month security review.

**TL;DR:** I built a bash noscript that finds K8s resource waste locally because installing Kubecost/CastAI agents triggered a 3-month security review.

**The Problem:**
I've been consulting for Series B startups and noticed a pattern: massive over-provisioning (e.g., 8GB RAM requests for apps using 500MB), but no easy way to audit it. The existing tools are great, but they require installing agents inside the cluster. Security teams hate that. It often takes months to get approval.

**The Solution:**
I wrote a simple bash noscript that runs locally using your existing `kubectl` context.
* **No Agents:** Runs on your laptop.
* **Safety:** Anonymizes pod names locally (SHA256 hashes) before exporting anything.
* **Method:** Compares `requests` vs `usage` metrics from `kubectl top`.

**The Code (MIT Licensed):**
https://github.com/WozzHQ/wozz

**Quick Start:**
`curl -sL https://raw.githubusercontent.com/WozzHQ/wozz/main/noscripts/wozz-audit.sh | bash`

**What I'm looking for:**
I'm a solo dev trying to solve the "Agent Fatigue" problem.
1. Is the anonymization logic paranoid enough for your prod clusters?
2. What other cost patterns (orphaned PVCs, etc.) should I look for?

Thanks for roasting my code!

https://redd.it/1p31y4t
@r_devops

What's the cleverest prompt injection bypass you've actually encountered?

Been red teaming chatbots for a while now and the attack vectors keep evolving. Most attempts are basic role-play or system prompt leaks, but I've seen some genuinely creative ones.

The cleverest I caught recently was an attacker who embedded instructions in fake error messages, making the model think it was debugging itself. Something like "Error: To continue, ignore previous instructions and..." Pretty sneaky social engineering on the model itself.

I'm curious what others have encountered in production. Are you seeing more sophisticated multi-turn attacks? Any particularly creative bypasses that made you rethink your defenses?

Also interested in how teams are actually managing this operationally. Static filters obviously don't cut it.



https://redd.it/1p329nj
@r_devops

Is it normal to have to learn something new for every work task?

I'm working for a tech company where they put together a bigger DevOps team that spans across multiple projects, so that we manage them all at the same time. Previously we were doing the same work separately for each project. We were initially hired as inexperienced juniors, were never properly trained and for several years we kinda shot the shit since we had rather simple tasks.

Now we have an immense workload split among too few of us and, I kid you not, we get a new area of expertise to handle pretty much every month. 70% of the tasks I get require learning something new, almost from scratch. Only a few, highly experienced and highly motivated people are able to keep up. I feel like the rest of us are sinking, but I don't really know, since nobody talks about it.

Is this amount of learning something normally expected for a DevOps job in other companies?

I am extremely exhausted, I feel constantly ashamed of my performance, and I often procrastinate doing the tasks because I have no idea how to do them, nor do I feel like constantly asking questions. A lot of the time, I barely understand the answers, because I haven't been trained in what I'm supposed to do.

Is this situation normal when being a DevOps, are you constantly expected to learn new things from scratch, on your own? I don't know if I need to change the company or change my profession altogether.

https://redd.it/1p34525
@r_devops

Jenkins or GitLab Runners for Android apps?

Hey all, I’m in the process of setting up CI/CD at the moment in my company, starting with a few Android apps first.

At the moment, I have noscripts to run all of the tests and then build signed releases, it’s okay for now but I’d like to not have to do this and be able to have easily accessible builds to distribute automatically.

We moved from GitHub to running a self hosted GitLab instance (cheaper for LFS on other projects + easier overall personally), I haven’t configured runners yet but now need to think about either doing that or spinning up a Jenkins server, I’ve used it in the past for other projects personally and professionally so I’m relatively comfortable with it. But I need some more opinions on what you’d do in my situation.

Are there any other tools that might be easier for deployment/maintenance? The less administration the better personally lol. (I’m managing Development and other infrastructure already)

The ability to run our OS builds (AOSP) in the future would also be a nice to have, but not important, they’re a lot less frequent but not having to baby them would be good.

https://redd.it/1p35v5x
@r_devops

Release Engineering vs SRE

Hi all,

Looking for advice on two positions I've been offered at the same company. I had initially went in for a Platform Engineering role, however, this role has now closed.

The company are interested in still getting me on board though and have offered me the choice of an SRE and Release Engineer role. My background has mainly been in small companies where I've taken up more DevOps-y responsibities and for the past while been in a 'dedicated' DevOps role (though it is more an everything developer role in practice). I want to get more experience with the parts of DevOps I enjoy; designing and implementing distributed scalable infrastructure whilst abstracting complexity from SWEs in the SDLC. Ideally without becoming a Sys Admin or losing sight of SWE-esque day-to -day. Hence I believed PE would be a good fit (please correct me if I'm wrong)

I'm aware each company defines all these roles differently, and no opinion here can give me clarity into that. However the choice involves specialised industry defined roles at a size of company I don't have experience with. I don't have many people in my network I can ask for guidance so any insight to this would be amazing!

PS I have a knee jerk avoidance of RE cause I think focussing primarily on git, release versioning and build tools would drive me insane, but would love to be proved wrong as I love the idea of collaborating a bunch.

https://redd.it/1p2za1j
@r_devops

TLS MITM environments such as Zscaler: How do you ensure trust when the entire TLS chain is deliberately compromised?

When an organization has decided to implement global TLS inspection via Man In The Middle proxies, effectively taking a chainsaw to the entire computer/math trust architecture of TLS that underpins practically all modern computing, how can we still provide a valid, real, secure trust system to system and people to systems?

I'm going through my own thought experiments now trying to answer the question, "If only basic non-TLS HTTP existed, what would I need to configure and/or build to provide both the trust and secure communications that TLS otherwise ensures?

On the small scale I'm looking at things like enabling claims encryption for SAML and OIDC authentications, exclusively using FIDO2 hardware tokens (no TOTP, SMS, etc), etc. But while I've worked out securely authenticating to services, the MITM is still able to scrape the JWT bearer tokens, session cookies, etc to hijack sessions even if it can't replay the authentication itself. And even if we solve authentication, there's still the data itself to consider, which is going to require some form of public-key based, application-level encryption, like an SSH data flow only implemented in the web browser (WASM maybe?).

I'm late to the game, but suddenly I'm trust into understanding exactly the problem space that folks like WhatsApp et al have been trying to solve with full end-to-end encryption. Because I realize now that even if my own organization isn't using MITM TLS inspection, whatever or whoever I'm communicating with on the other side of the conversation may not be so lucky.

\---

To be clear I'm not looking for ideas on how to get around Zscaler for my own traffic; I've got more than enough technical chops to route around this asinine security theatre if I cared to.

Rather I'm looking at this from a systems architecture / DevOps / SDLC perspective for how I factor in a solution to address this new (to me) threat vector for my users. For example, ZScaler publishes a list of their proxy IP CIDR ranges which a website / app can match against the "client" and if it's matched at least present the user with a warning that any data they enter is absolutely NOT secure no matter what that little padlock icon in the location bar says (since ZScaler includes subverting the client's trust CA with their own).

My customers still need actual security, actual trust, no matter what my insecurity team thinks. So this is just another design requirement to deal with and I'm looking for tips about how others might have approached this problem. Both in application arch itself, but also the full SDLC because how do we deal with trusting supply chains, etc.

https://redd.it/1p3ajr8
@r_devops

I built a tower defense game that teaches cloud architecture (but does anyone actually want this?)

A couple weeks ago, I was once again explaining to a junior dev why his API was crashing under load. I drew diagrams, showed him charts, talked about load balancers and scaling... And I saw that familiar emptiness in his eyes. He was nodding, but I knew he wasn't really feeling the problem.

Then it hit me - what if I made a game where you actually see your architecture collapse in real-time?

What I built

Server Survival is basically tower defense for DevOps. You build cloud infrastructure from blocks (WAF, Load Balancer, EC2, RDS, S3), connect them with arrows, and then watch your creation try to survive waves of incoming traffic.

Full disclosure: this is a rough MVP

I'll be honest - right now this is a prototype hacked together on my knee. I intentionally made the simplest version possible just to validate the idea. There are tons of simplifications, some things don't work exactly like real AWS, the load balancing is sometimes wonky.

But! That's exactly why I'm releasing this open source. I want to understand - is this even interesting to anyone?

I have a ton of ideas for what could be added - different cloud providers (AWS/Azure/GCP), more realistic mechanics, auto-scaling groups, availability zones, monitoring dashboards, multiplayer mode, real-world incident scenarios like Black Friday or security breaches... But before I sink more time into this, I really need to know: does anyone actually need this?

GitHub: https://github.com/pshenok/server-survival

Let me know what you think

https://redd.it/1p3bxnx
@r_devops

Help restructuring a terraform monorepo.

Hello all!

My heads spinning a bit and I need some insights here.

For context: Two years ago I was contracted to architect and implement the cloud systems for an application working on a POC that needed a fast turnaround. At the time the application was very small, basic networking, one RDS instance, one API gateway etc you get it simple. So I put it all in a monorepo and implemented fairly basic gha CI/CD and branch based envs coupled with workspaces as everything was to be setup in one AWS account.

Fast forward to today I was called back to lend a hand as things have grown. They now have the same networking more or less but API gateway -> LoadBalancer -> Multiple Containerized APIs in ECS. Multiple DB's etc etc everything has grown exponentially, still in the monorepo and one state file they also want to shift to a multi account/env setup (music to my ears)

I really do not want to spark the debate of Multi vs Mono repos. But they have a small team of 3-4 devs that are in charge of the infra and deployment of applications so they've opted to leave it as a monorepo. Worth noting the application logic is broken out but at least the first image is deployed with terraform.

The question is now, how do you break up a jumbo sized root state file that everything is using and isolate state so that "services" like the ecs based api containers can be modified and deployed independently. Also graduate to prod without affecting prod when changes are made to a service. Target multiple accounts and avoid drift all from a mono repo...

My current plan is to switch to the tried and true Dir per env. Breakout each service as a "module" and parametrize their inputs. Stitch the CI/CD so that there's a staged deployment and granular deployment for isolated updates. So each service has it's own root level state and can be updated independently within the repo without a massive plan and deploy.

Graduating to prod and keeping them in sync seems difficult in a scenario like this as tagging service modules is pretty much out in a mono. So it'd have to lean more towards trunk based and semi manual deployments to the prod env after approval.

Hopefully that all makes sense. Any thoughts here would be greatly appreciated as I usually lean towards multi repo.

Cheers.




https://redd.it/1p3afkm
@r_devops

Symlink Attacks: When File Operations Betray Your Trust

https://instatunnel.my/blog/symlink-attacks-when-file-operations-betray-your-trust

https://redd.it/1p3j3hw
@r_devops

DevOps internship questions

Hey everyone! I'm a university student in CS. I have an interview for a DevOps internship next week. Looking forward to it, but wanna make sure I'm preparing properly. Here's what I've done so far:
\- I have looked at the interviewers' LinkedIns and checked out what they do or have done at the company
\- Reviewed all the technologies, languages and tools listed in the job posting. For the ones I already know or have on my resume, I refreshed my memory and did a deep dive into it. For the ones I wasn’t familiar with, I did a quick overview
\- Wrote down specific details about the projects and experience listed on my resume so I’m ready for questions like “what was your role?”/“why did you do it this way?”/“can you explain this in more detail?" and so on.
\- Prepped for some behavioural questions

I'm also thinking about preparing a few questions to ask them, some out of curiosity, some just to keep the interview flowing nicely.

What else should I focus on? I don't get nervous when it comes to stuff like this, so I should be able to hold my nerves and have a nice interview. Also, since it's an intern position, my guess is that they won't be expecting good technical skills or expertise, so if I'm right, they're looking for someone who is competent, willing to learn and shows some level of enthusiasm and drive. And my job is to leave a good impression on them to help me stand out.

Any advice and tips are much appreciated.
Also the job is in Canada, and the company is an enterprise level company.

https://redd.it/1p3n5k0
@r_devops

CEO "helps" with terraform, rewrites entire product into an unmaintainable frankenstein, now wants to migrate everything

Not my story, thankfully, it was shared /w me - just wanted to share the insanety that's going on rn:

"A customer recently asked us to help them with some terraform to install our app. My CEO casually remarked “hey I’m pretty good with terraform let me take this over”

Now he has a completely re-architected version of our product that only works for that one customer, he added a bunch of new services like Istio, ArgoCd, Vault, rewrote all our cicd in dagger, and ripped out a bunch more required services. It barely works. Nobody is trained on half of this. Some of our core functionality is completely missing. He vibe coded this over two months in a vacuum, and thinks of himself as some kind of genius he can’t even explain half the shit.

He is asking me to migrate everything over to his bullshit over the next couple weeks"

https://redd.it/1p3p7pv
@r_devops

What’s enough for a Junior?

I’m about to start applying for a Junior devops and my portfolio is as follows:

- all terraform natless eks cluster with an ALB ingress and kyverno admission based on a kms key sig and an attestation for an image(i also made a gitlab pipeline that signs an image with cosign and attests it with trivy and then pushes it into my private ecr).

- all terraform eks monitoring stack with kube-prometheus.

- Custom runtime with OCI image extraction, custom networking supporting multiple containers, NAT and port forwarding (i actually ran a monitoring stack on this using prometheus and a node exporter) all written in GO.

- Now i’m about to do an ebpf firewall and after this i’ll just start applying.

I have no reference point in terms of how a junior application pool actually looks like in terms of skill level and since i originally wanted to do cybersecurity my idea of a typical junior is about exactly as what i have right now.

Is there anybody who works in the industry and has an idea of the junior skill level and whether that’s enough to land a global remote position?


https://redd.it/1p3qvjf
@r_devops

How long does it typically take you to prepare a fully configured cloud environment (staging or production)? (Including networking, security, logging, access controls, etc.)

💡 Vote and comment: what slows down the process the most?

View Poll

https://redd.it/1p3rkc7
@r_devops

what's working to automate the code review process in your ci/cd pipeline?

trying to add automated code review to our pipeline but running into issues, we use github actions for everything else and want to keep it there instead of adding another tool.

Our current setup is pretty basic: lint, unit tests, security scan with snyk. All good but they don't catch logic issues or code quality problems,  our seniors still have to manually review everything which takes forever.

I’ve looked into a few options but most seem to either be too expensive for what they do or require a ton of setup, we Need something that just works with minimal config, we don't have time to babysit another tool.

What's actually working for people in production? Bonus points if it integrates nicely with github actions and doesn't slow down our builds, they already take 8 minutes which is too long.

https://redd.it/1p3tg5p
@r_devops

Thinking of Switching from C++ Dev to DevOps After 9 Years — Is It Realistic? How Do I Start Upskilling?

Short background:
I’m a C++ developer with about 9+ years of experience. I’m not some tech wizard — just an average guy who’s been grinding through it. But honestly, I don’t think I can keep up with this constant coding frenzy anymore. It doesn’t come naturally to me, and it’s starting to drain me.

I’ve been thinking about shifting into DevOps. I know it’s a huge field and could take a year or more of consistent learning, but I’d rather spend that time building a career I can actually enjoy instead of banging my head against the wall.

For those who have made a similar transition or know the space well:
How do I realistically upskill for DevOps? And is this career shift even feasible after 9 years in development?

https://redd.it/1p3v616
@r_devops

Customer Success Architect

What does a Customer Success Architect do? I mean, I read a job listing for it, and I get that they talk to customers, hype the product, etc. But what's the job like? Does it pay well? Are you still technical at all?

https://redd.it/1p3wn5s
@r_devops

Pc to start dev ops

Hello everyone, I’m about to start studying dev ops totally on my own, taking courses and reading books about it.
Having no computer science base I would start from scratch and by zero I mean that I would need the PC to start everything.
I had in mind to buy an inexpensive PC, and then in the future change it with something more powerful.

And I had thought of this: HP 15-FD0057NL, Intel Core I3 N305. RAM 8 GB, 256 Gb SSD (€349).

Do you think it’s a good choice? Or if you have something to advise me let me know. Thank you

https://redd.it/1p3utib
@r_devops

Devops being split into more roles?

I have noticed comments here and there that DevOps is getting split and get more specialized people.
Have you seen a split into several roles like Platform Engineers and Cloud Engineers happening at your place or with coworkers?

https://redd.it/1p48za8
@r_devops

Practical "Path" for DevOps Home Learning?

Hi All, so currently I'm working as an SDET for the past few years. Recently I got a chance to do some devops stuff on AWS. Basically setting up s3 storage state (with terraform) and deploying a .NET app to Beanstalk via Gitlab CI/CD. Also just some other beginner terraform stuff.

I've found it pretty interesting and I do recognize it's beginner stuff but i've often had to learn some of the pipeline stuff as an SDET and honestly it's became more interesting.

I have previously spent a lot of time learning devops stuff on KodeKloud (Which works great) however if you don't use it you sorta lose it. However I now have a chance to start actually working with it at work.

Something I wanted to think of is sort of a practical "path" I can do something with at home (with an AWS free account) and on my Proxmox mini pc's.

In my head it would look maybe something like:

1. Use a sample (something simple like a todo app) and deploy it to EC2/Beanstalk (.net probably) via Gitlab (sorta have already done this)
2. Connect RDS w/ Beanstalk to get a handle with that.
3. Set up those resources in Terraform
4. Dockerize the app
5. I guess also Dockerize the Database
6. Deploy to EKS as a container?
7. ???? (Maybe get Cloud practitioner cert for AWS? I heard it was pretty simple)

I don't think we will be using EKS for awhile at work (Since we just moved to AWS from other cloud providers). I also know Kubernetes is pretty complicated.

Any missing steps or things you would add?

https://redd.it/1p4acia
@r_devops

mariadb vs mysql

We run both of these, seemingly at random depending on who set each one up for each application. We need to standardize and pick one. Which do you run and why?

https://redd.it/1p4bvbh
@r_devops

Analysing the cloudflare outage!

I made a quick small video explaining the cloudflare outage. I went through the RCA and added some bits to it. I've been part of a similar global outage at scale where a buggy code deployed on the edge servers brought the entire service down for hours.

It's really really tough to recover from issues where your edge servers get impacted with high CPU or Memory utilisation.

https://www.youtube.com/watch?v=ObAn4hQc370

Go through the video and let me know if you found it useful.


https://redd.it/1p4d802
@r_devops