Structured logs' ROI: is itworth it?

I suggested we invest into structured logging at work. We've a microservices platform.
Been getting lots of resistance, ROI unclear, etc.

Currently it takes us up to a whole day to get a clear picture of complex platform related issues.

What's your experience been like?

https://redd.it/1nk2rpm
@r_devops

I've been cleaning up CI/CD breaches for 5 years. Please learn from other people's mistakes.

I'm tired of getting 3am calls from CTOs whose companies are falling apart because of preventable CI/CD security issues.

Last month, I watched a team of incredibly talented engineers cry in a conference room. Their startup - 3 years of work, 40 employees, families depending on them - lost their Series B funding because investors discovered an 8-month-old breach during due diligence.

The heartbreaking part? It started with something we've all done: a developer copied a long-lived AWS key into Jenkins on a Friday afternoon to unblock a release. "Just temporary," the commit message said.

I see this pattern constantly:

We lock down production like Fort Knox
We leave our CI/CD systems wide open
We tell ourselves "we'll fix the security debt next sprint"
We never do

Some hard truths from my experience:

Your CI/CD is 4x more likely to be breached than prod
Average cost when it happens: $4.9M
Average time to discover it: 287 days
Most devastating part: it's usually preventable

I'm not trying to scare you. I'm trying to help you avoid the pain I see teams go through.

Quick health check you can do right now:

# Secrets in Git history?
git log --all --source --grep="password\|key\|secret" | wc -l

# Overprivileged CI runners?
kubectl auth can-i '' '' --as=system:serviceaccount:ci:default

If those commands return anything scary, you're not alone. Every company I've helped started exactly there. I wrote up everything I've learned from 200+ incident responses - the attack patterns, the real costs, and most importantly, how to prevent it. Not trying to sell anything, just tired of seeing good teams get hurt by stuff we can fix. The goal isn't perfect security. It's avoiding the preventable disasters that destroy companies and careers.

Here's the link to the free guide: https://medium.com/@heinancabouly/the-50m-security-hole-in-your-ci-cd-pipeline-and-how-to-fix-it-before-attackers-find-it-9a1308fbb3dc?source=friends\_link&sk=5997988b9e9fbf2c31189f24dcf26e73

Hope this helps someone avoid a 3am call like the ones I get.

https://redd.it/1nk9h1y
@r_devops

OTEL Collector + Tempo: How to handle frontend traces without exposing the collector?

Hey everyone!

I’m working with an environment using OTEL Collector + Tempo. The app has a frontend in Nginx + React and a backend in Node.js. My backend can send traces to the OTEL Collector through the VPC without any issues.

My question is about the frontend: in this case, the traces come from the public IP of the client accessing the app.

Does this mean I have to expose the Collector publicly (e.g., HTTPS + Bearer Token), or is there a way to keep the Collector completely private while still allowing the frontend to send traces?

Current setup:

Using GCP
Frontend and backend are running as Cloud Run services
They send traces to the OTEL Collector running on a Compute Engine instance
The connection goes through a Serverless VPC Access connector

Any insights or best practices would be really appreciated!

https://redd.it/1nkcf43
@r_devops

OpenTelemetry Collector: What It Is, When You Need It, and When You Don’t

Understanding the OpenTelemetry Collector - what it does, how it works, real architecture patterns (with and without it), and how to decide if/when you should deploy one for performance, control, security, and cost efficiency.



https://oneuptime.com/blog/post/2025-09-18-what-is-opentelemetry-collector-and-why-use-one/view



https://redd.it/1nkd6vl
@r_devops

G-Man: Automatically (and securely) inject secrets into any command

I have no clue if anyone will find this useful but I wanted to share anyway!

I created this CLI tool called [G-Man](https://github.com/Dark-Alex-17/gman) whose purpose is to automatically fetch and pass secrets to any command securely from any secret provider backend, while also providing a unified CLI to manage secrets across any provider.

I've found this quite useful if you have applications running in AWS, GCP, etc. that have configuration files that pull from Secrets Manager or some other cloud secret manager. You can use the same secrets locally for development, without needing to manually populate your local environment or configuration files, and can easily switch between environment-specific secrets to start your application.

## What it does
* `gman` lets you manage your secrets in any of the supported secret providers (currently support the 3 major cloud providers and a local encrypted vault if you prefer client-side storage)
* Store secrets once (local encrypted vault or a cloud secret manager)
* Then use `gman` to inject secrets securely into your commands either via environment variables, flags, or auto-injecting into configuration files.
* Can define multiple run profiles per tool so you can easily switch environments, sets of secrets, etc.
* Can switch providers on the fly via the `--provider` flag
* Sports a `--dry-run` flag so you can preview the injected command before running it

## Providers
- Local: encrypted vault (Argon2id + XChaCha20‑Poly1305), optional Git sync.
- AWS Secrets Manager: select profile + region; delete is immediate (force_delete_without_recovery=true).
- GCP Secret Manager: ADC (`gcloud auth application-default login`) or `GOOGLE_APPLICATION_CREDENTIALS`; deleting a secret removes all versions.
- Azure Key Vault: `az login`/DefaultAzureCredential; deleting a secret removes all versions (subject to soft-delete/purge policy).

## CI/CD usage
- Use least‑privileged credentials in CI.
- Fetch or inject during steps without printing values:
- `gman --provider aws get NAME`
- `gman --provider gcp get NAME`
- `gman --provider azure get NAME`
- `gman get NAME` (the default-configured provider you chose)
- File mode can materialize config content temporarily and restore after run.

- Add & get:
- `echo "value" | gman add MY_API_KEY`
- `gman get MY_API_KEY`
- Inject env vars for AWS CLI:
- `gman aws sts get-caller-identity`
- This is more useful when running applications that actually use the AWS SDK and need the AWS config beforehand like Spring Boot projects, for example. But this gives you the idea
- Inject Docker env vars via the `-e` flags automatically
- `gman docker run my/image` injects `-e KEY=VALUE`
- Inject into a set of configuration files based on your run profiles
- `gman docker compose up`
- Automatically injects secrets into the configured files, and removes them from the file when the command ends

## Install
- `cargo install gman` (macOS/Linux/Windows).
- `brew install Dark-Alex-17/managarr/gman` (macOS/Linux).
- One-line bash/powershell install:
- `bash` (Linux/MacOS): `curl -fsSL https://raw.githubusercontent.com/Dark-Alex-17/gman/main/install.sh | bash`
- `powershell` (Linux/MacOS/Windows): `powershell -NoProfile -ExecutionPolicy Bypass -Command "iwr -useb https://raw.githubusercontent.com/Dark-Alex-17/gman/main/noscripts/install_gman.ps1 | iex"`
- Or grab binaries from the [releases page](https://github.com/Dark-Alex-17/gman/releases/latest).

### Links
- GitHub: https://github.com/Dark-Alex-17/gman

And to preemptively answer some questions about this thing:

* I'm building a much larger, separate application in Rust that has an `mcp.json` file that looks like Claude Desktop, and I didn't want to have to require my users put things like their GitHub tokens in plaintext in the file to configure their MCP servers. So I wanted a Rust-native way of storing and encrypting/decrypting and injecting values into the `mcp.json` file and I couldn't find another library

that did exactly what I wanted; i.e. one that supported environment variable, flag, and file injection into any command, and supported many different secret manager backends (AWS Secrets Manager, local encrypted vault, etc). So I built this as a dependency for that larger project.
* I also built it for fun. Rust is the language I've learned that requires the most practice, and I've only built 6 enterprise applications in Rust and 7 personal projects, but I still feel like there's a TON for me to learn.


So I also just built it for fun :) If no one uses it, that's fine! Fun project for me regardless and more Rust practice to internalize more and learn more about how the language works!


https://redd.it/1nkf33p
@r_devops

Ridiculous pay rate

I just came here to say I had a recruiter reach out and they were saying 24/hr pay rate for a DevOps engineer position.

What the hell is that pay, thankful I am already at a great FT job but that is absurd for DevOps work or really anything in IT.

And if was just a scam to steal my information they could have went higher on the pay rate to make me sending me resume over more enticing.

https://redd.it/1nkgnax
@r_devops

Im currently transitioning from help desk to devops at my job, how can I do the best I can? I was told it will be “a lot” and I’m already lost in the code


So we purchased puppet enterprise to help automate the configuration management of our servers. I was apart of the general puppet training but not involved in the configuration management side of training. There were two parts.

Now I was given this job and I have to automate the installation of all our security software and also our CIS benchmarks and there is some work done but there’s a ton left to do.

I’m not going to lie it feels like a daunting task and it was told to me that it was, and I’m not even “fully” in the role, I still have to “split time” which imo makes it even harder.

Right now I’m using my time at work to self study almost the whole day.

I kind of like the fact that I could make a job out of this here but there’s just so much code and different branches and I’m sitting here looking at some of the code and it overwhelms me how much I don’t know and what does this attribute do and why is the number here zero. It’s a lot and I do wish I had some work sponsored training cause I wasn’t invited for the second week of training.

https://redd.it/1nkj7m7
@r_devops

MLOps

Hi! Any MLOps engineers in the sub?

Looking to chat and know a bit about the tech stack you are working on. Please DM if you have a little extra time for a curious bobblehead in your day! Thanks!

https://redd.it/1nkjv2c
@r_devops

dumpall — CLI to aggregate project files into Markdown (great for CI/CD & debugging)

I built `dumpall`, a small CLI that aggregates project files into a single, clean Markdown doc.

Originally made for AI prompts, but it turned out pretty handy for DevOps workflows too.

🔧 DevOps uses:

\- Include a unified code snapshot in build artifacts

\- Generate Markdown dumps for debugging or audits

\- Pipe structured code into CI/CD noscripts or automation

\- Keep local context (no uploading code to 3rd-party tools)

✨ Features:

\- AI-ready Markdown output (fenced code blocks)

\- Smart exclusions (skip node_modules, .git, etc.)

\- --clip flag to copy dumps straight to clipboard

\- Pipe-friendly, plays nice in noscripts

Example:

npx dumpall . -e node_modules -e .git --no-progress > all_code.md

Repo 👉 https://github.com/ThisIsntMyId/dumpall

Docs/demo 👉 https://dumpall.pages.dev/

https://redd.it/1nkkroi
@r_devops

Who else is losing their mind with Bitnami?

Bitnami’s sunsetting images has been brutal.

I keep hitting endless ImagePullBackOff loops while re-deploying Postgres and Redis across prod, staging, and dev.

After hours of firefighting I’ve switched to CloudNativePG for Postgres and kept Bitnami legacy for Redis just to stay afloat.

Anyone found smoother migration paths or solid long-term replacements?

https://redd.it/1nknrco
@r_devops

I have no idea how you guys do it

Long time lurker, not even working in DevOps (but rather IT, doing a mix of sysadmin/support). But man, some of the shit you guys can do and need to know is mind blowing. DevOps is definitely my target in the next 5-8 years, just need to get exposed to it and keep working my way up.

So many names for so many applications/tools, hundreds of cloud services etc. What an absolute shitshow of a field! Yet still interesting to me. Reading through the posts all the time has my head spinning. Most of it might as well be a different language. Keep up the grind!

https://redd.it/1nkrbng
@r_devops

Testing a new rate-limiting service – feedback welcome

Hey all,

I’m building a project called Rately. It’s a rate-limiting service that runs on Cloudflare Workers (so at the edge, close to your clients).

The idea is simple: instead of only limiting by IP, you can set rules based on your own data — things like:

* URL params (/users/:id/posts → limit per user ID)
* Query params (?api\_key=123 → limit per API key)
* Headers (X-Org-ID, Authorization, etc.)

**Example:**

Say your API has an endpoint **/user/42/posts**. With Rately you can tell it: “apply a limit of 100 requests/min per **userId**”.

So user 42 and user 99 each get their own bucket automatically. No custom nginx or middleware needed.

It has two working modes:

1. **Proxy mode** – you point your API domain (CNAME) to Rately. Requests come in, Rately enforces your limits, then forwards to your origin. Easiest drop-in.

​

Client ---> Rately (enforce limits) ---> Origin API

1. **Control plane mode** – you keep running your own API as usual, but your code or middleware can call Rately’s API to ask “is this request allowed?” before handling it. Gives you more flexibility without routing all traffic through Rately.

​

Client ---> Your API ---> Rately /check (allow/deny) ---> Your API logic



I’m looking for a few developers with APIs who want to test it out. I’ll help with setup 🙏.

https://redd.it/1nkwyyw
@r_devops

Is going from plain APIs to agents always worth the extra complexity?

I have been building systems by wiring APIs together with HTTP endpoints and webhooks. It’s predictable, debuggable, and I know exactly where the logic lives. Now I keep seeing agent frameworks that promise to sit on top of APIs, handle decision logic, and “figure things out” on the fly.

For people who have gone beyond the demos THE ACTUAL PRODUCTION!!, what real problems did agents solve that you could not handle with direct API orchestration?? Was it worth the extra complexity in terms of debugging, reliability, and cost?



https://redd.it/1nkzb60
@r_devops

Kafka (Strimzi) and Topic Operator seems like a bad idea to me?

I’ve never done anything with kafka and need to set it up in kubernetes, so I naturally looked for an operator. It seems that strimzi is the way to go tho I don’t agree with their topics operator approach. To me it seems topics should be a concern of the application and not defined dependent on the infra. Developing in docker locally, now I have to define topics there. Or if a team needs a new topic suddenly they have to change infra components.

I googled and didn’t find a discussion about that. It seems teams are generally fine with that topic operator approach. Can you enlighten me why it should not be part of the application configurations Itself and rather part of the infrastructure yamls we use for kubernetes?

https://redd.it/1nl0ct1
@r_devops

Feedback on tools used to scan vuln NPM packages

Anyone else used the google tool to scan for vuln NPM packages any recommendations or is there a better way ? https://cyberdesserts.com/npm-scanner

https://redd.it/1nl14jy
@r_devops

How do you hire a DevOps contractor who’s way more technical than you?

I manage a mature SaaS product and I’ve ended up as the accidental DevOps person after replacing an offshore team that didn’t really have the role covered. I’m technical, but not at the level I need for where we’re headed, so it’s time to bring in someone who genuinely knows the space. Ideally on a contract to tackle the big projects , then hopefully keep them on part-time afterward for ongoing support.

This isn’t a job post (I’ll share that to r/devopsjobs soon), but I’m looking for advice from people here who’ve been on either side of this. If you want to DM with thoughts or recommendations, my inbox is open.

The main projects are things like finishing our Jenkins to ArgoCD migration, stabilizing the dev environment, upgrading Kubernetes and keycloak, fixing Terraform drift, and tightening up security by swapping bastion for SSM. Down the line we’ll need a coordinated Postgres upgrade and help implementing something like Flyway. I have a rough roadmap with phases, but I also want the person I hire to shape it once they’ve seen the guts.

Where I could use your help is figuring out the right approach.

First, what’s a sane way to interview and evaluate someone who’s supposed to outclass you? I'm thinking of one focused technical conversation to hear their high-level plan for the Jenkins migration, and then maybe a short, paid working session in a non-prod environment to see how they think. Is that a good signal, or is there a better way to assess real-world skills?

Second, where do you actually find great freelance talent these days beyond the job subreddits? Are places like Upwork, boutique agencies or certain communities worth cutting through the noise for?

Third, what's a safe but effective way to handle day one access? My instinct is to start with more limited permissions and expand as we build trust, but I don’t want to slow them down. How do you prefer to start when you join a new project?

Finally, I have a roadmap, but I want the person I hire to have ownership and help shape it. I want someone who’ll call out gaps in my plan, not just follow checklists. For the contractors here, what are the green flags that tell you a client will actually listen to your expertise, and what are the red flags that tell you to run?


Budget isn’t FAANG, but it’s sane. I care more about working with someone who’s proactive, communicates clearly, and leaves things tidier than they found them. If you’re interested, keep an eye out for the official post, but I’d really appreciate any advice on process, places to look, or things I might not know enough to ask yet. Thanks.



https://redd.it/1nl0bvw
@r_devops

Can splunk alerts be sent to another app via post request?

I noticed that people are able to send stack trace data in a splunk alerts which makes me wonder if these alerts can send a post request to a custom app for tracking purposes

https://redd.it/1nl7254
@r_devops

Tell me about your experience looking for job offers

Has anyone here found effective ways to increase their chances… like reaching out to strangers or companies on LinkedIn, or posting content there?

Did any of these work for you? Or can you share some tips from your own experience with job hunting?

https://redd.it/1nl9aw4
@r_devops

Skill Vs Money

So I have been a person who believe if we ace in our skill or niche( myn is devops) Money is automatically generated. But situations around me make me feel like this the shittiest thing I have ever done. Frnds who have graduated with me have been earning 20k -30 K inr per month. I have stucked to learning devops and doing an internship of 5k inr per month. Iam i foolish here or I need some patience to reach my devops dream role. What I mean by devops dream goal is that basic payofor frehser Or even some higher with acc to my skill

https://redd.it/1nlacf0
@r_devops

How common it is to be a DevOps engineer without (good) monitoring experience?

Hello community!

I am wondering how common it is for not having or having very little experience with monitoring for DevOps Engineers?

At the beginning of my career, when I worked as a system administrator, monitoring was a must-have skill because there was no segregation of duties (it was before Prometheus/Grafana and other fancy things were invented).


But since I switched to DevOps, I have worked very little to no with monitoring, because most often it was SRE's area of responsibility.


And now the consequences are that is it a blocker for most of the companies from hiring me, even with my 10+ YOE and 7+ years in DevOps.

https://redd.it/1nl91wh
@r_devops