Biometric Breach Vector: The reverse search tool that bypassed our basic IAM and access controls.

I run a small DevOps team, and we were doing a casual security audit on our personal digital footprints. We used a powerful biometric search engine to test the integrity of our own security posture.

The audit started with faceseek. I uploaded an ancient, low-res picture of one of our developers that was on a completely separate, non-work-related platform. The goal was to see if the tool could map that face to anything related to our company.

The terrifying discovery: It mapped that single photo to a non-face PFP used on a personal Gitlab repo that contained a legacy, exposed API key (the developer thought the repo was locked down and unindexed). The biometrics served as the unexpected bridge between personal life and professional exposure.

This is a serious security vector. It proves that the weakest link in our Identity and Access Management (IAM) isn't the password or the 2FA token; it's the permanently indexed biometric hash of our team members.

Question for r/devops: How are you integrating biometric threat awareness into your security pipeline? Is anyone using tooling in their CI/CD to audit their own employees' publicly indexable biometric data to preemptively find these kinds of cross-platform security vulnerabilities? We need to treat this as a systemic risk.

https://redd.it/1nw9fm5
@r_devops

Devops/sre engineer with 10 years of experience how to get into quant firms?

Hi all

I’ve been working as an SRE/DevOps engineer for 10 years (CI/CD, infra automation, deployments, monitoring etc). Lately I’ve been curious about roles in quant/prop trading firms.

For someone with my background, should I focus on: • Linux internals & low-level system performance? • Programming (C++/Python) for low-latency systems? • Or just keep building infra/data pipelines?

Also, what roles make sense for me — quant dev, trading infra engineer, low-latency SRE?

Anyone here actually doing SRE/infra at a quant shop — would love to hear what skills really matter and how different it is from regular tech companies.

Thanks!

https://redd.it/1nwaj4v
@r_devops

Would you say that devops has a higher element of customer service / people interaction than pure infrastructure work?

I see a common pattern among some pretty brilliant IT engineers I’ve worked with in the past who have transitioned to devops: they thrive on project work where they’re heads down and just build, but when it comes to explaining things to a stakeholder, reaching across silos to get tasks done or even just being a team player and helping someone figure shit out - they are unable to do the basic required things in these scenarios. We have had to let go two engineers at my company because we’re a contract shop and when we have clients who need us to lead the way, these guys just fall over. I don’t know how to suss personality and culture here because not every scenario is cut and dried “you need to be a people person” shit. Much of it is just common sense about owning your shit. Any advice?

https://redd.it/1nw8551
@r_devops

Development philosophies of error-handling for sysadmin-type tasks?

I don't know exactly how to search for what I'm looking for, so figured I'd ask here:

I have this codebase I've inherited that is basically one big Ansible project (sensibly broken up into roles, don't worry) that does a bunch of validations before running dnf update on a group of servers and reporting the results.

As you might expect there's a number of places during the process where we want it to stop and report back, like if you don't own the systems in question or if you're trying to run the procedure outside of your scheduled change window or if the servers can't be reached for some reason, etc.

As a sysadmin first and developer second, I've always kind of struggled with how to develop procedural tasks such as this in a way that they can fail gracefully at a given point without doing lots of "do task, if it fails report this specific error, otherwise do next task, if it fails this way do this error run otherwise do that one otherwise do next task" and so on. Are there any good resources on best practices / design patterns for this kind of work, preferably ones that a non-CompSci doofus can understand? They don't have to be Ansible-specific, I'm looking more for basic theory, if such a thing exists.

https://redd.it/1nwaxcq
@r_devops

Super micromanaging boss

So I joined this new org almost a year ago and our boss is really starting to get on my nerves woth micromanaging.

Our daily meetings are 1h long with 4ppl in team, coz he either talks about w/e stuff or asks about every detail of everything you do.

We have pretty strong team with competent ppl that know how to accomplish our goals. But he keeps on butting in in every small detail even if he does not understand the tech behind solutions we want to implement or argues with us we are wrong coz he heard or this or that.

Every single time he does that - later it turns out we were right and we did everything best we could and all needs are met.

We constantly have to do some refinements and knowledge sharing for w/e reason. I feel like he lost purpose and is trying to prove he is still useful for his boss, while we can pretty much replace him in any work he previously did and most likely better.

Worst part is when that micromanaging and will to „prove himself” makes us look bad when we discuss stuff with platform teams. Ppl that worked with F500s and we discuss stuff like ROI, complex architectures or new tools. Manager butts in, doesnt even understand what they talk about or refuses to accept some ideas (mostly good ones) and makes us look stupid. Imagine DTrump talking to ppl with 30 years of experience „ypu dont know shit”..

Its embarassing for us. I know he has good intentions but I dont know how to tell him to „chill out” on those actions, coz he has rly smart ppl on the board in team that know wth we are doing and we all worked at Senior to Staff level engineering roles.

Its just exhausing to sit through 2h meetings that could be 30m or discuss with architects have all planned and someone butts in to change stuff we already agreed on.

Anyone had such boss and could give some tips on how to politely fine-tune him ?

https://redd.it/1nwfu17
@r_devops

How to Handle a Career Gap While Applying for DevOps/SRE Roles?

Hey everyone,

I’d really appreciate some advice from folks who have gone through this.

I have about 4 years of professional experience as a DevOps/Infrastructure Engineer where I worked with AWS, Terraform, Ansible, Jenkins, Docker, Kubernetes, and monitoring tools like Prometheus/Grafana. Most of my work was around automating deployments, setting up CI/CD, and supporting production infrastructure.

Due to personal circumstances, I had to take a step back from full-time work for about 2 years. During that time, I focused on recovery, picked up certifications (AWS Cloud Practitioner, GitOps), and did some personal lab projects (Terraform IaC, Jenkins pipelines, K8s deployments). I also enrolled in a Master’s program to strengthen my technical foundation.

Now I’m actively applying for **DevOps/SRE roles in the U.S.** but I’m hitting a wall — lots of applications, very few responses. I suspect the unexplained gap is a red flag for recruiters.

My questions are:

* How should I present this gap on my resume/LinkedIn?
* Is it worth creating a “Career Break & Professional Development” section to show I was still learning/building projects?
* For those who’ve been in a similar spot, how did you explain the gap during interviews?

Any advice or real-world examples would be hugely helpful. I want to make sure this gap doesn’t overshadow the skills I bring to the table.

Thanks in advance!

https://redd.it/1nwe937
@r_devops

Is DevOps freelancing realistic, or should I pivot ?

I like cloud and recently passed the AWS SAA. I’m currently learning Terraform, and since I’m from a third-world country, freelancing is my main plan once I reach a solid skill level.

My concern is whether DevOps freelancing is realistic, especially for someone with no prior experience. Most DevOps work seems to require long-term trust and deep integration with company systems, which doesn’t always fit the “freelancer” model. That makes me wonder if I should instead invest my time in backend or full-stack development, which clearly have stronger freelancing markets.

So I’d love to hear from those with experience:

Is DevOps freelancing actually a viable path?

Or would it be smarter to focus on something else?

https://redd.it/1nwj3b7
@r_devops

Startup, Leadership wants to bring in people to all live in a mansion for a week to do intense collab when we work WFH, your thoughts?

Leadership wants to bring in core devs, devops, software dev leadership, and support, to have long collab sessions for a week in a large mansion essentially. They will provide all the accommodations, including lodging, tickets, food that the support (not tech support, more like people like project managers) will cook.

Would you embrace? Would you push back on it? Decline it?

https://redd.it/1nwgkuq
@r_devops

Thoughts on Aiven vs cloud vendor services?

Aiven offers managed versions of open-source tools (Postgres, MySQL, Kafka, Redis, ClickHouse, etc.) across multiple clouds. They pitch it as avoiding vendor lock-in while still getting fully managed infra. Anyone here using it in production? Worth it vs the native AWS/GCP/Azure databases?

https://redd.it/1nwjvpj
@r_devops

Curious how's the market right now in North America.

Who's looking for a DevOps position? How do you feel the market is right now? About remote, seems like companies want back to office even more.

https://redd.it/1nworvh
@r_devops

Are DevOps services actually worth it for small teams?

Hey all, not usually in this sub (I’m a product person, not ops), but I figured you’d know best.

Our small SaaS team is drowning in infra... suff like broken deploys, weird billing spikes and no one who really wants to own “ops.” I keep seeing DevOps services advertised as a way to fix this. They claim they’ll handle pipelines, monitoring, scaling, etc. so your devs can stay focused on product.

On paper it sounds amazing, but I’ve never talked to anyone who’s actually used them... guide me please

https://redd.it/1nwiu55
@r_devops

Dev team & operations team but no devops team.

My company are in the process of replacing all of our saas with in-house apps.

I work in the operations team and have been operating as a sort of translator between the devs and the rest of IT

I’d like to move into devops and I’m wondering the best way to position myself to do this given the opportunity.

We operate exclusively in azure.

I’m not sure any of the work iv done so far is what you would call real devops work, things like setting up SSO, recommending we setup defender for cloud so the security team has visibility into any vulnerabilities inside the code, configuring service principals for the applications to access different parts of our environment, iv recommended moving to azure devops and want to moving into more devops related work, so my question is, what can I do at this point to provide value and maybe gain some experience with working in devops?


https://redd.it/1nwsqnx
@r_devops

Need a devops partner

Hey I am a beginner in devops i need a partner to study devops u should be active atleast 1hr a day we can learn and grow together if anyone interested dm me

https://redd.it/1nwwgbu
@r_devops

Team culture, whinging

I’m in a team that has a culture of whinging, mostly other parts of the business being incompetent (which aren’t actually too bad and pay the bills), also external parties, but also other team members’ work, when those team member aren’t present. Additionally, a focus on technical aspects as opposed to business outcomes.

Have you ever seen such culture turn around and how?

https://redd.it/1nwz0uq
@r_devops

What I learned from hiring a software development partner for our startup

Our startup recently partnered with a custom software development team to build a scalable app. At first, I wasn’t sure how to pick the right vendor there are so many options out there.

What really made a difference was working with a team that communicated clearly, had proven experience across industries, and remained flexible as our requirements changed during development.

The structured approach saved us a lot of time and headaches, and it was a relief to see the project move smoothly. I’d love to hear how others choose development partners for their projects.

https://redd.it/1nwysm5
@r_devops

Git CI/CD Integration Testing

I’d like to get some opinions and advice on how to set up the basic structure of a test pipeline and repository structure in gitlab.

At my company, we’re starting a new project that integrates multiple components. Some of these components already exist and just provide Docker images. But several other components are being developed from scratch specifically for this project. My task is to write a test pipeline that brings all of these components together and runs tests.

My initial idea was to create a separate repository for each new component so we can version them properly. Then, have one dedicated repository for integration, which would only be responsible for deploying the different component images (for example, via Kubernetes) and running integration tests.

However, a colleague who has been with the company for many years suggested a different approach: a single project repository, with each component in its own folder, and one big pipeline that builds everything from source, runs unit tests and coverage checks for each component, and then also runs the integration tests.

Personally, I think it makes much more sense to separate the components. The downside I see, though, is that some components might need dependencies from others just to test themselves properly.

So my questions are:

What’s considered best practice here?

How do you usually structure something like this in a clean and maintainable way?

What are the pros and cons of each approach?


I’m open to hearing different strategies and experiences.

https://redd.it/1nx0zly
@r_devops

How to deal with a coworker who is almost never available and is un-fireable?

In our department, we essentially handle the entire stack from native app development to deploying our product into the cloud. I work with 3 platform engineers with the infrastructure architecture and deployment as well. One of the senior guys who’s the most knowledgeable one is barely ever around and does not do his portion of the work to get new features in for qa to test and deploy into a couple of the staging environments. So I and another engineer have to pick up his slack and get it done before the next release deadline.

That senior engineer in question is the son of the CTO of the company. So telling management about him goes nowhere. We’ve tried. I know we should leave, but job market seems pretty bad even for seniors. With that being said, I still love working here. I’m just trying to get some advice on what to do here with him in particular.

https://redd.it/1nx39k6
@r_devops

Octopus Deploy Pricing & Use Cases.. Feedback…

For those of you running Octopus Deploy day-to-day in the enterprise.. How are you finding it? Specifically:

Are you finding the value in audit trails, approvals, and environment management worth the premium?

If you’re using it for Kubernetes or multi-cloud, how does it compare to alternatives like ArgoCD or Flux… Would love to hear from other teams (especially mid-sized orgs or regulated industries) on how you’re using it and what’s been working.

https://redd.it/1nx2bv5
@r_devops

Need help setting up Clickhouse DC DR Setup

# What I already have

* Two Kubernetes clusters: **DC** and **DR**.
* Each cluster runs ClickHouse via the **Altinity Operator** using `ClickHouseInstallation` (CHI). Example names: `prod-dc` and `prod-dr`.
* Each cluster currently runs its own **ClickHouse Keeper** ensemble (StatefulSet + Service): e.g. `chk-clickhouse-keeper-dc` in DC and `chk-clickhouse-keeper-dr` in DR.
* ClickHouse server pods in DC point to the DC keeper; ClickHouse pods in DR point to the DR keeper.
* Networking: there is *flat networking* between clusters and FQDNs resolve (e.g. `pod.clickhouse.svc.cluster.local`), DNS resolution has been verified.

Tables use `ReplicatedMergeTree` engine with the usual ZooKeeper/keeper paths, e.g.:

CREATE TABLE db.table_local (
id UInt64,
ts DateTime,
...
) ENGINE = ReplicatedMergeTree('/clickhouse/tables/{shard}/table', '{replica}')
PARTITION BY toYYYYMM(ts)
ORDER BY (id);

# My goal / Question

I want **real-time replication** of data between DC and DR — i.e., writes in DC should be replicated to DR replicas with minimal replication lag and without manual sync steps. How can I achieve this with Altinity Operator + ClickHouse Keeper? Specifically:

* If separate keepers are kept in each cluster, how do I make `ReplicatedMergeTree` replicas in both clusters use the same replication / coordination store?
* Any recommended Altinity CHI config patterns, DNS / service setups, or example CRDs for a DC–DR setup that others use in production?

Any help is really appreciated. Thanking in advance.

https://redd.it/1nx4ssh
@r_devops

devops/SRE tasks with MCP server?

hey folks, I am a maintainer at SigNoz. We recently open sourced our MCP server ( https://github.com/signoz/signoz-mcp-server )

We have got some community members using it but wanted to get feedback from the community here also on what type of devops/SRE tasks do you expect to do with your MCP server of o11y tools? what are you doing currently

We have basic tools like getting metrics, logs, dashboards, alerts etc. Looking for feedback from the community before deciding on what tools to build next.

Also, if anyone uses AI SRE tools like Resolve AI, Traversal - would love to learn what you use it for? What things you expect to do using MCP servers and what you would do with a more full fledged AI SRE product?

https://redd.it/1nx36sa
@r_devops

Vpc and Networking

Practicing devops projects we can do it but I have a doubt or confusion. Suppose I need to setup ips and network how should I do i have seen many videos but i dont understand this concept of subnets /32 /16 and ip hashing and how can i allocate custome network for a projects and vpcs any resources.W

Tl:Dr
I need resources to learn about cloud and networking vpc, subnets from scratch

https://redd.it/1nx0xms
@r_devops