Got screwed on MLOps project payment - $11k paid out of $18k, need advice

Hey folks,
So I'm in some BS situation right now and honestly don't know if I'm being paranoid or actually getting shafted.
Started a contract gig ~4 months back. Client needed their ML stack unfucked - they had data scientists pushing models to prod with literally zero pipeline, no monitoring, nothing. My job was:
Spin up proper MLOps infra on AWS (SageMaker + custom containers), Get their LLM stuff production-ready (they were running GPT wrappers with no fallbacks lmao),
Build out some agentic workflows for their support chatbot, Set up proper observability - Prometheus/Grafana, cost tracking, the works
Lock down their IAM because it was a dumpster fire
Rate was $18k split across 3 milestones - $6k each for planning, implementation, and deployment/handoff.
Here's where it gets weird:
First $6k hit my account fine. Second milestone, I shipped the entire ML pipeline, containerized everything, got their models deploying automatically. Invoice them, get... $2.5k. Ask WTF, they say "we're reviewing costs quarterly now" and me be like Ok!.
I didn't go aggressive because tbh I had like $9k buffer saved up and my project pipeline was dry. Figured I would finish strong, they would see the value, make it right.
Fast forward - I'm basically done. Their LLM agents are handling 60% of tickets autonomously, inference costs down 40%, everything's monitored. I even wrote runbooks for their junior devs. Invoice the last $6k.
Two weeks of ghosting, then they schedule a call. Offer me $3.2k as "completion bonus" bringing total to like $11.7k.
Their reasoning: "timeline extended beyond scope and we had infrastructure costs we didn't anticipate."
Bro. The timeline extended because THEY kept pivoting on which LLM provider to use (we went OpenAI -> Anthropic -> back to OpenAI). The infra costs went DOWN because of my work. I literally showed them the FinOps dashboards.
I'm sitting here like...? Do I just take the L and move on? My savings are getting thin and I don't have another gig yet, so part of me is like "just take the $3k and don't make enemies."
But another part is pissed because the work is legitimately good and in production making them money.
What would you do & I should do?
Anyone been in something similar? I had some rascals earlier who didn't paid me , Ignored my reachouts after the contract work was done , They is a special place in hell for these guyzz ..

https://redd.it/1q7iv92
@r_devops

Former Cloudflare SRE building a tool to keep a live picture of what’s actually running. Looking for honest feedback

Hey everyone, I’m Kenneth, founder of OpsCompanion.

I spent years as a Senior SRE at Cloudflare. One thing that became painfully clear is that most outages, security issues, and compliance fire drills don’t come from a lack of tools. They come from missing context. People don’t know what’s running, how things connect, or what changed recently, especially once systems sprawl across clouds, repos, and teams.

That’s why I’m building OpsCompanion.

OpsCompanion helps engineers:

* Keep a live, visual picture of what’s running and how things connect
* Answer “what changed?” without digging through five tools, Slack threads, or the god-awful state of documentation most teams are dealing with today
* Preserve operational context so the next on-call isn’t starting from zero

This isn’t about adding more logs or alerts, or slapping AI onto existing platforms and calling it AGI. It’s about giving engineers the same mental model I used to carry in my head, but shared and kept up to date.

We’ve opened up free access for a small, curated group of engineers who work close to production. If it’s useful, great. If not, I genuinely want to know why and what would make it useful.

Free access here:
[https://opscompanion.ai/](https://opscompanion.ai/)

Everyone who signs up during this early window will get an life time deal once we that part up(I will reach out via email), the gratitude of myself, and to drive the road map of our product

I’ll be in the comments. Happy to answer questions, hear skepticism, get roasted a bit, or talk about what it actually takes to be an SRE or DevOps engineer in 2026.

https://redd.it/1q7xn5c
@r_devops

DevOps Engineer: Which certifications are worth doing for the future?

Hi everyone,

I’m a DevOps Engineer with a few years of experience and I’m looking to invest in certifications that will actually help me in the long run.

Which certifications would you recommend that are relevant now and also future proof.

Cloud, Kubernetes, security, SRE or anything else?

Would love to hear from people who’ve seen real career benefits from certs. Thanks!

https://redd.it/1q7hz9b
@r_devops

AI is making CI/CD the slowest part of DevOps , how are teams handling this?

Something I’m noticing more and more:



Code gets written incredibly fast now (AI + copilot + agents), but CI pipelines haven’t caught up.

A 10–15 min CI run used to be fine when coding took hours.

Now it’s the dominant part of the feedback loop.



Symptoms I’m seeing:

\- PRs pile up waiting on CI

\- Teams reduce test coverage just to move faster

\- More issues slip into prod because “CI was green but shallow”



Curious how others are handling this:

\- selective test execution?

\- better change-impact detection?

\- just throwing more compute at CI?

\- or accepting slower feedback?



Would love concrete approaches that actually worked.

https://redd.it/1q817dy
@r_devops

suggestion needed: How do you manage hundreds of minimal container images in an air gaped environment?

We operate in isolated networks where artifacts can’t be pulled from the internet. Updating minimal images while keeping security current is challenging. What strategies do you use to automate vulnerability updates safely?



https://redd.it/1q81va8
@r_devops

Anyone else finding observability for LLM workloads is a completely different beast?

We just started deploying some AI heavy services and honestly I feel like I'm learning monitoring all over again. Traditional metrics like CPU and memory barely tell you anything useful when your inference times are all over the place and token usage is spiking randomly. The unpredictability is killing me. One minute everything looks fine, next minute latency is through the roof because some user decided to send a novel length prompt. And dont even get me started on trying to correlate model performance with actual infrastructure costs. Its like playing whack a mole but the moles are invisible. Been spending the last few weeks trying to build out a proper observability framework for this stuff and realizing most of what I learned about traditional APM only gets you halfway there. You need visibility into token throughput, embedding latencies, model versioning, and somehow tie all that back to user experience metrics. Curious how everyone else is handling observability for their AI/ML infrastructure? What metrics are you actually finding useful vs what turned out to be noise?



https://redd.it/1q83pi8
@r_devops

How do you handle small webhook payload changes during local testing?

When testing webhooks locally, I often hit the same issue.

If one field in the payload needs to change, the usual options are to retrigger the external event or dig through a dashboard to resend something close enough. It works, but it’s slow and a bit clumsy.

Curious how others deal with this.
Do you have a workflow that makes small payload tweaks easier, or is this just how it is?

https://redd.it/1q82815
@r_devops

Where the Cloud Ecosystem is Heading in 2026: Top 5 Predictions

Wrote a blog about where I feel the cloud ecosystem is heading in 2026. Here's a summary of the blog:


1. The AI Vibe Check

The "just add AI" honeymoon phase is ending. At KubeCon London, sessions were packed based on buzzwords alone. By Atlanta, the mood shifted to skepticism. In 2026, organizations will stop chasing the hype wagon and start demanding proof of ROI, better security audits, and a clear plan for Day 2 operations before integrating AI features.

2. Kubernetes Moves to the "Back Seat"

Kubernetes is no longer the star of the show and is more like the engine under the hood. We’re seeing a massive surge in adoption of projects like Crossplane, kro, and Kratix. Platform teams are moving away from forcing developers to touch K8s primitives, instead favoring abstractions and self-service APIs. The goal for 2026: developer experience (DevEx) that hides the complexity of the cluster.

3. The Death of Local Dev Environments

Local environments can’t keep up with modern cloud complexity or the speed of AI coding agents. The "slow feedback loop" (waiting for CI/Staging) is the new bottleneck. 2026 will be the year of production-like cloud dev environments.

4. The "Specific" AI SRE

We aren't at the "autopilot cluster" stage yet. While tools like K8sGPT and kagent are gaining ground, we won't see general-purpose AI managing entire clusters. Instead, 2026 will favor task-specific agents with limited scope and strict permissions. It’s about empowering SREs, not replacing them.

5. Open Source Fatigue

Organizations are hitting a saturation point with overlapping CNCF projects. In 2026, the "cool factor" won't be enough to drive adoption. Teams are becoming hyper-selective, prioritizing long-term maintainability, community health, and clear roadmaps over whatever is currently trending on GitHub.

https://redd.it/1q83ft5
@r_devops

Got lucky with a Junior SRE role — how do I not waste it?

Honestly, I got lucky.


I recently moved from Helpdesk to a Junior SRE/DevOps role at a startup.

I have very little actual DevOps background, but I want to use this opportunity to build a serious career.

Since I'm the only SRE, I have full access to everything. I want to use this "sandbox" to fast-track to a solid level in 2 years. If you were me, how would you prioritize?

What paid off the most early on? (Terraform, CI/CD, networking, observability, etc.)
What real-world implementation taught you the most about how systems fit together?
Which tools/trends are noise early on?
How did you keep improving without burning out?

Note: I'm currently a CS student considering dropping out to focus 100% on this role. Is the practical experience worth more than the paper in the current market?

Thanks!

https://redd.it/1q85aco
@r_devops

Can I try DevOps, or am I missing something I should master first?

I need a professional opinion from someone in DevOps. I’ve had a turbulent and fragmented professional path, and I’d like to know if there’s anyone who can guide me and tell me from which point I should start over.

My story is a bit long:

I graduated in Computer Engineering, a 5-year program (2019–2023), with half of it (2020–2023) during the pandemic. That period came with difficulties in networking and a lack of hands-on practice due to the remote format via cellphone (I didn’t have enough income to buy my own equipment).

With a lot of difficulty, I managed to get 2 internships.

I interned at a construction company where the focus was industrial and residential automation. Naively, everything they taught me was how to request product quotations. I tried to learn by observing others, but it wasn’t enough and had no real connection to computing.

Despite that, in 3 months I managed to save enough money to build my first PC, and then I spent 4 months applying for other internship positions until I got a support role.

The support position was at a small company with 12 employees, focused on assisting elderly people, and my supervisor was a systems analyst.

In this new internship, I studied NDG Linux Essentials, CCNA1, Python, computer assembly and maintenance, Windows Server (application and network management with Active Directory), Flask, JavaScript, Docker, Docker Compose, Git, GitHub, and Nginx.

My supervisor left, and I was hired by the company to work in IT, but officially under the role of administrative assistant. I accepted because I needed the money, but today I believe it was a mistake.

Being the only IT person, I was very busy managing and maintaining everything, without knowing if I was doing things the right way.

What was supposed to be 3 months while I looked for another job ended up becoming 2 years, and now, in 2026, I feel obsolete and out of the job market (I don’t even have a LinkedIn profile).

Today, I have about 90% of my time free because I automated all my tasks.

After researching a lot, I’m thinking about starting a DevOps journey, but I’d like to know if it makes sense to try DevOps without having a developer portfolio and without even knowing how to create a website beyond a basic Flask app or WordPress.

I have few certifications, and unfortunately, from engineering I only have the degree noscript, since the course itself went through all that turbulence.

At the moment, I’m a “do-everything” person, with a bit of everything and not really good at anything. What should I do to build a solid foundation and a strong specialization?

https://redd.it/1q86v3m
@r_devops

Why incidents and failures matter more than perfect uptime

Over time, you encounter various challenges. Deployments fail, systems break, and some decisions don't work as expected. This is often how real experience is built.

When people are hired, the focus is usually on successful systems, uptime, and automation. Sometimes, though, you're asked about incidents, outages, or things that went wrong. And those moments often show real experience.

What kind of difficulties or mistakes did you face while working with production systems, and what did they teach you?

https://redd.it/1q88ybp
@r_devops

The SEO Ecosystem in 2026: Why Rankings Are Now Built, Not Chased

SEO in 2026 isn’t about chasing algorithms or isolated hacks anymore. It’s an interconnected ecosystem where multiple forces work together to determine search visibility and long-term performance. What you see on the surface, rankings and traffic, is the result of deeper signals operating in sync.

Search visibility today is shaped by AI-driven algorithms that constantly interpret user behavior and intent. Search engines are getting better at understanding why users search, not just what they type. That’s why search behavior analysis has become a core strategy, not an afterthought.

Content quality has also evolved. It’s no longer about volume or keywords, but about depth, clarity, topical authority, and usefulness across the entire journey. Pages that genuinely solve problems and demonstrate expertise naturally earn credibility and trust, reinforced by strong brand signals and authoritative backlinks.

Community input is another growing influence. Mentions, discussions, shared experiences, and real-world engagement help search engines validate relevance beyond the website itself. Supporting all of this are solid technical foundations that allow efficient crawling, indexing, and performance.

Finally, user signals act as continuous feedback loops. Engagement, satisfaction, and interaction confirm whether a page truly deserves its position. In 2026, SEO success comes from aligning all these elements into one cohesive strategy, built for sustainability, not shortcuts.

\#SEO2026 #SEOEcosystem #FutureOfSearch #AIAndSEO #ContentQuality #SearchVisibility #TechnicalSEO #DigitalStrategy

https://redd.it/1q89kei
@r_devops

Is building a full centralized observability system (Prometheus + Grafana + Loki + network/DB/security monitoring) realistically a Junior-level task if doing it independently?

Hi r/devops,

I’m a recent grad (2025) with \~1.5 years equivalent experience (strong internship at a cloud provider + personal projects). My background:

• Deployed Prometheus + Grafana for monitoring 50+ nodes (reduced incident response \~20%)

• Set up ELK/Fluent Bit + Kibana alerting with webhooks

• Built K8s clusters (kubeadm), Docker pipelines, Terraform, Jenkins CI/CD

• Basic network troubleshooting from campus IT helpdesk

Now I’m trying to build a full centralized monitoring/observability system for a pharmaceutical company (traditional pharma enterprise, \~1,500–2,000 employees, multiple factories, strong distribution network, listed on stock exchange). The scope includes:

1. Metrics collection (CPU/RAM/disk/network I/O) via Prometheus exporters

2. Full logs centralization (syslog, Windows Event Log, auth.log, app logs) with Loki/Promtail or similar

3. Network device monitoring (switches/routers/firewalls: SNMP traps, bandwidth per interface, packet loss, top talkers – Cisco/Palo Alto/etc.)

4. Database monitoring (MySQL/PostgreSQL/SQL Server: IOPS, query time, blocking/deadlock, replication)

5. Application monitoring (.NET/Java: response time, heap/GC, threads)

6. Security/anomaly detection (failed logins, unauthorized access)

7. Real-time dashboards, alerting (threshold + trend-based, multi-channel: email/Slack/Telegram), RCA with timeline correlation

I’m confident I can handle the metrics part (Prometheus + exporters) and basic logs (Loki/ELK), but the rest (SNMP/NetFlow for network, DB-specific exporters with advanced alerting, security patterns, full integration/correlation) feels overwhelming for me right now.

My question for the community:

• On a scale of Junior/Mid/Senior/Staff, what level do you think this task requires to do independently at production quality (scaleable, reliable alerting, cost-optimized, maintainable)?

• Is it realistic for a strong Junior+/early-Mid (2–3 years exp) to tackle this solo, or is it typically a Senior+ (4–7+ years) job with real production incident experience?

• What are the biggest pitfalls/trade-offs for beginners attempting this? (e.g., alert fatigue, storage costs for logs, wrong exporters)

• Recommended starting point/stack for someone like me? (e.g., begin with Prometheus + snmp_exporter + postgres_exporter + Loki, then expand)

I’d love honest opinions from people who’ve built similar systems (open-source or at work). Thanks in advance – really appreciate the community’s insights

https://redd.it/1q88ulk
@r_devops

SBOM generation for a .net app in a container

I'm trying to create a reliable way to track packages we use (for license and CVE issues). So far I'm using CycloneDX for .NET apps, and cyclonedx-npm for our React apps. This is working fine.



I'm now looking to make this work for a .NET app deployed via Docker, and I'm not sure how to proceed. Currently I'm generating two SBOMs:



1. CycloneDX for the .NET application code (captures NuGet packages with versions)

2. Syft for the container image (captures OS packages and other container dependencies)



My questions:

\- Should I merge these BOMs into one, or treat them as separate projects in Dependency-Track?

\- Syft doesn't seem to capture NuGet package versions properly - if I only use Syft's SBOM, I'm missing important .NET dependency details

\- Is there a better tool than Syft for .NET containers, or a way to make Syft scan the published app files properly?



What approach do you use for tracking both application dependencies AND container dependencies for .NET apps in Docker?

https://redd.it/1q8erp9
@r_devops

Cosmic Rundown: How developers think about creativity, infrastructure, and perception

Interesting read on how developers approach infrastructure and system design. The article explores the intersection of creativity and logistics.


https://www.cosmicjs.com/blog/cosmic-rundown-how-will-the-miracle-happen-london-calcutta-bus-protest-perception

https://redd.it/1q8ha0z
@r_devops

Where are you keeping your LLM logs?

LLM logs are crushing my application logging system. We recently launched AI features on our app and went from \~100mb/month of normal website logs to 3gb/month of llm conversation logs and growing. Our existing logging system was overwhelmed (queries timing out, etc), and costs started increasing. We’re considering how to re-architect our llm logs specifically so we can handle more users plus the increasing token use from things like reasoning models, tool calling, and multi-agent systems. I’m not selling any solutions here, genuinely curious what others are doing. Do you store them alongside APM logs? Dedicated LLM logging service? Build it yourself with open source tools?

https://redd.it/1q8g282
@r_devops

Recommendations for log monitoring tools

Hey everyone, hope you’re doing well.

I’m looking for recommendations for log monitoring tools with decent Webhook integration.

I currently use New Relic. I’ve set up Log + Alert Policies, but the best I could manage was getting generic alerts on Discord, like "Query result is > 0 on 'Error Log Detected'".

The problem is that this alert lacks context. It doesn't tell me what the error was. I’m forced to log into the New Relic dashboard, filter the time window, and manually hunt down the log just to see the stack trace. This is exactly the kind of manual toil I want to eliminate.

I need a tool that triggers a webhook and sends the actual log content (traceback/error message) directly in the notification body when my app throws an exception. I want to be able to glance at Discord and immediately know where the code broke.

Has anyone dealt with this? Any suggestions?

Thanks!

https://redd.it/1q897a8
@r_devops

Has anyone actually tried AWS DevOps Agent for incident response? Worth the setup effort?

Hey everyone,

I'm an SRE at a mid-sized company and we're drowning in incident response time. Our typical P1 takes 2-3 hours just to figure out what's actually broken - we're jumping between CloudWatch, Datadog, our deployment logs in GitHub, and trying to correlate what changed with what broke.

I saw AWS announced DevOps Agent at re:Invent and it sounds almost too good to be true - like it automatically correlates all this stuff and investigates incidents for you? But I'm skeptical because:

1. We have a pretty complex setup (multiple AWS accounts, microservices, the usual mess)
2. I don't want to spend a week integrating something that gives me generic "have you tried turning it off and on again" advice
3. It's in preview so I'm worried about stability/support

For those who've actually used it:

How long did setup take realistically and be actually useful?
Does it actually find root causes or just surface the same logs you'd find manually?
Is it useful for complex distributed system issues or just simple stuff?
Any gotchas with multi-account setups?

Our on-call rotation is brutal right now and management is asking why our MTTR is so high. If this tool actually works, it could be a game-changer. But if it's just AI hype, I'd rather spend my time improving our runbooks.

Thanks for any real-world experiences you can share!

https://redd.it/1q8k1ct
@r_devops

Transitioning into DevOps from Help Desk

Hi everyone! I've recently built my own home lab environment and I've thoroughly enjoyed the ups and downs of being able to host multiple services on my own. Currently not satisfied/no longer challenged with the work that I'm doing at my current job and I'm interested in transitioning into the DevOps industry but need some guidance as I'm unsure on what I should be focusing on first.

Background:

\- 27 yrs old

\- No degree. Dropped out in 2018. 1.6 GPA. School was never a strong suit for me growing up.

\- No certifications. Tried focusing on A+/Network+ a year ago, but I didn't have the passion that I have now to follow through with either certification. Likely will obtain either or this year.

\- 7 yrs of experience in IT at my current job. Started off as a part-time helpdesk tech and got promoted into various senior level help desk roles focusing on different parts of our product's support/installation efforts. Worked in a NOC environment, field service/product implementation support, led and managed a team of help desk techs and even had a year of experience as a project coordinator. Current role is senior field service operations engineer (leading a team that supports our technicians who are sent out to install and troubleshoot our product).

\- Absolutely despise inefficiencies. At my current job, if I see something that can either be automated or streamline to assist my team and the customer, I try to pitch to to leadership and sometimes it's appreciated and it sticks. But honestly, most of the time I'm told to "get back to solving tickets".

\- I thrive in DIY/hands-on learning. Primarily self-taught IT through building PCs, configuring my home network (VLAN segmentation/tagging, IDS/IPS, subnetting, etc.), and now my home lab environment. I also like to be thrown into the fire and be forced to learn, but on my own terms (might be a bad habit?).

Why am I thinking about DevOps?:

\- Started building my home lab on bare metal early last year with Proxmox. Deploying, breaking and fixing my services is what's now filling my free-time after work. I used to be a heavy PC gamer but the time I used to spending gaming is now spent maintaining and deploying new services. It's my primary driving point for trying to get into the DevOps world after successfully deploying multiple VMs and containers on my server. Currently hosting services such as a mail server, TrueNAS, Home Assistant, Portainer, Jellyfin, Nginx, Beszel and other niche services. Most of them have been deployed with Docker and I manage them in Portainer.

After lurking in this and other subreddits, I've heard that I should look into the following:

\- Understand the basics of CI/CD

\- Deploy and understand the uses of Grafana/Prometheus

\- Get comfortable with K8s/K3s

\- Learn Python/Go

\- Continue using Bash

I'm open to any and all suggestions on where I should go next with my journey. Perhaps I'm more suited for another industry? Feel free to ask questions. Thanks in advance, hope everyone's 2026 is starting off well :)

TL;DR - I'm a help desk grunt that wants more for his career than solving the same issues over and over. Found out about home labbing, enjoy deploying and maintaining docker containers, need advice on how to enter the DevOps industry and land my first junior dev ops role or bridge role.

https://redd.it/1q8q1of
@r_devops

Huge e-commerce brands buckle under the pressure of high volume sales. Why?

Hello devops! So this past holiday season I had a job at a call center where we did customer service for a few worldwide beauty brands. Why I´m making this post is that their sites could not handle the load for Cyber Monday and Black Friday sales. Irate almost-customers called in to complain how the ordering system didn´t allow them to get through checkout. False order confirmations, items in their shopping cart not making it through to the backend ordering system, customers having their orders frozen at checkout... As customer service agents we all use Salesforce on the backend. How do huge companies like these have such crappy websites? Is it the fault of the developers for the sites themselves? Is it a problem in the backend between the website and the Salesforce ordering system? I welcome any and all opinions on the matter. You never see Amazon having trouble like this with their website. Why do these big brands (think Versace, Gap, etc.) have such sucky e-commerce system?

https://redd.it/1q8r9tz
@r_devops

Planning a career transition, does my plan make sense? Pipeline TD ->DevOps -> MLops

I am currently a Senior Pipeline Technical Director (Pipe TD for short) at a VFX/CG studio in Vancouver, BC with 7 YOE. Lately I've been feeling like I'm stagnating both in terms of learning new skills and salary (getting close to the cap at the senior level). Also, the VFX industry is declining and it's hard to find a new pipe openings at other studios these days. I've been doing some research and found that DevOps role is similar to my current role. My current responsibilities:

\- manage the render farm for failing jobs/efficiency of renders, stuck frames etc

\- make sure the pipeline outputs clean data between different departments (layout/anim/lighting etc)

\-troubleshoot artists' broken anim/lighting scenes

\-patch bugs in code for artists tools

\-make plugins/noscripts to make artist's life easier

\-a lot of babysitting artists so that they can log off on time at 5pm and not having to worry about their things breaking


My plan to break into DevOps and eventually into MLops:

1. study and pass the AWS Certified Solutions Architect - Associate Certificate

2. learn about IOC (TerraForm)

3. learn Docker and Kubernetes

4. Apply for a devOps role (after 6-7 months of study and personal projects)

5. If I get accepted, learn as much as I can

6.While employed, go through https://github.com/DataTalksClub/mlops-zoomcamp and apply it to personal projects

7. Get MLOps related certs

8. start applying to MLOps roles when I have \~2 years of devOps experience


Is my plan feasible? are there are gaping holes?

https://redd.it/1q8pug6
@r_devops