No Kubernetes experience, Am I cooked?
Currently in a role which everything is deployed via AWS ECS Fargate containers. I have been supporting these applications for a little bit now. There is not a TON of net new things to work on and learn. Just browsing roles or Job Denoscriptions I am seeing a ton of companies asking for Kubernetes experience. It seems like 80-90% of the roles want this for a mid level engineer. Are this many companies actually using Kubernetes, whether it be AWS EKS or Azure AKS, or googles Kubernetes offering.
having no experience and frankly, Kubernetes for my current work application is overkill. So I wouldn't be able to gain on the job experience. That said, am I cooked in this Job market(outside of the Market already being doo-doo in general). I have come across posts of folks who study for the cert but seem to not have hands on experience - which I DONT want to go down this route, not sure what the though process is on that lol.
Thought about doing it on my spare time but kids and wife take a good majority of my weekend, and not sure what the best method is to learn about Kubernetes and which learning method would be the most effective which the community recommends.
https://redd.it/1oj5dlq
@r_devops
Currently in a role which everything is deployed via AWS ECS Fargate containers. I have been supporting these applications for a little bit now. There is not a TON of net new things to work on and learn. Just browsing roles or Job Denoscriptions I am seeing a ton of companies asking for Kubernetes experience. It seems like 80-90% of the roles want this for a mid level engineer. Are this many companies actually using Kubernetes, whether it be AWS EKS or Azure AKS, or googles Kubernetes offering.
having no experience and frankly, Kubernetes for my current work application is overkill. So I wouldn't be able to gain on the job experience. That said, am I cooked in this Job market(outside of the Market already being doo-doo in general). I have come across posts of folks who study for the cert but seem to not have hands on experience - which I DONT want to go down this route, not sure what the though process is on that lol.
Thought about doing it on my spare time but kids and wife take a good majority of my weekend, and not sure what the best method is to learn about Kubernetes and which learning method would be the most effective which the community recommends.
https://redd.it/1oj5dlq
@r_devops
Reddit
From the devops community on Reddit
Explore this post and more from the devops community
How N26 builds reliability at scale — with Bruno Paulino (Tech Lead at N26)
What does reliability actually look like when every deploy touches millions of bank customers?
In this episode of Señors @ Scale, Bruno Paulino (Tech Lead at N26) shares how his teams build resilient FinTech systems — from CI/CD pipelines and server-driven UIs to AI-powered customer support.
We cover:
Cutting deploy times from 1 hour to 5 minutes
Rolling out server-driven UI across mobile and web
Using LLMs and RAG to scale customer support
Statsig and safe experimentation in production
Balancing speed, compliance, and reliability in FinTech
Lessons from outages, testing, and developer culture
🎧 Watch or listen:
▶️ YouTube: https://youtu.be/XA42xUQlxRY
🎧 Spotify: https://open.spotify.com/episode/1cVpylsiGZphf8Pr6ocFgv
🍎 Apple Podcasts: https://podcasts.apple.com/us/podcast/reliability-at-scale-with-bruno-paulino-n26/id1827500070?i=1000733534640
If you’re into DevOps, platform engineering, or CI/CD at scale — this one’s for you.
https://redd.it/1oj50af
@r_devops
What does reliability actually look like when every deploy touches millions of bank customers?
In this episode of Señors @ Scale, Bruno Paulino (Tech Lead at N26) shares how his teams build resilient FinTech systems — from CI/CD pipelines and server-driven UIs to AI-powered customer support.
We cover:
Cutting deploy times from 1 hour to 5 minutes
Rolling out server-driven UI across mobile and web
Using LLMs and RAG to scale customer support
Statsig and safe experimentation in production
Balancing speed, compliance, and reliability in FinTech
Lessons from outages, testing, and developer culture
🎧 Watch or listen:
▶️ YouTube: https://youtu.be/XA42xUQlxRY
🎧 Spotify: https://open.spotify.com/episode/1cVpylsiGZphf8Pr6ocFgv
🍎 Apple Podcasts: https://podcasts.apple.com/us/podcast/reliability-at-scale-with-bruno-paulino-n26/id1827500070?i=1000733534640
If you’re into DevOps, platform engineering, or CI/CD at scale — this one’s for you.
https://redd.it/1oj50af
@r_devops
YouTube
Reliability at Scale – With Bruno Paulino (N26)
🎙 About the Podcast:
Señors @ Scale is a no-fluff engineering podcast hosted by Neciu Dan — diving into the real-world chaos of scaling systems, teams, and yourself. From production bugs to platform bets, we sit down with senior engineers to discuss the scars…
Señors @ Scale is a no-fluff engineering podcast hosted by Neciu Dan — diving into the real-world chaos of scaling systems, teams, and yourself. From production bugs to platform bets, we sit down with senior engineers to discuss the scars…
Octopus Deploy vs speed/safety tradeoffs
One of the biggest tensions in DevOps is shipping faster vs shipping safer. Octopus Deploy gives us approvals, audit logs, and runbooks, but those can also slow things down if overused.
How do you balance speed and safety in Octopus Deploy? Feature flags? Progressive deployments? Manual approvals only in certain environments? Would love to hear how other teams approach this.
https://redd.it/1oj5szo
@r_devops
One of the biggest tensions in DevOps is shipping faster vs shipping safer. Octopus Deploy gives us approvals, audit logs, and runbooks, but those can also slow things down if overused.
How do you balance speed and safety in Octopus Deploy? Feature flags? Progressive deployments? Manual approvals only in certain environments? Would love to hear how other teams approach this.
https://redd.it/1oj5szo
@r_devops
Reddit
From the devops community on Reddit
Explore this post and more from the devops community
Apple's new container runtime vs Docker Desktop
Hi everyone
I was curious how Apple’s new container system compares to Docker Desktop, so I ran some benchmarks.
I tested CPU, memory, disk I/O, and startup time.
|Category|Docker|Apple|Units|
|:-|:-|:-|:-|
|CPU 1 thread|10939.81|11080.05|events/s|
|CPU all threads|53881.70|55415.57|events/s|
|Memory|81634.45|108588.00|MiB/s|
|Startup time|0.21|0.92|seconds|
Full charts and results, are available here: Full Benchmark
Let me know if you’d like me to run additional tests
https://redd.it/1oj9wxs
@r_devops
Hi everyone
I was curious how Apple’s new container system compares to Docker Desktop, so I ran some benchmarks.
I tested CPU, memory, disk I/O, and startup time.
|Category|Docker|Apple|Units|
|:-|:-|:-|:-|
|CPU 1 thread|10939.81|11080.05|events/s|
|CPU all threads|53881.70|55415.57|events/s|
|Memory|81634.45|108588.00|MiB/s|
|Startup time|0.21|0.92|seconds|
Full charts and results, are available here: Full Benchmark
Let me know if you’d like me to run additional tests
https://redd.it/1oj9wxs
@r_devops
www.repoflow.io
Benchmarking Apple Containers vs Docker Desktop
We benchmarked Apple’s new container system against Docker Desktop on macOS. Discover which runs containers faster, uses less memory, and handles I/O more efficiently.
AWS took break, Azure Followed , Down Again
Azure down: Thousands of users complain about outage; here's Microsoft's latest statement
https://www.google.com/amp/s/m.economictimes.com/news/new-updates/azure-down-thousands-of-users-complain-about-outage-heres-microsofts-latest-statement/amp_articleshow/124910174.cms
https://azure.status.microsoft/en-gb/status
https://redd.it/1ojc3xj
@r_devops
Azure down: Thousands of users complain about outage; here's Microsoft's latest statement
https://www.google.com/amp/s/m.economictimes.com/news/new-updates/azure-down-thousands-of-users-complain-about-outage-heres-microsofts-latest-statement/amp_articleshow/124910174.cms
https://azure.status.microsoft/en-gb/status
https://redd.it/1ojc3xj
@r_devops
The Economic Times
Azure down: Thousands of users complain about outage; here's Microsoft's latest statement
Microsoft's Azure experienced an outage affecting nearly 11,500 users on Wednesday. The company confirmed it was investigating an issue with the Azure Portal, where customers faced access problems. Further details were promised shortly as the situation unfolded.
DevOps engineers: What Bash skills do you actually use in production that aren't taught in most courses?
I'm a DevOps Team Lead managing Kubernetes/AWS infrastructure at an FDA-compliant medical device company. My colleague works at Proofpoint doing security automation.
We've both noticed that most Bash courses teach toy examples, but production Bash is different. We're curious what real-world skills you wish you'd learned earlier:
* Are you parsing CloudWatch/Splunk logs?
* Automating CI/CD pipelines?
* Handling secrets management in noscripts?
* Debugging production incidents with Bash one-liners?
* Something else entirely?
What Bash skills have been most valuable in your DevOps career that you had to learn the hard way?
https://redd.it/1ojcrdo
@r_devops
I'm a DevOps Team Lead managing Kubernetes/AWS infrastructure at an FDA-compliant medical device company. My colleague works at Proofpoint doing security automation.
We've both noticed that most Bash courses teach toy examples, but production Bash is different. We're curious what real-world skills you wish you'd learned earlier:
* Are you parsing CloudWatch/Splunk logs?
* Automating CI/CD pipelines?
* Handling secrets management in noscripts?
* Debugging production incidents with Bash one-liners?
* Something else entirely?
What Bash skills have been most valuable in your DevOps career that you had to learn the hard way?
https://redd.it/1ojcrdo
@r_devops
Reddit
From the devops community on Reddit
Explore this post and more from the devops community
The Vi editor Survival Guide for devs like me
I have put together a simple guide to vi commands that actually helped me all these years when editing configs or noscripts on Linux.
Short, practical, and focused on real examples.
Let me know if I have missed some..would love to take feedbacks and make it an exhaustive list!
Read it here
https://redd.it/1ojac48
@r_devops
I have put together a simple guide to vi commands that actually helped me all these years when editing configs or noscripts on Linux.
Short, practical, and focused on real examples.
Let me know if I have missed some..would love to take feedbacks and make it an exhaustive list!
Read it here
https://redd.it/1ojac48
@r_devops
Medium
Vi /Vim Editor : Practical commands every developer, sysadmin, and DevOps engineer should know.
The vi editor has been around for decades, yet it remains one of the most powerful and lightweight text editors available on any Unix-like…
Do I build "api-core" layer as an always-on container (App Runner / Fargate) — or as event-driven Lambda functions?
Such as user auth, billing, usage. Think core business logic that my webapps will call about my customers (B2C/B2B)
Where the api-core is like an internal service, with its own ci/cd pipeline
https://redd.it/1ojgtza
@r_devops
Such as user auth, billing, usage. Think core business logic that my webapps will call about my customers (B2C/B2B)
Where the api-core is like an internal service, with its own ci/cd pipeline
https://redd.it/1ojgtza
@r_devops
Reddit
From the devops community on Reddit
Explore this post and more from the devops community
Taking the CKAD exam this week after CKS and CKA. Any advice?
Hi All!
I am taking the CKAD exam next week. I was urged to be a KUBERSTRONAUT by my co-workers. Any advice for me? I am yet to do the Killrsh practice tests (I want to do it just before the exams).
My past experiences with the exam have been that the questions are really not what you expect. Is it going to be the same with CKAD? I am going in with just a week's prep so I am feeling a bit unprepared. Should I work for another week?
Any particular topics that I should focus on?
Thanks in advance for all your help!
https://redd.it/1ojargr
@r_devops
Hi All!
I am taking the CKAD exam next week. I was urged to be a KUBERSTRONAUT by my co-workers. Any advice for me? I am yet to do the Killrsh practice tests (I want to do it just before the exams).
My past experiences with the exam have been that the questions are really not what you expect. Is it going to be the same with CKAD? I am going in with just a week's prep so I am feeling a bit unprepared. Should I work for another week?
Any particular topics that I should focus on?
Thanks in advance for all your help!
https://redd.it/1ojargr
@r_devops
Reddit
From the devops community on Reddit
Explore this post and more from the devops community
Does every DevOps role really need Kubernetes skills?
I’ve noticed that most DevOps job postings these days mention Kubernetes as a required skill. My question is, are all DevOps roles really expected to involve Kubernetes?
Is it not possible to have DevOps engineers who don’t work with Kubernetes at all? For example, a small startup that is just trying to scale up might find Kubernetes to be an overkill and quite expensive to maintain.
Does that mean such a company can’t have a DevOps engineer on their team? I’d like to hear what others think about this.
https://redd.it/1ojj08t
@r_devops
I’ve noticed that most DevOps job postings these days mention Kubernetes as a required skill. My question is, are all DevOps roles really expected to involve Kubernetes?
Is it not possible to have DevOps engineers who don’t work with Kubernetes at all? For example, a small startup that is just trying to scale up might find Kubernetes to be an overkill and quite expensive to maintain.
Does that mean such a company can’t have a DevOps engineer on their team? I’d like to hear what others think about this.
https://redd.it/1ojj08t
@r_devops
Reddit
From the devops community on Reddit
Explore this post and more from the devops community
Is there a way to get notified when a CVE in your container image is actually being exploited in the wild?
Getting tired of patching every theoretical CVE that scanners throw at us. Half of them never see real exploits but still create noise and patch fatigue.
Anyone know of tools or feeds that can tell you when a CVE in your container images is actually being exploited in the wild? Not just CVSS scores or theoretical impact, but real threat intel showing active exploitation.
Would love to prioritize patches based on actual risk instead of just severity numbers.
https://redd.it/1ojimb6
@r_devops
Getting tired of patching every theoretical CVE that scanners throw at us. Half of them never see real exploits but still create noise and patch fatigue.
Anyone know of tools or feeds that can tell you when a CVE in your container images is actually being exploited in the wild? Not just CVSS scores or theoretical impact, but real threat intel showing active exploitation.
Would love to prioritize patches based on actual risk instead of just severity numbers.
https://redd.it/1ojimb6
@r_devops
Reddit
From the devops community on Reddit
Explore this post and more from the devops community
Business Logic Flaws: The Vulnerabilities No Scanner Can Find 🧩
https://instatunnel.my/blog/business-logic-flaws-the-vulnerabilities-no-scanner-can-find
https://redd.it/1ojpr4n
@r_devops
https://instatunnel.my/blog/business-logic-flaws-the-vulnerabilities-no-scanner-can-find
https://redd.it/1ojpr4n
@r_devops
InstaTunnel
Business Logic Flaws: Vulnerabilities No Scanner Can Find
Discover business logic flaws that evade automated scanners. Learn about coupon stacking, negative quantity exploits, race conditions & prevention strategies
35 to DevOps too late?
Been doing QA for the past 5 years and it is getting toll on me. I feel like I can do more and I love tinkering linux. I don't hate my job God bless but feels like I can do more. I am more than your average user, but less than a professional DevOps I suppose. Appreciate your opinions.
https://redd.it/1ojr5vd
@r_devops
Been doing QA for the past 5 years and it is getting toll on me. I feel like I can do more and I love tinkering linux. I don't hate my job God bless but feels like I can do more. I am more than your average user, but less than a professional DevOps I suppose. Appreciate your opinions.
https://redd.it/1ojr5vd
@r_devops
Reddit
From the devops community on Reddit
Explore this post and more from the devops community
How do you write your first post about a new habit-building app?
I’ve recently finished developing my first product app that helps users build habits and achieve their goals step by step. Since I don’t have prior marketing experience, I’m planning to start with zero-cost marketing and rely mainly on organic posts. My goal is to share the story behind the app and invite feedback, but I’m unsure how to write that first post without sounding like I’m trying to sell something.
For those who’ve launched a product before, how did you craft your first post to make it feel authentic and engaging? What elements or structure helped you get genuine feedback instead of just promotional nois
https://redd.it/1ojs797
@r_devops
I’ve recently finished developing my first product app that helps users build habits and achieve their goals step by step. Since I don’t have prior marketing experience, I’m planning to start with zero-cost marketing and rely mainly on organic posts. My goal is to share the story behind the app and invite feedback, but I’m unsure how to write that first post without sounding like I’m trying to sell something.
For those who’ve launched a product before, how did you craft your first post to make it feel authentic and engaging? What elements or structure helped you get genuine feedback instead of just promotional nois
https://redd.it/1ojs797
@r_devops
Reddit
From the devops community on Reddit
Explore this post and more from the devops community
Stuck between honesty and overselling.
I’ve been working in DevOps for about 12 years now. Covering most aspects over the years: build and release management, infra provisioning and maintenance (cloud and on-prem), SRE work, config management, and a bit of DevSecOps too.
Here’s where my dilemma starts. Like most DevOps engineers in large orgs, I haven’t personally set up every layer of the stack. For instance,
* I know Kubernetes well enough to manage deployments, troubleshoot, and maintain clusters, but I wasn’t the one who built them from scratch.
* Same with Ansible, I write and manage playbooks daily, but I didn’t originally architect or configure the controller host.
* Similar story with Terraform, cloud infra setup, and WAF/network administration, I understand the moving parts and can work on them, but I didn’t create everything ground-up.
In interviews, when I explain this honestly, I can almost feel the interviewer’s interest drop the moment I say “I haven’t personally set up the cluster or administer it” or “I wasn’t responsible for the initial infra design.”
Yet, I see people who exaggerate their contributions land those same roles. People who, frankly, can’t even write solid production-ready manifests or pipelines. There are people who write manifests in Notepad++ who are hired in Lead DevOps role(same as me). It's frustrating working with these people.
So, here’s my question:
* Is it time I start “selling” myself more aggressively in interviews?
* Or is there a way to frame my experience truthfully without underselling what I actually know and can do?
I don’t want to lie, but I’m starting to feel that being 100% transparent is working against me. Has anyone else faced this? How do you balance credibility and confidence in technical interviews; especially in senior DevOps/SRE roles?
I don't like the feeling of getting rejected in final round of interviews. Or am I just overestimating my skills/capabilities and I'm far behind market/job expectations. What is it that I'm doing wrong?
https://redd.it/1ojrzhi
@r_devops
I’ve been working in DevOps for about 12 years now. Covering most aspects over the years: build and release management, infra provisioning and maintenance (cloud and on-prem), SRE work, config management, and a bit of DevSecOps too.
Here’s where my dilemma starts. Like most DevOps engineers in large orgs, I haven’t personally set up every layer of the stack. For instance,
* I know Kubernetes well enough to manage deployments, troubleshoot, and maintain clusters, but I wasn’t the one who built them from scratch.
* Same with Ansible, I write and manage playbooks daily, but I didn’t originally architect or configure the controller host.
* Similar story with Terraform, cloud infra setup, and WAF/network administration, I understand the moving parts and can work on them, but I didn’t create everything ground-up.
In interviews, when I explain this honestly, I can almost feel the interviewer’s interest drop the moment I say “I haven’t personally set up the cluster or administer it” or “I wasn’t responsible for the initial infra design.”
Yet, I see people who exaggerate their contributions land those same roles. People who, frankly, can’t even write solid production-ready manifests or pipelines. There are people who write manifests in Notepad++ who are hired in Lead DevOps role(same as me). It's frustrating working with these people.
So, here’s my question:
* Is it time I start “selling” myself more aggressively in interviews?
* Or is there a way to frame my experience truthfully without underselling what I actually know and can do?
I don’t want to lie, but I’m starting to feel that being 100% transparent is working against me. Has anyone else faced this? How do you balance credibility and confidence in technical interviews; especially in senior DevOps/SRE roles?
I don't like the feeling of getting rejected in final round of interviews. Or am I just overestimating my skills/capabilities and I'm far behind market/job expectations. What is it that I'm doing wrong?
https://redd.it/1ojrzhi
@r_devops
Reddit
From the devops community on Reddit
Explore this post and more from the devops community
Fresher DevOps Engineer (3 months in) — how can I best use my free time to upskill for a better WLB + higher paying role later?
Hey folks 👋
I joined 3 months ago as a Junior DevOps Engineer (fresher). My CTC is 3 LPA and there’s a 2-year bond (₹1L if I break it). The work is super light, so I get a lot of free time in office.
Here’s what I have access to:
Ubuntu VM with sudo access
ChatGPT
2 weekly offs (Sat & Sun)
Right now I know a bit of Linux, Jenkins, GitLab, SVN, and WinSCP.
My goal is to upskill in DevOps + Cloud, build hands-on projects, and later move to a remote or Hyderabad-based role with better pay + WLB.
My goal:
👉 Build solid DevOps + Cloud skills
👉 Create hands-on projects I can show later on GitHub
👉 Prepare for a better-paying role after my bond (ideally remote or Hyderabad-based)
👉 Maintain a good work-life balance
Can you suggest:
What should I focus on learning next (AWS, Docker, Kubernetes, Terraform, etc.)?
Any project ideas I can do on my Ubuntu VM?
Free resources, YouTube channels, or courses worth following?
How to plan a practical roadmap using ChatGPT + self-practice?
https://redd.it/1ojuqtw
@r_devops
Hey folks 👋
I joined 3 months ago as a Junior DevOps Engineer (fresher). My CTC is 3 LPA and there’s a 2-year bond (₹1L if I break it). The work is super light, so I get a lot of free time in office.
Here’s what I have access to:
Ubuntu VM with sudo access
ChatGPT
2 weekly offs (Sat & Sun)
Right now I know a bit of Linux, Jenkins, GitLab, SVN, and WinSCP.
My goal is to upskill in DevOps + Cloud, build hands-on projects, and later move to a remote or Hyderabad-based role with better pay + WLB.
My goal:
👉 Build solid DevOps + Cloud skills
👉 Create hands-on projects I can show later on GitHub
👉 Prepare for a better-paying role after my bond (ideally remote or Hyderabad-based)
👉 Maintain a good work-life balance
Can you suggest:
What should I focus on learning next (AWS, Docker, Kubernetes, Terraform, etc.)?
Any project ideas I can do on my Ubuntu VM?
Free resources, YouTube channels, or courses worth following?
How to plan a practical roadmap using ChatGPT + self-practice?
https://redd.it/1ojuqtw
@r_devops
Reddit
From the devops community on Reddit
Explore this post and more from the devops community
Stuck between a great PhD offer and a solid DevOps career any advice?
I’m currently working as a DevOps Engineer with a good salary, and I’m 27 years old.
Recently, I received an offer to pursue a PhD at a top 100 university in the world. The topic aligns perfectly with my passion — information security, WebAssembly, Rust, and cloud computing.
The salary is much lower than my current salary, and it will take around 5 years to finish the program, but I see this as a rare opportunity at my age to gain strong research experience and deepen my technical skills.
I’m struggling to decide is this truly a strong opportunity worth taking, or should I stay in the industry and keep building my professional experience?
Has anyone here gone through a similar situation? How did it impact your career afterward whether you stayed in academia or returned to industry?
After having a phd in information security, what are the opportunities to come back to the industry?
https://redd.it/1ojv4rf
@r_devops
I’m currently working as a DevOps Engineer with a good salary, and I’m 27 years old.
Recently, I received an offer to pursue a PhD at a top 100 university in the world. The topic aligns perfectly with my passion — information security, WebAssembly, Rust, and cloud computing.
The salary is much lower than my current salary, and it will take around 5 years to finish the program, but I see this as a rare opportunity at my age to gain strong research experience and deepen my technical skills.
I’m struggling to decide is this truly a strong opportunity worth taking, or should I stay in the industry and keep building my professional experience?
Has anyone here gone through a similar situation? How did it impact your career afterward whether you stayed in academia or returned to industry?
After having a phd in information security, what are the opportunities to come back to the industry?
https://redd.it/1ojv4rf
@r_devops
Reddit
From the devops community on Reddit
Explore this post and more from the devops community
Offloading SQL queries to read-only replica
What's the best strategy? One approach is to redirect all reads to replica and all writes to master. This is too crude, so I choose to do things manually, think
Database.on_replica do
# code here
end
However this has hidden footguns. For one thing the code should make no writes to the database. This is easy to verify if it's just a few lines of code, but becomes much more difficult if there are calls to procedures defined in another file, which call other files, which call something in a library. How can a developer even know that the procedure they're modifying is used within a read-only scope somewhere high up in the call chain?
Another problem is "mostly reads". This is find_or_create method semantics. It does a SELECT most of the time, but for some subset of data it issues an INSERT.
And yet another problem is automated testing. How to make sure that a bunch of queries are always executed on a replica? Well, you have to have a replica in test environment. Ok, that's no big deal, I managed to set it up. However, how do you get the data in there? It is read-only, so naturally you have to write to the master. This means you have to commit the transaction, otherwise replica won't see anything. Committing transactions is slow when you have to create and delete thousands of times per each test suit run.
There has to be a better way. I want my replica to ease the burden of master database because currently it is mostly idle.
https://redd.it/1ojv8gv
@r_devops
What's the best strategy? One approach is to redirect all reads to replica and all writes to master. This is too crude, so I choose to do things manually, think
Database.on_replica do
# code here
end
However this has hidden footguns. For one thing the code should make no writes to the database. This is easy to verify if it's just a few lines of code, but becomes much more difficult if there are calls to procedures defined in another file, which call other files, which call something in a library. How can a developer even know that the procedure they're modifying is used within a read-only scope somewhere high up in the call chain?
Another problem is "mostly reads". This is find_or_create method semantics. It does a SELECT most of the time, but for some subset of data it issues an INSERT.
And yet another problem is automated testing. How to make sure that a bunch of queries are always executed on a replica? Well, you have to have a replica in test environment. Ok, that's no big deal, I managed to set it up. However, how do you get the data in there? It is read-only, so naturally you have to write to the master. This means you have to commit the transaction, otherwise replica won't see anything. Committing transactions is slow when you have to create and delete thousands of times per each test suit run.
There has to be a better way. I want my replica to ease the burden of master database because currently it is mostly idle.
https://redd.it/1ojv8gv
@r_devops
Reddit
From the devops community on Reddit
Explore this post and more from the devops community
payment processing went down for 2 minutes. engineering said p3. finance said p1
we had a payment gateway timeout friday that lasted barely 2 minutes. during that time customers couldnt complete checkouts.
engineering immediately called it p3. its a known issue with the third party provider. happens occasionally. self resolved. no code changes needed.
finance lost their minds. called it p1. ran the numbers and we lost significant revenue because its black friday weekend. customers who hit errors abandoned carts and didnt come back.
support sided with finance because they got slammed with tickets and customers were threatening chargebacks on social media.
product sided with engineering because technically the system worked as designed. timeout and retry logic did exactly what it should.
spent the entire postmortem arguing about severity instead of talking about improvements. finance wants anything touching payments to be p1 automatically. engineering says that makes severity meaningless.
the problem is both are right. from technical standpoint it was minor. from business standpoint we literally lost money during peak shopping weekend.
calling on fintech and ecommerce people: how do you handle this kinda scenario, looking for some advice.?
https://redd.it/1ojz89l
@r_devops
we had a payment gateway timeout friday that lasted barely 2 minutes. during that time customers couldnt complete checkouts.
engineering immediately called it p3. its a known issue with the third party provider. happens occasionally. self resolved. no code changes needed.
finance lost their minds. called it p1. ran the numbers and we lost significant revenue because its black friday weekend. customers who hit errors abandoned carts and didnt come back.
support sided with finance because they got slammed with tickets and customers were threatening chargebacks on social media.
product sided with engineering because technically the system worked as designed. timeout and retry logic did exactly what it should.
spent the entire postmortem arguing about severity instead of talking about improvements. finance wants anything touching payments to be p1 automatically. engineering says that makes severity meaningless.
the problem is both are right. from technical standpoint it was minor. from business standpoint we literally lost money during peak shopping weekend.
calling on fintech and ecommerce people: how do you handle this kinda scenario, looking for some advice.?
https://redd.it/1ojz89l
@r_devops
Reddit
From the devops community on Reddit
Explore this post and more from the devops community
What do you do when Audit wants tickets and there are none?
For those in large public companies, do you ever work with Audit? What do you do when Audit comes around asking for tickets on work that was done using systems outside of Jira/ADO? Audit is breathing down our necks.
https://redd.it/1ojzvun
@r_devops
For those in large public companies, do you ever work with Audit? What do you do when Audit comes around asking for tickets on work that was done using systems outside of Jira/ADO? Audit is breathing down our necks.
https://redd.it/1ojzvun
@r_devops
Reddit
From the devops community on Reddit
Explore this post and more from the devops community
The problem I see with AI is if the person asking AI to do something doesn’t understand scale, they could end up with infrastructure issues at the foundation.
How many times have we had to talk our own people off a ledge for considering Kubernetes when we just need ECS or vice-versa? How many times has management come back from a conference with a new shiny and it then becomes the biggest maintenance headache for every one involved?
I think that we may not see it immediately but poorly architected architecture in middling companies that are trying to poorly execute AI agents will keep us busy for quite some time. The bubble isn’t a sudden pop. Its a slow realization that you screwed yourself over two years ago by blindly taking the recommendations of an advanced autocomplete program.
https://redd.it/1ok2g4q
@r_devops
How many times have we had to talk our own people off a ledge for considering Kubernetes when we just need ECS or vice-versa? How many times has management come back from a conference with a new shiny and it then becomes the biggest maintenance headache for every one involved?
I think that we may not see it immediately but poorly architected architecture in middling companies that are trying to poorly execute AI agents will keep us busy for quite some time. The bubble isn’t a sudden pop. Its a slow realization that you screwed yourself over two years ago by blindly taking the recommendations of an advanced autocomplete program.
https://redd.it/1ok2g4q
@r_devops
Reddit
From the devops community on Reddit
Explore this post and more from the devops community
❤1