data democratization aka automation and management of data platforms

Hi folks, Are you guys aware of any platforms that can help with management of a number of users on large datalakes, what i mean by this say u have a product like databricks and we want to "user-wise" manage how much access someone has, we wanna stream line this by maybe this flow , user raises a request somehwere -> automated noscript grants access -> access revoked automatically within a set time,
also log who had what access etc etc,
ofc a custom solution is possible but i was hoping for any opinions on if anything similar to this already exists.
Thanks for yuour time have agood one

https://redd.it/1oo9x3u
@r_devops

Learning friend

Is anyone here willing to learn Devops with me? I am a beginner

https://redd.it/1oo9js7
@r_devops

EKS Node Resource Limits

I am currently undertaking the task of auditing EKS Node resource limits, comparing the limits to the requests and actual usage for around 40 applications. I have to pinpoint where resources are being wasted and propose changes to limits/requests for these nodes.

My question for you all is, what percentage above average Usage should I set the resource limits? I know we still need some wiggle room, but say that an application is using on average 531m of Memory, but the limit is at 1000m (1Gb). That limit obviously needs to come down, but where should it come down to? 600m I think would be too close. Is there a rule of thumb to go by here?

Likewise, the same service uses 10.1mcores of CPU on average, but the limit is set to 1core. I know CPU throttling won't bring down an application, but I'd like to keep wiggle room there to, I'm just not sure how close to bring the limit to the average usage. Any advice?

https://redd.it/1oo78yq
@r_devops

GitOps role composition pattern for deployments?

Is anyone utilizing or has anyone utilized a cluster role-based composition pattern for deployments? Any other patterns?

Currently spinning up ArgoCD for current org and looking at efficiently implementing this for scalability.

At my previous org, we wound up having things a bit scattered about with \~30 AppSets and 30 applications (separate from appsets, for individual clusters).

It was manageable as we didn't change things much but I could see running into scaling issues as far as effort/maintenance goes down the road.

I would appreciate getting a second set of eyes to see if this makes sense or if I'm going to run into issues I haven't thought of: https://github.com/SelfhostedPro/ArgoCD-Role-Composition

https://redd.it/1ooejsr
@r_devops

How a tiny DNS fault brought down AWS us-east-1 and what devops engineers can learn from it

When AWS us-east-1 went down due to a DynamoDB issue, it wasn’t really DynamoDB that failed , it was DNS. A small fault in AWS’s internal DNS system triggered a chain reaction that affected multiple services globally.

It was actually a race condition formed between various DNS enacters who were trying to modify route53

If you’re curious about how AWS’s internal DNS architecture (Enacter, Planner, etc.) actually works and why this fault propagated so widely, I broke it down in detail here:

Inside the AWS DynamoDB Outage: What Really Went Wrong in us-east-1 https://youtu.be/MyS17GWM3Dk

https://redd.it/1ooi45v
@r_devops

What guardrails do you use for feature flags when the feature uses AI?

Before any flag expands, we run a preflight: a small eval set with known failure cases, observability on outputs, and thresholds that trigger rollback. Owners are by role and not by person, and we document the path to stable.

Which signals or tools made this smoother for you?

What do you watch in the first twenty four hours?

https://redd.it/1oo5u1m
@r_devops

LeetCode style interview for DevOps role

Curious if anyone has done any LeetCode style interviews recently?

Recently interviewed for a Senior DevOps role at a FAANG adjacent company which was a 6 stage process.

I thought I was doing pretty well after going though multiple stages doing system design, architecture, reliability engineering, scenario based troubleshooting etc, and even got through some coding exercises in Python.

One of the interviewers was changed last minute. I was told it would purely be a cultural fit type of interview but it ended up being a couple of LeetCode style problems which completely threw me off and I kinda of bombed and struggled to get through them.

I'm fairly experienced with Python but never learned DSA as I don't have a software engineering background and was frustrated to get failed on this after everything.

https://redd.it/1ookpme
@r_devops

Terraform + AWS Questions

So i'll try to keep this brief. I am an SDET learning Terraform as well as AWS. I think I mostly have "demo" stuff working but I wanted to just pose a list of questions off the top of my head:

1. Right now I think one s3 bucket per AWS account makes the most sense (for storing state). From my understanding the "key" is what determines both the terraform state file path as well as the LockID. However I am not sure if for example you define a backend s3.tf file, does the LockID use the key or the key+bucket name?
2. Sort of a follow up to #1, any suggestions for naming conventions when it comes to state files key? Something like environment+project+terraform/state.tf or similar?
3. When it comes to Terraform, I know there is the chicken and the egg sort of thing. What's the proper way to handle this? Some sort of bootstrap .tf file? From my understanding basically you would do that OR set up the s3 bucket manually and then import it? How does that usually go?
4. What are the main resources you think a newcomer should start focusing on as far as tracking? Right now i'm just doing the backend s3 and beanstalk (app and enviornment_ and rds currently.

https://redd.it/1ookd9a
@r_devops

Tofu/Terraform Modules for enterprise

So I'm looking to setup a tofu module repo, all the examples I can find show each module has to have its own git path to be loaded in.

Is there a way to load an entire repo of modules? Or do I have to roll a provider to do that?

I just want to put the classic stuff in place like tag requirements and sane defaults etc.

I got the backend config sorted but putting it in the pipeline templates so each init step gets the right settings. But struggling with the best way to centralize modules.

We are using tofu if that matters.

https://redd.it/1ooph4x
@r_devops

Need advice on deployment and dev ops

Built a simple wrapper around chatgpt for an internal audit my company and now they want it deployed company wide. I’ve never deployed something at a company, never even knew what a Linux box was until my IT team asked if I would be able to manage it which I obviously said yes too.

Looking for advice on how to best host and deploy because I’m going to have to be the one to manage it.

I have a python app wrapped in a fast api, that sends PDFs to OpenAI api for analysis and then returns the response on a basic streamlit UI. 2000-4000 6-10 page PDFs needs to be run through it monthly at scale. What’s the best way to get there. I’ve used render, but only on the free plan to demo it, now I’m pretty lost.

Any help would be great! My outsourced IT team says the solution is a Linux box which will take 10-14 days to set up. Company is ~90mm ARR, 300 employees.

I have no formal swe experience, I still have to ask the AI in cursor to run the commands to push things to GitHub. Please explain like I have basic knowledge, I will look up anything I don’t know.

https://redd.it/1oopug3
@r_devops

Insecure Direct Object References (IDOR): The $1 Billion Authorization Bug 🔢

https://instatunnel.my/blog/insecure-direct-object-references-idor-the-1-billion-authorization-bug

https://redd.it/1oossag
@r_devops

I wrote zigit, a tiny C program to download GitHub repos at lightning speed using aria2c

Hey everyone!
I recently made a small C tool called zigit — it’s basically a super lightweight alternative to git clone when you only care about downloading the latest source code and not the entire commit history.

zigit just grabs the ZIP directly from GitHub’s codeload endpoint using aria2c, which supports parallel and segmented downloads.

Check it out at : https://github.com/STRTSNM/zigit/

https://redd.it/1oownb2
@r_devops

Stateful or Stateless IaC?

I've been debating this topic relentlessly. What is better? Infra as Code, which maintains states or stateless that work directly with the resources?

View Poll

https://redd.it/1ooxi57
@r_devops

Feedback

We’re two founders building an AI system that automatically detects, predicts and fixes website/app errors in real time, think Tesla Autopilot for debugging in DevOps. 

We’d love to learn from you, engineers, founders or DevOps folks for 10 minutes about how you currently debug issues. 

Not selling anything, just trying to validate if this could save teams a significant amount time. 

Happy to share a summary of what we learn + offer early access! 

https://calendly.com/aarittaparia/30min 

If you don’t have time, we would appreciate if you could fill this form: https://rc60edu0zkd.typeform.com/to/YixyC7S7 

Thanks so much! 

https://redd.it/1ooyk0s
@r_devops

Any tips on places where i can train as aspiring devops?

Hi, currently working in small company and finishing my college degree in few months.

I got interested in devops around half year ago and trained linux, git, github, github actions + Jenkins, docker hub. Built pipelines on simple projets, even did some tests.
Also got my hands on deployment with kubctl but there is a lot i have to learn yet.

Back to the question. Coders have codewars and leetcode. I wonder if there is any site for devops?
I found Qwiklabs for GCP however i was wondering what about the rest? Like solving problems or using part of the knowledge to try fixing something more difficult?

I kind of want commercial experience..

https://redd.it/1oozs5r
@r_devops

Live Coding session for the community. Who is in? (Beginners friendly)

Wanted to give something back to the tech community, so I’ll be hosting a live coding session with cameras and mics on. Been coding for 12+ years, and the last 3 fully into AI.

We’ll code together, learn, talk about workflows, answer questions, and just have fun with it.

Tech stack (most probably):

n8n
Airtable
Apify
OpenRouter

Interested in joining?
Drop a comment saying interested or whatever you want <3
=> We’re organizing everything in a WhatsApp group to pick the best time.

Oh and yeah… the call is FREE of course.

P.S. - yesterday’s session was f****ing amazing and super fun :-)

Talk soon,
GG

https://redd.it/1op040q
@r_devops

Reduce CI CD pipeline time strategies that actually work? Ours is 47 min and killing us!

Need serious advice because our pipeline is becoming a complete joke. Full test suite takes 47 minutes to run which is already killing our deployment velocity but now we've also got probably 15 to 20% false positive failures.

Developers have started just rerunning failed builds until they pass which defeats the entire purpose of having tests. Some are even pushing directly to production to avoid the ci wait time which is obviously terrible but i also understand their frustration.

We're supposed to be shipping multiple times daily but right now we're lucky to get one deploy out because someone's waiting for tests to finish or debugging why something failed that worked fine locally.

I've tried parallelizing the test execution but that introduced its own issues with shared state and flakiness actually got worse. Looked into better test isolation but that seems like months of refactoring work we don't have time for.

Management is breathing down my neck about deployment frequency dropping and developer satisfaction scores tanking. I need to either dramatically speed this up or make the tests way more reliable, preferably both.

How are other teams handling this? Is 47 minutes normal for a decent sized app or are we doing something fundamentally wrong with our approach?

https://redd.it/1op2qri
@r_devops

Building control planes is part of devops

Hi all,

I'm a developer who loves operations. My take on DevOps is that any GitOps solution based on Terraform or Ansible could become a control plane. I think we should write our own control planes instead of gluing together off-the-shelf products, and DevOps engineers are developers with a broader understanding compared to backend engineers.

I've written a library in Clojure to prove my point, and this blog article outlines it.

https://bigconfig.it/blog/demystifying-the-control-plane-the-easy-upgrade-path-from-gitops-with-bigconfig/

https://redd.it/1oozepi
@r_devops

Terraform AWS "Bootstrap" Project

So i've seen a few people recommend a module or separate project that handles "bootstraping" Terraform. I'm still new to TF but from my understanding this would set a local state and create resources when you then migrate the local state to.

What would be a minimal example for this needed? I'm trying to sort of create a "base" bootstrap project for Terraform and AWS.

Seems like for a "base" level module I would only need the s3 resource for storing state, but I am sure there is more I am missing that would be "good to have".

I haven't really used modules, but I am guessing I could use them in some fashion to have a sort of "template" for different aws resources? (IE: I have 4-5 different .net projects that can use the same module?)

Thanks

https://redd.it/1op6xci
@r_devops

Terraform code review tool github

Hi Experts,
Are you using any tool which auto reviews the terraform code? Since our team is growing and lot of changes are coming in daily, I am looking for a free tool which can be integrated with github actions that auto reviews and comment on my PR.

Right now I am trying windsurf bot, since its already been used by developers. Works ok but not the best.

If you all are using any, what are those?


https://redd.it/1op7h8b
@r_devops

PyPIPlus.com 2.0 — explore Python packages better: full dependency trees, reverse dependents, OSV CVEs, licenses, offline bundles

I built **PyPIPlus.com** a tool to explore Python packages in depth and I’d love your feedback. In the past, two of my posts about this project went viral, and the feedback from the community helped shape it into what it is today.

Below is what the site currently does: **PyPIPlus.com** can be used to check a python package dependencies (incl. extras), reverse dependents, OSV CVEs, licenses, health score, purity, and to generate offline ready to install bundles.

Dependency tree: direct + transitive deps, extras, env markers
Reverse dependents: what other packages use this package
Security: OSV CVEs per version, affected/fixed ranges, CSV exports/copy
Licenses: per package and each sub-dependancy in a full tree view
Health score: 0–100 + A–F (last updates, security vuln, docs, etc.. )
Purity: pure-Python vs compiled via analysis wheel tags/build metadata (only marked pure python if the package and all dependancies are pure)
Offline bundles: all wheels + SBOM + licenses, reproducible and air-gapped

Bundle contents:

wheels/ → all dependency wheels
requirements.txt → pinned versions
install.py → universal installer (Windows/macOS/Linux)
sbom.cdx.json → CycloneDX SBOM for security scans
LICENSES.md → license summary for all packages
NOTICE → attribution (when required)

Install: `python` [`install.py`](http://install.py)
Scan: `osv-scanner --sbom sbom.cdx.json`

Live: [https://pypiplus.com](https://pypiplus.com)
Example (flask v2.3.1): [https://pypiplus.com/project/flask/2.3.1/](https://pypiplus.com/project/flask/2.3.1/)

Previous Posts:

If you’re new to the project:

I made PyPIPlus.com — a faster way to see all dependencies of any Python package


P.S: I hope I've added enough value in this project to be useful, my last attempt at sharing it in r/devops received some rough audience. Regardless, any feedback is better than no feedback.

https://redd.it/1op61jy
@r_devops