I built a Python ingestion pipeline to archive Reddit data locally.

I needed a way to archive and analyze large volumes of text data (specifically engineering career discussions) from Reddit without relying on the heavy overhead of Selenium, but using PRAW, cuz duh .

It's an ingestion pipeline (ORION) that runs locally.

The Architecture:

Ingestion: Python requests hitting Reddit's JSON endpoints directly rather than parsing HTML.
Rate Limiting: Implemented a custom delay logic to handle HTTP 429 backoffs without getting the IP blacklisted.
Transformation: Parses the raw nested JSON tree, cleans the data (removes stickies/automod spam), and structures it into linear text/PDF reports.
Resource Usage: Runs on minimal resources (no headless browser required).

It’s a specific tool for a specific job, but I thought the approach to handling the JSON endpoints might be interesting to anyone looking to build lightweight things

Source Code: https://mrweeb0.github.io/ORION-tool-showcase/

It's non promotional an fully open source, munch trho it.

Feedback on the error handling logic is welcome.

https://redd.it/1p8aujq
@r_devops

What’s the right way to deal with a QA team that slows down your workflow?

I am a dev and I’m running into some issues with my QA team. I’m trying to get a clear picture of what’s actually causing them because we keep seeing vague bug reports, inconsistent coverage, and build/test mismatches, and it slows things down more than it should. don't get me wrong, i’m not looking to blame anyone here, I’ve worked with brilliant QA teams before and clearly know how important the role is.

I just want to understand where these breakdowns usually start and how to go about addressing them without creating internal conflict, and what a healthy QA–dev process actually looks like. appreciate everyone's feedback

small ps: please be respectful and contribute productively to the thread.

https://redd.it/1p8em0j
@r_devops

Devops Job Titles question

I used to work for a AWS Ops Center, where mostly we monitored and tracked/recorded alerts thru cloudwatch.

After 2 years with the company they gave me AWS Admin rights, & the developers were not able to trigger the cards in Jenkins themselves since they were not admins, they trusted me to do so. Also the admin rights gave me rights to grant/deny access to instances/databases for developers for a certain amount of time (while they deploy their codes).

Since I do not have any coding background, I see that im not qualified to apply to DevOps positions. However, would there be any other positions i could apply to? Are there more job noscripts out there that are responsible for monitoring? Maybe i can learn how to create these alerts?

Is there a job noscripts for what i was doing? Or would it be worth while to learn the coding since i have experience of how Ci/CD works now.

https://redd.it/1p8f7wb
@r_devops

Which is the most popular CI/CD tool used nowadays?

SO, there are many CI/CD tools like Jenkins, Azure pipelines, GitHub Actions etc., Which one is the most popularly used in current market? I guess it would be GtHub actions based on its ease of use and flexibility. Any other tool apart from these that you can mention here? Thank you

https://redd.it/1p8glxi
@r_devops

Intel SGX alternative needed since they're killing attestation service

Earlier this year Intel announced killing SGX IAS which is attestation service for older trusted execution tech. Deadline April 2025 sounded far but migrations always take forever. Anyone built on SGX started scrambling to migrate to Intel TDX or AMD SEV. Problem was these aren't drop in replacements, APIs different and security models work differently.

I recently dug deep enough and saw an old post complaining about it and started thinking abt it. Back then companies were posting about this everywhere, lots of production workloads was still on SGX cause it was most mature for years and suddenly everyone was rebuilding. Silver lining is that newer stuff actually better with performance improved and less memory restrictions. Thankfully it wasn't just another migration for the sake of it. Still annoying tho when you build critical infrastructure on vendor hardware and they discontinue. Makes you think twice about single vendor dependence.

Wondering after some time passed, how widespread impact of this was how many production systems using SGX attestation need migration?

https://redd.it/1p8ilv5
@r_devops

How do you do CI/CD when you're not allowed to implement any automation

I'm currently looking into CI/CD options for a project I'm on. However, automated CI/CD is blocked indefinitely (even on a local machine not accessible to the Internet). I don't think I'd get approval for a simple Powershell automation either.

What are some ways to do some CI/CD like practices when automation is blocked indefinitely. I can't call it CI/CD or automation or it'll be blocked.

https://redd.it/1p8j5t0
@r_devops

Repository Firewall alternatives needed

Hi all,

I am evaluating the repository firewalls for a self hosted company (because npm)

The alternatives so far are:

Sonatype Repository Firewall
JFrog Curation: this might be the better option capability wise but also more expensive.

Do you use any other tools? Or have anything to say for/against them?


https://redd.it/1p8pee6
@r_devops

Postman Workspace Leaks: When Your API Testing Tool Becomes a Data Breach 📮

https://instatunnel.my/blog/postman-workspace-leaks-when-your-api-testing-tool-becomes-a-data-breach

https://redd.it/1p8tsy0
@r_devops

Zero downtime deployments without Kubernetes

Hey guys,


One of the nicest feature of Kubernetes are zero downtime deployments.


In general, thinking beyond Kubernetes, to have it for a web-based app that responds to requests, we must have some kind of proxy before it. Why? Because at the very moment of deploying a new app version, we cannot take down the previous one immediately; it must be up and running until the new one is ready.

What do you guys use to have zero downtime deployments when you do not use Kubernetes?

https://redd.it/1p8vyxi
@r_devops

Seeking Advice

I have a network administrator degree and want to get into devops. I've been looking at videos on YouTube and getting some experience with my homelab, but it is only taking me so far. I would like to find a bootcamp that has either live instruction or at least will be able to answer questions if I get stuck on a question. If any one in the community can point me in the right direction, that would be great.

https://redd.it/1p8wb9i
@r_devops

Black Friday Deals ?? join the thread !!

Hey there,

I found some interesting deals:

https://kodekloud.com/black-friday-sale

https://labs.iximiuz.com

https://codewithmosh.com/p/all-access

https://tryhackme.com/pricing

https://neetcode.io/pro

https://www.algoexpert.io/purchase



https://redd.it/1p8vmmr
@r_devops

Nexus choked to death

A funny incident happened today at my workplace. For context, our company enforced pulling from public repo strictly through Nexus proxy.

I had finish with hardening AL2023 minimal with Nexus proxy configured. Who would've thought DNF 3 packages during build stage would brought down our Nexus server which never happened until today. The platform guy thought "Huh guess it's time to scale up". He did with 16 vCPU and 64 GB of memory. Same thing happened. He couldn't believe it and like "Aight imma get Sonatype support for this".

Not long after a devops guy called me just want to see it live + trying to blame my dockerfile. He noticed there's a for-loop though it only disables repo lists except for Nexus's. I build the image to prove it to him and lo and behold, Nexus server died in front of his eyes. He laughed in disbelief for a good minute there.

In the end, he asked me to rebuild again so he can record and show to support.

Not sure what happened tbh but it's pretty funny ngl.

https://redd.it/1p8zgrd
@r_devops

Anyone here taken the CNPE (Cloud Native Platform Engineer) certification?

Hey all,


The CNPE certification is now available, and I’m curious, has anyone here taken it yet?
What was your experience? Difficulty level? Worth it for platform engineers?

Would love to hear your thoughts before I go for it.

https://redd.it/1p918pk
@r_devops

What is costing you money, sleep or sanity?

As the noscript suggests, I’m looking for “hair on fire” problem that you guys experience when dealing with the cloud or your hosting provider.

I’m not looking to make a startup of the problems you share. You see I am a SRE/DevOps Engineer with a YouTube channel around playing with infra. And I’m looking for ideas… I hope that wasn’t promotion btw !

https://redd.it/1p96q2k
@r_devops

Our AI coding workflow is creating tech debt faster than we can ship

Our team's been dealing with the usual AI coding mess, works fine in isolation, but the overall architecture is held together with duct tape. Nobody understands the systems we're building anymore.

Started trying something different: use AI to generate architecture plans first (been testing socratesai.dev), then feed those to Cursor/Claude for actual code. Idea is to separate the "what should this look like" from the "write the code." Too early to tell if it actually works or if I'm just adding extra steps.

But at least the Terraform modules are following some kind of pattern now instead of each PR being a creative interpretation of our standards. Anyone else trying to wrangle AI tools into following actual architecture? Or have you just accepted everything needs heavy refactoring anyway?

https://redd.it/1p9axg4
@r_devops

How did your team handle the Bitnami paid changes?

Joined a company earlier this year without knowing that they had been kicking the can down the road RE: how to handle Bitnami's change to paid for images.

The guy who hired me left and now I need to come up with an opinion on what to do.

How did you end up approaching?

https://redd.it/1p9cfoz
@r_devops

Hardcoded API Keys: The Rookie Mistake That Costs Millions 💎

https://instatunnel.my/blog/hardcoded-api-keys-the-rookie-mistake-that-costs-millions

https://redd.it/1p9eqld
@r_devops

how to find good open source projects?

Hello beautiful people,

Recently I got fired as a Cloud Support Engineer, I have only one year of experience and wanna boost my resume doing open source projects. As far as I've heard it's a good way to showcase your skills. I'll still do separate projects.

How to find good beginner-friendly projects and how to participate? What should I search for? Already asked GPT but thought it's a cool idea to ask here as well. Thank you in advance.

https://redd.it/1p9gwxf
@r_devops

Advent of DevOps?

..is there such a thing, that anyone can recommend? Kubernetes, Docker, containerization, etc would be great. Linux, networking, noscripting, etc too.

https://redd.it/1p9lw5e
@r_devops

Ephemeral Infrastructure: Why Short-Lived is a Good Thing

https://lukasniessen.medium.com/ephemeral-infrastructure-why-short-lived-is-a-good-thing-2cf26afd75ef

https://redd.it/1p9rzfs
@r_devops

Hardware for build agents

I'm considering whether it will be better to get multiple small low powered machines or a single high spec machine to run builds. I'm currently using Teamcity community so limited to 3 build agents. The build artifacts will all be docker images. I was thinking of a quad core i3 with 8gb ram for each agent which I can get for quite cheap. Alternatively will have to spend a bit more to build a higher spec machine with maybe an i5 with 32gb ram. Which do you think is better?

https://redd.it/1p9uj0l
@r_devops