your funniest mistakes

So I recently found that messed up my servers firewall rules slightly. I put the rate limit on new connection inadvertently also on outgoing connections.

Yep, I rate limited my outbound traffic and then was flummoxed by spurious DNS and HTTP requests failing.

Misery loves company so what's your latest "put head in paper bag" error?

https://redd.it/1pd74zt
@r_systemadmin

Are all large corporations a mess like where I work, Mastercard?

So about five years ago the company I was working at was as acquired by Mastercard. Then right when they started migrating last year I was offered a new position at the corporate office that after much discussion with my wife I accepted(discussed as in we had to move across the state).

Let me tell you, this has been an eye opener. We have so much management and so many levels of management it’s nearly impossible to get teams to agree to anything so every team is off doing their own thing that’s often either duplicate work or incompatible other systems (without hacking together a bunch of noscripts). There’s absolutely no ownership, it’s like everyone wants someone else to own the systems and they just want to be told what to do. And software development, it’s like our plan is write code then figure out how to manage it or actually sell it to customers. And the H1B hires, there are entire floors of H1B people, and many of them have so much attitude or trying to argue or avoiding responsibility.

I’ve read articles of large multinational corporations and the dumb shit they do but figured people spiced up the story for attention. Now I’m feeling is it really this bad at all of them.

https://redd.it/1pdlv9w
@r_systemadmin

HUMOR -- I play games to relax.. but this...

Somebody had to do it....

An IT/network infra game ......

I play games to relax... Not to bring work home!!

https://github.com/pshenok/server-survival


https://redd.it/1pdligm
@r_systemadmin

Automating Windows post-installs with a simple winget-based noscript

I’ve been standardizing my own Windows setups for a while, and ended up turning that process into a small open-source tool.
It uses winget + a config file to automate app installs, basic settings, and the usual “first 30 minutes” after imaging a machine.

Nothing fancy - just predictable, reproducible, and easy to adapt for personal or team use.
If anyone has ideas on making it more robust (or cleaner), I'm all ears.

Repo: https://github.com/kaic/win-post-install

Site: https://kaic.me/win-post-install/

https://redd.it/1pdpodg
@r_systemadmin

CVE-2025-55182 - React exploit - brown alert time?

Just reading up on this.... and starting to sweat about the vast quantity of react and react-based frameworks that are impacted from what appears to potentially be an *extremely* simple to achieve RCE... (sent request with some code in it, code runs, the end)


Anyone else sweating? I'm just trying to reverse engineer which customer products/tools/web servers might be impacted and the fastest way to find out/mitigate... Been playing with the React developer tools now but struggling with version profiling the servers.


More info here - CVE Record: CVE-2025-55182


Happy Thursday!



https://redd.it/1pdr09b
@r_systemadmin

Help needed: How do you debug super minimal containers?

We just shifted our apps to min container images, NO bash, NO extra, locked down tight to cut vuln. It’s definitely a big win for security, but devs and ops are lost when something BREAKS.

Zero shell or debug tools inside the container means every fix needs spinning up temp debug pods… really slowing us down!!

Is there any better approach to debug, or should we go back to normal container images since we prioritize speed?

https://redd.it/1pdt1sd
@r_systemadmin

In place upgrade domain controller oh my

Does anyone have anything good to say about going from server 2016 to server 2022 but a domain controller.

Ever boss I had says it’s going to tombstone our whole ad if we do….



https://redd.it/1pdudol
@r_systemadmin

Thickheaded Thursday - December 04, 2025

Howdy, /r/sysadmin!

It's that time of the week, Thickheaded Thursday! This is a safe (mostly) judgement-free environment for all of your questions and stories, no matter how silly you think they are. Anybody can answer questions! My name is AutoModerator and I've taken over responsibility for posting these weekly threads so you don't have to worry about anything except your comments!

https://redd.it/1pdw6w2
@r_systemadmin

Have we hit rock bottom for tech support yet?

I don't know if this is just me getting old. But I feel like the standard for tech support is at an all time low at the moment.

Over the past year I've had to raise cases with vendors & manufacturers & it just gets more & more painful. It seems that we've gone from

support being generally good > support being generally bad > lucky if support even know about the product > lucky if support will even attempt to address the issue insead of asking you to re-raise with another team.

Naming & shaming a few:

Microsoft (obviously): Like most IT operations worldwide, we use more than 1 MS product. Sometimes we use (wait for it....) more than 1 MS product at a time. But good luck raising a case with MS. As soon as they find out your using another MS product, or even the same product but a different version. Case closed, please do the needful & re-raise.

& yes that's with the top tier MS support.

Broadcom: It used to be the case that VMWare support was helpful. Now, the general level of knowledge on the support teams is shocking. Getting answers to basic questions can take weeks in some cases.

Cisco: I have an account issue with Cisco. 2 transfers later I'm still not with the 'right' team that can help me.

MSI: Personal one this time. Bought a new monitor last year & it's already broken with a failed LED. Product is under warranty but MSI won't repair because I don't have the origional box the monitor came in...

I know we're in an 'expensive IT' era where tech firms are slashing costs to compete on AI. Or maybe it's just because so many of these firms are quasi-monopolies.

But surely it can't get any worse?

Right....

https://redd.it/1pdxp69
@r_systemadmin

Cold calls are one thing. Unsolicited meeting invites are a whole new level of unacceptable.

I'm having to clear multiple pending appointments from my calendar every week because these shitbirds think it's acceptable to just send unsolicited meeting invitations.

Christ, I hate salespeople...

Rant over.

https://redd.it/1pdz65u
@r_systemadmin

Just got my cease & desist letter from Broadcom

Title. Small manufacturing company with an on prem setup & 6 vms. We are about done swapping over to hyper v, the Broadcom quote for a 1 year renewal for us was 25k, three years ago we renewed for 5k, absolutely crazy. Luckily I knew ahead of time the quote was going to be outrageous thanks to other posts in this sub, now to finish the upgrade before the 10 day deadline. Happy Thursday!

https://redd.it/1pe06o8
@r_systemadmin

Proxmox Datacenter Manager in stable version 1.0 available

« Proxmox Datacenter Manager is an open-source, centralized management solution to oversee and manage multiple, independent Proxmox-based environments. It provides an aggregated view of all your connected nodes and clusters and is designed to manage complex and distributed infrastructures, from local installations to globally scaled data centers. With multi-cluster management it enables management like live migrations of virtual guests without any cluster network requirements. »

Announcement post : https://forum.proxmox.com/threads/proxmox-datacenter-manager-1-0-stable.177321/
Release notes : https://pdm.proxmox.com/docs/roadmap.html#proxmox-datacenter-manager-1-0

https://redd.it/1pe10v0
@r_systemadmin

Phishing simulations helping ?? harming, or just annoying people?

We all know why they exist ...phishing is exploding, and no tool can catch everything.
But in real life? Some teams say simulations actually help. Others say they just frustrate people and break trust.....and there’s no decrease in click rates.

What’s your experience? Helpful, harmful… or just annoying?



https://redd.it/1pe25zc
@r_systemadmin

Microsoft being... not cheap??

Advancing Microsoft 365: New capabilities and pricing update | Microsoft 365 Blog

Am I reading this right, that they're now going to include some of the InTune suite capabilities as part of the M365 E5 licenses? Remote app, enterprise app management etc.? Has anyone had experience with those add-ons? The pricing for them previously was extortionate compared to 3rd party options.

https://redd.it/1pe3r94
@r_systemadmin

Does the “I feel dumb every day” phase ever end?

Looking for perspective - posting on a throwaway account for obvious reasons.

I’ve been in a new sysadmin role for a bit, working on a big project I’ve been labbing and POC testing for several months. The tech is somewhat interesting, but I’m realizing I don’t think I enjoy the work of actually building things. My previous job was mostly analyzing and monitoring. This one is all about building, architecting, and being responsible when something breaks, and I’ve been having a hard time with that transition.

I know I’m in a good situation and many on here would kill for problems like I have. I also know I can’t just shift careers and make the same amount, which adds even more pressure.

The part I’m struggling with most is that I want to be competent and confident, but the path to get there feels overwhelming. I feel dumb every day. It’s always “why won’t this box talk to that box” or “why did this work just now and now it doesn’t.” The stress of being responsible for a large network makes it worse, and the frustration makes it hard to study, hard to learn, and hard to stay motivated.

I’ve realized that confidence doesn’t actually come first — confusion does — but sitting in that confusion and frustration day after day is incredibly draining. I keep telling myself that growth is supposed to feel uncomfortable and that maybe the only way out is through, but right now it just feels like I’m constantly behind everyone else. The voice in my head tells me that they're regretting hiring me.

I don’t really click with my boss either, which adds its own layer of stress - I don't feel supported and left on my own.

I know this might sound like whining, but I’m genuinely looking for perspective or encouragement from people who’ve been in this spot. Did you go through this phase and eventually grow into the role? Did the constant “I feel dumb” feeling ever ease up? Did moving from monitoring to building click eventually? Or did you realize the work just wasn’t a good fit?

I’m trying to figure out whether this is normal growing pain or if I should be rethinking my path before I burn myself out.

Any insight/encouragement would really help right now.

https://redd.it/1pe3zl0
@r_systemadmin

A coworker died yesterday. There's an important lesson that some need to hear.

Yesterday, the company sent out an email announcing that Bill, who had been with the company 20 years, had passed away suddenly. Everyone liked Bill. He was one of those genuinely good people.

Today (the very next day), we had our annual catered lunch. The entire staff was there. And not a single word was said about Bill. No moment of silence. No acknowledgment at all. For 20 years this dude had been there, and everyone just went on with their day, eating and chatting like nothing had happened. Like the guy never existed.

We spend 40 hours a week with our coworkers, that’s more time than many of us spend with our own families. And I want to be clear: I like where I work. It’s a great place. The pay is good, and the people are solid. Truly.

But this really drove something home for me. As the holidays come up, management LOVES to pile on projects for IT on these days when everyone is gone, system upgrades, network changes, “quick” maintenance windows. And I say fuck. that. noise.

No one will remember you swapping a phone system on New Year’s or replacing a UPS on Christmas Eve. But your family will sure as shit remember you not being there. And if you die suddenly on a Monday, odds are your coworkers will have moved on by Wednesday.

Say no. Stand your ground. Protect your time. Be with your family.

https://redd.it/1pe971k
@r_systemadmin

What tools did you use to automate onboarding?

Onboarding for us, and some of you I’m sure, is a very annoying, labor-intensive process, all because there is very little automation.

For the past year as a back-burner side project, I’ve been gathering requirements from each department that touches the new hire process in any way.

At this point, I’m just blind to my options because I’ve never done this before in my career. In my research, I am considering Power Automate and set up as may triggers and dependencies as I can, and leave certain things to manual process, but other than that, I have no direction or knowledge of the COTS solutions out there.

What do you do for onboarding? I’m not looking for what happens during your personal business process. I’m asking specifically about what tools and solutions worked for you in your org? Hoping to get some traction and places to look.

https://redd.it/1pe09dz
@r_systemadmin

Sys admin sucks update

Prev post

I was going to post this update sooner as I recently walked out one day due to harrassment.

This rant will include things that I have heard or that a colleague has heard.

storage of plaintext passwords for crucial staff members

you require AD to run a simulated phishing campaign through email

Scripting is not allowed as it'll automate us out of a job. "Scripting isn't allowed because there's no way to know if it worked." (I noscript anyways)

It isn't possible to have a netlogon noscript not include their password in plaintext

"You can't be expecting these changes to happen right away it takes time" you've been working on AD for how long? there is no progress.

in my interpretation, privacy law violations. (plaintext passwords)

no longer required to use 2/3 of the programs I described in my last post

So far I've heard an IT guy at another organization receive more on the job training from the sysadmin than I have (not that I want to learn anything from this guy anyways)

One of my colleagues set up AD for one of our departments and the sysadmin convinced a higher up that we "weren't ready" for AD and then he got paid overtime to delete the entire server and rebuild it from scratch with local accounts.

There was a day where he had a 30 minute rant about AI hacking your pc and uploading everything if you use it once (chatgpt, copilot)

"Hackers are in the cloud, so we don't recommend storing anything there."

If you get "hacked" through your email on a work laptop you have to let him wipe your personal phone if you at any point logged into your email on your phone or if you even use teams.

He does not wipe work laptops when they've been infected, just runs virus scans.

I'm just collecting a paycheck at this point and have mentally checked out. There is still so much more but this is more of the current stuff.

https://redd.it/1pe6291
@r_systemadmin

Replace Server 2008 DC with Server 2025?

EDIT: Great news! We convinced the customer to terminate the old domain with extreme prejudice and just create a new one. Every single employee was a domain admin on the old domain and there were tons of other problems with it. Win-win.

Am I fucked? Everything I'm seeing says I literally have to install a temporary 2012 server first.

The 2025 server won't promote because the forest functional level is too low. The 2008 functional level says it is as high as it can be.

Do I really have to do a temporary server?

edit: because I have a tiny amount of pride, this is a customer. I've done some stupid shit, but I take zero responsibility for having a 17 year old DC.

https://redd.it/1pe8955
@r_systemadmin

Adobe Acrobat Pro 2020 end of support Nov 30 – new signed PDFs already broken, need cheap 2024 perpetual / term license alternatives fast

Inbox on Dec 2nd ruined my week.

380 seats of legit Acrobat Pro 2020 we bought outright back in 2020.
Adobe email hits: “EOS Nov 30 2025, your installs no longer validate ISO 32000-2 signatures. New signed docs already show validation errors.”

Every single contract or invoice we get now opens with the giant yellow “SIGNATURE VALIDITY UNKNOWN” banner. Legal is losing their minds, compliance audits looming.

Adobe quote to stay legal:

$72k one-time for 2024 3-year term licenses
or $90k+ yearly subnoscription forever

Foxit pilot was a disaster, redaction sucks.

Anyone found a real volume reseller still moving cheap Acrobat Pro 2024 term / perpetual licenses with proper CLP paperwork? Or are we all just getting forced into the subnoscription hell at this point?

https://redd.it/1pehlel
@r_systemadmin

NSF I-Corps research: What are the biggest pain points in managing GPU clusters or thermal issues in server rooms?

I’m an engineering student at Purdue doing NSF I-Corps interviews.

If you work with GPU clusters, HPC, ML training infrastructure, small server rooms, or on-prem racks, what are the most frustrating issues you deal with?
Specifically interested in:

• hotspots or poor airflow
• unpredictable thermal throttling
• lack of granular inlet/outlet temperature visibility
• GPU utilization drops
• scheduling or queueing inefficiencies
• cooling that doesn’t match dynamic workload changes
• failures you only catch reactively

What’s the real bottleneck that wastes time, performance, or money?

https://redd.it/1pem1yl
@r_systemadmin