got fired for screwing up incident response lol

Well that was fun... got walked out friday after completely botching a p0 incident 2am alert comes in, payment processing down. im oncall so my problem. spent 20 minutes trying to wake people up instead of just following escalation. nobody answered obviously database connection pool was maxed but we had zero visibility into why.

Spent an hour randomly restarting stuff while our biggest client lost thousands per minute. ceo found out from customer email not us which was awkward turns out it was a memory leak from a deploy 3 days ago. couldve caught it with proper monitoring but "thats not in the budget"

according to management 4 hours to fix something that shouldve taken 20 minutes. now im job hunting and every company has the same broken incident response shouldve pushed for better tooling instead of accepting that chaos was normal i guess

https://redd.it/1ndk9do
@r_systemadmin

RingCentral's Poor Customer Service

Just so others don't repeat my mistake, my recommendation is to avoid using RingCentral.

Pros:
\- Getting signed up was easy and the rep was very responsive during that process. And, for the most part, phone service was OK. But...

Cons:
\- Once you've signed, you'll never reach your rep again.
\- When you have a problem, getting help is almost impossible (especially billing concerns).
\- You're stuck with the number of lines you started with (you can increase, but never decrease).
\- And, when times are tight and you need to cancel service, they make it very difficult. You'll probably miss your window of time to cancel... then you're locked in for a couple more years (over-paying for average VOIP service).

IMPORTANT: If you do choose them, read and understand all the fine print of the contract, because you're locked in for a long time.



https://redd.it/1ndr1fw
@r_systemadmin

Tell One of Your Oops Stories

A couple of jobs ago, I built a SQL query for finance that would fix sales discrepancies at the end of each month. These discrepancies were mistakes and typos done by the closing manager in our retail stores.

Finance would tell me what they were expecting based on the deposits. I was methodical about the process. I summarized all the individual transactions and dumped everything into a temp table for review before writing it to the production table.

I was comparing the production table to my temp to make sure what was already correct, was still correct, and what wrong, is now fixed. I discovered an issue, which happened, so I wanted to make a change and run the query again. Anyway, for some ungodly reason, I decided to manually type "DELETE * FROM TABLE", when I had the DELETE statement already build into my query and all I had to do was rerun it. Anyway, the table name I typed was the production table. I already had the left mouse button pressed to execute the command. The signal from my brain to my right pointer finger to release the button was already sent. I knew I screwed up as my finger was releasing the mouse button. I just deleted the entire production table.

Instant meat-sweats. I panicked for about 10 minutes until I was able to gather my thoughts. In that time, someone from finance called me letting me know that something was now really wrong with the sales data, which I already knew. I'm dreading telling my boss that we have to pull a backup. I then realized I could rebuild the summary table by using a few other tables. It took me about four hours to write a new SQL query to rebuild the table and test the output. The finance employee validated and all was good.

No one in my department ever found out. As far as the finance employee knew, there was already an issue that I was working on.

https://redd.it/1ndx3a8
@r_systemadmin

RIFd after 14 years 355 days.

It happened.

Three hours into my shift in the middle of the workweek my boss is let go, within 5 minutes I get a ping and a meeting invite. I ask when I join if it’s about the boss, or me. It was for me.

10 days short of 15 years. Very different company now, different name a few times over, acquisitions, etc. Very few of the people I initially trained with are left, so it was bittersweet. The mental stress lifted immediately. I can’t feel like a failure when it’s part of a RIF action… but I definitely feel angry, or maybe just annoyed. And a little sad.


I met my (now) wife in the service desk when I was green, found out my son was ready to enter the world during an overnight shift. Grilling with the guys during clean ticket queues overnight. I was 19 and still in college. Now I’m 33, going on 34 in a month.

Haven’t interviewed since 2010, but I’ve been on so many bridge calls, P1 calls, technical discussions and troubleshooting sessions with vendors, carriers, end users, c suite… doesn’t make me feel nervous thinking about the interviews…. But making a resume again? That scares me.

Sorry to post this, it’s not particularly on topic. I just don’t really know how to feel. I know what to do, brushed up linked in, made phone calls to social network and put my feelers out, already have a call with a recruiter tomorrow to discuss some opportunities. Chatted with my wife, agreed we will get through this and she’s been primarily concerned with whether or not I’m okay. Bless her.

I dunno guys. I’m not a technologist, and I don’t eat live and breathe IT. I just like solving problems. I guess I just didn’t foresee having to solve this one.

https://redd.it/1ndzitt
@r_systemadmin

Manager yells at me in front of everyone and makes me do office-boy type work. Is this normal in desktop support?

I work in IT desktop support in a startup (just me and my manager). He’s nice most of the time, but he doesn’t explain things clearly, no documentation, and if I make a small mistake, he yells at me in front of everyone sometimes.

A lot of my work feels like grunt work like carrying laptops, swapping monitors, setting up meeting rooms, etc.

I’m also introverted, so being put in the spotlight is uncomfortable for me.

Is this actually normal for entry-level desktop support, or is my manager just dumping everything on me? And how do I deal with the public yelling part?

https://redd.it/1ne10u2
@r_systemadmin

Im gonna lose my job

I work for a developer of hotel property management. I see the end is near im 56. Sysadmin. Attrition is real both hotels and staff. We are legacy what do i do? We host in aws many properties but im a weird way


https://redd.it/1ndzva5
@r_systemadmin

wish i knew sooner

I was today years old when I learned how to actually use a tool I thought I already knew: SSH.

I stopped doing sysadmin work about two years ago to focus on my own projects. Now that I’m connecting my homelab to my business lab, I’ve started using SSH more and it blew my mind.

Back in my sysadmin days, I saved the day more than once with the CLI because not everyone was comfortable there. I used SSH constantly to configure servers and make changes without touching the web UI (i never read into SSH so never did my homework).

But yesterday I discovered SSH tunnels. Forwarding a remote web UI (like Jellyfin) straight to the machine I’m sitting at… insane!

And today… i not only forwarded a couple of webUIs, shared file systems and being able to browse (I2P) without having to install it machine im using! Got too exited and had to share my thoughts and i will start reading more docs on the tools i use.

https://redd.it/1ndzfh9
@r_systemadmin

Sysadmins: how are you handling M365 retention and backup for small orgs?

Got a couple of 20–80 seat orgs leaning completely on M365 and most of them honestly think Microsoft is just backing up everything for them. Spoiler: nope.
Stuff I keep running into:

Deleted items vanish way sooner than they expect.
SharePoint/OneDrive restores are… painful at best.
Nobody’s thinking about compliance or long-term archive.
And of course, users swear the recycle bin = backup 🤦.
For bigger orgs it’s usually sorted, they’ll pay for a proper tool. But for the small ones with tight budgets, I’m kinda stuck in the middle here.
So what are you all doing? Just cranking up retention policies? Rolling your own noscripts? Paying for something lightweight? Or just praying nothing gets nuked?

https://redd.it/1ndzzet
@r_systemadmin

Thickheaded Thursday - September 11, 2025

Howdy, /r/sysadmin!

It's that time of the week, Thickheaded Thursday! This is a safe (mostly) judgement-free environment for all of your questions and stories, no matter how silly you think they are. Anybody can answer questions! My name is AutoModerator and I've taken over responsibility for posting these weekly threads so you don't have to worry about anything except your comments!

https://redd.it/1ne4qv4
@r_systemadmin

How to stop execs screenshotting dashboards?

i caught a senior manager this morning screenshotting sensitive dashboards and sending them via whatsapp.



https://redd.it/1ne2qjs
@r_systemadmin

Security Operations with AI-Powered SASE

Our company has been juggling hybrid cloud apps, a few on-prem systems, and a remote-heavy workforce. Started looking into SASE vendors earlier this year and noticed every single one now talks about AI as a differentiator.

Some highlight AI-driven threat detection, others say it helps with policy automation or incident response. Hard to tell how much of it is real versus marketing fluff.

Has anyone here actually seen measurable benefits from AI inside their SASE deployments?

https://redd.it/1ne4txn
@r_systemadmin

What’s the biggest pain point of using a bunch of local, ad-hoc IT contractors?

We've had to rely on a handful of local contractors and freelancers to help with our on-site IT needs in different cities. While it's better than nothhing, it's a huge headache to manage. For those of you who go this route, what's your biggest frustration? For us, it's teh inconsistent pricing, the varying skill levels, and the time it takes to find and vet a new person every time we have an issue. It feels like we spend more time managing the people than getting the work done. I'm interested to hear if this is a common experience or if there’s a better way to handle

https://redd.it/1ne84uu
@r_systemadmin

Saved a user from spreadsheet-induced despair by introducing them to the mystical art of "Ctrl+F"

So the user walks in, eyes glazed, soul half-evaporated. They've been manually scanning a 12,000-row Excel sheet to find entries with a specific client name. One. Row. At A. Time. Like some kind of medieval monk transcribing sacred scrolls.

They say, “I’ve been at this for two hours and I’ve only found 8 matches.”

I blink. “Wait… you know you can just press Ctrl+F, right?”

Them: “Ctrl what?”
Me: “Ctrl+F. It opens a search box. Type the name. Boom. It jumps to every match. You can even highlight them all.”

It was like I had just summoned a productivity demon from the depths of the IT underworld.

User reaction: stunned silence
My internal soundtrack: Eye of the Tiger
Time saved: conservatively 3 hours, 1 existential crisis, and a potential keyboard smash

https://redd.it/1ne9c5a
@r_systemadmin

Whats this massive feeling about being inadequate all of a sudden.

Hey all,
I’ve posted here a few times before. I’m currently the sole IT person at a small tech company that focuses heavily on software development and managing databases for clients. It’s been about a year and a few months, and while I’ve learned a lot, I’m starting to feel like I’m hitting a brick wall.

\*\*I think this feeling really sank in after I saw a new DBA we hired speak so confidently and effortlessly with an external client. He was calm, direct, and probably secured a new deal for the company within minutes. Meanwhile, I just sat there thinking, “I could never do that.” I’m not a strong speaker, and I don’t have that kind of presence or self assurance. It made me question whether I’m really cut out for this path, or if I’m just pretending to keep up.\*\*

I’ve been trying to level up into a Junior DBA role (even going through Oracle learning materials/Udemy videos and labs), at the moment ive only built an internal Oracle 19c test environments from scratch (installing on Oracle Linux and install the database on Docker thanks to Network Chuck awsome video on this, configuring pluggable databases, automating backups via RMAN, etc.) but honestly… it’s starting to feel a bit anticlimactic with all the SQL queries i have to remember. I don't know if it's burnout or just the reality setting in, but the idea of grinding out that certification feels less exciting by the day.

That said, I’ve done a ton on my own here:

* Migrated our on-prem infrastructure from VMware to Proxmox VE, including critical production VMs.
* Replaced our legacy OpenVPN setup with modern alternatives (currently testing NetBird).
* Implemented/Coordinate firewall upgrades (FortiGate)
* Contributed to our successful ISO 27001 certification thus handled internal backup policy drafting, logging requirements, and infrastructure documentation.
* Managed AWS cost optimization by cleaning up snapshots, right-sizing instances, and coordinating with dev teams on resource usage.

I’ve been wearing every IT hat you can think of: sysadmin, network guy, backup guy, Oracle DBA-in-training, compliance tech, etc. But i have the feeling that im being seen as just the IT guy sitting and doing nothing and being billable for the company.


Im thinking to search for a position at a bigger company but im having the feeling that it would be the same, or maybe i should directly search for a company that delivers sysadmin like services to other cleints so i can be off site at clients most of the time.


Any one hitting the same wall as me? Man i want to just sit at the beach and watch a nice sunset now....

https://redd.it/1neay9h
@r_systemadmin

Employee passed away, can't open his Access database

An engineer reached out to me to help open an Access database that was managed by an employee who passed away. Said employee was the only one who maintained it and did not leave any documentation about his process. There is no password on the file itself, but when attempting to open the file as the former employee's user, it prompts for a password. We are assuming this is an old, cached password in the database.

I've tried to recover passwords using both Passware Kit Forensics, which finds no passwords on the file, and using Thegrideon Access Password, which was helpful to display the User and IDs, but didn't retrieve any passwords.

Has anyone ever delt with this issue on old Access Databases? We are kind of stuck and I guess this is a fairly important database (although why is there no documentation if it is so important...)

Any ideas would be helpful as I am stuck trying to find a working solution.

https://redd.it/1necrj9
@r_systemadmin

SecureBoot Certificate will expire today September 11th 2025

Microsoft Secureboot signing certificate will expire today, September 11, 2025

When I was checking something for a customer regarding the SecureBoot change in 2026, I noticed that the SecureBoot boot manager certificate for digital signatures expires on September 11, 2025 (tomorrow) on the client. I then checked this on various other clients with different manufacturers and operating systems and found that it was the same on all devices (except those purchased this year). According to Microsoft Support, it could be that these clients may no longer boot up - starting today after expiration.

This fix should apparently resolve the issue, but it is very risky and only works if the latest updates and firmware updates have been installed:

How to manage the Windows Boot Manager revocations for Secure Boot changes associated with CVE-2023-24932 - Microsoft Support

I believe this affects thousands of devices.. Because multiple devices I checked, whether client or server, was affected. Newer Clients (purchased in 2025) and Serves seem to be fine.

Here's how to check:

mountvol S: /S
Test-Path "S:\\EFI\\Microsoft\\Boot\\bootmgfw.efi"
(Get-PfxCertificate -FilePath "S:\\EFI\\Microsoft\\Boot\\bootmgfw.efi").Issuer

$cert = Get-PfxCertificate -FilePath "S:\\EFI\\Microsoft\\Boot\\bootmgfw.efi"
$cert.Issuer
$cert.GetExpirationDateString()

Output:

CN=Microsoft Windows Production PCA 2011, O=Microsoft Corporation, L=Redmond, S=Washington, C=US

Expiring date: 11.09.2025 22:04:07

Has anyone else noticed that?!

https://redd.it/1nedey1
@r_systemadmin

How do you handle service accounts and password rotation? (Disable "password never expires" + longer policy)

We’re running into an issue with our service accounts. Right now, they are all set to **"password never expires"**, which we know is a security risk.
The problem is: as soon as we turn that off, the accounts are immediately forced to change their password — which risks breaking services.

What we’d like to achieve:

* **No more "password never expires"**, but with a **longer password lifetime** than regular user accounts (e.g., 1365 days).

We already looked into Windows LAPS, but that’s mostly for local admin accounts and doesn’t solve this problem for domain-based service accounts.

Curious to hear your approaches — especially how you handled the migration without accidentally taking down services. 🙏

https://redd.it/1ne8p21
@r_systemadmin

spent 3 hours debugging a "critical security breach" that was someone fat fingering a config

This happened last week and I'm still annoyed about it. So Friday afternoon we get this urgent slack message from our security team saying there's "suspicious database activity" and we need to investigate immediately.

They're seeing tons of failed login attempts and think we might be under attack. Whole team drops everything. We're looking at logs, checking for sql injection attempts, reviewing recent deployments. Security is breathing down our necks asking for updates every 10 minutes about this "potential breach." After digging through everything for like 3 hours we finally trace it back to our staging environment.

Turns out someone on the QA team fat fingered a database connection string in a config file and our test suite was hammering production with the wrong credentials. The "attack" was literally our own automated tests failing to connect over and over because of a typo. No breach, no hackers, just a copy paste error that nobody bothered to check before escalating to defcon 1. Best part is when we explained what actually happened, security just said "well better safe than sorry" and moved on. No postmortem, no process improvement, nothing.

Apparently burning half the engineering team's Friday on a wild goose chase is just the cost of doing business. This is like the third time this year we've had a "critical incident" that turned out to be someone not reading error messages properly before hitting the panic button. Anyone else work somewhere that treats every hiccup like its the end of the world?

https://redd.it/1neixob
@r_systemadmin

A hard lesson was learned this week.

On Monday, I logged in at 8:00am like I normally do with my full cup of coffee ready to tackle the day. What I came to find out later that morning what happened ruined my week.

In our environment, we utilize Privileged Identity Management to grant us the Global Administrator role on a need basis. Now going back in time a couple months in June, we shifted all of our Microsoft 365 licenses from E5's to Business Premium and Business Basic. I stressed to senior management it needed to happen - being it was a huge waste of money since we didn't utilize all of the features. Inevitably, those licenses expired as they should of. This ended breaking PIM because I didn't take into realization that we needed additional Entra ID P2 licenses for PIM to work. Boom, PIM is broke. No big deal, right? I'll just login to our break-glass global admin account and temporarily assign us the global admin role while we work on fixing PIM. Little did I know that our global admin account was in a disabled state and we didn't have the password on file.... Thus - unable to do anything in our 365 tenant.

There was a hard lesson learned here today.... To all of you 365 admins out there, ensure you have a break-glass account, and you are able to log in.

Thanks to my stupid mistake for not checking on this, I am now waiting on Microsoft 365 Data Protection services to unlock and reset the password - and we all know how Microsoft support can be sometimes.

Once we can get logged back in, I am making sure that this never happens again and it's going to be apart of our DR testing every quarter, making sure we have the password, and we can get logged in.



https://redd.it/1nejbri
@r_systemadmin

Going to crash out over AutoDesk -SEND HELP

I work for a school district and we use SCCM still. We are moving to AutoDesk 2026 from 2023. It took a consultant to figure out an install application in SCCM. We now need to figure out how to uninstall AutoDesk from computers with SCCM.

I can’t figure it out. I followed the steps that AutoDesk lists for a clean uninstall and noscripted them all in PowerShell and then some. Nothing I do gets it to actually fully uninstall. I try deleting every folder I can find, but nothing gets rid of the icons. I noscripted the deletion of registry keys, every uninstall.exe that I can find, all the adskuninstallhelper.exe that I can find, deleting all the folders. IT WONT GO AWAY.

Does anyone have experience with this? I figured the steps for a clean uninstall would make it work. Also, why the hell does AutoDesk not make this fucking easier- I mean I am going to lose it.

https://redd.it/1nel5if
@r_systemadmin

After almost a decade of recovery, I'm back to being a sysadmin and I think I like it...

I thought I'd finally recovered and managed to fully join the ranks of recovered sysadmins when I finished my PhD and was made redundant from the software house I was worked for. Honestly it was a bit of a relief as I'd been ramping things down while I was studying - I'd gone from network administration to remotely babysitting the monthly M$ patch cycle for the servers we couldn't tolerate unplanned downtime on. Really I wasn't a sysadmin at this point, so I was thankful for the push.

I embraced the fresh start in academic life and jumped into research, working on a series of projects where the only admin I was doing was my own systems. No demands, no users, no on-call. Aside from the subtle battles with university IT to get what I needed (Yes I really do need that many systems, yes I do need IPv6, no you can't take my network ports...), life was bliss. Someone else was responsible for managing the big compute, I was "just" a user.

Then I made a mistake. As I moved up the greasy pole of academic positions, I started planning research and was pulled into teaching. Given my background, networking and computer architecture were the obvious specialities. Given how esoteric and experimental some of the technologies are, no one else knew how to manage them so I ended up admining a couple of systems with some fun FPGA accelerators in them. No big deal I thought, a little bit of automation and I can make this pretty painless.

That was a bit over three years ago and as you are probably expecting because I'm posting here, it didn't stop at a just a couple of systems. As the frequency of posts on alt.sysadmin.recovery diminished, my admin responsibilities increased. My colleagues realised I knew what I was doing and could get things done with University IT that they couldn't, and now I'm now responsible for managing multiple compute clusters that support several million $ of academic research. The sort of systems that corporate university IT don't want to touch with a barge pole, but are needed to make the research and teaching happen.

The shift back to being a sysadmin was inevitable I suppose, but the difference between then and now is that instead of business-critical Windows servers, I'm managing Linux systems with esoteric hardware that's held together by custom drivers I have to maintain. What does the future hold though?

University IT seems to go through cyclical phases of being more and less corporate. When it gets more corporate, the shadow IT run by academics increases, coalescing on a few who try to do it properly. My experience placed me perfectly for this downfall, but how far am I going to fall? Departments may even end up with their own pseudo-IT team to work around the central bureaucracy, only for these teams to be subsumed by central IT when it goes through a phase of being less corporate. Unfortunately the pendulum swings the other way and as things get more corporate, and the people who get pulled in like this often leave as the transition happens and they are tasked with more mundane responsibilities. Is this my destiny? To be dragged kicking and screaming back into corporate IT as I clutch to the weird and whacky, only to be cast out when I won't conform?

For now I seem to be embracing the life of a sysadmin again. I picked up some stickers at a recent open-source conference, and one of them (Moss in the fire) is proudly stuck on my office door proclaiming my place as a sysadmin. My beard even seems to agree with this path as I've started finding the occasional grey hair, my journey to a greybeard looks to be a certainty.

Despite falling out of recovery, I'm still an academic and I find myself wanting to know the truth: Is permanent recovery possible? Can one ever escape the life of a sysadmin? Or is it just an illusion? Do we become too used to having the power to do what we need to do, struggling to conform with the systems others force upon us, always destined to fall back