spent 3 hours debugging a "critical security breach" that was someone fat fingering a config

This happened last week and I'm still annoyed about it. So Friday afternoon we get this urgent slack message from our security team saying there's "suspicious database activity" and we need to investigate immediately.

They're seeing tons of failed login attempts and think we might be under attack. Whole team drops everything. We're looking at logs, checking for sql injection attempts, reviewing recent deployments. Security is breathing down our necks asking for updates every 10 minutes about this "potential breach." After digging through everything for like 3 hours we finally trace it back to our staging environment.

Turns out someone on the QA team fat fingered a database connection string in a config file and our test suite was hammering production with the wrong credentials. The "attack" was literally our own automated tests failing to connect over and over because of a typo. No breach, no hackers, just a copy paste error that nobody bothered to check before escalating to defcon 1. Best part is when we explained what actually happened, security just said "well better safe than sorry" and moved on. No postmortem, no process improvement, nothing.

Apparently burning half the engineering team's Friday on a wild goose chase is just the cost of doing business. This is like the third time this year we've had a "critical incident" that turned out to be someone not reading error messages properly before hitting the panic button. Anyone else work somewhere that treats every hiccup like its the end of the world?

https://redd.it/1neixob
@r_systemadmin

A hard lesson was learned this week.

On Monday, I logged in at 8:00am like I normally do with my full cup of coffee ready to tackle the day. What I came to find out later that morning what happened ruined my week.

In our environment, we utilize Privileged Identity Management to grant us the Global Administrator role on a need basis. Now going back in time a couple months in June, we shifted all of our Microsoft 365 licenses from E5's to Business Premium and Business Basic. I stressed to senior management it needed to happen - being it was a huge waste of money since we didn't utilize all of the features. Inevitably, those licenses expired as they should of. This ended breaking PIM because I didn't take into realization that we needed additional Entra ID P2 licenses for PIM to work. Boom, PIM is broke. No big deal, right? I'll just login to our break-glass global admin account and temporarily assign us the global admin role while we work on fixing PIM. Little did I know that our global admin account was in a disabled state and we didn't have the password on file.... Thus - unable to do anything in our 365 tenant.

There was a hard lesson learned here today.... To all of you 365 admins out there, ensure you have a break-glass account, and you are able to log in.

Thanks to my stupid mistake for not checking on this, I am now waiting on Microsoft 365 Data Protection services to unlock and reset the password - and we all know how Microsoft support can be sometimes.

Once we can get logged back in, I am making sure that this never happens again and it's going to be apart of our DR testing every quarter, making sure we have the password, and we can get logged in.



https://redd.it/1nejbri
@r_systemadmin

Going to crash out over AutoDesk -SEND HELP

I work for a school district and we use SCCM still. We are moving to AutoDesk 2026 from 2023. It took a consultant to figure out an install application in SCCM. We now need to figure out how to uninstall AutoDesk from computers with SCCM.

I can’t figure it out. I followed the steps that AutoDesk lists for a clean uninstall and noscripted them all in PowerShell and then some. Nothing I do gets it to actually fully uninstall. I try deleting every folder I can find, but nothing gets rid of the icons. I noscripted the deletion of registry keys, every uninstall.exe that I can find, all the adskuninstallhelper.exe that I can find, deleting all the folders. IT WONT GO AWAY.

Does anyone have experience with this? I figured the steps for a clean uninstall would make it work. Also, why the hell does AutoDesk not make this fucking easier- I mean I am going to lose it.

https://redd.it/1nel5if
@r_systemadmin

After almost a decade of recovery, I'm back to being a sysadmin and I think I like it...

I thought I'd finally recovered and managed to fully join the ranks of recovered sysadmins when I finished my PhD and was made redundant from the software house I was worked for. Honestly it was a bit of a relief as I'd been ramping things down while I was studying - I'd gone from network administration to remotely babysitting the monthly M$ patch cycle for the servers we couldn't tolerate unplanned downtime on. Really I wasn't a sysadmin at this point, so I was thankful for the push.

I embraced the fresh start in academic life and jumped into research, working on a series of projects where the only admin I was doing was my own systems. No demands, no users, no on-call. Aside from the subtle battles with university IT to get what I needed (Yes I really do need that many systems, yes I do need IPv6, no you can't take my network ports...), life was bliss. Someone else was responsible for managing the big compute, I was "just" a user.

Then I made a mistake. As I moved up the greasy pole of academic positions, I started planning research and was pulled into teaching. Given my background, networking and computer architecture were the obvious specialities. Given how esoteric and experimental some of the technologies are, no one else knew how to manage them so I ended up admining a couple of systems with some fun FPGA accelerators in them. No big deal I thought, a little bit of automation and I can make this pretty painless.

That was a bit over three years ago and as you are probably expecting because I'm posting here, it didn't stop at a just a couple of systems. As the frequency of posts on alt.sysadmin.recovery diminished, my admin responsibilities increased. My colleagues realised I knew what I was doing and could get things done with University IT that they couldn't, and now I'm now responsible for managing multiple compute clusters that support several million $ of academic research. The sort of systems that corporate university IT don't want to touch with a barge pole, but are needed to make the research and teaching happen.

The shift back to being a sysadmin was inevitable I suppose, but the difference between then and now is that instead of business-critical Windows servers, I'm managing Linux systems with esoteric hardware that's held together by custom drivers I have to maintain. What does the future hold though?

University IT seems to go through cyclical phases of being more and less corporate. When it gets more corporate, the shadow IT run by academics increases, coalescing on a few who try to do it properly. My experience placed me perfectly for this downfall, but how far am I going to fall? Departments may even end up with their own pseudo-IT team to work around the central bureaucracy, only for these teams to be subsumed by central IT when it goes through a phase of being less corporate. Unfortunately the pendulum swings the other way and as things get more corporate, and the people who get pulled in like this often leave as the transition happens and they are tasked with more mundane responsibilities. Is this my destiny? To be dragged kicking and screaming back into corporate IT as I clutch to the weird and whacky, only to be cast out when I won't conform?

For now I seem to be embracing the life of a sysadmin again. I picked up some stickers at a recent open-source conference, and one of them (Moss in the fire) is proudly stuck on my office door proclaiming my place as a sysadmin. My beard even seems to agree with this path as I've started finding the occasional grey hair, my journey to a greybeard looks to be a certainty.

Despite falling out of recovery, I'm still an academic and I find myself wanting to know the truth: Is permanent recovery possible? Can one ever escape the life of a sysadmin? Or is it just an illusion? Do we become too used to having the power to do what we need to do, struggling to conform with the systems others force upon us, always destined to fall back

into the patterns of old. How many of you have un-recovered after so long?



https://redd.it/1nesbza
@r_systemadmin

Hiring folks: why do you ask "tell me about yourself "

Im always torn on how to respond to this aside from answering it like John madden mixed in with Tony Romo.

What are you looking for? What is ai looking for?

https://redd.it/1neskw0
@r_systemadmin

How do you deal with incident amnesia?

Hey everyone,

I’ve been thinking about this problem I’ve had recently. For teams actively facing multiple issues a day, debugging here and there, how do you deal with incident amnesia? For both major and micro-incidents?

You’ve solved a problem before, it happens again after a span of time but you forget it was ever solved so you go through the pain of solving the issue again. How do you deal with this?

For me, I have to search slack for old conversations relating to the issue, sometimes I recall the issue vaguely but can’t get the right keywords to search properly. Or having to go to Linear to comb through past issues to see if I can find any similarities.

Your thoughts would be much appreciated!

https://redd.it/1nesw3j
@r_systemadmin

Weekly 'I made a useful thing' Thread - September 12, 2025

There is a great deal of user-generated content out there, from noscripts and software to tutorials and videos, but we've generally tried to keep that off of the front page due to the volume and as a result of community feedback. There's also a great deal of content out there that violates our advertising/promotion rule, from noscripts and software to tutorials and videos.

We have received a number of requests for exemptions to the rule, and rather than allowing the front page to get consumed, we thought we'd try a weekly thread that allows for that kind of content. We don't have a catchy name for it yet, so please let us know if you have any ideas!

In this thread, feel free to show us your pet project, YouTube videos, blog posts, or whatever else you may have and share it with the community. Commercial advertisements, affiliate links, or links that appear to be monetization-grabs will still be removed.

https://redd.it/1nez4pk
@r_systemadmin

Blocked password list - does it impact current passwords?

Morning all,

Finally got approval to put a blocked password list in place, recent pentest showed loads of people with the most basic passwords known to man.

Question is, say I add "Password12345" to the blocked password list, does this just impact future passwords going forward, or will it cause problems for any users with "Password12345" as their password?

Obviously I am forcing password changes etc, but just curious as to how the blocked password list works for currently set passwords.


We're Hybrid, so will be set in AD and synced over to 365.

https://redd.it/1neyiry
@r_systemadmin

C-suite has 12,000 Outlook folders and Outlook is eating a whole i7 alive

One of our execs has built his “system” in Outlook. The result:

12,000 folders
\~90,000 emails
50GB OST
Cache already limited to 6 months

Every 3 minutes Outlook Desktop spikes CPU to 100%, happily chewing \~40% of an i7 with 32GB RAM while the machine sits otherwise idle. This seems to close down other programs, making the computer basicly useless.

Normal exports die (even on a VM). Purview eDiscovery is the current desperate experiment. He refuses OWA. He insists on Outlook Desktop.

I feel like we’ve hit the actual architecture ceiling of Outlook, but I’m still expected to “fix it.” Has anyone here ever dragged a setup like this back from the brink? Or do I just tell him his workflow is literally incompatible with how Outlook/Exchange works?

https://redd.it/1nf1sm6
@r_systemadmin

MFA Entra AD - Break Glass Account

Hey guys,

today I received a message that Microsoft is enforcing MFA for Admin-Portals.
Which in itself is nothing new, I already configured CA for every Admin Account.

But the Message itself says, that every Admin needs it and that this rule will overwrite any CA-Rule.

>Notes:

>You can revisit this page to select a future enforcement date up to September 30, 2025 UTC.

>The portal enforcement will bypass any MFA exclusions configured via Conditional Access policies, security defaults or per-user MFA.

>You can determine if there are any users accessing these portals without MFA by using this PowerShell noscript or this multifactor authentication gaps workbook.

If I understand this correctly my Break Glass Account needs MFA aswell then? I always thought this was supposed to be the account to have direct access if everything else fails.

How do you guys do this?



https://redd.it/1nexxtw
@r_systemadmin

How should critical vulnerabilities be handled?

Another subreddit suggested I come here for advice on this.

Backstory:
I know it's probably different from company to company but I'm hoping to get some insight on this process. I'm in a support role for a mid-size company. It's unique in that it's tier 1/2 support but also some system administration. They're trying to squeeze all the work they can from their underpayed employees across the board, but it's getting me some valuable experience so I'm okay with it. For the most part. The Sr System Engineer is "retiring" soon. He wants to go 1099 and only work 20 hrs a week on certain projects. He's trying to unload this work on me in preparation of his retirement. I don't have an engineering background. Quite the opposite. I fell into IT and have no real technical education.

Here's the rub, Security will create Vulnerability Management tickets. It looks like they just copy/paste text from cve.org or Defender. It's usually a lot of information referencing several possibly affected programs requesting an update or patch to the affected program. I'm then expected to go in and update whatever needs to be updated. It usually involves a developer or analyst's laptop with non-standard software. I try to do my best and determine what software needs to be updated but 80% of the time the user will push back saying they don't have it or it will already be updated to the current version. If I don't see it listed in their programs I have to take their word for it. Or, for example, if it involves Apache Commons Text, I don't even know what that is or how to find it so if the user pushes back I have no choice but to take their word fur it. If it's already the current version, I don't what else I'm supposed to do. I can try to use AI for help but that involves a long remote session with the user while I troubleshoot and it rarely ends in success. The retiring engineer (who is actually a generally nice guy) will tell me I need to figure these things out because he's retiring soon and won't be around to do this. I don't feel like I have the education, experience, or knowledge to complete most of these tickets.

I also feel like the Security team is abdicating their responsibility to some degree on this. It's not the first time I've felt this way about Security. When I ask if software is security approved they tell us to search cve.org but when I come back and tell them that it says the program is high risk and I should deny it, they say it's not that simple and other factors need to be taken into consideration but they don't elaborate or follow-up on it. I'm not a security guy. I don't know how to make these determinations.

Is this how it's supposed to work? Am I just supposed to figure it out or just fail at the job? In short (too late for that I suppose, haha) am I the problem?

https://redd.it/1nf57a2
@r_systemadmin

Critical Cursor AI Flaw Allows Silent Code Execution via Malicious Repositories

**Date:** September 12, 2025

**TL;DR:**

* Cursor AI ships with Workspace Trust disabled by default, creating a silent code execution risk.
* Attackers can weaponize malicious repositories to run arbitrary code as soon as a folder is opened.
* Users must enable Workspace Trust and audit repositories to mitigate potential supply chain attacks.

A serious security flaw has been disclosed in the AI-powered code editor Cursor, a fork of Visual Studio Code. The vulnerability allows attackers to execute arbitrary code when a developer opens a maliciously crafted repository. The issue arises because Cursor ships with Workspace Trust disabled by default, which lets .vscode/tasks.json auto-run commands without user consent.

This flaw poses a significant threat to developers and security teams by opening the door to supply chain attacks. Sensitive credentials could be leaked, files modified, or systems compromised. To protect themselves, sysadmins and developers should enable Workspace Trust in Cursor, use alternative editors for untrusted code, and carefully review repositories before opening them.

**Full Story:**

[https://thehackernews.com/2025/09/cursor-ai-code-editor-flaw-enables.html](https://thehackernews.com/2025/09/cursor-ai-code-editor-flaw-enables.html)



https://redd.it/1nf7mhh
@r_systemadmin

Asked to be a guest speaker on IT security for individuals/micro businesses

Hello friends,

A client of mine asked me to be a guest speaker at an event in a very specific trade. Effectively, it's a bunch of micro businesses (1-2 employees), and they want me to offer advice on cyber security/etc.

I've never done this before, do you guys have any tips? She wants a 50 minute presentation but I don't know if I can blather about stuff that long, so I was thinking maybe a 30 minute session covering 6 topics at 5 minutes each, with 20 minutes of questions/answers.

She also asked me how much I would charge for this, but since I've never done this I don't know what to answer. I would think my hourly rate to prepare the presentation and the time to do the presentation.

https://redd.it/1nf6396
@r_systemadmin

MSP fixing vulnerabilities on our network - should fixes be included in our SLA or be chargeable?

It's not exactly clear if they are included in our SLA but you would imagine if our MSP is in charge of setting up and securing our network, that they would fix whatever vulnerabilities they find. How is this generally handled in other orgs who have an MSP? Thanks

https://redd.it/1nf6tf1
@r_systemadmin

Who needs 811 when an excavator can discover all the utilities at once?

I said what I said.

https://redd.it/1nfd1p3
@r_systemadmin

Is it normal that my team demands me to answer phone calls from them when I'm on vacation?

Half a year ago I went on 10 day vacation. Before leaving, I left our Project Manager a message with a quick guide on what was left to do with the project and a note, that she needs to pick someone from the team to continue with the tests.

When on vacation, I was doing tourist things and haven't really paid attention to my phone (also was out of service often). In the afternoon I've noticed few unanswered calls and a message from my colleague, asking about the details of the project - I messaged him, to write to the PM, so she can forward him the note with the guide. Few hours later I've noticed few new messages, where he asks me to talk about the project, so he doesn't have to message the PM. I got annoyed, told him the PM knows every detail and stopped answering.

After coming back from vacation, I got scolded by whole team, that I should answer the calls.

Now, half a year later, I'm going on vacation and my team member asked me how can he contact me in case he needs something.

Is it normal? I honestly wasn't expecting that kind of reaction from the whole team. And it's not some small company with 3 person IT dept - just a regular corporation.

https://redd.it/1nfeeqm
@r_systemadmin

Users storing passwords on personal gmail accounts

I work in healthcare IT and a user told me today that everyone in his department created a personal gmail account to store their work passwords on and that they use the same password for everything. They wanted me to reset their gmail accounts which I obviously don’t have access to do because they made it.

How do you all handle situations like this? I reported this to my manager due to my concern of PHI being accessed. Maybe I did the right thing reporting it but I also am worried that I am overreacting.



https://redd.it/1nfgl3k
@r_systemadmin

Did I do the right thing?

Hi all,

I recently handed my notice in at a job where I felt undervalued and stressed due to the chaotic nature of the business. In the last year I got the "extra" responsibilities of label printers, power BI connections and dashboards, creating and maintaining html apps for the business. All on top of the infrastructure of switches, hosts, storage etc. alongside this I was also teaching new IT recruits.
Small increase of 1.5k pay per year to cover.
This seems like a lot of work but I also think this is maybe the nature of being a sysadmin in a medium business? ~300 employees.
I recently landed a job as an infra engineer instead, for the same pay and a couple more hours a week but for a company with a slightly larger IT team.

I enjoyed the old place because it was varied and I liked most of the people, but I'm running out of steam and they wouldn't hire anyone else that's 3rd line level knowlege to help.

I feel like I've done the right thing, but what would your deciding factors be?

https://redd.it/1nfbdtu
@r_systemadmin

KB5014754 - AD Strong Certificate Mapping Enforcement. What are you doing? Help

I am trying to figure out how to handle this enforcement of strong certificate mapping for smart cards that Microsoft is enforcing next patching.

* Our PKI team uses Entrust and our certs are stored in an LDAP other than active directory so we cannot add the SID stamping from the AD account on their certificates.
* We have 2016 Domain controllers so we cannot use the GPO tuples for strong name based mapping
* Users self-renew their smart card certs any given day so there could be hundreds of newly-issued certificates between newly issued smart cards and renewed certs.

I have been running splunk searches against eventcode 39 and manually mapping the AltSecurityIdentities attribute to their AD account based off the events over the last month.

I need to set up some kind of a sync that connects from LDAP-A and can detect newly issued certificates, pulls the cert serialnumber/issuer, or SKI, whatever attribute we choose, and dumps it into LDAP-B (AD) account's altsecurityIdentities.

Is anybody else successfully doing this via powershell or python or anything? I am NOT a coder whatsoever. Starting to freak out.


[https://support.microsoft.com/en-us/topic/kb5014754-certificate-based-authentication-changes-on-windows-domain-controllers-ad2c23b0-15d8-4340-a468-4d4f3b188f16](https://support.microsoft.com/en-us/topic/kb5014754-certificate-based-authentication-changes-on-windows-domain-controllers-ad2c23b0-15d8-4340-a468-4d4f3b188f16)

https://redd.it/1nfcr26
@r_systemadmin

Exclusive: Vista-backed device management software firm Jamf explores a sale, sources say

https://www.reuters.com/technology/vista-backed-device-management-software-firm-jamf-explores-sale-sources-say-2025-09-12/


Seems like JAMF is going on the market.

A non-payed walled, brief article.

https://www.themiddlemarket.com/latest-news/vista%E2%80%90backed-jamf-reportedly-explores-sale

https://redd.it/1nflm6c
@r_systemadmin