Rant about our predecessors

The Sysadmin before I took over the job earlier this year was always super paranoid about cybersecurity. While we should always be aware, he was paranoid to the point of making the entire company change their passwords and running a full AV scan on the entire network every time one little thing went wrong with his PC, even if he was to blame.

Program crashed? Change passwords, run a scan.
PC automatically rebooted because of updates? reset passwords company wide, run a scan.
A website glitched and "doesn't look right"? reset passwords, run a scan.
He rebooted the PC and it took one minute longer to come back up? reset passwords, run a scan.
(I'm not kidding on any of these)

He went so far as to convince the owner to hire someone to do a full cybersecurity/vulnerability scan and pentest on the network and then spent weeks combing through the results and tweaking GPO's PC and Firewall settings to lock everything down.

So, imagine my surprise when yesterday, I was hunting down a firewall issue with our FortiGate, trying to get a VLAN access to a specific site and service and I was looking for DHCP logs and stumbled into the System Events page for the last 24 hours.

|Top Event|Level|Count|
|:-|:-|:-|
|Admin Login failed|Alert|25,244|
|Admin login disabled|Alert|2,643|

<insert "that's a lot of damage" meme>

Turns out, the HTTP and HTTPS access has been enabled on our external WAN interfaces this entire time. I looked at my first config backups back in March and the setting was there, so way before my time.

Luckily, no successful logins from the outside, but still......sigh.

https://redd.it/1o0c32q
@r_systemadmin

7 days into my new job and i want to quit

Just a rant to this dear community. As you can see from the noscript, here’s the deal: I started at a new software development company as a Senior IT Ops / DevOps Engineer, supposed to take care of the infrastructure with a team of about 10 people. The company has around 10 products, focusing on healthcare institutions and providers.But on my first day, nobody sent me any onboarding sessions or even contacted me. They just gave me a notebook with login credentials, and in there I could find a welcome mail and a default onboarding slot from the COO with all the newcomers.I proceeded to ping what was supposed to be my team lead, and he talked to me for 20 minutes, explaining the setup very poorly and just giving me the link to Confluence where I should read the documentation (less than 10% iss actually documented).I tried to organize myself by contacting the Product Owners for every product to give me a short intro into each, and they annoyingly just sent me invites for three weeks from now, apparently because they are “busy. I then proceeded to bother every name i could associated with the apps and finally got to hte IT support stuff who gave me at least some insight. In the country where I am, I’m the only one from the team here, and nobody in the office even knows what I’m supposed to do and where i should sit ( all the places are full and i am sitting at some confference room table. Apparently, the team is also split into smaller teams, where everyone takes one of the apps and maintains it. I’m supposed to take care of the two apps that nobody wants to deal with. What a chaos.

https://redd.it/1o0dsoq
@r_systemadmin

Is it normal to feel lonely?

Basically the noscript. I feel lonely. I want to talk to people that are interested in the things I'm interested in and progress my skills with the support of a community, but I'm not sure how to do that. Every time I try to interact with people, I feel like a vampire that isn't providing enough value to justify my presence. How do I put myself into a position to where I can interact with people that are interested in the same things as me while still providing value? I haven't had a job(other than freelance web development) in any of the fields I'm interested in, so I feel like that makes it even harder to relate to folks. Am I overthinking this?

I want to provide some context about myself. I thought for about a year that I was going to be a software engineer. It could still happen, but I've started to realize I'm more interested in the technology behind everything, rather than programming as a whole. I don't mind programming and wouldn't be upset if that's where I ended up. I've had a few interviews that didn't pan out, which is to be expected. I think I would really like to be a sysadmin, because my main goal from the beginning was to work in cybersecurity as a penetration tester and it would be cool to see things from the other side. I'm working towards my OSCP right now, but maybe I'm chasing a pipe dream that wouldn't be ideal for me?

Sorry for the word vomit and sorry if this post doesn't make a lot of sense. I'm just a bit lost and needed to write.

edit: Wording

https://redd.it/1o0f3lb
@r_systemadmin

Whats the point of terraform?

At first I thought Terraform sounded great. But now I honestly don’t get why it’s supposed to be so good for smaller organizations. Yeah, you can create VMs more consistently, but you still have to make those VMs manually first to use them as templates. It’s not like Terraform is easy to set up either. You need to create a template, set up SSH keys, configure cloud-init, then clean it up, and maybe even use modules, which just makes everything more complex and adds more maintenance work.
It is not like it makes manual work go away completely. Feels like it just better to invest time in packer tool and use ansible for config management.


I will spend some more time in my free time to learn more about terraform. Maybe I am wrong.

https://redd.it/1o0atyw
@r_systemadmin

Should I send the client a mini-pc so I can work on their network (since they have basically no IT dept?)

Short backstory: I have a client company which has virtually no IT department at all-- just a guy listed as the "help desk specialist". Anyway, I may need to have them run nightly jobs on prem where they do some basic queries to a database which can only be accessed from their network, and then upload CSVs of data to a SaaS which my company manages via SFTP or SCP.

Normally I wouldn't need to do this-- my clients are usually large companies with their own IT that can handle something relatively simple like this. But sometimes I get a client who is very small and outsources all of their IT, so they only keep like one person on-site to fix printers and such.

Anyway-- here's my question:

I see there are mini-PCs on Amazon for as low as $130 - $200. Low on specs, but I wouldn't need much at all for my situation. So, I've been thinking-- I could get one, install linux and configure it however I need, set up appropriate keys, noscripts, cron jobs, etc. Then, I just mail it to them and tell the IT guy to plug it into their network and turn it on (headless, no keyboard, etc). I would connect and work on it through SSH whenever I need to. And I can get the IT guy to physically turn it off or on if I ever need to.

So-- is this a really dumb idea? Are there security concerns I haven't considered?

Thanks for any advice.

https://redd.it/1o0duk4
@r_systemadmin

How do security guys get their jobs with their lack of knowledge

I Just dont understand how some security engineers get their jobs. I do not specialize in security at all but I know that I know far more than most if not all of our security team at my fairly large enterprise. Basically they know how to run a report and give the report to someone else to fix without knowing anything about it or why it doesnt make sense to remediate potentially? Like I look at the open security engineer positions on linkedin and they require to know every tool and practice. I just cant figure out how these senior level people get hired but know so little but looking at the job denoscriptions you need to know a gigantic amount.

For example, you need to disable ntlmv2. should be easy.

End rant

https://redd.it/1o0m82o
@r_systemadmin

Teams meeting AI note taker virus

We use teams to meet with external parties often. Occasionally someone will click on a link in a meeting that says it's an AI not taker. The user just clicks the link out of curiosity. Suddenly that AI is adding itself to every meeting that user is in and then it spreads to the rest of Teams. The one I'm dealing with right now is fireflies.ai. Seems like the only way to get it to stop is go to their site and delete the account. How is it possible that Microsoft would allow a vulnerability like this? Is there not a way to prevent this kind of thing? I have blocked the app as stated here https://learn.microsoft.com/en-us/answers/questions/4429002/removing-fireflies-ai-note-taker-bot-from-microsof
but that doesn't seem to fix the problem of the note taker messaging everyone after every meeting. Any advice?

https://redd.it/1o0njwy
@r_systemadmin

Anyone else getting Entra Connect Alerts today (10/7/25)?

Earlier I got "Password Hash Synchronization heartbeat was skipped in last 120 minutes". I restarted our Entra Connect server even though everything seemed to be running fine. I checked M365 admin center and the password and directory sync are working without errors. Now I get another warning "Health service data is not up to date". Is anyone else getting Microsoft Security emails about this or see it on the Azure portal? Running various powershell cmds and everything seems healthy on my server.

https://redd.it/1o0p1d2
@r_systemadmin

A PowerShell module to help recover from "oops, we deleted C:\Windows\Installer to save space"

So… you (or someone before you) tried to free up disk space by “cleaning” `C:\Windows\Installer` — maybe even ran one of those noscripts floating around that only checks the *Patches* registry keys (`HKLM\...\Installer\UserData\S-1-5-18\Patches`) and deletes everything else.

Congratulations, you just broke updates and uninstalls for half the apps on the server.
SQL Server? Exchange? Azure Arc Agent? Yeah, they’re all crying now.

The [FixMissingMSI](https://github.com/suyouquan/SQLSetupTools) tool can find and repair those missing cache files, but it’s GUI-only and not really practical when you have hundreds of systems.

I built [FixMissingMSI.PowerShell](https://github.com/ITJoeSchmo/FixMissingMSI.PowerShell) to automate that process.

* Runs FixMissingMSI *non-interactively* through .NET reflection (no GUI)
* Collects per-host CSV reports of missing MSI/MSP files
* Builds a **shared cache** that’s *demand-driven* \-- only uploads files that a server has reported actually missing
* Lets you re-run repair jobs after the cache fills so other hosts self-heal
* Includes `Get-InstallerRegistration` / `Remove-InstallerRegistration` for dealing with broken product registrations. Remove-InstallerRegistration is built off of the PowerShell within [Microsoft's Program Install and Uninstall Troubleshooter](https://support.microsoft.com/en-us/topic/fix-problems-that-block-programs-from-being-installed-or-removed-cca7d1b6-65a9-3d98-426b-e9f927e1eb4d) for scrubbing broken MSI registrations (when repair/uninstall is hopeless, this enables a clean install).

Repo: [github.com/ITJoeSchmo/FixMissingMSI.PowerShell](http://github.com/ITJoeSchmo/FixMissingMSI.PowerShell)
PSGallery: [powershellgallery.com/packages/FixMissingMSI.PowerShell/1.1.4](http://powershellgallery.com/packages/FixMissingMSI.PowerShell/1.1.4)

MECM deployment example: [FixMissingMSI.PowerShell/examples/MECM.ps1](https://github.com/ITJoeSchmo/FixMissingMSI.PowerShell/blob/main/examples/MECM.ps1)

Feel free to use, fork, and adapt. If you’ve been bitten by a "cleanup noscript" before, this might save you a rebuild.

https://redd.it/1o0ra4c
@r_systemadmin

I think our public facing IP is getting blacklisted

A few weeks ago a dev at our company thought it was a good idea to write a noscript to check the Apple website for the availability of an iPhone he was looking for. It was a python noscript that hit a web page every 180 seconds and looked for certain keywords. He ran it for a little over 24 hours until it appears Apple started blocking it. The requests were failing with a page not found - 541 error.

At this point he told me about the noscript, he shuts it down, and we move on. I think it's probably not a big deal, and just a temporary IP block or something at Apple.

Ever since then other sites have slowly been blocking traffic from our corp network., and Apple is still blocking -- not the main site, just when you try to put an item in your "bag" to purchase.

New sites that appears to be blocking us are:

>\- Try to open the Sign In page on Costco.com \- This site can't be reached Error - ERR_HTTP2_PROTOCOL_ERROR

>\- Today, try to track a package at UPS.com \- Access Denied - You don't have permission to access "http://www.ups.com/track?" on this server.

We can access these sites without issue if we connect to our guest Wi-Fi, which goes out via a different ISP.

Maybe it's not related, but it sure seems like something is going on. Anyone seen anything like this? Any suggestions to try or resolve?

https://redd.it/1o0nbvo
@r_systemadmin

Got a ticket from a director… couldn’t find him because his Teams photo looks AI-generated from 2004

Just joined a new company this week, still figuring out who’s who and which coffee machine actually works.

Got a ticket from one of the directors, so I thought I’d be proactive and reach out to him in the office.
Naturally, I check Teams to see what he looks like.

Click his profile.. and I’m greeted by what can only be described as an AI-generated headshot from the Windows XP era.
Perfect skin, mysterious blur, warm studio lighting.

So there I am, wandering around the office like a lost intern, trying to match this perfectly airbrushed corporate relic to an actual human.
Spoiler: the real guy looks nothing like that picture. Easily 20 years older


Anyone else notice this trend? Or is my new office stuck in a parallel timeline where everyone still looks like their 2003 LinkedIn profile? 😅

https://redd.it/1o0wpy1
@r_systemadmin

To sysadmins solo or in a small team, what sneaky things do you do that you probably shouldn't?

Nothing malicious or illegal of course, I'm talking minor "workarounds" that you probably shouldn't be doing but do anyway, because you can. Similar to jaywalking, yes you probably shouldn't do it, but it doesn't hurt anyone when you do it.

I'll start, we have a standard password reset policy every 90 or so days, and obviously you can't reuse a previous password. I'll change mine, then use AD to simply revert it back to my original. Before people scream this is a security violation, this is a non-elevated account with zero admin privilege (yes I also understand changing passwords helps against the hash being accessible locally on the machine, but unless you change passwords every few days, it won't matter that much). I wouldn't do this on any privilege accounts (we utilize a PAM solution anyway).

Understandably, in larger organizations, it's harder to "get away" with stuff like this.

https://redd.it/1o0vtkq
@r_systemadmin

Microsoft Simplifies File Transfers of Departing Employees

Microsoft is planning to introduce several enhancements to simplify OneDrive file transfers for departing employees.

Key enhancements include:

Automatic OneDrive access delegation, where access is granted to the manager or designated secondary owner when a user account is deleted.
New filters to help managers quickly identify shared and important files.
An enhanced Move and Share feature that enables bulk file transfers while preserving existing permissions.
More prominent account cleanup notifications, making it less likely for them to be missed.

https://redd.it/1o11o43
@r_systemadmin

Normalize invoicing recruiters for wasting your time.

I have done this twice now and gotten payed.
I am doing this when they insist going forward with a long shot or fishnet recruitment.

https://redd.it/1o14mkf
@r_systemadmin

How do you stop sensitive data leaking in ChatGPT at work?

Hey everyone, need advice please. Lately,In my team, I keep seeing they’re pasting client’s info and internal docs into ChatGPT for quick answers or summaries. The problem is, they’re literally copying and pasting emails, client data and internal docs into it. At first, it seemed harmless but now I’m really concerned. I’ve seen posts like this one where users noticed unexpected chats with their personal info, and this one where someone found internal emails from a real estate agency they never had access to.

I know this can leak sensitive company info, and honestly, it feels like a ticking time bomb. We want to let the team use AI but not risk anything confidential.

I’m trying to figure out what’s the best path

1. Turn off ChatGPT or other GenAI tools completely
2. Let them use but track or monitor what’s being pasted
3. Only allow a few trusted people to use it
4. Make strict rules on what can/can’t be shared
5. Get some tool that secures or governs AI use

I’m 100% sure someone at NASA, finance firms or other professional companies must have enterprise workflows for this. Open to any suggestion

thanks

https://redd.it/1o15g06
@r_systemadmin

Why is everything these days so broken and unstable?

Am I going crazy? Feels like these days every new software, update, hardware or website has some sort of issues. Things like crashing, being unstable or just plain weird bugs.

These days I am starting to dread when we deploy anything new. No matter how hard we test things, always some weird issues starting popping up and then we have users calling.

https://redd.it/1o15s25
@r_systemadmin

What is your happiest moment in I.T.

I see lots of posts in this group that are negative. From users being stupid, High maintenance owners and leadership teams pissing us off or messing things up, and technology just being unenjoyable to work with.
That being said lets here some stories from the community about the awesome moments of this line of work to give people a little bit of happiness and joy.

https://redd.it/1o1caxo
@r_systemadmin

ms entra and 365 admin portals?

We are getting 504 errors. Anyone else?

https://redd.it/1o1f1ud
@r_systemadmin

Open TCP/9100???

I was just asked to forward TCP/9100 so that a vendor can connect to an on premise printer from the outside. This, coming from the customer that claims to take security very, very seriously. Unless, of course, security means they have to use legitimate vendors.

😩

https://redd.it/1o1gug1
@r_systemadmin

So this was a first for me.

# Ever gone searching for help with an issue, only to find your own advice from years ago staring back at you? I just leveled that up in a weird way.

I was searching for an answer to a problem we’ve been having with Windows 11 updates breaking GPO-applied printers, when I happened across a long, well-written post that seemed to describe my issue exactly. I thought

>Finally! An answer!

Unfortunately, it didn’t really offer any solutions, but it did link to its source. That article, in turn, cited “reports on Reddit” as its source.

By this point, I was starting to have déjà vu. Guess what? Those reports on Reddit referred to my own damn post. 😂

That’s a first for me. Of course, I’ve found my own posts from the past past while searching a current problem before. But finding AI slop, sourced by AI slop, sourced by my own posts from the past was a whole new level.

https://redd.it/1o1l996
@r_systemadmin

Story of A Company that Lags Behind

Hello,

I was hired at my current place of employment about 6 months ago. I am proficient with AD and good with PowerShell and all things 365. My experience lies in my past work with the military and Level 3. I cannot share anymore, otherwise, too much personal info is given. Mind you, the company is around 50 to 100 people, and there are only two of us.

When I first started, my company had just gotten through a ransomware attack, and my COO, as well as the IT Manager, wore it like a badge of pride, even though the attackers got all of our data. It should have been a huge red flag to me when I was only two months in and they were bragging about an abject failure. Well, 4 months after this hits, I find that we don't have great backups, by accidentally deleting portions of critical data. A story in of itself and a mistake I admit too. We had to rebuild it all. I was told that it was my fault, but thinking back on it, why is our last good backup 3 months old? A month after, raising the flag on failed backups, which was tasked to the IT Manager, I find that we still have not gotten good backups. well, 4 months go by, and we finally make the switch to a real backup solution, which turns out to be VEAM. It works wonderfully, but the fact that it took 4 months past the initial discovery to build a working backup was mind-boggling. In the meantime, I had brought our original ticket queue that averaged 36 tickets down to 3 in the queue at any given time on top of all this. I have been given an unofficial corrective action due to my role in installing appropriate security measures without process approval. A process that was not written down.

So, I raised the red flag after my IT manager failed to come up with a solution which I had offered in writing 3 options. Thinking I was doing the right thing, as the company relies on us to properly function. Turns out, he is hiding everything we are not doing. I started to push BitLocker; as that was not on our devices and the passwords were not in Intune. We also had no DLP, TPM Delegation, MDM, or MAM; which I eventually deployed them all in the span of 3 months for our 168 devices. Over the course of each one, management requested an explanation and approval process, which was not written down. We have no formal process as we are so small.

In all of this, i have to write a page report for everything, no matter how simple a change. I want to implement basic DLP. Well, i will have to write a report with all of the technical steps on how to do it. Same with any other change that is not as basic as resetting a password. I have a OneNote with everything i do down to the detail, and have even shown my COO. Is this typical of Management, or am I in bad company?

Fast forward to month 6, and I am working on a project as well as handling IT helpdesk and Networking tickets. I have no problem with this and love staying busy. Well, it was a minor project that involved building our company portal applications and pushing them to devices through Intune. Something I have already done for multiple applications that we currently use. I was cussed out by one of our consultants after implementing this particular application, which can easily be reversed in Intune. All for creating a remote installer for software we already use (ShareFile). Well, this one was "not vetted" properly, so I was called in for a one-on-one, which I requested HR for. The COO then decided to come down 30 minutes before the meeting and bring me up to her office, so it became a one-on-one. While I should have held my ground, I did not. Well, the COO lectured me on trust and how i need to do more to earn it from her. She then sent an email explaining how the meeting went, and how my authority to operate will be greatly reduced in the days to come. This, coming from someone who doesn't support practicing industry standards, is aggravating, but she is the boss. I fear that when we do get breached, I will be scapegoated, even though only half of my security recommendations are even being looked