Local Admin vs. SYSTEM - Any difference in risk?

I'm looking at two different patch management solutions that seem to have different approach to how it installs (from what I can tell).

Any thoughts? Any meaningful difference in risk?

Product 1: It's a full RMM. Installs as "System" - and there's really no additional information beyond that (that I can tell) from the publicly available docs.

Product 2: It's a dedicated patch management platform. They use a service account - that has:

Read-only access to the Active Directory domain.
Logon as a service right on the local computer. The installer will attempt to automatically grant this right to the specified account.
Membership in the local Administrators group on the server where the Deployer service resides. You can add a dedicated domain account to local Administrators groups manually.
Membership in the local Administrators group on all of your managed endpoints. You can add a dedicated domain account to local Administrators groups manually, with a noscript, or via Group Policy.

And the credentials are encrypted and stored locally for Product 2. Product 1 is devoid of any additional information.

https://redd.it/1pu3d64
@r_systemadmin

Best practice for MFA on local admin accounts on network gear?

Our cybersecurity auditors want us to implement MFA for all local accounts on all our network gear, including routers. While that's relatively easy to do, it does make me wonder how we're supposed to get in if something goes wrong? If our router at our main office loses its WAN connection, for example, how will I be able to log into it and fix it if it can't send an MFA code or communicate with a third party identity provider?

Any known way to get around this? We have a Palo Alto, from what I can see the only supported options for MFA for local accounts are either third party online providers like Okta or Duo, or getting one of those on-prem RSA SecurID appliances, which are call-us-for-a-quote levels of expensive. Maybe that's my only option, but I wanted to check to make sure I'm not missing something.

EDIT: Specifically I'm wondering what happens if someone breaks something, like if one my coworkers edits a firewall rule poorly and blocks WAN access. Or if an update breaks something and needs to be rolled back. I don't want to be locked out of logging in and fixing it because it can't text me code due to the problem I'm trying to fix in the fist place.

https://redd.it/1pu597z
@r_systemadmin

"Just connect the LLM to internal data" - senior leadership said

Hey everyone,

I work at a company where there’s been a lot of pressure lately to connect an LLM to our internal data. You know how it goes, Business wants it yesterday. Nobody wants to be the one slowing things down.

A few people raised concerns along the way. I was one of them. I said that sooner or later someone would end up seeing the contents of files with sensitive stuff, without even realizing it was there – not because anyone was snooping, just overly permissive access that nobody noticed or cared enough to fix.

The response was basically – "we hear you." And that was it.

Fast forward to last week. Someone from a dev team asked the LLM a completely normal question, something like – can you summarize what’s been going on with X over the last couple of weeks?

What they got back wasn’t just a dev-side summary. Around the same time, legal was also dealing with issues related to X – and that surfaced too. Apparently, those files lived under legal, but the access around them was way more open than anyone realized.

It got shared inside the team, then forwarded, and suddenly people from completely unrelated teams were talking about a legal issue most of us didn’t even know existed – and now everyone is talking about it.

What’s driving me insane is that none of this feels surprising. I’m worried this is just the first version of this story. HR. Legal. Audits. Compensation. Pick your poison.

Genuinely curious – is this happening in other companies too? Have you seen similar things once LLMs get wired into internal data, or were we just careless in how this was connected?

https://redd.it/1pu79cx
@r_systemadmin

Anyone else been getting threatening letters from Broadcom?

Hi all

Just wanted to see if Broadcom has been sending you guys hate mail on VMware licensing? We purchased perpetual copies of VMWare 7 back in the day, then renewed to subnoscription (you were forced to) now they are trying to say that version 7 somehow transferred into their subnoscription model.

News flash is that we never upgraded to version 8 and now off of their shitty product thankfully.

https://redd.it/1pu7upy
@r_systemadmin

VMware to Hyper-V, Cease and Desist

Wow.... what a ride it has been. We started the process of migrating about 100 virtual servers across three vSphere clusters to Hyper-V clusters back in August. Finally shut down the last ESXi host a few weeks ago. Our licenses expired on December 20th and today, the 23rd, a cease and desist from Broadcom landed in my inbox. Gladly signed the form stating I've removed the product and sent it back.

To any other sysadmins dealing with this right now, stay strong! Onward to Hyper-V!

Or Proxmox ;)

https://redd.it/1pua2o8
@r_systemadmin

Preparing for the VMware VVF/VCF renewal? Watch out for the Core Floor and vSAN TiB math.

Hey folks,

I’ve been deep in the weeds auditing our clusters for the upcoming 2025 VMware renewal. Now that we’re moving from perpetual sockets to the Broadcom subnoscription model (VVF/VCF), there are two specific "gotchas" I’ve run into that can seriously mess up a budget if you aren't careful.

1. The 16-Core Minimum "Floor" Broadcom requires a minimum of 16 cores per physical CPU. If you’re running older hardware with dual 8-core or 12-core chips, you are still billed for 32 cores per host. This "ghost cost" is a major OpEx jump for smaller environments that were previously socket-heavy.

2. The vSAN Ennoscriptment Gap The difference in storage ennoscriptments between the tiers is massive:

VVF: Includes 100GiB per licensed core.
VCF: Includes 2TiB per licensed core. If you have high storage density but low core counts, the "Add-on TiB" SKUs for VVF can actually make the full VCF stack cheaper.

How I’m Auditing This: Don't rely on manual counts. Use PowerShell 7 (PS 5.1 throws too many errors with the modern modules) and the Broadcom audit noscript.

Get-FoundationCoreAndTiBUsage -DeploymentType VVF Get-FoundationCoreAndTiBUsage -DeploymentType VCF

I've built a logic map and a web estimator to help my team visualize the "VVF + Add-on" vs. "VCF" break-even point. I'm happy to share the link or the raw logic if anyone is currently stuck in spreadsheet hell trying to justify these numbers to their CFO.

Curious if anyone else has found a "sweet spot" for core-to-storage ratios that makes VVF still make sense on larger clusters?

https://redd.it/1pua2ay
@r_systemadmin

NTFS Permissions

Hoping someone has insight on this problem because it is not making any sense to me. I am trying to setup up permissions so that users cannot rename a folder. I disable inheritance, set the user group to read only for (this folder, subfolders, or files), and any user is able to rename the folder. If I change to (subfolders and files), then users are not allowed to rename but they also cannot open the folder. How is it then when I try to apply read permissions to (this folder), the user with these permissions applied can rename the folder?

https://redd.it/1pu6c50
@r_systemadmin

Is there any backup software option that hasn’t gone completely off the deep end with pricing?

Local Gov IT here, on the hunt for a new backup software for better visibility and Linux support. I have 5 VMs on a single HA host pair and 4 job-specific “servers”, each with <500GB data, and a Synology SAN with ~25TB total data. Primary backups are on-prem to a separate building on the same property as my MDF, plus weekly (soon to be twice-weekly) runs to removable drives which get stored off-site.

Talked with Acronis and Veeam, and they’ve both apparently lost all touch with reality and basic common sense. Apparently it somehow has become accepted practice to charge by total data capacity even for on-prem? Not sure how the software or support team is doing anything different for 10GB or 10PB, but the quotes I’m getting of $4k/year and up are just ridiculous. Our current software cost around $750 one-time with a 20% yearly maintenance and still works fine 6 years later. I’d glad keep it going except that I now need Linux backup which they don’t offer.

Are there any solid options that haven’t become extortionists in the SaaS price gouging frenzy?

https://redd.it/1pudr0h
@r_systemadmin

So what do you do when people won't listen to you?

What do you do when somebody comes to you with a problem and you try to explain it and they won't listen to your solution. And then they go and try their own idea which doesn't work it just makes me furious like why did you come to me in the first place and ignore my advice. Especially since I've been doing this years longer than you have

https://redd.it/1pue3n4
@r_systemadmin

Automating UI-heavy workflows when APIs aren’t an option...

A lot of internal and vendor-provided systems we deal with still rely heavily on UIs with limited or no API surface. Automating these workflows reliably has been challenging, especially when UI changes or timing issues cause noscripts to break.

We’ve evaluated a range of approaches such as UiPath / Power Automate for RPA-style workflows, TestComplete / Ranorex for desktop and hybrid apps, and Lightweight image-based noscripting tools for targeted tasks. More recently, we’ve also evaluated AskUI, which works directly off what’s on screen instead of relying on internal UI structure. It’s been useful for certain edge cases, though it’s not something we’d use everywhere.

For other sysadmins dealing with similar constraints, how do you balance automation coverage vs ongoing maintenance? what workflows did you decide were not worth automating?

https://redd.it/1pucub4
@r_systemadmin

Why is sms so hard now

We’re trying to fix tier 0 alerts because slack is too noisy at 3am, but the carrier red tape for sms is insane. our "low volume" 10dlc campaigns keep getting stuck in manual review for weeks.

I’m testing an api that handles the compliance on its end so we can just pipe alerts through instantly.

How are you guys routing priority alerts to your team in 2026? are you fighting carriers or looking for a way to outsource the compliance?

https://redd.it/1puffn5
@r_systemadmin

Would you install a domain controller that isn't needed?

We have multiple domains. A remote site was using OLD domain and had a physical, long past EOL DC. All the DNS, DHCP etc is handled by the network gear - not the DC. Due to the logistics of the site it takes months to get equipment there. A replacement server was ordered ages ago and finally delivered.

But we've since moved all the clients to NEW domain and all are InTune joined. I can't send the server back or reroute it to another site. But as it's been paid for they want it installed, but nobody is clear for what. What would you do? It will do nothing on OLD domain. It will do nothing on NEW domain. Im thinking build it on NEW domain as a server (not a DC) and just let it sit there ( I'll have to patch it, monitor and the rest) with the option to promote if ever needed, rather than for no reason promote it now and introduce unnecessary complexity or risk.

https://redd.it/1puint8
@r_systemadmin

What do you use to write documentation?

This might be a basic question, but it’s something I’ve never seen done really well.

At my last job, we used Notion as an internal knowledge base. It looked good at first, but over time:

A lot of pages went out of date
Information felt scattered across too many places
It wasn’t always clear what was still “authoritative”

I’m curious how teams that do this well actually approach it:

What does your knowledge base include (runbooks, onboarding, decisions, docs, etc)?
How do you keep it up to date over time?
Who owns it?
What tools do you use (Notion, Confluence, markdown, wiki, something else)?
And what have you tried that didn’t work?

Not looking for tool recommendations as much as real-world practices. I’m trying to understand what actually scales beyond the first few months.

https://redd.it/1pui2vj
@r_systemadmin

‘1 engineer, 1 month, 1 million lines of code.’ - Microsoft to Replace All C/C++ Code With Rust by 2030

https://www.thurrott.com/dev/330980/microsoft-to-replace-all-c-c-code-with-rust-by-2030

>“My goal is to eliminate every line of C and C++ from Microsoft by 2030,” Microsoft Distinguished Engineer Galen Hunt writes in a post on LinkedIn. “Our strategy is to combine AI and Algorithms to rewrite Microsoft’s largest codebases.

I fail to see how this could possibly end any way other than amazingly bad.

https://redd.it/1pumbfj
@r_systemadmin

SolarWinds alternatives?

Hi all,

We have just had our renewal quote through for SolarWinds and it has more than tripled in price. This is not something we have budgeted for, and obviously not a business practice we as an organisation should be supporting so I wanted to know what alternatives you are using?

We primarily use it for alerting, monitoring server performance (CPU, Memory, Disk Latency, Network I/O etc). We also use it for application monitors, and pro-active restarting services etc.

Keen to hear your thoughts,

The Fat Fish

https://redd.it/1puleel
@r_systemadmin

mtu rabbit hole

Here's the rabbit hole I am trying to figure out.

\- Application using udp in a k8s pod will sometimes lag really badly even with adequate bandwidth.

\- all physical hosts and links uses 1500mtu. calico is using 1450 (default)

\- tried to increase host mtu to 1550 so that I can change calico to 1500. This breaks k8s host communication...


Why does changing mtu on the physical host break k8s when they are suppose to negotiate the largest size through icmp discovery?






https://redd.it/1puo4io
@r_systemadmin

Please tell me this is not a new trend

Okay so the bank I work at recently implemented a new change. They didn't remove our elevated security accounts, but they removed the admin rights to them. So now when we need to do literally anything that requires any level of elevation whatsoever, we have to go to two different portals.

One portal to request the password to our admin account, and another portal to request the admin access for our admin account.

And this is not a once a week or a once a day thing. Anytime we want to RDP to a server, or even run an elevated power shell command, we have to go through this.

Is this a new trend? Is it time to get out of IT?

I swear to God I will shoot my tits off

EDIT: RDP to a server, not pee on it

https://redd.it/1puu8qr
@r_systemadmin

Sometimes, they really are just stupid

Every time I hear “user X is an idiot” I typically have a conversation like “user X doesn’t have your technical background, that doesn’t mean they are stupid” or “if it wasn’t for people like user X I wouldn’t need your talent” etc.

Naturally I think this too every now and then and have to remind myself of the same thing.

Today, I was listening to an audiobook of 1984 when a user walks in my office. Never mind that my door was closed and I was working on a confidential document, I lock my screen and then pause the book and he says, “That sounded good, what is that?”

I said that it was an audiobook of 1984.

He says, “Is there any way you can send me a trannoscript of that?”

I said what do you mean, a trannoscript?

He says, “Well I don’t like listening to podcasts, but if it’s interesting, I’ll read the trannoscript of it.”

I said you want me to send you a trannoscript of *the book* 1984. He says, “Yes..”

I stared at him for at least five seconds thinking surely it would click and finally I just said sorry, what did you actually need help with and moved on with my life.

I could understand if it was some obscure novel or if I hadn’t said the word *book* a couple times, but this was a first-person experience of some next-level stupidity.

https://redd.it/1puvd85
@r_systemadmin

Pouring one out for the Steam sysadmins on call today

They must have at least a few people who work over the holiday there right?

Clearly they are dealing with some issue. Both downdector and steamstat.us are showing signs of a large outage. Anecdotally the store and partner portal have been acting up for myself for about an hour.

https://redd.it/1puxovk
@r_systemadmin

When you did V2V from VMware to Hyper-V what tools did you use?

Can anyone please tell me a detailed guide preferably for moving 180 Vms from vmware vcenter 8.0 onto hyper-v.

What tools, what methods for V2V did you use?

Details would be appreciated. As for Vms with static IP sql servers how did you move those?

https://redd.it/1pv3u7u
@r_systemadmin

How do you guys train the trainable classifiers for CUI?

So I'm trying to set up a DLP + label + trainable classifiers at my work. We are in Microsoft GCCHIGH environment with no on-prem.

I have tried many times to train the trainable classifers "CUI" to work, but since we do not have a actual CUI documents to work with, it keeps failing. Looks like we need at least 50 positive and 50 negative minimum. I tried generating some fake positive CUI and negatives but it failed...

Any sysadmins or Information Protection Engineers in CMMC space, how did you guys set up the trainable classifiers without using an actual CUI documents?

https://redd.it/1pv4psg
@r_systemadmin