Daily drift is real

Noticed something recently.

Most tenants I see have small changes happening daily.

Role assignments.

Conditional Access toggles.

Intune settings.

App permissions.

One percent here.

Two percent there.

After six months the environment is unrecognizable.

How do you all track drift without manually comparing JSON dumps?

https://redd.it/1p5avvw
@r_systemadmin

What did you know how to do before becoming a sysadmin?

I am on my journey to become a sysadmin. I have zero actual work experience. I'm 42. Been in manual labour since I was 16 and always felt my calling was working in IT. Finally decided to do it. Haven't owned a pc in 10 years. I brought a pc 6 months ago. Took the conptia tech+ a week later and passed. Took A+ the next month and passed. Took network+ a month later and passed.

Ive been doing everything I think i need to be able to get a junior role or 1st/2nd line support but my end goal is sysadmin. I have a home lab set up and I do regular daily practice when I finish my job (my job is 9-10 hours a day).

Ive learnt to use Linux and Windows server to monitor and manage users/servers. Learnt sql for some reason. Powershell. Excel. I got a m365 business account a few weeks ago and just messed about adding old devices through intune and made some policies.

My whole work life ive dealt with talking to the public and customers. I feel like im ready to get into the world of IT now. Ive applied for tons of jobs but not even an interview yet.

What did you guys know and do before becoming a sysadmin?

Edit: I appreciate all the great replies. This definitely feels like a sub where you're all just there for each other. Good stuff.

https://redd.it/1p5h3ua
@r_systemadmin

Are IT responsible for writing/owning the Business Continuity Plan?

I understand that IT input will be required at stages throughout the plan, but just wondering who is typically responsible for writing/owning an org’s BCP? Does it fall under IT Manager or a role under corporate/risk?



https://redd.it/1p5hqnq
@r_systemadmin

Quality of engineers is really going down

More and more people even with 4-5 YOE as just blind clickops zombies. They dont know anything about anything and when it comes to troobuleshoot any bigger issues its just goes beyond their head. I was not master with 4-5 years in the field but i knew how to search for stuff on the internet and sooner or later i would figure it out. Isnt the most important ability the ability to google stuff or even easier today to use a AI tool.But even for that you need to know what to search for.

https://redd.it/1p5ki7i
@r_systemadmin

RDS Server 2025 - High WMI usage 30%-90%

hi guys (and girls)


I'm troubleshooting an issue for a few weeks now, and feel like i'm stuck.
So I finally decided to aks you guys for any help:)



The Story

We recently upgraded a
customer from an RDS 2016 farm to RDS 2025. The old 2016 servers suffered from
very high CPU load for WMIPrvSE.exe.

When there ware 0 users logged on, the
problem was not there.
When there ware \~ 5 users logged on, it was
not that bad.
When there ware \~ 20 users logged on, it was
absolute disaster.... Like almost always 80% usage for this WMI process alone.

I was unable to find the
cause on the 2016 Farm, but ended up assigning only 1 CPU to this process.
Artificially limiting the CPU usage. This worked for years. Not the best way to
handle the issue, to be honest. 

Now I always assumed (my bad!) that whenever we replaced the 2016 server with a new server, this problem word just disappear. Boy was I Wrong!

The new server, having 32-core CPU (Hyper-v VM) is having the exact same issue!
WMIPrvSE.exe using between 30% and 80% of the CPU usage, all-dag-long.
But at the end of the day, when all users log out, it’s gone.

Now here is my big issue: I cant find why! I have been reading logs and traces for days…
My gut feeling is telling me it’s specific to this customers environment. Because we had the same with Server 2016 and with Server 2025. I never saw this on any other environemnt. So I feel like I can rull out any of the generic software tools we use (Antivirus/backup etc) that we run on all our customers. I feel like it must be client-specific software. Or maybe a printer driver for example.


I used Process Explorer to analyse WmiPrvSE.exe and this is the stack trace:

 

ntoskrnl.exe!KeSaveStateForHibernate+0x7d66ntoskrnl.exe!KeQueryPerformanceCounter+0x1c20

ntoskrnl.exe!KeWaitForSingleObject+0x1a9d

ntoskrnl.exe!KeWaitForSingleObject+0x71f

ntoskrnl.exe!KeQueryUnbiasedInterruptTimePrecise+0x2167

ntoskrnl.exe!ExReleaseFastMutexUnsafe+0xc6d

ntoskrnl.exe!KiCheckForKernelApcDelivery+0x32

ntoskrnl.exe!ExAcquirePushLockSharedEx+0x4fb

ntoskrnl.exe!ExAcquirePushLockSharedEx+0x4b9

ntoskrnl.exe!ExUuidCreate+0x1ec9

ntoskrnl.exe!ExUuidCreate+0x1ace

ntoskrnl.exe!WmiQueryTraceInformation+0x2243

ntoskrnl.exe!NtQuerySystemInformation+0xf54

ntoskrnl.exe!NtQuerySystemInformation+0x3e

ntoskrnl.exe!setjmpex+0x9215

ntdll.dll!NtQuerySystemInformation+0x14

cimwin32.dll+0x2dbc0

cimwin32.dll+0x116b4

framedynos.dll!CWbemProviderGlue::CreateInstanceEnumAsync+0x426

wmiprvse.exe+0x8ca9

wmiprvse.exe+0x8338

RPCRT4.dll!NdrServerCallNdr64+0x1c63

RPCRT4.dll!NdrStubCall2+0x30d

combase.dll!CStdStubBuffer_Invoke+0xdf

RPCRT4.dll!CStdStubBuffer_Invoke+0x46

combase.dll!RoClearError+0xc4e2

combase.dll!RoClearError+0xba56

combase.dll!RoClearError+0xb0a1

combase.dll!HBITMAP_UserSize+0x25c6

combase.dll!CoWaitForMultipleHandles+0x101a

combase.dll!CoWaitForMultipleHandles+0x6488

combase.dll!HMONITOR_UserFree+0x2123

RPCRT4.dll!I_RpcFreeBuffer+0x107

RPCRT4.dll!NDRSContextUnmarshall2+0xa24

RPCRT4.dll!NDRSContextUnmarshall2+0x17ea

RPCRT4.dll!RpcExceptionFilter+0x27e4

RPCRT4.dll!RpcBindingFromStringBindingW+0x325c

RPCRT4.dll!RpcImpersonateClient+0x123c

RPCRT4.dll!RpcImpersonateClient+0x3c3

RPCRT4.dll!I_RpcGetBufferWithObject+0x678

ntdll.dll!RtlSetThreadSubProcessTag+0x3bae

ntdll.dll!RtlSetThreadSubProcessTag+0x1cd3

KERNEL32.DLL!BaseThreadInitThunk+0x17

ntdll.dll!RtlUserThreadStart+0x2c

 

 

I you guys have suggestion how I can find the root cause of this then please, let me know!
I have been all over WMImon.exe and analysed logs for hours…

 

 

 

https://redd.it/1p5h75m
@r_systemadmin

Am I crazy?

So, I'm at another career crossroad. For the last decade or so, I've been a commercial truck driver. 12 weeks ago, I suffered an injury that almost took my eyesight and I'm not sure if I'm going to be getting back into the drivers seat.

Last week, a Linux for the Professional book bundle became available through Humble Bundles and I took the whole 22-book volume. I've been using Linux for years keeping old desktops and laptops alive for much longer than the average person would think possible and after starting with one on the books, I'm more into it than ever.

If I don't have a college degree and not a ton of money to work with, but I have a lot of work experience and the drive to learn everything I can, would there be a future in this industry for me?

TL;DR - I might need to find a new career and am wondering if I can teach myself enough to get into SysAdmin.

https://redd.it/1p5nzlx
@r_systemadmin

Microsoft support black hole – domain admin takeover stuck for 7 days, anyone have escalation tips?

Hoping someone here has been through this and can point me in the right direction.

I need to do an admin takeover for our company domain. It's stuck on an old M365 tenant where the admin account is locked behind MFA I can't reset. I've set up a new tenant and verified domain ownership with the TXT record—that part's done.

Opened a support ticket on 11/17 (Sev C), was told it would be escalated. Since then, complete silence. No calls, no emails, no updates. When I call support I get pointed back online. When I add notes to the ticket, nothing.

It's been 7 days on what was supposed to be a 48-hour escalation.

I've already:

* Emailed the executive team
* Posted on X tagging u/MicrosoftHelps
* Tried updating the ticket multiple times

Anyone have a trick for getting through to the domain/tenant team? Or a contact that actually works? This is holding up a compliance deployment with a hard deadline.

Ticket #2511180010000158 if any MS lurkers are feeling generous.

https://redd.it/1p5of5w
@r_systemadmin

Help orient a lost Linux guy on Microsoft? I've been doing *nix for 10 years and I'm terrified of being thrown into the deepend now.

I started as a front end web dev at my agency, and slowly became a full stack web dev, then moved into a cloud administration role all at the same organization. I have only ever worked with Linux and AWS.

My agency is wanting to make a hard pivot to Azure and has a great interest in Power Platform.

I have no idea how any of this works and even just starting to dip my toes in and already I feel very overwhelmed. Bringing this up to management is no longer an option and it's been made very clear to me that my options are "adapt or leave".

Never having had to deal with software licensing and now being thrown into the wolves with licensing is the scariest part so far in the early stages. Is there an ELI5 breakdown of how various Microsoft license tiers work? What does a PowerApps license even do for me? What IS a Power Platform?

My view on IT is very stuck in a self-hosting mindset (even if we do use AWS, we could move to on-prem very readily with the IaC I have). From what little I've seen of MS over my years in tech it seems like MS has pulled away from the DIY, self-hosted model at *lightning* speed and it's clear I don't even understand what they're offering.

Aside from AD and/or Entra, what kinds of workloads are you running in Azure? What roadblocks in my mindset as a relatively old-school Linux guy will I need to overcome? Is everything a hybrid of SaaS now? I'm so lost.

MS people, come laugh at me or commiserate as you see fit. If I can't find orientation, maybe at least you'll find shaudenfreude in my situation.

https://redd.it/1p5qs01
@r_systemadmin

Made a terminal-style text-only social network like it's 1987 :)

With full keyboard navigation including VIM bindings. Looks amazing in full-screen. With "IRC" style chat rooms, DMs, posts, topics, friends, etc.

Email doesn't need to be verified so you can use a throw-away to check it out. It's free.

What do you think?

https://cyberspace.online

https://redd.it/1p5qw9l
@r_systemadmin

What are we paying for health, dental, and vision insurance? (US only)

Considering testing the market and I'm trying to account for health, dental, and vision benefits when determining my total salary. At my current company I pay $115 a pay period or $230 a month for health, dental, and vision insurance. This is for a family plan (wife, son, and I). We've been fairly happy with the insurance, the medical plan is not an HSA which we have wanted to avoid. Would you say this is average, low, or high compared to other places?

https://redd.it/1p5rgsl
@r_systemadmin

I Warned them and they didn't Listen!

We are a VMware shop, when talks of the Broadcom acquisition started ramping up, I warned management that license renewals will cost more for us. they didn't listen because "our account managers are always good to us".

When the acquisition happened, I showed them articles about the pricing increases, management shrugged it off.

But when it came to our turn to get a renewal, BAM! big quote! and suddenly its "why do we need all of this?" "Is this correct?" "but it was cheaper last time?"

Sick of answering to management whose style is "closed eyes, fingers in ears" approach.

https://redd.it/1p5vqbz
@r_systemadmin

We're selling AI stuff but we barely use it internally

The noscript kind of says it all. We're an Enterprise Platform software company selling AI dreams to F500 and we barely use AI internally, not even the software engineers (only auto completion, not much). We have a fairly basic internal AI RAG system to find knowledge that no one really use. It works well, but only tech savvy people use it, Sales, Marketing, Management, very few people use or trust AI and yet, they are selling it for millions of dollars to some big companies out there.

Question: are we an outlier or the norm?

It kills me to be part of this sh*it show, I do use AI myself quite a bit, and some people are impressed with my work lol

Sometimes I feel bad for our customers but at the same time I feel like the first question they should ask (it happened once with a prospect) is: "since you're selling AI, can you tell me how changed your life in the last year or so?"

Just wanted to share this anecdote, and I am curious to hear about anyone else in the industry. Also if you're on the buyer-side, share your experience dealing with software vendors pushing for AI fluff all the times and curious about how you separate the wheat from the chaff

https://redd.it/1p5v6kn
@r_systemadmin

What needed to be in Windows ages ago?

Week numbers in the taskbar.
(if you ever worked in planning, procurement or production, you know)

Adding text in screenshots, why in earth didn't they add this yet? Now I'm writing in my nice mouse-gestures-font

https://redd.it/1p60jhr
@r_systemadmin

Why does identity in the Microsoft stack still feel so scattered?

Entra ID roles here.

Azure IAM there.

Intune permissions somewhere else.

Enterprise app settings in another menu.

CA policies in their own world entirely.

Every time I try to do a clean audit, I end up clicking through 10 different portals just to understand who can do what.

Is this just the permanent state of Microsoft cloud, or have any of you actually found a sane way to centralize identity governance?

https://redd.it/1p66n1h
@r_systemadmin

Genuinely curious - would you use AI more if your data actually stayed private?

Hey everyone, genuine and curious question here.

I've been talking to a bunch of people lately about AI at work - ChatGPT, Claude, Copilot, all that stuff. And I keep hearing the same thing over and over: "I'd use it way more, but I can't put client data into it" or "my compliance team would kill me."

So what happens? People either don't use AI at all and feel like they're falling behind, or they use it anyway and just... hope nobody finds out. I've even heard of folks spending 20 minutes scrubbing sensitive info before pasting anything in, which kind of defeats the whole point.

I've been researching this space trying to figure out what people actually want, and honestly I'm a bit confused.

Like, there's the self-hosting route (which I saw recently there's a post that went viral on self-hosting services). Full control, but from what I've seen the quality just isn't there compared to GPT-5 or Claude Opus 4.5 (which just came out and it's damn smart!). And you need decent hardware plus the technical know-how to set it up.

Then there's the "private cloud" option - running better models but in your company's AWS or Azure environment. Sounds good in theory but someone still needs to set all that up and maintain it.

Or you could just use the enterprise versions of ChatGPT and hope that "enterprise" actually means your data is safe. Easiest option but... are people actually trusting that?

I guess I'm curious about two different situations:

If you're using AI for personal stuff - do you even care about data privacy? Are you fine just using ChatGPT/Claude as-is, or do you hold back on certain things?

If you're using AI at work - how does your company handle this? Do you have approved tools, or are you basically on your own figuring out what's safe to share? Do you find yourself scrubbing data before pasting, or just avoiding AI altogether for sensitive work?

And for anyone who went the self-hosting route - is the quality tradeoff actually worth it for the privacy?

I'm exploring building something in this space but honestly trying to figure out if this is a real problem people would pay to solve or if I'm just overthinking it.

Would love to hear from both sides - whether you're using AI personally or at work.

Thanks :)

https://redd.it/1p65x1m
@r_systemadmin

Data leakage is happening on every device, managed or unmanaged. What does mobile compliance even mean anymore? Be real, all our sensitive company data and personal info we shouldn’t type into AI tools is already there...

We enforce MDM.
We lock down mobile policies.
We build secure BYOD frameworks.
We warn people not to upload internal data into ChatGPT, Perplexity, Gemini, or whatever AI tool they use.
Emails, internal forms, sensitive numbers, drafts, documents....everything gets thrown into these AI engines because it’s convenient.

The moment someone steals an employee’s phone…
or their laptop…
or even just their credentials…
all that AI history is exposed.

If this continues, AI tools will become the new shadow IT risk no one can control and we’re not ready
And because none of this is monitored, managed, logged, or enforced…
we will never know what leaked, where it ended up, or who has it
How are u handling mobile & AI data leakage ?
Anything that actually works?

https://redd.it/1p6absr
@r_systemadmin

Is anyone at a 2025 ADDS functional level?

Curious if anyone has been brave enough to go for it

https://redd.it/1p635sc
@r_systemadmin

Windows DNS forwarders validation error

Hy!

I have a DC, which are also DNS server. I try to set up the forwarders to dns1.fortiguard.net. When I entered the IP address of the DNS server 96.45.45.45, the GUI show: An unknown error occurred while validating the server.

I check the name resolution with nslookup from DC:

nslookup google.hu 96.45.45.45 and the result is success. I also check with PowerShell:

Test-NetConnection 96.45.45.45 \-Port 53

The result is success.

Why does it say the GUI the validation error?


Edit: The server operatin system is Windows Server 2022. I tried it on Windows Server 2019 and 2016, but the validation is OK in the same network. Is it a Windows Server 2022 bug?

https://redd.it/1p69vr5
@r_systemadmin

The original "Vibe Coding" wasn't AI. It was VisiCalc (1979)

I've been seeing the term "Vibe Coding" thrown around a lot lately regarding AI tools, and it sent me down a bit of a history rabbit hole.

I went back and looked at the launch of VisiCalc in 1979 and James Martin’s 1982 book Application Development Without Programmers. The parallels to what we are dealing with right now are actually kind of insane.

Back then, IT departments had multi-year backlogs. Managers started buying Apple IIs with their typewriter budgets just to run VisiCalc so they could bypass IT. That was the birth of "Shadow IT."

Everyone thinks macros were the start of user-gen coding, but VisiCalc didn't even have macros. It was just the sheer ability for a user to define logic without asking permission that broke the dam.

I wrote up a deeper dive on this, but the conclusion I came to is that we're trying to solve this the wrong way (again). In the 80s, IT tried to ban PCs. It failed. Then we tried to ignore spreadsheets. That failed. Eventually, we just accepted them.

We're currently in the "ban/ignore" phase with AI/Low-code tools. I think the only way out is what I'm calling "Governed Sandboxes"—basically giving users "IT-like" powers but inside a walled garden where we can still audit the data.

Curious if anyone here was around for the Lotus/Excel wars, or if you guys are seeing the exact same "Shadow IT" patterns popping up with things like Copilot or Power Platform right now?

https://redd.it/1p6ecnd
@r_systemadmin

Org goes all shadow IT

Anyone else find their org going all shadow IT? I get pulled in to fix stuff non-stop and never included from the start. Ready to jump off a roof.

https://redd.it/1p6eu8l
@r_systemadmin

Memory - Fair Warning

Folks, we've seen a few posts regarding Memory availability and pricing over the last week or two and just a quick update from what we are seeing on the VAR side.

Memory is becoming non-existent slowly, but surely.
The pricing since just August has more then doubled.
Anticipate system costs going up from here if they haven't already.

Dell for example will not sell certain modules unless its in a system build. I've seen this with servers and laptops at this time.

3rd parties like Axiom/Kingston/Crucial are basically running out of stock.

I don't believe there's a good solution to "Buy Now" or "Wait it out" this is just what to expect if any of your partners come back with exceptionally high pricing or long lead times. Also your ETA's should be expected to be extended at any time.

Just fair warning friends.

https://redd.it/1p6fq4h
@r_systemadmin