Kerberos update inflicted strange behavior

Asking for (expert) opinion. MSP tasked me with the assignment of updating a customers kerberos password after not changing it for more than 14 years as a security recommendation from their security partner.

After assessing the impact, checking domain controller replication for possible errors I changed the password once. The day after customer started noting problems with their citrix environment, being that application crashes occurrd, chrome.exe not working and log off issues.

The evening of changing the password I checked after changing the password for kerberos authentication errors on several servers, however I couldn’t find any. The problems have led to customer escalation and we however decided to go forward and change the Kerberos password for the second time to get rid of the golden ticket attack possibility.

The problems that are currently still occurring are focused on the customers Citrix environment with described problems above.

Customer is running an older but stable (prior to the change) version of FSLogix, in combination with Ivanti Workspace Manager, on Server 2022 Std edition.

I just want to rule out that changing the Kerberos password has anything to do with chrome.exe or pdf readers crashing. Strangely enough no eventlog registrations point us in any direction where the issue might come from.

After changing the password once and afterwards for the second time (there were 25 hours in between changing and default domain policy was set to 10 hours to expire tickets) we initiated a klist purge and rebooted the domain controllers one by one to see if this would make any difference. Further I have visually confirmed the keynumber version incrementally changed from 2 to 3 and from 3 to 4 on all domaincontrollers. This for me is an indication that the change went successfully.

I can image and understand the change could trigger something, yet crashing applications on a citrix server that have no dependencies with the domain is strange behavior. Also when not using FSLogix profiles no errors occur. When reverting back to FsLogix the issues occur. When using the most recent version of FsLogix the issue persists.

Please share your opinions and possible suggestions on how to investigate this further.

Thanks in advance.


https://redd.it/1nlwl41
@r_systemadmin

Large Enterprise ADFS Migration - Seeking Community Experiences

Hi all,

Our organization is a large enterprise that has been heavily invested in Active Directory Federation Services (ADFS) for years. We're now considering initiating a project to review and potentially trial more modern authentication mechanisms, but the scope feels daunting given our deep integration.

# Our Current Situation:

* Extensive ADFS deployment with numerous integrated applications
* Complex on-premises infrastructure dependencies
* Significant investment in existing ADFS customizations and configurations
* Large user base with established authentication workflows

# What We're Seeking:

I'd love to hear from others who have navigated similar transitions:

**Migration Experiences:**

* Has anyone here led or been part of a large-scale ADFS migration?
* What were the biggest challenges you encountered?
* How did you handle the transition timeline and user impact?
* What lessons learned would you share?

**Solution Comparisons:**

* **Microsoft Entra ID (Azure AD)**: Experiences with hybrid deployments, cost implications, feature gaps vs ADFS?
* **Third-party solutions** (Okta, Ping Identity, Auth0, etc.): How do they compare in enterprise environments?
* **Other modern alternatives**: What else should we be evaluating?

**Practical Considerations:**

* Cost analysis: Hidden costs beyond licensing?
* Integration challenges with legacy applications?
* Change management strategies that worked well?
* Security and compliance considerations during migration?

# Specific Questions:

1. For those who moved to Entra ID - was the cost savings as significant as Microsoft claims?
2. Any experiences with running parallel systems during transition?
3. How did you handle applications that were tightly coupled to ADFS?

Any insights, war stories, recommendations, or cautionary tales would be incredibly valuable as we plan our approach.

Thanks in advance for sharing your experiences!

https://redd.it/1nlvdm6
@r_systemadmin

IT now controls the light system

I kid you not the reasoning was "it plugs into an Ethernet cable".

I'm waiting for facilities to shove HVAC off to us as well because that's networked too. Maybe we disconnect it from the network so they can't use that argument. "Oh you're mad you cant control it from your desk anymore? I can control the lights from my desk it's nice"

https://redd.it/1nm0miq
@r_systemadmin

Military equivalent of DevOps

I’m active duty in the Army, working as a 35T. From what I can tell, my role lines up pretty closely with DevOps/sysadmin: I handle system integration, troubleshooting, networking, security, and keeping mission-critical systems running.

Here’s where I’m at:
Certs: Only have Security+ right now
Clearance: Active TS/SCI
Experience: 5 years in the field (all hands-on, operational environments)
Education: No degree yet — considering WGU’s Software Engineering BS/MS because of flexibility & cost

My questions:
•Would a degree from WGU or UMGC actually help me when I separate, or should I just keep stacking certs?
•For DevOps roles, which certs would you recommend I target next (AWS, Azure, Linux, Kubernetes, etc.)?
•For those who made the jump from military IT/maintenance into DevOps/SRE, what helped you the most when transitioning?

Trying to set myself up for success when I ETS. Appreciate any advice.

https://redd.it/1nm6t4w
@r_systemadmin

Lost Hardware

A help desk employee lost a brand new $2,500 piece of equipment and has no idea where it went. The department is a secure area, so there are cameras everywhere. Security has him on video putting the device in his backpack. He went over to pick it up and bring it back to the help desk for configuration. For some reason, people were asking me what to do, so they received "order a new one and tell his manager".

Chances are it was thrown out by accident because he comes across as too skittish to steal anything.

https://redd.it/1nm5xlu
@r_systemadmin

Company Provided Cell Phones

Just curious how many of you have companies that provide mobile devices for your end users?

How do you go about managing them day to day and how many total devices?

https://redd.it/1nm36eo
@r_systemadmin

How do y’all deal with people that just seem to hate IT?

I get a ticket from a user Monday about not receiving emails from a vendor they’re expecting. Now I like this person, I feel we used to have pretty good rapport but I work with them much less now that they’re in sales. I do a message trace, no dice, nothing in quarantine, I see that vendor has sent emails, just not the ones he’s looking for. I say hey I don’t see anything that shows it even hit the server, so it likely is on their end. Maybe they don’t send it, or they’re having issues with their system? Do you have anyone from there I can talk to and sort it out with?

I then get an email I believe he meant to forward but replied and added his boss (sales) asking if I knew what I was doing because I’m always pushing back and not fixing his problems, then suggests I should take some formal classes in IT because I’m not helpful.

I just didn’t reply from there but I’m like, bro what the fuck? Half the time I ask you questions on your tickets and you just don’t reply? I know I love the quick fixes, but shit am I not allowed to take more than one email to fix an issue? I talk to the sales guy and show him our tickets and he’s like no no, I get it. I know you’re just trying to help, no one else here is doubting your abilities.

But like, what do I even do for people like these? If I don’t do it via ticket it’s not documented so I hate to call them or come to their desk. Also, turned out vendor was mid migration and had some issues come out that was making one of their programs that sends email fail to send intermittently.

https://redd.it/1nmaypc
@r_systemadmin

How do you handle problems that resolve themselves?

Exactly as stated.

We recently had an issue where a large number of our pooled VDI machines lost contact with the with the DC's and started complaining about time differences. We didnt change anything to fix it, we just rebooted the unused machines in the pool and it seems to have cleared up. The group that controls the DC's swears it wasnt a time issue on their end and I know its not a time issue on the pooed VDI machines.

The issue just went away and im having trouble letting it go. I need to know the cause before I can move on and im struggling. Besides that, its hard to give a downtime summary to leadership when you cant confirm the cause for a fact.







https://redd.it/1nmats1
@r_systemadmin

M365 Apps unexpectedly closing on their own? Follow up

Hi Folks,

I posted a while back about an issue where M365 apps (New Teams, OneDrive, Edge, and New Outlook, Word, Excel) would all suddenly close at the same time without warning.

At the time I couldn’t reproduce it consistently, so it was hard to pin down. I thought the culprit might be the M365 cloud update, but u/martinnothnagel\_msft has confirmed the cloud update ONLY impacts apps that use the C2R. As this issue was still happening at least once per, week, whilst on the monthly enterprise channel that could not be the culprit. Further, this issue continued to occur after the cloud update was paused.

This week it finally happened on my own work laptop, and I was able to spend time digging in. Here’s what I found:

The trigger appears to be updates to the AppX package e.g. New Outlook app (olk.exe), which is delivered as an AppX package via the Microsoft Store.

Around the exact time the apps all closed, the folder C:\Program Files\WindowsApps\Microsoft.OutlookForWindows_* had a modified timestamp, confirming an Outlook update was applied.

When this happens, not only does New Outlook restart, but other M365 apps also crash. Edge also shows “Edge closed unexpectedly” when relaunched.

In order to workaround this, i've applied a GPO to block Microsoft Store auto-updates. I’m going to monitor this week to see if the crashes stop.

This seemed to manifested the same time we enrolled all our company windows devices into Intune.
I'm not saying Intune is responsible, it's just extremely timely...!

Has anyone else seen this behavior?

https://redd.it/1nm88vr
@r_systemadmin

Moving from folder redirection to OneDrive known folder move seamlessly and transparently to users?

I'm currently using Folder Redirection & would like to move to OneDrive Known Folder Move, and am wondering how others have done this, in environments where there is a high expectation of seamlessness, simplicity and reliability for end users (who are not tech savvy), and where desktops still exist and it's not strictly one user = one device.

I know I can upload people's Desktop, Documents, etc folders to their OneDrives with the SharePoint migration tool, and have tested that far already. I know I can make OneDrive redirect known folders by default once the user logs into OneDrive.

The things I am still struggling with are:

* The GPO or Intune settings to automatically sign in OneDrive can't (obviously) do so if MFA is required and not satisfied. I don't see any option to enforce that a user has to sign into OneDrive before saving files on the machine.
* I know Windows Hello for Business is the answer for users who carry a laptop; make their Windows sign in satisfy MFA. I am referring to users who sign into the shared desktop in whichever classroom they are in today.
* I can't seem to get a machine from our current folder redirection GPO to OneDrive KFM, without the machine trying to copy files anywhere (letting us handle it with the SharePoint Migration Tool).

One of the main issues is that folders OneDrive KFM refuses to touch, such as Videos, are redirected inside the documents folder rather than alongside it in the current setup. That causes KFM to refuse to touch Documents. I assume this will be a multi step process to get Folder Redirection to pull Videos and Pictures out of Documents, let clients get that policy, and then move to KFM.

The other issue is idiot proofing. Folder Redirection has had offline files disabled for many years because with it on, users don't pay attention to or resolve sync conflicts, and data is lost. Is there any way to make OneDrive more aggressive about resolving conflicts? The goal is to eliminate cases where user error can result in saving files locally that are not backed up.

https://redd.it/1nm81s7
@r_systemadmin

Exchange hybrid free busy issue

Working on an exchange hybrid migration. Have 2 exchange 2016 servers and 1 new exchange 2019 server being used for hybrid connection to Exo, no mailboxes. Migrated 2 users as test to Exo and when they check free busy of on prem users this happens. Some work and some don't, totally random. User could work one day and not the next. All mailboxes are on a single exchange 2016 server. Not always same database for mailboxes showing no information. Any ideas?

https://redd.it/1nmg4cd
@r_systemadmin

Do you all plan to further in your career or are you comfortable at sysadmin?

Do you all plan to further in your career or are you comfortable at sysadmin? Just curious on what ppl takes are on moving up in their career.

https://redd.it/1nmcgol
@r_systemadmin

Two domain controllers

Seeing what everyone’s input is for dns settings on two domain controllers. Do you put a loop back and then the address of the opposite dns server or
Do you use the il address of the server and the. The opposite ip address of the server

https://redd.it/1nmia3n
@r_systemadmin

What would you do if your boss was fired, and your new boss was not handling the workload of the old boss? Specifically in a small team (3) situation, where the manager typically is willing to stand up and work tickets as needed, but the new manager who promised to be hands-on is simply not?

It's a very difficult situation as we are understaffed, which they acknowledge, but still refuse to even look at the ticket system. They want us to screen share it, or give verbal assessments of what our weekly focus was, as though they are above doing the work their predecessor did. In this small team, typically the breakdown has been helpdesk (little or no IT experience), sysadmin (mid-level), and manager (senior admin who now also manages IT staff). This new leader is acting like they're in a huge company with a department of 10+ people, like we can afford to have someone who doesn't cover, even when IT members are out on vacation or sick. What's the solution for us?

https://redd.it/1nmo3ag
@r_systemadmin

I might be asking too much i think, but there's any autounattend noscript that installs excel, powerpoint and word while windows in installing?

Also if there's any way to auto connect on a Set up Wi-Fi too, i don't know if any of this is possible, but i thought i might just ask anyway

https://redd.it/1nmpam7
@r_systemadmin

Exchange SE install ends prematurely Event Logs: "Installation Completed Successfully" after cancelling first install.

Was trying to install Exchange SE tools only so I could shut down my last exchange server. During the install I was missing a prerequisite (.NET 4.8). So I installed that which asked for a restart. I closed out of the Exchange installer and restarted.

After restart, updates applied, etc...

Started the Exchange SE installer again. It went through the standard MSI installer "Gathering required information" then the installer just goes away.

Checking the Event log, I see these entries all within a second of each other.

>1040 Beginning a Windows Installer transaction: E:\\Exchangeserver.msi
1042 Ending a Windows Installer transaction: E:\\Exchangeserver.msi
11707 Product: Microsoft Exchange Server -- Installation completed successfully.
1033 Windows Installer installed the product. Product Name: Microsoft Exchange Server. Product Version: 15.0.2562.17. Product Language: 1033. Manufacturer: Microsoft Corporation. Installation success or error status: 0.

The logs in C:\\ExchangeSetupLogs are from the original installation attempt. I have checked the (suggested) registry for HKLM\\Software\\Microsoft\\ExchangeServer, there are no entries there. Control Panel does not show Exchange Server is installed. I also tried renaming the Program Files\\Microsoft\\Exchange Server folder. And of course, restart.

Guess I should have double checked the Prerequisites were installed before proceeding.

Any suggestions?

Can't seem to get the installer to come back up.


x r/exchangeserver

https://redd.it/1nmrwbf
@r_systemadmin

”Cloud is more secure”

I have been wondering when this will happen. Everyone saying ”cloud is more secure than on-prem”. Yeah, sure. https://www.theregister.com/2025/09/19/microsoft_entra_id_bug/

https://redd.it/1nmsj18
@r_systemadmin

Office environment question

Going to lead off this post with a "Sorry I am not really a Sysadmin" but I do frequent (lurk) this subreddit and it has been helpful in the past.

I am a really informal tech leader at a mid-sized architecture firm. Before I arrived, much of the contents of our server were stored in the cloud, and for the past few years, and in the time I have been here, we have worked primarily with a server that is stored physically in our office, monitored by our IT service providers.

Do you think it would be worth returning to the cloud? We have been somewhat frustrated with our IT company as of late and have a previously good relationship with a company that does phone services that otherwise also could provide IT services, just via a cloud environment, that would virtualize a lot of our system. We could also tap into their broader cloud infrastructure, but I lack the technical know how to that extent to really get a sense of what is better - and most importantly if it is worth the money

Tl:dr 50-75 employees distributed in two office, should we have a physical server or work more in the cloud? I am leaning the later.

https://redd.it/1nmtirh
@r_systemadmin

Company running VMware 5.5 in 2025

Found an enterprise running VMware vSphere 5.5 (from 2013!) with 500+ Windows Server 2008/2012 boxes. They're planning to upgrade to... VMware 6.x, which is.. yeah.

Someone should tell them about Broadcom pricing before they get destroyed. Yikes.

I keep finding companies like this, maybe 20-30 per week with seriously outdated infrastructure.

How do you even approach companies that are this far behind?

https://redd.it/1nmwbn3
@r_systemadmin

Hyper-V Issue – Weird but seems to be true

TL;DR: Could removing a DVD drive (not just ISO removal, actual drive removal) from a running Gen 2 VM cause a cluster host to briefly lose all its connections to everything?

So, I was doing some routine maintenance on our Hyper-V cluster (6 hosts, Nimble SAN) last week, live migrating some VMs, updating others and suddenly one of the hosts lost all connections to the iSCSI storage and the cluster. Some of the VMs on that host failed and rebooted, some did not. After a minute everything was fine.

I couldn’t figure out what happened, but 2 minutes earlier, I had, on a different host, removed a DVD Drive from a running Gen 2 VM because I couldn’t seem to migrate it from 2019 -> 2022 with the DVD drive attached. (Currently we still have some 2019 hosts)

I didn’t think much of the DVD drive removal as I was doing other maintenance and I couldn’t imagine it caused the problem and was running on a different host, but I did read later that you shouldn’t remove DVD drives while VMs are running, even Gen 2.

Two days go by, everything has been stable, I was doing more work and again removed a DVD drive from a running VM and again 2 minutes later one of the hosts lost its connections, some VMs rebooting, some not and again it recovered quickly.

So another day goes by and now I’m curious, I still don’t believe it was the DVD drive removal and assume it was something else I must have done, so I remove a DVD drive yet again from a different running VM and this time just waited and watched and sure enough, 2 minutes later, one of the hosts lost it’s connections to the storage and the network and then came back.

Again, I know now that you are not supposed to remove DVD drives from running VMs, but has anyone ever seen this and does anyone have any idea how or why removing a DVD drive from one VM could cause a host in a cluster to basically lose its mind for a minute?  I’m still not sure I believe it, but it’s hard to deny the pattern. This cluster has been running for years without ever glitching before.

https://redd.it/1nmtxud
@r_systemadmin

Have you ever, as a system administrator, come across any organization’s business secret like I did? If yes, what is that??

As a system administrator you may have come across with any organization's business secret

like one I had,

Our organisation is a textile manufacturing one. What I came to know is, they are selling organic cotton & through which getting huge margin of profit compared to the investment for raw materials and production cost. Actually, they got certificates by giving bribes, but in reality, they use synthetic yarn... yet sell this as organic into the UK.
........... likewise any business secrets??

https://redd.it/1nn0et2
@r_systemadmin