The AppStack Was Pirated.
In a previous role, I worked for a large healthcare company running a VMware Horizon non-persistent VDI environment.
If you’ve ever managed Horizon in a healthcare setting, you already know how “fun” that combination can be – duplicate sessions, pool unavailability, network issues, and the constant battle of keeping clinical staff happy without breaking anything.
VMware Horizon in Prod
Our setup was pretty complex: Windows 10 virtual desktops, a stack of AppVolumes packages, and a plethora of departmental customizations sprinkled across different user groups.
Most users lived their entire day inside a session that was completely wiped the moment they logged off. Everything they needed had to come from Network storage, logon noscripts, or an attached AppStack.
One of those AppStacks was Adobe Acrobat.
In fact, we had four different versions of Adobe Acrobat:
# How did we handle licensing?
We had a poorly documented process for each:
Adobe Reader free – the standard version of Adobe that all users got by default. No purchase/approval needed.
Adobe Acrobat Standard DC – subnoscription-based, assigned only to admin or power users who actually needed editing capabilities. We’d raise a ticket to procurement who would buy a license in the Adobe portal, and we’d make the user an Adobe Account and assign the license \+ appstack.
Adobe Acrobat Pro – subnoscription-based, used by executives and operational teams. We would pass these straight to our procurement team, they’d then purchase a license and assign it through an internal system, [then pass the ticket back](https://blog.chrispro.live/wp-content/uploads/2025/12/image-8-768x512.png) so we can update the users appstack.
Adobe Acrobat 2017 – perpetually licensed version that some operational teams used, because they had old licenses for it. We’d keep the licenses attached to tickets within our ITSM system, which made them easy to find – yeah…
# What went wrong?
Well, it all started on a Monday morning when I came into the office and saw an influx of tickets regarding Adobe being “broken”.
It turns out, our VDI guys had pushed new appstacks for Adobe Reader/Standard, updating them to the latest version – which had quite a different user interface from what they were used to.
Not much was actually broken, users just saw the app looking a bit different and got scared, the only real issue was that one of the PDF editing tools wouldn’t load correctly because it relied on a DLL or something that was omitted from the appstack.
So we got that sorted and thought all was well, but then we started to see some more tickets come in, from all over the business, about a licensing error.
Huh, nothing changed with licensing, what’s going on here?
# Where did our licenses go?
So it turns out that alongside Adobe Reader and Standard, Adobe Pro was updated as well.
Normally, this would be fine, however it revealed something rather scary, nobody knows where the Adobe Pro licenses for all of these users are, we only have the tickets with approvals from our procurement team, so we asked them about it.
>Oh, we have no action needed on our end for these, we just review the request and assign back to you guys for the technical work.
Well, that tells us exactly what I was afraid of, we had dozens of users without valid licenses for Adobe Pro, and they need it now.
And so began a procurement spree, getting a list of users that need Adobe Pro and actually buying licenses this time.
# Our packaging guy had a well-kept secret
The guy that actually packaged all our apps, for both VDI and non-VDI, was an older gentleman, who very much wanted to
In a previous role, I worked for a large healthcare company running a VMware Horizon non-persistent VDI environment.
If you’ve ever managed Horizon in a healthcare setting, you already know how “fun” that combination can be – duplicate sessions, pool unavailability, network issues, and the constant battle of keeping clinical staff happy without breaking anything.
VMware Horizon in Prod
Our setup was pretty complex: Windows 10 virtual desktops, a stack of AppVolumes packages, and a plethora of departmental customizations sprinkled across different user groups.
Most users lived their entire day inside a session that was completely wiped the moment they logged off. Everything they needed had to come from Network storage, logon noscripts, or an attached AppStack.
One of those AppStacks was Adobe Acrobat.
In fact, we had four different versions of Adobe Acrobat:
# How did we handle licensing?
We had a poorly documented process for each:
Adobe Reader free – the standard version of Adobe that all users got by default. No purchase/approval needed.
Adobe Acrobat Standard DC – subnoscription-based, assigned only to admin or power users who actually needed editing capabilities. We’d raise a ticket to procurement who would buy a license in the Adobe portal, and we’d make the user an Adobe Account and assign the license \+ appstack.
Adobe Acrobat Pro – subnoscription-based, used by executives and operational teams. We would pass these straight to our procurement team, they’d then purchase a license and assign it through an internal system, [then pass the ticket back](https://blog.chrispro.live/wp-content/uploads/2025/12/image-8-768x512.png) so we can update the users appstack.
Adobe Acrobat 2017 – perpetually licensed version that some operational teams used, because they had old licenses for it. We’d keep the licenses attached to tickets within our ITSM system, which made them easy to find – yeah…
# What went wrong?
Well, it all started on a Monday morning when I came into the office and saw an influx of tickets regarding Adobe being “broken”.
It turns out, our VDI guys had pushed new appstacks for Adobe Reader/Standard, updating them to the latest version – which had quite a different user interface from what they were used to.
Not much was actually broken, users just saw the app looking a bit different and got scared, the only real issue was that one of the PDF editing tools wouldn’t load correctly because it relied on a DLL or something that was omitted from the appstack.
So we got that sorted and thought all was well, but then we started to see some more tickets come in, from all over the business, about a licensing error.
Huh, nothing changed with licensing, what’s going on here?
# Where did our licenses go?
So it turns out that alongside Adobe Reader and Standard, Adobe Pro was updated as well.
Normally, this would be fine, however it revealed something rather scary, nobody knows where the Adobe Pro licenses for all of these users are, we only have the tickets with approvals from our procurement team, so we asked them about it.
>Oh, we have no action needed on our end for these, we just review the request and assign back to you guys for the technical work.
Well, that tells us exactly what I was afraid of, we had dozens of users without valid licenses for Adobe Pro, and they need it now.
And so began a procurement spree, getting a list of users that need Adobe Pro and actually buying licenses this time.
# Our packaging guy had a well-kept secret
The guy that actually packaged all our apps, for both VDI and non-VDI, was an older gentleman, who very much wanted to
retire, but he was just “too good” at what he did.
Arthur, was our (illustrative purposes only) app packaging engineer.
I had a chat with him after we figured out what the issue was, and he told me that the Adobe Pro appstack was basically built for 2 specific VIP’s in the company.
I let him know that there were about 50 users in it right now, and he proceeded to get very pissed off at us/SD.
In the end, someone from management got through to him, and found out that Adobe’s DRM had been bypassed when appstacking Adobe Pro – meaning we were effectively running a pirated Appstack for months.
# Well, that was fun.
Yep, the business sort of turned this into a bit of a cost-saving opportunity as well.
None of the managers wanted to use their IT budgets on this issue so we were advised to only procure a license if someone explicitly raised a ticket about it.
In the end, everyone already hated VDI, and we had a bad reputation because of it – so it didn’t really impact us all that badly.
Adobe, if you are reading this, know 2 things:
Our company paid for licenses immediately after identifying the issue.
I no longer work for this company, you could probably find out where I worked pretty easily, but know this issue was years ago, and I was just in a standard L2 role, not at all in a position where I could be accountable for this.
Hope you enjoyed!
https://redd.it/1pjrwbu
@r_systemadmin
Arthur, was our (illustrative purposes only) app packaging engineer.
I had a chat with him after we figured out what the issue was, and he told me that the Adobe Pro appstack was basically built for 2 specific VIP’s in the company.
I let him know that there were about 50 users in it right now, and he proceeded to get very pissed off at us/SD.
In the end, someone from management got through to him, and found out that Adobe’s DRM had been bypassed when appstacking Adobe Pro – meaning we were effectively running a pirated Appstack for months.
# Well, that was fun.
Yep, the business sort of turned this into a bit of a cost-saving opportunity as well.
None of the managers wanted to use their IT budgets on this issue so we were advised to only procure a license if someone explicitly raised a ticket about it.
In the end, everyone already hated VDI, and we had a bad reputation because of it – so it didn’t really impact us all that badly.
Adobe, if you are reading this, know 2 things:
Our company paid for licenses immediately after identifying the issue.
I no longer work for this company, you could probably find out where I worked pretty easily, but know this issue was years ago, and I was just in a standard L2 role, not at all in a position where I could be accountable for this.
Hope you enjoyed!
https://redd.it/1pjrwbu
@r_systemadmin
what is the best heatmap tool for a beginner?
so i’m trying to get a better understanding of how users interact with my website, and i’ve heard heatmaps are a good way to visualize all that. i’ve never used one before, but i’m really interested in seeing where people click the most, how far they scroll, etc. i’ve done some research and there are a ton of options out there, but i’m not sure which one’s worth the investment.
i’m looking for something that’s fairly easy to set up and doesn’t require too much of a learning curve, since i’m not super techy. also, i’m on a bit of a budget, so if there are good options that aren’t too pricey, that’d be a bonus.
a few things i’m wondering:
* are heatmap tools pretty accurate, or should i be using them alongside other analytics to get a fuller picture?
* do the free versions of heatmap tools actually give you enough data, or is it worth it to pay for a more robust plan?
* if you’ve used any heatmap tools for your website, which one has been the easiest to use and most helpful?
i’ve heard about a few like hotjar and crazy egg, but not sure if there are better alternatives out there. looking forward to hearing what you guys think
https://redd.it/1pjpnoq
@r_systemadmin
so i’m trying to get a better understanding of how users interact with my website, and i’ve heard heatmaps are a good way to visualize all that. i’ve never used one before, but i’m really interested in seeing where people click the most, how far they scroll, etc. i’ve done some research and there are a ton of options out there, but i’m not sure which one’s worth the investment.
i’m looking for something that’s fairly easy to set up and doesn’t require too much of a learning curve, since i’m not super techy. also, i’m on a bit of a budget, so if there are good options that aren’t too pricey, that’d be a bonus.
a few things i’m wondering:
* are heatmap tools pretty accurate, or should i be using them alongside other analytics to get a fuller picture?
* do the free versions of heatmap tools actually give you enough data, or is it worth it to pay for a more robust plan?
* if you’ve used any heatmap tools for your website, which one has been the easiest to use and most helpful?
i’ve heard about a few like hotjar and crazy egg, but not sure if there are better alternatives out there. looking forward to hearing what you guys think
https://redd.it/1pjpnoq
@r_systemadmin
Reddit
From the sysadmin community on Reddit
Explore this post and more from the sysadmin community
Thickheaded Thursday - December 11, 2025
Howdy, /r/sysadmin!
It's that time of the week, Thickheaded Thursday! This is a safe (mostly) judgement-free environment for all of your questions and stories, no matter how silly you think they are. Anybody can answer questions! My name is AutoModerator and I've taken over responsibility for posting these weekly threads so you don't have to worry about anything except your comments!
https://redd.it/1pjtu3w
@r_systemadmin
Howdy, /r/sysadmin!
It's that time of the week, Thickheaded Thursday! This is a safe (mostly) judgement-free environment for all of your questions and stories, no matter how silly you think they are. Anybody can answer questions! My name is AutoModerator and I've taken over responsibility for posting these weekly threads so you don't have to worry about anything except your comments!
https://redd.it/1pjtu3w
@r_systemadmin
Reddit
From the sysadmin community on Reddit
Explore this post and more from the sysadmin community
How I nuked the network at a small gaming facility with one line.
[There was a post requesting horror stories from helpdesk and my story was swept away by a sea of comments, please enjoy.\]
There was a general data segment for most of the computers at a small gaming facility i worked for before we granulized our segmentation. On this data segment you could find the computers for all of the departments and the POS up front. Printers, servers, switches, ATMs, gaming machines, phones, cameras and a few other devices were excluded from this segment and had their own. The departments affected were generally security, surveillance, cashier cage service counter, player club service counter, food services, counting room, gaming inspection, slot mgmt, tables mgmt, operations mgmt, facilities mgmt, custodial services, receiving and IT helpdesk.
Some context, the previous IT administrators were actually an outside consulting firm that came out and did IT work for both sites. Needless to say, they were great at talking up large goals for infrastructure change and development, and had absolutely zero follow through, ending up in a spaghettified network full of crap configurations, SPOFs, and general lack of foresight and ability. Only the main-site gaming facility a few cities away had a de facto network administrator, an overworked sysadmin who managed basically every application and server and the network configuration cleanup after that firm was terminated. The company would not approve a network technician for the off-site smaller gaming facility only a couple years after parting with that disaster.
I was working on helpdesk and was a fairly new unofficial off-site network technician working with approval and under the discretion of the main-site IT director. I was working on organizing and relabeling the IDF cables with verbally approved minimal downtimes for each endpoint, manually clearing out bad switch configuration lines and replacing them with our preferred agreed upon configurations, and in general documenting the wild frontier we were stuck with. These were the first major change these switches had seen in years, and it was clear that they had been manually configured at different times with different intents. Many also had common bad practices security holes that are easily fixed with a line or two. At this point too the IT budget was abysmal so there was no good remote management solution aside from the singular SecureCRT license afforded to the department, or custom PuTTY configs shared amongst us.
Well, one unlucky day on the gaming floor working on one unlucky access switch in particular, i was clearing the vlan database of unused entries. At this point, I was new and self-taught mostly alone, and I was unaware of a certain unpopular protocol that would be my ultimate doom. Did i mention our enterprise was Cisco? well, i was just getting started and picked the first vlan to clear - the data vlan. On this access switch, for its purposes of connecting slot machines back to the distribution layer, it did not need this one. So i simply did my thing as i had on a few other switches beforehand, getting the hang of it, and entered the command “no vlan <num>” and saved. I didn’t notice any immediate change. I didn’t even notice my Wi-fi went.
Away from me all around the gaming facility, departments erupted into chaos. Although the slot machines kept going so the patrons were mostly unphased, all the customer-facing service counters, the point of sales, the back of house, security and surveillance, gaming operations, even our helpdesk lost network connectivity. The phones worked. And i soon found out so did everyone’s legs and voices, as the IT office was swarmed a few moments after my return. I assured everyone I would look into the issue and get it resolved immediately, and I called up the IT director, who at this time was the best network engineer I knew with 20 years of experience, and I explained what happened and what I had been doing.
He instructed me to go to core switch at our site and manually connect to
[There was a post requesting horror stories from helpdesk and my story was swept away by a sea of comments, please enjoy.\]
There was a general data segment for most of the computers at a small gaming facility i worked for before we granulized our segmentation. On this data segment you could find the computers for all of the departments and the POS up front. Printers, servers, switches, ATMs, gaming machines, phones, cameras and a few other devices were excluded from this segment and had their own. The departments affected were generally security, surveillance, cashier cage service counter, player club service counter, food services, counting room, gaming inspection, slot mgmt, tables mgmt, operations mgmt, facilities mgmt, custodial services, receiving and IT helpdesk.
Some context, the previous IT administrators were actually an outside consulting firm that came out and did IT work for both sites. Needless to say, they were great at talking up large goals for infrastructure change and development, and had absolutely zero follow through, ending up in a spaghettified network full of crap configurations, SPOFs, and general lack of foresight and ability. Only the main-site gaming facility a few cities away had a de facto network administrator, an overworked sysadmin who managed basically every application and server and the network configuration cleanup after that firm was terminated. The company would not approve a network technician for the off-site smaller gaming facility only a couple years after parting with that disaster.
I was working on helpdesk and was a fairly new unofficial off-site network technician working with approval and under the discretion of the main-site IT director. I was working on organizing and relabeling the IDF cables with verbally approved minimal downtimes for each endpoint, manually clearing out bad switch configuration lines and replacing them with our preferred agreed upon configurations, and in general documenting the wild frontier we were stuck with. These were the first major change these switches had seen in years, and it was clear that they had been manually configured at different times with different intents. Many also had common bad practices security holes that are easily fixed with a line or two. At this point too the IT budget was abysmal so there was no good remote management solution aside from the singular SecureCRT license afforded to the department, or custom PuTTY configs shared amongst us.
Well, one unlucky day on the gaming floor working on one unlucky access switch in particular, i was clearing the vlan database of unused entries. At this point, I was new and self-taught mostly alone, and I was unaware of a certain unpopular protocol that would be my ultimate doom. Did i mention our enterprise was Cisco? well, i was just getting started and picked the first vlan to clear - the data vlan. On this access switch, for its purposes of connecting slot machines back to the distribution layer, it did not need this one. So i simply did my thing as i had on a few other switches beforehand, getting the hang of it, and entered the command “no vlan <num>” and saved. I didn’t notice any immediate change. I didn’t even notice my Wi-fi went.
Away from me all around the gaming facility, departments erupted into chaos. Although the slot machines kept going so the patrons were mostly unphased, all the customer-facing service counters, the point of sales, the back of house, security and surveillance, gaming operations, even our helpdesk lost network connectivity. The phones worked. And i soon found out so did everyone’s legs and voices, as the IT office was swarmed a few moments after my return. I assured everyone I would look into the issue and get it resolved immediately, and I called up the IT director, who at this time was the best network engineer I knew with 20 years of experience, and I explained what happened and what I had been doing.
He instructed me to go to core switch at our site and manually connect to
it, and check the VLAN database. Checking, I found that the entry for data vlan <num> was missing from the core switch. He instructed me to put it back and once I did and saved the config, everything came back up. He informed me that I had fallen prey to the aforementioned consulting firm’s sloppy management practices. They had VTP still on site-wide, and even worse was that some of the access-layer switches were in server mode. What I had so innocuously done from the access switch on the gaming floor brought down pretty much the whole site in a moment. Luckily the core switch was also in server mode, so once I put it back the change was basically undone. At that point we made it a policy to never allow VTP on the network.
Morals of the story/tldr
1. unnamed consulting firm sucks.
2. VTP bad.
3. trial by fire is the best way to learn.
4. thanks for not firing employees for mistakes like this.
https://redd.it/1pjtn4g
@r_systemadmin
Morals of the story/tldr
1. unnamed consulting firm sucks.
2. VTP bad.
3. trial by fire is the best way to learn.
4. thanks for not firing employees for mistakes like this.
https://redd.it/1pjtn4g
@r_systemadmin
Reddit
From the sysadmin community on Reddit
Explore this post and more from the sysadmin community
Reset KRBTGT Key - Which noscript
Hi!
I want to reset the KRBTGT-password on an old domain. There are so many noscripts and manuals out there - which one would you recommend?
This one here did not get any updates since 2020:
https://github.com/microsoftarchive/New-KrbtgtKeys.ps1/blob/master/New-KrbtgtKeys.ps1
This one is newer, but not the "Microsoft-one":
https://github.com/zjorz/Public-AD-Scripts/blob/master/Reset-KrbTgt-Password-For-RWDCs-And-RODCs.ps1
Best wishes
https://redd.it/1pjuy61
@r_systemadmin
Hi!
I want to reset the KRBTGT-password on an old domain. There are so many noscripts and manuals out there - which one would you recommend?
This one here did not get any updates since 2020:
https://github.com/microsoftarchive/New-KrbtgtKeys.ps1/blob/master/New-KrbtgtKeys.ps1
This one is newer, but not the "Microsoft-one":
https://github.com/zjorz/Public-AD-Scripts/blob/master/Reset-KrbTgt-Password-For-RWDCs-And-RODCs.ps1
Best wishes
https://redd.it/1pjuy61
@r_systemadmin
GitHub
New-KrbtgtKeys.ps1/New-KrbtgtKeys.ps1 at master · microsoftarchive/New-KrbtgtKeys.ps1
This noscript will enable you to reset the krbtgt account password and related keys while minimizing the likelihood of Kerberos authentication issues being caused by the operation. - microsoftarchive...
Windows Admin Center 2511 generally available
Microsoft announced the release of Windows Admin Center 2511
https://techcommunity.microsoft.com/blog/windows-admin-center-blog/windows-admin-center-version-2511-is-now-generally-available/4477048
https://redd.it/1pjvzk8
@r_systemadmin
Microsoft announced the release of Windows Admin Center 2511
https://techcommunity.microsoft.com/blog/windows-admin-center-blog/windows-admin-center-version-2511-is-now-generally-available/4477048
https://redd.it/1pjvzk8
@r_systemadmin
TECHCOMMUNITY.MICROSOFT.COM
Windows Admin Center version 2511 is now generally available! | Microsoft Community Hub
Today we’re announcing the general availability of Windows Admin Center version 2511! A special thank you to our customers, partners, and fans. Your...
Windows server ignores primary DNS, only queries secondary
Strange situation, at least to me.
A Windows Server 2019 has primary DNS set to the DC (which also has DNS role) and secondary DNS to a public DNS provider.
Now this server in question doesn't resolve any name that the primary DNS could answer for, because apparently it only queries the secondary DNS.
So for example
nslookup windows.domain
states that the secondary DNS has been queried and couldn't resolve the query.
If I however do
nslookup windows.domain IP.OF.PRIMARY.DNS
so that I force nslookup to query the primary DNS, then it will resolve just fine.
So apparently the server can reach the primary DNS without problems and the primary DNS would also resolve queries for the server without problems.
Still the server choses to only use the secondary DNS and ignores the primary DNS (unless I specifically force it to use the primary DNS when doing a nslookup).
Any idea how to troubleshoot this?
https://redd.it/1pjw1bn
@r_systemadmin
Strange situation, at least to me.
A Windows Server 2019 has primary DNS set to the DC (which also has DNS role) and secondary DNS to a public DNS provider.
Now this server in question doesn't resolve any name that the primary DNS could answer for, because apparently it only queries the secondary DNS.
So for example
nslookup windows.domain
states that the secondary DNS has been queried and couldn't resolve the query.
If I however do
nslookup windows.domain IP.OF.PRIMARY.DNS
so that I force nslookup to query the primary DNS, then it will resolve just fine.
So apparently the server can reach the primary DNS without problems and the primary DNS would also resolve queries for the server without problems.
Still the server choses to only use the secondary DNS and ignores the primary DNS (unless I specifically force it to use the primary DNS when doing a nslookup).
Any idea how to troubleshoot this?
https://redd.it/1pjw1bn
@r_systemadmin
Reddit
From the sysadmin community on Reddit
Explore this post and more from the sysadmin community
Server disappearing from Hyper-V
This morning a bunch of our servers disappeared from Hyper-V. There was no security alerts from huntress so I don’t think there is anything malicious going on.
We had to restore them from Veeam and now everything is ok. Has anyone run into this before? I’m not sure to be worried or not lol.
How do I prevent this from happening again?
https://redd.it/1pjyyuk
@r_systemadmin
This morning a bunch of our servers disappeared from Hyper-V. There was no security alerts from huntress so I don’t think there is anything malicious going on.
We had to restore them from Veeam and now everything is ok. Has anyone run into this before? I’m not sure to be worried or not lol.
How do I prevent this from happening again?
https://redd.it/1pjyyuk
@r_systemadmin
Reddit
From the sysadmin community on Reddit
Explore this post and more from the sysadmin community
Anyone else noticing that vendor support doesn't read tickets these days?
Yesterday, a support case was submitted to a certain Cloud AP Controller company. Can can put my APs on a certain firmware in their old portal, but their new one throws a specific error suggesting they need to enable that feature for me. So, I put in the details necessary so that they can just press the buttons they need to press on their end to enable a feature, or tell me what I need to do to make it work on my own - though Google Fu has me thinking it's the former.
Case arrives with the first technician and they basically reply: "Hello. Can you please provide details of the problem?"
In fairness, this case was opened as a courtesy by another tech after we resolved a different problem, and maybe they didn't relay all the info. So I go back to that email, copy the contents and paste them into this new email.
Ticket is transferred to another tech.
"Hello. What seems to be the problem?"
Copy/paste
Ticket is transferred to another tech.
"Hello. Please share any troubleshooting you have done."
Copy/paste
Now, I'm waiting on a yet another reply, but this is starting to get really old, and it's not just this company. Truthfully, it seems only Cisco is capable of reading ticket history before asking me any questions.
https://redd.it/1pk1buu
@r_systemadmin
Yesterday, a support case was submitted to a certain Cloud AP Controller company. Can can put my APs on a certain firmware in their old portal, but their new one throws a specific error suggesting they need to enable that feature for me. So, I put in the details necessary so that they can just press the buttons they need to press on their end to enable a feature, or tell me what I need to do to make it work on my own - though Google Fu has me thinking it's the former.
Case arrives with the first technician and they basically reply: "Hello. Can you please provide details of the problem?"
In fairness, this case was opened as a courtesy by another tech after we resolved a different problem, and maybe they didn't relay all the info. So I go back to that email, copy the contents and paste them into this new email.
Ticket is transferred to another tech.
"Hello. What seems to be the problem?"
Copy/paste
Ticket is transferred to another tech.
"Hello. Please share any troubleshooting you have done."
Copy/paste
Now, I'm waiting on a yet another reply, but this is starting to get really old, and it's not just this company. Truthfully, it seems only Cisco is capable of reading ticket history before asking me any questions.
https://redd.it/1pk1buu
@r_systemadmin
Reddit
From the sysadmin community on Reddit
Explore this post and more from the sysadmin community
How often do you expire MFA tokens on mobile devices?
We recently migrated our O365 tenant into our parent company. Their cybersecurity posture is much more strict than ours was previously. I now have execs complaining that they have to log into their email/calendar/teams on their phone every 7 days. I'm told this was a compromise because the standard is every 24 hours (mine is every 24 hours since i have a privileged account).
Is this true? Are you making people log into their office applications on their phones every day?
I feel like the MFA fatigue is setting in and people are starting to just respond to any prompt they see now since they get them all the time.
https://redd.it/1pk1zsv
@r_systemadmin
We recently migrated our O365 tenant into our parent company. Their cybersecurity posture is much more strict than ours was previously. I now have execs complaining that they have to log into their email/calendar/teams on their phone every 7 days. I'm told this was a compromise because the standard is every 24 hours (mine is every 24 hours since i have a privileged account).
Is this true? Are you making people log into their office applications on their phones every day?
I feel like the MFA fatigue is setting in and people are starting to just respond to any prompt they see now since they get them all the time.
https://redd.it/1pk1zsv
@r_systemadmin
Reddit
From the sysadmin community on Reddit
Explore this post and more from the sysadmin community
What's the biggest outage you caused?
I'll start.
Job 1: At a college, took down the student management systems in the middle of class enrollment. 15,000 students.
Job 2: Took down the HR systems in the middle of open enrollment. Thankfully it was back up inside of 10 minutes. 45,000 employees.
I sense a theme...
To be fair though, job 2's outage I and others honestly thought what I was doing would not have caused an outage. We even told our contact in HR "just in case". Job 1 was a "oops, wrong window" scenario.
https://redd.it/1pk274p
@r_systemadmin
I'll start.
Job 1: At a college, took down the student management systems in the middle of class enrollment. 15,000 students.
Job 2: Took down the HR systems in the middle of open enrollment. Thankfully it was back up inside of 10 minutes. 45,000 employees.
I sense a theme...
To be fair though, job 2's outage I and others honestly thought what I was doing would not have caused an outage. We even told our contact in HR "just in case". Job 1 was a "oops, wrong window" scenario.
https://redd.it/1pk274p
@r_systemadmin
Reddit
From the sysadmin community on Reddit
Explore this post and more from the sysadmin community
Those out there that still use/capture golden images for deployments... How do you handle updating of the golden image?
As the noscript suggests... I'm mostly asking about how to handle the golden image. You only get 4 SYSPREPs so how often and/or what do you do? It's been ages and we had too many "different" systems to do it properly so we just had one image per system type and we would just run updates after imaging which back then still cut tons of time off just having software pre-installed etc.
I believe technically I could do this:
1. Create my image
2. Clone it, set aside
3. SYSPREP image
4. GRAB the SYSPREPed image and deploy that
5. When Time comes to update the image, use Step 2 and start at Step 1 again, always keeping a 0 count SYSPREP image that I am working off of.
This also ensures that its the same drivers from the jump etc.
https://redd.it/1pk3f22
@r_systemadmin
As the noscript suggests... I'm mostly asking about how to handle the golden image. You only get 4 SYSPREPs so how often and/or what do you do? It's been ages and we had too many "different" systems to do it properly so we just had one image per system type and we would just run updates after imaging which back then still cut tons of time off just having software pre-installed etc.
I believe technically I could do this:
1. Create my image
2. Clone it, set aside
3. SYSPREP image
4. GRAB the SYSPREPed image and deploy that
5. When Time comes to update the image, use Step 2 and start at Step 1 again, always keeping a 0 count SYSPREP image that I am working off of.
This also ensures that its the same drivers from the jump etc.
https://redd.it/1pk3f22
@r_systemadmin
Reddit
From the sysadmin community on Reddit
Explore this post and more from the sysadmin community
LDAPS with Microsoft AD CS: Should applications trust Root CA or Intermediate CA?
Hi,
Let’s assume I need to configure LDAPS for an application, and a certificate is required for this purpose.
We are using a Microsoft two-tier Certificate Authority infrastructure.
On the Domain Controllers, the Kerberos Authentication certificate template is used for LDAPS.
My question is: Which certificate should be used on the application side in this scenario?
Additionally, for applications or appliances, should the Root CA certificate or the Intermediate CA certificate be used?
https://redd.it/1pk7dv0
@r_systemadmin
Hi,
Let’s assume I need to configure LDAPS for an application, and a certificate is required for this purpose.
We are using a Microsoft two-tier Certificate Authority infrastructure.
On the Domain Controllers, the Kerberos Authentication certificate template is used for LDAPS.
My question is: Which certificate should be used on the application side in this scenario?
Additionally, for applications or appliances, should the Root CA certificate or the Intermediate CA certificate be used?
https://redd.it/1pk7dv0
@r_systemadmin
Reddit
From the sysadmin community on Reddit
Explore this post and more from the sysadmin community
Free Windows post-install noscript generator for reproducible setups (+100 apps, configs, debloat)
I maintain a reproducible Windows post-install noscript.
It uses batch and bash for faster, drift-free provisioning.
Eventually, I packaged it into a public, free generator so teams and individuals can export their
own standardized .bat noscript without editing anything.
The generated noscript handles:
• 100+ application installs (winget-based)
• Performance defaults & tuning
• Privacy/telemetry settings
• Explorer/taskbar/UI configuration
• Optional bloatware removal
• Reversible changes
• Zero dependencies — just run the .bat on a fresh Windows install
• Generator runs entirely client-side
It’s not meant to replace enterprise tools like MDT/Intune, but for small teams, home labs, or
personal reproducible setups, it works surprisingly well.
How do you automate turning a fresh Windows image into a usable machine? Is there anything else you’d like to add?
Tool: https://kaic.me/win-post-install/
GitHub: https://github.com/kaic/win-post-install
https://redd.it/1pk8sds
@r_systemadmin
I maintain a reproducible Windows post-install noscript.
It uses batch and bash for faster, drift-free provisioning.
Eventually, I packaged it into a public, free generator so teams and individuals can export their
own standardized .bat noscript without editing anything.
The generated noscript handles:
• 100+ application installs (winget-based)
• Performance defaults & tuning
• Privacy/telemetry settings
• Explorer/taskbar/UI configuration
• Optional bloatware removal
• Reversible changes
• Zero dependencies — just run the .bat on a fresh Windows install
• Generator runs entirely client-side
It’s not meant to replace enterprise tools like MDT/Intune, but for small teams, home labs, or
personal reproducible setups, it works surprisingly well.
How do you automate turning a fresh Windows image into a usable machine? Is there anything else you’d like to add?
Tool: https://kaic.me/win-post-install/
GitHub: https://github.com/kaic/win-post-install
https://redd.it/1pk8sds
@r_systemadmin
kaic.me
Windows Post-Install Script Generator
Automate Windows setup with custom batch noscripts. Free browser tool.
Urgent: Important Security Update for ScreenConnect (Email sent out on December 11, 2025 at 14:46 GMT)
Dear Partner,
ConnectWise has issued a Security Bulletin on our Trust Center regarding a security update for ScreenConnect™ versions prior to 25.8.
This update addresses issues that, under specific conditions, could expose configuration data or allow authorized or administrative users to upload untrusted extensions. The ScreenConnect™ 25.8 patch includes enhancements to how ScreenConnect manages and validates extensions to ensure that only trusted components can be installed.
We strongly recommend that all partners: Upgrade to ScreenConnect™ version 25.8 as soon as possible. Cloud-hosted ScreenConnect instances have already been updated to the latest release. ScreenConnect On-prem partners will need to update manually to 25.8. Visit Download | ScreenConnect page to download and apply the update (access requires a valid on-premises license). If your license is out of maintenance, you must upgrade your license before installing the latest supported release of ScreenConnect. For instructions on updating to the newest release, please reference this doc: Upgrade an on-premise installation - ConnectWise Automate partners with a ScreenConnect integration should verify that their Automate ScreenConnect Extension is updated to version 4.4.0.16 before upgrading to ScreenConnect 25.8. Once the extension is confirmed, partners can visit the Automate Product Updates page to download and apply the ScreenConnect 25.8 update. For instructions on updating to the newest release, please reference this doc: Upgrade an on-premise installation - ConnectWise Link to release notes: ScreenConnect release notes - ConnectWise Review the Security Bulletin for additional details. For help with upgrading visit ConnectWise Chat to open a case or email help@connectwise.com for additional support.
ConnectWise Security Bulletin Please refer to the Security Bulletin posted to our Trust Center regarding this vulnerability for more detailed information.
Stay informed We are committed to transparency and will keep you informed of any further developments. For real-time updates, please subscribe to the ConnectWise security bulletin RSS feed.
Report a security incident To report a security or privacy incident, please visit the ConnectWise Trust Center.
We appreciate your continued partnership and trust in our products and services.
Thank you, ScreenConnect Team
https://redd.it/1pka0f9
@r_systemadmin
Dear Partner,
ConnectWise has issued a Security Bulletin on our Trust Center regarding a security update for ScreenConnect™ versions prior to 25.8.
This update addresses issues that, under specific conditions, could expose configuration data or allow authorized or administrative users to upload untrusted extensions. The ScreenConnect™ 25.8 patch includes enhancements to how ScreenConnect manages and validates extensions to ensure that only trusted components can be installed.
We strongly recommend that all partners: Upgrade to ScreenConnect™ version 25.8 as soon as possible. Cloud-hosted ScreenConnect instances have already been updated to the latest release. ScreenConnect On-prem partners will need to update manually to 25.8. Visit Download | ScreenConnect page to download and apply the update (access requires a valid on-premises license). If your license is out of maintenance, you must upgrade your license before installing the latest supported release of ScreenConnect. For instructions on updating to the newest release, please reference this doc: Upgrade an on-premise installation - ConnectWise Automate partners with a ScreenConnect integration should verify that their Automate ScreenConnect Extension is updated to version 4.4.0.16 before upgrading to ScreenConnect 25.8. Once the extension is confirmed, partners can visit the Automate Product Updates page to download and apply the ScreenConnect 25.8 update. For instructions on updating to the newest release, please reference this doc: Upgrade an on-premise installation - ConnectWise Link to release notes: ScreenConnect release notes - ConnectWise Review the Security Bulletin for additional details. For help with upgrading visit ConnectWise Chat to open a case or email help@connectwise.com for additional support.
ConnectWise Security Bulletin Please refer to the Security Bulletin posted to our Trust Center regarding this vulnerability for more detailed information.
Stay informed We are committed to transparency and will keep you informed of any further developments. For real-time updates, please subscribe to the ConnectWise security bulletin RSS feed.
Report a security incident To report a security or privacy incident, please visit the ConnectWise Trust Center.
We appreciate your continued partnership and trust in our products and services.
Thank you, ScreenConnect Team
https://redd.it/1pka0f9
@r_systemadmin
Reddit
From the sysadmin community on Reddit
Explore this post and more from the sysadmin community
Do you enjoy your job?
With all the “I’m burnt out” notions going around in tech, is there any positivity to go with this?
Are you able to work from home if you choose? Can you go into the office jf you choose?
Do you clock in at 9 and out by 5? Or are you on call?
Do you feel you have job security or always on edge?
Is AI going to be the I ROBOT sequel and take over our roles?
Now I hope this doesn’t turn into another IT hate thread, aiming for some good vibes
https://redd.it/1pkaaah
@r_systemadmin
With all the “I’m burnt out” notions going around in tech, is there any positivity to go with this?
Are you able to work from home if you choose? Can you go into the office jf you choose?
Do you clock in at 9 and out by 5? Or are you on call?
Do you feel you have job security or always on edge?
Is AI going to be the I ROBOT sequel and take over our roles?
Now I hope this doesn’t turn into another IT hate thread, aiming for some good vibes
https://redd.it/1pkaaah
@r_systemadmin
Reddit
From the sysadmin community on Reddit
Explore this post and more from the sysadmin community
Large company culture
So I took a senior admin job with a large company. Over 10k employees and a worldwide place etc.
Well, so far ive been there a month and am not really happy. Let me explain.
1. Keep being treated as if im new to IT. No access to half of the systems I need to work with.
2. Gatekeeping team. "Oh, well only bill does that. If you get a ticket on it just re assign. No we cant give you access to x systems.
3. Given 0 projects. 0 tickets. Month in. Literally today someone told me I could grab a ticket if I wanted. The tickets I can actually do with the access I have would be stupid things like expand a disk or add someone to a group.
4. Teams for every little thing. There is an o365 team. An iam/sso team. Phones team. Helpdesk line team. Desk side team. Network team. Security team. Ass wipe team. Piss team. You want to do anything nope... that's x team.
5. It doesnt make a difference if im there or not. Nothing is expected of me. No one cares how long your lunch is. Or when you start and stop.
6. Manager keeps saying how there is sooooo much work. OK where the fuck is it? Then im told they will get it going this week. Nope....
7. Im probably more experienced and capable at various things on my team yet im not allowed to even participate in any of it.
8. Again I was hired as a senior level admin making well over six figures and this company is completely wasting their money. I've never seen anything like this in my career. Im 40.
People who went to a big Corp after smaller or medium size places where you actually..... worked..... and fixed things.... does it get better? I hear some like and prefer this. I don't understand how you do? Im going to try to give it more time. One month is not enough. But I mean it feels like im going to end up being just a tier 3 helpdesk or some weird shit. Or like this is all an elaborate scam but my checks are still clearing.
https://redd.it/1pkd55q
@r_systemadmin
So I took a senior admin job with a large company. Over 10k employees and a worldwide place etc.
Well, so far ive been there a month and am not really happy. Let me explain.
1. Keep being treated as if im new to IT. No access to half of the systems I need to work with.
2. Gatekeeping team. "Oh, well only bill does that. If you get a ticket on it just re assign. No we cant give you access to x systems.
3. Given 0 projects. 0 tickets. Month in. Literally today someone told me I could grab a ticket if I wanted. The tickets I can actually do with the access I have would be stupid things like expand a disk or add someone to a group.
4. Teams for every little thing. There is an o365 team. An iam/sso team. Phones team. Helpdesk line team. Desk side team. Network team. Security team. Ass wipe team. Piss team. You want to do anything nope... that's x team.
5. It doesnt make a difference if im there or not. Nothing is expected of me. No one cares how long your lunch is. Or when you start and stop.
6. Manager keeps saying how there is sooooo much work. OK where the fuck is it? Then im told they will get it going this week. Nope....
7. Im probably more experienced and capable at various things on my team yet im not allowed to even participate in any of it.
8. Again I was hired as a senior level admin making well over six figures and this company is completely wasting their money. I've never seen anything like this in my career. Im 40.
People who went to a big Corp after smaller or medium size places where you actually..... worked..... and fixed things.... does it get better? I hear some like and prefer this. I don't understand how you do? Im going to try to give it more time. One month is not enough. But I mean it feels like im going to end up being just a tier 3 helpdesk or some weird shit. Or like this is all an elaborate scam but my checks are still clearing.
https://redd.it/1pkd55q
@r_systemadmin
Reddit
From the sysadmin community on Reddit
Explore this post and more from the sysadmin community
Reminder that AI can cause outages
Not an anti-AI post. I use it too. But I’ve now seen multiple cases where people blindly followed AI advice and it directly caused outages.
The core issue is simple: AI really wants to be helpful and sound correct. It does not like saying “I don’t know,” and it usually doesn’t lead with “this depends” or “check the vendor docs.” Instead, it gives very generic, confident-sounding answers that might apply… or might be completely wrong for your environment.
What I’m seeing lately is people using AI as a replacement for vendor documentation instead of a supplement. They’ll skip official docs because “AI already explained it” and then go change something in prod.
That’s how you end up breaking things.
AI doesn’t know: your firmware versions, your licensing, your exact product SKU, your vendor’s weird limitations, the 20-year-old legacy system someone put in place and never documented.
It just predicts an answer that sounds right.
Some patterns I’ve personally seen:
- generic registry or firewall changes applied without understanding side effects
- assumptions that features work the same across different vendors or versions
- config changes that directly contradict the vendor’s own “do not do this in production” notes
- people trusting AI output more than official documentation because it’s faster to read
AI is fine for:
- explaining what something does
- summarizing docs you already trust
- helping you think through risks
- sanity-checking an idea
AI is dangerous for:
- “tell me exactly what to change”
- “this is faster than reading the docs”
- production changes without validation
Treat AI like a junior admin who’s confident but doesn’t know your environment. Useful, but you still check their work.
Curious if others are starting to see this pop up too.
https://redd.it/1pke3mb
@r_systemadmin
Not an anti-AI post. I use it too. But I’ve now seen multiple cases where people blindly followed AI advice and it directly caused outages.
The core issue is simple: AI really wants to be helpful and sound correct. It does not like saying “I don’t know,” and it usually doesn’t lead with “this depends” or “check the vendor docs.” Instead, it gives very generic, confident-sounding answers that might apply… or might be completely wrong for your environment.
What I’m seeing lately is people using AI as a replacement for vendor documentation instead of a supplement. They’ll skip official docs because “AI already explained it” and then go change something in prod.
That’s how you end up breaking things.
AI doesn’t know: your firmware versions, your licensing, your exact product SKU, your vendor’s weird limitations, the 20-year-old legacy system someone put in place and never documented.
It just predicts an answer that sounds right.
Some patterns I’ve personally seen:
- generic registry or firewall changes applied without understanding side effects
- assumptions that features work the same across different vendors or versions
- config changes that directly contradict the vendor’s own “do not do this in production” notes
- people trusting AI output more than official documentation because it’s faster to read
AI is fine for:
- explaining what something does
- summarizing docs you already trust
- helping you think through risks
- sanity-checking an idea
AI is dangerous for:
- “tell me exactly what to change”
- “this is faster than reading the docs”
- production changes without validation
Treat AI like a junior admin who’s confident but doesn’t know your environment. Useful, but you still check their work.
Curious if others are starting to see this pop up too.
https://redd.it/1pke3mb
@r_systemadmin
Reddit
From the sysadmin community on Reddit
Explore this post and more from the sysadmin community
The Server Was “Obstructed”
Another story from Healthcare IT, in a previous role of mine.
We were going through our regular maintenance tasks, and noticed an alert in Dell OpenManage about a failed CMOS battery for one of our clinic’s servers.
[that looks like this](https://www.experts-exchange.com/questions/29133977/The-system-board-battery-has-failed-on-DELL-Poweredge-T320.html).
For context:
* Each of our clinic locations had 2 HyperV servers, setup to replicate to each other every few minutes.
* One of the servers was generally fairly modern and powerful, while the other was whatever we could scrap together to run legacy clinic VM’s, and be a replication partner – so we could fail over to it if something went bad.
* Each clinic had zero onsite IT staff, and often the nearest IT person was an hour drive away, they also had really dated Network links – I’m talking 10-20Mbit (in 2022).
* In many cases, the hardware was 10+ years old and EoL, and the software usually was too, we had plenty of 2008R2 and 2012R2 hosts/VM’s out there, so things broke regularly – the business was well aware of the risks of this.
Anyway, because we had servers in so many locations, we contracted out an external vendor to complete our hands on server maintenance tasks, let’s call our vendor **Outeractive**.
So when we saw the server alert, we followed our usual process:
* Log the issue on our maintenance tasks board.
* Fail-over any virtual machines from the problematic host to the replica, outside hours (this needed a change request).
* Create a service request to **Outeractive** on the following day, who would usually provide an ETA.
* Contact the clinic manager to let them know someone would be coming in to access the server room.
* Respond to any calls from **Outeractive**, providing them directions to the clinic site if needed (yes, we actually had to do this).
* Shutdown the affected host as **Outeractive** arrive onsite (so we have the most up-to-date possible replicas).
* **Outeractive** replace the required part.
* We do a final health check, and then schedule to fail back over the VM’s outside hours again.
# So our vendor arrived onsite…
We received a call from **Outeractive** as they arrived and were about to start the work, all was going well, and we left them to it.
Then they called back 10 minutes later.
>We can’t access the server.
Huh, what do you mean you can’t access the server?
Do you need us to speak to the clinic manager for the key?
>No no, we physically can’t get to the server, it’s obstructed.
It should be in the rack, able to slide right out, can you send us a photo of what you mean?
>Yep
[https://imgur.com/ZdoOQGx](https://imgur.com/ZdoOQGx)
This photo got shared around the office pretty quickly, and is pretty funny now that I’m seeing it again.
So the server that **Outeractive** needed to get to was wedged in between the UPS and another server/shelf.
So the only way to get to it safely, would be to somehow suspend the newer server that’s above it, and then lift out the older server from underneath.
To be clear, this is the server **Outeractive** had to replace parts in, and they needed clear access to the side panel, not just the front or back.
[Here’s another image](https://imgur.com/7iwWHiT) of all this, but from the side, the server in the middle, is basically unable to be safely removed/reinstalled without impacting the server above it.
# What do we do next?
Well, the most important thing anyone in Healthcare IT will say to you, is that we can **never** lose patient/clinical data.
This made any further actions from our **Outeractive** technician extremely high risk, so we organized with him to reschedule, and attend the site ourselves.
# Why was it high risk for a vendor to touch?
Remember earlier when I said our clinics only have 10-20Mbit links? – Yep, that applies to this site, and limited our offsite backup capabilities, you should know:
* The live database for this entire \~15 staff clinic was running on the top server. The clinic is currently trying to operate, seeing patients,
Another story from Healthcare IT, in a previous role of mine.
We were going through our regular maintenance tasks, and noticed an alert in Dell OpenManage about a failed CMOS battery for one of our clinic’s servers.
[that looks like this](https://www.experts-exchange.com/questions/29133977/The-system-board-battery-has-failed-on-DELL-Poweredge-T320.html).
For context:
* Each of our clinic locations had 2 HyperV servers, setup to replicate to each other every few minutes.
* One of the servers was generally fairly modern and powerful, while the other was whatever we could scrap together to run legacy clinic VM’s, and be a replication partner – so we could fail over to it if something went bad.
* Each clinic had zero onsite IT staff, and often the nearest IT person was an hour drive away, they also had really dated Network links – I’m talking 10-20Mbit (in 2022).
* In many cases, the hardware was 10+ years old and EoL, and the software usually was too, we had plenty of 2008R2 and 2012R2 hosts/VM’s out there, so things broke regularly – the business was well aware of the risks of this.
Anyway, because we had servers in so many locations, we contracted out an external vendor to complete our hands on server maintenance tasks, let’s call our vendor **Outeractive**.
So when we saw the server alert, we followed our usual process:
* Log the issue on our maintenance tasks board.
* Fail-over any virtual machines from the problematic host to the replica, outside hours (this needed a change request).
* Create a service request to **Outeractive** on the following day, who would usually provide an ETA.
* Contact the clinic manager to let them know someone would be coming in to access the server room.
* Respond to any calls from **Outeractive**, providing them directions to the clinic site if needed (yes, we actually had to do this).
* Shutdown the affected host as **Outeractive** arrive onsite (so we have the most up-to-date possible replicas).
* **Outeractive** replace the required part.
* We do a final health check, and then schedule to fail back over the VM’s outside hours again.
# So our vendor arrived onsite…
We received a call from **Outeractive** as they arrived and were about to start the work, all was going well, and we left them to it.
Then they called back 10 minutes later.
>We can’t access the server.
Huh, what do you mean you can’t access the server?
Do you need us to speak to the clinic manager for the key?
>No no, we physically can’t get to the server, it’s obstructed.
It should be in the rack, able to slide right out, can you send us a photo of what you mean?
>Yep
[https://imgur.com/ZdoOQGx](https://imgur.com/ZdoOQGx)
This photo got shared around the office pretty quickly, and is pretty funny now that I’m seeing it again.
So the server that **Outeractive** needed to get to was wedged in between the UPS and another server/shelf.
So the only way to get to it safely, would be to somehow suspend the newer server that’s above it, and then lift out the older server from underneath.
To be clear, this is the server **Outeractive** had to replace parts in, and they needed clear access to the side panel, not just the front or back.
[Here’s another image](https://imgur.com/7iwWHiT) of all this, but from the side, the server in the middle, is basically unable to be safely removed/reinstalled without impacting the server above it.
# What do we do next?
Well, the most important thing anyone in Healthcare IT will say to you, is that we can **never** lose patient/clinical data.
This made any further actions from our **Outeractive** technician extremely high risk, so we organized with him to reschedule, and attend the site ourselves.
# Why was it high risk for a vendor to touch?
Remember earlier when I said our clinics only have 10-20Mbit links? – Yep, that applies to this site, and limited our offsite backup capabilities, you should know:
* The live database for this entire \~15 staff clinic was running on the top server. The clinic is currently trying to operate, seeing patients,
Experts-Exchange
Solved: The system board battery has failed on DELL Poweredge T320 | Experts Exchange
Find answers to The system board battery has failed on DELL Poweredge T320 from the expert community at Experts Exchange