PSA: Recent Windows 10 update force-binds Copilot to Alt+C
If you have an app that uses Alt+C or happen to be Polish (unable to type "ć" as it is bound to Alt + C on the polish keyboard) and also happen to still have Windows 10 on some devices and you have not uninstalled Copilot from them yet, you are gonna stumble upon a funny situation / start getting not so funny calls soon.
There is no official solution apart from from uninstalling/disabling the Copilot app as of today. The issue does not occur on Windows 11.
My org was hit today but apparently others got hit earlier - relevant MS Q&A thread (in Polish): https://learn.microsoft.com/pl-pl/answers/questions/5541180/jak-wy-czy-skr-t-prawy-alt-c-uruchamiajacy-now-kon
https://redd.it/1ntcc2h
@r_systemadmin
If you have an app that uses Alt+C or happen to be Polish (unable to type "ć" as it is bound to Alt + C on the polish keyboard) and also happen to still have Windows 10 on some devices and you have not uninstalled Copilot from them yet, you are gonna stumble upon a funny situation / start getting not so funny calls soon.
There is no official solution apart from from uninstalling/disabling the Copilot app as of today. The issue does not occur on Windows 11.
My org was hit today but apparently others got hit earlier - relevant MS Q&A thread (in Polish): https://learn.microsoft.com/pl-pl/answers/questions/5541180/jak-wy-czy-skr-t-prawy-alt-c-uruchamiajacy-now-kon
https://redd.it/1ntcc2h
@r_systemadmin
Microsoft
Jak wyłączyć skrót Prawy ALT+c uruchamiajacy nową konwersację w COPILOT - Microsoft Q&A
Jak wyłączyć skrót klawiaturowy Prawy ALT+c żeby nie uruchamiał nowej konwersacji w COPILOT tylko wprowadzał polski znak "ć" (małe c z kreską). Jak Copilot jest całkowicie wyłączony to skrót działa poprawnie i daje literę ć, ale wystarczy że…
Font foundries and licensing
Those of you who use custom font foundries and host websites - how does one navigate the complicated font licensing world?
E.g.we want to use a font owned by Adobe. Adobe has three resellers and each gave us a different licensing interpretation and wildly different quotes. I want to host the font due to security requirements, use it in internal/dev sites, use it for official document templates.
https://redd.it/1ntcwvl
@r_systemadmin
Those of you who use custom font foundries and host websites - how does one navigate the complicated font licensing world?
E.g.we want to use a font owned by Adobe. Adobe has three resellers and each gave us a different licensing interpretation and wildly different quotes. I want to host the font due to security requirements, use it in internal/dev sites, use it for official document templates.
https://redd.it/1ntcwvl
@r_systemadmin
Reddit
From the sysadmin community on Reddit
Explore this post and more from the sysadmin community
Moronic Monday - September 29, 2025
Howdy, /r/sysadmin!
It's that time of the week, Moronic Monday! This is a safe (mostly) judgement-free environment for all of your questions and stories, no matter how silly you think they are. Anybody can answer questions! My name is AutoModerator and I've taken over responsibility for posting these weekly threads so you don't have to worry about anything except your comments!
https://redd.it/1ntdyto
@r_systemadmin
Howdy, /r/sysadmin!
It's that time of the week, Moronic Monday! This is a safe (mostly) judgement-free environment for all of your questions and stories, no matter how silly you think they are. Anybody can answer questions! My name is AutoModerator and I've taken over responsibility for posting these weekly threads so you don't have to worry about anything except your comments!
https://redd.it/1ntdyto
@r_systemadmin
Reddit
From the sysadmin community on Reddit
Explore this post and more from the sysadmin community
Microsoft Licenses / CSPs
We currently use Trusted Tech Team. We are ok with them, but we also want to make sure we are getting the best price possible. Your milage may vary, but on average are you willing to share how much you are paying monthly for and O365 E3? We are paying $30.96.
https://redd.it/1ntftob
@r_systemadmin
We currently use Trusted Tech Team. We are ok with them, but we also want to make sure we are getting the best price possible. Your milage may vary, but on average are you willing to share how much you are paying monthly for and O365 E3? We are paying $30.96.
https://redd.it/1ntftob
@r_systemadmin
Reddit
From the sysadmin community on Reddit
Explore this post and more from the sysadmin community
Is it just me, or does working in operations always come with having to babysit the helpdesk, no matter what position you move into?
I'm trying to move on from IT helpdesk, and while I'm technically no longer doing frontline support, I still get pulled back into it.
I work in operations now, but I'm stuck handling escalated tickets from the helpdesk and often end up babysitting the whole process. I don't do helpdesk work anymore, but I can't fully escape it either.
Now I'm being told I need to get ITIL certified. I'm starting to wonder if I've made a mistake in this transition. I just want to focus on real operations work or get into system builds and infrastructure. I'm honestly burnt out from anything helpdesk-related.
Has anyone else been in this situation? How did you get out of the helpdesk shadow for good
https://redd.it/1ntgtup
@r_systemadmin
I'm trying to move on from IT helpdesk, and while I'm technically no longer doing frontline support, I still get pulled back into it.
I work in operations now, but I'm stuck handling escalated tickets from the helpdesk and often end up babysitting the whole process. I don't do helpdesk work anymore, but I can't fully escape it either.
Now I'm being told I need to get ITIL certified. I'm starting to wonder if I've made a mistake in this transition. I just want to focus on real operations work or get into system builds and infrastructure. I'm honestly burnt out from anything helpdesk-related.
Has anyone else been in this situation? How did you get out of the helpdesk shadow for good
https://redd.it/1ntgtup
@r_systemadmin
Reddit
From the sysadmin community on Reddit
Explore this post and more from the sysadmin community
What are some cheaper docking stations that you recommend?
We allow our users to work hybrid. We provide everyone with an in office setup, but if they want to be hybrid, we do not provide a setup for at home. Some people just use their laptop at home, but recently we've been getting asked for recommendations on what to buy for home setups that are the same as work.
There is a PC salvage place near by that they grab decent monitors for $30-40 each. The salvage place never has any docks. Most people don't want to shell out the $175-250 for a new Dell dock.
I personally don't know much about docks outside of what I use at work which are WD19 and P2424HEB conference monitors.
Does anyone know of any decent docks that work with Dell Latitude 5420,5440, and 5450's that are on the cheaper side of things? under 75? under 50?
https://redd.it/1ntgthn
@r_systemadmin
We allow our users to work hybrid. We provide everyone with an in office setup, but if they want to be hybrid, we do not provide a setup for at home. Some people just use their laptop at home, but recently we've been getting asked for recommendations on what to buy for home setups that are the same as work.
There is a PC salvage place near by that they grab decent monitors for $30-40 each. The salvage place never has any docks. Most people don't want to shell out the $175-250 for a new Dell dock.
I personally don't know much about docks outside of what I use at work which are WD19 and P2424HEB conference monitors.
Does anyone know of any decent docks that work with Dell Latitude 5420,5440, and 5450's that are on the cheaper side of things? under 75? under 50?
https://redd.it/1ntgthn
@r_systemadmin
Reddit
From the sysadmin community on Reddit
Explore this post and more from the sysadmin community
Applying for Work in Today's Reality - A Tale of Two Job Openings
Context: I am a happily-employed person who is a hiring manager for technical roles in my division of a large global company. My notes below compare two recent roles I hired and hopefully provides some useful context to help those of you searching today get past some invisible barriers.
In the last few weeks, I've been through several rounds of interviews for a pair of open roles. Both were highly technical in nature and at every single step, they could not have gone more differently.
Role #1 - <Well Known ERP> Developer. Posting up for under a day, 2k+ resumes. Did all 2k get read? Absolutely not. It's not possible. After initially tossing plagarized resumes and completely non-applicable ones, HR read as many as they needed to match a handful of people to our skill matrix and screened them. They scheduled 5 over the next 2 weeks, working around the candidate schedule and ours.
One was great, but accepted an offer before we got through the rest. One was good, and we sent to round two. One showed up with an AI recording device active without mentioning it, and blatantly read us ChatGPT answers. (Hint: You might bluff HR, but the hiring manager will know. Knock that crap off.);4 and 5 were good, but not a match for our environment overall. If we see another open role that fits them, they'll get a call to see if they're interested.
HR pulled a few more, and one we side-barred literally mid-interview. I said I didn't care what the rules were, I wanted an offer on the table by the next day. They start in a few weeks, and the whole team is delighted.
What made candidates struggle to be seen in this scenario?
Firstly, AI-generated resumes, bot-nets representing applicants, humans plagarizing resumes, and humans spam-applying to every single role whether they match or not affect genuine candidates badly. You are a shining light in a pile of bullshit, and sadly there's a lot more of it than there is of you.
Secondly, we scoped this role to only require 3-5 years experience. The base skillset was one that can be self-studied, paper certified, and be honestly obtained without in-role professional experience. (I can say that because that's exactly how I learned it, once upon a time.)
None of that is bad or wrong, but it's an awful market right now. Even once we work past AI-generated resumes, bot-nets and spam applicants, you're up against actual peers in skill and for well-known tech there's a lot of y'all. That's before layoffs, where people with 3-4x your XP are applying too.
The one trait that really made candidates stand out in this category was their ability to show they understood the business context of how the technology is used. As an example, we brought up the vendor's plans to deprecate a very significant feature we rely heavily on in the next 1-2 years. We asked if they'd read about that or had any experience with a shift away from that feature.
To be clear, for a role with that level of XP, I never expected to have someone say, 'Yes, I've done that project...'. I was listening for something that let me know they understood how complex it was in general.
The candidates that winced, or somehow acknowledged how major/painful a project that would be were the ones we knew understood that feature, even without any technical answers.
Role #2 - <Large-but-Niche Proj Mgmt Tool> System Admin. HR told me they would pull the posting in a day expecting 1k+ resumes. I somehow kept the subnoscripts off my face and said we'd see how it went. 5 days later, we had 57 resumes. Most of those were from posts I'd personally made in forums for that specific technology. I personally read all 57. 2 I rejected as submitting plagarized resumes, and 3 were WILDLY unrelated (think 'car mechanic' applying for a Jira API developer role.)
From there, 14 made it to round 1 as resumes that listed experience in that tool. I asked HR to screen 5. One more reached out to me directly after the posting ended, and I sent them to screening because they were professionally known
Context: I am a happily-employed person who is a hiring manager for technical roles in my division of a large global company. My notes below compare two recent roles I hired and hopefully provides some useful context to help those of you searching today get past some invisible barriers.
In the last few weeks, I've been through several rounds of interviews for a pair of open roles. Both were highly technical in nature and at every single step, they could not have gone more differently.
Role #1 - <Well Known ERP> Developer. Posting up for under a day, 2k+ resumes. Did all 2k get read? Absolutely not. It's not possible. After initially tossing plagarized resumes and completely non-applicable ones, HR read as many as they needed to match a handful of people to our skill matrix and screened them. They scheduled 5 over the next 2 weeks, working around the candidate schedule and ours.
One was great, but accepted an offer before we got through the rest. One was good, and we sent to round two. One showed up with an AI recording device active without mentioning it, and blatantly read us ChatGPT answers. (Hint: You might bluff HR, but the hiring manager will know. Knock that crap off.);4 and 5 were good, but not a match for our environment overall. If we see another open role that fits them, they'll get a call to see if they're interested.
HR pulled a few more, and one we side-barred literally mid-interview. I said I didn't care what the rules were, I wanted an offer on the table by the next day. They start in a few weeks, and the whole team is delighted.
What made candidates struggle to be seen in this scenario?
Firstly, AI-generated resumes, bot-nets representing applicants, humans plagarizing resumes, and humans spam-applying to every single role whether they match or not affect genuine candidates badly. You are a shining light in a pile of bullshit, and sadly there's a lot more of it than there is of you.
Secondly, we scoped this role to only require 3-5 years experience. The base skillset was one that can be self-studied, paper certified, and be honestly obtained without in-role professional experience. (I can say that because that's exactly how I learned it, once upon a time.)
None of that is bad or wrong, but it's an awful market right now. Even once we work past AI-generated resumes, bot-nets and spam applicants, you're up against actual peers in skill and for well-known tech there's a lot of y'all. That's before layoffs, where people with 3-4x your XP are applying too.
The one trait that really made candidates stand out in this category was their ability to show they understood the business context of how the technology is used. As an example, we brought up the vendor's plans to deprecate a very significant feature we rely heavily on in the next 1-2 years. We asked if they'd read about that or had any experience with a shift away from that feature.
To be clear, for a role with that level of XP, I never expected to have someone say, 'Yes, I've done that project...'. I was listening for something that let me know they understood how complex it was in general.
The candidates that winced, or somehow acknowledged how major/painful a project that would be were the ones we knew understood that feature, even without any technical answers.
Role #2 - <Large-but-Niche Proj Mgmt Tool> System Admin. HR told me they would pull the posting in a day expecting 1k+ resumes. I somehow kept the subnoscripts off my face and said we'd see how it went. 5 days later, we had 57 resumes. Most of those were from posts I'd personally made in forums for that specific technology. I personally read all 57. 2 I rejected as submitting plagarized resumes, and 3 were WILDLY unrelated (think 'car mechanic' applying for a Jira API developer role.)
From there, 14 made it to round 1 as resumes that listed experience in that tool. I asked HR to screen 5. One more reached out to me directly after the posting ended, and I sent them to screening because they were professionally known
to me via networking. (Cheat-code here.) HR passed 3 of the 6 and I overruled to add one more to the pile. Those 4 all met me last week.
3 of them go to final round this week, and I'm already lobbying for 2 of them, if not all 3 to be placed somewhere in our org. I expect to tell HR to make an offer by Friday for the first one.
What made this role so very different from the first?
Primarily, the vendor has no option that allows someone to have hands-on time with the tool unless they work for a company that licenses it. You can read documentation or take their classes, but that's about it. That dramatically limits the applicant pool right away and also means the hiring manager really needs someone with experience.
Secondly, that the tool is not incredibly complex from a technical standpoint. An admin CAN do wildly complicated things, but the basic setup doesn't require a full IT background. Making that platform work effectively is way more about understanding how the users will interact with it to support business needs. That kind of collaboration with end-users is a very different model than a pure dev role.
On the complex side, there is a component of that tool that IS both highly complex and rare. I would have loved to get candidates with experience in it. But I also knew how rare it was, so HR were told to prioritize resumes that listed it but also pass resumes that had a specific list of other comparable tools. Ultimately no candidate had experience in it, but they all expressed excitement to get to work with it and frustration that their current firms wouldn't license it.
Takeaways:
Picking up a broadly applicable set of skills/technologies is good, but right now it's getting you buried in AI/bot traffic. You aren't doing anything wrong, the scammers/AI bots are, but real people are sadly paying for that. Getting past that barrier is hard, you either get called at random or you circumvent it entirely via technical/professional networking.
Applying for roles where you don't match the requirements can work in a strong market where we have time to teach. This isn't that market today. I'm sure the candidates I rejected could learn quickly, I just don't have time. If you send in a resume thinking, 'I know I could learn that fast!' You're probably right. But if I have to make a call between a candidate with 10 years experience in the platform, and teaching someone from scratch? My sanity needs the experienced one.
Learning less common technologies or platforms can be seen as a waste of time, but it can also be the difference between being one of 2k+ resumes and 57 resumes read directly by the hiring manager even before the HR screen.
I'm hoping that my notes and details here help those of you searching today to refine how you look. If there are questions/clarifications in comments, I'll answer as I can. (It's also Monday, so please pack patience! I might not be free until after hours for any long answers.)
https://redd.it/1ntk92q
@r_systemadmin
3 of them go to final round this week, and I'm already lobbying for 2 of them, if not all 3 to be placed somewhere in our org. I expect to tell HR to make an offer by Friday for the first one.
What made this role so very different from the first?
Primarily, the vendor has no option that allows someone to have hands-on time with the tool unless they work for a company that licenses it. You can read documentation or take their classes, but that's about it. That dramatically limits the applicant pool right away and also means the hiring manager really needs someone with experience.
Secondly, that the tool is not incredibly complex from a technical standpoint. An admin CAN do wildly complicated things, but the basic setup doesn't require a full IT background. Making that platform work effectively is way more about understanding how the users will interact with it to support business needs. That kind of collaboration with end-users is a very different model than a pure dev role.
On the complex side, there is a component of that tool that IS both highly complex and rare. I would have loved to get candidates with experience in it. But I also knew how rare it was, so HR were told to prioritize resumes that listed it but also pass resumes that had a specific list of other comparable tools. Ultimately no candidate had experience in it, but they all expressed excitement to get to work with it and frustration that their current firms wouldn't license it.
Takeaways:
Picking up a broadly applicable set of skills/technologies is good, but right now it's getting you buried in AI/bot traffic. You aren't doing anything wrong, the scammers/AI bots are, but real people are sadly paying for that. Getting past that barrier is hard, you either get called at random or you circumvent it entirely via technical/professional networking.
Applying for roles where you don't match the requirements can work in a strong market where we have time to teach. This isn't that market today. I'm sure the candidates I rejected could learn quickly, I just don't have time. If you send in a resume thinking, 'I know I could learn that fast!' You're probably right. But if I have to make a call between a candidate with 10 years experience in the platform, and teaching someone from scratch? My sanity needs the experienced one.
Learning less common technologies or platforms can be seen as a waste of time, but it can also be the difference between being one of 2k+ resumes and 57 resumes read directly by the hiring manager even before the HR screen.
I'm hoping that my notes and details here help those of you searching today to refine how you look. If there are questions/clarifications in comments, I'll answer as I can. (It's also Monday, so please pack patience! I might not be free until after hours for any long answers.)
https://redd.it/1ntk92q
@r_systemadmin
Reddit
From the sysadmin community on Reddit
Explore this post and more from the sysadmin community
Do you use an Enterprise Password Manager for hundreds or thousands of employees?
Hi,
The company I work for chose LastPass for our enterprise password manager a couple years ago. It sucks and everyone hates it. The person who has taken over the ownership of it wants to find something else. I used LastPass personal for a while, until they were dumb and I then changed to Bitwarden and never looked back. I know BW has an enterprise version, but I've never used it so can't speak to how well, or not, it works.
I'm just wondering what Password Manager other people might be using and how well they work. The main issue is how things are owned and shared amongst other people or teams in the company. I'm told we have 1000-1500 users and 4000+ actual passwords in the system. We need to have a good way to share the entries with other people so we don't have duplicates. We don't have that now which causes issues when I change a password and then break something for 10 other people who have duplicate entries for the system that I didn't know about and can't see myself.
Anyway, just looking for ideas.
Thanks.
https://redd.it/1ntkwvk
@r_systemadmin
Hi,
The company I work for chose LastPass for our enterprise password manager a couple years ago. It sucks and everyone hates it. The person who has taken over the ownership of it wants to find something else. I used LastPass personal for a while, until they were dumb and I then changed to Bitwarden and never looked back. I know BW has an enterprise version, but I've never used it so can't speak to how well, or not, it works.
I'm just wondering what Password Manager other people might be using and how well they work. The main issue is how things are owned and shared amongst other people or teams in the company. I'm told we have 1000-1500 users and 4000+ actual passwords in the system. We need to have a good way to share the entries with other people so we don't have duplicates. We don't have that now which causes issues when I change a password and then break something for 10 other people who have duplicate entries for the system that I didn't know about and can't see myself.
Anyway, just looking for ideas.
Thanks.
https://redd.it/1ntkwvk
@r_systemadmin
Reddit
From the sysadmin community on Reddit
Explore this post and more from the sysadmin community
Migrating Group Policies into Microsoft Intune?
Hey everyone, I’m looking for some advice. I just got thrown into an Intune Autopilot project after the person who was handling it before broke his leg, and I’m a bit lost. Does anyone here have experience with this or know of a solid guide I could follow? Any help would be hugely appreciated!
https://redd.it/1ntki4r
@r_systemadmin
Hey everyone, I’m looking for some advice. I just got thrown into an Intune Autopilot project after the person who was handling it before broke his leg, and I’m a bit lost. Does anyone here have experience with this or know of a solid guide I could follow? Any help would be hugely appreciated!
https://redd.it/1ntki4r
@r_systemadmin
Reddit
From the sysadmin community on Reddit
Explore this post and more from the sysadmin community
Worthless MSP
So we outsourced our help desk to a worthless MSP. These people are so incompetent they can’t reset basic 365 passwords. Yet we give them admin access.
Any good MSPs out there that can be trusted?
https://redd.it/1ntoayr
@r_systemadmin
So we outsourced our help desk to a worthless MSP. These people are so incompetent they can’t reset basic 365 passwords. Yet we give them admin access.
Any good MSPs out there that can be trusted?
https://redd.it/1ntoayr
@r_systemadmin
Reddit
From the sysadmin community on Reddit
Explore this post and more from the sysadmin community
Follow Up: The Previous Network Administrator 'Didn't Believe in VLANs'
Hello again. I posted this a while back and people seemed to enjoy reading it. Here's a follow up with some progress and more jank I've discovered since. This is not an exhaustive list of jank or progress, just stuff I thought was particularity funny.
Chat/IM
A serverless chat client that operated via multicast was in use and installed on all workstations. It kept local logs of all chats on each workstation in plaintext and used no authentication whatsoever. You set your own nickname and that got reported to all other online clients. Do you want to be the HR manager today? That was just two clicks away! (The HR manager reached out to me on the chat app my first day and asked. “Hey, is this LeftoverMonkeyParts?. This is HR Manager. Can you verify some of your details for me?” My nickname hadn’t been set yet, so they were just reaching out to the one user online with the default name.)
Status: Removed from all endpoints. Replaced with Teams
VNC
All remote support was handled through TightVNC. The server, and client, were installed on all employee workstations all utilizing a single, shared, six character password. To initiate a remote support connection, an IT employee was supposed to use the aforementioned chat application to get the IP address of the computer for the user they wanted to connect to. Did I mention the chat app would give you the IP address and hostnames of the remote clients?
Please be aware that ManageEngine Endpoint Central was deployed to all endpoints and already has a fully featured remote support tool built in with multi-monitor support and clipboard sharing. There was also no requirement that I get a users IP address as I can simply search by logged on user or hostname
Status: Removed from all endpoints. Replaced with ManageEngine
System Center DPM - Backups in general
I’ve never really figured out what their DR plan was. I don’t think they knew either. It was something they knew they should have, and a lot of the pieces were there, but they weren’t put together right or really at all. The best way I can describe it is “Put as many copies of what we think is important in as many places as possible and there’s no way they’ll get them all”.
The only real backup solution in place was Microsoft System Center DPM. It integrated fairly well with MSSQL Server and pretty poorly with everything else. It took backups of all the production SQL databases (Just the Databases, not images of the VMs) and documents that they thought were important and wrote them out to disk on a dedicated physical Windows domain joined Dell Server that was chuck-to-fuck full of 100+ TB of enterprise flash storage. The perfect backup hardware. Very fast. It also wrote out to tape on a daily basis using two dedicated SAS LTO-8 drives. If it were me, personally, I would have spent the 100 TB of flash storage money on an LTO autoloader…. But hey, that’s what the PC tech is for getting here at 6AM every morning to load tapes. “What? Let them run overnight? No. That would never be feasible!”
A lot more ‘work’ went into ‘Backing Up’ the SQL servers. In addition to DPM, all of the production databases were exported as SQL BAK files on a single SMB shared volume and were then automatically loaded onto a series of “DR” sql servers each night. Most of this was orchestrated using the SQL Agent jobs which were all running as a single shared account with domain admin privileges. All of the documents (4TBs of PDFs) were similarly scattergunned across a dozen different domain joined SMB shares via a series of robocopy scheduled tasks all also running with domain admin privileges. With the exception of the tapes, not a single warm copy of this data was stored anywhere that wasn't a windows domain joined endpoint.
No image level backups of VMs were being taken whatsoever. But that wasn’t for a lack of effort. System Center DPM does integrate
Hello again. I posted this a while back and people seemed to enjoy reading it. Here's a follow up with some progress and more jank I've discovered since. This is not an exhaustive list of jank or progress, just stuff I thought was particularity funny.
Chat/IM
A serverless chat client that operated via multicast was in use and installed on all workstations. It kept local logs of all chats on each workstation in plaintext and used no authentication whatsoever. You set your own nickname and that got reported to all other online clients. Do you want to be the HR manager today? That was just two clicks away! (The HR manager reached out to me on the chat app my first day and asked. “Hey, is this LeftoverMonkeyParts?. This is HR Manager. Can you verify some of your details for me?” My nickname hadn’t been set yet, so they were just reaching out to the one user online with the default name.)
Status: Removed from all endpoints. Replaced with Teams
VNC
All remote support was handled through TightVNC. The server, and client, were installed on all employee workstations all utilizing a single, shared, six character password. To initiate a remote support connection, an IT employee was supposed to use the aforementioned chat application to get the IP address of the computer for the user they wanted to connect to. Did I mention the chat app would give you the IP address and hostnames of the remote clients?
Please be aware that ManageEngine Endpoint Central was deployed to all endpoints and already has a fully featured remote support tool built in with multi-monitor support and clipboard sharing. There was also no requirement that I get a users IP address as I can simply search by logged on user or hostname
Status: Removed from all endpoints. Replaced with ManageEngine
System Center DPM - Backups in general
I’ve never really figured out what their DR plan was. I don’t think they knew either. It was something they knew they should have, and a lot of the pieces were there, but they weren’t put together right or really at all. The best way I can describe it is “Put as many copies of what we think is important in as many places as possible and there’s no way they’ll get them all”.
The only real backup solution in place was Microsoft System Center DPM. It integrated fairly well with MSSQL Server and pretty poorly with everything else. It took backups of all the production SQL databases (Just the Databases, not images of the VMs) and documents that they thought were important and wrote them out to disk on a dedicated physical Windows domain joined Dell Server that was chuck-to-fuck full of 100+ TB of enterprise flash storage. The perfect backup hardware. Very fast. It also wrote out to tape on a daily basis using two dedicated SAS LTO-8 drives. If it were me, personally, I would have spent the 100 TB of flash storage money on an LTO autoloader…. But hey, that’s what the PC tech is for getting here at 6AM every morning to load tapes. “What? Let them run overnight? No. That would never be feasible!”
A lot more ‘work’ went into ‘Backing Up’ the SQL servers. In addition to DPM, all of the production databases were exported as SQL BAK files on a single SMB shared volume and were then automatically loaded onto a series of “DR” sql servers each night. Most of this was orchestrated using the SQL Agent jobs which were all running as a single shared account with domain admin privileges. All of the documents (4TBs of PDFs) were similarly scattergunned across a dozen different domain joined SMB shares via a series of robocopy scheduled tasks all also running with domain admin privileges. With the exception of the tapes, not a single warm copy of this data was stored anywhere that wasn't a windows domain joined endpoint.
No image level backups of VMs were being taken whatsoever. But that wasn’t for a lack of effort. System Center DPM does integrate
Reddit
From the sysadmin community on Reddit
Explore this post and more from the sysadmin community
with VMWare and they did try to make it work several times. About once per year judging by the leftover service accounts. I initially hit the same roadblock they did, but I was able to overcome it via the secret troubleshooting magicks of “Looking in the event viewer.” It was a TLS version mismatch between DPM and vCenter.
Status: Replaced with Veeam. 100TB Flash Server is now a \wicked* fast VHR. All data is now backed up at the image level
Remote Access/Remote Work
They seem to have settled on VMWare Horizon VDI as their remote access solution of choice. 40 Windows 10 VMs running in the prod cluster, one machine per employee for remote access. Before this they had been issuing personal VPN hardware appliances out of employees to wack into their home networks. From what I can tell they initially allowed traffic through the firewall right to the Horizon servers. It was breached at some point soon after going online (because of course it was). They then added a VMWare horizon Secure Access Gateway which is \designed* to go into a DMZ to sit in-between the public facing internet and the Horizon servers, but they didn’t do that. It was just put in the same prod network as the VMWare cluster and Horizon servers. This solution, when it was working, resulted in some employees having essentially three devices. A Windows Desktop, a Windows Laptop, and a Windows VDI VM. One employee was using their laptop to connect to their VDI VM and then RDPing into their desktop.
Status: Replaced with Laptops/Docks and the OpenVPN implementation with 2FA that’s built into the firewall.
EDR
They paid for a modern EDR tool with a 24/7 SOC. Reliably deployed to every system, even the Server 2012 VMs. At first I was impressed, but then I dug deeper. They had disabled all alerting from the tool and forbid the SOC from taking any action in the event of a detection and not provided any phone/cell contact information to the SOC for anyone in the department. Here’s what they did instead:
One server called “ITUTIL1” ran a scheduled task (as domain admin) that would run a literal for loop to generate a list of every possible endpoint address within all of our subnets. It would then attempt to reach out with WinRM to all addresses and collect the event logs from Windows Defender for every successful connection. The data was then “formatted” and emailed twice daily to the IT Department director. The VM did other silly things too, like use the same logic to generate a list of all available IP addresses and email them to the director weekly.
Status: VM burned in a fire. Reporting for EDR tool enabled and SOC given full authorization to do whatever they want
FTP Servers
We have several FTP servers which are used to exchange data programmatically with a few different external entities. The entities are all known with fixed IP addresses, but the firewall rules for FTP are all set to allow any in the firewall. That’s because on the FTP server software they’ve set a *blacklist* with huge swaths of IP addresses blocked out
Ex:
…
80.0.0.0 - 82.255.255.255
83.0.0.0 - 85.255.255.255
…
They then have the “enabled” button unchecked for the particular range where an external entity sits, thus permitting the connection via FTP. I have no idea why they chose to do things this way. Other services for known entities that aren’t FTP have lists of allowed addresses in the firewall
Status: Confirmed external addresses with entities, added to firewall. Disabled dumb blacklist nonsense
Argentina
Some of the local subnets use Non RFC1918 addresses. It was a historical holdover required by an external entity from before NAT and RCF1918 existed as proper standards, but they never fixed it. Looking at the geoblocking config in the firewall I see all incoming connections with the exception of Canada, The United States, and Argentina are blocked. I wonder how that went down. Super Funny
There's so much more, but this is what I can share easily and without worry. To all the junior sysadmins out there I want you to know that I'm not
Status: Replaced with Veeam. 100TB Flash Server is now a \wicked* fast VHR. All data is now backed up at the image level
Remote Access/Remote Work
They seem to have settled on VMWare Horizon VDI as their remote access solution of choice. 40 Windows 10 VMs running in the prod cluster, one machine per employee for remote access. Before this they had been issuing personal VPN hardware appliances out of employees to wack into their home networks. From what I can tell they initially allowed traffic through the firewall right to the Horizon servers. It was breached at some point soon after going online (because of course it was). They then added a VMWare horizon Secure Access Gateway which is \designed* to go into a DMZ to sit in-between the public facing internet and the Horizon servers, but they didn’t do that. It was just put in the same prod network as the VMWare cluster and Horizon servers. This solution, when it was working, resulted in some employees having essentially three devices. A Windows Desktop, a Windows Laptop, and a Windows VDI VM. One employee was using their laptop to connect to their VDI VM and then RDPing into their desktop.
Status: Replaced with Laptops/Docks and the OpenVPN implementation with 2FA that’s built into the firewall.
EDR
They paid for a modern EDR tool with a 24/7 SOC. Reliably deployed to every system, even the Server 2012 VMs. At first I was impressed, but then I dug deeper. They had disabled all alerting from the tool and forbid the SOC from taking any action in the event of a detection and not provided any phone/cell contact information to the SOC for anyone in the department. Here’s what they did instead:
One server called “ITUTIL1” ran a scheduled task (as domain admin) that would run a literal for loop to generate a list of every possible endpoint address within all of our subnets. It would then attempt to reach out with WinRM to all addresses and collect the event logs from Windows Defender for every successful connection. The data was then “formatted” and emailed twice daily to the IT Department director. The VM did other silly things too, like use the same logic to generate a list of all available IP addresses and email them to the director weekly.
Status: VM burned in a fire. Reporting for EDR tool enabled and SOC given full authorization to do whatever they want
FTP Servers
We have several FTP servers which are used to exchange data programmatically with a few different external entities. The entities are all known with fixed IP addresses, but the firewall rules for FTP are all set to allow any in the firewall. That’s because on the FTP server software they’ve set a *blacklist* with huge swaths of IP addresses blocked out
Ex:
…
80.0.0.0 - 82.255.255.255
83.0.0.0 - 85.255.255.255
…
They then have the “enabled” button unchecked for the particular range where an external entity sits, thus permitting the connection via FTP. I have no idea why they chose to do things this way. Other services for known entities that aren’t FTP have lists of allowed addresses in the firewall
Status: Confirmed external addresses with entities, added to firewall. Disabled dumb blacklist nonsense
Argentina
Some of the local subnets use Non RFC1918 addresses. It was a historical holdover required by an external entity from before NAT and RCF1918 existed as proper standards, but they never fixed it. Looking at the geoblocking config in the firewall I see all incoming connections with the exception of Canada, The United States, and Argentina are blocked. I wonder how that went down. Super Funny
There's so much more, but this is what I can share easily and without worry. To all the junior sysadmins out there I want you to know that I'm not
complaining, I'm loving every second of this for now. Don't let posts like this discourage you from coming into this field.
https://redd.it/1ntqvyo
@r_systemadmin
https://redd.it/1ntqvyo
@r_systemadmin
Reddit
From the sysadmin community on Reddit
Explore this post and more from the sysadmin community
How are people coping with Bitnami shutting down their image distribution?
Fuck Broadcom.
https://redd.it/1ntqmkm
@r_systemadmin
Fuck Broadcom.
https://redd.it/1ntqmkm
@r_systemadmin
Reddit
From the sysadmin community on Reddit
Explore this post and more from the sysadmin community
I had the pleasure of speaking to Microsoft Support for the first time in ages this afternoon...
I was trying to troubleshoot an issue with a cross-tenant SharePoint migration, struggling to find any documentation on the error I was getting, so I figured I'd give MS support a shot...
They kept giving me Powershell commands containing parameters that don't actually exist, and letting me sit in complete silence for minutes at a time while they "looked into the issue"
If I wanted Powershell commands hallucinated by Copilot, I would talk to Copilot myself! Silly me for thinking they would do anything else 🙃
https://redd.it/1ntw4e8
@r_systemadmin
I was trying to troubleshoot an issue with a cross-tenant SharePoint migration, struggling to find any documentation on the error I was getting, so I figured I'd give MS support a shot...
They kept giving me Powershell commands containing parameters that don't actually exist, and letting me sit in complete silence for minutes at a time while they "looked into the issue"
If I wanted Powershell commands hallucinated by Copilot, I would talk to Copilot myself! Silly me for thinking they would do anything else 🙃
https://redd.it/1ntw4e8
@r_systemadmin
Reddit
From the sysadmin community on Reddit
Explore this post and more from the sysadmin community
Executive is convinced that former disgruntled IT employee set his account to auto-accept all incoming appointments
Which would be a little hilarious if true but how do I go about investigating this 😭
https://redd.it/1nu1606
@r_systemadmin
Which would be a little hilarious if true but how do I go about investigating this 😭
https://redd.it/1nu1606
@r_systemadmin
Reddit
From the sysadmin community on Reddit
Explore this post and more from the sysadmin community
Are DBAs generally psychopaths or sadists in your opinion?
I want to say every DBA I’ve ever dealt with has hurt my feelings.
I’m a very sensitive Windows admin/infrastructure guy.
https://redd.it/1nu5kdd
@r_systemadmin
I want to say every DBA I’ve ever dealt with has hurt my feelings.
I’m a very sensitive Windows admin/infrastructure guy.
https://redd.it/1nu5kdd
@r_systemadmin
Reddit
From the sysadmin community on Reddit
Explore this post and more from the sysadmin community
If you were designing a data center/server room today, what would you prioritize?
Hey folks,
I’m working on a network plan for a 12-story hospital and I’d love to tap into your experience. If you were given the chance to design a server room or small data center from scratch today, what would you focus on and how would you approach it?
Would you prioritize redundancy (power, cooling, networking) above all else?
How much attention would you give to scalability for the next 10–15 years?
What rack/cabling layout or standards would you follow?
Any advice for managing fiber vs. copper in a hospital setup?
What are the “gotchas” you wish you’d thought about before your own builds?
I’m not asking for free consulting, just trying to gather some real-world lessons and crowd wisdom from people who’ve actually done this.
Thanks in advance!
https://redd.it/1nu0o00
@r_systemadmin
Hey folks,
I’m working on a network plan for a 12-story hospital and I’d love to tap into your experience. If you were given the chance to design a server room or small data center from scratch today, what would you focus on and how would you approach it?
Would you prioritize redundancy (power, cooling, networking) above all else?
How much attention would you give to scalability for the next 10–15 years?
What rack/cabling layout or standards would you follow?
Any advice for managing fiber vs. copper in a hospital setup?
What are the “gotchas” you wish you’d thought about before your own builds?
I’m not asking for free consulting, just trying to gather some real-world lessons and crowd wisdom from people who’ve actually done this.
Thanks in advance!
https://redd.it/1nu0o00
@r_systemadmin
Reddit
From the sysadmin community on Reddit
Explore this post and more from the sysadmin community
Our "asset management" is a Google Sheet and I'm not even embarrassed anymore
Started as IT admin at a 200 person distributed company. Asked about our asset tracking system during onboarding.
"Oh yeah, it's in the shared drive. Really comprehensive spreadsheet."
This "comprehensive spreadsheet" has:
- 47 laptops marked as "somewhere in California"
- 12 entries that just say "John's laptop (which John?)"
- One MacBook Pro listed as "probably dead but maybe just sleeping"
- 3 different tabs with conflicting information
- Last updated 8 months ago
Found out we've been paying insurance on equipment that was returned 2 years ago. Also discovered we apparently own 15 monitors but nobody knows where they are.
CEO keeps asking for "better visibility into our IT assets" while I'm over here playing detective trying to figure out if Sarah in marketing actually has 2 laptops or if someone fat-fingered the spreadsheet.
Anyone else managing distributed IT with the technological sophistication of a lemonade stand?
https://redd.it/1nu7z4h
@r_systemadmin
Started as IT admin at a 200 person distributed company. Asked about our asset tracking system during onboarding.
"Oh yeah, it's in the shared drive. Really comprehensive spreadsheet."
This "comprehensive spreadsheet" has:
- 47 laptops marked as "somewhere in California"
- 12 entries that just say "John's laptop (which John?)"
- One MacBook Pro listed as "probably dead but maybe just sleeping"
- 3 different tabs with conflicting information
- Last updated 8 months ago
Found out we've been paying insurance on equipment that was returned 2 years ago. Also discovered we apparently own 15 monitors but nobody knows where they are.
CEO keeps asking for "better visibility into our IT assets" while I'm over here playing detective trying to figure out if Sarah in marketing actually has 2 laptops or if someone fat-fingered the spreadsheet.
Anyone else managing distributed IT with the technological sophistication of a lemonade stand?
https://redd.it/1nu7z4h
@r_systemadmin
Reddit
From the sysadmin community on Reddit
Explore this post and more from the sysadmin community
Do you bring your laptop on vacation?
I’ve been in IT for 18+ years in a bunch of different roles. Right now, I’m the network admin/manager at a mid-sized business, been here 7 years. I like the job and the company a lot.
Here’s the thing, I don’t have a backup for most of what I do. My IT Director handed this stuff off to me years ago and never looked back. Because of that, I’m basically on call all the time. I dont trust him if somthing were to break and needed to be fixed. Most of the time when hes working on somthing he ends up calling me to step in.
I’ve got a vacation coming up for my 20th anniversary with my wife, and she’s not thrilled that I’m planning to bring my laptop. Her thought is if I have to bring it, the company should pay for the carry-on fee. My point to her is, 99% of the time I don’t get calls. Once in a while, I do, usually something small I can fix in 10 minutes or just walk someone through over the phone. If it was ever a real disaster, I’d fly home anyway.
So, just to settle the debate—do you guys bring your work laptop on vacation “just in case,” or do you leave it at home?
https://redd.it/1nuacyi
@r_systemadmin
I’ve been in IT for 18+ years in a bunch of different roles. Right now, I’m the network admin/manager at a mid-sized business, been here 7 years. I like the job and the company a lot.
Here’s the thing, I don’t have a backup for most of what I do. My IT Director handed this stuff off to me years ago and never looked back. Because of that, I’m basically on call all the time. I dont trust him if somthing were to break and needed to be fixed. Most of the time when hes working on somthing he ends up calling me to step in.
I’ve got a vacation coming up for my 20th anniversary with my wife, and she’s not thrilled that I’m planning to bring my laptop. Her thought is if I have to bring it, the company should pay for the carry-on fee. My point to her is, 99% of the time I don’t get calls. Once in a while, I do, usually something small I can fix in 10 minutes or just walk someone through over the phone. If it was ever a real disaster, I’d fly home anyway.
So, just to settle the debate—do you guys bring your work laptop on vacation “just in case,” or do you leave it at home?
https://redd.it/1nuacyi
@r_systemadmin
Reddit
From the sysadmin community on Reddit
Explore this post and more from the sysadmin community