Recruiting

I'm not currently looking to leave my role, but I've been caught in a few waves in the past 10 years of horrible work environment that I had been looking to leave. I applied for a few jobs but they never really went too far, despite me (I think) being a pretty solid candidate. I've only ever had a helpdesk job at my college and then got an internship in college which led to a FTE where I've been in different internal roles ever since (so I've never really had to seriously go through the process). My company ended up hiring a few good people through a recruiting agency, but how does that work as a job seeker (I'd ask those people who are now my peers but I don't want them to think I'm looking to leave)? All I know is by looking on Indeed or just knowing what the big companies in my area are. I'm honestly just curious how it would work in case I do need to seriously look for a job again.

https://redd.it/1p9mo7i
@r_systemadmin

Okay, but how do you SSH into 1,000 devices??

My company has a few thousand devices in the field (vending machines). And recently my team got report that many machines is having a problem. We figured that those devices are using ‘develop’ branch of our kiosk application, instead of ‘production’ branch.

Th fix is to change git branch to production. But the problem is there's about 700 devices (that we know) that went out with ‘develop’ branch.

For this problem, my team already manual remote SSH into each devices and solve them all. Took us one whole day.

This isn't first time we need to do this. But mostly it wasn't as many devices as this.

I wonder if I can do something like sending same cli command to multiple SSH addresses at once of if there's any tool that let me do that. We use reverse tunnel for SSH endpoint.

Or if your company deals with similar fleet size. How are you dealing with such case?

https://redd.it/1p926bi
@r_systemadmin

Network segment is receiving DHCP address info but not communicating on LAN or internet

Hi all, this problem started late on Thurs and my normal networking consultant is bedridden with the flu and can't help. This one is stumping me.... I'm seeing symptoms that could be something like a network loop and I'm seeing symptoms that might be DNS/DHCP(?)

We have multiple managed switches in the building but this problem is only happening to devices connected to one of them.

SOME of the devices connected to this switch are fine but others can't communicate on the LAN or internet even though they are receiving valid DHCP address info.... no pings, traceroutes die right away.

I rebooted the switch and the devices, it didn't make any difference.

We have an access point plugged into the switch and I can see that access point on the network, it's accepting clients but the clients can't connect anything.

If I plug my laptop into any of the ports connected to that switch it will work normally.

I'm stumped and over my head - if anyone has any recommendations please let me know!


EDIT: Additional Info:

* the DHCP servers (a pair of Windows 2019 servers) are still giving out addresses within the last 24 hours and I have lease expirations of 12/7 (8 days from now)

* I have a DHCP range of (10.0.20.1 - 10.0.21.254) and all devices have addresses witihn that range so I don't think there is a rouge DHCP server on the network.


* Some of the "problem" devices seem to be able to ping the gateway but others cannot.

https://redd.it/1p9tkwa
@r_systemadmin

Intune Shared Device Configuration

Hi everyone

I’m setting up Android Enterprise Fully Managed devices as shared devices for first-line workers. Dedicated (COSU) isn’t an option because we need Microsoft Tunnel, which only works on Fully Managed.

What’s the best practice to make Fully Managed devices behave like shared/dedicated devices?

• ⁠Only specific apps
• ⁠No system settings
• ⁠No personal Play Store
• ⁠Clean sign-in/out between users

Do I need to create a separate “technician/staging account” for the enrollment, or is there another recommended way to handle the initial AAD login?

Thanks for any advice

https://redd.it/1p9tsfy
@r_systemadmin

LPIC-2 Preparation on 4linux

Hello everyone, I'm currently looking for preparation in Portuguese for the LPIC-2 certification and I came across the 4linux website, is their preparation really good for preparing for the exams?

https://redd.it/1p9tj62
@r_systemadmin

Microsoft

I feel like some years ago, there was a lot of messaging from Microsoft about how clean and green their datacenters were, they we trialing underwater DCs, Microsoft would be carbon negative by 2030 you we basically saving the planet by using Azure.

However lately, I don't hear the same narrative.

Has the need to save the planet gone away?
Did something happen in the last couple of years to move away from this messaging? /s



https://redd.it/1p9y58p
@r_systemadmin

Invoice / Monthly Payment tracker

Taking over the role of IT Manager in a couple of weeks - currently the Network Admin. Looking for a good tool to input and track all invoices and bills. A good way to track all monthly / yearly renewals. Current Manager has an Access database to input all invoices and Excel sheet to track monthly payments and yearly. Most of the bills arrive in email or hard copy so those are inputting into the firms invoicing database. I want my own IT db to track everything coming in. Any suggestions?

https://redd.it/1p9zi8w
@r_systemadmin

Which DLP is the better choice for a 10k-endpoint environment?

We’re evaluating three options right now: 1. Forcepoint 2. Trellix 3. Symantec

We have around 10,000 Windows endpoints, and Forcepoint is noticeably more expensive, especially when you include premium support.

If anyone has real-world experience with these tools—stability, policy management, support quality—would love to hear what you recommend.

We’re looking strictly for an on-prem deployment.

https://redd.it/1pa25ij
@r_systemadmin

Are you allowing any AI tools to touch production data at work?

I’m not a sysadmin myself, but I’m an engineer, and I’m trying to understand how this actually works in the real world.

In my previous role, I could use pretty much any AI tool I wanted, but I was working for a startup. I recently moved to a new company where I have a bit more responsibility and influence, and the situation is the complete opposite. We are not allowed to use any AI tool other than Copilot.

When I first raised this with our IT department, the response was basically “everything is a data breach.” But at the same time, I already see people using GPT or Claude anyway and just not talking about it. So there’s this weird gap between policy and reality.

Since I can now at least help influence some of these decisions, I’m trying to understand what’s actually normal out there right now. Are most organisations still in full lockdown mode? Are there environments where AI is formally allowed under strict controls? And do any of those setups actually feel secure in practice?

https://redd.it/1pa5jeg
@r_systemadmin

Do I have the right idea going into a DNS migration?

Hello, never done this before. Currently our domain is hosted on godaddy but the nameservers point to a third party provider. I just got access to a txt file with all the records. So from what I've gathered:

Move the name servers over to godaddy nsXX.domaincontrol.com

nsYY.domaincontrol.com and manually add the records 1 by 1?

Then wait for propogation?

Any help is appreciated, thank you.

https://redd.it/1pa81li
@r_systemadmin

NBNS HELP

Hi,

I am junior system engineer, and need guidance on a task. I have to disable NBNS. I ran Wireshark on one subnet and saw a lot of NBNS from laptops ( DC is broadcasting for laptops

My questions are:

If DNS records exist and work, why is there still NBNS/LLMNR traffic?

How can I tell if anything in the environment actually relies on these protocols before I disable them?
What is the safe way to test this in a production environment?
Are there any common things that usually break (apps, printers, shares) when people turn these off?


Please guide.


Thank you

https://redd.it/1paayg7
@r_systemadmin

Hypervisor Crawling to a stop

Hi everyone,

I just came across one of our hypervisors acting very strange.

We run backups on all the VM's (which have been running fine) via Acronis and these have started failing.

So I tried and connect via our RMM tool but nothing, RDP directly and it takes forever to connect and get a black screen.

So I connect via iLO and I can reach the desktop but its very very slow, windows take forever to open and respond.

I managed to get task manager open but nothing out of the ordinary and event logs shows some potential issues with WMI but not sure.

A reboot has been done but exactly the same issue, VM's are fine but the host seems to be fighting for its life.

Has anyone come across this or would have ideas on what to troubleshoot?

https://redd.it/1pa38u4
@r_systemadmin

How to Change Default SNMP Ports?

I'm setting up a monitoring lab with PRTG as the manager and two agents: a Windows VM and the physical host itself. The project has requirements:

· Must change default SNMP ports (161/162). Only ports 20000 and above are allowed.

The Problem: I can't get the Windows SNMP Service (on both the VM and physical host) to reliably listen on a custom port (e.g., 20000).

What I've Tried on the Windows Agents:

1. Registry Mods: Added TrapListenPort (DWORD) under HKLM\SYSTEM\CurrentControlSet\services\SNMP\Parameters and TrapPort under the snmptrap service path. After restarting the services, netstat -an shows the service is still listening on port 161, not the new port.
2. Service Reconfiguration: Tried using sc config to change the binary path for the SNMP service to include a -p 20000 parameter, but this seems to break the service.

The PRTG side is ready, but I'm stuck at this mandatory port change on the Windows agents. The goal is to have the SNMP service actively listening on, for example, UDP 20000, so PRTG can query it.

Question: What is the definitive, working method to change the listening port for the built-in Windows SNMP Service? Is it even possible without a third-party SNMP agent?

https://redd.it/1pacxhw
@r_systemadmin

What should I add next to my certifications as a Senior IT Infrastructure Engineer?

Hey everyone,

I’ve been in IT for about ten years, currently working as a Senior IT Infrastructure Engineer, and I’m trying to figure out what certifications or skill areas I should pursue next to stay competitive and keep growing.

Current certifications:
• VMware Certified Professional – DCV (VCP-DCV)
• VMware Certified Technical Associate – DCV (VCTA-DCV)
• Microsoft Certified: Azure Administrator Associate
• Microsoft Certified: Azure Fundamentals
• ITIL Foundation (IT Service Management)
• Cisco CCNA Routing & Switching
• CompTIA Security+

Given this background, what would you recommend I add next?

https://redd.it/1padhdt
@r_systemadmin

Switching from LDAP to LDAPS — how bad is the migration?

Our cybersecurity team just told us to disable LDAP and move to LDAPS. Anyone else dealing with this?

https://redd.it/1pagz4t
@r_systemadmin

Event ID 500, Desktop Window Manager uses too much VRAM. (I hope I'm in the right place, forgive me if not)

Have a wonderful day, everyone!

I'll just say this upfront, and I don't mean it disrespectfully, but I really need someone who does system integration or someone who's deeper into this stuff.

I've got result ID 500 in the Event Viewer under Diagnostics Performance:
"Video memory are over utilized and there is trashing happening. Reducing number of running programms and open Windows may help resolve this"
The task category is Desktop Window Manager Monitoring. So, that's where the error must be.

I have the current drivers.
Or rather, the newest one doesn't work because Adrenalin won't start, so I'm using the October version.
I've also tried almost all the settings that are supposed to fix the DWM (Desktop Window Manager) problem, but it's possible I'm missing something.

What's weird is that sometimes I'm not doing anything on the desktop, and my graphics card is using 12-14GB of VRAM even though I'm not doing anything. I've also made a few logs with HW info, and when it's running normally, like with YouTube or Twitch, it uses about 3-4GB of VRAM.

I have the feeling that the graphics card is also being throttled because I often have a pretty weird render distance in games. But when I look at the clock speeds, they're normally up at 3100mhz (9070xt Nitro+)

I've also reinstalled Windows multiple times. Still get event 500 every time.

I really need help, I've spent almost 100 hours on this and can't get rid of it.

https://redd.it/1pai4c9
@r_systemadmin

Power of VSCode Editor

TIL you can open an entire folder of noscripts in VSCode and do a quick Replace of a search string for all noscripts in that folder. I’m sure many of you already knew about this, but it sure saved me a few hours of work.

https://redd.it/1paio0g
@r_systemadmin

Just caused my first massive outage

Hello everyone,


I just got promoted to a new role 3 weeks ago. I made a new deployment to one of our tools and everything just crashed it caused an outage for around 12 hours!

There was nothing wrong with my deployment, and I was following the process word by word. The system just crashed all of a sudden and we believe it was a firmware bug.


I am still worried and scared of doing any more changes, I still haven't heard anything from management. I am filling some big shoes in the new role and my manager told me that when I started they believed in me being able to learn quickly so I got promoted. Everyone has been really supportive but I am still scared worried that I might not have a job next week...


How do you deal with those feelings? My manager told me numerous times that it's ok as we didn't do anything wrong, but I am still worried, how should I deal with the paranoia, or should I really be worried?

https://redd.it/1pan2u4
@r_systemadmin

Windows Event Collector freezing - suggestions?

Hi, and thanks in advance:

I was brought to a Windows Event Collector server, getting events from 2.5K endpoints. It is set to send fowarded events to c:/default-really??, and to rewrite itself after 20MB of data processed. Splunk Universal Forwarder is installed on the server to ingest stuff to Splunk.

Event logs on the server have nothing really useful (Com service (in Korean?) failed to start...) and the forwarded-log-file states last updated about 10min after the last event in the log.

I have not had a chance to see the server running after reboot to check resource use, and apparently after being rebooted - it runs 2-3 days before freezing the Windows Event Collector service so badly it cannot be stopped from the services menu.

The only ting I can think of (after glancing at it), is perhaps an interaction between Splunk UF, and the forwarded log getting full.

If anyone has suggestions: Thanks.
If not, Hope you had a good weekend.

Semi Ninja Edit

The Forwarded Event log states that there are ~2650 endpoints reporting, and the registry has under 3K hives in it.

https://redd.it/1pap4gq
@r_systemadmin

What's your process for technical vendor evaluations?

I'm leading a platform evaluation for my team and trying to improve our process. Currently we're looking at [category\] tools and I'm finding it takes way longer than it should.

Our current approach:

\- Download spec sheets/docs from each vendor

\- Manually pull key specs into a spreadsheet

\- Try to normalize different terminology

\- Takes 4-6 hours minimum

What does your evaluation process look like? Any frameworks or approaches that have worked well? Especially curious how larger teams handle this.

https://redd.it/1pamzx4
@r_systemadmin

Riverbird RMM

Hey everyone,
Do any of you from Riverbird use the RMM and use it for monitoring and RMM? Would you like to hear your experiences?
We want to use it as an MSP for our customers and replace ATERA.

https://redd.it/1paqfcv
@r_systemadmin