Windows Event Collector freezing - suggestions?

Hi, and thanks in advance:

I was brought to a Windows Event Collector server, getting events from 2.5K endpoints. It is set to send fowarded events to c:/default-really??, and to rewrite itself after 20MB of data processed. Splunk Universal Forwarder is installed on the server to ingest stuff to Splunk.

Event logs on the server have nothing really useful (Com service (in Korean?) failed to start...) and the forwarded-log-file states last updated about 10min after the last event in the log.

I have not had a chance to see the server running after reboot to check resource use, and apparently after being rebooted - it runs 2-3 days before freezing the Windows Event Collector service so badly it cannot be stopped from the services menu.

The only ting I can think of (after glancing at it), is perhaps an interaction between Splunk UF, and the forwarded log getting full.

If anyone has suggestions: Thanks.
If not, Hope you had a good weekend.

Semi Ninja Edit

The Forwarded Event log states that there are ~2650 endpoints reporting, and the registry has under 3K hives in it.

https://redd.it/1pap4gq
@r_systemadmin

What's your process for technical vendor evaluations?

I'm leading a platform evaluation for my team and trying to improve our process. Currently we're looking at [category\] tools and I'm finding it takes way longer than it should.

Our current approach:

\- Download spec sheets/docs from each vendor

\- Manually pull key specs into a spreadsheet

\- Try to normalize different terminology

\- Takes 4-6 hours minimum

What does your evaluation process look like? Any frameworks or approaches that have worked well? Especially curious how larger teams handle this.

https://redd.it/1pamzx4
@r_systemadmin

Riverbird RMM

Hey everyone,
Do any of you from Riverbird use the RMM and use it for monitoring and RMM? Would you like to hear your experiences?
We want to use it as an MSP for our customers and replace ATERA.

https://redd.it/1paqfcv
@r_systemadmin

How are you actually managing container vulnerability chaos at scale?

Our security team just dumped a report showing 500+ critical CVEs across our container fleet and wants everything patched immediately. Half are in base OS packages we don't even use, others are in dependencies 3 layers deep.

Currently running Trivy in CI but it's basically crying wolf on everything. Devs are getting frustrated with blocked builds over theoretical vulns while actual exploitable stuff gets lost in the noise.

Looking for real-world approaches that have worked for you:

How do you prioritize what actually needs fixing vs noise?
Any tools that give exploit context or EPSS scoring?
Automation workflows that don't break dev velocity?
Base image strategies that reduce your attack surface from the start?

Any advice would be appreciated.



https://redd.it/1pass85
@r_systemadmin

Little advice for a guy recently laid off, looking to update skills

Hey guys, like it says, laid off from a job I was sr admin and responsible for sccm, Citrix, DR/Backuos using Commvault. I have 25 years experience in everything from Cisco to all Windows stuff. As a guy in his 50’s I decided to go for a few certs while I had the time. (Not a lot of hiring in Q4)

I’ve started SSCP as a mid level security cert, was doing CCSP but I don’t have the year of actual cloud security. In addition I’m going after AWS and Azure certs. If there was an AI cert for agentic or generative AI I’d be interested in that.

Does sound like a solid plan?

https://redd.it/1pawr1f
@r_systemadmin

Task Scheduler Status

I'm trying to add/fix a custom task I had for Task Scheduler. A problem arose before where the task itself was not appearing in Event Viewer. In the limited searching of answers, I ended up deleting the task through File Explorer (C:/Windows/System32/Tasks/<task>), and deleted the associated registry keys in TaskCache/Tree and TaskCache/Task.

So the problem of Task Scheduler complaining about the task is over, but when I create a new task with the same exact name as the original (let's say "Backup Data"), it will then create, but not appear in Event Viewer, and looking through schtasks in CMD, it says the Status is N/A, which is probably why Get-ScheduledTasks in powershell complains about a parameter being incorrect.

How do I fix this issue? Any help is appreciated!


EDIT: Some additional info, looking at Event Viewer, this is something that came up with creating the task:
Task registered task "\\Backup Data" , but not all specified triggers will start the task. User Action: Ensure all the task triggers are valid as configured. Additional Data: Error Value: 2147942583.

https://redd.it/1pax1gv
@r_systemadmin

Our country is down

Our TLD (.vu) has gone offline. That's the country of Vanuatu.

Apparently GoDaddy is the registrar for .vu. As much as people crap on them, I wouldn't look there first for the cause. I would guess that whoever pays the bill for .vu, forgot to do so. That can't be quite right. According to digwebinterface.com, there are a handful of .vu domains that have records still, but most only return an SOA. So maybe someone at Godaddy did fat finger it, and deleted most .vu domains? I don't care. I just want it working again.

Contacting GoDaddy support is comedy gold. Can't get past level 1. They won't escalate. They can't get it into their heads the scope of this thing.

* Me: The entire .vu TLD is unavailable. Godaddy is the .vu TLD registrar.

* GoDaddy: To assist you further, we will need to check your account and website. I have sent a one-time code to the registered email address on your account for the validation process. Can you please help me with that code?

* Me: Can't do that since .vu is down our ********.vu email and web sites are also down.

* GoDaddy: I see, but we haven't received reports of similar errors from our other customers using this extension. To assist you further, we will need to check your account and website. For that first, we need to validate your account.

* Me: (Sigh)

Anyway, all you guys who think you've blown it because you took down the corporate DHCP server, give yourselves a break. This is next-level.

https://redd.it/1paytf4
@r_systemadmin

i feel like chatgpt is shrinking my skills

Before when I had to run that one basic task/command/noscripting thing I didn't fully remember I would have to either: google it, dabble thru man pages/help commands, get grilled on an IRC server/stack overflow by some elitist. And then burn that shit into my memory.


Now I just chatgpt it, ezpz no grilling. But also if I have to write an entire noscript that I KNOW how to write it correctly(given enough time and patience) I'll just hand it off to chatgpt.

https://redd.it/1paya84
@r_systemadmin

Domain controller upgrade

Hi, I currently have a few domain controllers running on Windows Server 2016. I want to upgrade them to Windows Server 2022 using new hardware and then retire the old servers. All of the domain controllers are in the same domain and within a single forest. What would be a reasonable cost for an MSP to handle this upgrade?

https://redd.it/1pb16vp
@r_systemadmin

How do you know if you have too much work ?

For context, I accepted a new job after months of difficult searching. I didn't really have a choice, so I took this Level 2 Helpdesk Technician job with some sysadmin and IT Project Manager responsibilities at a startup (a kind of modern MSP).

This is quite important, so I'll spell it out here: it's chat support, and we're contractually required to respond to every message within 10 minutes, which means that even while we're working on something else, we have to respond to messages at the same time.

There are two of us in this job, and between us we have about a hundred tickets (which is more than at Level 1), quite a few projects on the go, and a bunch of other stuff to do (procedures, different configurations for our clients, helping Level 1 support).

Recently, things have started to go pretty badly. I've lost quality in my daily work with all this flow to manage, and I can feel that it's starting to annoy my superiors.

I talked to my superiors about it, and they confirmed that there is a lot of work to be done, but “it's that time of year, it's normal, we're not going to hire a third person.”

How do you know if you have too much work, and how have you dealt with it ?

https://redd.it/1pb69pk
@r_systemadmin

Building a family Cloud Storage on JioFiber: How to bypass NAT for remote access?

Hi all,

I am in the process of building a personal cloud for my family using NextCloud. The main objective is to have a centralized place where everyone's mobile photos and videos automatically sync (mirroring Google Photos).

The Challenge: I am on a JioFiber connection, which puts me behind a double NAT/CGNAT. I had previously set up the LAN for single-user personal use, but now I need to open it up for remote access for the rest of the family.

Current Setup:

Storage: 1TB Primary Drive + 1TB Mirror Backup (RAID 1).
ISP: JioFiber (Wired).

Has anyone successfully set up a stable remote connection on Jio without a static IP? I am looking for the most reliable method to get around the port forwarding restrictions so the sync works seamlessly for my family.

Thanks in advance!

https://redd.it/1pb77s1
@r_systemadmin

AI drafted support tickets: Curse or blessing?

I honestly don't know where to stand on this one. The uptick in support requests that are clearly AI drafted is increasing steadily.

Pros: Legible.

Cons: A five paragraph word salad that either mentions the core issue in the opening line, or just wastes 10 minutes of my life while I try to unfuck whatever the user is trying to explain. With emoji-sirens.

Thoughts?

https://redd.it/1pb51xp
@r_systemadmin

December Microsoft 365 Changes: Quick Updates Roundup!

That was a busy November, right - where you started diving into all those Ignite updates! From Baseline Security Mode and Work IQ to Agent 365, the new Intune Agents, and the latest from Entra Internet Access, there was a lot to take in.

And now that we’ve officially stepped into December, let’s walk through what’s coming your way this month so you can plan smoothly.

In the Spotlight:

Tenant-owned Team Impersonation in Teams\- Teams will enhance security by expanding impersonation detection from brand-focused checks to include tenant-owned domain impersonation.
Retirement of Mailbox Audit Cmdlets \- The Search-MailboxAuditLog and New-MailboxAuditLogSearch cmdlets will retire by late December 2025. Admins must transition to Search-UnifiedAuditLog for audit searches.
Improved Identity Alert Precision in Defender XDR \- Microsoft will provide finer control over Entra ID Protection alert ingestion, letting admins choose whether to pull in only High-risk, High + Medium-risk, or all detections.

Here’s a quick overview of what’s coming:

Retirements: 6
New Features: 10
Enhancements: 7
Functionality Changes: 3
Action Required: 2

Retirements:

Microsoft will retire the Favorite Contacts feature in early December 2025, standardizing contact behavior across Microsoft 365 using the more accurate Frequent Contacts intelligence model.
The App Skills feature in Copilot for Excel, which provided automated insights inside spreadsheets, will be retired, as Microsoft shifts toward newer Copilot-driven experiences.
The TeamworkDevice (beta) API used to manage Windows-based room devices through Microsoft Graph will be retired, requiring admins to transition to newer device management APIs.
PowerPoint will discontinue the Reuse Slides feature on Windows and Mac, encouraging users to adopt modern content reuse and collaboration workflows.
Teams support for Android 8 devices will fully end by late December 2025, including all security updates and bug fixes.
For Viva Connections modernization, the Assignments and Courses ACEs and associated SharePoint dashboard web parts for Education tenants will be retired.

New Features:

Purview Data Lifecycle Management will introduce Priority Cleanup, allowing admins to override existing retention or legal hold settings to delete OneDrive and SharePoint content when necessary.
Teams presence will become more accurate by evaluating full device activity, ensuring users stay “Available” even when only the Teams tab is inactive.
Data Security Investigations will gain improved cost visibility with a lightweight estimator and a detailed usage dashboard for better budget planning.
Teams will automatically detect and set user work locations when devices connect to corporate Wi-Fi networks.
Purview IRM will integrate with DSI, allowing admins to initiate pre-scoped investigations directly from IRM cases for faster response to risky activities.
Backup-related events like policy updates, backup triggers, and restore operations will be captured in monitoring logs for better audit visibility.
DLP email notifications will soon let users take corrective actions such as stopping file sharing or deleting files directly from the notification.
The new Outlook for Windows will support seamless import of .pst files into user mailboxes, simplifying migrations and data recovery.
The ChatGPT Enterprise Connector will be added to the Purview Compliance Portal, enabling auditing and retention of prompts and responses generated through organizational ChatGPT use.
Purview eDiscovery (Premium) will support importing and reviewing non-Microsoft 365 data sources alongside traditional M365 content.

Enhancements:

Parent sensitivity labels will be replaced with Label Groupings, offering clearer classification while ensuring users assign actual labels rather than grouped parent buckets.
Organizational Messages will begin supporting Entra ID Hybrid-joined devices, expanding

message reach across mixed environments.
Purview Insider Risk Management limits will expand significantly: Variants per indicator: 3 → 10; Total variants: 100 → 400; Detection group items: 200 → 500
IRM policies will allow multiple DLP policies to act as triggers, enabling broader and more accurate risk detection scenarios.
Exchange Online GCC High and DoD tenants will gain inbound SMTP DANE with DNSSEC, improving email authentication and security.
The Microsoft 365 Backup service will roll out to GCC environments starting December 2025.
Microsoft Planner will receive Data Lifecycle Management support, allowing retention policies to protect Planner tasks and related content.

Existing Functionality changes:

The Teams app usage report will be replaced with the Integrated Apps usage report, offering a redesigned layout with improved charts and actionable usage insights.
Microsoft Intune network endpoints will move to Azure Front Door IPs. Tenants using firewall allowlists including those relying on Basic Mobility and Security must update them.
SharePoint agent usage reporting will shift from per-site views to a unified tenant-wide report, simplifying insight gathering for admins.

Action Needed:

Managed connectors for syncing UKG and Blue Yonder data into Teams Shifts will retire on December 7, 2025. Organizations must build custom integrations to maintain data sync.
The Visio Data Visualizer add-in will be removed from Excel on December 8, 2025. Admins should disable the add-in and instruct users to save diagrams locally as .vsdx files.

Act now to stay ahead and ensure these updates don't impact you!

https://redd.it/1pb9hdc
@r_systemadmin

Suspicious of new co-worker

I work fully remotely for a company based in the UK. We primarily work in both the UK and US with the odd worker scattered around other countries. If they work from these other countries they need explicit permission to do so.

The new worker supposedly works from Texas and appears to be a US employee. But I've seen quite a few red flags and I wonder if anyone has seen anything similar or what to do in this situation.

His LinkedIn doesn't make any sense. He supposedly worked as a technical architect over 10 years ago but now works in a more junior role. He has no links to any of his certifications on his LinkedIn. His last company was based on the "US" but when I went to check on the employees they were all based in Africa. His first few companies that he worked for are from Nigeria too.

His English isn't great either and it takes him a long time to say what he needs to say. He's supposedly very knowledgeable in devops but it's been 6 weeks and I've barely seen him do anything.

So I obviously had my suspicions and I have access to our logs which shows login location and IP. He has two IP's which he uses to login which are based in Boston and Texas. But when I look the IP's up they are both VPN's. This seems highly suspicious to me because that would mean he's using a VPN on his router and not his actual ISP IP.

Has anyone had anything similar? Is it worth worrying about?

https://redd.it/1pbankx
@r_systemadmin

Suggestions for alternative PDF-Tool?

We're running a Remote Desktop Services environment where we previously used a licensed Adobe PDF Reader. After migrating to Windows Server 2022, it seems that version is no longer supported. Adobe's new licensing model for Acrobat/Reader looks pretty terrible to me, unless I'm misunderstanding something.

We have around 60 users working directly in the RDS environment, and I'm looking for a solid alternative that can handle opening, merging, and ideally some light PDF editing.

Does anyone have reliable recommendations?

https://redd.it/1pb8fq8
@r_systemadmin

FreshService down?

I saw the UI update but I doubt it is the reason why.

https://redd.it/1pbd8cq
@r_systemadmin

Why is Microsoft documentation always accurate until you actually try to use it

Every time I troubleshoot something in M365 or Azure I start with the docs.

And for the first 30 seconds everything looks perfect.



Then I try to follow the steps.

Half the screenshots are from old portals.

Buttons are in different places.

Settings moved last week.

The important part is hidden behind a “See more” link.

And the feature behaves nothing like the example.



Feels like the docs are written by a version of Microsoft that does not exist in reality.



Is this just my luck or does everyone else hit the same wall?

https://redd.it/1pbenok
@r_systemadmin

Why are a lot of IT companies suddenly starting to push Hourly consulting roles

Why do companies feel the need to hire on an hourly basis and pay you less than 40 hours per week? Is it on prerequisite knowing that they can have you work overtime on overnight shifts? I want to know the reason for this shift

https://redd.it/1pbcka1
@r_systemadmin

How does your company handle on-call compensation?

I know this question gets asked every once in a while, but I feel like it's always good to have fresh input from folks.

The place I'm at currently is pressuring me to join the on-call rotation (something that, when I was originally hired, was exclusively handled by a different team).

The compensation for being on-call is as follows:

- No standby pay (no pay for simply being on-call)
- Only paid for calls that come in that result in work (i.e. if I get called at 2am, but the client declines the afterhours cost, no remuneration)
- With the current number of people in the rotation, it would be once every 12 weeks or so.

I'm inclined to decline it, mostly due to the no standby pay. I dislike the idea of putting portions of my personal life on hold on the off chance someone does call in, and not getting compensated for that. I'm curious what the common standard is currently for being on-call.


EDIT: In response to some of the answers already - I am salary, but would get no comp time unless the call was excessively long, i.e. no leaving early if I started my day early due to a call.

https://redd.it/1pbf49u
@r_systemadmin

Anyone Actually Tracking DORA Metrics in Their Org? Worth the Effort?

I keep hearing about DORA metrics lately (deployment frequency, lead time, MTTR, change failure rate) and how they’re supposed to help teams measure “DevOps performance.”



We’ve got a decent CI/CD setup and some monitoring, but none of this data lives in one place. Management keeps asking if we can start tracking the DORA metric stuff, but I’m not sure if it’s actually useful or just another vanity dashboard.



For those of you who’ve done it, did it make any real difference? How hard was it to set up? We’re mostly Kubernetes + GitLab + Grafana right now.





https://redd.it/1pbi2zx
@r_systemadmin