Microsoft back online. Excuse: too many servers were shut down during maintenance.

Preliminary root cause:
We identified that the issue was caused by elevated service load resulting from reduced capacity during maintenance for a subset of North America hosted infrastructure.”

For 9 and a half hours? You can’t shift the traffic to another region? You can’t abort the maintenance and turn it back on? This smells fishy….

https://redd.it/1qkjzh1
@r_systemadmin

Weekly 'I made a useful thing' Thread - January 23, 2026

There is a great deal of user-generated content out there, from noscripts and software to tutorials and videos, but we've generally tried to keep that off of the front page due to the volume and as a result of community feedback. There's also a great deal of content out there that violates our advertising/promotion rule, from noscripts and software to tutorials and videos.

We have received a number of requests for exemptions to the rule, and rather than allowing the front page to get consumed, we thought we'd try a weekly thread that allows for that kind of content. We don't have a catchy name for it yet, so please let us know if you have any ideas!

In this thread, feel free to show us your pet project, YouTube videos, blog posts, or whatever else you may have and share it with the community. Commercial advertisements, affiliate links, or links that appear to be monetization-grabs will still be removed.

https://redd.it/1qkmvws
@r_systemadmin

I lost my ssh key for a VPS I own. I'm locked out, aren't I?

I fucked up. I have a VPS that I use a SSH key to access. That ssh key was on my pc. Notice the past tense here. That pc has been formatted and sold about 2 weeks ago.

I obviously did not make a backup because why the fuck would I, that's something only smart people do.

This VPS is hosted by Hetzner if it makes any difference.

This is clearly a long shot but is there any way to access the VPS? I'll write an email to Hetzner telling them how much of a dumbass I am and maybe they can give me a hand since they have physical access to the VPS. Even so I sincerely doubt they'll help...god knows how many instances are on that machine.

https://redd.it/1qkqmb2
@r_systemadmin

Did everybody lose an unknown number of emails from M365 issues?

I sent some test emails during the M365 outage yesterday. The only ones that showed up in my inbox after the problem was resolved were the ones that I sent after the problem was resolved. The ones from earlier in the day never showed up in my M365 inbox and my GMail account never received any rejection.

I may never know exactly how many emails went to the bitbucket yesterday, but I suspect a lot.

Same thing all over?

https://redd.it/1qktxfs
@r_systemadmin

I did not abide.... Read Only Friday

Don't do it, no matter how many times you've done it before, no matter how trivial it typically is.... DON'T DO IT!!

Thought I could sneak a ticketing system upgrade in on a Friday before a few days off. I do not yet know how much of my time I've donated for "this one small thing".....

EDIT:

It was the classic, update blew up the config game. PTO rescued, happy Friday peeps!

https://redd.it/1qkrwp0
@r_systemadmin

Microsoft needs a wake up call

MORE issues with exchange today. "A recent code regression is causing crashes on a portion of mailbox infrastructure that handles access requests from Outlook on the web, New Outlook, Outlook for Mac, and mobile apps".

Get it the fuck together, Microsoft. Jesus christ.

Edit: grammar mistake

https://redd.it/1qkz2te
@r_systemadmin

Alerting Staff

So yesterday sh*t show with MS it was apparent that we need a way to mass communicate with staff that there is an outage in these types of situations in the event Teams or Mail (or both) go down. We currently don't have a company portal for these types of notifications. I'm wondering if anyone has gone down this path, and if so what they did?

https://redd.it/1qkuvkx
@r_systemadmin

It’s the dream on social media vs. reality

Everyone on social media says, "Oh, IT is great, I make $100k+." But unless you live in a high COL area, the reality is usually starting in Help Desk for $30k–$60k. That is often not far off from the regular job postings you might see for retail or delivery/warehouse in the area, just with the advantage that you're working at a desk or remote.


​I want to hear the real journey, not the highlight reel:

​The Start: What level did you start at (MSP, local shop, corporate, family biz, etc.)? What state were you in, what was the pay, and was it actually livable at the time?

​Location: Did you have to move to get better pay, or were you able to advance locally?

​The Work: Was your first job simple, or were you thrown into the fire? How long until you actually felt like you knew what you were doing?

​Education: Did college or certs actually help you in the real world, or were they just a specific checkbox to get hired?

​The Grind: What's the longest you've spent grinding on a single project?

​The Money: How long did you spend in the field before you actually hit $100k?

https://redd.it/1ql62va
@r_systemadmin

Anyone ever sit in a Colo during a severe weather event?

With all the crazy temps and likelihood of widespread power outages across the US, anyone ever look to weather the outage/recharge devices and such at your Colo? Do places usually have policies on this?

I’m north enough where I’ll be fine in this storm but the thought occurred to me!

Edit:

People seem confused by my question, I was referring to the context of charging devices and caching some Netflix on your phone stopping by for a few hours vs living there for 3 days

https://redd.it/1qkwshz
@r_systemadmin

Do you delay Windows updates?

Over the years windows patching has been of highly varying quality, and every conversation I can find around this has a lot of people on two very different sides. I've been trying to puzzle out an answer between "Always patch immediately" and "let someone else be the beta tester".

I don't see any good recent conversations on this topic in this sub in recent years that have swayed me one way or the other, so I'm hoping to get some more opinions here.

https://redd.it/1qlgcc5
@r_systemadmin

Ricoh MFP automation hell: CAP users + Address Book + PCL6 auth — is there REALLY no supported way?

I manage \~60 Ricoh MFPs and I’m stuck on CAP.

Address Book automation works (SOAP, PowerShell, RicohAddressBook).
Printer deployment and PCL6 drivers are fully automated.


But CAP users:

\- local to device
\- no API
\- no supported way to create users, assign badges, or link scan destinations


CSV import exists but isn’t automation-friendly. Has anyone solved this?

Is CAP-ES the only real option, or is CAP basically a dead end unless you buy Ricoh’s server stack?

I’m open to unsupported hacks if that’s the reality.



https://redd.it/1qljyhd
@r_systemadmin

Documentation - what do you use?

I’m just curious what other sysadmins are using for documentation, both for within your area, and to share with other areas of your company. In my experience, documentation needs to be as simple and easily accessible as possible, or no one will look for it or read it. Documentation will only get checked at all if it’s easier for the person to look at it rather than just ask you. In my opinion SharePoint is terrible for this, no one wants to look for word docs in a library, or try and navigate though potentially multiple sites to find it, the searching isn’t great, and overall it’s just a cluttered painful experience. I’m learning towards using markdown and a static site generator to render those into web pages. But I’m curious what other people do and how it works out for them.

https://redd.it/1qlnc1e
@r_systemadmin