Post-mortem sanity check: how do you handle “un-scannable” expiries (API keys, internal certs) without spreadsheets?

We just had a ~2 hour outage because a 3rd-party API key expired.

What’s annoying is that this wasn’t a surprise. We knew about it. It was written down in a “DevOps Secrets” spreadsheet, and someone even had a calendar reminder, but that person was on PTO, and honestly the spreadsheet hadn’t been opened in months.

We already use UptimeRobot for public SSL certs, so those are fine. But for the “offline” stuff , Apple Push certs, API tokens, internal signing keys, we don’t seem to have a good answer.

I’m honestly just trying to sanity-check how other teams deal with this:

Do you actually have a tool that tracks these properly?
Or is everyone using some shared spreadsheet + reminders and hoping it doesn’t get missed?

I’m tempted to hack together a cron job to nag us, but that feels like yet another fragile workaround that’ll probably rot over time.
Curious what’s actually worked for people.

https://redd.it/1pytylt
@r_systemadmin

Suggest arguments

I am working as a sysadmin intern in a university.
They have a site for professors to upload their recorded lectures and notes and for students to access them. It used aws s3 bucket to store the data.

There are many pcs that are outdated for general use with i3-7th gen and almost 50, 1tb hdd. I had setup a ubuntu microcloud's microceph, and set and integrated it with the site for proof of concept. I had configured redndancy, backup and for security i had seperated the server from all other nodes in the university network. For observability of the server i used Graphana and protheumus. We already have a power backup for the it department and the other servers, so power will not be an issue

The director argued that money is never the issue for tech. The current trend in tech is that we should pour money and since all big tech are using cloud and it is the "trend" why should we have an in-house cloud.

I had future plans of using microcloud's lxd and other features to implement most features offered by cloud providers so that students could learn the tech that goes into setting up these cloud services and allow students to use these in-house services for testing and research purposes. Obviously even for such a small scale cloud, it would require a couple good nodes with good configurations which would be a cost but the university anyways does not care about money.

I have created a document citing stanford's it department with their own cloud and other companies that have shifted to a hybrid model as well.

Please suggest good arguments and suggestion what should be done

https://redd.it/1pyu60k
@r_systemadmin

Bitlocker hardware encryption post

There was an excellent post a little while ago showing how to enable the new hardware enabled bitlocker as well as some performance comparisons and for some reason the mods nuked it? How is that not relevant to this sub?

https://redd.it/1pyx9or
@r_systemadmin

Question about building UPS bypass switches.

Hi, i do alot of UPS units (mostly up to about 5k), i have been around the larger (10k and above) and i know they have a bypass system and understand the concept, however something thats always seemed odd.

The building bypass generally has the supply and output from the UP with a switch for both, then the bypass which will basically connect the in power to the out connection there by putting the power and UPS in parallel, this means it will back feed in to the output side of the UPS, real bad idea... so how do these switches do this?


I would assume it would need to break the UPS output then make the bypass port, but this would drop the load for a fraction of a second, so how do they get around this issue?

https://redd.it/1pyvm2m
@r_systemadmin

Company Teams Backgrounds

We have a noscript that runs everyday that pushes out company-branded Teams backgrounds to all user computers. Users are also allowed to add custom backgrounds. 2025 was our 50th anniversary, and today we received a request to remove all Teams backgrounds that have our 50th logo on them. I was told by one of my colleagues that removing any backgrounds from our folder repository would also remove anyone's custom backgrounds. I would think there's a way around this and a more elegant way to implement this - GPO maybe?

https://redd.it/1pyzubf
@r_systemadmin

Is it AWS in general, or just us-west-1?

I'm enjoying some bad network connection issues and latency at 200x the normal levels.

Is this going to be a very long day, or is it some one else's problem?

https://redd.it/1pz0a61
@r_systemadmin

Best alternative to Veeam Backup & Replication?

Hello everyone!


My set-up consists of 4 servers. All of them using VMware ESXi. For backups, I've been using Veeam Backup & Replication Community Edition and must say it is really, really nice.


The problem is that the CE edition limits up to 10 workloads, and I definitively need more than that.


I'd love to hear alternatives cheaper than Veeam, or single-payment software, or even free :)



Thanks!!

https://redd.it/1pz2oqx
@r_systemadmin

Mail bomb defense?

I'm seeing an uptick in Mail Bomb attacks... thousands of incoming messages in a short time.

Other than Proofpoint, I'd love to hear some tools and solutions others have found that help with this in the real world, are there any good solutions?

Thanks in advance!

https://redd.it/1pyxw7o
@r_systemadmin

DFS file server configuration

Hi,

Currently a 3TB folder (One of Seven folders) is configured to replicate between 2 DFS file servers.

And I have configured 300GB to staging quota in DFS management for this 3TB folder.

* Is this value too big ?

I found "DfsrPrivate" folder is quite big (240GB) and it impact to backup window and backup storage.

For my experience, small value (\~10GB) caused replication failure (not instance replication ?)

Therefore would like to know your suggestion to for configuration to retain replication and better backup window.

Thanks and Happy New Year.

https://redd.it/1pz6gju
@r_systemadmin

System Admin job interview

Should I act excited during the interview? Or stern face?

Hey guys, I have a system administrator job interview tomorrow afternoon - and was wondering what advice some of you might have. I've done system admin work for about a year now, but have never actually interviewed for a system administrator noscript position. I was told the interview is only an hour long - anyone know what I might expect? Thanks! Its about Must understand Windows Server, ADDS, Hyper-v, Solarwinds, Vmware Esxi. SCCM, Citrix CVAD, (Minimum knowledge). Azure is a plus.

https://redd.it/1pz7022
@r_systemadmin

Given the insane pricing of ram for consumer, how has it affected this field for servers and such?

Just curious because at this point i can sell my cheap ddr5 32gb for almost $500. When I got my 64 gb I paid 230 now its over 900.

Curious how those who are replacing or plan to, have you seen price increase drastically?

https://redd.it/1pz8tk1
@r_systemadmin

How do you prevent network documentation from becoming outdated?

Hi everyone, sorry for the wall of text, but I could really use some advice.

Lately I’ve been running into issues with the way I document and manage my customers’ infrastructures.
This is my current workflow:

I design and document the network using draw.io, basically drawing a topological map of the network with IPs, devices, connections, etc.
Then I store all access credentials and connection methods (SSH, RDP, web UIs, etc.) in Devolutions RDM, which I use daily for remote access and support.

The problem is documentation drift.
For every small change (new device, IP change, VLAN tweak, whatever), the draw.io diagram often doesn’t get updated — sometimes by me, sometimes by colleagues. Over time this becomes a mess and starts to actively hurt troubleshooting and onboarding.

What I’m looking for:

A single source of truth for devices and network information
Inventory of devices (IPs, roles, locations, notes, etc.)
Ideally the ability to generate or at least visualize a network map/topology (even semi-manual is fine)
Bonus points if it’s self-hosted, but commercial is okay too if it’s worth it

I briefly looked at NetBox. It clearly looks powerful and well-respected, but my first impression was that it’s very complex and possibly overkill for this use case. I might be wrong, so I’m open to being corrected by people who actually use it daily.

So the real question is:
What do you use to keep network documentation, inventory, and topology sane and actually up to date in a multi-tech environment?

I’m less interested in “perfect on paper” and more in “people actually keep it updated”.

Thanks in advance to anyone willing to share real-world experience.

https://redd.it/1pzbso1
@r_systemadmin

Question about the normal of number IT team by the number of users

What the normal of number IT team by the number of users help desk/network/system and network admin


https://redd.it/1pzbmys
@r_systemadmin

Entra down in Germany?

Anyone else having problems? Same thing for Azure.

Edit: Seems to be an OpenDNS issue.

It's not DNS
Something Something
It's DNS

https://redd.it/1pzfqh0
@r_systemadmin