DC's starting to have amnesia

Seeing a trend of domain controllers forgetting who they are which wreaks all sorts of havoc with DNS, DHCP, AD, Kerberos, etc.

The fix is very easy - restart NLA Service - Network Location Awareness

Changes network location from private/public to Domain as it should be,

Anyways, I had a few different DC's do this over the weekend.

Has anyone seen this and/or have a more stable fix?

https://redd.it/1ozrd6f
@r_systemadmin

Barracuda started rejecting our emails

A few months ago, we started getting bouceback emails from a company that stated it was rejected due to suspected spam. As we were investigating why, we got another, and another. Eventually I figured out all those companies were using Barracuda as their email filter service.

I tried contacting Barracuda, but since we're not a customer, they just said contact the companies and have them put us on their whitelist. That and to use their reputation checker, which said our domain is not blocked/banned.

We use Exchange Online and have DMARC all set up correctly. Any ideas what may be happening or has anyone else experienced this? Maybe someone here using Barracuda that I could test with to see if you can see why it is getting marked as spam?

I sure hope this isn't it, but it sounds a lot like the issue in this post.

https://redd.it/1ozrrun
@r_systemadmin

Are we legally required to have a DPA?

We just got our first european customer inquiry. They're asking us to sign a DPA before they'll move forward with the trial. I had to google what a DPA even was because compliance wise I'm super uninformed
From what I understand it's a legal contract about how we handle their data which is required under GDPR. The only issue is that we've never had one before because all our customers have been based in the States ever since start.

I found some templates online but they're super technical with all this legal language about sub-processors and data transfers and SCCs (nobody here has a clue what compliance is unfortuantely)
Do most Saas companies have a standard DPA template they just send out? Or does it need to be customized for each customer? And if we sign one with this EU customer do we need to offer it to our US customers too? Sorry if these questions sound stupid but I just want to make sure that we're fully correct when it comes to compliance 

https://redd.it/1ozz238
@r_systemadmin

Had a good one today.

Ticket regarding failed SharePoint sync in one drive.

The cause was determined to be a folder name that was almost a paragraph long with a file in it named the same.

Unsure how one drive let the file and folder to be created but it sure didn’t want to sync after the fact.

https://redd.it/1ozzwe7
@r_systemadmin

Docking station advice for new hires

Our team just got a bunch of new hires and they’re all doing hybrid work. Company laptops are all MacBook Air M4, and somehow I got assigned to handle the docking station setup. I’m not really into this stuff. I just use a Samsung S80UD with a built-in KVM and it works fine when I plug in USB C.

I don’t want to deal with support tickets every week, so I’m planning to buy the same dock for everyone. Some of them might want to connect two monitors, so having both HDMI and DP port would be good. Looking for something in the 100 to 200 dollar range. Any recommendations?

https://redd.it/1ozxu29
@r_systemadmin

Why do hackers perform huge DDoS attacks on big names like Microsoft?

I saw this article (15 Tbps DDoS attack against Azure) and it made me wonder, why do they bother with attacks like this? Where's the money in attacks like this?

https://redd.it/1p04lyp
@r_systemadmin

Why does every “simple” change request turn into a full-blown fire drill?

Lately I feel like I’m losing my mind. Every week we get “small” change requests from the business. Things like “just add one group,” “just open one port,” “just update one app.” On paper these are 10 minute tasks.

But the moment I start touching anything, everything unravels.
Dependencies nobody documented, legacy configs from 2014, random noscripts someone wrote and never told anyone about, services that break for reasons that don’t make sense. Suddenly my whole day is spent tracing something that should have been trivial.

I’m starting to wonder if this is just how the job is now or if our environment is uniquely cursed.
Do you guys also feel like even basic changes trigger chaos because the stack is too old, too interconnected or too undocumented?

Just needed to vent and hear how others deal with this without burning out.

https://redd.it/1p05vv8
@r_systemadmin

Cloudflare down... again?

Seems so in the UK - can't even login to cloudflare lol

edit - the login button now works and I can get to 2FA - but upon entering it takes me back to the login page. So still broke

https://redd.it/1p09nye
@r_systemadmin

Cloudflare Global Network experiencing issues Official Update

# Cloudflare Global Network experiencing issues

Investigating \- Cloudflare is aware of, and investigating an issue which potentially impacts multiple customers. Further detail will be provided as more information becomes available.

From Official Status Page on https://www.cloudflarestatus.com/

Update #0

Investigating \- Cloudflare is aware of, and investigating an issue which impacts multiple customers: Widespread 500 errors, Cloudflare Dashboard and API also failing.
We are working to understand the full impact and mitigate this problem. More updates to follow shortly.
Nov 18, 2025 - 11:48 UTC



Update #1

Update \- We are continuing to investigate this issue.
Nov 18, 2025 - 12:03 UTC



Update #2

Update \- We are seeing services recover, but customers may continue to observe higher-than-normal error rates as we continue remediation efforts.
Nov 18, 2025 - 12:21 UTC

https://redd.it/1p0a1jq
@r_systemadmin

RIP: All the west coast admins that got woke up at 4am for an outage they had nothing to do with

Remember the good old days when people talked about how silly and ignorant clients were when they said 'the internet is down' and we'd be like 'really? the whole internet? wow.'
Turns out the joke was on us the whole time.

https://redd.it/1p0addr
@r_systemadmin

CloudFlare down... Better Check DownDetector... Oh...

When you think CloudFlare's down but you can't check DownDetector because that's down because CloudFlare's down lol

https://www.centrel-solutions.com/temp/irony.png

https://redd.it/1p0a797
@r_systemadmin

Downdetector is down due to Cloudfkare being down - Oh my

So.

https://redd.it/1p0a2ts
@r_systemadmin

The Website is Down #1: Sales Guy vs. Web Dude (Classic Cloudflare)

I am SURE it has been posted here COUNTLESS TIMES, but today - with Cloudflare on fire, we should all sit back, relax, and laugh our assess off with this historical nugget of internet gold.

https://youtu.be/uRGljemfwUE?si=TJhlwE5obrQbGyYJ

I'm always amazed by how many of the "new generation" of SysAdmins have never even heard of it. Sigh, kids these days. Maybe NSFW, but just a little.

https://redd.it/1p0cppu
@r_systemadmin

Is it just me or institutional knowledge is no longer valued?

I've been at the same place for close to 22 years now, and I've survived a LOT of layoffs. But I know plenty of old-timers that did not, and when they left, there was a massive amount of institutional knowledge that got lost. And management doesn't give a crap. They just tell you to figure it out when you need to reach out to someone that is no longer there.

When I started here 22 years ago, loyalty was rewarded. I met plenty of people that had been here 20+ years and managed to retire from this place.

Since the pandemic ended, I'm noticing that this place no longer rewards loyalty, and even having intimate knowledge on how something works, or being the company subject matter expert on something doesn't guarantee any kind of job security.

https://redd.it/1p0dxev
@r_systemadmin

Cloudflare is Down! Here's what you can do.

We have monitoring placed on all the system, we got bombarded with alerts back to back.

Instead of panicking we changed the DNS proxy and generated new SSL certs for all the proxied domains.

All of our customers are back online within 30 minutes from the outage started.

If you're unable login to Cloudflare, their API access is still working you can use the API keys to update the DNS records!

If you're unable to access cloudflare you can change your DNS from cloudflare to your domain provider OR can transfer it to Fastly, bunny or Akamai and use the alternative providers.

If you've purchased the domain from Cloudflare or they use cloudflare (namecheap 😒) sadly you will have to wait.

You can try emailing your domain provider to change the nameservers they will help you out, try cloudns or similar options.

https://redd.it/1p0d69g
@r_systemadmin

As CTO, I’m pleased to announce our platform outperformed Cloudflare during the incident,....

....maintaining flawless availability across our primary production environment at http://localhost:3000, a testament to the robustness of our enterprise architecture.

https://redd.it/1p0li15
@r_systemadmin

Hot take: The outage isn't the problem everyone going down at once is

It’s happening again. Cloudflare is down, and with it, a massive chunk of the internet has simply vanished. We see the usual panic: 500 errors on major platforms, broken APIs, and businesses bleeding revenue by the second.

But if we just treat this as "another technical glitch," we are missing the point.

This isn't a reliability issue; it’s a topology issue. We have allowed the internet (designed to be the ultimate decentralized network btw) to atrophy into a fragile oligopoly. When "the cloud" is effectively just three or four giant computers in Northern Virginia and Frankfurt, outages aren't accidents; they are statistical certainties.

https://redd.it/1p0g2c7
@r_systemadmin

So the Cloudflare outage was basically the Windows .LOG size bug on steroids?

https://www.axios.com/2025/11/18/cloudflare-outage-cause-systems-down

>What they're saying: Cloudflare spokesperson Jackie Dutton said the outage was caused by a "configuration file that is automatically generated to manage threat traffic."

>"The file grew beyond an expected size of entries and triggered a crash in the software system that handles traffic for a number of Cloudflare's services," Dutton said.

Seeing the larger explanation for this in the near future (assuming they actually give one) are probably going to make both eyes and heads roll. Going to guess that this one is going to take a while for people to trust again after they claim it to be fixed.

https://redd.it/1p0rb4i
@r_systemadmin

I built a DownDetector for DownDetector

After DownDetector went down with the CloudFlare outage today I decided to build a robust, independent tool which can act as a DownDetector for DownDetector

Hosted on AWS plus a static mirror on Netlify and also Vercel for triple redundancy !!!

https://redd.it/1p0svc7
@r_systemadmin

I can't take it anymore guys

"Oops, something went wrong!"

Buttons greyed out for no discernible reason with no explanation why.

Extra buttons loading so slow that your mouse is already there, and then you click the new button that just suddenly appeared on accident.

Email alerts that send you a link, make you log in, and then don't redirect you to the link.

Micropenissoft shitwindows changing your settings automatically for no reason.

Licenses to use features that already exist on hardware you already spent thousands of dollars on.

AI features I didn't ask for.

Updates that give you a "new and improved interface" that requires you to search for things to find them and click through more menus than before.

Popups that interrupt me in the middle of typing to tell me about some new feature I don't fucking care about.

I'm losing my mind, guys. Was it always this bad?



https://redd.it/1p0s7x3
@r_systemadmin

Spent 5 hours debugging AWS Elastic Beanstalk… turns out my client just hadn’t paid the bills.

So today I learned a very important lesson about AWS:
It won’t tell you *why* it’s ruining your life.

I’m working for a client, right?
Simple task: **“Can you deploy this updated Node backend on EB?”**
Cool, no problem. I’ve done this a hundred times.

Except today EB woke up and chose violence.

* Stuck at “Updating environment”
* Stuck at “No Data”
* Rebuild fails
* Auto Scaling group refuses to exist
* Logs won’t download
* Node 22 acting like it hates me
* Even a brand new environment wouldn’t launch
* EC2 keeps screaming “vCPU limit exceeded”
* Support rejects quota increase in 30 seconds flat

At this point I’m sweating thinking I corrupted their entire environment.
I’m googling every possible error under the sun.
I'm blaming my ZIP file, my code, my past life sins, everything.

**FOUR HOURS later…**

I open the billing section and see:

>

BRO.
AWS basically put the entire account into **timeout mode**, silently.
Didn’t tell me upfront.
Didn’t show a warning in EB.
Didn’t say “Hey genius, your client didn’t pay the bills.”
Just let me fight ghosts for half a day.

The whole infrastructure was literally **blocked** because the client hadn’t paid MONTHS of invoices.

And here I was debugging like I broke production.

*Me:* Why won’t EC2 launch??
*AWS:* 😐
*Me:* Why is my quota suddenly 1 vCPU??
*AWS:* 😐
*Me:* Why did you reject my quota request in 0.2 seconds??
*AWS:* 😐
*Billing page:* “Past due: ₹23,659.”
*Me:* OH.

Anyway, client is like “ohhh yeah, we forgot to pay that.”

So yeah, shoutout to AWS for letting me believe I destroyed the entire system, when the real root cause was basically, “We don’t run servers for broke people.”

Day ruined, self-esteem shattered, but at least I earned Reddit content.

https://redd.it/1p10mv9
@r_systemadmin