Today I screwed up

Well I guess it happens to all of us every now and then, but its always such a bad feeling when it happens. 4 years at this company and today, I screwed up production

It was a morning deployment to prod, a couple of quirks but nothing too special. And the actual deployment went fine actually. I did the post-deploy checks, all green. Closed the vpn connection and went on with my day.

Close to the end of the day we start getting tickets, users couldnt log in... me and my manager jumped into action and not even 30 seconds in we see a duplicated network on production, with my name all over it...

Fixing it took just a couple of clicks and I checked my command history and cannot find what I did but its my name on those logs and now Im just feeling like crap...

Anyways... hope your day is going better than mine

https://redd.it/1onjbmo
@r_systemadmin

Does every non-technical person state the order of HTTP redirects incorrectly? Or just the people around me?

This is just a small thing, but I'm baffled by it.

When a user asks me for help to create an HTTP redirect, if they are in a non-technical role such as marketing or education, they will almost always state it this way:

> Please help me create a redirect from www.new-site.com/new-path to www.old-site.com/old-path.

So, as a matter of course, I always have to reply with a narrative denoscription of how a redirect actually functions for a user. Something like:

> The user will enter www.new-site.com/new-path into their browser, and will get bounced over to a final destination of www.old-site.com/old-path. Are you sure that's what you want?

... It's just an extra email. And everyone has been gracious about the clarification. But I am just so surprised how widespread this inverted thinking seems to be among my users.

Among you web server sysadmins, have you noticed something similar?

https://redd.it/1onl1y9
@r_systemadmin

I hate printers

i work at a relatively small company and our IT dept is only about 5 people with very specific roles. so when more helpdesk-ish tickets come in, they're pretty much for whoever is free in that moment (Yes it sucks).


But ive been dealing with this stupid ass printer shit for soooo long now because some manager doesnt like the way the printer prints.

For context, its a citizen label printer. And i set it up with printix for whoever wants to use it but really just this specific department. You can print the labels, after some elbow grease they now look fantastic! Was even approved by the requester (a manager). But for whatever reason, you have to click portrait each time. ok... not a big deal! You can even tell itll be messed up if youre on landscape. So it should be an easy catch for anyone.

But this manager HATES that. So now he threatened to go to my boss about this whole situation... all because the user has to click portrait each time. Now really, im sure theres some way some how to write some command, noscript, or edit a driver or something so landscape just isnt an option that even appears. But what the shit are you really talking about!?!?!

Its just one click you have to do before printing out your labels! But he now wants to scrap the thousands of dollars we spent from our budget into these printers. All because of one more step to click and print these labels....

Am i overreacting??? or is this as ridiculous as he may think.

https://redd.it/1ono277
@r_systemadmin

Microsoft/Globalsign OCSP failure

It looks like there’s a DNS problem with Microsoft’s primary OCSP responder (I know, I know it’s always DNS).

The responder at “ocsp.msocsp.com”, which is configured in billions of certificates (I counted 58 billion on a quick check) issued by various Microsoft Certificate Authorities, normally has a CNAME pointing it to “hostedocsp.globalsign.com”.

This in turn should have a CNAME point to “api.globalsign.cloud”. This CNAME does not appear to exist anymore. This last name has working A records. The chain is broken between these last two globalsign records.

It’s unclear since when this is the case, one DNS history source said there had been no zone changes since October 31st.

What does this mean? Well, it means a large number of clients trying to validate one of those Microsoft certificates will usually try using OCSP first, and fail. It will then usually fallback to downloading the CRL, which can have a significant bandwidth and a small performance impact, as downloading a CRL is generally slower. It should not necessarily affect web browsing, as modern browsers tend to have their own CRL cache they prefill. But a large number of Windows and Microsoft services will not, and rely on the OS mechanism, which means a large number of failed requests to these OCSP servers. This can also affect non-Microsoft applications and services that use Azure, since these often use default Microsoft-supplied certificates on service endpoints, Front Door services, APIs endpoints and the like.

https://redd.it/1onm61v
@r_systemadmin

What is your dress code/attire for work?

My workplace is fairly lax unless we have customers coming. Normally I wear jeans/polo everyday and t-shirt on Friday. Shorts are fine through the summer.

https://redd.it/1onr38o
@r_systemadmin

FYI: Gmail/Google tightened their bulk sender guidelines - emails may now be rejected

https://support.google.com/a/answer/14229414

Previously Google was only putting non-compliant emails in Spam, they have now just said from this month that they may reject emails completely - following the lead of Microsoft here.

Just a reminder to setup your company DMARC policies if you haven't already, and also review bulk sender compliance rules if you're a bulk sender (sending 5,000+ emails per day).

https://redd.it/1onskta
@r_systemadmin

New Sysadmin, way out of his depth.

The Story:

Hi all, I'm mostly making this post out of desperation at this point. I'm a .net developer who's recently been forced to take over as the sole admin for our whole windows server after my boss decided he didn't like the last guy and well... "hey GenericEvilGenius, you're a computers guy right? you should just do it all then". So now if I want to keep getting paid I'm having to sink-or-swim at a job I'm woefully inexperienced at.

Not much later my boss tells me that we (by which he means I) have to manage migrating our entire business to a new server hosted by a new hosting provider, as our current servers are being EOL'd at the end of the month ... I'm so screwed.

After a few days of the hardest I've ever worked I've gotten everything like... 90% of the way there I think but after we do the DNS changeover to point everything towards the new server, it quickly becomes apparent that only like, 40%-50% of our usual traffic is actually reaching our API. This is swiftly confirmed by several irate phone calls from clients complaining that our services aren't working.

But the thing is, i tested this API beforehand, very thoroughly. Even now any tests I perform come back just fine (as it evidently does for roughly half of our clients). As a dev I understand that the first step to troubleshooting any problem is being able to re-create it, but no matter what i do i cant see any problem from my end, but i also can't understand why a problem might affect only some of our clients and not others. All of these people were able to use our API just fine literally yesterday.


The Technical Details:

Migrating from a Windows Server 2016 environment to a Windows Server 2025 one.
Server hosts an email server (hMail), a website (IIS), and a .net based API.
Some users are unable to reach the API after the move, I am unable to reproduce the problem or get any meaningful error information out of those who are experiencing it.
Confirmed firewall is not blocking requests, I can see that all clients requests are passing through the firewall okay, but it's showing those we have confirmed are experiencing the issue are getting a SERVER-RST response.

The only meaningful difference between the old server and new that i can see is that our old server had 3 IP addresses, one for each subdomain it was hosting.

1. mail.example.com for the email server.
2. www.example.com for the website.
3. services.example.com for the API.

It's my understanding that hosting all of these on one server with a single shared IP shouldn't be a problem, so long as people are addressing their SNI's correctly but this is the point at which I reach the limits of my knowledge. Do any of you have any idea why this might be happening? or what I can try looking into next?

https://redd.it/1ont8nu
@r_systemadmin

What are some "Rules for thee, but not for me" that you live by?

What are some things your users required to do, which you do not practice yourself?

For me, it's resetting cookies.

My daily workflow consists of at least 15-20 browser tabs for various admin consoles, ticket queues, monitoring dashboards, reports, etc. All set up and configured exactly how I want them (default page, menu order, column widths, etc.), so while it's not the end of the world if I need to reset my cookies, it is a major inconvenience to get everything set back up again.

https://redd.it/1onvq06
@r_systemadmin

What VOIP phone service is best for managing high-volume call center operations?

I'm in the process of evaluating a VOIP phone service for our call center, which handles a high volume of inbound and outbound calls daily. We need a reliable solution that integrates well with our CRM, offers call routing features, and scales as our team grows. Our call center is distributed, so remote capabilities are a must.

I've looked into a few options but am curious about what VOIP phone service you’d recommend for performance and ease of setup. Has anyone here set up a system that integrates well with Salesforce or HubSpot?

https://redd.it/1onz20l
@r_systemadmin

Junior employee doesn’t want to grow and I’m just telling the truth

We have a junior employee who has been with our company for several years now. Guys a good worker and will do what you ask him to do and will do a good job when he his tasked with something. But he isn’t a go getter, only cares about what’s in front of him. Doesn’t care about new technology, announcements, or what’s changing. If I tell him about a cool new feature in technology that will make us more efficient, he will respond- it’s works now why change.

He was supposed to be my replacement if I decided to leave the company but he doesn’t want my job. My role is a bit different, I don’t have to just deal with what’s in front of me but need to know what’s coming, how will it impact us, how do we prepare, etc. I’m more of an engineering/architect role and he doesn’t care to learn it. He really just wants to be an L3/4 support engineer.

Recently management has been asking me how he’s doing and I’m honest with them. I say he’s great when you tell him do to something but he will never get out of his comfort zone and you will not get him to grow here. I tried for years and just accepted that’s him. I don’t fell like I’m throwing him under the bus but telling management that if I bounce, you’ll need to find someone else.

https://redd.it/1oo56a6
@r_systemadmin

Shout out to my fellow solo IT Admins

Just wanted to give a shout out to my fellow solo's. We keep everything running at the places we work at.

What kind of infrastructure do you all look after?

I'm at about 60 users, about 50 pcs and laptops, printers, phones, wifi, cctv, website, network, currently 8 on-prem servers, only just starting to explore Azure.

Been doing it for over 12 years.

https://redd.it/1oo53wi
@r_systemadmin

What’s considered an acceptable website downtime per month ?

For SaaS founders and devs here, How much downtime per month do you consider “acceptable” ?

Example:

< 5 minutes
< 30 minutes
< 1 hour
Doesn’t matter much

Also curious, Do you actually track downtime or only learn when users complain ?

https://redd.it/1oo50qn
@r_systemadmin

But why, Microsoft? Why?!

https://tasks.microsoft.com = Outlook

https://tasks.office.com = Planner

https://redd.it/1ood847
@r_systemadmin

New Small Business solo sysadmin here: "Ethical Hacker" contacted our general email a few days ago to disclose several website vulnerabilities and is asking for a bug bounty. How do I handle this? Is this a con/shakedown?

The message I got from him was as follows:

>Hello Team,

As an Ethical Hacker I found some Vulnerabilities in your site few of them are as follows.

[various information describing the two vulnerabilities and how to fix them\]

if you have any other questions. I’m hoping to Receive a bounty reward for my current finding.

I will be looking forward to hearing from you on this and Will be reporting other vulnerabilities accordingly.

 Stay Safe & Healthy.

[2 screenshots showing the vulnerabilities\]

I didn't click on anything and I haven't responded because I wasn't sure if it was a scam or not. We're a small business with like 7 employees and outsource our website to a 3rd party company. We're also currently in the process of switching that company. I know ethical hackers exist but I thought businesses usually had to opt-in to bug bounty programs through a site like HackerOne? He never provided any way to pay him, just that he wants to be paid?

He sent a follow-up email today:

>Hello,

>Is there any update on this bug? I'm hoping to receive a bounty reward for responsible disclosure once your team has validated the issue.

>I will be waiting for your response.

>Kind Regards

I'm not even sure if our owner would authorize a bounty payment even if I could verify this guy's identity, nor am I sure how much to offer him, or how to do it, or even if it's legit or not?

What do I do?

https://redd.it/1oofvr1
@r_systemadmin

All new to me

So just got a new job as the only IT person at this company and we’re doing a move to a new office. I need help with getting some resources.

What sites do people use to help them procure equipment such as Ethernet cables in bulk or like network closet equipment? I’m very newbie to all this and pretty overwhelmed with being on a project management side for the first time.



Any help is appreciated!

EDIT: Based in the US. Sorry first post

https://redd.it/1ooaom7
@r_systemadmin

How old is the oldest production server you manage?

Asking because we have some dinosaurs out there... talking about 10 years or so. What are some of the oldest you have out there that you manage, and what are they running?

https://redd.it/1ooi08v
@r_systemadmin

The black screen of death is causing problems in terms of user recognition

Anyone else noticed that users now cannot recognize BSOD anymore?

With it being a black screen now, I am finding users are thinking its a windows update screen (because users don't read), but to be fair, when you look at it at first glance it does seem that way

See image here

We had a production machine that was BSOD and we did not know because everyone thought it was windows updates, and it happened randomly enough to not affect the shows.

And of course the tool we have to monitor that did not flag it until it happened after 3 times. Just a little frustration. I hated the old sad face smiley, but at least it was obvious.

Granted, BSOD are not normal and should not be happening in the first place, but still I think this was a negative change.

https://redd.it/1oojwbv
@r_systemadmin

Potentially dangerous elevated cabinet

Would you work or have anyone working for you work in this cabinet? Its 25+ feet off the ground.

https://i.postimg.cc/qMz8GGW2/IMG-2022.jpg

Background:

I took over a manufacturing facility last year that has its IDF for the production floor elevated about 25 feet off the ground. At some point before my time the cabinet was located in an office but they needed more floor space so they demoed the office and brought the cabinet straight up so they wouldn't have to rewire everything.

The network switches and UPSes in this cabinet are 10+ years old. I put in a budget request to rewire the plant and install a new cabinet and replace all switches and firewall with new units under support. I was denied the cost to rewire the facility but approved to replace the hardware.

My problem:

I have expressed concerns to my boss that its unsafe to work in the cabinet, that the plywood could break causing the whole cabinet to come crashing down taking down the facility. I was told "no one qualified has said this is a safety concern, we get audited by safety vendors all the time and no one has flagged this".

I actually haven't been in this cabinet since I am not a fan of heights and would prefer to not touch the thing. My low voltage vendor that was going to do the swap out said they wouldn't touch it as they consider it a safety hazard.

This thing is also located over a main walk way in the facility and while people are working on it will be roped off I just have a feeling that this thing could fall at any time.

My only course of action is to find someone to do the swap out for me and have a Cover Your Ass Email sent to my boss and his boss saying there is a potential risk for the cabinet to fall and against my better judgement we are going to replace the equipment in it rather than rewiring.

https://redd.it/1ool3ng
@r_systemadmin

How do you deal with general incompetence and failing from management?

90% certain colleagues read this sub and to be honest, if you're my colleague reading this, I don't care, I just hope you support these view points.


I've been working in the Defence sector for a while now, left a pretty prestigious company to go join a systems integrator who is running a project to create private clouds. And **everything** is a shit show.

* Architecture refuse to make LLDs.
* HLDs are scattered all over the place and when they're in the right place they're out of date.
* The project is 2 years old and there's **no monitoring**.
* Domain Admins is prevelant and some people use it as a daily driver.
* Tiering models exist however Domain Admins can login to everything which defeats the point of tiering and allows lateral movement exploitations.
* Barely anything is documented yet on the skills matrix most people are listed as 5/5.
* Management pretend to listen and do absolutely fuck all.
* Some "standards" exist but they're wholly inconsistent.
* Solution Architects are treating this project as their own homelab and trainset, getting defensive if people propose changes or try to enact a degree of change.


The job market is total shit. I'm being paid well here but it's just so fucking soul destroying sitting at a desk, being hired as an expert whilst you can't change anything meaningful because some power tripping asshole architect won't allow you to.


What do I actually do here? My attitude is getting more and more negative and it's going to get to the point where I tell them fuck you I quit.

https://redd.it/1ookzd9
@r_systemadmin

I think I have to leave

After being a member of this subreddit for a quite a while I feel stress when I see a thread from this subreddit pop up. It’s the same stress I feel while at work. Even through this is one of my favorite places to be on Reddit, I feel it’s best to leave. It’s been fun and Its great to have a community to share our opportunities with. However self care should come first.

https://redd.it/1ootjxy
@r_systemadmin

How do you guys do bare metal provisioning?

I recently started working with my dad who runs a small MSP. We have a few hundred active clients with each having anywhere from 10 to 300 devices. Around 90% of devices are Window machines. We often have 5 new machines to provision each week, although sometimes we do closer to 30. Currently I use a win 11 usb with unattend to install then a ps noscript to install apps. Some clients we have we setup with Datto rmm, but that's maybe 1/3 of them. I know a common recommendation is to use intune, but 0% chance we can move everyone there.

Any recommendations to speed up the process? Ideally something that is not another subnoscription.

https://redd.it/1ooqh9k
@r_systemadmin