Just had a sales rep try to upsell me while their platform is down globally.

I’m currently staring at a status page that’s been all red for 45 minutes. My inbox is filling up with "Is the email gateway down?" tickets from clients.

My phone rings. I answer it because I have a P1 ticket open with their support and I thought it might finally be an engineer calling back.

Nope.

"Hi, this is Josh. I saw you've been a partner for a while and wanted to see if you'd be interested in seeing a demo of our new analytics dashboard."

I actually sat there in silence for a few seconds. I asked him, "Are you aware that your entire US-East cluster is currently offline?"

He hesitated and said, "Oh, I heard there might be some latency, but the dashboard I'm selling actually helps visualize uptime!"

I didn't even have the energy to argue. I just hung up.

The disconnect between the sales floor and the NOC at these massive vendors never ceases to amaze me.



https://redd.it/1pqomk3
@r_systemadmin

CLOUDFLARE MY LIFE IS YOURS PLEASE

I guess it's fine that they keep things up and running 97% of the time, but man when it rains it pours.

Bunch of clients complaining about sudden weird behavior.

"Can't take inbound calls, but outbound is fine."

Firewall looks good.

Switches have had work done recently, but nothing that would break anything.

SIP trunk is showing registered???

Carrier not receiving replies to challenges though.

Carrier support whispers the magic words: "Make sure you're using a public DNS"

"Oh, I am, I know I am cause I always use google and cloudflare... let me just check my configuration."

There it is. Primary DNS server set to 1.1.1.1

I swap it with the secondary 8.8.8.8 and phones start working.

It's always DNS... always has been...

https://redd.it/1pqoof2
@r_systemadmin

Recommendations for Office 365 backups?

I have a small biz client asking for an Office 365 backup solution.



It needs to cover the following: Exchange Online, OneDrive, SharePoint Online and Teams. This would include things like permissions, calendars, mailbox-rules, etc etc.



Backups do not need to cover the more Azure oriented items (PC's in Intune/Defender/etc, VM's, SQL, and so forth), but ideally can fully restore a user-account. Worst-case would be creating a new user account and running a restore from a dead user to that account.



We should also be able to export the above services outside of O365 (eg ExO -> PST), and do so with some granularity (individual files/folders in SPO, folders or even emails in ExO, etc etc)



My go-to has been **afi.ai** for a while. However, it's also been a while since I've taken anything else out for a spin.



I believe the client would be open to both on-prem and cloud-based solutions. They do not have a plethora of on-prem servers, and do not have on-prem AD. Any on-prem solution would likely mean new hardware. They are bandwidth-limited on their upstream. Cost will be a factor.



Any recommendations?

https://redd.it/1pqq54m
@r_systemadmin

office.com changed again....

I used to be able to pin the admin button on the left pane but now its all just ai bs... Could they be anymore stupid....

https://redd.it/1pqt4fz
@r_systemadmin

Teams Down?

Something something 365 something something

Edit: appears to be back up as of \~2:20pm EST

https://redd.it/1pqtouz
@r_systemadmin

M365/Teams service degradation?

Anyone else seeing delays when sending chat messages in Microsoft Teams? images are also not loading.

We’ve had a few users report it, and I’m seeing the same thing from home as well, so it doesn’t seem tied to our office connection. Feels like a possible Microsoft service degradation, just checking if others are experiencing this too, or if I’m losing it. 😅

https://redd.it/1pqtoaz
@r_systemadmin

25+ plus years working in tech and never been on a real job interview.

I was chatting with my wife at lunch and talking about the “what ifs” due to the current job climate and I realized that I have never been on a real interview. First job I had was 17 years ago and I was hired on as a contractor to literally unlock the chassis on desktops because they had key locks and throw the key in the garbage. The job obviously progressed and when I left 17 years later, I “interviewed” for a new job and the director was super busy and talk to me for 3 minutes and left. I got the job and it’s now 8 years later.

https://redd.it/1pqwma8
@r_systemadmin

Company is trying to refresh hardware and it couldn’t be at a worse possible time…

I’m sure I’m not the only one talking about it… Prices are changing/going up every day and rapidly.

Well, it’s not January 1st yet, and it looks to me like prices are already approaching double their expected cost.

Thanks a lot AI hyperscalers! It’s going to be fun soon.



https://redd.it/1pqto7e
@r_systemadmin

Pour one out for Target admins

Gift cards and app based order fulfillment not working, employee apps tracking stock levels down. Rumors of db replication issues on the busiest shopping weekend of the year.

/r/target in shambles

https://redd.it/1pqzaik
@r_systemadmin

Recommendation for label maker with strong adhesive?

My ol' trusty P-touch label maker is dying and I'm looking for a replacement. This one was used for general label making and was great but on some surfaces the labels would come off after a while. So I'm looking for something that uses some kind of extra strong adhesive on the labels to help with that while also being able to make normal strength labels for the rest of surfaces. I see some P-touch units that accept extra strong tape but don't know how good they are. Did anybody use those or can recommend something?

https://redd.it/1pqxx7w
@r_systemadmin

Anyone else have regrets about their major choice and or think about going back to college?

Originally, and I'm talking 20 years ago, I was a computer science major. Things were going just dandy until the engineering calc and science classes hit...lol. It was clear to me that these were weed out classes and yeah I probably didn't put enough effort into them at the time. I wasted nearly two years and didn't learn a single thing about computers and or programming as there were so many general prereq and engineering related courses (math / science) to take.

I ended up transferring to another college and earned a Bachelors of Information Technology with a minor in computer security. At least a majority of those classes were tech focused. I was happy to learn about MS Server 2003, it was better then calculus! Just about everything from that degree is outdated of course but I suppose it did provide a decent foundation. I did need the degree to have the job where I'm at today and now have nearly 18 years of experience. I was able to graduate with about $12k in student loan debt thanks to working at the time (plus parents paying the first year), those loans have long been paid off.

Fast froward to today and I'm 40 years old. I make about $125k a year here in Ohio with good benefits and work remote 4 days a week. I'm thankful for what I have but part of me will always have a regret about my major choice and even college choice. I work with some people that went to big in state and out of state universities. When we talk about where we went to college I'm always saying "I just went to a local college named X". I've considered going back to college to earn a masters degree in a tech related concentration (Information Systems, or Master of Science in AI) from a reputable school. With a 2 1/2 year old son and being married I'm not sure I could even pull it off.

Anyone else have regrets about their major choice and or think about going back to college?

https://redd.it/1pr6pnu
@r_systemadmin

I need to open 10 years old .nsf (Lotus Notes) file. Is this possible?

Pretty much the noscript says it. For the ongoing case, I need to open old Lotus Notes file with all the email messages and etc. Is this even possible at this age? I did quick search, it seems .nsf files are propretary format of IBM and there is no free apps that can open it. So, I am thinking is purchasing LN license is only way?

https://redd.it/1pr7n7b
@r_systemadmin

Mozilla Firefox and CIS L1 with GPO

Hello folks,


I have a task to configure CIS Level 1 settings for Mozilla Firefox (Windows version) via GPO. When I look at the settings that need to be configured, many of them are listed under the GPO path “Preferences (Deprecated)”.


Example:

Computer Configuration\\Policies\\Administrative Templates\\Mozilla\\Firefox\\Preferences (Deprecated)\\dom.disable_window_flip


I tried reviewing the Firefox documentation, but I’m unsure what the replacement is for this deprecated GPO path. In GPO, there are settings called Preferences and Preferences (JSON on one line). If I understand correctly, I need to find these settings in a different format and paste them there. This feels a bit cumbersome.


Has anyone done this before?

https://redd.it/1pr8e64
@r_systemadmin

Toxic Senior/Mentor is sabotaging me and gatekeeping access, but I just had a baby and can't quit. How to survive 6-12 months?

I joined an IT team of 5 people about a year ago. I was assigned to "Senior Y" for onboarding. Initially, we bonded well over both being new parents, but things turned toxic once I started showing competence.

**The Dynamics:**

* **Sabotage & Credit Stealing:** Early on, he gave me bad advice that got me in trouble, then took credit for my successful work.
* **Gatekeeping:** After my probation (6 months), I was due for T0 (admin) access. Other team members got theirs, but Senior Y blocked mine, claiming I "wasn't ready" because a previous mentee of his broke a GPO once. I’m forced to beg colleagues to share screens to do my job.
* **The Shift:** Realizing he was unreliable, I started working with other competent team members. Senior Y got jealous, realized I didn't need him, and began excluding me from projects and attacking me in meetings.

**The Incident:** Recently, Senior Y struggled with a high-priority Microsoft issue for 2 weeks. I did the research, found the root cause, and sent him the solution. He told me to "stop wasting time," then used my info to fix it. When I asked for details on the fix (to document it properly), he stonewalled me. Feeling thrown under the bus, I maliciously complied and closed the ticket in a way that triggered a management escalation. The Senior and Manager got heat for the lack of documentation.

**Current Situation:** The Manager generally sides with the Senior (they are close). I am now totally isolated. I love the actual work/salary/perks, but the environment is hostile. I just had my second child yesterday (which likely protects me legally from immediate firing), but the IT market is tough.

**My Goal:** I need to survive here for 6-12 months to support my family before finding a new role.

**The Question:** How do you handle a "Mentor" who actively sabotages you? Should I go "Grey Rock," keep my head down, or fight back?

https://redd.it/1praux7
@r_systemadmin

Godaddy Rant (Yeah, I know)

So, before rebuilding my client's WordPress site over the last few days, we ran into major issues caused by GoDaddy’s server migration and infrastructure changes.
::screams into the void::

During a recent period when they added additional servers and shuffled accounts, our site’s database was duplicated and became inconsistent, though the site was still working till Thursday morning. This caused misaligned content, broken plugins, and pages displaying incorrectly. I hadn't logged in for a week and the system failed to send out warning emails lol. In fact, GoDaddy’s built-in backup tools failed to capture the site at all, leaving us unable to reliably restore meaningful data (phpadmin still had my NinjaForms data and a list of my plugins so that was pretty cool). Menus, posts, and caching were all affected, and the site often displayed outdated or broken content. Despite repeated attempts to get support, GoDaddy refused to accept responsibility for the problems caused by their migration (one guy told me they recently bought up some servers). They actually tried to sell me Premium WordPress support at one point just to "help" me (to be fair, it was only the Indian guy I spoke to that tried that) even though its fucking WordPress and my cat could figure it out.

Ultimately, the only solution was to wipe the entire database and rebuild the site from scratch.
::more screaming into the void and wife is staring at me::

To salvage essential content, we relied on Archive.org (shoutout to the GOAT) to recover data that had been lost or rendered inaccessible.

The experience obviously highlighted major vulnerabilities in GoDaddy’s handling of databases and backups and showed how quickly critical content can be compromised during server migrations. I'm back to doing manual backups and keeping a copy on my server.

Needless to say, we should have known better than to trust GoDaddy, and I am actively looking into a more reliable hosting solution for my client going forward. In 10+ years I haven't had any issues with Godaddy and now I see why everyone shits on them.


https://redd.it/1prbaxz
@r_systemadmin

My company was acquired

No general announcement has been made. I know because the acquiring company needed an inventory of physical hardware and VMs

We currently run in a datacenter, the acquiring company is strictly cloud. Our workloads are not cloud friendly generally, large sql databases and large daily transfers from clients. We run nothing in the cloud currently.

How screwed am I?

https://redd.it/1prilrh
@r_systemadmin

Zapier Excel enterprise app - permissions overly broad?

A user asked me to grant admin consent for him to use Zapier to add records to an Excel file in his OneDrive. Upon further inspection, the permissions that this app is requesting seem absurdly broad and unnecessary.

This app would like to:

Have full access to all files user can access.
Allows the app to read, create, update and delete all files the signed-in user can access.
Maintain access to data you have given it access to.
Allows the app to see and update the data you gave it access to, even when users are not currently using the app. This does not give the app any additional permissions.
Edit or delete items in all site collections
Allows the application to edit or delete documents and list items in all site collections on behalf of the signed-in user.
Sign in and read user profile
Allows users to sign-in to the app, and allows the app to read the profile of signed-in users. It also allows the app to read basic company information of signed-in users.

There doesn't seem to be any way to limit the app's access to just one excel file or just one folder, or even to limit it to just the one user's personal OneDrive. The fact that the app could access all SharePoint files in all sites which the user has access to is quite concerning. While I know that Zapier is a reputable software company, it still seems irresponsible to allow such excessive permissions. Has anyone crossed this bridge before? Any suggestions? The boss wants me to make this work but also appreciates security.

https://redd.it/1prelzy
@r_systemadmin

Transition to PAM

Hello Everyone, 

We’re rolling out a PAM solution  with a large number of Windows and Linux servers.



Current state:

1. Users (Infra, DB, Dev teams) log in directly to servers using their regular AD accounts

2. Privileges are granted via local admin, sudo, or AD group membership  

Target state:

1. Users authenticate only to the PAM portal using their existing regular AD accounts

2. Server access will  through PAM using managed privileged accounts  

Before enabling user access to PAM, we need to: 

1. Review current server access (who has access today and why)

2. Define and approve RBAC roles

3. Grant access based on RBAC  

We want to enforce RBAC before granting any PAM access

 

Looking for some advise:

 

1. How did we practically begin the transition?

2. How did we review existing access

3. What RBAC roles did you advise to create

4. How to map current access with new RBAC roles?  

Any sequencing advice to avoid disruption?

https://redd.it/1pre86x
@r_systemadmin

Dell enterprise support

Can anyone help with a UK number to call for Dell enterprise support?

My dell support account is fked so cant see our products, the supposed 24/7 number we have (0800 389 0621) is telling us its now out of hours and our account manager isnt responding to contact attempts!

https://redd.it/1prmo3b
@r_systemadmin

Anyone still doing physical data center decommissions?

We’re sunsetting an old on-prem setup and looking at what a full decommission would involve with things like racks, servers, drives, cables, and the works. Curious how folks are handling this today. Do you go with national vendors? Local scrappers?



Also... do you guys typically get paid for the gear or just pay for haul-away and data wiping?

https://redd.it/1prp5mj
@r_systemadmin

AD account lockouts happening only between 2-4 AM, can’t find the source 😭

Going crazy with this one. Got a user in accounting whose account keeps getting locked out, but only between 2-4 AM. She is definitely not working at that time and swears she doesn’t have any personal devices connected to company stuff.
What I have tried:
1. Ran Lockoutstatus.exe - points to one of our DCs but security logs just show the lockout, not the source
2. Checked scheduled tasks on her workstation, nothing running at those hours
3. Disabled her account on our wifi controller thinking maybe an old phone, lockouts still happened
The weird part is it started about 3 weeks ago and nothing changed on her end. Only thing that happened around that time was we migrated a few shared mailboxes to M365 but she wasn’t part of that project.
Third morning in a row I’m waking up to her helpdesk ticket. What am I missing?​​​​​​​​​​​​​​​​

https://redd.it/1prqfef
@r_systemadmin