Crown Castle 8:54 est outage

Anyone else see a 5 minute outage starting around 8:54am est today, SW CT area? Maybe still down or routing issue..
EDIT: came fully back up around 9:18am. seemed hard down for the first 5min, then routing issues for about 15 minutes.

https://redd.it/1owxgah
@r_systemadmin

Question about app provisioning and offboarding

Our company is expanding from one office in NYC to add remote hires in Mexico, Canada and the Philippines over the next 12 months.





HR is pushing for Rippling because it supposedly handles both onboarding and device/app provisioning in one flow. They’re saying I can kill three tools (Jamf, Okta and some manual Google Workspace noscripts). Has anyone used them? Does it really deprovision Google and Slack accounts when someone quits in another country or is that still a manual thing?

https://redd.it/1owyzxr
@r_systemadmin

follow up re: Microsoft has gotten too big to fail

Update to my ticket came in, the one I posted about in Microsoft has gotten too big to fail, and their support shows it. : r/sysadmin


After weeks of no contact, their support got back to me via email with big news: Can they call me to share this news?

Annoyed they had to call me and couldnt just email me, I said fine.

Here is the big news they shared with me: After many days troubleshooting this issue, spending countless hours on the ticket, their escalation engineers determined that.... I need to open a new ticket for this particular issue. My ticket is not in "scope" for this issue.


I fought back and refused to let them close my old ticket out. As someone who worked in helpdesk for many years, I know how SLAs work. You don't get to close my ticket out until its actually resolved.

https://redd.it/1ox14fe
@r_systemadmin

I swear search engines are getting dumber to force us to use AI

I used to open Bing and search "what is my IP" and in the top search box, I'd get my public IP address. This was helpful at work for servers or whatever else I needed it for.

It also worked if I typed speed test, it would run out like it's own mini Ookla thing, not push browser pages..

I get it, it's not actually "Dumber" they're probably just monitoring their search pages by giving those results over actual functionality. Just annoying that we're pushed (by these tech companies, not internally) to use Copilot or Gemini for searches just to make it look like it's doing something meaningful.

Anytime else notice this?

Can I also go out on a limb and say I feel like Gboard for Android is far less accurate at swipe texting than it used to be, as if trying to get me to use voice or Gemini options instead?

https://redd.it/1ox275t
@r_systemadmin

any experiences with bluetally for asset management?

Hey all - we’re reviewing asset management software for our org (roughly maybe 900+ users across multiple offices and some remote contractors). The team’s been running everything through excel and jira exports, and we’re experiencing a bit of slowdown with some processes because of the sheer number of users and workflows.

Team head asked us to demo a few platforms, and BlueTally came up in our shortlist because of the integrations. On paper, it looks clean with intune, jamf, slack/teams, SCIM, Dell/Lenovo warranty sync, etc.

But I know better than to believe the ads. paper and production are never the same thing. I’m now trying to figure out if anyone here’s actually using it at scale, to the tune of like 1k+ assets. Basically, how is it working for your team and would you recommend it?

Thanks

https://redd.it/1owzfw1
@r_systemadmin

Question to satisfy my curiosity: Why did you choose to use Oracle SQL this day and age, and was there a major reason why?

I can only think it would be due to legacy applications that use some type of special feature.

https://redd.it/1ox2d54
@r_systemadmin

Adobe Acrobat Alternatives

Hey everyone, our org is getting hit hard by Adobe Acrobat Pro/Standard renewal costs, so we need to switch ASAP. We just need something cheaper (or open-source) that can handle real editing, splitting/merging, form filling, and markup without being a pain to use.

https://redd.it/1ox2xfj
@r_systemadmin

Who does ITAD well?

In a new role. We have ongoing hardware turnover and need to decommission. have good recommendations for ITAD in the midwest? What security measures, certs, or otherwise should I be looking for? Thanks in advance

https://redd.it/1ox0skb
@r_systemadmin

has anyone tried smaller european cloud providers instead of aws or azure?

I've been looking into alternatives to the usual hyperscalers like aws, azure and google cloud for a few of our european clients who care a lot about data privacy and iso-certified hosting.

while checking options we found a few interesting european providers such as xelon, scaleway and hetzner. all of them offer iaas setups that look a bit simpler and more transparent than the big ones.

xelon caught my eye mainly because their data centers are swiss based and iso certified, which is really appealing for data protection. the interface also feels a lot cleaner and easier, especially for teams that don’t have a huge devops department.

curious if anyone here has used any of these smaller platforms for production workloads. how do they compare in performance, and support next to aws or azure?

https://redd.it/1ox0p8u
@r_systemadmin

Acrobat filling up the C:\Windows\Installer folder on a large number of computers?

I've had this issue on countless computers. The drive is full, I check what is taking up the space, and its always a 50GB+ C:\\Windows\\Installer folder, sometimes in the 100s


All I have to do is uninstall Acrobat and instantly the folder goes down to \~5GB


Anybody else have a similar problem?

https://redd.it/1oxcrmh
@r_systemadmin

Exchange to 365

got the quote below from the company we use for our IT management, we're upgrading our current 10 year old server and hoping to move from on premise exchange to M365, but the cost of just that migration they're saying $18k - $27kReview existing Exchange 2016 environment

Identify total mailbox count, mailbox sizes, shared mailboxes, and permissions

Determine migration method based on Microsoft requirements

Document mail flow, accepted domains, and connectors

Develop a migration schedule

Configure Exchange Online protection (EOP) and spam filtering policies

Assign appropriate Microsoft 365 licenses to user accounts

Set up baseline policies for retention

Configure Exchange Online and on-premises connectors for mail flow

Enable directory synchronization using Azure AD Connect

Verify synchronization of user accounts, groups, and passwords

Test mail flow between on-premises Exchange and Exchange Online.

Prepare mailboxes for migration

Migrate user and shared mailboxes to Exchange Online

Verify successful migration of mailboxes and permissions.

Update Outlook profiles and reconfigure mobile devices as needed

Perform delta sync or final data synchronization.

Update DNS records

Validate mail flow through Microsoft 365.

Decommission or disable mail flow from on-prem Exchange.

Configure MFA

End User Support as needed

Configure shared resources and room mailboxes.

sound legit for 25 email accounts?

https://redd.it/1ox902y
@r_systemadmin

Microsoft Support Needed 3 Months. Our T3 Needed 5 Minutes.

Apprentice at a small MSP here. We handle a bunch of schools, and last spring one of our users at a school I help manage with my T2 colleague reported that they couldn’t book certain rooms in Outlook.

We’ve got loads of resource mailboxes — basically every room on campus is its own mailbox — and for whatever reason this user could book a few but not the rest. Same permissions, same setup, but nope… Outlook insisted those rooms were off-limits. So I do what every apprentice does: the sacred rite of “Google absolutely everything before bothering T2.”

No dice.

T2 also has no clue.

So we open a ticket with Microsoft Support.

And then began three months (not exaggerating) of back-and-forth with the same support person. They asked for logs. Then more logs. Then the same logs formatted differently. Screenshots. Screen recordings. “Have you tried turning it off and on again?” I’m convinced at some point they were going to ask for the user’s star sign.

Eventually they come back with:

“This issue has never been seen before. We’ve escalated it.”

SLA paused because it’s “with a third party,” user found a workaround, but the ticket still needed a proper resolution so it stayed on hold… forever.

We were basically resigned to the idea that if Microsoft couldn’t figure it out, nobody could. So we left it on hold and moved on.

Fast-forward a few weeks: we’re on a group call with some colleagues, the cursed ticket comes up, and one of our T3s goes, “Eh, let me take a look.”

Five minutes later — I mean literally five minutes — he solves it.

What was the magical, arcane fix that Microsoft Support couldn’t uncover with three months of logs and “investigation”?

Assigning delegated access to the user for those resource mailboxes in Exchange Admin.

We had checked access in the main Microsoft 365 Admin Center when the ticket first came in and everything looked correct, but we didn’t think to look specifically in Exchange Admin again because… well… Microsoft said they were on it.

Three months.

Five minutes.

T3 absolutely legendary.

Microsoft Support absolutely not.

https://redd.it/1oxiss2
@r_systemadmin

Ansible management for non-AD servers?

We manage (most) servers with Active Directory. We manage user devices with Entra/Intune.

We have some devices and VMs that, for security reasons, we don't want to touch AD. It's mostly devices that we have lower trust of, such as HVAC systems. We still need to manage these systems and harden them to the best of our ability.

Most of these systems are Windows Server 2019 or Alma Linux.

I have never used Ansible. Is Ansible a good compromise, or am I barking up the wrong tree?

https://redd.it/1oxk91e
@r_systemadmin

I've deleted the ccmcache folder on a couple of servers. How screwed am I?

So I've deleted the content of the folder C:\\Windows\\ccmcache (not the folder itself) on at least 10 windows servers (2012 to 2002).

The thing is some of them had updated recently and It was pending a reboot.

Is there any chance of them to be affected at next boot?

Thanks!!

https://redd.it/1oxcnto
@r_systemadmin

Driver Management

Hi all, just looking for some tips on driver management for an array of devices. We have a mixture of HP, Lenovo and Surface devices. Currently we are co managed using Autopatch for deployment of drivers. We are quite strict with our deployment rings so the drivers adhere to the Windows update policy that is 2 days deferred. Which the drivers inherit. We do automatically approve each driver which yes is a bit of nightmare having to keep on top of this. We have had some complaints as you can control when the drivers install without setting maintenance windows which would be missed by the workforce shutting machines down. I'm looking to try find a way where the user can be warned that there are pending driver installs which will then prompt them to postpone but also enforce if not done within a certain amount of time. I know there are solutions per manufacturer but wondered if anyone has had the same problem or managed to get some decent to manage this. Thanks

https://redd.it/1oxmk62
@r_systemadmin

Patch manager for the 3 OS's

Hello, Currently trying to find a good patch manager for system and third-party applications on Windows, Mac, AND Linux (Ubuntu). That last one seems to be the kicker in all of this. We've tried ManageEngine, but their support is utterly horrid and I don't want to go with them for that reason even though the price is right. We demoed NinjaOne and it looks great, but it's pretty expensive and we only need a patch manager.


What are people using that cover the 3 OS's?

https://redd.it/1oxsvrc
@r_systemadmin

What is the rationale behind blocking mobile device native mail apps on MDM?

Title says it.

I’m trying to understand the philosophy my company adopted where if a mobile device joins our tenant (BYOD or company mobile), that device cannot add any company email profile to its native mail app tools like iOS Mail or Samsung Mail. Every user must use the Oulook Mobile App from Microsoft.

I’m not really for nor against it, I just don’t know the benefits to this decision.

https://redd.it/1oxurrz
@r_systemadmin

The Stage 4 Sysadmin

We've all seen it. An Engineer whose influence/meddling spreads like Cancer throughout an organisations IT systems. Chronically misconfigured systems and shockingly poor process because it made sense to 'them'. Employed as a friend of the CEO, or a self taught fiddler given power beyond their capabilities.

Bring forth your tales of woe and the amount of cleanup required to heal the org. Or was it a Terminal case the org never recovered from?

Edit: Who's to whose

https://redd.it/1oxvo2c
@r_systemadmin

IT Experts....What’s the One Thing You ALWAYS Triple Check During Office Moves?

Every office relocation I’ve been involved in ends with the same IT headaches
Wi-Fi dead zones, racks not working when turn on, ISP delays, cabling mistakes, mislabeled ports, missing equipments... bad infrastructure…

What’s the one thing you ALWAYS check before a company moves into a new office?
The one detail that saves you every time?


https://redd.it/1oxwj05
@r_systemadmin

Chasing problems in the infrastructure

I’m at a loss as to where I should be looking next, so figure I’d toss it out here and see what I might have missed.

To try and keep a long story short; we decided to pull the trigger on implementing a replacement ERP. Previous one was on prem so the board decided to keep this new one on prem instead of paying for the costs associated with cloud. Got the specs and requirements from the ERP vendor before implementation. Worked with our MSP to make the storage upgrades to the SAN as needed, otherwise they said our server meets and exceeds the requirements. However, since working in this ERP, many users have complained about performance issues. The ERP vendor and consultants have also indicated that the performance we are seeing is worse than they’d expect. They offered an AWS instance which was provisioned with half the specs of our on prem server, and it performs 60% better than what we are seeing on local workstations (though directly on the on prem server, performance is similar to AWS).

We’ve done iperf tests to see if its network, and latency is minimal, no packet loss or jitter between the local workstations and server. Monitoring the resources on the host show and it’s barely blinking when under a load. We’ve plugged a workstation as direct to the server as possible and it actually performed worse than before. All workstations are hardwired with a 1gbps connection. The only bottleneck neck that jump out are from our main aggregate to an Aruba that the host plugs into is also only 1Gbps. Our ISP is 600Mbps down/300Mbps up, so with the AWS instance working faster than our on prem doing the same processes now has me thinking it’s the host server. Though the host works as fast as AWS, has me thinking it is within the network instead somehow.

Got a call scheduled with HPE next week to see if there’s anything the MSP and I missed as far as server and Aruba configurations go, but I’m at a loss right now as there’s no smoking gun in the network so far. Literally just throwing everything I can at the wall to see what sticks. Any thoughts on what direction I should be throwing next?



https://redd.it/1oy2r53
@r_systemadmin

Need a cloud user directory management system

I have a very specific challenge. I need a web-based system/tool where I can create a user directory with permission assignment - group assignment. Administrators should be able to manage the user directory. It needs to have support for csv/Excel import, alternatively api integration. I want api integration to read out information from the directory. The directory should be able to have attributes such as name, username, email, social security number, and more fields for other things. Imagine an Active Directory but simpler.

The users in the directory do not need to be able to log in to the system.

The purpose is that I need a temporary solution (max 1 year) for a vocational college to administer its classes in a user-friendly way. Assign students to classes and courses. I want to integrate this data with our AD.

I do not want to let the school administrators into our AD. They need something simpler. In 1 year, our usual platform will be ready for them.

This ok if the solution will have a somewhat high cost. Although I think JumpCloud is a bit too expensive. But It will save us a lot of manually work during this year. It can be cloud or self-hosted.

https://redd.it/1oxuon1
@r_systemadmin