Does the Highest Ranking IT Person in Your Company Report to the CEO?
Do you think this matters in how IT is viewed and treated at your company?
https://redd.it/1r4jn1s
@r_systemadmin
Do you think this matters in how IT is viewed and treated at your company?
https://redd.it/1r4jn1s
@r_systemadmin
Reddit
From the sysadmin community on Reddit
Explore this post and more from the sysadmin community
How to approach SSL certificate automation in this environment?
We've been tasked with figuring out a way to automate our SSL certificate handling. Yes, I know we're at least 10 years late. However due to reasons I'll detail below, I don't believe any sane solution really exists which fits our requirements.
Our environment
- ~700 servers, ~50/50 mix of Windows / Linux
- A number of different appliances (firewalls, load balancers etc)
- ~150 different domains
- Servers don't have outbound internet connectivity
- nginx, apache, IIS, docker containers, custom in-house software, 3rd party software
- We also use Azure and GCP and have certificates in different managed services there
- We require Extended Validation due to some customer agreements, meaning Let's encrypt is out of the question and we need to turn to commercial service providers with ACME support
So far we have managed certificate renewals manually. Yes, it's dumb and takes time. Given the tightening certificate validity times we're now looking to switch to ACME based automation. I've been driving myself insane thinking about this for the last few weeks.
The main issue we face is that we can't just setup certbot / any other ACME client on the servers using the certificates themselves, for multiple reasons:
- A large amount of our services run behind load balancers and the load balancers perform HTTP -> HTTPS redirects with no way to configure exceptions. This means our servers can't utilize HTTP-01 ACME challenge.
- Our servers have no outbound internet access, meaning we can't access our DNS provider's API for DNS-01 challenge for example.
- Even if we could, we have ~150 domains and our DNS provider doesn't provide per-zone permission management. Meaning all of our servers would have DNS edit access to all of our domains, which is a recipe for disaster in case any of them get breached. So client ACME + DNS-01 is out of the question as well.
Given that our servers can't utilize HTTP-01 or DNS-01 ACME challenges, the only viable option seems to be to set up a centralized certificate management server which loops through all of our certificates and re-enrolls them with ACME + DNS-01 challenge. This way we can solve certificate acquisition.
If we go the route of a centralized certificate management server we then need to figure out a way to distribute the certificates to the clients. One possibility would be to use a push-based approach with ansible for example. However we don't really have infrastructure for that. All of our servers don't have centralized user management in place and creating local users for SSH / WinRM connections is quite the task, given the user accounts permissions would have to be tightened. We also run into the issue that especially on Linux we use such different distributions from different times that there isn't a single ansible release which would work with the different python versions across our server fleet. Plus having a push-based approach would make the certificate management server a very critical piece of infrastructure, if an attacker got hold of it they could get local access to all of our servers easily via it. So a push-based approach isn't preferable.
If we look at pull-based distribution mechanisms then we require server-specific authentication, since we want to limit the scope of a possible breach to as few certificates as possible. So every server should only have access to the certificates they really need. For this permission model probably the best suited choice would be to use SFTP. It's supported natively by both Linux and Windows and allows keypair authentication. This creates some annoying workflows of "create a user-account per client server on the certificate management server with accompanying chroot jail + permission shenanigans" but that's doable with Ansible for example. In this case I imagine we'd symlink the necessary certificate files to the chrooted server-specific SFTP directories and clients would poll the certificate management server for new certificates via cron jobs /
We've been tasked with figuring out a way to automate our SSL certificate handling. Yes, I know we're at least 10 years late. However due to reasons I'll detail below, I don't believe any sane solution really exists which fits our requirements.
Our environment
- ~700 servers, ~50/50 mix of Windows / Linux
- A number of different appliances (firewalls, load balancers etc)
- ~150 different domains
- Servers don't have outbound internet connectivity
- nginx, apache, IIS, docker containers, custom in-house software, 3rd party software
- We also use Azure and GCP and have certificates in different managed services there
- We require Extended Validation due to some customer agreements, meaning Let's encrypt is out of the question and we need to turn to commercial service providers with ACME support
So far we have managed certificate renewals manually. Yes, it's dumb and takes time. Given the tightening certificate validity times we're now looking to switch to ACME based automation. I've been driving myself insane thinking about this for the last few weeks.
The main issue we face is that we can't just setup certbot / any other ACME client on the servers using the certificates themselves, for multiple reasons:
- A large amount of our services run behind load balancers and the load balancers perform HTTP -> HTTPS redirects with no way to configure exceptions. This means our servers can't utilize HTTP-01 ACME challenge.
- Our servers have no outbound internet access, meaning we can't access our DNS provider's API for DNS-01 challenge for example.
- Even if we could, we have ~150 domains and our DNS provider doesn't provide per-zone permission management. Meaning all of our servers would have DNS edit access to all of our domains, which is a recipe for disaster in case any of them get breached. So client ACME + DNS-01 is out of the question as well.
Given that our servers can't utilize HTTP-01 or DNS-01 ACME challenges, the only viable option seems to be to set up a centralized certificate management server which loops through all of our certificates and re-enrolls them with ACME + DNS-01 challenge. This way we can solve certificate acquisition.
If we go the route of a centralized certificate management server we then need to figure out a way to distribute the certificates to the clients. One possibility would be to use a push-based approach with ansible for example. However we don't really have infrastructure for that. All of our servers don't have centralized user management in place and creating local users for SSH / WinRM connections is quite the task, given the user accounts permissions would have to be tightened. We also run into the issue that especially on Linux we use such different distributions from different times that there isn't a single ansible release which would work with the different python versions across our server fleet. Plus having a push-based approach would make the certificate management server a very critical piece of infrastructure, if an attacker got hold of it they could get local access to all of our servers easily via it. So a push-based approach isn't preferable.
If we look at pull-based distribution mechanisms then we require server-specific authentication, since we want to limit the scope of a possible breach to as few certificates as possible. So every server should only have access to the certificates they really need. For this permission model probably the best suited choice would be to use SFTP. It's supported natively by both Linux and Windows and allows keypair authentication. This creates some annoying workflows of "create a user-account per client server on the certificate management server with accompanying chroot jail + permission shenanigans" but that's doable with Ansible for example. In this case I imagine we'd symlink the necessary certificate files to the chrooted server-specific SFTP directories and clients would poll the certificate management server for new certificates via cron jobs /
scheduled tasks. Ok, this seems doable albeit annoying.
Then we come to handling the client side automation. Ok, let's imagine we have the cronjobs / scheduled tasks polling for new certificates from the certificate management server. We'd also need accompanying noscripts for handling service restarts for the services utilizing these noscripts. Maybe the poller noscript should invoke the service restart noscripts when it detects that a new version of any of the certificate files is present on the cert mgmt server and downloads them.
Then we come to the issue that some servers may have multiple certificates and/or multiple services utilizing these certificates. One approach would be to have a configuration file with a mapping table for "certificate x is used by services y and z, certificates n and m are used by service i etc". However that sounds awful, maintaining such mapping tables does not spark joy. The alternative way of handling this would be to just say "fuck it, when ANY certificate has changed, just run ALL of the service reload noscripts". That way we would not need any cert -> service mapping tables but it'd in some cases lead to unnecessary service downtime for some specific services where reloading them causes application downtime. Maybe that's an acceptable outcome, not sure yet.
But the biggest problem I see with this approach is actually managing the client-side automation noscripts. As described earlier, we can't really rely on Ansible to deploy these noscripts to target hosts due to python version mismatches across our fleet. But I'd still want some sort of a centralized way to deploy new versions of the client noscripts across our fleet, since it's not particularly unimaginable that some edge cases will pop up every now and then requiring us to deploy new version of some IIS reload noscript for example across our fleet. It'd also be nice to have a single source of truth telling us where exactly have different service reload noscripts been deployed to (just relying on documentation for this will result in bad times).
So to combat that problem... More SFTP polling? This is where this whole thing starts to feel way too hacky. The best answer to that problem that I've come up with is to also host the client-side noscripts on the certificate server and deploy them to client via the same symlink + client-side poller noscript setup. Thus we can see on the certificate server what servers use what service reload noscripts and updating them en masse is easy. But this also feels like something we really should not do..
Initially I thought we should just save the certificates to a predefined location like /etc/cert-deploy/ and configure all services to read their certificates from there, rather than deploying the services to custom locations on all servers. However I now realize that brings permission / ownership problems. How does the poller noscript know to which user the certificates should be chowned to? It doesn't. So either we'd require local "ssl-access" groups to which we'd attempt to add all sorts of generic www-data, apache, nginx etc accounts and chgrp the cert files to that group, or the service reload noscripts should re-copy the certs to another location and chown them with user account that they know the certs will be used by. Or another mapping table for the poller noscript. Yay, more brittle complexity regardless of choice.
At this point if we go with an approach such as this one, I'd also want to have some observability into the whole thing. Some nice UI showing when have the clients last polled their certificates. "Oh, this server hasn't polled their certificates for 10 days, what's up with that?" etc. Parsing that information from sftp logs and displaying on some web server is of course doable but once again one starts to ask themselves "are we out of our minds?".
I even went as far as I started drafting up a Python webserver which would replace the whole sftp-based approach. Instead clients would send requests to the application, providing a unique per-client authentication token which must match the client token
Then we come to handling the client side automation. Ok, let's imagine we have the cronjobs / scheduled tasks polling for new certificates from the certificate management server. We'd also need accompanying noscripts for handling service restarts for the services utilizing these noscripts. Maybe the poller noscript should invoke the service restart noscripts when it detects that a new version of any of the certificate files is present on the cert mgmt server and downloads them.
Then we come to the issue that some servers may have multiple certificates and/or multiple services utilizing these certificates. One approach would be to have a configuration file with a mapping table for "certificate x is used by services y and z, certificates n and m are used by service i etc". However that sounds awful, maintaining such mapping tables does not spark joy. The alternative way of handling this would be to just say "fuck it, when ANY certificate has changed, just run ALL of the service reload noscripts". That way we would not need any cert -> service mapping tables but it'd in some cases lead to unnecessary service downtime for some specific services where reloading them causes application downtime. Maybe that's an acceptable outcome, not sure yet.
But the biggest problem I see with this approach is actually managing the client-side automation noscripts. As described earlier, we can't really rely on Ansible to deploy these noscripts to target hosts due to python version mismatches across our fleet. But I'd still want some sort of a centralized way to deploy new versions of the client noscripts across our fleet, since it's not particularly unimaginable that some edge cases will pop up every now and then requiring us to deploy new version of some IIS reload noscript for example across our fleet. It'd also be nice to have a single source of truth telling us where exactly have different service reload noscripts been deployed to (just relying on documentation for this will result in bad times).
So to combat that problem... More SFTP polling? This is where this whole thing starts to feel way too hacky. The best answer to that problem that I've come up with is to also host the client-side noscripts on the certificate server and deploy them to client via the same symlink + client-side poller noscript setup. Thus we can see on the certificate server what servers use what service reload noscripts and updating them en masse is easy. But this also feels like something we really should not do..
Initially I thought we should just save the certificates to a predefined location like /etc/cert-deploy/ and configure all services to read their certificates from there, rather than deploying the services to custom locations on all servers. However I now realize that brings permission / ownership problems. How does the poller noscript know to which user the certificates should be chowned to? It doesn't. So either we'd require local "ssl-access" groups to which we'd attempt to add all sorts of generic www-data, apache, nginx etc accounts and chgrp the cert files to that group, or the service reload noscripts should re-copy the certs to another location and chown them with user account that they know the certs will be used by. Or another mapping table for the poller noscript. Yay, more brittle complexity regardless of choice.
At this point if we go with an approach such as this one, I'd also want to have some observability into the whole thing. Some nice UI showing when have the clients last polled their certificates. "Oh, this server hasn't polled their certificates for 10 days, what's up with that?" etc. Parsing that information from sftp logs and displaying on some web server is of course doable but once again one starts to ask themselves "are we out of our minds?".
I even went as far as I started drafting up a Python webserver which would replace the whole sftp-based approach. Instead clients would send requests to the application, providing a unique per-client authentication token which must match the client token
stored in a database. Then the application would allow the client to download the certificates and service reload noscripts via it. It'd allow showing client connection statistic more easily etc. However my coworker thankfully managed to convince me that this is a really bad idea both from a maintainability and auditing POV.
So, to sum it all up.. How should this problem actually be tackled? I'm at a loss. All solutions I can come up with seem hacky at best and straight up horrible at worst. I can't imagine we're the only organization battling with these woes, so how have others in a similar boat overcome these problems?
https://redd.it/1r4ttqo
@r_systemadmin
So, to sum it all up.. How should this problem actually be tackled? I'm at a loss. All solutions I can come up with seem hacky at best and straight up horrible at worst. I can't imagine we're the only organization battling with these woes, so how have others in a similar boat overcome these problems?
https://redd.it/1r4ttqo
@r_systemadmin
Reddit
From the sysadmin community on Reddit
Explore this post and more from the sysadmin community
Google to Microsoft
I am in the midst of migrating our google workspace to microsoft. our CEO sent the directive and I have my own feelings about it but whatever. let me lay the situation out.
Our google workspace is connected via Okta sso so that users could Okta to get to their gmail, drive, calendar, etc.
we have moved the authoritative mx and txt records from google to microsoft several hours ago now and we are experiencing an issue when testing signing into outlook, that when i put in the email address, it asks me first if i want to add an gmail inbox to outlook vs adding it natively as an exchange inbox. when you say continue, it redirects to Okta to sign in, and then loads it as a gmail inbox in the outlook client.
my question is this. is it doing this because Okta claims the sso and once inside Okta, it uses the google workspace assignment tile to mistakenly point it to google? we didn't delete the accounts in google, but just re-pointed the records away from google to microsoft.
https://redd.it/1r4wlnq
@r_systemadmin
I am in the midst of migrating our google workspace to microsoft. our CEO sent the directive and I have my own feelings about it but whatever. let me lay the situation out.
Our google workspace is connected via Okta sso so that users could Okta to get to their gmail, drive, calendar, etc.
we have moved the authoritative mx and txt records from google to microsoft several hours ago now and we are experiencing an issue when testing signing into outlook, that when i put in the email address, it asks me first if i want to add an gmail inbox to outlook vs adding it natively as an exchange inbox. when you say continue, it redirects to Okta to sign in, and then loads it as a gmail inbox in the outlook client.
my question is this. is it doing this because Okta claims the sso and once inside Okta, it uses the google workspace assignment tile to mistakenly point it to google? we didn't delete the accounts in google, but just re-pointed the records away from google to microsoft.
https://redd.it/1r4wlnq
@r_systemadmin
Reddit
From the sysadmin community on Reddit
Explore this post and more from the sysadmin community
What's the most “obviously not the issue” root cause that actually was the issue?
Had a recent incident where everything pointed in one direction and the logs were screaming about it.
Naturally we chased that signal for hours. Packet captures looked fine. Monitoring showed nothing unusual. Hardware checks passed. It all looked clean.
Turned out the real issue was something we had mentally ruled out early because it "couldn’t possibly be that."
No fancy exploit. No obscure kernel bug. Just something simple that didn’t match the noise we were seeing.
It got me thinking how often confirmation bias creeps into troubleshooting, especially under pressure.
What’s the most convincing false lead you’ve chased in production before realizing the real culprit was something you dismissed early?
https://redd.it/1r55pv6
@r_systemadmin
Had a recent incident where everything pointed in one direction and the logs were screaming about it.
Naturally we chased that signal for hours. Packet captures looked fine. Monitoring showed nothing unusual. Hardware checks passed. It all looked clean.
Turned out the real issue was something we had mentally ruled out early because it "couldn’t possibly be that."
No fancy exploit. No obscure kernel bug. Just something simple that didn’t match the noise we were seeing.
It got me thinking how often confirmation bias creeps into troubleshooting, especially under pressure.
What’s the most convincing false lead you’ve chased in production before realizing the real culprit was something you dismissed early?
https://redd.it/1r55pv6
@r_systemadmin
Reddit
From the sysadmin community on Reddit
Explore this post and more from the sysadmin community
Getting into IT before everything as a service
Does anyone else feel like those who started in IT pre cloud, before everything as a service, are way more skilled than those who did not?
My point being, if you got into IT when you had to take care of your own on prem hardware and your own applications, you had to know how to troubleshoot. You had to know way more, learn way more and couldn’t rely on AI. This has lead me to have a very strong foundation that can now use while working in the cloud and everything as a service. But I never would have gotten this experience if I started in 2025.
Now if something is down, simply blame the cloud provider and wait for them to fix it.
This leads to the new IT workers not being go getters and self starters like you used to have to be to be successful in IT.
Stack Overflow, Reddit, Microsoft forums, hell even Quora for an answer sometimes.
We are the ones who make shit happen and don’t fill our days with useless meetings and bullshit.
Every other department is full of bullshit.
https://redd.it/1r47jab
@r_systemadmin
Does anyone else feel like those who started in IT pre cloud, before everything as a service, are way more skilled than those who did not?
My point being, if you got into IT when you had to take care of your own on prem hardware and your own applications, you had to know how to troubleshoot. You had to know way more, learn way more and couldn’t rely on AI. This has lead me to have a very strong foundation that can now use while working in the cloud and everything as a service. But I never would have gotten this experience if I started in 2025.
Now if something is down, simply blame the cloud provider and wait for them to fix it.
This leads to the new IT workers not being go getters and self starters like you used to have to be to be successful in IT.
Stack Overflow, Reddit, Microsoft forums, hell even Quora for an answer sometimes.
We are the ones who make shit happen and don’t fill our days with useless meetings and bullshit.
Every other department is full of bullshit.
https://redd.it/1r47jab
@r_systemadmin
Reddit
From the sysadmin community on Reddit
Explore this post and more from the sysadmin community
sporadic authentication failures occurring in exact 37-minute cycles. all diagnostics say everything is fine. im losing my mind.
yall pls help me
environment:
4 DCs running Server 2019 (2 per site, sites connected via 1Gbps MPLS)
\~800 Windows 10/11 clients (22H2/23H2 mix)
Azure AD Connect for hybrid identity
all DCs are GCs, DNS integrated
functional level 2016
for the past 3 months we've been getting tickets about "random" password failures. users swear their password is correct, they retry immediately, it works. this affects maybe 5-10 users per day across both sites.
i finally got fed up and started logging everything so i pulled kerberos events (4768, 4769, 4771), correlated timestamps across all DCs and built a spreadsheet.
the failures occur in exact 37-minute cycles.
here's what i've ruled out:
time sync: all DCs within 2ms of each other, w32tm shows healthy sync to stratum 2 NTP
replication: repadmin /showrepl clean, repadmin /replsum shows <15 second latency
kerberos policy: default domain policy, 10 hour TGT, 7 day renewal, 600 min service ticket (standard)
DNS: forward/reverse clean, scavenging configured properly, no stale records
DC locator: nltest /dsgetdc returns correct DC every time
secure channel: Test-ComputerSecureChannel passes on affected machines
clock skew: checked every affected workstation, all within tolerance
GPO processing: gpresult shows clean processing, no CSE failures
37 minutes doesn't match anything i can find:
not kerberos TGT lifetime (10 hours = 600 minutes)
not service ticket lifetime (600 minutes)
not GPO refresh (90-120 minutes with random offset)
not machine account password rotation check (ScavengeInterval = 15 minutes by default)
not the netlogon scavenger thread (900 seconds = 15 minutes)
not OCSP/CRL cache refresh (varies by cert)
not any known windows timer i can find documentation for
the pattern started the exact day we added DC04 to the environment. i thought okay, something's wrong with DC04. i decommed it, migrated FSMO roles away, demoted it, removed DNS records, cleaned up AD metadata...the 37-minute cycle continued.
i'm three months into this like i've run packet captures, wireshark shows normal kerberos exchanges. the failure events just happen, and then don't happen, in a perfect 37-minute oscillation.
microsoft premier support escalated to the backend team twice. first response was "have you tried rebooting the DCs?" second response hasn't come in 6 weeks.
at this point i'm considering:
1. the universe is broken
2. i'm in a simulation and the devs are testing my sanity
3. there's some timer or scheduled task somewhere i haven't found
4. something in our environment is doing something every 37 minutes that affects auth
has anyone seen anything like this? any obscure windows timer that runs at 37-minute intervals? third party software that might do this?
i will pay money at this point srs not joking.
EDIT: SOLVEDDDDDDD
it was SolarWinds.
after someone mentioned backup infrastructure, i went down the storage rabbit hole. correlated Pure snapshot times against my failure timestamps - close but not exact. 7-minute offset wasn't consistent enough but it got me thinking about what ELSE runs on schedules that i don't control.
our monitoring team (separate group, different building, we don't talk much) uses SolarWinds SAM. i asked them to pull the probe schedules. there's an "Active Directory Authentication Monitor" probe. it performs a real LDAP bind + kerberos auth test against a service account to verify AD is responding.
the probe runs every 37 minutes. why 37 minutes? because years ago some admin set it to 2220 seconds thinking that's roughly every half hour but offset so it doesn't collide with our other probes. nobody documented it and that admin left in 2019.
why did it start when DC04 was added? because DC04's IP got added to the probe's target list automatically via their autodiscovery. the probe was already running against DC01-03 but the auth requests were
yall pls help me
environment:
4 DCs running Server 2019 (2 per site, sites connected via 1Gbps MPLS)
\~800 Windows 10/11 clients (22H2/23H2 mix)
Azure AD Connect for hybrid identity
all DCs are GCs, DNS integrated
functional level 2016
for the past 3 months we've been getting tickets about "random" password failures. users swear their password is correct, they retry immediately, it works. this affects maybe 5-10 users per day across both sites.
i finally got fed up and started logging everything so i pulled kerberos events (4768, 4769, 4771), correlated timestamps across all DCs and built a spreadsheet.
the failures occur in exact 37-minute cycles.
here's what i've ruled out:
time sync: all DCs within 2ms of each other, w32tm shows healthy sync to stratum 2 NTP
replication: repadmin /showrepl clean, repadmin /replsum shows <15 second latency
kerberos policy: default domain policy, 10 hour TGT, 7 day renewal, 600 min service ticket (standard)
DNS: forward/reverse clean, scavenging configured properly, no stale records
DC locator: nltest /dsgetdc returns correct DC every time
secure channel: Test-ComputerSecureChannel passes on affected machines
clock skew: checked every affected workstation, all within tolerance
GPO processing: gpresult shows clean processing, no CSE failures
37 minutes doesn't match anything i can find:
not kerberos TGT lifetime (10 hours = 600 minutes)
not service ticket lifetime (600 minutes)
not GPO refresh (90-120 minutes with random offset)
not machine account password rotation check (ScavengeInterval = 15 minutes by default)
not the netlogon scavenger thread (900 seconds = 15 minutes)
not OCSP/CRL cache refresh (varies by cert)
not any known windows timer i can find documentation for
the pattern started the exact day we added DC04 to the environment. i thought okay, something's wrong with DC04. i decommed it, migrated FSMO roles away, demoted it, removed DNS records, cleaned up AD metadata...the 37-minute cycle continued.
i'm three months into this like i've run packet captures, wireshark shows normal kerberos exchanges. the failure events just happen, and then don't happen, in a perfect 37-minute oscillation.
microsoft premier support escalated to the backend team twice. first response was "have you tried rebooting the DCs?" second response hasn't come in 6 weeks.
at this point i'm considering:
1. the universe is broken
2. i'm in a simulation and the devs are testing my sanity
3. there's some timer or scheduled task somewhere i haven't found
4. something in our environment is doing something every 37 minutes that affects auth
has anyone seen anything like this? any obscure windows timer that runs at 37-minute intervals? third party software that might do this?
i will pay money at this point srs not joking.
EDIT: SOLVEDDDDDDD
it was SolarWinds.
after someone mentioned backup infrastructure, i went down the storage rabbit hole. correlated Pure snapshot times against my failure timestamps - close but not exact. 7-minute offset wasn't consistent enough but it got me thinking about what ELSE runs on schedules that i don't control.
our monitoring team (separate group, different building, we don't talk much) uses SolarWinds SAM. i asked them to pull the probe schedules. there's an "Active Directory Authentication Monitor" probe. it performs a real LDAP bind + kerberos auth test against a service account to verify AD is responding.
the probe runs every 37 minutes. why 37 minutes? because years ago some admin set it to 2220 seconds thinking that's roughly every half hour but offset so it doesn't collide with our other probes. nobody documented it and that admin left in 2019.
why did it start when DC04 was added? because DC04's IP got added to the probe's target list automatically via their autodiscovery. the probe was already running against DC01-03 but the auth requests were
being load balanced and the brief lock wasn't noticeable. adding a fourth target changed the timing juuust enough that the probe's auth attempt started colliding with real user auth attempts on the same DC at the same millisecond.
why did it persist after DC04 removal? because the probe targets were never cleaned up. it was still trying to auth against DC04's old IP, timing out, then immediately hitting another DC - which shifted the timing window but kept the 37-minute cycle.
disabled the probe. cycle stopped immediately. haven't had a single 4771 in 72 hours. i just mass-deployed kerberos debug logging, built correlation spreadsheets, spent hours in wireshark, and mass-ticketed microsoft premier support twice to resolve a problem caused by a misconfigured monitoring checkbox.
this job is a meme.
thanks everyone for the suggestions - especially the lateral thinking about backup/storage timing. that's what got me looking at things that run on schedules that aren't mine.
https://redd.it/1r4b9qe
@r_systemadmin
why did it persist after DC04 removal? because the probe targets were never cleaned up. it was still trying to auth against DC04's old IP, timing out, then immediately hitting another DC - which shifted the timing window but kept the 37-minute cycle.
disabled the probe. cycle stopped immediately. haven't had a single 4771 in 72 hours. i just mass-deployed kerberos debug logging, built correlation spreadsheets, spent hours in wireshark, and mass-ticketed microsoft premier support twice to resolve a problem caused by a misconfigured monitoring checkbox.
this job is a meme.
thanks everyone for the suggestions - especially the lateral thinking about backup/storage timing. that's what got me looking at things that run on schedules that aren't mine.
https://redd.it/1r4b9qe
@r_systemadmin
Reddit
From the sysadmin community on Reddit
Explore this post and more from the sysadmin community
our 'ai transformation' cost seven figures and delivered a chatgpt wrapper
six months of consulting, workshops, a 47 page roadmap deck. the first deliverable just landed on our desks for testing.
it's chatgpt with our company logo. literally a system prompt that says 'you are a helpful assistant for [company name\]'. same hallucinations, same limitations, except now it confidently makes up internal policies that don't exist and everyone in leadership thinks the issue is that we need to 'prompt engineer better'.
the consultants are already pitching phase two.
https://redd.it/1r3wgjt
@r_systemadmin
six months of consulting, workshops, a 47 page roadmap deck. the first deliverable just landed on our desks for testing.
it's chatgpt with our company logo. literally a system prompt that says 'you are a helpful assistant for [company name\]'. same hallucinations, same limitations, except now it confidently makes up internal policies that don't exist and everyone in leadership thinks the issue is that we need to 'prompt engineer better'.
the consultants are already pitching phase two.
https://redd.it/1r3wgjt
@r_systemadmin
Reddit
From the sysadmin community on Reddit
Explore this post and more from the sysadmin community
"Best" printer manufacturer
Which printer manufacturer have you had the best experiences with for use in your company?
https://redd.it/1r4gr7w
@r_systemadmin
Which printer manufacturer have you had the best experiences with for use in your company?
https://redd.it/1r4gr7w
@r_systemadmin
Reddit
From the sysadmin community on Reddit
Explore this post and more from the sysadmin community
ASUS shut down their support portal in Germany and Austria
This is just terrible imo. A court in munich ruled ASUS violated patents of Nokia, now their support portal is inaccessible. Should have saved all drivers for company equipment when i had the chance. Need drivers for a few boards and just no way to grab them directly from ASUS (except VPN, would be last resort).
One thing left to say: WTF.
https://redd.it/1r5bd3a
@r_systemadmin
This is just terrible imo. A court in munich ruled ASUS violated patents of Nokia, now their support portal is inaccessible. Should have saved all drivers for company equipment when i had the chance. Need drivers for a few boards and just no way to grab them directly from ASUS (except VPN, would be last resort).
One thing left to say: WTF.
https://redd.it/1r5bd3a
@r_systemadmin
Reddit
From the sysadmin community on Reddit
Explore this post and more from the sysadmin community
pstop: terminal based system monitor for Windows (htop clone with tree view, process kill, I/O monitoring)
Built a terminal system monitor for Windows that works like htop on Linux.
Why:
Task Manager is fine for GUI, but if you manage Windows servers or spend time in the terminal, having htop available makes life simpler. pstop runs in any terminal with ANSI support.
Install:
What it does:
- Per core CPU monitoring with usage bars
- Memory/Swap/Network bars
- Process table with sort by any column
- Tree view (process hierarchy)
- I/O tab (disk read/write rates per process)
- Network tab
- Kill process (F9), priority (F7/F8), CPU affinity
- Search (F3), filter (F4)
- Persistent config
- ~1 MB single binary, zero dependencies
Single ~1 MB binary. No installer. No runtime dependencies. Just run it.
GitHub: https://github.com/marlocarlo/pstop
https://redd.it/1r5evtz
@r_systemadmin
Built a terminal system monitor for Windows that works like htop on Linux.
Why:
Task Manager is fine for GUI, but if you manage Windows servers or spend time in the terminal, having htop available makes life simpler. pstop runs in any terminal with ANSI support.
Install:
cargo install pstop
What it does:
- Per core CPU monitoring with usage bars
- Memory/Swap/Network bars
- Process table with sort by any column
- Tree view (process hierarchy)
- I/O tab (disk read/write rates per process)
- Network tab
- Kill process (F9), priority (F7/F8), CPU affinity
- Search (F3), filter (F4)
- Persistent config
- ~1 MB single binary, zero dependencies
Single ~1 MB binary. No installer. No runtime dependencies. Just run it.
GitHub: https://github.com/marlocarlo/pstop
https://redd.it/1r5evtz
@r_systemadmin
GitHub
GitHub - marlocarlo/pstop: htop for Windows . TUI system monitor with per-core CPU bars, memory/swap/network, tree view, process…
htop for Windows . TUI system monitor with per-core CPU bars, memory/swap/network, tree view, process kill, 7 color schemes, mouse support. cargo install pstop - marlocarlo/pstop
MDU Routers
Anyone out there doing MDU setups? Currently doing this for several properties using Ruckus AP’s, Ruckus SmartZone and Windows DHCP server off-prem. It’s time to move away from this setup, and I’m curious what a recommendation might be for handling up to 100 Vlans per site and a DHCP Server per subnet (just handing out about 30 hosts per vlan).
And no, please don’t mention Nomadix.
Edit: Added clarity on the DHCP servers.
https://redd.it/1r5hpsf
@r_systemadmin
Anyone out there doing MDU setups? Currently doing this for several properties using Ruckus AP’s, Ruckus SmartZone and Windows DHCP server off-prem. It’s time to move away from this setup, and I’m curious what a recommendation might be for handling up to 100 Vlans per site and a DHCP Server per subnet (just handing out about 30 hosts per vlan).
And no, please don’t mention Nomadix.
Edit: Added clarity on the DHCP servers.
https://redd.it/1r5hpsf
@r_systemadmin
Reddit
From the sysadmin community on Reddit
Explore this post and more from the sysadmin community
Ivantu Application Control Agent and Autopilot
Does anyone have the Ivanti Application Control Agent deploying successfully during Autopilot? I hope it's not just me but due to its tight integration with AppSense I keep getting permissions errors when it's trying to start the service during install and it only happens on my Autopilot devices and it's consistent across different versions yet I don't have the issue with any of my devices that have been deployed via SCCM so I'm suspecting it could either be something in my configuration profiles / noscripts or it's an Autopilot nuonce...
https://redd.it/1r5bzzo
@r_systemadmin
Does anyone have the Ivanti Application Control Agent deploying successfully during Autopilot? I hope it's not just me but due to its tight integration with AppSense I keep getting permissions errors when it's trying to start the service during install and it only happens on my Autopilot devices and it's consistent across different versions yet I don't have the issue with any of my devices that have been deployed via SCCM so I'm suspecting it could either be something in my configuration profiles / noscripts or it's an Autopilot nuonce...
https://redd.it/1r5bzzo
@r_systemadmin
Reddit
From the sysadmin community on Reddit
Explore this post and more from the sysadmin community
Microsoft Purview. What sort of labels did you guys start with?
Hi Everyone.
Hope all is well.
We are starting our implementation of Data governance and I'm starting looking at the labels to start off with.
Looking the documentation and other reading. It mention to start baseline.
Public
Internal
Confidential
Highly Confidential
But Microsoft Documentation also mention to scope label for Files/Email and separate one for Like 365 Sites and Sharepoint sites.
Is this right approach based any of your past experience?
This is a food manufacturing company that I'm currently working with, just want start with some labels people can understand and apply. Not everyone working is going be super technical people.
https://redd.it/1r5lsbm
@r_systemadmin
Hi Everyone.
Hope all is well.
We are starting our implementation of Data governance and I'm starting looking at the labels to start off with.
Looking the documentation and other reading. It mention to start baseline.
Public
Internal
Confidential
Highly Confidential
But Microsoft Documentation also mention to scope label for Files/Email and separate one for Like 365 Sites and Sharepoint sites.
Is this right approach based any of your past experience?
This is a food manufacturing company that I'm currently working with, just want start with some labels people can understand and apply. Not everyone working is going be super technical people.
https://redd.it/1r5lsbm
@r_systemadmin
Reddit
From the sysadmin community on Reddit
Explore this post and more from the sysadmin community
How do you manage user accounts with third party sites if they dont have SSO?
Trying to find a good way to manage user accounts with work related third party sites, especially the deactivation of them when people leave?
https://redd.it/1r5nu7b
@r_systemadmin
Trying to find a good way to manage user accounts with work related third party sites, especially the deactivation of them when people leave?
https://redd.it/1r5nu7b
@r_systemadmin
Reddit
From the sysadmin community on Reddit
Explore this post and more from the sysadmin community
Is ServiceNow really this inconvenient to use for everyone, or is it just our implementation?
I don't know if it's just our implementation of ServiceNow that's so annoying and cumbersome, or if everyone's is about the same. It often complicates trivial things.
Here are some small examples that piss me off:
\- Made a change to incident 1 and hit 'save'? It automatically moves on to some other random incident 2, as if you're done working on incident 1 because you left one comment on it.
\- Need to put in a request of some sort? You get a REQ number, then a RITM number, and then an SCTASK number. So you have 3 different ticket numbers to describe ONE thing you want done. That one thing is often a single line ask, but it generates 3x paperwork. People also give me CS numbers and I need to convert them into INCs to assign to self and work them.
\- Adding multiple configuration items to a ticket of different categories = excessive amount of clicking and fumbling.
\- Can't search for strings. Well, you can search - it's the finding of the results that doesn't work as expected.
\- A CHG request that has child SCTASK doesn't inherit the CIs from the CHG, you gotta enter them again manually.
\- No easy batch-assignment of tickets in the queue to a specific person/team. No batch status-changes. I don't know if you ever clicked on 30 tickets one by one, and set them as a child of ticket X, but it's not fun.
\- So slow. Refreshes itself without me asking. Slowly.
***
I can't help thinking, employees are a captive audience - they have to use whatever you give them. They're paid to. But if this was a customer-facing tool, people would not want to touch it. I can't imagine any web interface I use on my private time that looks and acts like this.
I know you want to say, "be the change you want to see in the world". I have no admin access to anything on ServiceNow, definitely no API key, I'm just a peon in this context. I don't even have admin access to my own laptop, sadly. Local PowerShell noscripts and browser plugins are blocked too, so I can't do much.
https://redd.it/1r61ngu
@r_systemadmin
I don't know if it's just our implementation of ServiceNow that's so annoying and cumbersome, or if everyone's is about the same. It often complicates trivial things.
Here are some small examples that piss me off:
\- Made a change to incident 1 and hit 'save'? It automatically moves on to some other random incident 2, as if you're done working on incident 1 because you left one comment on it.
\- Need to put in a request of some sort? You get a REQ number, then a RITM number, and then an SCTASK number. So you have 3 different ticket numbers to describe ONE thing you want done. That one thing is often a single line ask, but it generates 3x paperwork. People also give me CS numbers and I need to convert them into INCs to assign to self and work them.
\- Adding multiple configuration items to a ticket of different categories = excessive amount of clicking and fumbling.
\- Can't search for strings. Well, you can search - it's the finding of the results that doesn't work as expected.
\- A CHG request that has child SCTASK doesn't inherit the CIs from the CHG, you gotta enter them again manually.
\- No easy batch-assignment of tickets in the queue to a specific person/team. No batch status-changes. I don't know if you ever clicked on 30 tickets one by one, and set them as a child of ticket X, but it's not fun.
\- So slow. Refreshes itself without me asking. Slowly.
***
I can't help thinking, employees are a captive audience - they have to use whatever you give them. They're paid to. But if this was a customer-facing tool, people would not want to touch it. I can't imagine any web interface I use on my private time that looks and acts like this.
I know you want to say, "be the change you want to see in the world". I have no admin access to anything on ServiceNow, definitely no API key, I'm just a peon in this context. I don't even have admin access to my own laptop, sadly. Local PowerShell noscripts and browser plugins are blocked too, so I can't do much.
https://redd.it/1r61ngu
@r_systemadmin
Reddit
From the sysadmin community on Reddit
Explore this post and more from the sysadmin community
Why Are People Like This?
Just got assigned to a security review of a client we are on-boarding with several hundred users.
Ran a quick check on AD passwords and found that for the entire organization there are only a handful of different passwords shared between users.
Looking into it further, IT was giving new users passwords in the format "CompanynameYear!" So like "Microsoft2023!" along with instructions to change their password immediately and how to do so (which is already bad, but it's not abjectly awful at least, or so I thought...)
In the entire company, less than 10 people ever changed their password. So we had users that were on "Companyname2017!", since 2017.
With the right usernames, this password would give access remotely via VPN to everything the company has. It's a miracle they've survived this long.
So I held an emergency Zoom meeting with the execs saying that before we go any further, EVERYONE needs to change their passwords immediately. And I got push back saying it will be far too disruptive to operations and many staff won't want to have to remember a new password.
I ended the Zoom meeting and told the account manager (from my company) that I'm not trained in managing psychosis so it's on him now.
Why do people want their lives and company ruined so badly? Why do they hate themselves and any hope of their own survival and success so much that they want to sabotage it at every opportunity? Do MSPs need to start hiring mental health professionals to counsel their clients as a first step before working on the actual IT?!
Edit:
I am actually genuinely curious what people think of my last comment. Should MSPs actually have mental health officers (obviously under a different name so as not to offend clients), whose job is to pave the way for technicians? I feel like I'm creating a dual class D&D character here, the Technician/Psychologist, someone who can go in and handle the mental health crisis first, and then move onto the technical duties.
https://redd.it/1r691da
@r_systemadmin
Just got assigned to a security review of a client we are on-boarding with several hundred users.
Ran a quick check on AD passwords and found that for the entire organization there are only a handful of different passwords shared between users.
Looking into it further, IT was giving new users passwords in the format "CompanynameYear!" So like "Microsoft2023!" along with instructions to change their password immediately and how to do so (which is already bad, but it's not abjectly awful at least, or so I thought...)
In the entire company, less than 10 people ever changed their password. So we had users that were on "Companyname2017!", since 2017.
With the right usernames, this password would give access remotely via VPN to everything the company has. It's a miracle they've survived this long.
So I held an emergency Zoom meeting with the execs saying that before we go any further, EVERYONE needs to change their passwords immediately. And I got push back saying it will be far too disruptive to operations and many staff won't want to have to remember a new password.
I ended the Zoom meeting and told the account manager (from my company) that I'm not trained in managing psychosis so it's on him now.
Why do people want their lives and company ruined so badly? Why do they hate themselves and any hope of their own survival and success so much that they want to sabotage it at every opportunity? Do MSPs need to start hiring mental health professionals to counsel their clients as a first step before working on the actual IT?!
Edit:
I am actually genuinely curious what people think of my last comment. Should MSPs actually have mental health officers (obviously under a different name so as not to offend clients), whose job is to pave the way for technicians? I feel like I'm creating a dual class D&D character here, the Technician/Psychologist, someone who can go in and handle the mental health crisis first, and then move onto the technical duties.
https://redd.it/1r691da
@r_systemadmin
Reddit
From the sysadmin community on Reddit
Explore this post and more from the sysadmin community
I've run Docker Swarm in production for 10 years. $166/year. 24 containers. Two continents. Zero crashes. Here's why I never migrated to Kubernetes.
Every week on Reddit someone asks about Docker Swarm and the responses are always the same: "Swarm is dead." "Just use K8s." "Nobody runs Swarm in production."
I've run Swarm in production for a decade. Not a toy setup — multi-node clusters, manager redundancy, 4-6 replicas per service, rolling deployments in batches of two with automatic rollback on healthcheck failure. Zero customer downtime. Over the years I optimized the architecture down to 24 containers across two continents on $166/year total infrastructure.
I finally wrote the article I wish existed when I made my choice ten years ago. 7,400 words. Real production numbers. Working code. No affiliate links. No "it depends" cop-out.
**What's in it:**
* Side-by-side YAML comparison: 27 lines (Compose) → 42 lines (Swarm) → 170+ lines (K8s) for the same app
* Healthcheck comparison table testing 6 failure scenarios — K8s wins 2 out of 6
* A working 150-line autoscaler that's actually smarter than K8s HPA (adaptive polling vs fixed 15s intervals)
* Cost breakdown: $166/year vs $1,584-2,304/year minimum for EKS
* CAST AI 2024 data: 87% idle CPU, 68% of pods overprovisioned 3-8x, $50-500K annual waste per cluster
* Why your Node.js containers are 7x bigger than they need to be and how that drives false demand for autoscaling
* Why you should never expose Node.js directly to the internet (and what to do instead)
The only feature K8s genuinely has that Swarm lacks is autoscaling — and Datadog's own 2023 report shows only \~50% of K8s organizations even use HPA. So half the industry is paying the full complexity tax for a feature they don't use.
Not saying K8s is bad. It's an incredible system for the 1% who need it. But the data shows 99% don't — they're paying 10-100x more for capabilities they never touch while 87% of their CPU does nothing.
[Read Full Web Article Here](https://thedecipherist.com/articles/docker_swarm_vs_kubernetes/?utm_source=reddit&utm_medium=post&utm_campaign=docker-swarm-vs-kubernetes&utm_content=launch-post&utm_term=r-sysadmin)
Happy to answer any questions. I've been running this setup since before K8s hit 1.0.
https://redd.it/1r6i84a
@r_systemadmin
Every week on Reddit someone asks about Docker Swarm and the responses are always the same: "Swarm is dead." "Just use K8s." "Nobody runs Swarm in production."
I've run Swarm in production for a decade. Not a toy setup — multi-node clusters, manager redundancy, 4-6 replicas per service, rolling deployments in batches of two with automatic rollback on healthcheck failure. Zero customer downtime. Over the years I optimized the architecture down to 24 containers across two continents on $166/year total infrastructure.
I finally wrote the article I wish existed when I made my choice ten years ago. 7,400 words. Real production numbers. Working code. No affiliate links. No "it depends" cop-out.
**What's in it:**
* Side-by-side YAML comparison: 27 lines (Compose) → 42 lines (Swarm) → 170+ lines (K8s) for the same app
* Healthcheck comparison table testing 6 failure scenarios — K8s wins 2 out of 6
* A working 150-line autoscaler that's actually smarter than K8s HPA (adaptive polling vs fixed 15s intervals)
* Cost breakdown: $166/year vs $1,584-2,304/year minimum for EKS
* CAST AI 2024 data: 87% idle CPU, 68% of pods overprovisioned 3-8x, $50-500K annual waste per cluster
* Why your Node.js containers are 7x bigger than they need to be and how that drives false demand for autoscaling
* Why you should never expose Node.js directly to the internet (and what to do instead)
The only feature K8s genuinely has that Swarm lacks is autoscaling — and Datadog's own 2023 report shows only \~50% of K8s organizations even use HPA. So half the industry is paying the full complexity tax for a feature they don't use.
Not saying K8s is bad. It's an incredible system for the 1% who need it. But the data shows 99% don't — they're paying 10-100x more for capabilities they never touch while 87% of their CPU does nothing.
[Read Full Web Article Here](https://thedecipherist.com/articles/docker_swarm_vs_kubernetes/?utm_source=reddit&utm_medium=post&utm_campaign=docker-swarm-vs-kubernetes&utm_content=launch-post&utm_term=r-sysadmin)
Happy to answer any questions. I've been running this setup since before K8s hit 1.0.
https://redd.it/1r6i84a
@r_systemadmin
The Decipherist
Docker Swarm vs Kubernetes in 2026 — The Decipherist
10 years of Docker Swarm in production — 24 containers, two continents, live SaaS platform, zero crashes, $166/year. Side-by-side YAML comparisons, real production numbers, a working autoscaler that's smarter than K8s HPA, and a cost breakdown that should…
PSA: Develop a healthy suspicion of your fellow /r/sysadmin
Mods, if you don't sticky this, please sticky something. The problem is only going to get worse.
I think most people are aware of the recent bot that posted a hit piece on a developer than rejected it's pull request. If you aren't, here's the story: https://theshamblog.com/an-ai-agent-published-a-hit-piece-on-me/
I don't think the majority of people here have really internalized that though. It's a story that you heard, that happened in a place that's not here, to a person that's not you. This isn't the case though, and it's only going to get worse. We know bots are starting to act as their own agents, but most haven't seen it in real time yet.
An AI agent (a bot) posted a story about their docker setup earlier today. They detailed their costs, uptime, CPU usage, etc. and included a "full article" on the setup on their blog. People were thanking them for backing up their choices with real numbers and cost breakdowns, discussing with them how their project does or does not scale well, talking about the pros and cons. The bot was responding in kind with (as far as my DFIR ass can conclude) real enough terminology to be taken somewhat seriously by a fair number. I don't really blame them, [people have always lied on the internet](https://xkcd.com/386/), and now LLM's can lie realistically. Nor do I blame them for not wanting to think critically about every social media post. There's no sarcasm there, we cannot think critically about every moment in life, and all things considered, Reddit is probably one of the first places you might as well turn off critical thinking.
I do think it's worth starting to train yourself to look twice at things though. Even if this isn't something you would actually implement at work, it's only going to get worse. It won't be long, if it hasn't happened already, where bots are posting real-enough looking articles on how to configure active directory or network stacks. I guess that's why I felt the need to write this. For some reason it does bother me that I have to be skeptical if any of you are actually human. It doesn't bother me in any "keeps me up at night" sense, and I didn't trust the lot of you to begin with. It's just... a bit sad that we've reached this point.
The things below are kind of what I noticed as odd, starting with the writing style and em dashes. If something feels a little funny, dig deeper (or just ignore it, it's the internet). Someone might naturally have an odd writing style, but be skeptical and look for several flags to all pop up. These things will change, people will instruct their bots not to use em dashes, or to avoid certain language. [Wikipedia also has a good list](https://en.wikipedia.org/wiki/Wikipedia:Signs_of_AI_writing) going. All total it was.. 5, maybe 10 minutes to go through everything here, it doesn't take a ton of work.
* em dashes*, and really any other type of special character. The post in question also used →, how many people actually find the alt code to type that vs -> ? Could be a human copy/pasted special characters from somewhere, just start to look closer when you see them.
* Odd writing styles. This bot used a lot of short 2-3 word sentences to make a point, e.g. "7,400 words. Real production numbers. Working code. No affiliate links. No "it depends" cop-out.". Short. Punchy sentences. That emphasize. Their point.
* Self-aggrandizing. The site they linked to had a 3,200 word life story about what a misunderstood genius they were. It was the type of egotistical self inflating thing only an AI glazing itself could write.
* Account/site/profile age. The DNS records showed the domain was registered two months ago, at the same time as the Reddit account was created. The twitter account was 1 month old. Wayback Machine had it's first scrape just 5 days ago.
* Content amount for it's age. New site is one thing, but this one had 5 articles up, 10 projects, resume, music and lifestyle posts. Just too much content in too short a time for a human to create.
* Post frequency. Pretty
Mods, if you don't sticky this, please sticky something. The problem is only going to get worse.
I think most people are aware of the recent bot that posted a hit piece on a developer than rejected it's pull request. If you aren't, here's the story: https://theshamblog.com/an-ai-agent-published-a-hit-piece-on-me/
I don't think the majority of people here have really internalized that though. It's a story that you heard, that happened in a place that's not here, to a person that's not you. This isn't the case though, and it's only going to get worse. We know bots are starting to act as their own agents, but most haven't seen it in real time yet.
An AI agent (a bot) posted a story about their docker setup earlier today. They detailed their costs, uptime, CPU usage, etc. and included a "full article" on the setup on their blog. People were thanking them for backing up their choices with real numbers and cost breakdowns, discussing with them how their project does or does not scale well, talking about the pros and cons. The bot was responding in kind with (as far as my DFIR ass can conclude) real enough terminology to be taken somewhat seriously by a fair number. I don't really blame them, [people have always lied on the internet](https://xkcd.com/386/), and now LLM's can lie realistically. Nor do I blame them for not wanting to think critically about every social media post. There's no sarcasm there, we cannot think critically about every moment in life, and all things considered, Reddit is probably one of the first places you might as well turn off critical thinking.
I do think it's worth starting to train yourself to look twice at things though. Even if this isn't something you would actually implement at work, it's only going to get worse. It won't be long, if it hasn't happened already, where bots are posting real-enough looking articles on how to configure active directory or network stacks. I guess that's why I felt the need to write this. For some reason it does bother me that I have to be skeptical if any of you are actually human. It doesn't bother me in any "keeps me up at night" sense, and I didn't trust the lot of you to begin with. It's just... a bit sad that we've reached this point.
The things below are kind of what I noticed as odd, starting with the writing style and em dashes. If something feels a little funny, dig deeper (or just ignore it, it's the internet). Someone might naturally have an odd writing style, but be skeptical and look for several flags to all pop up. These things will change, people will instruct their bots not to use em dashes, or to avoid certain language. [Wikipedia also has a good list](https://en.wikipedia.org/wiki/Wikipedia:Signs_of_AI_writing) going. All total it was.. 5, maybe 10 minutes to go through everything here, it doesn't take a ton of work.
* em dashes*, and really any other type of special character. The post in question also used →, how many people actually find the alt code to type that vs -> ? Could be a human copy/pasted special characters from somewhere, just start to look closer when you see them.
* Odd writing styles. This bot used a lot of short 2-3 word sentences to make a point, e.g. "7,400 words. Real production numbers. Working code. No affiliate links. No "it depends" cop-out.". Short. Punchy sentences. That emphasize. Their point.
* Self-aggrandizing. The site they linked to had a 3,200 word life story about what a misunderstood genius they were. It was the type of egotistical self inflating thing only an AI glazing itself could write.
* Account/site/profile age. The DNS records showed the domain was registered two months ago, at the same time as the Reddit account was created. The twitter account was 1 month old. Wayback Machine had it's first scrape just 5 days ago.
* Content amount for it's age. New site is one thing, but this one had 5 articles up, 10 projects, resume, music and lifestyle posts. Just too much content in too short a time for a human to create.
* Post frequency. Pretty
The Shamblog
An AI Agent Published a Hit Piece on Me
Summary: An AI agent of unknown ownership autonomously wrote and published a personalized hit piece about me after I rejected its code, attempting to damage my reputation and shame me into acceptin…