How Discord Serves 15-Million Users on One Server.

I think this article was posted before on some of DevOps-related channels.

tl;dr: It's BEAM & Elixir. Also, every time you see an article noscriptd "how a chat app scales to crazy number of users", it's always BEAM.

Yet, in their case, Discord had to make some performance analysis and architectural optimizations with some data offloading. That part is interesting.

P.S. I'd love to learn Elixir someday, but I have zero experience with it or Erlang, or FP in general. So, if you're in the same position, and you're looking for a learning buddy, please, let me know :)

#elixir #architecture

And now it's time for a story of how Pinterest scaled to 11 million users with only 6 engineers.

No BEAM this time, though Pinterest isn't a chat app either. They managed to scale that much and stay so lean thanks to the architectural decisions (mostly around keeping it simple).

P.S. Also, Pinterest is probably the best social network out there (if you could call it a social network).

#architecture

I’m a CLI guy. For me it was always easier to use good old commands like find and cd to navigate around. At some point I even memorized the flags of tar.

Thus, it is still much easier to me to use plain kubectl with a couple of plugins to navigate the clusters.

However, I know that many folks prefer graphical interfaces or at least some TUI. Also, I remember that a lot of folks were pissed when Lens split into a community and a paid versions.

So, today I want to share a new native Kubernetes desktop client - Seabird.

I haven’t tried it for the reasons I mentioned above, but you may enjoy it.

#kubernetes

​​Today a new issue of CatOps Newsletter should’ve come out. However, it’s delayed for reasons.

P.S. If you’re attending Fosdem/CfgMgmt Camp, let’s meet for coffee!

​​A member of our community raises funds for a Mavic 3t for his classmate who serves in the 110 separate mechanized brigade.

You can donate on the Monobank jar

Or on a card directly:

5375 4112 1440 4548

#donations #Ukraine

A delayed CatOps Digest is out!

You can find it here:
https://newsletter.catops.dev/p/catops-digest-2024-02-05

#digest #newsletter

Did you know that Isovalent (a company behind Cilium) has some amazing labs that can teach you about using Cilium, Hubble, and Tetragon.

The labs have multiple tracks, such as: platform, network, security, etc.

These labs also cover topics like the new GatewayAPI. Doing some of these labs tight now at #cfgmgmtcamp24 and love them so far!

#kubernetes #networking #cilium #ebpf

​​For today’s Donations Monday I’d like to remind you about Pavlo and Naya, who raise funds for drones and some telecom equipment for our defenders.

Now they have a neat page with all the requisites in one place.

#donations #Ukraine

​​There was an interesting talk at CfgMgmtCamp 2024 about non blocking code reviews.

The good thing is that there’s also an article on this topic from the author

The general idea is that not all the code changes require a code review, especially when there are enough safety nets configured.

As a result, smaller code changes simply sit there and wait for be reviewed, which may take some time, especially in a remote setup.

The solution is to allow such changes as they are and add them to a backlog of pending reviews.

There are more details in the article. Also, here’s a picture from that presentation that kinda captures the spirit of this idea.

#programming #culture #devops

An overview of the progressive delivery principle by Buoyant.

There's also a comparison between Flagger and Argo Rollouts, which are probably the most famous (if not the only) tools designed for progressive delivery.

Now, Buoyant is the company behind Linkerd. So, obviously, their examples are based on Linkerd. However, you don't need any service mesh to adopt this technique. Service mesh will only help you with smarter traffic splitting.

Also, there's an error in this article: it claims that Flagger works with existing deployments, while Argo Rollouts requires its special CRD, which is only partially true. Argo Rollouts can also work with existing Deployment objects. So, in these regards, both tools are the same.

If you prefer a video format, I would also suggest watching a talk about progressive delivery by Carlos Sanchez at Fosdem 2024. There you can see Argo Rollouts in action.

#cicd

You might have heard that the WeaveWorks has gone out of business.

Yet, they did a right thing and open sourced their GitOps Enterprise product, which previously was only accessible to the paid customers.

#cicd #gitops

Friday is a great day to listen to our latest voice chat!

This time we have discussed what would be the next big thing in operations. And of course, the whole conversation was very AI focused.

You can find the latest episode on:

- YouTube
- Spotify
- Apple Podcasts
- Google Podcasts

#voice_chat #ai

​​AI, AI everywhere…

… we could say after checking Ben Evans's presentation, where he's exploring macro and strategic trends in the tech industry for 2024.

Or we can say that it's still too early and error-prone. Or that we are not sure what we want and what possibilities it provides.

In any case, that is a good analytic, so take your time if you'd like to understand the current state of AI and its near future.

https://www.ben-evans.com/presentations

#ai

A new issue of the CatOps digest is here! You can find it via:

https://newsletter.catops.dev/p/catops-digest-2024-02-18

#digest #newsletter

A friend of mine is raising money for an FPV-drone Bomber Insomnia for defenders from the 119 separate brigade of Territorial Defence.

You can donate directly to this Monobank Jar: https://send.monobank.ua/jar/LmRuSA8dm

#Donations #Ukraine

Linux Foundation has some discounts for its courses and certifications till the end of February. Including Kubernetes, ArgoCD, and Istio certifications.

They are still not cheap, but you can save up to 50%, which is nice.

#kubernetes #training #courses

For you to know: the full Kubernetes CKS (certified security specialist) is available on YouTube. There are both theory and practice, but obviously you will need to take the exam separately.

#kubernetes

Some time ago I had a task to split the helm template output into separate files per object.

So, I found this issue in the Helm’s repository. People were suggesting using AWK for that, but that didn’t work well for me at the time, so I opted out for YQ.

A couple of days ago someone left a comment to that issue that apparently there is a tool called Kubesplit that can do exactly that. So, feel free to use it if you need to achieve something similar to what I did.

#kubernetes

Today marks the 2 year of the russian full-scale invasion of Ukraine.

Today there are going to be rallies in many cities around the globe.

You can find the list here:

https://ukrainianvictory.org/publications/announcement-pro-ukrainian-rallies-marking-the-second-anniversary-of-the-russian-invasion-updated/

Go there, take your family, take your friends, take your colleagues.

Remind you local politicians that war is not over and that there would be no peace in this world until ruzzia exists!

​​For today’s Donations Monday I would like to remind you about Pavlo and Naya, who raise money on drones and telecommunication equipment.

P.S. Here’s a photo of a drone you helped to buy last week and it’s already on its way to the place.

Resend had a 12 hour outage on the 21st of February.

tl;dr:
> The database migration accidentally deleted data from production servers…
> … we performed a database migration command locally, but it incorrectly pointed to the production environment instead…

You can read it in more details is the article, but here are some of the action items from this postmortem:

- No accessible user role should have write privileges on the production database.
- Improve local development to reduce risks related to database migrations.
- Create redundancy to preserve sending function even during a database outage.
- Increase cadence for disaster recovery tests.
- Implement incident banner on Resend dashboard to inform users quickly.

So, I dunno, check your database. Maybe, you have such a risk as well.

Also, it’s kinda strange that people rarely talk about network isolation not only between their production and non-production environments, but also between their local environment and production. Make production access conscious. Put it on a separate role/VPN. Add some friction accessing it.

Moreover, for the love of god, validate your DB backups.

#postmortem #databases