If you haven’t read a Roblox’s postmortem on October‘s 73-hour outage, you definitely should!

Even though this event happened in October, the postmortem was released just a few days ago. And in this case, this is a very good decision! Especially, because this write up provides a detailed analysis on what happened at that time and what chain of events caused that.

It‘s cool to read postmortem the next day after an outage - we are all curious human beings. Unfortunately, those postmortems usually they are lacking many details. This is understandable: it‘s not enough time for a thorough analysis, also your team is probably already tired.

In this case, though, you can have a detailed overview of what happened as well as plans to prevent this chain of events happening again. Moreover, with some plans already implemented.

It‘s a pity that not may companies do similar postmortems. And I must say that this is probably in their disadvantage either. After reading this document I have a feeling that Roblox is a cool place to work, TBH.

#postmortem #hashicorp #consul

Just a friendly reminder that we still have our Kubernetes survey form opened!

We would appreciate it if you can spend some time filling it!

Cheers!

#kubernetes

RedHat presented their own minimal Kubernetes distribution targeted at edge devices - MicroShift.

Functionally, MicroShift repackages OpenShift core components into a single binary that weighs in at a relatively tiny 160MB executable (without any compression/optimization).

As a monolith, it provides an “all-or-nothing” start/stop behavior that works well with systemd and enables fast (re)start times of a few seconds.

So first of all, you can now install OpenShift on a Raspberry Pi as a single binary. Secondly, your car will probably run Kubernetes in the nearest future.

#kubernetes #OpenShift #RedHat

It's time to patch node OS for your Kubernetes clusters. If you manage nodes on your own, of course.

CVE-2022-0185 can allow a container in Kubernetes to escape using unshare Linux command. Usually, it's blocked by Docker's seccomp filter, which is disabled by default in Kubernetes.

#kubernetes #security

And another security update comes from Let's Encrypt.

Let's Encrypt will revoke all active certificates that were issued and validated with the TLS-ALPN-01 challenge before 00:48 UTC on 26 January 2022. Because of a bug, those certificates are considered mis-issued.

They estimate this is <1% of all active certificates, so it's not that bad, though.

#security #tls

Some Friday material.

This short article is a perfect analogy for programming and systems design in general!

Consul API Gateway is now in beta.

It also got TCPRoute Support as well as now it's supported in the official Helm Chart.

There is also a new learning material for you to take a closer look at it.

#hashicorp #kubernetes #networking #consul

So, if you want to know, how the real serverless looks like, here you are.

WebVM allows you to run fully functional Debian virtual machine in your browser.

The article provides an overview of the technology as well as the things that make this possible.

Obviously, there are not many businesses implementations for this at the moment, but I see great potential in it.

#serverless #wasm

Information from our chat.

Be careful committing to SOPS for your secret management.

It looks like they're looking for maintainers at the moment, which means that this project is not supported right now to a degree it was before.

P.S.: If you want to share any interesting information, feel free to join our chat! Although, keep in mind that a mix of Ukrainian and Russian is used there.

#security

​​​​​​​​​​​​HUG Kyiv #13: Q/A session with HashiCorp co-founders in 1 day!

When: Thursday 3rd February, 19:50 (Kyiv TZ)
Where: Online
Language: English

Feel free ask and vote for questions here

#event

This post will be interesting to those, who run Java application in Kubernetes. Specifically, if you use OkHttp client for Java.

Kubernetes network load balancing using OkHttp client - the name speaks for itself. Basically, the problem is that OkHttp uses persistent connections to communicate with the peers. Therefore, IPVS - the default K8s load balancer - is unable to properly balance those connections. Thus, you may have all the connections served by a single replica of your downstream service!

This article provides some useful insights on how to trace down this issue and how to fix it in your code, as well as it mentions tradeoffs of such fix.

#kubernetes #java #programming

We will start in 10 minutes

Youtube - https://youtu.be/GCvhy4I2bzU

Zoom - https://www.meetup.com/Kyiv-HashiCorp-User-Group/events/281541295/

#event

Docker Desktop had already became a paid option for corporate users and not all the companies are willing to pay. Some just haven't bought the licenses yet. This is bad news for Mac user stations, which are likely a majority in the corporate world.

However, there are projects out there, which aim to solve the problem of virtual machine to run your Docker containers.

This article tells a story of migration to Lima from Docker Desktop on MacOS.

There is also a complimentary project with a funny name - Colima, which simplifies the migration even further.

#docker #containers

From our subscribers.

A postmortem by Mozilla on the recent Firefox outage.

For those services hosted on Google Cloud Platform (GCP) these load balancers have settings related to the HTTP protocol they should advertise and one of these settings is HTTP/3 support with three states: “Enabled”, “Disabled” or “Automatic (default)”. Our load balancers were set to the “Automatic (default)” setting and on January 13, 2022 at 07:28 UTC, GCP deployed an unannounced change to make HTTP/3 the default.

Lessons Learned...

GCP’s deployment of HTTP/3 as default was unannounced. We are actively working with them to improve the situation. We realize that an announcement (as is usually sent) might not have entirely mitigated the risk of an incident, but it would likely have triggered more controlled experiments (e.g. in a staging environment) and deployment.

I'm yet to read this postmortem through.

P.S.: If you want to suggest any interesting materials for this channel, you can always do it in our chat. The chat itself is Ukrainian and Russian speaking.

There has been silence on this channel for a while now. I'm sorry for that. I didn't post anything, because I was not in the mood of doing so.

However, I have a formal excuse as well: I started processing the result of Kubernetes Cluster Operations survey that I had issued a few weeks before. So, expect a write-up soon! I would say "this week", but let's see.

In the meantime, you can read a short fiction story about two students, who are trying to figure out, what it actually means "to listen on a port"?

Cya!

GitHub presented Mermaid - a tool to include diagrams into Markdown files.

Basically, it allows to output a diagram in JS-enabled environments and a Markdown syntax in non-JS environments.

I haven't checked if it already works in READMEs on GitHub, but I assume it should.

#markdown #github #microsoft

CatOps exists for almost five years already as a non-profit hobby project. Yes, at some point we were advertising some technical events here, but we never took money for that.

In my opinion, keeping it independent of ad money is what makes all the fun. I'll be honest with you, from time to time I get ideas of starting a Patreon or something similar, but each time it turns out to be too complicated.

However, there is a way to support CatOps financially! Even though this is an indirect support!

Consider subscribing to the Patreon page of the "Come Back Alive" Foundation or you can, of course, donate directly. They have all the details on their website. Unfortunately, not all the information on the website is translated to English, but the main things are there.

This Foundation supports Ukrainian army as well as establish re-integration programs for the veterans.

By supporting people, who take care of us, you empower us in doing our day-to-day jobs and providing some platform engineering content here.

With love 💛💙
@grem1in

Good engineers ship good software, great engineers empower others to do so.

Become a +10% engineer explores some ways of how one can influence others in a good way and ease the work of entire team.

Yes, the name of this article is a word game with the "10x engineer" phrase. Yet, the main idea is that "Productivity of the team > Productivity of the individual"

#culture

I had an idea of writing something about Terraspace for quite some time now. I might have even had a draft somewhere! But of course, someone wrote it before me :)

Here's an article about Terraspace - a part of series about Terraform ecosystem. Other parts are:
- From Terralith to Terraservice with Terraform
- Terragrunt cheat sheet
- Another part with the comparison between Terragrunt and Terraspace is coming soon, but there are already some source files

I got this article from the Terraform Weekly subnoscription. It's curated by a proud member of our community - Anton Babenko. If you use Terraform in your work, you should definitely subscribe!

P.S. If you still think I should write something about Terraspace, let me know by putting 👍 emoji on this post. If you have more specific suggestions, you're always welcome in our chat (chat is in Ukrainian and Russian)!

#terraform #hashicorp #terragrunt #terraspace

​​Finally I got my things together and published the results of the Kubernetes Survey! Many thanks to everyone, who participated in this survey!

The first part is available in English. Also, you can find it in Ukrainian here.

To be honest, I’m not even sure what was easier: write the original post or translate it, haha.

The second part will be available soon. So, stay tuned!

#kubernetes

Ever wondered, how a TCP connection works in slow-mo?

Here's an article just about that. There's a video as well. The link is in the article.

TBH, would be nice if this article covers not only the basics of TCP, but other features as well. Like RST packets.

Also, here's an interesting investigative read, which is not exactly about TCP, but it's features played thy key role for the investigation. Or this case, that actually happened in my company

#networking #tcp