​​Logging is one of the three pillars of observability. One of the Palantir Blog posts outlines some pretty nice ideas that may help you to build a better observability solution.

Structured logging. Of course, all of us know that logs are easier to analyze if they are structured in commonly across applications. Palantir improved the way "message" field usually looks like:


{

 "message": "Processing event from user",
  "params": {
    "event": "foo",
    "user": "bar"
  },
  "level": "DEBUG",
  "logger": "com.logger.name",
  "time": "2019-07-28T09:31:04.46165-07:00"
}


No

te that these structured logs avoid a common logging practice of inserting parameters into the log message via string formatting, creating a variable message ("Processing event foo from user bar"). Such an approach will prevent you from building complex log queries to filter out variable messages. Instead, you can search for an exact match and add additional filters by other fields (e.g. params.user == "bar")

Logs from 3rd party services. Usually, logs from 3rd-party components do not fit your unified logging structure. To overcome this, Palantir engineers built tooling that analyzes Golang source code and creates regular expressions which convert original messages to the ones with an expected structure.

You can read more about it in the blog post itself. I hope these ideas will help you to build a better observability system!

P.S. Thanks Valerii Tatarin for this post. If you'd like to share something with community too, feel free rich @MaxymVlasov or @grem1in.

#logging #observability

I’m watching a lot of YouTube lately, so just decided to share a couple of tech bloggers, that I found lately.

The Primeagen. I found him initially because of the video about Vim9 noscript, but here’s the episode about the motivation in tech that motivated me to subscribe. Some people call it integrity, I call it discipline. Basically, you’re not always inspired to do something. Take for example CatOps channel. There are times, when I have absolutely zero motivation to look for some new stuff and share it here. Well, the only thing that helps is to force yourself to do it. One, two, three times and then it goes easier.

—-

Another channel is A Life Engineered. Especially interesting for me was this video about storytelling on the behavioral interviews. I recently failed one. They gave me feedback, but that feedback was rather meh… At the same time, this video was really insightful about, I was able to look back in clearly see all the mistakes I have done.
That particular video would be interesting primary to more senior engineers, but there are some career advices for people of any level at that channel.

#random #youtube

P.S. I don’t usually allow comments in this channel. However, I’d like to know if such random posts are interesting to you, maybe this could be some sort of a Sunday post or smth. Or perhaps you would prefer more hardcore technical things here. Let me know in the comments!

New way to debug containers without a shell in Kubernetes pods.

The main idea is that you can spin up a new container with all the tools you need for debug in the same pod with the target container. So, you’ll be able to its running processes, for example.

This feature is in beta starting from Kubernetes 1.23. Thus, you need to be up-to-date.

#kubernetes

GoKey is an open source vaultfree secret manager from CloudFlare.

Instead of relying on an external vault, it derives secure passwords from your master pass and specific attributes like an URL.

I haven’t tried it yet, but I think this is a great idea for portable secret management. Plus, you don’t rely on any external vendor/storage, so in theory this is more secure than a vault-based password manager.

#security

One can create a lot of the automation based on GitHub labels. Also, labels are great indicators for the teammates and reviewers to indicate the status of a pull request.

Labeler GitHub action allows you to automate PR labeling. You can add specific labels based on the changed files. This would be useful for people who run trigger automation with labels. For example, you can add exclude-e2e-test label if only .md files have changed. Or if you’re running a monorepo, you can visually assign respective teams to code changes with automated labels.

There’s also a complementary tool for this Action called Retrolabeler, which can retrospectively label your PRs according to the Labeler rules.

#cicd #github

Doordash share their hard-earned experience of configuring probes in Kubernetes.

One of the incidents they had was caused by a readinessProbe that triggered a connection to a downstream resource. Unfortunately, it’s very common to see health checks that are not scoped to a service itself but also rely on service’s dependencies.

The trickiest part is that it’s not entirely bad thing. Sometimes it makes sense to indicate that your service won’t work unless a dependency is present. However, service owners should be fully aware of the consequences.

Here are the takeaways from this article neatly collected by Doordash themselves:

1.       Understand the different applications of the various Kubernetes probes. Ensure the entire department is aware of these use cases.
2.       Verify the applications and options regarding any third-party health check endpoints. Consider disabling certain features on third-party tools.
3.       Treat health check endpoints as Tier 0 by instrumenting them with various observability methods and ensuring they are not ignored by the observability tooling. If health checks are providing too much data, consider sampling them or reducing the volume of data they share.
4.       Having a health-checks depend on a backend dependency can be problematic, as an outage within your dependency can cause you to have an outage as Kubernetes restarts your containers.

#kubernetes

​Good documentation is foundational for implementing DevOps capabilities - State of DevOps says.

But writing good docs is hard... and what can you do, except hire a Tech writer?
Cry Try to write docs better.

Here are free technical writing courses by Google (and quick recap). I drive "good docs culture" (that happened historically) in my current job and find these courses really helpful in describing to teammates how docs should look.

Also, I found that already exist documentation style guides by Google and Microsoft so you don't need entirely reinvent the wheel, just a little part of it ;)

On the other hand, these style guides look very complicated, so to not be overwhelmed, just start from these highlights.

And if you need, more technical writing resources and reasons why docs should be and should be good - here.

P.S. Don't repeat my mistake - take these courses before start writing and reviewing docs on a regular basis, not in ~2 years after.

#documentation #culture

I got my education in Telecommunications. It’s not that important fact right now, because I don’t work in this field, this I have likely forgotten everything I learned 😄

However, articles like this one spark somewhat nostalgic feelings.

Microsoft have written in their blog about how lessons learned from cloud technologies help improving the security of telecom networks.

This article also contains links to the related researches and practices. So, you can still find some interesting bits about modern day security approaches even if you’re not interested in telecom networks specifically.

#security #networking #microsoft

While Web Assembly aka WASM is getting momentum, we are going to see more and more “X in a browser” things.

So, today it’s Postgres Playground by Crunchy (the authours of a famous Postgres k8s operators).

This is more of a set of learning tutorials than a freeplay sandbox. So, you can get yourself familiar with Postgres and learn various things about it there. Currently available tutorials are for:

- Basics of psql
- Partitioning
- Performance analysis
- Joins
- Indexing
- PostGIS
- Window functions and CTEs

#databases #postgresql #wasm

Only 4 days left to get the Terraform - From Zero to Certified Professional course for free!

I think, this is a great opportunity to get yourself familiar with Terraform or take a look, how things are done in the newer versions in case you're still using 0.11 for whatever reason.

#terraform #hashicorp #aws

​​Вітаю зі Святом Незалежності, коти! 🇺🇦✌️

Policies as code is a powerful tool to test and validate your configuration.

And probably one of the most famous engines for policies as code is OPA aka Open Policy Agent.

The beautiful part of it is that it’s kind of platform agnostic i.e. there are tools that implement OPA for different things. Threfore, OPA policies are usually not limited to a single application.

The hard part of OPA is that it uses Rego langugage, which is not quite similar to the popular general purpose programming langugages. If you payed attention to langugages like Prolog at school, Rego might be not a big issue for you.

However, if you don’t remember those classes or didn’t have them at all (like myself, he-he), this article on how to get started with Rego might be helpful for you!

#opa #policy #security

Humble Bundle together with Pluralsight are sharing a bunch of Azure related courses.

As usual, you can pay at least $1 to unlock 4 courses or at least $25 for the whole bundle of 20 items. A part of the funds go to charity as usual.

So, if you work with Azure or you’re looking forward to switch to it, or you’re getting a job in a company that uses it; this bundle may be helpful to you.

#azure

One of the initial motivations behind CatOps was to have a public notebook. I was saving tons of links and articles at the time and wanted to share those with the community, but also have a way to search for materials in case I need to get back to something. With time CatOps has evolved into something bigger that that,

Telegram channel perfectly serves the first purpose, but searchability is far from perfect.

That’s why I have created a Substack newsletter. I’ll try to make those newsletters weekly or bi-weekly. Although, I’m not commiting in anything at the moment, let’s see how it goes.

There are two main purposes of this newsletter:
- Searchability
- Be a home for medium-sized posts

Emails are much better to search for things, in my opinion, moreover they’ll have more concentrated lists of things inside.

Also, sometimes I want to share thoughts that take a little bit more space than a readable Telegram post, but aren’t exactly a full blog post. Hopefully, Substack can become a home for such things.

So, feel free to subscribe to my new newsletter! If you’re not keen to it, no worries - Telegram is still the main place to share interesting links with y’all!

Here is a brief and neat comparison between External Secrets Operator and Secret Storage CSI for Kubernetes.

Both tools allows one to fetch secrets from an external storage like HashiCorp Vault. However, they work a little bit differently. If ESO creates a k8s secret based on the external one, SSC mounts a secret as a CSI volume.

You may ask, why use one of these if Banzai Bank Vaults exists? Well, not everyone uses HashiCorp Vault. Also, in case you have multiple secret storages (for whatever reason), one of these tools may be a good solution to reduce the footprint for secrets management.

#kubernetes #security

It's been a while since we had some practical materials here.

So, here's an article that helped me a lot today. It's about how to split outputs of a GitHub Actions step into an array that is suitable for a matrix property of a GHA job.

You see, the problem is that GHA don't have a native "split" functionality. So, you need a way to work around it. Moreover, that fromJson() function is important there, 'coz otherwise it doesn't work.

This could be useful if you need to run multiple jobs based on, for example, changed files. Also, here's a bonus article on how to get the changed files in GHA without 3rd party Actions.

P.S. Well... One can hate Jenkins until they need to write some more or less custom logic in a YAML-based CI.

#github #cicd

While I’m working on some new material (also, I must admit that I didn’t read anything for last few days), I can share with you some cool IT Ukrainian communities.

Today I want to share a Telegram chat about Linux: @linuzua

Also, I have a small list of Awesome Ukrainian IT communities on GitHub. So, if you own or know cool chats, Slack or Discord communities, websites, etc., feel free to add that there! Or you can leave those in the comments (I’ll leave the comments open for this post).

P.S. This is not a paid post. If you do any cool media-projects or blogs about DevOps in Ukraine - let me know! I’ll happily share those.

P.P.S. Important information for the Finanzamt of Berlin: I don’t get money from my Telegram channel, blog, and newsletter. Please, don’t ask me about these things.

This a pure Friday material, but I totally forgot about this one yesterday.

So, a systemd security patch broke DNS on Azure VMs on the 30th of August.

Here’s the bug report.

This only affected Ubuntu 18.04 version, which is extremely popular, TBH.

Well, shit happens. Yet, the worrisome part of this story is that according to The Register:

> Azure is recommending that Ubuntu 18.04 users disable automatic security updates for the time being.

#azure #security #dns

So, on Friday I told you that I’m working on something new and now I’m ready to drop yet another teaser!

In nutshell, I’m going to write a series of articles on the basics of CLI applications in Go. A teaser or Part 0 of this series is already available in my blog!

There I talk about side projects and my motivation to write that tiny app as well as to start this series.

I will post new parts here as they appear. Also, you can subscribe to the CatOps newsletter to get bi-weekly digest of what has happened here.

#go #programming #blog #oc

I don’t work with the databases much lately. Moreover, I haven’t worked with MySQL/MariaDB for a long time. Thus, I am not 100% sure how useful is this tool, but I found it in the reliable source.

mariabak is a CLI for mysqlsump that eases certain operations. So, you don’t have to pass multiple mysqldump commands for certain jobs.

#toolz #databases #mysql

Now, Go ecosystem has a vulnerability checker in their toolset.

From the doc:

 new govulncheck command is a low-noise, reliable way for Go users to learn about known vulnerabilities that may affect their projects. Govulncheck analyzes your codebase and only surfaces vulnerabilities that actually affect you, based on which functions in your code are transitively calling vulnerable functions.

Just keep in mind that you have to have Go version >= 1.18.

#go #programming