SMURF Testing

Google introduced new mnemonic for test quality attributes - SMURF:
📌 Speed: Unit tests are faster than other test types so they can be run more often.
📌 Maintainability: Cost of debugging and maintaining tests.
📌 Utilization: A good test suite optimizes resource utilization, fewer resources cost less to run.
📌 Reliability: Sorting out flaky tests wastes developer time and costs resources in rerunning the tests.
📌 Fidelity: High-fidelity tests come closer to approximating real operating conditions. So integration tests have more fidelity than unit tests.

In many cases improving one quality attribute can affect the others, so be careful and measure your costs and trade-offs.

#engineering #testing

API Governance at Scale

The most difficult part of API Governance is to ensure that developers follow provided guidelines and policies. Without proper controls, the real code will eventually drift from the guidelines—it’s only a matter of time. This doesn’t happen because developers are bad or unwilling to follow the rules, but because we’re human, and humans make mistakes. Mistakes accumulate and grow over time and as a result you can get APIs that are too far from initial recommendations.

In small teams with a small codebase, developers education can work, and trained reviewers can ensure the code follows the rules. But as your team or organization grows, this approach isn't enough. I strongly believe that only automation can maintain policy compliance over a large codebase or multiple teams.

Google recently published API Governance at Scale, sharing their experience and tools to control API guidelines execution.

They introduced 3 key components:
✏️ API Improvement Proposals (AIPs). This is a design document providing high-level documentation for API development. Each rule is introduced as a separate AIP that consists of a problem denoscription and a guideline to follow (Example, AIP-126).
✏️ API Linter . This tool provides real-time checks for compliance with existing AIPs.
✏️ API Readability Program. This is an educational program to prepare and certify API design experts, who then perform a code review for API changes.

While Google developed the AIPs concept, they encourage other companies to adopt the approach. Many of the rules are generic and easily reusable. They even provide a special guide on how to adopt AIPs. Adoption strategy is not finished now, but preparation status can be tracked via appropriate Github issue.

#engineering #api

Take a Vacation

Last week I was on vacation, so there was a little break in the publications😌. Therefore I would like to talk a little about the vacation and how important it is. High quality vacation is not just opportunity for relax but it is also a prevention mechanism for many serious diseases.

But it’s not enough just to take vacations regularly; the way you spend them determines if you re-charge your internal battery or not.

My tips for a good vacation:

✏️ Take Enough Time: Ideally, a vacation length should be at least 14 days (as a single period). If you feel heavily exhausted, then better to take 21 days. That time is usually enough to recharge.
✏️ Change the Scenery: Travelling to a new place (even a short trip) gives you new impressions, experience, fill you with new ideas, inspiration and energy. Spending time outside standard surroundings significantly decreases an overall strain level. The fact is also proved by German researchers.
✏️ Digital Detox: Don't touch your laptop, don't open working chats, don't read the news, minimize social networks usage. Give the rest to your brain from constant information noise.
✏️ Be Spontaneous: Don't try to plan everything: constant following the schedule makes vacation feel more work-like and doesn't allow to enjoy the moment. Spontaneous activities can provide more fun and satisfaction.
✏️ Do Nothing: Allow yourself to take time for idleness. That's really difficult as you feel just wasting time that can be spend more effectively😀. But that's the trick: state of nothingness rewires the brain, improve creativity and problem solving capabilities.

So take care of yourself and plan a proper rest during the year.

#softskills #productivity

Google ARM Processor

Last week, Google announced their own custom ARM-based processor for general-purpose workloads. They promised up to 65% better price-performance and up to 60% better energy-efficiency.

Why is it interesting? Until now, only AWS offered a custom cost-optimized ARM processor - AWS Graviton. And now Google joined the competition. This shows that interest in ARM processors still grows and continue to grow in the future.

From engineering perspective, it's not possible just to switch workload from one architecture to another as images need to be pre-built for a specific architecture. One of the ways to test ARM nodes and migrate smoothly on the new architecture is by using multi-architecture images (I wrote about that here)

#engineering #news

Uber’s Gen AI On-Call Copilot

GenAI continues its march in routine automation. This time Uber shared their experience with Genie - on-call support automation for internal teams.

The issue is very common for large companies with many teams: there is some channels (for Uber, it's slack with ~45 000 questions per month) where teams can put questions and request help with the service or technology. Of course, there are a lot of docs and relevant articles, but they are fragmented and spread across internal resources. It's really hard for users to find answers on their own. As a result, the number of repetitive questions grows, load and demand on support engineers increase.

Key elements of implemented solution:
✏️ RAG (Retrieval-Augmented Generation) Approach to work with LLM
✏️ Data Pipeline: Information from wikis, internal Stack Overflow, and engineering docs is scraped daily, transformed into vectors, and stored in an in-house vector database with the source links. Data pipeline is implemented on Apache Spark.
✏️ Knowledge Service: When a user posts a question in Slack, Genie’s backend converts it into a vector and fetches the most relevant chunks from the vector database.
✏️ User Feedback: Users can rank answers as Resolved, Helpful, Not Helpful, or Relevant, these ratings are used to analyze answer quality.
✏️ Source Quality Improvements: There is a separate evaluation process to improve source data quality. The LLM performs docs analysis and returns an evaluation score, explanations of the score and actionable suggestions to improve. All these information is collected to an evaluation report for further analysis and fixes.

Since Genie’s launch in September 2023, Uber reports it has answered 70,000 questions with 48.9% helpfulness rate, saving 13 000 engineering hours😲. It's impressive! I definitely want to have something similar at my work. Just a small hurdle left—get the budget and resources for implementation. No big deal, right? 😉

#engineering #usecase #ai

Columnar Databases

Traditional databases store data in a row-oriented approach that is optimized for transactional, single-entity data lookup. But if you need to aggregate data by a specific column, the system has to read all columns from disk, which slows down query performance and increase resource usage.
To solve the issue, columnar databases was introduced.

Columnar database is a type of a database that stores data in columns together on the disk.

Imagine the following sample:

|Account|LastName|FirstName|Purchase,$|
| 0122  | Jones  | Jason   | 325.5    |
| 0123  | Diamond| Richard | 500      |
| 0124  | Tailor | Alice   | 125      |

In row-database it will be stored as following:

   0122, Jones, Jason, 325.5;   
   0123, Diamond, Richard, 500; 
   0124,  Tailor, Alice, 125;        

In column-database:

   0122, 0123, 0124;
   Jones, Diamond, Tailor;
   Jason, Richard, Alice;
   325.5, 500, 125;

Benefits of the approach:
📍High data compression due to the similarity of data within a column
📍Enhanced querying and aggregation performance for in analytical and reporting tasks
📍Reduced I/O load as there is no need to process irrelevant data

The most popular columnar databases:
1. Amazon Redshift
2. Google Cloud BigTable
3. Microsoft Azure Cosmos DB
4. Apache Druid
5. Vertica
6. ClickHouse
7. Snowflake Data Cloud

Columnar databases are well-suited for building data warehouse, real-time analytics, statistics, storing and aggregating time-series data.

#engineering

Manage Your Energy Level

Recently I wrote about the importance of having high-quality vacations. What I didn’t share is that I went on vacation completely drained, with zero level of internal resources and even a diagnosis from a neurologist 😵‍💫. It is a tough state to be in, I never want to feel like that again.

So I reflected on how to prevent burning out in the future.

First of all, I understand that's my fault - not heavy work, urgent issues, or company changes. It's primary responsibility of any leader to support their internal resource and energy. That's very important. Leader cannot work without enough energy level, as it's not possible to drive anything or meet business goals in that state.

Next, I started to study different recommendations what to do. The advice is usually very common: exercise, walk, eat well, and have time for hobbies. Unfortunately, I already knew that, but it didn't help me. My issue is that I don't notice the point where I am completely drained and it's too late to go for a walk.

So I need to control internal state somehow. As technical people, we know that to control something we need to measure something. One resource recommends Welltory app that makes personal health analysis based on heart rate variability (Garmin watches have similar features already built-in). Additionally it uses info about sleep, steps, stress level, and more from mostly any smart watch. Looks like a magic, but there is real science under that. This isn’t an ad—just sharing a tool I found useful 🙂.

I've been using the app for about 2 weeks now. The algorithm is still training (about 35% done), but I’m already using its basic features. I periodically make measurements and check overall state: green, orange or red. Based on this, I’ve started taking short recovery breaks at work to avoid hitting zero. Also I control overall health trend to understand if my daily routine needs additional corrections like more exercise, walk, etc.

Burnout is very common problem in our industry that's why I decided to share my experience on what can be helpful to control internal state and support good level of motivation and energy. Of course, 2 weeks are not enough to say the approach works. Put likes if the topic is interesting and I'll share my results in 1-2 months.

Stay healthy and take care of yourself.

#softskills #productivity

Cloud Ecosystem Trends

This week CNCF published Emerging trends in the cloud native ecosystem with a list of trends that will continue to grow in 2025.

Top trends:
🚀 Cloud Cost Optimizations. With growing cloud adoption, businesses focus on controlling cloud costs using tools like Karpenter and OpenCost. The same trend was also highlighted by FinOps Foundation earlier this year.
🚀 Platform Engineering (I did overview there). Extend developer experience with platforms for observability, policies as a code, internal developer portals, security, CI/CD, and storage to speed up business development.
🚀 AI Synergy. The trend is to support AI training and operations in the cloud. New actively developed projects there:
- OPEA: a collection of cloud-native patterns for GenAI workloads
- Milvus: a high-performance vector database
- Kubeflow: a project to deploy machine-learning workflows on Kubernetes
- KServe: a toolset for serving predictive and generative machine-learning models
🚀 Observability Standards Unification. Projects like OpenTelemetry and the Observability TAG unify standards, minimize vendor locks, and reduce costs.
🚀 Security. Security is a top priority topic in CNCF. There are some newly graduated projects in that area (like Falco) and separate TAG-Security group that publishes white papers that offer directions to the industry on the topic of security.
🚀 Sustainability (more about GreenOps there).  Sustainability tools (like Kepler, OpenCost) measure carbon footprints of Kubernetes applications. The area is under active development now, but it already has promising open-source projects and standards.

It's interesting that overprovisioning and high resource waste is still the main problem in modern clouds. According to the Kubernetes Cost Benchmark Report clusters with 50 or more CPUs used only 13% of their provisioned capacity, memory utilization was at the level of 20%. This shows a huge opportunity for future optimizations.

#news

I'm introducing a new section on the channel: #aibasics !

Over the past two years, ML is the top trend in the industry with the huge interest not just in tech but across various business domains. ML helps to automate routine tasks and significantly decrease operational costs. And definitely this trend will continue to grow next few years or even more.

As engineers we should at least know the fundamentals of that technology. I mean not just using lots of GenAI tools in daily work but understanding how it works under the hood, its limitations, capabilities and applicability for business and engineering tasks. As of me, I have a significant knowledge gap here, which I plan to close next several months.

I plan to start with the following courses (they are absolutely free):
✏️ Mashing Learning Crash Course from Google that has fresh updates in November
✏️ LLM Course by Cohere

I will use those courses as a base and extend them with additional sources on demand.

So I'm starting my AI learning journey and will share my progress and key takeaways here 💪

ML Introduction

Let's start AI basics with the ML definition and their types.

Definition from Google ML Introduction course:

ML is the process of training a piece of software, called a model, to make useful predictions or generate content from data.

ML Types:
📍 Supervised Learning. The model is trained on lots of data with existing correct answers. It's "supervised" in the sense that a human gives the ML system data with the known correct results. This type is used for regressions and classifications.
📍 Unsupervised Learning. The model makes predictions using data that does not contain any correct answers. A commonly used unsupervised learning model employs a technique called clustering. The difference from classification is that categories are discovered during training and not defined by a human.
📍Reinforcement Learning. The model make predictions by getting rewards or penalties based the on actions performed. The goal is to find the best strategy to get the most rewards. Approach is used to train robots to execute different tasks.
📍Generative AI. The model creates content (text, images, music, etc.) from a user input.  These models learn existing patterns in data with the goal to produce new but similar data.

Each ML type has its own purpose, like making predictions, finding patterns, creating content, or automating routine tasks. Among them, Generative AI is the most popular and well-known today.

#aibasics

ML Basic Terms

To be on the same page with AI-experts we need to build a special vocabulary with basic terms and concepts:
✏️ Feature - input parameter for the model. Usually it represents some characteristic of the entity or facts for which the model makes prediction.
✏️ Label - existing answer for input data. Usually used to train supervised models: predicted value can be compared with labels to check the size of discrepancy.
✏️ Loss - the difference between predicted value and label. For different models different functions to calculate loss is used.
✏️ Learning Rate - a floating-point number that tells the optimization algorithm the step size for the iteration while moving toward a minimum of a loss function. If the learning rate is too low, the model can take a long time to converge. If the learning rate is too high, the model may never converge.

#aibasics

Linear Regression

Linear regression is the simplest supervised ML model that finds relationships between features and labels.

Mathematically it looks like:

 y'=b+w1*x1  + w2*x2 + ... + wn*xn

where
- y' - predicted value
- b - bias (calculated during training)
- wn - weight for a feature (calculated during training)
- xn - feature value (input to the model)

Loss for that type of model is usually calculated as a mean squared error(MSE) or mean absolute error (MAE):
- MSE is sensitive to outliers and adjusts the model toward them.
- MAE minimizes the absolute differences, making it less sensitive to outliers.

Training steps:
1. Calculate the loss with the current weight and bias.
2. Determine the direction to move the weights and bias that reduce loss.
3. Move the weight and bias values a small amount in the direction that reduces loss.
4. Return to step one and repeat the process until the model can't reduce the loss any further.

Example:
The model needs to predict taxi ride prices based on features like distance and ride duration. Past ride prices can be used as labels.

The model formula:

y'=b+w1*distance  + w2*ride_duration

The goal is to find values for b, w1, and w2 that minimize the MSE for the given labels. A well-trained model should converge after limited number of iterations, where the loss cannot be optimized anymore.

Use Cases:
✏️ Predicting Outcomes. Forecast values based on multiple inputs, e.g., taxi fares, apartment rentals, or flight prices.
✏️ Discovering Relationships. Reveal how variables are related and how changes in one variable affect the whole result.
✏️ Processes Optimizations. Optimize processes by understanding the relationships between different factors.

Studying linear regression made me realize why I learned linear algebra and statistics at university 😄. I really had some fun with the math and dynamic examples.

References:
- Google ML Crash Course: Linear Regression
- Understanding Multiple Linear Regression in ML

#aibasics

Minimum Viable Architecture

There is no one-size-fits-all architecture for all scales for all project phases. Architecture should evolve with the product and it should be adopted to the requirements at different stages of product lifecycle.

That's the main idea from Randy Shoup talk - Minimum Viable Architecture. He calls this approach “just enough architecture”- the architecture that's good enough for the product to be released at current project stage.

Product Stages and Their Architecture:

📍 Prototyping.
- Goal: proof business concept, test the market and acquire first customers.
- Rapid iterations, a lot of prototyping.
- Technology doesn't matter, use any tools that get results fast.
- No architecture
- Single team

📍 Starting.
- Goal: solve customer needs as cheap as possible, acquire more customers.
- Rapid learning and iterations.
- Use simple, familiar tech stack
- Typically monolith architecture with a single database
- Rely on cloud infrastructure and open-source tools.
- Focus on competency growth, outsource everything else.
- Number of teams grows.

📍 Scaling.
- Goal: stay ahead of rapidly growing business.
- Time to rearchitect: "Getting to rearchitect a system is a sign of success, not failure."
- Build scalable architecture, focus on latency and performance
- Perform migration from monolith to microservices
- Scale team numbers

📍 Optimizing.
- Goal: make a system more sustainable, efficient and effective.
- Focus on small, incremental improvements.
- No major architectural changes.
- Improve operational efficiency.
- Consolidate the teams

I like the idea of matching architecture to business priorities and not overcomplicating the solution on early stages. The talk also shares some tips when rearchitecturing is really needed and how to do it without breaking the existing solution. Some ideas and recommendations about architecture looks too dogmatic for me, but overall the talk is really good ad I recommend to check the full video.

#architecture

Visualized summary for main stages from the talk Minimum Viable Architecture.

#architecture

Today, I’m starting a topic with a picture first.

That's a D. Caruso Mood Map, a tool widely used in emotional intelligence techniques. The tool maps all our emotions on the grid with two scales:
- One scale is for a level of energy (low to high).
- The other scale is for a level of pleasantness (unpleasant to pleasant).

Explanation of how to use that will be shared in the next post 👇

#softskills #leadership

The Mood Map

Let's check what is interesting in that tool and how it can be used in our daily life and work.

✏️ Emotions Mapping. That's ability to recognize emotions in yourself, your colleagues, and your partners. One of the helpful techniques is mirroring — matching the other person’s speech rate and tone of voice. On the neurophysiology level it means “This person is like me!”, that makes communication more pleasant, gives the sense of safety and increases the chances to reach agreements.

✏️ Task Selection. The Mood Map helps to choose tasks for yourself or your team based on current emotional states. For example, anxiety can sharpen focus, happiness and joy are good for creativity, contentment improves chances to come to a consensus. The key idea is to either pick tasks that match your current state or shift your state to suit the task. This applies to the teams too: "If you have a brainstorming session and the team seems anxious, that’s not a good match. As a leader, you either have to change the tone of the room or change the agenda to match the tone".

✏️ Understanding. What makes you happy might not make someone else happy That's important to understand and learn what motivates and inspires your team members. At the same time, emotions have universal reasons. If you understand the root of someone’s behavior, you can address it effectively.

✏️ Changing Emotions. Agreements are hard to reach if you and the other person are in different quadrants of the Mood Map. Ideally, everyone needs to move to the `green` quadrant to reach a consensus. However, jumping directly from `red` to green is almost impossible. Instead, you can guide someone through smaller transitions, like red -> blue -> green. For example, if someone is in the red quadrant, speaking slowly and calmly can help reduce emotional intensity and shift them toward blue.

I used to be skeptical about emotional intelligence techniques, but that tool looks to be helpful and practical.

Additional trick there: during complex conversations, if emotions are escalating, pause and ask yourself, “What am I feeling right now? Why?”. Reflection helps to shift your brain from the emotional side to the logical side. Once you’re back in a logical state, you can better manage the situation and improve your chances of success.

References:
- David Caruso Youtube Channel
- Can emotional intelligence be learned?
- Emotional Intelligence in a Changing World

#softskills #leadership #communications #productivity

Binary Data Classification

In the previous #aibasics post, I briefly explained the basics of machine learning with Linear Regression. Today let's talk about another type of task - binary data classification. Typical example is determining whether an email is spam or not spam.

Key steps for binary classification:

1. Predict Probability. Take a Logistic Regression model that predict probability (mathematically it returns values between 0 and 1). For example, the probability of an input email being either spam or not spam. If the model predicts 0.72, this means there is a 72% chance the email is spam and 28% chance the email is not spam.

2. Set a Classification Threshold . The classification threshold determines how to assign a binary label (e.g., spam or not spam) based on the predicted probability. For example, the model predicts that a given email has a 75% chance of being spam. Does it mean the email is spam? Actually, no. If the threshold is set at 0.8, then email will be classified as not spam.

3. Evaluate the Model Using a Confusion Matrix. To measure how good our model is, we need to summarize the number of correct and incorrect predictions using confusion matrix:
- True Positive (TP): Correctly predicted positive cases.
- False Negative (FN): Positive cases incorrectly predicted as negative.
- False Positive (FP): Negative cases incorrectly predicted as positive.
- True Negative (TN): Correctly predicted negative cases.

4. Measure Classification Quality. The following metrics are used to define the effectiveness of the result model:
- Accuracy. The proportion of all classifications that were correct, whether positive or negative.
- Recall. The proportion of all actual positives that were classified correctly as positives.
- False Positive Rate. The proportion of all actual negatives that were classified incorrectly as positives.
- Precision. The proportion of all the model's positive classifications that are actually positive

The classification threshold and quality metrics should be adjusted based on the cost of errors for particular domain. If marking important emails as spam is costly, you may increase the threshold to reduce false positives. Conversely, if missing spam emails is more problematic, you may lower the threshold to prioritize catching them.

References:
- Google ML Course: Logistic Regression
- Google ML Course: Classification
- Confusion matrix in machine learning

#aibasics