Structure Eats Strategy

Many are familiar with Conway’s Law, which states that the architecture of a system reflects the communication structure of the organization that created it. If architecture mirrors the organizational structure, then having a solid structure is crucial, right? Jan Bosch explores this idea in his article "Structure Eats Strategy". He posits that many organizations prevent their growth due to ineffective organizational structures, communication, and distribution of responsibilities.

Bosch identifies four fundamental building blocks within any organization:
📍Business (B): Defines business strategy, revenue, and innovations.
📍Architecture (A): Establishes the technology stack and system structure.
📍Processes (P): Outlines activities and ways of working.
📍Organization (O): Defines departments, teams, and responsibilities.

Most organizations operate in an OPAB model (organization-first), meaning that the existing organizational structure dictates processes, leading to an accidental architecture shaped by the company's past.

Bosch argues that the BAPO model (business-first) is significantly more effective. This model begins with defining the business strategy, which shapes the architecture. The architecture then forms the processes and tools, and only after these elements are in place is the organizational structure defined.

The article offers a valuable framework for understanding the relationship between organizational structure and architecture. Structure should be a tool to help the business achieve its goals and guide architecture in the right direction for the future. Unfortunately, there are many examples where existing team boundaries lead to flawed architectural decisions. This results in support overheads, lower maintainability, reinvented wheels, and higher costs for the business.

#architecture #management

Pinterest Ad System Redesign

A recent article on the Pinterest Engineering blog, noscriptd "Redesigning Pinterest’s Ad Serving Systems with Zero Downtime," discusses the complete overhaul of Pinterest's recommendation system.
The ad-serving system is crucial to Pinterest's business, serving as the primary revenue channel. The previous system, known as "Mohawk," was implemented in 2014. However, as the company grew, Mohawk accumulated technical debt and complexity, making incident troubleshooting and bug analysis increasingly challenging.

The following architectural issues were identified as motivations for the reimplementation:
📍High coupling between business and infrastructure logic
📍Lack of modularization and ownership: Code from the same packages and files was owned by different teams.
📍No guarantees of data integrity
📍Unsafe multi-threading: Lack of error handling and potential race conditions.

The new design principles include:
📍Extensibility: Enable the addition of new features and the deprecation of old ones.
📍Separation of Concerns: Organize logic into separate modules, each owned by the appropriate teams.
📍Safety: Ensure the safe use of concurrency and enforce data integrity rules.
📍Development Velocity: Provide well-supported development environments and facilitate easy testing and debugging.

The team spent 2 years implementing the new version of the recommendation system - AdMixer. Rewritten in Java, like other Pinterest services, this new system helps unify the technological stack. It has been in production for three quarters now without significant outages. Additionally, it was designed to run on AWS Graviton instances (ARM architecture) to reduce AWS usage costs (more about that in Multi-Arch Images post ).

What I can say is that a complete system rewrite is a very expensive endeavor. It would be interesting to know the costs of this two-year project, as not all companies can afford such an exercise. An eight-year lifespan is not particularly long for a software system, indicating that serious architectural mistakes and unresolved technical debt prevented the previous system from evolving effectively.

Let's be honest: engineers often prefer to rewrite a system, especially one created by others😀. It’s fun, cool, and easier, and it makes a nice addition to a resume. However, it is far more challenging to build a system with the right architectural principles that support evolutionary rather than revolutionary development (do not confuse it with over-engineering!).

In any case, the Pinterest engineering team did a great job: they implemented a new approach, successfully delivered it to production, accelerated business features delivery, and increased developers satisfaction. I hope the new service incorporates lessons learned and it will be extensible for future growth and business needs without requiring a full rewrite.

#architecture #usecase #refactoring

Modern Software Engineering

Finally, I’ve finished reading Modern Software Engineering by D. Farley 📚. It took me longer than expected, as I spent a lot of time on the first two parts of the book, which focus more on general concepts rather than practical advice, making the initial experience a bit tedious for me.

Let's dive into the book to see what it's about. It is divided into four major sections:

Part I: What is Software Engineering?
The author provides a definition of software engineering that I find really great:

Software engineering is the application of an empirical, scientific approach to finding efficient, economic solutions to practical problems in software.

Each word in this definition is significant. Reflecting on them, the author concludes that we must manage the complexity of the systems we create and continuously learn new things and adapt to them.
Engineering is not just coding—it includes all aspects needed to create software: processes, tools, and culture. We should apply engineering practices everywhere to make effective decisions. And we need metrics to measure the effectiveness of those decisions. The author suggests the following metrics:
📍Stability: change failure rate, recovery time
📍Throughput: lead time (from idea to production), deployment frequency

Part II: Optimize for Learning
To effectively learn and adapt to changes, we need to:
Working iteratively
📍Employing fast, high-quality feedback
📍Working incrementally
📍Being experimental
📍Being empirical
As you can see, these principles align closely with modern agile practices.

Part III: Optimize to Manage Complexity
The fundamental principles for managing complexity include:
📍Modularity
📍Cohesion
📍Separation of concerns
📍Abstraction
📍Loose coupling
While these concepts are familiar to most programmers, the book provides a valuable review with practical examples and techniques for better understanding and application.

Part IV: Tools to Support Engineering in Software
To apply an engineering approach in practice, the following tools are suggested:
📍Testability
📍Deployability
📍Speed
📍Controlling variables
📍Continuous delivery
The author discusses specific tools and how they relate to fundamental principles. For example, achieving testability requires good abstraction, modularity, and separation of concerns.

For me, the book doesn't introduce new concepts but rather reinforces my current understanding of how we should engineer our systems. To conclude this review, I’d like to share a quote from the book that serves as a good rule for daily work:

If our ‘software engineering’ practices don’t allow us to build better software faster, then they aren't really engineering

#architecture #engineering #booknook

And now there is a more entertaining part of the book review: a sketchnote🔥 to help simplify and remember what the book is about.

#architecture #engineering #booknook #sketchnote

The Cornell Note Taking System

When I study, I like to take notes and write summaries about what I’ve learned. This helps me rethink what I’ve read and provides a short denoscription of useful ideas that I can revisit from time to time.

Recently, I discovered the Cornell Note-Taking System. It was developed by Walter Pauk, a professor at Cornell University, in the 1950s. The system was initially designed to improve the efficiency of college-level students (I wish someone had shown it to me when I was a student!).

The Cornell Note-Taking System is very simple and consists of three parts: notes, keywords/questions, and summary. Here’s how it works:

📍Divide the paper into two columns: the questions column (1/3 of the width of the page) and the notes column (2/3 of the width of the page). Leave 5-7 lines at the bottom of the page for the summary.
📍During study, make notes in the notes section. It’s recommended to keep one line between different statements and two lines between different chapters. This is how you should work during lectures and readings.
📍Add key terms and relevant questions to the questions column. This helps emphasize important points.
📍Write a summary in your own words to describe what you’ve learned. A good tip is to imagine you’re explaining it to one of your friends.

Various studies show that the Cornell Note-Taking System is more effective than other styles when students need to apply learned knowledge in practice. I’ve tested this approach and can confirm that it works much better than standard note-taking. It requires to rework the material, which improves the level of retention and understanding. Now it’s my primary method for taking notes.

#softskills #productivity

Practical Architecture

I highly recommend "Practical (a.k.a. Actually Useful) Architecture" video by Stefan Tilkov where the author presents 10 recommendations for pragmatic architectural work:

✏️Choose your perspective(s) consciously. The author identifies 3 main architecture perspectives:
- Business domain: How the system is divided into logical modules (boxes).
- Integration: How different modules communicate with each other.
- Technology: The technology choices within the boxes.
These perspectives are usually maintained by different people, changed with different frequencies and for different reasons. It's crucial to focus on the right perspective so that decisions at one level don't negatively impact another. For example, a framework usage shouldn't affect integration contracts to prevent dependencies on the wrong level for the wrong reason.
✏️Explicitly architect your team setup. Define team topology based on the domain modules.
✏️Match your organizational setup to project size:
- Single team. Some tasks are architectural with no explicit architecture roles.
- Small number of collaborating teams. Hold an architecture committee meeting once a week with members from each team.
- Large number of independent teams. Establish explicit architecture roles, possibly forming a separate architecture team.
✏️Don't be afraid to decide things centrally. Some decisions need to be made centrally:
- What is centralized vs. what is left to individual teams.
- Team responsibilities.
- Any aspects relevant to more than one team.
- Global policies and strategic aspects.
✏️Pick your battles wisely. Avoid trying to address everything simultaneously. Focus on the most important aspect first and plan other aspects for later.
✏️Enforce the least viable amount of rules, rigidly. Provide teams with the autonomy they need to get things done, but define clear boundaries to avoid chaos. For example, while the choice of programming language can be a team decision, API standards should be centralized. Define a very few critical rules but enforce them strictly.
✏️Balance prenoscriptive vs. denoscriptive architecture. The goal of architectural work is to produce decisions, not just documentation. Good decisions might not satisfy everyone, but decisions that do are often bad. Make firm decisions and encourage others to follow them.
✏️Don't aim for perfection - iterate. Architecture is not about creating a snapshot in time but about managing a series of changes. Build systems so they can evolve over time and continuously re-evaluate past decisions.
✏️Architect for delivery flow as much as for runtime quality. Design both the runtime environment and the development pipeline. The quality of a system's architecture is proportional to the number of bottlenecks limiting its evolution, development, and operations. More bottlenecks mean a worse architecture.
✏️Be boring & do more with less. Prefer simple, straightforward solutions to overly clever and complicated cool approaches.

The video offers excellent advice and practical recommendations, with nearly every idea and statement being truly significant. I'll conclude the review with a quote that I really like:

If you get architecture wrong, everything breaks down - yet even if you get it right, success is not guaranteed

#architecture

Architecture Without an End State

`Architecture Without an End State` is a concept introduced by M. Nygard. It describes architecture as a mix of past decisions, the current state, and future vision. This means transitioning a system from one architecture to another isn't possible; instead, iterative steps must be taken toward current business goals. But business goals and requirements may change over time, and this series of changes is endless.

Nygard suggests 8 rules for designing systems that adapt to change:

✏️Embrace Plurality: Avoid creating a single source of truth. Each system should have its own entity model with its own related data. Different representations are connected through a system of global identifiers (URNs and URIs). Assume an open world that can be extended anytime: new entities will be added, new integrations will appear, and new consumers will start using the system.
✏️Contextualize Downstream: Upstream changes can affect all downstream systems. Minimize the impact by localizing the context (since business rules have different interpretations and requirements across business units) and reducing the number of entities that all systems need to know about.
✏️Beware Grandiosity: Avoid enterprise modeling attempts and global object model creation, as they are often too complex to be practical. Instead, narrow the scope and build a model for a particular problem or area. Start small and make incremental changes based on feedback.
✏️Decentralize: This allows to make local optimization of resources and budgets.
✏️Isolate Failure Domains: Use modularity to define proper system boundaries and isolate failures. Resilient architecture admits componentwise changes
✏️Data Outlives Applications: Data lives longer than technology and specific applications. Use a hexagonal architecture (ports & adapters) to separate business logic from data storage.
✏️Applications Outlive Integrations: Integrations with other systems will change over time. Build a hexagonal architecture so that all systems are connected via explicit boundaries, making communication a boundary problem rather than a domain layer problem.
✏️Increase Discoverability of Information: Poor information flow within an organization leads to duplicate system functions and reinventing the wheel between different teams. Improving information discovery reduces costs of development (fewer duplicates). Some ideas include internal blogs, open code repositories, and modern search engines.

The only true end state for a system is when the company shuts down and all services are turned off. Main recommendation from the author is to stop chasing the end state; it's not achievable. Continuous adaptation is the most effective way to build a system, and these rules can help guide that process.

#architecture #systemdesign

Creative Thinking

According to the World Economic Forum Survey creative thinking is the most demanded skill in 2024. But is creativity important in software development?

Definitely yes. Robert E. Franken, in his book “Human Motivation,” defines creativity as

the tendency to generate or recognize ideas, alternatives, or possibilities that may be useful in solving problems, communicating with others, and entertaining ourselves and others

In our work, we frequently generate ideas to meet business requirements and solve technical issues (and occasionally to entertain others too! 😄). The more creative we are, the more ideas we can produce that lead to better and more efficient solutions.

The human brain generates new ideas based on existing knowledge, connecting it in innovative ways. This means the foundation for creativity is so-called `visual experience`: the more different elements we have in our minds, the more ideas we can produce.

In software development, it's essential to continually learn new things, even if they don't directly relate to our current technology stack or tasks. While it may seem pointless to study something unrelated to our immediate work because we might forget it quickly, this isn't entirely true. New knowledge serves as building blocks for generating new ideas. Designers even have a practice to study the works of others to develop a sense of good quality before creating their own style. Similarly, in software development, studying various use cases and technologies broadens our mind and enhances our visual experience.

Like any other skill, creativity can be trained. Many exercises to train creativity can be found on the Internet. Here are a few examples to start with:
✏️Study real use cases and patterns from your area of professional interest to expand your visual experience and sense of quality.
✏️Extend your standard subnoscriptions with content unrelated to your primary interests, such as science, drawing, medicine, etc. This helps introduce fresh ideas from other fields.
✏️ Select a simple object like a ruler, pen, or paper clip, and spend 10 minutes brainstorming as many alternative uses for it as possible. Try this exercise daily for a month and track whether the number of generated ideas increases.

To summarize, creativity is a valuable skill that need to be trained to enhance problem-solving abilities and boost professional efficiency.

#softskills #creativity

UUID vs ULID

UUID (Universally Unique Identifier) and ULID (Universally Unique Lexicographically Sortable Identifier) are unique identifier formats widely used in software development. While both are based on a 128-bit length and random generation, they differ fundamentally in structure and use cases.

UUID Key Implementation Details:
- Represented as a 32-character hexadecimal string. Example: 8ba8b814-9dad-11d1-80b4-00c04fd430c8
- Typically generated using the current time and a random number, but may include other input parameters
- Has 8 versions with different generation logic (e.g., v1 uses a MAC address and timestamp, while v4 uses only random or pseudorandom numbers), v4 is the most popular now
- Has an official RFC

ULID Key Implementation Details:
- Represented as a 26-character string. Example: 01ARZ3NDEKTSV4RRFFQ69G5FAV
- Generated based on the current time and a random number
- Common structure: 48 bits timestamp, 80 bits random number
- Lexicographically sortable, meaning they can be sorted in alphabetical order
- No official standard, but there is a spec on GitHub

Key Differences
✏️ Length: ULID is shorter.
✏️ Generation Speed: Some sources claim ULID is generated faster, but this depends on the implementation and technology used.
✏️ URL Safety: ULID is URL-safe without any encoding or escaping.
✏️ Sortability: ULID is sortable, which is useful for quickly sorting and searching large numbers of identifiers, it can be really efficient for data partitioning cases.
✏️ Security: UUID is generally safer from a security perspective as it is less predictable.
✏️ Adoption: UUID is widely adopted and can be generated at various levels (database, application).

Ultimately, the best choice depends on specific project requirements. Consider the importance of uniqueness, sortability, performance, and string length to make a final decision.

#architecture #systemdesign

Snowflake ID

One more popular algorithm for generating IDs in distributed systems is Snowflake.
It was initially created by Twitter (X) to generate IDs for tweets.

In 2010, Twitter migrated their infrastructure from MySQL to Cassandra. Since Cassandra doesn’t have an in-built id generator, the team needed an approach for ID generation that met the following requirements:
- Generate tens of thousands of ids per second in a highly available manner
- IDs need to be roughly sortable by time, the accuracy of the sort should be about 1 second (tweets within the same second are not sorted)
- ID have to be compact and fit into 64 bits

As a solution Snowflake service was introduced that generates IDs with the following structure:
✔️ Sign bit: The first bit is always 0 to keep the ID a positive integer.
✔️ Timestamp (41 bits): Time when ID was generated
✔️ Node ID (10 bits): Unique identifier of the worker node generating the ID
✔️Step (12 bits): A counter that is incremented for each ID generated within the same timestamp

Snowflake IDs are sortable by time, because they are based on the time they were created. Additionally, the creation time can be extracted from the ID itself. This can be used to get objects that were created before or after a particular date.

There is no official standard for Snowflake approach, but there are several implementations available on Github. The approach has also been adopted by major companies like Discord, Instagram and Mastodon.

#architecture #systemdesign

GeoSharding in Tinder

Let’s continue improving our visual experience in system design. Today we’ll check the Tinder search architecture. I hope everyone knows what Tinder is. One of its main components is the search with real-time recommendations. Initial implementation was based on a single Elasticsearch cluster with only 5 shards. Over time the number of shards grew and more replicas were added until the system met its scaling limits. In 2019 this led to a decision to re-architect the component to satisfy new performance requirements.

Main Challenges:
📍 Location-Based Search with a maximum distance of 100 miles. For example, when serving a user in California, there is no need to include the users in London.
📍 Performance: index size growth decreases performance linearly. Multiple smaller indexes demonstrated better performance results.

Decisions Made:
✏️ Split Data: Storing users who are physically near each other in the same geoshard (a Tinder-specific term for their sharding implementation).
✏️ Limit Numbers of Geoshards: 40–100 geoshards globally results a good balance of P50, P90, and P99 performance under average production load.
✏️ Use the Google S2Geometry library to work with geo data:
- The library is based on the Hilbert curve: two points that are close on the Hilbert curve are close in physical space.
- It allows hierarchical decomposition of the sphere into "cells", each cell can also be decomposed on smaller cells.
- Each smallest cell represents a small area of the earth.
- The library provides built-in functionality for location mapping.
- S2 supports different-sized cells, ranging from square centimeters to miles.
✏️ Balance Geoshards: Not all locations have the same population density. So it’s needed to define the proper size of the shard and balance data to avoid a hot-shard issue. S2 cells were scored and combined to the geoshards, as a result each geoshard can have a different number of cells of the same size.
✏️ Mapping: Create a mapping between geoshards and S2 cells. For queries, the data service gets S2 cells to cover the query circle using S2 library, then map all the S2 cells to geoshards using the shard mapping.

It’s reported that the new approach improved performance by 20 times compared to the previous implementation with a single index setup. More importantly, it has the capacity to scale more in the future by extending the number of the geoshards. For me it was really interesting to read about location-based search approach, S2Geometry library and concepts under its implementation.

#architecture #systemdesign #scaling #usecase

Behavior Hormones

When I reviewed `Leaders Eat Last` I highlighted that chemical elements like endorphin, dopamine, serotonin, and oxytocin can drive human behavior, so it’s good to know how they work. So let’s go into more details.

Endorphin:
- Biological painkiller that masks real pain to continue moving forward
- Produced during physical activities such as exercising, running, laughing
- Decreases stress level (it’s not possible to feel fear and laughing at the same time)
- Has short-time effect
- Practical Usage: make exercises to switch focus from the problem; use humor to defuse tense situations

Dopamine:
- Satisfaction from achievements
- Generated on getting things done
- Helps to focus on the task and end goal: the closer and clearer the goal, the more dopamine is released, increasing motivation
- Has short-time effect: intermediate doses are constantly needed to keep focus and motivation
- Can have negative effects as it causes different types of addictions: social media, KPIs, shopping, etc. Each event generates dopamine, leading to repetitive behaviors
- Practical Usage: set clear goals (for own tasks or for the team), plan intermediate steps, visualize the progress

Serotonin:
- Feeling of pride, respect, recognition
- Released when personal achievements are recognized by a group
- Improves self-confidence and responsibility towards the group
- Typically a person who supports and assists others is generally accepted as a leader by the group
- Has a long-time effect
- Practical Usage: recognize team members’ achievements, offer help and support when needed

Oxytocin:
- Friendship, trust, love. It’s everything that makes us social
- Produced when an individual is accepted as part of the group
- Being part of a group provides a sense of safety, calm, and happiness, reducing stress levels. Physical contacts (handshake, hugs, etc.) can also support those feelings
- Has a long-time effect
- Practical Usage: build an atmosphere of trust within your group, organize team buildings, invent something that indicates group membership

Endorphin, dopamine, serotonin, and oxytocin have significant impact on human behavior. Understanding basic principles of their work can help to improve team cohesion, motivation, collaboration and most importantly establish trust within the team.

#softskills #leadership

Our behavior drivers is also a good topic to visulize. Check it out!

#softskills #leadership #sketchnote

Distroless Images

The concept of distroless images was initially presented at the swampUP conference in 2017.

So what does it mean?

"Distroless" images contain only your application and its runtime dependencies. They do not contain package managers, shells or any other programs you would expect to find in a standard Linux distribution.

That’s how Google deploys software in production. But what problem does it solve? Why is a small distro image like alpine not enough?

So let’s start with what application actually needs to run:
- Compiled sources
- Dependencies
- Language runtime

It doesn’t need any package manager, shell utilities or other tools from OS distribution. But their existence increases image size and image download time, extends scope of compliance (security hardening, CVE scans). That's the problem Google tried to solve.

Initially Google used alpine as the smallest available distro. But Alpine contains an unnecessary package manager, BusyBox and it is based on musl libc that makes glibc usage mostly impossible. So Google decided to create images that contain only what is really needed. That’s how distroless was created.

Distroless images are based on Debian Linux distribution and have a variety of language support: go, python, java, cpp, nodejs. The smallest distroless image, gcr.io/distroless/static-debian11, is around 2 MiB. That's about 50% of the size of alpine (~5 MiB), and less than 2% of the size of debian (124 MiB).

Since March 2023, Distroless images are based on oci manifests and support multiple architectures (more about multi-arch images in the post ).

We have been using distroless for some time already, and the experience is really positive. Of course, in some cases I regret about bash absence in runtime 😃, but it drives us to improve other debug and observability tools. Additionally, I want to highlight that distros are actively supported by Google and they have regular updates including actual security patches.

#engineering

Hexagonal Architecture

The hexagonal architecture (also known as Ports & Adapters) is a quite popular concept pattern in the modern software world. Let’s check what it is and when it can be helpful.

The pattern was originally introduced by Alistair Cockburn in 2005 in his blog post Hexagonal Architecture and intended to solve the following problems:
- Undesired dependencies between system layers
- Mixed business logic with external interactions logic
- Poor business logic testability by automated integration tests

Key points of the suggested solution:
✏️ Split system representation on what is “inside” application and what is “outside”.“Inside” logic should not leak to “outside” part
✏️ Introduce “ports” - communication channels with outside entities better known as boundaries of application: contracts and interfaces
✏️ Introduce “adapters” - entities that transform one interface into another (web UI, mobile UI, databases, queue system integrations, etc.). Adapters are the components that allow the application to interact with specific technologies.
✏️ Each port can have multiple adapters
✏️ Adapters can be primary and secondary.
✏️ Primary adapters (also called “driving”) are responsible to receive external signals and convert them to a Port, such adapters initiate communication flow. Examples: different types of UI, direct calls to REST APIs, CLIs, etc.
✏️ Secondary adapters (also called “driven”) convert commands from Ports to a specific technology. Examples: read\write to database, message queue integration, storing files on s3 storage, etc.
✏️ Outside entities can interact with application only via defined ports

Usage examples: different OS support via separate adapters, integration with different types of storages like filesystem, s3, azure blob storage and others, separation of application logic and UI development, proper contracts between systems that are developed by different teams.

Pros:
- Testability improvements: it’s possible to test application logic without external dependencies
- Increased maintainability: separation of concerns and business logic decoupling make it easier to locate the code to be modified
- Flexibility: adapters can be easily swapped without any impact on core logic
- Adaptation to technology evolution: Technologies are changed more frequently than business logic. As the technology part is encapsulated in a specific adapter, it’s enough to change the adapter only.
- Put-off technologies decisions: focus on business logic, postpone decision about particular technologies for real production usage.

Cons:
- Complexity: application has a complex structure with a lot of modules and explicit dependencies defined between them
- Indirection: extra calls to methods when an adapter converts between port and specific technology interfaces
- In some contexts may be not applicable or inefficient: approach works well for cases with complex and stable business logic.

So the main idea of the pattern is to design a software around domain logic and isolate it from external factors and constantly changing technologies.

#architecture #systemdesign #patterns