Shopify’s Modular Monolith

In the age of microservices, exploring real-world examples of alternative architectures is really interesting. Today, we'll check Shopify's architecture through an interview with one of its principal engineers.

Why is it interesting? Shopify employs a modular monolith architecture, and their system seamlessly managed peaks of 60 million requests per minute on Black Friday

Key points from the interview:
- Shopify's architecture is based on Ruby on Rails, MySQL, Kafka, Elasticsearch, and Memcached\Redis
- Some parts of the system migrated to Rust for better performance
- Certain applications migrated to Vitess for better horizontal data sharding
- Applications are operated within Kubernetes on the Google Cloud platform
- The company actively contributes to open source projects used internally to improve their performance and scaling capabilities
- All shops on the platform are grouped within dedicated sets of database servers to minimize blast radius (the same pattern we saw in Netflix StatefulSet reliability approach)
- The majority of user-facing functionality is served by Shopify Core, a monolith divided into multiple modules focused on different business domains
- Shopify Core can be scaled horizontally so there are no plans to split it on separate services
- New features are rolled out to production using a canary approach

#architecture #scalability #usecase

Saga Design Pattern

Distributed systems are complex, handling transactions in distributed systems are even more complex. One way to address the issue is by using the saga pattern. A common use case for it is to manage single transaction over multiple services with their own databases.

Implementation logic:
- Define local transaction as atomic work performed by a service
- Organize local transaction into a sequence - saga
- After local transaction completion, publish a message or event to trigger the next local transaction
- In case of failure, execute a series of compensating transactions that undo the changes that were made by all previously executed local transactions
- Compensations must be idempotent because they might be called more than once within multiple retry attempts

Saga coordination options:
- Choreography. It's event-based approach where each local transaction publishes events that trigger local transactions in other services. Requires a mature event-driven architecture.
- Orchestration. This approached requires central orchestrator, that tells the services what local transactions to execute or rollback.

Benefits:
- It allows to implement non-blocking long-running transactions
- Local transactions are fully independent
- Enforce separation of concerns as participants may not know about each other

Drawbacks:
- Eventual data consistency
- Difficult to troubleshoot when number of participants grow up
- Design and implementation are complex and expensive (need to implement common logic and compensation logic for all steps in the sequence)

From my perspective, the pattern is too complex, and as we know complex logic tend to bring complex issues. So if you can avoid distributed transactions, please, avoid them.

References:
- Sagas
- Saga Pattern
- Data Consistency in Microservices Architecture

#architecture #systemdesign #patterns

Draw to Win

Did you know that 2/3 of our brain activity is occupied by processing visual information? Most of the information about the world is received from our eyes. Visualization is the most powerful way to communicate. So what? We as a leaders can use it to educate and persuade people, to share our vision and sell our ideas.

But how to do that in practice? That's what the Dan Roam book Draw to Win: a Crash Course on How Lead, Sell, and Innovate with Your Visual Mind is about.

Why drawing is important:
- Drawing is the oldest 'technology' in the world
- 90% of all information is visual
- Visualization attracts attention and improves clarity in communications
- Visual information is memorable

Knowing that, we can significantly enhance our presentation skills. No special knowledge is required to start drawing: we can easily draw lines, arrows, shapes and smiles. That's all you need to start drawing to explain or sell your ideas. The accuracy of result pictures is not important.

The author explains that our brain processes visual information to answer the following questions: who? what? how many? when? where? and why?
Organize your ideas into a visual story that provides that answers and you will need only 6 pictures or slides to explain everything. The book even includes an example of how to explain a salary increase to a manager using this technique, but I won’t spoil it—better to check out the original for the full story 😉!

Additionally the book contains practical tips to get started with drawing and use it to improve creative thinking. One that I really like: if you don't know what to draw, start with the circle, name it, continue adding circles until your idea takes shape.

To sum up, visualization is a powerful tool that can be used to manage, educate, sell, share, collaborate and innovate. I really enjoy the book, it's full of interesting facts and practical advice. There are more books by the same author on this topic, and I’ll definitely add them to my reading list!

#booknook #softskills #presentationskills

In one of the previous blog posts we broke down the Saga pattern and I recommended not to use it because of high complexity. However, it's really interesting to explore successful implementations of the pattern. Let’s take a look at how HALO scaled to 11.6 million users using the Saga design pattern.

HALO is a very popular shooting game that was initially introduced in 1999. At that time the game was based on a single SQL database to store the entire game data. Their growth was explosive, and single database became not enough soon.

So they set up a NoSQL database and partitioned it. Data for each player was kept in a dedicated database partition. It resolves scaling limitations, but brought new issues:
- Data writes are not atomic anymore
- Partitions may have non-consistent information
It means that players can have different game data that significantly impacts game experience.

So HALO team decided to set up Saga:
✏️ Each partition is changed within a local transaction only
✏️ Orchestrator manages update within all database partitions
✏️State of each local transaction is stored in durable distributed log that allows:
- Track if a sub-transaction failed
- Find compensating transactions that must be executed
- Track the state of compensating transactions
- Recover from failures
✏️The log is stored outside Orchestrator that makes it stateless
✏️ Orchestrator interacts with the log to identify local transaction or compensating actions to execute

The introduced technical solution enables further HALO usage growth, which still remains a popular game series for Xbox with millions of unique users.

#architecture #systemdesign #usecase

Elastic: Back to Open Source?

This week, I came across the surprising news that Elastic has decided to return Elasticsearch and Kibana to open source.

Let me remind you that 3 years ago Elastic changed their license from Apache 2.0 to semi-proprietary Server Side Public License. Teams who actively used ELK stack remember that. In response, AWS forked the latest open Elasticsearch and Kibana versions, creating OpenSearch project.

In a year, OpenSearch had 100 million downloads and gathered 8,760 pull requests from 496 contributors over the globe. It even launched its own OpenSearch Conference in 2023. The fork became extremely popular and successful.

Now, Elastic announced AGPLv3 license for Elasticsearch and Kibana products. Maybe it relates to the decrease of the interest in Elasticsearch as a product. There is also a good article on TheNewStack that makes an attempt to explain and understand the reasons of that unexpected decision, which I recommend to read if you're interested in the topic.

The main question is whether teams already using OpenSearch will switch back to Elasticsearch. I don't think so. That's easy to change the license, but it's harder to return community trust.

#news #technologies

Cassandra 5.0 is Officially Released

On September 5, the Cassandra 5 GA release was announced. Why is it important? First, Cassandra doesn't get updated very often; the last major release was in 2021. Second, the end-of-support for the 3.x series was announced at the same time. So, if you're still using 3.x, it's time to start planning an upgrade at least to 4.x.

Key changes:
- Storage Attached Indexes (SAI) (CEP-7). This is a new index implementation that replaces Cassandra secondary indexes, fixing their limitations. It allows creating indexes for multiple columns on the same table, improves query performance, reduces index storage overhead, and support complex queries (like numeric range, boolean queries).
- Trie Memtables and Trie SSTables (CEP-19, CEP-25). This is a change of the underlying data structures for the in-memory memtables and on-disk SSTables. These storage formats utilize tries and byte-comparable representations of database keys to improve Cassandra’s performance for reads and modification operations.
- Migration to JDK 17
- Unified Compaction Strategy (UCS) (CEP-26). It combines the tiered and levelled compaction strategies into a single algorithm. UCS has been designed to maximize the speed of compactions, using a unique sharding mechanism that compacts partitioned data in parallel.
- New Aggregation and Math Functions. Cassandra 5 adds new native CQL functions like count, max, min, sum, avg, exp, log, round and others. Users can also create their own custom functions.
- Approximate Nearest Neighbor Vector Search (CEP-30). The feature uses SAI and a new Vector CQL type. Vector is an array of floating-point numbers that show how similar specific objects or entities are to one another. It is a powerful technique for finding relevant content within large document collections and it can be used as a data-layer technology for AI/ML projects.

New Cassandra release makes significant optimizations in existing functionality and brings some really promising new features. For full details, you can check out the release notes here.

#news #technologies

Manage Your Day

Career growth always means taking more responsibilities within the team or company. The more responsibilities you have, the more tasks you need to manage. At some point of time, you may feel like a squirrel on a wheel—constantly responding to incoming requests, issues, one after another, with no time for actual work.

Do not let external requests manage your work. Manage them yourself. This sounds simple, but everyone who has been in this situation knows it's not so easy to do in practice.

Simple tips that might help:
✏️ You don't have to respond immediately to every question you receive. In most cases, nothing bad will happen if you check your messenger or email once in 2-3 hours.
✏️ You don't have to go to every meeting you're invited to. Review invitations carefully and decide which ones are really important. It’s ok to decline or ask to reschedule.
✏️ You don't need to execute any task immediately as you receive it. Ask about priorities and deadlines, estimate impact on other tasks, discuss and plan accordingly.
✏️ If a task takes less than 2 minutes, just do it (That's the only principle from Getting Things Done that really works for me).
✏️ Book time on the calendar to work on important tasks. Try to reserve at least a few hours a day for focused work.
✏️ Set task priorities. I like Covey model, that groups all task by importance and urgency. Choose an approach that works for you.
✏️ Do not try to keep everything in the head: write down all important ideas, tasks, agreements, requests, whatever is important to perform your job.

Additionally I recommend to read Time Management Techniques That Actually Work. The article also contains a bunch of useful recommendations for the same. Try different tools and methods, and see what works for you.

Also, please, feel free to share other recommendations that work for you in the comments.

#softskills #productivity

Canonical Logs

Logging is the oldest tool to troubleshoot issues with the software. But relevant information is spread across many individual log lines, making it difficult or even impossible to quickly search right details, perform some aggregation or analysis. That's where canonical logs concept can help.

Canonical log is the structured one long log line at the end of the request (or any other type of work) that includes fields with request’s key characteristics. Having that data collocated in single information-dense lines makes queries and aggregations over it faster to write, and faster to run.

Canonical log can include the following information:
- HTTP verb, path, response code and status
- Authentication related information
- Request ID, Trace ID
- Error ID and error message
- Service info: name, version, revision
- Timing information: operation duration, percentiles, time spent in database queries and others
- Remaining and total rate limits
- Any other useful information for your service

I want to highlight that log must be structured (key-value, json) to make it machine readable. Structured logs can be easily indexed by many of existing tools, providing an ability to search and aggregate collected data.

Simple canonical log sample:

[2019-03-18 22:48:32.999] canonical-log-line alloc_count=9123 auth_type=api_key database_queries=34 duration=0.009 http_method=POST http_path=/v1/charges http_status=200 key_id=mk_123 permissions_used=account_write rate_allowed=true rate_quota=100 rate_remaining=99 request_id=req_123 team=acquiring user_id=usr_123

Good practice is to formalize log contract across services and applications. As an example protobuf structure can be used for that purposes.

Canonical logs seems to be a lightweight, flexible, and technology-agnostic technique to improve overall system observability. It's easy to implement and extend any existing logging capabilities.

References:
- Using Canonical Log Lines for Online Visibility
- Fast and flexible observability with canonical log lines
- Logs Unchained: Exploring the benefits of Canonical Logs

#engineering #observability

True Inspiration from Pixar. Part 1: Core Values

One of the recent books that really inspired me is Creativity, Inc.: Overcoming the Unseen Forces That Stand in the Way of True Inspiration by Ed Catmull, co-founder of Pixar Animation. It’s presented as a book about creative leadership and management principles that helped Pixar to build unique culture and become the best animation studio in the world. But it's much more than that.

What makes this book special for me? It's not just practical advice, but the story of a long way for a dream came true.

Book starts with Pixar’s history. Did you know that Pixar originally began with creating graphic design programs and selling special computers for them? From a young age, Ed Catmull dreamed about making computer-animated feature films (Disney films were all hand-drawn). But no one believed in his idea, so Pixar was focused on improving how computers processed and displayed graphic data.

Things changed in 1977 with the release of Star Wars by George Lucas. The era of computer effects started. For a few years, Pixar was a part of Lucasfilm, but the dream of making a fully animated movie wasn’t achieved.

In 1986, Steve Jobs bought Pixar, and the team finally had the chance to work on their first film. Toy Story in 1995 became a Pixar triumph. It was first fully computer-animated film in the world, it was extremely successful and it really changed the animation industry. But it was only the beginning. It took a lot of efforts to create next movies with the same quality, scale company culture, support creativity, keep company core values with its growth. Today, Pixar is part of Disney but operates as a separate division with its own vision, ideas, and projects.

Lessons learnt from the growth:
✏️ Building the right team is the foundation for success.
✏️ Focus on teamwork, not on individual talents.
✏️ Focus on the people, their habits, values, help them to reveal their talents.
✏️ People are more important than ideas because people create ideas.
✏️ Quality must be a set as a main condition before project start.

To be continued....

#booknook #softskills #leadership

GenAI for Legacy Systems Modernization

While most people actively write about using GenAI tools to generate new code, there is a new Thoughtworks publication that focuses on the opposite — using AI to understand and refactor legacy systems.

What makes legacy systems modernization expensive?
- Lack of design and implementation details knowledge
- Lack of actual documentation
- Lack of automated tests
- Absence of human experts
- Difficulty to measure the impact of the change

To address these challenges Thoughtworks team developed a tool called CodeConcise. But the authors highlighted that you don't need exactly this tool, the approach and ideas can be used as a reference to implement your own solution.

Key concepts:
✏️ Treat code as data
✏️ Build Abstract Syntax Trees (ASTs) to identify entities and relationships in the code
✏️ Store these ASTs in graph database (neo4j)
✏️ Use a comprehension pipeline that traverses the graph using multiple algorithms, such as Depth-first Search with backtracking in post-order traversal, to enrich the graph with LLM-generated explanations at various depths (e.g. methods, classes, packages)
✏️ Integrate the enriched graph with a frontend application that implements Retrieval-Augmented Generation (RAG) approach
✏️ The RAG retrieval component pulls nodes relevant to the user’s prompt, while the LLM further traverses the graph to gather more information from their neighboring nodes to provide the LLM-generated explanations at various levels of abstraction
✏️ The same enrichment pipeline can be used to generate documentation for the existing system

For now the tool was tested with several clients to generate explanations for low-level legacy code. The next goal is to improve the model to provide answers at the higher level of abstraction, keeping in mind that it might not be directly possible by examining the code alone.

The work looks promising and could significantly reduce the time and cost of modernizing old systems (especially written on exotic languages like COBOL). It simplifies reverse-engineering and helps generate knowledge about the current system. The authors also promised to share results on improving the current model and provide more real life examples for the tool usage.

#news #engineering #ai

True Inspiration from Pixar. Part 2: Protect New Ideas

That's the second part of Creativity, Inc.: Overcoming the Unseen Forces That Stand in the Way of True Inspiration book overview (First part is there). It is about practices that can help to protect new ideas from bureaucracy, fear of feedback, and thinking "experienced people know best."

To support creativity and innovation, Pixar's leaders built a culture based on the following principles:

✏️ Candor. Ask people about candor not honesty, build processes that demonstrate value of the candor on all levels. Pixar uses Braintrust practice to train that:
- Regular offline meetings every 2-3 months.
- Teams present film fragments to identify issues.
- There is no gradation by official noscripts, everyone's opinion is important, there is no stupid or destructive feedback.
- Feedback should be focused on the problem not the person.
- Criticism is part of improving the work, not as competition.
- Create an atmosphere of trust where all member are interested in great results.

✏️ Value of Failures. Don’t fear the failure. Failure is a chance to learn. Start working, get feedback, learn, and try again. The author says that Pixar culture is unique as it doesn't just allow the people to make mistakes, but it expects them to make those mistakes.

✏️ Protect Ugly Babies. New ideas are usually not beautiful, so author called them the ugly babies. It takes time and patience for them to grow and shine. All Pixar films started as simple, sometimes awkward ideas, going through many iterations. It's important to protect new ideas from conservatism, habits to do something by only known ways, because it leads to predictable but mediocre results, killing true inspiration.

✏️ Change and Randomness. People don't like changes because it feels unsafe or overwhelming. Asking "What if?" question helps teams imagine possibilities and break through the fear barriers. One more important note there is about rules. Rules appear by some reason, but reasons are changed over time and can be not actual anymore. Outdated rules improves bureaucracy and kill creative atmosphere.

✏️ The Hidden. True leaders accept that employees often have a deeper understanding of problems. Managers don’t need to know everything, but they must encourage open communication to get a bigger picture of what’s happening in teams. Healthy cultures encourage employees share theirs opinions, report problems, make suggestions. Otherwise leaders can be in dangerous information isolation.

To be continued....

#booknook #softskills #leadership

True Inspiration from Pixar. Part 3: Broadening the View

That's the third part of Creativity, Inc.: Overcoming the Unseen Forces That Stand in the Way of True Inspiration book overview (Other parts: 1, 2).

Throughout the book the author shares his management ideas and principles. There is one though that I find very interesting from practical point if view. He says that people who live or work together tend to become closer and share similar mental models and behavior patterns (event if those models are wrong). That fact can be used to build right team culture and create atmosphere of creativity and innovation.

Practical tools to improve collaboration within the team or company:
✏️ Daily Meetings. Daily information exchange and team work analysis improves overall team productivity.
✏️ Research Trips. All our mental models are wrong, we need regularly clean our believes and get rid of cliches. Pixar sends their employees to the locations that are relevant to the topics of the movies they are working on (e.g. real student campus for Monster University movie to get better understanding of students life and environment).
✏️ Power of Restrictions. Limited resources help to focus on important things, improve decision making and optimize internal processes.
✏️ Technology and Art Integration. New tech tools should be actively used to automate the routine, freeing up time for more important and creative tasks.
✏️ Short Experiments. Perform short experiments to prove new ideas.
✏️ Train the Vision. Pixar offers drawing courses to all employees, because drawing improves observation skills, stimulates the right side of the brain activity that is responsible for human creativity.
✏️ Dissection. It's very similar to Agile retrospective. After a film release, teams collect their lessons learnt, aggregate good experience and discuss the mistakes. The author suggest the following practice: ask everyone to list 5 things they'd do again and 5 they wouldn’t.
✏️ Continuous Education. Pixar encourages constant learning across fields. Pixar University has classes for drawing, dancing, acting skills and others. Employees from different departments attend classes together, establishing connections outside of work roles. One more important idea behind the practice: if you do something that you don't usually do at work you keep your brain healthy.

The main message from the author to the leaders: people are the most valuable part of any company🫶. Invest time and effort in your teams, help them reveal their potential, trust your colleagues, and avoid trying to control everything, delegate. Encourage openness and trust, create a safe environment within your organization. It’s hard, ongoing work, but it's the only way to build and maintain a creative culture.

I grew up on Pixar movies, and I still go to the cinema for every new release. It was really interesting to read how it's organized inside. Surprisingly, film making is very similar to software development: like any project it has budget and resource restrictions, it starts from MVP, then there is set of iterations and experiments to build the product, teams have daily syncups, some standard project processes, multiple "release candidate", deadlines and, of course, restrospectives. So suggested practices and ideas are also applicable for our daily routine to improve creativity and collaboration.

#booknook #softskills #leadership