GenAI for Legacy Systems Modernization
While most people actively write about using GenAI tools to generate new code, there is a new Thoughtworks publication that focuses on the opposite — using AI to understand and refactor legacy systems.
What makes legacy systems modernization expensive?
- Lack of design and implementation details knowledge
- Lack of actual documentation
- Lack of automated tests
- Absence of human experts
- Difficulty to measure the impact of the change
To address these challenges Thoughtworks team developed a tool called CodeConcise. But the authors highlighted that you don't need exactly this tool, the approach and ideas can be used as a reference to implement your own solution.
Key concepts:
✏️ Treat code as data
✏️ Build Abstract Syntax Trees (ASTs) to identify entities and relationships in the code
✏️ Store these ASTs in graph database (neo4j)
✏️ Use a comprehension pipeline that traverses the graph using multiple algorithms, such as Depth-first Search with backtracking in post-order traversal, to enrich the graph with LLM-generated explanations at various depths (e.g. methods, classes, packages)
✏️ Integrate the enriched graph with a frontend application that implements Retrieval-Augmented Generation (RAG) approach
✏️ The RAG retrieval component pulls nodes relevant to the user’s prompt, while the LLM further traverses the graph to gather more information from their neighboring nodes to provide the LLM-generated explanations at various levels of abstraction
✏️ The same enrichment pipeline can be used to generate documentation for the existing system
For now the tool was tested with several clients to generate explanations for low-level legacy code. The next goal is to improve the model to provide answers at the higher level of abstraction, keeping in mind that it might not be directly possible by examining the code alone.
The work looks promising and could significantly reduce the time and cost of modernizing old systems (especially written on exotic languages like COBOL). It simplifies reverse-engineering and helps generate knowledge about the current system. The authors also promised to share results on improving the current model and provide more real life examples for the tool usage.
#news #engineering #ai
While most people actively write about using GenAI tools to generate new code, there is a new Thoughtworks publication that focuses on the opposite — using AI to understand and refactor legacy systems.
What makes legacy systems modernization expensive?
- Lack of design and implementation details knowledge
- Lack of actual documentation
- Lack of automated tests
- Absence of human experts
- Difficulty to measure the impact of the change
To address these challenges Thoughtworks team developed a tool called CodeConcise. But the authors highlighted that you don't need exactly this tool, the approach and ideas can be used as a reference to implement your own solution.
Key concepts:
✏️ Treat code as data
✏️ Build Abstract Syntax Trees (ASTs) to identify entities and relationships in the code
✏️ Store these ASTs in graph database (neo4j)
✏️ Use a comprehension pipeline that traverses the graph using multiple algorithms, such as Depth-first Search with backtracking in post-order traversal, to enrich the graph with LLM-generated explanations at various depths (e.g. methods, classes, packages)
✏️ Integrate the enriched graph with a frontend application that implements Retrieval-Augmented Generation (RAG) approach
✏️ The RAG retrieval component pulls nodes relevant to the user’s prompt, while the LLM further traverses the graph to gather more information from their neighboring nodes to provide the LLM-generated explanations at various levels of abstraction
✏️ The same enrichment pipeline can be used to generate documentation for the existing system
For now the tool was tested with several clients to generate explanations for low-level legacy code. The next goal is to improve the model to provide answers at the higher level of abstraction, keeping in mind that it might not be directly possible by examining the code alone.
The work looks promising and could significantly reduce the time and cost of modernizing old systems (especially written on exotic languages like COBOL). It simplifies reverse-engineering and helps generate knowledge about the current system. The authors also promised to share results on improving the current model and provide more real life examples for the tool usage.
#news #engineering #ai
martinfowler.com
Legacy Modernization meets GenAI
Lessons from building and using a GenAI tool to assist legacy modernization.
🔥3👍1
True Inspiration from Pixar. Part 2: Protect New Ideas
That's the second part of Creativity, Inc.: Overcoming the Unseen Forces That Stand in the Way of True Inspiration book overview (First part is there). It is about practices that can help to protect new ideas from bureaucracy, fear of feedback, and thinking "experienced people know best."
To support creativity and innovation, Pixar's leaders built a culture based on the following principles:
✏️ Candor. Ask people about candor not honesty, build processes that demonstrate value of the candor on all levels. Pixar uses Braintrust practice to train that:
- Regular offline meetings every 2-3 months.
- Teams present film fragments to identify issues.
- There is no gradation by official noscripts, everyone's opinion is important, there is no stupid or destructive feedback.
- Feedback should be focused on the problem not the person.
- Criticism is part of improving the work, not as competition.
- Create an atmosphere of trust where all member are interested in great results.
✏️ Value of Failures. Don’t fear the failure. Failure is a chance to learn. Start working, get feedback, learn, and try again. The author says that Pixar culture is unique as it doesn't just allow the people to make mistakes, but it expects them to make those mistakes.
✏️ Protect Ugly Babies. New ideas are usually not beautiful, so author called them the
✏️ Change and Randomness. People don't like changes because it feels unsafe or overwhelming. Asking "What if?" question helps teams imagine possibilities and break through the fear barriers. One more important note there is about rules. Rules appear by some reason, but reasons are changed over time and can be not actual anymore. Outdated rules improves bureaucracy and kill creative atmosphere.
✏️ The Hidden. True leaders accept that employees often have a deeper understanding of problems. Managers don’t need to know everything, but they must encourage open communication to get a bigger picture of what’s happening in teams. Healthy cultures encourage employees share theirs opinions, report problems, make suggestions. Otherwise leaders can be in dangerous information isolation.
To be continued....
#booknook #softskills #leadership
That's the second part of Creativity, Inc.: Overcoming the Unseen Forces That Stand in the Way of True Inspiration book overview (First part is there). It is about practices that can help to protect new ideas from bureaucracy, fear of feedback, and thinking "experienced people know best."
To support creativity and innovation, Pixar's leaders built a culture based on the following principles:
✏️ Candor. Ask people about candor not honesty, build processes that demonstrate value of the candor on all levels. Pixar uses Braintrust practice to train that:
- Regular offline meetings every 2-3 months.
- Teams present film fragments to identify issues.
- There is no gradation by official noscripts, everyone's opinion is important, there is no stupid or destructive feedback.
- Feedback should be focused on the problem not the person.
- Criticism is part of improving the work, not as competition.
- Create an atmosphere of trust where all member are interested in great results.
✏️ Value of Failures. Don’t fear the failure. Failure is a chance to learn. Start working, get feedback, learn, and try again. The author says that Pixar culture is unique as it doesn't just allow the people to make mistakes, but it expects them to make those mistakes.
✏️ Protect Ugly Babies. New ideas are usually not beautiful, so author called them the
ugly babies. It takes time and patience for them to grow and shine. All Pixar films started as simple, sometimes awkward ideas, going through many iterations. It's important to protect new ideas from conservatism, habits to do something by only known ways, because it leads to predictable but mediocre results, killing true inspiration. ✏️ Change and Randomness. People don't like changes because it feels unsafe or overwhelming. Asking "What if?" question helps teams imagine possibilities and break through the fear barriers. One more important note there is about rules. Rules appear by some reason, but reasons are changed over time and can be not actual anymore. Outdated rules improves bureaucracy and kill creative atmosphere.
✏️ The Hidden. True leaders accept that employees often have a deeper understanding of problems. Managers don’t need to know everything, but they must encourage open communication to get a bigger picture of what’s happening in teams. Healthy cultures encourage employees share theirs opinions, report problems, make suggestions. Otherwise leaders can be in dangerous information isolation.
To be continued....
#booknook #softskills #leadership
❤1👍1
True Inspiration from Pixar. Part 3: Broadening the View
That's the third part of Creativity, Inc.: Overcoming the Unseen Forces That Stand in the Way of True Inspiration book overview (Other parts: 1, 2).
Throughout the book the author shares his management ideas and principles. There is one though that I find very interesting from practical point if view. He says that people who live or work together tend to become closer and share similar mental models and behavior patterns (event if those models are wrong). That fact can be used to build right team culture and create atmosphere of creativity and innovation.
Practical tools to improve collaboration within the team or company:
✏️ Daily Meetings. Daily information exchange and team work analysis improves overall team productivity.
✏️ Research Trips. All our mental models are wrong, we need regularly clean our believes and get rid of cliches. Pixar sends their employees to the locations that are relevant to the topics of the movies they are working on (e.g. real student campus for Monster University movie to get better understanding of students life and environment).
✏️ Power of Restrictions. Limited resources help to focus on important things, improve decision making and optimize internal processes.
✏️ Technology and Art Integration. New tech tools should be actively used to automate the routine, freeing up time for more important and creative tasks.
✏️ Short Experiments. Perform short experiments to prove new ideas.
✏️ Train the Vision. Pixar offers drawing courses to all employees, because drawing improves observation skills, stimulates the right side of the brain activity that is responsible for human creativity.
✏️ Dissection. It's very similar to Agile retrospective. After a film release, teams collect their lessons learnt, aggregate good experience and discuss the mistakes. The author suggest the following practice: ask everyone to list 5 things they'd do again and 5 they wouldn’t.
✏️ Continuous Education. Pixar encourages constant learning across fields. Pixar University has classes for drawing, dancing, acting skills and others. Employees from different departments attend classes together, establishing connections outside of work roles. One more important idea behind the practice: if you do something that you don't usually do at work you keep your brain healthy.
The main message from the author to the leaders: people are the most valuable part of any company🫶. Invest time and effort in your teams, help them reveal their potential, trust your colleagues, and avoid trying to control everything, delegate. Encourage openness and trust, create a safe environment within your organization. It’s hard, ongoing work, but it's the only way to build and maintain a creative culture.
I grew up on Pixar movies, and I still go to the cinema for every new release. It was really interesting to read how it's organized inside. Surprisingly, film making is very similar to software development: like any project it has budget and resource restrictions, it starts from MVP, then there is set of iterations and experiments to build the product, teams have daily syncups, some standard project processes, multiple "release candidate", deadlines and, of course, restrospectives. So suggested practices and ideas are also applicable for our daily routine to improve creativity and collaboration.
#booknook #softskills #leadership
That's the third part of Creativity, Inc.: Overcoming the Unseen Forces That Stand in the Way of True Inspiration book overview (Other parts: 1, 2).
Throughout the book the author shares his management ideas and principles. There is one though that I find very interesting from practical point if view. He says that people who live or work together tend to become closer and share similar mental models and behavior patterns (event if those models are wrong). That fact can be used to build right team culture and create atmosphere of creativity and innovation.
Practical tools to improve collaboration within the team or company:
✏️ Daily Meetings. Daily information exchange and team work analysis improves overall team productivity.
✏️ Research Trips. All our mental models are wrong, we need regularly clean our believes and get rid of cliches. Pixar sends their employees to the locations that are relevant to the topics of the movies they are working on (e.g. real student campus for Monster University movie to get better understanding of students life and environment).
✏️ Power of Restrictions. Limited resources help to focus on important things, improve decision making and optimize internal processes.
✏️ Technology and Art Integration. New tech tools should be actively used to automate the routine, freeing up time for more important and creative tasks.
✏️ Short Experiments. Perform short experiments to prove new ideas.
✏️ Train the Vision. Pixar offers drawing courses to all employees, because drawing improves observation skills, stimulates the right side of the brain activity that is responsible for human creativity.
✏️ Dissection. It's very similar to Agile retrospective. After a film release, teams collect their lessons learnt, aggregate good experience and discuss the mistakes. The author suggest the following practice: ask everyone to list 5 things they'd do again and 5 they wouldn’t.
✏️ Continuous Education. Pixar encourages constant learning across fields. Pixar University has classes for drawing, dancing, acting skills and others. Employees from different departments attend classes together, establishing connections outside of work roles. One more important idea behind the practice: if you do something that you don't usually do at work you keep your brain healthy.
The main message from the author to the leaders: people are the most valuable part of any company🫶. Invest time and effort in your teams, help them reveal their potential, trust your colleagues, and avoid trying to control everything, delegate. Encourage openness and trust, create a safe environment within your organization. It’s hard, ongoing work, but it's the only way to build and maintain a creative culture.
I grew up on Pixar movies, and I still go to the cinema for every new release. It was really interesting to read how it's organized inside. Surprisingly, film making is very similar to software development: like any project it has budget and resource restrictions, it starts from MVP, then there is set of iterations and experiments to build the product, teams have daily syncups, some standard project processes, multiple "release candidate", deadlines and, of course, restrospectives. So suggested practices and ideas are also applicable for our daily routine to improve creativity and collaboration.
#booknook #softskills #leadership
❤1👍1
Observability 2.0
There is very charismatic talk from Charity Majors, Co-Author of "Observability Engineering", called Is it time for Observability 2.0? Sounds intriguing, so let's check what's inside.
Key ideas:
📍 Observability 1.0. It's traditional observability techniques based on 3 pillars: logs, metrics and traces. It's complex, expensive and requires skilled engineer to analyze correlations between different sources of data.
📍 Observability 1.0 has significant limitations:
- static dashboards
- lack of important details
- designed to answer to pre-defined questions
- multiple source of truth
- multiple systems to support
📍Observability 2.0 paradigm is based on the idea to use wide structured logs that contain all necessary information. It makes easy to aggregate data, zoom in and out for details when needed.
📍Observability 2.0 based on single source of truth - logs, all others is just visualization, aggregation and making dynamic queries. There is no data duplication. There is no need to install and maintain a set of tools for the telemetry. In that terms it's cheaper.
Implementation tips:
📍 Instrument the code using principles of Canonical Logs (we already checked the concept there)
📍Add Trace IDs and Span IDs to trace requests chain execution
📍 Feed the data into a columnar store, to move away from predefined schemas or indexes
📍 Use a storage engine that supports high cardinality
📍Adopt tools with explorable interfaces or dynamic dashboards
Our systems have become too hard and complicated. So it's critical to have effective observability tools and practices. The approach from the talk looks promising especially as it doesn't require any new tools to be developed. Let's see if it become a new trend in observability implementation.
#engineering #observability
There is very charismatic talk from Charity Majors, Co-Author of "Observability Engineering", called Is it time for Observability 2.0? Sounds intriguing, so let's check what's inside.
Key ideas:
📍 Observability 1.0. It's traditional observability techniques based on 3 pillars: logs, metrics and traces. It's complex, expensive and requires skilled engineer to analyze correlations between different sources of data.
📍 Observability 1.0 has significant limitations:
- static dashboards
- lack of important details
- designed to answer to pre-defined questions
- multiple source of truth
- multiple systems to support
📍Observability 2.0 paradigm is based on the idea to use wide structured logs that contain all necessary information. It makes easy to aggregate data, zoom in and out for details when needed.
📍Observability 2.0 based on single source of truth - logs, all others is just visualization, aggregation and making dynamic queries. There is no data duplication. There is no need to install and maintain a set of tools for the telemetry. In that terms it's cheaper.
Implementation tips:
📍 Instrument the code using principles of Canonical Logs (we already checked the concept there)
📍Add Trace IDs and Span IDs to trace requests chain execution
📍 Feed the data into a columnar store, to move away from predefined schemas or indexes
📍 Use a storage engine that supports high cardinality
📍Adopt tools with explorable interfaces or dynamic dashboards
Our systems have become too hard and complicated. So it's critical to have effective observability tools and practices. The approach from the talk looks promising especially as it doesn't require any new tools to be developed. Let's see if it become a new trend in observability implementation.
#engineering #observability
YouTube
Is It Time To Version Observability? (Signs Point To Yes) • Charity Majors • GOTO 2024
This presentation was recorded at GOTO Amsterdam 2024. #GOTOcon #GOTOams
https://gotoams.nl
Charity Majors - CTO at honeycomb.io & Co- Author of "Observability Engineering" @CharityMajors
RESOURCES
https://twitter.com/mipsytipsy
https://github.com/charity…
https://gotoams.nl
Charity Majors - CTO at honeycomb.io & Co- Author of "Observability Engineering" @CharityMajors
RESOURCES
https://twitter.com/mipsytipsy
https://github.com/charity…
👍3🔥1
Shipping Threads in 5 Months
As developers, we often prefer writing new code rather then reusing the older one. Of course, new code will be better, faster, more readable, maintainable, it will use newer tools and frameworks and definitely will be great. Spoiler: No 😉
Old battle-tested code can be significantly better because it's already tested, covered by automations, it has common features in place, little learning encoded, it doesn't have stupid mistakes already, it's mature.
That's why I think there is really interesting experience that shared by Meta team how they reuse Instagram code to build Threads app from scratch.
Initially, the team had an ambitious goal to build a service to compete with X (Twitter) in a couple of months. To achieve that, they decided to use existing Instagram code with core features like profiles, posts, recommendations, followers as the base for the new service. Additionally, Threads was built on existing Instagram infra with support of existing product teams. This approach allows to deliver new fully-featured service in 5 months.
Key findings during the process:
✏️ You need the deep knowledge of legacy code to successfully reuse it
✏️ Code readability really matters
✏️ Repurposing and customization existing code for the new requirements brough additional technical debt that should be paid in the future
✏️ Shared infrastructure and product teams can significantly reduce development costs
✏️ Old code is already tested in real conditions and contains less issues then the new code
So don't rush to rewrite the existing code. Thoughtful evolutionary approach can bring more business benefits, reducing time to market and overall development costs.
#engineering #usecase
As developers, we often prefer writing new code rather then reusing the older one. Of course, new code will be better, faster, more readable, maintainable, it will use newer tools and frameworks and definitely will be great. Spoiler: No 😉
Old battle-tested code can be significantly better because it's already tested, covered by automations, it has common features in place, little learning encoded, it doesn't have stupid mistakes already, it's mature.
That's why I think there is really interesting experience that shared by Meta team how they reuse Instagram code to build Threads app from scratch.
Initially, the team had an ambitious goal to build a service to compete with X (Twitter) in a couple of months. To achieve that, they decided to use existing Instagram code with core features like profiles, posts, recommendations, followers as the base for the new service. Additionally, Threads was built on existing Instagram infra with support of existing product teams. This approach allows to deliver new fully-featured service in 5 months.
Key findings during the process:
✏️ You need the deep knowledge of legacy code to successfully reuse it
✏️ Code readability really matters
✏️ Repurposing and customization existing code for the new requirements brough additional technical debt that should be paid in the future
✏️ Shared infrastructure and product teams can significantly reduce development costs
✏️ Old code is already tested in real conditions and contains less issues then the new code
So don't rush to rewrite the existing code. Thoughtful evolutionary approach can bring more business benefits, reducing time to market and overall development costs.
#engineering #usecase
InfoQ
Simplicity, Speed, and Re-Use. Shipping Threads in 5 Months
In Jan 2023, we received word that we’d need to build a microblogging service to compete with Twitter in a couple of months. A small team was assembled to take on that challenge, and we shipped a new social network in July. This article describes how we developed…
👍2🔥1
Put Your Own Mask On First
"Put your own mask on first before assisting your child 😷"—they always say it on the plane before the flight. It sounds clear and familiar for us. But this same rule can apply to other parts of our lives. Metaphorically of course.
What do we usually do to meet deadlines, deal with a pile of issues at work, or handle business pressures? The most common scenario is to work more and more, eventually leading to burnout. But a burned-out leader can't solve problems effectively or help their team to survive in the storm of difficulties or achieve business goals.
It may sound counterintuitive but more pressure and problems you face at work, the more time and care you need to give yourself: proper nutrition, walking, regular physical activity, full sleep and less overwork.
When you take care of yourself, you can better take care of your team. So put your own mask on first before assisting to others.
#softskills #leadership
"Put your own mask on first before assisting your child 😷"—they always say it on the plane before the flight. It sounds clear and familiar for us. But this same rule can apply to other parts of our lives. Metaphorically of course.
What do we usually do to meet deadlines, deal with a pile of issues at work, or handle business pressures? The most common scenario is to work more and more, eventually leading to burnout. But a burned-out leader can't solve problems effectively or help their team to survive in the storm of difficulties or achieve business goals.
It may sound counterintuitive but more pressure and problems you face at work, the more time and care you need to give yourself: proper nutrition, walking, regular physical activity, full sleep and less overwork.
When you take care of yourself, you can better take care of your team. So put your own mask on first before assisting to others.
#softskills #leadership
❤8👍2
API Governance
In modern distributed systems, where individual teams manage different services, it's pretty common for each team to create their own APIs in different ways. Each team tries to make their APIs unique and impressive. As a result, a company may have a lot of APIs that follow different rules and principles and reflect organizational structure instead of business domains (you remember Conway's Law, right?). This can be a full mess.
To avoid this, APIs must be properly managed to stay consistent.
API Governance is the set of practices, tools and policies to enforce API quality, security, consistency and compliance. It involves creating standards and processes for every stage in the API lifecycle, from design, development, and testing to deployment, management, and retirement.
API Governance consists of the following elements:
📍 Centralization. A single point where policies are created and enforced.
📍 API Contract. Standard specifications to define APIs like OpenAPI, gRPC, GraphQL, AsyncAPI and others.
📍Implementation Guidelines. Establish and enforce style guidelines for all APIs. Good examples are Google Cloud API Guidelines , Azure API Guidelines.
📍 Security Policies. Defining API security standards and policies that protect sensitive data from cyber threats and ensuring API compliance with regulations.
📍 Automation. Developers and other roles need to quickly make sure that APIs are compliant with the enterprise standards at various stages of the lifecycle.
📍Versioning.
📍Deprecation Policy.
📍API Discovery. Provide a way to easily search for and discover existing APIs.
API Governance provides the guardrails to develop high-quality consistent APIs within the company. But to make it work a good level of automation is required.
#engineering #api
In modern distributed systems, where individual teams manage different services, it's pretty common for each team to create their own APIs in different ways. Each team tries to make their APIs unique and impressive. As a result, a company may have a lot of APIs that follow different rules and principles and reflect organizational structure instead of business domains (you remember Conway's Law, right?). This can be a full mess.
To avoid this, APIs must be properly managed to stay consistent.
API Governance is the set of practices, tools and policies to enforce API quality, security, consistency and compliance. It involves creating standards and processes for every stage in the API lifecycle, from design, development, and testing to deployment, management, and retirement.
API Governance consists of the following elements:
📍 Centralization. A single point where policies are created and enforced.
📍 API Contract. Standard specifications to define APIs like OpenAPI, gRPC, GraphQL, AsyncAPI and others.
📍Implementation Guidelines. Establish and enforce style guidelines for all APIs. Good examples are Google Cloud API Guidelines , Azure API Guidelines.
📍 Security Policies. Defining API security standards and policies that protect sensitive data from cyber threats and ensuring API compliance with regulations.
📍 Automation. Developers and other roles need to quickly make sure that APIs are compliant with the enterprise standards at various stages of the lifecycle.
📍Versioning.
📍Deprecation Policy.
📍API Discovery. Provide a way to easily search for and discover existing APIs.
API Governance provides the guardrails to develop high-quality consistent APIs within the company. But to make it work a good level of automation is required.
#engineering #api
Google Cloud Documentation
API design guide | Cloud API Design Guide | Google Cloud Documentation
A set of guidelines for designing APIs that are consistent with Google AIPs.
❤1👍1
API Governance: Versioning
Let's continue today with API management and talk about versioning. To define you versioning policy you need to answer the following questions:
😲 What versioning method will be used?
🤔 When do you need to release a new version?
😬 What naming convention to use?
😵💫 How to keep compatibility with the clients?
The most popular versioning strategies:
✏️ No Versioning. Yes, that's also a choice 😀
✏️ Semantic Versioning. It's well-know strategy to version anything in software development world.
✏️ Stability Levels: alpha, beta, stable. Major version is changed on breaking changes. Examples: v1alpha, v2beta, v1aplha3, v2. More details in Google API Design Guide.
✏️ Release Numbers: Simple sequential versions like v1, v2, v3, updated mainly for breaking changes.
✏️ Product Release Version: Use your product’s version for APIs. Example: product version 2024.3 then API version 2024.3. In that case version is changed even if there are no major changes, but it really simplifies tracking compatibility between releases and APIs.
To reduce the impact of API changes the following compatibility strategies can be used:
✏️ Synchronized Updates: Both API and clients are updated and delivered together. Simple, fragile. It can be useful if you control and manage all API consumers.
✏️ Client Supports Multiple Versions: One client can work with multiple API versions, but outdated clients may stop working and require updates to match newer APIs.
✏️ API Serves Multiple Versions: New API version is added in parallel to the existing one on the same server. In that case you may serve as many versions as you need to support all your clients. To reduce API management overhead Hub-Spoke pattern can be used: the hub represents the main version, while spoke versions are converted from the hub. This approach is actively used in Kubernetes, so you can read more details in Kubebuilder Conversion Concepts.
Analyze your requirements and architecture, set clear rules, define the versioning and compatibility approach. It's really important to document those decisions and socialize them to your clients.
#engineering #api
Let's continue today with API management and talk about versioning. To define you versioning policy you need to answer the following questions:
😲 What versioning method will be used?
🤔 When do you need to release a new version?
😬 What naming convention to use?
😵💫 How to keep compatibility with the clients?
The most popular versioning strategies:
✏️ No Versioning. Yes, that's also a choice 😀
✏️ Semantic Versioning. It's well-know strategy to version anything in software development world.
✏️ Stability Levels: alpha, beta, stable. Major version is changed on breaking changes. Examples: v1alpha, v2beta, v1aplha3, v2. More details in Google API Design Guide.
✏️ Release Numbers: Simple sequential versions like v1, v2, v3, updated mainly for breaking changes.
✏️ Product Release Version: Use your product’s version for APIs. Example: product version 2024.3 then API version 2024.3. In that case version is changed even if there are no major changes, but it really simplifies tracking compatibility between releases and APIs.
To reduce the impact of API changes the following compatibility strategies can be used:
✏️ Synchronized Updates: Both API and clients are updated and delivered together. Simple, fragile. It can be useful if you control and manage all API consumers.
✏️ Client Supports Multiple Versions: One client can work with multiple API versions, but outdated clients may stop working and require updates to match newer APIs.
✏️ API Serves Multiple Versions: New API version is added in parallel to the existing one on the same server. In that case you may serve as many versions as you need to support all your clients. To reduce API management overhead Hub-Spoke pattern can be used: the hub represents the main version, while spoke versions are converted from the hub. This approach is actively used in Kubernetes, so you can read more details in Kubebuilder Conversion Concepts.
Analyze your requirements and architecture, set clear rules, define the versioning and compatibility approach. It's really important to document those decisions and socialize them to your clients.
#engineering #api
❤1
SMURF Testing
Google introduced new mnemonic for test quality attributes - SMURF:
📌 Speed: Unit tests are faster than other test types so they can be run more often.
📌 Maintainability: Cost of debugging and maintaining tests.
📌 Utilization: A good test suite optimizes resource utilization, fewer resources cost less to run.
📌 Reliability: Sorting out flaky tests wastes developer time and costs resources in rerunning the tests.
📌 Fidelity: High-fidelity tests come closer to approximating real operating conditions. So integration tests have more fidelity than unit tests.
In many cases improving one quality attribute can affect the others, so be careful and measure your costs and trade-offs.
#engineering #testing
Google introduced new mnemonic for test quality attributes - SMURF:
📌 Speed: Unit tests are faster than other test types so they can be run more often.
📌 Maintainability: Cost of debugging and maintaining tests.
📌 Utilization: A good test suite optimizes resource utilization, fewer resources cost less to run.
📌 Reliability: Sorting out flaky tests wastes developer time and costs resources in rerunning the tests.
📌 Fidelity: High-fidelity tests come closer to approximating real operating conditions. So integration tests have more fidelity than unit tests.
In many cases improving one quality attribute can affect the others, so be careful and measure your costs and trade-offs.
#engineering #testing
❤1
API Governance at Scale
The most difficult part of API Governance is to ensure that developers follow provided guidelines and policies. Without proper controls, the real code will eventually drift from the guidelines—it’s only a matter of time. This doesn’t happen because developers are bad or unwilling to follow the rules, but because we’re human, and humans make mistakes. Mistakes accumulate and grow over time and as a result you can get APIs that are too far from initial recommendations.
In small teams with a small codebase, developers education can work, and trained reviewers can ensure the code follows the rules. But as your team or organization grows, this approach isn't enough. I strongly believe that only automation can maintain policy compliance over a large codebase or multiple teams.
Google recently published API Governance at Scale, sharing their experience and tools to control API guidelines execution.
They introduced 3 key components:
✏️ API Improvement Proposals (AIPs). This is a design document providing high-level documentation for API development. Each rule is introduced as a separate AIP that consists of a problem denoscription and a guideline to follow (Example, AIP-126).
✏️ API Linter . This tool provides real-time checks for compliance with existing AIPs.
✏️ API Readability Program. This is an educational program to prepare and certify API design experts, who then perform a code review for API changes.
While Google developed the AIPs concept, they encourage other companies to adopt the approach. Many of the rules are generic and easily reusable. They even provide a special guide on how to adopt AIPs. Adoption strategy is not finished now, but preparation status can be tracked via appropriate Github issue.
#engineering #api
The most difficult part of API Governance is to ensure that developers follow provided guidelines and policies. Without proper controls, the real code will eventually drift from the guidelines—it’s only a matter of time. This doesn’t happen because developers are bad or unwilling to follow the rules, but because we’re human, and humans make mistakes. Mistakes accumulate and grow over time and as a result you can get APIs that are too far from initial recommendations.
In small teams with a small codebase, developers education can work, and trained reviewers can ensure the code follows the rules. But as your team or organization grows, this approach isn't enough. I strongly believe that only automation can maintain policy compliance over a large codebase or multiple teams.
Google recently published API Governance at Scale, sharing their experience and tools to control API guidelines execution.
They introduced 3 key components:
✏️ API Improvement Proposals (AIPs). This is a design document providing high-level documentation for API development. Each rule is introduced as a separate AIP that consists of a problem denoscription and a guideline to follow (Example, AIP-126).
✏️ API Linter . This tool provides real-time checks for compliance with existing AIPs.
✏️ API Readability Program. This is an educational program to prepare and certify API design experts, who then perform a code review for API changes.
While Google developed the AIPs concept, they encourage other companies to adopt the approach. Many of the rules are generic and easily reusable. They even provide a special guide on how to adopt AIPs. Adoption strategy is not finished now, but preparation status can be tracked via appropriate Github issue.
#engineering #api
research.google
API Governance at Scale
❤1👍1
Take a Vacation
Last week I was on vacation, so there was a little break in the publications😌. Therefore I would like to talk a little about the vacation and how important it is. High quality vacation is not just opportunity for relax but it is also a prevention mechanism for many serious diseases.
But it’s not enough just to take vacations regularly; the way you spend them determines if you re-charge your internal battery or not.
My tips for a good vacation:
✏️ Take Enough Time: Ideally, a vacation length should be at least 14 days (as a single period). If you feel heavily exhausted, then better to take 21 days. That time is usually enough to recharge.
✏️ Change the Scenery: Travelling to a new place (even a short trip) gives you new impressions, experience, fill you with new ideas, inspiration and energy. Spending time outside standard surroundings significantly decreases an overall strain level. The fact is also proved by German researchers.
✏️ Digital Detox: Don't touch your laptop, don't open working chats, don't read the news, minimize social networks usage. Give the rest to your brain from constant information noise.
✏️ Be Spontaneous: Don't try to plan everything: constant following the schedule makes vacation feel more work-like and doesn't allow to enjoy the moment. Spontaneous activities can provide more fun and satisfaction.
✏️ Do Nothing: Allow yourself to take time for idleness. That's really difficult as you feel just wasting time that can be spend more effectively😀. But that's the trick: state of nothingness rewires the brain, improve creativity and problem solving capabilities.
So take care of yourself and plan a proper rest during the year.
#softskills #productivity
Last week I was on vacation, so there was a little break in the publications😌. Therefore I would like to talk a little about the vacation and how important it is. High quality vacation is not just opportunity for relax but it is also a prevention mechanism for many serious diseases.
But it’s not enough just to take vacations regularly; the way you spend them determines if you re-charge your internal battery or not.
My tips for a good vacation:
✏️ Take Enough Time: Ideally, a vacation length should be at least 14 days (as a single period). If you feel heavily exhausted, then better to take 21 days. That time is usually enough to recharge.
✏️ Change the Scenery: Travelling to a new place (even a short trip) gives you new impressions, experience, fill you with new ideas, inspiration and energy. Spending time outside standard surroundings significantly decreases an overall strain level. The fact is also proved by German researchers.
✏️ Digital Detox: Don't touch your laptop, don't open working chats, don't read the news, minimize social networks usage. Give the rest to your brain from constant information noise.
✏️ Be Spontaneous: Don't try to plan everything: constant following the schedule makes vacation feel more work-like and doesn't allow to enjoy the moment. Spontaneous activities can provide more fun and satisfaction.
✏️ Do Nothing: Allow yourself to take time for idleness. That's really difficult as you feel just wasting time that can be spend more effectively😀. But that's the trick: state of nothingness rewires the brain, improve creativity and problem solving capabilities.
So take care of yourself and plan a proper rest during the year.
#softskills #productivity
❤4🔥4👍1
Google ARM Processor
Last week, Google announced their own custom ARM-based processor for general-purpose workloads. They promised up to 65% better price-performance and up to 60% better energy-efficiency.
Why is it interesting? Until now, only AWS offered a custom cost-optimized ARM processor - AWS Graviton. And now Google joined the competition. This shows that interest in ARM processors still grows and continue to grow in the future.
From engineering perspective, it's not possible just to switch workload from one architecture to another as images need to be pre-built for a specific architecture. One of the ways to test ARM nodes and migrate smoothly on the new architecture is by using multi-architecture images (I wrote about that here)
#engineering #news
Last week, Google announced their own custom ARM-based processor for general-purpose workloads. They promised up to 65% better price-performance and up to 60% better energy-efficiency.
Why is it interesting? Until now, only AWS offered a custom cost-optimized ARM processor - AWS Graviton. And now Google joined the competition. This shows that interest in ARM processors still grows and continue to grow in the future.
From engineering perspective, it's not possible just to switch workload from one architecture to another as images need to be pre-built for a specific architecture. One of the ways to test ARM nodes and migrate smoothly on the new architecture is by using multi-architecture images (I wrote about that here)
#engineering #news
Google Cloud Blog
Try C4A, the first Google Axion Processor | Google Cloud Blog
The custom Arm-based processor is designed for general-purpose workloads like web and app servers, databases, analytics, CPU-based AI, and more.
👍2🔥1
Uber’s Gen AI On-Call Copilot
GenAI continues its march in routine automation. This time Uber shared their experience with Genie - on-call support automation for internal teams.
The issue is very common for large companies with many teams: there is some channels (for Uber, it's slack with ~45 000 questions per month) where teams can put questions and request help with the service or technology. Of course, there are a lot of docs and relevant articles, but they are fragmented and spread across internal resources. It's really hard for users to find answers on their own. As a result, the number of repetitive questions grows, load and demand on support engineers increase.
Key elements of implemented solution:
✏️ RAG (Retrieval-Augmented Generation) Approach to work with LLM
✏️ Data Pipeline: Information from wikis, internal Stack Overflow, and engineering docs is scraped daily, transformed into vectors, and stored in an in-house vector database with the source links. Data pipeline is implemented on Apache Spark.
✏️ Knowledge Service: When a user posts a question in Slack, Genie’s backend converts it into a vector and fetches the most relevant chunks from the vector database.
✏️ User Feedback: Users can rank answers as Resolved, Helpful, Not Helpful, or Relevant, these ratings are used to analyze answer quality.
✏️ Source Quality Improvements: There is a separate evaluation process to improve source data quality. The LLM performs docs analysis and returns an evaluation score, explanations of the score and actionable suggestions to improve. All these information is collected to an evaluation report for further analysis and fixes.
Since Genie’s launch in September 2023, Uber reports it has answered 70,000 questions with 48.9% helpfulness rate, saving 13 000 engineering hours😲. It's impressive! I definitely want to have something similar at my work. Just a small hurdle left—get the budget and resources for implementation. No big deal, right? 😉
#engineering #usecase #ai
GenAI continues its march in routine automation. This time Uber shared their experience with Genie - on-call support automation for internal teams.
The issue is very common for large companies with many teams: there is some channels (for Uber, it's slack with ~45 000 questions per month) where teams can put questions and request help with the service or technology. Of course, there are a lot of docs and relevant articles, but they are fragmented and spread across internal resources. It's really hard for users to find answers on their own. As a result, the number of repetitive questions grows, load and demand on support engineers increase.
Key elements of implemented solution:
✏️ RAG (Retrieval-Augmented Generation) Approach to work with LLM
✏️ Data Pipeline: Information from wikis, internal Stack Overflow, and engineering docs is scraped daily, transformed into vectors, and stored in an in-house vector database with the source links. Data pipeline is implemented on Apache Spark.
✏️ Knowledge Service: When a user posts a question in Slack, Genie’s backend converts it into a vector and fetches the most relevant chunks from the vector database.
✏️ User Feedback: Users can rank answers as Resolved, Helpful, Not Helpful, or Relevant, these ratings are used to analyze answer quality.
✏️ Source Quality Improvements: There is a separate evaluation process to improve source data quality. The LLM performs docs analysis and returns an evaluation score, explanations of the score and actionable suggestions to improve. All these information is collected to an evaluation report for further analysis and fixes.
Since Genie’s launch in September 2023, Uber reports it has answered 70,000 questions with 48.9% helpfulness rate, saving 13 000 engineering hours😲. It's impressive! I definitely want to have something similar at my work. Just a small hurdle left—get the budget and resources for implementation. No big deal, right? 😉
#engineering #usecase #ai
❤1🔥1
Columnar Databases
Traditional databases store data in a row-oriented approach that is optimized for transactional, single-entity data lookup. But if you need to aggregate data by a specific column, the system has to read all columns from disk, which slows down query performance and increase resource usage.
To solve the issue, columnar databases was introduced.
Columnar database is a type of a database that stores data in columns together on the disk.
Imagine the following sample:
In row-database it will be stored as following:
In column-database:
Benefits of the approach:
📍High data compression due to the similarity of data within a column
📍Enhanced querying and aggregation performance for in analytical and reporting tasks
📍Reduced I/O load as there is no need to process irrelevant data
The most popular columnar databases:
1. Amazon Redshift
2. Google Cloud BigTable
3. Microsoft Azure Cosmos DB
4. Apache Druid
5. Vertica
6. ClickHouse
7. Snowflake Data Cloud
Columnar databases are well-suited for building data warehouse, real-time analytics, statistics, storing and aggregating time-series data.
#engineering
Traditional databases store data in a row-oriented approach that is optimized for transactional, single-entity data lookup. But if you need to aggregate data by a specific column, the system has to read all columns from disk, which slows down query performance and increase resource usage.
To solve the issue, columnar databases was introduced.
Columnar database is a type of a database that stores data in columns together on the disk.
Imagine the following sample:
|Account|LastName|FirstName|Purchase,$|
| 0122 | Jones | Jason | 325.5 |
| 0123 | Diamond| Richard | 500 |
| 0124 | Tailor | Alice | 125 |
In row-database it will be stored as following:
0122, Jones, Jason, 325.5;
0123, Diamond, Richard, 500;
0124, Tailor, Alice, 125;
In column-database:
0122, 0123, 0124;
Jones, Diamond, Tailor;
Jason, Richard, Alice;
325.5, 500, 125;
Benefits of the approach:
📍High data compression due to the similarity of data within a column
📍Enhanced querying and aggregation performance for in analytical and reporting tasks
📍Reduced I/O load as there is no need to process irrelevant data
The most popular columnar databases:
1. Amazon Redshift
2. Google Cloud BigTable
3. Microsoft Azure Cosmos DB
4. Apache Druid
5. Vertica
6. ClickHouse
7. Snowflake Data Cloud
Columnar databases are well-suited for building data warehouse, real-time analytics, statistics, storing and aggregating time-series data.
#engineering
👍1🔥1
Manage Your Energy Level
Recently I wrote about the importance of having high-quality vacations. What I didn’t share is that I went on vacation completely drained, with zero level of internal resources and even a diagnosis from a neurologist 😵💫. It is a tough state to be in, I never want to feel like that again.
So I reflected on how to prevent burning out in the future.
First of all, I understand that's my fault - not heavy work, urgent issues, or company changes. It's primary responsibility of any leader to support their internal resource and energy. That's very important. Leader cannot work without enough energy level, as it's not possible to drive anything or meet business goals in that state.
Next, I started to study different recommendations what to do. The advice is usually very common: exercise, walk, eat well, and have time for hobbies. Unfortunately, I already knew that, but it didn't help me. My issue is that I don't notice the point where I am completely drained and it's too late to go for a walk.
So I need to control internal state somehow. As technical people, we know that to control something we need to measure something. One resource recommends Welltory app that makes personal health analysis based on heart rate variability (Garmin watches have similar features already built-in). Additionally it uses info about sleep, steps, stress level, and more from mostly any smart watch. Looks like a magic, but there is real science under that. This isn’t an ad—just sharing a tool I found useful 🙂.
I've been using the app for about 2 weeks now. The algorithm is still training (about 35% done), but I’m already using its basic features. I periodically make measurements and check overall state: green, orange or red. Based on this, I’ve started taking short recovery breaks at work to avoid hitting zero. Also I control overall health trend to understand if my daily routine needs additional corrections like more exercise, walk, etc.
Burnout is very common problem in our industry that's why I decided to share my experience on what can be helpful to control internal state and support good level of motivation and energy. Of course, 2 weeks are not enough to say the approach works. Put likes if the topic is interesting and I'll share my results in 1-2 months.
Stay healthy and take care of yourself.
#softskills #productivity
Recently I wrote about the importance of having high-quality vacations. What I didn’t share is that I went on vacation completely drained, with zero level of internal resources and even a diagnosis from a neurologist 😵💫. It is a tough state to be in, I never want to feel like that again.
So I reflected on how to prevent burning out in the future.
First of all, I understand that's my fault - not heavy work, urgent issues, or company changes. It's primary responsibility of any leader to support their internal resource and energy. That's very important. Leader cannot work without enough energy level, as it's not possible to drive anything or meet business goals in that state.
Next, I started to study different recommendations what to do. The advice is usually very common: exercise, walk, eat well, and have time for hobbies. Unfortunately, I already knew that, but it didn't help me. My issue is that I don't notice the point where I am completely drained and it's too late to go for a walk.
So I need to control internal state somehow. As technical people, we know that to control something we need to measure something. One resource recommends Welltory app that makes personal health analysis based on heart rate variability (Garmin watches have similar features already built-in). Additionally it uses info about sleep, steps, stress level, and more from mostly any smart watch. Looks like a magic, but there is real science under that. This isn’t an ad—just sharing a tool I found useful 🙂.
I've been using the app for about 2 weeks now. The algorithm is still training (about 35% done), but I’m already using its basic features. I periodically make measurements and check overall state: green, orange or red. Based on this, I’ve started taking short recovery breaks at work to avoid hitting zero. Also I control overall health trend to understand if my daily routine needs additional corrections like more exercise, walk, etc.
Burnout is very common problem in our industry that's why I decided to share my experience on what can be helpful to control internal state and support good level of motivation and energy. Of course, 2 weeks are not enough to say the approach works. Put likes if the topic is interesting and I'll share my results in 1-2 months.
Stay healthy and take care of yourself.
#softskills #productivity
👍9✍4❤🔥2
Cloud Ecosystem Trends
This week CNCF published Emerging trends in the cloud native ecosystem with a list of trends that will continue to grow in 2025.
Top trends:
🚀 Cloud Cost Optimizations. With growing cloud adoption, businesses focus on controlling cloud costs using tools like Karpenter and OpenCost. The same trend was also highlighted by FinOps Foundation earlier this year.
🚀 Platform Engineering (I did overview there). Extend developer experience with platforms for observability, policies as a code, internal developer portals, security, CI/CD, and storage to speed up business development.
🚀 AI Synergy. The trend is to support AI training and operations in the cloud. New actively developed projects there:
- OPEA: a collection of cloud-native patterns for GenAI workloads
- Milvus: a high-performance vector database
- Kubeflow: a project to deploy machine-learning workflows on Kubernetes
- KServe: a toolset for serving predictive and generative machine-learning models
🚀 Observability Standards Unification. Projects like OpenTelemetry and the Observability TAG unify standards, minimize vendor locks, and reduce costs.
🚀 Security. Security is a top priority topic in CNCF. There are some newly graduated projects in that area (like Falco) and separate TAG-Security group that publishes white papers that offer directions to the industry on the topic of security.
🚀 Sustainability (more about GreenOps there). Sustainability tools (like Kepler, OpenCost) measure carbon footprints of Kubernetes applications. The area is under active development now, but it already has promising open-source projects and standards.
It's interesting that overprovisioning and high resource waste is still the main problem in modern clouds. According to the Kubernetes Cost Benchmark Report clusters with 50 or more CPUs used only 13% of their provisioned capacity, memory utilization was at the level of 20%. This shows a huge opportunity for future optimizations.
#news
This week CNCF published Emerging trends in the cloud native ecosystem with a list of trends that will continue to grow in 2025.
Top trends:
🚀 Cloud Cost Optimizations. With growing cloud adoption, businesses focus on controlling cloud costs using tools like Karpenter and OpenCost. The same trend was also highlighted by FinOps Foundation earlier this year.
🚀 Platform Engineering (I did overview there). Extend developer experience with platforms for observability, policies as a code, internal developer portals, security, CI/CD, and storage to speed up business development.
🚀 AI Synergy. The trend is to support AI training and operations in the cloud. New actively developed projects there:
- OPEA: a collection of cloud-native patterns for GenAI workloads
- Milvus: a high-performance vector database
- Kubeflow: a project to deploy machine-learning workflows on Kubernetes
- KServe: a toolset for serving predictive and generative machine-learning models
🚀 Observability Standards Unification. Projects like OpenTelemetry and the Observability TAG unify standards, minimize vendor locks, and reduce costs.
🚀 Security. Security is a top priority topic in CNCF. There are some newly graduated projects in that area (like Falco) and separate TAG-Security group that publishes white papers that offer directions to the industry on the topic of security.
🚀 Sustainability (more about GreenOps there). Sustainability tools (like Kepler, OpenCost) measure carbon footprints of Kubernetes applications. The area is under active development now, but it already has promising open-source projects and standards.
It's interesting that overprovisioning and high resource waste is still the main problem in modern clouds. According to the Kubernetes Cost Benchmark Report clusters with 50 or more CPUs used only 13% of their provisioned capacity, memory utilization was at the level of 20%. This shows a huge opportunity for future optimizations.
#news
CNCF
Emerging trends in the cloud native ecosystem
Member post by Jatinder Singh Purba, Principal, Infosys; Krishnakumar V, Principal, Infosys; Prabhat Kumar, Senior Industry Principal, Infosys; and Shreshta Shyamsundar, Distinguished Technologist…
❤1
I'm introducing a new section on the channel: #aibasics !
Over the past two years, ML is the top trend in the industry with the huge interest not just in tech but across various business domains. ML helps to automate routine tasks and significantly decrease operational costs. And definitely this trend will continue to grow next few years or even more.
As engineers we should at least know the fundamentals of that technology. I mean not just using lots of GenAI tools in daily work but understanding how it works under the hood, its limitations, capabilities and applicability for business and engineering tasks. As of me, I have a significant knowledge gap here, which I plan to close next several months.
I plan to start with the following courses (they are absolutely free):
✏️ Mashing Learning Crash Course from Google that has fresh updates in November
✏️ LLM Course by Cohere
I will use those courses as a base and extend them with additional sources on demand.
So I'm starting my AI learning journey and will share my progress and key takeaways here 💪
Over the past two years, ML is the top trend in the industry with the huge interest not just in tech but across various business domains. ML helps to automate routine tasks and significantly decrease operational costs. And definitely this trend will continue to grow next few years or even more.
As engineers we should at least know the fundamentals of that technology. I mean not just using lots of GenAI tools in daily work but understanding how it works under the hood, its limitations, capabilities and applicability for business and engineering tasks. As of me, I have a significant knowledge gap here, which I plan to close next several months.
I plan to start with the following courses (they are absolutely free):
✏️ Mashing Learning Crash Course from Google that has fresh updates in November
✏️ LLM Course by Cohere
I will use those courses as a base and extend them with additional sources on demand.
So I'm starting my AI learning journey and will share my progress and key takeaways here 💪
❤1👍1
ML Introduction
Let's start AI basics with the ML definition and their types.
Definition from Google ML Introduction course:
ML Types:
📍 Supervised Learning. The model is trained on lots of data with existing correct answers. It's "supervised" in the sense that a human gives the ML system data with the known correct results. This type is used for regressions and classifications.
📍 Unsupervised Learning. The model makes predictions using data that does not contain any correct answers. A commonly used unsupervised learning model employs a technique called clustering. The difference from classification is that categories are discovered during training and not defined by a human.
📍Reinforcement Learning. The model make predictions by getting rewards or penalties based the on actions performed. The goal is to find the best strategy to get the most rewards. Approach is used to train robots to execute different tasks.
📍Generative AI. The model creates content (text, images, music, etc.) from a user input. These models learn existing patterns in data with the goal to produce new but similar data.
Each ML type has its own purpose, like making predictions, finding patterns, creating content, or automating routine tasks. Among them, Generative AI is the most popular and well-known today.
#aibasics
Let's start AI basics with the ML definition and their types.
Definition from Google ML Introduction course:
ML is the process of training a piece of software, called a model, to make useful predictions or generate content from data.
ML Types:
📍 Supervised Learning. The model is trained on lots of data with existing correct answers. It's "supervised" in the sense that a human gives the ML system data with the known correct results. This type is used for regressions and classifications.
📍 Unsupervised Learning. The model makes predictions using data that does not contain any correct answers. A commonly used unsupervised learning model employs a technique called clustering. The difference from classification is that categories are discovered during training and not defined by a human.
📍Reinforcement Learning. The model make predictions by getting rewards or penalties based the on actions performed. The goal is to find the best strategy to get the most rewards. Approach is used to train robots to execute different tasks.
📍Generative AI. The model creates content (text, images, music, etc.) from a user input. These models learn existing patterns in data with the goal to produce new but similar data.
Each ML type has its own purpose, like making predictions, finding patterns, creating content, or automating routine tasks. Among them, Generative AI is the most popular and well-known today.
#aibasics
❤1👍1
ML Basic Terms
To be on the same page with AI-experts we need to build a special vocabulary with basic terms and concepts:
✏️ Feature - input parameter for the model. Usually it represents some characteristic of the entity or facts for which the model makes prediction.
✏️ Label - existing answer for input data. Usually used to train supervised models: predicted value can be compared with labels to check the size of discrepancy.
✏️ Loss - the difference between predicted value and label. For different models different functions to calculate loss is used.
✏️ Learning Rate - a floating-point number that tells the optimization algorithm the step size for the iteration while moving toward a minimum of a loss function. If the learning rate is too low, the model can take a long time to converge. If the learning rate is too high, the model may never converge.
#aibasics
To be on the same page with AI-experts we need to build a special vocabulary with basic terms and concepts:
✏️ Feature - input parameter for the model. Usually it represents some characteristic of the entity or facts for which the model makes prediction.
✏️ Label - existing answer for input data. Usually used to train supervised models: predicted value can be compared with labels to check the size of discrepancy.
✏️ Loss - the difference between predicted value and label. For different models different functions to calculate loss is used.
✏️ Learning Rate - a floating-point number that tells the optimization algorithm the step size for the iteration while moving toward a minimum of a loss function. If the learning rate is too low, the model can take a long time to converge. If the learning rate is too high, the model may never converge.
#aibasics
👍3🔥1