𝗛𝗼𝘄 𝗱𝗼𝗲𝘀 𝗗𝗶𝘀𝗰𝗼𝗿𝗱 𝗵𝗮𝗻𝗱𝗹𝗲 𝗮 𝗺𝗶𝗹𝗹𝗶𝗼𝗻 𝗼𝗻𝗹𝗶𝗻𝗲 𝘂𝘀𝗲𝗿𝘀 𝗶𝗻 𝗮 𝘀𝗶𝗻𝗴𝗹𝗲 𝘀𝗲𝗿𝘃𝗲𝗿?
As time passed, the overall size of Discord's user base, including its most prominent communities, has grown massively. This affected servers that started to slow down and hit their throughput limits. So, they needed to scale individual Discord servers from tens of thousands to millions of concurrent users.
Whenever someone sends a message on Discord or joins a channel, they need to update the date UI of everyone online on that server. They call that server a "𝗴𝘂𝗶𝗹𝗱," which runs in a 𝘀𝗶𝗻𝗴𝗹𝗲 𝗘𝗹𝗶𝘅𝗶𝗿 𝗽𝗿𝗼𝗰𝗲𝘀𝘀, while there is another process (a "𝘀𝗲𝘀𝘀𝗶𝗼𝗻") for each connected client. The guild process tracks sessions of users who are members of that guild and are responsible for actions to those sessions. When sessions get updates, forward them to the web socket socket to the client.
The main issue is that 𝗮 𝘀𝗶𝗻𝗴𝗹𝗲 𝗺𝗲𝘀𝘀𝗮𝗴𝗲 𝗻𝗲𝗲𝗱𝘀 𝘁𝗼 𝗴𝗼 𝘁𝗼 𝘁𝗵𝗲 𝗻𝘂𝗺𝗯𝗲𝗿 𝗼𝗳 𝗽𝗲𝗼𝗽𝗹𝗲 𝗼𝗻𝗹𝗶𝗻𝗲 on that server, which means if a server has 1000 people online and they all send a message once, that's 1 million notifications.
As time passed, the overall size of Discord's user base, including its most prominent communities, has grown massively. This affected servers that started to slow down and hit their throughput limits. So, they needed to scale individual Discord servers from tens of thousands to millions of concurrent users.
Whenever someone sends a message on Discord or joins a channel, they need to update the date UI of everyone online on that server. They call that server a "𝗴𝘂𝗶𝗹𝗱," which runs in a 𝘀𝗶𝗻𝗴𝗹𝗲 𝗘𝗹𝗶𝘅𝗶𝗿 𝗽𝗿𝗼𝗰𝗲𝘀𝘀, while there is another process (a "𝘀𝗲𝘀𝘀𝗶𝗼𝗻") for each connected client. The guild process tracks sessions of users who are members of that guild and are responsible for actions to those sessions. When sessions get updates, forward them to the web socket socket to the client.
The main issue is that 𝗮 𝘀𝗶𝗻𝗴𝗹𝗲 𝗺𝗲𝘀𝘀𝗮𝗴𝗲 𝗻𝗲𝗲𝗱𝘀 𝘁𝗼 𝗴𝗼 𝘁𝗼 𝘁𝗵𝗲 𝗻𝘂𝗺𝗯𝗲𝗿 𝗼𝗳 𝗽𝗲𝗼𝗽𝗹𝗲 𝗼𝗻𝗹𝗶𝗻𝗲 on that server, which means if a server has 1000 people online and they all send a message once, that's 1 million notifications.
👍51❤1👎1
So, how they solved it? 𝗧𝗵𝗲𝘆 𝗳𝗶𝗿𝘀𝘁 𝘁𝗿𝗶𝗲𝗱 𝘁𝗼 𝘂𝗻𝗱𝗲𝗿𝘀𝘁𝗮𝗻𝗱 𝗵𝗼𝘄 𝘁𝗵𝗲 𝘀𝘆𝘀𝘁𝗲𝗺 𝗽𝗲𝗿𝗳𝗼𝗿𝗺𝘀. They tracked what Elixir processes were doing, if they were stuck waiting on something, etc. They recorded the event types, how many of each kind of message they received, and their processing times. In addition, they tried to understand how much memory they use, the performances of garbage collectors, etc.
After the analysis, they 𝗰𝗿𝗲𝗮𝘁𝗲𝗱 𝘁𝗵𝗲 𝗳𝗼𝗹𝗹𝗼𝘄𝗶𝗻𝗴 𝘀𝘁𝗿𝗮𝘁𝗲𝗴𝘆:
𝟭. 𝗣𝗮𝘀𝘀𝗶𝘃𝗲 𝘀𝗲𝘀𝘀𝗶𝗼𝗻𝘀: Discord significantly reduced the amount of data processed and sent by differentiating between active and passive user connections, cutting the fanout work by 90% for large servers.
𝟮. 𝗥𝗲𝗹𝗮𝘆𝘀: Implementing a relay system (read - multithreading) allowed Discord to split the fanout process across multiple machines, enabling a single guild to utilize more resources and support more prominent communities. Relays maintain connections to the sessions instead of the guild and are responsible for doing fanout with permission checks.
𝟯. 𝗪𝗼𝗿𝗸𝗲𝗿 𝗽𝗿𝗼𝗰𝗲𝘀𝘀𝗲𝘀 𝗮𝗻𝗱 𝗘𝗧𝗦: To maintain server responsiveness, Discord employed worker processes and Erlang Term Storage (ETS) for operations requiring iteration over large sets of members, thus avoiding bottlenecks in the guild process. ETS is an in-memory database that supports the ability of multiple Elixir processes to access it safely. This enables the creation of a new worker process and passes the ETS table so this process can run expensive operations and offload the central guild server.
🔗 https://discord.com/blog/maxjourney-pushing-discords-limits-with-a-million-plus-online-users-in-a-single-server
After the analysis, they 𝗰𝗿𝗲𝗮𝘁𝗲𝗱 𝘁𝗵𝗲 𝗳𝗼𝗹𝗹𝗼𝘄𝗶𝗻𝗴 𝘀𝘁𝗿𝗮𝘁𝗲𝗴𝘆:
𝟭. 𝗣𝗮𝘀𝘀𝗶𝘃𝗲 𝘀𝗲𝘀𝘀𝗶𝗼𝗻𝘀: Discord significantly reduced the amount of data processed and sent by differentiating between active and passive user connections, cutting the fanout work by 90% for large servers.
𝟮. 𝗥𝗲𝗹𝗮𝘆𝘀: Implementing a relay system (read - multithreading) allowed Discord to split the fanout process across multiple machines, enabling a single guild to utilize more resources and support more prominent communities. Relays maintain connections to the sessions instead of the guild and are responsible for doing fanout with permission checks.
𝟯. 𝗪𝗼𝗿𝗸𝗲𝗿 𝗽𝗿𝗼𝗰𝗲𝘀𝘀𝗲𝘀 𝗮𝗻𝗱 𝗘𝗧𝗦: To maintain server responsiveness, Discord employed worker processes and Erlang Term Storage (ETS) for operations requiring iteration over large sets of members, thus avoiding bottlenecks in the guild process. ETS is an in-memory database that supports the ability of multiple Elixir processes to access it safely. This enables the creation of a new worker process and passes the ETS table so this process can run expensive operations and offload the central guild server.
Please open Telegram to view this post
VIEW IN TELEGRAM
👍47❤2
𝗛𝗼𝘄 𝘁𝗼 𝗰𝗼𝗱𝗲 𝘄𝗶𝘁𝗵 𝗚𝗶𝘁𝗛𝘂𝗯 𝗖𝗼𝗽𝗶𝗹𝗼𝘁?
A recent study by GitHub and Microsoft discovered that AI now authors 46% of new code. They also found that overall developer productivity surged by 55%, leading to more efficient coding processes. When we talk about AI-powered coding, we mainly talk about GitHub Copilot.
But 𝗵𝗼𝘄 𝗚𝗶𝘁𝗛𝘂𝗯 𝗖𝗼𝗽𝗶𝗹𝗼𝘁 𝘄𝗼𝗿𝗸𝘀?
The process goes in the following steps:
𝟭. 𝗦𝗲𝗰𝘂𝗿𝗲 𝗽𝗿𝗼𝗺𝗽𝘁 𝘁𝗿𝗮𝗻𝘀𝗺𝗶𝘀𝘀𝗶𝗼𝗻: Your prompts are securely sent to Copilot, ensuring data privacy.
𝟮. 𝗖𝗼𝗻𝘁𝗲𝘅𝘁𝘂𝗮𝗹 𝘂𝗻𝗱𝗲𝗿𝘀𝘁𝗮𝗻𝗱𝗶𝗻𝗴: Copilot analyzes the code around your cursor, the file type, and other open files to offer relevant suggestions.
𝟯. 𝗖𝗼𝗻𝘁𝗲𝗻𝘁 𝗳𝗶𝗹𝘁𝗲𝗿𝗶𝗻𝗴: It filters out personal data and inappropriate content, focusing solely on generating helpful code.
𝟰. 𝗖𝗼𝗱𝗲 𝗴𝗲𝗻𝗲𝗿𝗮𝘁𝗶𝗼𝗻: Based on the intent identified in your prompts, Copilot crafts code suggestions that align with your coding style and project standards.
𝟱. 𝗨𝘀𝗲𝗿 𝗶𝗻𝘁𝗲𝗿𝗮𝗰𝘁𝗶𝗼𝗻: Here, we can decide whether to use, tweak, or reject Copilot's suggestions.
𝟲. 𝗙𝗲𝗲𝗱𝗯𝗮𝗰𝗸 𝗹𝗼𝗼𝗽: Copilot learns from your interactions, improving its suggestions. Every time you tweak or reject its ideas, he knows from it. It employs techniques like zero-shot (asking without examples), one-shot (asking with an example), and few-shot learning (providing multiple examples) to adapt to our instructions, whether you provide examples or not.
𝟳. 𝗣𝗿𝗼𝗺𝗽𝘁 𝗵𝗶𝘀𝘁𝗼𝗿𝘆 𝗿𝗲𝘁𝗲𝗻𝘁𝗶𝗼𝗻: It remembers past prompts and interactions, making future suggestions more accurate.
A recent study by GitHub and Microsoft discovered that AI now authors 46% of new code. They also found that overall developer productivity surged by 55%, leading to more efficient coding processes. When we talk about AI-powered coding, we mainly talk about GitHub Copilot.
But 𝗵𝗼𝘄 𝗚𝗶𝘁𝗛𝘂𝗯 𝗖𝗼𝗽𝗶𝗹𝗼𝘁 𝘄𝗼𝗿𝗸𝘀?
The process goes in the following steps:
𝟭. 𝗦𝗲𝗰𝘂𝗿𝗲 𝗽𝗿𝗼𝗺𝗽𝘁 𝘁𝗿𝗮𝗻𝘀𝗺𝗶𝘀𝘀𝗶𝗼𝗻: Your prompts are securely sent to Copilot, ensuring data privacy.
𝟮. 𝗖𝗼𝗻𝘁𝗲𝘅𝘁𝘂𝗮𝗹 𝘂𝗻𝗱𝗲𝗿𝘀𝘁𝗮𝗻𝗱𝗶𝗻𝗴: Copilot analyzes the code around your cursor, the file type, and other open files to offer relevant suggestions.
𝟯. 𝗖𝗼𝗻𝘁𝗲𝗻𝘁 𝗳𝗶𝗹𝘁𝗲𝗿𝗶𝗻𝗴: It filters out personal data and inappropriate content, focusing solely on generating helpful code.
𝟰. 𝗖𝗼𝗱𝗲 𝗴𝗲𝗻𝗲𝗿𝗮𝘁𝗶𝗼𝗻: Based on the intent identified in your prompts, Copilot crafts code suggestions that align with your coding style and project standards.
𝟱. 𝗨𝘀𝗲𝗿 𝗶𝗻𝘁𝗲𝗿𝗮𝗰𝘁𝗶𝗼𝗻: Here, we can decide whether to use, tweak, or reject Copilot's suggestions.
𝟲. 𝗙𝗲𝗲𝗱𝗯𝗮𝗰𝗸 𝗹𝗼𝗼𝗽: Copilot learns from your interactions, improving its suggestions. Every time you tweak or reject its ideas, he knows from it. It employs techniques like zero-shot (asking without examples), one-shot (asking with an example), and few-shot learning (providing multiple examples) to adapt to our instructions, whether you provide examples or not.
𝟳. 𝗣𝗿𝗼𝗺𝗽𝘁 𝗵𝗶𝘀𝘁𝗼𝗿𝘆 𝗿𝗲𝘁𝗲𝗻𝘁𝗶𝗼𝗻: It remembers past prompts and interactions, making future suggestions more accurate.
👍74👎10❤2
Challenging programming projects you should try:
🔗 https://jamesg.blog/2024/02/28/programming-projects/https://jamesg.blog/2024/02/28/programming-projects/
Please open Telegram to view this post
VIEW IN TELEGRAM
👍41❤1
Use KeePassXC to sign your git commits
🔗 https://code.mendhak.com/keepassxc-sign-git-commit-with-ssh/
Please open Telegram to view this post
VIEW IN TELEGRAM
👍40
A library for building fast, reliable and evolvable network services
💻 https://github.com/cloudflare/pingora/tree/main
Please open Telegram to view this post
VIEW IN TELEGRAM
👍45❤1
This media is not supported in your browser
VIEW IN TELEGRAM
Direct-a-Video: driving Video Generation
Direct-a-Video is a text-to-video generation framework that allows users to individually or jointly control the camera movement and/or object motion. Authors: City University of HK, Kuaishou Tech & Tianjin.
𝐇𝐢𝐠𝐡𝐥𝐢𝐠𝐡𝐭𝐬:
✅ Decoupling camera/object motion in gen-AI
✅ Allowing users to independently/jointly control
✅ Novel temporal cross-attention for cam motion
✅ Training-free spatial cross-attention for objects
✅ Driving object generation via bounding boxes
Paper: https://arxiv.org/pdf/2402.03162.pdf
Project: https://direct-a-video.github.io/
Direct-a-Video is a text-to-video generation framework that allows users to individually or jointly control the camera movement and/or object motion. Authors: City University of HK, Kuaishou Tech & Tianjin.
𝐇𝐢𝐠𝐡𝐥𝐢𝐠𝐡𝐭𝐬:
Paper: https://arxiv.org/pdf/2402.03162.pdf
Project: https://direct-a-video.github.io/
Please open Telegram to view this post
VIEW IN TELEGRAM
👍34❤3👎3
This tutorial will show you how to create an interactive table using Flet.
Moreover, with search and filtering functions, which is very cool
Please open Telegram to view this post
VIEW IN TELEGRAM
Please open Telegram to view this post
VIEW IN TELEGRAM
👍39
Harvard CS50 – Free Computer Science Course (2023 Edition)
Here are the lectures included in this course:
Lecture 0 - Scratch
Lecture 1 - C
Lecture 2 - Arrays
Lecture 3 - Algorithms
Lecture 4 - Memory
Lecture 5 - Data Structures
Lecture 6 - Python
Lecture 7 - SQL
Lecture 8 - HTML, CSS, JavaScript
Lecture 9 - Flask
Lecture 10 - Emoji
Cybersecurity
https://www.freecodecamp.org/news/harvard-university-cs50-computer-science-course-2023/
Here are the lectures included in this course:
Lecture 0 - Scratch
Lecture 1 - C
Lecture 2 - Arrays
Lecture 3 - Algorithms
Lecture 4 - Memory
Lecture 5 - Data Structures
Lecture 6 - Python
Lecture 7 - SQL
Lecture 8 - HTML, CSS, JavaScript
Lecture 9 - Flask
Lecture 10 - Emoji
Cybersecurity
https://www.freecodecamp.org/news/harvard-university-cs50-computer-science-course-2023/
freeCodeCamp.org
Harvard CS50 – Free Computer Science Course (2023 Edition)
Harvard University's CS50 is one of the most popular beginner computer science courses in the world. We just released the entire 26-hour CS50 course on the freeCodeCamp.org YouTube channel. This is the updated 2023 edition of the course. Dr. David J....
👍168👎5❤3😁1
Uber Migrates 1 Trillion Records from DynamoDB to LedgerStore to Save $6 Million Annually
🔗 https://www.infoq.com/news/2024/05/uber-dynamodb-ledgerstore/
Please open Telegram to view this post
VIEW IN TELEGRAM
👍39
New Regular expression (Regex) functions in Excel
🔗 https://insider.microsoft365.com/en-us/blog/new-regular-expression-regex-functions-in-excel
Please open Telegram to view this post
VIEW IN TELEGRAM
👍14
SQL at 50: What’s next for the structured query language?
🔗 https://www.infoworld.com/article/3715453/sql-at-50-whats-next-for-the-structured-query-language.html
Please open Telegram to view this post
VIEW IN TELEGRAM
👍26
It's time for Stack Overflow's annual Developer Survey! From your preferred programming languages, the new tech you want to explore, and much, much more, we're calling all who code to weigh in.
Take the survey: https://stackoverflow.az1.qualtrics.com/jfe/form/SV_6rJVT6XXsfTo1JI?utm_medium=social&utm_source=twitter&utm_campaign=dev-survey-2024&utm_content=
Take the survey: https://stackoverflow.az1.qualtrics.com/jfe/form/SV_6rJVT6XXsfTo1JI?utm_medium=social&utm_source=twitter&utm_campaign=dev-survey-2024&utm_content=
👍40👎3❤1
#bugs
Github CoPilot Re-Enables Itself After Being Disabled
🔗 https://github.com/microsoft/vscode-copilot-release/issues/1248
Github CoPilot Re-Enables Itself After Being Disabled
Please open Telegram to view this post
VIEW IN TELEGRAM
👍36
What's New in C# 13: Enhanced Params, Performance Boosts, and New Extension Types
🔗 https://www.infoq.com/news/2024/05/csharp-13-preview/
Please open Telegram to view this post
VIEW IN TELEGRAM
👍36🔥1
40 Tips From A Senior to a Junior developer.
Part I:
1. Master the Basics: Ensure you have a strong grasp of fundamental programming concepts.
2. Write Readable Code: Prioritize readability over cleverness.
3. Use Version Control: Learn Git and make frequent commits with meaningful messages.
4. Understand Your Tools: Get familiar with your IDE, debugger, and terminal.
5. Comment Wisely: Use comments to explain why, not what.
6. Practice Problem-Solving: Regularly work on coding challenges to improve your skills.
7. Learn Design Patterns: Study common design patterns and their use cases.
8. Ask Questions: Don’t hesitate to seek help when stuck.
9. Read Documentation: Thoroughly read the documentation for libraries and frameworks you use.
10. Write Tests: Develop a habit of writing unit and integration tests.
11. Refactor Regularly: Continuously improve your codebase.
12. Learn Debugging: Master the art of debugging to solve issues efficiently.
13. Understand Algorithms: Study common algorithms and their complexities.
14. Keep Learning: Stay updated with new technologies and industry trends.
15. Build Projects: Work on side projects to apply what you learn.
16. Use Code Reviews: Participate in code reviews to learn from others and improve your code.
17. Learn SQL: Know how to work with databases and write efficient queries.
18. Learn about the business domain you’re working in.
19. Build relationships with other developers.
20. Stay Organized: Use task management tools to keep track of your work.
Part I:
1. Master the Basics: Ensure you have a strong grasp of fundamental programming concepts.
2. Write Readable Code: Prioritize readability over cleverness.
3. Use Version Control: Learn Git and make frequent commits with meaningful messages.
4. Understand Your Tools: Get familiar with your IDE, debugger, and terminal.
5. Comment Wisely: Use comments to explain why, not what.
6. Practice Problem-Solving: Regularly work on coding challenges to improve your skills.
7. Learn Design Patterns: Study common design patterns and their use cases.
8. Ask Questions: Don’t hesitate to seek help when stuck.
9. Read Documentation: Thoroughly read the documentation for libraries and frameworks you use.
10. Write Tests: Develop a habit of writing unit and integration tests.
11. Refactor Regularly: Continuously improve your codebase.
12. Learn Debugging: Master the art of debugging to solve issues efficiently.
13. Understand Algorithms: Study common algorithms and their complexities.
14. Keep Learning: Stay updated with new technologies and industry trends.
15. Build Projects: Work on side projects to apply what you learn.
16. Use Code Reviews: Participate in code reviews to learn from others and improve your code.
17. Learn SQL: Know how to work with databases and write efficient queries.
18. Learn about the business domain you’re working in.
19. Build relationships with other developers.
20. Stay Organized: Use task management tools to keep track of your work.
👍172❤3👎3
40 Tips From A Senior to a Junior developer.
Part II:
21. Optimize Code: Write efficient code but avoid premature optimization.
22. Understand basic security practices.
23. Maintain good documentation for your projects.
24. Follow Best Practices: Adhere to industry best practices and coding standards.
25. Work on Soft Skills: Improve communication and teamwork skills.
26. Stay Humble: Be open to feedback and willing to admit mistakes.
27. Contribute to Open Source: Participate in open source projects to gain experience.
28. Practice Pair Programming: Collaborate with peers to solve problems together.
29. Understand the Full Stack: Gain knowledge of both front and back-end development.
30. Automate Repetitive Tasks: Use noscripts and tools to automate repetitive tasks.
31. Monitor Performance: Use tools to monitor and improve application performance.
32. Keep Code Simple: Aim for simplicity in your code.
33. Learn from Mistakes: Analyze and learn from your errors.
34. Set Goals: Define and work towards personal and professional goals.
35. Be Patient: Development takes time; be patient with your learning process.
36. Read Books: Invest time in reading programming and technology books.
37. Join Communities: Engage in developer communities.
38. Stay Healthy: Maintain a healthy work-life balance.
39. Use Modern Practices: Embrace agile, DevOps, and continuous integration/deployment practices.
40. Mentor Others
Part II:
21. Optimize Code: Write efficient code but avoid premature optimization.
22. Understand basic security practices.
23. Maintain good documentation for your projects.
24. Follow Best Practices: Adhere to industry best practices and coding standards.
25. Work on Soft Skills: Improve communication and teamwork skills.
26. Stay Humble: Be open to feedback and willing to admit mistakes.
27. Contribute to Open Source: Participate in open source projects to gain experience.
28. Practice Pair Programming: Collaborate with peers to solve problems together.
29. Understand the Full Stack: Gain knowledge of both front and back-end development.
30. Automate Repetitive Tasks: Use noscripts and tools to automate repetitive tasks.
31. Monitor Performance: Use tools to monitor and improve application performance.
32. Keep Code Simple: Aim for simplicity in your code.
33. Learn from Mistakes: Analyze and learn from your errors.
34. Set Goals: Define and work towards personal and professional goals.
35. Be Patient: Development takes time; be patient with your learning process.
36. Read Books: Invest time in reading programming and technology books.
37. Join Communities: Engage in developer communities.
38. Stay Healthy: Maintain a healthy work-life balance.
39. Use Modern Practices: Embrace agile, DevOps, and continuous integration/deployment practices.
40. Mentor Others
👍149❤2