What Actually Makes You Senior
https://www.reddit.com/r/programming/comments/1p6iql6/what_actually_makes_you_senior/
submitted by /u/Acceptable-Courage-9 (https://www.reddit.com/user/Acceptable-Courage-9)
[link] (https://terriblesoftware.org/2025/11/25/what-actually-makes-you-senior/) [comments] (https://www.reddit.com/r/programming/comments/1p6iql6/what_actually_makes_you_senior/)
https://www.reddit.com/r/programming/comments/1p6iql6/what_actually_makes_you_senior/
submitted by /u/Acceptable-Courage-9 (https://www.reddit.com/user/Acceptable-Courage-9)
[link] (https://terriblesoftware.org/2025/11/25/what-actually-makes-you-senior/) [comments] (https://www.reddit.com/r/programming/comments/1p6iql6/what_actually_makes_you_senior/)
System Design Interview: Design Twitter/X Timeline - A Frontend Deep Dive
https://www.reddit.com/r/programming/comments/1p6kt64/system_design_interview_design_twitterx_timeline/
submitted by /u/sdxyz42 (https://www.reddit.com/user/sdxyz42)
[link] (https://newsletter.systemdesign.one/p/system-design-interview-twitter) [comments] (https://www.reddit.com/r/programming/comments/1p6kt64/system_design_interview_design_twitterx_timeline/)
https://www.reddit.com/r/programming/comments/1p6kt64/system_design_interview_design_twitterx_timeline/
submitted by /u/sdxyz42 (https://www.reddit.com/user/sdxyz42)
[link] (https://newsletter.systemdesign.one/p/system-design-interview-twitter) [comments] (https://www.reddit.com/r/programming/comments/1p6kt64/system_design_interview_design_twitterx_timeline/)
VGG19 Transfer Learning Explained for Beginners
https://www.reddit.com/r/programming/comments/1p6lhiz/vgg19_transfer_learning_explained_for_beginners/
<!-- SC_OFF -->For anyone studying transfer learning and VGG19 for image classification, this tutorial walks through a complete example using an aircraft images dataset. It explains why VGG19 is a suitable backbone for this task, how to adapt the final layers for a new set of aircraft classes, and demonstrates the full training and evaluation process step by step. written explanation with code: https://eranfeit.net/vgg19-transfer-learning-explained-for-beginners/ video explanation: https://youtu.be/exaEeDfbFuI?si=C0o88kE-UvtLEhBn This material is for educational purposes only, and thoughtful, constructive feedback is welcome. <!-- SC_ON --> submitted by /u/Feitgemel (https://www.reddit.com/user/Feitgemel)
[link] (https://eranfeit.net/vgg19-transfer-learning-explained-for-beginners/) [comments] (https://www.reddit.com/r/programming/comments/1p6lhiz/vgg19_transfer_learning_explained_for_beginners/)
https://www.reddit.com/r/programming/comments/1p6lhiz/vgg19_transfer_learning_explained_for_beginners/
<!-- SC_OFF -->For anyone studying transfer learning and VGG19 for image classification, this tutorial walks through a complete example using an aircraft images dataset. It explains why VGG19 is a suitable backbone for this task, how to adapt the final layers for a new set of aircraft classes, and demonstrates the full training and evaluation process step by step. written explanation with code: https://eranfeit.net/vgg19-transfer-learning-explained-for-beginners/ video explanation: https://youtu.be/exaEeDfbFuI?si=C0o88kE-UvtLEhBn This material is for educational purposes only, and thoughtful, constructive feedback is welcome. <!-- SC_ON --> submitted by /u/Feitgemel (https://www.reddit.com/user/Feitgemel)
[link] (https://eranfeit.net/vgg19-transfer-learning-explained-for-beginners/) [comments] (https://www.reddit.com/r/programming/comments/1p6lhiz/vgg19_transfer_learning_explained_for_beginners/)
Readonly Character Ranges in Editors Are a Big Deal
https://www.reddit.com/r/programming/comments/1p6ln0s/readonly_character_ranges_in_editors_are_a_big/
submitted by /u/alexeyr (https://www.reddit.com/user/alexeyr)
[link] (https://matklad.github.io/2025/11/10/readonly-characters.html) [comments] (https://www.reddit.com/r/programming/comments/1p6ln0s/readonly_character_ranges_in_editors_are_a_big/)
https://www.reddit.com/r/programming/comments/1p6ln0s/readonly_character_ranges_in_editors_are_a_big/
submitted by /u/alexeyr (https://www.reddit.com/user/alexeyr)
[link] (https://matklad.github.io/2025/11/10/readonly-characters.html) [comments] (https://www.reddit.com/r/programming/comments/1p6ln0s/readonly_character_ranges_in_editors_are_a_big/)
Chimera - an innovative (?) db interface
https://www.reddit.com/r/programming/comments/1p6ln5j/chimera_an_innovative_db_interface/
<!-- SC_OFF -->Today I’m sharing the first architectural preview of Chimera, a project I’ve been building in my free time to both explore system architecture and sharpen my modern C++ skills. Chimera is designed to simplify interaction with heterogeneous databases by offering a single, consistent interface for PostgreSQL, MongoDB, and Oracle. It currently provides: Autogenerated DAO classes from existing DB schemas A speculative in-memory model for each table/collection Multiple synchronization modes (sync, async, real-time) It is aimed to be used in two ways: - Embedded Mode (linked directly into a CSCI) - Hosted Server Mode (exposed via REST API) The goal is to reduce boilerplate, standardize data access, and make multi-DB environments easier to manage — especially in complex, high-reliability systems. I’m sharing the first architecture diagram below ⬇️ and I’d genuinely appreciate feedback, ideas, or constructive criticism. Your insights will help me guide its next steps while continuing to grow as a C++ engineer. Thanks in advance! Ps: At the moment the name is chimera for the Three Adapters, I shall find another mythical animal if I decide to add another one 😂 <!-- SC_ON --> submitted by /u/Altruistic_Pizza_766 (https://www.reddit.com/user/Altruistic_Pizza_766)
[link] (https://www.linkedin.com/posts/marco-pecorella-20b005183_cplusplus-softwarearchitecture-backend-activity-7399115638218240000-vTrn?utm_source=share&utm_medium=member_ios&rcm=ACoAACs_0usBleIFhb1GQ4bxucSy5eHFA94T4aY) [comments] (https://www.reddit.com/r/programming/comments/1p6ln5j/chimera_an_innovative_db_interface/)
https://www.reddit.com/r/programming/comments/1p6ln5j/chimera_an_innovative_db_interface/
<!-- SC_OFF -->Today I’m sharing the first architectural preview of Chimera, a project I’ve been building in my free time to both explore system architecture and sharpen my modern C++ skills. Chimera is designed to simplify interaction with heterogeneous databases by offering a single, consistent interface for PostgreSQL, MongoDB, and Oracle. It currently provides: Autogenerated DAO classes from existing DB schemas A speculative in-memory model for each table/collection Multiple synchronization modes (sync, async, real-time) It is aimed to be used in two ways: - Embedded Mode (linked directly into a CSCI) - Hosted Server Mode (exposed via REST API) The goal is to reduce boilerplate, standardize data access, and make multi-DB environments easier to manage — especially in complex, high-reliability systems. I’m sharing the first architecture diagram below ⬇️ and I’d genuinely appreciate feedback, ideas, or constructive criticism. Your insights will help me guide its next steps while continuing to grow as a C++ engineer. Thanks in advance! Ps: At the moment the name is chimera for the Three Adapters, I shall find another mythical animal if I decide to add another one 😂 <!-- SC_ON --> submitted by /u/Altruistic_Pizza_766 (https://www.reddit.com/user/Altruistic_Pizza_766)
[link] (https://www.linkedin.com/posts/marco-pecorella-20b005183_cplusplus-softwarearchitecture-backend-activity-7399115638218240000-vTrn?utm_source=share&utm_medium=member_ios&rcm=ACoAACs_0usBleIFhb1GQ4bxucSy5eHFA94T4aY) [comments] (https://www.reddit.com/r/programming/comments/1p6ln5j/chimera_an_innovative_db_interface/)
The tech debt elephant: A product perspective
https://www.reddit.com/r/programming/comments/1p6nca9/the_tech_debt_elephant_a_product_perspective/
submitted by /u/ArtisticProgrammer11 (https://www.reddit.com/user/ArtisticProgrammer11)
[link] (https://www.hyperact.co.uk/blog/tech-debt-a-product-perspective) [comments] (https://www.reddit.com/r/programming/comments/1p6nca9/the_tech_debt_elephant_a_product_perspective/)
https://www.reddit.com/r/programming/comments/1p6nca9/the_tech_debt_elephant_a_product_perspective/
submitted by /u/ArtisticProgrammer11 (https://www.reddit.com/user/ArtisticProgrammer11)
[link] (https://www.hyperact.co.uk/blog/tech-debt-a-product-perspective) [comments] (https://www.reddit.com/r/programming/comments/1p6nca9/the_tech_debt_elephant_a_product_perspective/)
Announcing Unison 1.0
https://www.reddit.com/r/programming/comments/1p6o8pe/announcing_unison_10/
submitted by /u/Successful_Answer_66 (https://www.reddit.com/user/Successful_Answer_66)
[link] (https://www.unison-lang.org/unison-1-0/) [comments] (https://www.reddit.com/r/programming/comments/1p6o8pe/announcing_unison_10/)
https://www.reddit.com/r/programming/comments/1p6o8pe/announcing_unison_10/
submitted by /u/Successful_Answer_66 (https://www.reddit.com/user/Successful_Answer_66)
[link] (https://www.unison-lang.org/unison-1-0/) [comments] (https://www.reddit.com/r/programming/comments/1p6o8pe/announcing_unison_10/)
Generalizing the Shunting Yard Algorithm Part 3
https://www.reddit.com/r/programming/comments/1p6pp16/generalizing_the_shunting_yard_algorithm_part_3/
submitted by /u/error-errorfruituser (https://www.reddit.com/user/error-errorfruituser)
[link] (https://syntax-slander.hashnode.dev/generalizing-the-shunting-yard-algorithm-part-3) [comments] (https://www.reddit.com/r/programming/comments/1p6pp16/generalizing_the_shunting_yard_algorithm_part_3/)
https://www.reddit.com/r/programming/comments/1p6pp16/generalizing_the_shunting_yard_algorithm_part_3/
submitted by /u/error-errorfruituser (https://www.reddit.com/user/error-errorfruituser)
[link] (https://syntax-slander.hashnode.dev/generalizing-the-shunting-yard-algorithm-part-3) [comments] (https://www.reddit.com/r/programming/comments/1p6pp16/generalizing_the_shunting_yard_algorithm_part_3/)
Simulating Unix v7 on PDP-11 with In-Depth Explanations
https://www.reddit.com/r/programming/comments/1p6ujl6/simulating_unix_v7_on_pdp11_with_indepth/
<!-- SC_OFF -->I hope I did a good job uncovering the technical details and the relevant history. <!-- SC_ON --> submitted by /u/baziotis (https://www.reddit.com/user/baziotis)
[link] (https://sbaziotis.com/uncat/simulating-pdp-11-with-in-depth-explanations.html) [comments] (https://www.reddit.com/r/programming/comments/1p6ujl6/simulating_unix_v7_on_pdp11_with_indepth/)
https://www.reddit.com/r/programming/comments/1p6ujl6/simulating_unix_v7_on_pdp11_with_indepth/
<!-- SC_OFF -->I hope I did a good job uncovering the technical details and the relevant history. <!-- SC_ON --> submitted by /u/baziotis (https://www.reddit.com/user/baziotis)
[link] (https://sbaziotis.com/uncat/simulating-pdp-11-with-in-depth-explanations.html) [comments] (https://www.reddit.com/r/programming/comments/1p6ujl6/simulating_unix_v7_on_pdp11_with_indepth/)
Reading Code Is a Useful (but sometimes overlooked) Skill
https://www.reddit.com/r/programming/comments/1p6w0ya/reading_code_is_a_useful_but_sometimes_overlooked/
<!-- SC_OFF -->This video goes into why becoming better at reading code can lead to becoming a better programmer overall and why it might be a skill that can be easy to overlook. <!-- SC_ON --> submitted by /u/stumblingtowards (https://www.reddit.com/user/stumblingtowards)
[link] (https://youtu.be/Apf9OuU7jdQ) [comments] (https://www.reddit.com/r/programming/comments/1p6w0ya/reading_code_is_a_useful_but_sometimes_overlooked/)
https://www.reddit.com/r/programming/comments/1p6w0ya/reading_code_is_a_useful_but_sometimes_overlooked/
<!-- SC_OFF -->This video goes into why becoming better at reading code can lead to becoming a better programmer overall and why it might be a skill that can be easy to overlook. <!-- SC_ON --> submitted by /u/stumblingtowards (https://www.reddit.com/user/stumblingtowards)
[link] (https://youtu.be/Apf9OuU7jdQ) [comments] (https://www.reddit.com/r/programming/comments/1p6w0ya/reading_code_is_a_useful_but_sometimes_overlooked/)
The Evolution of std::optional - From Boost to C++26 - Steve Downey - CppCon 2025
https://www.reddit.com/r/programming/comments/1p71oz6/the_evolution_of_stdoptional_from_boost_to_c26/
submitted by /u/BlueGoliath (https://www.reddit.com/user/BlueGoliath)
[link] (https://www.youtube.com/watch?v=fTbTF0MUsPA) [comments] (https://www.reddit.com/r/programming/comments/1p71oz6/the_evolution_of_stdoptional_from_boost_to_c26/)
https://www.reddit.com/r/programming/comments/1p71oz6/the_evolution_of_stdoptional_from_boost_to_c26/
submitted by /u/BlueGoliath (https://www.reddit.com/user/BlueGoliath)
[link] (https://www.youtube.com/watch?v=fTbTF0MUsPA) [comments] (https://www.reddit.com/r/programming/comments/1p71oz6/the_evolution_of_stdoptional_from_boost_to_c26/)
Concept-based Generic Programming - Bjarne Stroustrup - CppCon 2025
https://www.reddit.com/r/programming/comments/1p71o1g/conceptbased_generic_programming_bjarne/
submitted by /u/BlueGoliath (https://www.reddit.com/user/BlueGoliath)
[link] (https://www.youtube.com/watch?v=VMGB75hsDQo) [comments] (https://www.reddit.com/r/programming/comments/1p71o1g/conceptbased_generic_programming_bjarne/)
https://www.reddit.com/r/programming/comments/1p71o1g/conceptbased_generic_programming_bjarne/
submitted by /u/BlueGoliath (https://www.reddit.com/user/BlueGoliath)
[link] (https://www.youtube.com/watch?v=VMGB75hsDQo) [comments] (https://www.reddit.com/r/programming/comments/1p71o1g/conceptbased_generic_programming_bjarne/)
Could C++ Developers Handle an ABI Break Today? - Luis Caro Campos - CppCon 2025
https://www.reddit.com/r/programming/comments/1p71pfx/could_c_developers_handle_an_abi_break_today_luis/
submitted by /u/BlueGoliath (https://www.reddit.com/user/BlueGoliath)
[link] (https://www.youtube.com/watch?v=VbSKnvldtbs) [comments] (https://www.reddit.com/r/programming/comments/1p71pfx/could_c_developers_handle_an_abi_break_today_luis/)
https://www.reddit.com/r/programming/comments/1p71pfx/could_c_developers_handle_an_abi_break_today_luis/
submitted by /u/BlueGoliath (https://www.reddit.com/user/BlueGoliath)
[link] (https://www.youtube.com/watch?v=VbSKnvldtbs) [comments] (https://www.reddit.com/r/programming/comments/1p71pfx/could_c_developers_handle_an_abi_break_today_luis/)
Agentic AI: The Hidden Engineering Traps in Building Autonomous Workflows
https://www.reddit.com/r/programming/comments/1p70pol/agentic_ai_the_hidden_engineering_traps_in/
<!-- SC_OFF -->Been building production agentic AI (agents that execute, not just chat) for the last year. Most demos crash in the real world because: RAG chunking/embedding choices destroy retrieval quality (we ended up hybrid Pinecone + FAISS + strict thresholds) ReAct loops run away and burn 10× tokens (memory caps + probabilistic exits fixed it) No real error handling for flaky APIs or garbage input (custom backoff + fallback LLMs) Governance logging without killing speed (event sourcing + replayable traces) End result: 60% less human touch, but only after sweating these details. What’s the nastiest production surprise you’ve hit with agentic builds? <!-- SC_ON --> submitted by /u/Futurismtechnologies (https://www.reddit.com/user/Futurismtechnologies)
[link] (https://www.futurismai.com/solutions/futurism-agentic-ai-solutions/?utm_source=reddit&utm_medium=social) [comments] (https://www.reddit.com/r/programming/comments/1p70pol/agentic_ai_the_hidden_engineering_traps_in/)
https://www.reddit.com/r/programming/comments/1p70pol/agentic_ai_the_hidden_engineering_traps_in/
<!-- SC_OFF -->Been building production agentic AI (agents that execute, not just chat) for the last year. Most demos crash in the real world because: RAG chunking/embedding choices destroy retrieval quality (we ended up hybrid Pinecone + FAISS + strict thresholds) ReAct loops run away and burn 10× tokens (memory caps + probabilistic exits fixed it) No real error handling for flaky APIs or garbage input (custom backoff + fallback LLMs) Governance logging without killing speed (event sourcing + replayable traces) End result: 60% less human touch, but only after sweating these details. What’s the nastiest production surprise you’ve hit with agentic builds? <!-- SC_ON --> submitted by /u/Futurismtechnologies (https://www.reddit.com/user/Futurismtechnologies)
[link] (https://www.futurismai.com/solutions/futurism-agentic-ai-solutions/?utm_source=reddit&utm_medium=social) [comments] (https://www.reddit.com/r/programming/comments/1p70pol/agentic_ai_the_hidden_engineering_traps_in/)
Generalizing the Shunting Yard Algorithm Part 2
https://www.reddit.com/r/programming/comments/1p72hjq/generalizing_the_shunting_yard_algorithm_part_2/
submitted by /u/error-errorfruituser (https://www.reddit.com/user/error-errorfruituser)
[link] (https://syntax-slander.hashnode.dev/generalizing-the-shunting-yard-algorithm-part-2) [comments] (https://www.reddit.com/r/programming/comments/1p72hjq/generalizing_the_shunting_yard_algorithm_part_2/)
https://www.reddit.com/r/programming/comments/1p72hjq/generalizing_the_shunting_yard_algorithm_part_2/
submitted by /u/error-errorfruituser (https://www.reddit.com/user/error-errorfruituser)
[link] (https://syntax-slander.hashnode.dev/generalizing-the-shunting-yard-algorithm-part-2) [comments] (https://www.reddit.com/r/programming/comments/1p72hjq/generalizing_the_shunting_yard_algorithm_part_2/)
Where do you draw the line between Python and a faster language?
https://www.reddit.com/r/programming/comments/1p730ev/where_do_you_draw_the_line_between_python_and_a/
<!-- SC_OFF -->I love Python for speed of development, readability, and libraries but I sometimes feel guilty when performance becomes a bottleneck. Curious how others decide when it’s time to switch to C++/Rust/Go instead of optimizing Python. Do you usually push Python as far as possible first, or do you switch early when performance matters? <!-- SC_ON --> submitted by /u/Creepy-Valuable-3685 (https://www.reddit.com/user/Creepy-Valuable-3685)
[link] (https://benchmarksgame-team.pages.debian.net/benchmarksgame/) [comments] (https://www.reddit.com/r/programming/comments/1p730ev/where_do_you_draw_the_line_between_python_and_a/)
https://www.reddit.com/r/programming/comments/1p730ev/where_do_you_draw_the_line_between_python_and_a/
<!-- SC_OFF -->I love Python for speed of development, readability, and libraries but I sometimes feel guilty when performance becomes a bottleneck. Curious how others decide when it’s time to switch to C++/Rust/Go instead of optimizing Python. Do you usually push Python as far as possible first, or do you switch early when performance matters? <!-- SC_ON --> submitted by /u/Creepy-Valuable-3685 (https://www.reddit.com/user/Creepy-Valuable-3685)
[link] (https://benchmarksgame-team.pages.debian.net/benchmarksgame/) [comments] (https://www.reddit.com/r/programming/comments/1p730ev/where_do_you_draw_the_line_between_python_and_a/)
Any one try out googles anti gravity thing yet?
https://www.reddit.com/r/programming/comments/1p73imu/any_one_try_out_googles_anti_gravity_thing_yet/
<!-- SC_OFF -->How is it? <!-- SC_ON --> submitted by /u/No-Doubt-6825 (https://www.reddit.com/user/No-Doubt-6825)
[link] (https://youtu.be/SVCBA-pBgt0?si=k2-rmdTZzSMhSLHA) [comments] (https://www.reddit.com/r/programming/comments/1p73imu/any_one_try_out_googles_anti_gravity_thing_yet/)
https://www.reddit.com/r/programming/comments/1p73imu/any_one_try_out_googles_anti_gravity_thing_yet/
<!-- SC_OFF -->How is it? <!-- SC_ON --> submitted by /u/No-Doubt-6825 (https://www.reddit.com/user/No-Doubt-6825)
[link] (https://youtu.be/SVCBA-pBgt0?si=k2-rmdTZzSMhSLHA) [comments] (https://www.reddit.com/r/programming/comments/1p73imu/any_one_try_out_googles_anti_gravity_thing_yet/)
Ilya Sutskever(Former Chief scientist at OpenAI) and Yann LeCun(former Meta Chief AI scientist) both say that just scaling LLMs won't give us any more useful results
https://www.reddit.com/r/programming/comments/1p7c5ul/ilya_sutskeverformer_chief_scientist_at_openai/
submitted by /u/Frequent-Football984 (https://www.reddit.com/user/Frequent-Football984)
[link] (https://www.abzglobal.net/web-development-blog/ilya-sutskever-yann-lecun-and-the-end-of-just-add-gpus) [comments] (https://www.reddit.com/r/programming/comments/1p7c5ul/ilya_sutskeverformer_chief_scientist_at_openai/)
https://www.reddit.com/r/programming/comments/1p7c5ul/ilya_sutskeverformer_chief_scientist_at_openai/
submitted by /u/Frequent-Football984 (https://www.reddit.com/user/Frequent-Football984)
[link] (https://www.abzglobal.net/web-development-blog/ilya-sutskever-yann-lecun-and-the-end-of-just-add-gpus) [comments] (https://www.reddit.com/r/programming/comments/1p7c5ul/ilya_sutskeverformer_chief_scientist_at_openai/)
Linus Torvalds vs. Ambiguous Abstractions: When a Helper Function Hides the Intent
https://www.reddit.com/r/programming/comments/1p7gj94/linus_torvalds_vs_ambiguous_abstractions_when_a/
submitted by /u/teivah (https://www.reddit.com/user/teivah)
[link] (https://read.thecoder.cafe/p/linus-torvalds-ambiguous-abstractions) [comments] (https://www.reddit.com/r/programming/comments/1p7gj94/linus_torvalds_vs_ambiguous_abstractions_when_a/)
https://www.reddit.com/r/programming/comments/1p7gj94/linus_torvalds_vs_ambiguous_abstractions_when_a/
submitted by /u/teivah (https://www.reddit.com/user/teivah)
[link] (https://read.thecoder.cafe/p/linus-torvalds-ambiguous-abstractions) [comments] (https://www.reddit.com/r/programming/comments/1p7gj94/linus_torvalds_vs_ambiguous_abstractions_when_a/)
Learned about vendor lock-in the hard way during my internship. does anyone talk about this at school?
https://www.reddit.com/r/programming/comments/1p7mvxw/learned_about_vendor_lockin_the_hard_way_during/
<!-- SC_OFF -->Just finished my internship at a fintech company (think of Robinhood) and learned something they definitely don't teach you in school. So I was on the data platform team and there was this whole drama while I was there. Basically the company went all-in on one of the big cloud vendors for their data stuff a couple years ago. Everything was integrated, their catalog, governance, the whole stack. Then out of nowhere the vendor raised prices like 40% with barely any notice. The engineering managers were losing their minds. The problem? All the company's metadata was trapped in the vendor's proprietary system. Like literally years of data lineage, access policies, everything. They couldn't just switch because migrating would take almost a year of engineering time. So they just had to pay lol. One of the senior engineers told me this happens all the time. He called it "the walled garden trap." Big companies like Databricks, Snowflake, AWS all want you locked into their ecosystem. Once you're in, switching costs are insane. Made me start reading about open source alternatives. Found this article that explains it way better than I can: https://medium.com/datastrato/if-youre-not-all-in-on-databricks-why-metadata-freedom-matters-35cc5b15b24e](https://medium.com/datastrato/if-youre-not-all-in-on-databricks-why-metadata-freedom-matters-35cc5b15b24e (https://medium.com/datastrato/if-youre-not-all-in-on-databricks-why-metadata-freedom-matters-35cc5b15b24e%5D(https://medium.com/datastrato/if-youre-not-all-in-on-databricks-why-metadata-freedom-matters-35cc5b15b24e) There's this Apache project called Gravitino that's basically an open source metadata layer so you're not dependent on one vendor. Kinda interesting from a systems design perspective tbh. Anyway this whole experience made me think: Do companies actually consider vendor lock-in when choosing tech stacks or do they just go with whatever's easiest? For those who've worked at startups vs big tech, is this more of a startup problem? Is open source infra actually viable for most companies or is it too much ops overhead? Anyone else learn stuff like this during internships that completely changed how you think about tech decisions? Feels like we learn algorithms and data structures but nobody talks about the actual business/infrastructure tradeoffs you deal with in industry. <!-- SC_ON --> submitted by /u/Icy-Perception0 (https://www.reddit.com/user/Icy-Perception0)
[link] (https://medium.com/datastrato/if-youre-not-all-in-on-databricks-why-metadata-freedom-matters-35cc5b15b24e) [comments] (https://www.reddit.com/r/programming/comments/1p7mvxw/learned_about_vendor_lockin_the_hard_way_during/)
https://www.reddit.com/r/programming/comments/1p7mvxw/learned_about_vendor_lockin_the_hard_way_during/
<!-- SC_OFF -->Just finished my internship at a fintech company (think of Robinhood) and learned something they definitely don't teach you in school. So I was on the data platform team and there was this whole drama while I was there. Basically the company went all-in on one of the big cloud vendors for their data stuff a couple years ago. Everything was integrated, their catalog, governance, the whole stack. Then out of nowhere the vendor raised prices like 40% with barely any notice. The engineering managers were losing their minds. The problem? All the company's metadata was trapped in the vendor's proprietary system. Like literally years of data lineage, access policies, everything. They couldn't just switch because migrating would take almost a year of engineering time. So they just had to pay lol. One of the senior engineers told me this happens all the time. He called it "the walled garden trap." Big companies like Databricks, Snowflake, AWS all want you locked into their ecosystem. Once you're in, switching costs are insane. Made me start reading about open source alternatives. Found this article that explains it way better than I can: https://medium.com/datastrato/if-youre-not-all-in-on-databricks-why-metadata-freedom-matters-35cc5b15b24e](https://medium.com/datastrato/if-youre-not-all-in-on-databricks-why-metadata-freedom-matters-35cc5b15b24e (https://medium.com/datastrato/if-youre-not-all-in-on-databricks-why-metadata-freedom-matters-35cc5b15b24e%5D(https://medium.com/datastrato/if-youre-not-all-in-on-databricks-why-metadata-freedom-matters-35cc5b15b24e) There's this Apache project called Gravitino that's basically an open source metadata layer so you're not dependent on one vendor. Kinda interesting from a systems design perspective tbh. Anyway this whole experience made me think: Do companies actually consider vendor lock-in when choosing tech stacks or do they just go with whatever's easiest? For those who've worked at startups vs big tech, is this more of a startup problem? Is open source infra actually viable for most companies or is it too much ops overhead? Anyone else learn stuff like this during internships that completely changed how you think about tech decisions? Feels like we learn algorithms and data structures but nobody talks about the actual business/infrastructure tradeoffs you deal with in industry. <!-- SC_ON --> submitted by /u/Icy-Perception0 (https://www.reddit.com/user/Icy-Perception0)
[link] (https://medium.com/datastrato/if-youre-not-all-in-on-databricks-why-metadata-freedom-matters-35cc5b15b24e) [comments] (https://www.reddit.com/r/programming/comments/1p7mvxw/learned_about_vendor_lockin_the_hard_way_during/)
I built a distributed message streaming platform from scratch that's faster than Kafka
https://www.reddit.com/r/programming/comments/1p7oqbc/i_built_a_distributed_message_streaming_platform/
<!-- SC_OFF -->I've been working on Walrus, a message streaming system (think Kafka-like) written in Rust. The focus was on making the storage layer as fast as possible. Performance highlights: 1.2 million writes/second (no fsync) 5,000 writes/second (fsync) Beats both Kafka and RocksDB in benchmarks (see graphs in README) How it's fast: The storage engine is custom-built instead of using existing libraries. On Linux, it uses io_uring for batched writes. On other platforms, it falls back to regular pread/pwrite syscalls. You can also use memory-mapped files if you prefer. Each topic is split into segments (~1M messages each). When a segment fills up, it automatically rolls over to a new one and distributes leadership to different nodes. This keeps the cluster balanced without manual configuration. Distributed setup: The cluster uses Raft for coordination, but only for metadata (which node owns which segment). The actual message data never goes through Raft, so writes stay fast. If you send a message to the wrong node, it just forwards it to the right one. You can also use the storage engine standalone as a library (walrus-rust on crates.io) if you just need fast local logging. I also wrote a TLA+ spec to verify the distributed parts work correctly (segment rollover, write safety, etc). Code: https://github.com/nubskr/walrus Would love to hear what you think, especially if you've worked on similar systems! <!-- SC_ON --> submitted by /u/Ok_Marionberry8922 (https://www.reddit.com/user/Ok_Marionberry8922)
[link] (https://github.com/nubskr/walrus) [comments] (https://www.reddit.com/r/programming/comments/1p7oqbc/i_built_a_distributed_message_streaming_platform/)
https://www.reddit.com/r/programming/comments/1p7oqbc/i_built_a_distributed_message_streaming_platform/
<!-- SC_OFF -->I've been working on Walrus, a message streaming system (think Kafka-like) written in Rust. The focus was on making the storage layer as fast as possible. Performance highlights: 1.2 million writes/second (no fsync) 5,000 writes/second (fsync) Beats both Kafka and RocksDB in benchmarks (see graphs in README) How it's fast: The storage engine is custom-built instead of using existing libraries. On Linux, it uses io_uring for batched writes. On other platforms, it falls back to regular pread/pwrite syscalls. You can also use memory-mapped files if you prefer. Each topic is split into segments (~1M messages each). When a segment fills up, it automatically rolls over to a new one and distributes leadership to different nodes. This keeps the cluster balanced without manual configuration. Distributed setup: The cluster uses Raft for coordination, but only for metadata (which node owns which segment). The actual message data never goes through Raft, so writes stay fast. If you send a message to the wrong node, it just forwards it to the right one. You can also use the storage engine standalone as a library (walrus-rust on crates.io) if you just need fast local logging. I also wrote a TLA+ spec to verify the distributed parts work correctly (segment rollover, write safety, etc). Code: https://github.com/nubskr/walrus Would love to hear what you think, especially if you've worked on similar systems! <!-- SC_ON --> submitted by /u/Ok_Marionberry8922 (https://www.reddit.com/user/Ok_Marionberry8922)
[link] (https://github.com/nubskr/walrus) [comments] (https://www.reddit.com/r/programming/comments/1p7oqbc/i_built_a_distributed_message_streaming_platform/)