The exabyte club: LinkedIn’s journey of scaling the Hadoop Distributed File System | LinkedIn Engineering
https://engineering.linkedin.com/blog/2021/the-exabyte-club--linkedin-s-journey-of-scaling-the-hadoop-distr
https://engineering.linkedin.com/blog/2021/the-exabyte-club--linkedin-s-journey-of-scaling-the-hadoop-distr
Linkedin
The exabyte club: LinkedIn’s journey of scaling the Hadoop Distributed File System
Forwarded from Kseniia Tomak
У нас готов новый выпуск того самого дайджеста, о котором мало кто слышал:
https://digest.deordie.org/04_A_spark_of_the_summer_sun/
_____
Напомню, что у нас есть issue tracker: https://github.com/deordie/deordie-digest/issues
https://digest.deordie.org/04_A_spark_of_the_summer_sun/
_____
Напомню, что у нас есть issue tracker: https://github.com/deordie/deordie-digest/issues
Top 10 Announcements from Databricks Data + AI Summit 2021 - The Databricks Blog
https://databricks.com/blog/2021/06/04/dont-miss-these-top-10-announcements-from-data-ai-summit.html
https://databricks.com/blog/2021/06/04/dont-miss-these-top-10-announcements-from-data-ai-summit.html
Databricks
Top 10 Announcements from Databricks Data + AI Summit 2021
Couldn’t make it to this year’s event or missed some sessions? No worries, we’ve got the TL;DR for you in one place. Continue to the blog for 10 key announcements from the event, including the latest updates to Apache Spark, Delta Lake, MLflow and Databricks.
5 Reasons Why Rust Is The Future | Rust Functional Programming
https://scalac.io/blog/5-reasons-why-rust-is-the-future-rust-functional-programming/
https://scalac.io/blog/5-reasons-why-rust-is-the-future-rust-functional-programming/
Scalac - Software Development Company - Akka, Kafka, Spark, ZIO
5 Reasons Why Rust Is The Future
Rust functional programming. In the Stack overflow 2020 survey, Rust was picked as #1 most loved programming language, thanks to 86% of devs
A good reference for partitioning in Kafka.
https://docs.microsoft.com/en-us/azure/architecture/reference-architectures/event-hubs/partitioning-in-event-hubs-and-kafka
https://docs.microsoft.com/en-us/azure/architecture/reference-architectures/event-hubs/partitioning-in-event-hubs-and-kafka
Docs
Partitioning in Event Hubs and Kafka - Azure Architecture Center
Learn about partitioning in Kafka and Event Hubs with Kafka, see how many partitions to use in ingestion pipelines, and also see how to assign events to partitions.
Forwarded from DataEng
Сказка про Apache Kafka: https://www.gentlydownthe.stream/
Old but good blog post.
https://azure.microsoft.com/en-us/blog/individually-great-collectively-unmatched-announcing-updates-to-3-great-azure-data-services/
https://azure.microsoft.com/en-us/blog/individually-great-collectively-unmatched-announcing-updates-to-3-great-azure-data-services/
Microsoft
Individually great, collectively unmatched: Announcing updates to 3...
As Julia White mentioned in her blog today, we’re pleased to announce the general availability of Azure Data Lake Storage Gen2 and Azure Data Explorer. We also announced the preview of Azure Data F...
Forwarded from Инжиниринг Данных (Dmitry Anoshin)
Data Engineering Manifesto:
📌We are Engineers
📌Data is a Product
📌Enable by providing self-service solutions
📌Embrace Cloud & Managed services
📌Batch processing is easier than stream
📌Aim for simplicity through consistency
📌Notebooks are for research, not for development
📌Data scientists are colleagues, not enemies
📌SQL is the lowest common denominator
📌We are Engineers
📌Data is a Product
📌Enable by providing self-service solutions
📌Embrace Cloud & Managed services
📌Batch processing is easier than stream
📌Aim for simplicity through consistency
📌Notebooks are for research, not for development
📌Data scientists are colleagues, not enemies
📌SQL is the lowest common denominator
New free course on Azure Data Explorer (Kusto).
Docs
Introduction to Azure Data Explorer - Learn
Evaluate whether Azure Data Explorer is appropriate to process and analyze your big data.
The Definitive Guide to Delta Lake by O’Reilly- Free digital book - Download Now in Early Release - The Databricks Blog
https://databricks.com/blog/2021/06/22/get-your-free-copy-of-delta-lake-the-definitive-guide-early-release.html
https://databricks.com/blog/2021/06/22/get-your-free-copy-of-delta-lake-the-definitive-guide-early-release.html
Databricks
The Definitive Guide to Delta Lake by O’Reilly-- Free digital book - Download Now in Early Release
Download the FREE definitive guide book published by O’Reilly and from the creators of the popular open-source project now taking the data community by storm, and get the help and guidance needed to build or optimize your own lakehouse architecture.
👍1
Announcing GitHub Copilot
https://www.linkedin.com/pulse/announcing-github-copilot-kevin-scott
https://www.linkedin.com/pulse/announcing-github-copilot-kevin-scott
LinkedIn
Announcing GitHub Copilot
Today, GitHub, OpenAI and Microsoft launched a technical preview of GitHub Copilot, a collaboratively developed new AI pair programmer that helps you write better code. You can learn more about it here and sign up for the preview today—spots are limited,…
GitHub is introducing a lot of new features. I think this is the future, to have easy and safe setup accessible from any device.
It is seems future ia close. I personally believe one person needs one device for both work and home use, and anything else. And that device should only be light, have good screen and battery life. (I currently suffer from opposite situation 😁) Gaming industry is moving in thay direction and now it is time for the rest. I mean anyway we backup most important things to a cloud. So why not to have whole PC or Mac in the cloud. At least the one needed for work. The only question remains is price.
Microsoft News
Introducing a new era of hybrid personal computing: the Windows 365 Cloud PC
Updated on August 2, 2021: Windows 365 is now generally available to businesses of all sizes. Learn more about Windows 365 and experience it today. As some regions begin to make their way out of the challenges and disruption of the past 18 months, we’re seeing…
Super cool foundry program from dbt. But I am not sure if it is available remote or not.
Transform data in your warehouse
Announcing: The Foundry Program
Over the past few months, the dbt Labs’ recruiting team has partnered with leaders from our Engineering and Analytics Engineering organizations to design the Foundry Program. This program will provide a gateway for those that are new to the field or looking…
Forwarded from DataEng
Свет увидела новая распределенная SQL база данных на Rust (учебная): https://github.com/erikgrinaker/toydb
Автор также подробно описал её архитектуру: https://github.com/erikgrinaker/toydb/blob/master/docs/architecture.md
Автор также подробно описал её архитектуру: https://github.com/erikgrinaker/toydb/blob/master/docs/architecture.md
GitHub
GitHub - erikgrinaker/toydb: Distributed SQL database in Rust, written as an educational project
Distributed SQL database in Rust, written as an educational project - erikgrinaker/toydb
Forwarded from Инжиниринг Данных (Dmitry Anoshin)
Dremio будет проводить панель по озерам данных с топовыми спикерами онлайн https://www.dremio.com/subsurface/live?_ga=2.203573072.1949877702.1626636985-1007574036.1626636985
Dremio
Save the Date for Subsurface LIVE 2023!