A good reference for partitioning in Kafka.
https://docs.microsoft.com/en-us/azure/architecture/reference-architectures/event-hubs/partitioning-in-event-hubs-and-kafka
https://docs.microsoft.com/en-us/azure/architecture/reference-architectures/event-hubs/partitioning-in-event-hubs-and-kafka
Docs
Partitioning in Event Hubs and Kafka - Azure Architecture Center
Learn about partitioning in Kafka and Event Hubs with Kafka, see how many partitions to use in ingestion pipelines, and also see how to assign events to partitions.
Forwarded from DataEng
Сказка про Apache Kafka: https://www.gentlydownthe.stream/
Old but good blog post.
https://azure.microsoft.com/en-us/blog/individually-great-collectively-unmatched-announcing-updates-to-3-great-azure-data-services/
https://azure.microsoft.com/en-us/blog/individually-great-collectively-unmatched-announcing-updates-to-3-great-azure-data-services/
Microsoft
Individually great, collectively unmatched: Announcing updates to 3...
As Julia White mentioned in her blog today, we’re pleased to announce the general availability of Azure Data Lake Storage Gen2 and Azure Data Explorer. We also announced the preview of Azure Data F...
Forwarded from Инжиниринг Данных (Dmitry Anoshin)
Data Engineering Manifesto:
📌We are Engineers
📌Data is a Product
📌Enable by providing self-service solutions
📌Embrace Cloud & Managed services
📌Batch processing is easier than stream
📌Aim for simplicity through consistency
📌Notebooks are for research, not for development
📌Data scientists are colleagues, not enemies
📌SQL is the lowest common denominator
📌We are Engineers
📌Data is a Product
📌Enable by providing self-service solutions
📌Embrace Cloud & Managed services
📌Batch processing is easier than stream
📌Aim for simplicity through consistency
📌Notebooks are for research, not for development
📌Data scientists are colleagues, not enemies
📌SQL is the lowest common denominator
New free course on Azure Data Explorer (Kusto).
Docs
Introduction to Azure Data Explorer - Learn
Evaluate whether Azure Data Explorer is appropriate to process and analyze your big data.
The Definitive Guide to Delta Lake by O’Reilly- Free digital book - Download Now in Early Release - The Databricks Blog
https://databricks.com/blog/2021/06/22/get-your-free-copy-of-delta-lake-the-definitive-guide-early-release.html
https://databricks.com/blog/2021/06/22/get-your-free-copy-of-delta-lake-the-definitive-guide-early-release.html
Databricks
The Definitive Guide to Delta Lake by O’Reilly-- Free digital book - Download Now in Early Release
Download the FREE definitive guide book published by O’Reilly and from the creators of the popular open-source project now taking the data community by storm, and get the help and guidance needed to build or optimize your own lakehouse architecture.
👍1
Announcing GitHub Copilot
https://www.linkedin.com/pulse/announcing-github-copilot-kevin-scott
https://www.linkedin.com/pulse/announcing-github-copilot-kevin-scott
LinkedIn
Announcing GitHub Copilot
Today, GitHub, OpenAI and Microsoft launched a technical preview of GitHub Copilot, a collaboratively developed new AI pair programmer that helps you write better code. You can learn more about it here and sign up for the preview today—spots are limited,…
GitHub is introducing a lot of new features. I think this is the future, to have easy and safe setup accessible from any device.
It is seems future ia close. I personally believe one person needs one device for both work and home use, and anything else. And that device should only be light, have good screen and battery life. (I currently suffer from opposite situation 😁) Gaming industry is moving in thay direction and now it is time for the rest. I mean anyway we backup most important things to a cloud. So why not to have whole PC or Mac in the cloud. At least the one needed for work. The only question remains is price.
Microsoft News
Introducing a new era of hybrid personal computing: the Windows 365 Cloud PC
Updated on August 2, 2021: Windows 365 is now generally available to businesses of all sizes. Learn more about Windows 365 and experience it today. As some regions begin to make their way out of the challenges and disruption of the past 18 months, we’re seeing…
Super cool foundry program from dbt. But I am not sure if it is available remote or not.
Transform data in your warehouse
Announcing: The Foundry Program
Over the past few months, the dbt Labs’ recruiting team has partnered with leaders from our Engineering and Analytics Engineering organizations to design the Foundry Program. This program will provide a gateway for those that are new to the field or looking…
Forwarded from DataEng
Свет увидела новая распределенная SQL база данных на Rust (учебная): https://github.com/erikgrinaker/toydb
Автор также подробно описал её архитектуру: https://github.com/erikgrinaker/toydb/blob/master/docs/architecture.md
Автор также подробно описал её архитектуру: https://github.com/erikgrinaker/toydb/blob/master/docs/architecture.md
GitHub
GitHub - erikgrinaker/toydb: Distributed SQL database in Rust, written as an educational project
Distributed SQL database in Rust, written as an educational project - erikgrinaker/toydb
Forwarded from Инжиниринг Данных (Dmitry Anoshin)
Dremio будет проводить панель по озерам данных с топовыми спикерами онлайн https://www.dremio.com/subsurface/live?_ga=2.203573072.1949877702.1626636985-1007574036.1626636985
Dremio
Save the Date for Subsurface LIVE 2023!
Great website to explore and learn about Azure in an interactive way.
https://azurecharts.com/learning
https://azurecharts.com/learning
Azurecharts
Azure Learning Explorer
Discover published Azure Learning modules and learning paths for services of your interest
A comprehensive technology-agnostic course on how to architect distributed cloud applications.
YouTube
Architecting Distributed Cloud Applications
A comprehensive technology-agnostic course on how to architect distributed cloud applications. PLEASE SHARE MY CHANNEL WITH OTHERS
Ars Technica: Big tech companies are at war with employees over remote work.
https://arstechnica.com/gadgets/2021/08/vaccines-reopenings-and-worker-revolts-big-techs-contentious-return-to-the-office/
https://arstechnica.com/gadgets/2021/08/vaccines-reopenings-and-worker-revolts-big-techs-contentious-return-to-the-office/
Ars Technica
Big tech companies are at war with employees over remote work
CEOs want workers back at their desks. Employees and the virus have other plans.
I know that "remote vs office" topic is all over the place, but still, what do you think, does commute or other perks worth going to office, or do you miss your colleagues?
Anonymous Poll
34%
Remote
5%
Office
61%
Hybrid