In data engineering landscape there are always new interesting project. And sometimes only way you hear about them is by talking to other data professionals. So here are two cool projects I learned about from a friend:
1. Presidio: Data protection and anonymization library from Microsoft
2. Trino: a new query engine from creators of Presto
1. Presidio: Data protection and anonymization library from Microsoft
2. Trino: a new query engine from creators of Presto
microsoft.github.io
Home - Microsoft Presidio
PII anonymization for text, images, and structured data.
It seems one tool, dbt, is driving demand for new analytics engineer specialization. Spark is popular too,and often considered as main tool for data engineers, but it did not create a specialization.
Medium
Analytics Engineer: The Newest Data Career Role
Is it analytics? Is it engineering? What do these people do?
Data Engineering Annotated Monthly – August 2021 | The Big Data Tools Blog
https://blog.jetbrains.com/big-data-tools/2021/09/06/data-engineering-annotated-monthly-august-2021/
https://blog.jetbrains.com/big-data-tools/2021/09/06/data-engineering-annotated-monthly-august-2021/
The JetBrains Blog
Data Engineering Annotated Monthly – August 2021 | The Big Data Tools Blog
August is usually a quiet month, with vacations taking their toll. But data engineering never stops. I’m Pasha Finkelshteyn and I will be your guide through this month’s news, my impressions of the de
Beginner's Series to Rust | Channel 9
https://channel9.msdn.com/Series/Beginners-Series-to-Rust?ocid=eml_pg293709_gdc_comm_az&mkt_tok=MTU3LUdRRS0zODIAAAF_aGNH8C6GnJuFIXOAMh12cOOZzlysGY-QSiGzExcs0QGgifOYInHhOMlSCA6styKmVWUcN3lrkTDBm1Bx3q8FUqYN0pt0w9Iqv4MXAoq-GZ-CDNm86IIZOw
https://channel9.msdn.com/Series/Beginners-Series-to-Rust?ocid=eml_pg293709_gdc_comm_az&mkt_tok=MTU3LUdRRS0zODIAAAF_aGNH8C6GnJuFIXOAMh12cOOZzlysGY-QSiGzExcs0QGgifOYInHhOMlSCA6styKmVWUcN3lrkTDBm1Bx3q8FUqYN0pt0w9Iqv4MXAoq-GZ-CDNm86IIZOw
Msdn
Beginner's Series to Rust | Channel 9
Rust has been ranked as one of the most loved languages by developers. In this series, you will learn the fundamentals of Rust development. We'll start by downloading the tools you need to program wit
I can say from my own experience that this is much better then post-factum analytics integration with traditional ETL. I just did not know that the term is IDT.
https://medium.com/whispering-data/the-end-of-etl-as-we-know-it-92166c19084c
https://medium.com/whispering-data/the-end-of-etl-as-we-know-it-92166c19084c
Medium
The End of ETL As We Know It
If you’re as sick of this three-letter phrase as I am, you’ll be happy to know there is another way.
Great presentation about Data Mesh,a term coined by Zhamak Dehghani, in her original article.
YouTube
Data Mesh in Practice: How Europe's Leading Online Platform for Fashion Goes Beyond the Data Lake
The Data Lake paradigm is often considered the scalable successor of the more curated Data Warehouse approach when it comes to democratization of data. However, many who went out to build a centralized Data Lake came out with a data swamp of unclear responsibilities…
There are almost no books on data engineering which focus on concepts and problems rather than on specific technologies. But this book is one of this rare ones. It is a collection of advises, problems and solutions, or just ideas to reflect on.
Goodreads
97 Things Every Data Engineer Should Know: Collective W…
Take advantage of today's sky-high demand for data engi…
I wrote a summary where I compare Azure Synapse, Databricks and Azure Data Explorer focusing on the features that I find important.
https://medium.com/@gorros/azure-synapse-databricks-and-azure-data-explorer-kusto-73a3a0339cf2
https://medium.com/@gorros/azure-synapse-databricks-and-azure-data-explorer-kusto-73a3a0339cf2
Medium
Azure Synapse, Databricks, and Azure Data Explorer (Kusto)
Which analytical platform to choose?
HP introduced its new Unified analytics platform HPE GreenLake
Hpe
HPE GreenLake Announcement
See why enterprises are embracing a cloud-everywhere strategy with HPE GreenLake. Join HPE President and CEO Antonio Neri for a special broadcast on April 4. Learn about the next set of cloud services launching on HPE GreenLake edge-to-cloud platform.
#pandas #vaex #dask #polars
Top 3 Alternative Python Packages for Pandas | by Cornellius Yudha Wijaya | Towards Data Science
https://towardsdatascience.com/top-3-alternative-python-packages-for-pandas-d125627ce349?gi=3e25591d0cdf
Top 3 Alternative Python Packages for Pandas | by Cornellius Yudha Wijaya | Towards Data Science
https://towardsdatascience.com/top-3-alternative-python-packages-for-pandas-d125627ce349?gi=3e25591d0cdf
Medium
Top 3 Alternative Python Packages for Pandas
For many modern data scientists, Python is the programming language that was used for their everyday work — as a consequence, the data analysis would be done using one of the most data packages…
Forwarded from Инжиниринг Данных (Dmitry Anoshin)
SQL with Squid Games.pdf
424.7 KB
Базовый SQL на примере Squid Games. Хороший подход, сразу понятно для тех, кто смотрел сериал.
Now Azure data explorer engineer is available in Azure Synapse.
TECHCOMMUNITY.MICROSOFT.COM
Introducing Azure Synapse data explorer for log and telemetry analytics
Digital transformation is a key aspect of the new programming model of intelligent devices across the cloud. One of the key types of data that's arisen from this new application paradigm is telemetry. Telemetry data is everywhere: IoT sensors, app logs, web…
Data Analyst Certificate & Training - Grow with Google
https://grow.google/dataanalytics/#?modal_active=none
https://grow.google/dataanalytics/#?modal_active=none
Microsoft Releases Azure Open AI Service Including Access to Powerful GPT-3 Models
https://www.infoq.com/news/2021/11/azure-openai-service-gpt3/?utm_campaign=infoq_content&utm_source=infoq&utm_medium=feed&utm_term=AI%2C+ML+%26+Data+Engineering
https://www.infoq.com/news/2021/11/azure-openai-service-gpt3/?utm_campaign=infoq_content&utm_source=infoq&utm_medium=feed&utm_term=AI%2C+ML+%26+Data+Engineering
InfoQ
Microsoft Releases Azure Open AI Service Including Access to Powerful GPT-3 Models
At its recent Ignite conference, Microsoft announced the new Azure OpenAI Service in preview, allowing access to OpenAI’s API through the Azure platform. This new Azure Cognitive Service will give customers access to OpenAI’s powerful GPT-3 models, along…