Source: https://www.linkedin.com/posts/timo-dechau_in-our-little-data-world-are-we-naming-things-activity-6925303646817529856-Nu-U/
---
In our little data world are we naming things too much based on our marketing perspective. And is there serious over-selling going on.

Maybe yes.

Let’s do some examples:

dbt is not a data model tool. I see this notion quite often. It’s first a SQL orchestration and testing tool. Of course, I can use it to build and manage a data model. But this requires me to do the thinking not dbt

Snowflake and BigQuery are not data warehouses. Great people like .Rogier Werschkull. and Chad Sanderson remind us about that. They are analytical databases in the cloud. Of course, you can build a data warehouse with them. But this requires you to come up with a concept and architecture.

Fivetran and Airbyte are not ELT tools - they extract and load for you. And you are in charge of the transformation. They are basically supermarkets with self-checkout. Great idea but you have to do more.

Segment and Rudderstack are not really CDPs - Arpit Choudhury has written a great piece about it - they are customer data infrastructure, the collection and identity stitching layer

Reverse ETL is just ETL


Why is this important?

Because often these labels create expectations about the solution that these tools can’t fulfill.

When I set up Snowflake and think that I have a data warehouse now - I create huge expectations in my organization that I can’t fulfill.

Same with dbt - Ok, we need a data model, let’s use dbt for this. And then you add one sql file to the next one and call it a model.

Tools are tools, just that.

Pandas vs PySpark DataFrame With Examples

Let’s learn the difference between Pandas vs PySpark DataFrame, their definitions, features, advantages, how to create them and transform one to another with Examples.

👉 @devops_dataops

https://sparkbyexamples.com/pyspark/pandas-vs-pyspark-dataframe-with-examples/

Артем Шутак — Вставить в ClickHouse и не умереть

Презентация

Подборка ClickHouse + Spark - Altinity Knowledge Base
-----------
👉 @devops_dataops

Github Actions - Введение в CI/CD

00:00 - О чем курс
03:50 - Github вводный курс
12:35 - Начало работы с Github Actions
18:20 - Пишем первый workflow
29:17 - Автоматически тестируем React
37:57 - Что такое Actions
48:25 - Усложняем workflow (практика)
53:40 - Зависимость job и их порядок
01:00:18 - Context & Events
01:21:19 - Добавление cache
01:28:13 - Matrix
01:35:44 - Artifacts
01:45:25 - Environment & Secrets

https://www.youtube.com/watch?v=e0A2hDObLmg

Интересная модель монетизации у этого софта, вроде опенсоурс, но и есть разумные плюшки, которые можно получить только в платной версии (пользователи и роли + поддержка).
Ну и сама идея появления платформ с low-code подходом как open-source тоже интересная.
----
Tooljet | Open-source low-code platform to build internal tools

Extensible low-code framework for building business applications. Connect to databases, cloud storages, GraphQL, API endpoints, Airtable, etc and build apps using drag and drop application builder. Built using JavaScript/TypeScript.

https://www.tooljet.com/

Kubernetes Explained in 6 Minutes | k8s Architecture - YouTube
https://m.youtube.com/watch?v=TlHvYWVUZyc

ChatGPT Tutorial for Developers - 38 Ways to 10x Your Productivity

https://www.youtube.com/watch?v=sTeoEFzVNSc

Data Build Tool (dbt). Transformation in Modern data stack | by Amit Singh Rathore | Jan, 2023 | Dev Genius
https://blog.devgenius.io/data-build-tool-dbt-1f0b03d97cc6

Как запушить в Gitlab пакет npm, помогло 👍

Publishing your private npm packages to Gitlab NPM Registry

https://shivamarora.medium.com/publishing-your-private-npm-packages-to-gitlab-npm-registry-39d30a791085

Очередная подборочка инструментов Awesome-Selfhosted

A list of Free Software network services and web applications which can be hosted on your own servers

https://github.com/awesome-selfhosted/awesome-selfhosted

Prescriber-ETL-data-pipeline

An End-to-End ETL data pipeline that leverages pyspark parallel processing to process about 25 million rows of data coming from a SaaS application using Apache Airflow as an orchestration tool and various data warehouse technologies and finally using Apache Superset to connect to DWH for generating BI dashboards for weekly reports

https://github.com/judeleonard/Prescriber-ETL-data-pipeline

airflow-docker

This is my Apache Airflow Local development setup on Windows 10 WSL2/Mac using docker-compose. It will also include some sample DAGs and workflows.

https://github.com/anilkulkarni87/airflow-docker

Неплохая сравнительная табличка по инструментам metadata management

Awesome Data Discovery and Observability
https://github.com/opendatadiscovery/awesome-data-catalogs

2023 — Rockstar Data Engineer Roadmap
https://akaabachi.medium.com/2023-rockstar-data-engineer-roadmap-c990da37f625

Примеры из курса про Apache Airflow 2.0
https://github.com/adilkhash/apache-airflow-course-materials