Еще один open-source проект, который в первую очередь предназначен для команд, которые работают с dbt

██████╗░██████╗░████████╗
██╔══██╗██╔══██╗╚══██╔══╝
██║░░██║██████╦╝░░░██║░░░
██║░░██║██╔══██╗░░░██║░░░
██████╔╝██████╦╝░░░██║░░░
╚═════╝░╚═════╝░░░░╚═╝░░░

Open-source data observability for analytics engineers

💬 Data anomalies monitoring as dbt tests - Collect metrics and metadata over time, detect anomalies, as native dbt tests in your project!
💬 Data observability report - Generate a report for all dbt tests and share with your team.
💬 dbt artifacts uploader
💬 Slack alerts
💬 Data lineage made simple, reliable, and automated

👉 @devops_dataops

https://github.com/elementary-data/elementary

Deep Dive on ClickHouse Sharding and Replication Webinar

Join the Altinity experts as we dig into ClickHouse sharding and replication, showing how they enable clusters that deliver fast queries over petabytes of data. We’ll start with basic definitions of each, then move to practical issues. This includes the setup of shards and replicas, defining schema, choosing sharding keys, loading data, and writing distributed queries. We’ll finish up with tips on performance optimization.

#ClickHouse

👉 @devops_dataops

https://www.youtube.com/watch?v=Vuh6NOluIxo

Подборочка статей

1. HOW TO OPERATE APACHE AIRFLOW WITH GITLAB CI/CD
2. Running Airflow with Docker on EC2 + CI/CD with GitLab
3. Building CI/CD with Airflow, GitLab and Terraform in GCP

👉 @devops_dataops

GitHub - kestra-io/kestra

Kestra is an infinitely scalable orchestration and scheduling platform, creating, running, scheduling, and monitoring millions of complex pipelines.

👉 @devops_dataops

https://github.com/kestra-io/kestra

Настраиваем iptables с нуля
Хорошее видео, чтобы вникнуть в концепцию firewall на Linux
https://www.youtube.com/watch?v=Q0EC8kJlB64

IPTables: NAT и Port forwarding

Продолжение про iptables

https://www.youtube.com/watch?v=u_a3ouarrVU

Apache Airflow 2.4 — Everything You Need to Know

Data-driven scheduling

👉 @devops_dataops

https://medium.com/@astronomer.io/apache-airflow-2-4-everything-you-need-to-know-4eaccde15936

Конференция DevOops 2022 - доклады/видео

▫️ https://devoops.ru/schedule/days/
▫️ https://www.youtube.com/c/DevOops_conf/videos

https://github.com/evgeniy-kharchenko/DevOps-Roadmap - репозиторий одного из докладчика DevOps Roadmap

Google vs IBM Data Engineer Certificate - BEST Certificate for Data Engineers

0:00 Intro
1:24 Overview Google
2:42 Overview IBM
3:56 Outcomes
6:16 Similarities
7:53 Differences
11:47 My Personal Recommendation

https://youtu.be/UpMBdfg2ZrE

Data Engineering

A collection of one-off topics or videos that do not fall neatly into any other existing playlist.

1. A Brief History of Data Engineering | What is Data Engineering?
2. How to Become a Data Engineer (with no experience)
3. ETL vs ELT | Modern Data Architectures
4. YAML Tutorial | Learn YAML in 10 Minutes
5. What is Data Streaming?
6. 3 Must-Know Trends for Data Engineers | DataOps
7. What skills do you need as a Data Engineer?
8. What is Reverse ETL?
9. What tools should you know as a Data Engineer?
10. Intro to BASH // Command Line for Beginners
11. Getting Started w/ Airbyte! | Open Source Data Integration
12. Data Warehouse vs Data Lake | Explained (non-technical)
13. Data Modeling in the Modern Data Stack
14. Getting Started w/ Metabase | Open Source Data Visualization Tool
15. What do you actually do as a data engineer?

👉 @devops_dataops

https://www.youtube.com/playlist?list=PLy4OcwImJzBKg3rmROyI_CBBAYlQISkOO

How to Make Data Documentation Sexy

The job that nobody wants to do gets a rebrand.

https://medium.com/geekculture/how-to-make-data-documentation-sexy-c0ef0d696f78

Article collection about data analytics workflows and building pipelines:

🔸Capturing Data Analytics Workflows and System Requirements
🔸DevOps for DataOps: Building a CI/CD Pipeline for Apache Airflow DAGs
🔸Intro to data science on Google Cloud
🔸Building a Spark and Airflow development environment with Docker
🔸Apache Airflow for Beginners - Build Your First Data Pipeline

📍 @devops_dataops

(DataCamp) Introduction to Airflow in Python

This is a memo to share what I have learnt in Apache Airflow, capturing the learning objectives as well as my personal notes. The course is taught by Mike Metzger from DataCamp, and it includes 4 chapters:
▫️ Intro to Airflow
▫️ Implementing Airflow DAGs
▫️ Maintaining and monitoring Airflow workflows
▫️ Building production pipelines in Airflow

https://github.com/JNYH/DataCamp_Introduction_to_Airflow

Personal Notes:
https://medium.com/swlh/introduction-to-airflow-in-python-67b554f06f0b

Через 30 минут начнётся move data конференция (в 21 по мск)

https://movedata.airbyte.com/

Airflow Tutorial for Beginners - Full Course in 2 Hours 2022

Throughout the course, you will learn:
00:00 - Airflow Introduction
03:06 - Run Airflow in Python Env
10:44 - Run Airflow in Docker
17:55 - Airflow Basics and Core Concepts
21:55 - Airflow Task Lifecycle
26:19 - Airflow Basic Architecture
28:14 - Airflow DAG with Bash Operator
40:09 - Airflow DAG with Python Operator
45:04 - Data Sharing via Airflow XComs
52:53 - Airflow Task Flow API
57:56 - Airflow Catch-Up and Backfill
01:02:09 - Airflow Scheduler with Cron Expression
01:07:25 - Airflow Connection to Postgres
01:08:58 - Airflow Postgres Operator
01:19:30 - Airflow Docker Install Python Package 2 ways
01:29:34 - Airflow AWS S3 Sensor Operator
01:42:37 - Airflow Hooks S3 PostgreSQL
02:00:43 - Course Bonus

https://www.youtube.com/watch?v=K9AnJ9_ZAXE

Apache Spark / PySpark Tutorial: Basics In 15 Mins

This video gives an introduction to the Spark ecosystem and world of Big Data, using the Python Programming Language and its PySpark API. We also discuss the idea of parallel and distributed computing, and computing on a cluster of machines.

https://youtu.be/QLQsW8VbTN4

Mastering Ubuntu Server - Third Edition

https://github.com/PacktPublishing/Mastering-Ubuntu-Server_Third-Edition

Yacht vs. Portainer - Docker dashboard comparison - Virtualization Howto
https://www.virtualizationhowto.com/2022/12/yacht-vs-portainer-docker-dashboard-comparison/