NEW BOT Телеграм, страница

I came across Argo in an AWS blog post. In particular with Argo Workflows which is an orchestration tool like Airflow which you can use if you already have K8s cluster.

argoproj.github.io

Home

Open source Kubernetes native workflows, events, CI and CD

585 views12:43

Data1984

Kubernetes is probably the only major topic in our field that I never had a chance to work or interact with, but it seems it starts to serve as a meta OS or abstraction layer for major data engineering (and not only) platforms or projects.

596 views12:49

Data1984

Amazon Redshift announces public preview of Streaming Ingestion for Kinesis Data Streams
https://aws.amazon.com/about-aws/whats-new/2022/02/amazon-redshift-public-preview-streaming-ingestion-kinesis-data-streams/

Amazon

Amazon Redshift announces public preview of Streaming Ingestion for Kinesis Data Streams

👍1

581 views07:47

Data1984

https://youtu.be/6nGM37ThEsU

YouTube

How AI is Deciding Who Gets Hired

The job hunt has changed as artificial intelligence scores resumes, runs interviews and decides who gets access to opportunity. Lawmakers and activists are now pushing back on the threat of computerized bias while others work to outsmart the machine.

#FutureOfWork…

569 views17:45

Data1984

The ecosystem around dbt is growing.
https://github.com/re-data/re-data

GitHub

GitHub - re-data/re-data: re_data - fix data issues before your users & CEO would discover them 😊

re_data - fix data issues before your users & CEO would discover them 😊 - re-data/re-data

671 viewsedited 09:22

Data1984

Machine Learning CI/CD Pipeline with Github Actions and Amazon SageMaker

Medium

Machine Learning CI/CD Pipeline with Github Actions and Amazon SageMaker

When you begin working on ML project for a real Business Case, you should consider two essential questions.

394 viewsedited 16:32

Data1984

This article contains combination of multiple individually useful techniques. Especially, I like idea of indexing of S3 files with a cluster of Lambda functions.

Amazon

Doing more with less: Moving from transactional to stateful batch processing | Amazon Web Services

Amazon processes hundreds of millions of financial transactions each day, including accounts receivable, accounts payable, royalties, amortizations, and remittances, from over a hundred different business entities. All of this data is sent to the eCommerce…

395 viewsedited 19:04

Data1984

https://databricks.com/blog/2022/02/24/databricks-ventures-partners-with-dbt-labs-to-welcome-analytics-engineers-to-the-lakehouse.html

Databricks

Databricks Ventures Partners With dbt Labs to Welcome Analytics Engineers to the Lakehouse

Learn more about Databricks Ventures’ investment in dbt Labs and the Databricks-dbt Labs partnership, as well as the release of new enhancements, such as the native Databricks adapter and automatic query acceleration for Photon workloads.

392 views16:08

Data1984

I think there are three major platforms I would like to work/play with to get more experience:
1. Google Could Platform
2. Databricks (not just Spark)
3. Kubernetes (maybe to run Spark)

Google Cloud

Google Cloud Platform Services Summary

A complete list of services that form a part of Google Cloud.

416 viewsedited 16:53

Data1984

463 views17:00

Data1984

Every product in the Google Cloud family described in the visual sketchnote format to grasp the capability of the tools quickly and easily.

GitHub

GitHub - priyankavergadia/GCPSketchnote: If you are looking to become a Google Cloud Engineer , then you are at the right place.…

If you are looking to become a Google Cloud Engineer , then you are at the right place. GCPSketchnote is series where I share Google Cloud concepts in quick and easy to learn format. - priyankaverg...

602 views17:38