Here are some major updates for Lambda which probably will make you rethink your serverless architecture 😉
#reInvent #AWS #Lambda
✅ New for AWS Lambda – Container Image Support
✅ New for AWS Lambda – 1ms Billing Granularity Adds Cost Savings
✅ New for AWS Lambda – Functions with Up to 10 GB of Memory and 6 vCPUs

#reInvent
Amazon S3 Update – Strong Read-After-Write Consistency:
Effective immediately, all S3 GET, PUT, and LIST operations, as well as operations that change object tags, ACLs, or metadata, are now strongly consistent!
This is especially import if you are using S3 as a Data Lake and process data via EMR.

Version control for data.
https://podcasts.google.com/?feed=aHR0cHM6Ly93d3cuZGF0YWVuZ2luZWVyaW5ncG9kY2FzdC5jb20vZmVlZC9tcDMv&ep=14&episode=cG9kbG92ZS0yMDIwLTExLTAydDIzOjUxOjQzKzAwOjAwLTBhYTlmMWYyODQ1ZTEzYQ&pe=1&pep=0

#reInvent #AWS
It seems ML is becoming standard feature everywhere.
https://aws.amazon.com/about-aws/whats-new/2020/12/aws-announces-amazon-redshift-ml-preview/

#Scala #Kafka
I will leave this here just in case
https://www.confluent.io/blog/kafka-scala-tutorial-for-beginners/

#reInvent #AWS
Data analytics and engineering related updates:
✅ Announcing Amazon Redshift data sharing (preview)
✅ Amazon EMR Studio (Preview): A new notebook-first IDE experience with Amazon EMR
✅ Amazon Redshift announces native console integration with partners (Preview)
✅ Amazon Redshift announces support for native JSON and semi-structured data processing (preview)
✅ Simplify running Apache Spark jobs with Amazon EMR on Amazon EKS

#AWS #reInvent #Redshift
✅ Amazon Redshift announces Automatic Table Optimization
✅ Amazon Redshift now includes Amazon RDS for MySQL and Amazon Aurora MySQL databases as new data sources for federated querying (Preview)
✅ Amazon Redshift launches RA3.xlplus nodes with managed storage

A great review of metadata catalog evolution.
https://engineering.linkedin.com/blog/2020/datahub-popular-metadata-architectures-explained

#Scala
Old but good article about cake pattern for dependency injection in #Scala
https://medium.com/rahasak/scala-cake-pattern-e0cd894dae4e

Valid points about data teams.
https://podcasts.google.com/?feed=aHR0cHM6Ly93d3cuZGF0YWVuZ2luZWVyaW5ncG9kY2FzdC5jb20vZmVlZC9tcDMv&ep=14&episode=cG9kbG92ZS0yMDIwLTEyLTA4dDE1OjM2OjU0KzAwOjAwLTQ2NjQzODM3YzViMTI2NQ

Lakehouse = DW + Data Lake.

Примеры lakehouse:
- Redshift + Redshift Spectrum
- Snowflake
- Databrics Delta Lake
- Azure Synapse Analytics

Попался очень интересный paper, который был только недавно опубликован основателями Databricks.

This paper argues that the data warehouse architecture as we know it today will wither in the coming years and be replaced by a new architectural pattern, the Lakehouse, which will (i) be based on open direct-access data formats, such as Apache Parquet, (ii) have first class support for machine learning and data science, and (iii) offer state-of-the-art performance. Lakehouses can help address several major challenges with data warehouses, including data staleness, reliability, total cost of ownership, data lock-in, and limited use-case support.

#Scala #cheatsheet
https://alvinalexander.com/downloads/scala/Scala-Cheat-Sheet-devdaily.pdf

I am using #dbt for my latest project and it definitely is a great tool. I wouldn't compare it with #Spark which is great tool to crunch data both on a single machine and cluster. But for me #dbt is great since you can store your reports in version control and deploy them via CI/CD.

У data сообщества большие планы на dbt.

#Tableau #Visualization
https://news.1rj.ru/str/rockyourdata/2054

#cheatsheet
#cheatsheet
https://twitter.com/gp_pulipaka/status/1312573918255951875?s=09

New #opensource projects from #AWS #LinkedIn and #Netflix:
1. Optimizing data warehouse storage
2. FastIngest: Low-latency Gobblin with Apache Iceberg and ORC format
3. Testing data quality at scale with PyDeequ

Data visualization tools comparison from Dropbox.
https://dropbox.tech/application/why-we-chose-apache-superset-as-our-data-exploration-platform