It seems that #AWS is improving #Redshift on a weekly basis. Here is another cool feature.
https://aws.amazon.com/about-aws/whats-new/2020/11/amazon-redshift-announces-automatic-refresh-and-query-rewrite-for-materialized-views/

A comparison of data version control tools.
https://dagshub.com/blog/data-version-control-tools/

A short series of articles from Lyft about Gevent #Python library.
https://eng.lyft.com/what-the-heck-is-gevent-4e87db98a8
https://eng.lyft.com/gevent-part-2-correctness-22e3b7998382
https://eng.lyft.com/gevent-part-3-performance-e64303fa102b
https://eng.lyft.com/applying-gevent-learnings-to-deliver-value-to-users-part-4-of-4-36ad932deea8

Introduction to Apache Pinot, a real-time distributed OLAP datastore from LinkedIn and Uber
https://docs.pinot.apache.org/

Some important updates from #AWS :
✅ Amazon Kinesis Data Streams enables data stream retention up to one year.
✅ Now you can export your Amazon DynamoDB table data to your data lake in Amazon S3 to perform analytics at any scale.
✅ Amazon Redshift now supports modifying column compression encodings to optimize storage utilization and query performance
✅ Amazon Athena announces availability of engine version 2

➡️ Discover the new syntax for implicits in #Scala 3.

➡️ Learn how to express extension methods, implicit parameters, implicit conversions, and typeclasses in #Scala 3!

https://t.co/BYFnTVc3yh

#AWS updates:
✅ Amazon EMR now provides up to 35% lower cost and up to 15% improved performance for Spark workloads on Graviton2-based instances
✅ AWS Glue Streaming ETL jobs support reading records in the Apache Avro format
✅ Control the evolution of data streams using the AWS Glue Schema Registry

Learn how to design large-scale systems. Prep for the system design interview. Includes Anki flashcards.
https://github.com/donnemartin/system-design-primer

Finally, there is a managed service for #Airflow on #AWS!
https://news.1rj.ru/str/dataeng/252

I have created a template project to organize CI/CD for AWS Lambdas written in Python (and any other AWS infrastructure) using Terraform and Github Actions.
https://github.com/gorros/python-lambda-terraform-template

Обратная сторона облаков https://www.theverge.com/2020/11/25/21719396/amazon-web-services-aws-outage-down-internet

You now can use a SQL-compatible query language to query, insert, update, and delete table data in Amazon DynamoDB
https://aws.amazon.com/about-aws/whats-new/2020/11/you-now-can-use-a-sql-compatible-query-language-to-query-insert-update-and-delete-table-data-in-amazon-dynamodb/

Talkdesk is looking for a savvy Senior Data Engineer (REMOTE).

Վերջնաժամկետ 28.12.2020

Այս և մնացած բոլոր աշխատատեղերի մասին ամբողջական տեղեկատվություն կարող եք ստանալ մեր կայքից անցնելով հետևյալ հղումով `

https://www.talkdesk.com/careers/td/remote/engineering/senior-data-engineer-2445777?gh_jid=2445777

This is big!
https://www.linkedin.com/posts/iamabhishekchoudhary_dataengineering-machinelearning-deeplearning-activity-6739247941371932672-fc9n

#ML #AWS #Docker
https://towardsdatascience.com/serverless-comes-to-machine-learning-with-container-image-support-in-aws-lambda-ee9d729d48d7

Here are some major updates for Lambda which probably will make you rethink your serverless architecture 😉
#reInvent #AWS #Lambda
✅ New for AWS Lambda – Container Image Support
✅ New for AWS Lambda – 1ms Billing Granularity Adds Cost Savings
✅ New for AWS Lambda – Functions with Up to 10 GB of Memory and 6 vCPUs

#reInvent
Amazon S3 Update – Strong Read-After-Write Consistency:
Effective immediately, all S3 GET, PUT, and LIST operations, as well as operations that change object tags, ACLs, or metadata, are now strongly consistent!
This is especially import if you are using S3 as a Data Lake and process data via EMR.

Version control for data.
https://podcasts.google.com/?feed=aHR0cHM6Ly93d3cuZGF0YWVuZ2luZWVyaW5ncG9kY2FzdC5jb20vZmVlZC9tcDMv&ep=14&episode=cG9kbG92ZS0yMDIwLTExLTAydDIzOjUxOjQzKzAwOjAwLTBhYTlmMWYyODQ1ZTEzYQ&pe=1&pep=0

#reInvent #AWS
It seems ML is becoming standard feature everywhere.
https://aws.amazon.com/about-aws/whats-new/2020/12/aws-announces-amazon-redshift-ml-preview/