It seems that #AWS is improving #Redshift on a weekly basis. Here is another cool feature.
https://aws.amazon.com/about-aws/whats-new/2020/11/amazon-redshift-announces-automatic-refresh-and-query-rewrite-for-materialized-views/
https://aws.amazon.com/about-aws/whats-new/2020/11/amazon-redshift-announces-automatic-refresh-and-query-rewrite-for-materialized-views/
Amazon Web Services, Inc.
Amazon Redshift announces automatic refresh and query rewrite for materialized views
A comparison of data version control tools.
https://dagshub.com/blog/data-version-control-tools/
https://dagshub.com/blog/data-version-control-tools/
DagsHub Blog
Comparing Data Version Control Tools - 2020
Data versioning is one of the keys to automating a team's machine learning model development. While it can be very complicated if your team attempts to develop its own system to manage the process, this doesn’t need to be the case.
A short series of articles from Lyft about Gevent #Python library.
https://eng.lyft.com/what-the-heck-is-gevent-4e87db98a8
https://eng.lyft.com/gevent-part-2-correctness-22e3b7998382
https://eng.lyft.com/gevent-part-3-performance-e64303fa102b
https://eng.lyft.com/applying-gevent-learnings-to-deliver-value-to-users-part-4-of-4-36ad932deea8
https://eng.lyft.com/what-the-heck-is-gevent-4e87db98a8
https://eng.lyft.com/gevent-part-2-correctness-22e3b7998382
https://eng.lyft.com/gevent-part-3-performance-e64303fa102b
https://eng.lyft.com/applying-gevent-learnings-to-deliver-value-to-users-part-4-of-4-36ad932deea8
Medium
What the heck is gevent?
Overview
Introduction to Apache Pinot, a real-time distributed OLAP datastore from LinkedIn and Uber
https://docs.pinot.apache.org/
https://docs.pinot.apache.org/
docs.pinot.apache.org
Introduction | Apache Pinot Docs
Apache Pinot is a real-time distributed OLAP datastore purpose-built for low-latency, high-throughput analytics, and perfect for user-facing analytical workloads.
Some important updates from #AWS :
✅ Amazon Kinesis Data Streams enables data stream retention up to one year.
✅ Now you can export your Amazon DynamoDB table data to your data lake in Amazon S3 to perform analytics at any scale.
✅ Amazon Redshift now supports modifying column compression encodings to optimize storage utilization and query performance
✅ Amazon Athena announces availability of engine version 2
✅ Amazon Kinesis Data Streams enables data stream retention up to one year.
✅ Now you can export your Amazon DynamoDB table data to your data lake in Amazon S3 to perform analytics at any scale.
✅ Amazon Redshift now supports modifying column compression encodings to optimize storage utilization and query performance
✅ Amazon Athena announces availability of engine version 2
Amazon
Amazon Kinesis Data Streams enables data stream retention up to one year
➡️ Discover the new syntax for implicits in #Scala 3.
➡️ Learn how to express extension methods, implicit parameters, implicit conversions, and typeclasses in #Scala 3!
https://t.co/BYFnTVc3yh
➡️ Learn how to express extension methods, implicit parameters, implicit conversions, and typeclasses in #Scala 3!
https://t.co/BYFnTVc3yh
www.scala-lang.org
Explicit term inference with Scala 3
#AWS updates:
✅ Amazon EMR now provides up to 35% lower cost and up to 15% improved performance for Spark workloads on Graviton2-based instances
✅ AWS Glue Streaming ETL jobs support reading records in the Apache Avro format
✅ Control the evolution of data streams using the AWS Glue Schema Registry
✅ Amazon EMR now provides up to 35% lower cost and up to 15% improved performance for Spark workloads on Graviton2-based instances
✅ AWS Glue Streaming ETL jobs support reading records in the Apache Avro format
✅ Control the evolution of data streams using the AWS Glue Schema Registry
Amazon
Amazon EMR now provides up to 35% lower cost and up to 15% improved performance for Spark workloads on Graviton2-based instances
Learn how to design large-scale systems. Prep for the system design interview. Includes Anki flashcards.
https://github.com/donnemartin/system-design-primer
https://github.com/donnemartin/system-design-primer
GitHub
GitHub - donnemartin/system-design-primer: Learn how to design large-scale systems. Prep for the system design interview. Includes…
Learn how to design large-scale systems. Prep for the system design interview. Includes Anki flashcards. - donnemartin/system-design-primer
I have created a template project to organize CI/CD for AWS Lambdas written in Python (and any other AWS infrastructure) using Terraform and Github Actions.
https://github.com/gorros/python-lambda-terraform-template
https://github.com/gorros/python-lambda-terraform-template
GitHub
GitHub - gorros/python-lambda-terraform-template: A template project to organize CI/CD for AWS Lambdas written in Python (and…
A template project to organize CI/CD for AWS Lambdas written in Python (and any other AWS infrastructure) using Terraform and Github Actions. - GitHub - gorros/python-lambda-terraform-template: A ...
Forwarded from Инжиниринг Данных (Dmitry Anoshin)
Обратная сторона облаков https://www.theverge.com/2020/11/25/21719396/amazon-web-services-aws-outage-down-internet
The Verge
Prolonged AWS outage takes down a big chunk of the internet
The issue seems fairly widespread.
You now can use a SQL-compatible query language to query, insert, update, and delete table data in Amazon DynamoDB
https://aws.amazon.com/about-aws/whats-new/2020/11/you-now-can-use-a-sql-compatible-query-language-to-query-insert-update-and-delete-table-data-in-amazon-dynamodb/
https://aws.amazon.com/about-aws/whats-new/2020/11/you-now-can-use-a-sql-compatible-query-language-to-query-insert-update-and-delete-table-data-in-amazon-dynamodb/
Amazon
You now can use a SQL-compatible query language to query, insert, update, and delete table data in Amazon DynamoDB
Forwarded from VG Recruiting Agency (IT) (Вазген Галоян)
Talkdesk is looking for a savvy Senior Data Engineer (REMOTE).
Վերջնաժամկետ 28.12.2020
Այս և մնացած բոլոր աշխատատեղերի մասին ամբողջական տեղեկատվություն կարող եք ստանալ մեր կայքից անցնելով հետևյալ հղումով `
https://www.talkdesk.com/careers/td/remote/engineering/senior-data-engineer-2445777?gh_jid=2445777
Վերջնաժամկետ 28.12.2020
Այս և մնացած բոլոր աշխատատեղերի մասին ամբողջական տեղեկատվություն կարող եք ստանալ մեր կայքից անցնելով հետևյալ հղումով `
https://www.talkdesk.com/careers/td/remote/engineering/senior-data-engineer-2445777?gh_jid=2445777
#ML #AWS #Docker
https://towardsdatascience.com/serverless-comes-to-machine-learning-with-container-image-support-in-aws-lambda-ee9d729d48d7
https://towardsdatascience.com/serverless-comes-to-machine-learning-with-container-image-support-in-aws-lambda-ee9d729d48d7
Medium
Serverless comes to machine learning with container image support in AWS Lambda.
Today AWS Lambda released an astonishing new feature that could ease things a lot for machine learning practitioners.
Here are some major updates for Lambda which probably will make you rethink your serverless architecture 😉
#reInvent #AWS #Lambda
✅ New for AWS Lambda – Container Image Support
✅ New for AWS Lambda – 1ms Billing Granularity Adds Cost Savings
✅ New for AWS Lambda – Functions with Up to 10 GB of Memory and 6 vCPUs
#reInvent #AWS #Lambda
✅ New for AWS Lambda – Container Image Support
✅ New for AWS Lambda – 1ms Billing Granularity Adds Cost Savings
✅ New for AWS Lambda – Functions with Up to 10 GB of Memory and 6 vCPUs
Amazon
New for AWS Lambda – Container Image Support | Amazon Web Services
February 9, 2021: Post updated with the current regional availability of container image support for AWS Lambda. With AWS Lambda, you upload your code and run it without thinking about servers. Many customers enjoy the way this works, but if you’ve invested…
#reInvent
Amazon S3 Update – Strong Read-After-Write Consistency:
Effective immediately, all S3 GET, PUT, and LIST operations, as well as operations that change object tags, ACLs, or metadata, are now strongly consistent!
This is especially import if you are using S3 as a Data Lake and process data via EMR.
Amazon S3 Update – Strong Read-After-Write Consistency:
Effective immediately, all S3 GET, PUT, and LIST operations, as well as operations that change object tags, ACLs, or metadata, are now strongly consistent!
This is especially import if you are using S3 as a Data Lake and process data via EMR.
Amazon
Amazon S3 Update – Strong Read-After-Write Consistency | Amazon Web Services
When we launched S3 back in 2006, I discussed its virtually unlimited capacity (“…easily store any number of blocks…”), the fact that it was designed to provide 99.99% availability, and that it offered durable storage, with data transparently stored in multiple…
Version control for data.
https://podcasts.google.com/?feed=aHR0cHM6Ly93d3cuZGF0YWVuZ2luZWVyaW5ncG9kY2FzdC5jb20vZmVlZC9tcDMv&ep=14&episode=cG9kbG92ZS0yMDIwLTExLTAydDIzOjUxOjQzKzAwOjAwLTBhYTlmMWYyODQ1ZTEzYQ&pe=1&pep=0
https://podcasts.google.com/?feed=aHR0cHM6Ly93d3cuZGF0YWVuZ2luZWVyaW5ncG9kY2FzdC5jb20vZmVlZC9tcDMv&ep=14&episode=cG9kbG92ZS0yMDIwLTExLTAydDIzOjUxOjQzKzAwOjAwLTBhYTlmMWYyODQ1ZTEzYQ&pe=1&pep=0
Google Podcasts
Data Engineering Podcast - Add Version Control To Your Data Lake With LakeFS
Data lakes are gaining popularity due to their flexibility and reduced cost of storage. Along with the benefits there are some additional complexities to consider, including how to safely integrate new data sources or test out changes to existing pipelines.…
#reInvent #AWS
It seems ML is becoming standard feature everywhere.
https://aws.amazon.com/about-aws/whats-new/2020/12/aws-announces-amazon-redshift-ml-preview/
It seems ML is becoming standard feature everywhere.
https://aws.amazon.com/about-aws/whats-new/2020/12/aws-announces-amazon-redshift-ml-preview/
Amazon
AWS announces Amazon Redshift ML (preview)
#Scala #Kafka
I will leave this here just in case
https://www.confluent.io/blog/kafka-scala-tutorial-for-beginners/
I will leave this here just in case
https://www.confluent.io/blog/kafka-scala-tutorial-for-beginners/
Confluent
Apache Kafka and Scala - A Beginner’s Tutorial
Introduction to Apache Kafka and Scala. Learn about Kafka clients, how to use it in Scala, the Kafka Streams Scala module, and popular Scala integrations with code examples.