AWS Lambdas - Python vs Rust. Performance and Cost Savings. - Confessions of a Data Guy
https://www.confessionsofadataguy.com/aws-lambdas-python-vs-rust-performance-and-cost-savings/
https://www.confessionsofadataguy.com/aws-lambdas-python-vs-rust-performance-and-cost-savings/
Confessions of a Data Guy
AWS Lambdas - Python vs Rust. Performance and Cost Savings. - Confessions of a Data Guy
Save money, save money!! Hear Hear! Someone on Linkedin recently brought up the point that companies could save gobs of money by swapping out AWS Python lambdas for Rust ones. While it raised the ire of many a Python Data Engineer, I thought it sounded like…
👍2
Guide to Partitions Calculation for Processing Data Files in Apache Spark - DZone
https://dzone.com/articles/guide-to-partitions-calculation-for-processing-dat
https://dzone.com/articles/guide-to-partitions-calculation-for-processing-dat
DZone
Guide to Partitions Calculation for Processing Data Files in Apache Spark
Get to Know how Spark chooses the number of partitions implicitly while reading a set of data files into an RDD or a Dataset.
👍1
Build a poor man’s data lake from scratch with DuckDB | Dagster Blog
https://dagster.io/blog/duckdb-data-lake
https://dagster.io/blog/duckdb-data-lake
dagster.io
Build a Data Lake with DuckDB + Dagster
Use DuckDB, Python, and Dagster to build a lightweight data lake with SQL transforms and Parquet file support.
Pandas 2.0 and its Ecosystem (Arrow, Polars, DuckDB) | Airbyte
https://airbyte.com/blog/pandas-2-0-ecosystem-arrow-polars-duckdb
https://airbyte.com/blog/pandas-2-0-ecosystem-arrow-polars-duckdb
Airbyte
Pandas 2.0 and its Ecosystem (Arrow, Polars, DuckDB) | Airbyte
Dive deeper into the power of Pandas and how leveraging it can benefit your organization. Explore a new way to work with data and unlock powerful insights!
Home - Apache Doris
https://doris.apache.org/
https://doris.apache.org/
doris.apache.org
Apache Doris: Open source data warehouse for real time data analytics - Apache Doris
Apache Doris is an open-source database based on MPP architecture,with easier use and higher performance. As a modern data warehouse, apache doris empowers your Olap query and database analytics.
GitHub - MaterializeInc/datagen: Generate authentic looking mock data based on a SQL, JSON or Avro schema and produce to Kafka in JSON or Avro format.
https://github.com/MaterializeInc/datagen
https://github.com/MaterializeInc/datagen
GitHub
GitHub - MaterializeInc/datagen: Generate authentic looking mock data based on a SQL, JSON or Avro schema and produce to Kafka…
Generate authentic looking mock data based on a SQL, JSON or Avro schema and produce to Kafka in JSON or Avro format. - MaterializeInc/datagen
GitHub - eto-ai/lance: Modern columnar data format for ML implemented in Rust. Convert from parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, with more integrations coming..
https://github.com/eto-ai/lance
https://github.com/eto-ai/lance
GitHub
GitHub - lancedb/lance: Modern columnar data format for ML and LLMs implemented in Rust. Convert from parquet in 2 lines of code…
Modern columnar data format for ML and LLMs implemented in Rust. Convert from parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, Du...
👍1
Lightning fast aggregations by distributing DuckDB across AWS Lambda functions | by BoilingData.com | Medium
https://boilingdata.medium.com/lightning-fast-aggregations-by-distributing-duckdb-across-aws-lambda-functions-e4775931ab04
https://boilingdata.medium.com/lightning-fast-aggregations-by-distributing-duckdb-across-aws-lambda-functions-e4775931ab04
Medium
Lightning fast aggregations by distributing DuckDB across AWS Lambda functions
DuckDB is rapidly changing the way data scientists and engineers work. It’s efficient and internally parallelised architecture means that a…