This post introduces Arrow Flight SQL, a protocol for interacting with SQL databases over Arrow Flight. We have been working on this protocol over the last six months, and are looking for feedback, interested contributors, and early adopters.

As early as 2016, we set out a bold, new vision reimagining batch data processing through a new “incremental” data processing stack - alongside the existing batch and streaming stacks.

A tutorial on the most important features and idioms of Scala that you need to use Spark's Scala APIs. - deanwampler/JustEnoughScalaForSpark

We (or rather, @sriramsubram ) tried to make sense of all the data stuff out there, by organizing it by their user persona. WDYT?

This article you'll learn how to create a free cluster, ingest data, and run queries to gain insights into your data using your free cluster.

Learn more about AWS Cloud Quest - https://bit.ly/3v2tX6X
Play AWS Cloud Quest and have fun building AWS Cloud Skills. AWS Cloud Quest is the first and only role-playing game to help you build in-demand AWS Cloud skills. Collect gems and earn points as you…

Reviewing 'Essential Scala' and 'Functional Programming Simplified', while explaining why Spark has nothing to do with Scala, and asking why learning Functional Programming is such a pain. A (maybe) productive rant (or an opinionated essay).

Turns out developers and plants need mostly the same things.

Update July 12, 2022 – The Available Now section below has been edited for clarity to ensure the best experience for our readers.  Serverless applications are event-driven, using ephemeral compute functions ranging from web APIs, mobile backends, and streaming…

Valuable analytical data continues to grow and is increasingly distributed. Data silos mean that enterprises often need to move data across storage tiers first to perform analytics, which is expensive. Built on years of investment in BigQuery, BigLake is…

Organizations are adopting microservices architectures to build resilient and scalable applications using AWS Lambda. These applications are composed of multiple serverless functions that implement the business logic. Each function is mapped to API endpoints…

🚀 Hot off the Delta Lake #press!!!!!
😎 Delta Lake community is excited to announce the release Version 1.2.0. 

#Key features in this release:
⭐ Support…

Integrating Great Expectations with a ubiquitous Big data engineering platform

WOW. Bloomberg finally opensourced memray—a new versatile memory profile for Python. Can't way to use it. Thread 👇

Our career path is how we spend our time, how we support our lifestyles, how we make our impact, and even sometimes how we define our identity. Let’s make sure we’re on the right track.

When we first launched Amazon Athena, our mission was to make it simple to query data stored in Amazon Simple Storage Service (Amazon S3). Athena customers found it easy to get started and develop analytics on petabyte-scale data lakes, but told us they needed…

BigLake unifies data warehouses and data lakes into a consistent format for faster data analytics across Google Cloud and open source formats and engines.

dbt has established itself as one of the most popular tools in the modern data stack, and is aiming to bring analytics engineering to everyone. The dbt tool makes it easy to develop and implement complex data processing pipelines, with mostly SQL, and it…