Understand the CAP and PACELC theorems in distributed systems. Learn how to navigate tradeoffs between consistency, availability, and partition tolerance for optimal system design.

Discover the crucial Architecturally Significant Requirements (ASR) for distributed systems, including Availability, Durability, Resiliency, Reliability, and Scalability. Learn how these factors impact system design and performance.

Caching... There is so much in that word - the pain of invalidation and the joy of reusing computation. In Spark, this is known as an optimization technique

Machine Learning is based on the idea that analytic systems can learn to identify patterns and make decisions with minimal human involvement

So much hype around serverless architectures but what it's really bringing to the table for us? Is it the next standard in application development?

Learn more about Glow, the open-source toolkit for genomics data analytics that scales to population levels and the testing and benchmarking that shows it is up to 10x faster than competitors.

Learn how to use Ray on the Databricks Lakehouse Platform for reinforcement learning and with custom distributed Python pipeline new use cases and optimizations.

There are many pitfalls that inexperienced engineers may encounter when building pipelines based on Cloud Dataproc, let's look into them.

In this post, we'll explore unbounded data, what data stream processing is, its characteristics, workflow, and why should we leverage the cloud for it?

Presto on Spark enables more use cases for data analytics, providing a unified SQL experience for both interactive and batch use cases.

Another portion of tips to Apache Spark usage, now it's about caching and checkpointing data

I am very annoyed that all sorts of big data engineers confuse S3 and HDFS systems, assuming that S3 is the same as HDFS. That’s not true.

Today, we’re pleased to announce the latest service in the Amazon SageMaker suite that will make labeling datasets easier than ever before. Ground Truth Plus is a turn-key service that uses an expert workforce to deliver high-quality training datasets fast…

Today, we’re pleased to announce Amazon SageMaker Inference Recommender — a brand-new Amazon SageMaker Studio capability that automates load testing and optimizes model performance across machine learning (ML) instances. Ultimately, it reduces the time it…

Today, we’re pleased to announce Amazon SageMaker Training Compiler, a new Amazon SageMaker capability that can accelerate the training of deep learning (DL) models by up to 50%. As DL models grow in complexity, so too does the time it can take to optimize…

As a woman working in information technology (IT) for many years, it has always been close to my heart to challenge long-standing gender stereotypes and inspire more young learners to consider a career in tech. With artificial intelligence (AI) and machine…

Learn how we optimized our workflow at Intelematics by creating a library to work with geospatial data using python on the Databricks Lakehouse platform.

Managing, monitoring, and auditing IP address allocation for at-scale networks, as the growth in cloud workloads and connected devices continues at a rapid pace, is a complex, time-consuming, and potentially error-prone task. Traditionally, network administrators…

Principal component analysis (PCA) is an unsupervised machine learning technique. Perhaps the most popular use of principal component analysis is dimensionality reduction. Besides using PCA as a data preparation technique, we can also use it to help visualize…