💡⚔️Sensei will tell you
Sensei is a relatively new Python tool for generating synthetic data using systems such as OpenAI, MistralAI or AnthropicAII.
To start, you need to make the following preset:
pip install openai mistralai numpy
The developers also wrote detailed instructions for setup.

📊😎💡Selection of services for working with Big Data and integration with various DBMSs
DBeaver is a service that is suitable for integration with various databases, such as MySQL or Oracle. This application is designed for database management. The JDBC interface helps it interact with relational databases. The DBeaver editor allows you to use a large number of additional plugins and gives hints on filling out the code, highlighting the syntax. The application manager supports over 80 databases.
Mixpanel is a system for analytics and analysis of user behavior. It includes features such as:
1. User segmentation
2. Send in-app notifications to your users
3. A/B testing for various notifications
4. Integrating custom surveys into applications via Mixpanel Surveys
App Annie is a service for analytics and obtaining reliable data to make important decisions at all stages of the mobile application business. App Annie will help you study competitors, market conditions, track app downloads, revenue, usage, engagement and advertising. The service also allows you to optimize products for app stores and increase the effectiveness of promotion methods, retention rates and effectively support your target audience. App Annie includes market analytics, multi-store app analytics, and competitor analytics.
Adjust is an optimizer for all product promotion processes. Collects information about where users came to your app page from. It provides a set of measurement and analytics tools that marketers can use to monitor and guide the development of their applications throughout the product lifecycle

⚔️😎💡ClickHouse vs Greenplum
Clickhouse and GreenPlum are well-known DBMSs for big data analysis that are very popular. However, there are criteria by which it is necessary to unambiguously choose which of the DBMS data to use in a given situation. To do this, let's look at their main advantages and disadvantages.
Advantages of ClickHouse:
1. High performance: ClickHouse is designed for analytical tasks and has a high speed of executing requests for reading large amounts of data. This makes it an ideal choice for data analytics and OLAP (Online Analytical Processing)
2. Efficient data compression: ClickHouse uses various data compression methods, which can significantly reduce the amount of stored information without loss of performance.
3. Horizontal scaling: ClickHouse easily scales horizontally, which allows you to increase system performance by adding new nodes.
Disadvantages of ClickHouse:
1. Limited transaction support: ClickHouse is mainly focused on analytical tasks and does not have full transaction support, which can be a problem for some applications.
2. Limited feature set: Despite its performance, ClickHouse may not be sufficient for some complex analytical tasks due to the limited set of built-in features.
Greenplum benefits:
1. Transaction Support: Greenplum provides full support for transactions and ACID (Atomicity, Consistency, Isolation, Durability), making it an ideal choice for OLTP (Online Transactional Processing) and OLAP applications.
2. Wide Range of Features: Greenplum offers a rich set of built-in features and analytical processing capabilities, making it suitable for various types of analytical tasks.
3. Support for distributed transactions: Greenplum provides support for distributed transactions and scales horizontally to handle large volumes of data.
Disadvantages of Greenplum:
1. Complexity to manage: Greenplum may require more effort and experience to manage and configure, especially when dealing with large clusters.
2. Less efficient data compression: Compared to ClickHouse, Greenplum may not provide the same high level of data compression, which may result in higher disk space usage and lower performance
Ultimately, the choice between ClickHouse and Greenplum depends on the specific needs of the task. ClickHouse is better suited for analytical workloads with high performance requirements, while Greenplum may be the preferred choice for applications where transaction support and a wide range of features are important.

📚💡Selection of books on various Big Data processing technologies
Spark: The Definitive Guide - the book tells learn how to use, deploy, and maintain Apache Spark with this comprehensive guide, written by the creators of the open-source cluster computing framework code.
Hadoop. Подробное руководство - a book in which it is described thoroughly and clearly You have all the features of Apache Hadoop.
Apache Kafka. Потоковая обработка и анализ данных - the book describes the design principles of the Big Data broker Kafka, reliability guarantees, key APIs and architectural details
Kubernetes в действии - the book talks in detail about Kubernetes - Google's open source software for automating the deployment, scaling and management of applications, scaling and management of Big Data applications
Cassandra: The Definitive Guide: Distributed Data at Web Scale - this guide explains how the Cassandra database management system processes hundreds of terabytes of data while maintaining high availability across multiple processing centers data
MongoDB: полное руководство - This book takes a detailed look at MongoDB, a powerful database management system. Here you can also learn how this secure, high-performance system provides flexible data models, high data availability and horizontal scalability.

💡😎Datasets for the task of converting text to sound
FAIR has released a project for a system for converting text into sound into the public domain.
In addition to the main project, there are also datasets in JSON format.
Detailed instructions for using datasets can be found here

💡📊Training data used in ComCLIP
CLIP (Contrastive Language-Image Pre-Training) is a neural network developed by OpenAI to perform visual as well as language comprehension tasks. The algorithms aim to understand the relationship between text and images.
ComCLIP is an improved version of CLIP for matching text and graphic representations. ComCLIP can mitigate spurious correlations introduced by pre-trained CLIP models and dynamically estimate the importance of each component. Experiments were conducted on four datasets for compositional matching between images and texts.
These datasets are publicly available on the Internet and can be found at this link

💡😎💊Google published a new dataset of skin condition images
SCIN is an open access dataset that contains data on skin condition. This dataset was collected from volunteer Google Search users in the United States using a special application.
SCIN contains 10,000+ images for common dermatological diseases. Materials include images, medical history and symptom information, and self-reported Fitzpatrick skin type.

📄Documentation

☁️Download link

🌎TOP DS-events all over the world in April
Apr 2-3 - Healthcare NLP Summit - Online - https://www.nlpsummit.org/healthcare-2024/
Apr 9 - Data Architecture New Zealand - Hilton Auckland - https://data-architecture-nz.coriniumintelligence.com/
Apr 9-11 - Google Cloud Next - Las Vegas, US - https://cloud.withgoogle.com/next
Apr 16 - CISO Perth - Perth, Australia - https://ciso-perth.coriniumintelligence.com/
Apr 17-18 - CDAO Germany - https://cdao-germany.coriniumintelligence.com/
Apr 17-18 - AI in Finance – New York, United States - https://ny-ai-finance.re-work.co/
Apr 22-24 - PyCon DE & PyData Berlin 2024 - Berlin, Germany - https://2024.pycon.de/
Apr 22-24 - Machine Learning Prague 2024 - Prague, Czech Republic - https://www.mlprague.com/
Apr 23-24 - The Martech Summit - Singapore - https://themartechsummit.com/singapore
Apr 23-25 - ODSC - Boston, USA - https://odsc.com/boston/schedule-overview/
Apr 24-25 - Gartner Data & Analytics Summit - Mumbai, India - https://www.gartner.com/en/conferences/apac/data-analytics-india
Apr 24-26 - Symposium on Intelligent Data Analysis (IDA) - Stockholm, Sweden - https://ida2024.org/

⚔️Relational DBMS vs NOSQL DBMS: advantages and disadvantages
Database implementation is a fundamental element of modern information technology. In the world of databases, there are two main paradigms: relational DBMS and NoSQL DBMS. Each of them has its own advantages and disadvantages, which should be taken into account when choosing the right one for a particular task.
Relational databases are based on a data model known as the relational model. In such databases, data is stored in the form of tables, which consist of rows (records) and columns (fields). The data structure is defined by a predefined schema that describes the data types of each column.
Advantages of relational DBMS:
1. Data structure: Relational DBMS stores data in the form of tables, which makes it easy to understand and organized.
2. ACID properties: Guarantees atomicity, consistency, isolation and durability of transactions, making them reliable for applications that require a high degree of data integrity.
3. SQL Language: A powerful and widely used query language that provides standardization and ease of working with data.
Flaws:
1. Vertical scaling: Relational DBMSs can face vertical scaling limitations, which means that when they reach their performance limits they will have to be migrated to more powerful, and often more expensive, servers.
2. Schema Complexity: Changing the data schema can be difficult and require additional effort and time.
3. Difficulty of horizontal scaling: Even with data partitioning techniques, horizontal scaling of relational DBMSs can be complex and require additional configuration and optimization work.
NoSQL databases are designed to work with unstructured and semi-structured data. They offer a flexible data schema, which allows you to store data without explicitly defining the schema in advance.
Advantages of NOSQL:
1. Flexibility of data structure: NoSQL DBMSs allow you to store unstructured data, making them an ideal choice for applications with changing data requirements.
2. Horizontal scalability: Many NoSQL databases are designed to scale horizontally, making them suitable for handling large amounts of data and high workloads.
Flaws:
1. Lack of ACID properties: Unlike relational DBMSs, NoSQL databases can sacrifice some ACID properties in favor of performance and scalability.
2. Limited support for SQL query language: Some NoSQL DBMSs may have limited query language functionality, which can make it difficult to perform complex queries or analytical operations.
The choice between relational and NoSQL DBMS depends on the specific requirements and characteristics of the project. Relational DBMSs provide high data integrity, while NoSQL DBMSs allow you to work with large volumes of unstructured data and provide flexibility and scalability.

📊😎💡The two largest open datasets for text recognition have been released
Data sets contain millions of real documents, images and texts for text recognition, analysis and document parsing tasks
VQA is a dataset used to develop and evaluate machine learning models capable of answering image-related questions. In the dataset, questions are assigned to each image, as well as the correct answers to these questions. This dataset is supplemented with annotations from Britten's idl_data project. The supplemented dataset can be loaded using a Python noscript:
from datasets import load_dataset
dataset = load_dataset("pixparse/idl-wds")
PDFA is a set of documents filtered from the SafeDocs corpus, aka CC-MAIN-2021-31-PDF-UNTRUNCATED. This corpus is intended for comprehensive analysis of pdf documents. The supplemented dataset can be loaded using a Python noscript:
from datasets import load_dataset
dataset = load_dataset("pixparse/pdfa-eng-wds")

📊📉Selection of Python libraries for working with spatial data
Earth Engine API - allows you to access Google Earth Engine's vast collection of geospatial data and perform analysis tasks using Python.
TorchGeo (PyTorch) - Provides tools and utilities for working with geospatial data in PyTorch.
Arcpy (Esri) is a Python library provided by Esri for working with geospatial data on ArcGIS platform. It allows you to automate geoprocessing tasks and perform spatial analysis.
Rasterio is a library for reading and writing geospatial raster datasets. It provides efficient access to raster data and allows you to perform various operations with geodata.
GDAL (Open-Source Geospatial Foundation) is a powerful library for reading, writing and manipulating geospatial raster and vector data formats.
Shapely is a library for geometric operations in Python. It allows you to create, manipulate and analyze geometric objects.
RSGISLib - has functions for processing thermal images, including radiometric correction, earth surface temperature estimation.
WhiteboxTools is a library for geospatial analysis and data processing. It offers a complete set of tools for tasks such as terrain analysis, hydrological modeling and LiDAR data processing.

💡😎📊Open Source Synthetic Text-to-SQL Dataset
Gretel releases largest open source Text-to-SQL dataset to speed up training of AI models
As of April 2024, the dataset is believed to be the largest and most diverse synthetic text-to-SQL conversion dataset available today, according to the developers.
The dataset contains approximately 23 million tokens, including approximately 12 million SQL tokens, and a wide range of SQL complexity levels, including subqueries, single joins, multiple joins, aggregations, window functions, and set operations.
To load a dataset via the Python API, you need to write the following noscript:
from datasets import load_dataset
dataset = load_dataset("gretelai/synthetic_text_to_sql")

😎📊Data used when training the MA-LMM model
MA-LMM (Memory-Augmented Large Multimodal Model) is a large memory-augmented multimodal model for understanding the context of long videos.
The model allows the use of long context by significantly reducing GPU memory usage. Instead of trying to process more frames at once, like most existing models, MA-LMM processes video online while storing past information in a memory bank.
The data on which the model was trained was made publicly available. This data consists of 2 very large datasets that can be downloaded from this link

💡A large dataset for speech detection with a size of more than 150 thousand hours in 6000+ languages has been published
The dataset contains about 150 thousand hours of audio in more than 6,000 languages. The number of unique ISO codes in a given dataset does not coincide with the actual number of languages, since similar languages can be encoded with the same code.
The data was labeled for the voice detection task at a time sampling of approximately 30 milliseconds (or 512 samples at a sampling rate of 16 kilohertz).

😎💡Where can I get the data? Multiple open repositories
Awesome Data - Github repository. List of open datasets with direct download links. There is data with videos, pictures, audio, and everything in general.
Open ML is a source that includes 20k+ datasets. There are also libraries for Python and R.
Open Data Registry - data warehouse from AWS. There are some datasets here that you won't find anywhere else.
Papers with Code. - collections of datasets that were used in real studies
Dagshub - a repository in which Datasets are conveniently divided by application areas (NLP, CV, etc.)

💡SQL vs NoSQL: advantages and disadvantages
SQL and NoSQL are the two main approaches to data storage and processing. Each has its own advantages and disadvantages, and the choice between them depends on the specific needs of the project. Let's look at the main differences between them.
SQL (Structured Query Language) databases are relational databases as well as DBMS of the Hadoop ecosystem that use structured tables to store data.
Benefits of SQL:
1. Data structure: Tables, relationships and diagrams make data easily understandable and manageable.
2. ACID Consistency: SQL databases ensure compliance with ACID principles (atomicity, consistency, isolation, durability), ensuring transaction reliability.
3. Universal Query Language: SQL provides a rich and versatile set of tools for performing complex queries and analytics. In some DBMSs there may be only slight deviations in the form of SQL dialects
Disadvantages of SQL:
1. Horizontal scaling: Traditional SQL databases often face scaling limitations when dealing with large volumes of data.
2. Schema Complexity: Changing the data schema can be a complex and costly process.
3. Limited flexibility: Some SQL databases may have restrictions on data types or structures that may not be suitable for some types of data.
NoSQL databases, on the other hand, do not use traditional tables but instead offer flexible data models.
Advantages of NoSQL:
1. Data structure flexibility: NoSQL databases can easily scale and change without the need to rebuild the schema.
2. Horizontal scaling: Many NoSQL databases easily scale horizontally, providing high performance for large volumes of data.
3. Support for unstructured data: NoSQL databases are well suited for storing and processing unstructured data such as text, images and videos.
Disadvantages of NoSQL:
1. Lack of ACID support: Many NoSQL databases sacrifice ACID consistency for performance or flexibility.
2. Consistency Difficulty: When scaling and distributing data, NoSQL databases can face challenges maintaining data consistency.
Thus, depending on the project requirements and priorities for performance, scalability and flexibility, the choice between SQL and NoSQL databases may be different.

💡Dataset for detecting problems in code
SWE-bench is a dataset that was designed to provide a diverse set of codebase issues that could be verified using unit tests in repositories. The full SWE-bench split includes 2,294 issue-commit pairs across 12 python repositories.
Thus, the dataset offers a new task: solving problems in the presence of a complete repository and an issue on GitHub.
To load a dataset using a Python noscript, you can use the following command:
from datasets import load_dataset
dataset = load_dataset("princeton-nlp/SWE-bench")

📊😎💡🤖Dataset catalog for object detection and segmentation
SAM + Optical Flow = FlowSAM
FlowSAM is a new tool for detecting and segmenting moving objects in video, which significantly outperforms all previous models, both for a single object and for multiple objects
To train the model, many datasets were used, which became available for download at this link

📉📊Selection of tools for working with Big Data
Drill - Layers on top of multiple data sources, allowing users to query a wide range of information in a variety of formats, from Hadoop sequence files and server logs to NoSQL databases and cloud-based object stores.
Druid (https://druid.apache.org/) is a real-time analytics database that provides low query latency, high concurrency, multi-user capabilities, and instant visibility into streaming data. According to its proponents, multiple end users can simultaneously query data stored in Druid without any performance impact.
HPCC Systems is a big data platform developed by LexisNexis and open sourced in 2011. In accordance with its full name - High-Performance Computing Cluster - the technology is essentially a cluster of computers created on the basis of standard hardware for processing, managing and delivering big data.
Iceberg is an open table format used to manage data in lakes, achieved in part by tracking individual files of information in tables rather than directories. Created by Netflix for use with its petabyte-sized tables, Iceberg is now an Apache project. Iceberg is typically "used in production, where a single table can contain tens of petabytes of data."
Kylin is a distributed information storage and analytics platform for big data. It provides an analytical information processing (OLAP) engine designed to work with very large data sets. Because Kylin is built on top of other Apache technologies, including Hadoop, Hive, Parquet and Spark, its proponents say it can easily scale to handle large volumes of data.
Samza is a distributed stream processing system created by LinkedIn and is currently an open source project managed by Apache. The system can run on top of Hadoop YARN or Kubernetes, and a standalone deployment option is also offered. According to the developers, Samza can process "several terabytes" of data state information with low latency and high throughput for fast analysis.

💡Selection of ETL services for Big Data
Renta Marketing ETL - A cloud solution that allows you to integrate 28 enterprise data sources with popular data warehouses like Snowflake and BigQuery, the service allows a team of engineers and analysts to integrate third-party tools and create pipelines in a couple of minutes data without code. For example, you can set up Facebook Ads integration with BigQuery in four clicks. There is no need to involve developers to work with Renta Marketing ETL.
Fivetran - Cloud-based software that allows users to quickly and easily create pipelines. The platform supports more than 90 sources. Fivertran provides a set of ready-made integrations, so even novice developers can understand the service.
Hevo Data - the service provides users with more than 150 ready-made integrations. You can set up integrations in three simple steps. The result is a pipeline that copies data to storage and requires no maintenance.
Matillion is a low-code application for creating pipelines. With Matillion, teams can create pipelines and automate data processing. The service has a simple interface, so a user who is far from programming can create and change data. Marillion supports real-time processing. The tool supports popular data sources and makes it easy to identify and resolve data problems.
Supermetrics is an ETL solution designed for small businesses and marketers who primarily use Facebook Ads, Google Ads and Google Analytics. The tool has a built-in application on the Google Cloud Platform that allows you to export data directly to Google BigQuery.

🌎TOP DS-events all over the world in May
May 4 - SQL Saturday - Jacksonville, USA - https://sqlsaturday.com/2024-05-04-sqlsaturday1068/
May 7-9 - Real-Time Analytics Summit - San Jose, USA - https://www.rtasummit.com/
May 8 - Data Connect West - Portland, USA - https://www.dataconnectconf.com/dccwest/conference
May 8-9 - UNLEASH THE POWER OF YOUR DATA - Boston, USA - https://www.dbta.com/DataSummit/2024/default.aspx
May 8-9 - Data Innovation Summit - Dubai, UAE - https://mea.datainnovationsummit.com/
May 9 - Conversational AI Innovation Summit - San Francisco, USA - https://confx-conversationalai.com/
May 15-17 - World Data Summit - Amsterdam, The Netherlands - https://worlddatasummit.com/#up
May 16 - Spatial Data Science Conference 2024 - London, UK - https://spatial-data-science-conference.com/2024/london
May 18 - DSF MAYDAY - London, UK - https://datasciencefestival.com/event/mayday-2024/
May 21 - Deployment, Utilization & Optimization of Enterprise Generative AI - Silicon Valley, USA - https://ent-gen-ai-summit-west.com/events/enterprise-generative-ai-summit-west-coast
May 23-24 - The Data Science Conference - Chicago, USA - https://www.thedatascienceconference.com/