💥Top 5 Data Engineering Trends in 2022: Astronomer Research
Astronomer, which commercializes and promotes the popular batch automation tool for working with data, Apache AirFlow, conducted a series of interviews with experts in the field of data engineering to identify the most pressing trends in the IT field.
• Data lineage, Data provenance and Data Quality
• Decentralization of data across different contexts and teams, but within a single consistent infrastructure with centralization of resources
• Consolidation of data tools, including orchestration of processing pipelines
• Data Mesh, eliminating silos between processing teams through the connection of used platforms
• mutual integration of DataOps, MLOps, AIOps for more efficient and faster use of consistent data and tools for seamless work with them.
https://www.astronomer.io/blog/top-data-management-trends-2022

🗣SQL queries against csv file with csvkit
csvkit is a command line toolkit for converting and working with CSV files. This utility allows you to perform the following operations in plain Python:
• Convert Excel and JSON files to CSV
• Display only column names
• Slice data
• change the order of columns
• find rows with matching cells
• convert CSV to JSON
• generate summary statistics
• refer to CSV using SQL queries
• import data into databases and extract from them
• parse CSV data
• work with column delimiters
The pip package manager will help you install csvkit: pip install csvkit
And the syntax for accessing a CSV file via an SQL query on the command line will look like this:
csvsql --query "SQL Query Here - source file name as table name (without .CSV)" source_filename> target_filename
To use this in your Python noscript you should
1) first import CSVSQL from csvkit utility
from csvkit.utilities.csvsql import CSVSQL
2) further define the arguments as a list of values, for example:
args = ['--query', 'select distinct manufacturer from playground', 'payground.csv']
3) then call CSVSQL with arguments
result = CSVSQL (args)
3) finally, the results can be shown
print (result.main ())
https://csvkit.readthedocs.io/en/latest/index.html
https://medium.com/data-engineering-ramstkp/sql-queries-on-csv-using-python-24a472fe53b1

🚀Accelerating Big Data Analytics: Expedia Group Case Study with Apache Druid and DataSketches
When analyzing big data, problematic queries often arise that do not scale, since they require enormous computational resources and time to obtain accurate results. For example, counting individual items, quantiles, most frequent items, table joins in SQL queries, matrix calculations and graph analysis. If the approximate results for such calculations are acceptable, there are special streaming algorithms or sketches that run several orders of magnitude faster with acceptable errors. The sketches helped Yahoo successfully reduce processing time from days or hours to minutes or seconds. One such tool is the open-source library Apache DataSketches.
It is used by the large travel company Expedia Group to speed up time series analysis in Apache Druid, where table joins are limited, requiring a single dataset to be put into memory. DataSketches supports set operations, including join, intersection, and difference, with little loss in precision. This is useful when looking for and booking tickets. With DataSketches, each dataset can be queried independently of Druid to get the desired object for each dataset for preliminary and then final calculation. Since Druid did not initially support merging DataSketches objects, Expedia Group engineers had to write their own Java code. Moreover, the DataSketches object takes up very little memory space, despite the large size of the set. As a result, Apache Druid, a column-based DBMS for quickly receiving huge amounts of event data and submitting queries with low latency, became even faster.
https://datasketches.apache.org/
https://medium.com/expedia-group-tech/fast-approximate-counting-using-druid-and-datasketch-f5f163131acd

🌏5 Essential Components of Gartner's Digital Government Technology Platform
The Digital Government Technology Platform (DGTP) makes digital transformation a reality, but requires dedicated leadership. According to a Gartner study, by 2023, more than 80% of government digital implementations that are not based on a technology platform will fail.
DGTP is a set of end-to-end, integrated, horizontal capabilities that coordinate government services across multiple domains by integrating five platforms:
• Citizen Experience platform provides interfaces and technologies, implements policies and procedures for citizen-business interaction, and measures the experience of its users;
• Ecosystem platform – a set of digital interfaces that implement policies and procedures for governments and ecosystem partners to share data and services.
• Internet of Things (IoT) platform provides interfaces, data management and context, and implements policies and procedures for collecting and processing data from IoT sensors
• Information System Platform - Corporate information systems are at the heart of government IT efforts today. The information system platform provides the technologies, policies and procedures for integrating these back office systems into the DGTP
• Intelligence Platform provides advanced analytics, geospatial and location analytics, robotic process automation (RPA) and AI capabilities to process data collected or stored in any area of the platform.
The key reusable components in DGTP are applications and services that can provide a seamless mix of data, services, and capabilities that work together within DGTP and are accessible across networks and devices. DGTP is not a turnkey solution, but it gives government agencies the ability to innovate, reduce costs, and deliver new capabilities quickly and flexibly.
https://www.gartner.com/en/articles/government-cios-here-s-an-essential-piece-of-the-digital-transformation-puzzle

Компоненты технологической платформы цифрового правительства от Gartner

🍏Bayesian statistics with PyMC3: brief overview
Frequency statistics rely on long-term event rates (data points) to calculate the desired variable. The Bayesian method can also work without a lot of events, even with a single data point. Frequency analysis gives a point estimate, while Bayesian analysis gives a distribution that can be interpreted as the confidence that the mean of the distribution is a good estimate for the variable. However, there is an uncertainty in the form of the standard deviation.
The Bayesian approach is useful in ML problems where estimates and validity are important. For example, today it could rain with a 60% chance.” The main formula underlying the Bayesian approach is Bayes' theorem, which allows you to calculate the posterior probability P(A|B ) of event A depending on event B.
• P(B|A) is called the probability that if event A happened, how likely is event B to happen?
• P(A) – probability of event A, a prior (initial) assumption about the variable of interest.
• P(B) is the probability of event B (evidence), which is usually difficult to calculate when estimating the posterior probability.
You can quickly calculate the Bayesian probability using the PyMC3 Python library https://docs.pymc.io/en/v3/. It allows you to write models using an intuitive syntax to describe the data generation process. PyMC3 allows you to tune an ML model with gradient-based MCMC algorithms like NUTS, with ADVI for fast approximate inference, including a mini-batch ADVI for scaling to large datasets, or with Gaussian processes to build Bayesian non-parametric models. PyMC3 includes a complete set of predefined statistical distributions that can be used as the building blocks of a Bayesian model.
This probabilistic programming package for Python allows users to fit Bayesian models using various numerical methods, most notably Markov Chain Monte Carlo (MCMC) and Variational Inference (VI). Instead of providing a basic model specification and fitting functions, PyMC3 includes functions for summarizing output and diagnosing the model.
PyMC3 aims to make Bayesian modeling as simple and painless as possible by allowing users to focus on their scientific problem rather than the methods used to solve it. The package uses Theano as a computational backend to quickly evaluate an expression, compute the gradient automatically, and perform computations on the GPU.
PyMC3 also has built-in support for modeling Gaussian processes, allowing you to generalize models and build graphs. There's model validation and convergence detection, custom stepwise methods, and unusual probability distributions. Bayesian models obtained using PyMC3 can be embedded in larger programs, and the results can be analyzed using any Python tools.
https://medium.com/@akashkadel94/bayesian-statistics-overview-and-your-first-bayesian-linear-regression-model-ba566676c5a7

💥5 YOUTUBE channels for a data engineer from popular DS bloggers
• Ken Jee https://www.youtube.com/c/KenJee1/videos - 183 thousand subscribers and about 200 videos about Data Science, big data engineering, ML and sports analytics
• Karolina Sowinska https://www.youtube.com/c/KarolinaSowinska/videos 30+ thousand subscribers and almost 60 great videos about AirFlow, AI, ETL and the career of a data engineer;
• Shashank Mishra https://www.youtube.com/c/LearningBridge/video 40+ thousand subscribers and more than 150 videos about everyday life data engineers, DS course reviews, interview recommendations and personal experience of the author who worked at Amazon , McKinsey&Company, PayTm and other large corporations, as well as startups.
• Seattle Data Guy https://www.youtube.com/c/SeattleDataGuy/videos almost 20 thousand subscribers and more than 100 videos about the soft and hard skills of a data engineer, life hacks for solving daily tasks of collecting and aggregating data using Python and not only, SQL best practices, introduction to R and much more
• Andreas Kretz https://www.youtube.com/c/andreaskayy/videos about 27 thousand subscribers and more than 500 videos vanilla and proprietary Hadoop, Spark, Kafka, AWS services and other cloud platforms, ETL basics, installation details and practical use different Big Data technologies and features of the data engineer profession.

🏸Zingg + TigerGraph combo for deduplication and big data graph analytics
Graph databases with built-in relationship patterns are great for record disambiguation and entity resolution. For example, TigerGraph is a powerful graph analytics system. And if you supplement it with the open ML tool Zingg (https://github.com/zinggAI/zingg), you can find duplicate and ambiguous records even faster.
Imagine, the same person in different systems is written differently. Therefore, it is very difficult to analyze its user behavior, for example, to generate a personal marketing offer or inclusion in loyalty programs. Zingg have built-in locking mechanisms that only calculate pairwise similarity for selected records. This reduces computation time and helps scale to large datasets. You don't have to worry about manually linking/grouping records: the internal entity resolution framework takes care of that. So with Zingg and TigerGraph you can combine the best simple and scalable entity resolution and further graph analysis.
https://towardsdatascience.com/entity-resolution-with-tigergraph-add-zingg-to-the-mix-95009471ca02

LaMDA: Safe, Grounded, and High-Quality Dialog Model from Google AI
LaMDA is created by fine-tuning a family of dialogue-specific Transformer-based neural language models with model parameters up to 137B and training the models to use external knowledge sources. LaMDA has three key goals:
• Quality, which is measured in terms of Sensibleness, Specificity, and Interestingness. These indicators are evaluated by people. Reasonableness indicates the presence of meaning in the context of the dialogue, for example, the absence of absurd answers from the ML-model and contradictions with earlier answers. Specificity indicates whether the system's response is specific to the context of the previous dialog. Interestingness measures the emotional reaction of the interlocutor to the answers of the ML model.
• Safety so that the model's responses do not contain offensive and dangerous statements.
• Groundedness - modern language models often generate statements that seem plausible, but in fact contradict the true facts in external sources. Groundedness is defined as the percentage of responses with statements about the outside world that can be verified by reputable external sources. A related metric, Informativeness, is defined as the percentage of responses with information about the outside world that can be confirmed by known sources.
LaMDA models undergo two-stage training: pre-training and fine-tuning. The first stage was performed on a data set of 1.56 thousand words from publicly available dialogue data and public web documents. After tokenizing the data set of 2.81T tokens, the model was trained to predict each next token in the sentence, given the previous ones. The pretrained LaMDA model has also been widely used for NLP research at Google, including program synthesis, zero-shot learning, and more.
In the fine-tuning phase, LaMDA is trained to combine generative tasks to generate natural language responses in given contexts and classification tasks to determine the safety and quality of the model. This results in a single multitasking model: the LaMDA generator is trained to predict the next token in the dialogue dataset, and the classifiers are trained to predict the security and response quality scores in context using annotated data.
The test results showed that LaMDA significantly outperforms the pre-trained model in every dimension and at every scale. Quality metrics improve as the number of model parameters increases, with or without fine-tuning. Safety is not improved by scaling the model alone, but compensated for by fine-tuning. Groundedness improves as the size of the model grows, due to the ability to remember unusual knowledge. And fine-tuning allows the model to access external sources and effectively transfer part of the burden of remembering knowledge to them. By fine-tuning, the human-level quality gap can be reduced, although the performance of the model remains below human-level in terms of safety and Groundedness.
https://ai.googleblog.com/2022/01/lamda-towards-safe-grounded-and-high.html

сравнение метрик LAMDA с человеческими оценками

👀Upscaling video games with NVIDIA's DLDSR
DLDSR (Deep Learning Dynamic Super Resolution) is a video game image enhancement technology that uses a multilayer neural network that requires fewer pixels. The 2.25X DLDSR is comparable in quality to the 4X resolution of previous generation DSR technology. At the same time, DLDSR performance is much higher thanks to the tensor cores of RTX video cards, which accelerate neural networks several times. You can try DLDSR on your gaming computer by updating your video card driver and setting the desired settings.
https://www.rockpapershotgun.com/nvidias-deep-learning-dynamic-super-resolution-tech-is-out-now-heres-how-to-enable-it

🌦TOP-10 Data Science conferences in February 2022:
1. 02 Feb - Virtual conference DataOps Unleashed https://dataopsunleashed.com/
2. 03 Feb - Beyond Big Data: AI/Machine Learning Summit 2022, Pittsburgh, USA https://www.pghtech.org/events/BeyondBigData2022
3. 10 Feb - Online-summit AICamp ML Data Engineering https://www.aicamp.ai/event/eventdetails/W2022021009
4. 12-13 Feb - IAET International Conference on Machine Learning, Smart & Nanomaterials, Design Engineering, Information Technology & Signal Processing. Budapest, Hungary https://institute-aet.com/mns-22/
5. 16 Feb - DSS Hybrid Miami: AI & ML in the Enterprise. Miami, FL, USA & Virtual https://www.datascience.salon/miami/
6. 17-18 Feb - RE.WORK San Francisco, CA, USA and Online
Reinforcement Learning Summit: https://www.re-work.co/events/reinforcement-learning-summit-2022
Deep Learning Summit: https://www.re-work.co/events/deep-learning-summit-2022 Enterprise AI Summit: https://www.re-work.co/events/enterprise-ai-summit-2022
7. 18-20 Feb - International Conference on Compute and Data Analysis (ICCDA 2022). Sanya, China http://iccda.org/
8. 21-25 Feb - WSDM'22, The 15th ACM International WSDM Conference. Online. http://www.wsdm-conference.org/2022/
9. 22-23 Feb - AI & ML Developers Conference. Virtual. https://cnvrg.io/mlcon
10. 26-27 Feb - 9th International Conference on Data Mining and Database (DMDB 2022). Vancouver, Canada https://ccseit2022.org/dmdb/

🚗Yandex Courier Robots in Seoul
As early as last year, Yandex's autonomous courier robots began delivering orders in Russia, food from restaurants in the US city of Ann Arbor, Michigan, and other US student campuses. And in January 2022, Yandex entered into an agreement of intent with a large South Korean telecommunications company, KT Corporation, for delivery by autonomous robots in Seoul. So already this year, South Korea will become the first country in East Asia where Yandex rovers operate. The company is also preparing to launch this technology in Dubai.
http://www.koreaherald.com/view.php?ud=20220118000709

🎂Terality - super fast serverless engine instead of slow Pandas
Terality is a serverless data processing engine that runs on giant clusters to work with datasets of any size. Thanks to the serverless paradigm, you don’t have to worry about scaling resources in clusters or other infrastructure: there are practically no limits on memory, and therefore on the size of a data set. It only needs a good internet connection to handle hundreds of GB, even on a simple office laptop with 4GB of RAM. Terality allows you to run Pandas code 10 times faster: Terality syntax is similar to Pandas. It only takes one line of code to switch from Pandas to Terality: import teratiyu as te. The Python package sends HTTPS requests to the Terality engine when you call Pandas functions. The engine processes the data and the command and returns the result. However, Terality is not just a Python package, but freemium software with a free 1TB plan. This counts every API call, not just data reads.
https://docs.terality.com/
https://towardsdatascience.com/good-bye-pandas-meet-terality-its-evil-twin-with-identical-syntax-455b42f33a6d

🥁🥁Undouble - Python library for detecting duplicate images using hash functions
Finding identical or similar photos manually is a long and tedious task. ITS can not be solved simply by comparing the size and file name, because. photos are taken from different sources (mobile devices, social networking applications, etc.), which results in differences in these attributes and creates differences in resolution, scaling, compression, and brightness. Hash functions are ideal for detecting identical and similar photos due to their resistance to minor changes. This idea is the basis of Undouble - the Python library that works using a multi-stage image preprocessing process (grayscale, normalization and scaling), image hash calculation, and image grouping. Threshold 0 will group images with identical image hash. The results can be easily examined using the plotting function, and the images can be moved using the move function. When moving images, the image from the group with the highest resolution is copied, and all other images are moved to the "undouble" subdirectory.
To try this open source library (https://github.com/erdogant/undouble), you first need to install it: pip install undouble, then import the package into your project: from undouble import Undouble. Then, by setting the hash method and hash size, duplicates can be detected using undouble. In this case, the following steps are performed: recursively reading all images from the directory with the specified extensions, computing the hash, and grouping similar images.
See an example with explanations here:
https://towardsdatascience.com/detection-of-duplicate-images-using-image-hash-functions-4d9c53f04a75

😜How to create an exe-file from a py-noscript
Although you can run a Python noscript in a terminal or text editor, sometimes you need to hide all the code in a py file by wrapping it inside an executable (.exe) file. For example, to schedule a job that runs an executable at a specific time. This can be done in 2 ways:
• via the GUI of the auto-py-to-exe package (https://pypi.org/project/auto-py-to-exe/), which must first be installed via pip install auto-py-to-exe, then run and follow 4 steps in sequence.
• in the terminal with the PyInstaller library, which should also be installed first: pip install pyinstaller, and then go to the directory with the desired py file and create an executable based on it using the pyinstaller --onefile name_of_noscript.py command
In fact, the first method is a visualization of the 2nd in a visual GUI. And those who work with the CLI interface can immediately use PyInstaller without any additional wrappers.
https://towardsdatascience.com/how-to-easily-convert-a-python-noscript-to-an-executable-file-exe-4966e253c7e9

😜Neural networks for selfies on Google Pixel 6: accurate alpha matting in portrait mode
Matting an image is the process of extracting a precise alpha mask that separates the foreground and background objects of an image. This is not only necessary for professional designers when designing advertising photos, but has also become a popular entertainment for smartphone users. Send friends a selfie with the Eiffel Tower in the background while in a room with a grandmother's carpet? Easy with Google Pixel 6: a convolutional neural network from a sequence of encoder-decoder blocks to gradually evaluate high-quality alpha matting will preserve all the details, including fine hairs.
The input RGB image is combined with a coarse alpha matte (generated with a low resolution people segmenter) which is passed as input to the network. The new Portrait Matting model uses the MobileNetV3 backbone and a shallow decoder with few layers to first predict the low resolution advanced alpha mask. Then a shallow codec and a series of residual blocks are applied to process the high resolution image and the refined alpha mask from the previous step. The shallow codec relies more on lower level functions than the previous MobileNetV3 backbone, focusing on high resolution structural functions to predict the final transparency values for each pixel. This way the model can refine the original foreground alpha mask and accurately extract very fine details. This neural network architecture works effectively on Pixel 6 using Tensorflow Lite. The ML model also uses a variety of training datasets that cover a wide range of skin tones and hairstyles.
https://ai.googleblog.com/2022/01/accurate-alpha-matting-for-portrait.html

🍏Clustimage - Python library for image clustering
Unsupervised clustering in image recognition is a multi-step process. It includes preprocessing, feature extraction, similarity clustering, and estimation of the optimal number of clusters using a quality measure. All of these steps are implemented in the Clustimage Python package, which takes only paths or raw pixel values as input.
The goal of clustimage is to detect natural groups or clusters of images using the ilhouette, dbindex and their derivatives methods, in combination with clustering methods (agglomerative, kmeans, dbscan and hdbscan). Clustimage helps you determine the most robust clustering by efficiently searching by parameter and evaluating clusters. In addition to image clustering, the model can also be used to find the most similar images for a new invisible sample.
To try this open source library (https://github.com/erdogant/clustimage), you first need to install it: pip install clustimage, then import the package into your project: from clustimage import Clustimage.
See an example with explanations here: https://towardsdatascience.com/a-step-by-step-guide-for-clustering-images-4b45f9906128

👍🏻Don't like documenting code? Give it to the AI!
AI Doc Writer is a VS Code extension that documents code using AI. Simply select the necessary lines of code in the development environment and press Cmd / Ctrl +. AI Doc Writer will create a short denoscription of each feature and options. The tool from Microsoft (https://marketplace.visualstudio.com/items?itemName=mintlify.document) supports Python, JavaScript, TypeScript, PHP and Java languages, as well as JSX and TSX files. Of course, this cannot be called software documentation in the form in which the Customer understands it, however, the presence of understandable comments makes the code maintainable and readable. More examples:
https://betterprogramming.pub/too-lazy-to-write-documentation-let-the-ai-write-it-for-you-8574f7cd11b2

🔥Hide a spicy photo from strangers? Easy with nudity detection API
Python program by DeepAI (https://deepai.org/machine-learning-model/nsfw-detector)
evaluates the image and estimates the likelihood that it covers areas of the human body that are usually found under clothing. The nudity check is a dynamic label, a certainty algorithm, or a false percentage. The user can set a different threshold in their app for what it considers to be nudity, and the image detection algorithm will return the percentage chance that the image contains natural content. This ML system is applicable not only to photos, but also to videos: neural networks analyze the video stream and give probabilistic feedback on the “adultness” of consumption.
See an example of using the API here: https://medium.com/mlearning-ai/i-tried-a-python-nude-detector-with-my-photo-446dba1bbfc8