👍🏻Sentiment analysis in social networks in Python with VADER without developing an ML model
Not every classification problem needs machine learning models: sometimes even simple approaches can give excellent results. For example, VADER (Valence Aware Dictionary and sEntiment Reasoner) is a vocabulary and rule based sentiment analysis model. The project source code is available on Github under the MIT license: https://github.com/cjhutto/vaderSentiment
VADER can efficiently handle dictionaries, abbreviations, capital letters, repetitive punctuation marks, emoticons (😢 , 😃 , 😭, etc.), etc., which are commonly used in social networks to express sentiment, making it an excellent text sentiment tool. The advantage of VADER is that it evaluates the mood of any text without prior training of ML models. The result generated by VADER is a dictionary of 4 keys neg, neu, pos and components (compound):
• neg, neu and pos mean negative, neutral and positive respectively. Their sum must be equal to 1 or close to it in a floating point operation.
• Compound corresponds to the sum of the valency scores of each word in the lexicon and determines the degree of mood, and not the actual value, unlike the previous ones. Its value ranges from -1 (the strongest negative mood) to +1 (the strongest positive mood). The use of a composite score may be sufficient to determine the main tone of the text. Compound ≥ 0.05 for positive mood, compound ≤ -0.05 for negative mood, compound ranges from -0.05 to 0.05 for neutral mood
Try Google Colab: https://colab.research.google.com/drive/1_Y7LhR6t0Czsk3UOS3BC7quKDFnULlZG?usp=sharing
Example: https://towardsdatascience.com/social-media-sentiment-analysis-in-python-with-vader-no-training-required-4bc6a21e87b8

🦋Yandex DataLens: Lightweight BI from Yandex.Cloud
Yandex DataLens is a free data visualization and analysis service. Main features:
• many data sources: ClickHouse, PostgreSQL, Greenplum, MySQL, CSV files, Google spreadsheets, Metrica and AppMetrica in direct access mode;
• diagrams, tables and data access UI elements for building dashboards;
• support for geodata and integration with maps;
• easy creation of the necessary dashboards without deep knowledge in DS;
• all documentation in Russian and a lot of understandable demos.
Service: https://cloud.yandex.ru/services/datalens
Documentation: https://cloud.yandex.ru/docs/datalens/quickstart

💥Fusion plasma control with DL
To solve the global crisis, find sources of clean, limitless energy. For example, nuclear fusion, which powers the stars in the universe. On earth, atomic batteries can be used for this, breaking and fusing them under extreme conditions in a tokamak device - a vacuum surrounded by magnetic coils. In it the plasma radiation is hotter than the core of the Sun. The norm of the device in the operating mode is very difficult: the control system must coordinate many magnetic current coils and the voltage on them is several times less in order to achieve that the plasma never touches the walls of the vessel, which can lead to heat loss and, possibly, loss. Deep reinforcement learning has been successfully applied to this problem to create controllers that maintain plasma stability and stable control of various shapes.
Existing control systems for plasma complications and requiring rare control for each of the subsequent magnetic coils. Each controller uses algorithms to evaluate plasma properties in real time and measure magnet voltages. The architecture from the renowned Deep Mind AI Center and the Swiss Center for Plasma Research uses a single neural network to control all coils simultaneously, automatically judging which voltages are best for building plasma, especially with sensors.
https://deepmind.com/blog/article/Accelerating-fusion-science-through-learned-plasma-control

🤦🏼‍♀️Biopass - REST API of a SaaS product for face recognition
Biopass is a platform for processing biometric data and artificial intelligence for creating ID products. The Biopass ID RESTfull API allows developers to enroll, manage, verify people, match and extract biometric data, manage fingerprint image compression and decompression, face detection, analyze face fakes, anonymize faces, and perform quality checks.
BioPass ID is an online cloud service that provides powerful multi-biometric and artificial intelligence technology for the development of any Internet-enabled service, software or platform. As a SaaS (Biometrics as a Service) product, BioPass ID supports any programming language, sensor model, camera or platform, enabling fast and easy implementation and system integration.
Images are common options in BioPass ID requests. To send them in API requests, you need to encode them into base64 strings. If the string is not in base64 string format, the call will return a bad request response with the message "Invalid JSON format".
https://www.biopassid.com/

🤜🏻ML to protect against DDos attacks
Machine learning algorithms are actively used in cybersecurity, for example, to identify atypical user behavior due to unauthorized access. ML can also be used to protect against DDOS attacks. The goal of a DDoS attack is to disrupt an organization by flooding a network, Internet-connected service, or technical infrastructure surrounding the target with unwanted traffic. The amount of traffic directed to the target can severely limit or disable availability.
DDoS attacks use Internet-connected devices that have already been compromised by malware. An attacker exploits existing vulnerabilities in dozens, hundreds, thousands, or even millions of devices to gain remote control. Thanks to the ubiquity of IoT devices, when even a home refrigerator goes online, protection against DDOS attacks is relevant for both businesses and private households.
A 2017 Kaspersky Lab survey found that the cost of sustaining a DDoS attack for small and medium businesses was $120,000. For large enterprises, this figure has risen to $2 million. And a 2018 study estimated the cost of downtime for a large organization to range from $300,000 to $540,000. In the US, the average global cost of a data breach was $8.46 million, according to a 2020 IBM report.
Using ML, you can build a binary classification model that would mitigate the impact of a DDoS attack on an organization's activities by correctly distinguishing safe traffic from malicious traffic. Here it is necessary to reduce the rate at which the ML model incorrectly identified safe traffic as malicious, as well as mitigate the consequences of a DDoS attack by correctly identifying malicious traffic with a probability of at least 90%.
Implementation example with Dask, XGBoost and Stacked Ensembling:
https://towardsdatascience.com/mitigating-ddos-attacks-with-classification-models-aa75ea813d85

📝MLOps basics: 5 formats for transferring ML models
For ML systems, portability between different stages of the life cycle, from development to deployment in production, is important. For example, a Data Scientist writes code in notebooks like Jupyter Notebook or Google Colab. When porting this code to a production environment, it should be converted to a lightweight interchange format, compressed and serialized, that is independent of the development language. These formats are as follows:
• Pickle is a binary version of a Python object for serialization and deserialization of its structure, ie. converting a hierarchy of Python objects to a stream of bytes and vice versa;
• ONNX (Open Neural Network Exchange) is an open source format for ML models that provides a common set of operators and a universal file format for various platforms and tools. The ONNX format describes a computation graph (input, output, and operations) and is self-contained. It is deep learning focused, supported by Microsoft and Facebook, and works great with TensorFlow and PyTorch.
• PMML (Predictive Model Markup Language) is an XML-based predictive model exchange format that allows you to develop a model in one system for one application and deploy it to another using another application by passing an XML configuration file.
• PFA (Portable Format for Analytics) is a standard for statistical models and data transformation engines that is easily portable between different systems and models. Pre-processing and post-processing functions can be chained together and built into complex workflows. A PFA can be a simple raw data transformation or a complex set of parallel data mining models with a JSON or YAML configuration file.
• NNEF (Neural Network Exchange Format) is a format that facilitates the process of deploying machine learning, allowing you to use a set of neural network training tools for applications on various devices and platforms.
There are also framework-specific formats, such as POJO/MOJO for the H2O AutoML platform and Spark MLWritable for Apache Spark.

🦋Useful ML Services: Everypixel API for Image Recognition
We continue to get acquainted with useful ML tools. Meet the Everypixel API, a simple yet powerful visual recognition method that uses machine learning to understand images.
The API uses a set of pre-trained models that parse images and return useful information. It processes images and then tags them with relevant keywords, which helps in their categorization and moderation. In addition, it evaluates images according to their quality and aesthetic value. Great for online stores and marketplaces to complement product and image data. Allows you to upload images without writing denoscriptions, as they are filled in automatically. Thanks to the generation of keywords for images, it will help in SEO tasks, and the categorization of images will improve search and directory navigation.
Pros of Everypixel API:
• works even when the end user takes a picture at the wrong angle or in poor lighting conditions;
• sees images the way a person sees them;
• can create keywords associated with images;
• selects the best shot from several similar photos;
• Can rate images from 0 to 100 depending on their quality.
Disadvantages of Everypixel API:
• The free plan is limited to 100 requests per day;
• cannot rate historical photographs, illustrations, or 3D renderings.
https://labs.everypixel.com/api

🌸TOP-15 Data Science conferences in April 2022:
• Apr 5-6, Healthcare NLP Summit (Online training takes place Apr 12-15) https://www.nlpsummit.org/healthcare-2022/
• Apr 6, Google Data Cloud Summit. Virtual. https://cloudonair.withgoogle.com/events/summit-data-cloud-2022
• Apr 13-14, Unite 2022: The Collaborative Intelligence Summit. Atlanta, GA, USA. https://unite2022.com/
• Apr 13, Analytics Summit 2022. Cincinnati, OH, USA. https://web.cvent.com/event/c6511810-01df-4e56-8c98-9c649301e3e4/
• Apr 14-16, WAICF: World AI Cannes Festival. Cannes, France. https://worldaicannes.com/
• Apr 19-21, ODSC East: Open Data Science, Boston, MA, USA. https://odsc.com/boston/
• Apr 20, DSS Virtual: AI & ML in the Enterprise. Virtual. https://www.datascience.salon/virtual-ai-and-ml-enterprise/
• Apr 21-22, RE.WORK AI in Finance Summit. New York, NY, USA https://www.re-work.co/events/ai-in-finance-summit-new-york-2022
• Apr 21-22, RE.WORK AI in Insurance Summit. New York, NY, USA https://www.re-work.co/events/ai-in-insurance-summit-new-york-2022
• Apr 25-27, Data Governance, Quality, and Compliance https://tdwi.org/events/seminars/april/data-governance-quality-compliance/home.aspx
• Apr 25-26, Chief Data & Analytics Officers, APEX East. Fort Myers, FL, USA. https://cdao-apex-east.coriniumintelligence.com/
• Apr 25-29, International Conference on Learning Representations (ICLR) https://www.iclr.cc/Conferences/2022
• Apr 26-27, Insurance AI & Innovative Tech USA 2022. Chicago, IL, USA. https://events.reutersevents.com/insurance/insuranceai-usa
• Apr 27, 4-6PM GMT, Natural Language Generation: Financial services, humans + AI together. London, UK. https://www.meetup.com/london-nlg-meetup-group/events/284525082/
• Apr 27, Computer Vision Summit. San Jose, CA, USA. https://computervisionsummit.com/

🙌🏻Generation of 3D scenes from 2D photos with NVIDIA's NeRF
Inverse rendering has long used AI to approximate the behavior of light in the real world, allowing a 3D scene to be reconstructed from multiple 2D images taken from different angles. The NVIDIA research team has developed an approach that solves this problem almost instantly by combining ultra-fast neural network training and fast rendering.
NVIDIA has taken this approach to a popular new technology called Neural Radiation Fields, or NeRF. The result, dubbed Instant NeRF, is the fastest NeRF technology to date, achieving over 1000x speedup in some cases. It only takes a few seconds for the model to learn from a few dozen still photos - plus the camera angles they were taken from - and then it can render the resulting 3D scene in tens of milliseconds.
NeRFs use neural networks to represent and render realistic 3D scenes based on an input collection of 2D images. Collecting data for NeRF transmission is reminiscent of the work of a photographer on the red carpet: the neural network needs several dozen images taken from different points of the scene, as well as the position of the camera of each of them.
Typically, creating a 3D scene using traditional methods takes several hours or more, depending on the complexity and resolution of the rendering. Bringing AI into the picture speeds things up. Early NeRF models rendered crisp, artifact-free scenes in minutes, but took hours to learn. Instant NeRF reduces rendering time by several orders of magnitude. It is based on multi-resolution hash mesh encoding that is optimized to run efficiently on NVIDIA GPUs. This way you can achieve high-quality results using a fast and small neural network.
The model was developed using the NVIDIA CUDA toolkit and the Tiny CUDA neural network library. Due to its lightness, the neural network can be trained and run on a single NVIDIA GPU - it runs fastest on cards with NVIDIA Tensor Cores.
This technology will be useful for training robots and self-driving cars so that they can understand the size and shape of objects in the real world by capturing 2D images or video recordings of them. It can also be used in architecture and entertainment to quickly create digital representations of real environments that creators can modify and use.
https://blogs.nvidia.com/blog/2022/03/25/instant-nerf-research-3d-ai/

🗣👂🏻Noise Reduction in Quantum Computing: An MIT Study
Quantum computers are very sensitive to noise interference caused by imperfect control signals, environmental disturbances, and unwanted interactions between qubits. Therefore, researchers at MIT have created QuantumNAS, a framework that can identify the most robust quantum circuit for a particular computational problem and generate a mapping pattern tailored to the target quantum processor's qubits. device. QuantumNAS is much less computationally intensive than other search methods and can identify quantum circuits that improve the accuracy of machine learning and quantum chemistry problems. In classical neural networks, including more parameters often improves model accuracy. But in variational quantum computing, more parameters require more quantum gates, which introduces more noise.
To do this, a super-circuit was first designed with all possible parameterized quantum elements in the design space. This circuit was then trained and used to search for circuit architectures with high noise tolerance. The process includes a simultaneous search for quantum circuits and qubit mappings using an evolutionary search algorithm. This algorithm generates several candidates for displaying quantum circuits and qubits, and then evaluates their accuracy using a noise model or on a real machine. The results are fed back into the algorithm, which chooses the most efficient parts and uses them to restart the process until it finds the perfect candidates. The developers have collected the results of the study into the TorchQuantum open source library https://github.com/mit-han-lab/torchquantum.
https://news.mit.edu/2022/quantum-circuits-robust-noise-0321

📝Auto-generate summaries from Google Docs
Google Docs now automatically generate summaries of their content. summaries of content when available. While all users can add summaries, auto-generated suggestions are currently only available to Google Workspace business customers.
This is achieved through natural language understanding (NLU) and natural language generation (NLG) ML models, especially Transformer and Pegasus. A popular technique for combining NLU and NLG is to train a machine learning model using sequence-to-sequence learning, where the input is the words of the document and the output is the final words. The neural network then learns to map input tokens to output tokens. Early applications of the sequence-to-sequence paradigm used recurrent neural networks (RNNs) for both the encoder and decoder.
The introduction of Transformers has provided a promising alternative to RNNs due to internal attention for better modeling of long input and output dependencies, which is critical when summarizing documents. However, these models require large amounts of manually labeled data for sufficient training, so the appearance of Transformers alone was not enough to make significant progress in the field of document summarization.
The combination of Transformers with self-supervised preconditioning (BERT, GPT, T5) has led to major breakthroughs in many NLU problems for which limited labeled data is available. In self-supervised pre-learning, the model uses large amounts of unlabeled text to learn general language understanding and generation capabilities. Then, in a subsequent fine-tuning step, the model learns to apply these abilities to a specific task, such as debriefing or answering questions.
Pegasus' work takes this idea one step further by introducing a pre-workout goal tailored to abstract generalization. In Pegasus pre-training, also called Sentence Gap Prediction (GSP), full sentences from untagged news articles and web documents are masked from the input and a model is required to reconstruct them based on the remaining untagged sentences. In particular, GSP attempts to mask sentences that are considered important to the document with various heuristics to make pre-training as close to a debriefing task as possible. Pegasus has achieved state-of-the-art results on a diverse set of summation datasets.
Taking advantage of Transformer and Pegasus, the Google AI researchers carefully cleaned and filtered the fine-tuning data to contain training examples that were more consistent and presented a coherent definition of the summary text. Despite the reduction in the amount of training data, this resulted in a better model. Then the problem of maintaining a high-quality model in production was solved. Although the Transformer version of the encoder-decoder architecture is the dominant approach to model training for sequential sequence transformation problems such as abstract summation, it can be inefficient and impractical for use in real world applications. The main inefficiency is associated with the Transformer decoder, where the output summary token is generated sequentially through autoregressive decoding. The decoding process becomes noticeably slower as summaries get longer as the decoder processes all previously generated tokens at each step. RNNs are a more efficient architecture for decoding, since there is no internal attention when using the previous tokens, as in the Transformer model.
After transferring knowledge from a large model to a more efficient smaller model to transform the Pegasus model into a hybrid architecture of the Transformer encoder and RNN decoder, the number of layers of the RNN decoder was reduced to improve efficiency. The resulting model has improved delays and memory, while maintaining the original quality.
https://ai.googleblog.com/2022/03/auto-generated-summaries-in-google-docs.html

📝Dataframe validation with Pandera
In large DS projects, the Great Expectations framework can be used to validate the dataset and check the quality of the data. However, smaller tasks require simpler tools. For example, the lightweight Python library Pandera, which explicitly checks information in dataframes at runtime. Pandera allows you to define a data schema once using a class-based API with pydantic syntax and use it to validate various types of dataframes, including pandas, dask, modin, and pyspark.pandas. You can check the types and properties of columns in pd.DataFrame or values in pd.Series, perform more complex statistical testing such as hypothesis testing. You can synthesize data from schema objects for property-based testing using pandas data structures.
Function decorators allow you to integrate with existing data analysis/processing pipelines using function decorators. With lazy validation, you can validate dataframes before errors occur. Finally, compatibility with other Python tools such as pydantic, fastapi, and mypy makes Pandera a useful tool for the ML developer and data analyst.
Documentation: https://pandera.readthedocs.io/en/stable/
Example: https://towardsdatascience.com/validate-your-pandas-dataframe-with-pandera-2995910e564

💥Why you need Modin: Pandas alternative for fast big data processing
Handling large frames of data with Pandas is slow because this Python library does not support working with data that does not fit in available memory. As a result, Pandas workflows that work well for prototyping a few MB of data don't scale to a real or hundreds of real GB dataset. Therefore, due to the single-threaded execution of operations in RAM, Pandas is not very suitable for processing really large data sets. with a wide range of data. There is an alternative - the Modin, Python-library with a Pandas-like API that scales to all processor cores using the Dask or Ray engine.
Modin supports working with data that won't fit in, so you can comfortably work with hundreds of GB without worrying about massive memory slowdowns or memory errors. With support for the cluster and beyond the core, Modin represents the use of a DataFrame with exceptional performance on a single node and high scalability in a cluster.
In the context of an algorithm (no cluster), Modin will create and manage a local (Dask or Ray) cluster for execution. There is no need to suggest how to evaluate the data, or even know how many cores the system has. Extraction, you can use code with Pandas by simply changing the library import statement from pandas to modin.pandas and getting a significant speedup even on a single machine. Modin speeds up to 4x on a laptop with 4 main cores.
Docs: https://modin.readthedocs.io/en/latest/index.html
Github: https://github.com/modin-project/modin

Z-scoring for simple and fast anomaly detection
Anomaly detection is a fairly common problem that covers many scenarios, from financial fraud to computer network failures. Some problems require complex machine learning models, but most often some simpler and cheaper methods are sufficient. For example, you have sales data over a period of time where you want to flag days with abnormally high volumes or highlight customers with abnormally high credit card swipes for risk testing.
For such cases, a simple statistical method of marking outliers, called Z-scoring, will do. The score is equal to the difference between the current and mean values, divided by the standard deviation. Z-scoring assumes the classical normal distribution of random variables. Converting nominal scale values to a logarithmic scale will improve the ability of most ML models to discern relationships and improve the ability of Z-scores to flag outliers.

In practice the implementation of Z-score is very simple: it can be written as a small software noscript or even a set of SQL queries to quickly get a lightweight MVP and quickly test a hypothesis.
https://towardsdatascience.com/anomaly-detection-in-sql-2bcd8648f7a8