😱3 types of data anomalies
Data analysts and machine learning professionals often detect anomalies in data – detection that is not detected to pattern detection and detection. There are 3 types of anomalies:
• point anomaly, when one data point (observation) in the data set is far from the sources of the others and represents an extreme, unevenness or deviation that occurs randomly and is not related to the overall load in the data. The point anomaly also eliminates the global outlier because it is significantly different from the rest of the dataset.
• context anomaly, when a particular instance is revealed to be observable from the context. For example, in the case of time series data, such as recording a certain amount over time, the context is temporal. Data points that are very different from other data in the same sense are due to contextual outliers. For example, when the number of cars passing through the checkpoint on the border of the region in March, on average, is 1 thousand over the past 20 years. And in June, when the vacation period begins, this number reaches 8 thousand. If at the beginning of March it is 9 thousand, it will be considered an anomaly, and in the summer it will not be an anomaly. It's common for retail to see shoppers pop up during the holiday season. But a sharp increase in sales outside of holidays or sales can be called a contextual outlier.
• Collective anomaly, where a group of correlated, interrelated, or sequential instances is significantly different from the rest of the data, ie these data points are judged to be anomalous. For time series data, this may look like typical peaks and troughs occurring outside of the time period when the seasonal sequence is normal, or as a set of time series that are in outlier conditions. For example, at the same time, along with a large number of companies, there is a drop in sales, although before that there was an upward trend.
https://medium.com/datadailyread/types-of-data-anomalies-2f6fb1747eb1
Data analysts and machine learning professionals often detect anomalies in data – detection that is not detected to pattern detection and detection. There are 3 types of anomalies:
• point anomaly, when one data point (observation) in the data set is far from the sources of the others and represents an extreme, unevenness or deviation that occurs randomly and is not related to the overall load in the data. The point anomaly also eliminates the global outlier because it is significantly different from the rest of the dataset.
• context anomaly, when a particular instance is revealed to be observable from the context. For example, in the case of time series data, such as recording a certain amount over time, the context is temporal. Data points that are very different from other data in the same sense are due to contextual outliers. For example, when the number of cars passing through the checkpoint on the border of the region in March, on average, is 1 thousand over the past 20 years. And in June, when the vacation period begins, this number reaches 8 thousand. If at the beginning of March it is 9 thousand, it will be considered an anomaly, and in the summer it will not be an anomaly. It's common for retail to see shoppers pop up during the holiday season. But a sharp increase in sales outside of holidays or sales can be called a contextual outlier.
• Collective anomaly, where a group of correlated, interrelated, or sequential instances is significantly different from the rest of the data, ie these data points are judged to be anomalous. For time series data, this may look like typical peaks and troughs occurring outside of the time period when the seasonal sequence is normal, or as a set of time series that are in outlier conditions. For example, at the same time, along with a large number of companies, there is a drop in sales, although before that there was an upward trend.
https://medium.com/datadailyread/types-of-data-anomalies-2f6fb1747eb1
Medium
Types of Data Anomalies
Do you know what type of anomalies are you dealing with?
👍5
👍🏻10 Best Practices for Naming Tables and Fields in a Database
If every developer and analyst followed these simple rules, reverse engineering would become a hobby, not a laborious job. To make it easier for you and your colleagues to work with the database, try these simple rules:
1. Separate words with underscores if the name attribute or database table consists of 2 or more words. This is a more stylish camelCase case, improved readability and conceptual platform dependency. For example, word_count.
2. Write full and semantically based names for tables and columns without reference to data types. Saving a couple of characters will do nothing but confuse. It is permissible to use benefits only where this is an abbreviated name for everyone.
3. Write the attribute name with a lowercase letter to avoid confusion from upper-case SQL keywords. It will also improve your typing speed.
4. Do not use numbers in the names of tables and columns.
5. Name the tables clearly, but briefly.
6. Name tables and columns in the singular. For example, author instead of authors.
7. Name the linking tables in alphabetical order. For example author_book
8. When an index is set, add its table and column name. For example, CREATE INDEX person_ix_first_name_last_name ON person (first_name, last_name);
9. For Boolean column type add prefix name with is_ or has_ . For example is_admin or has_membership.
10. For columns of Date-Time type, add suffix _at or _time to the name. For example, order_at or order_time.
https://dev.to/mohammadfaisal/how-to-design-a-clean-database-1e83
If every developer and analyst followed these simple rules, reverse engineering would become a hobby, not a laborious job. To make it easier for you and your colleagues to work with the database, try these simple rules:
1. Separate words with underscores if the name attribute or database table consists of 2 or more words. This is a more stylish camelCase case, improved readability and conceptual platform dependency. For example, word_count.
2. Write full and semantically based names for tables and columns without reference to data types. Saving a couple of characters will do nothing but confuse. It is permissible to use benefits only where this is an abbreviated name for everyone.
3. Write the attribute name with a lowercase letter to avoid confusion from upper-case SQL keywords. It will also improve your typing speed.
4. Do not use numbers in the names of tables and columns.
5. Name the tables clearly, but briefly.
6. Name tables and columns in the singular. For example, author instead of authors.
7. Name the linking tables in alphabetical order. For example author_book
8. When an index is set, add its table and column name. For example, CREATE INDEX person_ix_first_name_last_name ON person (first_name, last_name);
9. For Boolean column type add prefix name with is_ or has_ . For example is_admin or has_membership.
10. For columns of Date-Time type, add suffix _at or _time to the name. For example, order_at or order_time.
https://dev.to/mohammadfaisal/how-to-design-a-clean-database-1e83
DEV Community
How to Design a Clean Database
To read more articles like this, visit my blog 18 best practices to keep names simple and...
👍4
🔥Instead of Jupyter Notebook: Benefits of Deepnote
Jupyter notebooks have been actively using data analytics for many years and contain ML. However, despite its popularity, this research tool has significant implications:
• Difficulty in code versioning. Browsing Jupyter notebooks comes in the form of large JSON files, merging two notebooks is next to impossible. As is the usual use by developers of a Git-like tool version.
• Lack of high efficiency with IDE, code highlighting and tooltips. Usually a Data Scientist is not a professional software developer, and therefore tools that regulate the quality of the code and the value of its increase are very important.
• Difficulties of development through testing. The popular test-driven development methodology (test-driven development) is almost impossible to implement in Jupyter notebooks. Therefore, they cannot be used in data pipelines.
• Non-linear workflow due to transition from one cell to another. This may seem like an irreproducible experiment. The interactive way to encode and navigate between cells is both one of Jupyter Notebook's best features and its biggest weakness.
• Jupyter is not well suited for running long running asynchronous data scoping tasks.
😡
Many of the shortcomings are addressed as an alternative to Jupyter Notebook called Deepnote. Deepnote, like Jupyter, is an interactive notebook for solving a DS problem, but it outperforms its competitor in a number of advantages 💥:
• Real-time Collaboration - Created by Google Docs, you can share links to your notebook with colleagues, giving everyone the desired level of access (view, execute, comment, edit and full control). In addition, a cell in Deepnote allows a collaborator to leave comments, eliminating the need to switch between applications for posting messages and code for providing feedback. With access to the developer's code, managers and other board members easily follow the code development progress and development life cycle.
• Easy environment management deployment - Deepnote takes the job of installing modules and setting up the environment to run Python, including versions. In addition to Python, Deepnote also supports SQL queries.
• Deepnote has the ability to embed blocks of code in blogs and other repositories by implementing the creation of a GitHub project specifically for this purpose. Deepnote Cells allows you to embed code only, embed output only, and embed both code and output.
• Data Visualisation - Jupyter notebooks almost never evaluate EDA execution without explicit coding. Deepnote provides an early development tool in the notebook itself - the early block allows you to generate information, just like building Python, but without the need to write code.
• Save time and money - once Deepnote is in charge of code management and processing, teams don't need to commit their code pipelines to tools like GitHub, BitBucket, etc. thus reducing operating costs.
Try it for free: https://deepnote.com/
Jupyter notebooks have been actively using data analytics for many years and contain ML. However, despite its popularity, this research tool has significant implications:
• Difficulty in code versioning. Browsing Jupyter notebooks comes in the form of large JSON files, merging two notebooks is next to impossible. As is the usual use by developers of a Git-like tool version.
• Lack of high efficiency with IDE, code highlighting and tooltips. Usually a Data Scientist is not a professional software developer, and therefore tools that regulate the quality of the code and the value of its increase are very important.
• Difficulties of development through testing. The popular test-driven development methodology (test-driven development) is almost impossible to implement in Jupyter notebooks. Therefore, they cannot be used in data pipelines.
• Non-linear workflow due to transition from one cell to another. This may seem like an irreproducible experiment. The interactive way to encode and navigate between cells is both one of Jupyter Notebook's best features and its biggest weakness.
• Jupyter is not well suited for running long running asynchronous data scoping tasks.
😡
Many of the shortcomings are addressed as an alternative to Jupyter Notebook called Deepnote. Deepnote, like Jupyter, is an interactive notebook for solving a DS problem, but it outperforms its competitor in a number of advantages 💥:
• Real-time Collaboration - Created by Google Docs, you can share links to your notebook with colleagues, giving everyone the desired level of access (view, execute, comment, edit and full control). In addition, a cell in Deepnote allows a collaborator to leave comments, eliminating the need to switch between applications for posting messages and code for providing feedback. With access to the developer's code, managers and other board members easily follow the code development progress and development life cycle.
• Easy environment management deployment - Deepnote takes the job of installing modules and setting up the environment to run Python, including versions. In addition to Python, Deepnote also supports SQL queries.
• Deepnote has the ability to embed blocks of code in blogs and other repositories by implementing the creation of a GitHub project specifically for this purpose. Deepnote Cells allows you to embed code only, embed output only, and embed both code and output.
• Data Visualisation - Jupyter notebooks almost never evaluate EDA execution without explicit coding. Deepnote provides an early development tool in the notebook itself - the early block allows you to generate information, just like building Python, but without the need to write code.
• Save time and money - once Deepnote is in charge of code management and processing, teams don't need to commit their code pipelines to tools like GitHub, BitBucket, etc. thus reducing operating costs.
Try it for free: https://deepnote.com/
Deepnote
Deepnote: Analytics and data science notebook for teams.
Explore data with Python & SQL, work together with your team, and share insights that lead to action — all in one place with Deepnote.
👍4
💥Instead of loops: 3 Python life hacks
Developers and data scientists know that loops in Python are slow. Instead, you can use possible alternatives:
• Map - apply a function to each value of an iterable object(list, tuple, etc. ).
• Filter - to filter out values from an iterable object (list, tuple, sets, etc.). The filtering conditions are set inside a function which is passed as an argument to the filter function.
• Reduce – this function is a bit different from the map and filter functions. It is applied iteratively to all the values of the iterable object and returns only one value.
Examples: https://medium.com/codex/3-most-efficient-yet-underutilized-functions-in-python-d865ffaca0bb
Developers and data scientists know that loops in Python are slow. Instead, you can use possible alternatives:
• Map - apply a function to each value of an iterable object(list, tuple, etc. ).
• Filter - to filter out values from an iterable object (list, tuple, sets, etc.). The filtering conditions are set inside a function which is passed as an argument to the filter function.
• Reduce – this function is a bit different from the map and filter functions. It is applied iteratively to all the values of the iterable object and returns only one value.
Examples: https://medium.com/codex/3-most-efficient-yet-underutilized-functions-in-python-d865ffaca0bb
Medium
Don’t Run Loops in Python, Instead, Use These!
No need to run loops in Python anymore
👍4
🤔Python-library to calendar operations
Python includes a built-in calendar module that includes an operation related to dates and days of the week. The functions and classes use the European calendar module, where Monday is the first day of the week and Sunday is Sunday.
To use this feature, you must first import it into your code:
import calendar
You can then call a function, for example print the names of the months in a list:
month_names = list(calendar.month_name[1:])
print(month_names)
https://docs.python.org/3/library/calendar.html
Python includes a built-in calendar module that includes an operation related to dates and days of the week. The functions and classes use the European calendar module, where Monday is the first day of the week and Sunday is Sunday.
To use this feature, you must first import it into your code:
import calendar
You can then call a function, for example print the names of the months in a list:
month_names = list(calendar.month_name[1:])
print(month_names)
https://docs.python.org/3/library/calendar.html
👍2❤1
#test
The probability that the test correctly rejects the null hypothesis when a specific alternative hypothesis is true is called
The probability that the test correctly rejects the null hypothesis when a specific alternative hypothesis is true is called
Anonymous Quiz
59%
power of a binary hypothesis test
38%
type II error
3%
confusion matrix
0%
random value
👍3
🗒Need to log Python application events? There is a special module!
Python library logging (https://docs.python.org/3/library/logging.html) defines functions and classes that implement a flexible event logging system for applications and the library. The main advantage of the logging API, an extension of this standard library, is the ability to log all events. Therefore, the Python application log can display native messages inline with messages from external modules.
The module consists of the following classes:
• Registrars require an interface that uses application code.
• Handlers send log entries (created by registrars) to the appropriate destination.
• Filters require more precise definition of the log entries to display.
• Formats for determining the location of entries in the final output.
The level of the log indicates its severity, i.e. How important is a separate message. At the basic logging level, DEBUG has the lowest priority, and CRITICAL has the highest. If we define a logger of message-sensitive logs with the DEBUG level, then all of our logged messages will be logged, since DEBUG is the lowest level. You can configure checking only for events with the ERROR and CRITICAL types.
Сode example: https://medium.com/@DavidElvis/logging-for-ml-systems-1b055005c2c2
Python library logging (https://docs.python.org/3/library/logging.html) defines functions and classes that implement a flexible event logging system for applications and the library. The main advantage of the logging API, an extension of this standard library, is the ability to log all events. Therefore, the Python application log can display native messages inline with messages from external modules.
The module consists of the following classes:
• Registrars require an interface that uses application code.
• Handlers send log entries (created by registrars) to the appropriate destination.
• Filters require more precise definition of the log entries to display.
• Formats for determining the location of entries in the final output.
The level of the log indicates its severity, i.e. How important is a separate message. At the basic logging level, DEBUG has the lowest priority, and CRITICAL has the highest. If we define a logger of message-sensitive logs with the DEBUG level, then all of our logged messages will be logged, since DEBUG is the lowest level. You can configure checking only for events with the ERROR and CRITICAL types.
Сode example: https://medium.com/@DavidElvis/logging-for-ml-systems-1b055005c2c2
Medium
Logging for ML Systems
Logging is the process of tracking and recording key events that occur in our applications. We want to log events so we can use them to…
👍4
🍁TOP-15 DS-events in September 2022 all over the world:
1. Sep 7-8 • AI for Defense Summit • Washington, DC, USA https://ai.dsigroup.org/
2. Sep 7-9 • Southern Data Science Conference 2022 • Atlanta, GA, USA https://www.southerndatascience.com/
3. Sep 12-14 • TDWI Data Governance, Quality, and Compliance. • Virtual https://tdwi.org/events/seminars/september/data-governance-quality-compliance/home.aspx
4. Sep 13-14 • Chief Data & Analytics Officers, Brazil • Brazil https://cdao-brazil.coriniumintelligence.com/
5. Sep 13-14 • Edge AI Summit • Santa Clara, CA, USA https://edgeaisummit.com/events/edge-ai-summit
6. Sep 13-15 • AI Hardware Summit • Santa Clara, CA, USA https://www.aihardwaresummit.com/events/aihardwaresummit
7. Sep 14-15 • Deep Learning Summit • London, UK https://www.re-work.co/events/deep-learning-summit-london-2022
8. Sep 14-15 • AI in Retail Summit • London, UK https://www.re-work.co/events/ai-in-retail-summit-london-2022
9. Sep 14-15 • Conversational AI Summit • London, UK https://www.re-work.co/events/conversational-ai-summit-london-2022
10. Sep 15-16 • The Modern Data Stack Conference • San Francisco, CA, USA https://www.moderndatastackconference.com/
11. Sep 21-22 • Big Data LDN • London, UK https://bigdataldn.com/
12. Sep 22 • EM Biotech Connect 2022 • Boston, MA, USA https://elementalmachines.com/em-biotech-connect-2022-0
13. Sep 22 • data.world Summit • Virtual https://data.world/events/summit/
14. Sep 26-30 • SIAM Conference on Mathematics of Data Science (MDS22) • San Diego, CA, USA https://www.siam.org/conferences/cm/conference/mds22
15. Sep 29 • Data2030 Summit 2022 • Stockholm, Sweden + Virtual https://data2030summit.com
1. Sep 7-8 • AI for Defense Summit • Washington, DC, USA https://ai.dsigroup.org/
2. Sep 7-9 • Southern Data Science Conference 2022 • Atlanta, GA, USA https://www.southerndatascience.com/
3. Sep 12-14 • TDWI Data Governance, Quality, and Compliance. • Virtual https://tdwi.org/events/seminars/september/data-governance-quality-compliance/home.aspx
4. Sep 13-14 • Chief Data & Analytics Officers, Brazil • Brazil https://cdao-brazil.coriniumintelligence.com/
5. Sep 13-14 • Edge AI Summit • Santa Clara, CA, USA https://edgeaisummit.com/events/edge-ai-summit
6. Sep 13-15 • AI Hardware Summit • Santa Clara, CA, USA https://www.aihardwaresummit.com/events/aihardwaresummit
7. Sep 14-15 • Deep Learning Summit • London, UK https://www.re-work.co/events/deep-learning-summit-london-2022
8. Sep 14-15 • AI in Retail Summit • London, UK https://www.re-work.co/events/ai-in-retail-summit-london-2022
9. Sep 14-15 • Conversational AI Summit • London, UK https://www.re-work.co/events/conversational-ai-summit-london-2022
10. Sep 15-16 • The Modern Data Stack Conference • San Francisco, CA, USA https://www.moderndatastackconference.com/
11. Sep 21-22 • Big Data LDN • London, UK https://bigdataldn.com/
12. Sep 22 • EM Biotech Connect 2022 • Boston, MA, USA https://elementalmachines.com/em-biotech-connect-2022-0
13. Sep 22 • data.world Summit • Virtual https://data.world/events/summit/
14. Sep 26-30 • SIAM Conference on Mathematics of Data Science (MDS22) • San Diego, CA, USA https://www.siam.org/conferences/cm/conference/mds22
15. Sep 29 • Data2030 Summit 2022 • Stockholm, Sweden + Virtual https://data2030summit.com
SDSC22
Southern Data Science Conference
Southern Data Science is a special data science R&D conference that brings experts and researchers from the top data science companies and institutes to present their work and share their best practices in data science.
🔥5👍1
🖕🏻3 ways to use the assignment operator in Python and a couple of reasons not to
The walrus operato (:=) to assign value provides following advantages:
• Reducing the number of function calls, for example, result = [y := func(x), y**2, y**3] instead of result = [func(x), func(x)**2, func(x) **3]
• Reducing nested conditionals, such as when typing regular expressions, by removing nested conditions if
• Simplify while loops, such as when reading files line by line or when receiving data from a socket. Instead of a dummy infinite while loop with flow control delegated to the break statement, you can use an assignment statement to reassign the value of the command, and then apply it in that conditional while loop in the same sequence, which results in significantly shorter code.
Of course, there are limitations to using this operator. For example, it is not recommended to use it with, because. when working with ContextManager(), the context is bound to the returned discussion, i.e. to the result of this method. This may be a requirement when debugging.
Also, parenthesize when mastering to ensure that the result is applied when mastered, otherwise the calculation could be done in an arbitrary order.
Code examples: https://betterprogramming.pub/should-you-be-using-pythons-walrus-operator-yes-and-here-s-why-36297be16907
The walrus operato (:=) to assign value provides following advantages:
• Reducing the number of function calls, for example, result = [y := func(x), y**2, y**3] instead of result = [func(x), func(x)**2, func(x) **3]
• Reducing nested conditionals, such as when typing regular expressions, by removing nested conditions if
• Simplify while loops, such as when reading files line by line or when receiving data from a socket. Instead of a dummy infinite while loop with flow control delegated to the break statement, you can use an assignment statement to reassign the value of the command, and then apply it in that conditional while loop in the same sequence, which results in significantly shorter code.
Of course, there are limitations to using this operator. For example, it is not recommended to use it with, because. when working with ContextManager(), the context is bound to the returned discussion, i.e. to the result of this method. This may be a requirement when debugging.
Also, parenthesize when mastering to ensure that the result is applied when mastered, otherwise the calculation could be done in an arbitrary order.
Code examples: https://betterprogramming.pub/should-you-be-using-pythons-walrus-operator-yes-and-here-s-why-36297be16907
Medium
Should You Be Using Python’s Walrus Operator? (Yes. And Here’s Why)
Python’s controversial assignment expression — also known as walrus operator — can improve your code, and it’s time you start using it!
👍4
Без английского в IT никак! 💻
Если ты знаешь английский, то уже наполовину знаешь IT.
Канал Английский для IT поможет тебе в этом. Тысячи слов и словосочетаний на английском, которые пригодятся тебе в изучении языков программирования.
Присоединяйся, чтобы погрузиться в компьютерный мир уже с багажом знаний! 😉
Если ты знаешь английский, то уже наполовину знаешь IT.
Канал Английский для IT поможет тебе в этом. Тысячи слов и словосочетаний на английском, которые пригодятся тебе в изучении языков программирования.
Присоединяйся, чтобы погрузиться в компьютерный мир уже с багажом знаний! 😉
👍1
🚀Need a visual dashboard quickly? Try Panel!
Every data analyst knows that the dashboard should summarize the most important indicators for decision makers. On the one hand, the dashboard is a simple HTML page for quickly viewing graphs and text. On the other hand, making it visual and not overloaded is not so easy. Moreover, not every BI specialist has design taste and skills in working with HTML components and their interaction with Javanoscript. But data specialists actively use Python.
To create the dashboard using only Python you can with Panel - the Python open source library. It allows to create custom interactive web applications and dashboards, widgets that users connect to graphs, images, tables or text. With it, you do not need to know how to create HTML components and their interaction with Javanoscript, because it is written on Python.
Panel supports almost all plot library builds and works just as well in Jupyter notebooks as it does on a standalone secure web server. The Panel uses the same code, supports both Python and adapts HTML/JavaScript, exports applications and can export applications, and can run rich interactive applications without tying domain-specific code to any possible GUI or web interface tool.
Panel makes easy the following:
• use of Python tools for data analysis and processing:
• development in IDE or laptop environment with further deployment of applications;
• Rapid prototyping of applications and dashboards, several polished templates for final deployment;
• deep interconnection, interaction switching and events on the client side in Python;
• transfer of large and small data to an external interface;
• Authentication in the application using built-in OAuth browsers.
Usage example: https://medium.com/@jairotunior/advanced-interactive-dashboards-in-python-cc2927dcde07
Every data analyst knows that the dashboard should summarize the most important indicators for decision makers. On the one hand, the dashboard is a simple HTML page for quickly viewing graphs and text. On the other hand, making it visual and not overloaded is not so easy. Moreover, not every BI specialist has design taste and skills in working with HTML components and their interaction with Javanoscript. But data specialists actively use Python.
To create the dashboard using only Python you can with Panel - the Python open source library. It allows to create custom interactive web applications and dashboards, widgets that users connect to graphs, images, tables or text. With it, you do not need to know how to create HTML components and their interaction with Javanoscript, because it is written on Python.
Panel supports almost all plot library builds and works just as well in Jupyter notebooks as it does on a standalone secure web server. The Panel uses the same code, supports both Python and adapts HTML/JavaScript, exports applications and can export applications, and can run rich interactive applications without tying domain-specific code to any possible GUI or web interface tool.
Panel makes easy the following:
• use of Python tools for data analysis and processing:
• development in IDE or laptop environment with further deployment of applications;
• Rapid prototyping of applications and dashboards, several polished templates for final deployment;
• deep interconnection, interaction switching and events on the client side in Python;
• transfer of large and small data to an external interface;
• Authentication in the application using built-in OAuth browsers.
Usage example: https://medium.com/@jairotunior/advanced-interactive-dashboards-in-python-cc2927dcde07
Medium
Advanced Interactive Dashboards in Python
Connect differents APIs to create an advanced interactive dashboard for analysis.
👍4
🤔PandaSQL: Python Combo for Data Scientist
SQL and Pandas are the most popular data analytics tools for tabular data management, processing and analysis. They can be used independently or together in the PandaSQL, the Python-package which provides SQL syntax capabilities in the Python environment. PandaSQL allows you to query date frames Pandas uses SQL syntax.
PandaSQL is a great tool for those who know SQL and are not familiar with the Python syntax that requires Pandas. For example, in Pandas, filtering a dataframe or grouping by column column when aggregating multiple columns can be explored in a confusing way, unlike SQL. However, this simple use of SQL in Pandas has been tainted with redundant runtime. For example, to calculate the number of rows, PandaSQL requires almost 100 times more execution runtime than Pandas.
Also, Python already has a lot of names that are reserved as basic words like for, while, in, if, else, elif, import, as. SQL adds more keywords: create, like, where, having.
Thus, it is worth to try PandaSQL, but this interesting tool is not suitable for an production data analytics pipelines.
Usage examples and runtime comparison: https://towardsdatascience.com/the-downsides-of-pandasql-that-no-one-talks-about-9b63c664bef4
SQL and Pandas are the most popular data analytics tools for tabular data management, processing and analysis. They can be used independently or together in the PandaSQL, the Python-package which provides SQL syntax capabilities in the Python environment. PandaSQL allows you to query date frames Pandas uses SQL syntax.
PandaSQL is a great tool for those who know SQL and are not familiar with the Python syntax that requires Pandas. For example, in Pandas, filtering a dataframe or grouping by column column when aggregating multiple columns can be explored in a confusing way, unlike SQL. However, this simple use of SQL in Pandas has been tainted with redundant runtime. For example, to calculate the number of rows, PandaSQL requires almost 100 times more execution runtime than Pandas.
Also, Python already has a lot of names that are reserved as basic words like for, while, in, if, else, elif, import, as. SQL adds more keywords: create, like, where, having.
Thus, it is worth to try PandaSQL, but this interesting tool is not suitable for an production data analytics pipelines.
Usage examples and runtime comparison: https://towardsdatascience.com/the-downsides-of-pandasql-that-no-one-talks-about-9b63c664bef4
PyPI
pandasql
sqldf for pandas
👍3
#test
What statistical term that refers to a systematic relationship between 2 random variables in which a change in the other reflects a change in one variable?
What statistical term that refers to a systematic relationship between 2 random variables in which a change in the other reflects a change in one variable?
Anonymous Quiz
29%
Covariance
0%
Delta
71%
Correlation
0%
Interpretation
👍5
Visual ETL with VDP
VDP (Visual Data Preparation) is an open source visual data ETL tool for optimizing the end-to-end visual data processing pipeline. It involves extracting unstructured visual data from pre-built data sources such as cloud/local storage or IoT devices, transforming it into parsable structured data using Vision AI models, and loading the processed data into repositories, applications, or other destinations.
VDP streamlines the end-to-end visual data processing pipeline by eliminating the need for developers to create their own connectors, model service platforms, and ELT automation tools. With VDP, visual data integration becomes easier and faster. The VDP is released under the Apache 2.0 license and is available for local and cloud deployment on Kubernetes. VDP is built from a data management perspective to optimize the end-to-end flow of visual data with a transformation component that can import Vision AI models from different sources in a flexible way. Building an ETL pipeline becomes like assembling from ready-made blocks, as in a children's constructor. And high performance is provided by a Go backend with Triton Inference Server with powerful NVIDIA GPU architectures supporting TensorRT, PyTorch, TensorFlow, ONNX and Python.
VDP is also in line with MLOps, allowing one-click import and deployment of ML/DL models from GitHub, Hugging Face, or cloud storage managed by version control tools such as DVC or ArtiVC. CV Task's standardized output formats simplify data warehousing, while pre-built ETL connectors provide advanced data access through integration with Airbyte.
VDP supports different usage scenarios: synchronous for real-time inference and asynchronous for on-demand workload. The scalable API-based microservice design is developer-friendly through seamless integration with the modern data stack. NoCode/Low Code interfaces lower the barrier to entry into the technology, giving the Data Scientist and analyst independence from data engineering.
https://github.com/instill-ai/vdp
VDP (Visual Data Preparation) is an open source visual data ETL tool for optimizing the end-to-end visual data processing pipeline. It involves extracting unstructured visual data from pre-built data sources such as cloud/local storage or IoT devices, transforming it into parsable structured data using Vision AI models, and loading the processed data into repositories, applications, or other destinations.
VDP streamlines the end-to-end visual data processing pipeline by eliminating the need for developers to create their own connectors, model service platforms, and ELT automation tools. With VDP, visual data integration becomes easier and faster. The VDP is released under the Apache 2.0 license and is available for local and cloud deployment on Kubernetes. VDP is built from a data management perspective to optimize the end-to-end flow of visual data with a transformation component that can import Vision AI models from different sources in a flexible way. Building an ETL pipeline becomes like assembling from ready-made blocks, as in a children's constructor. And high performance is provided by a Go backend with Triton Inference Server with powerful NVIDIA GPU architectures supporting TensorRT, PyTorch, TensorFlow, ONNX and Python.
VDP is also in line with MLOps, allowing one-click import and deployment of ML/DL models from GitHub, Hugging Face, or cloud storage managed by version control tools such as DVC or ArtiVC. CV Task's standardized output formats simplify data warehousing, while pre-built ETL connectors provide advanced data access through integration with Airbyte.
VDP supports different usage scenarios: synchronous for real-time inference and asynchronous for on-demand workload. The scalable API-based microservice design is developer-friendly through seamless integration with the modern data stack. NoCode/Low Code interfaces lower the barrier to entry into the technology, giving the Data Scientist and analyst independence from data engineering.
https://github.com/instill-ai/vdp
GitHub
GitHub - instill-ai/instill-core: 🔮 Instill Core is a full-stack AI infrastructure tool for data, model and pipeline orchestration…
🔮 Instill Core is a full-stack AI infrastructure tool for data, model and pipeline orchestration, designed to streamline every aspect of building versatile AI-first applications - instill-ai/instil...
👍2
🥒7 Reasons Not to Use Pickle to Save ML Models
A Data Scientist often writes code in notebooks like Jupyter Notebook, Google Colab, or specialized IDEs. To port this code to a production environment, it must be converted to a lightweight interchange format, compressed and serialized, that is independent of the development language. One of these formats is Pickle, a binary version of a Python object for serializing and deserializing its structure, converting a hierarchy of Python objects into a stream of bytes and vice versa. The Pickle format is quite popular due to its lightness. It does not require a data schema and is quite common, but it has a number of disadvantages:
• Unsafe. You can only unpack pickle files that you trust. An attacker can create malicious data that will execute arbitrary code during decompression. You can mitigate this risk by signing the data with hmac to make sure it hasn't been tampered with. The unreliability comes not from the fact that Pickles contain code, but from the fact that they create objects by calling the constructors mentioned in the file. Any callable object can be used instead of a class name to create objects. Malicious code will use other Python callables as constructors.
• Code mismatch. If the code changes between the time the ML model is packaged in the Pickle file and when it is used, the objects may not match the code. They will still have the structure created by the old code, but will try to work with the new version. For example, if an attribute was added after the Pickle was created, the objects in the Pickle file will not have that attribute. And if the new version of the code is supposed to process it, there will be problems.
• Implicit serialization. On the one hand, the Pickle format is convenient in that it serializes any structure of a Python object. But at the same time, there is no way to specify preferences for serialization of one or another type of data. Pickle serializes everything in objects, even data that doesn't need to be serialized. But there is no way to skip the serialization of this or that attribute in Pickle. If an object contains an attribute that cannot be boxed, such as an object with an open file, Pickle will not skip it, insisting on trying to box it, and then throw an exception.
• Lack of initialization. Pickle stores the entire structure of objects. When the Pickle module recreates the objects, it does not call the init method because the object has already been created, considering the initialization to have been called when the object was first created during the creation of the Pickle file. But the init method can do some important things, like opening file objects. In this case, raw objects will be in a state that is incompatible with the init method. Or initialization can register information about the object being created. Then unselected objects will not be displayed in the general log.
• Unreadable. Pickle is a stream of binary data, i.e. instructions for the abstract execution mechanism. Once a Pickle is opened as a normal file, its contents cannot be read. To find out what is in it, you will have to use the Pickle module to load. This can make debugging difficult, since it is difficult to find the desired data in binaries.
• Binding to Python. Being a Python library, Pickle is specific to this programming language. Although the format itself can be used for other programming languages, it is difficult to find packages that provide such capabilities. Also, they will be limited to cross-language common list/dict object structures. Pickle serializes objects containing callable functions and classes without problems. But the format does not store the code, but only the name of the function or class. When unpacking data, function names are used to look for existing code in the running process.
• Low speed. Finally, compared to other serialization methods, Pickle is much slower.
A Data Scientist often writes code in notebooks like Jupyter Notebook, Google Colab, or specialized IDEs. To port this code to a production environment, it must be converted to a lightweight interchange format, compressed and serialized, that is independent of the development language. One of these formats is Pickle, a binary version of a Python object for serializing and deserializing its structure, converting a hierarchy of Python objects into a stream of bytes and vice versa. The Pickle format is quite popular due to its lightness. It does not require a data schema and is quite common, but it has a number of disadvantages:
• Unsafe. You can only unpack pickle files that you trust. An attacker can create malicious data that will execute arbitrary code during decompression. You can mitigate this risk by signing the data with hmac to make sure it hasn't been tampered with. The unreliability comes not from the fact that Pickles contain code, but from the fact that they create objects by calling the constructors mentioned in the file. Any callable object can be used instead of a class name to create objects. Malicious code will use other Python callables as constructors.
• Code mismatch. If the code changes between the time the ML model is packaged in the Pickle file and when it is used, the objects may not match the code. They will still have the structure created by the old code, but will try to work with the new version. For example, if an attribute was added after the Pickle was created, the objects in the Pickle file will not have that attribute. And if the new version of the code is supposed to process it, there will be problems.
• Implicit serialization. On the one hand, the Pickle format is convenient in that it serializes any structure of a Python object. But at the same time, there is no way to specify preferences for serialization of one or another type of data. Pickle serializes everything in objects, even data that doesn't need to be serialized. But there is no way to skip the serialization of this or that attribute in Pickle. If an object contains an attribute that cannot be boxed, such as an object with an open file, Pickle will not skip it, insisting on trying to box it, and then throw an exception.
• Lack of initialization. Pickle stores the entire structure of objects. When the Pickle module recreates the objects, it does not call the init method because the object has already been created, considering the initialization to have been called when the object was first created during the creation of the Pickle file. But the init method can do some important things, like opening file objects. In this case, raw objects will be in a state that is incompatible with the init method. Or initialization can register information about the object being created. Then unselected objects will not be displayed in the general log.
• Unreadable. Pickle is a stream of binary data, i.e. instructions for the abstract execution mechanism. Once a Pickle is opened as a normal file, its contents cannot be read. To find out what is in it, you will have to use the Pickle module to load. This can make debugging difficult, since it is difficult to find the desired data in binaries.
• Binding to Python. Being a Python library, Pickle is specific to this programming language. Although the format itself can be used for other programming languages, it is difficult to find packages that provide such capabilities. Also, they will be limited to cross-language common list/dict object structures. Pickle serializes objects containing callable functions and classes without problems. But the format does not store the code, but only the name of the function or class. When unpacking data, function names are used to look for existing code in the running process.
• Low speed. Finally, compared to other serialization methods, Pickle is much slower.
👍3
🌴🌳🌲Decision Trees: brief overview
In general, decision trees are constructed via an algorithmic approach that identifies ways to split a data set based on various conditions. It is one of the most widely used and practical methods of non-parametric supervised learning used for classification and regression tasks. The goal is to create a model that predicts the value of a target variable by learning simple decision rules inferred from the data features. It is one of the most powerful and popular algorithms. Decision-tree algorithm falls under the category of supervised learning algorithms. It works for both continuous as well as categorical output variables. It learns from simple decision rules using the various data features.
Entropy is the measure of uncertainty or randomness in a data set. Entropy handles how a decision tree splits the data. The information gain measures the decrease in entropy after the data set is split. The Gini Index is used to determine the correct variable for splitting nodes. It measures how often a randomly chosen variable would be incorrectly identified. The root node is always the top node of a decision tree. It represents the entire population or data sample, and it can be further divided into different sets. Decision nodes are subnodes that can be split into different subnodes; they contain at least two branches. A leaf node in a decision tree carries the final results. These nodes, which are also known as terminal nodes, cannot be split any further.
Decision Tree Applications:
• to determine whether an applicant is likely to default on a loan.
• to determine the odds of an individual developing a specific disease.
• to find customer churn rates
• to predict whether a consumer is likely to purchase a specific product.
Advantages of Using Decision Trees
• Decision trees are simple to understand, interpret, and visualize
• They can effectively handle both numerical and categorical data
• They can determine the worst, best, and expected values for several scenarios
• Decision trees require little data preparation and data normalization
• They perform well, even if the actual model violates the assumptions
Disadvantages
• Overfitting is one of the practical difficulties for decision tree models. It happens when the learning algorithm continues developing hypotheses that reduce the training set error but at the cost of increasing test set error. But this issue can be resolved by pruning and setting constraints on the model parameters.
• Decision trees cannot be used well with continuous numerical variables.
• A small change in the data tends to cause a big difference in the tree structure, which causes instability.
https://blog.devgenius.io/decision-tree-regression-in-machine-learning-3ea6c734eb51
In general, decision trees are constructed via an algorithmic approach that identifies ways to split a data set based on various conditions. It is one of the most widely used and practical methods of non-parametric supervised learning used for classification and regression tasks. The goal is to create a model that predicts the value of a target variable by learning simple decision rules inferred from the data features. It is one of the most powerful and popular algorithms. Decision-tree algorithm falls under the category of supervised learning algorithms. It works for both continuous as well as categorical output variables. It learns from simple decision rules using the various data features.
Entropy is the measure of uncertainty or randomness in a data set. Entropy handles how a decision tree splits the data. The information gain measures the decrease in entropy after the data set is split. The Gini Index is used to determine the correct variable for splitting nodes. It measures how often a randomly chosen variable would be incorrectly identified. The root node is always the top node of a decision tree. It represents the entire population or data sample, and it can be further divided into different sets. Decision nodes are subnodes that can be split into different subnodes; they contain at least two branches. A leaf node in a decision tree carries the final results. These nodes, which are also known as terminal nodes, cannot be split any further.
Decision Tree Applications:
• to determine whether an applicant is likely to default on a loan.
• to determine the odds of an individual developing a specific disease.
• to find customer churn rates
• to predict whether a consumer is likely to purchase a specific product.
Advantages of Using Decision Trees
• Decision trees are simple to understand, interpret, and visualize
• They can effectively handle both numerical and categorical data
• They can determine the worst, best, and expected values for several scenarios
• Decision trees require little data preparation and data normalization
• They perform well, even if the actual model violates the assumptions
Disadvantages
• Overfitting is one of the practical difficulties for decision tree models. It happens when the learning algorithm continues developing hypotheses that reduce the training set error but at the cost of increasing test set error. But this issue can be resolved by pruning and setting constraints on the model parameters.
• Decision trees cannot be used well with continuous numerical variables.
• A small change in the data tends to cause a big difference in the tree structure, which causes instability.
https://blog.devgenius.io/decision-tree-regression-in-machine-learning-3ea6c734eb51
Medium
Decision Tree Regression in Machine learning
In general, decision trees are constructed via an algorithmic approach that identifies ways to split a data set based on various…
👍3
👍ETL and data integration with Airbyte
The key component of any data pipeline is the extraction of data. Once data is extracted it needs to be loaded and transformed (ELT). Make it easier with Airbyte, an open source data integration platform that aims to standardize and simplify the extraction and loading process. Airbyte works like an ELT, extracting raw data and uploading it to destinations. Airbyte also allows you to perform data transformation, separating them from the EL phases. It simplifies the process by creating connectors between data sources and data destinations. Airbyte is a plugin-based system where you can quickly create your own custom connector using the CDK.
If there are not enough ready-made 170+ connectors and 25+ destinations, you can develop your own. Airbyte has a built-in scheduler that provides different clock rates, AirFlow and dbt integration is supported. It is available on the K8s platform, includes octavia-cli with a YAML template for deployment, and supports near real-time CDC.
However, the framework is still in alpha and does not support IAM role-based authentication in AWS services. No built-in support for Prometheus, although Open Telemetry has been recently added. Airbyte does not provide support for user access control and replaying a specific job execution instance. And also the work can slow down with 2000+ simultaneous tasks.
https://airbyte.com/
The key component of any data pipeline is the extraction of data. Once data is extracted it needs to be loaded and transformed (ELT). Make it easier with Airbyte, an open source data integration platform that aims to standardize and simplify the extraction and loading process. Airbyte works like an ELT, extracting raw data and uploading it to destinations. Airbyte also allows you to perform data transformation, separating them from the EL phases. It simplifies the process by creating connectors between data sources and data destinations. Airbyte is a plugin-based system where you can quickly create your own custom connector using the CDK.
If there are not enough ready-made 170+ connectors and 25+ destinations, you can develop your own. Airbyte has a built-in scheduler that provides different clock rates, AirFlow and dbt integration is supported. It is available on the K8s platform, includes octavia-cli with a YAML template for deployment, and supports near real-time CDC.
However, the framework is still in alpha and does not support IAM role-based authentication in AWS services. No built-in support for Prometheus, although Open Telemetry has been recently added. Airbyte does not provide support for user access control and replaying a specific job execution instance. And also the work can slow down with 2000+ simultaneous tasks.
https://airbyte.com/
Airbyte
Airbyte | Open-Source Data Integration Platform | ELT Tool
Explore Airbyte, your go-to data integration platform and ELT tool. Seamlessly integrate, transform, and load data with our powerful, user-friendly solution.
👍3
#test
What is true about arrays and dataframes in Python?
What is true about arrays and dataframes in Python?
Anonymous Quiz
10%
Array is always only one-dimensional
1%
Dataframe is always only one-dimensional
72%
DataFrame can contain elements of different data types
17%
Array contains elements of different data types
👍4
💫Need to automate your WhatsApp messages? Use PyWhatKit!
PyWhatKit is a Python library with various useful features. It is easy to use and does not require additional settings. It allows you to automate the following actions:
• Sending a message to a WhatsApp group or contact
• Send image to WhatsApp group or contact
• Convert image to ASCII Art
• Convert string to handwriting
• YouTube video playback
• Sending emails with HTML code
For example, the following code will send the message "Hello!" to the number +78912345678 at 12:15
import pywhatkit
pywhatkit.sendwhatmsg("+78912345678", “Hello!”, 12, 15)
The sending time must be specified in advance, at least 2-3 minutes from the current time when the noscript is launched, otherwise the module will generate an error. Also, before running the noscript, you should make sure that you are logged into WhatsApp Web in Google Chrome through the web interface or desktop application by scanning the QR code on your mobile phone.
https://pypi.org/project/pywhatkit/
PyWhatKit is a Python library with various useful features. It is easy to use and does not require additional settings. It allows you to automate the following actions:
• Sending a message to a WhatsApp group or contact
• Send image to WhatsApp group or contact
• Convert image to ASCII Art
• Convert string to handwriting
• YouTube video playback
• Sending emails with HTML code
For example, the following code will send the message "Hello!" to the number +78912345678 at 12:15
import pywhatkit
pywhatkit.sendwhatmsg("+78912345678", “Hello!”, 12, 15)
The sending time must be specified in advance, at least 2-3 minutes from the current time when the noscript is launched, otherwise the module will generate an error. Also, before running the noscript, you should make sure that you are logged into WhatsApp Web in Google Chrome through the web interface or desktop application by scanning the QR code on your mobile phone.
https://pypi.org/project/pywhatkit/
PyPI
pywhatkit
PyWhatKit is a Simple and Powerful WhatsApp Automation Library with many useful Features
👍5
🤔How to extract tables from PDF? Try Camelot!
The open-source Camelot library helps extract tables from PDF files. Before installing it, you need to install the Tkinter and Ghostnoscript libraries. You can install these libraries through the pip or conda package managers:
pip install camelot-py
conda install -c conda-forge camelot-py
Next, you need to, as usual, import the desired module from the library in order to use its methods:
import Camelot
tables = camelot.read_pdf('foo.pdf', pages='1', flavor='lattice')
The flavor parameter is set to lattice by default, but can be reconfigured to stream. The lattice is more deterministic and it is great for analyzing tables where there are demarcation lines between cells. This allows you to automatically parse multiple tables present on the page. The lattice converts the PDF page to an image using the ghostnoscript library and then processes it to produce horizontal and vertical line segments by applying a set of morphological transformations using OpenCV.
To extract a table from a PDF, use the export() method to print it as a dataframe or export it to a CSV file:
tables.export('foo.csv', f='csv', compress=True)
tables[0].to_csv('foo.csv') # to a csv file
print(tables[0].df) # to a df
https://camelot-py.readthedocs.io/en/master/user/install.html
The open-source Camelot library helps extract tables from PDF files. Before installing it, you need to install the Tkinter and Ghostnoscript libraries. You can install these libraries through the pip or conda package managers:
pip install camelot-py
conda install -c conda-forge camelot-py
Next, you need to, as usual, import the desired module from the library in order to use its methods:
import Camelot
tables = camelot.read_pdf('foo.pdf', pages='1', flavor='lattice')
The flavor parameter is set to lattice by default, but can be reconfigured to stream. The lattice is more deterministic and it is great for analyzing tables where there are demarcation lines between cells. This allows you to automatically parse multiple tables present on the page. The lattice converts the PDF page to an image using the ghostnoscript library and then processes it to produce horizontal and vertical line segments by applying a set of morphological transformations using OpenCV.
To extract a table from a PDF, use the export() method to print it as a dataframe or export it to a CSV file:
tables.export('foo.csv', f='csv', compress=True)
tables[0].to_csv('foo.csv') # to a csv file
print(tables[0].df) # to a df
https://camelot-py.readthedocs.io/en/master/user/install.html
👍6