Сбор данных для машинного обучения: этапы, методики и рекомендации

Все успешные компании постоянно собирают данные. Они отслеживают поведение людей в Интернете, организуют опросы, мониторят отзывы, слушают сигналы от умных устройств, извлекают значимые слова из электронных писем и предпринимают другие шаги для накопления фактов и чисел, помогающих им принимать бизнес-решения. Хотя современный мир изобилует данными, сбор ценной информации включает в себя множество организационных и технических трудностей, которые мы и рассмотрим в этой статье. Особое внимание мы уделим методикам сбора данных и инструментам для аналитики и проектов машинного обучения.


Читать: https://habr.com/ru/articles/753212/

Дайджест статей про YDB

Статьи про YDB публиковались на Хабре ещё до выхода в open source, а отдельным блогом мы обзавелись всего несколько недель назад. В связи с этим проведём небольшую ретроспективу — что пишут про YDB в других хабах.


Читать: https://habr.com/ru/companies/ydb/articles/753686/

Best of Breed: commercetools on Building Composable Commerce on MongoDB

The text discusses the partnership between commercetools and MongoDB to build a modern data-centric composable commerce platform. Commercetools uses MongoDB as their trusted database advisor to manage their data and focus on commerce. MongoDB Atlas, a fully managed database service, allows commercetools to scale their operation based on demand. Commercetools is considered a thought leader in the software industry and is part of the MACH Alliance, which promotes microservice, API-first, cloud-native SaaS, and headless architecture practices. The partnership with MongoDB enables commercetools to deliver innovation at speed and integrate AI into their retail applications. The MongoDB for VS Code Extension is now generally available, allowing developers to build applications and work with MongoDB data directly from the VS Code IDE. The extension provides features such as data exploration, JavaScript playgrounds, MongoDB Shell access, and integration with Terraform. The extension has been well-received by developers and continues to improve with new features. The text also mentions upcoming changes to the findOneAnd* API in the Node.js driver, where the modified document will be returned by default. The text discusses changes in the MongoDB Node.js Driver related to the behavior of the findOneAnd* family of APIs. It introduces a new option called "includeResultMetadata" that, when set to false, allows the APIs to return the requested document. Prior to this change, the document could only be accessed through a separate property. The text also mentions that this new behavior will become the default in the upcoming 6.0.0 release. The text provides examples of code and mentions that if TypeScript is used, type hinting will indicate any issues with the code related to this change. The text concludes by suggesting upgrading the MongoDB Node.js Driver to adapt to the new behavior before the 6.0.0 release.

Read: https://www.mongodb.com/blog/post/best-breed-commercetools-building-composable-commerce

Записки оптимизатора 1С (часть 2). Полнотекстовый индекс или как быстро искать по подстроке

Сегодня речь пойдет про ускорение поиска по подстроке в высоконагруженных базах данных 1С. А точнее об альтернативе, которую можно предложить взамен полнотекстового поиска от 1С или MS SQL.

Речь пойдет о поисковых запросах с конструкцией LIKE ‘%текст%’. Именно с двумя %%. В этом случае стандартные индексы не работают и SQL производит полное сканирование таблиц.


Читать: https://habr.com/ru/companies/softpoint/articles/753848/

Mixing SQL and NoSQL with MariaDB and MongoDB

Read: https://mariadb.com/?p=37190

Что такое обезличивание и с чем его едят?

Три года назад, весной 2020, пандемия COVID-19 только начиналась и все переходили в режим самоизоляции. Мы в один миг оказались в ситуации, когда у нас имеется огромное количество данных, доступ к которым можно получить лишь из офиса, в который, разумеется, невозможно попасть. Работать без доступа к документам нереально, нам критически необходимо продолжать размечать данные, а значит нужно разработать подход, при котором можно это делать удаленно. Именно тогда мы в очередной раз подняли тему, которая и так периодически возникала в наших обсуждениях — а так ли нам нужны настоящие данные? Ответ очевидный: нет, если мы сохраняем их вид (ФИО меняем на ФИО, адрес на адрес и т. д.). Данный процесс называется обезличиванием.


Читать: https://habr.com/ru/companies/sberbank/articles/749906/

Empowering Automation in Jira: Harnessing the Power of ScriptRunner

This text discusses the use of ScriptRunner, a plugin for Jira, to enhance automation capabilities in the project management tool. The article presents a specific case where the ScriptRunner plugin was leveraged to create a custom automation solution for matching text against a list of URLs. This solution allowed the team to prioritize work on specific documentation pages and make informed decisions based on the ranking of URLs. The article emphasizes the importance of automation for program managers, systems administrators, and engineering teams, and highlights the benefits of automation, such as enhanced productivity, improved accuracy, faster response times, scalability and consistency, and increased transparency and visibility. The automation project described in the article successfully improved the team's understanding of ticket impact and prioritization, ultimately leading to more efficient workflows and better content improvement. The text discusses the upcoming changes to the findOneAnd* APIs in the Node.js driver for MongoDB. Currently, these APIs return a ModifyResult object instead of the requested document. However, starting with version 6.0.0 of the driver, the modified or original document will be returned by default. The text also mentions that a new option called includeResultMetadata has been introduced in version 5.7.0 of the driver, which allows users to specify whether they want the requested document to be returned. Additionally, the text mentions the use of TypeScript and how it affects the handling of the returned document. The text then transitions to discussing the collaboration between MongoDB and commercetools, a composable commerce platform. It explains how commercetools uses MongoDB as their trusted advisor and how MongoDB Atlas, the fully managed database service, allows commercetools to focus on commerce rather than data management. It also highlights commercetools' leadership in the software industry and their use of AI and machine learning in conjunction with MongoDB. The text concludes by emphasizing the synergy between MongoDB and commercetools and the success they have achieved together.

Read: https://www.mongodb.com/blog/post/empowering-automation-jira-harnessing-power-noscript-runner

CAP-теорема: принципы согласованности, доступности и устойчивости

CAP-теорема, сформулированная Эриком Брюэром в 2000 году, сразу же приковала внимание специалистов в области распределенных систем и стала неотъемлемой частью арсенала знаний для разработчиков, стремящихся к созданию эффективных и устойчивых систем.

Теорема Брюэра гласит, что в распределенной системе невозможно одновременно обеспечить полное выполнение всех трех принципов: согласованности, доступности и устойчивости к разделению (partition tolerance). То есть, при наличии разделения сети между узлами, система должна выбирать между согласованностью и доступностью.


Читать: https://habr.com/ru/companies/otus/articles/754514/

MariaDB Community Server Q3 2023 maintenance releases

Read: https://mariadb.com/?p=37315

MongoDB Completes Infosec Registered Assessors Program (IRAP) Assessment for MongoDB Atlas



Read: https://www.mongodb.com/blog/post/mongodb-completes-infosec-registered-assessors-program-irap-assessment-atlas

4 Big Reasons to Upgrade to MongoDB 7.0

The text highlights four big reasons to upgrade to MongoDB 7.0. The first reason is enhanced performance, especially when working with time series data. The improvements in storage optimization, compression, and query performance allow for better handling of high cardinality data and improved scalability. The second reason is smoother migrations, with updates to cluster-to-cluster sync enabling more efficient data migration in various scenarios. The third reason is a streamlined developer experience, with enhancements to the aggregation pipeline that provide greater flexibility and performance in indexing and querying data. The fourth reason is stronger security, with the introduction of Queryable Encryption, which allows customers to encrypt sensitive data and run equality queries on encrypted data. The completion of the Infosec Registered Assessors Program (IRAP) assessment also highlights MongoDB's commitment to security and enables Australian government agencies to use MongoDB Atlas for their data management needs.

Read: https://www.mongodb.com/blog/post/4-big-reasons-upgrade-mongodb-7-0

MongoDB Announces Queryable Encryption with Equality Query Type Support

MongoDB has announced the general availability of Queryable Encryption, a technology that allows for end-to-end encryption of sensitive data while still being able to run equality queries on that encrypted data. This feature helps customers meet strict data privacy requirements. The technology allows developers to query encrypted sensitive data in a simple and intuitive way. The release includes the equality query type, with plans to add support for other query types in future releases. Queryable Encryption was first introduced in preview in MongoDB 6.0 in 2022 and has been fine-tuned through partnerships with customers in the healthcare and insurance industries. The encryption keys remain within the application and cannot be accessed by the database server, ensuring the security and privacy of the data. This technology can help customers meet data privacy requirements such as HIPAA, GDPR, CCPA, and PCI. MongoDB has also completed the Infosec Registered Assessors Program (IRAP) Assessment for MongoDB Atlas, ensuring the highest security standards for Australian government agencies. MongoDB Atlas offers advanced security features, data sovereignty, multi-cloud capabilities, scalability, and operational efficiency, making it a trusted data platform for government entities. MongoDB 7.0, the latest release, includes enhancements in performance, migrations, developer experience, and security. These features make it easier for organizations to boost the productivity of their development teams and build modern, distributed applications.

Read: https://www.mongodb.com/blog/post/mongodb-announces-queryable-encryption

MySQL 5.7 EOL: Move to MariaDB Server with 11 Easy Commands

Read: https://mariadb.com/?p=37307

Join Us for the Latest Technology Updates on Exadata and Database Cloud

It has been a busy year at Oracle and we are inviting you to join us at DatabaseWorld at CloudWorld to hear about the latest technology updates on Exadata and the Database Cloud in Las Vegas on September 18-23, 2023!

Read: https://blogs.oracle.com/database/post/latest-technology-updates-exadata-dbcloud-dbw2023

Building AI with MongoDB: Announcing the First Qualifiers for the Innovators Program

The text discusses the MongoDB AI Innovators Program and announces the first three startups to graduate from the program. The startups include Cloneable, which provides an application layer for bringing AI to devices at the edge of the network; ExTrac, which uses AI-powered analytics to locate, track, and forecast risks to public safety; and CetoAI, which provides predictive analytics for the maritime industry. The text also mentions the release of MongoDB 7.0, which offers enhanced performance, smoother migrations, a streamlined developer experience, and stronger security features. Additionally, the text introduces Queryable Encryption, a technology that allows for end-to-end encryption of sensitive data while still being able to run queries on that data.

Read: https://www.mongodb.com/blog/post/building-ai-mongodb-announcing-first-qualifiers-innovators-program

Beyond Borders: How MongoDB’s Talent Sourcing Team Fuels Global Hiring

The text discusses MongoDB's Talent Sourcing team and their role in fueling global hiring. The team is centralized in India but provides global support and is divided into specialized sourcing groups for different business units. They work towards generating a steady flow of candidates, reducing the time and cost of hiring. The team contributes to 25% of the overall hiring globally. Several team members share their experiences, highlighting the opportunities for growth, learning, and development that MongoDB provides. The text also mentions MongoDB's Queryable Encryption technology, which allows for end-to-end encryption of sensitive data while still allowing equality queries on that data. The benefits of this technology are outlined, including faster application development, reduced operational risk, and strong technical controls for data privacy. The text also briefly mentions MongoDB's AI Innovators Program, which provides support for startups building AI applications on top of MongoDB Atlas. The text discusses two companies, ExTrac and CetoAI, and their use of AI and data analysis in different industries.

ExTrac is focused on managing large data sets and improving performance as the business scales. They are using AI to analyze data in various formats, such as text, images, and videos, to model trends, track narratives, and predict risk for their customers. They aim to utilize the technical expertise provided by the AI Innovators Program to achieve these goals.

CetoAI, on the other hand, operates in the maritime industry and provides predictive analytics. They use high-frequency data and AI to reduce machinery breakdowns, decrease carbon emissions, and manage operational risk. The company collects real-time data from sensors installed on each vessel and uses AI models for predictive maintenance, fuel optimization, and carbon intensity forecasting. CetoAI chose MongoDB, a database system, to handle the complex data feeds generated by their sensors and to support their business growth. They are exploring additional capabilities offered by MongoDB, such as Atlas Device Sync and Atlas Vector Search, to expand their services.

Overall, both companies are leveraging AI and data analysis to improve their operations and provide better services to their customers.

Read: https://www.mongodb.com/blog/post/beyond-borders-mongodbs-talent-sourcing-team-fuels-global-hiring

Зачем инструмент dbt нужен аналитику

Представляем подробный гайд по dbt — Data Build Tool — одному из лучших фреймворков для трансформации данных.

Читать: «Зачем инструмент dbt нужен аналитику»

Cloud Strides at MongoDB.local Auckland

The MongoDB.local Auckland event highlighted the increasing need for strong cloud strategies to support data-driven businesses in New Zealand. Kiwi companies, both local and global, are avid users of cloud services and technologies like MongoDB. Recent commitments from Microsoft, Amazon, and Google to build cloud regions in New Zealand will make it easier to build and deploy cloud projects that meet data sovereignty requirements. MongoDB announced its intention to fully support each cloud provider's regions on MongoDB Atlas, bringing a local presence to the leading cloud database platform. The event also featured presentations from customers and partners showcasing their experiences with MongoDB Atlas. Additionally, the blog post discusses three startups that have qualified for MongoDB's AI Innovators Program, including Cloneable, ExTrac, and CetoAI. These startups are using MongoDB Atlas to develop and deploy AI-powered solutions in various industries. The post concludes with an overview of MongoDB's Talent Sourcing team, highlighting the team's contributions to global hiring and the company's supportive work culture. The author, Garima Fulara, is a Senior Sourcer in the Corporate & Marketing department at MongoDB. She takes pride in her role in driving the company's growth by finding the right professionals for the right positions. The company has experienced significant expansion during her time there. The team looks for individuals with strong research skills and knowledge of LinkedIn Recruiter, particularly for niche roles and geographies. The interview process involves a sourcing test and interviews with various stakeholders. The rest of the interview rounds focus on culture, relationship-building skills, communication, and articulation. The company is currently hiring for several open roles on their Talent Sourcing team.

Read: https://www.mongodb.com/blog/post/cloud-strides-mongodb-local-auckland

Tarantool 2.11 LTS: Рассказываем про новые возможности администрирования и безопасности

Привет. Меня зовут Владимир Салыкин, я директор по продукту Tarantool. Мы выпустили Tarantool 2.11 LTS — стабильный релиз с долгим циклом поддержки. Работа над ним началась в мае 2022 года, и сейчас релиз включает в себя более тысячи коммитов от 42 авторов. Мы все много работали над решением основных проблем с обслуживанием и администрированием, с которыми ранее сталкивались наши пользователи.

В этой статье мы хотим рассказать про ключевые фичи, которые были добавлены в релизе.


Читать: https://habr.com/ru/companies/vk/articles/739134/