For those who reacted with 🦀 on a previous post. I wrote a Twitter thread on how I am building Qdrant with Rust. It is on Twitter because the development is still in progress, and I would like to tell you about some interesting details without a special blog-post.

Some topics of the thread:

- How Qdrant is useful?
- How it stores data and build indexes?
- How to keep data always available for search?
- How do I auto-generate documentation in Rust?

Your comments are welcome here and on Twitter!

​​Metric Learning Tips & Tricks

Hi everyone, I don't often have posts on this channel.
But today, I want to share some insights about what I'm working on.

Over the last year, I have been developing on a job matching system, and in the process, I have solved some interesting problems related to metric(similarity) learning.

I decided to collect all the interesting solutions into an article.

Here are some highlights:

- We have embeddings for professions and you can play with them online
- There is a way to train a model without labeled data, but it requires some tricks
- Hard Negative Mining does not work, but you can increase batch size instead
- It is possible to estimate embedding confidence
- We can micro-manage the model without re-training. Introducing the neural rules
- How do we deploy metric learning in production. Spoiler: with Qdrant

Neural Search Step-by-Step

We made a tutorial on Semantic Embeddings and Neural Search. With this guide, you will build your own semantic search service from scratch.

You won't need any complicated training of the neural network. Moreover, you can do all preparation steps in the Google Colab notebook.

Tutorial includes:
- What is the Neural Search?
- Getting embeddings from BERT Encoder
- Using vector search engine Qdrant
- Creating an API server with FastAPI.


If you want to learn how to build projects like this, the tutorial is for you.

Hello everyone,

The largest Russian-speaking data science community ODS.ai organizes the Summer School, designed after the famous Google SoC, and I participate in it as a mentor for the Metric Learning track with Qdrant.

During the track, participants are challenged to research the fine-tuning for the similarity learning, build a working prototype and contribute to Open Source.

Today at 18:00 MSK, there will be a first meetup of the track. I will be talking about:
- Which datasets are suitable for similarity matching, how can you obtain self-supervised
- Approaches to fine-tuning encoders. Selection of method depending on the amount of available annotation


I invite everyone interested to Spatial Chat ODS, at 18:00 MSK. Password: odssummerofcodeison.

Language of the event: Russian.

Materials will be available in English later on the channel.

​​ODS.ai Summer of Code results

Hi everyone, ODS SoC has officially finished in the last week, and it is time to present the results.
First of all, the winner of the Metric Learning track Tatiana Grechishcheva has published a detailed article on her work of fine-tuning and deploying metric learning models.
She fine-tuned the ViT model for matching similar clothing and put together a detailed tutorial of how you can deploy such a model to production.
An online demo is also included!

There are also some exciting results on fine-tuning transformers with different types of head layers.
In a nutshell, the result is that it is enough to have only a couple hundred examples to improve the similarity matching result without overfitting.
I will make a separate post about it and further plan on making metric learning practical.

​​Hi!
Check out the first-handed experience of building neural search solutions with Qdrant from its user in the latest Vector Podcast

Awesome Metric Learning
The Metric Learning approach to data science problems is heavily underutilized. There are a lot of academic research papers around it but much fewer practical guides and tutorials.
So we decided that we could help people adopt metric learning by collecting related materials in one place.
We are publishing a curated list of awesome practical metric learning tools, libraries, and materials - https://github.com/qdrant/awesome-metric-learning!
This collection aims to put together references to all relevant materials for building your application using Metric Learning.
If you know some exciting article, helpful tool, or a blog post that helped you apply metric learning - feel free to PR your proposal!

​​Triplet loss - Advanced Intro

Loss functions in metric learning are all chasing the same goal - to make positive pairs closer and negative further.
But the way they achieve this leads to different results and different side effects.

In today's post, we describe the differences between Triplet and Contrastive loss, why the use of Triplet loss can give an advantage, especially in the context of fine-tuning.
It also covers the approach to an efficient implementation of batch-all triplet mining.

Metric Learning for Anomaly Detection

Anomaly detection is one of those tasks to which it is challenging to apply classical ML methods directly.
The balancing of normal and abnormal examples and the internal inconsistency of anomalies make classifier training a challenging task.

And the difficulty is often related to data labeling, which in the case of anomalies may not be trivial.

The metric learning approach avoids the explicit separation into classes while combining the advantage of modeling the subject domain with the knowledge of specific anomalous examples.

In our case study, we are solving the problem of estimating the quality of coffee beans ☕️🌱 and determining the type of defects.

We trained Auto-Encoder on unlabeled samples and made fine-tuning on a small fraction of labeled ones.
This approach achieves results equivalent to conventional classification but requires orders of magnitude less labeled data.

Similarity Learning lacks a framework. So we built one.

Many general-purpose frameworks allow you to train Computer Vision or NLP tasks quickly. However, Similarity Learning has peculiarities, which usually require an additional layer of complexity on top of the usual pipelines.
So, for example, the batch size in the training of similarity models has a much greater role than in other models. Labels either do not exist or are handled in a completely different way. In many cases, the model is already pre-trained, which also adjusts the process.

Developing Similarity Learning models one after another, we began to notice patterns that helped us generalize and bring all our experience with training and fine-tuning such models into one package.
Yesterday we published Quaterion — an open-source, blazing-fast, customizable, scalable framework for training and fine-tuning similarity learning models.

One of the main features of the framework is caching. It allows you to infer large models only once and then use cached vectors during the training. It speeds up the process x100 times, simultaneously allowing you to use batch sizes that are unattainable in other ways.

(gif)

​​How many layers to fine-tune?

Model fine-tuning allows you to improve the quality of the pre-trained models with just a fraction of the resources spent on training the original model. But there is a trade-off between the number of layers you tune and the precision you get.
Using fewer layers allows for faster training with a larger batch size, while more layers increase the model's capacity.
We've done experiments so you can make more educated choices.
Highlights:

- Training only the head of a model (5% of weights) gives x2 boost on metrics, while full training gives only x3.
- Training only a head layer allows using larger models with bigger batch sizes, compensating for the precision.
- If you only have a small dataset, full model tuning will give a more negligible effect

Vector Similaruty beyond Search

Vector similarity offers a range of powerful functions that go far beyond those available in traditional full-text search engines and the conventional kNN search.

We just scratched the surface of the topic but already found a lot of new ways to interact with the data, including:

- Dissimilarity search - that can be applied to anomaly detection, mislabeling detection, and data cleaning.
- Diversity search - that can be used for giving a better overview of the data, with no query at all.
- Recommendations - where we can do beyond the single query vector and use positive and negative examples to find the most relevant items.
- Discovery or Exploration - where we can invert the logic behind triplet-loss to provide real-time improvements of the search results.


In the article, we are talking about a new toolbox for unstructured data exploration, where the search is just one of the instruments.
And maybe you will find there a tool to implement your next big idea 🙂


https://qdrant.tech/articles/vector-similarity-beyond-search/

Attention all Berliners!

You are invited to our first offline meetup!
Join us for talks on vector search, machine learning, and more.
I will also be participating, providing an overview of Qdrant's progress and future plans.

Still not convinced? We will have free pizza and beer!

The event is scheduled for December 8, 2023, at 18:00, in Berlin.
Please register https://lu.ma/vectorspace.

https://techcrunch.com/2024/01/23/qdrant-open-source-vector-database/

We are so hiring!

​​BM42: the next hybrid search baseline

You probably heard that embedding similarity struggles with exact keyword matches, especially when the keyword is a rare word, a name, or some kind of ID.

Usually, this problem is solved by combining the embedding similarity with an exact keyword search like BM25.
However, BM25 relies on pure statistics and has no idea about the meaning of the words.

This works well for large documents, but for short texts and chunks, it's not that great.
We propose a new approach for exact keyword search based on Sparse Vectors and Attention from Transformers.

We can leverage the intelligence of the transformer to score the importance of each word in a sentence,
while still being able to combine it with collection-wide statistics like IDF.

This approach works especially well with the latest release of Qdrant.
It supports hybrid search out of the box, and handles IDF computation for you.

Read more in the article.

​​MiniCOIL: Contextualized per-word embeddings

We continue our experiments with unorthodox methods to embed texts.

This time we want to solve the following problems of BM25:

- It can't differentiate homonyms: "bat" as and animal and a baseball "bat" are the same thing for BM25.
- It does loose the information during the stemming process: "information" and "informant" are the same for BM25.

How are we doing this?

Inspired by COIL paper, we generate embeddings per-word.
But instead of relying on learned relevance, which usually fails out of domain, we use learned semantic similarity + stasticical relevance.

We obtain miniCOIL vectors by reducing the dimensionality of the token-level embeddings of the Sentence Transformers.
The whole process luckily doesn't require explicit labels, and we can train it word-by-word on a single CPU.

Read more in our latest article

​​Embeddings Confidence

Let's say we want to know which embeddings are good and which are noisy.
Normally, you can't do this, because embedding model tries to represent the whole variety of objects.

Even if input data is random, it would still match with similar random data.

Using the score threshold is also not a good idea, embedding models were not trained to predict absolute scores,
only relative scores makes sense.

But what if we could make emeddings more like a traditional classifiers, but instead of predicting class,
we would predict presence of some features of the object.

Then, we could apply the very same techniques to estimate confidence of the embeddings - use SoftMax.

An attentive reader will notice that a similar approach has already been discussed on this channel.
But the last the proposed approach involved a full model training.
This time, we only train a light-weight adapter for a ready-made model and dataset.

Tried this approach on a food dataset, encoded with CLIP:

​​Relevance Feedback
Did you know that Google search results used to have 👍/👎 buttons?
It was the way to collect signals from users and improve the ranking.

Do you know why you don't see those buttons anymore?
People are either too lazy to click or they abuse the buttons to their advantage.

In 2026, however, real people don't have to rank raw search results anymore - there are all sorts of AI in the middle.
And here we are talking not only about Agents.
From a raw retrieval to a final answer, there are multiple stages of progressively smarter re-ranking models,
each of which refines the result a little bit better than the previous one.
Each of those models also costs a little more than the model in the previous stage.

And so far, the process of re-ranking was linear: for better results, you need to pre-select more candidates.
That means that you pay proportionally more for compute.

Relevance feedback changes this pattern.
Instead of linear re-ranking, you can take feedback information and reiterate the retrieval process to source better candidates.
And for the first time, we implemented a relevance feedback mechanism directly in the vector search engine.

For the same or less number of calls of the expensive model, you can get better results.
And unlike previous methods, we don't need to re-score hundreds of candidates to extract the signal for relevance feedback,
meaning that this approach is efficient enough to be used in real-time applications.