Записали, наконец, с https://news.1rj.ru/str/boris_again видео-разбор статьи Adam: A Method for Stochastic Optimization ( https://arxiv.org/abs/1412.6980 ). Получилось, конечно, не так, как задумывалось изначально, но все равно интересно.

По итогу, в первой части видео рассказывается про общие принципы того, как устроен градиентный спуск с моментом и интуицию, стоящую за методом Adam. Во второй части видео докладчик проходится по самой статье, и мы постепенно переключаемся в режим свободного обсуждения формул и теорем. Поскольку мы не являемся специалистами по теории оптимизации, правильность всего сказанного в видео не гарантируется. А если вы нашли ошибку или можете дополнить обсуждение, не забудьте рассказать об этом в комментариях!

Ссылка на демонстрацию и статью, показанную в первом видео: https://distill.pub/2017/momentum/ .

————————————————————

Ссылки на видео:

https://www.youtube.com/watch?v=vqIwkVQnq4w&ab_channel=WMax (1 часть)

https://www.youtube.com/watch?v=ZnKmWDKBlGg&ab_channel=WMax (2 часть)

————————————————————

Спасибо большое @btseytlin за рассказ, а @unfinity - за загрузку на YouTube!
#объяснения_статей

⁉️ Today I will be interviewing Boris aka @boris_again about AI.

INTERVIEW PART 1.

❓Introduce yourself.
Hello. My name is Boris. I work at Planet Farms, an agrotech company making vertical farms. I am an ML teamlead, building ML from scratch, from infra to models. Previously I was a Researcher at Toloka among other things.

I might be known for teaching ML at HSE and OTUS, a telegram channel about all things life @boris_again and consulting people on job search in ML and IT in general.

❓Tell me about your education.
I graduated from the Data Science program at Faculty of Computer Science HSE. Highly recommended.

❓What about your work experience?
It’s hard to gauge, but I guess full-time is about 8 years now.

❓Top-3 articles you would recommend reading.
A Metric Learning Reality Check – personal favorite. A great story how the whole metric learning community was not making any progress for 10 years due to bad experiments, a great showcase of how ML benchmarks and publishing culture fail and how they don’t, a solid work on finding a better solution to the metric learning problem that is also easy to read.

An Industrial-Strength Audio Search Algorithm – the paper about Shazam. Just check this quote: “This procedure reduces the search problem to a kind of “astronavigation”... ”. This is so awesome: explaining hard stuff with metaphors in a research paper. Very easy to read.

Deep learning is robust to massive label noise – just an excellent work.

❓What do you think about Kaggle competitions?
Personally, I'm not interested in participating.

Often the large ensemble of models wins and most top results use similar ensembles. It seems that whoever is lucky with the seed wins. It's very rare to find a situation where the most creative solution wins. On the other hand, competitions are increasingly taking place in narrow areas such as medical images and bioinformatics.

After all, Kaggle is now a whole industry. There are people winning gold medals for money. There are entire teams with fleets of servers full of GPUs solving competitions on a full-time basis. All in all, to me, this creates an atmosphere where the expectation of money and interest on Kaggle is less than it is at work.

❓What is the specialty of MLOps?
MLOps is backend + ML.

One day people realized that models are not useful if they are not implemented in the product. It turned out that ML models have some peculiarities in terms of implementation.

So MLOps was born as a term to describe everything related to the infrastructure around deploying ML models in production: how data goes into the model, how models are trained, how predictions are made, how it all scales under load, how to retrain and monitor models.

The industry is now moving to the point where a Data Scientist without knowledge of MLOps is becoming increasingly rare.

INTERVIEW PART 2.

❓The mathematical gap between ML and DL How effective do you think you can be in DL if you only know the basics?
To be honest, the mathematics in the DL is far from being the most complicated, especially at the level of practical applications. Take the derivative? Write the likelihood function? Understand how the chain rule works in mathematical analysis? It's not easy, but it's within everyone's reach.

But mathematics helps a lot at the level of intuition. Especially probability and information theory. I don't have a mathematical background myself and I'm always catching up. You can do without deep mathematics, but every time I dive deeper, my capabilities are greatly expanded.

Of course, a deep knowledge of mathematics is necessary if you want to do research.

❓Let's talk about your articles.
I have several articles. Two of them are my coursework and my master's thesis. They are about metric learning: search, where the input is an image, and the output is other, similar images.

There's an article from Toloka where I used our original method of learning on crowdsourced markup , where for each example you get several noisy labels from markups are given to solve a scientific competition.

There is also a special article. After the first wave of the coronavirus epidemic, I came across information that there was a "British strain" that could cause a second wave. I decided to check this information and found that it was almost certainly true. It took me three months to write the article on my own. I had no previous experience of epidemiology, so it was very difficult to research the literature on the subject. It was particularly difficult because I was doing it alone. However, I persevered and wrote an article in which I used a predictive model to show that a strain with a virality coefficient like the British coronavirus strain would cause a wave of disease. I wanted to show that this was not the time to relax and remove restrictions.

There was a funny episode: In the middle of my research, I ran into a dead end and asked friends for help. They put me in touch with specialists from the Gamaleya Institute, real epidemiologists. I showed them my research and they said: "Great, that's what we do". I was shocked that in two months, with no experience in the field, I had done something that looked like real work.

After posting the article on arxiv, I contacted journalist friends, made a presentation, and started sending it out to the mass media. In the result, it turned out that the coronavirus was no longer a hot topic.

In the end, it was only possible to publish it on mail.ru news. You can read the publication on mail.ru here and a more technical version in this habr post.

👍 Thanks.
I would like to express special thanks to Boris for his answers and especially for his time.