#books #kagglebook #ctf

Читаю The Kaggle book, оказывается , Энтони Голдблюм по образованию экономист, как и я )
А Джереми Ховард перед основанием fast.ai трудился в Kaggle.

Хинтон тоже не избежал участия в соревах каггл, и даже выиграл MerckActiviy. Уже интересно!

"Professor Donoho does not refer to Kaggle specifically, but to all data science competition platforms. Quoting computational linguist Mark Liberman, he refers to data science competitions and platforms as being part of a Common Task Framework (CTF) paradigm that has been silently and steadily progressing data science in many fields during the last decades. He states that a CTF can work incredibly well at improving the solution of a problem in data science from an empirical point of view, quoting the Netflix competition and many DARPA competitions as successful examples. The CTF paradigm has contributed to reshaping the best-in-class solutions for problems in many fields.

The system works the best if the task is well defined and the data is of good quality. In the long run, the performance of solutions improves by small gains until it reaches an asymptote. The process can be sped up by allowing a certain amount of sharing among participants (as happens on Kaggle by means of discussions, and sharing Kaggle Notebooks and extra data provided by the datasets found in the Datasets section). According to the CTF paradigm, competitive pressure in a competition suffices to produce always-improving solutions. When the competitive pressure is paired with some degree of sharing among participants, the improvement happens at an even faster rate – hence why Kaggle introduced many incentives for sharing."

#cv

"Actually, if you watch carefully the data, it seems like data distributions are segregated into specific portions of space, something reminiscent ot me of the Madelon dataset created by Isabelle Guyon.

I therefore tried to stratifiy my folds based on a k-means clustering of the non-noisy data and my local cv has become more reliable (very correlated with the public leaderboard) and my models are performing much better with cv prediction."

https://www.kaggle.com/code/lucamassaron/are-you-doing-cross-validation-the-best-way

#umap #tsne #dimreducers #manifold

Понравилась интерактивная визуализация кластеров датасета одежды. Ну и мамонт, конечно.

"UMAP is an incredibly powerful tool in the data scientist's arsenal, and offers a number of advantages over t-SNE.

While both UMAP and t-SNE produce somewhat similar output, the increased speed, better preservation of global structure, and more understandable parameters make UMAP a more effective tool for visualizing high dimensional data.

Finally, it's important to remember that no dimensionality reduction technique is perfect - by necessity, we're distorting the data to fit it into lower dimensions - and UMAP is no exception.

However, by building up an intuitive understanding of how the algorithm works and understanding how to tune its parameters, we can more effectively use this powerful tool to visualize and understand large, high-dimensional datasets."

https://pair-code.github.io/understanding-umap/

Думаю, это была свёрточная нейросеть, а не "ИИ".

"В USP был проведён эксперимент, в котором ИИ самостоятельно анализировал фотографии лошадей, сделанные до и после хирургического вмешательства, а также до и после приёма обезболивающих препаратов. ИИ изучал глаза, уши и рот лошадей, определяя наличие болевого синдрома. Согласно результатам исследования, ИИ сумел выявить признаки, указывающие на боль, с точностью 88 %, что подтверждает эффективность такого подхода и открывает перспективы для дальнейших исследований."

https://3dnews.ru/1118376/ii-nauchilsya-raspoznavat-emotsii-givotnih-po-virageniyu-mordi

#hustles

Digital Product мне особенно нравится, но никак не складывается пока что (

https://medium.com/the-data-entrepreneurs/data-side-hustles-you-can-start-today-844863769827

#ai #gpt #llms

Интересная идея, что с ИИ продуктивнее говорить, чем переписываться.

https://medium.com/the-efficient-entrepreneur/why-your-deepseek-prompts-are-falling-short-and-how-to-close-the-gap-5b517caad388

#hustles #ai #gpt #llms #rag

https://medium.com/towards-data-science/5-ai-projects-you-can-build-this-weekend-with-python-c57724e9c461

#optuna #optunahub #hpo #hpt #smac3

Что интересно, оптимизатор smac3, который я недавно независимо для себя открыл, добавлен в оптуну через хаб.

https://medium.com/optuna/announcing-optuna-4-2-98148689e626

#python #codestyle #codegems

Есть о чём задуматься, типа конструкций for elem in mylist or []:

https://levelup.gitconnected.com/12-production-grade-python-code-styles-ive-picked-up-from-work-ad32d8ae630d

#dash #panel #streamlit #dashboards

https://pub.towardsai.net/which-python-dashboard-is-better-dash-panel-and-streamlit-showdown-8d4f8bf744f9

Дорогие подписчики и гости канала! И американская и российская сторона форсируют события вокруг организации встречи президента США Дональда Трампа и человека назначенного президентом России и похожего на Владимира Путина. При этом становится, временами, непонятно кому эта встреча больше нужна. Трамп пытается решать все проблемы нахрапом, особо не задумываясь над всякой ерундой в виде каких-то "планов". Никакого плана Трампа по прекращению войны не существует в природе, а есть желание сблизить переговорные позиции России и Украины настолько, чтобы стороны согласились на перемирие. В сущности, группы переговорщиков на всех площадках только тем и занимаются, что "сближают позиции". Трамп уже в курсе желаний российского руководства, которые были сильно скорректированы на сегодняшний день и считает, что может во время личной встречи с "Путиным", что называется, дожать до необходимого минимума уступок. Скорость, с которой разворачиваются события действительно впечатляет. Ещё на прошлой неделе Илону Маску был согласован визит в Москву со встречей с "президентом", как уже на выходных практически договорились о встрече "Путина" и Трампа в Саудовской Аравии, а Маску предложили перенести визит на другие даты. Встреча Маска с "Путиным" была интересна политбюро, как способ повлиять на Трампа в преддверии личной встречи с президентом США, а теперь понимания нужен ли контакт с Маском нет. Политбюро ждёт этой встречи в основном с одним расчётом- договорённости на снятие с России большинства санкций уже в ближайшее время, и готовы за это на уступки. Это главное, на что настраивают "Путина".

#politics

Интересно, соответствует ли информация об участии Маска действительности. Этот канал информации уже не раз подтверждал свою осведомлённость, и по косвенным признакам действительно можно судить о политической позиции Маска, благоприятствующей нынешнему руководству России.

#books #kagglebook

Закончил чтение The Kaggle Book (English Edition). Общие замечания к книге:

Много места потеряно ради сомнительной "академической широты". Зачем было тратить десятки страниц на определение метрик? Лучше бы вместо этого рассказали про трюк с ансамблированием моделек, затюненных на разные метрики. Сами же в начале сказали, что книга подразумевает наличие определённой базы, и "основы линейной регрессии" рассказывать не будут.

А тюнеры? Зачем было приводить код использования GridSearchCV? К чему эта академическая широта картины, не лучше ли было дать совет, каким тюнером пользоваться и в чем его практические преимущества? Зачем рекламировать skopt, который на момент написания книги не имел коммитов уже 2 года (а на текущий момент 5 лет)?

Ну ладно, раз вы потратили десятки страниц на описание этих тюнеров (80% из которых в реале никто не будет использовать) и примеры кода, почему не удосужились их все запустить на каком-то датасете и сравнить хотя бы для примера?

Теперь, их объяснения какие параметры тюнить у бустингов, ну честно, это на уровне школьников, не гроссмейстеров каггл.

В то же время, некоторые главы действительно изобилуют ценным личным опытом и советами, особенно глава про ансамбли, это как раз то, чего я ждал.

Понравились главы по компьютерному зрению (CV) и обработке текстов (NLP), в первой много внимания уделено аугментации изображений, в последней приведены хорошие примеры конвейеров (pipelines).

Преимуществ в целом больше, чем недостатков, и книгу я рекомендую к прочтению для начального и среднего уровня в DS.

Далее размещу несколько постов с идеями, которые мне понравились, показались полезными или неожиданными. Иногда будут мои комментарии. Основной контент на английском.

Custom losses in boosting
Metrics, Dimensionality reduction, Pseudo-labeling
Denoising with autoencoders, Neural networks for tabular competitions
Ensembling часть 1
Ensembling часть 2
Stacking variations

Также понравилась серия постов/мини-интервью с гроссами каггл, приведу интересное:

Часть 1
Часть 2
Часть 3
Часть 4

#books #kagglebook #interviews

Paweł Jankiewicz

I tend to always build a framework for each competition that allows me to create as many experiments as possible.
You should create a framework that allows you to change the most sensitive parts of the pipeline quickly.

Я тоже пытаюсь сделать свой фреймворк, чтобы не начинать каждый раз с нуля. для области DS этим неизбежно становится automl.


What Kaggle competitions taught me is the importance of validation, data leakage prevention, etc. For example, if data leaks happen in so many competitions, when people who prepare them are the best in the field, you can ask yourself what percentage of production models have data leaks in training; personally, I think 80%+ of production models are probably not validated correctly, but don’t quote me on that.


Software engineering skills are probably underestimated a lot. Every competition and problem is slightly different and needs some framework to streamline the solution (look at https://github.com/bestfitting/ instance_level_recognition and how well their code is organized). Good code organization helps you to iterate faster and eventually try more things.


Andrew Maranhão

While libraries are great, I also suggest that at some point in your career you take the time to implement it yourself. I first heard this advice from Andrew Ng and then from many others of equal calibre. Doing this creates very in-depth knowledge that sheds new light on what your model does and how it responds to tuning, data, noise, and more.

Over the years, the things I wished I realized sooner the most were:
1. Absorbing all the knowledge at the end of a competition
2. Replication of winning solutions in finished competitions

Это сильнейшая идея. Развивая её дальше, можно сказать, что учиться надо и по закончившимся соревам, в которых ты НЕ участвовал, и даже по синтетическим, которые ты сам создал!

In the pressure of a competition drawing to a close, you can see the leaderboard shaking more than ever
before. This makes it less likely that you will take risks and take the time to see things in all their detail.
When a competition is over, you don’t have that rush and can take as long as you need; you can also
replicate the rationale of the winners who made their solutions known.

If you have the discipline, this will do wonders for your data science skills, so the bottom line is: stop when
you are done, not when the competition ends. I have also heard this advice from an Andrew Ng keynote,
where he recommended replicating papers as one of his best ways to develop yourself as an AI practitioner.


Martin Henze

In many cases, after those first few days we’re more than 80% on the way to the ultimate winner’s solution, in terms of scoring metric. Of course, the fun and the challenge of Kaggle are to find creative ways to get those last few percent of, say, accuracy. But in an industry job, your time is often more efficiently spent in tackling a new project instead.

I don’t know how often a hiring manager would actually look at those resources, but I frequently got the impression that my Grandmaster noscript might have opened more doors than my PhD did. Or maybe it was a combination of the two. In any case, I can much recommend having a portfolio of public Notebooks.

Even if you’re a die-hard Python aficionado, it pays off to have a look beyond pandas and friends every once in a while. Different tools often lead to different viewpoints and more creativity.


Andrada Olteanu

I believe the most overlooked aspect of Kaggle is the community. Kaggle has the biggest pool of people, all gathered in one convenient place, from which one could connect, interact, and learn from. The best way to leverage this is to take, for example, the first 100 people from each Kaggle section (Competitions, Datasets, Notebooks – and if you want, Discussions), and follow on Twitter/LinkedIn everybody that has this information shared on their profile. This way, you can start interacting on a regular basis with these amazing people, who are so rich in insights and knowledge.

#books #kagglebook #interviews

Yifan Xie

In terms of techniques, I have built up a solid pipeline of machine learning modules that allow me to quickly apply typical techniques and algorithms on most data problems. I would say this is a kind of competitive advantage for me: a focus on standardizing, both in terms of work routine and technical artifacts over time. This allows for quicker iteration and in turn helps improve efficiency when conducting data experiments, which is a core component of Kaggle.

I am a very active participant on Numerai. For me, based on my four reasons to do data science, it is more for profit, as they provide a payout via their cryptocurrency. It is more of a solitary effort, as there is not really an advantage to teaming; they don’t encourage or forbid it, but it is just that more human resources don’t always equate to better profit on a trading competition platform like Numerai.

Ryan Chesler

For me, error analysis is one of the most illuminating processes; understanding where the model is failing and trying to find some way to improve the model or input data representation to address the weakness.

I started from very little knowledge and tried out a Kaggle competition without much success at first. I went to a local meetup and found people to team up with and learn from. At the time, I got to work with people of a much higher skill level than me and we did really well in a competition, 3rd/4500+ teams. After this, the group stopped being as consistent and I wanted to keep the community going, so I made my own group and started organizing my own events. I’ve been doing that for almost 4 years and I get to be on the opposite side of the table teaching people and helping them get started.


Bojan Tunguz

For a while I was really into the NLP competitions, but those have always been rare on Kaggle. One constant over the years, though, has been my interest in tabular data problems. Those used to be the quintessential Kaggle competition problems but have unfortunately become extinct. I am still very interested in that area of ML and have moved into doing some basic research in this domain. Compared to the other areas of ML/DL, there has been very little progress on improving ML for tabular data, and I believe there is a lot of opportunity here.

Some of Kaggle techniques are also applicable to my day-to-day modeling, but there one important aspect is missing – and that’s the support and feedback from the community and the leaderboard. When you are working on your own or with a small team, you never know if what you are building is the best that can be done, or if a better solution is possible.

Поэтому, думаю, должен быть еще 1 этап жизненного цикла ML модели (или даже бизнес-задачи): вывод её на уровень kaggle или подобной платформы, с хорошими призами, чтобы понять границы возможного.

The single biggest impact on your model’s performance will come from very good features. Unfortunately, feature engineering is more of an art than a science and is usually very model- and dataset-dependent. Most of the more interesting feature engineering tricks and practices are rarely, if ever, taught in standard ML courses or resources. Many of them cannot be taught and are dependent on some special problem-specific insights. But the mindset of looking into feature engineering as default is something that can be cultivated. It will usually take many years of practice to get good at it.

#books #kagglebook #interviews #todo

Jean-François Puget

I like competitions with a scientific background, or a background I can relate to. I dislike anonymous data and synthetic data, unless the data is generated via a very precise physics simulation. More generally, I like Kaggle competitions on domains I don’t know much about, as this is where I will learn the most. It is not the most effective way to get ranking points, but it is the one I entertain most.

What I often do is plot samples using two features or derived features on the x and y axis, and a third feature for color coding samples. One of the three features can be the target. I use lots of visualization, as I believe that human vision is the best data analysis tool there is.

К этому совету стоит отнестись серьёзно. Я сам видел, как в одной сореве Джиба что-то странное заметил в повторении некоторых чисел в дальних столбцах огромной матрицы, что привело к выявлению утечки. Ну кто в работе вообще смотрит на таблицу данных, тем более здоровую?

Hyperparameter tuning is one of the best ways to overfit, and I fear overfitting a lot.

Kazuki Onodera

-In your experience, what do inexperienced Kagglers often overlook?
-Target analysis.

-What mistakes have you made in competitions in the past?
-Target analysis. Top teams always analyze the target better than others.

Xavier Conort

-Tell us about a particularly challenging competition you entered, and what insights you used to tackle the task.

-My favorite competition is GE Flight Quest, a competition organised by GE where competitors had to predict arrival time of domestic flights in the US.

I was very careful to exclude the name of the airport from my primary feature lists. Indeed, some airports hadn’t experienced bad weather conditions during the few months of history. So, I was very concerned that my favorite ML algorithm, GBM, would use the name of the airport as a proxy for good weather and then fail to predict well for those airports in the private leaderboard. To capture the fact that some airports are better managed than others and improve my leaderboard score slightly, I eventually did use the name of the airport, but as a residual effect only. It was a feature of my second layer of models that used as an offset the predictions of my first layer of models. This approach can be considered a two-step boosting, where you censor some information during the first step. I learnt it from actuaries applying this approach in insurance to capture geospatial residual effects.

I would advise inexperienced Kagglers not to look at the solutions posted during the competition but to try to find good solutions on their own. I am happy that competitors didn’t share code during the early days of Kaggle. It forced me to learn the hard way.

-What mistakes have you made in competitions in the past?
-One mistake is to keep on competing in competitions that are badly designed with leaks. It is just a waste of time. You don’t learn much from those competitions.

-What’s the most important thing someone should keep in mind or do when they’re entering a competition?
-Compete to learn. Compete to connect with other passionate data scientists. Don’t compete only to win.

Chris Deotte

I enjoy competitions with fascinating data and competitions that require building creative novel models.
My specialty is analyzing trained models to determine their strengths and weaknesses. Feature engineering and quick experimentation are important when optimizing tabular data models. In order to accelerate the cycle of experimentation and validation, using NVIDIA RAPIDS cuDF and cuML on GPU are essential.

Laura Fink

My favorite competitions are those that want to yield something good to humanity. I especially like all
healthcare-related challenges.

#books #kagglebook #interviews

Shotaro Ishihara

Kaggle assumes the use of advanced machine learning, but this is not the case in business. In practice, I try to find ways to avoid using machine learning. Even when I do use it, I prefer working with classical methods such as TF-IDF and linear regression rather than advanced methods such as BERT.

Gilberto Titericz

I always end a competition with at least one Gradient Boosted Tree model and one deep learning-based approach. A blend of such diverse approaches is very important to increase diversity in the predictions and boost the competition metric.

Headhunters all around the world look at Kaggle to find good matches for their positions and the knowledge and experience gained from competitions can boost any career.

The number of medals I have in Kaggle competitions is my portfolio; up to now (11/2021) it’s 58 gold and 47 silver, which summarizes well the ML experience I got from Kaggle. Taking into account that each competition runs for at least 1 month, this is more than 105 consecutive months of experience doing competitive ML.

The data scientist must take into account how the model is going to be used in the future, and make the validation as close as possible to that.

Keep in mind that what makes a competition winner is not just replicating what everyone else is doing, but thinking out of the box and coming up with novel ideas, strategies, architectures, and approaches.

Gabriel Preda

There are also potential employers that make very clear that they do not consider Kaggle relevant. I disagree with this view; personally, before interviewing candidates, I normally check their GitHub and Kaggle profiles. I find them extremely relevant.

A good Kaggle profile will demonstrate not only technical skills and experience with certain languages, tools, techniques, or problem-solving skills, but also how well someone is able to communicate through discussions and Notebooks. This is a very important quality for a data scientist.

Jeong-Yoon Lee

In 2012, I used Factorization Machine, which was introduced by Steffen Rendle at KDD Cup 2012, and improved on prediction performance by 30% over an existing SVM model in a month after I joined a new company. At a start-up I co-founded, our main pitch was the ensemble algorithm to beat the market-standard linear regression. At Uber, I introduced adversarial validation to address covariate shifts in features in the machine learning pipelines.

The more ideas you try, the better chance you have to do well in a competition. The principle applies to my day-to-day work as well.

#books #kagglebook #todo

Теперь пройдёмся по основному контенту и поглядим, чего там было интересного.

Custom losses in boosting

If you need to create a custom loss in LightGBM, XGBoost, or CatBoost, as indicated in their respective documentation, you have to code a function that takes as inputs the prediction and the ground truth, and that returns as outputs the gradient and the hessian. Подумал, что хорошо бы учесть разные лоссы при написании свего тюнера гиперпараметров.

From a code implementation perspective, all you have to do is to create a function, using closures
if you need to pass more parameters beyond just the vector of predicted labels and true labels.

Here is a simple example of a focal loss (a loss that aims to heavily weight the minority class in
the loss computations as described in Lin, T-Y. et al. Focal loss for dense object detection: https://
arxiv.org/abs/1708.02002) function that you can use as a model for your own custom functions:

from scipy.misc import derivative
import xgboost as xgb

def focal_loss(alpha, gamma):
    def loss_func(y_pred, y_true):
        a, g = alpha, gamma

        def get_loss(y_pred, y_true):
            p = 1 / (1 + np.exp(-y_pred))
            loss = (
                -(a * y_true + (1 - a) * (1 - y_true))
                * ((1 - (y_true * p + (1 - y_true) * (1 - p))) ** g)
                * (y_true * np.log(p) + (1 - y_true) * np.log(1 - p))
            )
            return loss

        partial_focal = lambda y_pred: get_loss(y_pred, y_true)
        grad = derivative(partial_focal, y_pred, n=1, dx=1e-6)
        hess = derivative(partial_focal, y_pred, n=2, dx=1e-6)
        return grad, hess

    return loss_func


xgb = xgb.XGBClassifier(objective=focal_loss(alpha=0.25, gamma=1))

In the above code snippet, we have defined a new cost function, focal_loss, which is then fed into an XGBoost instance’s object parameters. The example is worth showing because the focal loss requires the specification of some parameters in order to work properly on your problem (alpha and gamma). The more simplistic solution of having their values directly coded into the function is not ideal, since you may have to change them systematically as you are tuning your model. Instead, in the proposed function, when you input the parameters into the focal_loss function, they reside in memory and they are referenced by the loss_func function that is returned to XGBoost. The returned cost function, therefore, will work, referring to the alpha and gamma values that you have initially instantiated.

Another interesting aspect of the example is that it really makes it easy to compute the gradient and the hessian of the cost function by means of the derivative function from SciPy. If your cost function is differentiable, you don’t have to worry about doing any calculations by hand. However, creating a custom objective function requires some mathematical knowledge and quite a lot of effort to make sure it works properly for your purposes. You can read about the difficulties that Max Halford experienced while implementing a focal loss for the LightGBM algorithm, and how he overcame them, here: https://maxhalford.github.io/blog/lightgbm-focal-loss/. Despite the difficulty, being able to conjure up a custom loss can really determine your success in a Kaggle competition where you have to extract the maximum possible result from your model.