RepVGG: Making VGG-style ConvNets Great Again

Прикол

On ImageNet, RepVGG reaches over 80% top-1 accuracy, which is the first time for a plain model, to the best of our knowledge

https://arxiv.org/pdf/2101.03697.pdf
https://github.com/DingXiaoH/RepVGG

Meta Pseudo Labels
https://twitter.com/quocleix/status/1349443438698143744
https://arxiv.org/abs/2003.10580

Striding Toward the Minimum
(Source: The Batch)

When you’re training a deep learning model, it can take days for an optimization algorithm to minimize the loss function. A new approach could save time.
What’s new: Juntang Zhuang and colleagues at Yale, University of Illinois at Urbana-Champaign, and University of Central Florida proposed AdaBelief, a more efficient variation on the popular Adam optimizer.
Key insight: The popular optimization methods of stochastic gradient descent (SGD) and Adam sometimes take small steps, requiring more time to reach their destination, when they could take larger ones. Given a small learning rate and a point in a large, steep area of a loss function’s landscape, SGD takes small steps until the slope becomes steeper, while Adam’s steps become smaller as it progresses. In both scenarios, an ideal optimizer would predict that the slope is long and take larger steps.
How it works: AdaBelief adjusts its step size depending on the difference between the current gradient and the average of previous gradients.

Like Adam, AdaBelief moves along a function step by step and calculates an exponential moving average of the gradient, assigning exponentially smaller weights to previous gradients. Also like Adam, at each step, a steeper average gradient generally calls for a larger step size.
Unlike Adam, AdaBelief treats the weighted average as a prediction of the gradient at the next step. If the difference between the prediction and the actual gradient is small, the function’s steepness probably isn’t changing much, and AdaBelief takes a relatively larger step. Conversely, if the difference is large, the landscape is changing, and AdaBelief decreases the step size.

Results: The authors provide videos showing that, in experiments on functions with known minimums, AdaBelief was faster than both Adam and SGD with momentum (as shown above). To demonstrate their method’s accuracy, they compared AdaBelief to SGD, Adam, and other adaptive optimizers on tasks including image classification, image generation, and language modeling. AdaBelief basically matched SGD’s accuracy and exceeded that of all other adaptive optimizers. For instance, on ImageNet, AdaBelief increased a ResNet18’s highest top-1 accuracy, or accuracy of its best prediction, to 70.08 percent, on par with SGD’s 70.23 percent and 2 percent better than the best adaptive optimizers.
Why it matters: Faster optimization means faster training, and that means more time to experiment with different models.
We’re thinking: The authors’ video demonstrations suggest that AdaBelief could be a valuable alternative to Adam. However, they don’t supply any numbers that would make for a precise speed comparison. We look forward to the authors of the Deep Learning Optimizer Benchmark Suite, who have evaluated over a dozen optimizers in various tasks, running AdaBelief through its paces.

- GAN Inversion: A Survey. (arXiv:2101.05278v1 [cs.CV])
http://arxiv.org/abs/2101.05278

AI врывается там, где его меньше всего ждали. Теперь нейросети могут видеть стереограммы. А ты нет.

Я даже учил когда-то, что стереограммы работают через восприятие глубины и смещением картинки между двумя глазами, но, давайте согласимся, вероятность того, что это чистая магия — не нулевая.

Статья: https://arxiv.org/pdf/2012.15692.pdf

Визуализация алгоритма конвертации SDF в меш https://youtu.be/B_xk71YopsA

Всем привет! Я предпринял попытку законспектировать всю основную теорию по RL. Попробовал скомпилировать в единое повествование материалы из нескольких основных курсов, чтобы детально объяснить, как устроены алгоритмы RL и почему они выглядят именно так, а не иначе. Надеюсь, кому-нибудь будет полезно)) Может быть, текст пригодится кому и для погружения в RL с нуля (от читателя предполагается только знание базового ML / DL).

Вероятно, будут фиксы / обновления; актуальная версия будет здесь:
https://github.com/FortsAndMills/RL-Theory-book

Любая пепер вообще:))

Это очень смешно))

«Rendering research papers in essence»

🌳 Neural-Backed Decision Trees

Demo: https://research.alvinwan.com/neural-backed-decision-trees/

Github: https://github.com/alvinwan/neural-backed-decision-trees

Paper: https://arxiv.org/abs/2004.00221

Code: https://colab.research.google.com/github/alvinwan/neural-backed-decision-trees/blob/master/examples/load_pretrained_nbdts.ipynb

Dataset: https://pytorch.org/docs/stable/torchvision/datasets.html

@ai_machinelearning_big_data

Github Patreon — теперь со специальным обучающим контентом для патронов.

https://onlysponsors.dev/

Еще не пробовала применять эту работу, но выглядит очень интересно. Фокус на интерпретируемости результатов классификации с построением иерархии чисто по весам уже обученной любой нейронки-классификатора.

In this work, we propose Neural-Backed Decision Trees (NBDTs) to make state-of-the-art image classification models interpretable. An NBDT is a hierarchical classifier that, unlike its predecessors,(1) uses a hierarchy derived from model parameters, to avoid overfitting, (2) can be created from any existing classification neural network without architectural modifications, and (3) retains interpretability by using a single model, sequential discrete decisions, and pure leaves. NBDTs are built in 2 steps: construct a tree structure using the weights of a trained network, dubbed an induced hierarchy.
Then, re-train or fine-tune that classification network with an extra hierarchy-based loss term, called a tree supervision loss.
For the forward pass, we run the fully-connected layer as embedded decision rules – variants of oblique decision rules for arbitrary branching factors.

Pinscreen - компания, занимающаяся созданием виртуальных аватаров, показала обновление своей платформы на базе UE4 и назвала ее PaGAN (Photoreal GAN).

По сути, у нас есть исходник с трекингом в движке, а поверх накладывается дипфейк.

И ещё, Pinscreen теперь на мегагранте, так что ждать внедрения осталось недолго.