PyTorch3D v0.4.0 released with support for implicit functions, volume rendering and a reimplementation of NeRF.

И сразу вдогонку еще одна статья про применение дифференцируемого рендера для регистрации облаков точек с RGB-D сенсора.
* github
* project page

Aligning partial views of a scene into a single whole is essential to understanding one's environment and is a key component of numerous robotics tasks such as SLAM and SfM. Recent approaches have proposed end-to-end systems that can outperform traditional methods by leveraging pose supervision. However, with the rising prevalence of cameras with depth sensors, we can expect a new stream of raw RGB-D data without the annotations needed for supervision. We propose UnsupervisedR&R: an end-to-end unsupervised approach to learning point cloud registration from raw RGB-D video. The key idea is to leverage differentiable alignment and rendering to enforce photometric and geometric consistency between frames.
#differentiable_rendering #indoor

"Moscow never sleeps"

Advadnoun сделал colab notebook (https://colab.research.google.com/drive/1Q-TbYvASMPRMXCOQjkxxf72CXYjR_8Vp), где декодер от DALL-E используется для синтеза изображений из латентного вектора, который оптимизируется CLIP по критерию текстового промпта. Можно дать на вход текст и поэкспериментировать.

https://arxiv.org/pdf/2102.12092.pdf
DALL-E paper
* github

Memo запилил пост на тему eco-friendly cryptoArt с подробным введением в тему.
* твитор
* пост на гитхабе
#nft

IBRNet: Learning Multi-View Image-Based Rendering

* paper
* project page
* code

Unlike neural scene representation work that optimizes per-scene functions for rendering, we learn a generic view interpolation function that generalizes to novel scenes. We render images using classic volume rendering, which is fully differentiable and allows us to train using only multi-view posed images as supervision. Experiments show that our method outperforms recent novel view synthesis methods that also seek to generalize to novel scenes. Further, if fine-tuned on each scene, our method is competitive with state-of-the-art single-scene neural rendering methods.

Related works
1. https://alexyu.net/pixelnerf/
2. https://github.com/alextrevithick/GRF

#nerf

D-NeRF: Neural Radiance Fields for Dynamic Scenes

* project page
* paper
* no code yet

We propose D-NeRF, a method for synthesizing novel views, at an arbitrary point in time, of dynamic scenes with complex non-rigid geometries. We optimize an underlying deformable volumetric function from a sparse set of input monocular views without the need of ground-truth geometry nor multi-view images.
#nerf

Wow! Blender 2.9.2 has 2D interpolation
* notes
#blender

Попробовала адаптировать этот ноутбук к лицам людей.
Сетап очень простой — параметрическая голова человека без текстуры и даже глазных яблок. Параметрическую голову можно взять тут или тут.
Оптимизация была сделана с текстурой. Интересный момент, что многие детали меша на картинке появляются именно за счет текстуры с такой оптимизацией, при этом сам меш меняется не сильно. Если же оптимизировать только меш, то результаты на нем заметнее.

Кратко о Microsoft Mesh на текущий момент.
Но да, может быть ждать осталось не долго.

Тема не новая, но применение 3d photo inpainting к генеративным картинкам выглядит прикольно

Neural 3D Video Synthesis
[Facebook Reality Labs]
* Project page
* Paper
We propose a novel approach for 3D video synthesis that is able to represent multi-view video recordings of a dynamic real-world scene in a compact, yet expressive representation that enables high-quality view synthesis and motion interpolation. Our approach takes the high quality and compactness of static neural radiance fields in a new direction: to a model-free, dynamic setting. At the core of our approach is a novel time-conditioned neural radiance fields that represents scene dynamics using a set of compact latent codes.
Our learned representation is highly compact and able to represent a 10 second 30 FPS multi-view video recording by 18 cameras with a model size of just 28MB. We demonstrate that our method can render high-fidelity wide-angle novel views at over 1K resolution, even for highly complex and dynamic scenes.
#nerf #video #3d #novel_view

Neural Geometric Level of Detail:
Real-time Rendering with Implicit 3D Surfaces
[Nvidia]

Real-time rendering of high quality neural 3d SDFs with a sparse octree feature volume and a tiny MLP. 🔥

* Project page
* Paper
* twitter
#implicit_geometry #LOD #octree #sdf #real_time #3d

Multimodal Neurons in Artificial Neural Networks
https://distill.pub/2021/multimodal-neurons/

Yannic's drop: https://youtu.be/Z_kWZpgEZ7w

The latest generation of adversarial image attacks is, uh, somewhat simpler to carry out

https://openai.com/blog/multimodal-neurons/

https://twitter.com/moyix/status/1367575109305794563/photo/1

Tensorflow & PyTorch теперь поддерживают AMD
https://hub.docker.com/r/rocm/pytorch
https://hub.docker.com/r/rocm/tensorflow
#tools

Пока без комментариев.
Гляньте на ночь.
Генеративный контент в 3д.
ИИ, создающий миры под запрос.
https://youtu.be/hA0MsGWvmzs