Decomposing 3D Scenes into Objects via Unsupervised Volume Segmentation
* project page
* abs
* pdf
We present ObSuRF, a method which turns a single image of a scene into a 3D model represented as a set of NeRFs, with each NeRF corresponding to a different object. A single forward pass of an encoder network outputs a set of latent vectors describing the objects in the scene. These vectors are used independently to condition a NeRF decoder, defining the geometry and appearance of each object. We make learning more computationally efficient by deriving a novel loss, which allows training NeRFs on RGB-D inputs without explicit ray marching. We find that after training ObSuRF on RGB-D views of training scenes, it is capable of not only recovering the 3D geometry of a scene depicted in a single input image, but also to segment it into objects, despite receiving no supervision in that regard.
#nerf #segmentation #depth

ReStyle: A Residual-Based StyleGAN Encoder via Iterative Refinement
* pdf, abs
* project page
* github
* colab
Recognizing the limitations of current inversion approaches, in this work we present a novel inversion scheme that extends current encoder-based inversion methods by introducing an iterative refinement mechanism. Instead of directly predicting the latent code of a given real image using a single pass, the encoder is tasked with predicting a residual with respect to the current estimate of the inverted latent code in a self-correcting manner. Our residual-based encoder, named ReStyle, attains improved accuracy compared to current state-of-the-art encoder-based methods with a negligible increase in inference time. We analyze the behavior of ReStyle to gain valuable insights into its iterative nature. We then evaluate the performance of our residual encoder and analyze its robustness compared to optimization-based inversion and state-of-the-art encoders.
#gan #inversion

We also introduce a new technique for solving the image toonification task using the iterative nature of our encoders.
- twitter thread
#gan #inversion

gradSim: Differentiable simulation for system identification and visuomotor control

* youtube
* project page
* paper under review

Our main contributions are:
* gradSim, a differentiable simulator that demonstrates the ability to backprop from video pixels to the underlying physical attributes.
* We demonstrate recovering many physical properties exclusively from video observations, including friction, elasticity, deformable material parameters, and visuomotor controls (sans 3D supervision)
* A PyTorch framework facilitating interoperability with existing machine learning modules.
#differentiable_rendering #physics #simulation

NPMs: Neural Parametric Models
for 3D Deformable Shapes

* project page
* abs
* pdf
Parametric 3D models have enabled a wide variety of tasks in computer graphics and vision, such as modeling human bodies, faces, and hands. However, the construction of these parametric models is often tedious, as it requires heavy manual tweaking, and they struggle to represent additional complexity and details such as wrinkles or clothing.

To this end, we propose Neural Parametric Models (NPMs), a novel, learned alternative to traditional, parametric 3D models, which does not require hand-crafted, object-specific constraints. In particular, we learn to disentangle 4D dynamics into latent-space representations of shape and pose, leveraging the flexibility of recent developments in learned implicit functions. Crucially, once learned, our neural parametric models of shape and pose enable optimization over the learned spaces to fit new observations, similar to the fitting of a traditional parametric model, e.g., SMPL. This enables NPMs to achieve a significantly more accurate and detailed representation of observed deformable sequences.

We show that NPMs improve notably over both parametric and non-parametric state of the art in reconstruction and tracking of monocular depth sequences of clothed humans and hands. Latent-space interpolation as well as shape / pose transfer experiments further demonstrate the usefulness of NPMs.
#implicit_geometry #depth #non_rigid_reconstruction #avatars

Putting NeRF on a Diet: Semantically Consistent Few-Shot View Synthesis
* project page
* abs
* pdf

* Task: render a scene from novel poses given just a few photos.
* Neural Radiance Fields (NeRF) generate crisp renderings with 20-100 photos, but overfit with only a few.
* Problem: NeRF is only trained to render observed poses, leading to artifacts when few are available.
* Key insight: Scenes share high-level semantic properties across viewpoints, and pre-trained 2D visual encoders can extract these semantics. "An X is an X from any viewpoint."
* Our proposed DietNeRF supervises NeRF from arbitrary poses by ensuring renderings have consistent high-level semantics using the CLIP Vision Transformer.
* We generate plausible novel views given 1-8 views of a test scene.

#nerf #clip #one_shot_learning

Swapping Autoencoder for Deep Image Manipulation
The code for this amazing tool was released today.
* youtube (must watch)
* project page
* code (yes)

We propose the Swapping Autoencoder, a deep model designed specifically for image manipulation, rather than random sampling. The key idea is to encode an image with two independent components and enforce that any swapped combination maps to a realistic image. In particular, we encourage the components to represent structure and texture, by enforcing one component to encode co-occurrent patch statistics across different parts of an image. As our method is trained with an encoder, finding the latent codes for a new input image becomes trivial, rather than cumbersome. As a result, it can be used to manipulate real input images in various ways, including texture swapping, local and global editing, and latent code vector arithmetic.

#gan #autoencoder #image_editing #single_image

Complex step differentiation is a technique that employs complex arithmetic to obtain the numerical value of the first derivative of a real valued analytic function of a real variable, avoiding the loss of precision inherent in traditional finite differences.
- twitter

Shedding light on fairness in AI with a new data set

Facebook released a dataset of 45,168 videos of diverse people, designed to evaluate the fairness of computer vision systems.

Dataset Characteristics
* Total number of subjects/actors: 3,011
* Total number of video recordings: 45,186
* Average per video length: ~1 Minute

Nature Of Content
Video recordings of individuals, who are asked random questions from a pre-approved list, to provide their “unnoscripted” answer

Labels
* Age (self-provided) 3,011
* Gender (self-provided) 3,011
* Skin Tone (human labelled) 3,011
* Lighting (human labelled) 45,186
#dataset

Anthropomorphic Webcam
Open-Source and Open-Hardware
* youtube
* project page
* paper 👁👁
Eyecam is a webcam shaped like a human eye. It can see, blink, look around and observe you.
The purpose of this project is to speculate on the past, present and future of technology. We are surrounded by sensing devices. From surveillance camera observing us in the street, Google or Alexa speakers listen to us or webcam in our laptop, constantly looking at us. They are becoming invisible, blending into our daily lives, up to a point where we are unaware of their presence and stop questioning how they look, sense, and act.

Хорошая иллюстрация того, что можно делать с новым full-body tracking в Lens Studio.

Слышите стук? Это новое поколение фильтров уже за дверью.

На experiments.withgoogle появился раздел с xR-экспериментами в хроме. Работает на устройствах андроид, в последней версии браузера. Пока что можно посмотреть 3 проекта от креативной студии гугла:

🌏 Floom — тоннель на другую сторону Земли
😷 Sodar — радар социальной дистанции
📏 Measure Up — AR-рулетка

и еще один проект, который скоро появится:
📷 Picturescape — ваша галерея Google Photo в AR

Кстати, запустить свои эксперименты может любой желающий https://experiments.withgoogle.com/submit

Стартовала Nvidia GTC. Все в онлайне, все доступно.
Ну и конечно же все началось с keynote от Jensen Huang. Идет прям щас.

Самое интересное (пока), что самой первой темой стала Omniverse. Платформа вот с таким позиционированием в 2021: shared virtual worlds.
И если в прошлом году акцент был в основном на возможность работы над одной сценой из разных мест и одно демо с мраморными шариками, то в этом году риторика поменялась в сторону концепции Digital Twins - возможности создания копии нашего ветхого мира и обитания в нем. Да, создания и обитания.
В этом смысле Дженсен еще раз упомянул Snow Crash и показал пару не то, что примеров, а кейсов ипользования Omniverse, которая выходит из беты.
Там упоминалась ILM с виртуальными декорациями, Activision, которые присунули туда 100 000 своих ассетов и сделали систему работы с ними на базе Omniverse.
WPP показали ... виртуальный продакшен в Omniverse! И тут стало понятно, что амбиции Omniverse лежат далеко за пределами дизайна и коллективной работы.
Bentley показала цифровые двойники заданий и кварталов, которые они проектируют, собранные в Omniverse.
Но круче всех выступила BMW, они показали цифровые копии своих фабрик. Грубо говоря, вместо камер наблюдения и тысяч экранов с видео, там полная 3Д-копия фабрики, по которой можно перемещаться и следить за любыми деталями в реалтайме. Мало того, что там копии роботов, там еще и копии людей. Показали, как инженеры в мокап-костюмах настраивают конвеерную линию в виртуальном пространстве, оптимизируя расположение инструментов и безопасность. Общаясь при этом, как в игре. Чистый киберпанк с копией фабрик, роботов и работников!
В общем Omniverse это реальная киберпанк-амбиция.
Ну и не зря я тут задумал конференцию DIGITAL HUMANS - все эти виртуальные миры надо будет заселять нашими цифровыми двойниками. А для них надо будет сделать годные оболочки. По новым правилам, которых пока никто не знает. Мир стремительно оцифровывается и валит в параллельную вселенную...

P/S/ Ну блин, только что показали Drive Sim - цифровые двойники РЕАЛЬНЫХ автомобилей, которые катятся по виртуальным дорогам копии этого мира. В 3Д. В Омниверсе.
Ладно, я пошел, ищите меня там. Аватар не забудьте.

P/S/S/ Максин просто меня убила. 3Д-женюсь!

Само выступление Дженсена Хуанга
https://youtu.be/eAn_oiZwUXA