code for this paper is available ☺️
https://github.com/utkarshojha/few-shot-gan-adaptation

dualFace: Two-Stage Drawing Guidance for Freehand Portrait Sketching (CVMJ)
[JAIST, University of Tokyo]
* youtube
* project page
* code
* abs, pdf
In this paper, we propose dualFace, a portrait drawing interface to assist users with different levels of drawing skills to complete recognizable and authentic face sketches. dualFace consists of two-stage drawing assistance to provide global and local visual guidance: global guidance,
which helps users draw contour lines of portraits (i.e., geometric structure), and local guidance, which helps users draws details of facial parts (which conform to user-drawn contour lines), inspired by traditional artist workflows in portrait drawing. In the stage of global guidance, the user draws several contour lines, and dualFace then searches several relevant images from an internal database and displays the suggested face contour lines over the background of the canvas. In the stage of local guidance, we synthesize detailed portrait images with a deep generative model from user-drawn contour lines, but use the synthesized results as detailed drawing guidance.
#sketch #retrieval #face

Sceneformer: Indoor Scene Generation with Transformers
[ Technical University of Munich]
* project page
* github
* pdf, abs
We address the task of indoor scene generation by generating a sequence of objects, along with their locations and orientations conditioned on a room layout. Large-scale indoor scene datasets allow us to extract patterns from user-designed indoor scenes, and generate new scenes based on these patterns. Existing methods rely on the 2D or 3D appearance of these scenes in addition to object positions, and make assumptions about the possible relations between objects. In contrast, we do not use any appearance information, and implicitly learn object relations using the self-attention mechanism of transformers. Our method is also flexible, as it can be conditioned not only on the room layout but also on text denoscriptions of the room, using only the cross-attention mechanism of transformers.
#indoor

Explaining in Style: Training a GAN to explain a classifier in StyleSpace
[Google research]
* project page
* pdf
Image classification models can depend on multiple different semantic attributes of the image. An explanation of the decision of the classifier needs to both discover and visualize these properties. Here we present StylEx, a method for doing this, by training a generative model to specifically explain multiple attributes that underlie classifier decisions. We apply StylEx to multiple domains, including animals, leaves, faces and retinal images. For these, we show how an image can be modified in different ways to change its classifier output. Our results show that the method finds attributes that align well with semantic ones, generate meaningful image-specific explanations, and are human-interpretable as measured in user-studies.
#gan

New book on geometric deep learning
https://geometricdeeplearning.com/
#courses

KeypointDeformer: Unsupervised 3D Keypoint Discovery for Shape Control
[University of Oxford, UC Berkeley, Stanford University, Google Research]
* project page
* demo
* pdf, abs
* code (plan to release on April 30)

We present KeypointDeformer, a novel unsupervised method for shape control through automatically discovered 3D keypoints. Our approach produces intuitive and semantically consistent control of shape deformations. Moreover, our discovered 3D keypoints are consistent across object category instances despite large shape variations. Since our method is unsupervised, it can be readily deployed to new object categories without requiring expensive annotations for 3D keypoints and deformations. Our method also works on real-world 3D scans of shoes from Google scanned objects.
#3d #unsupervised

Total Relighting:
Learning to Relight Portraits for Background Replacement
[Google Research]
* youtube
* project page
* pdf
We propose a novel system for portrait relighting and background replacement. Our technique includes alpha matting, relighting, and compositing. We demonstrate that each of these stages can be tackled in a sequential pipeline without the use of priors (e.g. background or illumination) and with no specialized acquisition techniques, using only a single RGB portrait image and a novel, target HDR lighting environment as inputs. We train our model using relit portraits of subjects captured in a light stage computational illumination system, which records multiple lighting conditions, high quality geometry, and accurate alpha mattes. To perform realistic relighting for compositing, we introduce a novel per-pixel lighting representation in a deep learning framework, which explicitly models the diffuse and the specular components of appearance.
#single_image #relighting #matting

Emerging Properties in Self-Supervised Vision Transformers (DINO)
[Facebook AI Research, Inria, Sorbonne University]
(probably everyone already noticed)
* pdf, abs
* github
* blog
* YC review
tldr; DINO — self-distillation with no labels.
Non-contrastive SSL which looks like BYOL with some tweaks: local & global image crops sent to the student network, while only global crops sent to the teacher (multi-crop thing); cross entropy loss between teacher & student representations. Tested with resnet and vision_transformers, achieving better results with the latter. Works extremely well as a feature extractor for knn and simple linear models. It is also shown, that ViTs extract impressive class-specific segmentation maps in this unsupervised setting, which look much better than in supervised setting.
Also, batch size matters, and multi-crop matters. For example, the performance is 72.5% after 46 hours of training without multi-crop (i.e. 2×224^2) while DINO in 2×224^2+10×96^2 crop setting reaches 74.6% in 24 hours only. Memory usage is 15,4G vs 9.3G. So, the best result of 76.1 top1 accuracy is achieved using 16 GPUs for 3 days. And it is still better computational result than in the previous unsupervised works.
It is possible to use this method with small batches without multi-crop, but this setup wasn't studied well. Results with the smaller batch sizes (bs = 128) are slightly below our default training setup of bs = 1024, and would certainly require to re-tune hyperparameters like the momentum rates for example. We have explored training a model with a batch size of 8, reaching 35.2% after 50 epochs, showing the potential for training large models that barely fit an image per GPU.
See images in comments.
#self_supervised

Vector Neurons: A General Framework for SO(3)-Equivariant Networks
[Stanford University, NVIDIA Research, Google Research, University of Toronto]
* pdf, abs
* github
* project page

Invariance and equivariance to the rotation group have been widely discussed in the 3D deep learning community for pointclouds. We introduce a general framework built on top of what we call Vector Neuron representations for creating SO(3)-equivariant neural networks for pointcloud processing. Extending neurons from 1D scalars to 3D vectors, our vector neurons enable a simple mapping of SO(3) actions to latent spaces thereby providing a framework for building equivariance in common neural operations -- including linear layers, non-linearities, pooling, and normalizations. Due to their simplicity, vector neurons are versatile and, as we demonstrate, can be incorporated into diverse network architecture backbones.
We also show for the first time a rotation equivariant reconstruction network.
#3d #point_clouds #implicit_geometry

The code is now available: https://github.com/sergeyprokudin/smplpix
And it is also possible to use SMPLpix renderer with DECA to make only face renders.
#smpl #neural_rendering

It's a bit late, but still

Project Starline: Feel like you're there, together
- blogpost
Very impressive press release from Google that pushes the boundaries of telepresence.
Imagine looking through a sort of magic window, and through that window, you see another person, life-size and in three dimensions. You can talk naturally, gesture and make eye contact.
There's not much information about the technical part yet. The project was under development for a few years and some trial deployments are planned this year.

To make this experience possible, we are applying research in computer vision, machine learning, spatial audio and real-time compression. We've also developed a breakthrough light field display system that creates a sense of volume and depth that can be experienced without the need for additional glasses or headsets.

Small release for win64
https://ephtracy.github.io/index.html?page=magicacsg

review: https://youtu.be/rgwNsNCpbhg?t=208
#sdf #tools