princeton-vl

Procedural Adversarial Learning For many computer vision tasks, neural networks can be trained to produce a low average error, yet they still fail on seemingly simple scenes. Reducing these failures requires first finding them, but they may not be in a fixed dataset. We study procedural adversarial learning (PAL), a framework of minimax training and adversarial evaluation which searches the input space of a procedural generator for difficult instances. The procedural generator is an interpretable program mapping a low-dimensional vector of control parameters (e.g. layout, lighting, pose) to a rendered image and its label. We introduce the Simple-PAT algorithm, which uses zeroth-order optimization in the procedural generator's parameter space to propose adversarial training samples that are generated on demand. We also create PAL-Dojo, a collection of efficient procedural generators for surface normal estimation, image segmentation, and human pose estimation. Against a cross-model black-box adversary with 50M aggregate queries, Simple-PAT reduces worst-case error by a geometric mean of 3.15x compared to standard training and 2.63x compared to the strongest robust training baseline. Code is available at princeton-vl/procedural-adversarial-learning.

All scenes are generated from fully procedural code, so no external assets are required. We hope you (or your agent) will give it a try at princeton-vl/infinigen! For more updates, follow @princetonvl. This performance update was led by @jiadeng and I. 2/3

You can see images of every new thing in the update here: infinigen.cs.princeton.edu Examples docs: infinigen.cs.princeton.edu Github: princeton-vl/infinigen Changelog: github.com/princeton-vl/infinigen/releases/tag/2.…