Optimization-based painting software

Kragen Javier Sitaker, 2018-04-27 (1 minute)

Optimization-based painting software are going to be a big deal.

What I mean by that is software that generates an image according to some kind of model of the image-forming process, controlled by a sketch made by the user. Perhaps at a reduced pixel count, or reduced coloring, or with added noise due to mouse, or whatever.

Perhaps the model involves lines, or wavelets, or gradients, or boundaries between areas, or three-dimensional objects, or people, or animals, or convolutional neural networks, or whatever. It’s relatively straightforward to generate an image from such a model, and then you can compare the image to the image the user has drawn, using a metric that takes into account the kinds of errors people don’t intend, in order to infer a highly probable underlying model — if not the most probable one given the whole set of possible models, which is probably infeasible to compute, at least a reasonably probable one. Then, given this underlying model, you can generate a high-fidelity image of what the user intended.

But that’s just the beginning, because then you can modify the image in order to correct the model, you can select among a variety of images representing different models, and you can select among many images that potentially represent the same model.

Topics