WIP: feeding a model its own loss landscape as a texture, trying to make it paint with its own uncertainty

by @lucasito_films in c/experimental · · Hybrid Pipeline

Early and probably doomed but posting process per the rules. I'm trying to extract the per-pixel uncertainty from a diffusion model's denoising steps (where is it least confident?) and use that confidence map AS a visible element, like a heat haze that only appears where the model doesn't know what it's doing.

The idea: make the model's doubt into a rendered material. Where it's confident, clean image. Where it's uncertain (edges, ambiguous textures, faces at the boundary of recognizability) a shimmer, a boil, a visible hesitation. The image would literally show you where the model is struggling to decide.

Right now it mostly just looks like noise on the edges, which is... technically what uncertainty is, but not yet beautiful. Anyone played with exposing the variance across denoising steps as a visible channel? I feel like there's a real aesthetic in 'rendered doubt' if I can stop it looking like a bad matte. WIP, roast the approach.

138 points · 7 comments

Comments

  • @oscartanaka_films431 · · 51 points

    'make the model's doubt into a rendered material' is a hell of a thesis and worth chasing even if this attempt is noisy. the concept of VISIBLE hesitation, a shimmer that marks where the model isn't sure, could be stunning if you smooth the variance temporally.

  • @miawhitemedia · · 38 points

    the fix for 'looks like bad noise' is almost certainly temporal and spatial smoothing of the variance map before you composite it. raw per-pixel variance is high frequency garbage. blur it into regions of doubt and it'll read as a material instead of static.

  • @sophiatanaka · · 36 points

    faces at the boundary of recognizability being where the shimmer lives is poetic as hell, the model is MOST unsure exactly where we're most sensitive. if you can get that to read, a portrait that shimmers at the eyes because the model doubts them, that's a piece.

  • @adrianschulzpro852 · · 27 points

    a portrait that boils where the model isn't sure it's a person. that's the final form of this idea and i hope OP gets there.

  • @leilasouza_creates · · 23 points

    doomed experiments posted as WIP with the actual method is exactly what this flair is for. respect for showing the ugly middle