I made a thing to visualize how CLIP "sees". Images are split into a large number of smaller chunks, and these cuts are what CLIP scores for accuracy! #aiart #generativeart #vqgan #clip

Nov 16, 2021 · 1:56 PM UTC

There's a natural bias to the center with the way the cut regions are picked, which seems to create "central focus" to compositions.
There are also some different algorithms for picking the cuts. The technique in @jbusted1's notebook tends to favor picking between just two sizes (min and max). This seems to give a bit more even coverage.
Here's what a histogram looks like for the sizes picked by @jbusted1's technique:
The cut_pow technique (which is visualized in the first video) has a uniform distribution of sizes at cut_pow 1.0, but the distribution shifts as the cut pow changes.
One of the reasons panoramic images are harder to generate is because the edges tend to have bad coverage:
One thing I'm experimenting with is a method that will allow oversampling the edges (inspired by @moultano), and will allow cuts that encompass the entire image by allowing letterboxing. Here's an earlier version visualizing what I mean:
Originally I was picking the cut size similar to the cut_pow technique, but my most recent experiment was to use a normal distribution and clip similar to how @jbusted1's technique does it.
Here's a garden generated with this sort of technique
And a secluded swim spot
My favorite of the night was this early generation
The letterboxing approach was creating a nasty artifact on the bottom edge of the image. I am testing other padding methods rather than just filling with black, like repeating the edge pixels or reflecting the image.
What I've noticed is that by getting more distributed cut coverage I'm able to make less repetitive looking panorama's, but they also tend to have weaker compositions as they lack the CLIP attention focal point.
This makes me curious of what would happen if instead of trying to get even coverage I instead set it up to let me pick seed locations that the cuts will be centered around. An experiment for another day!
There's also several other techniques that @moultano created for his Doorways project, like using multiple prompts across the image, and weighting cuts by their size, which I think are also hugely important to making good panoramas. moultano.wordpress.com/2021/…