Experimenting with CLIP as a feature extractor for NIMA (neural image assessment) type things... a simple linear regression yielded a direction in CLIP latent space that can increase the "aestheticness" of an output. Here's "a datura witch" and a couple of Starry Night variants:

Dec 18, 2021 · 10:36 PM UTC

11
26
5
239
Replying to @RiversHaveWings
May I ask which NIMA model you're using? I recently wanted to add this repo to my pipeline, but all the pretrained model links are broken... github.com/kentsyx/Neural-IM…

GitHub - yunxiaoshi/Neural-IMage-Assessment: A PyTorch Implementation of Neural IMage Assessment

A PyTorch Implementation of Neural IMage Assessment - GitHub - yunxiaoshi/Neural-IMage-Assessment: A PyTorch Implementation of Neural IMage Assessment

github.com
2
6
I trained it myself, it takes CLIP ViT-B/16 embeddings as input rather than features output by VGG.
1
5
more replies
Replying to @RiversHaveWings
wow! how did you train the linear weights, just good and bad examples? and do you have a notebook with it somewhere? :D
1
3
I computed ViT-B/16 embeddings for everything in the AVA train set (using the prefiltered CSVs from github.com/kentsyx/Neural-IM…). Then I did a linear regression vs their mean rating in scikit-learn and copied the weights and bias into a PyTorch nn.Linear layer.

GitHub - yunxiaoshi/Neural-IMage-Assessment: A PyTorch Implementation of Neural IMage Assessment

A PyTorch Implementation of Neural IMage Assessment - GitHub - yunxiaoshi/Neural-IMage-Assessment: A PyTorch Implementation of Neural IMage Assessment

github.com
2
1
20
more replies
Replying to @RiversHaveWings
You should try an SVM with RBF kernels. With about 1k labels it gets really good on CLIP latents.
1
14
Huh! I have around 220k CLIP embeddings and distributions of ratings in the ranges 1-10 for them, it looks like scikit-learn will have trouble scaling to that many samples?
1
2
more replies
Replying to @RiversHaveWings
Ahh Datura, Dangerous and profound!
1
Replying to @RiversHaveWings
@Darsh_Panchal bruh
1
Replying to @RiversHaveWings @nvnot_
This is cool ☮️
1
Replying to @RiversHaveWings
That's a fantastic idea
1
Load more