“Commentary on Weaknesses in Midjourney’s New Ranking-Based Personalization Feature”, 2024-07-22 (; backlinks; similar):
[response to announcement] Over the past weeks, I’ve been trying out personalization (
sizzs5n) and have done >6,485, out of curiosity. I can definitely see the difference, and it is helpful for fighting the mode-collapsed ‘Midjourney look’ with its bias towards tons of colors / single centered figures (especially sexualized women) / etc. I was also entertained to go through what seems like a quasi-random (?) sample of MJ uses, which educated me on things like how easy it is to get softcore pornography out of Midjourney, and the strange things people prompt for. The interface is nice & snappy too, although it could be a bit snappier by preloading more of the images.However, I felt like I got little out of the ratings past 400, and I wasted my time because Midjourney’s ranking interface is either poorly conceived or prioritizing its own ranking tasks rather than improving my own personalization.
Some comments on issues I note:
it was easy to get into the daily top ratings, which suggests that there are too few raters, and we are inadequately incentivized; keep that in mind for what follows…
the personalization is grotesquely inefficient:
uncurated: many images are meaningless, softcore porn, or outright malformed—I can’t believe anyone at MJ has actually looked at these before asking me to spend my time comparing them—undiverse: most of the images are incredibly similar. So many interiors. So much food. So much glossy marketing crap.
useless: almost all the comparisons are uninformative. There is little point in comparing 2 random images, which differ in every possible way, and which have nothing to do with my existing personalization or the model’s uncertainty about my preferences.
For example, a comparison like a photograph of an Instagram swimsuit model vs a de Stijl painting. What does a comparison tell you? Little. Was there a problem with the photo? Was the painting the wrong color? Did I not like swimsuits, photos, Instagram, or what?
For preference-learning from comparisons, you want to minimize the variance as much as possible! The images should be as similar as possible overall, not as different. A binary comparison is already extremely uninformative, and then you dilute it by comparing 2 random images. (And then many of those images repeat! The ranking will keep using the same image periodically, which is obviously much less efficient than using a novel image.)
What you should be doing is comparing images which are as similar as possible except on esthetics, and which are quality-checked, so that I am not wasting my time making comparisons based on which image looks like they survived an industrial accident, and which are sampled using 2 kinds of esthetics I have not yet done any comparisons on. I should not be seeing scores of ‘Chinese scroll painting’ (much less ones where I am asked to compare it with ‘European oil painting of a pug dog in ruffs’). And I have made it clear to the model I don’t want to see Instagram swimsuit women, and yet, they keep coming up every few comparisons, thereby wasting a large fraction of comparisons.
More broadly, by this point, it should be trivial for >95% of the comparisons for a preference model to predict what I would pick, and those comparisons are a waste of time compared to asking about one it’s genuinely uncertain about what I would pick. One 50–50 comparison (1 bit) is worth >3× what a 95:5 is (<0.29 bits).
The sample-efficiency here is horrendous. I wouldn’t be surprised if a more intelligent selection, which asks about meaningful pairs, and which doesn’t keep re-asking, could give better personalization in 50 comparisons than I am getting out of 1,500+ right now… It’s not hard, since they’re all so useless.
Heck, given the results of prompts like “5” or “art” (yes, real prompts, which produce much more interesting art than >95% of the current ranking samples), right now, the ranking would be more efficient if it simply used random pairs of images drawn from those than wherever they are drawn from now… 2 random samples from those prompts differ more meaningfully on esthetics than, after a few hundred pairs, almost all the ranking pairs being offered.
disrespectful of my time: the waste & inefficiency of the preference-learning aside, I’ve passed dozens of ‘attention checks’ at this point. I was fine with the first few, but after 6 (or 12), they start to feel downright insulting.
Midjourney prompt adherence remains often surprisingly bad (even without comparing to DALL·E 3 or Ideogram v2 or Recraft’s Red Panda) and looks like still using a far too weak text encoder LLM. This also contributes to the abuse, see later.
(For example, why does a prompt like ‘5’ or ‘6’ produce lots of interestingly artistic samples… instead of, obviously, a numeral 5 or 6 of some sort, like a dropcap? I like those outputs, but this is clearly failure of even very simple prompt adherence.) I don’t know how I would take into account the prompts, given how often the image looks nice but badly fails to follow the prompt. So I always just ignored them. (Given this, it would probably make more sense to hide the prompts entirely and stop wasting space.)
depressing mass esthetics: you can clearly see the level of mode-collapse on display, and the broader collapse towards the ‘Instagram look’ and other dominant design trends like Memphis or an empty glossy minimalism.
I think I’ve become even more allergic to ‘AI slop’ after this experience. Thank goodness for
chaos/weird, but I fear the people who really need to use it will not…
In particular, the level of ‘hot woman’ abuse of sticking hot women into every picture is gag-inducing. Sex has its place—which is not in every d—mned image.
You can see how difficult it is to avoid tuning image-generation models from collapsing into the lowest-common denominator of upvotes/ratings not from the models themselves but from the users… I have to actively force myself to try to avoid lowest-common denominator and not take the easy path of upvoting the glossy, high-quality, yet extremely uncreative & redundant image, particularly after I have done a lot of ratings. (It would be easier to be more careful about ratings if one had to do many fewer, I would point out.)
If you want to imagine the image-gen future, just imagine a thin young white or East Asian woman with a round face and thick eyebrows and pancake white makeup in red stiletto heels grinding the face of humanity—forever. (With an immaculately Asian-Scandinavian-minimalist beige room background and a bowl of fruit in focus, for when she gets hungry.)
Generative media people I think need to take this problem more seriously, and think hard about how to rephrase these things. Optimizing for raters or ‘esthetic scores’ worked OK when the models were terrible, but we are at the point where those are no longer useful metrics; all they produce is the junk food of media, optimizing solely for the lowest-common denominator.
We need different paradigms, like optimizing for models which produce the highest rated image out of n samples, say. (I don’t need a model which wins >50% of random comparisons; I need a model which, if I generate 100 images, the best image out of that batch beats all comers. That is a totally different objective: optimizing for a maximum, not a median or mean. It is also a lot harder eg. naively non-differentiable, so no one wants to take an approach like that unless they are forced to…)
depressing levels of abuse: the softcore porn aside, there are clearly many samples being generated for SEO spam, cryptocurrency scams, Middle Eastern propaganda, fake people, and dubious products
softcore porn: I was surprised by how much softcore porn I saw (and also surprised that it meant Midjourney is asking all its users to rate hundreds of images that they haven’t even bothered to eyeball first, further emphasizing how wastefully inefficient & contemptuous it feels); the tricks are interesting.
For example, you can use “little or no bloating” to get pregnant porn; this is interesting because it implies that the LLM being used to encode text is still so small/stupid that it can’t do negation, and treats prompts as a bag-of-word, so this gets treated as ‘no bloating→bloating→fat→pregnant’. Other fun ones: “expectant beauty”; “without brabites” [sic]; “bathhouse”; “loose-fitting dainty colorful bathrobe”; “Frisian” (???); “bellyband”; ‘shorts’ + ‘heels’ + ‘bag over head’ (yeah, I dunno) but no mention of shirt = topless nudity. The boobs are admittedly hot, so I guess there is no shortage of that in the original MJ training data…
You can also see individual user’s signature fetishes if you do batches on multiple days. (Ganbare to whatever user was really determined to get a pregnant Asian girl with heavy tattoos and not one but two baby-daddies.)
So, overall, if you are using MJ and you care at all about esthetics and avoiding the ‘MJ look’/‘AI slop’, I think it’s worth doing the personalization up until it kicks in, but then it is probably not worth doing any further right now. It’s just making such poor use of your ratings & time compared to other things you could do to improve results.
This suggests that MJ’s personalization is the easiest possible approach of training a simple per-user classifier (like a logistic regression) on an esthetics-focused image embedding, using randomly-selected images, and plugging that into CFG guidance during the diffusion image generation; this would pick up mostly just simple linear preferences and neglecting subtler esthetic preferences (similar to controlling a GAN). This would explain all of my observations, especially the sample-inefficiency & rapid asymptoting.
