“Voice Conversion With Just Nearest Neighbors”, Matthew Baas, Benjamin van Niekerk, Herman Kamper2023-05-30 (AI music, retrieval AI)⁠:

Any-to-any voice conversion aims to transform source speech into a target voice with just a few examples of the target speaker as a reference. Recent methods produce convincing conversions, but at the cost of increased complexity—making results difficult to reproduce and build on.

Instead, we keep it simple. We propose k-nearest neighbors voice conversion (kNN-VC): a straightforward yet effective method for any-to-any conversion. First, we extract self-supervised representations of the source and reference speech [eg. dog barks]. To convert to the target speaker, we replace each frame of the source representation with its nearest neighbor in the reference. Finally, a pretrained vocoder synthesizes audio from the converted representation.

Objective and subjective evaluations show that kNN-VC improves speaker similarity with similar intelligibility scores to existing methods.

Code, samples, trained models: https://bshall.github.io/knn-vc/.