“Powered by AI: Advancing Product Understanding and Building New Shopping Experiences”, 2020-05-19 (; similar):
Today we’re announcing:
We’ve built and deployed GrokNet, an universal computer vision system designed for shopping. It can identify fine-grained product attributes across billions of photos—in different categories, such as fashion, auto, and home decor.
GrokNet is powering new Marketplace features for buyers and sellers today and we’re testing automatic product tagging on Facebook Pages to help make photos shoppable.
We’re also introducing Rotating View, a state-of-the-art 3D-like photo capability that allows anyone with a camera on their phone to capture multi-dimensional panoramic views of their listings on Marketplace.
And we’ve advanced research by creating a state-of-the-art technique to predict occluded or layered objects in photos (like a shirt beneath a jacket).
These advancements are part of the foundation we’re building to develop an entirely new way to shop on our platforms—making it easier for individuals and small businesses to showcase their products to billions of people, and for buyers to find exactly what they’re looking for.
…We built, trained, and deployed a model with 83 loss functions across seven data sets to combine multiple verticals into a single embedding space. This universal model allows us to leverage many more sources of information, which increases our accuracy and outperforms our single vertical-focused models…In the GrokNet training architecture, a major challenge is managing 7 datasets and 83 loss functions, so that they all perform well simultaneously. To solve this, we adjust the batch sizes and loss weights, using more images per batch and higher loss weights for the challenging tasks. We also use weakly supervised learning to automatically generate additional training data, further improving accuracy.
…Our long-term vision is to build an all-in-one AI lifestyle assistant that can accurately search and rank billions of products, while personalizing to individual tastes. That same system would make online shopping just as social as shopping with friends in real life. Going one step further, it would advance visual search to make your real-world environment shoppable. If you see something you like (clothing, furniture, electronics, etc.), you could snap a photo of it and the system would find that exact item, as well as several similar ones to purchase right then and there…While these systems are fragmented right now, incorporating everything into one system is the ambitious challenge we’ve set out to achieve. Building these systems across all Facebook platforms would enable shoppers to connect with their friends and family to get an opinion on an automatically generated 360-degree 3D view of an item. These friends can weigh in on which sneakers they like most or which size painting looks best in the shopper’s kitchen. By combining state-of-the-art computer vision with advancements in other AI domains, such as language understanding, personalization, and social-first experiences, we’re well positioned to transform online shopping for everyone.