See Also

Miscellaneous

• https://csm.ai/commonsim-1-generating-3d-worlds/ (Transformer NN⁠, video generation)

• https://openai.com/blog/vpt/ (RL exploration⁠, model-based RL; backlinks)

Link Bibliography

1. https://arxiv.org/abs/2302.05442#google: “Scaling Vision Transformers to 22 Billion Parameters”⁠, Mostafa Dehghani, Josip Djolonga, Basil Mustafa, Piotr Padlewski, Jonathan Heek, Justin Gilmer, Andreas Steiner:

   link-bibliography

2. https://arxiv.org/abs/2212.05051: “VindLU: A Recipe for Effective Video-and-Language Pretraining”⁠, Feng Cheng, Xizi Wang, Jie Lei, David Crandall, Mohit Bansal, Gedas Bertasius:

   link-bibliography

3. https://arxiv.org/abs/2212.04979#google: “Video-Text Modeling With Zero-Shot Transfer from Contrastive Captioners”⁠, Shen Yan, Tao Zhu, Zirui Wang, Yuan Cao, Mi Zhang, Soham Ghosh, Yonghui Wu, Jiahui Yu:

   link-bibliography

4. https://arxiv.org/abs/2209.14156: “TVLT: Textless Vision-Language Transformer”⁠, Zineng Tang, Jaemin Cho, Yixin Nie, Mohit Bansal:

   link-bibliography

5. https://arxiv.org/abs/2208.03550: “EVL: Frozen CLIP Models Are Efficient Video Learners”⁠, Ziyi Lin, Shijie Geng, Renrui Zhang, Peng Gao, Gerard de Melo, Xiaogang Wang, Jifeng Dai, Yu Qiao, Hongsheng Li:

   link-bibliography

6. https://arxiv.org/abs/2207.07285#alibaba: “X-CLIP: End-to-End Multi-grained Contrastive Learning for Video-Text Retrieval”⁠, Yiwei Ma, Guohai Xu, Xiaoshuai Sun, Ming Yan, Ji Zhang, Rongrong Ji:

   link-bibliography

7. https://arxiv.org/abs/2206.11795#openai: “Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videos”⁠, Bowen Baker, Ilge Akkaya, Peter Zhokhov, Joost Huizinga, Jie Tang, Adrien Ecoffet, Brandon Houghton, Raul Sampedro:

   link-bibliography

8. https://arxiv.org/abs/2206.08356#facebook: “OmniMAE: Single Model Masked Pretraining on Images and Videos”⁠, Rohit Girdhar, Alaaeldin El-Nouby, Mannat Singh, Kalyan Vasudev Alwala, Arm, Joulin, Ishan Misra:

   link-bibliography

9. https://arxiv.org/abs/2206.07160#microsoft: “LAVENDER: Unifying Video-Language Understanding As Masked Language Modeling”⁠, Linjie Li, Zhe Gan, Kevin Lin, Chung-Ching Lin, Zicheng Liu, Ce Liu, Lijuan Wang:

   link-bibliography

10. https://arxiv.org/abs/2205.10747: “VidIL: Language Models With Image Descriptors Are Strong Few-Shot Video-Language Learners”⁠, Zhenhailong Wang, Manling Li, Ruochen Xu, Luowei Zhou, Jie Lei, Xudong Lin, Shuohang Wang, Ziyi Yang:

    link-bibliography

11. https://arxiv.org/abs/2205.09113#facebook: “Masked Autoencoders As Spatiotemporal Learners”⁠, Christoph Feichtenhofer, Haoqi Fan, Yanghao Li, Kaiming He:

    link-bibliography

12. https://arxiv.org/abs/2204.00598#google: “Socratic Models: Composing Zero-Shot Multimodal Reasoning With Language”⁠, Andy Zeng, Adrian Wong, Stefan Welker, Krzysztof Choromanski, Federico Tombari, Aveek Purohit, Michael Ryoo:

    link-bibliography

13. https://arxiv.org/abs/2203.11096: “CLIP Meets GamePhysics: Towards Bug Identification in Gameplay Videos Using Zero-shot Transfer Learning”⁠, Mohammad Reza Taesiri, Finlay Macklon, Cor-Paul Bezemer:

    link-bibliography

14. https://arxiv.org/abs/2201.12086#salesforce: “BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation”⁠, Junnan Li, Dongxu Li, Caiming Xiong, Steven Hoi:

    link-bibliography

15. https://arxiv.org/abs/2111.11432#microsoft: “Florence: A New Foundation Model for Computer Vision”⁠, Lu Yuan, Dongdong Chen, Yi-Ling Chen, Noel Codella, Xiyang Dai, Jianfeng Gao, Houdong Hu, Xuedong Huang:

    link-bibliography

16. https://arxiv.org/abs/2107.14795#deepmind: “Perceiver IO: A General Architecture for Structured Inputs & Outputs”⁠, Andrew Jaegle, Sebastian Borgeaud, Jean-Baptiste Alayrac, Carl Doersch, Catalin Ionescu, David Ding, Skanda Koppula:

    link-bibliography

17. https://arxiv.org/abs/2106.11097: “CLIP2Video: Mastering Video-Text Retrieval via Image CLIP”⁠, Han Fang, Pengfei Xiong, Luhui Xu, Yu Chen:

    link-bibliography

18. https://arxiv.org/abs/2103.07579#google: “Revisiting ResNets: Improved Training and Scaling Strategies”⁠, Irwan Bello, William Fedus, Xianzhi Du, Ekin D. Cubuk, Aravind Srinivas, Tsung-Yi Lin, Jonathon Shlens:

    link-bibliography

19. https://ai.facebook.com/blog/learning-from-videos-to-understand-the-world/: “Learning from Videos to Understand the World”⁠, Geoffrey Zweig, Polina Kuznetsova, Michael Auli, Francois Fagan:

    link-bibliography

20. https://arxiv.org/abs/2103.03206#deepmind: “Perceiver: General Perception With Iterative Attention”⁠, Andrew Jaegle, Felix Gimeno, Andrew Brock, Andrew Zisserman, Oriol Vinyals, Joao Carreira:

    link-bibliography

21. https://arxiv.org/abs/2102.00719: “Video Transformer Network”⁠, Daniel Neimark, Omri Bar, Maya Zohar, Dotan Asselmann:

    link-bibliography

22. https://cdn.openai.com/papers/Learning_Transferable_Visual_Models_From_Natural_Language_Supervision.pdf: “Learning Transferable Visual Models From Natural Language Supervision”⁠, Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry:

    link-bibliography

23. https://arxiv.org/abs/2012.08508#deepmind: “Object-based Attention for Spatio-temporal Reasoning: Outperforming Neuro-symbolic Models With Flexible Distributed Architectures”⁠, David Ding, Felix Hill, Adam Santoro, Matt Botvinick:

    link-bibliography

24. https://arxiv.org/abs/2008.09037: “Accuracy and Performance Comparison of Video Action Recognition Approaches”⁠, Matthew Hutchinson, Siddharth Samsi, William Arcand, David Bestor, Bill Bergeron, Chansup Byun, Micheal Houle:

    link-bibliography

25. https://arxiv.org/abs/1905.00546#facebook: “Billion-scale Semi-supervised Learning for Image Classification”⁠, I. Zeki Yalniz, Hervé Jégou, Kan Chen, Manohar Paluri, Dhruv Mahajan:

    link-bibliography

26. https://arxiv.org/abs/1811.11721: “CCNet: Criss-Cross Attention for Semantic Segmentation”⁠, Zilong Huang, Xinggang Wang, Yunchao Wei, Lichao Huang, Humphrey Shi, Wenyu Liu, Thomas S. Huang:

    link-bibliography

27. https://arxiv.org/abs/1808.01340#deepmind: “A Short Note about Kinetics-600”⁠, Joao Carreira, Eric Noland, Andras Banki-Horvath, Chloe Hillier, Andrew Zisserman:

    link-bibliography

28. https://arxiv.org/abs/1705.07750#deepmind: “Quo Vadis, Action Recognition? A New Model I3D and the Kinetics Dataset”⁠, Joao Carreira, Andrew Zisserman:

    link-bibliography