xifan.uno
大模型

Multimodal neurons in artificial neural networks

We’ve discovered neurons in CLIP that respond to the same concept whether presented literally, symbolically, or conceptually. This may explain CLIP’s accuracy in classifying surprising visual renditio

溯源：OpenAI News

Footnotes

  1. A

Note that the released CLIP models are intended strictly for research purposes. See the associated model card⁠(opens in a new window).

References

  1. 1

Quiroga, R. Q., Reddy, L., Kreiman, G., Koch, C., & Fried, I. (2005). Invariant visual representation by single neurons in the human brain⁠(opens in a new window). Nature, 435(7045), 1102-1107.

  1. 2

He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep residual learning for image recognition⁠(opens in a new window). In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 770-778).

  1. 3

Erhan, D., Bengio, Y., Courville, A., & Vincent, P. (2009). Visualizing higher-layer features of a deep network⁠(opens in a new window). University of Montreal, 1341(3), 1.

  1. 4

Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I., & Fergus, R. (2013). Intriguing properties of neural networks⁠(opens in a new window). arXiv preprint arXiv:1312.6199.

  1. 5

Mahendran, A., & Vedaldi, A. (2014). Understanding Deep Image Representations by Inverting Them⁠(opens in a new window). arXiv preprint arXiv:1412.0035.

  1. 6

Nguyen, A., Yosinski, J., & Clune, J. (2015). Deep neural networks are easily fooled: High confidence predictions for unrecognizable images⁠(opens in a new window). In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 427-436).

  1. 7

Øygard, A. (2015). Visualizing GoogLeNet Classes⁠(opens in a new window). Accessed in.

  1. 8

Mordvintsev, A., Olah, C., & Tyka, M. (2015). Inceptionism: Going deeper into neural networks⁠(opens in a new window).

  1. 9

Nguyen, A., Dosovitskiy, A., Yosinski, J., Brox, T., & Clune, J. (2016). Synthesizing the preferred inputs for neurons in neural networks via deep generator networks⁠(opens in a new window). arXiv preprint arXiv:1605.09304.

  1. 10

Nguyen, A., Clune, J., Bengio, Y., Dosovitskiy, A., & Yosinski, J. (2017). Plug & play generative networks: Conditional iterative generation of images in latent space⁠(opens in a new window). In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 4467-4477).

  1. 11

Nguyen, A., Yosinski, J., & Clune, J. (2016). Multifaceted feature visualization: Uncovering the different types of features learned by each neuron in deep neural networks⁠(opens in a new window). arXiv preprint arXiv:1602.03616.

  1. 12

Olah, C., Mordvintsev, A., & Schubert, L. (2017). Feature visualization⁠(opens in a new window). Distill, 2(11), e7.

  1. 13

Goh, G., et al. (2021). Multimodal Neurons in Artificial Neural Networks⁠(opens in a new window). Distill.

  1. 14

Miller, G. A. (1995). WordNet: a lexical database for English⁠(opens in a new window). Communications of the ACM, 38(11), 39-41.

  1. 15

Crawford, K. & Paglen, T. (2019). Excavating AI: the politics of images in machine learning training sets⁠(opens in a new window). Excavating AI.

  1. 16

Hanna, A., Denton, E., Amironesei, R,, Smart A., Nicole, H. Lines of Sight⁠(opens in a new window). Logic Magazine.

  1. 17

Fried, I., MacDonald, K. A., & Wilson, C. L. (1997). Single neuron activity in human hippocampus and amygdala during recognition of faces and objects⁠(opens in a new window). Neuron, 18(5), 753-765.

  1. 18

Kreiman, G., Koch, C., & Fried, I. (2000). Category-specific visual responses of single neurons in the human medial temporal lobe⁠(opens in a new window). Nature neuroscience, 3(9), 946-953.

  1. 19

Radford, A., Jozefowicz, R., & Sutskever, I. (2017). Learning to generate reviews and discovering sentiment⁠(opens in a new window). arXiv preprint arXiv:1704.01444.

  1. 20

Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013). Efficient estimation of word representations in vector space⁠(opens in a new window). arXiv preprint arXiv:1301.3781.

  1. 21

Brown, T. B., Mané, D., Roy, A., Abadi, M., & Gilmer, J. (2017). Adversarial patch⁠(opens in a new window). arXiv preprint arXiv:1712.09665.

  1. 22

Crawford, K. & Paglen, T. (2019). Excavating AI: the politics of images in machine learning training sets⁠(opens in a new window). Excavating AI.

Authors

Gabriel Goh, Chelsea Voss, Daniela Amodei, Shan Carter, Michael Petrov, Justin Jay Wang, Nick Cammarata, Chris Olah

Acknowledgments

Sandhini Agarwal, Greg Brockman, Miles Brundage, Jeff Clune, Steve Dowling, Jonathan Gordon, Gretchen Krueger, Faiz Mandviwalla, Vedant Misra, Reiichiro Nakano, Ashley Pilipiszyn, Alec Radford, Aditya Ramesh, Pranav Shyam, Ilya Sutskever, Martin Wattenberg & Hannah Wong