EECS Department Special Colloquium

Monday, March 29, 2004
521 Cory Hall, Hogan Room
4:00-5:30 p.m.

Professor Toby Berger

Electrical and Computer Engineering,
Cornell University


An Information-Theoretic Treatment of Real Neural Nets with Feedback




Inspired by the fact that the anatomical structure of primate visual cortex can be well modeled as a hierarchy of time-discrete, finite-state channels with feedback, we explore such channels from the perspective of information theory. We sketch the proof of a coding theorem and converse which establish that the constrained limit superior of the Marko-Massey directed information rate generates the Shannon capacity-cost function of finite-state channels with feedback. We also show that any input process which drives such a channel at its constrained capacity produces an output process which is such that the joint input/output process is Markovian. Moreover, said output process is marginally Markovian, though the input process itself need not be. The findings explicate how the brain is able to utilize its RNNs in an effectively Shannon optimum manner robustly in the average cost despite the fact that it employs neither encoders nor decoders in the traditional sense of these terms in communication and information theory.


Toby Berger, the Irwin and Joan Jacobs Professor of Engineering, conducts research interests in information theory, random fields, communication networks, wireless communications, video compression, voice and signature compression and verification, neuroinformation theory, quantum information theory, and coherent signal processing. He is the author of Rate Distortion Theory, a co-author of Digital Compression for Multimedia and a co-author of Information Measures for Discrete Random Fields. Berger has served as editor-in-chief of the IEEE Transactions on Information Theory and as president of the IEEE Information Theory Group. He has been a Fellow of the Guggenheim Foundation, the Japan Society for Promotion of Science, the Ministry of Education of the People's Republic of China and the Fulbright Foundation. In 1982 he received the Frederick E. Terman Award of the American Society for Engineering Education, and in 2002 he received the Shannon Award from the IEEE Information Theory Society. Berger is a Fellow of the IEEE, a life member of Tau Beta Pi, and a blues harmonica player.