EECS Department Special Colloquium  
Monday, March 29, 2004 Professor Toby Berger Electrical and Computer Engineering, 

An InformationTheoretic Treatment of Real Neural Nets with Feedback 

Abstract: 

Inspired by the fact that the anatomical structure of primate visual cortex can be well modeled as a hierarchy of timediscrete, finitestate channels with feedback, we explore such channels from the perspective of information theory. We sketch the proof of a coding theorem and converse which establish that the constrained limit superior of the MarkoMassey directed information rate generates the Shannon capacitycost function of finitestate channels with feedback. We also show that any input process which drives such a channel at its constrained capacity produces an output process which is such that the joint input/output process is Markovian. Moreover, said output process is marginally Markovian, though the input process itself need not be. The findings explicate how the brain is able to utilize its RNNs in an effectively Shannon optimum manner robustly in the average cost despite the fact that it employs neither encoders nor decoders in the traditional sense of these terms in communication and information theory.


Biography:  
Toby Berger, the Irwin and Joan Jacobs Professor of Engineering, conducts research interests in information theory, random fields, communication networks, wireless communications, video compression, voice and signature compression and verification, neuroinformation theory, quantum information theory, and coherent signal processing. He is the author of Rate Distortion Theory, a coauthor of Digital Compression for Multimedia and a coauthor of Information Measures for Discrete Random Fields. Berger has served as editorinchief of the IEEE Transactions on Information Theory and as president of the IEEE Information Theory Group. He has been a Fellow of the Guggenheim Foundation, the Japan Society for Promotion of Science, the Ministry of Education of the People's Republic of China and the Fulbright Foundation. In 1982 he received the Frederick E. Terman Award of the American Society for Engineering Education, and in 2002 he received the Shannon Award from the IEEE Information Theory Society. Berger is a Fellow of the IEEE, a life member of Tau Beta Pi, and a blues harmonica player. 