Last update: 10/22/2007

Technical Notes on Linear Regression and Information Theory

Asari, H.


To understand how the brain processes sensory information, it is important to study the relationship between input stimuli and the output neural responses. Neuroscientists have typically looked at two complementary aspects of neural representations. The first, and best studied, is the encoding process by which a stimulus is converted by the nervous system into neural activity. Less studied is the decoding process, by which experimenters attempt to use neural activity to reconstruct the stimulus that evoked it. To characterize these processes, various methods have been developed to model the stimulus-response functions and to test their performance.

Here we briefly overview the basics and logics of these methods. The first part reviews linear regression methods with a certain regularization to find the best linear models. In particular, we will go through how ridge regression is related to the singular value decomposition. The second part shows how to apply information theory to test the quality of linear filters. We will discuss the connection of correlation functions to entropy and information, and a way to compute information by exploiting singular value decomposition (SVD).

Preprint (107KB, PDF)
See also my dissertation (Appendix A).