Direct method

In this paper we will use a direct method [DeWeese, 1995, DeWeese, 1996, Stevens and Zador, 1996, de Ruyter van Steveninck et al., 1997] to estimate the mutual information. Direct methods use another form of the expression eq. (5) for mutual information,
 
The first term is the entropy of the output spike train itself, while the second is the conditional entropy of the output given the inputs. The first term measures of the variability of the spike train in response to the ensemble of different inputs, while the second measures the reliability of the response to repeated presentations of the same inputs. The second term depends on the reliability of the synapses and spike generating mechanism: to the extent the same inputs produce the same outputs, this term approaches zero.

The direct method has two advantages over the reconstruction method in the present context. First, it does not require the construction of a ``reconstructor'' for estimating the input from the output. Although the optimal linear reconstructor is straightforward to estimate, the construction of more sophisticated (i.e. nonlinear) reconstructors can be a delicate art. Second, it provides an estimate of information that is limited only by the errors in the estimation of and ; the reconstruction method by contrast provides only a lower bound on the mutual information that is limited by the quality of the reconstructor.

As noted above, the estimation of and can require vast amounts of data. If, however, interspike intervals (ISIs) in the output spike train were independent, then the entropies could be simply expressed in terms of the entropy of the associated ISI distributions. The information per spike is then given simply by
 
where H(T) are are total and conditional entropies, respectively, of the ISI distribution. The information rate (units: bits/second) is then just the information per spike (units: bits/spike) times the firing rate R (units: spikes/second),
 

The representation of the output spike train as a sequence of firing times is entirely equivalent (except for edge effects) to the representation as a sequence of ISIs , where . The advantage of using ISIs rather than spike times is that H(T) depends only on the ISI distribution p(T), which is a univariate distribution. This dramatically reduces the amount of data required.

In the sequel we assume that spike times are discretized at a finite time resolution . The assumption of finite precision keeps the potential information finite. If this assumption is not made, each spike has potentially infinite information capacity; for example, a message of arbitrary length could be encoded in the decimal expansion of a single ISI.

Eq. 8 represents the information per spike as the difference between two entropies. The first term is the total entropy per spike,
 
where is the probability that the length of the ISI was between and . The distribution of ISIs can be obtained from a single long (ideally, infinite) sequence of spike times.

The second term is the conditional entropy per spike. The conditional entropy is just the entropy of the ISI distribution in response to a particular set m of input spikes , averaged over all possible sets of inputs spikes
 
where represents average. Here is the probability of obtaining an ISI of length in response to a particular particular set of input spikes .

We used the following algorithm for estimating the conditional entropy:

1. Generate ensemble of input spikes. Some particular ensemble of input spikes (corresponding, for example, to m=17) is generated, , where are independent homogenous Poisson processes (for convenience we assume they have the same rate, but this is not essential).
2. Compute conditional ISI distribution. The conditional distribution of ISIs of the model neuron is obtained by measuring the ISIs on a large (ideally, an infinite) number of trials in which a synaptic current is generated from using the synaptic noise equations eq. 2 or eq. 3. If the noise is nonzero, then each realization of the synaptic current is slightly different, leading to variability in the output ISI.
3. Compute conditional entropy for this input ensemble. From the conditional distribution, the conditional entropy in response to this particular input ensemble is computed as . This ISI distribution depends on the amount of synaptic noise assumed; if there is no noise, the output distribution assumes only a single value and the conditional entropy is zero.
4. Repeat, and average over conditional entropies for other ensemble. The average conditional entropy per spike is calculated by repeating this procedure for a large (ideally, infinite) number of input patterns and averaging over the resulting conditional entropies.

In summary, we have described the three steps required to compute the information rate in our model. First, the total entropy per spike is computed from Eq. 10, and the conditional entropy per spike is computed from Eq. 11. Next, the information per spike is computed from Eq. 8. Finally, the information rate (information per time) is computed from Eq. 9.