Natural Image Statistics

1 Aapo Hyv arinenJarmo HurriPatrik O. HoyerNatural Image StatisticsA probabilistic approach to earlycomputational visionFebruary 27, 2009 SpringerContents overview1 Introduction..1 Part I Background2 Linear filters and frequency analysis.. 253 Outline of the visual system.. 514 Multivariate probability and Statistics .. 69 Part II Statistics of linear features5 Principal components and whitening.. 976 Sparse coding and simple cells.. 1377 Independent component analysis.. 1598 Information-theoretic interpretations.. 185 Part III Nonlinear features & dependency of linear features9 Energy correlation of linear features & normalization.. 20910 Energy detectors and complex cells.. 22311 Energy correlations and topographic organization.. 24912 Dependencies of energy detectors: Beyond V1.. 27313 Overcomplete and non-negative models.. 28914 Lateral interactions and feedback.. 307 Part IV Time, colour and stereo15 Colour and stereo images.

2 32116 Temporal sequences of Natural images.. 337 Part V Conclusion17 Conclusion and future prospects.. 379 Part VI Appendix: Supplementary mathematical tools18 Optimization theory and algorithms.. 39319 Crash course on linear algebra.. 41520 The discrete Fourier transform.. 42321 Estimation of non-normalized statistical models.. 437 Index.. 445 References.. 453vContents1 Introduction.. What this book is all about .. What is vision? .. The magic of your visual system .. Importance of prior information .. adaptation provides prior information .. models and latent quantities .. onto the retina loses information .. inference and priors .. Natural images .. Image space .. of Natural images .. Redundancy and information .. theory and Image coding .. reduction and neural coding .. Statistical modelling of the visual system .. information theory and Bayesian inference.

3 Vs. descriptive modelling of visual system.. predictive theoretical neuroscience .. Features and statistical models of Natural images .. representations and features .. of features .. features to statistical models .. The statistical-ecological approach recapitulated .. References .. 21 Part I Background2 Linear filters and frequency analysis.. Linear filtering .. response and convolution .. Frequency-based representation .. in one and two dimensions .. representation and linear filtering.. and mathematical details .. Representation using linear basis .. idea .. representation as a basis .. Space-frequency analysis .. analysis and Gabor filters .. localization vs. spectral accuracy .. References .. Exercices .. 483 Outline of the visual system.. Neurons and firing rates .. From the eye to the cortex.

4 Linear models of visual neurons .. to visual stimulation .. cells and linear models .. models and selectivities of simple cells .. channels .. Nonlinear models of visual neurons .. in simple-cell responses .. cells and energy models .. Interactions between visual neurons .. Topographic organization .. Processing after the primary visual cortex .. References .. Exercices .. 674 Multivariate probability and Statistics .. Natural images patches as random vectors .. Multivariate probability distributions .. and motivation .. density function .. Marginal and joint probabilities .. Conditional probabilities .. Independence .. Expectation and covariance .. and covariance in one dimension .. matrix .. and covariances .. Bayesian inference .. example .. Rule .. priors.

5 Inference as an incremental learning process .. Parameter estimation and likelihood .. , estimation, and samples .. likelihood and maximum a posteriori .. and large samples .. References .. Exercices .. 92 Part II Statistics of linear features5 Principal components and whitening.. DC component or mean grey-scale value .. Principal component analysis .. basic dependency of pixels in Natural images .. one feature by maximization of variance .. many features by PCA .. implementation of PCA .. implications of translation-invariance .. PCA as a preprocessing tool .. reduction by PCA .. by PCA .. by PCA .. Canonical preprocessing used in this book .. Gaussianity as the basis for PCA .. probability model related to PCA .. as a generative model .. synthesis results .. Power spectrum of Natural images .. 1/fFourier amplitude or 1/f2power spectrum.

6 Between power spectrum and covariances .. importance of amplitude and phase .. Anisotropy in Natural images .. Mathematics of principal component analysis* .. decomposition of the covariance matrix .. and translation-invariance .. Decorrelation models of retina and LGN * .. and redundancy reduction .. decorrelation .. decorrelation .. Concluding remarks and References .. Exercices .. 1356 Sparse coding and simple cells.. Definition of sparseness .. Learning one feature by maximization of sparseness .. sparseness: General framework .. sparseness using kurtosis .. sparseness using convex functions of square .. case of canonically preprocessed data .. feature learned from Natural images .. Learning many features by maximization of sparseness .. decorrelation .. decorrelation .. of feature vs. sparseness of representation .. Sparse coding features for Natural images.

7 Set of features .. of tuning properties .. How is sparseness useful? .. modelling .. modelling .. economy .. Concluding remarks and References .. Exercices .. 1567 Independent component analysis.. Limitations of the sparse coding approach .. Definition of ICA .. model .. for preprocessed data .. Insufficiency of second-order information .. whitening does not find independent components .. components have to be non-gaussian .. The probability density defined by ICA .. Maximum likelihood estimation in ICA .. Results on Natural images .. of features .. synthesis using ICA .. Connection to maximization of sparseness .. as a measure of sparseness .. sparseness measures .. Why are independent components sparse? .. forms of non-gaussianity .. in Natural images .. is sparseness dominant? .. General ICA as maximization of non-gaussianity.

8 Limit Theorem .. Non-gaussian is independent .. coding as a special case of ICA .. Receptive fields vs. feature vectors .. Problem of inversion of preprocessing .. Frequency channels and ICA .. Concluding remarks and References .. Exercices .. 1838 Information-theoretic interpretations.. Basic motivation for information theory .. Entropy as a measure of uncertainty .. of entropy .. as minimum coding length .. entropy .. entropy .. Mutual information .. Minimum entropy coding of Natural images .. compression and sparse coding .. information and sparse coding .. entropy coding in the cortex .. Information transmission in the nervous system .. of information flow and infomax .. infomax with linear neurons .. with nonlinear neurons .. with non-constant noise variance .. Caveats in application of information theory .. Concluding remarks and References.

9 Exercices .. 204 Part III Nonlinear features & dependency of linear features9 Energy correlation of linear features & normalization.. Why estimated independent components are not independent .. vs. theoretical components .. the number of free parameters .. Correlations of squares of components in Natural images.. Modelling using a variance variable .. Normalization of variance and contrast gain control .. Physical and neurophysiological interpretations .. the effect of changing lighting conditions .. surfaces .. of cell responses .. Effect of normalization on ICA .. Concluding remarks and References .. Exercices .. 22110 Energy detectors and complex cells.. Subspace model of invariant features .. Why linear features are insufficient .. Subspaces or groups of linear features .. Energy model of feature detection .. Maximizing sparseness in the energy model.

10 Definition of sparseness of output .. One feature learned from Natural images .. Model of independent subspace analysis .. Dependency as energy correlation .. Why energy correlations are related to sparseness .. Spherical symmetry and changing variance .. Correlation of squares and convexity of nonlinearity .. Connection to contrast gain control .. ISA as a nonlinear version of ICA .. Results on Natural images .. Emergence of invariance to phase .. The importance of being invariant .. Grouping of dependencies .. Superiority of the model over ICA .. Analysis of convexity and energy correlations* .. Variance variable model gives convexh.. Convexhtypically implies positive energy correlations .. Concluding remarks and References .. 24811 Energy correlations and topographic organization.. Topography in the cortex .. Modelling topography by statistical dependence.