A Gentle Introduction to Empirical Process Theory and ...

1 A Gentle Introduction to Empirical Process Theory andApplicationsBodhisattva SenJuly 19, 2022 Contents1 Introduction to Empirical Notation .. (or Empirical risk minimization) .. Why study weak convergence of stochastic processes? .. Asymptotic equicontinuity ..142 Size/complexity of a function Covering numbers .. Bracketing numbers ..203 Glivenko-Cantelli (GC) classes of GC by bracketing .. Preliminaries .. s inequality for the sample mean .. random variables/processes .. Symmetrization.

2 Proof of GC by entropy .. applications .. ofM/Z-estimators .. of least squares regression .. Bounded differences inequality a simple concentration inequality .. Supremum of the Empirical Process for a bounded class of functions ..3914 Chaining and uniform Dudley s bound for the supremum of a sub-Gaussian Process .. s bound when the metric space is separable .. Maximal inequality with uniform entropy .. Maximal inequalities with bracketing .. Bracketing number for some function classes.

3 515 Rates of convergence The rate theorem .. Some examples .. parameter .. non-standard example .. in high-dimensional regression ..596 Rates of convergence of infinite dimensional Least squares regression on sieves .. Least squares regression: a finite sample inequality .. Oracle inequalities .. sparse linear regression .. Density estimation via maximum likelihood ..737 Vapnik- Cervonenkis (VC) classes of VC classes of Boolean functions .. Covering number bound for VC classes of sets.

4 VC classes of functions .. Examples and Permanence Properties .. Exponential tail bounds: some useful inequalities ..938 Talagrand s concentration inequality for the suprema of the Preliminaries .. Talagrand s concentration inequality .. Empirical risk minimization and concentration inequalities .. formal result on excess risk in ERM .. risk in bounded regression .. Kernel density estimation .. 1149 Review of weak convergence in complete separable metric Weak convergence of random vectors inRd.

5 Weak convergence in metric spaces and the continuous mapping theorem .. (T), the Borel -field ofT.. general continuous mapping theorem .. Weak convergence in the spaceC[0,1] .. and relative compactness .. and weak convergence inC[0,1] .. Non-measurablilty of the Empirical Process .. [0,1] with the ball -field .. 13010 Weak convergence in non-separable metric Bounded stochastic processes .. Spaces of locally bounded functions .. 14211 Donsker classes of Donsker classes under bracketing condition.

6 Donsker classes with uniform covering numbers .. Donsker theorem for classes changing with sample size .. 14612 Limiting distribution Argmax continuous mapping theorems .. Asymptotic distribution .. A non-standard example .. 15613 Concentration Efron-Stein inequality .. Concentration and logarithmic Sobolev inequalities .. The Entropy method .. Gaussian concentration inequality .. Bounded differences inequality revisited .. Suprema of the Empirical Process : exponential inequalities.

7 1693 AbstractThis document provides an Introduction to the Theory of Empirical processes. Thestandard references on this topic ( , [van der Vaart and Wellner, 1996]) usually de-velop all the abstract concepts in detail before they address the statistical this is certainly the right approach to provide a rigorous treatment of theapplications, I believe that this has somewhat hindered some graduate students fromappreciating the usefulness of the topic. In this set of lecture notes, I try to address afew statistical applications at the end of every section and do not go into the rigoroustreatment of certain topics to make the material more accessible to a broader document arose from the lecture notes that I delivered at Stanford in Spring most graduate students in statistics now-a-days are not necessarily exposed tothe Theory of weak convergence of stochastic processes ( , in the spaceC[0,1] orD[0,1])

8 This document tries to give the reader a brief overview of this classical Theory (in Section 9). I hope this will make the transition to the Theory of weak convergenceon abstract spaces would like to thank Aditya Guntuboyina for several helpful discussions and forsharing his lecture notes on this subject (indeed, the treatment of some of the topics inthis document is taken from his lecture notes1). I am thankful to Axel Munk and TobiasKley2(who discussed some of these lecture notes in one of their seminar classes) andthe students3in their class, and to the members of the Empirical Processes ReadingGroup coordinated by Chao Zheng4at University of Southampton, for pointing outnumerous typos, inconsistencies, etc.

9 In the notes. I am also thankful to ChaowenZheng (University of York), Myoung-Jin Keay (South Dakota State), Huiyuan Wang(Peking University), Zhen Huang (Columbia University) for pointing out further typosand of the examples given in this document is borrowed from the following books:[Gin e and Nickl, 2016], [Koltchinskii, 2011], [Pollard, 1984], [Wainwright, 2019], [van de Geer, 2000],[van der Vaart, 1998], [van der Vaart and Wellner, 1996].1 ~aditya/ Shayan Hundrieser, Marcel Klatt, and Thomas Staudt3 Tobias W.

10 Wegel, Erik Pudelko, Jan N. D uhmert, Meggie Marschner, Antonia Seifrid, Jana B ohm,Robin Requadt, Oliver D. Gauselmann, Tobias Weber, Huaiqing Gou, Leo H. Lehmann, Michel Groppe4 Introduction to Empirical processesIn this chapter we introduce the main object of study ( , Empirical processes), highlightthe main questions we would like to answer, give a few historically important statisticalapplications that motivated the development of the field, and lay down some of the broadquestions that we plan to investigate in this Process Theory began in the 1930 s and 1940 s with the study of the empiricaldistribution function and the corresponding Empirical .