Probability with Engineering Applications

1 Probability with Engineering ApplicationsECE 313 Course NotesBruce HajekDepartment of Electrical and Computer EngineeringUniversity of Illinois at Urbana-ChampaignAugust 2021c 2021 by Bruce HajekAll rights reserved. Permission is hereby given to freely print and circulate copies of these notes so long as the notes are leftintact and not reproduced for commercial purposes. Email to pointing out errors or hard to understandpassages or providing comments, is Embracing uncertainty .. Axioms of Probability .. Calculating the size of various sets .. Probability experiments with equally likely outcomes .. Sample spaces with infinite cardinality .. Short Answer Questions .. Problems .. 212 Discrete-type random Random variables and Probability mass functions .. The mean and variance of a random variable .. Conditional probabilities .. Independence and the binomial distribution .. independent events.

2 Random variables (of discrete-type) .. distribution .. distribution .. Geometric distribution .. Bernoulli process and the negative binomial distribution .. The Poisson distribution a limit of binomial distributions .. Maximum likelihood parameter estimation .. Markov and Chebychev inequalities and confidence intervals .. The law of total Probability , and Bayes formula .. Binary hypothesis testing with discrete-type observations .. Maximum likelihood (ML) decision rule .. Maximum a posteriori Probability (MAP) decision rule .. Reliability .. Union bound .. Network outage Probability .. distribution of the capacity of a flow network .. Analysis of an array code .. Reliability of a single backup .. Short Answer Questions .. Problems .. 763 Continuous-type random Cumulative distribution functions .. Continuous-type random variables.

3 Uniform distribution .. Exponential distribution .. Poisson processes .. Bernoulli processes .. and properties of Poisson processes .. Erlang distribution .. Linear scaling of pdfs and the Gaussian distribution .. rule for pdfs .. Gaussian (normal) distribution .. central limit theorem and the Gaussian approximation .. ML parameter estimation for continuous-type variables .. Functions of a random variable .. distribution of a function of a random variable .. a random variable with a specified distribution .. area rule for expectation based on the CDF .. Failure rate functions .. Binary hypothesis testing with continuous-type observations .. Short Answer Questions .. Problems .. 1484 Jointly Distributed Random joint cumulative distribution functions .. joint Probability mass functions .. joint Probability density functions .. Independence of random variables.

4 Of independence for two random variables .. from a pdf whether independence holds .. distribution of sums of random variables .. of integer-valued random variables .. of jointly continuous-type random variables .. Additional examples using joint distributions .. joint pdfs of functions of random variables .. of pdfs under a linear mapping .. of pdfs under a one-to-one mapping .. of pdfs under a many-to-one mapping .. Correlation and covariance .. Minimum mean square error estimation .. estimators .. estimators .. estimators .. Law of large numbers and central limit theorem .. Law of large numbers .. Central limit theorem .. joint Gaussian distribution .. From the standard 2-d normal to the general .. Key properties of the bivariate normal distribution .. Higher dimensional joint Gaussian distributions .. Short Answer Questions .. Problems .. 2255 Wrap-up2396 Some notation.

5 Some sums .. Frequently used distributions .. discrete-type distributions .. continuous-type distributions .. Normal tables .. Answers to short answer questions .. Solutions to even numbered problems .. 248viCONTENTSP refaceA key objective of these notes is to convey how to deal with uncertainty in both qualitative andquantitative ways. Uncertainty is typically modeled as randomness. We must make decisions withpartial information all the time in our daily lives, for instance when we decide what activities topursue. Engineers deal with uncertainty in their work as well, often with precision and challenge in applying reasoning to real world situations is to capture the main issues in amathematical model. The notation that we use to frame a problem can be critical to understandingor solving the problem. There are often events, or variables, that need to be given theory is widely used to model systems in Engineering and scientific notes adopt the most widely used framework of Probability , namely the one based on Kol-mogorov s axioms of Probability .

6 The idea is to assume a mathematically solid definition of themodel. This structure encourages a modeler to have a consistent, if not completely accurate, also offers a commonly used mathematical language for sharing models and of the process of learning to use the language of Probability theory is learning classificationsof problems into broad areas. For example, some problems involve finite numbers of possiblealternatives, while others concern real-valued measurements. Many problems involve interaction ofphysically independent processes. Certain laws of nature or mathematics cause some probabilitydistributions, such as the normal bell-shaped distribution often mentioned in popular literature, tofrequently appear. Thus, there is an emphasis in these notes on well-known Probability distributionsand why each of them arises frequently in notes were written for the undergraduate course,ECE 313: Probability with EngineeringApplications,offered by the Department of Electrical and Computer Engineering at the Universityof Illinois at Urbana-Champaign.

7 The official prerequisites of the course ensure that students havehad calculus, including Taylor series expansions, integration over regions in the plane, the use ofpolar coordinates, and some basic linear author gratefully acknowledges the students and faculty who have participated in thiscourse through the years. He is particularly grateful to Professor D. V. Sarwate, who first intro-duced the course, and built up much material for it on the HajekAugust 2021viiCONTENTS1 OrganizationChapter 1 presents an overview of the many Applications of Probability theory, and then explainsthe basic concepts of a Probability model and the axioms commonly assumed of Probability probabilities are assigned to possible outcomes based on symmetry. For example, when a sixsided die is rolled, it is usually assumed that the Probability a particular numberishows is 1/6,for 1 i this reason, we also discuss in chapter 1 how to determine the sizes of variousfinite sets of possible variables are introduced in chapter 2 and examined in the context of a finite, orcountably infinite, set of possible outcomes.

8 Notions of expectation (also known as mean), variance,hypothesis testing, parameter estimation, multiple random variables, and well known probabilitydistributions Poisson, geometric, and binomial, are covered. The Bernoulli process is considered itprovides a simple setting to discuss a long, even infinite, sequence of event times, and provides atie between the binomial and geometric Probability focus shifts in chapter 3 from discrete-type random variables to continuous-type randomvariables. The chapter takes advantage of many parallels and connections between discrete-typeand continuous-type random variables. The most important well known continuous-type distribu-tions are covered: uniform, exponential, and normal (also known as Gaussian). Poisson processesare introduced they are continuous-time limits of the Bernoulli processes described in Chapter2. Parameter estimation and binary hypothesis testing are covered for continuous-type randomvariables in this chapter as they are for discrete-type random variables in chapter 4 considers groups of random variables, with an emphasis on two random include describing the joint distribution of two random variables, covariance and correla-tion coefficient, and prediction or estimation of one random variable given observation of more advanced notions from calculus come in here, in order to deal with joint probabilitydensities, entailing, for example, integration over regions in two answer questions and problems can be found at the end of each chapter with answers tothe questions and even numbered problems provided in the appendix.

9 Most of the short answerquestions also have video links to solutions which should open if you click on [video] in the pdfversion of these notes, if you have a browser and internet connection. A brief wrap up is givenin chapter 5. A small number of other videos are provided for examples. These videos are notmeant to substitute for reading the notes; it is recommended students attempt solving problemsand watch the videos afterwards if Embracing uncertaintyWe survive and thrive in an uncertain world. What are some uses of Probability in everyday life?In Engineering ? Below is an incomplete list: Call centers and other staffing problems:Experts with different backgrounds are neededto staff telephone call centers for major financial investment companies, travel reservationservices, and consumer product support. Management must decide the number of staff and themix of expertise so as to meet availability and waiting time targets. A similar problem is facedby large consulting service companies, hiring consultants that can be grouped into multipleoverlapping teams for different projects.

10 The basic problem is to weigh future uncertaindemand against staffing costs. Electronic circuits:Scaling down the power and energy of electronic circuits reduces thereliability and predictability of many individual elements, but the circuits must neverthelessbe engineered so the overall circuit is reliable. Wireless communication:Wireless links are subject to fading, interference from othertransmitters, doppler spread due to mobility and multipath propagation. The demand, suchas the number of simultaneous users of a particular access point or base station, is also timevarying and not fully known in advance. These and other effects can vary greatly with timeand location, but yet the wireless system must be engineered to meet acceptable call qualityand access probabilities. Medical diagnosis and treatment:Physicians and pharmacologists must estimate themost suitable treatments for patients in the face of uncertainty about the exact condition ofthe patient or the effectiveness of the various treatment options.