Host Power Management in VMware vSphere 7

1 VMware , I nc. 3401 Hillview Avenue Palo Alto CA 94304 USA Tel 877-486-9273 Fax 650-427-5001 www. VMware . com Copyright 2021 VMware , Inc. All rights reserved. This product is protected by and international copyright and intellectual property laws. VMware products are covered by one or more patents listed at VMware is a registered trademark or trademark of VMware , I nc. in the United States and/or other jurisdictions. All other marks and names mentioned herein may be trademarks of their respective companies. Host Power Management in VMware vSphere performance Study for Optimal Power Consumption September 28, 2021 Host Power Management in VMware vSphere | Page 2 Table of Contents Executive Summary .. 3 Introduction .. 3 Background .. 4 C-States (I dle States/ Power States) .. 4 P-State (Operational States/ performance State): Demand-Based Switching (DBS) .. 4 Power Management Settings .. 5 BI OS Settings.

2 6 Host Power Management in vSphere 7. 0 .. 6 Defining Custom Power Policy in vSphere .. 8 Summary: vSphere Power Policies .. 8 Experimental Setup .. 9 Hardware .. 9 vSphere Power Management and esxtop .. 10 Results .. 12 I mpact of P-States and C-States on Host Power Consumption .. 12 Observation 1: Only P-State Management is Enabled .. 13 Observation 2: Both P-State and C-State Management are Enabled .. 13 LoadGen .. 14 SPEC Power .. 15 VMmark .. 17 View Planner (VDI) .. 20 Impact of vSphere Power Policies on Turbo 23 Best Practices .. 25 References .. 26 About the Authors .. 27 Acknowledgments .. 27 Appendix A .. 28 Host Power Management in VMware vSphere | Page 3 Executive Summary Modern datacenters, which are vital to the computing infrastructure, are expensive to build and operate. A significant part of the operational cost of a modern data center can be attributed to Power consumption and cooling.

3 In an ideal world, we would want to minimize Power consumption without any impact on application performance . H owever, practical cases require a compromise. Finding the right balance entails considerable financial gains. VMware vSphere (ESXi) provides several pre-configured policies for Power Management , targeting applications to optimize for latency, throughput , and Power consumption. Prior technical papers have described the available Power policies in vSphere , their functionality, and present performance and Power consumption data [1] [2]. This technical paper provides an updated guide to vSphere 7. 0 users on different Power policies performance and Power tradeoffs. In addition, the article uses experimental results to gain insights regarding the Power Management capabilities of VMware vSphere on the latest processors using modern workloads. Note that there are no algorithmic changes in the implementation of Power Management in vSphere I nstead, the article evaluates the vSphere Power Management on the latest processors with several recent workloads.

4 While observing the performance and Power consumption on a range of workloads, the paper recommends the use of Power policies as follows: (a) The Balanced policy is the default recommended policy in vSphere , because it maximizes performance -per-watt overall and provides an optimal point between performance and Power consumption for a wide variety of application characteristics. (b) The High performance profile is preferred in the case of latency-sensitive applications at the cost of higher Power consumption, and (c) Low Power profile for low utilization servers, resulting in higher Power savings at the cost of performance . Introduction The Power used by servers and data centers accounts for ~ 1. 5% of the total Power consumption in the world [3]. CPUs in modern data centers consume ~ 30% of the overall system Power . As CPU hardware complexity has grown significantly in recent years, computer architects have introduced several Power -focused enhancements.

5 These designs have enabled operating systems to capitalize on architectural tools to improve performance -per-watt on various applications. The introduction of demand-based switching (DBS) from Intel has been the critical factor behind Power savings in modern processors. Host Power Management in VMware vSphere | Page 4 The Advanced Configuration and Power Interface (ACPI) specification is an open standard initially developed by several hardware vendors and software developers [4]. It establishes standard interfaces that enable operating system directed motherboard device configuration and Power Management . ACPI can appl y to both individual hardware components and the entire system. In addition, it helps monitor the system s status and employs Power Management algorithms by changing the CPU operating frequency and putting unused components to sleep [5]. Background The ACPI standard defines C-States (commonly known as idle states or Power states) and P-States (widel y kn own as operational states or performance states).

6 Hardware vendors such as Intel and AMD have introduced hardware support for P-States and C-States, which are accessible through the on-chip Power control unit (PCU) [6] [ 7]. C-States (Idle States/ Power States) C-States are Power states that aid in saving Power by turning off sub-sections of the CPU when not in use. CPUs are designed to support various Power levels. C0-state is the operational state where all the components are active, and the processor can actively execute instructions. C1 is a shallow state where the clock is gated (switched off). However, all the modules remain active, and the processor can go back to the active C0 state instantaneously. Furthermore, C2-Cn are sleep states where specific sections of the CPU are turned off. The higher the C-State, the deeper into sleep mode the CPU goes. Thus, higher C-States result in significant Power savings. However, it takes more time for the CPU to return to the operational state from deeper sleep states.

7 Therefore, changing the C-state in the BIOS setting doesn t have an impact on throughput . H owever, the latency to get back from a deeper sleep state would be higher. P-State (Operational States/ performance State): Demand-Based Switching (DBS) P-States correspond to different performance levels that are applied while the processor is actively executing instructions. P-States are relevant only when the processor is in the active C0-state. P-State is both a frequency and voltage operating point defined as performance states in the ACPI specification. Both frequency and voltage are scaled as the P-State increases. This process is referred to as dynamic voltage and frequency scaling (DVFS). Hardware vendors such as Intel incorporate several hardware-level P-States that divide the energy and frequency demands into several tiers. P0 is the highest frequency (with the highest voltage). I t is often referred to as the turbo mode.

8 P1 is the nominal/base operating frequency. Higher P-States correspond to lower operating frequencies. P-States pl ay a crucial role in saving CPU Power when the workload does not fully load a CPU. Host Power Management in VMware vSphere | Page 5 Figure 1 shows how P-States and C-States relate to each other. The P-States and C-States form the basis for Power Management in modern processors. The number of available P-States and C-States vary with vendors and processor generations. VMware vSphere utilizes the available P- and C-States in the Power Management algorithms to provide easy-to-use Power profi l es for end-users. Figure 1: Representation of ACPI P-States and C-States Power Management Settings VMware vSphere includes a comprehensive set of Power Management capabilities. These are designed to save Power during idle times and when the host is underutilized or is inactive. The user is recommended to configure the BI OS settings to OS control mode or the equivalent to allow vSphere the most flexibility to use available hardware features for optimal Power consumption.

9 An example of how to select the BI OS settings to allow vSphere to control Power Management in a Dell PowerEdge system is shown below in Figure 2. Similar options can be found in servers from other vendors as well. C0P0P1P2P3C1C2 CnPn.. performance States (P-States) Power States (C-States) Host Power Management in VMware vSphere | Page 6 BIOS Settings To set the recommended BIOS settings, follow these steps before deployment. 1. Enter the System BI OS settings by pressing F2 during bootup. 2. Select System Setup > System Profile Settings. 3. The default System Profile on Dell systems is performance Per Watt (DAPC). Change this to performance Per Watt (OS) to transfer control to vSphere /ESXi. 4. All other settings are selected by default. Figure 2: Sample BIOS setting on a Dell PowerEdge R640 host Host Power Management in vSphere vSphere offers four different Power Management policies that utilize the ACPI P-States ( performance States) and ACPI C-States ( Power States).

10 The available Power policies are as follows: High performance : This policy tries to maximize performance by disabling C-State and P-State Management . It always keeps the CPU in the highest possible operating frequency (P0-State) and only uses the top two shallow C-States (C0 when running and C1 when idle). The High- performance policy is geared toward latency-sensitive applications and provides predictable and consistent performance . Host Power Management in VMware vSphere | Page 7 Balanced: This policy is designed to minimize host Power consumption while having little to no impact on performance . The balanced policy is the default Power policy in vSphere The policy determines the system load and selects the appropriate P-State. The Demand Based Switching (DBS) implementation here has little to no impact on application performance . vSphere chooses a suitable deep C-State (like C2) based on its prediction of when the CPU cores need to be active again.