Understanding Virtual Memory In Red Hat Enterprise Linux 4

1 Understanding Virtual Memory In Red HatEnterprise Linux 4 Neil HormanVersion - DRAFT EDITD ecember 13, 20051 Contents1 Introduction22 What Comprises a VM .. MMU .. Zoned Buddy Allocator .. Slab Allocator .. kernel Threads .. Components that use the VM .. 73 The Life of A Page84 Tuning the blockdump .. laptopmode .. legacyvalayout .. nrpdflushthreads .. dirtybackgroundratio .. overcommitmemory .. overcommitratio .. dirtyexpirecentisecs .. dirtywritebackcentisecs .. lowerzoneprotection .. dirtyratio .. maxmapcount .. page-cluster .. minfreekbytes .. swappiness .. hugetlbshmgroup .. nrhugepages .. vfscachepressure.

2 16 List of Figures1 High level overview of VM subsystem .. 32 Illustration of a Virtual to physical Memory translation .. 43 Diagram of the VM page state machine .. 911 IntroductionOne of the most important aspects of an operating system is the VirtualMemory Management system. Virtual Memory (VM) allows an operatingsystem to perform many of its advanced functions, such as process isolation,file caching, and swapping. As such, it is imperative that an administratorunderstand the functions and tunable parameters of an operating system svirtual Memory manager so that optimal performance for a given workloadmay be achieved. This article is intended to provide a system administratora general overview of how a VM works, specifically the VM implementedin Red Hat Enterprise Linux 4 (RHEL4).

3 After reading this document, thereader should have a rudimentary Understanding of the data the RHEL4VM controls and the algorithms it uses. Further, the reader should havea fairly good Understanding of general Linux VM tuning techniques. It isimportant to note that Linux as an operating system has a proud legacy ofoverhaul. Items which no longer serve useful purposes or which have betterimplementations as technology advances are phased out. This implies thatthe tuning parameters described in this article may be out of date if youare using a newer or older kernel . This is particularly true of this release ofRed Hat Enterprise Linux , as it is the first RHEL release making use of kernel series. Fear not however!

4 With a well grounded Understanding ofthe general mechanics of a VM, it is fairly easy to convert ones knowledgeof VM tuning to another VM. The same general principles apply, and docu-mentation for a given kernel (including its specific tunable parameters), canbe found in the corresponding kernel source tree under the file Documenta-tion/ DefinitionsTo properly understand how a Virtual Memory Manager does its job, it helpsto understand what components comprise a VM. While the low level detailsof a VM are overwhelming for most, a high level view is nonetheless helpfulin Understanding how a VM works, and how it can be optimized for variousworkloads. A high level overview of the components that make up a Virtualmemory manager is presented in Figure 1 What Comprises a VMFigure 1: High level overview of VM subsystem3 While a VM is actually far more complicated than illustrated in Figure1, the high level function of the system is accurate.

5 The following sectionsdescribe each of the listed components in the MMUThe Memory Management Unit (MMU) is the hardware base that make aVirtual Memory system possible. The MMU allows software to referencephysical Memory by aliased addresses, quite often more than one. It ac-complishes this through the use ofpagesandpage tables. The MMU usesa section of Memory to translate Virtual addresses into physical addressesvia a series of table lookups. Various processor architectures preform thisfunction is slightly different ways, but in general figure 2 illustrates how atranslation is preformed from a Virtual address to a physical address:Figure 2: Illustration of a Virtual to physical Memory translationEach table lookup provides a pointer to the base of the next table, as1 AUTHORS NOTE: The Overview figure may need updating to bring itinto line w/ kernel architecture4well as a set of extra bits which provide auxiliary data regarding that pageor set of pages.

6 This information typically includes the current page status,access privileges, and size. A separate portion of the Virtual address beingaccessed provides an index into each table in the lookup process. The finaltable provides a pointer to the start of the physical page corresponding tothe Virtual address in RAM, while the last field in the Virtual address selectsthe actual word in the page being accessed. Any one of the table lookupsduring this translation, may direct the lookup operation to terminate anddrive the operating system to preform another action. Some of these actionsare somewhat observable at a system level, and have common names orreferences Segmentation Violation- A user space process requests a Virtual ad-dress, and during the translation the kernel is interrupted and informedthat the requested translation has resulted in a page which it has notallocated, or which the process does not have permission to access.

7 Thekernel responds by signaling the process that it has attempted to accessan invalid Memory region, after which it is terminated. Swapped out- During a translation of an address from a user spaceprocess, the kernel was interrupted and informed that the page tableentry lists the page as accessible, but not present in RAM. The kernelinterprets this to mean that the requested address is in a page whichhas been swapped to disk. The user process requesting the address isput to sleep and an I/O operation is started to retrieve the Zoned Buddy AllocatorThe Zoned Buddy Allocator is responsible for the management of page allo-cations to the entire system. This code manages lists ofphysically contiguouspages and maps them into the MMU page tables, so as to provide other ker-nel subsystems with validphysicaladdress ranges when the kernel requeststhem (Physical to Virtual Address mapping is handled by a higher layer ofthe VM and is collapsed into the kernel subsystems block of Figure 1 ).

8 Thename Buddy Allocator is derived from the algorithm this subsystem uses tomaintain it free page lists. All physical pages in RAM are cataloged by thebuddy allocator and grouped into lists. Each list represents clusters of 2npages, where n is incremented in each list. There is a list of single pages,a list of 2 page clusters, a list of 4 page cluster, and so on. When a requestcomes in for an amount of Memory , that value is rounded up to the nearestpower of 2, and a entry is removed from the appropriate list, registered in thepage tables of the MMU and a corresponding physical address is returned to5the caller, which is then mapped into a Virtual address for kernel use. If noentries exist on the requested list, an entry from the next list up is brokeninto two separate clusters, and 1 is returned to the caller while the other isadded to the next list down.

9 When an allocation is returned to the buddyallocator, the reverse process happens. The allocation is returned to therequisite list, and the list is then examined to determine if a larger clustercan be made from the existing entries on the list which was just algorithm is advantageous in that itautomatically returns pages to thehighest order free list possible. That is to say, as allocations are returned tothe free pool, they automatically form larger clusters, so that when a needarises for a large amount of physically contiguous Memory ( for a DMAoperation), it is more likely that the request can be satisfied. Note that thebuddy allocator allocates Memory in page multiples only. Other subsystemsare responsible for finer grained control over allocation size.

10 For more in-formation regarding the finer details of a buddy allocator, refer to [1]. Notethat the Buddy allocator also manages memoryzones, which define pools ofmemory which have different purposes. Currently there are three memorypools which the buddy allocator manages accesses for: DMA- This zone consists of the first 16 MB of RAM, from whichlegacy devices allocate to perform direct Memory operations NORMAL- This zone encompasses Memory addresses from 16 MBto 1 GB2and is used by the kernel for internal data structures, as wellas other system and user space allocations. HIGHMEM- This zone includes all Memory above 1 GB and is usedexclusively for system allocations (file system buffers, user space allo-cations, etc).