VMware ESX 4.0, Management Manual

Page 1: ...r Server 4 0 This document supports the version of each product listed and supports all subsequent versions until the document is replaced by a new edition To check for more recent editions of this do...

Page 2: ...are com Copyright 2006 2011 VMware Inc All rights reserved This product is protected by U S and international copyright and intellectual property laws VMware products are covered by one or more patent...

Page 3: ...ce Pools 38 Create Resource Pools 39 Add Virtual Machines to a Resource Pool 40 Removing Virtual Machines from a Resource Pool 41 Resource Pool Admission Control 41 5 Creating a DRS Cluster 45 Admissi...

Page 4: ...5 Resource Management in NUMA Architectures 76 Specifying NUMA Controls 77 A Performance Monitoring Utilities resxtop and esxtop 81 Using the esxtop Utility 81 Using the resxtop Utility 81 Using esxto...

Page 5: ...es the update history of the vSphere Resource Management Guide Revision Description EN 000107 02 Included a point in Multicore Processors on page 19 section EN 000107 01 Removed references to CPU Mach...

Page 6: ...vSphere Resource Management Guide 6 VMware Inc...

Page 7: ...Sphere Resource Management Guide covers ESX ESXi and vCenter Server Intended Audience This manual is for system administrators who want to understand how the system manages resources and how they can...

Page 8: ...phone_support html Support Offerings To find out how VMware support offerings can help meet your business needs go to http www vmware com support services VMware Professional Services VMware Educatio...

Page 9: ...ty vary over time Resource management allows you to dynamically reallocate resources so that you can more efficiently use available capacity Resource Types Resources include CPU memory power storage a...

Page 10: ...e user Goals of Resource Management When managing your resources you should be aware of what your goals are In addition to resolving resource overcommitment resource management can help you accomplish...

Page 11: ...ine memory Low 500 shares per virtual CPU 5 shares per megabyte of configured virtual machine memory For example an SMP virtual machine with two virtual CPUs and 1GB RAM with CPU and memory shares set...

Page 12: ...s Select resource allocation settings shares reservation and limit that are appropriate for your ESX ESXi environment The following guidelines can help you achieve better performance for your virtual...

Page 13: ...OK 6 Right click the marketing virtual machine VM Marketing and select Edit Settings 7 In the CPU panel change the value in the Reservation field to the desired number 8 Click OK If you select the cl...

Page 14: ...o displays a chart showing the resource pools and virtual machines in the DRS cluster with the following CPU or memory usage information To view CPU or memory information click the CPU button or Memor...

Page 15: ...s allocated to the virtual machine based on user configured resource allocation policies for example reservation shares and limit and with the assumption that all virtual machines in the cluster consu...

Page 16: ...have not yet been reserved Based on the available unreserved resources the system determines whether it can guarantee the reservation for which the virtual machine is configured if any This process is...

Page 17: ...same results Emulation provides portability and runs software designed for one platform across several platforms When CPU resources are overcommitted the ESX ESXi host time slices the physical process...

Page 18: ...ualization adds varying amounts of overhead depending on the workload and the type of virtualization used An application is CPU bound if it spends most of its time executing instructions rather than w...

Page 19: ...of virtual machines Intel and AMD have each developed processors which combine two or more processor cores into a single integrated circuit often called a package or socket VMware uses the term socke...

Page 20: ...technology allows a single physical processor core to behave like two logical processors The processor can run two independent applications at the same time To avoid confusion between logical and phys...

Page 21: ...possible to meet the resource demands of these virtual machines Ensure that any custom affinity settings make sense for a hyperthreaded system Enable Hyperthreading To enable hyperthreading you must f...

Page 22: ...help in case of unusual workloads that interact badly with hyperthreading For example an application with cache thrashing problems might slow down an application sharing its physical core You can pla...

Page 23: ...virtual CPUs for example a uniprocessor virtual machine with affinity to at least two CPUs or a two way SMP virtual machine with affinity to at least three CPUs NOTE CPU affinity specifies virtual ma...

Page 24: ...SpeedStep and Enhanced AMD PowerNow CPU power management technologies For the VMkernel to take advantage of the power management capabilities provided by these technologies you might need to first ena...

Page 25: ...kernel dedicates part of this managed machine memory for its own use The rest is available for use by virtual machines Virtual machines use machine memory for two purposes each virtual machine require...

Page 26: ...can allocate to the virtual machine The virtual machine s memory allocation is also implicitly limited by its configured size Overhead memory includes space reserved for the virtual machine frame buff...

Page 27: ...ddress translation overhead after the shadow page tables are set up Because the translation look aside buffer TLB on the processor caches direct virtual to machine mappings read from the shadow page t...

Page 28: ...software memory virtualization In particular hardware assistance eliminates the overhead required to keep shadow page tables in synchronization with guest page tables However the TLB miss latency when...

Page 29: ...ciently ESX ESXi sets this level at approximately six percent of the memory available for running virtual machines An ESXi host uses additional system memory for management agents that run in the serv...

Page 30: ...y MB 1 VCPU 2 VCPUs 3 VCPUs 4 VCPUs 5 VCPUs 6 VCPUs 7 VCPUs 8 VCPUs 256 113 17 159 43 200 53 241 62 293 15 334 27 375 38 416 50 512 116 68 164 96 206 07 247 17 302 75 343 88 385 02 426 15 1024 123 73...

Page 31: ...memory tax rate with the Mem IdleTax option Use this option together with the Mem SamplePeriod advanced attribute to control how the system determines target memory allocations for virtual machines S...

Page 32: ...memory is overcommitted and delete a swap file ESX ESXi hosts use swapping to forcibly reclaim memory from a virtual machine when the vmmemctl driver is not available or is not responsive n It was nev...

Page 33: ...e cluster s hosts in the vSphere Client inventory panel and click the Configuration tab 5 Select Virtual Machine Swapfile Location 6 Click the Swapfile Datastore tab 7 From the list provided select th...

Page 34: ...that were using swap files those swap files continue to exist and take up disk space even after the ESX ESXi host restarts These swap files can consume many gigabytes of disk space so ensure that you...

Page 35: ...n ESX ESXi host differ Consider the example in the following figure Two virtual machines are running on an ESX ESXi host Each block represents 4 KB of memory and each color letter represents a differe...

Page 36: ...red by virtual machines To determine this look at the machine memory and count the number of blocks that have more than one arrow pointing at them There are three such blocks so Memory Shared Common i...

Page 37: ...ces The resource pools at a higher level are called parent resource pools Resource pools and virtual machines that are at the same level are called siblings The cluster itself represents the root reso...

Page 38: ...dependently of the actual hosts that contribute to the resources If you replace three 2GB hosts with two 3GB hosts you do not need to make changes to your resource allocations This separation allows a...

Page 39: ...allocation settings to manage a resource pool Table 4 1 is a summary of the attributes that you can specify for a resource pool Table 4 1 Resource Pool Attributes Field Description Name Name of the n...

Page 40: ...spective resource pools Change Resource Pool Attributes After a resource pool is created you can change its attributes Procedure 1 Select the resource pool in the vSphere Client inventory panel 2 In t...

Page 41: ...rag the virtual machine to another resource pool You do not need to power off a virtual machine if you only move it When you remove a virtual machine from a resource pool the total number of shares as...

Page 42: ...on virtual machines with reservations but does not know how much each user will need to reserve Making the reservations for S1 and S2 expandable allows the administrator to more flexibly share and in...

Page 43: ...available locally n When you try to power on VM K2 RP KID has no unreserved CPU capacity so it checks its parent RP MOM has only 1GHz of unreserved capacity available 5GHz of RP MOM are already in us...

Page 44: ...vSphere Resource Management Guide 44 VMware Inc...

Page 45: ...ou first power on a virtual machine in the cluster DRS attempts to maintain proper load balancing by either placing the virtual machine on an appropriate host or making a recommendation See Admission...

Page 46: ...mendations provided have multiple lines showing each of the prerequisite actions The user can either accept this entire recommendation or cancel powering on the virtual machine Group Power On You can...

Page 47: ...see Figure 5 1 The three hosts on the left side of this figure are unbalanced Assume that Host 1 Host 2 and Host 3 have identical capacity and all virtual machines have the same configuration and loa...

Page 48: ...ion is computed using the load imbalance metric of the cluster This metric is displayed as Current host load standard deviation in the cluster s Summary tab in the vSphere Client A higher load imbalan...

Page 49: ...onfiguration options Processor Compatibility To avoid limiting the capabilities of DRS you should maximize the processor compatibility of source and destination hosts in the cluster VMotion transfers...

Page 50: ...rt raw disks or migration of applications clustered using Microsoft Cluster Service MSCS n VMotion requires a private Gigabit Ethernet migration network between all of the VMotion enabled managed host...

Page 51: ...elect the Enable individual virtual machine automation levels check box 5 Select an individual virtual machine or select multiple virtual machines 6 Right click and select an automation mode 7 Click O...

Page 52: ...vSphere Resource Management Guide 52 VMware Inc...

Page 53: ...RS affinity and anti affinity rules An affinity rule specifies that two or more virtual machines be placed on the same host An anti affinity rule is limited to two virtual machines and it requires tha...

Page 54: ...e cluster s Settings dialog box appears 3 In the left pane under VMware DRS select Rules 4 Select a rule in the right pane and click Edit 5 Make changes in the dialog box and click OK Disable DRS Rule...

Page 55: ...virtual machines in the cluster s root resource pool vCenter Server removes all existing resource pools of the host and the virtual machines in the host s hierarchy are all attached to the root Becau...

Page 56: ...g directly to a host within a cluster is not allowed because the resource pool controls the resources Remove Hosts from a Cluster You can remove hosts from a cluster Prerequisites Before you remove a...

Page 57: ...does not recommend or perform in fully automated mode any virtual machine migrations off of a host entering maintenance or standby mode if the VMware HA failover level would be violated after the hos...

Page 58: ...yellow because cluster requirements are no longer met When considering cluster validity scenarios you should understand these terms Reservation A fixed guaranteed allocation for the resource pool inp...

Page 59: ...shown in the Reserved Capacity field for the cluster n RP1 was created with a reservation of 4GHz Two virtual machines VM1 and VM7 of 2GHz each are powered on Reservation Used 4GHz No resources are le...

Page 60: ...ld not power on the additional virtual machine n RP2 was created with a reservation of 5GHz Two virtual machines of 1GHz and 2GHz are powered on Reservation Used 3GHz 2GHz remains unreserved RP3 was c...

Page 61: ...nning n The cluster is now running virtual machines that require a total of 6GHz The cluster still has 8GHz available which is sufficient to meet virtual machine requirements The resource pool reserva...

Page 62: ...on 6G Reservation Used 2G Unreserved 4G 0G VM1 1G VM7 3G VM2 3G VM3 1G VM4 1G VM5 1G VM6 1G Managing Power Resources The VMware Distributed Power Management DPM feature allows a DRS cluster to reduce...

Page 63: ...remote power on commands If you plan to use IPMI or iLO as a wake protocol you must configure the BMC BMC configuration steps vary according to model See your vendor s documentation for more informati...

Page 64: ...gged into should be set to auto negotiate the link speed and not set to a fixed speed for example 1000 Mb s Many NICs support WOL only if they can switch to 100 Mb s or less when the host is powered o...

Page 65: ...ory while a priority five recommendation brings only slight improvement The threshold is configured under Power Management in the cluster s Settings dialog box Each level you move the VMware DPM Thres...

Page 66: ...ministrator or to send notification using an SNMP trap By default this alarm is cleared after vCenter Server is able to successfully connect to that host To monitor VMware DPM activity you can also cr...

Page 67: ...the fields that appear in those sections Cluster Summary Tab General Section The General section of the cluster s Summary tab provides general information about your cluster Table 7 1 General Section...

Page 68: ...chart that provides CPU and memory utilization information VMware DRS Resource Distribution Chart The VMware DRS Resource Distribution chart displays CPU and memory utilization information Open this c...

Page 69: ...l or Automatic Migration Threshold Priority level or higher of DRS recommendations to apply Power Management Threshold Priority level or higher of VMware DPM recommendations to apply Additionally the...

Page 70: ...umption n Power off a specific host n Increase cluster capacity n Balance CPU or memory reservations n Maintain unreserved capacity DRS recommendations are configurable only using vCenter Server Migra...

Page 71: ...Details of the action taken Time Timestamp of when the action occurred By default the information on this page is maintained for four hours and it is preserved across sessions you can log out and when...

Page 72: ...vSphere Resource Management Guide 72 VMware Inc...

Page 73: ...kload such as a scientific computing application can be constrained by memory bandwidth This problem is amplified on symmetric multiprocessing SMP systems where many processors must compete for bandwi...

Page 74: ...ophisticated NUMA scheduler to dynamically balance processor load and memory locality or processor load balance 1 Each virtual machine managed by the NUMA scheduler is assigned a home node A home node...

Page 75: ...llows one of the remaining virtual machines to run remotely on the second node it suffers a serious performance penalty because all its memory remains on its original node Dynamic Load Balancing and P...

Page 76: ...cal machines As a result higher levels of overcommitment can be supported efficiently Transparent page sharing for ESX ESXi systems has also been optimized for use on NUMA systems On NUMA systems page...

Page 77: ...rtual machines having two virtual processors cannot reside within a NUMA node that has a single core such as the single core Opteron processors This also means they cannot be managed by the ESX ESXi N...

Page 78: ...nity for different NUMA nodes NOTE You must manually select the boxes for all processors in the NUMA node CPU affinity is specified on a per processor not on a per node basis Associate Memory Allocati...

Page 79: ...gs 2 Select Options and click Advanced 3 Click the Configuration Parameters button 4 In the vSphere Client set memory affinity for the NUMA node to 1 Completing these two tasks ensures that the virtua...

Page 80: ...vSphere Resource Management Guide 80 VMware Inc...

Page 81: ...esxtop4rc This configuration file consists of eight lines The first seven lines contain lowercase and uppercase letters to specify which fields appear in which order on the CPU memory storage adapter...

Page 82: ...line and the command defaults to localhost Using esxtop or resxtop in Interactive Mode By default resxtop and esxtop run in interactive mode Interactive mode displays statistics in different panels A...

Page 83: ...tatistics Columns and Order Pages You can define the order of fields displayed in interactive mode If you press f F o or O the system displays a page that specifies the field order on the top line and...

Page 84: ...ce pool and virtual machine CPU utilization Resource pools running virtual machines or other worlds are at times called groups For worlds belonging to a virtual machine statistics for the running virt...

Page 85: ...entage of time the resource pool virtual machine or world was idle IDLE Percentage of time the resource pool virtual machine or world was idle Subtract this percentage from WAIT to see the percentage...

Page 86: ...gles whether CPU statistics are displayed expanded or unexpanded The expanded display includes CPU resource utilization statistics broken down by individual worlds belonging to a resource pool or virt...

Page 87: ...ard and low High means that the machine memory is not under any pressure and low means that it is COSMEM MB Displays the memory statistics as reported by the ESX service console All numbers are in meg...

Page 88: ...local MEMSZ MB Amount of physical memory allocated to a resource pool or virtual machine GRANT MB Amount of guest physical memory mapped to a resource pool or virtual machine The consumed host machine...

Page 89: ...atistic is applicable on NUMA systems only Table A 7 displays the interactive commands you can use with the memory panel Table A 7 Memory Panel Interactive Commands Command Description M Sort resource...

Page 90: ...mmand in milliseconds KAVG cmd Average ESX ESXi VMkernel latency per command in milliseconds GAVG cmd Average virtual machine operating system latency per command in milliseconds DAVG rd Average devic...

Page 91: ...can be expanded r Sorts by READS s column w Sorts by WRITES s column R Sorts by MBREAD s read column T Sorts by MBWRTN s written column N Sorts first by ADAPTR column then by CID column within each AD...

Page 92: ...latency per command in milliseconds GAVG cmd Average guest operating system latency per command in milliseconds QAVG cmd Average queue latency per command in milliseconds DAVG rd Average device read...

Page 93: ...er device basis Table A 12 Virtual Machine Storage Panel Statistics Column Description ID Resource pool ID of the running world s resource pool or the world ID of the running world GID Resource pool I...

Page 94: ...d latency per read operation in milliseconds QAVG rd Average queue read latency per read operation in milliseconds DAVG wr Average device write latency per write operation in milliseconds KAVG wr Aver...

Page 95: ...ans the corresponding link is up N means it is not SPEED Link speed in Megabits per second FDUPLX Y means the corresponding link is operating at full duplex N means it is not USED BY Virtual network d...

Page 96: ...e you must first prepare for batch mode Procedure 1 Run resxtop or esxtop in interactive mode 2 In each of the panels select the columns you want 3 Save this configuration to a file by default esxtop4...

Page 97: ...rver The default port is 443 and unless this is changed on the server this option is not needed resxtop only username The user name to be authenticated when connecting to the remote host You are promp...

Page 98: ...The configuration file can be the default esxtop4rc configuration file or a user defined configuration file b Runs esxtop in Batch mode c filename Load a user defined configuration file If the c optio...

Page 99: ...e optimum result Procedure 1 In the vSphere Client inventory panel select the host to customize 2 Click the Configuration tab 3 In the Software menu click Advanced Settings 4 In the Advanced Settings...

Page 100: ...which memory activity is monitored to estimate working set sizes 60 Mem BalancePeriod Specifies the periodic time interval in seconds for automatic memory reallocations Significant changes in the amo...

Page 101: ...running virtual machines More frequent rebalancing can also improve fairness 2000 Numa RebalanceCoresTotal Specifies the minimum number of total processor cores on the host required to enable the NUM...

Page 102: ...ne This boolean value defaults to True If you set it to False for a virtual machine this turns off memory sharing sched swap persist Specifies whether the virtual machine s swap files should persist o...

Page 103: ...plications 18 custom automation mode 51 D DPM and admission control 16 automation level 65 enabling 64 Last Time Exited Standby 65 monitoring 66 overrides 65 threshold 65 DRS disabling 51 fully automa...

Page 104: ...er on 46 H home nodes NUMA 75 host local swap DRS cluster 33 standalone host 33 hosts adding to DRS clusters 55 advanced attributes 99 as resource providers 9 entering maintenance mode 57 removing fro...

Page 105: ...MigEnable 101 Numa RebalanceCoresNode 101 Numa RebalanceCoresTotal 101 Numa RebalanceEnable 101 Numa RebalancePeriod 101 O Opteron 76 overcommitted DRS clusters 60 overhead memory 25 P page migration...

Page 106: ...em Resource Allocation Table SRAT 74 T technical support 7 U updated information 5 V valid DRS clusters 58 vCenter Server events 66 virtual machine affinity 53 virtual machine anti affinity 53 Virtual...