Problem Statement
What is the most suitable hardware specifications for this environments ESXi hosts?
Requirements
1. Support Virtual Machines of up to 16 vCPUs and 256GB RAM
2. Achieve up to 400% CPU overcommitment
3. Achieve up to 150% RAM overcommitment
4. Ensure cluster performance is both consistent & maximized
5. Support IP based storage (NFS & iSCSI)
6. The average VM size is 1vCPU / 4GB RAM
7. Cluster must support approx 1000 average size Virtual machines day 1
8. The solution should be scalable beyond 1000 VMs (Future-Proofing)
9. N+2 redundancy
Assumptions
1. vSphere 5.0 or later
2. vSphere Enterprise Plus licensing (to support Network I/O Control)
3. VMs range from Business Critical Application (BCAs) to non critical servers
4. Software licensing for applications being hosted in the environment are based on per vCPU OR per host where DRS “Must” rules can be used to isolate VMs to licensed ESXi hosts
Constraints
1. None
Motivation
1. Create a Scalable solution
2. Ensure high performance
3. Minimize HA overhead
4. Maximize flexibility
Architectural Decision
Use Two Socket Servers w/ >= 8 cores per socket with HT support (16 physical cores / 32 logical cores) , 256GB Ram , 2 x 10GB NICs
Justification
1. Two socket 8 core (or greater) CPUs with Hyper threading will provide flexibility for CPU scheduling of large numbers of diverse (vCPU sized) VMs to minimize CPU Ready (contention)
2. Using Two Socket servers of the proposed specification will support the required 1000 average sized VMs with 18 hosts with 11% reserved for HA to meet the required N+2 redundancy.
3. A cluster size of 18 hosts will deliver excellent cluster (DRS) efficiency / flexibility with minimal overhead for HA (Only 11%) thus ensuring cluster performance is both consistent & maximized.
4. The cluster can be expanded with up to 14 more hosts (to the 32 host cluster limit) in the event the average VM size is greater than anticipated or the customer experiences growth
5. Having 2 x 10GB connections should comfortably support the IP Storage / vMotion / FT and network data with minimal possibility of contention. In the event of contention Network I/O Control will be configured to minimize any impact (see Example VMware vNetworking Design w/ 2 x 10GB NICs)
6. RAM is one of the most common bottlenecks in a virtual environment, with 16 physical cores and 256GB RAM this equates to 16GB of RAM per physical core. For the average sized VM (1vCPU / 4GB RAM) this meets the CPU overcommitment target (up to 400%) with no RAM overcommitment to minimize the chance of RAM becoming the bottleneck
7. In the event of a host failure, the number of Virtual machines impacted will be up to 64 (based on the assumed average size VM) which is minimal when compared to a Four Socket ESXi host which would see 128 VMs impacted by a single host outage
8. If using Four socket ESXi hosts the cluster size would be approx 10 hosts and would require 20% of cluster resources would have to be reserved for HA to meet the N+2 redundancy requirement. This cluster size is less efficient from a DRS perspective and the HA overhead would equate to higher CapEx and as a result lower the ROI
9. The solution supports Virtual machines of up to 16 vCPUs and 256GB RAM although this size VM would be discouraged in favour of a scale out approach (where possible)
10. The cluster aligns with a virtualization friendly “Scale out” methodology
11. Using smaller hosts (either single socket, or less cores per socket) would not meet the requirement to support supports Virtual machines of up to 16 vCPUs and 256GB RAM , would likely require multiple clusters and require additional 10GB and 1GB cabling as compared to the Two Socket configuration
12. The two socket configuration allows the cluster to be scaled (expanded) at a very granular level (if required) to reduce CapEx expenditure and minimize waste/unused cluster capacity by adding larger hosts
13. Enabling features such as Distributed Power Management (DPM) are more attractive and lower risk for larger clusters and may result in lower environmental costs (ie: Power / Cooling)
Alternatives
1. Use Four Socket Servers w/ >= 8 cores per socket , 512GB Ram , 4 x 10GB NICs
2. Use Single Socket Servers w/ >= 8 cores , 128GB Ram , 2 x 10GB NICs
3. Use Two Socket Servers w/ >= 8 cores , 512GB Ram , 2 x 10GB NICs
4. Use Two Socket Servers w/ >= 8 cores , 384GB Ram , 2 x 10GB NICs
5. Have two clusters of 9 hosts with the recommended hardware specifications
Implications
1. Additional IP addresses for ESXi Management, vMotion, FT & Out of band management will be required as compared to a solution using larger hosts
2. Additional out of band management cabling will be required as compared to a solution using larger hosts
Related Articles
1. Example Architectural Decision – Network I/O Control for ESXi Host using IP Storage (4 x 10 GB NICs)
2. Example VMware vNetworking Design w/ 2 x 10GB NICs
3. Network I/O Control Shares/Limits for ESXi Host using IP Storage
4. VMware Clusters – Scale up for Scale out?
5. Jumbo Frames for IP Storage (Do not use Jumbo Frames)
6. Jumbo Frames for IP Storage (Use Jumbo Frames)
Thank You….. This is very good information, especially for the beginners.
Hi Josh,
As a matter of interest, when calculating overcommitment of CPU or RAM, where do you include the calculations? In the target host specifications or when determining total compute requirements for VMs?
Andrew
Hi Andrew
Sorry for the (very) late reply.
I have a spreadsheet which calculates overcommitment by taking the compute resources per host, multiplying that by the number of hosts in the proposed cluster, minus the percentage reserved for HA, then using capacity planner data , add up all the vCPU/RAM and calculates overcommitment.
This method is pretty conservative from a CPU perspective, as HA doesn’t reserve CPU cores for HA so the cluster can leverage the cores for scheduling purposes. For RAM, I think it works well since as you well know RAM doesn’t overcommit anywhere near as well as CPU.
Once I have the overcommitment numbers, I target
<250% overcommitment (CPU) for large VMs or Business Critical Apps
<=400% overcommitment (CPU) for general mixed servers workloads (where most workloads are <4 vCPUs) <=125% overcommitment (RAM) for large VMs or Business Critical Apps <=150% overcommitment (RAM) for general mixed servers workloads (where most workloads are <=8GB RAM and suitable for TPS) Hope that makes sense. - Hope to catch up with you at VMworld so you can tell me about your vChampion Lunch trip 🙂
Josh, thanks for that example!
1) If we use two socket 8 cores servers, and there are some “monster” VMs up to 16 cores, 256Gb RAM each (as per requirement #1) –
that means there will be ESX hosts where ALL physycal cores are occupied by single monster VM, leaving nothing to hypervisor. The system resources won’t be fairly scheduled, as host will need to wait until all 16 cores are available. I would expect a high %READY time on that machine.
There is a potential in HyperThreading to relieve the situation, but I wouldn’t count on that in production design.
2) Same story applies to memory. If there are monster VMs with up to 256GB of memory – that leaves nothing to hypervisor and VM overhead. That will result in memory shortage, which may or may not be relieved by ballooning and kernel swapping. I wouldn’t count on that in production design.
3) Most of two-socket systems are limited in number of memory slots, 16 on average. You’lll need 16Gb modules to achieve 256 per server. 16Gb modules are more expensive per GB than 8GB, there is potential of high costs and long ROI. In addition to that, all slots will be occupied – no room to grow.
If I couldn’t influence monster VM sizing (worst case scenario), I would use 10 four socket 8 cores servers, 336Gb each.
Because it would give enough room for worst case hypervisor CPU and memory overhead, and provide some room for memory to grow in a future. That is 9% more expensive in terms of HA overhead, but if cost is the concern – it would be slashed by 42 8Gb memory modules instead of 16Gb
What do you think?