Tag Archives: host failures cluster tolerates

Example Architectural Decision – VMware HA – Percentage of Cluster resources reserved for HA

Posted on by
8

Problem Statement

The decision has been made to use “Percentage of cluster resources reserved for HA” admission control setting, and use Strict admission control to ensure the N+1 minimum redundancy level is maintained. However, as most virtual machines do not use  “Reservations” for CPU and/or Memory, the default reservation is only 32Mhz and 0MB+overhead for a virtual machine. In the event of a failure, this level of resources is unlikely to provide sufficient compute to operate production workloads. How can the environment be configured to ensure a minimum level of performance is guaranteed in the event of one or more host failures?

Requirements

1. All Clusters have a minimum requirement of N+1 redundancy
2. In the event of a host failure, a minimum level of performance must be guaranteed

Assumptions

1. vSphere 5.0 or later (Note: This is Significant as default reservation dropped from 256Mhz to 32Mhz, RAM remained at 0MB + overhead)

2. Percentage of Cluster resources reserved for HA is used and set to a value as per Example Architectural Decision – High Availability Admission Control

3. Strict admission control is enabled

4. Target over commitment Ratios are <=4:1 vCPU / Physical Cores | <=1.5 : 1 vRAM / Physical RAM

5. Physical CPU Core speed is >=2.0Ghz

6. Virtual machines sizes in the cluster will vary

7. A limited number of mission critical virtual machines may be set with reservations

8. Average VM size uses >2GB RAM

9. Clusters compute resources will be utilized at >=50%

Constraints

1. Ensuring all compute requirements are provided to Virtual machines during BAU

Motivation

1. Meet/Exceed availability requirements
2. Minimize complexity
3. Ensure the target availability and performance is maintained without significantly compromising  over commitment ratios

Architectural Decision

Ensure all clusters remain configured with the HA admission control setting use
“Enable – Do not power on virtual machines that violate availability constraints”

and

Use “Percentage of Cluster resources reserved for HA” for the admission control policy with the percentage value based on the following Architectural Decision – High Availability Admission Control

Configure the following HA Advanced Settings

1. “das.vmMemoryMinMB” with a value of “1024″
2. “das.vmCpuMinMHz” with a value of “512”

Justification

1. Enabling admission control is critical to ensure the required level of availability.
2. The “Percentage of cluster resources reserved for HA” setting allows a suitable percentage value of cluster resources to reserved depending on the size of each cluster to maintain N+1
3.The potentially inefficient slot size calculation used with “Host Failures cluster tolerates” does not suit clusters where virtual machines sizes vary and/or where some mission Critical VMs require reservations

  • 4.
  • Using advanced settings “das.vmCpuMinMHz” & “das.vmMemoryMinMB” allows a minimum level of performance (per VM) to be guaranteed in the event of one or more host failures
  • 5.
  • Advanced settings have been configured to ensure the target over commit ratios are still achieved while ensuring a minimum level of resources in a the event of a host failure
  • 6.
  • Maintains an acceptable minimum level of performance in the event of a host failure without requiring the administrative overhead of setting and maintaining “reservations” at the Virtual machine level
  • 7.
  • Where no reservations are used, and advanced settings not configured, the default reservation would be 32Mhz and 0MB+ memory overhead is used. This would likely result in degraded performance in the event a host failure occurs.

Alternatives

1. Use “Specify a fail over host” and have one or more hosts specified
2. “Host Failures cluster tolerates” and set it to appropriate value depending on hosts per cluster without using advanced settings
3.Use higher Percentage values
4. Use Higher / Lower values for “das.vmMemoryMinMB” and “das.vmCpuMinMHz”
5. Set Virtual machine level reservations on all VMs

Implications

1. The “das.vmCpuMinMHz” advanced setting applies on a per VM basis, not a per vCPU basis, so VMs with multiple vCPUs will still only be guarenteed 512Mhz in a HA event

2. This will reduce the number of virtual machines that can be powered on within the cluster (in order to enforce the HA requirements)

CloudXClogo

 

 

Common Mistake: Inefficient cluster sizes

Link

Reply

In my day job, I regularly come across environments which are running poorly and have inefficient designs.

One of the most common issues I see is VMware environments which cannot power on VMs due to being out of compute resources, but not for the reasons you may expect.

While the environments may have less than optimal HA settings / policies, the most common issues I see is customers (for whatever reason) having multiple clusters with only a few nodes. (ie: 2/3/4 etc)

Some of the time, there are corporate policies which may require this type of setup, but alot of the time, you can comply with these policies while still optimizing the environment.

It seems that even with virtualisation having been common place for many years, the basics are still mis-understood by a significant percentage of industry professionals. I have heard comments event recently saying you need 2 node clusters for maximum HA efficiency, They couldn’t be more Wrong!

So, why are small clusters a potential problem?

Depending on what HA setting you choose (Host failures cluster tolerates , Percentage of cluster resources reserved for HA, or Failover Host/s), the clusters have a large amount of “waste”.

What is “Waste”?

“Waste”, is the amount of the compute power within the cluster, that cannot be used to ensure in a HA event, VMs can be restarted on the remaining hosts.

Now at this stage, let me point out, some “Waste” is a good thing. We need to have some spare capacity for HA events, but the challenge is to minimize the waste without compromising HA.

So, in a recent environment I reviewed, there was 4 clusters using similar IBM x3850 Servers.

Cluster 1 : 2 Nodes

Cluster 2 : 2 Nodes

Cluster 3: 3 Nodes

Cluster 4 : 2 Nodes

In all clusters, HA was enabled (as it should be) and the HA admission control setting was “Percentage of Cluster resources reserved for HA” (which I prefer).

The 2 node clusters HA reservation percentage was set to 50%, and the 3 node cluster was 33%, which would be the settings I would choose if I had to stick with the 4 cluster design.

Because the environment (in its current state) was unable to host any more VMs, the customer wanted to purchase another 2 new Hosts, and form a new cluster.

At this stage we have the equivalent of 4 hosts of “waste” within the environment, and with a new cluster we would have 5 hosts “wasted”.

Now after a quick check of the VMware EVC KB: 1003212 all CPUs are compatible with EVC and support the EVC mode “Intel® “Merom” Generation”.

So, we can form a single new cluster using the existing 9 hosts and maintain full cluster functionality by enabling EVC.

Lets assume the hosts are all in a cluster and we’re configuring HA, How do we ensure we have more available compute for the new virtual machines?

Simple, we Enable HA (as you always should), Enable admission control, and set the HA policy to “Percentage of Cluster resources reserved for HA, But what percentage should we choose?

Well, it depends of what level of redundancy you require.

Generally, I recommend for

<8 hosts = N+1 – Note: If you require N+1 during maintenance you need N+2

>8 hosts < 16 hosts = N+2

>16 hosts <24 hosts = N+3

>24 hosts = N+4

The reason for the above, is as you add more hosts, your chance of a host failure, and a subsequent host failure increases. Therefore the more hosts you have, the more redundancy you need, Similar concept to RAID.

So in this example, we’re right on the line in terms of N+1 or N+2.

Lets be conservative, and choose N+2, therefore setting “Percentage of Cluster resources reserved for HA” to 22% (N+2 is actually 22.5%, but we use round numbers).

So what have we achieved?

The previous setup had only N+1 and an average HA overhead of 45.75% (50%+50%+50%+33% divide 4).

The new 9 node cluster now with N+2 redundancy and only has an overhead of 22%. A NET gain of 23.75% of available compute resources without purchasing new hardware.

What else do we gain by having a single larger cluster:

1. Increased DRS flexibility

2. Increase redundancy (previously N+1, now N+2)

3. Less chance of contention

4. No need to purchase new hardware!!

The above is a simple example of how to increase efficiency within a VMware environment without purchasing new hardware.

Now for those of you wanting to know more about HA/DRS, this has been covered in great detail in other blogs, I would recommend you first have a read of the following blog and get a copy of “vSphere 5.0 Clustering technical deep dive” book.

Yellow Bricks (Duncan Epping) – HA Admission control Pros and Cons