Category Archives: HA

Example Architectural Decision – Host Isolation Response for IP Storage

Posted on by
Reply

Problem Statement

What are the most suitable HA / host isolation response when using IP based storage (In this case, Netapp HA Pair in 7-mode) when the IP storage runs over physically separate network cards and switches to ESXi management?

Assumptions

1. vSphere 5.0 or greater (To enable use of Datastore Heartbearting)
2. vFiler1 & vFiler2 reside on different physical Netapp Controllers (within the same HA Pair in 7-mode)
3. Virtual Machine guest operating systems with an I/O timeout of 190 seconds to allow for a Controller fail-over (Maximum 180 seconds)

Motivation

1. Minimize the chance of a false positive isolation response
2.Ensure in the event the storage is unavailable that virtual machines are promptly shutdown to minimize impact on the applications/data.

Architectural Decision

Turn off the default isolation address and configure the below specified isolation addresses, which check connectivity to multiple Netapp vFilers (IP storage) on the vFiler management VLAN and the IP storage interface.

Utilize Datastore heartbeating, checking multiple datastores hosted across both Netapp controllers (in HA Pair) to confirm the datastores themselves are accessible.

Services VLANs
das.isolationaddress1 : vFiler1 Mgmt Interface 192.168.1.10
das.isolationaddress2 : vFiler2 Mgmt Interface 192.168.2.10

IP Storage VLANs
das.isolationaddress3 : vFiler1 vIF 192.168.10.10
das.isolationaddress4 : vFiler2 vIF 192.168.20.10

Configure Datastore Heartbeating with “Select any of the clusters datastores taking into account my preference” and select the following datastores

  • One datastore from vFiler1 (Preference)
  • One datastore from vFiler2 (Preference)
  • A second datastore from vFiler1
  • A second datastore from vFiler2

Configure Host Isolation Response to: Power off.

Justification

1. The ESXi Management traffic is running on a standard vSwitch with 2 x 1GB connections which connect to different physical switches to the IP storage (and Data) traffic (which runs over 10GB connections). Using the ESXi management gateway (default isolation address) to deter main isolation is not suitable as the management network can be offline without impacting the IP storage or data networks. This situation could lead to false positives isolation responses.
2. The isolation addresses chosen test both data and IP storage connectivity over the converged 10Gb network
3. In the event the four isolation addresses (Netapp vFilers on the Services and IP storage interfaces) cannot be reached by ICMP, Datastore heartbeating will be used to confirm if the specified datastores (hosted on separate physical Netapp controllers) are accessible or not before any isolation action will be taken.
4. In the event the two storage controllers do not respond to ICMP on either the Services or IP storage interfaces, and both the specified datastores are inaccessible, it is likely there has been a catastrophic failure in the environment, either to the network, or the storage controllers themselves, in which case the safest option is to shutdown the VMs.
5. In the event the isolation response is triggered and the isolation does not impact all hosts within the cluster, the VM will be restarted by HA onto a surviving host.

Implications

1. In the event the host cannot reach any of the isolation addresses, and datastore heartbeating cannot access the specified datastores, virtual machines will be powered off.

Alternatives

1. Set Host isolation response to “Leave Powered On”
2. Do not use Datastore heartbeating
3. Use the default isolation address

For more details, refer to my post “VMware HA and IP Storage“

Example Architectural Decision – Virtual Machine swap file location

Posted on by
1

Problem Statement

When using shared storage where deduplication is utilized along with an array level snapshot based backup solution, what can be done to minimize the wasted capacity of snapping transient files in backups and the CPU overhead on the storage controller having to attempt to deduplicate data which cannot be deduped?

Assumptions

1. Virtual machine memory reservations cannot be used to reduce the vswap file size

Motivation

1. Reduce the snapshot size for backups without impacting the ability to backup and restore
2. Minimize the overhead on the storage controller for deduplication processing
3. Optimize the vSphere / Storage solution for maximum performance

Architectural Decision

1. Configure the HA swap file policy to store the swap file in a datastore specified by the host.
2. Create a new datastore per cluster which is hosted on Tier 1 storage and ensure deduplication is disabled on that volume
3. Configure all Host’s within a cluster to use the same specified datastore for vswap files
4. Ensure the new datastore is not part of any backup job

Justification

1. With minimal added complexity, backup jobs now exclude the VM swap file which reduces the backup size by the total amount of vRAM assigned to the VMs within the environment
2. As the vswap file is recreated at start up, loosing this file has no consequence
3. Decreasing Tier 1 storage requirements
4. The storage controller will not waste CPU cycles attempting to dedupe data which will not dedupe
5. Setting high percentages of memory reservation may impact the overcommitment in the environment where specifying a datastore for vswap reduces overhead without any significant downside

Implications

1. A datastore will need to be created for swapfiles
2. HA will need to be configured to store the swap file in a datastore specified by the host
3. The host (via Host Profiles) will need to be configured to use a specified datastore for vswap
4. vMotion performance will not be impacted where a VM is vMotion’d between two hosts that do not have a common vswap datastore as one datastore per cluster will be used for vswap files
5. The datastore will need to be sized to take into account the total vRAM assigned to VMs within the cluster

Alternatives

1. Set Virtual machine memory reservations of 100% to eliminate the vswap file
2. Store the swap file in the same directory as the virtual machine and accept the overhead on backups & dedupe
3. Use multiple datastores for vSwap across the cluster and accept the impact on vMotion

 

High Availability Admission Control Setting and Policy

Posted on by
1

High Availability Admission Control Setting and Policy

Problem Statement

In a self service IaaS cloud, the virtual machine numbers and compute requirements will likely vary significantly and without notice. The environment needs to achieve the maximum consolidation ratio possible without impacting the ability to provide redundancy at a minimum of

1. N+1 for clusters of up to 8 hosts
2. N+2 for clusters of >8 hosts but <=16
3. N+3 for clusters of >16 hosts but <=24
4. N+4 for clusters of >24 hosts but <=32

What is the most efficient HA admission control policy / setting and configuration for the vSphere cluster?

Assumptions

1. Virtual machine workloads will vary from small ie: 1 vCPU / 1GB RAM up to large VMs of >=8vCPU and >=64GB Ram
2. Redundancy is mandatory as per the problem statement
3. ESXi hosts can support the maximum VM size required by the offering
4. vSphere 5.0 or later is being used

Motivation

1. Ensure maximum consolidation ratios in the cluster
2. Ensure optimal compute resource utilization
3. Prevent HA overhead being increased by the potentially inefficient slot size based HA algorithms
4. Make maximum use of hardware investment

Alternatives

1. Use “Specify a fail over host”
2. Set “Host failure cluster tolerates” to 1, 2,3 or 4 depending on cluster size

Justification

1. Enabling admission control is critical to ensure the required level of availability
2. The admission control settings that rely on the Slot size based HA algorithms do not suit clusters with varying VM sizes
3. Percentage setting being rounded up adds minimal additional HA overhead and helps ensure performance in a HA event
4. Ensure maximum CPU scheduling efficiency by having all hosts within the cluster running virtual machines
5. Ensure optimal DRS flexibility by having all hosts within the cluster active to be able to run virtual machines

Architectural Decision

For the HA Admission control setting use “Enable – Do not power on virtual machines that violate availability constraints”

For the HA admission control policy use “Percentage of cluster resources reserved for HA” and set the percentage of cluster resources as per the below table.

Note: Percentage values that do not equate to a full number will be rounded up.

HAPercentages

Note: Check out this cool HA admission control percentage calculator by Samir Roshan of ThinkingLoudOnCloud

Implications

1. The Percentage of cluster resources reserved for HA uses VM level CPU and Memory reservations to calculate cluster capacity. If not reservations are set performance in the event of a failure may be impacted
2. The default Mhz reserved for HA is 32mhz per VM –  CPU Reservations should be considered for critical VMs to ensure performance is not significantly degraded in a HA event
3. The default memory reserved for HA is 0MB + VM overhead – Memory reservations should be considered for critical VMs to ensure performance is not significantly degraded in a HA event