Nutanix | Scalability, Resiliency & Performance | Index

Posted on June 13, 2018 by Josh Odgers

In my opinion there are three key areas for a solution such as the Nutanix platform to deliver consistent successful business outcomes.

They are of course Scalability, Resiliency and Performance.

If a product is strong in just one of these areas and weak in the other two, put simply it’s very unlikely to be a minimally viable product.

Even two out of three isn’t enough, you need a product which delivers all three to a high standard if you want to have a successful business outcome.

For example, if you have a product which performs well, but doesn’t scale then a customer who has the potential of growth should not consider that product. If a product is very scalable but isn’t resilient is unlikely to be suitable for any production workloads and if a solution is resilient but not scalable, then again we have major constraints which make it unattractive in most if not all scenarios.

I’ve previously written about “Things to consider when choosing infrastructure” where I discussed avoiding silos and by combining workloads on a modern platform like Nutanix, the environment can be more performant, resilient and obviously scalable.

When you have a product or product/s which have all three of these qualities, you are very likely to be able to deliver a great business outcome.

I’ve previously compared “peak” performance with “real world” performance where I concluded among other things that “Peak performance is rarely a significant factor for a storage solution” and even though it’s very common when doing proof of concepts (POCs), “Do not waste time performing absurd testing of “Peak performance”.

The purpose of this series is to demonstrate how the Nutanix platform performs in all three areas and to show examples to help prospective customers make informed decisions.

These three areas are my primary focus day to day in my role as Principal Architect at Nutanix and I am working to ensure the core of the Nutanix platform continues to improve equally in all areas.

Readers of this series will notice that many of the posts will refer to back to each other and in many cases cover similar areas. This is deliberate as Scalability, Resiliency and Performance are tied together and essential for an enterprise grade solution.

This series will continue to be updated as new capabilities, enhancements etc are released.

I hope this series is informative and helps existing/prospective customers as well as industry analysts learn more about the evolving Nutanix platform.

Scalability

Part 1 – Storage Capacity
Part 2 – Compute (CPU/RAM)
Part 3 – Storage Performance for a single Virtual Machine
Part 4 – Storage Performance for Monster VMs with AHV!
Part 5 – Scaling Storage Performance for Physical Machines

More coming soon.

Resiliency

Index:
Part 1 – Node failure rebuild performance
Part 2 – Converting from RF2 to RF3
Part 3 – Node failure rebuild performance with RF3
Part 4 – Converting RF3 to Erasure Coding (EC-X)
Part 5 – Read I/O during CVM maintenance or failures
Part 6 – Write I/O during CVM maintenance or failures
Part 7 – Read & Write I/O during Hypervisor upgrades
Part 8 – Node failure rebuild performance with RF3 & Erasure Coding (EC-X)
Part 9 – Self healing
Part 10: Nutanix Resiliency – Part 10 – Disk Scrubbing / Checksums

Bonus: Sizing Nutanix for Resiliency with Design Brews

More coming soon.

Performance

Part 1 – Coming soon.

Referenced Posts:

Things to consider when choosing infrastructure
Enterprise Architecture & Avoiding tunnel vision
Peak performance vs Real World Performance

Nutanix Resiliency – Part 7 – Read & Write I/O during Hypervisor upgrades

Posted on June 11, 2018 by Josh Odgers

If you haven’t already review Parts 1 through 4, please do so as they cover critical resiliency factors around speed of recovery from failures, increasing resiliency on the fly by converting from RF2 to RF3 as well as using Erasure Coding (EC-X) to save capacity while providing the same resiliency level.

Parts 5 and 6 covered how Read and Write I/O function during CVM maintenance or failure and in this Part 7 of the series, we look at what impact hypervisor (ESXi, Hyper-V, XenServer and AHV) upgrades have on Read and Write I/O.

This post will refer to Parts 5 and 6 heavily so they are required reading to fully understand this post.

As covered in Part 5 and 6, no matter what the situation with the CVM, read and write I/O continues to be served and data remains in compliance with the configured Resiliency Factor.

In the event of a hypervisor upgrade, Virtual Machine are first migrated off the node and continue normal operations. In the event of a hypervisor failure the Virtual Machine would be restarted by HA and then resume normal operations.

Whether it be a hypervisor (or node) failure or hypervisor upgrade, ultimately both scenarios result in the VM/s running on a new node and the original node (Node 1 in the diagram below) being offline with the data on it’s local drives unavailable for a period of time.

Now how does Read I/O work in this scenario? The same way as was described in Part 5 with reads being serviced remotely OR if the 2nd replica happens to be on the node the Virtual Machine migrated (or was restarted by HA) onto, then the read is serviced locally. If a remote read occurs the 1MB extent is localised to ensure subsequent reads are local.

How about Write I/O? Again as per Part 6, all writes are always in compliance with the configured Resiliency Factor no matter if it’s a hypervisor upgrade OR CVM, Hypervisor, Node, Network, Disk or SSD failure with one replica being written to the local node and the subsequent one or two replica/s distributed throughout the cluster based on the clusters current performance and capacity per node.

It really is that simple, and this level of resiliency is achieved only thanks to the Acropolis Distributed Storage Fabric.

Summary:

A hypervisor failure never impacts the write path of ADSF
Data integrity is ALWAYS maintained even in the event of a hypervisor (node) failure
A hypervisor upgrade is completed without disruption to the read/write path
Reads continue to be served either locally or remotely regardless of upgrades, maintenance or failure
During hypervisor failures, Data Locality is maintained with writes always keeping one copy locally where the VM resides for optimal read/write performance during upgrades/failure scenarios.

Nutanix Resiliency – Part 6 – Write I/O during CVM maintenance or failures

Posted on June 8, 2018 by Josh Odgers

In Part 5 we covered how Read I/O is serviced during CVM maintenance or failure so now we need to cover the arguably more difficult and critical task of servicing write I/O during the same maintenance or failure scenarios.

For those of you who read Part 5, this next section will look familiar. For those who have not read Part 5 I would ask that you please do so but let’s quickly cover off again the basics of how Nutanix ADSF writes and protects data.

Looking an the following diagram we see a three node cluster with a single Virtual Machine. The VM has written some data represented by a,b,c & d & under normal circumstances all writes will have one replica written to the host running the VM (in this case Node 1) and the other replica (or replicas in the case of RF3) are distributed throughput the cluster based on disk fitness values. The disk fitness values (or what I call “Intelligent replica placement”) ensure data is placed in the most optimal place the first time based on capacity and performance.

If one or more nodes are added to the cluster, the Intelligent replica placement will send proportionally more replicas to those nodes until the cluster is in a balanced state. In the very unlikely even no new writes are occurring, ADSF has a background disk balancing process which will balance the cluster as a low priority.

Now that we know the basics of how Nutanix protects data using multiple replicas (called “Resiliency Factor”) let’s talk about what happens during a Nutanix ADSF storage layer upgrade.

Upgrades are initiated by a one-click process and performed in rolling style one controller VM (CVM) at a time regardless of the configured Resiliency Factor and if Erasure Coding (EC-X) is used or not. The rolling upgrade put simply takes one CVM offline at a time, performs the upgrade, performs and self check and then rejoins the cluster and then repeats the process on the next CVM.

One of the many advantages of Nutanix decoupling the storage from the hypervisor (i.e.: not embedding storage into the kernel) is that upgrades and even storage layer failures do not impact the running Virtual machines.

VMs do not need to be restarted (i.e.: Like a HA event) nor do they need to migrate (e.g.: vMotion) to another node. VMs continue without interruption to storage traffic even when the local controller is offline for any reason.

If the local CVM is down for maintenance or due to failure, the write I/O is dynamically re-directed throughout the cluster.

Let’s look at a Write I/O when the CVM local to a VM is offline (for any reason).

The local CVM being offline means the physical drives (NVMe, SSD, HDD etc) are not available meaning the local data (replicas) is unavailable.

All write I/O will be continue to function and remain in compliance with the configured Resiliency Factor (RF), however rather than one replica being written locally, it will be written to a remote CVM over the network as will the other replica/s.

In the example below, we have a three node cluster so the VM on Node 1 is writing both replicas for “E” over the network to Node 2 and 3. This is how new data is serviced.

If more nodes existed in the cluster, the write traffic would be distributed evenly using Intelligent Replica Placement across all nodes within the cluster as shown below.

In the event data is being overwritten (as opposed to net new data) and the local replica is unavailable due to the CVM being offline, Nutanix ensures data integrity is maintained by overwriting the available replica and writing a second (or third for RF3) copy on another node in the cluster.

This is critical because if data is not always kept in compliance with it’s resiliency factor (FTT for VMware vSAN) a subsequent drive or node failure would cause data loss.

A major resiliency advantage Nutanix has over vSAN is the fact we always remain in compliance with the configured Resiliency Factor including during all failure and maintenance scenarios. vSAN however does not maintain it’s configured FTT level during all host maintenance and failure scenarios. For VMs on vSAN configured with FTT=1, in the event the host hosting one vSAN disk “object” is offline for maintenance, new overwrites are not protected so a single drive failure can result in data loss.

Chief Technologist at VMware, Duncan Epping recently posted an article titled: “VSAN 6.2 : Why going forward FTT=2 should be your new default” where he recommended FTT=2 as the new default for vSAN customers.

I have to agree with Duncan, but I wouldn’t say vSAN should be set to FTT=2, I would say it MUST be set to FTT=2 as FTT=1 creates a single point of failure for over-writes during maintenance or failures and this is unacceptable for most production workloads with VDI being one of a potential few exceptions in some cases.

Nutanix on the other hand does not have the same architectural flaw as vSAN and as such, RF2 is extremely resilient and suitable for even the most critical environments as explained in this series.

That and the fact ADSF is able to restore resiliency in such a timely manner, RF2 has far superior resiliency compared to vSAN FTT=1.

In the next part we will cover the critically important topic of how VMs are impacted during hypervisor (ESXi, Hyper-V, XenServer and AHV) upgrades.

Summary:

Write I/O continues uninterrupted if the local CVM is offline
Write I/O is distributed throughout the cluster evenly thanks to Intelligent Replica Placement
All new data is written in compliance with the configured Resiliency Factor
Overwrites of existing data is always written in compliance with the configured Resiliency Factor by writing a new replica where the original replica is not available.
Data integrity is ALWAYS maintained regardless of a CVM being under maintenance or failure.
Nutanix RF2 is more resilient than vSAN FTT=1 despite each claiming to maintain two copies of data.

CloudXC

By Josh Odgers – VMware Certified Design Expert (VCDX) #90

Category Archives: Nutanix

Nutanix | Scalability, Resiliency & Performance | Index

Nutanix Resiliency – Part 7 – Read & Write I/O during Hypervisor upgrades

Nutanix Resiliency – Part 6 – Write I/O during CVM maintenance or failures

Share this:

Share this:

Share this: