This video shows how to configure Network I/O Control (NIOC) as per Nutanix Best Practices, however this configuration is also applicable to any IP based Storage.
For more information see the Nutanix vNetworking Best Practices Guide.
Related Articles:
1. Network I/O Control Shares/Limits for ESXi Host using IP Storage
2. Network I/O Control for ESXi Host using IP Storage (4x10Gb NICs)
This is the fifth part of a series of posts covering how the Integrity of Write I/O is ensured for Virtual Machines when writing to VMDK/s (Virtual SCSI Hard Drives) running on NFS datastores presented via VMware’s ESXi hypervisor as a “Datastore”.
This part will focus on Data Corruption.
As a reminder from the first post, this post is not talking about presenting NFS direct to Windows.
So why am I covering data corruption? Simple, because there is a misconception that SCSI commands are not properly supported for VMs running on NFS datastores which leads to corruption. This was covered in Part 1, so Part 5 will focus on data corruption not specific to NFS, but which can effect all storage platforms and how it occurs, then how storage solutions can mitigate the risk of data corruption issues.
The following data is a summary of the data provided in An analysis of data corruption in the storage stack.
Netapp conducted a large scale study into data corruption, which covered >1 Million HDDs across tens of thousands of Netapp systems over 41 months (2004 – 2007) and long story short, Netapp detected a level of data corruption which surprised me and seems to disprove many things like advertised MTBF for HDDs.
The following shows a breakdown of the problems found.
The first thing I noticed in the above pie charts is the vast difference between the percentage of failures in Enterprise grade disks (left) and nearline based disks (right).
It also shows physical interconnects to be a large percentage of failures, which highlights the need for simplicity in the storage solution. In addition, one of the more surprising results in the level of storage protocol and performance based failures being the cause of corruption.
Note: In this study, the majority of systems deployed were FC (Block storage based) based, this highlights that a storage protocol itself regardless of being block or file based storage, can have issues if improperly implemented. So regardless of storage protocol, corruption can occur.
The below summary of corruption type and percentage of disks effected shows the dramatic 10x more issues with SATA drives compared to Enterprise grade drives.
The above also shows bit corruptions or Torn Writes effect more disks compared to lost or misdirected writes, which highlights the importance of Torn I/O Protection (covered in Part 4).
The article summarizes in the following points:
The main take away from my perspective is:
1. The requirement to have corruption handling mechanisms for any environment running workloads which require data integrity.
2. Data should be spread out (ideally across disks) to minimize the chance of issues.
The article went on to form these conclusions:
In Summary:
1. Data corruption can occur on JBOD , enterprise grade storage solutions and everything in between.
2. SATA drives have a much higher rate (~10x) of corruption.
3. Enterprise grade drives are much better from a data integrity perspective.
4. Corruption handling via sector and ideally block based checksums is essential on writes.
5. Using a checksum on Read helps detect corrupted data.
6. Corruption can occur even when no ECC errors are reported by a physical HDD.
7. Any storage protocol implementation can have bugs which can lead to corruption.
8. Backup / Recovery solutions are essential. Reliance solely on primary storage or application level backups using disks puts your data at risk.
9. Solutions solely dependant on application level data protection on disk are at risk of corrupted data being replicated to other active/passive or backup copies.
My final point, in an enterprise grade storage solutions which use checksums to verify data integrity on write and reads, have a much lower risk of data corruption regardless of media type and storage protocol.
JBOD style deployments using SATA drives have a significantly higher risk of data corruption which is contributed to by the SATA drives 10x higher corruption rates and the lack of enterprise grade checksum features found in some shared storage (SAN/NAS) solutions.
Integrity of Write I/O for VMs on NFS Datastores Series
Part 1 – Emulation of the SCSI Protocol
Part 2 – Forced Unit Access (FUA) & Write Through
Part 3 – Write Ordering
Part 4 – Torn Writes
Part 5 – Data Corruption
Nutanix Specific Articles
Part 6 – Emulation of the SCSI Protocol (Coming soon)
Part 7 – Forced Unit Access (FUA) & Write Through (Coming soon)
Part 8 – Write Ordering (Coming soon)
Part 9 – Torn I/O Protection (Coming soon)
Part 10 – Data Corruption (Coming soon)
Related Articles
1. What does Exchange running in a VMDK on NFS datastore look like to the Guest OS?
2. Support for Exchange Databases running within VMDKs on NFS datastores (TechNet)
3. Microsoft Exchange Improvements Suggestions Forum – Exchange on NFS/SMB
4. Virtualizing Exchange on vSphere with NFS backed storage
This is the fourth part of a series of posts covering how the Integrity of Write I/O is ensured for Virtual Machines when writing to VMDK/s (Virtual SCSI Hard Drives) running on NFS datastores presented via VMware’s ESXi hypervisor as a “Datastore”.
This part will focus on Torn Write I/O.
As a reminder from the first post, this post is not talking about presenting NFS direct to Windows.
Some of you are probably wondering “What is a Torn Write”?
A Torn write can occur if there is a problem (e.g.: Power or HW failure) during a multi sector block being written.
The below shows what a Torn Write looks like, which is basically where part of data A and B remain after a Torn write, resulting in corrupted data.
Image Source: Silent data corruption in disk arrays: A solution
The article Toward I/O-Efficient Protection Against Silent Data Corruptions in RAID Arrays describes a Torn Write (I/O) as:
Torn write: When a disk write is issued to a chunk, only a
portion of sectors in the chunk are successfully updated,
and the chunk contains some stale sectors in the end part.
The issue with the write I/O being written across multiple sectors is that in the event a power outage impacting the write back cache or a hardware issue such as a drive failing, the I/O may be partially written (or “Torn”). This means the data was not fully written, but some data was written overwriting the existing data causing corruption.
In this case, if the storage solution provides a write acknowledgement and the data is partially or not written to persistent media this results in what is known as silent data corruption as data being read back will be part of the new data and part of the old data.
It should be noted RAID does not protect against Torn writes, nor can it help correct the situation once it has occurred.
The next question is, does the issue of Torn writes impact VMs on ESXi backed by NFS datastores. The answer is, Yes because Torn Writes can potentially occur on any storage solution regardless of the abstracted storage protocol.
So do Torn Writes occur VMs on ESXi backed by NFS datastores? The answer again would be Yes, but importantly, this would not be as a result of anything at the hypervisor layer, it would be as a result of a failure impacting the underlying storage.
Note: This issue equally impacts block and file based storage presented to ESXi, so it is not a NFS specific issue.
So what is required to provide protection against Torn Writes?
The best method to protect against Torn Writes is to use checksums, specifically Block level checksums which can check the integrity of writes which span multiple sectors, therefore in the event of a torn write, the checksum will fail and a write acknowledgement will not be sent. The important fact here is the underlying storage is responsible for this process, not ESXi , the VMDK or storage protocol (FC,FCoE,iSCSI, NFS!) presenting the storage to ESXi.
In summary, Torn Writes are not an issue with VMs running on ESXi backed by NFS datastores where the underlying storage performs Block level checksums.
I have requested VMware create a Knowledge base article on Torn Writes for formal reference and will update this post with the reference if/when this is done.
In part five, I will discuss Data Corruption.
Integrity of Write I/O for VMs on NFS Datastores Series
Part 1 – Emulation of the SCSI Protocol
Part 2 – Forced Unit Access (FUA) & Write Through
Part 3 – Write Ordering
Part 4 – Torn Writes
Part 5 – Data Corruption
Nutanix Specific Articles
Part 6 – Emulation of the SCSI Protocol (Coming soon)
Part 7 – Forced Unit Access (FUA) & Write Through (Coming soon)
Part 8 – Write Ordering (Coming soon)
Part 9 – Torn I/O Protection (Coming soon)
Part 10 – Data Corruption (Coming soon)
Related Articles
1. What does Exchange running in a VMDK on NFS datastore look like to the Guest OS?
2. Support for Exchange Databases running within VMDKs on NFS datastores (TechNet)
3. Microsoft Exchange Improvements Suggestions Forum – Exchange on NFS/SMB
4. Virtualizing Exchange on vSphere with NFS backed storage