Name: Anand Prakash
Title: Virtualization Specialist
Twitter: @ananprak32
Profile: VCA-WM, VCA-DCV , VCA-Cloud , VCP3, VCP-DCV 4,5

Problem Statement

This is new company which is virtualization to VMware vSphere 5.x environment. As they have been managing lot of physical environment, they want to carry forward those settings/practices in Virtual Environment:
1.Tiering of all disks allocated to virtual machines. Using different Tiers for different mount point inside Windows Guest. So that they can separate OS/Data/Application Log with different performance levels.
2.They want to use dedicated LUN’s for all the VMDK’s assigned to the Guest. Possibly using  RAID inside guest to enhance performance .
3.Having multiple of virtual NIC’s for Backup/Management/Data per virtual machine.
4.Creating larger machines with more memory and CPU so that applications runs faster. While creating virtual machines it is considered for future workloads (Minimum 5 years). This includes Memory/CPU/Storage.
5.Using traditional methods of DR/Backup in virtual environment. Having similar dedicated environment at DR site. The servers are running all the time so that they can used during and event of DR.
6.The environment has been designed and implemented to meet above requirements. Now they have are having multiple of performance issues and outages. The engineers spend most of their time in troubleshooting and fixing/reworking issues.
7.With so much investment they have hit the limit of consolidation. They have some legacy servers for which hardware warranty has expired and cannot be virtualized. Customer do not want to extend the warranty of these servers and it is difficult to find spares or upgrade them.

Assumptions

1.Environment is running VMware vSphere 5.0 (VMware vCenter 5.0 and ESXi 5.0)
2.There are 2 sites with 12 servers of following Specs:
HP ProLiant SL210t Gen8 Server with 512GB of Memory and 4 – 12 Core CPU, Embedded NIC – Intel’s x2 1G NIC on board
3.There are clusters made up 4 hosts each. There 2 clusters at primary site (1 Prod and 1 Non Prod). The other cluster at secondary site is dedicated to DR and not running any workload.
4.They are using Symmetrix VMAX storage array with VPLEX solution.
5.They have replication technologies to replicate data at secondary site.
6.Out of 2- 1 GB NIC’s, one is dedicated for vMotion, Management and other NIC is dedicated for virtual machine traffic.
7.Network is capable to supporting 10GB NIC’s.
8.The Non-Prod environment are not required to be running during DR.

Constraints

1.Organisations wants to remediate this issues with minimum investment and hardware purchases.
2.The Production and Non-Prod environment should be physically separate and should not communicate with each other.
Motivation

1.The environment should easy to manage and should not require too much time and effort to support it.
2.The environment should provide higher consolidation ratio and better performance.
3.They should be standard across environment and flexible to changing needs.
4.More network redundancy for failures and load balancing.
5.This should provide customer satisfaction and cost savings.
6.Provide better options to manage capacity.

Architectural Decision

1.Recommended to use of Automated Storage Tiering (AST) at Storage Array. There are policies which can be set at storage level of assign infrequently used data to slower, less-expensive SATA storage but allow it to be automatically moved to higher-performing SAS or solid-state drives (SSDs) as it becomes more active. We can create many different groups based on business need (Bronze/Silver/Gold and Replicated/Non-Replicated). Create Storage DRS cluster and simply deploy the virtual machines to that DRS cluster.
Note: Make sure you disable uncheck “Enable I/O metric for SDRS recommendations.” if you are using Automated Storage Tiering (AST) at Storage Array.
2.Recommended to use large 4TB LUN’s instead of using small LUN per vmdk. Move the virtual machines to appropriate SDRS Group. User single disks from datastore to create single drive.
3.Use single NIC’s per virtual machine and use network less backups method like VMware vSphere Data Recovery. No management and backup NIC’s.
4.Deploy VMware vCenter Operations in environment. Start analysis of whole environments and right size the virtual machine based on their active usage.  And right size machine based on analysis of vCenter Operations manager. Also if required use converter to right size disks (It is easy to increase it but you will need rework to decrease the disk allocation).
5.Buy additional 2 – Dual Port 10GB NIC’s and install them on all 12 servers. Then create 2 Addition virtual switches on ESXi hosts. Connect one uplink of each 10GB NIC to Prod Network and other uplink to Non-Prod Network. Have both onboard uplinks connected to Management and vMotion VMkernal interfaces. Use ESXi host profiles to standardize this configuration.
Once the above configuration are done to ESXi host we can run both Prod and Non-Prod load on Same ESXi host. Now instead of using backup/restore or data disk replication we can setup the replication of all large datastores. Move the Guest OS on these datastores which require DR. We will run the Non-Prod load on the cluster at DR sites. We can use VMware Site Recovery to configure DR Plan to shutdown Non-Prod Guests at DR site and power on the Production Guests.
Merge both Prod and Non-Prod Clusters in the same cluster at primary site. Setup high priority/Reservation to Production Guests.

Alternatives

1.You can use vSphere Storage Profiles to maintain Tiers or Storage I/O Control.
2.Leave as current configuration
3.Leave as current configuration
4.You can ask to use to use VMware Capacity Planner to review the environment.
5.Leave as current configuration

Justification

1.You can use vSphere Storage Profiles and SIOC in combination. But the use of storage profiles post implementation is tedious task. Also if you do not use storage profiles you will loose current configuration and location of disk tiers (During storage vMotion if advanced option is not used). This type of machines with Tiers are difficult to storage vmotion and recovery. When you take snapshot of this type of Virtual machine and all IO’s goes to working directory and the naming structure of snapshot files and base disks become non-standard. If there was the snapshot and virtual machine is crashes (Configuration is lost) it will not be easy to recover. Even if you dedicate the Gold tier (SSD Disk) for application you will not know if it is utilized fully.
2.If you use multiple disks with dedicated LUN you will hit the limit of LUN (256) and no. of paths (1024). Again every time you create a new disk you will loose some free space for VMFS Metadata. Also there will whole lot of free space which gets wasted. If you do not maintain free space your vCenter will be full of alerts. And it will be difficult to distinguish between genuine and false alerts. Hence you will be limited in how many virtual machines you can create. Creating RAID inside guest can invalidate the impact of RAID created at storage array.
3.When you configure the network less backup you do not need dedicated backup NIC. As data on all NIC’s travel through same uplink, there is no point in having dedicated virtual NIC’s for management and data. Also if have some data traffic issues which is unlikely on 10GB uplink you can manage the virtual machine using console. This is critical when you are using Nexus 1000v with limited no. of ports. You could hit the limit of port much before you hit the limit of memory and CPU
4.Using VMware Capacity Planner can help one but there is need for continuous capacity planing. So VMware Operations Manager is a better option. The virtual machines themselves are not aware that they are running virtual hardware and they think that all resource allocated to them are only meant for them and try to use it. But in reality they are shared with others (Hence the machine really need resources suffer). In virtual it is very easy to increase capacity so we only provision for current workload unlike physical.
5.With current configuration there is no redundancy for all the networks. As network is capable of 10GB network and onboard NIC’s are not capable of utilizing that. Also the current configuration requires dedicate host for prod and Non-Prod network. This setup will create standardization of ESXi hosts and make it possible to use for all Guests (Prod/Non-Prod). This can also give you flexibility to group similar (Window/Linux) family to group together based on host (If same type of Guest OS running then they can better utilize Memory TPS). Also if Guest OS are placed on same datastore back-end storage can make use of de-duplication and save lot of storage.

Implications

1.Simple easy management of Virtual machines as VMware admin does not require to spread machine over datastores and can use simple storage vmotion to move virtual machines. You will not need top maintain documentation or keep checking compliance of disk periodically. The storage array can manage load effectively by monitoring current usage. This will ensure all the Tiers are used efficiently and on need basis. Whenever there is low space on datastore cluster you add new datastore to cluster and it manages the re-balancing automatically.
2.More free space on the datastores. Combined free space of many machines results in more savings. There are no alerts in vCenter for free space usage.  More deployment of virtual machine and increased efficiency. Environment will have spare capacity to deploy more storage or larger disks machines if required.
3.This will increase the speed of backup and do not clog your network. This will also result in cost savings as you do need tradition agents for backing up individual virtual machine. This will also result in less utilization of memory and CPU of ESXi host. This will provide you more ports to deploy more virtual machines. This will increase consolidation ratio and effective use of physical resources. This will in turn help to recover initial cost of hardware early.
4.Based on analysis we can release the memory, CPU and storage from other virtual machines. This will result in additional resources for other virtual machines and reduced CPU wait times. This will also allow business to virtualize more servers and retire old hardware which cannot be upgraded. More savings for business which can be used to by 10GB NIC’s.
5.As we have host running plenty of memory and CPU, the network can become bottleneck. Deploying additional 10GB NIC’s will overcome this bottleneck. This will provided better performance and more consolidation and efficient use of hardware. This will make it possible to make use of hardware which was dedicated for DR and was using power and space. This will further result in cost saving and higher customer satisfaction.

Back to Competition Main Page or Competition Submissions