Recently I was putting together a design for a vCenter Operations for View 1.0.x solution for a customer who has approx 6000 virtual desktops and it got me thinking, what would best way to implement vCenter Operations for View for a 10,000 user “Pod” for a View 5.0 or 5.1 environment?

Before we begin, I would like to clarify this solution is designed to work with a standard View “Pod” design. If your environment does not follow the VMware View Reference Architecture then I do not recommend this architecture. eg: Managing Server and Desktop workloads via the same vCentrer may cause issues for this solution.

Example: It is assumed if a customer is deploying a VMware View solution with greater than 2000 users, that a management cluster will be used, or that the View Management VMs (View Connection Server / View Security Server / View Composer etc) are hosted somewhere other than the View Blocks themselves.

For more details on why using a Management cluster for View Management VMs is preferred , see my post “Example Architectural Decision – Supporting VMware View Infrastructure Servers”

The below is the basic concept of the View “Pod” which is made up of five (5) “Blocks”. Each block is a vSphere cluster which supports up to 2000 view users.

This above graphic is courtesy of John Dodge in his blog post Demystifying VMware View Large Scale Designs

In summary a 10,000 user View “Pod” is made up of

* Five (5) vCenter Servers
* Five (5) View Composer Servers (Note: Can be installed on the vCenter server)
* Seven (7) View Connection Servers (Brokers)
* Five (5) View “Blocks”

Now we need to confirm what is required to implement vCenter Operations for View.

Lets look at the system requirements for the vC Ops View Adapter server.

Reference: Pg 11 of vCenter Operations for View Integration Guide

So based on the above, to support a 10,000 user pod we would require a “Monster” VM with 20 vCPUs and 40gb Ram!

This would require an ESXi host with at least two physical CPUs w/ 10 cores each and the “Monster” VM would basically monopolize the host, so this doesn’t seem like a viable solution for the vast majority of customers.

Alternatively, if we take a scale out approach then we can use five (5) VMs with 4 cores and 8Gb ram. This sounds perfectly reasonable, and would fit within the majority of vSphere clusters currently deployed.

Next lets look at the requirements for the vCenter Operation Manager vApp.

Reference: Pg 13 of the vCenter Operations Manager 5.0 Installation Guide

Here we can see three examples, that support up to 1500, 3000 and 6000 virtual machines.

From these numbers, if we used a single vC Ops manager vApp it would require greater than 8vCPUs each for the UI and Analytics VMs, which could monopolize smaller management ESXi hosts and/or cause CPU scheduling difficulties or reduced consolidation ratios for the management cluster.

So similarly to the View Adapter server, I am proposing a “scale out” approach for the vC OPS Manager vApp.

In this case, I want to comfortably support 5 “Blocks” of 2000 virtual machines, therefore allowing for some head room, the “up to 3,000 virtual machine” solution appears to be the best option.

Therefore we will require five  vCenter Operations Manager vApp deployments for this solution.

It is also important to consider the storage capacity and performance requirements, which are shown below.

Reference: Pg 13 of the vCenter Operations Manager 5.0 Installation Guide

From a capacity/performance perspective, the solution for 10,000 users needs too be sized to support between

* 15,000 & 30,000 IOPS
* ~6TB & ~12TB

At this stage we have determined we require the following

5 x vCenter Operations for View Adapter Servers

5 x vCenter Operations Manager vApp’s

Next we need to work out how best to configure each View Adapter server.

If you review the vCenter Operations for View Integration Guide on page 19 you will see the below graphic which states “Enter the name of a View connection server in your VDI environment….”

So the question is, which View Connection Server (Broker) should we use?

Here I have come up with two Solutions which are overall very similar but have a couple of differences, which are

1. How many connection brokers are used to service user connections
2. What connection broker/s the View Adapter servers connect too.

Lets go over both solutions as well as a potential option 3 which needs further investigation. (I will follow up with another article on Option 3)

Solution 1 : Dedicated Connection Brokers for vCenter Operations for View Adapter Servers

The concept here is there is a total of 7 connection brokers servicing the 10,000 user “Pod”. We remove two (2) of the brokers (in this example, Number 6 and 7) from the round robin on the load balancer and configure the View Adapter servers to use either Connection Broker 6 or 7.

Here is a diagram showing the solution

Note: Even though Connection Brokers 6 & 7 are not included in the round robin to service user connections, they continue to replicate between all other brokers.

Here are the Pros and Cons for Solution 1

PROS

* Traffic from vC Ops for View does not impact the performance of the connection brokers which users connect too

CONS

* Only five (5) connection brokers are available to service user connections (Note: 5 should be sufficient as each broker can support 2000 connections)
* In the event one (or more) of the five connection brokers has an issue user connection times may be impacted

Solution 2 : One to One mappings between View Adapter Servers and Connection Brokers

The concept here is there is a total of 7 connection brokers and 5 View Adapter servers. Each View Adapter server is configured to connect on a one to one basis to a specific connection broker eg: View Adapter Server 1 connects to Connection Broker 1, and so on.

Here is a diagram showing the solution

Here are the Pros and Cons for Solution 2

PROS

* All seven (7) connection brokers are available to the load balancer to service user connections
* The performance of all the connection servers (Brokers) should be consistent as they have a fairly equal workload (except for 6 & 7 which dont service View Adapter traffic)

CONS

* Traffic from vC Ops for View may impact the performance of the connection brokers which users are using to connect

Solution 3 – Configuring the View Adapter servers to use the load balancer address

What about configuring the View Adapter servers to use the load balancer address, rather than connecting to a specific connection broker? I am currently investigating this option, and from some discussions with a number of the VMware EUC team, there may be reasons this wont work. I will post an update once I have further investigated and tested this option.

Moving on, Once you decide which of the above two Solutions suit your environment best, the following applies to all three of the above solutions.

For each “Block” (ran by a dedicated vCenter server), one (1) vC OPS for View Adapter server and one (1) vCenter Operations Manager vApp are deployed into the Management cluster.

For a complete 10,000 user “Pod”, this will mean a total of five (5) vC OPS for View Adapter server and five (5) vCenter Operations Manager vApps.

Each vCenter Operations Manager server will be configured on a one to one basis with one vCenter eg: Block 1 vC Ops vApp connects to Block 1 vCenter.

Next we need to ensure each vC Ops for View Adapter server isn’t monitoring all desktop pools in the pod (which it will by default). This is very important otherwise each View Adapter server will be saturated with all pools, and therefore managing  all desktops in the “Pod” (up to 10,000 desktops!). This would cause major performance issues and result in the vC Ops for View environment hitting some hard limits.

To avoid this issue we select the tick box “Specify desktop Pools” (shown below) and enter all pool names (separated by a “,”)  for the “block” that is being monitored by this View Adapter server, similar to the below.

Now each of the View Adapter servers configured to only monitor one block, so a maximum of 2000 users.

However, there is a catch, as the pool filter is configured on the View adapter server/s, there is still an ongoing overhead (both compute & network) on both the connection broker/s and the view adapter server/s as the View pods topology is still proceeded by the adapter server before being filtered out (by specifying the desktop pools as in the above screen capture).  It is important to understand this overhead does not impact the vC Ops Manager vApp in any way as it is associated with a vCenter which will only manage up to 2,000 users.

So in addition to the standard components of the 10,000 user View “Pod” discussed earlier, we now have

* Five (5) vC Ops for View Adapter Servers
* Five (5) vCenter Operations Manager vApps

In summary, here are the Pros and Cons of the overall concept.

PROS

1. Avoid’s the requirement for very large (>8vCPU) vC OPS UI / Analysis & View Adapter VMs to support the solution
2. Prevents large VMs from potentially monopolizing management cluster ESXi hosts
3. Prevents an increased CPU scheduling overhead in scheduling large vCPU VMs for the Management ESXi hosts
4. In the event of a View Connection Server (Broker) , View Adapter Server and/or vC OPS Manager vApp failure, only part of the monitoring solution is impacted.
5. Prevents an increased CPU scheduling overhead in scheduling large vCPU VMs for the Management ESXi hosts
6. The solution is scalable and can start from a smaller deployment of <2000 users (with one vC Ops vApp and one View Adapter Server) and easily scale with linear performance even beyond a 10,000 user pod as its based on a repeatable model supporting 2000 users. In short, this solution can scale basically without limit.
7. No single View Connection Server (Broker) is managing all vC OPS for View traffic
8. Increased DRS flexibility/efficiency for the management cluster as smaller VMs are easier for DRS to load balance
9. Flexibility in ESXi host hardware, ie: The ability to have smaller & potentially cheaper (2 socket / 8 core) servers for management

CONS

1. Additional installation / configuration time as five (5) View Adapter servers & five (5) vCenter Operations Manager vApps need to be deployed
2. Increased Microsoft Windows 2008 licensing – although if you can justify Windows 2008 Datacenter edition licenses on the management cluster/s allow unlimited windows VMs as such avoiding this issue.
3. Additional maintenance effort in patching/upgrading five (5) View Adapter servers & five (5) vCenter Operations Manager vApps instead of one.
4. You do not get a single pane of glass for monitoring the entire 10,000 user Pod, you will have five (5) separate vC OPS web interfaces, one per block.

In Conclusion, the above architecture is scalable and allows the deployment of vCenter Operations for View version 1.0.x which avoids a number of potential “gotchas”, some of which may degrade the performance of your View environment. Each View Adapter Server and vC Ops Manager vApp will only service one (1) block of 2000 users. When adding additional Blocks or Pods (new Pods are required for >10000 users) the solution will scale and support 2000 users at a time.

My advise would be to use Solution 1, as the main advantage is as the Adapter server still has to process all topology data before filtering it out, this solution ensures there is no impact on the connection brokers servicing user connections. This means the CPU/Network overhead (discussed earlier) only impacts the connection brokers not servicing users.

Looking forward to the upcoming version of vCops for View (1.5), it will bring increased scalability and is penciled in for late Q1 2013, so the above architecture is really an interim solution until the new product is released.

Note: In VMware View 5.2 (which will also be released first half of 2013) there are some major improvements in scalability which may change the “Block” and “Pod” architecture discussed in this post as well as some improvements to the View agent, again these changes will likely change the vC Ops for View architecture.

I will be following up this article, with a similar post on vC Ops for View 1.5 architecture closer to the release date.

I would like to Thank John Dodge , David Wooten , David Homoki & Tim Whiffen from VMware EUC team, as well as Michael Webster (@vcdxnz001) & Andre Leibovici (@andreleibovici) for there input into this article.

I hope this article has been helpful and I welcome any constructive feedback / comments etc.