The ultimate Hyper-V Resources Guide

Source

It’s all available at : http://technet.microsoft.com/en-us/dd565807.aspx

New Offline Virtual Machine Servicing Tool (OVMST) 2.1 now available!

with keeping your offline virtualized environment safe from attack and in compliance? Download the Microsoft Virtual Machine Servicing Tool 2.1. You’ll get free, tested guidance and tools that automatically service your offline virtualized machines with the latest operating system and application updates, eliminating cumbersome manual processes.

Windows Server 2008 R2 Live Migration – “The devil may be in the networking details.”

Source: Ask the Core Team

Windows Server 2008 R2 has been publicly available now for only a short period of time, but we are already seeing a good adoption rate for the new Live Migration functionality as well as the new Cluster Shared Volumes (CSV) feature. I personally have worked enough issues now where Live Migration is failing that I felt a short blog on what process I have followed to work through these may have some value.

It is important to mention right up front that there is information publicly available on the Microsoft TechNet site that discusses Live Migration and Cluster Shared Volumes. This content also includes some troubleshooting information. I acknowledge that a lot of people do not like to sit in front of a computer monitor and read a lot of text to try and figure out how to resolve an issue. I am one of those people. Having said that, let’s dive in.

It has been my experience thus far that issues that prevent Live Migration from succeeding have to do with proper network configuration. In this blog, I will address the main network related configuration items that need to be reviewed in order to be sure Live Migration has the best chance of succeeding. I begin with an initial set of assumptions which include the R2 Hyper-V Failover Cluster has been properly configured and all validation tests have passed without failure, the highly available VM(s) have been created using cluster shared storage, and the virtual machine(s) are able to start on at least one node in the cluster.

I start off by identifying the virtual machines that will not Live Migrate between nodes in the cluster. While it should not be necessary in Windows Server 2008 R2, I recommend first running a ‘refresh’ process on each virtual machine experiencing an issue with Live Migration. I say it should not be necessary because a lot of work was done by the Product Group to more tightly integrate the Failover Cluster Management interface with Hyper-V. Beginning with R2, virtual machine configuration and management can be done using the Failover Cluster Management interface. Here is a sample of some of the actions that can be executed using the Actions Pane in Failover Cluster Manager.

If virtual machine configuration and management is accomplished using the Failover Cluster Management interface, any configuration changes made to a virtual machine should be automatically synchronized across all nodes in the cluster. To ensure this has happened, I begin by selecting each virtual machine resource individually and executing a Refresh virtual machine configuration process as shown here –

The process generates a report when it completes. The desired result is shown here –

If the process completes with a Warning or Failure, examine the contents of the report and fix the issue(s) that was reported and run the process again until it successfully completes.

If the refresh process completes without Failure, try to Quick Migrate the virtual machine to each node in the cluster to see if it succeeds.

If a Quick Migration completes successfully, that confirms the Hyper-V Virtual Networks are configured correctly on each node and the processors in the Hyper-V servers themselves are compatible. The most common problem with the Hyper-V Virtual Network configuration is that the naming convention used is not the same on every node in the cluster. To determine this, open the Hyper-V Management snap-in, select the Virtual Network Manager in the Actions pane and examine the settings.

The information shown below (as seen in my cluster) must be the same across all the nodes in the cluster (which means each node must be checked). This includes not only spelling but ‘case’ as well (i.e. PUBLIC is not the same as Public) –

It is important to be able to successfully Quick Migrate all virtual machines that cannot be Live Migrated before moving forward in this process. If the virtual machine can Quick Migrate between all nodes in the cluster, we can begin taking a closer look at the networking piece.

Start verifying the network configuration on each node in the cluster by first making sure the network card binding order is correct. In each cluster node, the Network Interface Card (NIC) supporting access to the largest routable network should be listed first. The binding order can be accessed using the Network and Sharing Center, Change adapter settings. In the Menu bar, select Advanced and from the drop down list choose Advanced Settings. An example from one of my cluster nodes is shown here where the NIC (PUBLIC-HYPERV) that has access to the largest routable network is listed first.

Note: You may also want to review all the network connections that are listed and Disable those that are not being used by either the Hyper-V server itself or the virtual machines.

On each NIC in the cluster, ensure Client for Microsoft Networks and File and Printer Sharing for Microsoft Networks is enabled (i.e. checked). This is a requirement for CSV which requires SMB (Server Message Block).

Note: Here is where people get into trouble usually because they are familiar with clusters and have been working with them for a very long time, maybe even as far back at NT 4.0 days. Because of that, they have developed a habit for configuring cluster networking which basically is outlined in KB 258750. This article does not apply to Windows Server 2008.

Note: If CSV is configured, all cluster nodes must reside on the same non-routable network. CSV (specifically for re-directed I/O) is not supported if cluster nodes reside on separate, routed networks.

Next, verify the local security policy and ensure NTLM security is not being restricted by a local or domain level policy. This can be determined by Start > Run > gpedit.msc > Computer Configuration > Windows Settings > Security Settings > Local Policies > Security Options. The default settings are shown here –

In the virtual machine resource properties in the Failover Cluster Management snap-in, set the Network for Live Migration ordering such that the highest speed network that is enabled for cluster communications and is not a Public network is listed first. Here is an example from my cluster. I have three networks defined in my cluster –

The Public network is used for client access, management for the cluster, and for cluster communications. It is configure with a Default Gateway and has the highest metric defined in the cluster for a network the cluster is allowed to use for its own internal communications. In this example, since I am also using ISCSI, the ISCSI network has been excluded from cluster use. The corresponding listing on the virtual machine resource in the Network for live migration tab looks like this –

Here, I have unchecked the iSCSI network as I do not want Live Migration traffic being sent over the same network that is supporting the storage connection. The Cluster network is totally dedicated to cluster communications only so I have moved that to the top as I want that to be my primary Live Migration network.

Note: Once the live migration network priorities have been set on one virtual machine, they will apply to all virtual machines in the cluster (i.e. it is a Global setting).

Once all the configuration checks have been verified and changes made on all nodes in the cluster, execute a Live Migration and see if it completes successfully.

Bonus material:

There are configurations that can be put in place that can help live migrations run faster and CSV to perform better. One thing that can be done, is to Disable NetBIOS on the NIC that will be supporting the primary network used by CSV for re-directed I/O. This should be a dedicated network and should not be supporting any other traffic other than internal cluster communications, redirected I/O for CSV and\or live migration traffic.

Additionally, on the same network interface supporting live migration, you can enable larger packet sizes to be transmitted between all the connected nodes in the cluster.

If, after making all the changes discussed here, live migration is still not succeeding, then perhaps it is time to open a case with one of our support engineers.

Thanks again fro you time, and I hope you have found this information useful. Come back again.

Additional resources:

Using Live Migration with Cluster Shared Volumes in Windows Server 2008 R2

High Availability Product Team Blog

Hyper-V and Virtualization on Microsoft TechNet

Windows Server 2008 R2 Hyper-V Forum

Windows Server 2008 R2 High Availability Forum

Chuck Timon
Senior Support Escalation Engineer
Microsoft Enterprise Platforms Support

VMM Tricks: How to get the audit trail records from a VMM Job

Source

Today, in VMM, you can open the Administrator Console and click on the Jobs view to see all the Jobs in VMM. When you select an individual job, you can click on the “Change Tracking” tab and view any changes that were recorded in VMM as part of this job.

In this post, i will show you how to do this in PowerShell since it is not as intuitive. Basically, audit trail records that are placed in the change tracking tab are not cached on the client computer by default. the only way to surface them in powershell is to individually invoke the get-job commandlet and pass it the Job parameter while asking for the FULL job to be downloaded. the following powershell script does exactly that and displayed the audit trail records for all jobs in the system.

#get all jobs in the system
$jobs = get-job -all
foreach ($job in $jobs)
{

#get the details including the audit records for each single job in VMM
$singlejob = get-job -job $job -full
$auditrecordsexist = $singlejob | select AreAuditRecordsAvailable
if ($auditrecordsexist)
{

#if any audit records exist, print them out
foreach ($record in $singlejob.AuditRecords)
{
$record
$prev = $record.Previous
$newr = $record.New
$prev
$newr
}
}
}

Cluster Disk With Identifier (identifier) has a Persistent Reservation on it

One of my customer’s team member had destroy the Hyper-V Cluster by mistake. He formatted the cluster nodes without evict them before doing that . The nodes of the clsuter used to be  part of old cluster that was destroyed by mistake.

I tried to build the nodes again from the scratch and Create new cluster, When I run the validation wizard I got this error:

Cluster disk with identifier (identifier) has a persistent reservation on it ,the disk might be part of other cluster. removing the disk from other validation set.

My SAN is HP EVA. The Cluster is not able to see any of my LUNS although I can see them from disk management and can’t create the cluster.

This error due to the fact that the LUNs still keeping the old identifiers from the old cluster, You have to use Cluster command line to clear the reservation by that command:

cluster.exe node %nodename% /clear:disknumber

Now everything should work fine and you can pass the validation wizard

VMM tricks: How to troubleshoot the “Not Responding” host status in VMM 2008

This content is published in Microsoft KB 976640

Step 1: Use Virtual Machine Manager Configuration Analyzer

The Virtual Machine Manager Configuration Analyzer is a diagnostic tool that you can use to evaluate important configuration settings for computers that either are serving or might serve VMM roles or other VMM functions. The Virtual Machine Manager Configuration Analyzer does the following:

• Scans the hardware and software configurations of the computers that you specify
• Evaluates these configurations against a set of predefined rules
• Displays error messages and warnings for any configurations that are not optimal for the VMM role or other VMM functions that you have specified for the computer

System Requirements:

Before you install the Virtual Machine Manager Configuration Analyzer, you must download and install the 64-bit version of Microsoft Baseline Configuration Analyzer. To download the MBCASetup64.msi file, visit the following Microsoft Web site:

http://go.microsoft.com/fwlink/?LinkID=97952

Note The system requirements for Microsoft Baseline Configuration Analyzer indicate that only Windows Server 2003 is supported. However, the Virtual Machine Manager Configuration Analyzer and Microsoft Baseline Configuration Analyzer have been tested and are supported on 64-bit versions of Windows Server 2008.

You must install and run the Virtual Machine Manager Configuration Analyzer on the computer that either currently is or will become your VMM 2008 server. To review the system requirements for the VMM 2008 server, visit the following Microsoft Web site:

http://go.microsoft.com/fwlink/?LinkId=133516

To download the Virtual Machine Manager Configuration Analyzer, visit the following Microsoft Web site:

http://www.microsoft.com/downloads/details.aspx?FamilyID=02d83950-c03d-454e-803b-96d1c1d5be24&displaylang=en

Step 2: Check required hotfixes

On any servers that host a VMM role together with Hyper-V or Virtual Server hosts, you apply the following hotfixes:

956589 Description of the Hyper-V update for issues that may occur when you manage the Hyper-V role on the 64-bit editions of Windows Server 2008 by using SCVMM

956774 A Background Intelligent Transfer Service (BITS) client cannot handle files that have paths that contain the volume GUID in Windows Server 2008 or in Windows Vista

958124 A wmiprvse.exe process may leak memory when a WMI notification query is used heavily on a Windows Server 2008-based or Windows Vista-based computer

954563 Memory corruption may occur with the Windows Management Instrumentation (WMI) service on a computer that is running Windows Server 2008 or Windows Vista Service Pack 1

955805 Certain applications become very slow on a Windows Server 2008-based or Windows Vista S955805-based computer when a certificate with SIA extension is installed

The following are some common error messages that are addressed in these hotfixes:

Error (2911)
Insufficient resources are available to complete this operation on the server.domainname.com server.
(Not enough storage is available to complete this operation (0×8007000E))

Error (2912)
An internal error has occurred trying to contact an agent on the server.domainname.com server.
(No more threads can be created in the system (0×800700A4))

Error (2916)
VMM is unable to complete the request. The connection to the agent server.domainname.com was lost.
(Unknown error (0×80338012))

Error (2915)
The WS-Management Service cannot process the request. Object not found on the server.domainname.com server.
(Unknown error (0×80041002))

In addition to these hotfixes, you must also apply the following hotfix on the VMM server:

961983 Description of the hotfix rollup package for System Center Virtual Machine Manager 2008: April 14th, 2009

Step 3: Disable TCP Offloading

You must disable TCP Offloading in Windows, in the registry, and in any network adapter teaming management software that is being used. You must check all these locations to make sure that TCP Offloading is completely disabled. This operation must be performed on both the VMM server and the host computer.

Locate all network adapters in the registry under the following subkey:

HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\Class\{4D36E972-E325-11CE-BFC1-08002BE10318}

There are additional subkeys under this subkey that are named with four digits, starting with “0000.” Locate the subkeys that show the physical network adapter names on the details pane. Find the “DriverDesc” value on the details pane. This value should contain the name of a network adapter, such as “HP NC360T PCIe Gigabit Server Adapter.” For each of these subkeys, make the following changes:

Disable all vendor-specific offloading. Set values for any entries that include the word “Offload” to “0″ (disabled). For example, these entries include the following:

Collapse this tableExpand this table

Entry	Description
*FlowControl	No description available
*IPChecksumOffloadIPv4	Describes whether the device enabled or disabled the calculation of IPv4 checksums
*TCPChecksumOffloadIPv4	Describes whether the device enabled or disabled the calculation of TCP Checksum over IPv4 packets
*TCPChecksumOffloadIPv6	Describes whether the device enabled or disabled the calculation of TCP checksum over IPv6 packets
*UDPChecksumOffloadIPv4	Describes whether the device enabled or disabled the calculation of UDP Checksum over IPv4 packets
*UDPChecksumOffloadIPv6	Describes whether the device enabled or disabled the calculation of UDP Checksum over IPv6 packets
*LsoV1IPv4	Describes whether the device enabled or disabled the segmentation of large TCP packets over IPv4 for large send offload
*LsoV2IPv4	Describes whether the device enabled or disabled the segmentation of large TCP packets over IPv4 for large send offload
*LsoV2IPv6	Describes whether the device enabled or disabled the segmentation of large TCP packets over IPv6 for large send offload
*IPsecOffloadV1IPv4	Describes whether the device enabled or disabled the calculation of IPsec headers over IPv4.
*IPsecOffloadV2	Describes whether the device enabled or disabled IPsec offload version 2 (IPsecOV2).
*IPsecOffloadV2IPv4	Describes whether the device enabled or disabled IPsecOV2 for IPv4 only.
*RSS	Receive side scaling
*TCPUDPChecksumOffloadIPv4	Describes whether the device enabled or disabled the calculation of TCP or UDP checksum over IPv4
*TCPUDPChecksumOffloadIPv6	Describes whether the device enabled or disabled the calculation of TCP or UDP checksum over IPv6

To disable TCP Offloading in Windows, use the following registry entry for task offloading for the TCP/IP protocol:

Subkey: HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\TCPIP\Parameters
Entry: DisableTaskOffload
Type: REG_DWORD

You can set this registry entry to 1 to disable all task-offloading from the TCP/IP transport.

Many vendors have some forms of offloading capabilities built into their teaming management software. Such offloading can appear in many forms and is usually vendor-specific.

For more information about offloading, visit the following Microsoft Developer Network Web site:

http://msdn.microsoft.com/en-us/library/aa938424.aspx

Step 4: Check the Svchost.exe process of the Windows Remote Management service

Virtual Machine Manager depends very much on the Windows Remote Management service for underlying communication. Therefore, the “Not Responding” status is very likely to occur because of an error in the underlying Windows Remote Management communication between the VMM server and the host computer. In this case, the status is “OK” shortly after you restart the host computer. However, the status changes to “Not Responding” after 3 to 4 hours. Additionally, if you stop the Windows Remote Management service at a command prompt, it takes much longer than usual to be completed. Sometimes, it can take up to five minutes to stop.

This problem can occur if the shared Svchost.exe process that hosts the Windows Remote Management service is backed up.

To resolve this problem, configure the Windows Remote Management service to run in a separate Svchost.exe process. To do this, open an elevated command prompt, type the following command, and then press ENTER.

Note Make sure that you type the command exactly as it appears here. Notice the space after the “=” symbol.

c:\>sc config winrm type= own

If the command is completed successfully, you see the following output:

[SC] ChangeServiceConfig SUCCESS

Step 5: Check the VMM server computer account

This problem also occurs because the VMM server computer account is removed from the local Administrators group on the host computer. This setting may be caused by the “Restrictive Groups” Group Policy setting.

For more information about this cause, click the following article number to view the article in the Microsoft Knowledge Base:

969164 Virtual Server or Hyper-V host may have a status of “Not Responding” or “Needs Attention” due to Restricted Groups group policy setting

To resolve this problem, move the VMM server and host computers to a new organizational unit (OU) that blocks inheritance of all Group Policy objects.

Step 6: Check for other causes

Some other causes that are potential causes of this problem include the following:

• The VMM agent is not running.
• Some antivirus software is scanning ports or protocols.

VMM accounts membership

You can put the VMM accounts into the appropriate groups according to the following.

VMM server machine account:

• The Administrators group on the VMM server and on all host computers
• The Virtual Machine Manager Servers local group on the VMM server

The account for actions in VMM:

• The Local Administrators group on the VMM server and on all host computers

Getting Started with Citrix Essentials for Hyper-V

Citrix learning is providing new training for Citrix Essentials for Hyper-V. Go and catch your seat.

Description

This course is available at no cost for a limited time only!  Register now to add this course to your learning transcript and lock in the promotional offer. This course will remain available in your learning transcript for future access. Note: New users will need to create an account.

In this Essentials for Hyper-V online training course, learners will be introduced to Citrix Essentials for Hyper-V and gain the knowledge needed to perform basic installation and configuration tasks. Through online simulations, students will install and configure key features and explore product capabilities, such as virtual storage management, provisioning services, lifecycle management and workflow studio.

New content added!
This course is now updated to include features and enhancements aligned with the Windows Server 2008 R2 release.

Register

VMM tricks: SCVMM Service may take up lots of memory

Source: Hyper-V notes from the field

In a large SCVMM environment we noticed that the SCVMM Service (vmmservice.exe) allocates lots of memory (>4GB).  You may notice that you are unable to create new console sessions or existing sessions lose their connection when all available memory is used. This is not a leak as the memory is freed over time.

The reason for this allocations was the large number of jobs that had run in the past. By default SCVMM keeps the last 90 days in the Database and the Console shows this in the Jobs  pane. In the title area you can see the number of jobs in brackets. In the customer case we had almost 10.000 jobs.

The problem can be solved by setting a shorter history with the following registry key:

HKLM\Software\Microsoft\Microsoft System Center Virtual Machine Manager Server\settings\sql
DWORD Value: TaskGC
Enter the history length in days. (e.g. 7)

SCVMM starts a maintenance procedure with this number of days every 20h. You may not see a complete reduction immediately, as the maintenance procedure limits itself in the number of objects it deletes in one run. So you may need to monitor this for some days.

Cheers

Robert

Micorosft Lab Validation Report for Hyper-V

Microsoft Hyper-V : Scalable, Native Server Virtualization for the Enterprise

Microsoft just published a Lab Validation Report for Windows Server 2008 Hyper-V, which was written by Enterprise Strategy Group.  This report goes over the installation and configuration of Windows Server 2008 Hyper-V and management of those servers with Virtual Machine Manager 2008.

The report reviews the performance of Windows Server 2008 Hyper-V in comparison with physical systems.

Performance

In this section, we’ll take a look at the results of ESG Lab testing of the performance of applications running on a physical server and on a Hyper-V virtual machine.

ESG Lab Testing

ESG Lab used four real-world application workloads to evaluate the physical and virtual performance of Microsoft Windows 2008 Data Center Edition R1:

1.  Application Install:  a timed installation of Visio 2007 using a distribution image stored on a network

attached shared drive within a private network.

2.  Directory level copy:  a timed copy an 860 MB directory with 2,014 files to a temporary directory.  The c:\windows\win32 directory was copied to a temporary directory on the same C: drive.

3.  Subsequent copies:  the directory level copy was repeated with much of the IO activity happening in cache. The average of three cached copy operations was recorded.

4.  SQL query:  a long running SQL select statement using a 25,000 row production database from ESG’s

internal IT operation was timed.  The SQL query performed a join of three tables. The average duration of three select statements was recorded.

The HP blade server used for this test was equipped with four 2.2 GHz dual-core AMD Opteron processors and eight gigabytes of RAM.   Comparing physical and virtual performance on the same server was accomplished after a reboot with Hyper-V role enabled and disabled.   During the virtual server testing, the server was configured with a  single virtual server, which used nearly all of the physically available hardware resources (all eight CPU cores, seven out of eight GB of RAM).

Physical and virtual testing was performed within a 40 GB logical C: drive.  The C: drive was built using a single LUN presented by a FC attached HP MSA storage array with six 15K SAS drives configured as a single RAID-5 group (5+1).

The Hyper-V C: drive was configured as a basic virtual hard disk (VHD).  The results are shown in Table 1.

Table 1. ESG Lab Performance Results

Operation	Physical	Virtual	Difference
Application install	00:05:52.000	00:06:09.000	4.8%
Directory level copy	00:00:41.680	00:00:33.660	7.1%
Subsequent copies	00:00:05.660	00:00:05.830	3.0%
SQL query	00:00:47.566	00:00:53.630	12.7%

What the Numbers Mean

It took five minutes and 52 seconds to install an application on the physical  server running Windows 2008 Data Center Edition SP1   It took six minutes and nine seconds to install the same application on the same hardware running the same operating system running within a Hyper-V enabled virtual machine  The difference in performance is relatively low (4.8%) The directory level copy and subsequent copies were also relatively low (7.1% and 3.0% respectively)  A long running Microsoft SQL query took 12.7% longer when running in a virtual server  The manageably low performance impact of Hyper-V won’t be detected by the vast majority of end-users and applications

Hyper-V Virtual Machines could not intialize

This is one of the problems that you may face, When you start the VMs after you rename the host or join the domain you can’t start the VMs on taht host and face a problem that (Could not find a usable certificate)

For hyper-V with windows 2008 R2

VMName’ failed to initialize.
Could not initialize machine remoting system. Error: ‘Unspecified error’ (0×800070490).
Could not find a usable certificate. Error: ‘Unspecified error’ (0×800070490).

For Hyper-V with Windows 2008

VMName’ failed to initialize.
Could not initialize machine remoting system. Error: ‘Unspecified error’ (0×80004005).
Could not find a usable certificate. Error: ‘Unspecified error’ (0×80004005).

This is a well known problem for Hyper-V V1 use this update to solve it as per KB Article 967902 that details the symptoms and resolution.

For Hyper-V V2 just restart the host and everything will be fine.