Planning for Windows iSCSI SAN boot on Private Cloud Bare Metal Hosts

Data Center Modernization has definitely reached critical mass. The message that came from TechEd 2013 was “It’s time to make Hybrid Cloud Real.” That, of course, starts with the modernizing your data center to be able to implement private clouds. On top of that, more and more data centers are migrating their hypervisors to Hyper-V in spite of the greater footprint a full Windows Server operating system has on the bare metal. The feature parity as well as cost savings that comes from Hyper-V as a feature (and the subsequent removal of the VMWare tax) offsets the hassle of the additional footprint.

Windows Server bare metal hosts running Hyper-V, like other hypervisors, support SAN boot of the operating system drive using iSCSI. It is important to realize that the iSCSI services depend on the underlying storage and iSCSI network being provisioned properly to accommodate the eccentricities of how Windows boots from SAN using network interface cards in place of traditional storage adapters or HBAs.

Understand the Supportability Parameters

The supportability of the storage support comes from the storage vendor. This also extends to iSCSI boot SAN scenarios per the KB article: https://support.microsoft.com/en-us/kb/305547/en-us. Even though the article does not mention Windows Server 2012 (or R2) it is still in place. Normally, this would not be complicated but in the case of iSCSI networks, the device may likely be using a NIC to locate the storage (especially if they are actually using NAS – network attached storage – i.e. NetApp) and not a traditional storage adapter or HBA.

Slipstream your 3^rd-party drivers if possible

The use of slipstreamed NIC/Storage drivers in the installation ISO will prevent any timing issues from swapping back and forth between driver media and OS media. The may be especially the case if you are controlling headless blade devices using KVM or some other solution. I have found that this resolves many of the issues outlined in this particular KB: https://support.microsoft.com/en-us/kb/2826787 – as well as the 0x80070057 error message when trying to format drives or create partitions during the operating system setup.

No Thin-Provisioning LUNs for the OS Boot Drive

LUNS on the NAS devices (i.e. NetApp Devices) need to be thick provisioned for the drive containing the OS instead of thin-provisioned. In addition LUNS for the host OS boot volume only should be 127GB or less. Remember this is only in the context of the LUN being used for host devices iSCSI boot volume.

Avoid using Default Gateways for iSCSI NICs

The NICs configured for the iSCSI SAN should avoid having a default gateway. This can cause issues such as slow throughput occurring during formatting of disks and the copying of files during installation. This has been an issue with the Windows iSCSI initiator in the past and has previous appeared in KB articles:

960104: If you start a system from iSCSI, the gateway specified in the iSCSI Boot solution will always be used by Windows to communicate with the iSCSI Target

http://support.microsoft.com/kb/960104/EN-US  

2727330: Default gateway is set to 0.0.0.0 if you start a Windows Vista-based, Windows 7-based, Windows Server 2008-based or Windows Server 2008 R2-based computer from an iSCSI boot device

http://support.microsoft.com/kb/2727330/EN-US  

In addition, the network ports connecting to the boot volume iSCSI interfaces on the iSCSI network’s switch should have ICMP redirect disabled.

If all else fails . . . revert to the old way!

If the interactive installation still fails, remember – there is the legacy way of deploying Windows Servers in an iSCSI SAN boot configuration outlined in:

https://technet.microsoft.com/en-us/library/ee619722%28v=ws.10%29.aspx

VMM: Options for Offline Servicing, P2V, and Building Virtual Networks

UPDATE: 10/21/2014: The MVMC 3.0 is now released with P2V functionality restored.

http://www.microsoft.com/en-us/download/details.aspx?id=42497

I work with SCVMM (System Center Virtual Machine Manager) frequently in many different contexts. I even do the occasional private cloud engagement specifically on VMM and Hyper-V. Most of the time however, I am using VMM in a peripheral context – be it personal lab work, proof-of-concept labs for customer or partners, etc. I have been very pleased with the evolution of Hyper-V and System Center products over the last few years. I find the largest issues that create pain points for me involve the constant need to service virtual machines, deal with physical-to-virtual conversions, and the cumbersome process of building test networks that demonstrate elements such as multi-tenancy that require me to super impose logical switches and other elements of software-defined networking on top of my switching fabric.

I field a lot of questions with regards to how to best go about these options with the most recent versions of SCVMM (particularly VMM 2012 R2.)

Virtual Machine Servicing

I don’t keep all of virtual machines running at the same time. In addition, I have many templates for which I reuse/import/export on a regular basis. In VMM 2012 there was the option of using a separate add-on utility called the Virtual Machine Servicing Tool. The problem is it only was for VMM 2012 RTM (or R1) and it does not work with VMM 2012 SP1 or VMM 202 R2. You will likely find many questions regarding this that appear in the comments section on my initial blog about the VSMT 2012 utility way back in 2012. http://blogs.technet.com/b/gladiatormsft/archive/2012/08/14/the-virtual-machine-servicing-tool-2012-is-now-available.aspx

So with there being no newer version of VSMT for 2012 SP1 or R2 and the fact that you cannot use VSMT 2012 on VMM 2012 SP1 or R2, what are your options going forward for servicing – particularly offline servicing? You have a few options:

• Customize a solution with DISM (Deployment Image Service and Management Toolkit) You should be very familiar with DISM as it is very useful for the consultant and IT Pro (like me) who does not always have access to System Center infrastructures. It can also be easily scripted to mount and service offline images for OS updates at the very least. You can become familiar with DISM servicing using the following link as it is a great introduction to the concept:  http://technet.microsoft.com/en-us/library/dd799267(v=WS.10).aspx. This walkthrough tells you how to mount a virtual disk online and then apply various servicing commands using the DISM tool. You can then apply updates using the tool to apply individual Windows Update packages (.MSU’s) although this can be cumbersome for many sets of updates. This does require scripting for effectiveness but I have found that I can get away with one set per OS so long as I have access to the individual .MSU files [DISM /image:C:MyDirMount /Add-Package /Packagepath:<file_path>] This way is still way quicker than standing up a VM running WSUS, keeping it in sync and then booting up every single VM and updating it through the WSUS server. There are also additional scripts out there that work with live WSUS servers and DISM that you can also try – for example http://gallery.technet.microsoft.com/Offline-Servicing-of-VHDs-df776bda#content – Offline Servicing of VHDs against WSUS
• Use Configuration Manager 2012 R2: Configuration Manager 2012 R2 has a VHD patching feature that allows you to apply software updates to VHDs that you created using task sequences. While this requires Configuration Manager, it is a great option for offline servicing. More information on this can be found here: http://technet.microsoft.com/en-us/library/dn448591.aspx
• Orchestration: You can use a solution provided by a SMA (Service Management Automation) Runbook. The following blog posts talks about a feature in the gallery that allows you to automate the process of offline servicing: http://blogs.technet.com/b/privatecloud/archive/2013/12/07/orchestrated-vm-patching.aspx The specific runbook is found in the Technet gallery here: http://gallery.technet.microsoft.com/Orchestrated-Offline-VM-c90492db

 

P2V

The built-in Physical to Virtual conversion component of VMM was deprecated with the release of SCVMM 2012 R2. I wrote about this and the alternative options earlier this year: http://blogs.technet.com/b/gladiatormsft/archive/2014/01/18/virtualize-but-how-p2v-i-thought-you-took-that-feature-out-of-vmm.aspx Many had hoped the feature would be included in the release of the Microsoft Virtual Machine Conversion utility (MVMC 2.0 ) http://blogs.technet.com/b/gladiatormsft/archive/2014/04/12/the-microsoft-virtual-machine-converter-2-0-is-now-live-on-the-microsoft-download-center.aspx but this was an erroneous speculation. P2V will be returning with the MVMC 3.0 release that will likely come later this fall. In the meantime use Disk2VHD as I mentioned in my post earlier as a viable alternative.

VMM Network Builder

Getting virtual networks set up properly in VMM and having everything in sync with the Hyper-V virtual switches, Host configurations, and the underlying switch fabric can be a cumbersome task. Up until now, I have been longing for a simplification of the process of setting up networking in VMM. Now we have the greatest single add-on utility (in my opinion) to come to SCVMM 2012: The VMM Network Builder. This is a free download that just became available from the Download Center (http://www.microsoft.com/en-us/download/details.aspx?id=43975) This is a tool that will simplify the process of creating virtual networks that utilize VLAN isolation through VMM.

This will ensure that the Host NICs have the proper consistent settings for all of your virtual networks so all of your virtual machines will be able to be set properly to the appropriate virtual network associated with the correct VLAN. This will reduce the instances of having to troubleshoot network configuration which can be a common pain point given the many levels where things can be set incorrectly. With this utility, you can do a simple basic networking setup that can be applied to all of your hosts.

Online Event: Virtualizing Your Data Center with Hyper-V and System Center

Wednesday, February 19th from 9am – 5pm PST

If you're new to virtualization, or if you have some experience and want to see the latest R2 features of Windows Server 2012 Hyper-V or Virtual Machine Manager, join us for a day of free online training with live Q&A to get all your questions answered. Learn how to build your infrastructure from the ground up on the Microsoft stack, using System Center to provide powerful management capabilities. Microsoft virtualization experts Symon Perriman and Matt McSpirit (who are also VMware Certified Professionals) demonstrate how you can help your business consolidate workloads and improve server utilization, while reducing costs. Learn the differences between the platforms, and explore how System Center can be used to manage a multi-hypervisor environment, looking at VMware vSphere 5.5 management, monitoring, automation, and migration. Even if you cannot attend the live event, register today anyway and you will get an email once we release the videos for on-demand replay!  

Topics include:

•    Introduction to Microsoft Virtualization
•    Host Configuration
•    Virtual Machine Clustering and Resiliency
•    Virtual Machine Configuration
•    Virtual Machine Mobility
•    Virtual Machine Replication and Protection
•    Network Virtualization
•    Virtual Machine and Service Templates
•    Private Clouds and User Roles
•    System Center 2012 R2 Data Center
•    Virtualization with the Hybrid Cloud
•    VMware Management, Integration, and Migration

Register here: https://aka.ms/virtDC

Virtualize! But How? P2V? I thought you took that feature out of VMM!

UPDATE: 10/21/2014: The MVMC 3.0 is now released with P2V functionality restored.

http://www.microsoft.com/en-us/download/details.aspx?id=42497

One of the most asked questions I have received regarding the latest release of VMM 2012 has been “Where is the P2V conversion feature? Did it go away?” Well yes, The P2V (Physical-to-Virtual) conversion process was deprecated from System Center Virtual Machine Manager 2012 R2. My colleague Mike Briggs wrote about this in an excellent blog: http://blogs.technet.com/b/scvmm/archive/2013/10/03/how-to-perform-a-p2v-in-a-scvmm-2012-r2-environment.aspx  which also outlines a very specific process with a very helpful architectural diagram to get around this for those who still need to get servers migrated.

What was the objective of P2V?

The primary reason for P2V was to bring legacy physical servers into private cloud. Now that many of the operating systems that were commonly virtualized (Windows 2000 Server, Windows Server 2003, Windows Server 2003, and Windows Server 2008) are either out of support or in extended support. Newer operating systems housing business workloads are more hypervisor-aware and more often than not these days, they are being deployed to private clouds and corporate data centers as provisioned virtual machines as opposed to being deployed directly to bare metal installations.

Alternatives:

If you are planning to deploy Virtual Machine Manager 2012 R2 as part of your System Center hybrid/private cloud and are looking for a strategy to migrate all of your physical servers aboard, consider you have these options:

Use VMM 2012 SP1 in parallel: Setting up a parallel VMM server (just like specified in the above mentioned blog post) and a Hyper-V Host Running Windows Server 2012 can still serve as a viable staging environment as this will continue to be supported in the immediate future. Windows Server 2012 Hyper-V VM’s can be easily imported/migrated over to Windows Server 2012 R2. This will be especially important if you are looking to do on-line P2V’s with active servers and you want to minimize downtime. Documentation for performing P2V in VMM 2012 SP1 can be found here: http://technet.microsoft.com/en-us/library/hh427286.aspx

Disk2VHD: This free utility from the Sysinternals group (http://technet.microsoft.com/en-us/sysinternals/ee656415) can also perform online P2V’s completely originating from the source operating system creating a VHD or VHDX file ready for virtualization prime time. The latest version adds support for VHDX-formatted VHDs, WinRE volumes, removable media (for capturing,) and includes an option to capture live volumes instead of relying on volume shadow copy (VSS).

3^rd-Party Solutions: There are P2V converters out there. Some are rudimentary and inexpensive while others integrate this feature into their enterprise backup and recovery solutions (such as Acronis.) These are also viable options to consider.

Feed of Windows Azure Pack Gallery Resources and SCVMM Service Templates is now LIVE

The feed of Windows Azure Pack Gallery Resources and SCVMM Service Templates is now LIVE at:

http://www.microsoft.com/web/webpi/partners/servicemodels.xml

The initial set of service models are:

• Gallery Resources
  • o Windows Server 2012
  • o Windows Server 2012 R2
  • o Windows Server 2012 WebServer (IIS)
• SCVMM Service Templates
  • o Sharepoint 2013
  • o Service Template Example Kit

You can enable this feed and download your first service model in just a few easy steps (and the first 2 steps are one-time overhead).

1. Install the Microsoft Web Platform Installer from here: http://www.microsoft.com/web/downloads/platform.aspx
2. Add the Service Model Feed as a custom feed
3. Launch the Web Platform Installer
4. Select the Options link at the bottom right, next to the Install button
5. Enter the Feed URL into the Custom Feeds field  http://www.microsoft.com/web/webpi/partners/servicemodels.xml
6. Select the Add feed button
7. Select the OK button
8. You will now see a new Service Modelslink at the upper right of the Web Platform Installer UI
   1. Select the Service Model you want to download
9. Select the Service Models link at the top of the Web Platform Installer UI
10. Select the Add button next to whichever Service Model you would like to download
11. Select the Installbutton
    1. Accept the usage terms
12. Select the I Accept button
13. Select the Continue button
14. Select the Finish button
15. A Windows Explorer window will open, displaying the contents of the Service Model.  Service Models are extracted into your %SystemDrive% folder, according to type
    1. Gallery Resources – %SystemDrive%GalleryResources<resourcename>
    2. SCVMM Service Templates – %SystemDriveSCVMM Service Templates<resourcename>
    3. Follow the directions in the Service Model readme to load the model and prepare any dependent resources (VHDs, etc) for deployment.

 

 

Software Update to block RSA keylengths <1024 has been Released to the Download Center

Today (August 14th) an update was released that, once applied, will block RSA certificates with keys less than 1024 bits. The software update was released to the Download Center.

The security advisory is located at http://technet.microsoft.com/security/advisory/2661254.
The KB article is available at http://support.microsoft.com/kb/2661254.
 
The update is available now to allow organizations to assess the impact of this update and to reissue certificates with larger key sizes, if necessary, before the update is sent out through Windows Update. Previous blogs may have mentioned it being released to Windows Update this month. That is no longer the case. The update is planned to be sent out through Windows Update on October 9, 2012.

Please refer to the KB article for direct links to download the update for your supported version of Windows.

The Virtual Machine Servicing Tool 2012 is now available!

The latest version of the VMST has been released! The Virtual Machine Servicing Tool (VMST) 2012 coincides with System Center 2012 –Virtual Machine Manager (VMM), System Center 2012 Configuration Manager and Windows Server Update Services (WSUS) 3.0 SP2.

VMST 2012 is designed to help you reduce IT costs by providing a means to service your virtual machines, templates, and virtual hard disks offline with the latest operating system and application patches—without introducing vulnerabilities into your IT infrastructure.This has been a very popular solution accelerator from Microsoft and can be downloaded at the following URL:

http://www.microsoft.com/en-us/download/details.aspx?id=30470

You use different features in the Virtual Machine Servicing Tool to update offline virtual machines in a VMM library, a stopped virtual machine on a host, virtual machine templates, and to make updates directly to virtual hard disks (VHDs).

 

Bringing Legacy Blog Back to Cover Legacy Products

Just about a year ago, I moved all new posts over to Technet.com. In spite of that, this blog still continues to get much attention due to a lot of the existing content proving to be very useful for users. For that I am extremely happy to help and it recently gave me an idea. I have been mulling over how I should focus my current blog over at Technet with regards to information, guidance, and support tips. While I have a lot of great information coming (a lot of new products/product versions in the pipeline) I also have a wealth of information I’ve been needing to post tat was related to existing products and legacy products (Softgrid/App-V 4.x/MED-V V1, etc.) I also realize there is a strong user community and install base still present who may not be moving off until the products get closer to end of life.

– Steve Thomas

With this said, I decided that I would use this blog on WordPress in the future for legacy product information (App-V 4.x/Softgrid/MED-V V1/VMM 2008/VPC) while keeping my blog over at Technet more related to current and forward technologies (App-V 5.0/UE-V/Hyper-V 2012/Win8/Win2012.)

SCVMM 2008 R2: VMM Service Crash with UnauthorizedAccessException while Gathering Information on the Source Machine

Symptoms

When click clicking “SCAN SYSTEM” in the SCVMM P2V (Physical to Virtual) conversion wizard, the following error occurs:

“The connection to the Virtual Machine Manager server <SERVERNAME> was lost.”

 The job results will show the following error:

 “Error 1700”

“The Virtual Machine Manager service on the <HOSTNAME> server stopped while this job was running. This may have been caused by a system restart.”

Further analysis will reveal that there was not a system restart, but a VMM Service crash and restart.

Cause:

This can occur if there is a failure to access WMI on the source server or the source machine fails to initialize its WMI classes properly. You can verify specifics by browsing to some of the required WMI classes (Win32_processor, Win32_operating system, etc.) using a WMI browser (WMIC, WBEMTEST, etc.)

For example, you can test basic access by using the Computer management snap-in. Browse to Service and Applicatons – then to WMI control. Then right click and select properties and attempt to browse the common namespaces.

Resolution

If WMI returns error messages, be aware that they may not indicate problems in the WMI service or in WMI providers. Failures can originate in other parts of the operating system and emerge as errors through WMI. See the URL below with common error codes

http://msdn.microsoft.com/en-us/library/aa394559.aspx

• 0x800410xx and 0x800440xx are WMI Operational Errors
• 0x8007xxxx are Win32 errors, not WMI errors (Often DCOM security-related)
• 0x80040xxx are DCOM errors, not WMI errors (Often DCOM configuration-related)
• 0x80005xxx are ADSI errors (LDAP), not WMI errors

There is also a web site that contains these codes in detail:

http://msdn.microsoft.com/en-us/library/aa394559.aspx

If the entire WMI repository is corrupt, try salvaging and/or rebuilding the repository using the following method:

1.  Open an elevated command prompt.

2.    Verify the WMI repository is not corrupt by running the following command:

 winmgmt /verifyrepository

If the repository is not corrupted, a “WMI Repository is consistent” message will be returned. If you get something else, go to step 3. If the repository is consistent, perform a repair.

3. Run the following commands to repair WMI:

 winmgmt /salvagerepository

If the repository salvage fails to work, then run the following command to see if it resolves the issue:

winmgmt /resetrepository

After the last command, there should be a “WMI Repository has been reset” message returned that verifies the command was successful.

SCVMM: Avoid STOP 07XB Issues Post-P2V Migration by Ensuring Registry Hooks to Physical Storage Filters are Removed

Sometimes P2V migrations fail in SCVMM after the 70% completion phase. This is usually post copy/post virtualization fixup. This could also be a situation where it seems the job is hanging at 70% in the VMM console.  Another symptom that might be noticed would be that the P2V job has completed w/ Info. Often you will see this error:

Warning (13210)

Timeout occurred while waiting for VM integration services to be installed on virtual machine <name> residing on host <host>

Recommended Action

Ensure that the version of the VM integration services binaries matches the version of the guest operating system in the virtual machine and then try again.

The issue can be confirmed by starting the VM in Hyper-V. This will show it crashing upon bootup with a STOP 0x7B error. This may occur if certain registry entries that are added by 3^rd-party storage providers or physical hardware providers when these were installed are not removed because the software was not uninstalled prior to P2V process, the software was installed but these settings were not removed prior to P2V, or these were not removed during the P2V fixup phase. These registry entries are preventing your computer from starting correctly.

When this happens, you can do one of two things. Prior to P2V on the source machine search your registries for “UpperFilters” and “LowerFilters.” You can usually track down culprits of post P2V STOP 0x7B’s this way.

For example, you may find the following:

HKLMSYSTEMCurrentControlSetControlClass{71A27CDD-812A-11D0-BEC7-08002BE2092F}

UpperFilters = snapman timounter

Or

HKLMSYSTEMCurrentControlSetControlCriticalDeviceDatabasegendisk

LowerFilters = ZetSFD

Simplay removing these values prior to running a P2V, will resolve the issue although this could be time consuming. You could also mount the VHD in another OS (such as Windows 7 or Windows Server 2008 R2) and load the hives. Then you can search and make the same changes. Bear in mind, you will need to make sure the right ControlSet hive is actually modified (check HKEY_LOCAL_MACHINESYSTEMSelect first) or modify all ControlSets.