Data Center Modernization has definitely reached critical mass. The message that came from TechEd 2013 was “It’s time to make Hybrid Cloud Real.” That, of course, starts with the modernizing your data center to be able to implement private clouds. On top of that, more and more data centers are migrating their hypervisors to Hyper-V in spite of the greater footprint a full Windows Server operating system has on the bare metal. The feature parity as well as cost savings that comes from Hyper-V as a feature (and the subsequent removal of the VMWare tax) offsets the hassle of the additional footprint.
Windows Server bare metal hosts running Hyper-V, like other hypervisors, support SAN boot of the operating system drive using iSCSI. It is important to realize that the iSCSI services depend on the underlying storage and iSCSI network being provisioned properly to accommodate the eccentricities of how Windows boots from SAN using network interface cards in place of traditional storage adapters or HBAs.
Understand the Supportability Parameters
The supportability of the storage support comes from the storage vendor. This also extends to iSCSI boot SAN scenarios per the KB article: https://support.microsoft.com/en-us/kb/305547/en-us. Even though the article does not mention Windows Server 2012 (or R2) it is still in place. Normally, this would not be complicated but in the case of iSCSI networks, the device may likely be using a NIC to locate the storage (especially if they are actually using NAS – network attached storage – i.e. NetApp) and not a traditional storage adapter or HBA.
Slipstream your 3rd-party drivers if possible
The use of slipstreamed NIC/Storage drivers in the installation ISO will prevent any timing issues from swapping back and forth between driver media and OS media. The may be especially the case if you are controlling headless blade devices using KVM or some other solution. I have found that this resolves many of the issues outlined in this particular KB: https://support.microsoft.com/en-us/kb/2826787 – as well as the 0x80070057 error message when trying to format drives or create partitions during the operating system setup.
No Thin-Provisioning LUNs for the OS Boot Drive
LUNS on the NAS devices (i.e. NetApp Devices) need to be thick provisioned for the drive containing the OS instead of thin-provisioned. In addition LUNS for the host OS boot volume only should be 127GB or less. Remember this is only in the context of the LUN being used for host devices iSCSI boot volume.
Avoid using Default Gateways for iSCSI NICs
The NICs configured for the iSCSI SAN should avoid having a default gateway. This can cause issues such as slow throughput occurring during formatting of disks and the copying of files during installation. This has been an issue with the Windows iSCSI initiator in the past and has previous appeared in KB articles:
960104: If you start a system from iSCSI, the gateway specified in the iSCSI Boot solution will always be used by Windows to communicate with the iSCSI Target
2727330: Default gateway is set to 0.0.0.0 if you start a Windows Vista-based, Windows 7-based, Windows Server 2008-based or Windows Server 2008 R2-based computer from an iSCSI boot device
In addition, the network ports connecting to the boot volume iSCSI interfaces on the iSCSI network’s switch should have ICMP redirect disabled.
If all else fails . . . revert to the old way!
If the interactive installation still fails, remember – there is the legacy way of deploying Windows Servers in an iSCSI SAN boot configuration outlined in:
UPDATE: 10/21/2014: The MVMC 3.0 is now released with P2V functionality restored.
I work with SCVMM (System Center Virtual Machine Manager) frequently in many different contexts. I even do the occasional private cloud engagement specifically on VMM and Hyper-V. Most of the time however, I am using VMM in a peripheral context – be it personal lab work, proof-of-concept labs for customer or partners, etc. I have been very pleased with the evolution of Hyper-V and System Center products over the last few years. I find the largest issues that create pain points for me involve the constant need to service virtual machines, deal with physical-to-virtual conversions, and the cumbersome process of building test networks that demonstrate elements such as multi-tenancy that require me to super impose logical switches and other elements of software-defined networking on top of my switching fabric.
I field a lot of questions with regards to how to best go about these options with the most recent versions of SCVMM (particularly VMM 2012 R2.)
Virtual Machine Servicing
I don’t keep all of virtual machines running at the same time. In addition, I have many templates for which I reuse/import/export on a regular basis. In VMM 2012 there was the option of using a separate add-on utility called the Virtual Machine Servicing Tool. The problem is it only was for VMM 2012 RTM (or R1) and it does not work with VMM 2012 SP1 or VMM 202 R2. You will likely find many questions regarding this that appear in the comments section on my initial blog about the VSMT 2012 utility way back in 2012. http://blogs.technet.com/b/gladiatormsft/archive/2012/08/14/the-virtual-machine-servicing-tool-2012-is-now-available.aspx
So with there being no newer version of VSMT for 2012 SP1 or R2 and the fact that you cannot use VSMT 2012 on VMM 2012 SP1 or R2, what are your options going forward for servicing – particularly offline servicing? You have a few options:
- Customize a solution with DISM (Deployment Image Service and Management Toolkit) You should be very familiar with DISM as it is very useful for the consultant and IT Pro (like me) who does not always have access to System Center infrastructures. It can also be easily scripted to mount and service offline images for OS updates at the very least. You can become familiar with DISM servicing using the following link as it is a great introduction to the concept: http://technet.microsoft.com/en-us/library/dd799267(v=WS.10).aspx. This walkthrough tells you how to mount a virtual disk online and then apply various servicing commands using the DISM tool. You can then apply updates using the tool to apply individual Windows Update packages (.MSU’s) although this can be cumbersome for many sets of updates. This does require scripting for effectiveness but I have found that I can get away with one set per OS so long as I have access to the individual .MSU files [DISM /image:C:MyDirMount /Add-Package /Packagepath:<file_path>] This way is still way quicker than standing up a VM running WSUS, keeping it in sync and then booting up every single VM and updating it through the WSUS server. There are also additional scripts out there that work with live WSUS servers and DISM that you can also try – for example http://gallery.technet.microsoft.com/Offline-Servicing-of-VHDs-df776bda#content – Offline Servicing of VHDs against WSUS
- Use Configuration Manager 2012 R2: Configuration Manager 2012 R2 has a VHD patching feature that allows you to apply software updates to VHDs that you created using task sequences. While this requires Configuration Manager, it is a great option for offline servicing. More information on this can be found here: http://technet.microsoft.com/en-us/library/dn448591.aspx
- Orchestration: You can use a solution provided by a SMA (Service Management Automation) Runbook. The following blog posts talks about a feature in the gallery that allows you to automate the process of offline servicing: http://blogs.technet.com/b/privatecloud/archive/2013/12/07/orchestrated-vm-patching.aspx The specific runbook is found in the Technet gallery here: http://gallery.technet.microsoft.com/Orchestrated-Offline-VM-c90492db
The built-in Physical to Virtual conversion component of VMM was deprecated with the release of SCVMM 2012 R2. I wrote about this and the alternative options earlier this year: http://blogs.technet.com/b/gladiatormsft/archive/2014/01/18/virtualize-but-how-p2v-i-thought-you-took-that-feature-out-of-vmm.aspx Many had hoped the feature would be included in the release of the Microsoft Virtual Machine Conversion utility (MVMC 2.0 ) http://blogs.technet.com/b/gladiatormsft/archive/2014/04/12/the-microsoft-virtual-machine-converter-2-0-is-now-live-on-the-microsoft-download-center.aspx but this was an erroneous speculation. P2V will be returning with the MVMC 3.0 release that will likely come later this fall. In the meantime use Disk2VHD as I mentioned in my post earlier as a viable alternative.
VMM Network Builder
Getting virtual networks set up properly in VMM and having everything in sync with the Hyper-V virtual switches, Host configurations, and the underlying switch fabric can be a cumbersome task. Up until now, I have been longing for a simplification of the process of setting up networking in VMM. Now we have the greatest single add-on utility (in my opinion) to come to SCVMM 2012: The VMM Network Builder. This is a free download that just became available from the Download Center (http://www.microsoft.com/en-us/download/details.aspx?id=43975) This is a tool that will simplify the process of creating virtual networks that utilize VLAN isolation through VMM.
This will ensure that the Host NICs have the proper consistent settings for all of your virtual networks so all of your virtual machines will be able to be set properly to the appropriate virtual network associated with the correct VLAN. This will reduce the instances of having to troubleshoot network configuration which can be a common pain point given the many levels where things can be set incorrectly. With this utility, you can do a simple basic networking setup that can be applied to all of your hosts.
UPDATE: 10/21/2014: The MVMC 3.0 is now released with P2V functionality restored.
The Microsoft Virtual Machine Converter 2.0 is available! The Microsoft Virtual Machine Converter provides a supported, freely available, stand-alone solution for converting VMware-based virtual machines and virtual disks to Hyper-V-based virtual machines and virtual hard disks (VHDs).
There is also a release of an update to the Migration Automation Toolkit (MAT). This is a collection of PowerShell scripts that will automate conversions using MVMC. You can use it to convert several machines at once, on a single server – or scale it out and execute conversions on many servers at the same time.
With the release, you will be able to access many new features including:
- On-premises VM to Azure VM conversion
- PowerShell interface for scripting and automation support
- Added support for vCenter & ESX(i) 4.1, 5.0 and now 5.5
- VMware virtual hardware version 4 – 10 support
- Linux Guest OS support including CentOS, Debian, Oracle, Red Hat Enterprise, SuSE enterprise and Ubuntu.
- Migration Automation Toolkit support for MVMC 2.0
Migration Automation Toolkit (MAT)
MVMC Converter Download
Wednesday, February 19th from 9am – 5pm PST
If you're new to virtualization, or if you have some experience and want to see the latest R2 features of Windows Server 2012 Hyper-V or Virtual Machine Manager, join us for a day of free online training with live Q&A to get all your questions answered. Learn how to build your infrastructure from the ground up on the Microsoft stack, using System Center to provide powerful management capabilities. Microsoft virtualization experts Symon Perriman and Matt McSpirit (who are also VMware Certified Professionals) demonstrate how you can help your business consolidate workloads and improve server utilization, while reducing costs. Learn the differences between the platforms, and explore how System Center can be used to manage a multi-hypervisor environment, looking at VMware vSphere 5.5 management, monitoring, automation, and migration. Even if you cannot attend the live event, register today anyway and you will get an email once we release the videos for on-demand replay!
• Introduction to Microsoft Virtualization
• Host Configuration
• Virtual Machine Clustering and Resiliency
• Virtual Machine Configuration
• Virtual Machine Mobility
• Virtual Machine Replication and Protection
• Network Virtualization
• Virtual Machine and Service Templates
• Private Clouds and User Roles
• System Center 2012 R2 Data Center
• Virtualization with the Hybrid Cloud
• VMware Management, Integration, and Migration
Register here: https://aka.ms/virtDC
Often in Virtualization and Management Products like SCVMM, MED-V, Config Manager, UE-V, and App-V the symptom of an issue appears in the respective System Center or MDOP product but the root cause is often caused by an anomaly in an underlying operating system component. Often that component is WMI. For this reason, it is invaluable have a solid understanding of WMI and WMI troubleshooting. WMI is often a component that can cause problems due to one or more of the following WMI issues:
- Corrupted repository
- Incomplete namespace
- Access Denied
- Invalid String in WMI property/data
- Unexpected value
- Memory leak
- Code Defect by WMI Provider
One of the most common errors encountered is error 0x800706BA – RPC Server Unavailable.
This error has context. If it is during connecting to a WMI namespace, it is usually because:
- The machine does not exist.
- The machine cannot respond because the appropriate firewall exceptions have not been made. Check firewall settings.
If it is during operation, it is likely because:
- The client machine doesn’t have correct firewall settings for asynchronous call backs.
- Connecting to a machine which doesn’t exist.
First I would verify the firewall rules. I would make sure the following rules are set:
- WMI (ASync) Properties – In Program: %SYSTEMROOT%\System32\WBEM\unsecapp.exe
- WMI (DCOM) – In Port: TCP 135 Program: %SYSTEMROOT%\System32\svchost.exe
- WMI (WMI) In-Out Program: %SYSTEMROOT%\System32\svchost.exe
I deal with WMI problems all the time. I generally follow this little troubleshooting checklist for RPC errors:
- Use the WMI Control MMC (WMIC.MSC) to ensure that the service is working on the local system.
- If the problem involves communicating with a remote system then use the WMI Control to test the ability to connect to the remote system
- For Access Denied type issues verify that the DCOM and WMI Service settings are at default values, and the Network Service account has been granted impersonation rights.
- Check the service settings if the WMI service fails to start or if client programs cannot communicate with the service. In some cases you may need to reregister all the modules to recover the service.
The feed of Windows Azure Pack Gallery Resources and SCVMM Service Templates which are compatible with the RTM releases is now LIVE
Windows Azure pack information can be found here: http://www.microsoft.com/en-us/server-cloud/products/windows-azure-pack/default.aspx#fbid=w0JHn-YWZV6
The Windows Azure Pack is a collection of Windows Azure technologies available to Microsoft customers at no additional cost. Once installed in your datacenter, the Windows Azure Pack integrates with System Center and Windows Server to help provide a self-service portal for managing services such as websites, Virtual Machines, and Service Bus; a portal for administrators to manage resource clouds; scalable web hosting; and more.
The feed of Windows Azure Pack Gallery Resources and SCVMM Service Templates which are compatible with the RTM releases is now LIVE at:
This release includes more than 40 deployable templates to get you started including:
- SharePoint 2013
- SharePoint Foundation 2010
- Microsoft SQL Server 2013
- Domain Controller
- Windows Server 2012 R2 HA Network Gateway
- Windows Server 2012 R2
- Windows Server 2012
- Database Servers
- MySQL Server (Windows and Linux)
- Oracle Self Service Kit
- Linux applications including
- CentOS6 LAMP
- WordPress CentOS6
- Samples to enhance authoring
- Custom Script
- Custom Script with Environment Variables
- Linux Generic
- Service Template Example Kit
- and more
For information on how to use this feed and import items in the Windows Azure Pack you can use the following resources:
- Downloading and Installing Windows Azure Pack Gallery Resource:
- Video: Using the Service Models Web Platform Installer Feed – https://www.youtube.com/watch?v=eAvOUNqqYfc&feature=youtu.be
- Technet: Downloading and Installing Windows Azure Pack Gallery Resource – http://social.technet.microsoft.com/wiki/contents/articles/20194.downloading-and-installing-windows-azure-pack-gallery-resource.aspx
- Video: Virtual Hard Disk Requirements for the Virtual Machine Role Gallery – https://youtu.be/eAvOUNqqYfc
- Technet: System Center 2012 R2 Virtual Machine Role Authoring Guide – http://social.technet.microsoft.com/wiki/contents/articles/18272.system-center-2012-r2-virtual-machine-role-authoring-guide.aspx
The feed of Windows Azure Pack Gallery Resources and SCVMM Service Templates is now LIVE at:
The initial set of service models are:
- Gallery Resources
- o Windows Server 2012
- o Windows Server 2012 R2
- o Windows Server 2012 WebServer (IIS)
- SCVMM Service Templates
- o Sharepoint 2013
- o Service Template Example Kit
You can enable this feed and download your first service model in just a few easy steps (and the first 2 steps are one-time overhead).
- Install the Microsoft Web Platform Installer from here: http://www.microsoft.com/web/downloads/platform.aspx
- Add the Service Model Feed as a custom feed
- Launch the Web Platform Installer
- Select the Options link at the bottom right, next to the Install button
- Enter the Feed URL into the Custom Feeds field http://www.microsoft.com/web/webpi/partners/servicemodels.xml
- Select the Add feed button
- Select the OK button
- You will now see a new Service Modelslink at the upper right of the Web Platform Installer UI
- Select the Service Model you want to download
- Select the Service Models link at the top of the Web Platform Installer UI
- Select the Add button next to whichever Service Model you would like to download
- Select the Installbutton
- Accept the usage terms
- Select the I Accept button
- Select the Continue button
- Select the Finish button
- A Windows Explorer window will open, displaying the contents of the Service Model. Service Models are extracted into your %SystemDrive% folder, according to type
- Gallery Resources – %SystemDrive%GalleryResources<resourcename>
- SCVMM Service Templates – %SystemDriveSCVMM Service Templates<resourcename>
- Follow the directions in the Service Model readme to load the model and prepare any dependent resources (VHDs, etc) for deployment.