Data Center Modernization has definitely reached critical mass. The message that came from TechEd 2013 was “It’s time to make Hybrid Cloud Real.” That, of course, starts with the modernizing your data center to be able to implement private clouds. On top of that, more and more data centers are migrating their hypervisors to Hyper-V in spite of the greater footprint a full Windows Server operating system has on the bare metal. The feature parity as well as cost savings that comes from Hyper-V as a feature (and the subsequent removal of the VMWare tax) offsets the hassle of the additional footprint.
Windows Server bare metal hosts running Hyper-V, like other hypervisors, support SAN boot of the operating system drive using iSCSI. It is important to realize that the iSCSI services depend on the underlying storage and iSCSI network being provisioned properly to accommodate the eccentricities of how Windows boots from SAN using network interface cards in place of traditional storage adapters or HBAs.
Understand the Supportability Parameters
The supportability of the storage support comes from the storage vendor. This also extends to iSCSI boot SAN scenarios per the KB article: https://support.microsoft.com/en-us/kb/305547/en-us. Even though the article does not mention Windows Server 2012 (or R2) it is still in place. Normally, this would not be complicated but in the case of iSCSI networks, the device may likely be using a NIC to locate the storage (especially if they are actually using NAS – network attached storage – i.e. NetApp) and not a traditional storage adapter or HBA.
Slipstream your 3rd-party drivers if possible
The use of slipstreamed NIC/Storage drivers in the installation ISO will prevent any timing issues from swapping back and forth between driver media and OS media. The may be especially the case if you are controlling headless blade devices using KVM or some other solution. I have found that this resolves many of the issues outlined in this particular KB: https://support.microsoft.com/en-us/kb/2826787 – as well as the 0x80070057 error message when trying to format drives or create partitions during the operating system setup.
No Thin-Provisioning LUNs for the OS Boot Drive
LUNS on the NAS devices (i.e. NetApp Devices) need to be thick provisioned for the drive containing the OS instead of thin-provisioned. In addition LUNS for the host OS boot volume only should be 127GB or less. Remember this is only in the context of the LUN being used for host devices iSCSI boot volume.
Avoid using Default Gateways for iSCSI NICs
The NICs configured for the iSCSI SAN should avoid having a default gateway. This can cause issues such as slow throughput occurring during formatting of disks and the copying of files during installation. This has been an issue with the Windows iSCSI initiator in the past and has previous appeared in KB articles:
960104: If you start a system from iSCSI, the gateway specified in the iSCSI Boot solution will always be used by Windows to communicate with the iSCSI Target
2727330: Default gateway is set to 0.0.0.0 if you start a Windows Vista-based, Windows 7-based, Windows Server 2008-based or Windows Server 2008 R2-based computer from an iSCSI boot device
In addition, the network ports connecting to the boot volume iSCSI interfaces on the iSCSI network’s switch should have ICMP redirect disabled.
If all else fails . . . revert to the old way!
If the interactive installation still fails, remember – there is the legacy way of deploying Windows Servers in an iSCSI SAN boot configuration outlined in:
UPDATE: 10/21/2014: The MVMC 3.0 is now released with P2V functionality restored.
I work with SCVMM (System Center Virtual Machine Manager) frequently in many different contexts. I even do the occasional private cloud engagement specifically on VMM and Hyper-V. Most of the time however, I am using VMM in a peripheral context – be it personal lab work, proof-of-concept labs for customer or partners, etc. I have been very pleased with the evolution of Hyper-V and System Center products over the last few years. I find the largest issues that create pain points for me involve the constant need to service virtual machines, deal with physical-to-virtual conversions, and the cumbersome process of building test networks that demonstrate elements such as multi-tenancy that require me to super impose logical switches and other elements of software-defined networking on top of my switching fabric.
I field a lot of questions with regards to how to best go about these options with the most recent versions of SCVMM (particularly VMM 2012 R2.)
Virtual Machine Servicing
I don’t keep all of virtual machines running at the same time. In addition, I have many templates for which I reuse/import/export on a regular basis. In VMM 2012 there was the option of using a separate add-on utility called the Virtual Machine Servicing Tool. The problem is it only was for VMM 2012 RTM (or R1) and it does not work with VMM 2012 SP1 or VMM 202 R2. You will likely find many questions regarding this that appear in the comments section on my initial blog about the VSMT 2012 utility way back in 2012. http://blogs.technet.com/b/gladiatormsft/archive/2012/08/14/the-virtual-machine-servicing-tool-2012-is-now-available.aspx
So with there being no newer version of VSMT for 2012 SP1 or R2 and the fact that you cannot use VSMT 2012 on VMM 2012 SP1 or R2, what are your options going forward for servicing – particularly offline servicing? You have a few options:
- Customize a solution with DISM (Deployment Image Service and Management Toolkit) You should be very familiar with DISM as it is very useful for the consultant and IT Pro (like me) who does not always have access to System Center infrastructures. It can also be easily scripted to mount and service offline images for OS updates at the very least. You can become familiar with DISM servicing using the following link as it is a great introduction to the concept: http://technet.microsoft.com/en-us/library/dd799267(v=WS.10).aspx. This walkthrough tells you how to mount a virtual disk online and then apply various servicing commands using the DISM tool. You can then apply updates using the tool to apply individual Windows Update packages (.MSU’s) although this can be cumbersome for many sets of updates. This does require scripting for effectiveness but I have found that I can get away with one set per OS so long as I have access to the individual .MSU files [DISM /image:C:MyDirMount /Add-Package /Packagepath:<file_path>] This way is still way quicker than standing up a VM running WSUS, keeping it in sync and then booting up every single VM and updating it through the WSUS server. There are also additional scripts out there that work with live WSUS servers and DISM that you can also try – for example http://gallery.technet.microsoft.com/Offline-Servicing-of-VHDs-df776bda#content – Offline Servicing of VHDs against WSUS
- Use Configuration Manager 2012 R2: Configuration Manager 2012 R2 has a VHD patching feature that allows you to apply software updates to VHDs that you created using task sequences. While this requires Configuration Manager, it is a great option for offline servicing. More information on this can be found here: http://technet.microsoft.com/en-us/library/dn448591.aspx
- Orchestration: You can use a solution provided by a SMA (Service Management Automation) Runbook. The following blog posts talks about a feature in the gallery that allows you to automate the process of offline servicing: http://blogs.technet.com/b/privatecloud/archive/2013/12/07/orchestrated-vm-patching.aspx The specific runbook is found in the Technet gallery here: http://gallery.technet.microsoft.com/Orchestrated-Offline-VM-c90492db
The built-in Physical to Virtual conversion component of VMM was deprecated with the release of SCVMM 2012 R2. I wrote about this and the alternative options earlier this year: http://blogs.technet.com/b/gladiatormsft/archive/2014/01/18/virtualize-but-how-p2v-i-thought-you-took-that-feature-out-of-vmm.aspx Many had hoped the feature would be included in the release of the Microsoft Virtual Machine Conversion utility (MVMC 2.0 ) http://blogs.technet.com/b/gladiatormsft/archive/2014/04/12/the-microsoft-virtual-machine-converter-2-0-is-now-live-on-the-microsoft-download-center.aspx but this was an erroneous speculation. P2V will be returning with the MVMC 3.0 release that will likely come later this fall. In the meantime use Disk2VHD as I mentioned in my post earlier as a viable alternative.
VMM Network Builder
Getting virtual networks set up properly in VMM and having everything in sync with the Hyper-V virtual switches, Host configurations, and the underlying switch fabric can be a cumbersome task. Up until now, I have been longing for a simplification of the process of setting up networking in VMM. Now we have the greatest single add-on utility (in my opinion) to come to SCVMM 2012: The VMM Network Builder. This is a free download that just became available from the Download Center (http://www.microsoft.com/en-us/download/details.aspx?id=43975) This is a tool that will simplify the process of creating virtual networks that utilize VLAN isolation through VMM.
This will ensure that the Host NICs have the proper consistent settings for all of your virtual networks so all of your virtual machines will be able to be set properly to the appropriate virtual network associated with the correct VLAN. This will reduce the instances of having to troubleshoot network configuration which can be a common pain point given the many levels where things can be set incorrectly. With this utility, you can do a simple basic networking setup that can be applied to all of your hosts.
UPDATE: 10/21/2014: The MVMC 3.0 is now released with P2V functionality restored.
The Microsoft Virtual Machine Converter 2.0 is available! The Microsoft Virtual Machine Converter provides a supported, freely available, stand-alone solution for converting VMware-based virtual machines and virtual disks to Hyper-V-based virtual machines and virtual hard disks (VHDs).
There is also a release of an update to the Migration Automation Toolkit (MAT). This is a collection of PowerShell scripts that will automate conversions using MVMC. You can use it to convert several machines at once, on a single server – or scale it out and execute conversions on many servers at the same time.
With the release, you will be able to access many new features including:
- On-premises VM to Azure VM conversion
- PowerShell interface for scripting and automation support
- Added support for vCenter & ESX(i) 4.1, 5.0 and now 5.5
- VMware virtual hardware version 4 – 10 support
- Linux Guest OS support including CentOS, Debian, Oracle, Red Hat Enterprise, SuSE enterprise and Ubuntu.
- Migration Automation Toolkit support for MVMC 2.0
Migration Automation Toolkit (MAT)
MVMC Converter Download
Wednesday, February 19th from 9am – 5pm PST
If you're new to virtualization, or if you have some experience and want to see the latest R2 features of Windows Server 2012 Hyper-V or Virtual Machine Manager, join us for a day of free online training with live Q&A to get all your questions answered. Learn how to build your infrastructure from the ground up on the Microsoft stack, using System Center to provide powerful management capabilities. Microsoft virtualization experts Symon Perriman and Matt McSpirit (who are also VMware Certified Professionals) demonstrate how you can help your business consolidate workloads and improve server utilization, while reducing costs. Learn the differences between the platforms, and explore how System Center can be used to manage a multi-hypervisor environment, looking at VMware vSphere 5.5 management, monitoring, automation, and migration. Even if you cannot attend the live event, register today anyway and you will get an email once we release the videos for on-demand replay!
• Introduction to Microsoft Virtualization
• Host Configuration
• Virtual Machine Clustering and Resiliency
• Virtual Machine Configuration
• Virtual Machine Mobility
• Virtual Machine Replication and Protection
• Network Virtualization
• Virtual Machine and Service Templates
• Private Clouds and User Roles
• System Center 2012 R2 Data Center
• Virtualization with the Hybrid Cloud
• VMware Management, Integration, and Migration
Register here: https://aka.ms/virtDC
UPDATE: 10/21/2014: The MVMC 3.0 is now released with P2V functionality restored.
One of the most asked questions I have received regarding the latest release of VMM 2012 has been “Where is the P2V conversion feature? Did it go away?” Well yes, The P2V (Physical-to-Virtual) conversion process was deprecated from System Center Virtual Machine Manager 2012 R2. My colleague Mike Briggs wrote about this in an excellent blog: http://blogs.technet.com/b/scvmm/archive/2013/10/03/how-to-perform-a-p2v-in-a-scvmm-2012-r2-environment.aspx which also outlines a very specific process with a very helpful architectural diagram to get around this for those who still need to get servers migrated.
What was the objective of P2V?
The primary reason for P2V was to bring legacy physical servers into private cloud. Now that many of the operating systems that were commonly virtualized (Windows 2000 Server, Windows Server 2003, Windows Server 2003, and Windows Server 2008) are either out of support or in extended support. Newer operating systems housing business workloads are more hypervisor-aware and more often than not these days, they are being deployed to private clouds and corporate data centers as provisioned virtual machines as opposed to being deployed directly to bare metal installations.
If you are planning to deploy Virtual Machine Manager 2012 R2 as part of your System Center hybrid/private cloud and are looking for a strategy to migrate all of your physical servers aboard, consider you have these options:
Use VMM 2012 SP1 in parallel: Setting up a parallel VMM server (just like specified in the above mentioned blog post) and a Hyper-V Host Running Windows Server 2012 can still serve as a viable staging environment as this will continue to be supported in the immediate future. Windows Server 2012 Hyper-V VM’s can be easily imported/migrated over to Windows Server 2012 R2. This will be especially important if you are looking to do on-line P2V’s with active servers and you want to minimize downtime. Documentation for performing P2V in VMM 2012 SP1 can be found here: http://technet.microsoft.com/en-us/library/hh427286.aspx
Disk2VHD: This free utility from the Sysinternals group (http://technet.microsoft.com/en-us/sysinternals/ee656415) can also perform online P2V’s completely originating from the source operating system creating a VHD or VHDX file ready for virtualization prime time. The latest version adds support for VHDX-formatted VHDs, WinRE volumes, removable media (for capturing,) and includes an option to capture live volumes instead of relying on volume shadow copy (VSS).
3rd-Party Solutions: There are P2V converters out there. Some are rudimentary and inexpensive while others integrate this feature into their enterprise backup and recovery solutions (such as Acronis.) These are also viable options to consider.
I recently had an issue where I encountered something quite bizarre. In an effort to reduce size on disk of dynamically expanding virtual hard disks (VHDs) I found myself feeling like I needed to take medication. After sysprepping an operating system image on the disk, the current file consumption on the disk was approximately 12 GB and the size of the dynamically expanding virtual hard disk was 15 GB (with a capability of growing to 127 GB as that was the size designated when creating the VHD.)
I then mounted the disk as a drive in Windows 7 using diskpart.exe in order to perform some offline defragmentation, pre-compaction, and compaction. I found that after disk defragmentation was completed in Windows 7, the total size of the disk grew to the full 127 GB although the total file consumption the disk was still at 12 GB. I had never encountered this before. I have, in the past, seen defragmentation cause some gain in VHD size but only ever at a maximum of 10-20%. To top that off, pre-compaction and compaction did nothing to reduce the size.
Now, just to give some background, the Windows 7 virtual stack used to mount VHDs did not understand the TRIM command (which is what the file system started sending down in Win7 to let the storage stack know an area was no longer in use). Anytime defragmentation is run on a dynamically expanding VHD where the stack doesn’t understand TRIM will in fact result in a larger VHD than you started with. BUT NOT THIS MUCH! I even verified that volume snapshots were disabled on the volume as that can also explain a large increase in the size of the VHD.
Realizing this was done on a host machine using a customer’s corporate operating system image, I took a copy of the original VHD and mounted and defragmented the disk on one of my plain vanilla operating system images and found the behavior I expected – only a nominal increase in size. At this point, I realized something outside the norm of the operating system was causing this growth. I could have easily done the tedious approach of removing individual 3rd-party filters on the image (using the divide-and-conquer method) while running defragmentation but I wanted to see if what was doing this was even related to defragmentation. I decided to simply just mount the drive again and monitor the disk size while doing absolutely nothing interactively to affect the drive.
I went to lunch. I came back, the disk was already at 32 GB. By the end of the afternoon, it was back to 127 GB. There was obviously some file-system based software performing this. It turns out, there is a McAfee Encryption policy in place (they were running 3rd-party disk encryption software) that silently encrypts new logical drives as they are added. When I mounted the VHD through Windows 7’s Disk again while this software had been disengaged, the issue did not occur.
I hadn’t been taking crazy pills after all.
Join Microsoft MVP Pete Zerger for a free two-day Jump Start series covering the end-to-end process of implementing a Microsoft cloud solution.
Registration is free for both days: Sign-up now.
This June 18 Jump Start will be a scenario-based, bottoms-up approach to designing and building our private cloud on Windows Server 2012 and incorporating the full spectrum of System Center 2012 SP1 components.
MVP Pete Zerger and Microsoft Sr. Technical Evangelist Symon Perriman will focus on bringing a greater understanding to key topics related to the fabric – such as virtual networking, leveraging the storage and networking capabilities of WS2012, and creating service templates in VMM and on the Service Manager CMDB as we move up the management stack. Additionally, you will learn how the service-catalog comes together to deliver an intuitive self-service experience in a step-by-step approach that will address many common questions.
This June 20 Jump Start is a continuation of Day 1 and will focus on successfully monitoring and managing ongoing operation of a private cloud environment.
MVP Pete Zerger and Technical Product Manager Matt McSpirit will provide examples of how to integrate Windows Azure IaaS into our private cloud to deliver hybrid cloud capabilities in System Center 2012 SP1, explain how to develop hybrid cloud self-service scenarios in System Center 2012 App Controller and in the System Center Service Manager Self-Service Portal, and demonstrate full integration of private and public cloud with ITIL.