To continue my discussion regarding the importance of troubleshooting WMI, I want to move the focus to a devising a targeted approach when troubleshooting so you can optimize the time it takes for you to zero in on the issue.
WMI issues generally fall into the following areas:
Configuration Issues: These are issues relating to the configuration of WMI on the local (or mostly remote) machine including:
• DCOM Security\Permissions or Configuration
• Firewall Configuration
• WMI namespace security
Infrastructure Issues: These are issues related to WMI components including:
• WMI service setup
• DCOM registration problems
• Missing WMI classes
• Improper WMI provider registration
• Missing System files
• WMI repository corruption (*GASP*)
• Deleted WMI repository (*HEADDESK*)
WMI Managed Entity Issues: These may be issues related to the extensible WMI components including:
• Security requirements
• Not running (e.g service, application) or de-installed application
• External dependencies
As I mention in my last article, you obviously want to verify your firewall rules (which are built into versions of Windows since Windows XP.)
WMI (ASync) Properties – In
WMI (DCOM) – In
Port: TCP 135
WMI (WMI) In-Out
Then you will want to zero in on the error itself.
0x800706BA – RPC Server Unavailable
When this error appears during connecting to a WMI namespace:
• The machine does not exist.
• The machine cannot respond because the appropriate firewall exceptions have not been made. Check firewall settings.
When this error appears during operation it could be:
• The client machine doesn’t have correct firewall settings for asynchronous call backs.
• Connecting to a machine which doesn’t exist.
0x80070005 – E_ACCESS_DENIED
When this error occurs during connecting to a WMI namespace –
• The username/password does not exist.
• The user does not have the remote launch or remote activation options set.
• Check dcomcnfg.exe under the COM Security Tab.
When this error occurs during operation –
• The specific user does not have the DCOM permissions.
• Minimum authentication level needed for the namespace is more than what is used.
• Check the settings on the Default Properties tab of DCOMCNFG.EXE.
0x80041003 – WMI Access Denied
During connecting to a WMI namespace – The user does not have the appropriate WMI permissions on a namespace. Check WMIMGMT.EXE and permissions for that namespace.
During operation – Specific user doesn’t have WMI access permissions.
0x80041001 – Unknown Error
Ah, the UNKNOWN ERROR. Often this is cause by a 3rd-party provider or non-OS software that extends the Repository has been either removed from the environment and left WMI subscriptions in the repository or is malfunctioning.
Enable WMI Verbose logging on the server and review the WMI logs in %SYSTEMROOT%\system32\wbem\logs. The Wbemess log will show which WMI subscription was sending notifications when the criteria was met.
You will need to follow the steps below to remove the WMI subscriptions once you isolate them:
1. Click Start, run, type Wbemtest then type root\cimv2\applications\ and click “Connect” button
2. Click on ‘Enum Classes’, click the Recursive radio button, click OK.
3. Scroll down until you see _FilterToConsumerBinding class. Double-click on it.
4. Click the “Instances” button on the right hand side.
5. Choose those you isolated and click on the delete button.
When you retrieve a managed resource in a WMI script, the CIMOM (WMI service) looks for the managed resource’s blueprint (class definition) in the default namespace if no namespace is specified. If the CIMOM cannot find the managed-resource class definition in the default namespace, a WBEM_E_INVALID_CLASS (0x80041010) error is generated.
0x8007000E – Not enough Storage is available to complete this process
This usually indicates a problem with a provider, handle leak, memory leak, or other problem tied to WMI functionality.
1. Use the WMI Control to ensure that the service is working on the local system.
2. If the problem involves communicating with a remote system then use the WMI Control to test the ability to connect to the remote system
3. If the service appears to be working, use verbose logging to see the activity (queries) that is being processed by the service and to identify any problems. You can also use WMICHK and WMIDIAG to check the health of the service and the hosted providers.
4. For Access Denied type issues verify that the DCOM and WMI Service settings are at default values, and the Network Service account has been granted impersonation rights.
5. Check the service settings if the WMI service fails to start or if client programs cannot communicate with the service. In some cases you may need to reregister all the modules to recover the service.
6. If queries appear to be returning an incomplete results set, try increasing the buffer thresholds.
7. If problems persist, make a backup copy of the existing WMI database (repository), and then try building a new one.
If you have been using App-V 5 SP1 with RDS (Remote Desktop Services) or Citrix XenApp servers, you may have noticed that when you installed the App-V 5.0 Service Pack 1 RDS Client using the EXE installer (no extraction) – this creates shortcuts for all 24 language packs on the start menu for Windows Server 2008 R2 machines. This was indeed a mistake which has been corrected with the SP2 EXE installer.
However, you may have noticed the SP2 version of the EXE installer still installs all of the 24 language packs. You will not notice this in any shortcuts but it will be visible in the Programs Control Panel as well as the App-V client installation folder. This is by design as the RDS EXE installer installs all the language packs so as to be able to service users from multiple locales by default. If you only want to install a certain language pack, I would advise for you to extract the MSI for the client and subsequent language packs by using the /LAYOUT and /layoutdir switches to extract the MSI files out.
More information on these switches can be found here: http://technet.microsoft.com/en-US/library/jj713460.aspx
Then you can install the MSI separately and will also allow you to suppress reboots for silent deployments using the /norestart switch. Bear in mind the EXE installer also detects (and applies if not found) some (but not all) prerequisites. Remember these will have to be deployed in advance when installing the App-V Client using the MSI installer.
Often in Virtualization and Management Products like SCVMM, MED-V, Config Manager, UE-V, and App-V the symptom of an issue appears in the respective System Center or MDOP product but the root cause is often caused by an anomaly in an underlying operating system component. Often that component is WMI. For this reason, it is invaluable have a solid understanding of WMI and WMI troubleshooting. WMI is often a component that can cause problems due to one or more of the following WMI issues:
- Corrupted repository
- Incomplete namespace
- Access Denied
- Invalid String in WMI property/data
- Unexpected value
- Memory leak
- Code Defect by WMI Provider
One of the most common errors encountered is error 0x800706BA – RPC Server Unavailable.
This error has context. If it is during connecting to a WMI namespace, it is usually because:
- The machine does not exist.
- The machine cannot respond because the appropriate firewall exceptions have not been made. Check firewall settings.
If it is during operation, it is likely because:
- The client machine doesn’t have correct firewall settings for asynchronous call backs.
- Connecting to a machine which doesn’t exist.
First I would verify the firewall rules. I would make sure the following rules are set:
- WMI (ASync) Properties – In Program: %SYSTEMROOT%\System32\WBEM\unsecapp.exe
- WMI (DCOM) – In Port: TCP 135 Program: %SYSTEMROOT%\System32\svchost.exe
- WMI (WMI) In-Out Program: %SYSTEMROOT%\System32\svchost.exe
I deal with WMI problems all the time. I generally follow this little troubleshooting checklist for RPC errors:
- Use the WMI Control MMC (WMIC.MSC) to ensure that the service is working on the local system.
- If the problem involves communicating with a remote system then use the WMI Control to test the ability to connect to the remote system
- For Access Denied type issues verify that the DCOM and WMI Service settings are at default values, and the Network Service account has been granted impersonation rights.
- Check the service settings if the WMI service fails to start or if client programs cannot communicate with the service. In some cases you may need to reregister all the modules to recover the service.
A much desired, long-awaited, and highly anticipated white paper was released last week. If you are looking to understand how virtual applications are added, published, and delivered from publishing servers (especially the differences from previous versions) look no further. The document is available here!
UPDATE: 10/21/2014: The MVMC 3.0 is now released with P2V functionality restored.
One of the most asked questions I have received regarding the latest release of VMM 2012 has been “Where is the P2V conversion feature? Did it go away?” Well yes, The P2V (Physical-to-Virtual) conversion process was deprecated from System Center Virtual Machine Manager 2012 R2. My colleague Mike Briggs wrote about this in an excellent blog: http://blogs.technet.com/b/scvmm/archive/2013/10/03/how-to-perform-a-p2v-in-a-scvmm-2012-r2-environment.aspx which also outlines a very specific process with a very helpful architectural diagram to get around this for those who still need to get servers migrated.
What was the objective of P2V?
The primary reason for P2V was to bring legacy physical servers into private cloud. Now that many of the operating systems that were commonly virtualized (Windows 2000 Server, Windows Server 2003, Windows Server 2003, and Windows Server 2008) are either out of support or in extended support. Newer operating systems housing business workloads are more hypervisor-aware and more often than not these days, they are being deployed to private clouds and corporate data centers as provisioned virtual machines as opposed to being deployed directly to bare metal installations.
If you are planning to deploy Virtual Machine Manager 2012 R2 as part of your System Center hybrid/private cloud and are looking for a strategy to migrate all of your physical servers aboard, consider you have these options:
Use VMM 2012 SP1 in parallel: Setting up a parallel VMM server (just like specified in the above mentioned blog post) and a Hyper-V Host Running Windows Server 2012 can still serve as a viable staging environment as this will continue to be supported in the immediate future. Windows Server 2012 Hyper-V VM’s can be easily imported/migrated over to Windows Server 2012 R2. This will be especially important if you are looking to do on-line P2V’s with active servers and you want to minimize downtime. Documentation for performing P2V in VMM 2012 SP1 can be found here: http://technet.microsoft.com/en-us/library/hh427286.aspx
Disk2VHD: This free utility from the Sysinternals group (http://technet.microsoft.com/en-us/sysinternals/ee656415) can also perform online P2V’s completely originating from the source operating system creating a VHD or VHDX file ready for virtualization prime time. The latest version adds support for VHDX-formatted VHDs, WinRE volumes, removable media (for capturing,) and includes an option to capture live volumes instead of relying on volume shadow copy (VSS).
3rd-Party Solutions: There are P2V converters out there. Some are rudimentary and inexpensive while others integrate this feature into their enterprise backup and recovery solutions (such as Acronis.) These are also viable options to consider.
You may have noticed that in previously releases of Softgrid and App-V, it was recommended as a best practice to manually create a dummy (I hate that word – I prefer “artificial”) ODBC data source (DSN) connection on the sequencing prior to sequencing anything that interacts with ADO/ODBC. The reason for this has to do with how these settings get captured during the sequencing process and how default registry opacity is handled. If the settings for ODBC are empty than any settings that are captured during the sequencing will be reconciled as “Override” since there will be no existing ODBC registry keys on the sequencer.
The problem this creates is the override settings will prevent the virtual application from seeing any existing ODBC settings that may exist on the client workstation. Given that the application may need to interface with a database client or middleware that needs these settings, this could be problematic. If you create a dummy (as it has been called) ODBC connection in advance, then only the settings for the connection parameters (DSN) captured during sequencing will be collected and the opacity will be set to “Merge.”
With the advent of the 4.6 SP1, many sequencing best practices were implemented into the workflow of the sequencer software. Among them was this very creation of an artificial DSN. As a result, people stopped having to worry about it. Well . . .
It is time to start doing this again. With the 5.0 Sequencer (including the latest SP2 release) you will need to create an artificial ODBC connection in advance of sequencing. Otherwise, you will still run into registry opacity issues with ODBC settings that are captured during sequencing. In most cases, virtual machines are used so I would simply add this to the default configuration of your base sequencing machine.
Normally I do not use this space to advertise books except in the case of two exceptions: If it is a free e-book and is relevant, I will definitely recommend it – otherwise – it had better be good. After reading his first book on the Windows Azure platform, I was very happy to hear that Tejaswi Redkar has come out with a book on rapidly ramping on and deploying Windows Azure Websites – especially given the fact that this is one of the top use cases for moving to the cloud with Windows Azure. Yes, I am using this space to give a shameless plug for this fantastic book.
Do you want to know everything about the fastest growing service in Windows Azure? Do you want to build your own websites in minutes in literally automate EVERYTHING! Are you building a mobile application and need to ensure availability and reliability by implementing an always-on web service? Is your organization working on a cloud strategy?
The book is available in both 21st (e-book) and 20th century (paper) formats!
I first met Tejaswi during the Fall of 2010. He was an Architect and I was a Support Escalation Engineer. I was wading through future private cloud scenarios and he was educating us all on public cloud scenarios. We were both at an internal Microsoft conference up in Bellevue and he was taking me to school left and right on Microsoft Cloud technologies as he was already deep inside many things that neither of us could talk publicly about at the time. As more of a user of Azure (for my personal and day-to-day operations) I find myself on the end of many questions relating to Azure where I can shed light on my personal experiences, but not actually claim to be an expert. I reference Tejaswi's books often as great starting points.
Barnes and Noble Link: http://www.barnesandnoble.com/w/windows-azure-web-sites-tejaswi-redkar/1117494730