Since the dawn of the 21st century (and even before) you have been hearing many items related to acronyms interchangeably describing manageability features within Microsoft products (as well as others.) For example, WMI has been at the heart of most Microsoft Manageability products and solutions given the fact it is one of the primary interfaces within the Windows operating systems. While Microsoft’s WMI ties mostly to its products, it is based upon a series of open, universal standards. And this is the heart of deciphering how acronyms and standards can be interchangeably used to describe the same entity.
So let’s weave through the sometimes confusing relationship between these manageability acronyms – WBEM, WMI, CIM, DMI, DTMF, WSMAN, WinRM, and SNMP of protocols/interfaces/standards. In this little game, I will try to go through these acronyms within the average blog post attention span. WMI is Microsoft’s implementation of the open Web-Based Enterprise Management (WBEM), which comes from the Distributed Management Task Force (an industry organization.) WBEM relies on protocols – which can come from legacy standards such as RPC (Remote-Procedure Call) or DCOM (Distributed Component Object Model) or more modernized http-based SOAP standards such as WinRM (Windows Remote Management) based on the WS-MAN (Web-Services Management) standard. SOAP (Simple Object Access Protocol) itself, is is a command extension protocol designed to be used with HTTP (Hyper-Text Transport protocol – or the web) or SMTP (Simple Mail Transport Protocol – or internet email.)
The WMI interface – based upon the WBEM standard – is built upon an infrastructure centered upon the Common Information Model (CIM) and its respective Object Manager (CIMOM), is what links management applications and providers. The infrastructure also serves as the object-class store and, in many cases, as the storage manager for persistent object properties. WMI implements the store, or repository, as an on-disk database named the CIMOM Object Repository. As part of its infrastructure, WMI supports several APIs through which management applications access object data and providers supply data and class definitions.
Beyond WMI, WBEM’s architecture extends to a variety of underlying technologies besides WMI and Win32 because not everything is or will always be on Microsoft technologies – including the Desktop Management Interface (DMI), and the Simple Network Management Protocol (SNMP) Some of these standards define data storage schemas as well as interfaces. Some define commands within communication protocols. Some or more modernized. SNMP has been deprecated in the most recent versions of Windows in favor of technologies such as WinRM.
I like to use the relationship of WinRM and WMI (alongside their open counterpart standards WS-MAN and WBEM) by stating that one is a management protocol and one is a management interface.
To Read more, check out the standards themselves:
When I worked in support, I troubleshot WMI quite bit using many tools. One tool I still keep my eye on with regards to ongoing development was – and still is – the WMI Explorer utility. I am happy to report a new version of an excellent troubleshooting tool for WMI is now available:
WMI Explorer 2.0 is now available for download:
Microsoft .NET Framework 4.0 Full or .NET Framework 4.5.1
Minimum display resolution: 1024×768
Administrator rights to view some WMI objects
(Optional) Internet access for automatic update check
This is a very intuitive tool for visually troubleshooting WMI issues. It gives you a direct view into the WMI namespace.
New Features include:
New: Asynchronous mode for enumeration of classes and instances in the background.
New: Method execution.
New: SMS (System Center Configuration Manager) Mode.
New: Property tab showing properties of selected class.
New: Input & Output parameter information in Methods tab with Help information.
New: List View output mode for Query Results.
New: Update Notifications when a new version of WMI Explorer is available.
New: Connect to multiple computers at the same time.
New: Quick Filter for Classes and Instances.
New: User Preferences.
New: View WMI Provider Process Information.
Improved: UI display on higher scaling levels and resolution.
Improved: Connect As option to provide alternate credentials.
Improved: Display of embedded object names in Property Grid.
NOTE: This is not an official Microsoft tool, and is available “AS IS” with NO support.
For sequencing in App-V 5, the new ETW model simplifies the process and moves App-V to the Windows standards for event tracing. Even better, the sequencer not only has two logs to worry about (operational and administrative) but a simple process can occur to enable more verbose debug logging.
In App-V 4.6, the process was not that simple. While the logs did not write to the Event Viewer-able logs, all but one are text-based which makes for easy manipulation with your favorite log parser. I prefer Trace32 of course! These log files are stored in the logs subdirectory of the Sequencer installation directory which defaults to C:\Program Files\ Program Files\Microsoft Application Virtualization Sequencer\Logs. Certain logs pertain to specific functions so the relevancy will vary on whatever your troubleshooting scenario might be.
SFT-Seq-log.txt: The majority of sequencer logging occurs here (Uploads to virtual environment, downloads from the virtual environment, service starts and service stops, etc.)
SFTrbt.txt: This is the sequencer reboot log file. When the 4.6 sequencer simulates reboots, the elements that are processed will be tracked in this log file.
SFTCallBack.txt: This is a more simple logs that allows you to reconsile process starts and stops during sequencing. It works great in conjunction with a process monitor log.
Filter.log: Outside of working with Microsoft Support, this log is not very useful as it is obfuscated. It tracks file activity but must be decoded with an internal utility. You can enable further tracking into a file called files.txt which will contain a log of all files created in the VFS. This can be enabled (although it will increase sequencing time) by enabling the following value in the registry:
- Key: HKEY_LOCAL_MACHINE\Software\Microsoft\SoftGrid\4.5\Sequencer\Configuration\
- Value: FileManifest
- Data Type: REG_DWORD
- Data: 1
SFTrpc.txt: This is the log file created by the monitoring element SFTRPC.EXE and in addition to also capturing process startup and shutdowb, will also contain verbose diagnostic information about each monitored shortcut.
In addition to the sequencer logs, you can also leverage process monitor (http://technet.microsoft.com/en-us/sysinternals/bb896645.aspx) and verbose MSI logging (https://madvirtualizer.wordpress.com/2014/03/17/enabling-advanced-windows-installer-logging/) if you encounter errors within the application during sequencing.
App-V 4.6 is still very prevalent out there and will be for a while. With the releases of Windows 8 and Windows 8.1 brought additional service packs for the App-V 4.6 client which means upgrades and/new installs for newer operating systems. Several weeks back on this blog, I went over how to enable advanced MSI logging for troubleshooting MSI installs and upgrades (Remember VOICEWARMUP – https://madvirtualizer.wordpress.com/2014/03/17/enabling-advanced-windows-installer-logging/ ) but I would like to now address some follow up emails I received. Admins would like more specific information on how to go through and read that potentially enormous log in order to find out what is failing where and when.
I would never advising reading a verbose MSI installation log from start to finish especially when dealing with potentially asynchronous actions. MSI logs also have an excessive amount of rollback information incorporated into the log upon failed installations. The seasoned IT Pro often looks for specific keywords such as “error” and “failed” and that can be misleading as not all logs generate these types of messages. In addition, searching on the string “error” can also yield false positives as well.
When I am looking at Verbose MSI logs of App-V 4.6 client installs, I usually analyze the log by doing the following:
- Searching for the error string generated in the App-V Installer User Interface with quotations.
- Searching using the string “1603.” See if it indicates that a custom action has failed.
- Searching using the string “Value 3.” This will indicate an install error. This can also help to identify the custom action failure.
- Searching for string “IsInBadState()” can also be helpful if there is an issue with a failed driver install. This is especially useful in troubleshooting an upgrade. Usually when this occurs, you usually need to delete the driver configuration and state of the specific App-V file system driver specified in order to reattempt the upgrade.
Finally if you need to walk through the App-V custom actions, you can do so by searching by the strings ‘SWI” or “SGC” as all of the App-V custom actions begin with these prefixes.
CustomAction SWI41sp1UpgradeFix returned actual error code 1603
You can walk through the logging of each key App-V custom action. Once you’ve identified what custom action failed, you can then use the following reference to find out specifically what was being attempted with the custom action here: http://support.microsoft.com/kb/2465574. Even though it specifies SP1, it is still valid and helpful for SP2 and SP3. For example the action reference above would be:
Installer : Client
Method name: SWI41sp1UpgradeFix
Description: Modifies an installed instance of the Softgrid 4.1SP1/4.2 client application to correctly support upgrading to a later version.
You can then dive deeper into the timeline of the action and align it with a more deep logging utility such as Process Monitor.
App-V 4.5 and 4.6 virtualize at the user mode layer. One of the most identifying factors of seeing that a thread stack is that of a virtualized application is the presence of the SFTLDR.DLL file. This is what is injected into every process a virtual application will create. This file is responsible for ensuring proper redirections and translations necessary to make virtualization function properly by:
- File changes to included virtual directory and file paths are redirected to the VFS
- Registry changes hooked and redirected to the virtual registry
- The spoofing of objects
- The spoofing of COM GUIDS
In addition to the common troubleshooting methods such as disabling local interaction and disabling object spoofing, you can also take things further by disabling various virtualization components using the System Guard Overrides in App-V 4.x. These are not meant to be solutions but isolation factors in case you need to modify mappings. Many of these can be set at the registry level affecting the entire client or at the application level using the OSD file.
All of the registry values mentioned are located under HKLM\SOFTWARE\Microsoft\SoftGrid\4.5\SystemGuard\Overrides:
Disabling Virtual Services
You can disable virtual services on a per package basis by adding in the <VIRTUAL_SERVICES_DISABLED> tag under the <POLICY> XML element in the OSD file. You can disable the subsystem for the entire client by going adding the DisableVirtualServices DWORD value with a value of 1. If this is enabled, the sftldr.dll will not hook the service APIs.
Disabling the Virtual Registry
You can disable the virtual registry on a per package basis by adding in the <VIRTUAL_REGISTRY_DISABLED> tag under the <POLICY> XML element in the OSD file. You can disable the subsystem for the entire client by going adding the DisableVreg DWORD value with a value of 1. If this is enabled, the sftldr.dll will not hook the virtual registry calls.
Disabling the Virtual File System
You can disable the virtual file system on a per package basis by adding in the <VIRTUAL_FILE_SYSTEM_DISABLED> tag under the <POLICY> XML element in the OSD file. You can disable the subsystem for the entire client by going adding the DisableVFS DWORD value with a value of 1. If this is enabled, the sftldr.dll will not hook virtual file system calls.
Finally, if you are really interested in going to the extreme . . .
You can disable ALL hooking. Can be useful when you are launching an application that is locally installed but still being brought into the virtual bubble. This allows you to turn it on and off if troubleshooting odd behavior. This is done at the client level which is why it is definitely only a troubleshooting option. You can disable hooking by adding in the registry value DisableSftldr DWORD value with a value of 42. Why 42? Well because that is the answer to everything in the universe. This basically makes the sftldr.dll (which is the primary hook DLL) dormant. MAVINJECT32 (or MAVINJECT64 if a 64-bit system) will still inject this DLL though. It will just remain dormant. This is a last resort.
If you are working with an application that has been virtualized with App-V 4.x and that application is not functioning as expected, one of the first steps in troubleshooting should always be to try to determine if the issue is related to sequencing or related to virtualization. For the standard application, Devirtualization is pretty straight-forward and pointed out in the following article:
One you have determined that the issue is likely a virtualization issue (done by successfully de-virtualizing using the article above) then you can proceed to quickly troubleshoot the issues further using a variety of approaches. One good source for V 4 troubleshooting is John Behneman’s post on the App-V team blog: http://blogs.technet.com/b/appv/archive/2012/11/14/3494514.aspx
You can also take this further by enabling the SFTLDR unhandled exception log (SFTLDR.LOG) which tracks hidden errors within the SFTLDR hook DLL which is the most critical virtualization DLL.
To enable the SFTLDR.LOG, create a string (REG_SZ) value called UnhandledExceptionFilterLocation within the following registry key: HKLM\SOFTWARE\Microsoft\SoftGrid\4.5\SystemGuard
For the Value data, you will want to put in the path to an existing directory that is at least one level deep from the root of the drive (i.e. C:\\TEMP\\SFTLDR.LOG.) Also note that I have to double the slash count for paths.
The SFTLDR.LOG file looks like this:
[08/27/2014 18:21:26.379] [03ac] [0a38] [Q:\KK_HAWT.BTY\LISTA\RUN\FIRST\hail.exe] HookedLoadLibraryExW. lpLibFileName=Q:\KK_HAWT.BTY\LISTA\RUN\FIRST\test.dll
[08/27/2014 18:21:26.379] [03ac] [0a38] [Q:\KK_HAWT.BTY\LISTA\RUN\FIRST\hail.exe] HookedLoadLibraryExW: SXS start
[08/27/2014 18:21:26.379] [03ac] [0a38] [Q:\KK_HAWT.BTY\LISTA\RUN\FIRST\hail.exe] HookedLoadLibraryExW: No mapping for Q:\KK_HAWT.BTY\LISTA\RUN\FIRST\test.dll
[08/27/2014 18:21:26.410] [03ac] [0a38] [Q:\KK_HAWT.BTY\LISTA\RUN\FIRST\hail.exe] HookedLoadLibraryExW: LoadLibrary failed (1). dwLastError=14001