For sequencing in App-V 5, the new ETW model simplifies the process and moves App-V to the Windows standards for event tracing. Even better, the sequencer not only has two logs to worry about (operational and administrative) but a simple process can occur to enable more verbose debug logging.
In App-V 4.6, the process was not that simple. While the logs did not write to the Event Viewer-able logs, all but one are text-based which makes for easy manipulation with your favorite log parser. I prefer Trace32 of course! These log files are stored in the logs subdirectory of the Sequencer installation directory which defaults to C:\Program Files\ Program Files\Microsoft Application Virtualization Sequencer\Logs. Certain logs pertain to specific functions so the relevancy will vary on whatever your troubleshooting scenario might be.
SFT-Seq-log.txt: The majority of sequencer logging occurs here (Uploads to virtual environment, downloads from the virtual environment, service starts and service stops, etc.)
SFTrbt.txt: This is the sequencer reboot log file. When the 4.6 sequencer simulates reboots, the elements that are processed will be tracked in this log file.
SFTCallBack.txt: This is a more simple logs that allows you to reconsile process starts and stops during sequencing. It works great in conjunction with a process monitor log.
Filter.log: Outside of working with Microsoft Support, this log is not very useful as it is obfuscated. It tracks file activity but must be decoded with an internal utility. You can enable further tracking into a file called files.txt which will contain a log of all files created in the VFS. This can be enabled (although it will increase sequencing time) by enabling the following value in the registry:
- Key: HKEY_LOCAL_MACHINE\Software\Microsoft\SoftGrid\4.5\Sequencer\Configuration\
- Value: FileManifest
- Data Type: REG_DWORD
- Data: 1
SFTrpc.txt: This is the log file created by the monitoring element SFTRPC.EXE and in addition to also capturing process startup and shutdowb, will also contain verbose diagnostic information about each monitored shortcut.
In addition to the sequencer logs, you can also leverage process monitor (http://technet.microsoft.com/en-us/sysinternals/bb896645.aspx) and verbose MSI logging (https://madvirtualizer.wordpress.com/2014/03/17/enabling-advanced-windows-installer-logging/) if you encounter errors within the application during sequencing.
App-V 4.6 is still very prevalent out there and will be for a while. With the releases of Windows 8 and Windows 8.1 brought additional service packs for the App-V 4.6 client which means upgrades and/new installs for newer operating systems. Several weeks back on this blog, I went over how to enable advanced MSI logging for troubleshooting MSI installs and upgrades (Remember VOICEWARMUP – https://madvirtualizer.wordpress.com/2014/03/17/enabling-advanced-windows-installer-logging/ ) but I would like to now address some follow up emails I received. Admins would like more specific information on how to go through and read that potentially enormous log in order to find out what is failing where and when.
I would never advising reading a verbose MSI installation log from start to finish especially when dealing with potentially asynchronous actions. MSI logs also have an excessive amount of rollback information incorporated into the log upon failed installations. The seasoned IT Pro often looks for specific keywords such as “error” and “failed” and that can be misleading as not all logs generate these types of messages. In addition, searching on the string “error” can also yield false positives as well.
When I am looking at Verbose MSI logs of App-V 4.6 client installs, I usually analyze the log by doing the following:
- Searching for the error string generated in the App-V Installer User Interface with quotations.
- Searching using the string “1603.” See if it indicates that a custom action has failed.
- Searching using the string “Value 3.” This will indicate an install error. This can also help to identify the custom action failure.
- Searching for string “IsInBadState()” can also be helpful if there is an issue with a failed driver install. This is especially useful in troubleshooting an upgrade. Usually when this occurs, you usually need to delete the driver configuration and state of the specific App-V file system driver specified in order to reattempt the upgrade.
Finally if you need to walk through the App-V custom actions, you can do so by searching by the strings ‘SWI” or “SGC” as all of the App-V custom actions begin with these prefixes.
CustomAction SWI41sp1UpgradeFix returned actual error code 1603
You can walk through the logging of each key App-V custom action. Once you’ve identified what custom action failed, you can then use the following reference to find out specifically what was being attempted with the custom action here: http://support.microsoft.com/kb/2465574. Even though it specifies SP1, it is still valid and helpful for SP2 and SP3. For example the action reference above would be:
Installer : Client
Method name: SWI41sp1UpgradeFix
Description: Modifies an installed instance of the Softgrid 4.1SP1/4.2 client application to correctly support upgrading to a later version.
You can then dive deeper into the timeline of the action and align it with a more deep logging utility such as Process Monitor.
In previous blog posts, I’ve discussed items related to App-V’s virtual registry, including the VREG subsystem itself and the new Registry Staging System introduced with App-V 5. I have also been explaining concepts and App-V nomenclature as well so I thought it was very timely to discuss concepts and terminology for our new App-V users with regards to how App-V captures and handles registry information.
Registry Opacity and Translucency
The two most common concepts in virtualizing registry keys and values are registry opacity and translucency. When the sequencer detects that a registry key being captured does not exist, it will mark that key as Opaque. This means it will not merge with the native registry key and will be completely isolated. No native registry values or sub keys beneath this key will be visible in the virtual environment. If the sequencer detects that a registry key being captured already exists, then the key will be marked as translucent which means that the views of the virtual and local registry will be merged and any existing values or sub keys will be visible.
You can modify a registry key’s opacity/translucency setting in the advanced sequencing editor either during sequencing or by editing the package after the fact. In the Virtual Registry tab, you can right-click the registry key and view/set the opacity/translucency:
- Merge with Local Key = Translucent
- Override Local Key = Opaque
Do not confuse the concept of translucency with the concept of registry pass-through. In some cases, the virtual registry needs to be explicitly bypassed for specific keys regardless of a package’s virtual registry configuration particularly for registry writes. This means that data written to these keys will write directly to (or attempt to) the native registry. In App-V 5, these settings are configured at the client level in a REG_MULTI_SZ value called PassThroughPaths in HKEY_LOCAL_MACHINESoftwareMicrosoftMicrosoftAppVSubsystemVirtualRegistry
In App-V 4, this value was called VirtualRegistryPassthroughEx and is documented in the following KB article: http://support.microsoft.com/kb/2750869/en-us
There are default entries in both App-V 4.x and 5.x. I would advise against modifying these but you can add values as situations warrant. Bear in mind that any data that is written to these keys will remain even when the application is removed or repaired.
Registry reflection works with registry bitness virtualization in the sense that it provides replication between the 32-bit and 64-bit registries with Windows running on 64-bit platforms. It essentially copies the registry keys and values between the 32-bit views (Wow6432Node) and the native 64-bit view. These include registry keys that deal primarily with shell integration such as (but not limited to)
Registry Reflection is documented in more detail here: http://msdn.microsoft.com/en-us/library/aa384235(VS.85).aspx. When App-V 4.6 was released, it was the first version of App-V to provide support for 64-bit operating systems. This meant new logic was incorporated into the sequencer in order to copy the views between the 64-bit view and the 32-bit views when sequencing 32-bit applications. Virtual applications needed to be able to replicate what happens in native scenarios. Let’s look at one:
So you have a machine in which you have installed Windows 7 x64. By default, the 4-bit Wordpad is registered to handle .DOCX files. Native registry reflection also has this copied into the 32-bit registry view.
A change has been made and now Office 2010 32-bit is now installed. This will register 32-bit Microsoft Word to handle .DOCX files in the 32-bit registry view. Native registry reflection will then copy this information into the 64-bit registry view so both 32 and 64-bit applications will trigger Microsoft Word 32-bit for handling .DOCX files.
This would work in reverse as well. Let’s say some number cruncher was breathing down your neck demanding 64-bit Excel. So you install 64-bit Office. This would register 64-bit Excel to handle .XLSX files in the 64-bit registry view. Native registry reflection will then copy this information into the 32-bit registry view so both 32 and 64-bit applications will trigger Microsoft Excel 64-bit for handling .DOCX files.
In App-V 4.6, this required a special subsystem which can actually be turned off for troubleshooting purposes by creating a DWORD value called DisableRegistryReflection with a value of 1 in HKEY_LOCAL_MACHINESoftwareWow6432NodeMicrosoftSoftgrid4.5SystemGuardOverrides.
In App-V 5, reflection is handled on the client and is triggered based upon bitness of the process. Whether or not the 32-bit application was sequenced on a 32-bit or 64-bit sequencer does not matter due to the registry staging subsystem. On a 64-bit operating system, when the package was sequenced on a 32-bit Sequencer, the registry staging system will stage the reflected keys to both the 32-bit and 64-bit registry views.
App-V 4.5 and 4.6 virtualize at the user mode layer. One of the most identifying factors of seeing that a thread stack is that of a virtualized application is the presence of the SFTLDR.DLL file. This is what is injected into every process a virtual application will create. This file is responsible for ensuring proper redirections and translations necessary to make virtualization function properly by:
- File changes to included virtual directory and file paths are redirected to the VFS
- Registry changes hooked and redirected to the virtual registry
- The spoofing of objects
- The spoofing of COM GUIDS
In addition to the common troubleshooting methods such as disabling local interaction and disabling object spoofing, you can also take things further by disabling various virtualization components using the System Guard Overrides in App-V 4.x. These are not meant to be solutions but isolation factors in case you need to modify mappings. Many of these can be set at the registry level affecting the entire client or at the application level using the OSD file.
All of the registry values mentioned are located under HKLM\SOFTWARE\Microsoft\SoftGrid\4.5\SystemGuard\Overrides:
Disabling Virtual Services
You can disable virtual services on a per package basis by adding in the <VIRTUAL_SERVICES_DISABLED> tag under the <POLICY> XML element in the OSD file. You can disable the subsystem for the entire client by going adding the DisableVirtualServices DWORD value with a value of 1. If this is enabled, the sftldr.dll will not hook the service APIs.
Disabling the Virtual Registry
You can disable the virtual registry on a per package basis by adding in the <VIRTUAL_REGISTRY_DISABLED> tag under the <POLICY> XML element in the OSD file. You can disable the subsystem for the entire client by going adding the DisableVreg DWORD value with a value of 1. If this is enabled, the sftldr.dll will not hook the virtual registry calls.
Disabling the Virtual File System
You can disable the virtual file system on a per package basis by adding in the <VIRTUAL_FILE_SYSTEM_DISABLED> tag under the <POLICY> XML element in the OSD file. You can disable the subsystem for the entire client by going adding the DisableVFS DWORD value with a value of 1. If this is enabled, the sftldr.dll will not hook virtual file system calls.
Finally, if you are really interested in going to the extreme . . .
You can disable ALL hooking. Can be useful when you are launching an application that is locally installed but still being brought into the virtual bubble. This allows you to turn it on and off if troubleshooting odd behavior. This is done at the client level which is why it is definitely only a troubleshooting option. You can disable hooking by adding in the registry value DisableSftldr DWORD value with a value of 42. Why 42? Well because that is the answer to everything in the universe. This basically makes the sftldr.dll (which is the primary hook DLL) dormant. MAVINJECT32 (or MAVINJECT64 if a 64-bit system) will still inject this DLL though. It will just remain dormant. This is a last resort.
If you have not already heard, today MDOP (Microsoft Desktop Optimization Pack) 2013 R2 became generally available for download to our MDOP customers. More information can be found here:
This latest incarnation of MDOP includes the release of:
- Application Virtualization (App-V) 5.0 SP2 and the virtualization of Office 2013
- Application Virtualization (App-V) 4.6 SP3
- User Experience Virtualization (UE-V) 2.0
- Microsoft BitLocker Administration and Monitoring (MBAM) 2.0 SP1, which also offers multi-language support.
- Diagnostics and Recovery Toolset (DaRT) 8.1
- Advanced Policy Group Management (APGM) 4.0 SP2
Primary release pillars for MDOP 2013 R2 revolve around support for Windows 8.1.
If you are still using App-V 4.6 on Windows 7 (either 4.6 SP1 or SP2) or Windows 8 (4.6 SP2) you may also be aware that the installation of Windows 8.1 will be blocked if it detects the presence of any version of App-V 4.x. This will also include 4.6 SP3 if you decide to go ahead and pre-install SP3 prior to the upgrade to 8.1. This relates to the
operating-specific drivers that are used by App-V 4.6. If you are planning to use Windows 8.1 with App-V 4.6 SP3, you will need to first remove any installations of App-V 4.6. Then you can upgrade/install Windows 8.1. Once Windows 8.1 is installed, you can install App-V 4.6 SP3. Please note this approach only applies to App-V 4.6 and not App-V 5.0.
While the App-V Client settings and user application settings will be retained with the uninstallation/reinstallation approach, the App-V 4.6 cache will be reset so applications will have to be re-delivered (re-streamed, pre-cached, re-advertised, etc.) Normally to avoid having to do this in the past, administrators would simply do in-place upgrades. This one-time exception does not allow for this and will require likely adjustments if you have users currently using virtual applications with 4.6 and will be upgrading to Windows 8.1 from Windows 7 or Windows 8.
Virtual Objects: You have also possibly heard this referred to as virtual eventing. You have seen references to this in the context of troubleshooting. You may have possibly used this as a workaround when troubleshooting in 4.x:
Add an asterisk wildcard in the following registry path: HKEY_LOCAL_MACHINESOFTWAREMicrosoftSoftGrid4.5SystemGuardObjExclusions
Before you ever do the above again, please read this article.
The question is: what exactly are you doing when you do this and what are the ramifications? Well, first of all, this should NEVER be disabled ad-hominem unless you have a specific reason for doing so. This is especially important in versions prior to V5 where disabling this subsystem actually disabled it for the entire client. Second, if possible, only disable using specific markers for objects pertaining to the application in question. It is understandable to avoid this method as it can be a daunting and tedious task to track these down in the case of some applications. The concept of “disabling virtual objects” in App-V really refers to disabling eventing and essentially everything else under the App-V virtual object isolation subsystem by name or wildcard mask. To understand the function of the virtual object subsystem (or VOBJECT) it will help to have a good understanding of kernel objects in general.
When we speak of kernel objects we are talking about all of the object types that fall into this category, i.e. mutexes, semaphores, base named objects, events, file-mappings, etc. Applications work with objects through the use of handles. This is accomplished by calling a Win32 API that is specific to the kernel object being instantiated. If you want to know more about these individual object types and how they work within the operating system, I would suggest picking up the latest copy of Windows Internals.
When an object is instantiated it is often identified using a string. Because the object is managed by the kernel, this means this object can be leveraged by multiple applications. In the case of App-V, like COM, there is name obfuscation that serves as the isolating factor so it allows for the kernel to still manage these objects and the virtual object subsystem within App-V can allow multiple processes within the same virtual environment to share access to the object.
App1.exe —–> Handle —-> Named Object (ObjEvt1)
App2.exe —–> Handle —-> Named Object (ObjEvt1) (previously created)
Many kernel objects can be named using a string. Because the kernel manages the objects, the name can be used by multiple processes. In the case of the above, Both App1.exe and App2.exe have handles to the named object ObjEvt1. App1.exe can now wait for eventing from App2.exe.
Named Kernel objects are isolated by App-V intercepting calls made to create the kernel object and modifying the name in a manner that only the current virtual environment understands. So if you have multiple applications that need to communicate to each other this way, you will need to ensure they belong to the same virtual environment. If you have a virtual application and a local application that will need to communicate with each other in this way, then you will need to disable naming of these objects.
In the case of application conflict resolution, this isolation process allows the virtual applications running in different virtual environments to create the same kernel object, yet have these objects handled differently by the kernel because the names will be different as far as the kernel is concerned. To put it crudely, if two versions of an application called GROUCHOv1.EXE and GROUCHOv2.EXE respectively need to create and “use” the same object (we’ll call it “cigar”) – then virtualizing them separately under App-V will make this possible as one object will actually be called sg<16-hex digit random ID>_cigar and the other will be sg<16-hex digit different random ID>_cigar.
Now, a pre-populated list of exclusions will show up in the registry under both 4.x and 5.x in order for App-V to be happy in the world of Windows. Adding further exclusions and disabling objects as a whole can be done, however, do NOT remove any existing exclusions that are already populated in the client registry. There is absolutely no reason for you to REMOVE any existing exclusions that you did not put in to begin with. When you are troubleshooting the virtual object subsystem, you may want to add in a single “*” to disable eventing for all objects or add in a specific kernel object name or names (preferred.)
4.6 Object Exclusions
In version 4.6, the legacy list of excluded objects were listed here:
A quick way of troubleshooting would be to create a new string value (I’d usually use 1001001) and enter an asterisk (*) as the value. When the application started to work, I could now use Process Explorer or WinObj to help determine what objects were being created. If I weren’t lazy, I would go back and only enter those object names for exclusion. If I were lazy, I’d leave in the asterisk. This can be dangerous – especially if I were on RDS or Citrix servers where you may have multiple versions of applications/middleware.
One thing I want to drive home is that changes here were MACHINE WIDE. If you wanted to apply these changes to one package only, you would leverage the LOCAL_INTERACTION_ALLOWED policy in the application’s OSD file – however this would also disable some COM virtualization features which you may not want. Bear in mind this policy element in the OSD file still looks up the objects in the virtual environment, but if not found, it will fall back to the native and check there as well. This is why it was and still is a great troubleshooting AND workaround for App-V 4.x.
To enable local interaction, your OSD VIRTUALENV tag should look like the following:
App-V 5 and the VObjects Subsystem
There is not much new under the hood in V5 with regards to how isolation of kernel objects happens. There is more flexibility and granularity with configuration.
There is an element within the Deployment Configuration that you can use to turn on or off the VObjects subsystem for a package:
<Objects Enabled=”true” />
What is great about this is that it will allow you to disable the VObjects subsystem for a package but still allow for COM to be virtualized. During the start of a virtual environment, the subsystem will read the manifest and Deployment Configuration file and combines them together. If there are multiple packages within a connection group, they will be combined as well.
You also still have the method of using the registry to exclude specific objects although I see no reason to use this as a means of disabling the entire object subsystem since you can do this through the XML on a per-package basis.
The exclusion locations in V5 are found in:
Tools to Isolate Objects
So, when you launch a virtual application, you can easily compare on a clean test machine whether this newly added virtual application package creates kernel objects. All you do is look for the objects that start with the name “sg” then remove that as well as the 16 character numeric identifier preceeding the underscore and everything else to the right of the underscore character is the actual name of the object which you can then use to create a mask for a custom object exclusion. There are two free tools that are great for this:
When you run Process Explorer and turn on the bottom pane, there will be a view of objects. When you select the virtual processes, you can identify the objects that are virtualized. You will need to sort the object view by name in some cases to make this easier.
Winobj is another tool from the Sysinternals Toolkit that is great to use however, you need to ensure that you run it as an administrator in order to get the right session context since this does not view from a process context. It will allow you to catch any objects you may have missed with the above.
So, I am hanging out on the Technet Forums one Saturday night and I got inspired by a thread (http://social.technet.microsoft.com/Forums/en-US/9b136314-77be-4568-8bc6-6e2ed865f43e/this-document-could-not-be-opened-it-does-not-appear-to-be-a-crystal-report-document) to write this particular blog post. Yes, before I go any further, I will acknowledge the pathetic revelation of what I like to do on a Saturday night – because there is nothing more wild and crazy than hanging out on the forums. Anyway, I wanted to stress the importance of how you can use the sequencer as a troubleshooting mechanism to determine whether an issue with a problematic application is tied to the sequencing process or is an issue with virtualization (in general or with a particular virtualization subsystem.) So, it makes total sense that here I am late on a Friday revealing doing something I love to do – wax poetic about App-V on my blog!
So, if you encounter errors in your application and you suspect that an important asset was not captured during sequencing or you cannot eliminate whether or not a clean sequencing machine was used – you can always DEVIRTUALIZE! Devirtualization is a great step in troubleshooting as it helps you determine if an APPV package has all of the necessary file and registry assets. Here is what you need:
You will need a clean sequencing machine with . . .
- . . . the same version of the sequencer used to sequence the application.
- . . . any dependencies or middleware that the application needs to run (otherwise not included in the package.)
- . . . a troubleshooting toolkit (i.e. Process Monitor)
Once this is in place, copy the package to the sequencer machine and then launch the sequencer application. From the Tools menu, select “Expand Package to Local System.”
Select the APPV package. It will then expand the package to a local installation footprint. If the application was sequenced properly, then all of the file, registry configuration, environment variables, and extension points will “install” to the local operating system. Once the expansion has completed, you can then proceed to launch and test the application. If the application functions correctly, you know the problems you have encountered with the application are not a result of sequencing issues. You can then redirect your troubleshooting towards issues including but not limited to:
- Virtualization subsystems (Virtual COM, Virtual Objects, Virtual Services, VFS, etc.)
- Incompatibilities with virtualization as a whole (drivers, COM+, etc.)
- Information that could have been captured incorrectly in the FSMetadata.xml file (Short Names, Directory Opacity)
- Registry Opacity
- Static Dependencies
If the application fails to launch or function as expected after de-virtualization, then you can proceed to troubleshoot using your troubleshooting toolkit. Usually Process Monitor is quick to isolate these issues (missing files or registry entries.) However, you should always inspect the native equivalent of the extension points as well.