I have been meaning to follow-up my previous discussions of the App-V registry staging subsystem with an article on the virtual registry in App-V 5. I will admit I am a little late to the follow up, however, as they say “better late than never.” In previous article I discussed registry staging and its effect on publishing and streaming. Now the discussion continues with how the virtual registry handles operations once the virtual registry has been staged.
The Virtual Registry in App-V was implemented in a much more streamlined way in version 5 than in previous versions. First of all, real hive files are packaged with the APPV package format. In addition, the real registry is used where the actual locations are state-separated while the Virtual Registry component provides the correct redirection and COW (copy-on-write) operations merging 3 elements into a single registry view:
The native registry
The immutable staged package registry (built from the registry.dat file within the package)
The user and machine COW registry
Like with file assets, the view of each merged registry is done for each package and package group (virtual environment)
At runtime, registry operations are hooked, and special sauce is made to ensure the redirection, registry merging, and copy-on-write operations of changes are done.
Registry reads are done in an ordered approach by layer and you can easily confirm this with Process Monitor. The order is:
The COW registry is read first
Followed by the Package Registry (constructed from REGISTRY.DAT)
Finally the native registry for the requested location.
For registry writes, things are much simpler. Registry writes always go to the COW location corresponding to the original key that was opened. Registry data that is written to the user’s roaming registry is tokenized so that it is portable across machine boundaries.
Bear in mind, this is based upon the predication that the registry location is viewed as virtual (opaque) to begin with and was not excluded or configured to be translucent in the sequencer. There are special pointers contained in the registry that will govern opacity that I will mention later.
Registry COW Locations
The Virtual Registry will manage and track all COW locations for registry storage. The locations will vary depending on security context and type.
Roaming User Registry COW data will go here:
Non Roaming User Registry and non-elevated Machine Registry COW data will go here:
For Machine registry data coming from elevated processes, the Registry COW data will go here:
Special Registry Key and Value Metadata
You may have noticed that for some keys, you will also see some additional data:
This data is not on all keys but when it is, it is reflective of the specific state of the key when responding to registry operations. Particularly, whether the key was previous deleted or if it was configured to be opaque and not merged with the other registry layers.
0x00010000: Key Deleted
0x00020000: Value Deleted
0x00040000: Key Opaque
If the value above also contains a 1 at the end of the type, this means there are sub keys present stored within the value data.
Purging the COW
The Registry COW data is purged along with the rest of the user state when the package has been deleted or when the package is repaired with the –userstate switch.
App-V 5: Do you Still have to Run Process Monitor within the App-V Bubble when Troubleshooting Applications?
If – by that question you want to know if you must start an instance of Process Monitor within the virtual application like you did in 4.x – no. You could run process Monitor inside the virtual bubble, but it will not yield you much more results. The reason behind this is simple: unlike previous versions of App-V, the REAL registry as well as the native file system – NTFS – is used in App-V 5.
In App-V 4.6 and earlier, if you did not launch process Monitor in the virtual application’s environment (usually through a command prompt) all you would capture related to the virtual application would be operations to file and registry resources outside the virtual environment. In Version 5, running Process Monitor as normal will capture access to the actual locations including registry, package store, as well as VFS (Virtual File System) COW (Copy-on-Write) locations. Why just that? Because that’s where things “actually” are located. What you will have to understand is that once the operations to where the application “thinks” it is located has been hooked in the file system and/or registry every subsequent operation will continue as such to include operations only to the:
Integration junction points to the Package Store
Actual package Store paths
VFS COW (Copy-on-Write) checks and locations
You can see examples of these in the following screenshots from Process Monitor:
Procmon and File Operations:
As you can see above, the query to the initial location of where the virtual applications thinks it is browsing (C:program Files (x86)Java) is clearly not natively where it thinks it is. The App-V engine picks up for this (through relationships in memory) and the operations are redirected to the appropriate converged locations in both the User VFS COW and Package Store.
Procmon and Registry Operations
You will see similar operations when tracing specific activities to registry entries. First, where the application thinks it is supposed to be located followed by subsequent operations to the actual state-separated locations in the actual registry.
Why does Process Monitor not show every Single Operation to “Virtual Paths”
The answer is quite simple – because the App-V Client takes care of all of this behind the scenes which is why you will need to have access to and understand the FileSystemMetadata.xml file as it contains all of the file system mappings for both non-tokenized paths and tokenized paths. The easiest and most automatic relationships are the KNOWNFOLDERID paths which automatically resolve to App-V tokenized paths in memory. For non-tokenized paths, it is handled differently on process creation.
Altitude Adjustment of ProcMon Driver
When you look at the altitudes of the App-V file system drivers and their relationship to the driver altitude, you can see that the Procmon driver sits at a lower altitude by default.
This might make you explore the possibility of raising the altitude to see if Process Monitor will capture more information. Please be careful doing this as this could create problems and system instability. Altitudes are managed and allocated by Microsoft (https://msdn.microsoft.com/en-us/library/windows/hardware/dn641617.) When developers want to register altitude locations for their filter driver, they fill out a special request form. That is how tightly controlled they are in order to prevent instability.
In previous blog posts, I’ve discussed items related to App-V’s virtual registry, including the VREG subsystem itself and the new Registry Staging System introduced with App-V 5. I have also been explaining concepts and App-V nomenclature as well so I thought it was very timely to discuss concepts and terminology for our new App-V users with regards to how App-V captures and handles registry information.
Registry Opacity and Translucency
The two most common concepts in virtualizing registry keys and values are registry opacity and translucency. When the sequencer detects that a registry key being captured does not exist, it will mark that key as Opaque. This means it will not merge with the native registry key and will be completely isolated. No native registry values or sub keys beneath this key will be visible in the virtual environment. If the sequencer detects that a registry key being captured already exists, then the key will be marked as translucent which means that the views of the virtual and local registry will be merged and any existing values or sub keys will be visible.
You can modify a registry key’s opacity/translucency setting in the advanced sequencing editor either during sequencing or by editing the package after the fact. In the Virtual Registry tab, you can right-click the registry key and view/set the opacity/translucency:
- Merge with Local Key = Translucent
- Override Local Key = Opaque
Do not confuse the concept of translucency with the concept of registry pass-through. In some cases, the virtual registry needs to be explicitly bypassed for specific keys regardless of a package’s virtual registry configuration particularly for registry writes. This means that data written to these keys will write directly to (or attempt to) the native registry. In App-V 5, these settings are configured at the client level in a REG_MULTI_SZ value called PassThroughPaths in HKEY_LOCAL_MACHINESoftwareMicrosoftMicrosoftAppVSubsystemVirtualRegistry
In App-V 4, this value was called VirtualRegistryPassthroughEx and is documented in the following KB article: http://support.microsoft.com/kb/2750869/en-us
There are default entries in both App-V 4.x and 5.x. I would advise against modifying these but you can add values as situations warrant. Bear in mind that any data that is written to these keys will remain even when the application is removed or repaired.
Registry reflection works with registry bitness virtualization in the sense that it provides replication between the 32-bit and 64-bit registries with Windows running on 64-bit platforms. It essentially copies the registry keys and values between the 32-bit views (Wow6432Node) and the native 64-bit view. These include registry keys that deal primarily with shell integration such as (but not limited to)
Registry Reflection is documented in more detail here: http://msdn.microsoft.com/en-us/library/aa384235(VS.85).aspx. When App-V 4.6 was released, it was the first version of App-V to provide support for 64-bit operating systems. This meant new logic was incorporated into the sequencer in order to copy the views between the 64-bit view and the 32-bit views when sequencing 32-bit applications. Virtual applications needed to be able to replicate what happens in native scenarios. Let’s look at one:
So you have a machine in which you have installed Windows 7 x64. By default, the 4-bit Wordpad is registered to handle .DOCX files. Native registry reflection also has this copied into the 32-bit registry view.
A change has been made and now Office 2010 32-bit is now installed. This will register 32-bit Microsoft Word to handle .DOCX files in the 32-bit registry view. Native registry reflection will then copy this information into the 64-bit registry view so both 32 and 64-bit applications will trigger Microsoft Word 32-bit for handling .DOCX files.
This would work in reverse as well. Let’s say some number cruncher was breathing down your neck demanding 64-bit Excel. So you install 64-bit Office. This would register 64-bit Excel to handle .XLSX files in the 64-bit registry view. Native registry reflection will then copy this information into the 32-bit registry view so both 32 and 64-bit applications will trigger Microsoft Excel 64-bit for handling .DOCX files.
In App-V 4.6, this required a special subsystem which can actually be turned off for troubleshooting purposes by creating a DWORD value called DisableRegistryReflection with a value of 1 in HKEY_LOCAL_MACHINESoftwareWow6432NodeMicrosoftSoftgrid4.5SystemGuardOverrides.
In App-V 5, reflection is handled on the client and is triggered based upon bitness of the process. Whether or not the 32-bit application was sequenced on a 32-bit or 64-bit sequencer does not matter due to the registry staging subsystem. On a 64-bit operating system, when the package was sequenced on a 32-bit Sequencer, the registry staging system will stage the reflected keys to both the 32-bit and 64-bit registry views.
App-V 4.5 and 4.6 virtualize at the user mode layer. One of the most identifying factors of seeing that a thread stack is that of a virtualized application is the presence of the SFTLDR.DLL file. This is what is injected into every process a virtual application will create. This file is responsible for ensuring proper redirections and translations necessary to make virtualization function properly by:
- File changes to included virtual directory and file paths are redirected to the VFS
- Registry changes hooked and redirected to the virtual registry
- The spoofing of objects
- The spoofing of COM GUIDS
In addition to the common troubleshooting methods such as disabling local interaction and disabling object spoofing, you can also take things further by disabling various virtualization components using the System Guard Overrides in App-V 4.x. These are not meant to be solutions but isolation factors in case you need to modify mappings. Many of these can be set at the registry level affecting the entire client or at the application level using the OSD file.
All of the registry values mentioned are located under HKLM\SOFTWARE\Microsoft\SoftGrid\4.5\SystemGuard\Overrides:
Disabling Virtual Services
You can disable virtual services on a per package basis by adding in the <VIRTUAL_SERVICES_DISABLED> tag under the <POLICY> XML element in the OSD file. You can disable the subsystem for the entire client by going adding the DisableVirtualServices DWORD value with a value of 1. If this is enabled, the sftldr.dll will not hook the service APIs.
Disabling the Virtual Registry
You can disable the virtual registry on a per package basis by adding in the <VIRTUAL_REGISTRY_DISABLED> tag under the <POLICY> XML element in the OSD file. You can disable the subsystem for the entire client by going adding the DisableVreg DWORD value with a value of 1. If this is enabled, the sftldr.dll will not hook the virtual registry calls.
Disabling the Virtual File System
You can disable the virtual file system on a per package basis by adding in the <VIRTUAL_FILE_SYSTEM_DISABLED> tag under the <POLICY> XML element in the OSD file. You can disable the subsystem for the entire client by going adding the DisableVFS DWORD value with a value of 1. If this is enabled, the sftldr.dll will not hook virtual file system calls.
Finally, if you are really interested in going to the extreme . . .
You can disable ALL hooking. Can be useful when you are launching an application that is locally installed but still being brought into the virtual bubble. This allows you to turn it on and off if troubleshooting odd behavior. This is done at the client level which is why it is definitely only a troubleshooting option. You can disable hooking by adding in the registry value DisableSftldr DWORD value with a value of 42. Why 42? Well because that is the answer to everything in the universe. This basically makes the sftldr.dll (which is the primary hook DLL) dormant. MAVINJECT32 (or MAVINJECT64 if a 64-bit system) will still inject this DLL though. It will just remain dormant. This is a last resort.
A much desired, long-awaited, and highly anticipated white paper was released last week. If you are looking to understand how virtual applications are added, published, and delivered from publishing servers (especially the differences from previous versions) look no further. The document is available here!
UPDATE: PLEASE READ THIS AFTER READING THIS ARTICLE: http://blogs.technet.com/b/gladiatormsft/archive/2014/08/22/app-v-5-revisiting-registry-staging.aspx
The App-V 5.0 Virtual Registry is isolated for each application (or virtual environment when using Connection Groups.) With this release, there is a better view into the Virtual Registry subsystem. In fact, the App-V 5 virtual registry is divided into two main parts: The Virtual Registry itself (VREG) and the registry staging subsystem.
The APPV package file is a human-readable set of assets. As many of you may have already figured out, you can simply rename the file to a ZIP extension and browse it from within Explorer. Some of you have discovered the hard ay that you destroy the package once you try to modify it this way. Yes – it is a mechanism for VIEWING ONLY. If you have had a chance to do this, you have probably noticed that each APPV package contains its own registry hive file (.DAT)
This is where all of the registry assets captured during sequencing reside. When a package is being published, part of the overall package publishing includes the staging of the registry. Both registry machine data and user data will occur. The staging allows the virtual registry not just to be made available, but to also set up the opacity configuration and other important relationships with the native registry. This is to ensure that the applications have a consistent view of both. If you launch REGEDIT.EXE inside one of the App-V bubbles, you will see this converged registry.
If you are using a traditional App-V 5 server-based infrastructure, you might find yourself in an interesting situation. When synchronizing with a publishing server for the first time, all of the publishing blocks for all of the applications targeted to that machine or user will be passed down and each application will be published on the App-V client either to the user or globally depending on the target methodology. In addition to shortcuts, file type associations, and other extension point registrations, there will also be the staging of the virtual registry occurring in the background. This staging can produce some significant overhead during these sync and on first launch – and when many applications are being used (as well as applications with thick registries) this overhead can delay availability of those applications. This can be especially critical when using pooled VDI scenarios incorporating SCS (Shared Content Store) because you also have the additional overhead of sparse file creation.
Registry staging occurs upon the initial launching of a package if it has not been completed. It extracts the .DAT file from the APPV Package Root (which was copied from the APPV file) and then copies it again over to %PROGRAMDATA%MicrosoftAppVClientVREG. It will then mount the hive file through a background thread and recreate it in the package registry location inside the native registry. Registry staging will occur for both individual packages and connection groups. The time it takes depends on several factors including the size of the hive.
The native registry locations where these are stored are:
Package Registry Staging: HKLMSoftwareMicrosoftAppVClientPackages<PACKAGE_GUID>Versions<VERSION_GUID>
Connection Group Registry Staging: HKLMSoftwareMicrosoftAppVClientPackageGroups<CG_GUID>Versions<VERSION_GUID>
Package Registry Staging: HKCUSoftwareClassesAppVClientPackages<PACKAGE_GUID>Versions<VERSION_GUID>
Connection Group Registry Staging: HKCUSoftwareClassesAppVClientPackageGroups<CG_GUID>Versions<VERSION_GUID>
You will also be able to tell if the registry staging for a specific application has been completed as there will be an additional key called RegistryStagingFinished (beneath the <VERSION_GUID> entry.) Registry staging can occur in the background or on-demand. In VDI environments, we are seeing heavy CPU utilization with background staging in some circumstances. You can gain significant improvement by switching to on-demand staging. You can do this by adding the following registry value in the HKEY_LOCAL_MACHINESoftwareMicrosoftAppVSubsystem key
Data Type: DWORD
This is showing to improve first launch experiences in App-V 5 VDI environments.
Troubleshooting virtualized Office add-ins are always fun. Correction: troubleshooting Office add-ins are always fun for people like me – not necessarily normal human beings. Whether you are using add-ins with a local instance of Office or are packaging add-ins with virtualized Office, you need to be to understand how Office knows which add-ins to load and how these will be loaded.
There are generally three ways to virtualize add-ins for Office:
1.) Where Office is installed locally and the Add-in is virtualized – The Office application must have a shortcut/OSD file provisioned in order to be brought into the virtual environment. Starting with App-V 4.6 SP2, the App-V Sequencer has a specific workflow that allows for this.
Be advised that when you do this, you will be bringing in the local instance of Office into the virtual environment in order to load the add-in. What this can lead to is other add-ins which may not be virtualized not loading if the configuration is not set properly for this. Having a clear understanding of your Office add-in ecosystem will help in developing a strategy in advance to avoid this.
2.) Through Connection Groups or Dynamic Suite Composition (Depending on the Version) – You can also take advantage of both the Add-in/Plug-in workflow and the “Expand Package to Local Disk” feature of the sequencer in order to package Office and Add-in applications separately but still connect them into a common virtual environment.
This method works best (in my opinion) when your strategy is to virtualize both Office and the add-ins. This is also the only way I use Add-ins with Office 2013.
3.) Office and add-ins packaged together into one package (often sequenced using a single pass.) This way is the least common and not always recommended as it may require you to deploy multiple Office packages.
Verify the Add-in Registry Keys
When an add-in is installed, it can be registered to either the user or to the whole machine. When an add-in is sequenced, it will also be virtualized and depending on the registry opacity configuration, will override whatever is locally present. User-registered add-ins go to HKEY_CURRENT_USER while add-ins registered to the entire machine go to HKEY_LOCAL_MACHINE. Whether Office is local and being brought into the virtual environment or Office is virtualized with the Add-in, Office needs to be able to understand the “Load Behavior” of the add-in. This will be found under
Each Add-in appears as a subkey beneath this key. The LoadBehavior DWORD value determines how it will load. This key is the #1 culprit when it comes to Add-ins not working right whether it is a “1st launch-only” issue or an “always-launch” issue. It is a DWORD value. By default, this entry is set to 3, which specifies that the add-in is loaded at startup but an many cases, especially if the add-in was launched during sequencing, you may run into a situation where the value may be set to something else depending on what was done during sequencing. If the value is 0, then the add-in will have to be enabled once the user launches the application. If the package is repaired it will have to be enabled again. The value 1 can be misleading and I encounter this a lot in virtual add-in troubleshooting. Add-ins dialog box indicates that the add-in is loaded after the application starts, the add-in isn’t actually loaded. If the application successfully loads the add-in, the LoadBehavior value changes to 0, and remains at 0 after the application closes. The value 2 means If the application successfully loads the add-in, the LoadBehavior value changes to 3, and remains at 3 after the application closes. With App-V this means potential issues in prompts including annoying “success messages.”
LoadBehavior at 3 is good for App-V
Hey, I made a rhyme! If the LoadBehavior value is set to 3, it will always try to load the add-in and if the add-in fails, will set it back to 2. This is what we want in the virtual registry of our App-V package most of the time. You can usually verify/set this in the Virtual Registry tab after sequencing prior to saving the package.
Should I Verify Registry Opacity?
Yes. While you are there you may want to verify your registry opacity settings. Are those Add-in keys set to “override” or “merge” with the local key. If you have local add-ins that you will use alongside of these virtual add-ins, you will want to ensure that the Addin key is set to merge.
What about Outlook Add-Ins?
For Outlook, you will most assuredly need to manually import/add these virtual registry entries during sequencing (or package upgrade) since you do not want to be launching Outlook during sequencing.
Check the Event Logs when Troubleshooting
While the event logs may not have the event handlers registered for office (since it is virtual, you will still see unregistered event ID’s of Outlook (Event ID 45) which will list all of the add-ins loaded and their subsequent load times.
Event ID: 45
Outlook loaded the following add-in(s)
Office 2013 and Mismatched Virtual Subsystem Settings
If you are using Virtual Office 2013 with App-V 5, this is the #1 issue. Given all of the changes with Office 2013 virtualization, one would think the Add-in story would become complicated. It is not. The recommended practices are pretty straight-forward:
On the Sequencing Machine:
1.) Install the App-V 5 Sequencer.
2.) Install Office 2013.
3.) Start a new package.
4.) On the “Type of Application” page, select “Add-on or Plug-in.”
5.) Select the installer for the plugin.
6.) In the “Install Primary” page, select “I have installed the primary parent program” and continue.
7.) Install the plugin and save the package.
8.) Depending on the add-in, you may need to run it during sequencing.
1.) Copy your Office 2013 to the client machine
2.) Copy the sequenced add-in to the client machine
3.) Modify the add-in’s DeploymentConfig.xml file and modify the following:
4.) Search for:
5.) Search for:
“<Objects Enabled=”true” />”
“<Objects Enabled=”false” />”
The Connection Group will fail if the above changes are not made.
6.) Build a Connection Group document for both Office 2013 and the Add-In. You can do this with stand-alone testing by using the following resource on creating a Connection Group XML document: http://technet.microsoft.com/en-us/library/jj713474.aspx
7.) Enable Package Scripts
Set-AppvClientConfiguration -EnablePackageScripts 1
8.) Add the Office 2013 package; publish it Globally
Add-AppvClientPackage –path <path_to_office_package_>.APPV | Publish-AppvClientPackage –Global
9.) Add the Add-In package; publish it Global
Add-AppvClientPackage –path <path_to_office_addin>.APPV | Publish-AppvClientPackage –Global
10.) Add the Connection Group pointing to the Connection Group document you created earlier. Enable it Globally
Add-AppvClientConnectionGroup –path <PATH_TO_CG_DOC>.xml |Enable-AppvClientConnectionGroup –Global
11.) Launch the Office application the add-in uses.