I’ve been blogging lately (well . . . not exactly lately) about debugging virtual applications that misbehave – whether by hanging, crashing, or spewing some strange error message. In discussing the use of tools, one question I often get is the relationship of attaching the debuggers to the virtual application inside or outside the virtual environment. With App-V 5, it only matters in certain circumstances at the user mode level as virtualized applications work from a state-separated registry and file system that leverage the native file system and registry.
When a developer is troubleshooting a virtual application that they have developed, or an IT Pro Debugger is attempting to reverse engineer an application issue, I usually recommend they attach to the virtual process as they would any other application at first. You may decide due to necessity or preference to attach a debugger or a tool to the virtual application process within the context of the virtual environment by launching it in the application’s “bubble.” You may want to be advised that once you do this, the debugging tool will also be running virtually and will behave accordingly.
I bring this up due to an interesting issue that was encountered by one of our partners a while back: There was an issue with a plug-in to Internet Explorer causing it to hang shortly after the BHO had been triggered while running in standard isolation (i.e. no Dynamic Virtualization involved.) Given that the individual had developed the code for the plug-in, they wanted to capture a full user mode memory dump of the instance of Internet Explorer that was running virtually. The issue could not be reproduced in the developer’s environment so there was the usual suspicion of something environmental being a factor. Since standard WER (Windows Error Reporting) is somewhat limited by default, the customer was leveraging ProcDump -h to capture a user mode memory dump of the IEXPLORE.EXE process.
Here’s the thing: while Procdump appeared to attach to and generate a dump successfully, there were no dumps to be found per the developer. Upon further inquiry, I found that the developer was doing the following:
- Using C: as a target. This is not good on many levels.
- Running ProcDump from a command prompt in the application’s (IEXPLORE.EXE) virtual environment.
While running it in the bubble was not necessarily a bad thing, I had the developer simply redirect the output to %TEMP% instead of C:. The dumps showed up as normal. When asked why C: mattered, I told him that due to coupling factors (specifying an unexcluded folder, VFS Write mode, running as an administrator, and launching in the bubble, the dump file was treated as application state data and was redirected to the VFS CoW location. Upon a quick demonstration, it was discovered buried beneath the user’s VFS folder.
I should note that there shouldn’t be any concern with regards to the usability of these redirected dumps. I will also note that upon repairing the App-V package in questions, the user freed up around 12GB of disk space. I guess they had been trying the ProcDump command quite a bit. J
I have been meaning to follow-up my previous discussions of the App-V registry staging subsystem with an article on the virtual registry in App-V 5. I will admit I am a little late to the follow up, however, as they say “better late than never.” In previous article I discussed registry staging and its effect on publishing and streaming. Now the discussion continues with how the virtual registry handles operations once the virtual registry has been staged.
The Virtual Registry in App-V was implemented in a much more streamlined way in version 5 than in previous versions. First of all, real hive files are packaged with the APPV package format. In addition, the real registry is used where the actual locations are state-separated while the Virtual Registry component provides the correct redirection and COW (copy-on-write) operations merging 3 elements into a single registry view:
The native registry
The immutable staged package registry (built from the registry.dat file within the package)
The user and machine COW registry
Like with file assets, the view of each merged registry is done for each package and package group (virtual environment)
At runtime, registry operations are hooked, and special sauce is made to ensure the redirection, registry merging, and copy-on-write operations of changes are done.
Registry reads are done in an ordered approach by layer and you can easily confirm this with Process Monitor. The order is:
The COW registry is read first
Followed by the Package Registry (constructed from REGISTRY.DAT)
Finally the native registry for the requested location.
For registry writes, things are much simpler. Registry writes always go to the COW location corresponding to the original key that was opened. Registry data that is written to the user’s roaming registry is tokenized so that it is portable across machine boundaries.
Bear in mind, this is based upon the predication that the registry location is viewed as virtual (opaque) to begin with and was not excluded or configured to be translucent in the sequencer. There are special pointers contained in the registry that will govern opacity that I will mention later.
Registry COW Locations
The Virtual Registry will manage and track all COW locations for registry storage. The locations will vary depending on security context and type.
Roaming User Registry COW data will go here:
Non Roaming User Registry and non-elevated Machine Registry COW data will go here:
For Machine registry data coming from elevated processes, the Registry COW data will go here:
Special Registry Key and Value Metadata
You may have noticed that for some keys, you will also see some additional data:
This data is not on all keys but when it is, it is reflective of the specific state of the key when responding to registry operations. Particularly, whether the key was previous deleted or if it was configured to be opaque and not merged with the other registry layers.
0x00010000: Key Deleted
0x00020000: Value Deleted
0x00040000: Key Opaque
If the value above also contains a 1 at the end of the type, this means there are sub keys present stored within the value data.
Purging the COW
The Registry COW data is purged along with the rest of the user state when the package has been deleted or when the package is repaired with the –userstate switch.
App-V 5: Do you Still have to Run Process Monitor within the App-V Bubble when Troubleshooting Applications?
If – by that question you want to know if you must start an instance of Process Monitor within the virtual application like you did in 4.x – no. You could run process Monitor inside the virtual bubble, but it will not yield you much more results. The reason behind this is simple: unlike previous versions of App-V, the REAL registry as well as the native file system – NTFS – is used in App-V 5.
In App-V 4.6 and earlier, if you did not launch process Monitor in the virtual application’s environment (usually through a command prompt) all you would capture related to the virtual application would be operations to file and registry resources outside the virtual environment. In Version 5, running Process Monitor as normal will capture access to the actual locations including registry, package store, as well as VFS (Virtual File System) COW (Copy-on-Write) locations. Why just that? Because that’s where things “actually” are located. What you will have to understand is that once the operations to where the application “thinks” it is located has been hooked in the file system and/or registry every subsequent operation will continue as such to include operations only to the:
Integration junction points to the Package Store
Actual package Store paths
VFS COW (Copy-on-Write) checks and locations
You can see examples of these in the following screenshots from Process Monitor:
Procmon and File Operations:
As you can see above, the query to the initial location of where the virtual applications thinks it is browsing (C:program Files (x86)Java) is clearly not natively where it thinks it is. The App-V engine picks up for this (through relationships in memory) and the operations are redirected to the appropriate converged locations in both the User VFS COW and Package Store.
Procmon and Registry Operations
You will see similar operations when tracing specific activities to registry entries. First, where the application thinks it is supposed to be located followed by subsequent operations to the actual state-separated locations in the actual registry.
Why does Process Monitor not show every Single Operation to “Virtual Paths”
The answer is quite simple – because the App-V Client takes care of all of this behind the scenes which is why you will need to have access to and understand the FileSystemMetadata.xml file as it contains all of the file system mappings for both non-tokenized paths and tokenized paths. The easiest and most automatic relationships are the KNOWNFOLDERID paths which automatically resolve to App-V tokenized paths in memory. For non-tokenized paths, it is handled differently on process creation.
Altitude Adjustment of ProcMon Driver
When you look at the altitudes of the App-V file system drivers and their relationship to the driver altitude, you can see that the Procmon driver sits at a lower altitude by default.
This might make you explore the possibility of raising the altitude to see if Process Monitor will capture more information. Please be careful doing this as this could create problems and system instability. Altitudes are managed and allocated by Microsoft (https://msdn.microsoft.com/en-us/library/windows/hardware/dn641617.) When developers want to register altitude locations for their filter driver, they fill out a special request form. That is how tightly controlled they are in order to prevent instability.
In previous blog posts, I discussed what happens when a package is added/configured, published, and when streamed. But what happens when an application is double-clicked? App-V needs to determine if the application needs to be virtualized, and if so, which virtual environment. The Shortcut is launched. The Junction point is parsed to the right version location in the package cache. The executable’s process is created but there will be a process notify callback by the App-V Virtual Environment Manager (AppVVemgr.sys) to determine if the process should be virtualized or not. Now, I previously blogged about this part of the process last year and gave a convoluted diagram (http://blogs.technet.com/b/gladiatormsft/archive/2013/09/12/app-v-5-to-virtualize-or-not-to-virtualize-how-app-v-5-answers-that-question-when-you-launch-a-shortcut.aspx) which was met with entertaining emails.
Assuming the application is going to be virtualized, after that workflow has completed, then either the AppVEntSubsystems32.dll or the AppVEntSubsystems64.dll is injected into the virtual process depending on the bitness of the application. Of course, it will always be the 32-bit DLL if working exclusively on X86 platforms. You will notice that once an application is running, there are a few ways you can tell if an application is running virtualized or not. I’ve previously mentioned ListDlls.exe from Sysinternals but you can also use Process Explorer, but not for the reason you may think. The “Virtualized” tab within Process Explorer refers to whether the application is being run within WoW64 (for 32-bit applications) rather native x64. Nothing to do with App-V just as the “Package Name” field refers to the base AppX package and applies to modern applications.
But you can use other features of Process Explorer to determine if a process is virtualized with App-V. For example, you verify the path and command line as they will show launched from the location of the package cache.
That is not completely full proof because you could also have an application that is locally installed but has been configured to run inside a package’s virtual environment. This also means the application will have the AppVEntSusbsystemsXX.DLL injected as well. So just search for that. You can do it through the stack itself but that will require you to walk to the individual thread stacks.
Instead you could just simply do a search in Process Explorer for the DLL and you will get a response back of every process for which that DLL has been injected
Besides injecting this DLL into the process, several other events are taking place. The App-V Virtual Environment Manager will also create all of the elements needed for that virtual environment including the attaching of the VFS COW Filter driver to the appropriate volumes and send the COW registry and file mappings to the driver assuming they have been staged properly – otherwise, that must also take place. The Virtual subsystem controller must also establish the subsystems with the proper settings based on the packages catalog (COM, Objects, etc.) which is why it is mandatory that these settings match when you are creating and enabling Connection Groups as the settings for the entire Package Group Catalog would also have to be created should this be the first application that is launching in a published Connection Group.
If you are running any version of the App-V 5 client or Sequencer prior to App-V 5.0 Service Pack 2 Hotfix 4 – stop reading. This does not apply to your environment. If you are running HF4 or sooner, you need to have a good understanding of the net effects of toggling VFS Mode on and/or off during package upgrade.
VFS Write Mode was introduced to (my words) “fix bad brake jobs” when it comes to application development. Applications that need to be able to read and write configurations files in normally protected directories or have modified ACL’s for standard users in order to write to program locations that only administrators would normally have access to (Notepad++ has a way to go back and forth between good and bad program configuration – http://blogs.technet.com/b/gladiatormsft/archive/2014/05/24/app-v-5-on-sequencing-using-tokenized-paths-pvad-s-vfs-and-vfs-write-mode.aspx.)
While VFS Write Mode involves setting a specific attribute within the package manifest, the effects on how the VFS (Virtual File System) and COW (Copy-on-Write) filter are handled for the user are significant. As a result, making any changes to this setting during package upgrade could have some effects.
Scenario 1: Package Upgrade where the VFS Write Mode setting is not changed
This one is pretty straight-forward. No considerations are needed. Nothing is changed in how the VFS handles things.
Scenario 2: Package Upgrade where VFS Write Mode setting is turned on when previously not enabled.
When this happens and directories previously existed in the COW but there was no “S” directory, then the original directory will be renamed to the “S” version and the permissions appropriately adjusted. The new “regular” directory will be created. So instead of just one directory (i.e. ProgramFilesX86) there will be two (adding ProgramFilesX86S.) For more information on the “S” directories, please refer to my previous post here (http://blogs.technet.com/b/gladiatormsft/archive/2014/08/29/app-v-5-on-the-now-sometimes-immutable-package-cache-location.aspx.)
Scenario 3: Package Upgrade where VFS Write Mode setting is turned off when previous enabled.
In this scenario, there will likely be existing two structures: directories appended with “S” and the regular directory with the relaxed permissions. When this happens, the relaxed permissions directories will be deleted and the “S” directories will be renamed back to their original names. If this is ever done unintentionally, you can imagine some of the issues that may happen so be careful doing this.
I would advise always avoiding Scenario 3. Scenario 2 will be quite common as many sequencers and packagers are now upgrading their pre-SP2 HF4 App-V packages in order to take advantage of the new VFS Write mode feature. The question I am getting asked a lot lately is whether it is better to re-sequence with VFS Write Mode on or to simply just perform a package upgrade. I would advise trying the package upgrade first to turn on the value. In most cases, this should work – but as always, I await your comments.
When you add/configure and publish a package in App-V, the primary file assets and source registry hives are stored in the package cache location. It defaults to %PROGRAMDATA%App-V. The package cache will store packages in a <PACKAGE_GUID><VERSION_GUID> directory structure for each package added to the machine regardless of targeting. This package cache will exist for the purposes of marking key assets in both stream-to-disk scenarios in stream-to-memory (Shared Content Store) scenarios.
In many cases you may want to redirect the location of the cache without relocating the location of %PROGRAMDATA%. This can be done by changing the location of PackageInstallationRoot through the registry, PowerShell, or through Group Policy. Technically you might be able to get away with not having to restart the client if you do it through PowerShell, but I would advise that you do it anyway.
- Value: PackageInstallationRoot
- Data Type: REG_SZ
Set-appvclientconfiguration –PackageInstallationRoot <Path>
You may have noticed that with the defaults, nothing shows up (even the default directory) until the first package is created. Once the first package is configured, the directory is created as a hidden directory with administrators, system, and trusted installer having control with the directory ownership going to SYSTEM. Because of the ACL’s, removing this out of band requires taking ownership and resetting permissions prior to deletion if you ever have to manually remove this.
If you change the location to another directory or another drive without pre-creating and setting the proper permissions, you will notice that you encounter add/publish package failures with error 0c-80070005. At the bare minimum, you would need to redirect to a directory where system does have full control. If the package cache is ever moved without being properly reconfigured, the App-V client service will fail to start.
If you redirect the App-V package cache to a persistent disk in non-persistent scenarios and also redirect catalog data, you can create situations where stale catalog data is trying to be sourced upon reversion. Non-persistent scenarios must avoid situations where catalogs from previous installations pre-exist in the redirected package cache locations. If you redirect the entire %ProgramData% directory to a persistent disk, you will likely run into this in non-persistent (or in the case of the use of a persistent disk.)
If you are suspecting that you are getting this issue, you can confirm it by enabling advanced debug logging of the Client-Orchestration, Client-Service, and the Streaming Manager logs. The Client-Orchestration and Client-Service log will throw error 0XA560212C-0x3. The StreamingManager log will give the following error:
Get DACL from directory failed. Error: The system cannot find the path specified. <PATH>
The quickest solution is to remove the old catalog from %ProgramData%MicrosoftAppVClientCatalog and restart the App-V Client service.
How does VFS Write Mode fit in?
With the redevelopment of the virtual file system including a new user-mode VFS and the COW (copy-on-write) filter, new challenges arose and this package cache format preventing applications from functioning correctly in certain scenarios such as:
- Needing to write to Common App Data or a virtualized %PROGRAMDATA%
- Creating Folder and files beneath %ProgramFiles%
- Adjusting permissions on User COW locations (which, in user targeting, mirror the native directories.)
Enter VFS Write Mode. This reverts back to the notion that the user who owns the COW (which for global would mean potentially any user) gets full access to the top-level COW directories it would normally have access to. Additional compliance in line with Windows Resource Protection and UAC do remain in effect however. To differentiate, there will be directories denoted with an “S”. Standard users will read and write from the regular directory (with relaxed ACL’s) while elevated processes will read and write to the “S” version.
The VFS attribute is a very simple one that is stored in the manifest and can also be used with sequencing templates. It is not yet available with dynamic configuration. Yeah . . . that might/ought to get changed.
But what happens when you complicate things with Connection Groups?
The expected behavior for Connection Groups is actually pretty simple. If you add a package with VFS Write Mode turned on to a Connection Group, then the net effect of that will be the entire Connection Group has this turned on since the Connection Group shares a common COW. This setting will persist.
COW restrictions by extension have not changed. All of the executable extensions and common application binary extensions are still restricted from being modified even with VFS Write Mode turned on. These include: ASP, BAT, CER, CHM, CMD, COM, CPL, CRT, DLL, DRV, EXE, FON, GRP, HLP, HTA, JS, LNK, MSI, MSP, MST, NLS, OCX, PIF, REG, SCR, SYS, URL, VB, VBE, VBS, WSF, and WSH
A much desired, long-awaited, and highly anticipated white paper was released last week. If you are looking to understand how virtual applications are added, published, and delivered from publishing servers (especially the differences from previous versions) look no further. The document is available here!