This morning, I was on a joint webinar with Flexera (http://learn.flexerasoftware.com/AR-WBNR-Microsoft-AppV-Best-Practices) on App-V best practices and I reminded many on the call why I always prefer to use the term "current recommended practices" as opposed to "best practices." Today I explained that I know longer insist on avoiding the use of the PVAD (Primary Virtual Application Directory) and sticking to strictly VFS sequences. True, I did make this recommendation nearly a year and a half ago (http://blogs.technet.com/b/gladiatormsft/archive/2014/08/25/app-v-5-installing-to-the-pvad-don-t-do-it-yes-i-said-it.aspx ) however, at the time, App-V 5.0 Service Pack 2 required VFS sequencing to ensure connection group convergence as well as a few others mentioned in the blog post. With the addition of the merged roots feature as well as the correction of issues with the convergence of environment variables, there is no longer any major reason to force packages to VFS sequences for the purposes of making connection groups work.
Some Applications Need the PVAD for Proper Functionality
As App-V 5.0 SP3 was released over a year ago and App-V 5.1 was released a few months back, it has become known that due to issues with pathing limits and other issues related to the App-V VFS subsystem and its native NTFS integration, there are some applications which still require the use of the PVAD. By this, I mean to actually expose the Primary Virtual Application Directory (PVAD) within the App-V sequencer and selecting that same directory during the installation of the application during the monitoring phase of the sequencer. You will find many examples of applications within the App-V community. While the number of applications requiring the use of the PVAD is a relatively small percentage, the applications that are affected represent a significant footprint that involve nearly every major enterprise organization's application library. For example, Office 2010 is one of those applications. In fact, if you were to look at the virtual file system structure of the App-V package generated by the ODT (Office Deployment Tool) you would also find that the flattened Office 2013 and Office 2016 packages are actually PVAD sequences.
App-V 5: Do you Still have to Run Process Monitor within the App-V Bubble when Troubleshooting Applications?
If – by that question you want to know if you must start an instance of Process Monitor within the virtual application like you did in 4.x – no. You could run process Monitor inside the virtual bubble, but it will not yield you much more results. The reason behind this is simple: unlike previous versions of App-V, the REAL registry as well as the native file system – NTFS – is used in App-V 5.
In App-V 4.6 and earlier, if you did not launch process Monitor in the virtual application’s environment (usually through a command prompt) all you would capture related to the virtual application would be operations to file and registry resources outside the virtual environment. In Version 5, running Process Monitor as normal will capture access to the actual locations including registry, package store, as well as VFS (Virtual File System) COW (Copy-on-Write) locations. Why just that? Because that’s where things “actually” are located. What you will have to understand is that once the operations to where the application “thinks” it is located has been hooked in the file system and/or registry every subsequent operation will continue as such to include operations only to the:
Integration junction points to the Package Store
Actual package Store paths
VFS COW (Copy-on-Write) checks and locations
You can see examples of these in the following screenshots from Process Monitor:
Procmon and File Operations:
As you can see above, the query to the initial location of where the virtual applications thinks it is browsing (C:program Files (x86)Java) is clearly not natively where it thinks it is. The App-V engine picks up for this (through relationships in memory) and the operations are redirected to the appropriate converged locations in both the User VFS COW and Package Store.
Procmon and Registry Operations
You will see similar operations when tracing specific activities to registry entries. First, where the application thinks it is supposed to be located followed by subsequent operations to the actual state-separated locations in the actual registry.
Why does Process Monitor not show every Single Operation to “Virtual Paths”
The answer is quite simple – because the App-V Client takes care of all of this behind the scenes which is why you will need to have access to and understand the FileSystemMetadata.xml file as it contains all of the file system mappings for both non-tokenized paths and tokenized paths. The easiest and most automatic relationships are the KNOWNFOLDERID paths which automatically resolve to App-V tokenized paths in memory. For non-tokenized paths, it is handled differently on process creation.
Altitude Adjustment of ProcMon Driver
When you look at the altitudes of the App-V file system drivers and their relationship to the driver altitude, you can see that the Procmon driver sits at a lower altitude by default.
This might make you explore the possibility of raising the altitude to see if Process Monitor will capture more information. Please be careful doing this as this could create problems and system instability. Altitudes are managed and allocated by Microsoft (https://msdn.microsoft.com/en-us/library/windows/hardware/dn641617.) When developers want to register altitude locations for their filter driver, they fill out a special request form. That is how tightly controlled they are in order to prevent instability.
App-V 4.5 and 4.6 virtualize at the user mode layer. One of the most identifying factors of seeing that a thread stack is that of a virtualized application is the presence of the SFTLDR.DLL file. This is what is injected into every process a virtual application will create. This file is responsible for ensuring proper redirections and translations necessary to make virtualization function properly by:
- File changes to included virtual directory and file paths are redirected to the VFS
- Registry changes hooked and redirected to the virtual registry
- The spoofing of objects
- The spoofing of COM GUIDS
In addition to the common troubleshooting methods such as disabling local interaction and disabling object spoofing, you can also take things further by disabling various virtualization components using the System Guard Overrides in App-V 4.x. These are not meant to be solutions but isolation factors in case you need to modify mappings. Many of these can be set at the registry level affecting the entire client or at the application level using the OSD file.
All of the registry values mentioned are located under HKLM\SOFTWARE\Microsoft\SoftGrid\4.5\SystemGuard\Overrides:
Disabling Virtual Services
You can disable virtual services on a per package basis by adding in the <VIRTUAL_SERVICES_DISABLED> tag under the <POLICY> XML element in the OSD file. You can disable the subsystem for the entire client by going adding the DisableVirtualServices DWORD value with a value of 1. If this is enabled, the sftldr.dll will not hook the service APIs.
Disabling the Virtual Registry
You can disable the virtual registry on a per package basis by adding in the <VIRTUAL_REGISTRY_DISABLED> tag under the <POLICY> XML element in the OSD file. You can disable the subsystem for the entire client by going adding the DisableVreg DWORD value with a value of 1. If this is enabled, the sftldr.dll will not hook the virtual registry calls.
Disabling the Virtual File System
You can disable the virtual file system on a per package basis by adding in the <VIRTUAL_FILE_SYSTEM_DISABLED> tag under the <POLICY> XML element in the OSD file. You can disable the subsystem for the entire client by going adding the DisableVFS DWORD value with a value of 1. If this is enabled, the sftldr.dll will not hook virtual file system calls.
Finally, if you are really interested in going to the extreme . . .
You can disable ALL hooking. Can be useful when you are launching an application that is locally installed but still being brought into the virtual bubble. This allows you to turn it on and off if troubleshooting odd behavior. This is done at the client level which is why it is definitely only a troubleshooting option. You can disable hooking by adding in the registry value DisableSftldr DWORD value with a value of 42. Why 42? Well because that is the answer to everything in the universe. This basically makes the sftldr.dll (which is the primary hook DLL) dormant. MAVINJECT32 (or MAVINJECT64 if a 64-bit system) will still inject this DLL though. It will just remain dormant. This is a last resort.
If you are running any version of the App-V 5 client or Sequencer prior to App-V 5.0 Service Pack 2 Hotfix 4 – stop reading. This does not apply to your environment. If you are running HF4 or sooner, you need to have a good understanding of the net effects of toggling VFS Mode on and/or off during package upgrade.
VFS Write Mode was introduced to (my words) “fix bad brake jobs” when it comes to application development. Applications that need to be able to read and write configurations files in normally protected directories or have modified ACL’s for standard users in order to write to program locations that only administrators would normally have access to (Notepad++ has a way to go back and forth between good and bad program configuration – http://blogs.technet.com/b/gladiatormsft/archive/2014/05/24/app-v-5-on-sequencing-using-tokenized-paths-pvad-s-vfs-and-vfs-write-mode.aspx.)
While VFS Write Mode involves setting a specific attribute within the package manifest, the effects on how the VFS (Virtual File System) and COW (Copy-on-Write) filter are handled for the user are significant. As a result, making any changes to this setting during package upgrade could have some effects.
Scenario 1: Package Upgrade where the VFS Write Mode setting is not changed
This one is pretty straight-forward. No considerations are needed. Nothing is changed in how the VFS handles things.
Scenario 2: Package Upgrade where VFS Write Mode setting is turned on when previously not enabled.
When this happens and directories previously existed in the COW but there was no “S” directory, then the original directory will be renamed to the “S” version and the permissions appropriately adjusted. The new “regular” directory will be created. So instead of just one directory (i.e. ProgramFilesX86) there will be two (adding ProgramFilesX86S.) For more information on the “S” directories, please refer to my previous post here (http://blogs.technet.com/b/gladiatormsft/archive/2014/08/29/app-v-5-on-the-now-sometimes-immutable-package-cache-location.aspx.)
Scenario 3: Package Upgrade where VFS Write Mode setting is turned off when previous enabled.
In this scenario, there will likely be existing two structures: directories appended with “S” and the regular directory with the relaxed permissions. When this happens, the relaxed permissions directories will be deleted and the “S” directories will be renamed back to their original names. If this is ever done unintentionally, you can imagine some of the issues that may happen so be careful doing this.
I would advise always avoiding Scenario 3. Scenario 2 will be quite common as many sequencers and packagers are now upgrading their pre-SP2 HF4 App-V packages in order to take advantage of the new VFS Write mode feature. The question I am getting asked a lot lately is whether it is better to re-sequence with VFS Write Mode on or to simply just perform a package upgrade. I would advise trying the package upgrade first to turn on the value. In most cases, this should work – but as always, I await your comments.
When you add/configure and publish a package in App-V, the primary file assets and source registry hives are stored in the package cache location. It defaults to %PROGRAMDATA%App-V. The package cache will store packages in a <PACKAGE_GUID><VERSION_GUID> directory structure for each package added to the machine regardless of targeting. This package cache will exist for the purposes of marking key assets in both stream-to-disk scenarios in stream-to-memory (Shared Content Store) scenarios.
In many cases you may want to redirect the location of the cache without relocating the location of %PROGRAMDATA%. This can be done by changing the location of PackageInstallationRoot through the registry, PowerShell, or through Group Policy. Technically you might be able to get away with not having to restart the client if you do it through PowerShell, but I would advise that you do it anyway.
- Value: PackageInstallationRoot
- Data Type: REG_SZ
Set-appvclientconfiguration –PackageInstallationRoot <Path>
You may have noticed that with the defaults, nothing shows up (even the default directory) until the first package is created. Once the first package is configured, the directory is created as a hidden directory with administrators, system, and trusted installer having control with the directory ownership going to SYSTEM. Because of the ACL’s, removing this out of band requires taking ownership and resetting permissions prior to deletion if you ever have to manually remove this.
If you change the location to another directory or another drive without pre-creating and setting the proper permissions, you will notice that you encounter add/publish package failures with error 0c-80070005. At the bare minimum, you would need to redirect to a directory where system does have full control. If the package cache is ever moved without being properly reconfigured, the App-V client service will fail to start.
If you redirect the App-V package cache to a persistent disk in non-persistent scenarios and also redirect catalog data, you can create situations where stale catalog data is trying to be sourced upon reversion. Non-persistent scenarios must avoid situations where catalogs from previous installations pre-exist in the redirected package cache locations. If you redirect the entire %ProgramData% directory to a persistent disk, you will likely run into this in non-persistent (or in the case of the use of a persistent disk.)
If you are suspecting that you are getting this issue, you can confirm it by enabling advanced debug logging of the Client-Orchestration, Client-Service, and the Streaming Manager logs. The Client-Orchestration and Client-Service log will throw error 0XA560212C-0x3. The StreamingManager log will give the following error:
Get DACL from directory failed. Error: The system cannot find the path specified. <PATH>
The quickest solution is to remove the old catalog from %ProgramData%MicrosoftAppVClientCatalog and restart the App-V Client service.
How does VFS Write Mode fit in?
With the redevelopment of the virtual file system including a new user-mode VFS and the COW (copy-on-write) filter, new challenges arose and this package cache format preventing applications from functioning correctly in certain scenarios such as:
- Needing to write to Common App Data or a virtualized %PROGRAMDATA%
- Creating Folder and files beneath %ProgramFiles%
- Adjusting permissions on User COW locations (which, in user targeting, mirror the native directories.)
Enter VFS Write Mode. This reverts back to the notion that the user who owns the COW (which for global would mean potentially any user) gets full access to the top-level COW directories it would normally have access to. Additional compliance in line with Windows Resource Protection and UAC do remain in effect however. To differentiate, there will be directories denoted with an “S”. Standard users will read and write from the regular directory (with relaxed ACL’s) while elevated processes will read and write to the “S” version.
The VFS attribute is a very simple one that is stored in the manifest and can also be used with sequencing templates. It is not yet available with dynamic configuration. Yeah . . . that might/ought to get changed.
But what happens when you complicate things with Connection Groups?
The expected behavior for Connection Groups is actually pretty simple. If you add a package with VFS Write Mode turned on to a Connection Group, then the net effect of that will be the entire Connection Group has this turned on since the Connection Group shares a common COW. This setting will persist.
COW restrictions by extension have not changed. All of the executable extensions and common application binary extensions are still restricted from being modified even with VFS Write Mode turned on. These include: ASP, BAT, CER, CHM, CMD, COM, CPL, CRT, DLL, DRV, EXE, FON, GRP, HLP, HTA, JS, LNK, MSI, MSP, MST, NLS, OCX, PIF, REG, SCR, SYS, URL, VB, VBE, VBS, WSF, and WSH
Update 12/5/2014: The PVAD feature is now optional as of App-V 5.0 SP3. Also SP3 allows for merged roots with Connection Groups. Read more here: http://technet.microsoft.com/en-us/library/dn858700.aspx#BKMK_pvad_hidden
If you have ever dealt with me directly as a customer, attended one of my presentations, or even simply stomached one of my diatribes in a casual, technical conversation, you have no doubt heard about one of my pet peeves (no, not “Tips –n- Tricks” – we’ll visit that another time) but the term “best practices.”
I loathe that term. I know we are guilty of it at Microsoft – so are just about every single technology-based organizations. In our ever-evolving industry, to put something down in print as THE ABSOLUTE BEST PRACTICES goes against the very nature of the word “practice.” It can seem arrogant. It implies way too much finality. It also can mean serious disruption when a practice changes. I work in a consulting practice. As products, regulations, politics, and trends change, so do our recommended practices. This is why I purposely try to use “under the current recommended practice” or simply just “current recommended practices.”
When it comes to recommended practices involving the sequencing of applications with App-V, I have been evolving my recommended practices particularly in the area of whether or not to use the PVAD. Specifically, whether to ever recommended to match the installation directory to the directory that was used for the Primary Virtual Application Directory. You may have also heard this as “Sequence to the PVAD, Install to the PVAD.”
Back in May, I expanded upon the subject of the PVAD and how it served as a placeholder for root-based files and directories (http://blogs.technet.com/b/gladiatormsft/archive/2014/05/24/app-v-5-on-sequencing-using-tokenized-paths-pvad-s-vfs-and-vfs-write-mode.aspx.) These differ from VFS-based resources (i.e. those beneath rootvfs) in how the App-V 5 file system handles file operations. How you use the PVAD does have implications in terms of how:
- KNOWNFOLDERID’s always get translated to proper App-V Tokens
- Directories and Files get properly merged within Connection Groups
- Environment variables get properly translated
- VFS Write Mode is effective
Well after adjusting recommendations for App-V 5 regarding sequencing for connection groups, tokenization, and environment variables, I am now making the following recommendation to ensure that the above items always happens: It is best to always sequence with a fake PVAD which means create a false PVAD that will be different from the installation directory.
Now the next logical question would be: Are there any circumstances where this would create problems with newly sequenced virtual applications? I don’t think so. But I await your comments. 🙂
Update 12/5/2014: The PVAD feature is now optional as of App-V 5.0 SP3. Also SP3 allows for merged roots with Connection Groups. Read more here: http://technet.microsoft.com/en-us/library/dn858700.aspx#BKMK_pvad_hidden
Update 12/5/2014: The PVAD is now considered optional due to changes in App-V 5Service Pack 3. Read more here: http://technet.microsoft.com/en-us/library/dn858700.aspx#BKMK_pvad_hidden
When sequencing applications, you often never know how the application was written with regards to what paths are actually referenced via hard-coded paths or through tokenized, or well-known paths or KNOWNFOLDERIDS. In previous versions of App-V and Softgrid, these paths were referenced using %CSIDL% variables. These are well-documented on MSDN (http://msdn.microsoft.com/en-us/library/windows/desktop/bb762494(v=vs.85).aspx). If the application was developed to reference CSIDLs, they usually function when virtualized just fine with App-V. CSIDLs are still supported for backward compatibility, but going forward starting with App-V 5, a different tokenization process is used to align better with KNOWNFOLDERIDs. The new App-V Tokens and their default locations upon installations are well documented in the App-V 5 sequencing guide:
Tokenization helps to remove a specific application’s tie-in to specific folder paths and to allow for greater portability especially when these KNOWNFOLDERIDS have been redirected for specific configurations.
It is important to understand that for sequencing applications that will be used for Connection Groups or to better take advantage of the App-V 5.0 SP2 HF4 VFS Write Mode, all files will need to be located in the VFS structures. This will drastically affect the sequencing process.
Sequencing to a PVAD, Installing to the PVAD
If you specify a Primary Virtual Application Directory and then install to that same location, the base directory of installation files will not be tokenized. For many older applications, this may be the best route to take. For example, if the Primary Virtual Application Directory (PVAD) is specified as C:DIRECTORY1 and the sequencer monitors an install during sequencing to C:DIRECTORY1 then everything created beneath DIRECTORY1 goes beneath Root and will not be tokenized or located beneath the VFS folder. There may be additional files that do in fact get written to tokenized paths and they will be tokenized but you will now have a mixture of both “rooted” file assets and VFS file assets.
In the example below, Notepad ++ is sequenced to C:Notepad and installed to C:Notepad:
As you can see above, the majority of the files appear below the Root structure and are not contained with the VFS tokenized directories. Also notice the folders beneath Root cannot be adjusted for opacity:
When the files are deployed to the App-V immutable package cache, they will be laid out the same way.
In addition, in order to troubleshoot and navigate the file system within the bubble, non-VFS folders do not appear in directory listings.
You will be able to change to virtual directory by using cd Notepad+
Sequencing to the PVAD, Installing to Program Files or a Different Location other than the PVAD (Fake PVAD)
If you specify a PVAD and then install to a different location, all of the files will be written to the VFS location inside the package. For example, if the PVAD is C:DIRECTORY1 and the sequencer monitors an install to C:Program FilesDIRECTORY1 during sequencing, then everything gets put beneath RootVFS and root installation files will get tokenized if they are written to write to a KNOWNFOLDERID or CSIDL during installation.
In the example below, Notepad++ is sequenced using a fake PVAD and installed into C:Program Files
As you can see, all of the files are stored beneath the VFS directory structure.
When files are VFS’ed, you can see the directory listings as normal directories converged with the local directories when you launch a command inside the App-V bubble.
What About Using a KNOWNFOLDERID as the PVAD?
This question comes up quite a bit. Many packagers like to match PVAD with the installation directory (especially now that the Q: requirement is gone.) This is fine but understand that when you install to a PVAD during sequencing, all installation files beneath that folder that are not otherwise set to install to another explicit KNOWNFOLDERID or CSIDL path will be treated as rooted files.
For example, if you choose C:Program FilesDIRECTORY as the PVAD and the sequencer monitors an install to C:Program FilesDIRECTORY1 during sequencing, then most files will still be beneath root and NOT VFS except for tokenization of explicit KNOWNFOLDERID paths.
In the example below, when sequencing Notepad++, a PVAD of C:Program FilesNotepad++ is specified.
During the monitoring of the installation, notice the specified install directory is also C:Program FilesNotepad++.
Upon completion of sequencing, you will then notice that the package files are laid out just like any other PVAD-sequenced package.
The base installation files are located beneath Root.
Like the example earlier, when you open up a command prompt inside the bubble, you will not see the root directory in a directory listing.
However, you can still change to it manually inside the bubble.
So What about this VFS Write Mode?
For a lot of applications, especially older applications, they are written to use configuration files beneath the root of the installation directory. Natively installed applications often have to use shims to redirect these modifiable configuration files to safer directories. Combine this with the fact that most packagers are not always sure where to place the installation files in relationship to the PVAD (we cannot always read the minds of developers) and you can run into problems when virtualizing applications. While using shims is possible with virtualized applications, this can drastically increase the sequencing and remediation time for an application.
Enter VFS Write Mode. The App-V team added this “silver bullet” of a feature into HF4 of App-V 5 SP2. Using the advanced configuration options in sequencing, you can now select this option to enable an application to normally open files for modification within the package.
To demonstrate how this can be helpful, once again we will use Notepad++. When you install Notepad++, you have the option to load/write the configuration files from the install directory instead of using the %APPDATA% directory which would be properly tokenized.
Prior to the invention of VFS Write Mode, if you selected “Don’t use %APPDATA%” during sequencing, this is what would happen when you tried to run the program virtualized. First you would get an error message like this:
This would be followed by a crash.
When you troubleshoot the application with Process Monitor, you will notice a series of failures when trying to load the files:
In HF4, you can open up the package in the new sequencer, navigate to the “Advanced” tab, and check the box enabling VFS Write Mode.
As a result, Notepad++ will open and function as expected.
You will also notice in Process Monitor, the file operation to open the XML configuration files in the installation directory (with Read/Write/Delete) is successful.
The great thing about VFS Write Mode is that it can be enabled simply by opening up the package in the App-V 5 SP2 HF4 sequencer. You can now put this into your arsenal of App-V AppCompat secret weapons.
Update 12/5/2014: The PVAD is now considered optional due to changes in App-V 5 Service Pack 3. Read more here: http://technet.microsoft.com/en-us/library/dn858700.aspx#BKMK_pvad_hidden