If you are running any version of the App-V 5 client or Sequencer prior to App-V 5.0 Service Pack 2 Hotfix 4 – stop reading. This does not apply to your environment. If you are running HF4 or sooner, you need to have a good understanding of the net effects of toggling VFS Mode on and/or off during package upgrade.
VFS Write Mode was introduced to (my words) “fix bad brake jobs” when it comes to application development. Applications that need to be able to read and write configurations files in normally protected directories or have modified ACL’s for standard users in order to write to program locations that only administrators would normally have access to (Notepad++ has a way to go back and forth between good and bad program configuration – http://blogs.technet.com/b/gladiatormsft/archive/2014/05/24/app-v-5-on-sequencing-using-tokenized-paths-pvad-s-vfs-and-vfs-write-mode.aspx.)
While VFS Write Mode involves setting a specific attribute within the package manifest, the effects on how the VFS (Virtual File System) and COW (Copy-on-Write) filter are handled for the user are significant. As a result, making any changes to this setting during package upgrade could have some effects.
Scenario 1: Package Upgrade where the VFS Write Mode setting is not changed
This one is pretty straight-forward. No considerations are needed. Nothing is changed in how the VFS handles things.
Scenario 2: Package Upgrade where VFS Write Mode setting is turned on when previously not enabled.
When this happens and directories previously existed in the COW but there was no “S” directory, then the original directory will be renamed to the “S” version and the permissions appropriately adjusted. The new “regular” directory will be created. So instead of just one directory (i.e. ProgramFilesX86) there will be two (adding ProgramFilesX86S.) For more information on the “S” directories, please refer to my previous post here (http://blogs.technet.com/b/gladiatormsft/archive/2014/08/29/app-v-5-on-the-now-sometimes-immutable-package-cache-location.aspx.)
Scenario 3: Package Upgrade where VFS Write Mode setting is turned off when previous enabled.
In this scenario, there will likely be existing two structures: directories appended with “S” and the regular directory with the relaxed permissions. When this happens, the relaxed permissions directories will be deleted and the “S” directories will be renamed back to their original names. If this is ever done unintentionally, you can imagine some of the issues that may happen so be careful doing this.
I would advise always avoiding Scenario 3. Scenario 2 will be quite common as many sequencers and packagers are now upgrading their pre-SP2 HF4 App-V packages in order to take advantage of the new VFS Write mode feature. The question I am getting asked a lot lately is whether it is better to re-sequence with VFS Write Mode on or to simply just perform a package upgrade. I would advise trying the package upgrade first to turn on the value. In most cases, this should work – but as always, I await your comments.
When you add/configure and publish a package in App-V, the primary file assets and source registry hives are stored in the package cache location. It defaults to %PROGRAMDATA%App-V. The package cache will store packages in a <PACKAGE_GUID><VERSION_GUID> directory structure for each package added to the machine regardless of targeting. This package cache will exist for the purposes of marking key assets in both stream-to-disk scenarios in stream-to-memory (Shared Content Store) scenarios.
In many cases you may want to redirect the location of the cache without relocating the location of %PROGRAMDATA%. This can be done by changing the location of PackageInstallationRoot through the registry, PowerShell, or through Group Policy. Technically you might be able to get away with not having to restart the client if you do it through PowerShell, but I would advise that you do it anyway.
- Value: PackageInstallationRoot
- Data Type: REG_SZ
Set-appvclientconfiguration –PackageInstallationRoot <Path>
You may have noticed that with the defaults, nothing shows up (even the default directory) until the first package is created. Once the first package is configured, the directory is created as a hidden directory with administrators, system, and trusted installer having control with the directory ownership going to SYSTEM. Because of the ACL’s, removing this out of band requires taking ownership and resetting permissions prior to deletion if you ever have to manually remove this.
If you change the location to another directory or another drive without pre-creating and setting the proper permissions, you will notice that you encounter add/publish package failures with error 0c-80070005. At the bare minimum, you would need to redirect to a directory where system does have full control. If the package cache is ever moved without being properly reconfigured, the App-V client service will fail to start.
If you redirect the App-V package cache to a persistent disk in non-persistent scenarios and also redirect catalog data, you can create situations where stale catalog data is trying to be sourced upon reversion. Non-persistent scenarios must avoid situations where catalogs from previous installations pre-exist in the redirected package cache locations. If you redirect the entire %ProgramData% directory to a persistent disk, you will likely run into this in non-persistent (or in the case of the use of a persistent disk.)
If you are suspecting that you are getting this issue, you can confirm it by enabling advanced debug logging of the Client-Orchestration, Client-Service, and the Streaming Manager logs. The Client-Orchestration and Client-Service log will throw error 0XA560212C-0x3. The StreamingManager log will give the following error:
Get DACL from directory failed. Error: The system cannot find the path specified. <PATH>
The quickest solution is to remove the old catalog from %ProgramData%MicrosoftAppVClientCatalog and restart the App-V Client service.
How does VFS Write Mode fit in?
With the redevelopment of the virtual file system including a new user-mode VFS and the COW (copy-on-write) filter, new challenges arose and this package cache format preventing applications from functioning correctly in certain scenarios such as:
- Needing to write to Common App Data or a virtualized %PROGRAMDATA%
- Creating Folder and files beneath %ProgramFiles%
- Adjusting permissions on User COW locations (which, in user targeting, mirror the native directories.)
Enter VFS Write Mode. This reverts back to the notion that the user who owns the COW (which for global would mean potentially any user) gets full access to the top-level COW directories it would normally have access to. Additional compliance in line with Windows Resource Protection and UAC do remain in effect however. To differentiate, there will be directories denoted with an “S”. Standard users will read and write from the regular directory (with relaxed ACL’s) while elevated processes will read and write to the “S” version.
The VFS attribute is a very simple one that is stored in the manifest and can also be used with sequencing templates. It is not yet available with dynamic configuration. Yeah . . . that might/ought to get changed.
But what happens when you complicate things with Connection Groups?
The expected behavior for Connection Groups is actually pretty simple. If you add a package with VFS Write Mode turned on to a Connection Group, then the net effect of that will be the entire Connection Group has this turned on since the Connection Group shares a common COW. This setting will persist.
COW restrictions by extension have not changed. All of the executable extensions and common application binary extensions are still restricted from being modified even with VFS Write Mode turned on. These include: ASP, BAT, CER, CHM, CMD, COM, CPL, CRT, DLL, DRV, EXE, FON, GRP, HLP, HTA, JS, LNK, MSI, MSP, MST, NLS, OCX, PIF, REG, SCR, SYS, URL, VB, VBE, VBS, WSF, and WSH
If you are working with an application that has been virtualized with App-V 4.x and that application is not functioning as expected, one of the first steps in troubleshooting should always be to try to determine if the issue is related to sequencing or related to virtualization. For the standard application, Devirtualization is pretty straight-forward and pointed out in the following article:
One you have determined that the issue is likely a virtualization issue (done by successfully de-virtualizing using the article above) then you can proceed to quickly troubleshoot the issues further using a variety of approaches. One good source for V 4 troubleshooting is John Behneman’s post on the App-V team blog: http://blogs.technet.com/b/appv/archive/2012/11/14/3494514.aspx
You can also take this further by enabling the SFTLDR unhandled exception log (SFTLDR.LOG) which tracks hidden errors within the SFTLDR hook DLL which is the most critical virtualization DLL.
To enable the SFTLDR.LOG, create a string (REG_SZ) value called UnhandledExceptionFilterLocation within the following registry key: HKLM\SOFTWARE\Microsoft\SoftGrid\4.5\SystemGuard
For the Value data, you will want to put in the path to an existing directory that is at least one level deep from the root of the drive (i.e. C:\\TEMP\\SFTLDR.LOG.) Also note that I have to double the slash count for paths.
The SFTLDR.LOG file looks like this:
[08/27/2014 18:21:26.379] [03ac] [0a38] [Q:\KK_HAWT.BTY\LISTA\RUN\FIRST\hail.exe] HookedLoadLibraryExW. lpLibFileName=Q:\KK_HAWT.BTY\LISTA\RUN\FIRST\test.dll
[08/27/2014 18:21:26.379] [03ac] [0a38] [Q:\KK_HAWT.BTY\LISTA\RUN\FIRST\hail.exe] HookedLoadLibraryExW: SXS start
[08/27/2014 18:21:26.379] [03ac] [0a38] [Q:\KK_HAWT.BTY\LISTA\RUN\FIRST\hail.exe] HookedLoadLibraryExW: No mapping for Q:\KK_HAWT.BTY\LISTA\RUN\FIRST\test.dll
[08/27/2014 18:21:26.410] [03ac] [0a38] [Q:\KK_HAWT.BTY\LISTA\RUN\FIRST\hail.exe] HookedLoadLibraryExW: LoadLibrary failed (1). dwLastError=14001
To continue the discussion of application troubleshooting, I wanted to provide some clarification on a certain type of application failure. In a blog post a while back (http://blogs.technet.com/b/gladiatormsft/archive/2013/05/15/app-v-process-exit-codes-and-why-they-matter-when-troubleshooting-virtual-applications.aspx) I discussed using Process Monitor to dissect silent application failures or exits. Process Monitor allows us to look at process exit codes to determine leads on application failures. I wanted to dive into another type of failure that may or may not leave a proper exit code because it is not “completely” silent. It’s what I refer to as a “Fly-By-Night” error message. [Incidentally, it should be surprising to no one that a blog such as this would eventually have a subtle reference to the rock band Rush.]
Fly-by-Night Error Messages refer to those error messages which appear to flash some code but is often interpreted by the human eye as a quick flash of a black screen:
Fly-by error messages occur when you try to call a script, batch file, internal command (using CMD.EXE triggered by CONHOST.EXE) or any type of Win32 Console application (also triggered by CONHOST.EXE) and it fails to launch.
If you are lucky, these can easy to troubleshoot if you attempt to launch the application without requiring a console host process to launch it (CONHOST.EXE.) For example, let’s use the example of the DB2 client. You have virtualized the middleware and you are attempting to run the Batch file that triggers the DB2 Command Prompt Plus (CLPLUS.BAT.) You simply run the path to executable within an already existing command prompt to grab the error message.
In the above example, it looks like the issue is pretty straight forward. The application requires Java and I do not have Java installed locally or virtually. In most cases, you will get an error message in the console – straight forward or not – you will still have something to go on.
So what if this yielded no error?
You now have a silent exit on your hands. If you do not get an error message at all, you will need to investigate it further using Process Monitor and I recommend referencing my previous blog on silent exit codes (see link above.) For example, if I get a silent exit from a console-based application, I will first load my filter for major process operations – filtering on PROCESS CREATE, PROCESS START, LOAD IMAGE, and PROCESS EXIT.
Assuming I stopped the trace right after the error, I can then start at the last exit of CONHOST.EXE and work backwards towards the exits of the CMD.EXE and the actual console application. If this were a batch file, it would just be CMD.EXE. In the example above, it is the db2cmd.exe process. In the above example, the exit code for db2cmd.exe was -2029059760.
[Incidentally, this was an easy fix (thank you VFS Write Mode.)]
As you may have figured out by now, App-V 5 has components that are referred to by multiple names depending on context. In some cases, App-V 5 basically takes the synonym concept to interesting levels. This can be misleading if misinterpreted. This can lead ISV’s and IT Pros down the wrong path with regards to troubleshooting, integrating with, or developing for – App-V. App-V 5 also brought us a whole bunch of new things that were really only new to the App-V product itself and not necessarily new to Windows as a whole. This is pretty clear once you are pointed back to the API reference. So to clear up potential confusion. I would like to take the time to point some of this information out. For those of you who are already App-V 5 experts, you may find this a little rudimentary and (dare I say it) boring – but we do have users who are new to App-V now following this blog on a regular basis. ?
A Virtual Process is also an Application
Whereas we had packages that contained applications in App-V 4.x and each application had its own set of configuration metadata, now we have metadata tied only to the package and the applications are launched by executables directly. Virtual applications are specifically pointed out and enabled within the package metadata. Applications are referred to as virtual processes when it comes to manipulation through PowerShell (i.e. the Start-AppVVirtualProcess cmdlet), but when turning on or off applications within Configuration Manager or the App-V Management Console, we are speaking still of Applications.
Targeting Reflects Publishing Method
User Targeting within a management system translates to the application being published to the user on the client machine. Machine Targeting within a management system, translates to the application being published to the machine globally.
A Connection Group is a Virtual Environment . . . and also a Package Group
A Connection Group is a set of XML metadata that defines a virtual environment. When a Connection Group is defined manually or through the App-V Publishing System, it is referred to as a Connection Group. When a Connection Group is created and deployed via Configuration Manager, it is referred to as a Virtual Environment. You may have already been made aware of that but also be aware that Connection Groups are also referred to as Package Groups in the context of the Client Catalog. Speaking of the Client Catalog . . .
A Manifest is Configuration but not Policy
The Manifest is the embedded configuration XML metadata that stored in the AppXManifest.XML within the APPV package. Dynamic Configuration is an external configuration policy that is tied to users and/or computers. Dynamic Configuration is also referred to as policy. The Catalog is not just the manifest but rather the Catalog is the combination of the two configurations for each packages and package groups.
Integration Links are Actually Junctions
For both User and Global Publishing, the integration links for shortcuts are created to sync up with the most recent package version within the immutable package cache. This saves us from having to recreate these when a package version changes. These links contain junction points not symbolic links. A symbolic link can be either a file or a directory. It looks to be the same as a Junction Point when viewed in Explorer, but it is spelled out as a Symbolic link to a file or a directory when viewed through the command prompt:
A symbolic link is defined here: http://msdn.microsoft.com/en-us/library/windows/desktop/aa365680(v=vs.85).aspx. A Junction Point is just that, a Junction point. Not a hard link. Junction Points are implemented through reparse points and are defined here:
Can you create your own integration points? Yes, you can. You can use the http://technet.microsoft.com/en-us/sysinternals/bb896768.aspx instead of mklink /d which would actually create a symbolic link.
A Shadow File is a Sparse File . . . and Friggin’ Awesome as Well.
Shadow files are what are created in the immutable package cache when a package is published. This is where a file is created that denotes the total size but does not consume disk space other than the allocation unit required to create the NTFS file record. Data will then populate these files upon streaming the application’s files to the disk. In the case of the Shared Content Store, these files will not get populated on disk (with some exceptions for portions of primary executables.) Shadow files are implemented using NTFS Sparse files defined here:
Sparse files appear in Explorer with a “x” to denote it has not been completely populated with data. The difference between the appearance is outlined visually below:
When viewing sparse files in the command prompt, you will see that the sparse file size indicator has parenthesis, if there are no parentheses, they are fully cached files.
To Mount is to Stream
To load or basically pre-cache, or pre-stream, we use the “Mount-AppVClientPackage” cmdlet in PowerShell. So if you wanted to manually pre-cache all of the binaries of a virtual application, you would Mount the package. In previous versions of App-V, this would also be referred to as loading.
Update 12/5/2014: The PVAD feature is now optional as of App-V 5.0 SP3. Also SP3 allows for merged roots with Connection Groups. Read more here: http://technet.microsoft.com/en-us/library/dn858700.aspx#BKMK_pvad_hidden
If you have ever dealt with me directly as a customer, attended one of my presentations, or even simply stomached one of my diatribes in a casual, technical conversation, you have no doubt heard about one of my pet peeves (no, not “Tips –n- Tricks” – we’ll visit that another time) but the term “best practices.”
I loathe that term. I know we are guilty of it at Microsoft – so are just about every single technology-based organizations. In our ever-evolving industry, to put something down in print as THE ABSOLUTE BEST PRACTICES goes against the very nature of the word “practice.” It can seem arrogant. It implies way too much finality. It also can mean serious disruption when a practice changes. I work in a consulting practice. As products, regulations, politics, and trends change, so do our recommended practices. This is why I purposely try to use “under the current recommended practice” or simply just “current recommended practices.”
When it comes to recommended practices involving the sequencing of applications with App-V, I have been evolving my recommended practices particularly in the area of whether or not to use the PVAD. Specifically, whether to ever recommended to match the installation directory to the directory that was used for the Primary Virtual Application Directory. You may have also heard this as “Sequence to the PVAD, Install to the PVAD.”
Back in May, I expanded upon the subject of the PVAD and how it served as a placeholder for root-based files and directories (http://blogs.technet.com/b/gladiatormsft/archive/2014/05/24/app-v-5-on-sequencing-using-tokenized-paths-pvad-s-vfs-and-vfs-write-mode.aspx.) These differ from VFS-based resources (i.e. those beneath rootvfs) in how the App-V 5 file system handles file operations. How you use the PVAD does have implications in terms of how:
- KNOWNFOLDERID’s always get translated to proper App-V Tokens
- Directories and Files get properly merged within Connection Groups
- Environment variables get properly translated
- VFS Write Mode is effective
Well after adjusting recommendations for App-V 5 regarding sequencing for connection groups, tokenization, and environment variables, I am now making the following recommendation to ensure that the above items always happens: It is best to always sequence with a fake PVAD which means create a false PVAD that will be different from the installation directory.
Now the next logical question would be: Are there any circumstances where this would create problems with newly sequenced virtual applications? I don’t think so. But I await your comments. 🙂
Update 12/5/2014: The PVAD feature is now optional as of App-V 5.0 SP3. Also SP3 allows for merged roots with Connection Groups. Read more here: http://technet.microsoft.com/en-us/library/dn858700.aspx#BKMK_pvad_hidden
UPDATE: 10/21/2014: The MVMC 3.0 is now released with P2V functionality restored.
I work with SCVMM (System Center Virtual Machine Manager) frequently in many different contexts. I even do the occasional private cloud engagement specifically on VMM and Hyper-V. Most of the time however, I am using VMM in a peripheral context – be it personal lab work, proof-of-concept labs for customer or partners, etc. I have been very pleased with the evolution of Hyper-V and System Center products over the last few years. I find the largest issues that create pain points for me involve the constant need to service virtual machines, deal with physical-to-virtual conversions, and the cumbersome process of building test networks that demonstrate elements such as multi-tenancy that require me to super impose logical switches and other elements of software-defined networking on top of my switching fabric.
I field a lot of questions with regards to how to best go about these options with the most recent versions of SCVMM (particularly VMM 2012 R2.)
Virtual Machine Servicing
I don’t keep all of virtual machines running at the same time. In addition, I have many templates for which I reuse/import/export on a regular basis. In VMM 2012 there was the option of using a separate add-on utility called the Virtual Machine Servicing Tool. The problem is it only was for VMM 2012 RTM (or R1) and it does not work with VMM 2012 SP1 or VMM 202 R2. You will likely find many questions regarding this that appear in the comments section on my initial blog about the VSMT 2012 utility way back in 2012. http://blogs.technet.com/b/gladiatormsft/archive/2012/08/14/the-virtual-machine-servicing-tool-2012-is-now-available.aspx
So with there being no newer version of VSMT for 2012 SP1 or R2 and the fact that you cannot use VSMT 2012 on VMM 2012 SP1 or R2, what are your options going forward for servicing – particularly offline servicing? You have a few options:
- Customize a solution with DISM (Deployment Image Service and Management Toolkit) You should be very familiar with DISM as it is very useful for the consultant and IT Pro (like me) who does not always have access to System Center infrastructures. It can also be easily scripted to mount and service offline images for OS updates at the very least. You can become familiar with DISM servicing using the following link as it is a great introduction to the concept: http://technet.microsoft.com/en-us/library/dd799267(v=WS.10).aspx. This walkthrough tells you how to mount a virtual disk online and then apply various servicing commands using the DISM tool. You can then apply updates using the tool to apply individual Windows Update packages (.MSU’s) although this can be cumbersome for many sets of updates. This does require scripting for effectiveness but I have found that I can get away with one set per OS so long as I have access to the individual .MSU files [DISM /image:C:MyDirMount /Add-Package /Packagepath:<file_path>] This way is still way quicker than standing up a VM running WSUS, keeping it in sync and then booting up every single VM and updating it through the WSUS server. There are also additional scripts out there that work with live WSUS servers and DISM that you can also try – for example http://gallery.technet.microsoft.com/Offline-Servicing-of-VHDs-df776bda#content – Offline Servicing of VHDs against WSUS
- Use Configuration Manager 2012 R2: Configuration Manager 2012 R2 has a VHD patching feature that allows you to apply software updates to VHDs that you created using task sequences. While this requires Configuration Manager, it is a great option for offline servicing. More information on this can be found here: http://technet.microsoft.com/en-us/library/dn448591.aspx
- Orchestration: You can use a solution provided by a SMA (Service Management Automation) Runbook. The following blog posts talks about a feature in the gallery that allows you to automate the process of offline servicing: http://blogs.technet.com/b/privatecloud/archive/2013/12/07/orchestrated-vm-patching.aspx The specific runbook is found in the Technet gallery here: http://gallery.technet.microsoft.com/Orchestrated-Offline-VM-c90492db
The built-in Physical to Virtual conversion component of VMM was deprecated with the release of SCVMM 2012 R2. I wrote about this and the alternative options earlier this year: http://blogs.technet.com/b/gladiatormsft/archive/2014/01/18/virtualize-but-how-p2v-i-thought-you-took-that-feature-out-of-vmm.aspx Many had hoped the feature would be included in the release of the Microsoft Virtual Machine Conversion utility (MVMC 2.0 ) http://blogs.technet.com/b/gladiatormsft/archive/2014/04/12/the-microsoft-virtual-machine-converter-2-0-is-now-live-on-the-microsoft-download-center.aspx but this was an erroneous speculation. P2V will be returning with the MVMC 3.0 release that will likely come later this fall. In the meantime use Disk2VHD as I mentioned in my post earlier as a viable alternative.
VMM Network Builder
Getting virtual networks set up properly in VMM and having everything in sync with the Hyper-V virtual switches, Host configurations, and the underlying switch fabric can be a cumbersome task. Up until now, I have been longing for a simplification of the process of setting up networking in VMM. Now we have the greatest single add-on utility (in my opinion) to come to SCVMM 2012: The VMM Network Builder. This is a free download that just became available from the Download Center (http://www.microsoft.com/en-us/download/details.aspx?id=43975) This is a tool that will simplify the process of creating virtual networks that utilize VLAN isolation through VMM.
This will ensure that the Host NICs have the proper consistent settings for all of your virtual networks so all of your virtual machines will be able to be set properly to the appropriate virtual network associated with the correct VLAN. This will reduce the instances of having to troubleshoot network configuration which can be a common pain point given the many levels where things can be set incorrectly. With this utility, you can do a simple basic networking setup that can be applied to all of your hosts.