Microsoft Application Virtualization 4.6 Service Pack 2 is now available via Microsoft Update. This also means that you can leverage WSUS as well as the software updates feature of Configuration Manager to deliver this update to your RDS and desktop clients. This is the third service pack that App-V has delivered via
Microsoft Update. You can view all the Microsoft Application Virtualization updates available via Microsoft Update from the following link: http://catalog.update.microsoft.com/v7/site/search.aspx?q=application%20virtualization
Here is another interesting one-off issue that was happening on a few machines in one of my customer’s environments. They were using App-V 4.6 with Configuration Manager 2012 managing the packages. The virtual applications were distributed fully cached to the clients (download and execute.) The problem was that the download to the cache would never be able to progress beyond 99% thus the application would never become available to the client. This was happening on all virtual applications for the affected clients. The Configuration Manager CAS.LOG showed the following:
Download completed for content Content_74cfa5bd-d3981-21fc-2316-4c3e8659f7a690.1 under context System ContentAccess 12/5/2012 11:15:35 AM 4460 (0x116C)
CreateFileW failed for c:windowsccmcache11xxxxxxxx.sft ContentAccess 12/5/2012 11:15:35 AM 4460 (0x116C)
???? failed; 0x80070020 ContentAccess 12/5/2012 11:15:35 AM 4460 (0x116C)
?????t failed; 0x80070020 ContentAccess 12/5/2012 11:15:35 AM 4460 (0x116C)
????????? failed; 0x80070020 ContentAccess 12/5/2012 11:15:35 AM 4460 (0x116C)
The specific HREF error code 80070020 translates to “The process cannot access the file because it is being used by another process.”
Process Explorer to the Rescue
Using Process Explorer (found here: http://technet.microsoft.com/en-us/sysinternals/bb896653) we found that the “System” process had an open handle to all of the various SFT files in the CCM cache (C:Windowsccmcache11xxxxxxxx.sft.) Using MSConfig and disabling all 3rd-party services and startup items (as well as the Configuration Manager client service (SMS Agent Host) we still found that the system STILL had an open handle to all of these SFT files in the CCM cache. Further investigation of the stack revealed there was a mini-filter driver attaching to the SFT files. The filter was identified in Process Explorer as AppVFltrPort. This corresponded to the SFTVIEW.SYS file. This file was part of the Microsoft Application Virtualization SFT View application (that is available from http://www.microsoft.com/en-us/download/details.aspx?id=8897). It has a mini-filter driver that attaches to SFT files even when you are not using the program. The problem shows up as soon as something uses the file system near (one level down) to a SFT file on a client computer.
Uninstall SFTVIEW from Clients
In the above case, the solution was to simply uninstall SFTVIEW or disengage the AppVFltrPort driver. The SFTVIEW tool was meant to be installed outside of production until you are ready for deployment onto content stores. The purpose of having this application on content stores is to provide read-only access to on-access anti-virus scanners so they can scan the contents of the SFT files. If you are looking to view content information or extract meta-data from SFT files, use the SFT Parser instead when working on clients. You can get that here: http://www.microsoft.com/en-us/download/details.aspx?id=12350. If you want anti-virus scanners to be able to scan the App-V client cache, use Service Inclusions instead. More information on Service Inclusions can be found here: http://blogs.technet.com/b/gladiatormsft/archive/2012/08/01/app-v-4-6-using-service-and-process-inclusions.aspx
I have encountered an interesting issue where a series of applications that have been virtualized with App-V 4.6 will fail due to an operational error. Ironically it has nothing to do with any kind of limits within App-V or is even related to the virtualization model of App-V. It happens because a key component of App-V that allows for the processing of OSD information fails due to interference from third-party software.
Crash Followed by Error
The problem begins when the majority of the applications experience an error with the SFTLP process. SFTLP.EXE will crash with the following error:
SFTLP.EXE – Application Error
The instruction at “0x0000XXXX” referenced memory at “0x0000XXXX”. The memory could not be “read.”
NOTE: You will see a slightly different error if Windows Error Reporting (WER) has been turned on.
Since SFTLP is an integral part of building the application’s virtual environment and it has crashed while in process, the launch will not complete leading to the following client launch error:
The Application Virtualization Client could not launch
An unexpected error occurred.
Report the following error code to your System Administrator. Error Code: xxxxxx-xxxxxx19-00001001
Further troubleshooting proved difficult as we were even unable to properly collect a user dump via WER (Windows Error Reporting) to determine why the SFTLP.EXE process was crashing. We tried attaching several debuggers to the AEDebug key and eventually was able to get further information on the crash. The reason WER was not very helpful was an inability to access the protected information within the bubble but we were at least able to know it was an access violation occurring. Other interesting elements also played a factor in the difficulty of troubleshooting:
- When the SFTLOG.TXT was set to verbose, the only thing that was revealed was that the SFTLP process encountered access violation exit code of 5 which we already knew.
- Use of the UnhandledExceptionLogger file SFTLDR.LOG also revealed nothing since this error occurred well before the primary executable of the virtual application was launched.
- Launching Procmon (Process Monitor) – either through the use of a command prompt or /externalcapture failed as the launch would start and immediately stop.
At this point, we determined we needed to look at something that was likely happening inside the operating system at the kernel level. Whenever something like this happens and I suspect that it may be third-party software with filter/kernel drivers, I will look at disabling unknown filter drivers. Bear in mind disabling non-Microsoft services and startup items using MSCONFIG will NOT disable third-party filters. You have to disable these filters by using Device Manager (I would advise this rather than directly editing the registry values when troubleshooting.) In Device Manager, you can help yourself identify these 3rd-party filter drivers by opening it up directly as an admin using devmgmt.msc or from the System Control Panel – and under “View” selecting “Show Hidden Devices.” This could be a tedious process, but absent any other troubleshooting data, it may be your best bet for narrowing down the issue. You can disable the driver from this interface and reboot in order to reproduce the issue with the driver disengaged.
AUTORUNS – A Shortcut
I use Autoruns as a quick way of telling me which filter drivers are 3rd-party. The Drivers tab will list of the drivers and their publisher. It helps me to quickly isolate the non-Microsoft filter drivers. NOTE: If there is a driver WITHOUT a publisher, I would definitely start with those. You can download Autoruns here: http://technet.microsoft.com/en-us/sysinternals/bb963902
In the issue above, we proceeded to disabled unknown filters in the Kernel until we came to one called “SysPlant for NT.” This particular driver registered under: HKEY_LOCAL_MACHINESYSTEMCurrentControlSetServicesSysPlant. The specific driver build in question came from Symantec and was at version 12.1.1000.157. Sure enough, when we disengaged the driver and rebooted all was well. Further research revealed that it was tied to Symantec Endpoint Protection. Even after trying Service and Process Inclusions for the SysPlant driver and the SepMasterService, it still was not helping with the driver engaged. We found that as long as we installed Symantec Endpoint Protection without the Device and Application control feature, we were able to avoid this problem as well.