Archive

Posts Tagged ‘cm2012’

App-V 5: On the LocationProvider and the IgnoreLocationProvider Feature


In a previous blog entry, (http://blogs.technet.com/b/gladiatormsft/archive/2014/12/10/app-v-5-on-the-packagesourceroot.aspx) I discussed the PackageSourceRoot override and how it can be used to control source content locations for packages. There is another option for overriding source content locations for App-V packages: the LocationProvider registry value located in HKEY_LOCAL_MACHINESOFTWAREMicrosoftAppVClientStreaming.

This registry value is not designed to be changed or adjusted manually. It is simply a configuration item that denotes the COM interface and its subsequent registration. When this value is empty, that means there is no LocationProvider interface registered. If one is registered {by GUID} than whatever its setting for the package source root per package takes precedent over the PackageSourceRoot registry setting or other per-package settings. This is how the Configuration Manager client hooks into the App-V client. It uses the COM provider called the VAppLaunchManager which essentially takes over package management with an event-driven methodology.

From the context of how is overrides the PackageSourceRoot, think of this as being a replacement for the manual registry setting of the OverrideURL setting done in previous versions of App-V and SCCM integration. If the application has not been streamed or fully loaded, the App-V Streaming Subsystem will reference this interface to retrieve the Override URL for the package (i.e. the SCCM Distribution Point) from which the package will stream from. This will happen under each time there is a:

  • First connection to a package.

  • Reconnection after a previous session was closed or a user has logged off.

  • Change in the network (move to new network, network interface reset, etc.)

The interface will be registered initially once the clients receives the first targeted advertisement of an App-V 5 virtual application from Configuration Manager. This is a much improved experience from the implementation of Configuration Manager 2007 and App-V 4.6 as existing packages will remain on the client

Now this brings up another likely question: Can you create exceptions to clients being controlled by the Configuration Manager client or some other ISV that might leverage the LocationProvider interface? Let’s say you have a subset of computers within a collection that you not only do not want receiving virtual advertisements from Configuration Manager, but you may also desire managing the applications by way of another mean altogether. In previous implementations of Configuration Manager and App-V integration, field resources came up with using custom policy exceptions (see Rob York’s old blog here: http://blogs.technet.com/b/virtualworld/archive/2010/07/07/using-sccm-local-policy-to-selectively-restrict-app-v-integration.aspx) and this worked.

So if you wanted to globally manage all of your resource physical machines, virtual machines, and devices through Configuration Manager (including the delivery of virtual applications) except for possibly a subset of machines in which you may want to manage the applications in a stand-alone fashion (i.e. RDS Servers, etc.) – how do you go about setting that exception in App-V 5? You could probably easily go about the same process – but what if you wanted to use Configuration Manager to publish the applications but still take advantage of the PackageSourceRoot? Why? Well, for reasons such as:

  • Having multiple App-V delivery systems but would like to reduce duplicity on content between content servers and distribution servers.

  • You want to manage affinity with content locations out-of-band from Configuration Manager

  • You want to provide streaming high availability with better failover than the distribution point failovers in Configuration Manager (which are not instantaneous as load-balanced shares.)  

You can have this by setting the value IgnoreLocationProvider to 0x1 (DWORD) in HKEY_LOCAL_MACHINESOFTWAREMicrosoftAppVClientStreaming. This setting will force the client to ignore the path returned by the LocationProvider interface and instead use the Package Source Root. This was first introduced in App-V 5 SP2 but it was somewhat problematic. The feature works well now in Service Pack 3.

VMM: Options for Offline Servicing, P2V, and Building Virtual Networks

August 23, 2014 1 comment

UPDATE: 10/21/2014: The MVMC 3.0 is now released with P2V functionality restored.

http://www.microsoft.com/en-us/download/details.aspx?id=42497

I work with SCVMM (System Center Virtual Machine Manager) frequently in many different contexts. I even do the occasional private cloud engagement specifically on VMM and Hyper-V. Most of the time however, I am using VMM in a peripheral context – be it personal lab work, proof-of-concept labs for customer or partners, etc. I have been very pleased with the evolution of Hyper-V and System Center products over the last few years. I find the largest issues that create pain points for me involve the constant need to service virtual machines, deal with physical-to-virtual conversions, and the cumbersome process of building test networks that demonstrate elements such as multi-tenancy that require me to super impose logical switches and other elements of software-defined networking on top of my switching fabric.

I field a lot of questions with regards to how to best go about these options with the most recent versions of SCVMM (particularly VMM 2012 R2.)

Virtual Machine Servicing

I don’t keep all of virtual machines running at the same time. In addition, I have many templates for which I reuse/import/export on a regular basis. In VMM 2012 there was the option of using a separate add-on utility called the Virtual Machine Servicing Tool. The problem is it only was for VMM 2012 RTM (or R1) and it does not work with VMM 2012 SP1 or VMM 202 R2. You will likely find many questions regarding this that appear in the comments section on my initial blog about the VSMT 2012 utility way back in 2012. http://blogs.technet.com/b/gladiatormsft/archive/2012/08/14/the-virtual-machine-servicing-tool-2012-is-now-available.aspx

So with there being no newer version of VSMT for 2012 SP1 or R2 and the fact that you cannot use VSMT 2012 on VMM 2012 SP1 or R2, what are your options going forward for servicing – particularly offline servicing? You have a few options:

  • Customize a solution with DISM (Deployment Image Service and Management Toolkit) You should be very familiar with DISM as it is very useful for the consultant and IT Pro (like me) who does not always have access to System Center infrastructures. It can also be easily scripted to mount and service offline images for OS updates at the very least. You can become familiar with DISM servicing using the following link as it is a great introduction to the concept:  http://technet.microsoft.com/en-us/library/dd799267(v=WS.10).aspx. This walkthrough tells you how to mount a virtual disk online and then apply various servicing commands using the DISM tool. You can then apply updates using the tool to apply individual Windows Update packages (.MSU’s) although this can be cumbersome for many sets of updates. This does require scripting for effectiveness but I have found that I can get away with one set per OS so long as I have access to the individual .MSU files [DISM /image:C:MyDirMount /Add-Package /Packagepath:<file_path>] This way is still way quicker than standing up a VM running WSUS, keeping it in sync and then booting up every single VM and updating it through the WSUS server. There are also additional scripts out there that work with live WSUS servers and DISM that you can also try – for example http://gallery.technet.microsoft.com/Offline-Servicing-of-VHDs-df776bda#content – Offline Servicing of VHDs against WSUS
  • Use Configuration Manager 2012 R2: Configuration Manager 2012 R2 has a VHD patching feature that allows you to apply software updates to VHDs that you created using task sequences. While this requires Configuration Manager, it is a great option for offline servicing. More information on this can be found here: http://technet.microsoft.com/en-us/library/dn448591.aspx
  • Orchestration: You can use a solution provided by a SMA (Service Management Automation) Runbook. The following blog posts talks about a feature in the gallery that allows you to automate the process of offline servicing: http://blogs.technet.com/b/privatecloud/archive/2013/12/07/orchestrated-vm-patching.aspx The specific runbook is found in the Technet gallery here: http://gallery.technet.microsoft.com/Orchestrated-Offline-VM-c90492db

 

P2V

The built-in Physical to Virtual conversion component of VMM was deprecated with the release of SCVMM 2012 R2. I wrote about this and the alternative options earlier this year: http://blogs.technet.com/b/gladiatormsft/archive/2014/01/18/virtualize-but-how-p2v-i-thought-you-took-that-feature-out-of-vmm.aspx Many had hoped the feature would be included in the release of the Microsoft Virtual Machine Conversion utility (MVMC 2.0 ) http://blogs.technet.com/b/gladiatormsft/archive/2014/04/12/the-microsoft-virtual-machine-converter-2-0-is-now-live-on-the-microsoft-download-center.aspx but this was an erroneous speculation. P2V will be returning with the MVMC 3.0 release that will likely come later this fall. In the meantime use Disk2VHD as I mentioned in my post earlier as a viable alternative.

VMM Network Builder

Getting virtual networks set up properly in VMM and having everything in sync with the Hyper-V virtual switches, Host configurations, and the underlying switch fabric can be a cumbersome task. Up until now, I have been longing for a simplification of the process of setting up networking in VMM. Now we have the greatest single add-on utility (in my opinion) to come to SCVMM 2012: The VMM Network Builder. This is a free download that just became available from the Download Center (http://www.microsoft.com/en-us/download/details.aspx?id=43975) This is a tool that will simplify the process of creating virtual networks that utilize VLAN isolation through VMM.


This will ensure that the Host NICs have the proper consistent settings for all of your virtual networks so all of your virtual machines will be able to be set properly to the appropriate virtual network associated with the correct VLAN. This will reduce the instances of having to troubleshoot network configuration which can be a common pain point given the many levels where things can be set incorrectly. With this utility, you can do a simple basic networking setup that can be applied to all of your hosts.

App-V 5: On Asset Intelligence


Asset Intelligence has been used for years to enhance the inventory capabilities of Microsoft System Center Configuration Manager (SCCM) by extending hardware inventory and adding license management functionality. The Asset Intelligence features of SCCM (2012, 2012 SP1, and 2012 R2) can report application data such as digital PID, MSI product codes, and publisher names for each virtual application registered on a client computer. With App-V 4.6 virtual applications, this was hindered by some limitations including lack of full integration into the SFT package as well as FB1 definitions to be defined for every applications.

App-V 5 improves upon the limitations of SCCM collecting Asset Intelligence data by incorporating Asset Intelligence information into the package.

Captured During Sequencing

Using a special process, the Sequencer will capture Asset Intelligence metadata during the monitor phase. The data is then placed inside the AppXManifest file embedded into the APPV package. This is the only supported time the information can be placed inside the package. In most cases, App-V 5 packages will have asset intelligence unless the application itself specifically does not install that metadata or is an older legacy application without an installer.

 

Manifest Values

The package manifest stores this information under the <AssetIntelligence> element. Additional properties are then specified under the sub-element <AssetIntelligenceProperties>:

<AssetIntelligenceProperties>

     <SoftwareCode/>

     <ProductName/>

     <ProductVersion/>

     <Publisher/>

     <ProductID/>

     <Language/>

     <ChannelCode/>

     <InstallDate/>

     <RegisteredUser/>

     <InstalledLocation/>

     <CM_DSLID/>

     <VersionMajor/>

     <VersionMinor/>

     <ServicePack/>

     <UpgradeCode/>

     <OsComponent/>

     </AssetIntelligenceProperties>

</AssetIntelligence>

Please note that Asset Intelligence data is optional and not required for valid AppV operation for an application. This information is provided for the purposes of inventory with SCCM.

Appearance on Client

The Asset Intelligence marker is attached to the package record in the registry and WMI to inform the AppVHandler in SCCM 2012 SP1 and R2 whether or not the package has asset intelligence.

You can see this when you type Get-AppvClientPackage in PowerShell under the HasAssetIntelligence value.

Why would a package not have Asset Intelligence?

There are a few reasons why an App-V 5 virtual application would not have this information in the manifest:

  • Timed out during sequencing: If the AI sequencing collector fails to collect the information within a specific period, it will time out. You can disable this value by adding a DWORD value called DisableTimeout to HKLMSoftwareMicrosoftAppVSequencerAssetIntelligence (Will need to create key by default.) Set this value to 1.

  • No attempt to collect: If the element is missing from the manifest, it means that no Asset Intelligence was gathered during sequencing. You will always see this if the package was converted but it also could be tied to an error collecting it on the sequencing workstation.  This means that SCCM will have to manual scan this package as if it were a legacy package by injecting itself into the virtual environment. 

  • Attempt was made but failed: If the element exists but is empty, it means that collection was attempted and nothing was gathered. 

 

App-V and CM 2012: GRRRRRRRR! Where did “Remove this virtual application when it is no longer advertised” go!?!?!?

May 22, 2013 6 comments

With Configuration Manager 2007 Integration in App-V 4.6, there was a very popular feature that allowed for what we refer to as “implicit uninstall.” This means that when a user is no longer part of collection, the virtual applications advertised to them would then be removed. This was very easy to configure by selecting “Remove this virtual application when it is no longer advertised” when importing the virtual application. This meant the client would uninstall the virtual application if they are removed from the target collection.

This exact feature is not implemented this exact same way with Configuration Manager 2012. The APP-V 5/Configuration Manager 2012 Integration whitepaper has been updated to include a methodology for achieving this same effect through custom collections – both through the console or through WQL.

Page 30 of the updated Configuration Manager White Paper discusses this:

http://download.microsoft.com/download/C/0/1/C0199142-8A8F-42A3-9D3F-6E807CB248F9/Managing_AppV5_with_Configuration_Manager_2012SP1.pdf

Categories: Uncategorized Tags: , , ,

Must have (User Experience Virtualization) Deployment and Management Resources!


It was no coincidence that the inaugural release of Microsoft User Experience Virtualization (UE-V) directly coincided with the release of APP-V 5.0 For most modern enterprise user scenarios (BYOD, flexible workspace, mobile computing, etc.) the need to have portable applications that roam user settings across application delivery formats is an essential ask from customers. UE-V also allows more granularity to users and administrators allowing key applications to roam settings without the bloat and overhead of roaming user profiles (yes, I said it.)

As I have been doing with APP-V 5.0 and Office 2013, I am continuing on my theme of aggregating essential content for deploying key elements of newest Windows client solutions. I have compiled this list of essential UE-V resources and like the others, I will update this as additional key content becomes available.

 

White Papers:

UE-V deployment Guide:
http://www.microsoft.com/en-us/download/details.aspx?id=35495

UE-V Cmdlet Resources
http://www.microsoft.com/en-us/download/details.aspx?id=41195

Technet Resources:

Technet Online Administrative Guidance:
http://technet.microsoft.com/en-us/library/jj680015.aspx

UE-V Technet Main Page:
http://technet.microsoft.com/en-us/windows/hh943107

UE-V Technet Forum:
http://social.technet.microsoft.com/Forums/en-US/mdopuev/threads

Technet Lab: UE-V Deployment
https://msevents.microsoft.com/CUI/EventDetail.aspx?EventID=1032537552&culture=en-US

 

UE-V Template Resources:

UE-V Community Template Gallery
http://gallery.technet.microsoft.com/site/search?f%5B0%5D.Type=RootCategory&f%5B0%5D.Value=UE-V&f%5B0%5D.Text=UE-V&sortBy=Date

 

UE-V TechNet Wiki Articles:

UE-V and SkyDrive Integration Not Supported
http://social.technet.microsoft.com/wiki/contents/articles/14425.ue-v-and-skydrive-integration-not-supported.aspx

How to repair an unsuccessful UE-V Uninstall
http://social.technet.microsoft.com/wiki/contents/articles/14489.how-to-repair-an-unsuccessful-ue-v-uninstall.aspx

UE-V and High Availability
http://social.technet.microsoft.com/wiki/contents/articles/16575.ue-v-and-high-availability.aspx

UE-V File Versions
http://social.technet.microsoft.com/wiki/contents/articles/16727.ue-v-file-versions.aspx

How to Verify a Successful UE-V Installation
http://social.technet.microsoft.com/wiki/contents/articles/16936.how-to-verify-a-successful-ue-v-installation.aspx

Technet Videos:

UE-V Overview:
http://technet.microsoft.com/en-us/windows/hh925634

UE-V Deployment and Setup:
http://technet.microsoft.com/en-us/windows/jj568979

Group Policy Management of UE-V:
http://technet.microsoft.com/en-us/windows/jj591520

Deploying the UE-V Agent with Config Manager
http://technet.microsoft.com/en-us/windows/jj573734

Deploying the UE-V Agent with MDT
http://technet.microsoft.com/en-us/windows/jj573733

 

KB Articles:

UE-V Template Verification Steps and Best Practices
http://support.microsoft.com/kb/2769981

UE-V Registry Settings
http://support.microsoft.com/kb/2770042

How to repair a corrupted UE-V install
http://support.microsoft.com/kb/2769631

Microsoft Office Support in Microsoft User Experience Virtualization (UE-V)
http://support.microsoft.com/kb/2841206/en-US

How to troubleshoot UE-V replication issues
http://support.microsoft.com/kb/2844022/en-US

UE-V settings replicated by Internet Explorer
http://support.microsoft.com/kb/2847017/en-US

UE-V Registry Settings
http://support.microsoft.com/kb/2770042

UE-V Template Verification Steps and Best Practices
http://support.microsoft.com/kb/2769981

UE-V Agent installation fails with error: The Offline Files service is not running
http://support.microsoft.com/kb/2769614

UE-V does not update the theme on RDS or VDI sessions
http://support.microsoft.com/kb/2769570

 

Add-on Downloads:

Microsoft Desktop Optimization Pack Administrative Templates
http://www.microsoft.com/en-us/download/details.aspx?id=35516

System Center 2012 Configuration Pack for Microsoft User Experience Virtualization 2.0
http://www.microsoft.com/en-us/download/details.aspx?id=40913

 

 

Categories: Uncategorized Tags: , , , , , , ,

Must Have App-V 5.0 Migration Resources!

January 25, 2013 7 comments

Yes, there is a learning curve with the new release of App-V. This is why it is extremely important that – Before you embark on and App-V 5.0 deployment or migration, that you look to the following resources for assistance in migrating to App-V 5.0.

App-V 5.0 Published Whitepapers:

App-V 5.0 Trial Guide:

http://download.microsoft.com/download/F/7/8/F784A197-73BE-48FF-83DA-4102C05A6D44/App-V%20%205.0%20Trial%20Guide.docx

App-V 5.0 Sequencing Guide:

http://download.microsoft.com/download/F/7/8/F784A197-73BE-48FF-83DA-4102C05A6D44/App-V%205.0%20Sequencing%20Guide.docx

Managing App-V 5.0 with Configuration Manager 2012 SP1

http://download.microsoft.com/download/C/0/1/C0199142-8A8F-42A3-9D3F-6E807CB248F9/Managing_AppV5_with_Configuration_Manager_2012SP1.pdf

Integrating App-V with Microsoft VDI White Paper

http://download.microsoft.com/download/B/B/7/BB782D16-67A9-4E54-8BB3-28A898DFFBA6/Integrating_App-V_with_Microsoft_VDI.pdf

App-V and Citrix Integration Overview

http://download.microsoft.com/download/C/4/1/C416F9FD-1B51-4E1D-9DBE-87C12AB329A2/App-V5_and_Citrix_Integration_Overview.docx

App-V 5 SP2 Publishing and Client Interaction

http://www.microsoft.com/en-us/download/details.aspx?id=41635

App-V 5.0 Supplemental Downloads

Microsoft Application Virtualization Server 5.0 Best Practice Analyzer

http://www.microsoft.com/en-us/download/details.aspx?id=38844 

System Center Management Pack for Microsoft Application Virtualization Server 5.0

http://www.microsoft.com/en-us/download/details.aspx?id=38418

Microsoft Desktop Optimization Pack Administrative Templates 2.0

http://www.microsoft.com/en-us/download/details.aspx?id=41183

 

TechNet Resources: 

Online App-V 5.0 Administrator’s guide:

http://technet.microsoft.com/en-us/library/jj713487.aspx

Deploying the App-V 5.0 Sequencer and Client

http://technet.microsoft.com/en-us/library/jj713463.aspx

TechNet Forum

http://social.technet.microsoft.com/Forums/en-US/mdopappv/threads

Technet Virtual Lab:

https://cmg.vlabcenter.com/default.aspx?moduleid=61991289-6281-43f6-bc7c-22a7696b3c6d

Main Resource page:

http://technet.microsoft.com/en-us/windows/hh826068.aspx?ocid=wc-int-news

How to Install the App-V Client for Shared Content Store Mode

http://technet.microsoft.com/en-us/library/jj713455.aspx

Blog Posts:

Microsoft Application Virtualization 5.0 Integration

http://blogs.technet.com/b/appv/archive/2013/01/03/microsoft-application-virtualization-5-0-integration.aspx

Scripting and Embedded Scripting for AppV 5.0 (Dynamic Deployment and User Configuration Scripting)

http://blogs.technet.com/b/appv/archive/2012/12/10/scripting-and-embedded-scripting-for-appv-5-0-dynamic-deployment-and-user-configuration-scripting.aspx

Sequencing for Connection Groups

http://blogs.technet.com/b/appv/archive/2012/11/29/sequencing-for-connection-groups.aspx

App-V 5.0 Client PowerShell Deep Dive

http://blogs.technet.com/b/appv/archive/2012/12/03/app-v-5-0-client-powershell-deep-dive.aspx

How to Collect App-V 5.0 Debug Traces

http://blogs.technet.com/b/appv/archive/2013/01/23/how-to-collect-app-v-5-0-debug-traces.aspx

App-V Scripting

http://blogs.technet.com/b/gladiatormsft/archive/2013/05/02/app-v-5-scripting-change.aspx

How to run processes inside the App-V Virtual Bubble

http://blogs.technet.com/b/appv/archive/2013/04/24/how-to-launch-native-local-processes-within-the-app-v-5-0-virtual-environment.aspx


Knowledge Base Articles:

Known issues when using App-V v5 Package Converter

http://support.microsoft.com/kb/2774436

Important changes in the App-V v5 Sequencer

http://support.microsoft.com/kb/2777777

How to remove a cached copy of an unpublished package in Microsoft App-V v5

http://support.microsoft.com/kb/2768945

Five common App-V v5 implementation scenarios

http://support.microsoft.com/kb/2780807

Supported Configuration Manager and Microsoft Application Virtualization scenarios

http://support.microsoft.com/kb/2780810

How To Use Microsoft User Experience Virtualization With App-V Applications

http://support.microsoft.com/kb/2850582

Known issues when creating or using an App-V 5.0 Office 2010 package

http://support.microsoft.com/kb/2828619

How to adjust the App-V v5 Publishing Server Refresh schedule

http://support.microsoft.com/kb/2780177/en-us

How to use PowerShell to find packages in App-V v5 client connection groups

http://support.microsoft.com/kb/2780809/en-us

How to troubleshoot publishing server refresh failures in App-V v5

http://support.microsoft.com/kb/2778168/en-us

Add-on Downloads

Microsoft App-V 5.0 Client UI Application Download

http://www.microsoft.com/en-us/download/details.aspx?id=41186

Microsoft App-V 5.0 Server Best Practice Analyzer

http://www.microsoft.com/en-us/download/details.aspx?id=38844

 

Office 2013 and APP-V 5.0

Supported scenarios for deploying Microsoft Office as a sequenced App-V Package

http://support.microsoft.com/kb/2772509

Download: Office Customization Tool for Click-to-Run

http://www.microsoft.com/en-us/download/details.aspx?id=36778

Customizing Office 2013 using the Office Click-to-Run Deployment Tool

http://technet.microsoft.com/en-us/library/jj219424(v=office.15)

Customization Overview for Click-to-Run

http://technet.microsoft.com/en-us/library/jj219428(v=office.15)

Deployment Options for Office 2013

http://technet.microsoft.com/en-us/library/ee624360.aspx#Virtualization

 

Office 2010 and App-V 5.0

Microsoft Office 2010 Sequencing Kit for Microsoft Application Virtualization 5.0

http://www.microsoft.com/en-us/download/details.aspx?id=38399

How to sequence Microsoft Office 2010 in Microsoft Application Virtualization 5.0

http://support.microsoft.com/kb/2830069

 

 

The Case of the Mysterious Open SFT Handle

December 6, 2012 Leave a comment

Here is another interesting one-off issue that was happening on a few machines in one of my customer’s environments. They were using App-V 4.6 with Configuration Manager 2012 managing the packages. The virtual applications were distributed fully cached to the clients (download and execute.) The problem was that the download to the cache would never be able to progress beyond 99% thus the application would never become available to the client.  This was happening on all virtual applications for the affected clients. The Configuration Manager CAS.LOG showed the following:
 
Download completed for content Content_74cfa5bd-d3981-21fc-2316-4c3e8659f7a690.1 under context System      ContentAccess   12/5/2012 11:15:35 AM   4460 (0x116C)
CreateFileW failed for c:windowsccmcache11xxxxxxxx.sft      ContentAccess   12/5/2012 11:15:35 AM   4460 (0x116C)
???? failed; 0x80070020 ContentAccess   12/5/2012 11:15:35 AM   4460 (0x116C)
?????t failed; 0x80070020       ContentAccess   12/5/2012 11:15:35 AM   4460 (0x116C)
????????? failed; 0x80070020    ContentAccess   12/5/2012 11:15:35 AM   4460 (0x116C)

The specific HREF error code 80070020 translates to “The process cannot access the file because it is being used by another process.”

Process Explorer to the Rescue

Using Process Explorer (found here: http://technet.microsoft.com/en-us/sysinternals/bb896653) we found that the “System” process had an open handle to all of the various SFT files in the CCM cache (C:Windowsccmcache11xxxxxxxx.sft.) Using MSConfig and disabling all 3rd-party services and startup items (as well as the Configuration Manager client service (SMS Agent Host) we still found that the system STILL had an open handle to all of these SFT files in the CCM cache. Further investigation of the stack revealed there was a mini-filter driver attaching to the SFT files. The filter was identified in Process Explorer as AppVFltrPort. This corresponded to the SFTVIEW.SYS file. This file was part of the Microsoft Application Virtualization SFT View application (that is available from http://www.microsoft.com/en-us/download/details.aspx?id=8897).  It has a mini-filter driver that attaches to SFT files even when you are not using the program.  The problem shows up as soon as something uses the file system near (one level down) to a SFT file on a client computer. 

Uninstall SFTVIEW from Clients

In the above case, the solution was to simply uninstall SFTVIEW or disengage the AppVFltrPort driver. The SFTVIEW tool was meant to be installed outside of production until you are ready for deployment onto content stores. The purpose of having this application on content stores is to provide read-only access to on-access anti-virus scanners so they can scan the contents of the SFT files. If you are looking to view content information or extract meta-data from SFT files, use the SFT Parser instead when working on clients. You can get that here: http://www.microsoft.com/en-us/download/details.aspx?id=12350. If you want anti-virus scanners to be able to scan the App-V client cache, use Service Inclusions instead. More information on Service Inclusions can be found here: http://blogs.technet.com/b/gladiatormsft/archive/2012/08/01/app-v-4-6-using-service-and-process-inclusions.aspx