App-V 5: Do you Still have to Run Process Monitor within the App-V Bubble when Troubleshooting Applications?
If – by that question you want to know if you must start an instance of Process Monitor within the virtual application like you did in 4.x – no. You could run process Monitor inside the virtual bubble, but it will not yield you much more results. The reason behind this is simple: unlike previous versions of App-V, the REAL registry as well as the native file system – NTFS – is used in App-V 5.
In App-V 4.6 and earlier, if you did not launch process Monitor in the virtual application’s environment (usually through a command prompt) all you would capture related to the virtual application would be operations to file and registry resources outside the virtual environment. In Version 5, running Process Monitor as normal will capture access to the actual locations including registry, package store, as well as VFS (Virtual File System) COW (Copy-on-Write) locations. Why just that? Because that’s where things “actually” are located. What you will have to understand is that once the operations to where the application “thinks” it is located has been hooked in the file system and/or registry every subsequent operation will continue as such to include operations only to the:
Integration junction points to the Package Store
Actual package Store paths
VFS COW (Copy-on-Write) checks and locations
You can see examples of these in the following screenshots from Process Monitor:
Procmon and File Operations:
As you can see above, the query to the initial location of where the virtual applications thinks it is browsing (C:program Files (x86)Java) is clearly not natively where it thinks it is. The App-V engine picks up for this (through relationships in memory) and the operations are redirected to the appropriate converged locations in both the User VFS COW and Package Store.
Procmon and Registry Operations
You will see similar operations when tracing specific activities to registry entries. First, where the application thinks it is supposed to be located followed by subsequent operations to the actual state-separated locations in the actual registry.
Why does Process Monitor not show every Single Operation to “Virtual Paths”
The answer is quite simple – because the App-V Client takes care of all of this behind the scenes which is why you will need to have access to and understand the FileSystemMetadata.xml file as it contains all of the file system mappings for both non-tokenized paths and tokenized paths. The easiest and most automatic relationships are the KNOWNFOLDERID paths which automatically resolve to App-V tokenized paths in memory. For non-tokenized paths, it is handled differently on process creation.
Altitude Adjustment of ProcMon Driver
When you look at the altitudes of the App-V file system drivers and their relationship to the driver altitude, you can see that the Procmon driver sits at a lower altitude by default.
This might make you explore the possibility of raising the altitude to see if Process Monitor will capture more information. Please be careful doing this as this could create problems and system instability. Altitudes are managed and allocated by Microsoft (https://msdn.microsoft.com/en-us/library/windows/hardware/dn641617.) When developers want to register altitude locations for their filter driver, they fill out a special request form. That is how tightly controlled they are in order to prevent instability.
I have been working issues where it was difficult to determine why a specific application in a virtual application would not launch or trigger under certain circumstances. In particular, when another application called the application directly in order to pass data to that virtual application. Normally, when a native application is brought into the virtual environment through a package or a connection group, it can call other applications and virtual applications for the purposes of sharing data. In App-V 5, a native application, through the shell, can also call a virtual process and pass a file to it. In most cases, this will work with some exceptions – however – I have been able to overcome most of these issues through troubleshooting.
In this particular case the application being called is a virtual application. The application calling it was a home-grown line of business (LOB) application written by the customer to retrieve archival data from a secure portal. When the data was retrieved the data would open up within their preferred ZIP archival utility 7-Zip. 7-Zip was virtualized in this environment with App-V 5. While the virtual application would run would run fine when launched from a shortcut, when called from another virtual application within the same package or from another shell-based application as I later found, it would fail. Monitoring the customer’s custom application from Process Monitor yielded a 135 (hex 0xc0000135 or decimal 11073741515) exit code and the application yielded a pretty direct error message:
Now, before going down any deep troubleshooting, I did my due diligence and attempted reproduction on other machines including a machine as CLEAN as possible. On the clean machine and all but two machines, this issue was NOT reproducible. The plot thickens. Time to eliminate variables.
No, it wasn’t Bad Environment Variables
This was one of the first things I verified. In fact, neither the working or non-working setups had 7-Zip in their search path. Surely including it would fix the problem, and it did make the issue go away – but – while this is a fix – we had not found root cause. I don’t like not having root cause. Besides, the workaround was obviously not needed on the machines not experiencing the issue.
Time for Procmon
I took two different Procmon traces – one from a working machine and one from a non-working machine. I then created a simple filter with “7Zfm.exe” – the primary executable for 7-Zip – under the where “Path” “Contains” filter:
Once I looked at the non-working one through this filter, I had all of the answers in hand. I just needed confirmation from the customer’s developer. You will notice in the trace below, before any search of the path occurred, there was a query to the app paths key within the registry:
The app path returned a local path and not a virtual path. The path was also not existent on the machine. Once that failed, the search path was parsed and the subsequent searches failed. Upon confirming with the developer, the application was calling 7-Zip using the ShellExecuteEx function (http://msdn.microsoft.com/en-us/library/windows/desktop/bb762154(v=vs.85).aspx) which can be interrupted easily by a “rogue” app path. Upon digging into the registry, we found that the working machines had the correct app path in HKCU . . .
And the Procmon trace showed it was quickly found in the app path:
. . . but the non-working machines had the wrong app path in HKCU:
Removing this rogue app path (which came from their roaming profile) resolved the issue.
More on App Paths
One of the reasons why a native application can call a virtual application is due to the support of app paths in App-V version 5. We did not have this luxury in previous versions of App-V. App Paths allow the shell to call the application by executable without having to use the PATH environment variable. Now with App-V 5, you can call a virtual application by typing in the application in the search dialog box or the “Run” menu or even through API’s (ShellExecuteEx.) App Paths are further documented here: http://msdn.microsoft.com/en-us/library/windows/desktop/ee872121(v=vs.85).aspx
When publishing the package to the user in App-V 5, the path to the application is registered in HKCU SoftwareMicrosoftWindowsCurrentVersionApp Paths. When the package is published globally, it will register in HKLMSoftwareMicrosoftWindowsCurrentVersionApp Paths
Prior to version 5 of App-V, when you sequenced an application that required a reboot, the reboot was simulated in that the sequencer would process the reboot action including the Pending File Operations and the RunOnce registry keys (RunOnce, RunOnceEx, GuiRunOnce, etc.) when monitoring ceased. In most cases, the simulation would be fine and all of the necessary reboot tasks would be processed properly. There were some exceptions – notably the famous disappearing Google Chrome application famously mentioned in “Stealth Puppy’s” [I love that handle] “Case of the Disappearing Application During Sequencing http://stealthpuppy.com/the-case-of-the-disappearing-application-during-sequencing/ “ where after the reboot tasks processed and the monitoring of the installation of GoogleChrome completed, the Program Files directory simply disappeared.
In defense of the sequencer, this was not as much the fault of the App-V sequencing process, but also an issue with the mechanism of the installation. This was easy to prove by going through the sequencing process and verifying the contents prior to stopping the monitoring process. The files were indeed correctly located prior to the stopping of monitoring:
However, when we ran PENDMOVES.EXE (http://technet.microsoft.com/en-us/sysinternals/bb897556) from Sysinternals, we found this:
The operation pending was to DELETE the actual files. This would explain why the files got deleted after the monitoring process ended. The next logical step would be to see if this was a properly captured operation and not just some bug in the sequencer and install the application natively. Prior to rebooting the machine, run the PENDMOVES.exe utility again. Upon doing this, you will find the EXACT same pending file operation – DELETE C:Program FilesGoogleChrome.
So the native installation does indeed remove the files upon a reboot. The difference is after the reboot, the files are back where they need to be! How is this possible? Google triggers updates using the Google Update service which is set to start automatically. The missing files would force the Google Update service to pull down the most up to date version. Indeed a prime example of some of the exceptions to the rule that the 4.6 sequencer was not catching.
Going back to the original issue – and never passing up an opportunity to showcase a Sysinternals utility – I would recommend a workaround. Given the issue was a DELETE operation, you could leverage another Sysinternals tool to add in a recopy function as a workaround with 4.6. Before actually stopping the monitoring process, first copy the files in question to a safe area.
Then I used the MOVEFILES.EXE utility (http://technet.microsoft.com/en-us/sysinternals/bb897556) from Sysinternals to have them moved back during the simulated reboot.
Changes in the V5 Sequencer
So to ensure that the sequencer works better with installations that require reboots (or even multiple reboots) the sequencer was changed to actually reboot. When the sequencer detects that the application is requesting a reboot, all tasks are recorded and deleted (except for the PendingFileRenames key) to force resume only after the system is rebooted and the monitoring has resumed, otherwise, these tasks could occur prior to the sequencer resuming. This information is recorded to disk (although obfuscated) and the system will reboot as normal
After the reboot, the PendingFileNames key will be processed as it would in a normal reboot. Upon the user logon, the sequencer will be launched (as it registered itself to do that prior to reboot.) Upon relaunch of the sequencer, the sequencer reads its state from the registry and the file system and restarts monitoring. The first tasks that will be processed will be the RunOnce tasks that were captured prior to the reboot.
While the Sequencer User Interface allows for the preservation of settings and a real system reboot, this is not the case for applications sequenced using the PowerShell Sequencer Cmdlets. Since this is mostly reserved for scripted MSI packages, those usually have suppressed reboots anyway.
In previous blog posts, I discussed what happens when a package is added/configured, published, and when streamed. But what happens when an application is double-clicked? App-V needs to determine if the application needs to be virtualized, and if so, which virtual environment. The Shortcut is launched. The Junction point is parsed to the right version location in the package cache. The executable’s process is created but there will be a process notify callback by the App-V Virtual Environment Manager (AppVVemgr.sys) to determine if the process should be virtualized or not. Now, I previously blogged about this part of the process last year and gave a convoluted diagram (http://blogs.technet.com/b/gladiatormsft/archive/2013/09/12/app-v-5-to-virtualize-or-not-to-virtualize-how-app-v-5-answers-that-question-when-you-launch-a-shortcut.aspx) which was met with entertaining emails.
Assuming the application is going to be virtualized, after that workflow has completed, then either the AppVEntSubsystems32.dll or the AppVEntSubsystems64.dll is injected into the virtual process depending on the bitness of the application. Of course, it will always be the 32-bit DLL if working exclusively on X86 platforms. You will notice that once an application is running, there are a few ways you can tell if an application is running virtualized or not. I’ve previously mentioned ListDlls.exe from Sysinternals but you can also use Process Explorer, but not for the reason you may think. The “Virtualized” tab within Process Explorer refers to whether the application is being run within WoW64 (for 32-bit applications) rather native x64. Nothing to do with App-V just as the “Package Name” field refers to the base AppX package and applies to modern applications.
But you can use other features of Process Explorer to determine if a process is virtualized with App-V. For example, you verify the path and command line as they will show launched from the location of the package cache.
That is not completely full proof because you could also have an application that is locally installed but has been configured to run inside a package’s virtual environment. This also means the application will have the AppVEntSusbsystemsXX.DLL injected as well. So just search for that. You can do it through the stack itself but that will require you to walk to the individual thread stacks.
Instead you could just simply do a search in Process Explorer for the DLL and you will get a response back of every process for which that DLL has been injected
Besides injecting this DLL into the process, several other events are taking place. The App-V Virtual Environment Manager will also create all of the elements needed for that virtual environment including the attaching of the VFS COW Filter driver to the appropriate volumes and send the COW registry and file mappings to the driver assuming they have been staged properly – otherwise, that must also take place. The Virtual subsystem controller must also establish the subsystems with the proper settings based on the packages catalog (COM, Objects, etc.) which is why it is mandatory that these settings match when you are creating and enabling Connection Groups as the settings for the entire Package Group Catalog would also have to be created should this be the first application that is launching in a published Connection Group.
Now that Hotfix 4 for App-V 5.0 SP2 has been out now for several weeks, many of you have likely already seen our Updated Guidance for Performance Recommendations now available on Technet (http://technet.microsoft.com/en-us/library/dn659478.aspx.) It almost goes without saying that the new stream-to-disk model of populating individual state-separated sparse files at the native NTFS level yielded an approach to streaming that came with some caveats and albeit, a few glitches, at first.
It was a big change and a lot of the move from an isolated virtual drive to a state separate immutable package cache directory meant that there would have to be some re-engineering and with that brought changes – including changes at the philosophical level.
The basic concepts remain the same. They are just implemented differently:
In addition, a switch to the use of standard protocols already deeply rooted into Windows occurred completing the change of the streaming landscape. Many customers have asked me to clarify some of the new performance recommendations and the rationale behind some of the choices.
SMB or HTTP?
Some concepts require us to rethink how we implement our packages. For example, intra-datacenter streaming, especially for Shared Content Store scenario will yield much better results with SMB or file-based streaming. HTTP, or web streaming, will be better suited for standard streaming especially to devices external to the data center. In the case of Internet-facing scenarios, especially where data would need to be encrypted, HTTPS would be the way to go.
Feature Block 1
When the package needs to stream content for a first launch, the StreamMap.xml file (which is already cached upon publishing) will be parsed.
Once all of the files listed in FB1 (the actual element is PrimaryFeatureBlock) are downloaded, the application can proceed to launch. Our updated guidance mentions the concept of automatic stream faulting upon first launch (which is what always happens in SCS) where if there is no FB1 (like in the example below) the file will be instantly pulled down to populate the sparse file and loaded into memory thus often resulting in a much quicker launch (especially in a scenario where HTTP streaming is being used.) This significant performance hit does not reveal its ugly head as drastic with SMB streaming as it is often going to be faster especially over higher speed LAN links.
Remember – this also not a simple file retrieval process. The files are inside a compressed package. So extraction also has to occur and that’s going to be somewhat less costly with SMB than with HTTP.
When the application launch succeeds, the PreviouslyUsed value in HKEY_LOCAL_MACHINESOFTWAREMicrosoftAppVClientStreamingPackages<GUID><GUID> wil be marked to 1. This also means that if you are using the default configuration, the Autoload settings mean that a background load task will be queued to begin. This will not happen in SCS mode. No Autoload, No FB1.
Removing/Updating Package Content with Opened Packages
We use metadata to govern version and package lineage from a streaming perspective and as a result, many administrators are choosing to forgo the appending of package files to include a version stamp. That is certainly an option but overwriting content on package stores may yield problems. Usually it relates to packages being in use. There are several tools at the ready to verify on your content servers what files may still be open for streaming by clients. You can use the Handle.exe tool for Sysinternals http://technet.microsoft.com/en-us/sysinternals/bb896655 to view and close open packages although I would not advise closing packages, especially in Shared Content Store scenarios as that could create application crashes and lead to data loss.
Since Anonymous authentication is not supported and access is authenticated, you can use the handle command with the –u switch to get user information. You could also revert to the old fashioned NET FILE command if you are using SMB for streaming as well.
It will also reveal the user.
I’ve noticed, and you likely have as well that when working in stream-to-disk scenarios, the handles to the packages can remain open for quite a while – even after the file has been fully cached sometimes. That is actually by design. Each App-V Client will maintain a connection to the package file for each user on a per-package basis. Like the case with previous versions of App-V, this means that there will be a lot of sessions coming from RDS clients. BUT – unlike previous versions of App-V, we do NOT have to deal with limitations caused by ephemeral ports, individual connection on a per-application basis, constant re-authentication on each application launches. There will be one session per package/user combination for stream-to-disk scenarios. NOTE – In Shared Content Store mode you may see multiple sessions depending on stream faults.
With these improvements come some caveats. The client will keep an open handle to the file when using SMB streaming. This is because these sessions involve connect to compressed packages and that can be an expensive process if you need to consistently reconnect. As a result, the App-V Client will cache open sessions for up to 30 minutes past use. I my opinion, this is a small price to pay for the added benefits and scalability that come with the changes.
Have you ever been testing a virtualized application and it fails with a bizarre application error that is either extremely vague (unknown error) or coupled with some random hex code? Your first reaction usually is “Where’d that error come from?” That is a good reaction to have as it is the first major hurdle in determining how to troubleshoot what has gone wrong. If you have not narrowed down the target of where the application error is coming from, you leave yourself open for spending a tremendous amount of wasted time down “troubleshooting rabbit-holes.”
Is it an App-V Operational Issue?
The first thing you will want to narrow down is whether or not it is an operational issue coming from the App-V Client itself, or is it related to the package (Application not functioning as expected.) I always recommend to first look at the source of the error window. If the error message is originating from a “Microsoft Application Virtualization” window, it is likely an operational issue tied to one of the client engine components or perhaps a streaming issue.
If it looks to be an operational issue, I would advise you leverage some scripts and tools written by Dave Falkus, one of my colleagues in the UK: http://blogs.technet.com/b/virtualworld/archive/2014/04/12/app-v-5-0-etw-tracing-automation.aspx
In addition, one of my previous articles will assist you in dissecting App-V error codes: http://blogs.technet.com/b/gladiatormsft/archive/2013/11/13/app-v-on-operational-troubleshooting-of-the-v5-client.aspx
So I’ve determined it is an error that occurs within the virtualized application itself, now what?
Whether you are isolating an issue that may be caused by virtualization or by bad sequencing, if you have determined the error is coming from the virtual application then it is time to start doing a little reverse engineering with a couple of Sysinternals tools. The first thing I do for an application completely new to me is map out the EXE and DLL launch tree. You can do this by capturing the issue with Process Monitor ensuring that you start capturing at application launch.
You can then run a quick capture filter (I usually load a saved filter with these filter settings)
My filters contain these operation events:
If the application is devirtualized on the sequencer, you will be looking to start from the shell process that initiated the Process Create operation (CMD.EXE or EXPLORER.EXE likely.) On a side note, svchost.exe will spawn modern apps, but that is out of scope here. J
If the application is running virtualized on the client, you will see that the AppVClient.exe process interjects spawning the mavinject32.exe or mavinject64.exe process depending on application bitness – followed by the shell creating the process. The example below using virtualized SongSmith displays this in Process Monitor:
Incidentally, if you want to know more about how the AppVClient.exe determines when to run the MAVInjector, read this previous article:
OK, I have identified the EXEs and DLLs, Now What?
You can do one of two things. You can search the Processes for strings using Process Explorer, or in the case of DLLs and EXE’s, you can simply leverage the Sysinternals STRINGS.EXE tool. The Process Explorer tool is great for finding error strings (by going to the properties of a process, selecting the “Strings” dialog box, clicking the “Find” button.)
The reason I do not use this is because it requires the process to still be active and it is not easy for searching DLL modules if they are not still in memory. I use instead the STRINGS.EXE utility (http://technet.microsoft.com/en-us/sysinternals/bb897439) as it allows for more search flexibility.
Case in point, this bizarre error in Notepad++:
I can confirm the error came from the EXE NOTEPAD++.EXE by using the following command:
Strings-o “C:Program FilesNotepad++Notepad++.exe” | findstr /I scintilla
Putting it All Together
When you get that unknown application error, you can then proceed to investigate towards a resolution using a clear path by:
- Determining whether it *IS* an application error and not an operational error.
- Determining if it is a sequencing or virtualization issue.
- Collecting the Process/DLL load/execution order.
- Identify which EXE or DLL is contains the error string.
You can then proceed to continue troubleshooting by focusing around that EXE or DLL with Process Monitor, SpyStudio, APIMon, or whatever tool you prefer to further debug the issue.
I've discussed running native applications within virtual environments and the many ways we can bring applications like Internet Explorer into the bubble. One of the many reasons we would need to bring IE into the virtual environment is for web applications that use different versions of Java. When you virtualize different versions of Java, the shortcuts that are created will launch inside the bubble (and also through the /appvve switch.) For example, in the figure below, I have two specific Java packages deployed – each with its own Internet Explorer shortcut configured to run inside a separate Java bubble. I use the URL javatester.org to verify the version.
I can also view the java processes from the sysinternals utility Process Explorer running successfully launched from their own respective immutable package directory locations.
I can use another Sysinternals utility ListDLLS.exe to show that the App-V hook DLL has been injected into the Internet Explorer and Java processes.
You will also notice that if you configure RDS RunVirtual (where you use the AppV RunVirtual registry key to allow Internet Explorer to run inside a virtual package) to load Java into Internet Explorer, that you can see the Java application spawned by the IEXPLORE.EXE process inside of Process Explorer.
Differences from Dynamic Virtualization in SP2
You may notice in SP2 (as I believed I also mentioned in my last blog post) that when you launch Explorer and Internet Explorer, you will find that these also contain the injected AppVEntSubsystems32.dll due to the default dynamic virtualization configuration – however – just because you see EXPLORER.EXE and IEXPLORE.EXE hooked with LISTDLLS does not mean they will automatically behave the same way as if the application where configured for Run Virtual. App-V 5.0 SP2 introduced this feature as response to the historical desire for the virtualization of shell extensions. Actually shell extension virtualization is just one of the side benefits of JITV (Just-in-Time Virtualization – or “Dynamic Virtualization” as it is officially called.) Beginning with Office 2013, there was the capability of virtualization within a process being dynamically activated and de-activated on a per-thread process.
With SP2, this feature was extended to shell extensions AND ActiveX controls implemented as an in-proc COM object. This is why you will see injections of the App-V Hook DLL into Explorer or Internet Explorer even when no virtualized package is in use. This leads to an important statement: Just because the application is hooked, doesn’t always mean it is running virtualized if it appears as a process under the ProcessesUsingVirtualComponents registry value. This will be done at the thread level.
When an ActiveX OCX or a DLL that implements a shell extension is loaded from a native process or a process from another virtual application, App-V generates an additional virtual environment on demand linking the package containing the OCX or DLL with the process. Then dynamic virtualization is turned on for that particular thread. Once the thread exits, dynamic virtualization is turned off. If the said thread with dynamic virtualization spawns another thread, that thread too will be virtualized. This also helps to prevent RunVirtual collisions.
When a single native process attempts to launch within more than one primary virtual environment without connection groups, you have colliding virtual environments – and it is not pretty.
In the context of Internet Explorer, they look like this:
Often this will be accompanied by the illustrious “spinning donut!”
How to tell if you have a RunVirtual Collision:
- The native application is immediately unresponsive upon launch and it remains unresponsive.
- The launch of the native applications spawns multiple packages in use when no connection groups are configured. These are the collisions.
- The CPU will be completely pegged:
And the battle waging will be the native processes – usually at least one per package involved in the collision.
Finally, the smoking gun will be running LISTDLLS or Process Explorer against it looking for the actual executable for the virtual package. You will notice that while the native app is hooked, no virtual processes are able to start and get hooked (in this particular example, Java)
The cause will most likely be launching the application through a run virtual command (ie Process.exe /appve:<GUID>_<GUID>) configured to run inside one particular virtual environment – while at the same time – that same native application is configured to run virtual inside another virtual application environment using the RDS RunVirtual registry key.
In the case of the above example, that was the exact cause. Internet Explorer was added as a process in the RunVirtual registry key and was configured to run in the Java 1.4 package. If you ran a normal instance of Internet Explorer, everything behaved properly. However, if you ran a shortcut to Internet Explorer configured to run inside a separate instance of Java using the /appvve switch, then the collision occurs.