Productive virtual application debugging requires an understanding of the basic fundamentals of debugging compiled software code. For this part of my series on debugging virtual applications, I will be focusing exclusively on these fundamentals. If you are already familiar with these concepts, please allow me to quickly recap these to those readers which may be either not familiar, or only somewhat and looking to solidify these concepts.
Types of Debugging
There are several categories of debugging and the descriptions will vary by vendor, publication, and academic degrees of description. There is almost a guaranteed point of view when it comes to applying it to a specific product or series of products. Being that my discussion primarily revolves around products that run on top of the Windows operating system, my point of view, or slant, is obviously geared towards the types and toolsets that come with Windows.
Live debugging refers to the mechanism of attaching to a running program or process either invasively or non-invasively. A debugger may attach to a process and wait for exceptions or set a specific breakpoint. The debugger can insert those breakpoints in once attached to the process. The easiest way to think of a breakpoint is to understand its most basic definition: a breakpoint is a place or time at which an interruption or change is made. More information on breakpoints and different breakpoint types within the Windows context can be found here: https://msdn.microsoft.com/en-us/library/windows/hardware/ff538928(v=vs.85).aspx. In addition, live debugging is also commonly used to troubleshoot and analyze code within the developer environment. In those situations the types of breakpoints will vary. For example, you can refer to the examples of breakpoints that are available within the Visual Studio development environment here: https://msdn.microsoft.com/en-us/library/bb161312.aspx. Once attached to the process, a debugger can then step through threads and functions as the application is live.
Print or Trace Debugging
This is the most common method for troubleshooting software applications and operating systems as technically, this can cover a wide scope of methods. An application can run at specific diagnostic levels generating additional output and information that can be collected into a file or database that can be used to isolate and issue. Event traces, log files, debug output all fall into this category. Strictly speaking within Windows, applications can leverage the OutputDebugString or ODS to have an application, service, or operating system component generate what is referred to as “debug spew” and you can use various tools to collect or view this debug trace information. The most popular tool for viewing ODS traces is the Debug View utility (DBGVIEW) from the Sysinternals suite (https://technet.microsoft.com/en-us/sysinternals/debugview.aspx) although this is not the only one. More information on the OutPutDebugString can be found here (https://msdn.microsoft.com/en-us/library/windows/desktop/aa363362(v=vs.85).aspx.)
In addition, there are tools that can hook into the Windows operating system to capture Win32 API and other application functions through the use of a simple user mode monitor (like the API Monitor tool) are even deeper through the use of a kernel-level filter driver (Like Process Monitor.) Literally troubleshooting outside the box and on to the wire – you can use network trafficprotocol analysis tools like Wireshark or Message Monitor (https://www.microsoft.com/en-us/download/details.aspx?id=44226) to capture network traces. These are all forms of trace debugging.
Windows Integrated Tracing and Instrumentation
Prior to Windows Vista, there were Event Logs, ODS tracing, text-based log files, etc. all within Windows each requiring their own tools and APIs. Starting with Windows 2000, Microsoft began incorporating Event Tracing for Windows (ETW) into the operating system and soon, applications and windows components were using this common engine for enabling diagnostics and collecting detailed debug tracing. Viewing of these traces was soon integrated into the Windows Event Viewer and users of App-V 5 are able to often resolve issues using this very mechanism.
The instrumentation mechanism is discussed here ( https://msdn.microsoft.com/en-us/library/zs6s4h68(v=vs.110).aspx) and probably the greatest technical reference on ETW can be found here (https://msdn.microsoft.com/en-us/library/windows/desktop/bb968803(v=vs.85).aspx)
Remote Debugging is a form of live debugging where the process of debugging occurs on a system that is different from the debugger. In most Windows cases, this is where there is an issue that needs to be debugged at the kernel mode level prior to the completion of an operating system boot or a system level crash. To start remote debugging, a debugger connects to a remote computer over a network or via a serial cable. The debugger can then control the execution of the program on the remote system and retrieve information about its state. In Windows, this is often done serially or via Firewire.
Post Mortem Debugging
Post Mortem debugging is a very common method of troubleshooting problems within software because it involves viewing a historical point-in-time snapshot of a hang, system, or application crash. This is where a debugger will read in a snapshot of debugging data called a “dump file” which contains existing memory and instruction pointers. The degrees of debugging depend on how much data is collected in the dump file as dump files can vary in what they collect. When it comes to application and system dumps, these can be controlled by the operating system’s default handlers (once called Dr. Watson for user mode applications) as to what information is collected in the dump file.
I was first introduced to the concept of post-mortem debugging reading an article by Matt Pietrek way back in 1992 in Dr. Dobbs Journal. Matt historically is one of the earliest writers on the subject of Windows debugging going back nearly three decades. The amazing thing is you can still read this article I am citing as it is available online: http://www.drdobbs.com/tools/postmortem-debugging/184408832
Execution Modes of Debugging in Windows
When we are speaking of execution modes in Windows, were talking about code that runs either in user mode or kernel mode. The execution mode affects the methodologies and tools you will leverage in order to properly debug the issue. Software is ultimately driven by the processor (CPU.) For a computer running Windows, the CPU runs in two different modes – user mode and kernel mode. The CPU switches between the two depending on the code.
The kernel and other operating system components run in kernel mode, hence the term. Rather than a macrokernel like other operating systems, Windows runs a smaller microkernel that runs as process SYSTEM. Like an application loads and uses DLL (dynamically linked libraries) the kernel also loads special modules called executive components and/or filter drivers alongside device drivers. There is essentially only one process running and that is what shows up in the Windows task Manager as “System” and if this application crashes . . . well . . . so does the entire computer. With debugging, when we are debugging in kernel mode, we are essentially debugging this process – however, it also serves as the governor of all of the other processes running on the system in user mode. All code that runs in kernel mode runs in a single virtual address space. This means that a faulty kernel-mode driver is not isolated from other drivers and the operating system itself.
Regular applications, middleware, plug-ins, and most services run in user mode. When you start a user-mode application, Windows creates a process for the application. This process will execute one or more threads. I use the description of the process itself being innate in nature. It just owns a private virtual address space, a private handle table, and contains at least one primary thread for execution. This description of a process comes from Jeffrey Richter who has written many books on Win32 programming. Because these processes are isolated from each other, an application is unable to screw up the operation of another separated process if it crashes. Other applications and the operating system are not affected by the crash. Data can be exchanged between these processes through interprocess communication mechanisms but they cannot directly write to address spaces directly. Limiting the virtual address space of a user-mode application prevents the application from altering, and possibly damaging, critical operating system data.
The App-V product is especially complex when using it as an example because it contains code at both kernel and user mode. The App-V client engine consist of kernel level drivers, a primary service, and user-mode DLL’s that are injected into the processes of virtualized applications.
Next Up: Debugging Misbehaving Application Scenarios
For many application packagers, virtual application sequencers, and general IT pros, the concept of actual “debugging” can take on many meanings. Often the words “troubleshooting” and “debugging” are interspersed – especially when reading articles and blogs dealing with the topic of trying to dissect what may be occurring when a virtual application is not functioning as expected. When we speak of the word “debugging” in the context of its meaning with regards to programming and compiled software code, it is simply the dissection and reverse engineering of binaries to determine the root cause of an issue or basically “find the bug” in the code.
The level of depth may vary depending on the tools being leveraged and the amount of access to code or symbols. For example, open-sourced code projects on the web are very easy to debug because – well – the code is distributed alongside of the binaries. In addition, special files called “symbols” are also often available if need be. This is especially helpful in the world of Windows debugging. For Closed-Source binaries – like Windows, access is limited to what API’s are exposed and documented within MSDN, the SDK’s, and publically available symbol files. Still, this tremendously aids our ISV partners when they are troubleshooting issues with their own code running on top of the Windows platform.
Enter the Virtual Application
What makes this especially complex in the world of virtual applications is that the surface area expands to not only include the original application but also the virtualization engine that is maintaining its sandbox – specifically its isolation and/or state separation mechanisms. With this, you have essentially increased your variables for issues. Whereas a native application involves one vendor; running on top of another vendor’s operations system, a virtual application now deals with potentially three different vendors (not even counting the potentially amount of 3rd-party vendors that could also be hooked into the kernel via filter and device drivers.) In the case of Microsoft and App-V, if the application being virtualized is a Microsoft application, there are unlimited resources internally to work on that application. In most cases, that represents less than one-hundredth of one percent of the applications out there in the ecosystem – at best. Most cases, the application is external. When that is the case, the debugger must determine the following:
Is the application open sourced or closed source?
If the application is open-sourced, the application can be easily investigated alongside the virtualization subsystems and likely debugged pending that the individual doing the debugging understand the source code and has the proper tools to debug the application from within the share operating system (in the case of App-V – that would be Windows.)
If the application is closed source, what resources are available from the vendor?
This is where it can be challenging. When you are debugging a closed source application running virtually, it requires significant insight into the application – especially if the application is running in native code. While Microsoft makes public symbols available for ISV’s to help with debugging, often the opposite is not true. As a result, the debugging is “best-effort” at best and is usually limited to basic reverse engineering tools like Process Monitor, API Monitor, or DbgView. One exception to this – that I have encountered – have been situations where the application encounters specific issues when virtualized – and those issues cannot be reproduced on a natively installed instance of the application. In those cases, the focus can shift to the virtualization engine however, even in these situations, working in triangulation with the application vendor yields more success – much quicker.
Is the application using a 3rd-party application virtualization engine by a vendor different from the vendor of the underlying operating system?
In this scenario, the application is written by one vendor, running on top of an operating system by a different vendor, and then sandboxed using an application virtualization by yet another vendor. In the case of Windows, the application is using a non-Microsoft virtualization solution. There have been many times where I was working support for App-V and a customer would call in with an issue they were having virtualizing a version of Office or Visual Studio on a non-Microsoft platform. I would always re-direct the customer to the vendor of the app virt stack – even though we were the vendor of the application being virtualized as well as the underlying operating system. I would then direct the customer to reach out to the Office or Visual Studio team as well to work in triangulation.
Relationships of Application to Support Vendors
When debating the best source for debugging virtual applications, please feel free to leverage the following matrix I constructed to assist you in reaching out to the most likely resources that will be able to help resolve the issue.
Operating System Vendor
AppVirt Stack Vendor
Best Vendor(s) for Virt Debugging
Best Case Scenario
Rare in the Windows World
Typical 3rd-party AppVirt Scenario
Vendor C first & Vendor A optional
Most Common at Microsoft
Vendor B first & Vendor A optional
The reason I make the above recommendations is because at some point the application, the application virtualization engine, or even perhaps the operating system may require some debugging – especially if there is a potential bug. If the resources troubleshooting the issue do not have access to the resources and tools needed to debug the issue – then you are essential throwing darts against the wall – and it will lead you potentially down a rabbit hole.
Why Discuss Debugging?
I have decided to start discussing the topic of virtual application debugging to serve the following purposes:
To demystify the concept for application packagers and IT Pros in the Application Virtualization space. There are tools and concepts that can help these professionals to further arm their skills and enhance their arsenals and toolboxes. Many reverse engineering tools such as ProcMon can only go so far.
To aid software vendors in how to debug applications running under App-V and how their applications may be affected.
To aid customers in how to gather and collect the appropriate debugging information to help Microsoft and other software vendors diagnose issues, isolate root cause, and resolve problems and bugs quicker.
Next Up Part 2: Types, Modes, and Situations
App-V 5: Do you Still have to Run Process Monitor within the App-V Bubble when Troubleshooting Applications?
If – by that question you want to know if you must start an instance of Process Monitor within the virtual application like you did in 4.x – no. You could run process Monitor inside the virtual bubble, but it will not yield you much more results. The reason behind this is simple: unlike previous versions of App-V, the REAL registry as well as the native file system – NTFS – is used in App-V 5.
In App-V 4.6 and earlier, if you did not launch process Monitor in the virtual application’s environment (usually through a command prompt) all you would capture related to the virtual application would be operations to file and registry resources outside the virtual environment. In Version 5, running Process Monitor as normal will capture access to the actual locations including registry, package store, as well as VFS (Virtual File System) COW (Copy-on-Write) locations. Why just that? Because that’s where things “actually” are located. What you will have to understand is that once the operations to where the application “thinks” it is located has been hooked in the file system and/or registry every subsequent operation will continue as such to include operations only to the:
Integration junction points to the Package Store
Actual package Store paths
VFS COW (Copy-on-Write) checks and locations
You can see examples of these in the following screenshots from Process Monitor:
Procmon and File Operations:
As you can see above, the query to the initial location of where the virtual applications thinks it is browsing (C:program Files (x86)Java) is clearly not natively where it thinks it is. The App-V engine picks up for this (through relationships in memory) and the operations are redirected to the appropriate converged locations in both the User VFS COW and Package Store.
Procmon and Registry Operations
You will see similar operations when tracing specific activities to registry entries. First, where the application thinks it is supposed to be located followed by subsequent operations to the actual state-separated locations in the actual registry.
Why does Process Monitor not show every Single Operation to “Virtual Paths”
The answer is quite simple – because the App-V Client takes care of all of this behind the scenes which is why you will need to have access to and understand the FileSystemMetadata.xml file as it contains all of the file system mappings for both non-tokenized paths and tokenized paths. The easiest and most automatic relationships are the KNOWNFOLDERID paths which automatically resolve to App-V tokenized paths in memory. For non-tokenized paths, it is handled differently on process creation.
Altitude Adjustment of ProcMon Driver
When you look at the altitudes of the App-V file system drivers and their relationship to the driver altitude, you can see that the Procmon driver sits at a lower altitude by default.
This might make you explore the possibility of raising the altitude to see if Process Monitor will capture more information. Please be careful doing this as this could create problems and system instability. Altitudes are managed and allocated by Microsoft (https://msdn.microsoft.com/en-us/library/windows/hardware/dn641617.) When developers want to register altitude locations for their filter driver, they fill out a special request form. That is how tightly controlled they are in order to prevent instability.
I have been working issues where it was difficult to determine why a specific application in a virtual application would not launch or trigger under certain circumstances. In particular, when another application called the application directly in order to pass data to that virtual application. Normally, when a native application is brought into the virtual environment through a package or a connection group, it can call other applications and virtual applications for the purposes of sharing data. In App-V 5, a native application, through the shell, can also call a virtual process and pass a file to it. In most cases, this will work with some exceptions – however – I have been able to overcome most of these issues through troubleshooting.
In this particular case the application being called is a virtual application. The application calling it was a home-grown line of business (LOB) application written by the customer to retrieve archival data from a secure portal. When the data was retrieved the data would open up within their preferred ZIP archival utility 7-Zip. 7-Zip was virtualized in this environment with App-V 5. While the virtual application would run would run fine when launched from a shortcut, when called from another virtual application within the same package or from another shell-based application as I later found, it would fail. Monitoring the customer’s custom application from Process Monitor yielded a 135 (hex 0xc0000135 or decimal 11073741515) exit code and the application yielded a pretty direct error message:
Now, before going down any deep troubleshooting, I did my due diligence and attempted reproduction on other machines including a machine as CLEAN as possible. On the clean machine and all but two machines, this issue was NOT reproducible. The plot thickens. Time to eliminate variables.
No, it wasn’t Bad Environment Variables
This was one of the first things I verified. In fact, neither the working or non-working setups had 7-Zip in their search path. Surely including it would fix the problem, and it did make the issue go away – but – while this is a fix – we had not found root cause. I don’t like not having root cause. Besides, the workaround was obviously not needed on the machines not experiencing the issue.
Time for Procmon
I took two different Procmon traces – one from a working machine and one from a non-working machine. I then created a simple filter with “7Zfm.exe” – the primary executable for 7-Zip – under the where “Path” “Contains” filter:
Once I looked at the non-working one through this filter, I had all of the answers in hand. I just needed confirmation from the customer’s developer. You will notice in the trace below, before any search of the path occurred, there was a query to the app paths key within the registry:
The app path returned a local path and not a virtual path. The path was also not existent on the machine. Once that failed, the search path was parsed and the subsequent searches failed. Upon confirming with the developer, the application was calling 7-Zip using the ShellExecuteEx function (http://msdn.microsoft.com/en-us/library/windows/desktop/bb762154(v=vs.85).aspx) which can be interrupted easily by a “rogue” app path. Upon digging into the registry, we found that the working machines had the correct app path in HKCU . . .
And the Procmon trace showed it was quickly found in the app path:
. . . but the non-working machines had the wrong app path in HKCU:
Removing this rogue app path (which came from their roaming profile) resolved the issue.
More on App Paths
One of the reasons why a native application can call a virtual application is due to the support of app paths in App-V version 5. We did not have this luxury in previous versions of App-V. App Paths allow the shell to call the application by executable without having to use the PATH environment variable. Now with App-V 5, you can call a virtual application by typing in the application in the search dialog box or the “Run” menu or even through API’s (ShellExecuteEx.) App Paths are further documented here: http://msdn.microsoft.com/en-us/library/windows/desktop/ee872121(v=vs.85).aspx
When publishing the package to the user in App-V 5, the path to the application is registered in HKCU SoftwareMicrosoftWindowsCurrentVersionApp Paths. When the package is published globally, it will register in HKLMSoftwareMicrosoftWindowsCurrentVersionApp Paths
To continue the discussion of application troubleshooting, I wanted to provide some clarification on a certain type of application failure. In a blog post a while back (http://blogs.technet.com/b/gladiatormsft/archive/2013/05/15/app-v-process-exit-codes-and-why-they-matter-when-troubleshooting-virtual-applications.aspx) I discussed using Process Monitor to dissect silent application failures or exits. Process Monitor allows us to look at process exit codes to determine leads on application failures. I wanted to dive into another type of failure that may or may not leave a proper exit code because it is not “completely” silent. It’s what I refer to as a “Fly-By-Night” error message. [Incidentally, it should be surprising to no one that a blog such as this would eventually have a subtle reference to the rock band Rush.]
Fly-by-Night Error Messages refer to those error messages which appear to flash some code but is often interpreted by the human eye as a quick flash of a black screen:
Fly-by error messages occur when you try to call a script, batch file, internal command (using CMD.EXE triggered by CONHOST.EXE) or any type of Win32 Console application (also triggered by CONHOST.EXE) and it fails to launch.
If you are lucky, these can easy to troubleshoot if you attempt to launch the application without requiring a console host process to launch it (CONHOST.EXE.) For example, let’s use the example of the DB2 client. You have virtualized the middleware and you are attempting to run the Batch file that triggers the DB2 Command Prompt Plus (CLPLUS.BAT.) You simply run the path to executable within an already existing command prompt to grab the error message.
In the above example, it looks like the issue is pretty straight forward. The application requires Java and I do not have Java installed locally or virtually. In most cases, you will get an error message in the console – straight forward or not – you will still have something to go on.
So what if this yielded no error?
You now have a silent exit on your hands. If you do not get an error message at all, you will need to investigate it further using Process Monitor and I recommend referencing my previous blog on silent exit codes (see link above.) For example, if I get a silent exit from a console-based application, I will first load my filter for major process operations – filtering on PROCESS CREATE, PROCESS START, LOAD IMAGE, and PROCESS EXIT.
Assuming I stopped the trace right after the error, I can then start at the last exit of CONHOST.EXE and work backwards towards the exits of the CMD.EXE and the actual console application. If this were a batch file, it would just be CMD.EXE. In the example above, it is the db2cmd.exe process. In the above example, the exit code for db2cmd.exe was -2029059760.
[Incidentally, this was an easy fix (thank you VFS Write Mode.)]
Update 12/5/2014: The PVAD is now considered optional due to changes in App-V 5Service Pack 3. Read more here: http://technet.microsoft.com/en-us/library/dn858700.aspx#BKMK_pvad_hidden
When sequencing applications, you often never know how the application was written with regards to what paths are actually referenced via hard-coded paths or through tokenized, or well-known paths or KNOWNFOLDERIDS. In previous versions of App-V and Softgrid, these paths were referenced using %CSIDL% variables. These are well-documented on MSDN (http://msdn.microsoft.com/en-us/library/windows/desktop/bb762494(v=vs.85).aspx). If the application was developed to reference CSIDLs, they usually function when virtualized just fine with App-V. CSIDLs are still supported for backward compatibility, but going forward starting with App-V 5, a different tokenization process is used to align better with KNOWNFOLDERIDs. The new App-V Tokens and their default locations upon installations are well documented in the App-V 5 sequencing guide:
Tokenization helps to remove a specific application’s tie-in to specific folder paths and to allow for greater portability especially when these KNOWNFOLDERIDS have been redirected for specific configurations.
It is important to understand that for sequencing applications that will be used for Connection Groups or to better take advantage of the App-V 5.0 SP2 HF4 VFS Write Mode, all files will need to be located in the VFS structures. This will drastically affect the sequencing process.
Sequencing to a PVAD, Installing to the PVAD
If you specify a Primary Virtual Application Directory and then install to that same location, the base directory of installation files will not be tokenized. For many older applications, this may be the best route to take. For example, if the Primary Virtual Application Directory (PVAD) is specified as C:DIRECTORY1 and the sequencer monitors an install during sequencing to C:DIRECTORY1 then everything created beneath DIRECTORY1 goes beneath Root and will not be tokenized or located beneath the VFS folder. There may be additional files that do in fact get written to tokenized paths and they will be tokenized but you will now have a mixture of both “rooted” file assets and VFS file assets.
In the example below, Notepad ++ is sequenced to C:Notepad and installed to C:Notepad:
As you can see above, the majority of the files appear below the Root structure and are not contained with the VFS tokenized directories. Also notice the folders beneath Root cannot be adjusted for opacity:
When the files are deployed to the App-V immutable package cache, they will be laid out the same way.
In addition, in order to troubleshoot and navigate the file system within the bubble, non-VFS folders do not appear in directory listings.
You will be able to change to virtual directory by using cd Notepad+
Sequencing to the PVAD, Installing to Program Files or a Different Location other than the PVAD (Fake PVAD)
If you specify a PVAD and then install to a different location, all of the files will be written to the VFS location inside the package. For example, if the PVAD is C:DIRECTORY1 and the sequencer monitors an install to C:Program FilesDIRECTORY1 during sequencing, then everything gets put beneath RootVFS and root installation files will get tokenized if they are written to write to a KNOWNFOLDERID or CSIDL during installation.
In the example below, Notepad++ is sequenced using a fake PVAD and installed into C:Program Files
As you can see, all of the files are stored beneath the VFS directory structure.
When files are VFS’ed, you can see the directory listings as normal directories converged with the local directories when you launch a command inside the App-V bubble.
What About Using a KNOWNFOLDERID as the PVAD?
This question comes up quite a bit. Many packagers like to match PVAD with the installation directory (especially now that the Q: requirement is gone.) This is fine but understand that when you install to a PVAD during sequencing, all installation files beneath that folder that are not otherwise set to install to another explicit KNOWNFOLDERID or CSIDL path will be treated as rooted files.
For example, if you choose C:Program FilesDIRECTORY as the PVAD and the sequencer monitors an install to C:Program FilesDIRECTORY1 during sequencing, then most files will still be beneath root and NOT VFS except for tokenization of explicit KNOWNFOLDERID paths.
In the example below, when sequencing Notepad++, a PVAD of C:Program FilesNotepad++ is specified.
During the monitoring of the installation, notice the specified install directory is also C:Program FilesNotepad++.
Upon completion of sequencing, you will then notice that the package files are laid out just like any other PVAD-sequenced package.
The base installation files are located beneath Root.
Like the example earlier, when you open up a command prompt inside the bubble, you will not see the root directory in a directory listing.
However, you can still change to it manually inside the bubble.
So What about this VFS Write Mode?
For a lot of applications, especially older applications, they are written to use configuration files beneath the root of the installation directory. Natively installed applications often have to use shims to redirect these modifiable configuration files to safer directories. Combine this with the fact that most packagers are not always sure where to place the installation files in relationship to the PVAD (we cannot always read the minds of developers) and you can run into problems when virtualizing applications. While using shims is possible with virtualized applications, this can drastically increase the sequencing and remediation time for an application.
Enter VFS Write Mode. The App-V team added this “silver bullet” of a feature into HF4 of App-V 5 SP2. Using the advanced configuration options in sequencing, you can now select this option to enable an application to normally open files for modification within the package.
To demonstrate how this can be helpful, once again we will use Notepad++. When you install Notepad++, you have the option to load/write the configuration files from the install directory instead of using the %APPDATA% directory which would be properly tokenized.
Prior to the invention of VFS Write Mode, if you selected “Don’t use %APPDATA%” during sequencing, this is what would happen when you tried to run the program virtualized. First you would get an error message like this:
This would be followed by a crash.
When you troubleshoot the application with Process Monitor, you will notice a series of failures when trying to load the files:
In HF4, you can open up the package in the new sequencer, navigate to the “Advanced” tab, and check the box enabling VFS Write Mode.
As a result, Notepad++ will open and function as expected.
You will also notice in Process Monitor, the file operation to open the XML configuration files in the installation directory (with Read/Write/Delete) is successful.
The great thing about VFS Write Mode is that it can be enabled simply by opening up the package in the App-V 5 SP2 HF4 sequencer. You can now put this into your arsenal of App-V AppCompat secret weapons.
Update 12/5/2014: The PVAD is now considered optional due to changes in App-V 5 Service Pack 3. Read more here: http://technet.microsoft.com/en-us/library/dn858700.aspx#BKMK_pvad_hidden
Have you ever been testing a virtualized application and it fails with a bizarre application error that is either extremely vague (unknown error) or coupled with some random hex code? Your first reaction usually is “Where’d that error come from?” That is a good reaction to have as it is the first major hurdle in determining how to troubleshoot what has gone wrong. If you have not narrowed down the target of where the application error is coming from, you leave yourself open for spending a tremendous amount of wasted time down “troubleshooting rabbit-holes.”
Is it an App-V Operational Issue?
The first thing you will want to narrow down is whether or not it is an operational issue coming from the App-V Client itself, or is it related to the package (Application not functioning as expected.) I always recommend to first look at the source of the error window. If the error message is originating from a “Microsoft Application Virtualization” window, it is likely an operational issue tied to one of the client engine components or perhaps a streaming issue.
If it looks to be an operational issue, I would advise you leverage some scripts and tools written by Dave Falkus, one of my colleagues in the UK: http://blogs.technet.com/b/virtualworld/archive/2014/04/12/app-v-5-0-etw-tracing-automation.aspx
In addition, one of my previous articles will assist you in dissecting App-V error codes: http://blogs.technet.com/b/gladiatormsft/archive/2013/11/13/app-v-on-operational-troubleshooting-of-the-v5-client.aspx
So I’ve determined it is an error that occurs within the virtualized application itself, now what?
Whether you are isolating an issue that may be caused by virtualization or by bad sequencing, if you have determined the error is coming from the virtual application then it is time to start doing a little reverse engineering with a couple of Sysinternals tools. The first thing I do for an application completely new to me is map out the EXE and DLL launch tree. You can do this by capturing the issue with Process Monitor ensuring that you start capturing at application launch.
You can then run a quick capture filter (I usually load a saved filter with these filter settings)
My filters contain these operation events:
If the application is devirtualized on the sequencer, you will be looking to start from the shell process that initiated the Process Create operation (CMD.EXE or EXPLORER.EXE likely.) On a side note, svchost.exe will spawn modern apps, but that is out of scope here. J
If the application is running virtualized on the client, you will see that the AppVClient.exe process interjects spawning the mavinject32.exe or mavinject64.exe process depending on application bitness – followed by the shell creating the process. The example below using virtualized SongSmith displays this in Process Monitor:
Incidentally, if you want to know more about how the AppVClient.exe determines when to run the MAVInjector, read this previous article:
OK, I have identified the EXEs and DLLs, Now What?
You can do one of two things. You can search the Processes for strings using Process Explorer, or in the case of DLLs and EXE’s, you can simply leverage the Sysinternals STRINGS.EXE tool. The Process Explorer tool is great for finding error strings (by going to the properties of a process, selecting the “Strings” dialog box, clicking the “Find” button.)
The reason I do not use this is because it requires the process to still be active and it is not easy for searching DLL modules if they are not still in memory. I use instead the STRINGS.EXE utility (http://technet.microsoft.com/en-us/sysinternals/bb897439) as it allows for more search flexibility.
Case in point, this bizarre error in Notepad++:
I can confirm the error came from the EXE NOTEPAD++.EXE by using the following command:
Strings-o “C:Program FilesNotepad++Notepad++.exe” | findstr /I scintilla
Putting it All Together
When you get that unknown application error, you can then proceed to investigate towards a resolution using a clear path by:
- Determining whether it *IS* an application error and not an operational error.
- Determining if it is a sequencing or virtualization issue.
- Collecting the Process/DLL load/execution order.
- Identify which EXE or DLL is contains the error string.
You can then proceed to continue troubleshooting by focusing around that EXE or DLL with Process Monitor, SpyStudio, APIMon, or whatever tool you prefer to further debug the issue.