On the Art of Troubleshooting App-V Applications

Lately I have been dealing with the big challenge of teaching something that I have become to know as more of an art than a science. That is, the troubleshooting of virtual applications not functioning correctly in App-V. I have seen many articles that walk through how someone was able to figure out why one particular application was failing and how they were able to provide a resolution. We always appreciate these brave pioneers who prevent us from having to reinvent the wheel when we happen to have the same exact application problem. Through a quick search of Bing, we were able to find that hero who would rescue us from our current plight. The App-V community is full of these heroes – especially in the MVP space. What we do not know in many cases is how they went down that particular path of troubleshooting. What we will often only see (and I myself am quite guilty of this) is the abbreviated cleaned-up version of how the problem was resolved. We do not see the many paths traversed or the rationale behind the particular troubleshooting approach used.

How do you learn to do that?

I got asked this question bluntly by a customer. I was helping them remediate an application and they came right out and asked me. This got me thinking. What about the packager who is new to App-V? How does one learn how to fix a broken virtualized application, or at the very, be able to determine the actual root cause as to why the application was failing? Where does one begin when troubleshooting a broken application? What are the best tools to use?

Before we can answer these questions, we need to explore why a lot of people struggle with this. What makes troubleshooting a virtual application challenging is the lack of source codes, symbols, and more importantly, an application SME. Every time you troubleshoot a virtual application for the first time, you are basically having to learn the very depths of the application. I used to work escalation-level support and many of the rules of troubleshooting and reverse engineering that I employed applied to virtual applications. But it is one thing to support a broken application when you have every developmental resource at your disposal because you happen to work for the company that makes the very software you are troubleshooting. It is another monumental task to have to isolate and troubleshoot one that is from another vendor. Fortunately, the scope of App-V applications are limited to applications that run on windows-based operating systems.

To start, let us talk about how to avoid rabbit holes. We need to focus first on what not to do. The reasons people struggle and often never resolve application issues is usually tied to one or more of the following rabbit holes:

• They apply a uniform . . . . rote . . . rather a “checklist” mentality where they run through the same 10-15 things to try that may possibly resolve the issue (just like purchasing a bunch of lottery tickets may land you some cash.) But when it does not work – you are back at square one and you now have several hours of your life you will never get back.
• They assume that a method that they discovered on the Internet (and used successfully) to troubleshoot one application successfully will work on others. After all, each problematic application is the same, right? No, each application is like a snowflake – no two are alike.
• They troubleshoot only the error and not the context of the error. While true, the error is the first clue, just keying the error into Bing and looking at the first entry may possibly lead you down a rabbit hole.
• They assume the more they learn about one particular tool, the better they will be at troubleshooting virtual applications. The fastest way to be unproductive with Process Monitor is to apply a rudimentary blanket universal approach every time you use it. You will head straight down a rabbit hole.
• The use the wrong tool to try to find/fix the wrong problem. Sometimes Process Monitor may not be the right tool.

 

Speaking of Process Monitor . . .

So let’s look at troubleshooting virtual applications using Process Monitor. How do you learn how to effectively do it? In my opinion, it boils down to combining several areas of knowledge –

Foundation: This is knowledge of the platform. In order to understand how applications break, you need to understand how applications work. In this case, we are dealing with Windows-based applications. A strong foundation of knowledge can be obtained by reading up on the design and internals of the windows operating system, the Win32 API, the native NT API, etc. all help to build this foundation. You do not necessarily have to know everything there is to know about the windows operating system, but having a good understanding and the ability to reference development references will suffice. Even if you are not a programmer by trade, reading the Windows Internals book (sometimes several times) will open your eyes to how critical elements such as processes, threads, and objects work. The Windows Internals book (by Mark Russinovich, David Solomon, and Alex Ionescu) is now in its 6^th edition and has been split into two volumes due to its mammoth size. In addition, you have several references online that you can research and reference. The Windows API is online at MSDN (http://msdn.microsoft.com/en-us/library/cc433218%28VS.85%29.aspx) and the Windows Drivers Kit (not needed as much for virtual applications) is also online at the WDK site (http://msdn.microsoft.com/en-us/windows/hardware/gg487428 for free. You can even have an offline reference for the Win32 and .NET API’s through the downloadable SDK’s (Windows 7 and .NET 4 here at http://www.microsoft.com/en-us/download/details.aspx?id=8279)

Operational: This is knowledge of the tool. Assume the example is Process Monitor, you have to be able to efficiently leverage the right feature when needed. Since Process Monitor is the most commonly used tool used in troubleshooting virtual applications, it is essential that you understand all of the features. For any Sysinternals Tool, I would recommend the Sysinternals Administrator’s Reference by Mark Russinovich and Aaron Margosis. In the case of Process Monitor, everything useful regarding the tool is discussed. Once you know how the tool is used, you can then proceed to apply its uses to the right task.

Application Context: This is the context of the failure within the application. You cannot teach a tool if you do not know the right context of the tool especially when the use of the tool will vary from issue to issue. The application that yields an “UNKNOWN ERROR” must be troubleshot differently from an application that performs poorly, crashes, freezes, disappears without a trace, etc. I find this to be the most important skillset – the “X” factor, or so-called “Art” of troubleshooting. Those who are great at this tend to know context and thus, are able to isolate quicker.

You may find this overwhelming at first . . .

 

 

. . . and it will take practice and repetition. But like any “art” the improvement will come and you will begin to develop your own style.

Sequencing is not Virtualization and Virtualization is not Sequencing

A final note: There are big differences between “sequencing problems” and problems that are actually virtualization issues where an application is not functioning as expected. Often virtualization problems are labeled as “sequencing problems” when in fact, no problems occurred during sequencing. The problem occurred *AFTER* sequencing. Now it is true that the resolution to the virtualization problem may result in a modification to the sequencing process (i.e. a custom recipe.) True “sequencing” issues rear their ugly heads during the actual sequencing of an application.

MDOP 2013 embeds APP-V MSI’s inside of EXE files

If you are looking for the MSI installer files for APP-V 5 SP1, you will need to manually extract them from the EXE files. The App-V 5.0 SP1 (Client/Server) MSI’s can be extracted using this command:

appv_xxx_setup.exe /Layout /LayoutDir=”<path>”

 

i.e.

c:temp>appv_client_setup.exe /Layout /LayoutDir=”c:temp”)

 

The language pack MSI will be laid out with the client or server MSI in the location/path specified in /LayoutDir.

 

This command is documented here http://technet.microsoft.com/en-us/library/dn144770.aspx.

App-V 5.0: Launching Native/Local Processes Within the Virtual Environment

Update 12/6/2014: With the release of App-V 5.0 SP3, there is an additional option now available. You can now use the RunVirtual registry for user-targeted applications. You might prefer this to the /appvve switch. Please read more about it here: http://technet.microsoft.com/en-us/library/dn858700.aspx#BKMK_runvirtual_reg_key

Back in 4.x, we pretty much had one of two ways to launch an application inside the virtual environment or “bubble” as it is often called. One way was through the use of a pre-launch script. This was cumbersome in that you had to either modify the cached OSD or even worse, modify the original OSD file and re-deploy/refresh. Since I always used this for troubleshooting purposes, I found it easier to modify the cached copy of the OSD file. Starting in version 4.5, we also had a way of launching an alternative native executable inside the virtual environment by simply adding parameters to the SFTTRAY.EXE command:

C:Program Files (x86)Microsoft Application Virtualization Clientsfttray.exe" /exe cmd.exe /launch "DefaultApp MFC Application 1.0.0.1

App-V 5 offers us even more flexibility in this regard. To start off, do not use the scripting elements in the dynamic configuration XML. It is completely unnecessary and a waste of time. APP-V 5 offers many ways of getting inside "the bubble."

Powershell

You can use the Start-AppVVirtualProcess cmdlet such as the following example to retrieve the package name and start a local process within the specified package's virtual environment:

$AppVName = Get-AppvClientPackage *name*

Start-AppvVirtualProcess -AppvClientObject $AppVName cmd.exe

This way allows you to do something similar to the previous variant of SFTTRAY in which you can initialize an App-V virtual environment substituting a local command as the process running within that virtual process.

 

The Command Line Hook Switch “/appvpid”

This allows you to apply the /appvpid switch to any command which will allow the command to run within the virtual process of the virtual process you selected by its PID (Process ID) as in the example below:

cmd.exe /appvpid:8108

The Command Line Hook Switch “/appvve”

Where the /appvpid switch requires the virtual process to already be running, this switch allows you to start a local command and allow it to run within the virtual environment of an App-V package and will initialize it. The syntax is as follows:

cmd.exe /appvve:<PACKAGEGUID_VERSION_GUID>

The syntax is a but cumbersome as it requires both the package GUID and the version GUID:

cmd.exe /appvve:bcfdbe1b-8bff-4502-b4e2-5a773e82d86c_2ef3aaf9-9864-44a3-85dd-bf99a6b4d8f2

Run Virtual

If you are working within RDS environments, and have a package that is published globally, you can also take advantage of the “Run Virtual” feature. You basically add process executable names as subkeys of the following key:

HKEY_LOCAL_MACHINESOFTWAREMicrosoftAppVClientRunVirtual

For example, if I have a locally installed application called MyApp.exe and I would like this application to run within the virtual environment, I would create a subkey called MyApp.exe (perhaps a helper application like Adobe Acrobat that may need to be called from a virtualized web application.) I would then put in as the default entry a REG_SZ value that contains the package GUID and the version GUID separated by an underscore (i.e. <GUID>_<GUID> or such as in the figure below:

 
Each native process that needs to run locally will require its own subkey beneath the Run Virtual key. As long as there is one version of the EXE on the system, placing the packageversion GUID combination in the default key value will suffice. If there is more than one version of the EXE installed on the machine, you will need to create a new REG_SZ value beneath the subkey and specify the value name by using the full path to the EXE.

Flexibility

The above options give you much flexibility in how you merge native processes with virtual environments. With App-V 5, these not only add interoperability options, but they enhance our troubleshooting toolkits.  There will not be any reason to go into dynamic configuration XML in order to do this.

PS: Yes, I know I have not been blogging enough on APP-V 5. I promise to change that. I do have customers to tend to, you know! 😉

Update 12/6/2014: With the release of App-V 5.0 SP3, there is an additional option now available. You can now use the RunVirtual registry for user-targeted applications. You might prefer this to the /appvve switch. Please read more about it here: http://technet.microsoft.com/en-us/library/dn858700.aspx#BKMK_runvirtual_reg_key