I have been working issues where it was difficult to determine why a specific application in a virtual application would not launch or trigger under certain circumstances. In particular, when another application called the application directly in order to pass data to that virtual application. Normally, when a native application is brought into the virtual environment through a package or a connection group, it can call other applications and virtual applications for the purposes of sharing data. In App-V 5, a native application, through the shell, can also call a virtual process and pass a file to it. In most cases, this will work with some exceptions – however – I have been able to overcome most of these issues through troubleshooting.
In this particular case the application being called is a virtual application. The application calling it was a home-grown line of business (LOB) application written by the customer to retrieve archival data from a secure portal. When the data was retrieved the data would open up within their preferred ZIP archival utility 7-Zip. 7-Zip was virtualized in this environment with App-V 5. While the virtual application would run would run fine when launched from a shortcut, when called from another virtual application within the same package or from another shell-based application as I later found, it would fail. Monitoring the customer’s custom application from Process Monitor yielded a 135 (hex 0xc0000135 or decimal 11073741515) exit code and the application yielded a pretty direct error message:
Now, before going down any deep troubleshooting, I did my due diligence and attempted reproduction on other machines including a machine as CLEAN as possible. On the clean machine and all but two machines, this issue was NOT reproducible. The plot thickens. Time to eliminate variables.
No, it wasn’t Bad Environment Variables
This was one of the first things I verified. In fact, neither the working or non-working setups had 7-Zip in their search path. Surely including it would fix the problem, and it did make the issue go away – but – while this is a fix – we had not found root cause. I don’t like not having root cause. Besides, the workaround was obviously not needed on the machines not experiencing the issue.
Time for Procmon
I took two different Procmon traces – one from a working machine and one from a non-working machine. I then created a simple filter with “7Zfm.exe” – the primary executable for 7-Zip – under the where “Path” “Contains” filter:
Once I looked at the non-working one through this filter, I had all of the answers in hand. I just needed confirmation from the customer’s developer. You will notice in the trace below, before any search of the path occurred, there was a query to the app paths key within the registry:
The app path returned a local path and not a virtual path. The path was also not existent on the machine. Once that failed, the search path was parsed and the subsequent searches failed. Upon confirming with the developer, the application was calling 7-Zip using the ShellExecuteEx function (http://msdn.microsoft.com/en-us/library/windows/desktop/bb762154(v=vs.85).aspx) which can be interrupted easily by a “rogue” app path. Upon digging into the registry, we found that the working machines had the correct app path in HKCU . . .
And the Procmon trace showed it was quickly found in the app path:
. . . but the non-working machines had the wrong app path in HKCU:
Removing this rogue app path (which came from their roaming profile) resolved the issue.
More on App Paths
One of the reasons why a native application can call a virtual application is due to the support of app paths in App-V version 5. We did not have this luxury in previous versions of App-V. App Paths allow the shell to call the application by executable without having to use the PATH environment variable. Now with App-V 5, you can call a virtual application by typing in the application in the search dialog box or the “Run” menu or even through API’s (ShellExecuteEx.) App Paths are further documented here: http://msdn.microsoft.com/en-us/library/windows/desktop/ee872121(v=vs.85).aspx
When publishing the package to the user in App-V 5, the path to the application is registered in HKCU SoftwareMicrosoftWindowsCurrentVersionApp Paths. When the package is published globally, it will register in HKLMSoftwareMicrosoftWindowsCurrentVersionApp Paths
Prior to version 5 of App-V, when you sequenced an application that required a reboot, the reboot was simulated in that the sequencer would process the reboot action including the Pending File Operations and the RunOnce registry keys (RunOnce, RunOnceEx, GuiRunOnce, etc.) when monitoring ceased. In most cases, the simulation would be fine and all of the necessary reboot tasks would be processed properly. There were some exceptions – notably the famous disappearing Google Chrome application famously mentioned in “Stealth Puppy’s” [I love that handle] “Case of the Disappearing Application During Sequencing http://stealthpuppy.com/the-case-of-the-disappearing-application-during-sequencing/ “ where after the reboot tasks processed and the monitoring of the installation of GoogleChrome completed, the Program Files directory simply disappeared.
In defense of the sequencer, this was not as much the fault of the App-V sequencing process, but also an issue with the mechanism of the installation. This was easy to prove by going through the sequencing process and verifying the contents prior to stopping the monitoring process. The files were indeed correctly located prior to the stopping of monitoring:
However, when we ran PENDMOVES.EXE (http://technet.microsoft.com/en-us/sysinternals/bb897556) from Sysinternals, we found this:
The operation pending was to DELETE the actual files. This would explain why the files got deleted after the monitoring process ended. The next logical step would be to see if this was a properly captured operation and not just some bug in the sequencer and install the application natively. Prior to rebooting the machine, run the PENDMOVES.exe utility again. Upon doing this, you will find the EXACT same pending file operation – DELETE C:Program FilesGoogleChrome.
So the native installation does indeed remove the files upon a reboot. The difference is after the reboot, the files are back where they need to be! How is this possible? Google triggers updates using the Google Update service which is set to start automatically. The missing files would force the Google Update service to pull down the most up to date version. Indeed a prime example of some of the exceptions to the rule that the 4.6 sequencer was not catching.
Going back to the original issue – and never passing up an opportunity to showcase a Sysinternals utility – I would recommend a workaround. Given the issue was a DELETE operation, you could leverage another Sysinternals tool to add in a recopy function as a workaround with 4.6. Before actually stopping the monitoring process, first copy the files in question to a safe area.
Then I used the MOVEFILES.EXE utility (http://technet.microsoft.com/en-us/sysinternals/bb897556) from Sysinternals to have them moved back during the simulated reboot.
Changes in the V5 Sequencer
So to ensure that the sequencer works better with installations that require reboots (or even multiple reboots) the sequencer was changed to actually reboot. When the sequencer detects that the application is requesting a reboot, all tasks are recorded and deleted (except for the PendingFileRenames key) to force resume only after the system is rebooted and the monitoring has resumed, otherwise, these tasks could occur prior to the sequencer resuming. This information is recorded to disk (although obfuscated) and the system will reboot as normal
After the reboot, the PendingFileNames key will be processed as it would in a normal reboot. Upon the user logon, the sequencer will be launched (as it registered itself to do that prior to reboot.) Upon relaunch of the sequencer, the sequencer reads its state from the registry and the file system and restarts monitoring. The first tasks that will be processed will be the RunOnce tasks that were captured prior to the reboot.
While the Sequencer User Interface allows for the preservation of settings and a real system reboot, this is not the case for applications sequenced using the PowerShell Sequencer Cmdlets. Since this is mostly reserved for scripted MSI packages, those usually have suppressed reboots anyway.
Before you ever want to begin testing client publishing, you want to ensure that your entitlements (assignments of packages and package configurations to groups) are properly set and the publishing metadata document is properly updated with the package, configuration, and entitled groups. You could otherwise find yourself in a sea of frustration testing client publishing especially when there is nothing new to retrieve.
Assuming all of the services were installed properly, you will have two primary web services responsible for entitlement and publishing. They may reside on the same machine or they may reside on different machines. Both the Management Service and the Publishing Services run as worker processes within IIS as all services are web-based leveraging RESTful (representational state transfer) technologies. (http://en.wikipedia.org/wiki/REST.) The clients sync with the Publishing Service to retrieve data from the Publishing Service only. The clients do not sync with the Management Service which means the Management Service and the back SQL database no longer create the potential for a single point of failure as was the potential with previous versions of App-V.
Package Entitlement Process
Understanding the steps from entitlement to publishing will help you if you need to isolate actions for troubleshooting issues with packages are not properly publishing. When you open the Silverlight-based management console, just like in previous versions, you are communicating via the management web service to the App-V data store.
When a new package is added or modified in any way, after administrative access is verified, the manifest is extracted from the APPV package to obtain key information and configuration (shortcut’s and FTA’s). This information is then placed into the database. The method in which the package is added governs what the package URL location format is captured. This of course can be overridden at the client with the PackageSourceRoot configuration item. In addition to having appropriate administrative access for the user running the Management Console, the services will need to have proper access to the content being added.
If you specify an SMB (UNC) path to the package and file share is located on the same machine as the Management Service, you must ensure that the NT AUTHORITYNetwork Service account has at least read access to the package location. If you specify an SMB (UNC) path to the package and file share is located on a different machine than the Management Service, the computer running the management service must have at least read access to the packages. If you specify the source using an HTTP-based URL, then the computer running the management service must have at least read access to the packages. Take note that you would replace the computer account with a service account should you elect to use a service account for the Management Service work process.
Publishing Sequence Number
Upon successful add or modification of the package the next step is to update the Publishing Sequencing Number. Any change in Package Configuration warrants a change in overall publishing data. This begins by incrementing the Publishing Sequence Number in the database in the Configuration table.
This is a crucial value as when the Publishing Service syncs with the Management Service, the Management Service compares the sequence number between the one contained in the publishing metadata versus the one in the database. If the metadata and database match sequence numbers, the existing metadata is returned to the Publishing Service. If the database number is different, then new metadata will be generated to the file system then returned to the Publishing Service. The Publishing Service will then store the data in its file system (PublishingMetadata.xml.)
The Publishing Service syncs with the Management Server under the following conditions:
1.) At a periodic interval: The registry value PUBLISHING_MGT_SERVER_REFRESH_INTERVAL located in HKEY_LOCAL_MACHINESOFTWAREMicrosoftAppVServerPublishingService governs the interval. It defaults to 600 seconds (10 minutes.) You can tweak this value but remember it is in effect whether the Publishing Service runs on the same machine or on separate machines. If the Publishing Services runs on a spate machine, it must be added as a publishing server in the Management Console.
2.) Upon StartRecycling of the Publishing Service Worker Process. Since the services exist as worker processes in IIS and not as services controlled in the Service Control Manager, you must restart the services by either rebooting, resetting IIS through IISRESET, or cycling the service directly by stopping and starting the worker process within the IIS Manager. If the Management Service and the Publishing Service reside on the same machine, I would be careful using IISRESET to cycle the services as you cannot control the order as to which service starts up first. You could possibly run into sync issues where the Publishing Service is unable to sync with the Management Service because it may still be in the process of restarting.
Take Note of Hotfix 2 for the App-V 5.0 Service Pack 1 Publishing Service
Hotfix 2 for App-V 5 SP2 (http://support.microsoft.com/kb/2897087/en-us) updates the Publishing Service to query Active Directory directly for the user’s group membership rather than using the Access Token. This allows for entitlements to take effect without the user having to log off (or in the case of computer targeting) having to reboot. Be advised that in addition to the hotfix being applied, you must make the following registry adjustment:
Data Type: DWORD
App-V 5: On App-V Package Modernization with the OPC (Open Package Container) or: One Package Container to rule them all!
Much of the changes in App-V 5 revolved around moving to more integrated components (NTFS, a “real” registry, native API’s on the client, representational state transfer on the server side, etc.) – but one of the most major key developments was the modernization of the AppV Package format. The reason behind this was simple. Using this new standard based upon an open standard allows for the modernization of your legacy applications with regards to package format. This provides synergy going forward by synchronizing the packaging of legacy application with the native packaging format of your modern applications existing (or to come.)
This drastic change makes a huge difference. The package is open and human readable. No more closed, custom package format full of mystery. Pre-requisites, installation process, experience opt-in settings, configurations, and site-specific deployments can be simplified into a single-command deployment. In addition, the package format exists in an installed state that follows the principles of the open package specification outlined in the ISO standards for the Open Package Specification (OPC) – http://www.iso.org/iso/iso_catalogue/catalogue_tc/catalogue_detail.htm?csnumber=51459
This is also clarified in the following magazine article from MSDN: http://msdn.microsoft.com/en-us/magazine/cc163372.aspx
The OPC Standard
The Open Package Conventions, or as it also referred to as Open Packaging Container, are a set of standards spearheaded by Microsoft to store binary data and XML metadata in a single entity leveraging Open XML and ZIP formats. OPC-based file formats combine the advantages of leaving the independent file entities embedded in the document intact and resulting in much smaller files compared to normal use of XML. With OPC, you have a package in ZIP format that can contain mandated descriptor files, content definitions by MIME type, followed by application-specific metadata and resources files. With the embedded MIME definitions it also removes many issues associated with simple document file type extensions.
The container could be treated as a package or a document – as in the case with Office documents. Regardless, each is technically a ZIP archive containing resources identified as parts and each part has a unique path identifier by URI and by a specified content type expressed as a MIME type to describe the actual data stored in that part. You can see this when you browse an APPV, APPX, or even a .DOCX file using a ZIP utility.
In addition to a hierarchy of directories and parts, OPC packages representing document types commonly use relationships to access content through associations. These Relationships are composed of four elements:
- The identifier (ID)
- The optional source (the package or a part within the package)
- The relationship type (a URI-style expression that defines the type of the relationship)
- A target (a URI to another part within the package or to an external resource)
The extension ".rels" is reserved for storing relationships metadata within "/_rels" subfolders. The subfolder name "_rels", the file extension ".rels" within such directory, and the filename "[Content_Types].xml" in any folder are the only three reserved names for files stored in an OPC package. Otherwise, it is extensible to associate additional resource type with their specific package and application standards (for example – those resources tied to App-V i.e. StreamMap.xml, PackageHistory.xml, etc.)
Notice in an APPV package while the [Content_Types].xml is present, the .rels components are not. AppX and AppV handles these relationships differently using its own AppXBlockMap.xml and AppXManifest.XML files. More information can be found in that excellent MSDN article I referenced earlier: http://msdn.microsoft.com/en-us/magazine/cc163372.aspx
AppX (Modern Applications)
AppX is the OPC-compliant package format for both Modern applications and Windows RT packages (what we commonly refer to as store applications) as referenced in: http://msdn.microsoft.com/en-us/library/windows/apps/hh464929(v=VS.85).aspx
The AppX package can be viewed with a ZIP extraction utility just like the AppV package. Like the APPV file format, the APPX package uses declarative XML to define requirements, dependencies, and relationships defined in “AppxManifest.xml” instead of a relationship resource folder.
AppV (Legacy Applications)
While the AppX package format can be generated during modern application development (http://blogs.msdn.com/b/windowsappdev/archive/2012/12/04/designing-a-simple-and-secure-app-package-appx.aspx) your existing applications can be sequenced with the App-V sequencer and deployed, managed, and serviced as App-V packages that have the binary equivalency of the Windows Store APPX format. Even better, these can be deployed to Windows 7 computers running the App-V 5 Client (where store-based modern applications cannot.) Be advised however, that while App-V 5 uses the same file format as modern applications, it does not mean you can deploy or publish an App-V package as a Windows Store application. They only share the same modern package format.
UPDATE 4/29/2015 – The AppV package format will be leveraged for Store deployments of Win32 applications going forward in Windows 10.
That’s not all . . .
Document Formats have gone that way as well. Ever opened a DOCX or XLSX file in 7-ZIP? You can see the OPC-based layout this way! This is by no means new. This began with Office 2007. In fact, other vendors besides Microsoft have started to adapt this standard including Autodesk, Siemans, and Mathworks. There is an excellent historical reference on MSDN that introduced the OPC document format and relationships back in 2009 that is very helpful in understanding the Open Packaging standards as they apply to Office document formats here: http://msdn.microsoft.com/en-us/library/ee361919(v=office.11).aspx
For sequencing in App-V 5, the new ETW model simplifies the process and moves App-V to the Windows standards for event tracing. Even better, the sequencer not only has two logs to worry about (operational and administrative) but a simple process can occur to enable more verbose debug logging.
In App-V 4.6, the process was not that simple. While the logs did not write to the Event Viewer-able logs, all but one are text-based which makes for easy manipulation with your favorite log parser. I prefer Trace32 of course! These log files are stored in the logs subdirectory of the Sequencer installation directory which defaults to C:\Program Files\ Program Files\Microsoft Application Virtualization Sequencer\Logs. Certain logs pertain to specific functions so the relevancy will vary on whatever your troubleshooting scenario might be.
SFT-Seq-log.txt: The majority of sequencer logging occurs here (Uploads to virtual environment, downloads from the virtual environment, service starts and service stops, etc.)
SFTrbt.txt: This is the sequencer reboot log file. When the 4.6 sequencer simulates reboots, the elements that are processed will be tracked in this log file.
SFTCallBack.txt: This is a more simple logs that allows you to reconsile process starts and stops during sequencing. It works great in conjunction with a process monitor log.
Filter.log: Outside of working with Microsoft Support, this log is not very useful as it is obfuscated. It tracks file activity but must be decoded with an internal utility. You can enable further tracking into a file called files.txt which will contain a log of all files created in the VFS. This can be enabled (although it will increase sequencing time) by enabling the following value in the registry:
- Key: HKEY_LOCAL_MACHINE\Software\Microsoft\SoftGrid\4.5\Sequencer\Configuration\
- Value: FileManifest
- Data Type: REG_DWORD
- Data: 1
SFTrpc.txt: This is the log file created by the monitoring element SFTRPC.EXE and in addition to also capturing process startup and shutdowb, will also contain verbose diagnostic information about each monitored shortcut.
In addition to the sequencer logs, you can also leverage process monitor (http://technet.microsoft.com/en-us/sysinternals/bb896645.aspx) and verbose MSI logging (https://madvirtualizer.wordpress.com/2014/03/17/enabling-advanced-windows-installer-logging/) if you encounter errors within the application during sequencing.
App-V 4.6 is still very prevalent out there and will be for a while. With the releases of Windows 8 and Windows 8.1 brought additional service packs for the App-V 4.6 client which means upgrades and/new installs for newer operating systems. Several weeks back on this blog, I went over how to enable advanced MSI logging for troubleshooting MSI installs and upgrades (Remember VOICEWARMUP – https://madvirtualizer.wordpress.com/2014/03/17/enabling-advanced-windows-installer-logging/ ) but I would like to now address some follow up emails I received. Admins would like more specific information on how to go through and read that potentially enormous log in order to find out what is failing where and when.
I would never advising reading a verbose MSI installation log from start to finish especially when dealing with potentially asynchronous actions. MSI logs also have an excessive amount of rollback information incorporated into the log upon failed installations. The seasoned IT Pro often looks for specific keywords such as “error” and “failed” and that can be misleading as not all logs generate these types of messages. In addition, searching on the string “error” can also yield false positives as well.
When I am looking at Verbose MSI logs of App-V 4.6 client installs, I usually analyze the log by doing the following:
- Searching for the error string generated in the App-V Installer User Interface with quotations.
- Searching using the string “1603.” See if it indicates that a custom action has failed.
- Searching using the string “Value 3.” This will indicate an install error. This can also help to identify the custom action failure.
- Searching for string “IsInBadState()” can also be helpful if there is an issue with a failed driver install. This is especially useful in troubleshooting an upgrade. Usually when this occurs, you usually need to delete the driver configuration and state of the specific App-V file system driver specified in order to reattempt the upgrade.
Finally if you need to walk through the App-V custom actions, you can do so by searching by the strings ‘SWI” or “SGC” as all of the App-V custom actions begin with these prefixes.
CustomAction SWI41sp1UpgradeFix returned actual error code 1603
You can walk through the logging of each key App-V custom action. Once you’ve identified what custom action failed, you can then use the following reference to find out specifically what was being attempted with the custom action here: http://support.microsoft.com/kb/2465574. Even though it specifies SP1, it is still valid and helpful for SP2 and SP3. For example the action reference above would be:
Installer : Client
Method name: SWI41sp1UpgradeFix
Description: Modifies an installed instance of the Softgrid 4.1SP1/4.2 client application to correctly support upgrading to a later version.
You can then dive deeper into the timeline of the action and align it with a more deep logging utility such as Process Monitor.
Whether it is integration with XenApp, XenDesktop, PVS, or UPM, App-V 5 is the recommended application virtualization and streaming model for Citrix Session and VDI environments. I work with a lot of customers who leverage App-V in this capacity and they have been asking me to maintain a “living post” that contains links and references to Citrix and App-V 5 integration resources.
So with help of App-V 5 SEE John Behneman, (you may recognize him from the official App-V support blog) below is a collection of Citrix KB and documentation links related to App-V 5.0 when integrated with Citrix technologies:
(Page will be updated)
Citrix Official App-V 5 Version Support Matrix for XenDesktop or XenApp and App-V 5.0
Choosing an application and desktop delivery method
XenDesktop 7.5 and XenApp 7.5 Features and Entitlements
Reviewer’s Guide XenDesktop 7.5
Citrix App-V 5 Integration: Internals
User Profile Management and App-V 5 – Required UPM exclusions for App-V
Random Application Fails When Using Standard Image Mode with the “Write Cache on Target Hard Disk” Option
Troubleshooting App-V 5
Citrix EDocs on App-V XenApp 6.5 Publishing: Note that they redirect you to how to publish App-V 5 apps with Server 2012 RDS:
Citrix EDocs on App-V 5 XenDesktop 7.1 Studio Publishing Integration:
App-V 5 and Citrix Integration
Microsoft-Citrix Virtual Desktop Infrastructure Technical Whitepaper
App-V: A Configuration Template for Deploying to Stateless RDS Clients on Citrix Published Desktops with Citrix UPM for Profile Management