For those of you still using App-V 4 (hopefully at least App-V 4.6 on the client side and 4.5 SP2 on the server side due to supportability) you may have been reading about how App-V 5 resolves a lot of limitations of App-V 4 – especially those that revolve around scalability. I dealt with many customers in support and still get questions on existing App-V 4.x deployments. The most common one revolves around keeping the existing 4.5 server(s) running optimally. I figured I would let the users out there (who are still using App-V 4.5 servers) know that I know they are still out there and remind them of the key factors that lead to performance issues
Watch the Cores and Dispatcher
App-V 4.5 runs usually on four instances of the core process (SGHWVR.EXE or SQLQVR.EXE if using the lightweight streaming server) as well as the dispatcher service. These are usually the key processes to monitor for CPU spikes. In normal operation, utilization should be evenly dispersed. In some cases you may see one or more spiked out due to likely one of the related issues listed below:
Update 4.5 Server to at least 4.5 SP2 plus these Hotfixes.
You should be running at the very least HF2 for App-V 4.5 SP2 with this hotfix being applied:
“Hotfix Package 2 for Microsoft Application Virtualization 4.5 SP2: March 2011” http://support.microsoft.com/kb/2507096/en-us
The App-V 4.5 Management Server should be updated to at least version plus the following should you be working with SQL database mirroring:
“FIX: An App-V 4.5 SP2 server cannot recover when an application virtualization database fails over” http://support.microsoft.com/kb/2873468/en-us
In addition to these hotfixes, you may want to also include this out of band fix which involves adjusting the back end database. I personally worked on this issue while I was in support and I can tell you that this fix makes a tremendous impact but must be done with caution:
“A publishing refresh might time out and return the 0A-10000005 error code on an App-V 4.5 client” us
Check the Database Values of Server Objects
If you are running with a less than optimal amount of cores or suspect performance issues after pre-creation of server objects prior to scaling out, you could be falling victim to the default limited values that come with pre-creating server objects. Check out this article to resolve it: http://blogs.technet.com/b/appv/archive/2010/05/10/pre-creation-of-server-objects-may-yield-certain-sub-optimal-values-in-the-app-v-sql-database.aspx
Large SFT-Server.LOG file
When the server log gets too large or is set to a high verbosity level, it can impact performance. Regular maintenance of the log files (planned purging) as well as setting an optimal verbosity level (2 likely) will prevent this from occurring. The logging level for the Application Virtualization Management Server can be changed in the registry at:
Keeping it at a level no greater than 3 is recommended.
Large APPLICATION_USAGE and MESSAGE_LOG tables
These tables can grow very large if unchecked, be it intentionally (i.e. no SQL agent running, running SQL Express) or unintentionally. The background of this long standing issues can be found here: http://blogs.technet.com/b/appv/archive/2008/08/04/troubleshooting-softgrid-database-growth-issues.aspx
The trouble is that you might be encountering this issue because the SQL jobs are not running properly or at all. Especially if you are running a more recent version of SQL Server or have just migrated databases. The Technet gallery has some scripts that can help you fix these jobs should they be failing:
SQL script that creates the SQL 4 Jobs that are required by the App-V DB: http://gallery.technet.microsoft.com/SQL-script-that-creates-b6345446
SQL script to allow App-V Check Usage History job to run on SQL 2008: http://gallery.technet.microsoft.com/SQL-script-to-allow-App-V-959bc1d4
SQL script to allow App-V Check Usage History job to run on SQL 2008 #2: http://gallery.technet.microsoft.com/SQL-script-to-allow-App-V-f656ac69
Limitations of Ephemeral Ports Used for RTSP
Delays in launching and launch timeouts can be traced to a defining limit in the amount of service requests an App-V management or streaming server can hold. This is due to the ephemeral port range defined by the default App-V 4.5 server configuration – http://blogs.technet.com/b/gladiatormsft/archive/2011/08/31/how-to-adjust-increase-reduce-the-rtcp-rtp-port-ranges-for-use-with-rtsp.aspx
Streaming Performance due to improperly configured Offloading or Block Size
Both the Underlying TCP offloading stack configuration and the block size for RTSP can be factors to slow streaming performance:
TCP Offloading Information: http://support.microsoft.com/kb/951037
RTSP Streaming Issue with block size: http://blogs.technet.com/b/gladiatormsft/archive/2011/11/28/the-default-block-size-of-64kb-in-app-v-4-6-and-later-may-cause-slow-streaming-with-certain-network-configurations.aspx
As App-V 4.5 is in extended support, the likelihood of new problems is low but not impossible with App-V 4.6 SP3 running on modern operating systems. However, I would recommend you verify the above before contacting support.
The Default Block Size of 64KB in App-V 4.6 and Later may Cause Slow Streaming with Certain Network Configurations
Starting with the 4.6 sequencer, the default block size for sequenced packages was changed to 64K. In addition, the option to adjust this was removed from the sequencer. The block size used to be an issue when the network bandwidth was limited and large blocks could not be transferred. Now with more robust networks in place, this is not a problem anymore.
With certain 4.6 client configurations (using packages sequenced on 4.6 sequencers and later) involving RTSP streaming from management servers or streaming servers, users may notice significantly longer RTSP streaming times.
If you want to set the block size lower than 64K with the App-V 4.6 sequencer, you can still do this via the command-line sequencer. This will require an installation program, script, and/or batch file that will run completely unattended. The command line parameters are found here:
You can use the /BLOCKSIZE option to specify a block size parameter of less than 64 (i.e. 32.)
To rectify this issue post-sequencing, you will need to make the following adjustments on the App-V Management server and/or streaming servers when using RTSP:
1.) If using Windows Server 2003 for your App-V Management server, you can modify the TCP/IP settings on the Windows Server 2003 App-V Management server to immediately acknowledge incoming TCP segments per the following KB article: “Slow performance occurs when you copy data to a TCP server by using a Windows Sockets API program” (http://support.microsoft.com/kb/823764).
2.) Use an alternative protocol for streaming (HTTP or SMB.)
3.) Use a third-party sequencer/packager/encoder to adjust the block size.
4.) Some of our customers have had success with adjusting TCP optimizations. For example, on servers running Windows Server 2003 Service Pack 2, you can turn on the TCP optimization feature “Receive-Side-Scaling” on by enabling the following registry parameter:
Data Type: DWORD
NOTE: This is only one part of the SNP (Scalable Network Pack features) that needs to be turned on. Other features such as offloading or TCP chimney do not need to be enabled. Please be aware of the fact that other applications that may be running on that server may not be tuned well for TCP optimizations. You will also need to reboot the server for this to take effect.
5.) For servers running Windows Server 2008 and later, it is recommended to also have the receive window auto-tuning level set to either normal or experimental in addition to RSS being enabled.
To enable TCP Auto-Tuning and RSS:
- From an elevated command prompt, run the following command:
netsh interface tcp set global rss=enabled autotuninglevel=normal
- Reboot the server.
To stream applications via RTSP, the App-V server uses three channels that are carried through three TCP sockets. First, the App-V client uses the RTSP channel to set up a connection with the App-V Management or Streaming Server. The server opens two private ports (one for the RTP channel and one for the RTCP channel.) The server also sends the port numbers to the client in a response. The client then opens two sockets. Then, the client connects to the private ports that are created on the server. The application is then streamed over the RTP channel. The RTCP channel provides real-time control over the data channel.
For each application that is streamed to an App-V client, a new set of RTP and RTCP channels are created. However, all streamed applications use one common RTSP channel. The RTSP channel is also used to discover available applications on the App-V server.
The default port range is from 49152 to 65535. This can be adjusted on either the streaming server (lightweight) or the Management server (heavyweight) but each one uses different configuration sources.
This defines the lowminimumstart range for the private ports. This will be used for both RTP and RTCP. For the App-V Streaming Server, it is found in the registry at the following location:
Data Type: DWORD
The default value is 49152 but the accepted ranges are from 1,025 – 65535.
For the App-V Management Server, this configuration comes from the SQL App-V Datastore. The database table that stores this is SOFTGRID_SERVERS and the column is rtp_min_port. The default value is 49152 but the accepted ranges are from 1,025 – 65535.
This defines the highmaximumstop range for the private ports. This will be used for both RTP and RTCP. For the App-V Streaming Server, it is found in the registry at the following location:
Data Type: DWORD
The default value is 65535 but the accepted ranges are from 1,025 – 65535.
For the App-V Management Server, this configuration comes from the SQL App-V Datastore. The database table that stores this is SOFTGRID_SERVERS and the column is rtp_max_port. The default value is 65535 but the accepted ranges are from 1,025 – 65535.
Installation Options on the Streaming Server
Both RTPTCPMINPORT and RTPTCPMAXPORT are available command line options during the installation of the App-V Streaming Server. These options are not available on the App-V Management Server installer.
Chances are, if you have deployed App-V in a large-scale TS/RDS environment, you may have come across this intermittent error message:
Application Virtualization Error – The System is too busy to complete the request. If the problem persists, please report it to your system administrator
Sometimes it is followed by a specific error code.
Error code 1690140A-200001CD
The error may even follow another error such as:
The Application Virtualization client could not be started. The Application Virtualization Service is not running. Report the following error code to your System Administrator: Error Code: 4602847-0FA10612-00006003.
Normally, when we get messages about the “System” being too busy to complete the request it is misleading. What is really too busy to complete the request is the App-V front end component.
When The SFTTRAY command launches, if it sees another instance of itself running, it tries to pass off whatever it was supposed to do to the existing sfttray process. If it sees that other instance but the handoff times out because the other instance isn’t responsive, it will return with this message. The error message really is telling you that SFTTRAY (or another front end component such as VAPPLAUNCHER or SFTDDE) is unable to complete the request because an App-V client component is too busy to complete the request.
There could be many underlying causes to this resulting from the SFTLIST.EXE (client service) being too busy or even the desktop configuration controller component (SFTDCC.EXE) not being responsive. If this is tied to a global DC refresh, or periodic DC refresh we could have a known issue with SFTDCC not releasing handles.
The 6th Hotfix for 4.6.0 addresses this and is available:
We recently also came across another cause of this issue and it is outlined here:
In this case, the RTSPS protocol was the culprit. RTSPS (RTSP over SSL) is limited to 255 channels and each package takes two channels. If the total number of channels exceeds this value then the error above is generated. To resolve the issue, decrease the maximum number of channels in use at any given time by scaling out the number of RTSPS servers.
Along that line, even if you are using RTSP, this could also be tied to scaling issues. If your DC/publishing refresh intervals are set to frequently (especially for terminal servers) you may be putting too much of a load on the SFTDCC process. Try reducing the frequency of periodic refreshes and/or increase the number of App-V Management servers for load-balancing purposes.
If you are running Citrix XenApp on a Terminal Server/RDS Server, you could be waiting on another seamless session to logoff. A good thing to verify is that the Citrix WFSHELL process releases SFTDCC properly:
- On the Citrix server, start Registry Editor.
- Locate and then click the following registry key:
- Right-click LogoffCheckSysModules, and then click Modify.
- In the Value data box, type sftdcc.exe, and then click OK.
Note If multiple values exist, separate the values by using a comma. For example, use the following format:
- Exit Registry Editor.
And finally, as always, try to follow the guidelines as recommended in the following KB article (if you are running Windows 2003 x86 – all of them.)
Under the section, “General Terminal Server Recommendations when running the App-V client” some applicability varies depending on the operating system platform and version.
- “Pre-cache all applications 100%. “(Applies to all operating systems and platforms.)
- “If you use Windows Server 2003 or older, implement a reboot cycle of the Terminal Servers . . .” (Should not need to do this on Windows Server 2008 and later.)
- “Implement UPHClean on the Terminal servers.” (not needed on Windows Server 2008 and later.)
- “Implement the following registry changes and reboot the servers” (Page Pool Memory settings not needed on Server 2008 X64 and later)
- “Verify that none of the App-V Application packages have following Virtual services if so remove them or set them to disabled.” (Still a good idea for all TS/RDS servers.)
What about the 00006003 error?
Ah yes, I alluded to that earlier. Often when this is happening, a third part filter driver is preventing the App-V Client service (SFTLIST.EXE) from starting. Check for encryption, or storage security-related applications.
After upgrading to the Application Virtualization 4.5 Management Server (Service Pack 1 or 2), you may get the following error when streaming applications via RTSP:
- A network error occurred error : 4524284-15603808-10000003
- You will also get this error if the installation is a clean installation. Publishing/Refresh still works but streaming applications fails.
This can occur if the content path is set incorrectly in the registry. This is likely caused by an incorrect value set during the installation. Please note that this error is different from previous versions and does not yield the normal 1F4 or 194 error.
Solution is to correct the value for the content folder under: