On the Bloggers, Analysts, and the Shareholders.

Over the past 20 years, I have had many different roles in IT. I’ve been a helpdesk jockey, professor/instructor, sysadmin, developer, support engineer, escalation engineer, and now consultant. I’ve worked with a variety of industries as well. I’ve been both a customer and an employee of a Fortune 100 software company. As I have moved into various roles through my career, I’ve simultaneously watched the growth of the IT community in pontificating in various mediums ranging from community forums to full-blown tabloid tech journalism. I’ve learned what kind of statements garner respect and attention and what are often dismissed as hyperbole or sensationalism.

The Bloggers

The bloggers are supposed to represent the users and/or IT pros – the “pulse” of the community. In many cases the quality of the bloggers are positive as they derived excellent content and insights due to one or more of the following factors:

Experience: A blogger will likely be taken seriously if they have the experience to back up what they are talking about. This is why the best insights often come buried deep inside of community forums and not necessarily on the site of a full-time blogger or tech journal. Why? Because blogging is not their job. They ARE an IT Pro. Blogging is merely a hobby.

Depth of Analytical Thought: They demonstrate an outstanding aptitude for critical thinking. Even if the source is focused towards a specific vendor (or as many say – biased) the analysis is spot on.

Depth of Technical Thought: Simply – they know the technology inside and out. They yield a wealth of technical information and for that reason alone, they often command respect.

I am here to tell you the influence bloggers have on software vendors and products often depend on how they engage and embrace the community around the vendor and its products – regardless of how they may “bash” a product or feature or “praise” it. If the community respects the blogger, their stature increases with the software vendor. If the blogger is simply ranting or spilling out hyperbole for the sole purpose of “click-bait,” that can come back to haunt them. This is often a challenge for full-time bloggers who are often selling advertisements to generate revenue or perhaps are freelancing for a journal who pays them literally by the click.

The Analysts

When you build up that large amount of overhead you need to keep those clicks and ad views going, the blogger has no choice but to be a provocateur to remain relevant in the IT tabloid media that those same bloggers helped to create. When an IT analyst or an IT research firm publishes opinions or assessments, they are always taken more seriously as they represent a wealth of combined experiences and knowledge bases. They approach product, technology, and industry analysis in a much more scientific and data-driven process. The research firms publish both the analytical and the technical depth in every case.

The Shareholders

Since most major software vendors, at least in the US, are publicly traded, it is Wall Street that ultimately has the most influence on its direction. in IT, your shareholders are often your customers as well.

The Inspiration

I’d been wanting to write an article on this subject for a while, but this week, I was inspired to the write this article after reading three distinct articles relating to RDS/VDI –a technology I worked in extensively. I have the unique opportunity to cite examples of an attempt of influence by a blogger, a group of analysts, and a group of investors in a very busy week for the VDI industry.

The Blogger:

http://www.brianmadden.com/blogs/brianmadden/archive/2015/06/10/server-os-based-vdi-is-an-official-quot-feature-quot-of-windows-server-2016-apparently-microsoft-plans-to-continue-screwing-us-for-years-to-come.aspx – Basically, Brian Madden still hates how Microsoft does VDI. In other news, the Sun came out this morning.

The Analysts: http://www.gartner.com/document/3072722 – a brutally honest assessment by Gartner on why VDI is not ready for the cloud and what it will take to get VDI to a true cloud-based DaaS (Desktop as a Service.)

The Investors: http://www.valuewalk.com/2015/06/citrix-systems-inc-ctxs-elliott-associates-letter/ The investment Group Elliott Management reveal their desires for change at Citrix (the leader in VDI) in an open letter to its CEO and Board of Directors.

Which of those three articles that I mentioned do I pay the most attention to? Well, I always trust analysis over hyperbole – but money trumps all.

Categories: Uncategorized

Application Troubleshooting: An Error Message, Without Context, is Worthless

Note: This blog contains free stock photos that were too hilarious to pass up.

I was inspired to write this article based off an internal discussion I was involved with where someone was requesting a comprehensive list of all possible event log messages delivered in Windows Server. I have always been interested in the reasoning behind an “ask” such as this because it could be misguided. Where to begin? First of all, it is quite a loaded “ask” as it is not specifying whether we are talking about all of the in-box default error messages relating to Windows – the operating system specifically, or are we also talking about all of the roles and features which also contain event providers? Where would all of these go? What would this be used for? Second of all, what is the purpose?


Answers always baffle me: “Oh, for the help desk.” “As a reference.” “For our in-house troubleshooting guide.” The last example leads me to what often results in a bi-directional obtuse conversation where I re-ask what are the use cases and contexts of these error messages. Will the person requesting the error messages also be attempting to follow-up on each error message to determine all of the possible situations in which these may occur? That would be quite an insurmountable task. In most cases, no, these will be either copied-and-pasted into a “guide” – sometimes even being printed out into a very thick, but mostly useless material reference.

HRESULTS vs. Event Logs

First of all, the event log message tells you everything you need to know about the ERROR itself. This has evolved significantly over the past decade with advancements in the Windows. Often, you will get errors/exceptions thrown from an application and the level of description will depend on the generosity of the programmer/developer of the application. One thing is for sure – at the very minimum in most cases, an integer-based or hex-based response will occur. The most common of these are HRESULTS. A few years ago, due to the nature of overlap between operating system components and external software, a tool called ERR.EXE was made available on the Microsoft Download Center (http://www.microsoft.com/en-us/download/details.aspx?id=985.) This tool parsed all of the known header files for windows and applications to provide the corresponding strings and descriptions of known HRESULTS and error codes within the Microsoft software ecosystem. The use of the utility would yield all known matches such as the example below:


This utility was especially useful because it could also automatically translate the decimal-based equivalent of an HRESULT:


But . . . Here’s the Thing. Where do you go from Here?

Well, how did you get there? What was occurring when you encountered that error? For example: Let’s say you get an HRESULT 0x80000005. You use the ERR tool (or you know from memory) that it is “Access Denied.” What was occurring when this happened? Let’s say this occurred within an application. You could then leverage Process Monitor or another tool to trace the issue to see what it file/registry entry/etc. the program was trying to accessed. There is no way to give anything more than a generic recommendation without additional context related to the error.

In the case of an event log entry, there is much more information. The source component, Event ID, description, and more – along with additional XML detail. In the example below, what else would you gather from this simply looking up Event ID 36888 other than what you see in the dialog boxes?


That is where the scenario in which it happened plays an important role in determine what the root cause of this error is. In some cases, only one situation warrants a particular error and resolution. These are the ones we love – yet they are rare. In most cases there will be more than one scenario.

Why are Some Event Messages More Descriptive than Others?

Like with applications errors, event log entries vary regarding the degree of detail. In many cases during the development and evolution of a component, particular errors are hypothetically conceived during the development cycle and are laid out with the event tracing framework. These get further nailed down in the test and beta phases and the events are adjusted accordingly. At the release time as many clear-cut, known issues are mapped out through release notes and knowledge base articles. This process continues through the lifecycle of the component or software.


But again, these are only events. On all of my machines I have ever worked with, I would venture to say that I have only ever encountered a small fraction of all of the potentially available windows events which warrant errors or warnings. There are easily over 200,000 ETW and legacy windows events. Why reinvent the wheel and create a huge list for searching when you already have BING at your fingertips. And with Bing, you can search the event\error with additional contents.

What is probably the most beneficial troubleshooting assistant is the known resolution – in the form of a knowledge base article. The “Symptom(s)” section of most types of knowledge base articles is where the context is mapped out with as much detail as possible. It is usually in the form of “You are running [SOFTWARE\COMPONENT] and you are performing [ACTION]” or “you are attempting to [CONFIGURE\LAUNCH\RUN\CLICK ACTION] and you encounter the following [ERROR\MESSAGE]


The Symptom(s) section is usually followed by the “Cause” section. Often this section is prefaced with “This issue may be caused by . . .” indicating that this may not be the only cause of the issue. Yet this particular cause will be remediated\resolved by the “Resolution” section. That format is what makes the knowledge base article so great. It cuts right down to the break-fix scenario. Symptom-Error-Cause-Resolution.

Building knowledge bases “from the error up” is not an optimal way of building out a framework for a help desk. These are built and drive by the overall experience of the issue itself. Working in support for many years, one of many elements that separated the true escalation engineers from what could automated with a diagnostic utility or front-line support was the ability to resolve an issue for the first time. While the first question was usually “What’s the Error Message?” it was the second question that was most important – “What were you doing?”


Manageability Lingo, Standards, & Acronyms, Oh My!

January 10, 2015 Leave a comment


Since the dawn of the 21st century (and even before) you have been hearing many items related to acronyms interchangeably describing manageability features within Microsoft products (as well as others.) For example, WMI has been at the heart of most Microsoft Manageability products and solutions given the fact it is one of the primary interfaces within the Windows operating systems. While Microsoft’s WMI ties mostly to its products, it is based upon a series of open, universal standards. And this is the heart of deciphering how acronyms and standards can be interchangeably used to describe the same entity.

So let’s weave through the sometimes confusing relationship between these manageability acronyms – WBEM, WMI, CIM, DMI, DTMF, WSMAN, WinRM, and SNMP of protocols/interfaces/standards. In this little game, I will try to go through these acronyms within the average blog post attention span. WMI is Microsoft’s implementation of the open Web-Based Enterprise Management (WBEM), which comes from the Distributed Management Task Force (an industry organization.) WBEM relies on protocols – which can come from legacy standards such as RPC (Remote-Procedure Call) or DCOM (Distributed Component Object Model) or more modernized http-based SOAP standards such as WinRM (Windows Remote Management) based on the WS-MAN (Web-Services Management) standard. SOAP (Simple Object Access Protocol) itself, is is a command extension protocol designed to be used with HTTP (Hyper-Text Transport protocol – or the web) or SMTP (Simple Mail Transport Protocol – or internet email.)

The WMI interface – based upon the WBEM standard – is built upon an infrastructure centered upon the Common Information Model (CIM) and its respective Object Manager (CIMOM), is what links management applications and providers. The infrastructure also serves as the object-class store and, in many cases, as the storage manager for persistent object properties. WMI implements the store, or repository, as an on-disk database named the CIMOM Object Repository. As part of its infrastructure, WMI supports several APIs through which management applications access object data and providers supply data and class definitions.

Beyond WMI, WBEM’s architecture extends to a variety of underlying technologies besides WMI and Win32 because not everything is or will always be on Microsoft technologies – including the Desktop Management Interface (DMI), and the Simple Network Management Protocol (SNMP)  Some of these standards define data storage schemas as well as interfaces. Some define commands within communication protocols. Some or more modernized. SNMP has been deprecated in the most recent versions of Windows in favor of technologies such as WinRM.

I like to use the relationship of WinRM and WMI (alongside their open counterpart standards WS-MAN and WBEM) by stating that one is a management protocol and one is a management interface.


To Read more, check out the standards themselves:

WMI Explorer 2.0 is now on Codeplex!

November 11, 2014 1 comment

When I worked in support, I troubleshot WMI quite bit using many tools. One tool I still keep my eye on with regards to ongoing development was – and still is – the WMI Explorer utility. I am happy to report a new version of an excellent troubleshooting tool for WMI is now available:

WMI Explorer 2.0 is now available for download:



Microsoft .NET Framework 4.0 Full or .NET Framework 4.5.1
Minimum display resolution: 1024×768
Administrator rights to view some WMI objects
(Optional) Internet access for automatic update check

This is a very intuitive tool for visually troubleshooting WMI issues. It gives you a direct view into the WMI namespace.

New Features include:

New: Asynchronous mode for enumeration of classes and instances in the background.
New: Method execution.
New: SMS (System Center Configuration Manager) Mode.
New: Property tab showing properties of selected class.
New: Input & Output parameter information in Methods tab with Help information.
New: List View output mode for Query Results.
New: Update Notifications when a new version of WMI Explorer is available.
New: Connect to multiple computers at the same time.
New: Quick Filter for Classes and Instances.
New: User Preferences.
New: View WMI Provider Process Information.
Improved: UI display on higher scaling levels and resolution.
Improved: Connect As option to provide alternate credentials.
Improved: Display of embedded object names in Property Grid.

NOTE: This is not an official Microsoft tool, and is available “AS IS” with NO support.

Categories: Management, WMI Tags: ,

App-V 4.6 Sequencing: What are the logs really for?

September 20, 2014 2 comments

For sequencing in App-V 5, the new ETW model simplifies the process and moves App-V to the Windows standards for event tracing. Even better, the sequencer not only has two logs to worry about (operational and administrative) but a simple process can occur to enable more verbose debug logging.

In App-V 4.6, the process was not that simple. While the logs did not write to the Event Viewer-able logs, all but one are text-based which makes for easy manipulation with your favorite log parser. I prefer Trace32 of course! These log files are stored in the logs subdirectory of the Sequencer installation directory which defaults to C:\Program Files\ Program Files\Microsoft Application Virtualization Sequencer\Logs. Certain logs pertain to specific functions so the relevancy will vary on whatever your troubleshooting scenario might be.

SFT-Seq-log.txt: The majority of sequencer logging occurs here (Uploads to virtual environment, downloads from the virtual environment, service starts and service stops, etc.)

SFTrbt.txt: This is the sequencer reboot log file. When the 4.6 sequencer simulates reboots, the elements that are processed will be tracked in this log file.

SFTCallBack.txt: This is a more simple logs that allows you to reconsile process starts and stops during sequencing. It works great in conjunction with a process monitor log.

Filter.log: Outside of working with Microsoft Support, this log is not very useful as it is obfuscated. It tracks file activity but must be decoded with an internal utility. You can enable further tracking into a file called files.txt which will contain a log of all files created in the VFS. This can be enabled (although it will increase sequencing time) by enabling the following value in the registry:

  • Key: HKEY_LOCAL_MACHINE\Software\Microsoft\SoftGrid\4.5\Sequencer\Configuration\
  • Value: FileManifest
  • Data Type: REG_DWORD
  • Data: 1

SFTrpc.txt: This is the log file created by the monitoring element SFTRPC.EXE and in addition to also capturing process startup and shutdowb, will also contain verbose diagnostic information about each monitored shortcut.

In addition to the sequencer logs, you can also leverage process monitor (http://technet.microsoft.com/en-us/sysinternals/bb896645.aspx) and verbose MSI logging (https://madvirtualizer.wordpress.com/2014/03/17/enabling-advanced-windows-installer-logging/) if you encounter errors within the application during sequencing.

Categories: App-V Tags: , , ,

App-V 4.6: So your Client Installation or Upgrade has failed: How do you break down the Logging?

September 9, 2014 1 comment

App-V 4.6 is still very prevalent out there and will be for a while. With the releases of Windows 8 and Windows 8.1 brought additional service packs for the App-V 4.6 client which means upgrades and/new installs for newer operating systems. Several weeks back on this blog, I went over how to enable advanced MSI logging for troubleshooting MSI installs and upgrades (Remember VOICEWARMUP – https://madvirtualizer.wordpress.com/2014/03/17/enabling-advanced-windows-installer-logging/ ) but I would like to now address some follow up emails I received. Admins would like more specific information on how to go through and read that potentially enormous log in order to find out what is failing where and when.

I would never advising reading a verbose MSI installation log from start to finish especially when dealing with potentially asynchronous actions. MSI logs also have an excessive amount of rollback information incorporated into the log upon failed installations. The seasoned IT Pro often looks for specific keywords such as “error” and “failed” and that can be misleading as not all logs generate these types of messages. In addition, searching on the string “error” can also yield false positives as well.

When I am looking at Verbose MSI logs of App-V 4.6 client installs, I usually analyze the log by doing the following:

  • Searching for the error string generated in the App-V Installer User Interface with quotations.
  • Searching using the string “1603.” See if it indicates that a custom action has failed.
  • Searching using the string “Value 3.” This will indicate an install error. This can also help to identify the custom action failure.
  • Searching for string “IsInBadState()” can also be helpful if there is an issue with a failed driver install. This is especially useful in troubleshooting an upgrade. Usually when this occurs, you usually need to delete the driver configuration and state of the specific App-V file system driver specified in order to reattempt the upgrade.

Finally if you need to walk through the App-V custom actions, you can do so by searching by the strings ‘SWI” or “SGC” as all of the App-V custom actions begin with these prefixes.

CustomAction SWI41sp1UpgradeFix returned actual error code 1603

You can walk through the logging of each key App-V custom action. Once you’ve identified what custom action failed, you can then use the following reference to find out specifically what was being attempted with the custom action here: http://support.microsoft.com/kb/2465574. Even though it specifies SP1, it is still valid and helpful for SP2 and SP3. For example the action reference above would be:

Installer :             Client

CA:                       SWI41sp1UpgradeFix

Method name:     SWI41sp1UpgradeFix

Description:         Modifies an installed instance of the Softgrid 4.1SP1/4.2 client application to correctly support upgrading to a later version.

You can then dive deeper into the timeline of the action and align it with a more deep logging utility such as Process Monitor.

Categories: App-V Tags: , , ,

App-V 4 Application Troubleshooting: Breaking Down Virtualization Issues Pt. II

September 2, 2014 1 comment

App-V 4.5 and 4.6 virtualize at the user mode layer. One of the most identifying factors of seeing that a thread stack is that of a virtualized application is the presence of the SFTLDR.DLL file. This is what is injected into every process a virtual application will create. This file is responsible for ensuring proper redirections and translations necessary to make virtualization function properly by:

  • File changes to included virtual directory and file paths are redirected to the VFS
  • Registry changes hooked and redirected to the virtual registry
  • The spoofing of objects
  • The spoofing of COM GUIDS

In addition to the common troubleshooting methods such as disabling local interaction and disabling object spoofing, you can also take things further by disabling various virtualization components using the System Guard Overrides in App-V 4.x. These are not meant to be solutions but isolation factors in case you need to modify mappings. Many of these can be set at the registry level affecting the entire client or at the application level using the OSD file.

All of the registry values mentioned are located under HKLM\SOFTWARE\Microsoft\SoftGrid\4.5\SystemGuard\Overrides:

Disabling Virtual Services

You can disable virtual services on a per package basis by adding in the <VIRTUAL_SERVICES_DISABLED> tag under the <POLICY> XML element in the OSD file. You can disable the subsystem for the entire client by going adding the DisableVirtualServices DWORD value with a value of 1. If this is enabled, the sftldr.dll will not hook the service APIs.

Disabling the Virtual Registry

You can disable the virtual registry on a per package basis by adding in the <VIRTUAL_REGISTRY_DISABLED> tag under the <POLICY> XML element in the OSD file. You can disable the subsystem for the entire client by going adding the DisableVreg DWORD value with a value of 1. If this is enabled, the sftldr.dll will not hook the virtual registry calls.

 Disabling the Virtual File System

You can disable the virtual file system on a per package basis by adding in the <VIRTUAL_FILE_SYSTEM_DISABLED> tag under the <POLICY> XML element in the OSD file. You can disable the subsystem for the entire client by going adding the DisableVFS DWORD value with a value of 1. If this is enabled, the sftldr.dll will not hook virtual file system calls.

Finally, if you are really interested in going to the extreme . . .

You can disable ALL hooking. Can be useful when you are launching an application that is locally installed but still being brought into the virtual bubble. This allows you to turn it on and off if troubleshooting odd behavior. This is done at the client level which is why it is definitely only a troubleshooting option. You can disable hooking by adding in the registry value DisableSftldr DWORD value with a value of 42. Why 42? Well because that is the answer to everything in the universe. This basically makes the sftldr.dll (which is the primary hook DLL) dormant. MAVINJECT32 (or MAVINJECT64 if a 64-bit system) will still inject this DLL though. It will just remain dormant. This is a last resort.



Categories: App-V Tags: , , , , ,

Get every new post delivered to your Inbox.

Join 29 other followers