Archive

Posts Tagged ‘dotnet’

On Debugging Virtual Applications: Part 3 – Situations where a Debugger is most needed for Virtual Applications

February 23, 2016 1 comment

In parts 1 and 2 of this series, I’ve covered some of the basic fundamentals of the concept of debugging compiled software code which, for the most part, has been a black box for many working with virtual applications. In part 3, I would like to now move the discussion towards those situations that warrant further debugging analysis. Note that for this topic, I will be limiting the scope to specifically user mode applications as application virtualization products like App-V are pretty much limited to these.

Native Code vs. Managed Code

Managed Code refers to computer source code that will only executive under the management of a special runtime module. Technically managed code refers to any code that requires a specific run-time but even native code may be dependent on underlying libraries contained within middleware (i.e. VC runtimes, java.) Microsoft coined the term “Managed Code” to describe code that requires the .NET Common Language Runtime (CLR.) It is written in .NET languages such as C# or Visual Basic .NET. Managed code’s memory, reference counting, and garbage collection is done by the underlying run-time – hence the term managed.

Native code refers to the classic Windows code that has been written in C, C++, Visual basic, and others in rare cases. While native code can easily be debugged through classic debuggers, managed code requires additional extensions. Native code is compiled to work directly with the underlying operating system while managed code is precompiled and then processed by the JIT (Just-in-time) .NET compiler and converted to native code at run-time allowing for greater portability. Native code was actually compiled for the operating system and hardware architecture. Porting this code to other architectures requires re-compiling.

Situation 1: Application Crashes

When an application crashes, it stops – immediately. Data is lost. It’s dead. It usually happens because the application has an encountered an unexpected exception and the potential for the exception has not been anticipated therefore the exception is not able to be properly handled within the application. Because of this, a special program within the operating system must deal with the aftermath. How the crash is intercepted and handled will depend on specific configurations within the operating system. How much data (active memory snapshots) that are actually saved for diagnostics also varies on how things are configured. Whether the module is native code or managed code will also potentially affect how it is handled.

Native Code Unexpected ErrorsExceptions

Depending on the operating system, when a normal application crashes, you will be interrupted with a dialog telling you that the application has stopped working and will be closed. Older users of windows may recognize the older 16-bit Windows blue screen that stated this. In recent versions of Windows (Windows Vista) and later WER (Windows Error Reporting) kicks in and cleans up the virtual address space and makes a record of the information even collecting a basic mini-dump. You will see a message like the one below.


WER is the modern version of an older program which would collect basic diagnostic data and a mini-dump. This program was called Dr. Watson and has been around since the dawn of Windows. WER takes this a step further and can report the crash to Microsoft. This is especially helpful if the program crashing is a Microsoft application. Other ISV’s often incorporate JIT’s (just-in-Time) debuggers to collect information for their diagnostic purposes as well.

Developers, Support Technicians, and Diagnostic Engineers will often register an alternative JIT debugger to intercept an application crash. Often times this is so they can do live debugging, or substitute their preferred collection tool for collecting user mode dump files for further debugging. For example, if you have either Windows Debugger (WINDBG) installed or Visual Studio, you will see this instead:

 

Note that in the above example, the debugger is able to intercept at the exception point (just in time) so there could even be further live/step-through debugging done.

The Windows Error Reporting model (https://msdn.microsoft.com/en-us/library/windows/desktop/bb513641(v=vs.85).aspx) is really a newer generation of what was formerly Dr. Waston in that it integrates with Microsoft for diagnostic and troubleshooting telemetry data called Software Quality Management (or SQM) data. WER defaults to collect SQM data and minidumps, but can easily be managed centrally, especially through GPOs. By default, minimal data is collected in the realm of user mode dumps and the dump files go into a special queue for upload to microsoft or potential another central repository.

 

You can configure WER to generate bigger dumps that can be more useful when debugging application crashes. You must provide some additional configuration to WER. Since I usually use this as a temporary measure (where I cannot install additional tools) I usually remove these once I have the data that I need.

First, create a folder where the dumps will be stored out of band from the WER queue (i.e. C:dumps.) Then import the below text as a .reg file:

Windows Registry Editor Version 5.00

 [HKEY_LOCAL_MACHINESOFTWAREMicrosoftWindowsWindows Error ReportingLocalDumps]

"DumpFolder"=hex(2):63,00,3a,00,5c,00,64,00,75,00,6d,00,70,00,73,00,00,00

"DumpCount"=dword:0000000a

"DumpType"=dword:00000002

Alternatively, you could also batch this out as well:

mkdir C:Dumps

reg add "HKLMSOFTWAREMicrosoftWindowsWindows Error ReportingLocalDumps" /f

reg add "HKLMSOFTWAREMicrosoftWindowsWindows Error ReportingLocalDumps" /v DumpType /t REG_DWORD /d 2 /f

reg add "HKLMSOFTWAREMicrosoftWindowsWindows Error ReportingLocalDumps" /v DumpCount /t REG_DWORD /d 100 /f

reg add "HKLMSOFTWAREMicrosoftWindowsWindows Error ReportingLocalDumps" /v DumpFolder /t REG_EXPAND_SZ /d C:Dumps /f


Double-click the .reg file to import it into the registry. If you wish to change the path where dumps are stored, you can edit the following key to reflect the preferred location:   

HKEY_LOCAL_MACHINESOFTWAREMicrosoftWindowsWindows Error ReportingLocalDumpsDumpFolder

Even when I use the above, I still have trouble collecting good dumps when dealing with virtual application crashes. Not to mention, in my case I often have Visual Studio and the Windows Debugger installed together. I prefer (and strongly) recommend that you use the Sysinternals tool PROCUMP (https://technet.microsoft.com/en-us/sysinternals/dd996900.aspx) to intercept application crashes to collect dumps for post-mortem debugging.

One great feature of ProcDump is that you can also configure it to override WER (and other debuggers) as the default debugger by registering it with the JIT Application Exception Debugger key (AEDebug) and also control where the user mode dumps are stored. Once that happens, you will then see ProcDump intercept and collect dumps by default when application crashes occur.

 

Managed Code Unhandled Exceptions

For .NET applications that encounter unhandled exceptions within managed code, you get a different experience. The runtime incorporates elements for this very type of thing giving the user a different option than what they would normally get with WER or the alternative registered with AEDebug.

 

Exception Handling

Since we have discussed the fact that many application crashes come from unhandled exceptions – often access violations or what it used to referred to – GP faults – let’s discuss actual exception handling and why it is important. Exception handling is literally what it means, a method through code of anticipating and controlling how these exceptions will be handled through special processes or something relatively easy to track (often through interface messages and/or logging.) One of the many reasons we have WER (and historically Dr. Watson) in Windows was to ensure that the operating system would have a last resort method of handling these issues. Every exception cannot be caught with programming and often, the usefulness of how it is handled will vary from developer to developer. ūüôā

In early Windows, primarily the early days of Win32, one of the touted features was the capability of being able to leverage SHE (Structured Exception handling) in the Win32 API. I first read and learned about SEH again, from master Matt Pietrick back in the NT 4 days reading MSJ (the earlier version of MSDN magazine. To my excitement and amazement, the article is STLL available online and worth a read to this day (https://www.microsoft.com/msj/0197/Exception/Exception.aspx.) Exception handling leverages trap and try/catch statements in most programming languages. Here is an implementation in its simplest form using native code (C++)

 

               __try

               {

                              *lpstr = '';         

               }

               __except (GetExceptionCode() == EXCEPTION_ACCESS_VIOLATION ?

                                                            EXCEPTION_EXECUTE_HANDLER :

                                                            EXCEPTION_CONTINUE_SEARCH)

               {

               MessageBox(NULL,"EXCEPTION_ACCESS_VIOLATION","CRAPPYAPP.EXE",MB_OK);

               g.fHandledViolation = FALSE;                     

               }

 

This exception would yield the following message but still allow the program to continue.

 

 

How exception levels differ

Whether the exception comes in the form of a popup window or is just simply logged into a file or to ETW, it is up to the programmer to ensure this. It is also up to the developer as to the level of information that will be collected. To use a managed code example, the following exception handler will sufficiently catch the exception:

 

            try

            {

                 .

                 .

                  .

            }

            catch (Exception e)

            {

                Console.WriteLine("ERROR: Unexpected error.");

                Console.WriteLine("DETAILS: {0}", e.Message);

                return 3;

            }

 

But the example below would serve even better as it would give more diagnostic information for troubleshooting:

 

       try

            {

                .

                .

                .

            }

            catch (Exception e)

            {

                Console.WriteLine("ERROR: Unexpected error.");

                Console.WriteLine("EXECPTION TYPE: {0}", e.GetType().Name);

                Console.WriteLine("DETAILS: {0}", e.Message);

                Console.WriteLine("STACKTRACE: {0}", e.StackTrace);

                return 3;

            }

 

This will log more detailed information including a trace of the current thread stack. Not sure what “Stack Trace” means? Do not worry, that will be covered in a later post.

 

Situation 2: Hangs

When an application hangs, you can break into the process using a debugger or you can use a debugging utility to save the existing data in memory into a dump file for post-mortem analysis. Windows has a similar mechanism for user mode hangs as it does for application crashes using WER. With WER, you are often given options to either wait, close the program and destroy the hung memory space, or capture a small dump and upload it to Microsoft to see if it is possibly a known issue with even perhaps a known solution.

 

In the case of virtual application, the data collected can often be unreliable so in place of using WER, I will use WinDBG or ProcDump to collect a hang dump for further analysis.

 

Situation 3: Heap Corruption

Another type of issue that can happen often with virtual applications is Heap Corruption. The Process heap is a special section of usable memory that is initially created when a process is created. The size is determined the linking portion of the compiler. Subsequent heaps may be created allocated, deallocated, and destroyed throughout the life of a process. With managed code, the heap is managed by the runtime (.NET CLR) to allocate objects and to provide its memory services like for example the Garbage Collector.

 

Heap corruption usually occurs when a process allocates a block of heap memory of a given size and then a thread writes to and/or frees memory addresses beyond the requested size of the heap block. Heap corruption can also occur when a process writes to block of memory that has already been freed.  Sometimes it is pretty obvious and the debugger is able to quickly assess it

0xc0000374 – A heap has been corrupted.

Usually during a heap API operation on the thread stack:

 

77b888c8 77b15c49 ntdll!RtlpFreeHeap+0x59c49

77b888cc 77abb4c8 ntdll!RtlFreeHeap+0x268

77b888d0 76eecb60 kernelbase!GlobalFree+0xc0

77b888d4 7796cd18 kernel32!GlobalFreeStub+0x28

77b888d8 000a1871 shttyapp!CorruptDatHeap+0xd1

77b888dc 000a2525 shttyapp!WndProc+0x345

77b888e0 777d84f3 user32!_InternalCallWinProc+0x2b

77b888e4 777b6c40 user32!UserCallWinProcCheckWow+0x1f0

77b888e8 777b6541 user32!DispatchMessageWorker+0x231

77b888ec 777d6f30 user32!DispatchMessageA+0x10

77b888f0 000a1242 shttyapp!WinMain+0x152

77b888f4 000a2c19 shttyapp!__tmainCRTStartup+0x11a

77b888f8 779638f4 kernel32!BaseThreadInitThunk+0x24

77b888fc 77ae5e13 ntdll!__RtlUserThreadStart+0x2f

77b88900 77ae5dde ntdll!_RtlUserThreadStart+0x1b

 

Most of the time, you have to incorporate more specific tools to track it down. Speaking of that:

 

Next up: more on the tools!

Advertisements

App-V: More on that Failed Office Add-in

August 1, 2013 3 comments

Continuing from my previous post a few days ago, let’s talk more about application errors when Office add-ins are virtualized. It is pretty easy to resolve most add-in issues with the right methodology. One of the first steps in my methodology is to first determine if the issue is related to the add-in loading or a general application error occurring after the add-in has loaded.

“First Launch Only” Issue or “Every Launch” Issue?

If you have isolated the problem to a load issue, now determine if the problem occurs on first application launch only or on every launch. If it is a first launch issue, the issue resides most likely with the package. If it is on every launch, the issue could be easily in the package or within the configuration and/or user state of the client machine. As mentioned in my previous blog post on the subject, the first place you want to check is the LoadBehavior key in the registry. After that I follow a basic process where on first launch issues, I eliminate LoadBehavior as an issue first, then I determine if registry opacity might be a factor. Registry opacity is where the virtual registry key is configured to either “merge” with the local key or “override” the local key.

If it is an issue that is occurring on all launches, I first eliminate user state as a potential culprit whether it is an incorrect configuration or corrupted configuration. I’ll repair the package first as well as eliminate any other elements related to user state management before I start actually digging into the package configuration.

If the add-in is still failing to load or function properly after loading then you may have to look to ensuring that you have done the basics of planning prior to virtualization. Time to back track a little:

Have We Verified Bitness?

Do we have matching bitness – (i.e. a 64-bit Office application requires a 64-bit COM add-in.) App-V does not change the fact that Office requires the add-ins to match the bitness of the office application. In other words, if you are using a 64-bit version of Office, you do not want to be loading 32-bit add-ins

Are you sure we aren’t missing requirements somewhere?

Have you confirmed that any pre-requisites or middleware required for the add-in have been either included in the add-in package or are installed locally on the client machine? I have been burned by this before. This is the most common reason why an add-in may actually “load” according to Office, but may actually still fail (by either crashing or giving a strange error.) Sometimes the application dependency may appear to be installed but may actually be partially installed. Let’s look at an example:

The Case of the Crashing Add-in

Now here is an interesting case: Let’s say the Office Add-in loads according to the Event Viewer, yet the Add-in still errors out. What is the best way to go about troubleshooting it? In the example below, a component called Cisco ViewMail for Outlook has failed to load in Outlook 2010. We initially get the following error:

‘VMOAddIn’ has fired an exception. Click the ‘Details’ button to see the detailed information about the error.


This error will contain details including a stack trace. This information may prove to be valuable later. This error is exactly what it says. It is not a load error – it is a handled exception or a crash. In this example, it is then followed by a more user-friendly message:

Cisco ViewMail encountered an error while starting and was not able to recover. ViewMail functionality is not available.


So when you get this error, the first thing you will want to do is take a look at the details of the crash. Based upon the error screen – this appears to be a managed code exception (.NET.) When you get these errors – look at these key elements:

Assembly Codebase: file:///Q:/Microsoft_Outlook_2010_V1/VFS/CSIDL_PROGRAM_FILES/Cisco Systems/ViewMail for Outlook/VMOLibrary.DLL

Assembly Full Name:    VMOLibrary, Version=9.0.2.1, Culture=neutral, PublicKeyToken=null

Assembly Version:      9.0.2.1

Exception Source:      VMOLibrary

Exception Type:  System.IO.FileNotFoundException

Exception Message:     Could not load file or assembly ‘Microsoft.DirectX.DirectSound, Version=1.0.2902.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35’ or one of its dependencies. The system cannot find the file specified.

So in the case of the above error, we see something very straight forward: A missing file – Microsoft.DirectX.DirectSound. Looking at the list of installed programs and updates, I see that the DirectX components are indeed installed.

Know the Add-in

Before you troubleshoot further – it would probably be a good idea to have a known good installation (where the add-in is installed and working normally on a native installation.) I usually compare Process Monitor captures of the known good with a trace of the reproduced error. Before I even do this, though, I need to know what I will be looking for. In most cases with add-ins, there will be a loading of the add-in (usually a DLL) by a base executable (in this case, the Office application is Outlook so the EXE will be OUTLOOK.EXE.)

What are the EXE’s and DLL’s for ViewMail?

To get this information, I can either look at a local installation or I can open a command prompt in the bubble and browse the VFS. Once I know what these DLL’s are and where they are located, I can then proceed to weed through a Process Monitor capture of the failure. I first add a PATH filter in ProcMon using the CONTAINS operand and the use the keyword “ViewMail.” This takes my filter down from nearly 30,000 events down to 1,217 items. Next I look to add another filter. Since the error “Could not load file or assembly ‘Microsoft.DirectX.DirectSound” was in regards to a file or assembly, I now add an additional filter for the
operation “LoadImage.” This knocked the entries down to 29 events but EVERYONE OF THEM SHOWED SUCCESS!

Does this mean we are taking crazy pills? No, this just means you have eliminated the simple issue of a failure to load a DLL. The “file” aspect of that error has been eliminated. We are now simply chasing down a .NET assembly. So now we need to hone in on the codebase of the error – narrowing the context:

Assembly Codebase:    file:///Q:/Microsoft_Outlook_2010_V1/VFS/CSIDL_PROGRAM_FILES/Cisco Systems/ViewMail for Outlook/VMOLibrary.DLL

I’m going to go back to the Process Monitor capture and return to the point where the successful LoadImage operation of VMOLibrary.DLL occurred. I reset the filter from this point (making sure this selection remains highlighted.) Anything before this point is irrelevant. I begin to start walking down the trace, systematically eliminating irrelevant processes (right-clicking them and selecting “exclude” – BUT –  BE SURE YOU KNOW WHAT YOU ARE EXCLUDING IS IRRELEVANT.)

I walk through the loading of the file VMOLibrary.DLL until the result is END_OF_FILE which means the file has been completely loaded.  Continuing on in the trace, I notice it is now walking the .NET assembly registrations and finding begins to hit PATH_NOT_FOUND error and events start to occur where it begins to write to an error log. I take note of the assembly name in the error: ‘Microsoft.DirectX.DirectSound.’ It is as if it thinks that the DirectSound component of DirectX is not installed – but I verified it was installed prior to troubleshooting. However, I go to the C:WindowsAssemblies folder and sure enough, it is not there.

How did I know to look there?

Activity to this special folder (C:Windows Assemblies) involves looking for actual activity going to the .NET Global Assembly Cache (GAC.) When looking at Process Monitor traces, all activity  in the Global Assembly Cache is logged by direct path in ProcMon as

  • C:WINNTassemblyGAC_MSIL
  • C:WINNTassemblyGAC
  • C:WINNTassemblyGAC_32

So the problem we had was that while DirectX was installed, the proper .NET DirectX assemblies were not registered in the GAC. Now if this ever happens you want to make sure the binaries for the DirectX managed Code components are there. In this case – the files were actually in C:WINNTMicrosoft.NETDirectX
for Managed Code1.0.2902.0 BUT they were not actually registered to the GAC. If the files were not even on the machine – a reinstallation of DirectX is in order (plus updates.) If the files are there but not registered in the global assembly cache then you could leverage REGASM:

REGASM.exe – http://msdn.microsoft.com/en-us/library/tzat5yw6.aspx

And register those assemblies manually – my recommendation would be to simplify things and just reinstall DirectX. Then you will see the assemblies properly registered.

 

 

NOTE: Unregistered assemblies may not show up in a DXDiag report but the .NET assemblies inventory for DirectX is part of the output as demonstrated with a before and after comparison using BeyondCompare.

App-V: Error Message when trying to launch a Virtualized Instance of Visual Studio 2010

January 16, 2012 Leave a comment

Microsoft App-V is the only application virtualization product capable of virtualizing Visual Studio 2010. However, you may receive the following error message when attempting to launch the primary development environment (DEVENV.EXE) within a Visual Studio 2010 package virtualized with Microsoft App-V:

The ‘Environment Package Window Management’ package did not load correctly.

The problem may have been caused by a configuration change or by the installation of another extension. You can get more information by running the application together with the /log parameter on the command line, and then examining the file ‘C:users<USERNAME>AppDataRoamingMicrosoftVisualStudio10.0ActivityLog.xml’.

Continue to show this error message?

 This is nothing to be alramed about. This can happen if one or more of the following is missing/configured incorrectly on the App-V Client where the Visual Studio package is being deployed:

– The Microsoft .NET Framework 4 Full Profile
– The KB 2468871 Update
– Interference from the Windows Presentation Foundation Font Cache Service.

 

To resolve this issue, be sure to complete the following steps on all App-V Clients to which the virtualized package of Visual Studio 2010 will be deployed.

1. Install Microsoft .NET Framework 4 Full Profile.  Using Microsoft Update, install all updates for Windows and .NET Framework.  The full download for the .NET Framework 4 Full Profile can be found here:

http://www.microsoft.com/download/en/details.aspx?id=17718

2. Make sure that the following update was installed by Microsoft Update.  If not, install it manually from the following location: http://www.microsoft.com/download/en/details.aspx?id=3556

3. Run services.msc and disable the Windows Presentation Foundation Font Cache service.

 

MED-V 1.0: MED-V service is fails to start with the following event in the System Log – Event ID 0


You may encounter problems with a Windows XP host when the MED-V service  is trying to start. It will fail to start if any of the critical .NET components are missing or corrupt with the following event in the System Log:

Source: MEDVService
EventID: 0
Description:
Cannot start service. System.IO.FileNotFoundException: Could not load file or assembly ‘System.Data, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089’ or one of its dependencies. The system cannot find the file specified.
 
File Name : ‘System.Data, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089’
   at: Kidaro.Foundation.SafeXmlSerializer.Create(Type type)
   at: Kidaro.Foundation.Config.XmlConfiguration.get_Item(String path)
   at: Kidaro.Foundation.Config.CompositeConfiguration.get_Item(String path)
   at: Kidaro.Foundation.Config.Configuration.Get(String path)
   at: Kidaro.Policy.CurrentUserSecurityPolicy.get_ClientPolicy()
   at: Kidaro.Policy.Initializers.ServicePolicyInitializer.ReportPolicyChange()
   at: Kidaro.Policy.Initializers.ServicePolicyInitializer.InitializeComponents()
   at: Kidaro.Features.BaseFeature`1.Initialize()
   at: Kidaro.Features.FeatureManager.Initialize(IFeature feature)
   at: Kidaro.Service.KidaroService.OnStart(String[] args)
¬†¬† at: System.ServiceProcess.ServiceBase.ServiceQueuedMainC…

Again, this problem is most likely happening because one of the following:

1. The file System.Data.dll has been moved or deleted (or one of its dependent assemblies)

2. The proper version of .NET was never installed (although the MED-V install should check for that)

3. The proper version of .NET was uninstalled after MED-V was installed

To resolve this issue, you should be able to run the .NET Framework 2.0 configuration utility to verify the existence of the System.Data.dll file. It is found in Control Panel under administrative tools.

After that is confirmed check the local directory:

C:\Windows\Microsoft.NET\Framework\v2.0.50727>dir System.Data.dll
 Volume in drive C has no label.
 Volume Serial Number is 36E4-4246

 Directory of C:\Windows\Microsoft.NET\Framework\v2.0.50727

06/10/2009  03:23 PM         2,933,248 System.Data.dll
               1 File(s)      2,933,248 bytes

– You can also use the .NET Verifier to confirm that the .NET 2.0 runtimes are correctly installed and registered.

More information about the .NET Verifier can be found here:

http://blogs.msdn.com/b/astebner/archive/2008/10/13/8999004.aspx

Categories: MED-V Tags: , , ,