Blogs

Monitoring and troubleshooting PVS 7.6

By 10 september 2015 No Comments

Monitoring and troubleshooting PVS 7.6

I am currently engaged in a project for a customer which involves Citrix Provisioning Services. So during this project we encountered some things that needed to be troubleshooted. Mainly the boot process was an issue

I would like to break this down into of steps and how to troubleshoot

To understand the boot process lets see how a normal VM or PC will boot

  1. The machine starts up
  2. The BIOS starts the POST Process
  3. The BIOS the identifies a boot device
  4. Next step is that the operating system is loaded

Now how does this compare to a target device which is booted.

  1. The machine starts up
  2. The BIOS starts the POST Process
  3. PXE Boot:
  4. Get IP Address
  5. Get Network Boot Program Name / Location
  6. Get Network Boot Program
  7. Contact the TFTP Server
  8. Download Bootstrap
  9. Load Operating System

 

Detailed information about the boot process can be found in this nice diagram;

http://support.citrix.com/article/CTX136378

So the PVS Boot Process is build in five steps

  1. IP Acquisition
  2. Bootstrap Download
  3. PVS Logon Process
  4. Single Read Mode
  5. BNISTACK / MIO Read Mode

 

Troubleshooting IP Acquisition

The purpose of this portion of the boot process if to give the target a way to communicate on the network and let it know where it can get the network boot program

The first step in the Provisioning Services Boot Process is for the PVS target device to acquire an IP address.

This can be accomplished either statically or dynamically

Dynamic

  • Booting from the NIC using DHCP
  • Booting from a PVS Boot Device Management (BDM) disk or .iso that is configured to use DHCP

Static

  • Create a boot disk using the PVS Boot Device Management (BDM) with a specified static IP address

 

Network tracing is the best method to be used for DHCP/IP acquisition issues. We can use WireShark https://www.wireshark.org/download.html on the PVS server. This will give you a pretty nice picture what is going on server side.

PVS_001

 

Capture Guidelines

  • If possible capture both server-side and target side trace
  • If possible capture a good a bad trace an compare both
  • Get as much IP information as possible

Target Side Capture Methods

 

  • Hyper-V

 

http://blogs.technet.com/b/mspfe/archive/2012/11/29/virtual-machine-network-monitoring-the-easy-way-with-hyper-v-in-windows-server-2012.aspx

 

  • VMware

 

http://kb.vmware.com/selfservice/microsites/search.do?cmd=displayKC&externalId=1004099

 

  • XenServer

 

http://support.citrix.com/article/CTX120869

 

 

  • Physical Server

 

Port Mirroring, Hubbing out, using a Tap

 

 

Troubleshooting Bootstrap Download

 

Once the target device obtains an IP address, it goes through the phase of downloading the bootstrap file.

Using the address configured in DHCP option 66, the PVS target device contacts the TFTP server (or boot server) and requests the boot file that is configured in DHCP option 67. This bootstrap file is called ARDBP32.bin.

Using BDM a PVS target will download TSBBDM.bin over port 6969

Consider the following options when creating the Bootstrap File.

PVS_002

“Interups Safe Mode” should only be used during troubleshooting.

In order to see the information to troubleshoot this in particular, you should also use WireShark and focus on DHCP and TFTP, ARP request’s.

PVS Logon Process Troubleshooting.

After the target device gets an IP address and downloads the bootstrap file it proceeds to login to a PVS server to start streaming the vDisk image.

The list of login servers is provided in the bootstrap file that is downloaded from the TFTP server

the steps involved with the login process;

  1. Login Port Request
  2. Present MAC address
  3. Load Balancing
  4. Transfer to I/O

 

You can also see this in the event log of the PVS server.

PVS_003

The windows eventlog will give enough information at this point. For instance;

  • Incorrect Boot strap information
  • Stream Service is not started
  • Target Device does not exist in the database
  • Or UDP ports are blocked.

 

Troubleshooting Single Read Mode Streaming

After a target device has logged into PVS and has been directed to a PVS server for streaming, the bootstrap file (ARDBP32.bin, TSBBDM.bin or TSBOROM.bin) will then intercept any requests.

The bootstrap directs any block level requests for data as to the PVS server as single read requests

The single read mode procedure will continue until the Microsoft Windows Operating System starts loading drivers and BNISTACK is successfully loaded

After the login the target will continue to process traffic using the UNDI driver.  (Universal Network Driver Interface and will send one to one packets.

During this time the target will display the windows logo.  Low level drivers are being loaded here

PVS_004

 

If the server does not respond to a request within 5 seconds the target device will request the same data again. This will happen a maximum of 3 time before the target will attempt to reconnect and login to the PVS server again

This is generally the longest portion of the boot process due to the amount of data that has to be transferred

Response delays here contribute the most to long boot times

Consider the following;

  • Large Send Offload
  • Network instability
  • Conflicts with UNDI Driver

http://support.citrix.com/article/CTX117374

 

Troubleshooting BNISTACK / MIO Read Mode

MIO (Multiple Input/Output) mode begins when BNISTACK is loaded by the operating system. BNISTACK is loaded into memory and takes over for the bootstrap, it manages the communication between the target device and the PVS server. At this point the following information is exchanged:

  • Image Mode
  • Active Directory Password Management Option
  • Write Cache Type and Size
  • Client Name
  • Licensing

BNISTACK uses multiple threads to communicate with the PVS server

MIO uses a single request -> multiple response structure

PVS_005

Once UNDI hands off to BNIStack (PVS Target Device’s Main Filter Driver) the target will begin to request larger amounts of data.  This activity will continue up to the windows login screen.

Consider the following;

  • Firewall/Antivirus Services
  • AV Updates are being loaded at this point
  • Domain Profile Creation.
  • Filter Driver conflicts

http://support.citrix.com/article/CTX124185

 

Logs

Provisioning Services has been providing the ability to capture logs since the time of PVS version 4.5.  Using the Log4cxx framework, PVS logging has been able to provide information about the goings on of the PVS console, the Soap Service, and the Stream Service.  These logs have provided valuable historical data when trying to troubleshoot issues.

There have been many improvements to the logging capability and as the feature set for PVS grew Citrix introduced new logs as well as refined some existing logs.  With the introduction of PVS 7.0 the logging format has been changed from Log4cxx to CDF.  Citrix has long used CDF as a mainstay of our logging mechanism across many products

 

This means a couple of things for PVS.  Since CDF tracing is reactive and not enabled by default there is no built in historical logging in PVS 7.x

CDFMonitor

With the introduction of Provisioning Services 7.0, the Provisioning Services logging mechanism has changed from persistent log files to CDF tracing. Provisioning Services no longer logs streaming event to a log file. You must configure CDF tracing to capture persistent logging for determining the root cause of a problem.

Procedure

To enable continuous CDF tracing in Provisioning Services 7.0, complete the following procedure:

  1. Download CDFMonitor.
  1. Download the Provisioning Services specific configuration file;

http://support.citrix.com/content/dam/supportWS/kA260000000TalyCAC/ctx138698.zip

  1. Extract the files to the C:cdfmonitor folder.
  1. Replace the CDFMonitor.exe.config that comes with CDFMonitor with the one downloaded

PVS_006

  1. Open a command line interface and open the C:cdfmonitor folder.
  2. Run the following command: cdfmonitor.exe /installservice

PVS_007

 

CDFMonitor runs as a service that persists even after the server restarts. Continuous CDF trace is then captured on all modules. The trace file will grow to 100 MB in size and will rotate between five files. The moment the CDFMonitor service is restarted, a new CDF trace file is created.

Gathering Log Files

To collect the log files, complete the following procedure:

Stop the CDFMonitor service from the Services applet.

Copy trace files from the c:windowscdfmonitor folder.

PVS_008

Start the CDFMonitor service from the Services applet, as shown in the following screen shot:

PVS_009

So at this time we have a trace file with the *.etl extension

 

CDF Analyzer

It is possible to open this in CDFAnalyzer

http://support.citrix.com/servlet/KbServlet/download/21460-102-720206/CDFAnalyzer.zip

Download and just run the tool, there is no need to install.

PVS_010

 

When you open the tool and load one of the ETL files that was created with CDF monitor you can see that the format is unknown

PVS_011

That means, the we didn’t have “symbols” to solve that GUID strings. For downloading that files we will use another Citrix tool – CDFMonitor

Start the command line, go to CDFMonitor directory and type:

CDFMonitor.exe /downloadtmfs

 

PVS_012

You need to provide you’re username and then continue with the download

Now, when we have all the TMF files we can configure CDFAnalyzer to use them. From menu choose “Tools” and the “Properties” and put the whole path where TMF files are located.

PVS_013

 

So at this point start CDFAnlyzer again and load the ETL file again..

PVS_014

 

Citrix DataCollector

Download the PVSData Collector from the following link

http://support.citrix.com/article/CTX136079

Just extract the zip file, there is no need to install the software..

Select the application.

PVS_015

 

The application will start automatically

PVS_016

 

At a certain point administrator credentials needs to be filled in.

PVS_017

The PVSDatacollector wil continue to collect the information and provide the progress.

PVS_018

the PVSDataCollector will complete providing the name of the zip file when the collection and compression is complete. Instructions are provided for where to upload the file.

PVS_019

The following is an example what is in the trace file

PVS_020

Once the data is captured you can upload this to Citrix Insight Services

Proceed to the following URL

https://taas.citrix.com

PVS_021

Login with you’re MyCitrix Credentials and click on Lets Go to continue

Click on the Upload Button to uplad the zip file you have created with PVSDatacollector

PVS_022

Enter the apropiate information and continue with the Upload File button

PVS_024

The zip file is being uploaded

After uploading the file, the file is analyzed and the results are displayed

PVS_025

CDFControl

CDFControl is an event tracing controller/consumer, geared towards capturing Citrix Diagnostic Facility (CDF) trace messages that are output from the various Citrix tracing providers. Now in this version, CDFControl offers new capabilities, such as inserting trace, parsing filter support, and more.

 

http://support.citrix.com/filedownload/CTX111961/CDFControl.zip

 

 

Reference materials used: Citrix.com, Support.citrix.com and the E-Docs website.