Monitoring and troubleshooting PVS 7.6
I am currently engaged in a project for a customer which involves Citrix Provisioning Services. So during this project we encountered some things that needed to be troubleshooted. Mainly the boot process was an issue
I would like to break this down into of steps and how to troubleshoot
To understand the boot process lets see how a normal VM or PC will boot
- The machine starts up
- The BIOS starts the POST Process
- The BIOS the identifies a boot device
- Next step is that the operating system is loaded
Now how does this compare to a target device which is booted.
- The machine starts up
- The BIOS starts the POST Process
- PXE Boot:
- Get IP Address
- Get Network Boot Program Name / Location
- Get Network Boot Program
- Contact the TFTP Server
- Download Bootstrap
- Load Operating System
Detailed information about the boot process can be found in this nice diagram;
So the PVS Boot Process is build in five steps
- IP Acquisition
- Bootstrap Download
- PVS Logon Process
- Single Read Mode
- BNISTACK / MIO Read Mode
Troubleshooting IP Acquisition
The purpose of this portion of the boot process if to give the target a way to communicate on the network and let it know where it can get the network boot program
The first step in the Provisioning Services Boot Process is for the PVS target device to acquire an IP address.
This can be accomplished either statically or dynamically
- Booting from the NIC using DHCP
- Booting from a PVS Boot Device Management (BDM) disk or .iso that is configured to use DHCP
- Create a boot disk using the PVS Boot Device Management (BDM) with a specified static IP address
Network tracing is the best method to be used for DHCP/IP acquisition issues. We can use WireShark https://www.wireshark.org/download.html on the PVS server. This will give you a pretty nice picture what is going on server side.
- If possible capture both server-side and target side trace
- If possible capture a good a bad trace an compare both
- Get as much IP information as possible
Target Side Capture Methods
- Physical Server
Port Mirroring, Hubbing out, using a Tap
Troubleshooting Bootstrap Download
Once the target device obtains an IP address, it goes through the phase of downloading the bootstrap file.
Using the address configured in DHCP option 66, the PVS target device contacts the TFTP server (or boot server) and requests the boot file that is configured in DHCP option 67. This bootstrap file is called ARDBP32.bin.
Using BDM a PVS target will download TSBBDM.bin over port 6969
Consider the following options when creating the Bootstrap File.
“Interups Safe Mode” should only be used during troubleshooting.
In order to see the information to troubleshoot this in particular, you should also use WireShark and focus on DHCP and TFTP, ARP request’s.
PVS Logon Process Troubleshooting.
After the target device gets an IP address and downloads the bootstrap file it proceeds to login to a PVS server to start streaming the vDisk image.
The list of login servers is provided in the bootstrap file that is downloaded from the TFTP server
the steps involved with the login process;
- Login Port Request
- Present MAC address
- Load Balancing
- Transfer to I/O
You can also see this in the event log of the PVS server.
The windows eventlog will give enough information at this point. For instance;
- Incorrect Boot strap information
- Stream Service is not started
- Target Device does not exist in the database
- Or UDP ports are blocked.
Troubleshooting Single Read Mode Streaming
After a target device has logged into PVS and has been directed to a PVS server for streaming, the bootstrap file (ARDBP32.bin, TSBBDM.bin or TSBOROM.bin) will then intercept any requests.
The bootstrap directs any block level requests for data as to the PVS server as single read requests
The single read mode procedure will continue until the Microsoft Windows Operating System starts loading drivers and BNISTACK is successfully loaded
After the login the target will continue to process traffic using the UNDI driver. (Universal Network Driver Interface and will send one to one packets.
During this time the target will display the windows logo. Low level drivers are being loaded here
If the server does not respond to a request within 5 seconds the target device will request the same data again. This will happen a maximum of 3 time before the target will attempt to reconnect and login to the PVS server again
This is generally the longest portion of the boot process due to the amount of data that has to be transferred
Response delays here contribute the most to long boot times
Consider the following;
- Large Send Offload
- Network instability
- Conflicts with UNDI Driver
Troubleshooting BNISTACK / MIO Read Mode
MIO (Multiple Input/Output) mode begins when BNISTACK is loaded by the operating system. BNISTACK is loaded into memory and takes over for the bootstrap, it manages the communication between the target device and the PVS server. At this point the following information is exchanged:
- Image Mode
- Active Directory Password Management Option
- Write Cache Type and Size
- Client Name
BNISTACK uses multiple threads to communicate with the PVS server
MIO uses a single request -> multiple response structure
Once UNDI hands off to BNIStack (PVS Target Device’s Main Filter Driver) the target will begin to request larger amounts of data. This activity will continue up to the windows login screen.
Consider the following;
- Firewall/Antivirus Services
- AV Updates are being loaded at this point
- Domain Profile Creation.
- Filter Driver conflicts
Provisioning Services has been providing the ability to capture logs since the time of PVS version 4.5. Using the Log4cxx framework, PVS logging has been able to provide information about the goings on of the PVS console, the Soap Service, and the Stream Service. These logs have provided valuable historical data when trying to troubleshoot issues.
There have been many improvements to the logging capability and as the feature set for PVS grew Citrix introduced new logs as well as refined some existing logs. With the introduction of PVS 7.0 the logging format has been changed from Log4cxx to CDF. Citrix has long used CDF as a mainstay of our logging mechanism across many products
This means a couple of things for PVS. Since CDF tracing is reactive and not enabled by default there is no built in historical logging in PVS 7.x
With the introduction of Provisioning Services 7.0, the Provisioning Services logging mechanism has changed from persistent log files to CDF tracing. Provisioning Services no longer logs streaming event to a log file. You must configure CDF tracing to capture persistent logging for determining the root cause of a problem.
To enable continuous CDF tracing in Provisioning Services 7.0, complete the following procedure:
- Download CDFMonitor.
- Download the Provisioning Services specific configuration file;
- Extract the files to the C:cdfmonitor folder.
- Replace the CDFMonitor.exe.config that comes with CDFMonitor with the one downloaded
- Open a command line interface and open the C:cdfmonitor folder.
- Run the following command: cdfmonitor.exe /installservice
CDFMonitor runs as a service that persists even after the server restarts. Continuous CDF trace is then captured on all modules. The trace file will grow to 100 MB in size and will rotate between five files. The moment the CDFMonitor service is restarted, a new CDF trace file is created.
Gathering Log Files
To collect the log files, complete the following procedure:
Stop the CDFMonitor service from the Services applet.
Copy trace files from the c:windowscdfmonitor folder.
Start the CDFMonitor service from the Services applet, as shown in the following screen shot:
So at this time we have a trace file with the *.etl extension
It is possible to open this in CDFAnalyzer
Download and just run the tool, there is no need to install.
When you open the tool and load one of the ETL files that was created with CDF monitor you can see that the format is unknown
That means, the we didn’t have “symbols” to solve that GUID strings. For downloading that files we will use another Citrix tool – CDFMonitor
Start the command line, go to CDFMonitor directory and type:
You need to provide you’re username and then continue with the download
Now, when we have all the TMF files we can configure CDFAnalyzer to use them. From menu choose “Tools” and the “Properties” and put the whole path where TMF files are located.
So at this point start CDFAnlyzer again and load the ETL file again..
Download the PVSData Collector from the following link
Just extract the zip file, there is no need to install the software..
Select the application.
The application will start automatically
At a certain point administrator credentials needs to be filled in.
The PVSDatacollector wil continue to collect the information and provide the progress.
the PVSDataCollector will complete providing the name of the zip file when the collection and compression is complete. Instructions are provided for where to upload the file.
The following is an example what is in the trace file
Once the data is captured you can upload this to Citrix Insight Services
Proceed to the following URL
Login with you’re MyCitrix Credentials and click on Lets Go to continue
Click on the Upload Button to uplad the zip file you have created with PVSDatacollector
Enter the apropiate information and continue with the Upload File button
The zip file is being uploaded
After uploading the file, the file is analyzed and the results are displayed
CDFControl is an event tracing controller/consumer, geared towards capturing Citrix Diagnostic Facility (CDF) trace messages that are output from the various Citrix tracing providers. Now in this version, CDFControl offers new capabilities, such as inserting trace, parsing filter support, and more.
Reference materials used: Citrix.com, Support.citrix.com and the E-Docs website.