wiki:Commissioning

Version 16 (modified by jayce, 5 years ago) (diff)

Updated the list of TBN scripts.

LWA1 Commissioning Scripts

Overview and Requirements

This is a collection of scripts use for commissioning and testing at the LWA1 station. All of the scripts depend on the LSL 0.6.x branch.

The Commissioning scripts build off of the LSL 0.6.x framework for working with LWA data and also depends on the following python modules:

  • h5py
  • pytz
  • wxPython

No installation (e.g., python setup.py install) is required to use the software. Simply run the scripts in the Commissioning directory.

Obtaining

The easiest way to obtain the latest version of the commissioning scripts is via subversion export/checkout:

svn checkout http://fornax.phys.unm.edu/lwa/subversion/trunk/Commissioning

Contents

General

checkDRSU.sh

Shell script to run on Linux machines with attached and powered-on DRSU to report the firmware version of Seagate drives with model number "ST31000525SV". This is useful for identifying disks with the newer (and less reliable) firmware version "CV12".

time2time.py

Given a date/time in the format of "YYYY/MM/DD HH:MM:SS[.SSS]" in local time, compute the corresponding MJD and MPM in UTC. If no date/time is specified, the current local date/time is used for the conversion.

mjd2local.py

For a given MJD value or list of MJD values, return the range of local times associated with that MJD.

astroevents2.py

List the rise, transit, and set times for the brightest radio objects in the sky.

estimateData.py

Script to estimate the data volume of TBN and DRX observations.

findMyStand.py

Simple script to locate and label where stands are in the array.

plotShelterTemp.py

Short script to read in a shelter.txt file (log of shelter temperature with each line consisting of a UNIX time stamp and a shelter temperature) and plot all of the available data.

Update: This script is also now compatiable with the /data/thermometer##.txt files created by the new SHL MCS.

plotPDUVoltages.py

Short script to plot the input PDU voltages over time from the /data/rack##.txt files generated by the new SHL MCS software.

plotBoardTemps.py

Short script to read in a temp.txt file (log of DP FPGA temperatures) and plot all of the available data.

plotChassisTemps.py

Script to plot colormaps of the FPGA temperatures as a function of time, chassis, and physical slot.

TBW

checkTimetags.py

Script for checking for missing frames and bad time tags in a full DP TBW capture. If problems are found the error are noted and printed out. It shouldn't be too hard to modify this for fewer inputs.

tbw2hdf.py

Export select stands from a TBW file to HDF5.

stationMaster.py

Modified version of tbwSpectra.py that is included with LSL (version >= 0.4.0) that estimated the frequency (in MHz) of each dipole's resonance point and saves that information along with time-average spectra to a NPZ file.

tinyMaster.py

Modified version of stationMaster.py that uses Numpy memory maps to work on systems with less than 16 GB of memory.

stationMaster2.py

Modified version of stationMaster.py that uses the 'ClipLevel?' keyword in LSL >= 0.4.2 to blank impulsive RFI events.

tinyMaster2.py

Modified version of stationMaster2.py that uses Numpy memory maps to work on systems with less than 16 GB of memory.

smGUI.py

GUI that interfaces with a NPZ file created by stationMaster.py that makes looking for problems with dipoles/mappings/etc. a point-and-click exercise.

rfiCheck.py

Script to take a single TBW capture and create a RFI-centered HDF5 file for stands 1, 10, 54, 248, 251, and 258 (the outlier). These stands correspond to the four corners of the array, the center, and the outlier. The HDF5 contains values for the spectral kurtosis estimated from the data and various statistics about the timeseries (mean, std. dev., percentiles, etc.)

plotRFI.py

Script to take the output of rfiCheck.py and make a series of plots to look at.

TBN

checkTimetags.py

Simple script for making sure that time tags update properly in a TBN file. The script not only checks time tags but also checks for dropped packets and synchronization problems.

eyeDiagram.py

Create an eye diagram for some portion of a TBN file.

eyeDiagram2.py

Look for glitches in a TBN file by fitting a quantized sine wave to the data.

tbnTimeseries.py

Script to plot out time series I/Q data from a TBN file.

tbnFileCheck.py

Script to skip through a TBN file and check for clipping on stand 10.

checkMissing.py

Look at what is coming out of the TBN ring buffer and plot of what frames are missing, if any, as a function of time in the file.

DRX

decodeStatus.py

Take the status code returned by a DP_ BOARD_STAT query and break it down into its constituent parts.

gain.py

JPL-created script for creating gain files to be used by DRX.

delay.py

JPL-create script for created delay files to be used by DRX.

test_analog_in2.py

Script for setting up a test DRX observation with a variety of tuning, frequency, bandwidth, and gain settings. Needs gain.py and delay.py to work.

generateDelays.py

Generate gain and delay files for a particular topocentric pointing (azimuth and elevation in degrees) at a specified frequency.

genernateDelays.py generates text files for gains and delays that need to be converted to binary packed files with the MCS/Executive commands megfg and medfg.

estimateBeam.py

Use the LSL misc.beamformer's integer delay and sum to estimate the beam shape towards a particular azimuth and elevation and save the beam to a Numpy NPZ file.

driftcurve.py

Using the beam created by estimateBeam.py, generate a temperature as a function of time for that pointing over an entire day. This script is based on the LSL driftcurve.py script.

checkTimetags.py

Simple script to check the time tag sequence in a DRX file and look for strange jumps and other badness. It can also be used to check the DRX decimate (sample rate) field and if the time has been set correctly on the various boards.

getTuningOffset.py

Check for an offset in the time tags associated with the two tunings beyond what is controlled by the time offset stored in the frame header. This should read in the first complete set of frame from all four beamtunepols and compare tunings 1 and 2. Any differences between the times are noted.

checkTuningCoherency.py

Given a a DRX file with both tunings set to the same parameters, check for coherency across the tunings via cross-correlation. For each set of data, print out the lag (in DRX samples) where the cross-correlation function has its peak and the normalized correlation magnitude relative to the previous set of tuning 1 data.

beamCoherency.py

Given two or more DRX files from different beams, check for coherency between the beams and make sure that the beams agree with the T_NOM values. For each beam/tuning/pair, print out the lag (converted to ticks), the time tag differences (both raw and T_NOM corrected) and the normalized correlation magnitude relative to the same-beam tuning cross-correlation.

drxTimeseries.py

Script for plotting time series data from a DRX file. This file is also available as part of the LSL package (version >= 0.4.0) but is included here since this is the script we actually used at JPL for pre-acceptance testing.

drxPower.py

Modified version of drxTimeseries.py that plots the power for each DRX sample instead of the raw I/Q values. The power is summed for the integration time (listed as average) and displayed for the duration that it is computed.

drxFileCheck.py

Script for inspecting one second of DRX data every 15 minutes to check the gains setting.

drxSpectra.py

Script for reading in DRX data and created time-averaged power spectral density plots. Like drxTimeseries.py, this file is also available as part of the LSL package (version >= 0.4.0) but is included here since this is the script we actually used at JPL for pre-acceptance testing.

drxWaterfall.py

Read in DRX files and create a collection of time-averaged spectra. These spectra are saved to a NPZ file called <filename>-waterfall.npz.

aggregateWaterfalls.py

Combine individual NPZ files generated by drxWaterfall.py at a high time resolution into a single lower time resolution NPZ file for use with plotWaterfall.py. All files should be processed with the same parameters (integration time, duration, FFT size, etc.) for the combining to make sense.

plotWaterfall.py

Given the NPZ output of drxWaterfall, plot it in an interactive way.

eyeDiagram.py

Create an eye diagram for some portion of a DRX file.

eyeDiagram2.py

Look for glitches in a DRX file by fitting a quantized sine wave to the data.

DRX-Mode Dipole-Dipole and Beam-Dipole Fringing

fringeSets.py

Read in SSMIF file and create a set of DRX gain files (zero-delay for dipoles-only) that puts a single dipole or beam on the X pol and the outlier on the other. The gain files are constructed such that all data is from X pol.

fringeDipole.py

Script to fringe DRX files that have one dipole on X pol. and another dipole (the outlier probably) on Y pol. Accepts three command line arguments:

  1. stand number on X pol.
  2. stand number on Y pol.
  3. DRX filename(s)

At the end of the correlation, the visibilities are written to a NPZ file.

fringeBeam.py

Similar to fringeDipole.py, but expects the beam to be on X pol. and the dipole on

Y pol. Accepts two command line arguments:

  1. stand number on Y pol.
  2. DRX filename(s)

plotFringes.py

Simple script to load in a collection of NPZ files generated by fringeDipole.py/ fringeBeam.py and plot the visibility amplitude over time.

plotFringes2.py

Similar to plotFringes.py but displays a waterfall of the visibility and includes the integrated bandpass.

simulateFringesBright.py

Given a collection of NPZ files generated by fringeDipole.py, simulate the fringes using the lsl.sim.vis.buildSimData() function and the bright sources listed in lsl.sim.vis.srcs.

DRX Processing Scripts with HDF5 Output

hdfWaterfall.py

Version of drxWaterfall.py that writes to HDF5 rather than NPZ to provide an easier path to getting the data into other packages, e.g., Matlab. The output file format is described here.

plotHDF.py

Version of plotWaterfall.py for viewing HDF5 files.

waterfall2hdf.py

Simple script to convert existing NPZ files generated by drxWaterfall.py/ drxWaterfallDRSU.py into HDF5 files that are compatible with the above HDF5 organization.

drspec2hdf.py

Simple script to convert a binary DR spectrometer file into an HDF5 file that is compatible with the above HDF5 organization.

drxWaterfallStokes.py

Similar to drxWaterfall.py, but computes all four Stokes parameters. The output file format is described here.

plotHDFStokes.py

This script is to plotHDF.py as drxWaterfallStokes.py is to drxWaterfall.py.