LWA Development Primitives - A set of utilities that should make developing new analysis software easier. These functions wrap the nitty gritty of the file reading and unpacking behind Python objects.
Also included is the LWA1DataFile function that take a filename and tries to determine the correct data format object to use.
Read and return the entire TBW capture. This function returns a three-element tuple with elements of:
- the actual duration of data read in,
- the time tag for the first sample, and
- a 2-D Numpy array of data.
The time tag is returned as seconds since the UNIX epoch by default. However, the time tags can be returns as samples at fS if the timeInSamples keyword is set.
The sorting order of the output data array is by digitizer number - 1.
Note
Setting the ‘duration’ keyword has no effect on the read process because the entire capture is always read in.
Read and return a single lsl.reader.tbw.Frame instance.
Estimate the n-sigma level for the absolute value of the voltages. Returns a list with indicies that are the digitizer numbers minus one.
Offset a specified number of seconds in an open TBN file. This function returns the exact offset time.
Note
The offset provided by this function is relatively crude due to the structure of TBN files.
Read in a chunk (in seconds) of TBN data. This function returns a three-element tuple with elements of:
- the actual duration of data read in,
- the time tag for the first sample, and
- a 2-D Numpy array of data.
The time tag is returned as seconds since the UNIX epoch by default. However, the time tags can be returns as samples at fS if the timeInSamples keyword is set.
The sorting order of the output data array is by digitizer number - 1.
Read and return a single lsl.reader.tbn.Frame instance.
Estimate the n-sigma level for the absolute value of the voltages. Returns a list with indicies corresponding to: 0) Tuning 1, X pol. 1) Tuning 1, Y pol. 2) Tuning 2, X pol. 3) Tuning 2, Y pol.
Offset a specified number of seconds in an open DRX file. This function returns the exact offset time.
Given an open DRX file and an amount of data to read in in seconds, read in the data and return a three-element tuple of the actual duration read in, the time for the first sample, and the data as numpy array.
This function always returns a 2-D array with the first dimension holding four elements. These elements contain, in order:
- Tuning 1, polarization X
- Tuning 1, polarization Y
- Tuning 2, polarization X
- Tuning 2, polarization Y
Read and return a single lsl.reader.drx.Frame instance.
Class to make it easy to interface with a DR Spectrometer file. Methods defined for this class are:
- getInfo - Get information about the file’s contents
- getRemainingFrameCount - Get the number of frames remaining in the file
- offset - Offset a specified number of seconds into the file
- readFrame - Read and return a single lsl.reader.drspec.Frame instance
- read - Read a chunk of data in and return it as a numpy array
Offset a specified number of seconds in an open DR spectrometer file. This function returns the exact offset time.
Given an open DR spectrometer file and an amount of data read in in seconds, read in the data and return a three-element tuple of the actual duration read in, the times at the beginning of each stream, and the data as numpy array.
Read and return a single lsl.reader.drspec.Frame instance.
Wrapper around the various classes defined here that takes a file, determines the data type, and initializes and returns the appropriate LDP class.
Python module to reading in data from both 12-bit and 4-bit TBW files. This module defines the following classes for storing the TBW data found in a file:
object that contains all data associated with a particular TBW frame. The primary consituents of each frame are:
- FrameHeader - the TBW frame header object and
- FrameData - the TBW frame data object.
Combined, these two objects contain all of the information found in the original TBW frame.
For reading in data, use the readFrame function. It takes a python file- handle as an input and returns a fully-filled Frame object. readFrame is designed to work with both 4-bit and 12-bit observations.
For describing the format of data in the file, two function are provided:
read in the first several frames to see how many stands are found in the data. .. note:
This function is a little flaky on TBW data sets that have less
than a full complement or 12M (36M) samples.
Changed in version 0.4.0: Switched over from pure Python readers to the new C-base Go Fast! readers.
Class that stores the information found in the header of a TBW frame. All three fields listed in the DP ICD version H are stored as well as the original binary header data.
Function to parse the TBW ID field and return the size of number of bits that comprise the data. 12 is returned for 12-bit data, and 4 for 4-bit data.
Function to check if the data is really TBW and not TBN by examining the TBW ID field. Returns True if the data is TBW, false otherwise.
Function to parse the TBW ID field and return the stand number.
Class that stores the information found in the data section of a TBW frame. Both fields listed in the DP ICD version H are stored.
Function to convert the time tag from samples since the UNIX epoch (UTC 1970-01-01 00:00:00) to seconds since the UNIX epoch.
Class that stores the information contained within a single TBW frame. It’s properties are FrameHeader and FrameData objects.
Convenience wrapper for the Frame.FrameHeader.getDataBits function.
Convenience wrapper for the Frame.FrameData.getTime function.
Convenience wrapper for the Frame.FrameHeader.isTBW function.
Convenience wrapper for the Frame.FrameHeader.parseID function.
Function to read in a single TBW frame (header+data) and store the contents as a Frame object. This function wraps readerHeader and readData[(12)|4].
Find out the number of data bits used in the file be reading in the first frame.
Find out how many frames are present per observation by examining the first frames for what would be 260 stands. This is done by reading two frames and then skipping the next 30,000.
Note
Post-IOC it is probably simpler to adopt a value of the number of frames per observation of 260 rather than try to find it from the file.
Python module for reading data in from TBN files.This module defines the following classes for storing the TBN data found in a file:
object that contains all data associated with a particular TBN frame. The primary constituents of each frame are:
- FrameHeader - the TBN frame header object and
- FrameData - the TBN frame data object.
Combined, these two objects contain all of the information found in the original TBN frame.
In addition to storing the data available in the frame, the Frame object also has attributes for holding information about the gain, central frequency, and filter code used for the observations.
For reading in data, use the readFrame function. It takes a python file- handle as an input and returns a fully-filled Frame object. The readBlock function reads in a (user-defined) number of TBN frames and returns a ObservingBlock object.
For describing the format of data in the file, two function are provided:
Changed in version 0.4.0: Switched over from pure Python readers to the new C-base Go Fast! readers.
Changed in version 0.5.0: Support for ECR 11 TBN header format change.
Class that stores the information found in the header of a TBW frame. All three fields listed in the DP ICD version H are stored as well as the original binary header data.
Changed in version 0.5.0: Added various attributes to retrieve the central frequnecy and gain that are part of the ECR 11 changes.
Convert the tuning word to a frequency in Hz.
Function to convert the sample rate in Hz to a filter code.
Get the current TBN gain for this frame.
Function to check if the data is really TBN and not TBW by examining the TBN ID field. Returns True if the data is TBN, false otherwise.
Function to parse the TBN ID field and return a tuple of the stand number and polarization.
Function to set the sample rate of the TBN data in Hz.
Class that stores the information found in the data section of a TBN frame. Both fields listed in the DP ICD version H are stored.
Changed in version 0.5.0: Removed various attributes related to storing a central frequnecy and gain that aren’t needed with ECR 11.
Function to convert the time tag from samples since the UNIX epoch (UTC 1970-01-01 00:00:00) to seconds since the UNIX epoch.
Class that stores the information contained within a single TBN frame. It’s properties are FrameHeader and FrameData objects.
Changed in version 0.5.0: Removed various attributes related to storing a central frequnecy and gain that aren’t needed with ECR 11.
Convenience wrapper for the Frame.FrameHeader.getCentralFreq function.
Convenience wrapper for the Frame.FrameData.getFilterCode function.
Convenience wrapper for the Frame.FrameHeader.getGain function.
Convenience wrapper for the Frame.FrameData.getTime function.
Convenience wrapper for the Frame.FrameHeader.isTBN function.
Convenience wrapper for the Frame.FrameHeader.parseID function.
Convenience wrapper for the Frame.FrameData.setSampleRate function.
Function to read in a single TBN frame (header+data) and store the contents as a Frame object.
Function to read in a single TBN block (frames set by the nFrames keyword) and store the contents as a ObservingBlock object. This function wraps readFrame.
Find out what the sampling rate/filter code is from consecutive sets of observations. By default, the rate in Hz is returned. However, the corresponding filter code can be returned instead by setting the FilterCode keyword to True.
Find out how many frames are present per observation by examining the first 2,080 TBN frames. Return the number of frames per observations as a two- element tuple, one for each polarization.
So many TBN frames are read in order to try to compensate for the inter- leaving of the packets from the various DP1 boards during the recording.
Note
Post-IOC it is probably simpler to adopt a value of the number of frames per observation of 520 rather than try to find it from the file.
Python module to read in DRX data. This module defines the following classes for storing the DRX data found in a file:
object that contains all data associated with a particular DRX frame. The primary constituents of each frame are:
- FrameHeader - the DRX frame header object and
- FrameData - the DRX frame data object.
Combined, these two objects contain all of the information found in the original DRX frame.
For reading in data, use the readFrame function. It takes a python file- handle as an input and returns a fully-filled Frame object. The readBlock function reads in a (user-defined) number of DRX frames and returns a ObservingBlock object.
For describing the format of data in the file, two function are provided:
Finally, there are also two experimental functions defined in this file for averaging DRX observations. averageObservations performs a straight channel- by-channel average of a collection of DRX frames, which averageObservations2 allows for both temporal and spatial averaging.
Changed in version 0.4.0: Switched over from pure Python readers to the new C-base Go Fast! readers.
Class that stores the information found in the header of a DRX frame. All six fields listed in the DP ICD version H are stored as well as the original binary header data.
Function to convert the sample rate in Hz to a filter code.
Return the sample rate of the data in samples/second.
Parse the DRX ID into a tuple containing the beam (1 through 4), tunning (1 and 2), and polarization (0 and 1).
Class that stores the information found in the data section of a DRX frame. All three fields listed in the DP ICD version H are stored.
Function to set the central frequency of the DRX data in Hz.
Function to set the gain of the DRX data.
Class that stores the information contained within a single DRX frame. It’s properties are FrameHeader and FrameData objects.
Convenience wrapper for the Frame.FrameData.getCentralFreq function.
Convenience wrapper for the Frame.FrameHeader.getFilterCode function.
Convenience wrapper for the Frame.FrameHeader.getSampleRate function.
Function to convert the time tag from samples since the UNIX epoch (UTC 1970-01-01 00:00:00) to seconds since the UNIX epoch.
Convenience wrapper for the Frame.FrameHeader.parseID function.
Convenience wrapper for the Frame.FrameData.setGain function.
Class that stores all frames associates with a particular beam at a particular time.
Convenience wrapper for the Frame.FrameData.getCentralFreq function.
Note
This returned a two-element tuple giving the frequncies for both tunings.
Convenience wrapper for the Frame.FrameData.getFilterCode function.
Convenience wrapper for the Frame.FrameData.getTime function.
Convenience wrapper for the Frame.FrameData.setGain function.
Function to read in a single DRX frame (header+data) and store the contents as a Frame object. This function wraps readerHeader and readData.
Function to read in a single DRX block (four frames) and store the contents as a ObservingBlock object. This function wraps readFrame.
Find out what the sampling rate/filter code is from a single observations. By default, the rate in Hz is returned. However, the corresponding filter code can be returned instead by setting the FilterCode keyword to true.
This function is included to make easier to write code for TBN analysis and modify it for DRX data.
Find out how many beams are present by examining the first 16 DRX records. Return the number of beams found.
Find out how many frames are present per beam by examining the first 16 DRX records. Return the number of frames per observations as a four- element tuple, one for each beam.
New in version 0.5.
Python module to read in DR spectrometer data. This module defines the following classes for storing the spectra found in a file:
object that contains all data associated with a particular spectrometer frame. The primary constituents of each frame are:
- FrameHeader - the spectrometer frame header object and
- FrameData - the spectral data object.
Combined, these two objects contain all of the information found in the original spectrometer frame.
For reading in data, use the readFrame function. It takes a python file- handle as an input and returns a fully-filled Frame object For describing the format of data in the file, three function are provided:
- getSampleRate - get the sample rate in the file
- getFrameSize - get the total (header + data) frame size
- getFFTsPerIntegration - get the number of FFT windows per integration
- getTransformSize - get the FFT length
- getIntegrationTime - get the integration time
Note
This reader works with the most current version of the DR spectrometer data format as specified in the version 1.7. To read data created with previous versions of the DR spectrometer, use LSL version 0.5.2.
Changed in version 1.0.1: Added in new functions to help better describe the contents of a DR spectrometer file.
Class that stores the information found in the header of a DR spectrometer/DRX frame.
Return whether or not the frame contains linear polarization products or not.
New in version 0.6.0.
Return whether or not the frame contains Stokes polarization parameters or not.
New in version 0.6.0.
Return a list of data products contained in the file.
New in version 0.6.0.
Return the number of FFT windows per integration.
New in version 1.0.1.
Function to convert the sample rate in Hz to a filter code.
Return the sample rate of the data in samples/second.
Return the beam the frame corresponds to.
Class that stores the information found in the data section of a DR spectrometer/ DRX frame.
Changed in version 0.5.3: Added the saturations field to keep up with saturation counts.
Changed in version 0.6.0: The attributes that store the data are not defined until a frame is read in order to account for the fact that spectrometer files can store either linear or Stokes data.
Function to set the central frequency of the DRX data in Hz.
Class that stores the information contained within a single DR spectrometer/ DRX frame. It’s properties are FrameHeader and FrameDataLinear/FrameDataStokes objects.
Changed in version 0.6.0: By default the data contained with in a frame is normalized by the number of fills (header.fills parameter). For data products that are a function of more than one primary input, i.e., XY* or I, the minimum fill of X and Y are used for normalization.
Convenience wrapper for the Frame.FrameHeder.containsLinearData function.
Convenience wrapper for the Frame.FrameHeder.containsStokesData function.
Convenience wrapper for the Frame.FrameData.getCentralFreq function.
Convenience wrapper for the Frame.FrameHeder.getDataProducts function.
Conveinence wrapper for the Frame.FrameHeader.getFFTsPerIntegration function.
New in version 1.0.1.
Convenience wrapper for the Frame.FrameHeader.getFilterCode function.
Return the integration time for data in seconds.
New in version 1.0.1.
Convenience wrapper for the Frame.FrameHeader.getSampleRate function.
Function to convert the time tag from samples since the UNIX epoch (UTC 1970-01-01 00:00:00) to seconds since the UNIX epoch.
Find out what the transform size is.
New in version 1.0.1.
Convenience wrapper for the Frame.FrameHeader.parseID function.
Convenience wrapper for the Frame.FrameData.setGain function.
Function to read in a single DR spectrometer/DRX frame (header+data) and store the contents as a Frame object.
Find out the data products contained in the file by looking at a frame.
Find out if the file contains linear polarization products or not.
Find out if the file contains Stokes parameters or not.
Find out what the sampling rate/filter code is from a single observations. By default, the rate in Hz is returned. However, the corresponding filter code can be returned instead by setting the FilterCode keyword to true.
Find out what the frame size in a file is at the current file location. Returns the frame size in bytes.
Find out what the number of FFT windows per integration is at the current file location.
New in version 1.0.1.
Find out what the transform size in a file is at the current file location.
Find out what the integration time is at the current file location.
Module that contains the error classes for the DRX, TBN, and TBW readers. These errors are currently meant to deal with file I/O problems.
Base class for file I/O problem during numpy.fromfile calls and out-of- sync Mark5C headers.
Extension to the base class for dealing with EOF errors. The error code is 1.
Extension to the base class for dealing with errors that occur when numpy.fromfile tried to read more data can exists. This is a specialized form of EOF error. The error code is 2.
Extension to the base class for dealing with Mark 5C header sync word problems. If the sync word doesn’t match what is expected. The error code is 3.
Extenstion to the base class for dealing with trying to read in TBW data with a TBN reader. The error code is 4.
Extenstion to the base class for dealing with trying to read in TBN data with a TBW reader. The error code is 5.
Function to provide a list of errors defined in this file. It alternatively takes an error code using the ‘errno’ keyword and returns its description.
Buffer for dealing with out-of-order/missing frames.
Changed in version 0.5: Removed support for DRX FrameBuffers since they shouldn’t be needed.
Changed in version 0.6: Removed support for TBW FrameBuffers since they didn’t really work.
Changed in version 1.0.0: Added back in the DRX FrameBuffers since they might be useful. Dropped TBW support from within the FrameBuffer class.
Bases: object
Frame buffer for re-ordering TBN and DRX frames in time order. This class is filled with frames and a returns a frame list when the ‘nSegments’ starts filling. In that case, the oldest segment is returned.
The buffer also keeps track of what has already been read out so that tardy frames are just dropped. For buffers that are read out, missing frames are replaced with frames filled with zeros.
Note
Due to the nature of the buffer, it is possible that there will still be ‘nSegments’-1 segements in the buffer that are either full or partially full. This can be retrieved using the buffer’s ‘flush()’ function.
Append a new frame to the buffer with the appropriate time tag. True is returned if the frame was added to the buffer and False if the frame was dropped because it belongs to a buffer that has already been returned.
Calculate the maximum number of frames that we expect from the setup of the observations and a list of tuples that describes all of the possible stand/pol combination.
This will be overridden by sub-classes of FrameBuffer.
Create a ‘fill’ frame of zeros using an existing good packet as a template.
This will be overridden by sub-classes of FrameBuffer.
Figure of merit for storing/sorting frames in the ring buffer.
This will be overridden by sub-classes of FrameBuffer.
Return a list of lists containing all remaining frames in the buffer from buffers that are considered ‘full’. Afterwards, delete all buffers. This is useful for emptying the buffer after reading in all of the data.
Note
It is possible for this function to return list of packets that are mostly invalid.
Return a list of frames that consitute a ‘full’ buffer. Afterwards, delete that buffer and mark it as closed so that any missing frames that are recieved late are dropped. If none of the buffers are ready to be dumped, None is returned.
Print out the status of the buffer. This contains information about: 1. The current buffer fill level 2. The numer of full and partial buffer dumps preformed 3. The number of missing frames that fill packets needed to be created
for
Bases: lsl.reader.buffer.FrameBuffer
A sub-type of FrameBuffer specifically for dealing with TBN frames. See lsl.reader.buffer.FrameBuffer for a description of how the buffering is implemented.
Keywords: stands
list of stand to expect packets for
The number of segements in the ring can be converted to a buffer time in seconds:
| Segments | TBN Filter Code | ||||||
| 1 | 2 | 3 | 4 | 5 | 6 | 7 | |
| 10 | 5.1 | 1.6 | 0.8 | 0.4 | 0.2 | 0.1 | 0.05 |
| 20 | 10.2 | 3.3 | 1.6 | 0.8 | 0.4 | 0.2 | 0.10 |
| 30 | 15.4 | 4.9 | 2.5 | 1.2 | 0.6 | 0.3 | 0.15 |
| 40 | 20.5 | 6.6 | 3.3 | 1.6 | 0.8 | 0.4 | 0.20 |
| 50 | 25.6 | 8.2 | 4.1 | 2.0 | 1.0 | 0.5 | 0.26 |
| 100 | 51.2 | 16.4 | 8.2 | 4.1 | 2.0 | 1.0 | 0.51 |
Calculate the maximum number of frames that we expect from the setup of the observations and a list of tuples that describes all of the possible stand/pol combination.
Create a ‘fill’ frame of zeros using an existing good packet as a template.
Figure of merit for sorting frames. For TBN it is: <frame timetag in ticks>
Bases: lsl.reader.buffer.FrameBuffer
A sub-type of FrameBuffer specifically for dealing with DRX frames. See lsl.reader.buffer.FrameBuffer for a description of how the buffering is implemented.
Keywords: beams
list of beam to expect packets for
The number of segements in the ring can be converted to a buffer time in seconds:
| Segments | DRX Filter Code | ||||||
| 1 | 2 | 3 | 4 | 5 | 6 | 7 | |
| 10 | 0.16 | 0.08 | 0.04 | 0.02 | 0.01 | 0.004 | 0.002 |
| 20 | 0.33 | 0.16 | 0.08 | 0.04 | 0.02 | 0.008 | 0.004 |
| 30 | 0.49 | 0.25 | 0.12 | 0.06 | 0.03 | 0.013 | 0.006 |
| 40 | 0.66 | 0.33 | 0.16 | 0.08 | 0.03 | 0.017 | 0.008 |
| 50 | 0.82 | 0.41 | 0.20 | 0.10 | 0.04 | 0.021 | 0.010 |
| 100 | 1.64 | 0.82 | 0.41 | 0.20 | 0.08 | 0.042 | 0.021 |
Calculate the maximum number of frames that we expect from the setup of the observations and a list of tuples that describes all of the possible stand/pol combination.
Create a ‘fill’ frame of zeros using an existing good packet as a template.
Figure of merit for sorting frames. For DRX it is: <frame timetag in ticks>