Data Readers

TBW

Python module to reading in data from both 12-bit and 4-bit TBW files. This module defines the following classes for storing the TBW data found in a file:

Frame

object that contains all data associated with a particular TBW frame. The primary consituents of each frame are:

• FrameHeader - the TBW frame header object and
• FrameData - the TBW frame data object.

Combined, these two objects contain all of the information found in the original TBW frame.

The functions defined in this module fall into two class:

1. convert a frame in a file to a Frame object and
2. describe the format of the data in the file.

For reading in data, use the readFrame function. It takes a python file- handle as an input and returns a fully-filled Frame object. readFrame is designed to work with both 4-bit and 12-bit observations.

For describing the format of data in the file, two function are provided:

getDataBits

read in the first frame of an open file handle and return whether or not the data is 12 or 4-bit

getFramesPerObs

read in the first several frames to see how many stands are found in the data. .. note:

This function is a little flaky on TBW data sets that have less 
than a full complement or 12M (36M) samples.

Changed in version 0.4.0: Switched over from pure Python readers to the new C-base Go Fast! readers.

class lsl.reader.tbw.FrameHeader(frameCount=None, secondsCount=None, tbwID=None)

Class that stores the information found in the header of a TBW frame. All three fields listed in the DP ICD version H are stored as well as the original binary header data.

getDataBits()

Function to parse the TBW ID field and return the size of number of bits that comprise the data. 12 is returned for 12-bit data, and 4 for 4-bit data.

isTBW()

Function to check if the data is really TBW and not TBN by examining the TBW ID field. Returns True if the data is TBW, false otherwise.

parseID()

Function to parse the TBW ID field and return the stand number.

class lsl.reader.tbw.FrameData(timeTag=None, samples=400, xy=None)

Class that stores the information found in the data section of a TBW frame. Both fields listed in the DP ICD version H are stored.

getTime()

Function to convert the time tag from samples since the UNIX epoch (UTC 1970-01-01 00:00:00) to seconds since the UNIX epoch.

class lsl.reader.tbw.Frame(header=None, data=None)

Class that stores the information contained within a single TBW frame. It’s properties are FrameHeader and FrameData objects.

getDataBits()

Convenience wrapper for the Frame.FrameHeader.getDataBits function.

getTime()

Convenience wrapper for the Frame.FrameData.getTime function.

parseID()

Convenience wrapper for the Frame.FrameHeader.parseID function.

lsl.reader.tbw.readFrame(filehandle, Verbose=False)

Function to read in a single TBW frame (header+data) and store the contents as a Frame object. This function wraps readerHeader and readData[(12)|4].

lsl.reader.tbw.getDataBits(filehandle)

Find out the number of data bits used in the file be reading in the first frame.

lsl.reader.tbw.getFramesPerObs(filehandle)

Find out how many frames are present per observation by examining the first frames for what would be 260 stands. This is done by reading two frames and then skipping the next 30,000.

Note

Post-IOC it is probably simpler to adopt a value of the number of frames per observation of 260 rather than try to find it from the file.

TBN

Python module for reading data in from TBN files.This module defines the following classes for storing the TBN data found in a file:

Frame

object that contains all data associated with a particular TBN frame. The primary constituents of each frame are:

• FrameHeader - the TBN frame header object and
• FrameData - the TBN frame data object.

Combined, these two objects contain all of the information found in the original TBN frame.

ObservingBlock

object that stores a collection of Frames for all stands/polarizations for a particular time.

In addition to storing the data available in the frame, the Frame object also has attributes for holding information about the gain, central frequency, and filter code used for the observations.

The functions defined in this module fall into two class:

1. convert a frame in a file to a Frame object and
2. describe the format of the data in the file.

For reading in data, use the readFrame function. It takes a python file- handle as an input and returns a fully-filled Frame object. The readBlock function reads in a (user-defined) number of TBN frames and returns a ObservingBlock object.

For describing the format of data in the file, two function are provided:

getSampleRate

read in the few frame of an open file handle and return the sampling rate of the data

getFramesPerObs

read in the first several frames to see how many stands are found in the data.

Changed in version 0.4.0: Switched over from pure Python readers to the new C-base Go Fast! readers.

class lsl.reader.tbn.FrameHeader(frameCount=None, secondsCount=None, tbnID=None)

Class that stores the information found in the header of a TBW frame. All three fields listed in the DP ICD version H are stored as well as the original binary header data.

Note

Although this version of FrameHeader does not fully support the ECR 11 TBN header format change, the central frequency of the TBN tuning can be obtained via: >>> from lsl.common.dp import fS >>> freq = fS*FrameHeaderInstance.secondsCount/2**32

isTBN()

Function to check if the data is really TBN and not TBW by examining the TBN ID field. Returns True if the data is TBN, false otherwise.

parseID()

Function to parse the TBN ID field and return a tuple of the stand number and polarization.

class lsl.reader.tbn.FrameData(timeTag=None, iq=None)

Class that stores the information found in the data section of a TBN frame. Both fields listed in the DP ICD version H are stored.

getFilterCode()

Function to convert the sample rate in Hz to a filter code.

getTime()

Function to convert the time tag from samples since the UNIX epoch (UTC 1970-01-01 00:00:00) to seconds since the UNIX epoch.

setCentralFreq(centralFreq)

Function to set the central frequency of the TBN data in Hz.

setGain(gain)

Function to set the gain of the TBN data.

setSampleRate(sampleRate)

Function to set the sample rate of the TBN data in Hz.

class lsl.reader.tbn.Frame(header=None, data=None)

Class that stores the information contained within a single TBN frame. It’s properties are FrameHeader and FrameData objects.

Note

Although this version of FrameHeader does not fully support the ECR 11 TBN header format change, the central frequency of the TBN tuning can be obtained via: >>> from lsl.common.dp import fS >>> freq = fS*FrameInstance.header.secondsCount/2**32

getFilterCode()

Convenience wrapper for the Frame.FrameData.getFilterCode function.

getTime()

Convenience wrapper for the Frame.FrameData.getTime function.

parseID()

Convenience wrapper for the Frame.FrameHeader.parseID function.

setCentralFreq(centralFreq)

Convenience wrapper for the Frame.FrameData.setCentralFreq function.

setGain(gain)

Convenience wrapper for the Frame.FrameData.setGain function.

setSampleRate(sampleRate)

Convenience wrapper for the Frame.FrameData.setSampleRate function.

lsl.reader.tbn.readFrame(filehandle, SampleRate=None, CentralFreq=None, Gain=None, Verbose=False)

Function to read in a single TBN frame (header+data) and store the contents as a Frame object.

lsl.reader.tbn.readBlock(filehandle, nFrames=520, SampleRate=None, CentralFreq=None, Gain=None, Verbose=False)

Function to read in a single TBN block (frames set by the nFrames keyword) and store the contents as a ObservingBlock object. This function wraps readFrame.

..warning::

Since the various TBN frames are interleaved on the network before recording, the frames returned by a single call to readBlock may not all have the same frame count/time tag. It is recommended that the lsl.reader.buffer.TBNFrameBuffer be used in order to deal with the out-of-order frames.

lsl.reader.tbn.getSampleRate(filehandle, nFrames=None, FilterCode=False)

Find out what the sampling rate/filter code is from consecutive sets of observations. By default, the rate in Hz is returned. However, the corresponding filter code can be returned instead by setting the FilterCode keyword to True.

lsl.reader.tbn.getFramesPerObs(filehandle)

Find out how many frames are present per observation by examining the first 2,080 TBN frames. Return the number of frames per observations as a two- element tuple, one for each polarization.

So many TBN frames are read in order to try to compensate for the inter- leaving of the packets from the various DP1 boards during the recording.

Note

Post-IOC it is probably simpler to adopt a value of the number of frames per observation of 520 rather than try to find it from the file.

DRX

Python module to read in DRX data. This module defines the following classes for storing the DRX data found in a file:

Frame

object that contains all data associated with a particular DRX frame. The primary constituents of each frame are:

• FrameHeader - the DRX frame header object and
• FrameData - the DRX frame data object.

Combined, these two objects contain all of the information found in the original DRX frame.

ObservingBlock

object that stores a collection of Frames for all beams/tunings/ polarizations for a particular time.

The functions defined in this module fall into two class:

1. convert a frame in a file to a Frame object and
2. describe the format of the data in the file.

For reading in data, use the readFrame function. It takes a python file- handle as an input and returns a fully-filled Frame object. The readBlock function reads in a (user-defined) number of DRX frames and returns a ObservingBlock object.

For describing the format of data in the file, two function are provided:

getBeamCount

read in the first few frames of an open file handle and return how many beams are present in the file.

getFramesPerObs

read in the first several frames to see how many frames (tunings/polarizations) are associated with each beam.

Finally, there are also two experimental functions defined in this file for averaging DRX observations. averageObservations performs a straight channel- by-channel average of a collection of DRX frames, which averageObservations2 allows for both temporal and spatial averaging.

Changed in version 0.4.0: Switched over from pure Python readers to the new C-base Go Fast! readers.

class lsl.reader.drx.FrameHeader(frameCount=None, drxID=None, secondsCount=None, decimation=None, timeOffset=None)

Class that stores the information found in the header of a DRX frame. All six fields listed in the DP ICD version H are stored as well as the original binary header data.

getFilterCode()

Function to convert the sample rate in Hz to a filter code.

getSampleRate()

Return the sample rate of the data in samples/second.

parseID()

Parse the DRX ID into a tuple containing the beam (1 through 4), tunning (1 and 2), and polarization (0 and 1).

class lsl.reader.drx.FrameData(timeTag=None, flags=None, iq=None)

Class that stores the information found in the data section of a DRX frame. All three fields listed in the DP ICD version H are stored.

Note

Although this version of FrameData does not fully support the DRX data format change, the central frequency of the DRX tuning can be obtained via: >>> from lsl.common.dp import fS >>> freq = fS*((FrameDataInstance.flags >> 32) & (2**32-1))/2**32

setCentralFreq(centralFreq)

Function to set the central frequency of the DRX data in Hz.

setGain(gain)

Function to set the gain of the DRX data.

class lsl.reader.drx.Frame(header=None, data=None)

Class that stores the information contained within a single DRX frame. It’s properties are FrameHeader and FrameData objects.

getFilterCode()

Convenience wrapper for the Frame.FrameHeader.getFilterCode function.

getSampleRate()

Convenience wrapper for the Frame.FrameHeader.getSampleRate function.

getTime()

Function to convert the time tag from samples since the UNIX epoch (UTC 1970-01-01 00:00:00) to seconds since the UNIX epoch.

parseID()

Convenience wrapper for the Frame.FrameHeader.parseID function.

setCentralFreq(centralFreq)

Convenience wrapper for the Frame.FrameData.setCentralFreq function.

setGain(gain)

Convenience wrapper for the Frame.FrameData.setGain function.

class lsl.reader.drx.ObservingBlock(x1=None, y1=None, x2=None, y2=None)

Class that stores all frames associates with a particular beam at a particular time.

getFilterCode()

Convenience wrapper for the Frame.FrameData.getFilterCode function.

getTime()

Convenience wrapper for the Frame.FrameData.getTime function.

setCentralFreq(centralFreq)

Convenience wrapper for the Frame.FrameData.setCentralFreq function.

setGain(gain)

Convenience wrapper for the Frame.FrameData.setGain function.

lsl.reader.drx.readFrame(filehandle, CentralFreq=None, Gain=None, Verbose=False)

Function to read in a single DRX frame (header+data) and store the contents as a Frame object. This function wraps readerHeader and readData.

lsl.reader.drx.readBlock(filehandle)

Function to read in a single DRX block (four frames) and store the contents as a ObservingBlock object. This function wraps readFrame.

lsl.reader.drx.getSampleRate(filehandle, nFrames=None, FilterCode=False)

Find out what the sampling rate/filter code is from a single observations. By default, the rate in Hz is returned. However, the corresponding filter code can be returned instead by setting the FilterCode keyword to true.

This function is included to make easier to write code for TBN analysis and modify it for DRX data.

lsl.reader.drx.getBeamCount(filehandle)

Find out how many beams are present by examining the first 16 DRX records. Return the number of beams found.

lsl.reader.drx.getFramesPerObs(filehandle)

Find out how many frames are present per beam by examining the first 16 DRX records. Return the number of frames per observations as a four- element tuple, one for each beam.

Reader Error Codes

Module that contains the error classes for the DRX, TBN, and TBW readers. These errors are currently meant to deal with file I/O problems.

exception lsl.reader.errors.baseReaderError(strerror, errno='-1')

Base class for file I/O problem during numpy.fromfile calls and out-of- sync Mark5C headers.

exception lsl.reader.errors.eofError

Extension to the base class for dealing with EOF errors. The error code is 1.

exception lsl.reader.errors.numpyError

Extension to the base class for dealing with errors that occur when numpy.fromfile tried to read more data can exists. This is a specialized form of EOF error. The error code is 2.

exception lsl.reader.errors.syncError(sync1=None, sync2=None, sync3=None, sync4=None)

Extension to the base class for dealing with Mark 5C header sync word problems. If the sync word doesn’t match what is expected. The error code is 3.

exception lsl.reader.errors.notTBNError

Extenstion to the base class for dealing with trying to read in TBW data with a TBN reader. The error code is 4.

exception lsl.reader.errors.notTBWError

Extenstion to the base class for dealing with trying to read in TBN data with a TBW reader. The error code is 5.

lsl.reader.errors.listErrorCodes(errno=None)

Function to provide a list of errors defined in this file. It alternatively takes an error code using the ‘errno’ keyword and returns its description.

Direct DRSU Access

Module to provide direct access to the files stored on a DRSU.

Warning

Direct access to DRSU is currently an experimental feature that has only been tested to a limited extent. The direct access module also only compiles on Linux systems and may not work with 32-bit Linux installation.

class lsl.reader.drsu.File(device, name, size, ctime=-1, mtime=-1, atime=-1, mode=None)

Object to provide direct access to a file stored on a DRSU. The File object supports:

• open
• seek
• tell
• read
• close

close()

Close the file object.

getFilehandle()

Return the underlying file object associated with an open file. None is returned if the file isn’t open.

open()

Open the ‘file’ on the device and ready the File object for reading.

read(size=0)

Read in a specified amount of data.

seek(offset, whence=0)

Seek in the file. All three ‘whence’ modes are supported.

tell()

Return the current position in the file by using the internal ‘bytesRead’ attribute value.

lsl.reader.drsu.listFiles(device)

Function to return a list of File instances describing the files on a the specified DRSU device.

Note

Currently, the user needs to have read/write privileges to the device in question. This typically means running the script calling this function via sudo.

lsl.reader.drsu.getFileByName(device, filename)

Function to return a File instance corresponding to the specified filename on the provided device. None is returned if the filename does not exists on the device.

lsl.reader.drsu.shepherdedReadFrame(File, reader)

Given a (open) File object and a reader module (e.g., lsl.reader.tbn), read in a single Frame instance. This function wraps the reader’s readFrame method and ‘shepherds’ the reading such that the file size specified by the File.size attribute is enforced on the reading process. This ensures that data beyond the ‘official’ end of the file are not read in.

Note

Currently, the user needs to have read/write privileges to the device in question. This typically means running the script calling this function via sudo.

Reader Ring Buffer

Buffer for dealing with out-of-order/missing frames.

Changed in version 0.5: Removed support for DRX FrameBuffers since they shouldn’t be needed.

Changed in version 0.6: Removed support for TBW FrameBuffers since they didn’t really work.

class lsl.reader.buffer.FrameBuffer(mode='TBN', stands=[], pols=[], samples=12000000, bits=12, nSegments=6, ReorderFrames=False)

Bases: object

Frame buffer for re-ordering TBW and TBN frames in time order. This class is filled with frames and a returns a frame list when the ‘nSegments’ starts filling. In that case, the oldest segment is returned.

The buffer also keeps track of what has already been read out so that tardy frames are just dropped. For buffers that are read out, missing frames are replaced with frames filled with zeros.

Note

Due to the nature of the buffer, it is possible that there will still be ‘nSegments’-1 segements in the buffer that are either full or partially full. This can be retrieved using the buffer’s ‘flush()’ function.

Note

DRX should _not_ need a ring buffer since the four beam outputs are, effectively, on different networks.

append(frame)

Append a new frame to the buffer with the appropriate time tag. True is returned if the frame was added to the buffer and False if the frame was dropped because it belongs to a buffer that has already been returned.

figureOfMerit(frame)

Figure of merit for storing/sorting frames in the ring buffer.

This will be overridden by sub-classes of FrameBuffer.

flush()

Return a list of lists containing all remaining frames in the buffer from buffers that are considered ‘full’. Afterwards, delete all buffers. This is useful for emptying the buffer after reading in all of the data.

Note

It is possible for this function to return list of packets that are mostly invalid.

get()

Return a list of frames that consitute a ‘full’ buffer. Afterwards, delete that buffer and mark it as closed so that any missing frames that are recieved late are dropped. If none of the buffers are ready to be dumped, None is returned.

status()

Print out the status of the buffer. This contains information about:

1. The current buffer fill level
2. The numer of full and partial buffer dumps preformed
3. The number of missing frames that fill packets needed to be created for
4. The number of frames that arrived too late to be incorporated into one of the ring buffers

class lsl.reader.buffer.TBNFrameBuffer(stands=[], pols=[, 0, 1], nSegments=20, ReorderFrames=False)

Bases: lsl.reader.buffer.FrameBuffer

A sub-type of FrameBuffer specifically for dealing with TBN frames. See lsl.reader.buffer.FrameBuffer for a description of how the buffering is implemented.

Keywords: stands

list of stand to expect packets for

pols

list of polarizations to expect packets for

nSegments

number of ring segments to use for the buffer (default is 20)

ReorderFrames

whether or not to reorder frames returned by get() or flush() by stand/polarization (default is False)

The number of segements in the ring can be converted to a buffer time in seconds:

Segments	TBN Filter Code
	1	2	3	4	5	6	7
10	5.1	1.6	0.8	0.4	0.2	0.1	0.05
20	10.2	3.3	1.6	0.8	0.4	0.2	0.10
30	15.4	4.9	2.5	1.2	0.6	0.3	0.15
40	20.5	6.6	3.3	1.6	0.8	0.4	0.20
50	25.6	8.2	4.1	2.0	1.0	0.5	0.26
100	51.2	16.4	8.2	4.1	2.0	1.0	0.51

figureOfMerit(frame)

Figure of merit for sorting frames. For TBN it is:

<frame count of frame>