Defining Observations and Observation Metadata

Session Definition Files

Module that contains all of the relevant class to build up a representation of a session definition file as defined in MCS0030v5. The hierarchy of classes is:

  • Project - class that holds all of the information about the project (including the observer) and one or more sessions. Technically, a SD file has only one session but this approach allows for the generation of multiple SD files from on Project object.

  • Observer - class that hold the observer’s name and numeric ID

  • Session - class that holds all of the observations associated with a particular DP output.

  • Observations - class that hold information about a particular observation. It includes a variety of attributes that are used to convert human-readable inputs to SDF data values. The observation class is further subclasses into: - TBW - class for TBW observations - TBN - class for TBN observations - DRX - class for general DRX observation, with sub-classes:

    • Solar - class for solar tracking
    • Jovian - class for Jovian tracking
    • Stepped - class for stepped observations
  • BeamStep - class that holds the information about a particular step in a Stepped Observation

All of the classes, except for Stepped and BeamStep, are complete and functional. In addition, most class contain ‘validate’ attribute functions that can be used to determine if the project/session/observation are valid or not given the constraints of the DP system.

In addition to providing the means for creating session definition files from scratch, this module also includes a simple parser for SD file. It is mostly complete but does not currently support some of the extended session/observation parameters.

Also included as part of this module are utilities to convert delays (in ns) and gains into the data formated expected by DP and required for MCS0030v5

Session Structure

lsl.common.sdf provides a means to represent a set of observations as Python objects. For each lsl.common.sdf.Project, there is:

  1. An observer (lsl.common.sdf.Observer)
  2. The project office comments (lsl.common.sdf.ProjectOffice)
  3. A single session that defines the SDF (lsl.common.sdf.Session)

The session contains one or more observerions (lsl.common.sdf.Observation). Each observing mode supported by the LWA is sub-classed (see below).

class lsl.common.sdf.Project(observer, name, id, sessions=None, comments=None, projectOffice=None)

Class to hold all the information about a specific session for a project/proposal.

render(session=0, verbose=False)

Create a session definition file that corresponds to the specified session. Returns the SD file’s contents as a string.

validate(verbose=False)

Examine all of the sessions and all of their observations to check for validity. If everything is valid, return True. Otherwise, return False.

class lsl.common.sdf.Observer(name, id, first=None, last=None)

Class to hold information about an observer.

class lsl.common.sdf.ProjectOffice(project=None, sessions=None, observations=None)

Class to hold comments from the LWA object office. This class isn’t really needed to create SD files, but it is helpful for parsing SD files.

class lsl.common.sdf.Session(name, id, observations=None, dataReturnMethod='DRSU', comments=None)

Class to hold all of the observations in a session.

setConfigurationAuthority(value)

Set the configuration request authority to a particular value in the range of 0 to 65,535. Higher values provide higher authority to set FEE and ASP parameters.

setDRXBeam(value)

Set the beam to use in the range of 1 to 4 or -1 to let MCS decide.

setMIBRecordInterval(component, interval)

Set the record interval for one of the level-1 subsystems (ASP, DP_, etc.) to a particular value in minutes. A KeyError is raised if an invalid sub-system is specified.

Special Values are:
  • -1 = use the MCS default interval
  • 0 = never record the MIB entries (the entries are still updated, however)
setMIBUpdateInterval(component, interval)

Set the update interval for one of the level-1 subsystems (ASP, DP_, etc.) to a particular value in minutes. A KeyError is raised if an invalid sub-system is specified.

Special Values are:
  • -1 = use the MCS default interval
  • 0 = request no updates to the MIB entries
setSpectrometerChannels(value)

Set the number of spectrometer channels to generate, 0 to disable.

setSpectrometerIntegration(value)

Set the number of spectrometer FFT integrations to use, 0 to disable.

validate(verbose=False)

Examine all of the observations associated with the session to check for validity. If everything is valid, return True. Otherwise, return False.

class lsl.common.sdf.Observation(name, target, start, duration, mode, ra, dec, frequency1, frequency2, filter, MaxSNR=False, comments=None)

Class to hold the specifics of an observations. It currently handles TBW, TBN, TRK_RADEC, TRK_SOL, TRK_JOV, and Stepped

computeVisibility(station=<lsl.common.stations.LWAStation object at 0x50f8650>)

Place holder for functions that return the fractional visibility of the target during the observation period.

estimateBytes()

Place holder for functions that return the estimate size of the data set being defined by the observation.

getBeamType()

Return a valid value for beam type based on whether maximum S/N beam forming has been requested.

getDuration()

Parse the self.duration string with the format of HH:MM:SS.SSS to return the number of milliseconds in that period.

getFixedBody()

Place holder for functions that return ephem.Body objects (or None) that define the pointing center of the observation.

getFrequency1()

Return the number of “tuning words” corresponding to the first frequency.

getFrequency2()

Return the number of “tuning words” corresponding to the second frequency.

getMJD()

Return the modified Julian Date corresponding to the date/time of the self.start string.

getMPM()

Return the number of milliseconds between the date/time specified in the self.start string and the previous UT midnight.

update()

Update the computed parameters from the string values.

validate(verbose=False)

Place holder for functions that evaluate the observation and return True if it is valid, False otherwise.

Observing Modes

class lsl.common.sdf.TBW(name, target, start, samples, bits=12, comments=None)

Sub-class of Observation specifically for TBW observations. It features a reduced number of parameters needed to setup the observation and provides extra information about the number of data bits and the number of samples.

Note

TBW read-out times in ms are calculated using (samples/196000+1)*5000 per MCS

Required Arguments:
  • observation name
  • observation target
  • observation start date/time (UTC YYYY/MM/DD HH:MM:SS.SSS string)
  • integer number of samples
Optional Keywords:
  • bits - number of data bits (4 or 12)
  • comments - comments about the observation
estimateBytes()

Estimate the data volume for the specified type and duration of observations. For TBW:

bytes = samples / samplesPerFrame * 1224 bytes * 260 stands

validate(verbose=False)

Evaluate the observation and return True if it is valid, False otherwise.

class lsl.common.sdf.DRX(name, target, start, duration, ra, dec, frequency1, frequency2, filter, MaxSNR=False, comments=None)

Sub-class of Observation specifically for DRX observations.

Required Arguments:
  • observation name
  • observation target
  • observation start date/time (UTC YYYY/MM/DD HH:MM:SS.SSS string)
  • observation duration (HH:MM:SS.SSS string)
  • observation RA in hours, J2000.0
  • observation Dec in degrees, J2000.0
  • observation tuning frequency 1 (Hz)
  • observation tuning frequency 1 (Hz)
  • integer filter code
Optional Keywords:
  • MaxSNR - specifies if maximum signal-to-noise beam forming is to be used (default = False)
  • comments - comments about the observation
computeVisibility(station=<lsl.common.stations.LWAStation object at 0x50f8650>)

Return the fractional visibility of the target during the observation period.

estimateBytes()

Estimate the data volume for the specified type and duration of observations. For DRX:

bytes = duration * sampleRate / 4096 * 4128 bytes * 2 tunings * 2 pols.

getFixedBody()

Return an ephem.Body object corresponding to where the observation is pointed. None if the observation mode is either TBN or TBW.

validate(verbose=False)

Evaluate the observation and return True if it is valid, False otherwise.

class lsl.common.sdf.Solar(name, target, start, duration, frequency1, frequency2, filter, MaxSNR=False, comments=None)

Sub-class of DRX specifically for Solar DRX observations. It features a reduced number of parameters needed to setup the observation.

Required Arguments:
  • observation name
  • observation target
  • observation start date/time (UTC YYYY/MM/DD HH:MM:SS.SSS string)
  • observation duration (HH:MM:SS.SSS string)
  • observation tuning frequency 1 (Hz)
  • observation tuning frequency 1 (Hz)
  • integer filter code
Optional Keywords:
  • MaxSNR - specifies if maximum signal-to-noise beam forming is to be used (default = False)
  • comments - comments about the observation
getFixedBody()

Return an ephem.Body object corresponding to where the observation is pointed. None if the observation mode is either TBN or TBW.

class lsl.common.sdf.Jovian(name, target, start, duration, frequency1, frequency2, filter, MaxSNR=False, comments=None)

Sub-class of DRX specifically for Jovian DRX observations. It features a reduced number of parameters needed to setup the observation.

Required Arguments:
  • observation name
  • observation target
  • observation start date/time (UTC YYYY/MM/DD HH:MM:SS.SSS string)
  • observation duration (HH:MM:SS.SSS string)
  • observation tuning frequency 1 (Hz)
  • observation tuning frequency 1 (Hz)
  • integer filter code
Optional Keywords:
  • MaxSNR - specifies if maximum signal-to-noise beam forming is to be used (default = False)
  • comments - comments about the observation
getFixedBody()

Return an ephem.Body object corresponding to where the observation is pointed. None if the observation mode is either TBN or TBW.

class lsl.common.sdf.Stepped(name, target, start, filter, steps=[], RADec=True, comments=None)

Sub-class of Observation for dealing with STEPPED-mode observations. It features a reduced number of parameters needed to setup the observation and added support for the individual steps.

Required Arguments:
  • observation name
  • observation target
  • observation start date/time (UTC YYYY/MM/DD HH:MM:SS.SSS string)
  • integer filter code
Optional Keywords:
  • steps - array of BeamStep objects that specify the different steps
  • comments - comments about the observation
append(newStep)

Add a new BeamStep step to the list of steps.

computeVisibility(station=<lsl.common.stations.LWAStation object at 0x50f8650>)

Return the fractional visibility of the target during the observation period.

estimateBytes()

Estimate the data volume for the specified type and duration of observations. For DRX:

bytes = duration * sampleRate / 4096 * 4128 bytes * 2 tunings * 2 pols.

getDuration()

Parse the list of BeamStep objects to get the total observation duration as the number of milliseconds in that period.

validate(verbose=False)

Evaluate the observation and return True if it is valid, False otherwise.

class lsl.common.sdf.BeamStep(c1, c2, duration, frequency1, frequency2, RADec=True, MaxSNR=False, SpecDelays=None, SpecGains=None)

Class for holding all of the information (pointing center, tuning frequencies, etc.)associated with a particular step.

Required Keywords:
  • pointing coordinate 1 (RA or azimuth)
  • pointing coordinate 2 (dec or elevation/altitude) * observation duration (HH:MM.SS.SSS string)
  • observation tuning frequency 1 (Hz)
  • observation tuning frequency 1 (Hz)
Optional Keywords:
  • RADec - whether the coordinates are in RA/Dec or Az/El pairs (default=RA/Dec)
  • MaxSNR - specifies if maximum signal-to-noise beam forming is to be used (default = False)
  • SpecDelays - 520 list of delays to apply for each antenna
  • SpecGains - 260 by 2 by 2 list of gains ([[XY, XY], [YX, YY]]) to apply for each antenna

Note

If SpecDelays is specified, SpecGains must also be specified. Specifying both SpecDelays and SpecGains overrides the MaxSNR keyword.

getBeamType()

Return a valid value for beam type based on whether maximum S/N beam forming has been requested.

getDuration()

Parse the self.duration string with the format of HH:MM:SS.SSS to return the number of milliseconds in that period.

getFixedBody()

Return an ephem.Body object corresponding to where the observation is pointed. None if the observation mode is either TBN or TBW.

getFrequency1()

Return the number of “tuning words” corresponding to the first frequency.

getFrequency2()

Return the number of “tuning words” corresponding to the second frequency.

update()

Update the settings.

validate(verbose=False)

Evaluate the step and return True if it is valid, False otherwise.

MCS Metadata Tarball Utilities

Module for working with a MCS meta-data tarball and extracting the useful bits out it and putting those bits into Python objects, e.g, lsl.common.stations.LWAStation and lsl.common.sdm.SDM.

lsl.common.metabundle.readSESFile(filename)

Read in a session specification file (MCS0030, Section 5) and return the data as a dictionary.

lsl.common.metabundle.readOBSFile(filename)

Read in a observation specification file (MCS0030, Section 6) and return the data as a dictionary.

lsl.common.metabundle.readCSFile(filename)

Read in a command script file (MCS0030, currently undocumented) and return the data as a list of dictionaries.

lsl.common.metabundle.getSDM(tarname)

Given a MCS meta-data tarball, extract the information stored in the dynamic/sdm.dat file and return a lsl.common.sdm.SDM instance describing the dynamic condition of the station.

If a sdm.dat file cannot be found in the tarball, None is returned.

lsl.common.metabundle.getStation(tarname, ApplySDM=True)

Given a MCS meta-data tarball, extract the information stored in the ssmif.dat file and return a lsl.common.stations.LWAStation object. Optionally, update the lsl.common.stations.Antenna instances associated whith the LWAStation object using the included SDM file.

If a ssmif.dat file cannot be found in the tarball, None is returned.

lsl.common.metabundle.getSessionMetaData(tarname)

Given a MCS meta-data tarball, extract the session meta-data file (MCS0030, Section 7) and return a dictionary of observations that contain dictionaries of the OP_TAG (tag), OBS_OUTCOME (outcome), and the MSG (msg).

lsl.common.metabundle.getSessionSpec(tarname)

Given a MCS meta-data tarball, extract the session specification file (MCS0030, Section 5) and return a dictionary of parameters.

lsl.common.metabundle.getObservationSpec(tarname, selectObs=None)

Given a MCS meta-data tarball, extract one or more observation specification file (MCS0030, Section 6) and return a list of dictionaries corresponding to each OBS file. If the selectObs keyword is set to a list of observation numbers, only observations matching the numbers in selectObs are returned.

lsl.common.metabundle.getSessionDefinition(tarname)

Given a MCS meta-data tarball, extract the session specification file, the session meta-data file, and all observation specification files to build up a SDF-representation of the session.

Note

This function returns a full lsl.common.sdf.Project instance with the session in question stored under project.sessions[0] and the observations under project.sessions[0].observations.

lsl.common.metabundle.getCommandScript(tarname)

Given a MCS meta-data tarball, extract the command script and parse it. The commands are returned as a list of dictionaries (one dictionary per command).

Supporting Functions

Conversion to/from MJD and MPM

These functions convert Python datetime instances to modified Julian Data (MJD) and milliseconds past midnight (MPM) pairs.

lsl.common.mcs.datetime2mjdmpm(dt)

Convert a UTC datetime instance to a MJD, MPM pair (returned as a two-element tuple).

Based off: http://paste.lisp.org/display/73536

New in version 0.5.2.

lsl.common.mcs.mjdmpm2datetime(mjd, mpm)

Convert a MJD, MPM pair to a UTC-aware datetime instance.

New in version 0.5.2.

Specifiying Delay and Gains for the Digital Processor

These functions are intended to help define observations that are run in Stepped mode with the beamforming method set to “SPEC_DELAYS_GAINS”.

lsl.common.dp.delaytoDPD(delay)

Given a delay in ns, convert it to a course and fine portion and into the final format expected by DP (big endian 16.12 unsigned integer).

lsl.common.dp.DPDtodelay(combined)

Given a delay value in the final format expect by DP, return the delay in ns.

lsl.common.dp.gaintoDPG(gain)

Given a gain (between 0 and 1), convert it to a gain in the final form expected by DP (big endian 16.1 signed integer).

lsl.common.dp.DPGtogain(combined)

Given a gain value in the final format expected by DP, return the gain as a decimal value (0 to 1).

Interpretting MCS Numeric Codes

These functions convert various MCS numeric codes found in the metatdata into strings.

lsl.common.mcs.status2string(code)

Convert a numerical MCS status code to a string.

lsl.common.mcs.summary2string(code)

Convert a numerical MCS overall status code to an explination.

lsl.common.mcs.sid2string(sid)

Convert a MCS subsystem ID code into a string.

lsl.common.mcs.cid2string(cid)

Convert a MCS command code into a string.

lsl.common.mcs.mode2string(mode)

Convert a MCS numeric observing mode into a string.