Defining Observations and Observation Metadata

Session Definition Files

Module that contains all of the relevant class to build up a representation of a session definition file as defined in MCS0030v5. The hierarchy of classes is:

• Project - class that holds all of the information about the project (including the observer) and one or more sessions. Technically, a SD file has only one session but this approach allows for the generation of multiple SD files from on Project object.

• Observer - class that hold the observer’s name and numeric ID

• Session - class that holds all of the observations associated with a particular DP output.

• Observations - class that hold information about a particular observation. It includes a variety of attributes that are used to convert human-readable inputs to SDF data values. The observation class is further subclasses into: - TBW - class for TBW observations - TBN - class for TBN observations - DRX - class for general DRX observation, with sub-classes:

  • Solar - class for solar tracking
  • Jovian - class for Jovian tracking

  • Stepped - class for stepped observations

• BeamStep - class that holds the information about a particular step in a Stepped Observation

All of the classes, except for Stepped and BeamStep, are complete and functional. In addition, most class contain ‘validate’ attribute functions that can be used to determine if the project/session/observation are valid or not given the constraints of the DP system.

In addition to providing the means for creating session definition files from scratch, this module also includes a simple parser for SD files.

Changed in version 1.0.0: Added the getObservationStartStop() function. Renamed parseTimeString() to parseTime() parseTime() can now accept dates/times as timezone-aware datetime instances Observations can now be initialized with durations as timedelta instances Observations can now be initialized with RA/dec/az/alt as ephem.hours and ephem.degrees instances

Session Structure

lsl.common.sdf provides a means to represent a set of observations as Python objects. For each lsl.common.sdf.Project, there is:

1. An observer (lsl.common.sdf.Observer)
2. The project office comments (lsl.common.sdf.ProjectOffice)
3. A single session that defines the SDF (lsl.common.sdf.Session)

The session contains one or more observerions (lsl.common.sdf.Observation). Each observing mode supported by the LWA is sub-classed (see below).

class lsl.common.sdf.Project(observer, name, id, sessions=None, comments=None, projectOffice=None)

Class to hold all the information about a specific session for a project/proposal.

append(newSession)

Add a new Session to the list of sessions.

render(session=0, verbose=False)

Create a session definition file that corresponds to the specified session. Returns the SD file’s contents as a string.

update()

Update the various sessions that are part of this project.

validate(verbose=False)

Examine all of the sessions and all of their observations to check for validity. If everything is valid, return True. Otherwise, return False.

class lsl.common.sdf.Observer(name, id, first=None, last=None)

Class to hold information about an observer.

class lsl.common.sdf.ProjectOffice(project=None, sessions=None, observations=None)

Class to hold comments from the LWA object office. This class isn’t really needed to create SD files, but it is helpful for parsing SD files.

class lsl.common.sdf.Session(name, id, observations=None, dataReturnMethod='DRSU', comments=None)

Class to hold all of the observations in a session.

append(newObservation)

Add a new Observation to the list of observations.

setConfigurationAuthority(value)

Set the configuration request authority to a particular value in the range of 0 to 65,535. Higher values provide higher authority to set FEE and ASP parameters.

setDRXBeam(value)

Set the beam to use in the range of 1 to 4 or -1 to let MCS decide.

setMIBRecordInterval(component, interval)

Set the record interval for one of the level-1 subsystems (ASP, DP_, etc.) to a particular value in minutes. A KeyError is raised if an invalid sub-system is specified.

Special Values are:

• -1 = use the MCS default interval
• 0 = never record the MIB entries (the entries are still updated, however)

setMIBUpdateInterval(component, interval)

Set the update interval for one of the level-1 subsystems (ASP, DP_, etc.) to a particular value in minutes. A KeyError is raised if an invalid sub-system is specified.

Special Values are:

• -1 = use the MCS default interval
• 0 = request no updates to the MIB entries

setSpectrometerChannels(value)

Set the number of spectrometer channels to generate, 0 to disable.

setSpectrometerIntegration(value)

Set the number of spectrometer FFT integrations to use, 0 to disable.

setSpectrometerMetatag(value)

Set the spectrometer metatag, ‘’ to disable.

update()

Update the various observations in the session.

validate(verbose=False)

Examine all of the observations associated with the session to check for validity. If everything is valid, return True. Otherwise, return False.

class lsl.common.sdf.Observation(name, target, start, duration, mode, ra, dec, frequency1, frequency2, filter, gain=-1, MaxSNR=False, comments=None)

Class to hold the specifics of an observations. It currently handles TBW, TBN, TRK_RADEC, TRK_SOL, TRK_JOV, and Stepped

Changed in version 1.0.0: Added support for RA/dec values as ephem.hours/ephem.degrees instances

computeVisibility(station=<ephem.Observer date='2016/12/12 20:05:45' epoch='2000/1/1 12:00:00' lon=-107:37:40.8 lat=34:04:12.0 elevation=2133.6m horizon=0:00:00.0 temp=15.0C pressure=0.0mBar>)

Place holder for functions that return the fractional visibility of the target during the observation period.

estimateBytes()

Place holder for functions that return the estimate size of the data set being defined by the observation.

getBeamType()

Return a valid value for beam type based on whether maximum S/N beam forming has been requested.

getDuration()

Parse the self.duration string with the format of HH:MM:SS.SSS to return the number of milliseconds in that period.

getFixedBody()

Place holder for functions that return ephem.Body objects (or None) that define the pointing center of the observation.

getFrequency1()

Return the number of “tuning words” corresponding to the first frequency.

getFrequency2()

Return the number of “tuning words” corresponding to the second frequency.

getMJD()

Return the modified Julian Date corresponding to the date/time of the self.start string.

getMPM()

Return the number of milliseconds between the date/time specified in the self.start string and the previous UT midnight.

setStart(start)

Set the observation start time.

update()

Update the computed parameters from the string values.

validate(verbose=False)

Place holder for functions that evaluate the observation and return True if it is valid, False otherwise.

Observing Modes

class lsl.common.sdf.TBW(name, target, start, samples, bits=12, comments=None)

Sub-class of Observation specifically for TBW observations. It features a reduced number of parameters needed to setup the observation and provides extra information about the number of data bits and the number of samples.

Note

TBW read-out times in ms are calculated using (samples/196000+1)*5000 per MCS

Required Arguments:

• observation name
• observation target
• observation start date/time (UTC YYYY/MM/DD HH:MM:SS.SSS string)
• integer number of samples

Optional Keywords:

• bits - number of data bits (4 or 12)
• comments - comments about the observation

estimateBytes()

Estimate the data volume for the specified type and duration of observations. For TBW:

bytes = samples / samplesPerFrame * 1224 bytes * 260 stands

update()

Update the computed parameters from the string values.

validate(verbose=False)

Evaluate the observation and return True if it is valid, False otherwise.

class lsl.common.sdf.TBN(name, target, start, duration, frequency, filter, gain=-1, comments=None)

Sub-class of Observation specifically for TBN observations. It features a reduced number of parameters needed to setup the observation.

Required Arguments:

• observation name
• observation target
• observation start date/time (UTC YYYY/MM/DD HH:MM:SS.SSS string or timezone- aware datetime instance)
• observation duration (HH:MM:SS.SSS string or timedelta instance)
• observation frequency (Hz)
• integer filter code

Optional Keywords:

• comments - comments about the observation

estimateBytes()

Estimate the data volume for the specified type and duration of observations. For TBN:

bytes = duration * sampleRate / 512 * 1048 bytes * 260 stands * 2 pols.

setDuration(duration)

Set the observation duration.

setFrequency1(frequency1)

Set the frequency in Hz corresponding to tuning 1.

validate(verbose=False)

Evaluate the observation and return True if it is valid, False otherwise.

..note::

This version of sdf allows for TBN tuning between 5 and 93 MHz.

class lsl.common.sdf.DRX(name, target, start, duration, ra, dec, frequency1, frequency2, filter, gain=-1, MaxSNR=False, comments=None)

Required Arguments:

• observation name
• observation target
• observation start date/time (UTC YYYY/MM/DD HH:MM:SS.SSS string or timezone- aware datetime instance)
• observation duration (HH:MM:SS.SSS string or timedelta instance)
• observation RA in hours, J2000.0 or ephem.hours instance
• observation Dec in degrees, J2000.0 or ephem.hours instance
• observation tuning frequency 1 (Hz)
• observation tuning frequency 1 (Hz)
• integer filter code

Optional Keywords:

• MaxSNR - specifies if maximum signal-to-noise beam forming is to be used (default = False)
• comments - comments about the observation

setDec(dec)

Set the pointing Dec.

setRA(ra)

Set the pointing RA.

class lsl.common.sdf.Solar(name, target, start, duration, frequency1, frequency2, filter, gain=-1, MaxSNR=False, comments=None)

Sub-class of DRX specifically for Solar DRX observations. It features a reduced number of parameters needed to setup the observation.

Required Arguments:

• observation name
• observation target
• observation start date/time (UTC YYYY/MM/DD HH:MM:SS.SSS string or timezone- aware datetime instance)
• observation duration (HH:MM:SS.SSS string or timedelta instance)
• observation tuning frequency 1 (Hz)
• observation tuning frequency 1 (Hz)
• integer filter code

Optional Keywords:

• MaxSNR - specifies if maximum signal-to-noise beam forming is to be used (default = False)
• comments - comments about the observation

getFixedBody()

Return an ephem.Body object corresponding to where the observation is pointed. None if the observation mode is either TBN or TBW.

class lsl.common.sdf.Jovian(name, target, start, duration, frequency1, frequency2, filter, gain=-1, MaxSNR=False, comments=None)

Sub-class of DRX specifically for Jovian DRX observations. It features a reduced number of parameters needed to setup the observation.

Required Arguments:

• observation name
• observation target
• observation start date/time (UTC YYYY/MM/DD HH:MM:SS.SSS string or timezone- aware datetime instance)
• observation duration (HH:MM:SS.SSS string or timedelta instance)
• observation tuning frequency 1 (Hz)
• observation tuning frequency 1 (Hz)
• integer filter code

Optional Keywords:

• MaxSNR - specifies if maximum signal-to-noise beam forming is to be used (default = False)
• comments - comments about the observation

getFixedBody()

Return an ephem.Body object corresponding to where the observation is pointed. None if the observation mode is either TBN or TBW.

class lsl.common.sdf.Stepped(name, target, start, filter, steps=None, RADec=True, gain=-1, comments=None)

Sub-class of Observation for dealing with STEPPED-mode observations. It features a reduced number of parameters needed to setup the observation and added support for the individual steps.

Required Arguments:

• observation name
• observation target
• observation start date/time (UTC YYYY/MM/DD HH:MM:SS.SSS string or timezone- aware datetime instance)
• integer filter code

Optional Keywords:

• steps - array of BeamStep objects that specify the different steps
• comments - comments about the observation

append(newStep)

Add a new BeamStep step to the list of steps.

computeVisibility(station=<ephem.Observer date='2016/12/12 20:05:45' epoch='2000/1/1 12:00:00' lon=-107:37:40.8 lat=34:04:12.0 elevation=2133.6m horizon=0:00:00.0 temp=15.0C pressure=0.0mBar>)

Return the fractional visibility of the target during the observation period.

estimateBytes()

Estimate the data volume for the specified type and duration of observations. For DRX:

bytes = duration * sampleRate / 4096 * 4128 bytes * 2 tunings * 2 pols.

getDuration()

Parse the list of BeamStep objects to get the total observation duration as the number of milliseconds in that period.

setBeamDipoleMode(stand, beamGain=0.04, dipoleGain=1.0, pol='X', station=<ephem.Observer date='2016/12/12 20:05:45' epoch='2000/1/1 12:00:00' lon=-107:37:40.8 lat=34:04:12.0 elevation=2133.6m horizon=0:00:00.0 temp=15.0C pressure=0.0mBar>)

Convert the current observation to a ‘beam-dipole mode’ observation with the specified stand. Setting the stand to zero will disable the ‘beam-dipole mode’ for this observation’.

Keywords:

• beamGain - BAM gain to use for each dipole in the beam

  default: 0.04; range: 0.0 to 1.0

• dipoleGain - BAM gain to use for the single dipole

  default: 1.0; range: 0.0 to 1.0

• pol - Polarization to record default: “X”

• station - lsl.common.stations instance to use for mapping

  default: lsl.common.stations.lwa1

update()

Update the computed parameters from the string values.

validate(verbose=False)

Evaluate the observation and return True if it is valid, False otherwise.

class lsl.common.sdf.BeamStep(c1, c2, duration, frequency1, frequency2, RADec=True, MaxSNR=False, SpecDelays=None, SpecGains=None)

Class for holding all of the information (pointing center, tuning frequencies, etc.)associated with a particular step.

Required Keywords:

• pointing coordinate 1 (RA [hours] or azimuth [degrees] or ephem.hours/ephem.degrees instance)
• pointing coordinate 2 (dec or elevation/altitude [degrees] or ephem.degrees instance)
• observation duration (HH:MM:SS.SSS string or timedelta instance)
• observation tuning frequency 1 (Hz)
• observation tuning frequency 1 (Hz)

Optional Keywords:

• RADec - whether the coordinates are in RA/Dec or Az/El pairs (default=RA/Dec)
• MaxSNR - specifies if maximum signal-to-noise beam forming is to be used (default = False)
• SpecDelays - 520 list of delays to apply for each antenna
• SpecGains - 260 by 2 by 2 list of gains ([[XY, XY], [YX, YY]]) to apply for each antenna

Note

If SpecDelays is specified, SpecGains must also be specified. Specifying both SpecDelays and SpecGains overrides the MaxSNR keyword.

Changed in version 1.0.0: Added support for azimuth/altitude and RA/dec values as ephem.hours/ephem.degrees instances

getBeamType()

Return a valid value for beam type based on whether maximum S/N beam forming has been requested.

getDuration()

Parse the self.duration string with the format of HH:MM:SS.SSS to return the number of milliseconds in that period.

getFixedBody()

Return an ephem.Body object corresponding to where the observation is pointed. None if the observation mode is either TBN or TBW.

getFrequency1()

Return the number of “tuning words” corresponding to the first frequency.

getFrequency2()

Return the number of “tuning words” corresponding to the second frequency.

setC1(c1)

Set the pointing c1.

setC2(c2)

Set the pointing c2

setDuration(duration)

Set the observation duration.

setFrequency1(frequency1)

Set the frequency in Hz corresponding to tuning 1.

setFrequency2(frequency2)

Set the frequency in Hz correpsonding to tuning 2.

update()

Update the settings.

validate(verbose=False)

Evaluate the step and return True if it is valid, False otherwise.

MCS Metadata Tarball Utilities

Module for working with an MCS meta-data tarball and extracting the useful bits out it and putting those bits into Python objects, e.g, lsl.common.stations.LWAStation and lsl.common.sdm.SDM.

lsl.common.metabundle.readSESFile(filename)

Read in a session specification file (MCS0030, Section 5) and return the data as a dictionary.

lsl.common.metabundle.readOBSFile(filename)

Read in a observation specification file (MCS0030, Section 6) and return the data as a dictionary.

lsl.common.metabundle.readCSFile(filename)

Read in a command script file (MCS0030, currently undocumented) and return the data as a list of dictionaries.

lsl.common.metabundle.getSDM(tarname)

Given an MCS meta-data tarball, extract the information stored in the dynamic/sdm.dat file and return a lsl.common.sdm.SDM instance describing the dynamic condition of the station.

If a sdm.dat file cannot be found in the tarball, None is returned.

lsl.common.metabundle.getStation(tarname, ApplySDM=True)

Given an MCS meta-data tarball, extract the information stored in the ssmif.dat file and return a lsl.common.stations.LWAStation object. Optionally, update the lsl.common.stations.Antenna instances associated whith the LWAStation object using the included SDM file.

If a ssmif.dat file cannot be found in the tarball, None is returned.

lsl.common.metabundle.getSessionMetaData(tarname)

Given an MCS meta-data tarball, extract the session meta-data file (MCS0030, Section 7) and return a dictionary of observations that contain dictionaries of the OP_TAG (tag), DRSU Barcode (drsu), OBS_OUTCOME (outcome), and the MSG (msg).

Changed in version 0.6.5: Update to the new _metadata.txt format

lsl.common.metabundle.getSessionSpec(tarname)

Given an MCS meta-data tarball, extract the session specification file (MCS0030, Section 5) and return a dictionary of parameters.

lsl.common.metabundle.getObservationSpec(tarname, selectObs=None)

Given an MCS meta-data tarball, extract one or more observation specification file (MCS0030, Section 6) and return a list of dictionaries corresponding to each OBS file. If the selectObs keyword is set to a list of observation numbers, only observations matching the numbers in selectObs are returned.

lsl.common.metabundle.getSessionDefinition(tarname)

Given an MCS meta-data tarball, extract the session specification file, the session meta-data file, and all observation specification files to build up a SDF-representation of the session.

Note

This function returns a full lsl.common.sdf.Project instance with the session in question stored under project.sessions[0] and the observations under project.sessions[0].observations.

lsl.common.metabundle.getCommandScript(tarname)

Given an MCS meta-data tarball, extract the command script and parse it. The commands are returned as a list of dictionaries (one dictionary per command).

lsl.common.metabundle.getASPConfiguration(tarname, which='beginning')

Given an MCS meta-data tarball, extract the ASP MIB contained in it and return a dictionary of values for the filter, AT1, AT2, and ATSplit. The ‘which’ keyword is used to specify whether or not the configuration returned is at the beginning (default) or end of the session.

New in version 0.6.5.

lsl.common.metabundle.getASPConfigurationSummary(tarname, which='beginning')

Similar to getASPConfiguration, but returns only a single value for each of the four ASP paramters: filter, AT, AT2, and ATSplit. The values are based off the mode of the parameter.

New in version 0.6.5.

Supporting Functions

Conversion to/from MJD and MPM

These functions convert Python datetime instances to modified Julian Data (MJD) and milliseconds past midnight (MPM) pairs.

lsl.common.mcs.datetime2mjdmpm(dt)

Convert a UTC datetime instance to a MJD, MPM pair (returned as a two-element tuple).

Based off: http://paste.lisp.org/display/73536

New in version 0.5.2.

lsl.common.mcs.mjdmpm2datetime(mjd, mpm)

Convert a MJD, MPM pair to a UTC-aware datetime instance.

New in version 0.5.2.

Specifiying Delay and Gains for the Digital Processor

These functions are intended to help define observations that are run in Stepped mode with the beamforming method set to “SPEC_DELAYS_GAINS”.

lsl.common.mcs.delaytoMCSD(delay)

Given a delay in ns, convert it to a course and fine portion and into the form expected by MCS in a custom beamforming SDF (little endian 16.12 unsigned integer).

New in version 0.6.3.

lsl.common.mcs.MCSDtodelay(delay)

Given delay value from an OBS_BEAM_DELAY field in a custom beamforming SDF, return the delay in ns.

New in version 0.6.3.

lsl.common.mcs.gaintoMCSG(gain)

Given a gain (between 0 and 1), convert it to a gain in the form expected by MCS in a custom beamforming SDF (little endian 16.1 signed integer).

lsl.common.mcs.MCSGtogain(gain)

Given a gain value from an OBS_BEAM_GAIN field in a custom beamforming SDF, return the decimal equivalent.

New in version 0.6.3.

Interpretting MCS Numeric Codes

These functions convert various MCS numeric codes found in the metatdata into strings.

lsl.common.mcs.status2string(code)

Convert a numerical MCS status code to a string.

lsl.common.mcs.summary2string(code)

Convert a numerical MCS overall status code to an explination.

lsl.common.mcs.sid2string(sid)

Convert a MCS subsystem ID code into a string.

lsl.common.mcs.cid2string(cid)

Convert a MCS command code into a string.

lsl.common.mcs.mode2string(mode)

Convert a MCS numeric observing mode into a string.