SUNWORM

SUN based seismic data acquisition and processing system

SUNWORM is a cross between HAWK and EARTHWORM

Steve Malone (steve@ess.washington.edu)
Dept. of Earth and Space Sciences, UofW
Version 1.4, Nov 26, 1995


OVERVIEW

SUNWORM is a series of programs running on a SUN SPARCStation with an Analyx A/D S-Bus (model ADDA-1418-166 or AD-1416-166) board which will digitize up to 256 channels of seismic data at nominally 100 samples per second per channel and save time segments of seismic interest (triggered events) to UNIX files in the UW2 trace format. Follow-on programs do automatic processing of these event files to determine preliminary earthquake locations and magnitudes and distribute these results to the seismological community. Data from other sources such as digital telemetered data arriving asynchronously can be merged into the UW2 files in near real-time so that an analyst sees all traces associated with an event. SUNWORM is based on the EARTHWORM concept developed by Alex Bittenbinder and Carl Johnson which uses Ethernet broadcast messages of waveform data by a PC-based data acquisition system and listener processes running on other PCs to process the waveforms. In the SUNWORM model shared memory within a single SPARCStation takes the place of the EARTHWORM Ethernet and standard UNIX Inter-Process Communication (IPC) facilities provide the effective function of a broadcast/listener data exchange. The basis of both systems is that a producer of data simply broadcasts the data for others to listen to and never needs to hear back from any of the listeners. The producer process for a data stream only needs the data source and from none to many listeners may receive the data without the producer process knowing or caring.

The current SUNWORM system at the University of Washington acquires data from an analog telemetered short-period network (160 channels), a digital telemetered broad-band station (3 channels), two USNSN VSAT telemetered broad-band station (3 channels), and six dial-up broad-band stations. Accurate timing is assured by using four independent time sources (Computer clock locked to Internet NTP time, recorded WWVB, recorded time-code generator, and recorded GPS pulses) and a program to compare them. Data from four dial-up broad-band stations are acquired using a modified version of the IRIS SPYDER® system which extracts time-windows of data based on event location and size from preliminary analysis. The system provides initial pager alarms within ~40 seconds of origin time of larger events (M>2.8) and updated alarms and e-mail at the conclusion of event recording (as much as 7 minutes following origin time). Within a minute of event termination, real-time data from all streams are merged into a UW2 format data file on which an analyst can use the picking program, xped to review and modify picks on all data streams (dial-up stations which may take several 10s of minutes to acquire). All waveform and derived data are archived locally and converted to SEED format for archiving at the IRIS Data Management Center.

SUNWORM EXECUTABLES

Currently there are a number of processes associated with the SUNWORM, of which most run on a single SPARCStation. One other process runs as a watch-dog alarm process on a different SPARCStation and yet another collects data from a digital telemetry receiver running on a third SPARCStation. Figure 1 is a block schematic of the main parts of SUNWORM. Manual interaction with the system is done through the program, worm which is described in a later section. The processes currently running or available as part of SUNWORM are the following:

  1. worm_ad- controls the Analyx A/D board, receives driver interrupts when data are ready, generates worm messages of type-5 , time stamps these messages with the UNIX time-of-day clock, and broadcasts these messages. (see below for message types and IPC library information)
  2. worm_bar- listens to the messages of type-5 coming from worm_ad and generates characteristic averages for every channel (short-term and long-term average and rectified average) and broadcasts these as message type-9.
  3. worm_trig- Listens to the messages of type-9 coming from worm_bar and runs a traditional carl_trig algorithm based on subnet and trigger parameters in a configuration file. Trigger-on and trigger-off signals are sent to worm_save using a file named Triggered which contains an event name based on the date-time of the trigger minus pre-event memory (about 20 seconds).
  4. worm_save- Listens to the trace data messages of type-5 and keeps them in a circular buffer nominally 20 seconds long and writes these messages and any additional incoming ones to an event trace data file when the file named, Triggered exists.
  5. worm_proc- Is a shell script which polls for the existence of an event trace data file and then processes that file by first demultiplexing it into the UW-1 trace data format, converts that to UW-2 format, and then runs a snaps like processing script called final_proc to try to get a preliminary event solution for each trigger and farther processing.
  6. keep_alive- Is a shell script which just loops periodically ( 60 seconds) checking on the condition of the other processes and can shut down, clean up, and start up everything if it ever finds any problems with any of the other routines. Any time it does a restart it sends e-mail to the sunworm user and anyone in its .forward file.
  7. worm_check- Is a shell script which runs periodically (every 10 minutes) by a crontab job on a different SPARCStation (currently gwiz) which checks on the keep_alive process running on SUNWORM's main machine (wiggles) and will send e-mail and pager alarms if it sees a problem.
  8. worm_view- Is an X11 program which will display selected channels (up to 48 at once) on a graphics X11 window in real time.
  9. worm_grab- Is a special use program which listens to trace data messages of type-5 and extracts any individual set of channels (including all of them) and save them in a demultiplexed UW-2 format file on a regular interval.
  10. worm_client, worm_serve- Are a pair of programs used to duplicate any worm type message stream from one machine onto another over the Internet. worm_client is given the name of a server machine and the worm message type to receive and it contacts and starts the worm_serve routine on the remote machine.
  11. readref- Is a program which reads telemetered Refraction Technology DAS data being received by a Reftek RT-112 and being acquired by a serial line S-Bus interface card on a SPARCStation (currently quake) and writes these data into a file as well as a generate worm data type-10 for broadcast.
  12. ref_save- Is a program which listens for worm data type-10 from Reftek telemetered stations and saves it in a file for periods when the Triggered file exists.
  13. worm_nsn- Is a program which listens to a serial line from a USNSN VSAT system of continuous loop-back data and extracts channel trace data and saves it into three individual channel files when the Triggered file exists.
  14. trig_nsn- Is a program to listens to a serial line from a USNSN VSAT system containing triggered data. It saves all NSN triggered time segments in a file whose name is based on the start time of the data for later merging with the other data.
  15. bb_proc- Is a script to convert reftek data files to UW2 and merge them with the proper UW2 trace data file. It also finds all NSN data (both continuous and triggered) and converts it to UW2 format and merges them with the appropriate UW2 trace data file.
  16. quick_proc- Is a script run by worm_proc if a preliminary pick file is created by worm_trig in the ~sunworm/GOOD directory containing the preliminary picks and information about an event in progress. It computes a preliminary location and magnitude for alert purposes.

USER COMMANDS and CONTROL FILES

Many of the fundamental parameters of SUNWORM operation are compile time options contained in the sunworm.h file in the source directory. See this file for the details. All other operational parameters and control are in the ~/RUN subdirectory of the sunworm login home directory on the main SPARCStation, wiggles or in the worm_run script file in the ~/BIN subdirectory. Some files in the ~/RUN directory are used to configure or modify a running SUNWORM system; others are used by a running system and should not be changed or deleted by hand. The system needs little or no routine attention other than to review and process the events it detects and records; however, changing the sensitivity of the trigger parameters may be desirable from time to time based on background seismic noise. Changing such parameters or starting and stopping the system are done using the program, worm and files in the ~/RUN directory as described below.

WORM Command

Control of the SUNWORM system is done using the worm command which must be run from within the ~/RUN directory by the sunworm user. This command takes one and only one argument, the choice of which is:

  • start - which will start a complete SUNWORM system running if it is NOT running.
  • restart - which will shutdown, cleanup and restart a complete SUNWORM if it is already running.
  • stop - will shutdown to a completely stopped state a running SUNWORM system.
  • reset - will cause a running SUNWORM system to reread the subnets file for its trigger process (see below).

    Stopping SUNWORM will likely cause the watch-dog alarm (worm_check) running on a different system (gwiz at UofW) to go off and send a message to the seismic pager (currently code 00999999) as well as e-mail to the sunworm user and anyone in its .forward file. If a restart is done fast enough, then this alarm MAY not be triggered since it only looks for problems once every ten minutes.

    WORM_RUN Configuration Script

    The script, ~/BIN/worm_run configures the actual processes which are to be used in the standard system. It should never be run by a user directly. The worm command is used to stop and start things and it uses the worm_run script for the current configuration. This standard configuration can be changed by editting the worm_run script which will take effect the next time the system is started with worm start or worm restart. A knowledgeble user may start and stop other processes by hand; however those started by hand but not configured into the worm_run script will not be running after a restart of the system. If something seems to be wrong with the current run there may be diagnostic information of interest in the wormrun.log file in the ~/RUN directory.

    Trigger sensitivity and subnets: subnets files

    The options having to do with the subnet and trigger parameters are contained in the file, subnets. This file can be edited by hand at any time but only takes effect after executing the command, worm reset. It has the following structure:

  • line 1: Comment line starting with #
  • line 2: Three integer numbers for the trigger parameters, enumer, edenom, equiet for the carl_trig routine
  • line 3-n: Each line defines a subnet where the first number is the subnet number and should be unique in this file. The next number is the number of stations to trigger this subnet. The names of the stations for this subnet follows as a blank delimited list. Only the vertical component of a three-component station is used to trigger.
  • line last: The list is terminated by a line with zero (0).
    The first few lines of an example subnets file:

    # Next line contains only enumer edenom equiet
    9 4 4
    0 3 CRF EPH GBL MDW OT3 RC1Z VT2 WA2 WRD
    1 4 BRV ET3 GBL MDW OT3 PRO RSW WA2 WIW MJ2 WG4
    2 3 ET3 JBO LNO PAT PRO RSW WIW WG4
    .....
    0

    To desensitize a specific subnet one can increase the number of stations need to trigger (2nd value on a subnet line). To decrease the over all sensitivity of all individual station triggers increase the enumer value. The convention is to keep a backup file called subnets.master file which is semi-permanent and if the regular subnets file is changed to desensitize any thing, the subnets.master file can be copied to the subnets file to resensitize to a standard level. Thus when adding or removing stations, both of these files should be changed.

    There is a bug in the worm_trig routine which will cause it to possibly continuously trigger if the number of stations in the subnets list is ever decreased. If the subnets file is changed other than to change the sensitivity the whole SUNWORM system should be restarted with the worm restart command.

    Station Names: sta.list> file

    The file, sta.list in the ~/RUN directory is a mapping of A/D channel numbers to station names. Each line contains six items, the first is arbitrary but is usually the word, station, the next is the A/D channel number (starting at channel 0) and should be unique within this file, the next is the station name or code, followed by the component code and that followed by the channel ID code (should be `_', underscore, if not defined), and the last is the minimum number in seconds that a trigger of this channel should stay active following a transition to its triggered state. This should be at least the time that a seismic wave will take to travel from this station to its nearest several neighbors. This file is read by both the worm_trig process and by the demultiplexing routine run by worm_proc. If stations are removed from this file they MUST also be removed from the subnets files.
    The first few lines of an example sta.list file:
    .nf station 0 TCG TIM _ 0
    station 1 SPW EHZ _ 10
    station 2 GMW EHZ _ 10
    station 3 GSM EHZ _ 10
    ....

    On a temporary basis there is also a station list file, sta.list.old which is used by the routine to convert from UW1 to UW2 after an event is demultiplexed. The demultiplexer is being rewritten to generate UW2 directly, but until this is finished and debugged one must update both of these station files for everything to work properly.

    Other files

    The file in this directory, run_num contains the current run number of the SUNWORM system and is automatically incremented each time the system is started by worm_ad. It should not be changed by hand unless installing a whole new system.

    Some other files in this directory have suffixes of .out which indicate that they are the diagnostic output files of currently running process and probably only have interest if there is trouble. Files with a suffix of .log are the routine logging files of current processes. Most of these type files start over with each run and older versions will have a suffix of their run number added to them automatically when they are no longer current. These older versions may be moved to the directory called ~/LOGS after a new run of SUNWORM is under way. The file, keep_alive.log is never deleted or renamed and contains information about every current and previous run of the SUNWORM system.

    EVENT PROCESSING

    Currently the event processing part of SUNWORM is becoming fairly stable but may continue to evolve over time. The event processing is started by the script, worm_proc which also spins off other scripts which run independently. Currently the worm_proc script runs the program, worm_dux which demultiplexes the basic trace-data saved by worm_save into UW-1 trace data format. The current version of worm_dux uses the file, ~/RUN/sta.list.old to map channel numbers into station names using the UW1 station/channel naming convention. The trace-data are then converted to UW-2 format using uw1-2. A script called fixtime is then run which uses the program, timecode2 to compare timing information contained in the data and possibly change the time in the data headers if needed. This script will also send e-mail to the sunworm user and anyone in its .forward file if something seems to have gone very wrong with the time code systems. After running fix_time, worm_proc synchronizes the pick-file name generated by worm_trig and the data file name so they will be the same, moves them both to the ~/REVIEW directory, and runs the script, final_proc on the pick-file in the background.

    The script final_proc is the current script configured to emulate the snaps processing scheme. The current version does not use cpick since there may still be problems with that program. Picks based on trigger time are used to try and locate the event using both spong2 and teles2. Tests of the solutions are used to determine what type of event it is rather than using aol at this time. If the event is well located and large enough, e-mail of the event solution is sent to the sunworm user as well as alarms using the do_alarm script. Different types of alarms are sent for different events at different times (see the Emergency SUNWORM procedures for details). The ~/REVIEW directory is where the analyst can review the data to determine which events to process and which to discard. Final_proc compresses each event data file and moves it to the /data5/net_ar directory from which it is automatically backed up onto Exebyte tape periodically. The last thing final_proc does is merge in broad-band data with the UW2 trace data using the scripts, bb_proc to merge telemetered RefTek data, and nsn_proc to merge USNSN data and notify Bob Crosson's computer of the existence of the new complete event data.

    EARLY ALERT

    There is an early alert mechanism running now which can provide an alert within about 40 seconds of the origin time of a large earthquake. This is done by having the worm_trig process generate a preliminary pick-file after 40 seconds if more than a certain number of stations have been triggered. This pick-file also contains the integrated LTA of all triggered stations up to the time the file is written. If this integrated LTA value exceeds a threshold and the preliminary pick-file contains a reasonable solution within the seismic network a pager alert is sent using the dobeep program with an approxmate magnitude value and an audio output message is given on several of the lab SPARCStations.

    BACKUP SUNWORM

    The SUNWORM system is now running on a completely separate SPARCStation-1 called waggles as a backup to the main system on wiggles. This backup system only runs the main data collection part of SUNWROM, and does no event processing, alerting, or BB data merging. Data are collected in the ~sunworm/REVIEW directory and the oldest files are removed periodically by a crontab job (every hour) called worm_clean to keep the the disk free space above 200MB.

    Since waggles is only a SPARC-1 wthout a lot of CPU power its data collection duties take up much of its time and so it SHOULD NOT BE USED FOR ANY OTHER PURPOSES SUCH AS DATA PROCESSING. If data need to be recovered from waggles because of a failure or problem with wiggles it should be "rcp'ed" to another machine before analysis.

    DIRECTORY STRUCTURE

    The following directories contain different types of files and information: All are relative to the sunworm user's home directory, /sunworm on the machine called wiggles. UL>

  • RUN Contains the configuration and log files for the current run as well as files used internally by SUNWORM and temporary files used during event processing.
  • BIN Contains all of the executable code and scripts specific to the SUNWORM system. Other programs used for event analysis may be found in ~snaps/bin, ~snaps/Scripts, and ~carl.
  • SRC Contains all the source code for executable programs as well as the compile time configuration file, sunworm.h. Below this directory is one called TRANSPORT which contains the library for the IPC shared memory communications code which emulates the Ethernet broadcast/listener model of the USGS EARTHWORM. This code has been kindly provided by the talented Lynn Dietz working under the direction of the magnificent Alex Bittenbinder, both of the USGS, Menlo Park.
  • REVIEW Contains the trace data and preliminary pick-files after the automatic event processing is finished. A seismic analyst goes through a review process in this directory using the program, xped. A script in this directory called mov_reviewed moves each event's files to other directories for further processing.
  • TRACE Contains the UW-2 format trace data files for all reviewed events which were determined to be of interest. This directory is actually on the file-system called bigdata which is also remote mounted on tic.
  • DELETE Contains the trace data for all events to be discarded. Files in this directory can be removed at any time to make more room on the file system.
  • P/YYMM Is a symbolic link to the current pick-file directory under the ~seis/P area
  • CDATA Is a directory possibly containing continuous data generated by worm_grab in UW2 format. This directory can grow quite rapidly. Previously a crontab job compressed and makes a tar archive in /bigdata/OREGON once a day and then removed the files from CDATA.
  • BACKUP Is a directory where event files are placed in compressed form before moving them to another machine for automatic backup on Exebyte tape.
  • LOGS This directory is an archive of many of older log files.
  • DOC Contains manuals and documents for SUNWORM

    ACKNOWLEDGMENTS

    The fundamental worm idea is due to the dynamic duo of Alex Bittenbinder and Carl Johnson who have designed and redesigned countless seismic data acquisition systems over the past 20 years and should have gotten it right by now. The transport layer code which implements the basic broadcast-listen mode of WORM data messages on a SPARCStation was written by Lyn Dietz of the USGS. The guts of the worm_client/worm_server code was lifted from code kindly provided by Phil Maechling of Caltech. Code to unscramble and decompress USNSN data was provided by Paul O'Keefe of Lamont-Doherty. The UW2 format, and routines for seismic trace display were writen by Bob Crosson and Eric Crosson. The timecode interpretation program as well as several seismic analysis programs were written by Tony Qamar. Sandy Stromme, who maintains the IRIS SPYDER® system wrote the conversion programs for generating SEED data archives from both UW1 and UW2 trace data files and has modified and debugged many other data conversion routines. Ruth Ludwin, Sandra Hebert, and Bill Steele have provided suggestions, and feed-back and have been patient with the problems of living through change. The development of the SUNWORM system and the operation of the Pacific Northwest Sesimograph Network is supported by the US Geological Survey, Westinghouse Hanford Co. (DOE), and the State of Washington.

    Back to:... UW GEOPHYSICS PROGRAM ..... SEISMOLOGY INFO..... PNW EARTHQUAKES

    University of Washington Dept. of Earth and Space Sciences, Box 351310 Seattle, WA, 98195-1310
    This is file /SEIS/PNSN/SHOP/sunworm.html; modified 10/11/96 - If you see any problems e-mail: seis_web@ess.washington.edu