Something Scary at the PNSN just before Halloween

Something Scary at the PNSN just before Halloween

October 29, 2018

by Steve Malone

The computer staff at the PNSN has been working hard over the past couple of years and very hard in the past few months to switch our ANSS Quake Monitoring System (AQMS) to more modern hardware and software.  Developed primarily in California over 10 years ago AQMS is now used by most seismic networks within the US.  It was adopted by the PNSN in 2011 and has been our main system since then, running on the same hardware and with few significant changes in the software.  All good things need to evolve and improve.  In particular the trend to increasingly use Open Software and standardized hardware means a faster and easier-to-maintain system.

 

On Oct 30 we will be switching from our legacy AQMS system to the new Linux based AQMS system, and with luck no one outside the PNSN staff will know it happened.  From the outside world (our web page, pnsn.org, our catalogs and all other products) everything should look pretty much the same.  There may be minor formatting differences or map bases, but mostly our web pages will look and feel just as before.  However, results should be available quite a bit faster, and things should be easier for those of us behind the scenes running the operation. 

 

Of course, if the Halloween gremlins are out a day early, and a glitch gets tossed into the works then things may get scary.  But don’t worry.  The legacy system will stay running behind the scenes, and if the goblins start dancing on the new system we can easily revert back with only a bit of embarrassment resulting.  In fact, we will keep the legacy system running for at least a few weeks if not months just as a check on the new system.

 

This post will be updated after Halloween with news of how the switch went and with any new features or bugs we find that others should be made aware of.


Nov. 8, 2018 - Not scary at all

The switch of hardware and software for the PNSN ANSS Quake Monitoring System (AQMS) went off with only a few minor glitches.  The process of transfering all of the past data from the old to the new system took some time, but once complete we were up and running almost as before.  At first things were quite slow when using the analysis tools but after some adjustments to the database those things speeded up.  So far, over a week after the switch, most all of the minor problems have been solved and we are running as smooth or maybe better than before.  In a later post we may give some of the nitty-gritty about the new system but for now, the crew is taking a well earned break from the hard work put in to make the switch.


Nov 23, 2018 - A bit of a late scare

A couple of days ago in the middle of the day all of a sudden all of our pagers and phones and e-mail started going off with messages one after another about a magnitude 4 earthquakes in the Olympic mountains.  Indeed, there had been one the night before that was handled by the computer system and our staff with only minor glitches.  Here it was 12 hours later with repeated, insistent messages about the same event suddenly being broadcast over and over again.  Something was wrong.  Digging into the guts of our alamring system indeed there was a subtle bug that given a certain combination of updates and requests of the system sent it into hyperventalating a stream of alert messages to our staff pagers and phones.  Embarassingly enough many e-mails also were sent to many of our clients and stake holders.  We quickly turned off the alerting system and monitored things by hand for a while.  After diagnostics, fixing the bug and doing multiple tests we are again running normally with a hope that all is well.  But, we are keeping a close watch on things to make sure nothing else sneaks up on us.