obspy.db, obspy-indexer

Hi everyone,

I've been enjoying Obspy over the last few months, and building upon the tools it provides. I've also become very interested in obspy.db and obspy-indexer, but I'm having trouble learning about it from the code and docs. Can anyone help with the following questions?:

1) I want to use obspy-indexer to build an sqlite database from a collection of mseed or sac files, so that I can connect to it using my own tools. Is this (partly) what obspy-indexer does?

2) If so, obspy-indexer doesn't seem to be working for me. I have a collection of ~35 sac files (with minimal headers) ending in ".s", but when I use obspy-indexer, this happens:

[$>] sudo obspy-indexer -d '/path/to/files=*.s'
2011-05-06 14:33:21,490 [INFO] Starting indexer localhost:8081 ...

The command stays open (it is serving a webpage?), and when I point my browser at localhost:8081, I get a blank page with some empty brackets in it. When I connect to and query the indexer.sqlite database, it appears to be empty (although the table metadata/DDL are all there). Am I misunderstanding what's happening?

3) Assuming I get past 1) and 2), can you suggest the best way to connect to indexer.sqlite to use a different (css3.0) schema?

Thanks in advance for all your help. I have found that Obspy developers are working on many of the problems I encounter in my work, so I'm very happy for your efforts (and would love to contribute, if my Python-fu ever becomes good enough).

Best,
Jonathan

Hi Jonathan,

I've been enjoying Obspy over the last few months, and building upon
the tools it provides. I've also become very interested in obspy.db
and obspy-indexer, but I'm having trouble learning about it from the
code and docs. Can anyone help with the following questions?:

I'm really not surprised that you run into issue using obspy.db - mainly
its not yet an "official" extension of ObsPy. It's not documented at all
and has a very weak test suite - however it's used in Munich within our
local Database solution SeisHub (http://www.seishub.org) as an external
indexer of the file based waveform MiniSEED/GSE2 archive. Originally it
was included into the SeisHub database but was separated in order to
have the indexing run on another process/computer.

1) I want to use obspy-indexer to build an sqlite database from a
collection of mseed or sac files, so that I can connect to it using
my own tools. Is this (partly) what obspy-indexer does?

partly correct - obspy.db is suited for indexing all kind of waveform
data but does not give you any convenient tools to extract the indexed
data from the generated database - instead you have to use the
SQLAlchemy module directly which on the other hand gives you all the
flexibility to request whatever you need

Also using SQLAlchemy allows to use all databases supported by
SQLAlchemy - for now we tested it with sqlite and postgresql

2) If so, obspy-indexer doesn't seem to be working for me. I have a
collection of ~35 sac files (with minimal headers) ending in ".s",
but when I use obspy-indexer, this happens:

[$>] sudo obspy-indexer -d '/path/to/files=*.s' 2011-05-06
14:33:21,490 [INFO] Starting indexer localhost:8081 ...

first of all don't quote the values - also try using the verbose flag -v
for more output. I attached some scripts we are using here - they should
give you a hint how to set the parameters correctly.

The command stays open (it is serving a webpage?), and when I point
my browser at localhost:8081, I get a blank page with some empty
brackets in it. When I connect to and query the indexer.sqlite
database, it appears to be empty (although the table metadata/DDL are
all there). Am I misunderstanding what's happening?

The indexer process is supposed to run as a daemon - so it will continue
to crawl the given directories searching for changed files - so you
should move it to the background using "&" and using the logfile option
- see the two start and run scripts.

If you want to index your waveform only once you may use the
"--run_once" flag - the service will quit automatically after it went
through.

Ignore the web interface for now - it's not a functional part yet of the
indexing process but may be explored in the future. (The indexer is
essentially build within the event loop of the Python class
BaseHTTPServer.HTTPServer.)

3) Assuming I get past 1) and 2), can you suggest the best way to
connect to indexer.sqlite to use a different (css3.0) schema?

although I never worked with CSS, I'm pretty sure this won't work - the
tables are hardcoded within obspy.db/db.py
(Home · obspy/obspy Wiki · GitHub) - you
could create the wished CSS schema by generating SQL views - but again I
don't know anything about CSS and its requirements

Thanks in advance for all your help. I have found that Obspy
developers are working on many of the problems I encounter in my
work, so I'm very happy for your efforts (and would love to
contribute, if my Python-fu ever becomes good enough).

Thank you for the cheers - we are always looking forward to any
contributions :wink:

Best regards,
Robert

indexer-start.sh (310 Bytes)

indexer-stop.sh (38 Bytes)

indexer-2007.sh (297 Bytes)

Hello Robert,

Thank you for your helpful and quick response…

  1. If so, obspy-indexer doesn’t seem to be working for me. I have a

collection of ~35 sac files (with minimal headers) ending in “.s”,

but when I use obspy-indexer, this happens:

[$>] sudo obspy-indexer -d ‘/path/to/files=*.s’ 2011-05-06

14:33:21,490 [INFO] Starting indexer localhost:8081 …

first of all don’t quote the values - also try using the verbose flag -v
for more output. I attached some scripts we are using here - they should
give you a hint how to set the parameters correctly.

The indexer works for me now.

  1. Assuming I get past 1) and 2), can you suggest the best way to

connect to indexer.sqlite to use a different (css3.0) schema?

although I never worked with CSS, I’m pretty sure this won’t work - the
tables are hardcoded within obspy.db/db.py
(http://obspy.org/browser/obspy/trunk/obspy.db/obspy/db/db.py) - you
could create the wished CSS schema by generating SQL views - but again I
don’t know anything about CSS and its requirements

I am familiar with SQLAlchemy, and it wouldn’t be hard to write a schema translation script, e.g. “indexer2css.py.” The only concern, of course, is which of the sac/mseed file headers are put into the obspy.db tables. For example, if a SAC file header is fairly complete (e.g. has arrival and origin information), it looks like the tables and fields in indexer.sqlite don’t have definitions for that information. Alternately, ObsPy also makes it easy to write my own parser of Trace object headers. Perhaps it makes the most sense for me to do this, as obspy-indexer seems to be mostly geared towards SeisHub.

Thanks again for all your hard work with ObsPy!

Best,
Jonathan