errors in MiniSeed reading - while mseed2sac is OK

ThomasLecocq · February 19, 2014, 12:36pm

Hi all,

I'm encountering errors in reading miniseed data :

    from obspy.core import read
    # 28MB Warning!
    !wget http://geophysique.be/tmp/liege83
    st = read('liege83',format='MSEED')

outputs:

339 values= unpack('%sHHBBBxHHhhBBBxlxxH' % endian, data)

    /home/seisvolc/anaconda/lib/python2.7/site-packages/obspy-0.9.0-py2.7-linux-x86_64.egg/obspy/core/utcdatetime.pyc in__init__(self, *args, **kwargs)
         322 kwargs['second'] = int(args[5])
         323 args= args[0:5]
    --> 324 dt = datetime.datetime(*args, **kwargs)
         325 self._fromDateTime(dt)
         326

TypeError: Required argument 'month' (pos 2) not found

Using mseed2sac on the same machine,

    seisvolc@seisvolc:~/tmp/mseed2sac-2.0$ ./mseed2sac liege83
    GR_GRB2__BHZ_D: Warning: Data integrity check for Steim-1 failed,
    last_data=-132, xn=-127
    Wrote 1728000 samples to GR.GRB4..BHZ.D.1983.312.000000-1.SAC
    Wrote 1728000 samples to GR.GRA2..BHZ.D.1983.312.000000-1.SAC
    Wrote 1728000 samples to GR.GRB2..BHZ.D.1983.312.000000-1.SAC
    Wrote 1728000 samples to GR.GRA3..BHZ.D.1983.312.000000-1.SAC
    Wrote 1728000 samples to GR.GRB3..BHZ.D.1983.312.000000-1.SAC
    Wrote 1728000 samples to GR.GRC3..BHZ.D.1983.312.000000-1.SAC
    Wrote 1728000 samples to GR.GRC1..BHZ.D.1983.312.000000-1.SAC
    Wrote 1728000 samples to GR.GRC4..BHZ.D.1983.312.000000-1.SAC
    Wrote 1728000 samples to GR.GRA1..BHZ.D.1983.312.000000-1.SAC
    Wrote 1728000 samples to GR.GRC2..BHZ.D.1983.312.000000-1.SAC
    Wrote 1728000 samples to GR.GRA4..BHZ.D.1983.312.000000-1.SAC
    Wrote 1728000 samples to GR.GRB5..BHZ.D.1983.312.000000-1.SAC
    Wrote 1728000 samples to GR.GRB1..BHZ.D.1983.312.000000-1.SAC

It seems some errors are caught by libmseed, and mseed2sac passes successfully to process the data, while obspy crashes. For what I can tell, obspy and mseed2sac, both latest versions, use the latest libmseed 2.12.

Other read error, but with different data (steim1, 4096):

    from obspy.core import read
    !wget http://geophysique.be/tmp/gra1_831108_0000.bhz
    st = read('gra1_831108_0000.bhz',format='MSEED')

gives:

    /home/seisvolc/anaconda/lib/python2.7/site-packages/obspy-0.9.0-py2.7-linux-x86_64.egg/obspy/mseed/util.pyc
    in _getRecordInformation(file_object, offset, endian)
         412
         413 info['number_of_records'] = long(info['filesize'] //
    --> 414 info['record_length'])
         415 info['excess_bytes'] = long(info['filesize'] %
    info['record_length'])
         416

KeyError: 'record_length'

which comes from block_size to be 0 just a few lines before.

but mseed2sac gives:

    seisvolc@seisvolc:~/tmp/mseed2sac-2.0$ ./mseed2sac
    /mnt/30years/data/1983/gra1_831108_0000.bhz
    Wrote 1300000 samples to .GRA1..BHZ.D.1983.312.000000.SAC
    Wrote 11200 samples to .GRA1..BHZ.D.1983.312.180330.SAC
    Wrote 3800 samples to .GRA1..BHZ.D.1983.312.181300.SAC
    Wrote 412400 samples to .GRA1..BHZ.D.1983.312.181620.SAC

Which seems to indicate that the trace 1) has no network code, 2) has gaps.

Finally, none of the 2 files above are readable using SeisGram2K60 from A. Lomax.

Any idea, lead to find the error ?

Best regards from Brussels,

Thomas

LionKrischer · February 19, 2014, 1:18pm

Hi Thom,

the first file is actually a tar file (at least in the beginning.) My machine only manages to unpack a single small file from it. It might be worth to try with some really old tar version. But in essence it is not a valid mini seed file. Neither is the second one. This one lacks almost all header values…

I have no clue how mseed2sac manages to read it but maybe it used to be something common so routines to deal with this are included. I would just convert them with mseed2sac and then use ObsPy for the rest. If you can figure out how these files were created we might include some option to read it in ObsPy.

Cheers!

Lion

ThomasLecocq · February 19, 2014, 1:37pm

Hey Lion,

Thanks for this, indeed it is a tar file. It got broken when I uploaded it to geophysique.be, but on my machine I can indeed read it without problems and extract the content. That’s fine then !

For the second type, I’m still trying to figure out how/why that works with mseed2sac…

Cheers

Thom

Sebastien_Bonaime · February 19, 2014, 1:39pm

On my mac, with PQL II version2010.268beta I can read those files but:

liege83 contains only 2x3770 samples
gra1_831108_0000.bhz: no network and 3 gaps

Peter_Danecek1 · February 19, 2014, 1:47pm

Hi Thom,

if you need to inspect `mseed` content, you might want to try out `msi` which is based on libmseed as well. I found it quite helpful mainly to see the exact timing and alignment of data blocs. But it gives you some other info as well (See below). Actually I am just working with it to understand some issue here.

I would like to see some of this functionality in obspy as well at some point (to have a single tool for diagnostics and quality control). But honestly for the moment I have not understood how the libmseed / Python wrapper works exactly ;-(.

~petr

--- snip ---

petr% msi IV.FAGN..HHZ.D.2012.300 |head
IV_FAGN__HHZ, 865997, D, 512, 412 samples, 100 Hz, 2012,300,00:00:02.860000
IV_FAGN__HHZ, 865998, D, 512, 412 samples, 100 Hz, 2012,300,00:00:06.980200
IV_FAGN__HHZ, 865999, D, 512, 412 samples, 100 Hz, 2012,300,00:00:11.100000
IV_FAGN__HHZ, 866000, D, 512, 412 samples, 100 Hz, 2012,300,00:00:15.220000
IV_FAGN__HHZ, 866001, D, 512, 412 samples, 100 Hz, 2012,300,00:00:19.340000
IV_FAGN__HHZ, 866002, D, 512, 412 samples, 100 Hz, 2012,300,00:00:23.460000
IV_FAGN__HHZ, 866003, D, 512, 412 samples, 100 Hz, 2012,300,00:00:27.580200
IV_FAGN__HHZ, 866004, D, 512, 412 samples, 100 Hz, 2012,300,00:00:31.700200
IV_FAGN__HHZ, 866005, D, 512, 412 samples, 100 Hz, 2012,300,00:00:35.820000
IV_FAGN__HHZ, 866006, D, 512, 412 samples, 100 Hz, 2012,300,00:00:39.940000

ThomasLecocq · February 19, 2014, 2:05pm

Hi Peter,

Thanks for the tip.

The difference between the "new" (=extracted from the .tar file) and the "old" files for the same day is surprising (see attached).

It seems that they differ, in addition to the missing network name, from blockette 348 onwards :

extract from the diff :

--- "a/new.txt"
+++ "b/old.txt"
-GR_GRA1__BHZ, 000346, D, 4096, 3772 samples, 20 Hz, 1983,312,17:55:13.400000
-GR_GRA1__BHZ, 000347, D, 4096, 3772 samples, 20 Hz, 1983,312,17:58:22.000000
-GR_GRA1__BHZ, 000348, D, 4096, 3770 samples, 20 Hz, 1983,312,18:01:30.600000
-GR_GRA1__BHZ, 000349, D, 4096, 3772 samples, 20 Hz, 1983,312,18:04:39.100000
-GR_GRA1__BHZ, 000350, D, 4096, 3772 samples, 20 Hz, 1983,312,18:07:47.700000
-GR_GRA1__BHZ, 000351, D, 4096, 3764 samples, 20 Hz, 1983,312,18:10:56.300000
-GR_GRA1__BHZ, 000352, D, 4096, 3764 samples, 20 Hz, 1983,312,18:14:04.500000
-GR_GRA1__BHZ, 000353, D, 4096, 3772 samples, 20 Hz, 1983,312,18:17:12.700000
-GR_GRA1__BHZ, 000354, D, 4096, 3772 samples, 20 Hz, 1983,312,18:20:21.300000
-GR_GRA1__BHZ, 000355, D, 4096, 3744 samples, 20 Hz, 1983,312,18:23:29.900000
-GR_GRA1__BHZ, 000356, D, 4096, 3764 samples, 20 Hz, 1983,312,18:26:37.100000
-GR_GRA1__BHZ, 000357, D, 4096, 3772 samples, 20 Hz, 1983,312,18:29:45.300000
-GR_GRA1__BHZ, 000358, D, 4096, 3772 samples, 20 Hz, 1983,312,18:32:53.900000
-GR_GRA1__BHZ, 000359, D, 4096, 3772 samples, 20 Hz, 1983,312,18:36:02.500000
-GR_GRA1__BHZ, 000360, D, 4096, 3772 samples, 20 Hz, 1983,312,18:39:11.100000
-GR_GRA1__BHZ, 000361, D, 4096, 3494 samples, 20 Hz, 1983,312,18:42:19.700000
-GR_GRA1__BHZ, 000362, D, 4096, 3772 samples, 20 Hz, 1983,312,18:45:14.400000
-GR_GRA1__BHZ, 000363, D, 4096, 3772 samples, 20 Hz, 1983,312,18:48:23.000000
-GR_GRA1__BHZ, 000364, D, 4096, 3772 samples, 20 Hz, 1983,312,18:51:31.600000

+_GRA1__BHZ, 000346, D, 4096, 3772 samples, 20 Hz, 1983,312,17:55:13.400000
+_GRA1__BHZ, 000347, D, 4096, 3772 samples, 20 Hz, 1983,312,17:58:22.000000
+_GRA1__BHZ, 000348, D, 4096, 2188 samples, 20 Hz, 1983,312,18:01:30.600000
+_GRA1__BHZ, 000349, D, 4096, 3770 samples, 20 Hz, 1983,312,18:03:30.000000
+_GRA1__BHZ, 000350, D, 4096, 3772 samples, 20 Hz, 1983,312,18:06:38.500000
+_GRA1__BHZ, 000351, D, 4096, 3658 samples, 20 Hz, 1983,312,18:09:47.100000
+_GRA1__BHZ, 000352, D, 4096, 3770 samples, 20 Hz, 1983,312,18:13:00.000000
+_GRA1__BHZ, 000353, D, 4096, 30 samples, 20 Hz, 1983,312,18:16:08.500000
+_GRA1__BHZ, 000354, D, 4096, 3766 samples, 20 Hz, 1983,312,18:16:20.000000
+_GRA1__BHZ, 000355, D, 4096, 3772 samples, 20 Hz, 1983,312,18:19:28.300000
+_GRA1__BHZ, 000356, D, 4096, 3748 samples, 20 Hz, 1983,312,18:22:36.900000
+_GRA1__BHZ, 000357, D, 4096, 3760 samples, 20 Hz, 1983,312,18:25:44.300000
+_GRA1__BHZ, 000358, D, 4096, 3772 samples, 20 Hz, 1983,312,18:28:52.300000
+_GRA1__BHZ, 000359, D, 4096, 3772 samples, 20 Hz, 1983,312,18:32:00.900000
+_GRA1__BHZ, 000360, D, 4096, 3772 samples, 20 Hz, 1983,312,18:35:09.500000
+_GRA1__BHZ, 000361, D, 4096, 3772 samples, 20 Hz, 1983,312,18:38:18.100000
+_GRA1__BHZ, 000362, D, 4096, 3510 samples, 20 Hz, 1983,312,18:41:26.700000
+_GRA1__BHZ, 000363, D, 4096, 3756 samples, 20 Hz, 1983,312,18:44:22.200000
+_GRA1__BHZ, 000364, D, 4096, 3772 samples, 20 Hz, 1983,312,18:47:30.000000

Thom

Le 19/02/2014 14:47, Peter Danecek a �crit :

Peter_Danecek1 · February 19, 2014, 2:16pm

Thom,

very probably the blocks 000348 and 000353, of the "old" file are not complete (correctly transmitted). There is a good chance that these blocks result in a checksum mismatch. The "new", file seems have been "repackaged". So the probably do not have the same origin (not just a question of tar/untar). Is this possible?

~petr

ThomasLecocq · February 19, 2014, 2:19pm

Petr,

Yes, definitively possible..

Thom

Le 19/02/2014 15:16, Peter Danecek a �crit :