Hi All,
I have a script that downloads windowed data using the FDSN client. It retrieves waveform data using get_waveforms and also retrieves some station information using get_stations. I’m getting an error periodically with the get_stations() call. Of course it would be easy to catch this, but I was interested in what is actually causing the error. Ie, is it the formatting of my station request or an error in the download (a timeout issue?).
A snippet of the error is below. I’m using obspy version: 0.10.3rc1.dev0+302.g104f5af5dc.obspy.master
inv=client.get_stations(network=stid[0],station=stid[1],location=stid[2],channel=stid[3],level=‘channel’)
File “/localdata/local/Software/ObsPy/obspy/obspy/clients/fdsn/client.py”, line 486, in get_stations
data_stream = self._download(url)
File “/localdata/local/Software/ObsPy/obspy/obspy/clients/fdsn/client.py”, line 1226, in _download
raise FDSNException(“Unknown HTTP code: %i” % code, server_info)
TypeError: %d format: a number is required, not NoneType
most likely a timeout error. I already fixed it in a branch of mine so it reports a better error - it’ll go in the master branch somewhen next week I guess.
To make sure you could just initialize the Client with Client(…, debug=True)
and manually download the shown URL with wget or whatever to see what happens and potentially report it to the datacenter.
I have gotten this same error, usually when I request a large set of waveforms. I think it might be a timeout issue, because when I have tried using the URL, it works. Even with debug=True, I have never been able to get an error code or message when this happens. Dividing the request into multiple, smaller requests seems to be more robust.
I have gotten this same error hundreds of times using a script found in antelope contrib called fetchirisdmc. fetchirisdmc was written to fetch all the waveforms from USArray with picks made by the array network facility. When you request several million waveforms you get pretty good statistics. I've found it always takes at least 3 passes (fetchirisdmc has 2 related scripts that do the bookeeping to handle this) to reduce the data loss to something acceptable. As best I can tell the problem is connected to that describe here - the server can get busy and eventually client (correctly) gives up. However, the log files get way out of control when you run this in debug mode so I am not 100% sure of that.
The fundamental problem is that web services is not designed for large numbers (meaning millions) of requests. fetchirisdmc really needs to be redesigned to use the bulk request mechanism, but it requires a major reworking to do that and I've not been motivated to fix it (the existing script works but is not optimal).
I wondered whether it was timeout related. I’ll try increasing the timeout a bit, otherwise just work around this sort of exception and repeat when I get it.
as Gary pointed out, if possible you should definitively try to bundle
your requests using get_stations_bulk() and get_waveforms_bulk(). In the
loops you're using you can instead of doing requests right away assemble
a list for a later bulk request.
It's much easier to repeat a single, failing bulk request if necessary,
than to check and repeat a couple of random failing single requests out
of hundreds or thousands.. also it's gonna be faster.. first because of
reduced http overhead negotiating with the FDSN server and also not
because you won't need to increase your timeout (into which you'll run
much more often with individual requests)..
Also, be nice to your FDSN service provider.. it's easier for them if
they get less, and bundled, bigger requests..