FDSN downloader spamming new processes

Dear ObsPy team,

I am facing a potential issue with the ObsPy FDSN client. I am using the EIDA routing client to download waveform data and metadata for many stations. The way I do it is a simple loop over all stations, using the get_waveforms() function in the loop. In principle it looks like this:

from obspy.clients.fdsn import RoutingClient as Client
client = Client(‘eida-routing’)
for station in long_list :
data = client.get_waveforms(…)

The problem is, that each time data is actually downloaded, ObsPy seems to start several new processes in the background, which are not being closed until the end of the python script. Thus, for each station in the for-loop new additional processes are started but not closed again. Since I have a very long stationlist and many days that I loop over, basically I end up spawning more processes than the systen can handle, ultimately resulting in the message: “Can’t start new thread”

This problem only occurs with the fdsn routing client, but e.g. not with the ArcLink client. I did not try the non-routed fdsn client. Same behaviour with ObsPy versions 1.1.0 and 1.1.1 …

Any suggestions how to avoid this? Or did I just code it in a stupid way?

Thanks in advance and best regards,

_____________________________________ Dr. Florian Fuchs Department of Meteorology and Geophysics University of Vienna UZA II, 2D508 Althanstraße 14 1090 Vienna Austria Phone: +43 1 4277 53726 Mobile: +43 664 60277 53726 www: http://imgw.univie.ac.at _____________________________________

Hi Florian,

you could try to use the get_waveforms_bulk() method instead of the for
loop, see if that helps. In general the bulk requests are more
efficient, as they avoid http comms overhead, but I'm not sure how this
plays out with the routed clients exactly..

But, this is definitly something we want to look into, so I'm opening an
issue on github for this.


Hi Florian,

Tobias opened the issue here: https://github.com/obspy/obspy/issues/2342

I'm having some troubles reproducing it. Can you join the discussion on
github and help us narrow it down?

All the best,