mass_downloader

Users Mailing List Archive
1

Hi.

I am trying to mass download some data of INGV, but I hit some problems on the way. I use the Continuous request as described here:
. The waveforms start do download, then at some point i start to get msg as: If i retry from different IP it works again for some time, and then I get blocked… What to do? are INGV blocking the mass downloading? Thanks, Blaž

2

Hi Blaz,

hmm - good question. I’ve downloaded a fair amount of data from the INGV and have not noticed that behavior - maybe they changed their policy. The mass downloader does send a lot of requests (it has to to work at all) but it tries to batch the request in a reasonable manner to be somewhat nice to data centers so I don’t think it does anything out of line. By default it launches up to 3 threads per data center to download in parallel (this seems to be a number the data centers are okay with) - you can try modifying this with mdl.download(..., thread_per_client=1).

Can you contact the INGV and ask how to best proceed? Otherwise I can also do it.

Cheers!

Lion

3

Thanks, I will contact the INGV and see what can be done. Anyway, if I get some useful info, I’ll write it here.

Cheers,

Blaž