Hi all,
I’m downloading a rather large noise data set from IRIS using the obspy.clients.fdsn.mass_downloader download(). I’ve broken the request down by station and year to ease the request on the DMC.
My request per network/station:
domain = RectangularDomain(minlatitude=min_lat, maxlatitude=max_lat,
minlongitude=min_lon, maxlongitude=max_lon)
restrictions = Restrictions(
starttime=start,
endtime=end,
chunklength_in_sec=86400,
reject_channels_with_gaps=False,
network=net,
station=sta,
location="",
channel=chan,
minimum_length=0.0,
sanitize=False)
Use IRIS,SCEDC,NCEDC
mdl = MassDownloader(providers=[“IRIS”, “SCEDC”,“NCEDC”])
mdl.download(domain, restrictions, mseed_storage=MSEED,
stationxml_storage=XML,threads_per_client=3,
download_chunk_size_in_mb=50)
I’m getting an http client error:
Traceback (most recent call last):
File “/Users/thclements/anaconda3/lib/python3.5/http/client.py”, line 541, in _get_chunk_left
chunk_left = self._read_next_chunk_size()
File “/Users/thclements/anaconda3/lib/python3.5/http/client.py”, line 508, in _read_next_chunk_size
return int(line, 16)
ValueError: invalid literal for int() with base 16: b’’
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File “/Users/thclements/anaconda3/lib/python3.5/http/client.py”, line 558, in _readall_chunked
chunk_left = self._get_chunk_left()
File “/Users/thclements/anaconda3/lib/python3.5/http/client.py”, line 543, in _get_chunk_left
raise IncompleteRead(b’’)
http.client.IncompleteRead: IncompleteRead(0 bytes read)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File “download_inter_station_dist.py”, line 249, in
min_lon, max_lon,MSEED,XML)
File “download_inter_station_dist.py”, line 144, in download
download_chunk_size_in_mb=50)
File “/Users/thclements/anaconda3/lib/python3.5/site-packages/obspy/clients/fdsn/mass_downloader/mass_downloader.py”, line 204, in download
threads_per_client=threads_per_client)
File “/Users/thclements/anaconda3/lib/python3.5/site-packages/obspy/clients/fdsn/mass_downloader/download_helpers.py”, line 836, in download_mseed
[(self.client, self.client_name, chunk) for chunk in chunks])
File “/Users/thclements/anaconda3/lib/python3.5/multiprocessing/pool.py”, line 260, in map
return self._map_async(func, iterable, mapstar, chunksize).get()
File “/Users/thclements/anaconda3/lib/python3.5/multiprocessing/pool.py”, line 608, in get
raise self._value
File “/Users/thclements/anaconda3/lib/python3.5/multiprocessing/pool.py”, line 119, in worker
result = (True, func(*args, **kwds))
File “/Users/thclements/anaconda3/lib/python3.5/multiprocessing/pool.py”, line 44, in mapstar
return list(map(*args))
File “/Users/thclements/anaconda3/lib/python3.5/site-packages/obspy/clients/fdsn/mass_downloader/download_helpers.py”, line 821, in star_download_mseed
*args, logger=self.logger)
File “/Users/thclements/anaconda3/lib/python3.5/site-packages/obspy/clients/fdsn/mass_downloader/utils.py”, line 222, in download_and_split_mseed_bulk
client.get_waveforms_bulk(bulk, filename=temp_filename)
File “/Users/thclements/anaconda3/lib/python3.5/site-packages/obspy/clients/fdsn/client.py”, line 910, in get_waveforms_bulk
data=bulk.encode(‘ascii’, ‘strict’))
File “/Users/thclements/anaconda3/lib/python3.5/site-packages/obspy/clients/fdsn/client.py”, line 1328, in _download
timeout=self.timeout, use_gzip=use_gzip)
File “/Users/thclements/anaconda3/lib/python3.5/site-packages/obspy/clients/fdsn/client.py”, line 1707, in download_url
data = io.BytesIO(f.read())
File “/Users/thclements/anaconda3/lib/python3.5/http/client.py”, line 455, in read
return self._readall_chunked()
File “/Users/thclements/anaconda3/lib/python3.5/http/client.py”, line 565, in _readall_chunked
raise IncompleteRead(b’’.join(value))
http.client.IncompleteRead: IncompleteRead(10571265 bytes read)
This is usually failing around 10500000 bytes read ~ 10.5 Mb. I’m only using 3 threads per client, which shouldn’t be a problem on my machine. Thanks!
Cheers,
Tim