iris web services client - anyone see this problme recently?

I may be missing something, but I see 17 web services at IRIS that are not part of the FDSN suite. Right now only a few are part of obspy.iris. Why remove obspy.iris? I understand standardizing the waveform requests to FDSN, but sometimes the "timeseries" request Gary was using is simpler, especially in educational applications.

The obspy team shouldn't write interfaces to all IRIS services, but keeping a web services client like obspy.iris that at least partially integrates with obspy is a good idea. Many users appreciate access to the non-fdsn IRIS services. At some point use of the MUSTANG services is going to catch on (if it hasn't already).

I imagine there are two parts to a client - building requests and unpacking the responses. I have not followed all the code of an obspy client through. Is it possible to open up the part of obspy.iris that builds a web-service request? That's tedious coding (if you add urllibs, error checking and reporting, etc.) that is already robustly complete in obspy.

You can leave the unpacking of non-fdsn services to the user. With a generic WS interface available, a user can acquire the raw server response using the client argument "filename=io.BytesIO()" and pull apart the xml, etc. That's what I usually do with stations and channels now anyway. Then I can construct dictionaries with exactly the information I need, not what obspy unpacks.

I guess that I am thinking of something like:

     irisClient = obspy.iris.Client()
     myOutput = irisClient(service="mustang/measurements",metric="Sample RMS", ...,format=json,filname=io.BytesIO())

It could be just a generic web service client, not just iris.

This may be already implemented or it may be a bad idea. I know it can all be done with local python. But removing obpsy.iris client without a way for us to use the IRIS services is not a good direction.

Message: 4
Date: Mon, 6 Jul 2015 17:24:27 -0400
From: Elliott Sales de Andrade <quantum.analyst@gmail.com>
To: users@obspy.org
Subject: Re: [obspy-users] iris web services client - anyone see this
  problme recently?
Message-ID:
  <CAJP_izd-KgwUKQ2A_sBNhxgMLiGfGR-jK_ax5rc1mk_urCyXQA@mail.gmail.com>
Content-Type: text/plain; charset=UTF-8

Hi Gary,

I'm not sure why you think obspy.fdsn was replaced by obspy.iris. In
fact, it is the opposite; obspy.iris is older, partly deprecated, and
possibly even soon to be removed. I do not even think IRIS leaves many
of those services available any more. You should go back to using
obspy.fdsn.

If you still have issues with obspy.fdsn, that is certainly something
that should be rectified.

Something seems to have drastically changed recently in the web service
interface to iris dmc. A script I last used about a year ago no longer
works. Several issues:
1. I formerly used the fdsn client, but the current document has no such
thing. I changed that line to

from obspy.iris import Client

This constructor then works, although it probably does next to nothing:
client=Client("IRIS")

2. Using ipython and some tracebacks from the exception handler I think
something is getting botched in the url generator code. You should be able
to cut and paste lines below to recreate the problem I'm seeing:

net='TA'
sta='D03D'
chan='BH*'

Note:
In [22]: print net, sta, chan
TA D03D BH*

Then:
from obspy import UTCDateTime
out='foo.dat'
stime=UTCDateTime(float(1388534670.01))
etime=UTCDateTime(float(1389137254.38))

Note:
In [26]: print stime
2014-01-01T00:04:30.010000Z

In [27]: print etime
2014-01-07T23:27:34.380000Z

Problem is when I then run:
client.timeseries(net,sta,"*","*",stime,etime,out)

I get this from ipython:

---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
<ipython-input-28-59c79d87a034> in <module>()
----> 1 client.timeseries(net,sta,"*","*",stime,etime,out)

/opt/antelope/python2.7.8/lib/python2.7/site-packages/obspy-0.10.2-py2.7-macosx-10.4-x86_64.egg/obspy/iris/client.pyc
in timeseries(self, network, station, location, channel, starttime, endtime,
filter, filename, output, **kwargs)
397 # build up query
398 try:
--> 399 data = self._fetch("timeseries", param_list=filter,
**kwargs)
400 except urllib.request.HTTPError as e:
401 msg = "No waveform data available (%s: %s)"

/opt/antelope/python2.7.8/lib/python2.7/site-packages/obspy-0.10.2-py2.7-macosx-10.4-x86_64.egg/obspy/iris/client.pyc
in _fetch(self, service, data, headers, param_list, **params)
135 req = urllib.request.Request(url=remoteaddr, data=data,
136 headers=headers)
--> 137 response = urllib.request.urlopen(req, timeout=self.timeout)
138 doc = response.read()
139 return doc

/opt/antelope/python2.7.8/lib/python2.7/urllib2.pyc in urlopen(url, data,
timeout)
125 if _opener is None:
126 _opener = build_opener()
--> 127 return _opener.open(url, data, timeout)
128
129 def install_opener(opener):

/opt/antelope/python2.7.8/lib/python2.7/urllib2.pyc in open(self, fullurl,
data, timeout)
394
395 req.timeout = timeout
--> 396 protocol = req.get_type()
397
398 # pre-process request

/opt/antelope/python2.7.8/lib/python2.7/urllib2.pyc in get_type(self)
256 self.type, self.__r_type = splittype(self.__original)
257 if self.type is None:
--> 258 raise ValueError, "unknown url type: %s" %
self.__original
259 return self.type
260

ValueError: unknown url type:
IRIS/timeseries/1/query?f&o&o&.&d&a&t&network=TA&station=D03D&location=%2A&starttime=2014-01-01T00%3A04%3A30.010&output=miniseed&endtime=2014-01-07T23%3A27%3A34.380&channel=%2A
---------------------------------------------------------------------------

I get a similar message from catching an exception in the original much
larger script:
[ 2015-07-06 11:53:14 ] timeseries threw an error, skipping TA : D03D Time
2014-01-01T00:14:30.009240Z Start 2014-01-01T00:04:30.009240Z End
2014-01-01T00:31:10.009240Z^M
Traceback (most recent call last):^M
File "/opt/antelope/5.5/contrib/bin/fetchirisdmc", line 130, in
fetchdmcthread^M
client.timeseries(net,sta,"*","?H*",stime,etime, filename=fname)^M
File
"/opt/antelope/python2.7.8/lib/python2.7/site-packages/obspy-0.10.2-py2.7-macosx-10.4-x86_64.egg/obspy/iris/client.py",
line 399, in timeseries^M
data = self._fetch("timeseries", param_list=filter, **kwargs)^M
File
"/opt/antelope/python2.7.8/lib/python2.7/site-packages/obspy-0.10.2-py2.7-macosx-10.4-x86_64.egg/obspy/iris/client.py",
line 137, in _fetch^M
response = urllib.request.urlopen(req, timeout=self.timeout)^M
File "/opt/antelope/python2.7.8/lib/python2.7/urllib2.py", line 127, in
urlopen^M
return _opener.open(url, data, timeout)^M
File "/opt/antelope/python2.7.8/lib/python2.7/urllib2.py", line 396, in
open^M
protocol = req.get_type()^M
File "/opt/antelope/python2.7.8/lib/python2.7/urllib2.py", line 258, in
get_type^M
raise ValueError, "unknown url type: %s" % self.__original^M
ValueError: unknown url type:
IRIS/timeseries/1/query?network=TA&station=D03D&location=%2A&starttime=2014-01-01T00%3A04%3A30.009&output=miniseed&endtime=2014-01-01T00%3A31%3A10.009&channel=%3FH%2A^M
---------------------------------------------------------------------------
NOTE: the url is clearly wrong. When I go the URL generator at iris dmc
and type in the similar data I used for the timeseries method of the iris
client it generates this URL:
http://service.iris.edu/fdsnws/dataselect/1/query?net=TA&sta=D03D&loc=&cha=&starttime=2014-01-01T00:04:30&endtime=2014-01-01T00:31:30&nodata=404

I notice:
a. The leading http:// ... to /1/query... is wrong
b. The obspy url following the "ValueError" appears to have some
formatting strings that do not have substitutions. Clearly botched.

Has anyone gotten client.timeseries to work recently from iris? Anyone see
anything like this before? I can turn this into a simpler script for
testing if there is reason to think I've found some bug.

--
Prof. Gary L. Pavlis
Director of Graduate Studies
Department of Geological Sciences
Indiana University
1001 East 10th Street
Bloomington, IN 47505
phone: 812 855 5141

_______________________________________________
obspy-users mailing list
obspy-users@lists.swapbytes.de
http://lists.swapbytes.de/mailman/listinfo/obspy-users

--
Elliott

------------------------------

_______________________________________________
obspy-users mailing list
obspy-users@lists.swapbytes.de
http://lists.swapbytes.de/mailman/listinfo/obspy-users

End of obspy-users Digest, Vol 56, Issue 5
******************************************

================================i====================================
Charles J. Ammon, Department of Geosciences
Penn State University / 440 Deike Bldg / University Park, PA 16802
VOICE: (814) 865 2310 / FAXES: (814) 863 7823 or (814) 863 8724
http://eqseis.geosc.psu.edu/~cammon/
http://www.personal.psu.edu/faculty/c/j/cja12/

Hi Charles,

Sorry for being unclear, but I was only referring to removing the API
from obspy.iris for the services that have already been replaced by
obspy.fdsn, and disabled at IRIS.

Hi all,

The obspy team shouldn't write interfaces to all IRIS services, but
keeping a web services client like obspy.iris that at least partially
integrates with obspy is a good idea. Many users appreciate access to
the non-fdsn IRIS services. At some point use of the MUSTANG services is
going to catch on (if it hasn't already).

Talking about Mustang: Like a year ago I wrote an `obspy.mustang` module but got stuck because I could not find a way to properly specify constraints when requesting multiple metrics at once.

For example here I would like to request the `max_stalta` metric with values larger then 10 and the `clock_locked` metric with values larger than 4000. In my tries it always applies the constrains to all requested metrics simultaneously.

http://service.iris.edu/mustang/measurements/1/query?metric=clock_locked&metric=max_stalta&sta=ANMO&cha=BHZ&loc=00&output=text&startafter=2013-01-01&endbefore=2013-03-01&value_gt=4000

Is the recommended procedure in these cases then to submit multiple requests or is there a way I am missing?

I imagine there are two parts to a client - building requests and
unpacking the responses. I have not followed all the code of an obspy
client through. Is it possible to open up the part of obspy.iris that
builds a web-service request? That's tedious coding (if you add urllibs,
error checking and reporting, etc.) that is already robustly complete in
obspy.

You can leave the unpacking of non-fdsn services to the user. With a
generic WS interface available, a user can acquire the raw server
response using the client argument "filename=io.BytesIO()" and pull
apart the xml, etc. That's what I usually do with stations and channels
now anyway. Then I can construct dictionaries with exactly the
information I need, not what obspy unpacks.

I guess that I am thinking of something like:

irisClient = obspy.iris.Client()
myOutput = irisClient(service="mustang/measurements",metric="Sample
RMS", ...,format=json,filname=io.BytesIO())

As Elliot already stated we have no intention of removing the still working parts of obspy.iris. Nonetheless what you intend to do here is just much simpler to do with a specialized package like this one: http://docs.python-requests.org

The advantage one gains by using ObsPy to do it is that we parse it to our internal representations which in the case of Station XML and QuakeML are fairly involved operations. Our internal objects don’t loose any information and its possible to round trip between file and object.

Out of curiosity: In what cases is it simpler to parse the XML files by yourself to create a dictionary instead of using the ObsPy classes and creating the dictionaries from them? We are always willing to make ObsPy easier to use :slight_smile:

Cheers!

Lion