problem running on Red Hat Enterprise Linux with Python 2.6.6

seismopaul · June 12, 2014, 7:02pm

Any clues on why we get this error message when loading obspy. Has anyone seen this one before??:

[ppicker@isdp01 ~]$ python
Python 2.6.6 (r266:84292, Jan 22 2014, 09:42:36)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-4)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import obspy
Traceback (most recent call last):
   File "<stdin>", line 1, in <module>
   File "/usr/lib64/python2.6/site-packages/obspy-0.9.2_497_g9d11-py2.6-linux-x86_64.egg/obspy/__init__.py", line 43, in <module>
     read.__doc__ % make_format_plugin_table("waveform", "read", numspaces=4)
   File "/usr/lib64/python2.6/site-packages/obspy-0.9.2_497_g9d11-py2.6-linux-x86_64.egg/obspy/core/util/base.py", line 405, in make_format_plugin_table
     "obspy.plugin.%s.%s" % (group, name), method)
   File "/usr/lib/python2.6/site-packages/setuptools-3.6-py2.6.egg/pkg_resources.py", line 356, in load_entry_point
     return get_distribution(dist).load_entry_point(group, name)
   File "/usr/lib/python2.6/site-packages/setuptools-3.6-py2.6.egg/pkg_resources.py", line 2431, in load_entry_point
     return ep.load()
   File "/usr/lib/python2.6/site-packages/setuptools-3.6-py2.6.egg/pkg_resources.py", line 2147, in load
     ['__name__'])
   File "/usr/lib64/python2.6/site-packages/obspy-0.9.2_497_g9d11-py2.6-linux-x86_64.egg/obspy/mseed/core.py", line 15, in <module>
     from obspy.mseed.headers import clibmseed, ENCODINGS, HPTMODULUS, \
   File "/usr/lib64/python2.6/site-packages/obspy-0.9.2_497_g9d11-py2.6-linux-x86_64.egg/obspy/mseed/headers.py", line 54, in <module>
     ENCODINGS = {0: ("ASCII", "a", np.dtype("|S1").type),
TypeError: data type not understood
>>>

We also tried version 0.9.2_660 with no luck....

barsch · June 12, 2014, 8:41pm

Hi Paul,

what NumPy version is installed?

Robert

Any clues on why we get this error message when loading obspy. Has
anyone seen this one before??:

[ppicker@isdp01 ~]$ python Python 2.6.6 (r266:84292, Jan 22 2014,
09:42:36) [GCC 4.4.7 20120313 (Red Hat 4.4.7-4)] on linux2 Type
"help", "copyright", "credits" or "license" for more information.

import obspy

Traceback (most recent call last): File "<stdin>", line 1, in
<module> File
"/usr/lib64/python2.6/site-packages/obspy-0.9.2_497_g9d11-py2.6-linux-x86_64.egg/obspy/__init__.py",

line 43, in <module>

read.__doc__ % make_format_plugin_table("waveform", "read",
numspaces=4) File
"/usr/lib64/python2.6/site-packages/obspy-0.9.2_497_g9d11-py2.6-linux-x86_64.egg/obspy/core/util/base.py",

line 405, in make_format_plugin_table

"obspy.plugin.%s.%s" % (group, name), method) File
"/usr/lib/python2.6/site-packages/setuptools-3.6-py2.6.egg/pkg_resources.py",

line 356, in load_entry_point

return get_distribution(dist).load_entry_point(group, name) File
"/usr/lib/python2.6/site-packages/setuptools-3.6-py2.6.egg/pkg_resources.py",

line 2431, in load_entry_point

return ep.load() File
"/usr/lib/python2.6/site-packages/setuptools-3.6-py2.6.egg/pkg_resources.py",

line 2147, in load

['__name__']) File
"/usr/lib64/python2.6/site-packages/obspy-0.9.2_497_g9d11-py2.6-linux-x86_64.egg/obspy/mseed/core.py",

line 15, in <module>

from obspy.mseed.headers import clibmseed, ENCODINGS, HPTMODULUS,
\ File
"/usr/lib64/python2.6/site-packages/obspy-0.9.2_497_g9d11-py2.6-linux-x86_64.egg/obspy/mseed/headers.py",

line 54, in <module>

ENCODINGS = {0: ("ASCII", "a", np.dtype("|S1").type), TypeError:
data type not understood

We also tried version 0.9.2_660 with no luck....

- --
Dr. Robert Barsch

EGU Office Munich
Luisenstr. 37
80333 Munich
Germany

Phone: +49-89-21806565
Fax: +49-89-218017855
eMail: barsch@egu.eu

seismopaul · June 13, 2014, 6:31pm

Thanks for checking in on this one Robert:

This version of Numpy (1.4.1) is installed:

/usr/lib64/python2.6/site-packages/numpy-1.4.1-py2.6.egg-info

and SCIPY is:

/usr/lib64/python2.6/site-packages/scipy-0.7.2-py2.6.egg-info

Cheers,

Paul

Robert Barsch wrote, On 6/12/14, 4:41 PM:

Mark_Williams · June 13, 2014, 6:40pm

Paul, Robert, looks like maybe a unicode issue with numpy?

#----------- On Scientific Linux 6.5 -------------------------------------------------------------#
markwill@bruce:~$ ipython
Python 2.6.6 (r266:84292, Nov 21 2013, 12:39:37)

In [1]: import numpy as np

In [2]: np.version
Out[2]: ‘1.4.1’

In [3]: np.dtype("|S1")
Out[3]: dtype(’|S1’)

In [4]: np.dtype(u"|S1")

barsch · June 13, 2014, 8:46pm

Thanks Mark - indeed numpy.dtype expects a str and not unicode (see
also dtype field names cannot be unicode (Trac #1814) · Issue #2407 · numpy/numpy · GitHub)

Paul, does it work for you if you patch line 54 in
/usr/lib64/python2.6/site-__packages/obspy-0.9.2_497___g9d11-py2.6-linux-x86_64.egg/__obspy/mseed/headers.py:

ENCODINGS = {0: ("ASCII", "a", np.dtype(str("|S1")).type)

Robert

Paul, Robert, looks like maybe a unicode issue with numpy?

#----------- On Scientific Linux 6.5
-------------------------------------------------------------#
markwill@bruce:~$ ipython Python 2.6.6 (r266:84292, Nov 21 2013,
12:39:37)

In [1]: import numpy as np

In [2]: np.__version__ Out[2]: '1.4.1'

In [3]: np.dtype("|S1") Out[3]: dtype('|S1')

In [4]: np.dtype(u"|S1")
---------------------------------------------------------------------------

TypeError Traceback (most recent call last)

/home/markwill/<ipython console> in <module>()

TypeError: data type not understood
#-----------------------------------------------------------------------------------------------------------#

Mark Williams Nevada Seismological Laboratory Nevada Seismic
Network University of Nevada, Reno tel: (775)-784-4317

Thanks for checking in on this one Robert:

This version of Numpy (1.4.1) is installed:

/usr/lib64/python2.6/site-__packages/numpy-1.4.1-py2.6.__egg-info

and SCIPY is:

/usr/lib64/python2.6/site-__packages/scipy-0.7.2-py2.6.__egg-info

Cheers,

Paul

Robert Barsch wrote, On 6/12/14, 4:41 PM:

Hi Paul,

what NumPy version is installed?

Robert

Any clues on why we get this error message when loading obspy. Has
anyone seen this one before??:

[ppicker@isdp01 ~]$ python Python 2.6.6 (r266:84292, Jan 22 2014,
09:42:36) [GCC 4.4.7 20120313 (Red Hat 4.4.7-4)] on linux2 Type
"help", "copyright", "credits" or "license" for more information.

import obspy

Traceback (most recent call last): File "<stdin>", line 1, in
<module> File
"/usr/lib64/python2.6/site-__packages/obspy-0.9.2_497___g9d11-py2.6-linux-x86_64.egg/__obspy/__init__.py",

line 43, in <module>

read.__doc__ % make_format_plugin_table("__waveform", "read",
numspaces=4) File
"/usr/lib64/python2.6/site-__packages/obspy-0.9.2_497___g9d11-py2.6-linux-x86_64.egg/__obspy/core/util/base.py",

line 405, in make_format_plugin_table

"obspy.plugin.%s.%s" % (group, name), method) File
"/usr/lib/python2.6/site-__packages/setuptools-3.6-py2.6.__egg/pkg_resources.py",

line 356, in load_entry_point

return get_distribution(dist).load___entry_point(group, name) File
"/usr/lib/python2.6/site-__packages/setuptools-3.6-py2.6.__egg/pkg_resources.py",

line 2431, in load_entry_point

return ep.load() File
"/usr/lib/python2.6/site-__packages/setuptools-3.6-py2.6.__egg/pkg_resources.py",

line 2147, in load

['__name__']) File
"/usr/lib64/python2.6/site-__packages/obspy-0.9.2_497___g9d11-py2.6-linux-x86_64.egg/__obspy/mseed/core.py",

line 15, in <module>

from obspy.mseed.headers import clibmseed, ENCODINGS, HPTMODULUS, \
File
"/usr/lib64/python2.6/site-__packages/obspy-0.9.2_497___g9d11-py2.6-linux-x86_64.egg/__obspy/mseed/headers.py",

line 54, in <module>

ENCODINGS = {0: ("ASCII", "a", np.dtype("|S1").type), TypeError:
data type not understood

We also tried version 0.9.2_660 with no luck....

- --
Dr. Robert Barsch

EGU Office Munich
Luisenstr. 37
80333 Munich
Germany

Phone: +49-89-21806565
Fax: +49-89-218017855
eMail: barsch@egu.eu

seismopaul · June 14, 2014, 12:49am

No joy, I get the same error making a str() as per your directions below.

Paul

Robert Barsch wrote, On 6/13/14, 4:46 PM:

LionKrischer · June 14, 2014, 7:45am

Hi Paul,

as you run the master you will have to use

from future.utils import native_str

...

ENCODINGS = {0: ("ASCII", "a", np.dtype(native_str("|S1")).type)

We performed the migration to a unified Python 2 + 3 codebase with the future package which essentially makes Python 2 behave like Python 3, e.g. all strings are unicode and other stuff.

We might have to adjust the whole code base, each time a dtype is set for older numpy versions Our test runners should have caught this but they somehow got stuck on an old ObsPy version…we’ll look into it.

Mark and Paul: Can you check if it happens for all dtypes, e.g. np.dtype(u”float32”), np.dtype(u”int32”), …

@Paul: Does the 0.9.2 release work? That was before the future migration.

Cheers and thanks!

Lion

seismopaul · June 17, 2014, 5:35pm

Hi Lion,

We tried to do the 0.9.2 install using the easy_install but that seems to blow up on RHEL (see below). Is there a clean source distro/tarball for the 0.9.2 that we could try? I couldn't see one easily linked in the www.obspy.org site, just installers and various other recipes.

Cheers,

Paul

[root@isdp01 mseed]# easy_install obspy

Traceback (most recent call last):

File "/usr/bin/easy_install", line 9, in <module>

load_entry_point('distribute', 'console_scripts', 'easy_install')()

File "/usr/lib/python2.6/site-packages/setuptools-3.6-py2.6.egg/pkg_resources.py", line 356, in load_entry_point

return get_distribution(dist).load_entry_point(group, name)

File "/usr/lib/python2.6/site-packages/setuptools-3.6-py2.6.egg/pkg_resources.py", line 2430, in load_entry_point

raise ImportError("Entry point %r not found" % ((group, name),))

ImportError: Entry point ('console_scripts', 'easy_install') not found

Lion Krischer wrote, On 6/14/14, 3:45 AM:

Mark_Williams · June 17, 2014, 6:58pm

Yes, it happens on all data types, this could be fixed in obspy by encoding the strings passed to dtype? A quick test looks like it works on 2/3, in python2 ‘encode’ casts it to ‘str’ and in python3 ‘encode’ casts to ‘bytes’, both of which are accepted by their respective numpys.

np.dtype(‘|S1’.encode())

But I don’t know how many of these exist in the codebase?

-Mark

LionKrischer · June 17, 2014, 11:37pm

Hi Paul,

something seems to be wrong with your Python installation, the following lines install a fully working ObsPy 0.9.2 on a clean Docker container running the latest CentOS version:

$ su -c ‘rpm -Uvh http://download.fedoraproject.org/pub/epel/6/i386/epel-release-6-8.noarch.rpm’

$ yum install numpy scipy
$ yum install python-matplotlib python-sqlalchemy0.8 python-suds python-lxml python-mock
$ yum install gcc-gfortran
$ yum install python-basemap python-basemap-data
$ easy_install flake8
$ easy_install obspy

It seems to use exactly the same Python and numpy versions as your machine:
http://tests.obspy.org/15235/

Otherwise you can always download source tarballs from the pypi page:
https://pypi.python.org/pypi/obspy/

We’ll look into the issues with the master and potentially even providing images for RHEL and consorts.

Cheers!

Lion

LionKrischer · June 17, 2014, 11:54pm

Hey Mark,

I also set up a machine to reproduce the error. future.utils.native_string does exactly what you describe.

The really bad news for us is that the same issue occurs for all numpy functions accepting a dtype argument, e.g.

np.arange(10, dtype=u"float32”)

also does not work with numpy 1.4

So we could with either:

replace all calls to numpy with dtype settings with

np.arange(10, dtype=native_string(“float32”))

do not use string dtypes, but instead

np.arange(10, dtype=np.float32)

Monkey patch most numpy calls
Cease support for NumPy 1.4 with the next release
Revert use of the future package in the code base

Number 5 is not really an option as otherwise the future package works remarkably well by now. Neither is 4 as Debian/Ubuntu/RHEL and other long term support linux distros still run that numpy. That leaves 1, 2, and 3.

Let’s move the discussion to github: https://github.com/obspy/obspy/issues/823

Cheers!

Lion

seismopaul · June 19, 2014, 2:44pm

Hi Lion,

Well things are going better now, in that we got obspy to install and it passes all tests using the formula provided below:

[root@isdp01 site-packages]# python
Python 2.6.6 (r266:84292, Jan 22 2014, 09:42:36)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-4)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import obspy
>>> quit()
[root@isdp01 site-packages]# obspy-runtests
......................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................

Mark_Williams · June 19, 2014, 3:26pm

Hi Paul,

Strange about pygtk2, I think yum pulled that in for me as a dep for matplotlib. If you installed from source, that would explain why you didn’t have any backends, though. If that’s the case, I would advise against using Tk on a server, for reasons below.

As far as I know, the ‘tk’ backend library checks for a display when imported, even if your script doesn’t need/open any windows. So if you are on a server, unless you ssh in and forward X-windows, using the Tk backend will not work. (This holds for Debian, Ubuntu as well). We have some internal services that generate figure images on demand without any ssh’ing and we set the backend to ‘Agg’ at the beginning of our scripts, and this seems to work fine.

I have a Salt state file (and an old Chef recipe) for installing obspy 0.8.x somewhere, I need to update it when I get a chance, but I will post it to GitHub. We should all put our heads together to get a working install procedure for RH? I would still like to build rpm’s when I have the time, but config management files would be just as good for me…

-Mark

Mark Williams
Nevada Seismological Laboratory
Nevada Seismic Network
University of Nevada, Reno
tel: (775)-784-4317