Unable to install obspy on RHEL 6.4

Glenn_Thompson · October 14, 2013, 1:53pm

I have a clean Red Hat Enterprise Linux 6.4 system. I’m a novice Python/ObsPy/easy_install/pip user.

I was able to install all the dependencies (https://github.com/obspy/obspy/wiki/Installation-on-Linux%3A-Dependencies) without issue with sudo yum. I was also able to update distribute as mentioned at https://github.com/obspy/obspy/wiki/Installation-on-Linux-via-PyPI. I am able to import numpy, scipy, lxml, suds, matplotlib etc.

But here is what happens when I try to add obspy:

[thompsong@feynman ~]$ sudo easy_install obspy
Traceback (most recent call last):
File “/usr/bin/easy_install”, line 9, in
load_entry_point(‘distribute’, ‘console_scripts’, ‘easy_install’)()
File “/usr/lib/python2.6/site-packages/setuptools-1.1.6-py2.6.egg/pkg_resources.py”, line 357, in load_entry_point
return get_distribution(dist).load_entry_point(group, name)
File “/usr/lib/python2.6/site-packages/setuptools-1.1.6-py2.6.egg/pkg_resources.py”, line 2393, in load_entry_point
raise ImportError(“Entry point %r not found” % ((group,name),))
ImportError: Entry point (‘console_scripts’, ‘easy_install’) not found
[thompsong@feynman ~]$ python
Python 2.6.6 (r266:84292, May 27 2013, 05:35:12)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-3)] on linux2
Type “help”, “copyright”, “credits” or “license” for more information.

So next I tried to install with pip using:

[thompsong@feynman ~]$ sudo pip install obspy

This seemed to work - hundreds of lines of output terminated with a message that obspy had been successfully installed. But when I try to import, nothing was found:

[thompsong@feynman ~]$ python
Python 2.6.6 (r266:84292, May 27 2013, 05:35:12)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-3)] on linux2
Type “help”, “copyright”, “credits” or “license” for more information.

import obspy
Traceback (most recent call last):
File “”, line 1, in
ImportError: No module named obspy

I notice that pip install added the following to /usr/bin:

[thompsong@feynman ~]$ ls -l /usr/bin/obs*
-rwxr-x—. 1 root root 319 Oct 14 09:41 /usr/bin/obspy-dataless2resp
-rwxr-x—. 1 root root 321 Oct 14 09:41 /usr/bin/obspy-dataless2xseed
-rwxr-x—. 1 root root 319 Oct 14 09:41 /usr/bin/obspy-flinn-engdahl
-rwxr-x—. 1 root root 307 Oct 14 09:41 /usr/bin/obspy-indexer
-rwxr-x—. 1 root root 303 Oct 14 09:41 /usr/bin/obspy-mopad
-rwxr-x—. 1 root root 333 Oct 14 09:41 /usr/bin/obspy-mseed-recordanalyzer
-rwxr-x—. 1 root root 301 Oct 14 09:41 /usr/bin/obspy-plot
-rwxr-x—. 1 root root 319 Oct 14 09:41 /usr/bin/obspy-reftek-rescue
-rwxr-x—. 1 root root 309 Oct 14 09:41 /usr/bin/obspy-runtests
-rwxr-x—. 1 root root 301 Oct 14 09:41 /usr/bin/obspy-scan
-rwxr-x—. 1 root root 321 Oct 14 09:41 /usr/bin/obspy-xseed2dataless

Did it install these to the wrong place?

I did already install ObsPy successfully on Windows, MacOS and Ubuntu, but since those are packaged, they are trivial.

Anyway, any help to get ObsPy working on RHEL 6.4 greatly appreciated.

Cheers,

Glenn

Peter_Danecek1 · October 14, 2013, 2:41pm

Hi Glenn,

I think this might derive from the fact that distribute merged with setuptools again. So you probably have setuptools installed not distribute. There is some compatibility layer, but I already realised that the setup.py script would fail.

I can send you my patch later this afternoon. Just need to have a quick look at it, there is some other stuff, which will not be relevant to you.

~petr

Peter_Danecek1 · October 15, 2013, 9:45am

Glenn,

I realise this thread is continuing offline. Maybe you want to send this to the list (CCed) as well. I am not that familiar with Linux right now, so maybe some other user can provide you better help here.

In any case find some details below.

Hi Peter,

Thanks so much for your help. I don't know if you can help further.

I have two different pythons on this Red Hat Linux server.

1) The standard python at /usr/bin/python which is version 2.6.
2) A 2.7 version distributed with Antelope at /opt/antelope/5.3/bin/python. This is the install I'd most like to get to work, but I figured it would be easier to get (1) working first.

Each of these installations will have a site-packages dir where all the packages you try to install go. The path will be something like ${prefix}/lib/python/site-packages/. For (1) this likely is /usr/lib/python2.6/site-packages (not sure about the version no.), for (2) could be something like /opt/antelope/5.3/lib/python. But you should be able to identify these when you look through the dirs, or search for them with `locate`. There you find the packages you installed.

Anyway, I thing the Obspy team proposes to install in an virtual environment. This might be an option for you, and you could create a virtual env for both interpreters and put there all missing packages and obspy into the virtualenv.

With each, I have a different problem.

For (2) it seemed that the install of obspy using the 'python setup.py build' and 'python setup.py install' method substantially worked. But it complained about gcc and gfortran not being available (they are). And then I found that dependencies lxml and scipy were not installed.

The error/warning message from the obspy script may be a "false alarm". I have seen this before. Just check if the C & fortran libs are correctly build and installed.

If you really like (or need) to install into the respective site-packages, you can do this with easy-install or pip, but you need to use correct version of these. On my system there is a set of these

easy_install-2.5 easy_install-2.6 easy_install-2.7 easy_install-3.2
pip-2.7

Maybe, look in /opt/antelope/5.3/bin for them.

Apparently each time I had tried to install these with "yum -y install obspy" they must have been installing against /usr/bin/python rather than /opt/antelope/5.3/bin/python (even though "which python" gave the latter). I don't know how to force yum to install against a non-standard location for python. Also the Antelope python doesn't have the yum module (it does have pip and easy_install though).

Well, that is how I would expect it would work. The `yum` package manager cannot install against something, which does not exist for it. `/opt/antelope/5.3/bin/python` is not managed by the 'yum' neither, right? So see above and go for the pip or easy_install option, or virtual env.

Attached are the logs from trying to install lxml and scipy (with easy_install or pip) and obspy (with python setup.py install).

Okay, the logs look like you are missing dependencies for lxml and scipy. obspy just runs into the same problem as lxml.

lxml: depends on libxml, libxslt (see logs) and as far as I know on zlib. Check if these are there. On Linux you will need to have the -dev (or -devel) packages installed as well, these contain the header files. The problem may also be that these are just not found, but it is not evident from the logs.

scipy: Here you are missing ATLAS library. Even if it is there, it is not found.

For (1) i.e. /usr/bin/python, all the dependencies are definitely in place (they all import). But my setuptools seem corrupt because I get the following:

# python setup.py build
Traceback (most recent call last):
File "setup.py", line 28, in <module>
   from setuptools import find_packages, setup
File "/usr/lib/python2.6/site-packages/setuptools-1.1.6-py2.6.egg/setuptools/__init__.py", line 11, in <module>
   from setuptools.extension import Extension
File "/usr/lib/python2.6/site-packages/setuptools-1.1.6-py2.6.egg/setuptools/extension.py", line 5, in <module>
   from setuptools.dist import _get_unpatched
File "/usr/lib/python2.6/site-packages/setuptools-1.1.6-py2.6.egg/setuptools/dist.py", line 16, in <module>
   import pkg_resources
File "/usr/lib/python2.6/site-packages/setuptools-1.1.6-py2.6.egg/pkg_resources.py", line 2813, in <module>
   add_activation_listener(lambda dist: dist.activate())
File "/usr/lib/python2.6/site-packages/setuptools-1.1.6-py2.6.egg/pkg_resources.py", line 686, in subscribe
   callback(dist)
File "/usr/lib/python2.6/site-packages/setuptools-1.1.6-py2.6.egg/pkg_resources.py", line 2813, in <lambda>
   add_activation_listener(lambda dist: dist.activate())
File "/usr/lib/python2.6/site-packages/setuptools-1.1.6-py2.6.egg/pkg_resources.py", line 2346, in activate
   list(map(declare_namespace, self._get_metadata('namespace_packages.txt')))
File "/usr/lib/python2.6/site-packages/setuptools-1.1.6-py2.6.egg/pkg_resources.py", line 1942, in declare_namespace
   _handle_ns(packageName, path_item)
File "/usr/lib/python2.6/site-packages/setuptools-1.1.6-py2.6.egg/pkg_resources.py", line 1912, in _handle_ns
   loader.load_module(packageName); module.__path__ = path
File "/usr/lib64/python2.6/pkgutil.py", line 238, in load_module
   mod = imp.load_module(fullname, self.file, self.filename, self.etc)
File "/usr/lib64/python2.6/site-packages/obspy/__init__.py", line 33, in <module>
   from obspy.core.util.base import _getVersionString
ImportError: cannot import name _getVersionString

The problem seems to be with obspy. Are you sure obspy is not already present? Maybe and older version? Try to import it and check the version.

Hope this all helps and consider posting to the list as well!
~petr

Glenn_Thompson · October 15, 2013, 3:08pm

(Peter thanks so much for your help. I’ll post to the wider group as suggested).

Peter_Danecek1 · October 15, 2013, 3:16pm

Be careful, if you clone for the repo. You will by default end up with the `master` branch, which is the development path. That by it self might not hurt, but if you want to install the release, you better download the tarball from the release tab.

Note also, the patch/setup.py is useful only for the release (0.8.4) branch. In master there should be a completely different setup.py and you can use that directly.

Peter_Danecek1 · October 15, 2013, 4:04pm

(Peter thanks so much for your help. I'll post to the wider group as suggested).

Each of these installations will have a site-packages dir where all the packages you try to install go.

Yes, they are /usr/lib/python/2.6/site-packages/ (no obspy, but everything else is there) and /opt/antelope/python2.7.2/lib/python2.7/site-packages (no lxml or scipy, does have obspy but the simple 'obspy.core.read' example from the tutorial fails).

ok, is more or less what we expected

I cloned the obspy github repo. Overwrote setup.py with the version Peter sent. Which I then try to build and install with "python setup.py build" and "python setup.py install". I've tried against 3 different pythons, checking that I'm calling the correct python in each case (always as root):

See the mail before (careful about the version)

(1) Against default python. All the dependencies are there, but the "python setup.py build" fails with errors seen in last email. (setuptools not working?)

Not sure what is the problem here. Maybe, try to do a pip freeze against this default python. Something suspect? Version for setuptools?

(2) Using virtualenv (from default python). None of the dependencies are there. The problem is how to install them, since "yum install PACKAGENAME" still builds against default python, not the virtualenv python. Is it possible to inherit them from default python site-packages? Or can I tell yum to install against virtualenv python?

I guess, with `yum` holds what I wrote in some different form before. It will install against default python. You have different options to create your virtualenv you can inherit your site-packages or avoid this. See virtualenv --help for the details. If you do not want to use site-packages you have to install these manually into the environment.

Be careful to active the correct environment. Then the virtualenv will provide you with pip and easy-install which install into the environment.

The reason I think setuptools for (1) are messed up is I am able to do a "python setup.py build" and a "python setup.py install" in case (2). However, it makes the same complaints as in the attachment to my previous email "obspy_install.log".

(1) Not sure, check the version of setuptools;
(2) Are the dependencies installed? The difference with respect to the default is that with yum you installed a prebuilt packages (no need for headers), for python 2.7 you are actually building the some C extension on your own and therefore need more dependencies.

(3) Against Antelope python. I'll ignore this for now. I think if I can get the virtualenv above working, the same procedure should allow me to get obspy working against Antelope python, because the problems I see are the same.

Just check if the C & fortran libs are correctly build and installed.

I know gcc and gfortran are installed to default locations and work. Is there anything else to check?

look for for example for the following files:
${prefix}/lib/python2.7/site-packages/obspy/lib/libmseed.so (C build)
${prefix}/lib/python2.7/site-packages/obspy/lib/libtaup.so (fortran build)

These require the compiles.

If you really like (or need) to install into the respective site-packages, you can do this with easy-install or pip, but you need to use correct version of these.

easy_install/pip: In a different virtualenv I did try building lxml, scipy, BLAS LAPACK, and other dependencies required to go the easy_install or pip install method, but it was time consuming and didn't work, whereas yum presumably takes care of dependencies itself?

I do not know the details of yum, but I would assume it checks for dependencies, takes an archive and unpacks it into default location. No build! No relocation (correct my if I am wrong!)
You can inherit the site-packages in your virtualenv (see above), but if you need to install lxml/scipy for via pip/easy-install, you run in similar problems, no matter if you install into site-packages or into the virtualenv.

Okay, the logs look like you are missing dependencies for lxml and scipy. obspy just runs into the same problem as lxml.

lxml: depends on libxml, libxslt (see logs) and as far as I know on zlib. Check if these are there. On Linux you will need to have the -dev (or -devel) packages installed as well, these contain the header files. The problem may also be that these are just not found, but it is not evident from the logs.

scipy: Here you are missing ATLAS library. Even if it is there, it is not found.

For default python, "yum install python-devel", yum install lxml" and "yum install scipy" worked, so do they just take care of dependencies automatically, like LAPACK, BLAS, ATLAS? Anyway to get a virtualenv python to inherit these? Or run yum against my virtualenv python?

Probalby you have atlas, libxml, libxslt, etc. installed. But you may miss libxml-devel, the header files (not sure about the names).

virtualenv see above;

The problem seems to be with obspy. Are you sure obspy is not already present? Maybe and older version? Try to import it and check the version.

obspy definitely does not exist for default python. Not in site-packages, and not found when try to import obspy. So I think it is a setuptools problem. I have tried moving the default python setuptools from site-packages and commenting out all references to them in easy_install.pth, then reinstalling setuptools but it appears to claim they are still there. How do I get a clean install of setuptools for my default python?

Does obspy install ok against EPD-Canopy - maybe I should try that?

I know of people having such a working installation.

(It was trivial to install on Windows, MacOS and Ubuntu - thanks to all the developers for such great tools and making them so easy to install - it's only on RHEL I'm having problems).

no comment!

And good luck!
~petr

Glenn_Thompson · October 15, 2013, 7:12pm

OK, was able to get at least a simple obspy.core.read working. I think the critical steps were:

a new virtualenv with the --system-site-packages option. This inherited everything I had installed with yum (numpy, scipy, matplotlib, lxml, suds, sqlalchemy). virtualenv also installed it’s own setuptools, which got me around default setuptools which appear to be broken.
remove any references to obspy from the site-packages in /usr/lib/python2.6 and /usr/lib64/python2.6. (I didn’t even know there was a separate /usr/lib64/python2.6 until just now).
download the latest obspy release from pip, added Petr’s setup.py, and then did a python setup.py build and a python setup.py install.

I don’t think this helps me get obspy working in conjunction with Antelope-Python, but for now this is ok.

Petr, thanks for your help.

Glenn

Mark_Williams · October 15, 2013, 7:41pm

HI Glenn,

I did get obspy working with Antelope on 6.4 by building scipy from source into an egg (via the instructions on the ObsPy wiki), and then easy_installing from that egg using Antelope python. Everything else (including all dependencies, distribute and obspy) was easy_install/pip installed using the Antelope python version. I have chef recipes somewhere for this, but they haven’t been fully updated and tested:

https://github.com/NVSeismoLab/cookbook-antelope/tree/master/recipes

The 2.6/2.7 mismatch with Antelope can be kind of a pain, in Ubuntu we are starting to just use the distro python and add Antelope through sitecustomize.py, but that doesn’t work for RH. We have been thinking about using scl-utils (https://fedorahosted.org/SoftwareCollections/) to install a distro python2.7, but there are limited distro packages for that version. We have had some luck treating Antelope as a virtualenv, but you need to be careful if you compile any of your own code to link to the libraries in the Antelope python folder. Good luck.

-Mark