CategoryDevelopment, Scripting, Python
DescriptionPython is an interpreted, object-oriented, high-level programming language with dynamic semantics. Its high-level built in data structures, combined with dynamic typing and dynamic binding, make it very attractive for Rapid Application Development, as well as for use as a scripting or glue language to connect existing components together. Python supports modules and packages, which encourages program modularity and code reuse.
|Legacy versions||Supported versions||New versions|
|2.7.12, 2.7.13, 2.7.6, 2.7.9, 3.3.3, 3.4.3, 3.6.0||2.7.14, 2.7.6_UCS4, 3.6.1|
|Version||Module load command||Additional modules loaded automatically|
|2.7.12||module load gcc/4.8.2 python/2.7.12||openblas/0.2.13_seq|
|2.7.13||module load gcc/4.8.2 python/2.7.13||openblas/0.2.13_seq sqlite/3.15.0|
|2.7.6||module load gcc/4.8.2 python/2.7.6||openblas/0.2.13_seq|
|2.7.9||module load gcc/4.8.2 python/2.7.9||openblas/0.2.13_seq|
|3.3.3||module load gcc/4.8.2 python/3.3.3||openblas/0.2.13_seq|
|3.4.3||module load gcc/4.8.2 python/3.4.3||openblas/0.2.13_seq|
|3.6.0||module load gcc/4.8.2 python/3.6.0||openblas/0.2.13_seq sqlite/3.15.0|
|2.7.14||module load new gcc/4.8.2 python/2.7.14||openblas/0.2.13_seq|
|2.7.6_UCS4||module load new gcc/4.8.2 python/2.7.6_UCS4||openblas/0.2.13_seq|
|3.6.1||module load new gcc/4.8.2 python/3.6.1||openblas/0.2.13_seq|
Interactive sessionIn order to start an interactive Python session on one of the login nodes, you can use the command python to start the Python interpreter.
[leonhard@euler06 ~]$ module load gcc/4.8.2 python/2.7.6 Autoloading openblas/0.2.13_seq [leonhard@euler06 ~]$ python Python 2.7.6 (default, Jan 20 2014, 14:02:29) [GCC 4.8.2] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import numpy >>> import a_module_that_is_not_installed_yet Traceback (most recent call last): File "<stdin>", line 1, in <module> ImportError: No module named a_module_that_is_not_installed_yet >>>Interactive sessions can be used to check if a particular package is already installed or not (if a package is not installed yet, the import statement will result in an error message).
How to submit a jobFor small tests, pre- and post-processing with Python, you can start an interactive sessions on one of the login nodes. All other Python jobs have to be submitted through the batch system. You can submit a Python job (my_python_script.py) in batch mode with the following command:
bsub [LSF options] "python my_python_script.py"
Here you need to replace [LSF options] with LSF parameters for the resource requirements of the job. Please find a documentation about the parameters of bsub on the wiki page about the batch system.
When you specify the Python interpreter on the first line of your script, then please use
Otherwise the Python interpreter from the operating system will be called when you directly execute the script with
python ./my_python_script.pyThe Python interpreter from the operating system is older and does not have the additional packages installed.
ExampleAs an example for running a Python script on the cluster, we use the built-in tests of the scientific Python packages numpy and scipy.
[leonhard@euler02 ~]$ module load python/3.3.3 Autoloading openblas/0.2.13_seq [leonhard@euler02 ~]$ cat test_python.py import numpy import scipy numpy.test() scipy.test() [leonhard@euler02 ~]$ bsub -n 1 -W 4:00 -R "rusage[mem=1024]" python ./test_python.py Generic job. Job <25518323> is submitted to queue <normal.4h>. [leonhard@euler02 ~]$ bjobs JOBID USER STAT QUEUE FROM_HOST EXEC_HOST JOB_NAME SUBMIT_TIME 25518323 leonhard PEND normal.4h euler02 *python.py Aug 23 15:05 [leonhard@euler02 ~]$ bjobs JOBID USER STAT QUEUE FROM_HOST EXEC_HOST JOB_NAME SUBMIT_TIME 25518323 leonhard RUN normal.4h euler02 e1408 *python.py Aug 23 15:05 [leonhard@euler02 ~]$ bjobs No unfinished job found [leonhard@euler02 ~]$ grep Ran lsf.o25518323 Ran 6139 tests in 114.985sRan 20195 tests in 366.931s
ExtensionsThe Python programming language can make use of packages, which extend the functionality of the core language. We provide a central Python package repository, with a range of installed packages that are available to all cluster users. If a package is not installed centrally, or if you need a newer version than the centrally installed one, then you can also install a Python package locally in your home directory, that can then be used together with the centrally installed Python interpreter.
Installing a Python package locally, using distutils
You can create your personal Python package repository. As an example, we create this directory in $HOME:
You are free to choose any other location for your package repository, but it has to contain a given sub-directory structure (which is shown here for python/2.7.6. In case of python/3.3.3, exchange the version number in the directory names from 2.7 to 3.3):
cd $HOME/python mkdir -p lib64/python2.7/site-packages
In order to make python aware of the local package repository, you should add a line to the $HOME/.bash_profile file:
and run this same command on the command line to make your current interactive shell aware of the additional python path. All of the the above steps should be done only once: before you install your first package.
After this preparation, one needs to load the Python module:
module load python/2.7.6
As a next step, you can download and extract the tarball with the source code of the additional package. Go into the directory containing the setup.py file (the directory is usually named package-version). Finally, you can run the command to install the package:
python setup.py install --prefix=$HOME/python
Here the prefix option assures that the package is installed in your local package repository.
With this setup, it is possible to have two versions of a package side-by-side. Python will always search in the locations given in PYTHONPATH first, and if a package is not found there, it will fall back to the central repository.
Installing a Python package, using PIP
Additionally, you can also install a package locally using pip
python -m pip install --user package
The packages will be installed in a local repository at $HOME/.local. Since this is the default path for user-installed python packages, there is no need to adjust PYTHONPATH.
In order to install a particular version of a package, you can use the command
python -m pip install --user package==version
Where "version" denotes the version number that you would like to install.But be aware that using pip to install a newer version of a package that has originally been installed with easy_install causes problems in the loading order, such that always the version that was installed with easy_install will be loaded unless you change the sys.path in the current Python session.