PETSc on a 64 bit Opteron Fedora Core 5 Cluster

Wextan is a 64 bit Opteron Linux Cluster with 32 Opteron 246 CPUS with 64 Gb Ram and 0.5 Terabytes of Hard Disk Storage running Fedora Core 5. A user wants to install Underworld v1.2 on this machine.

PETSc, is the Portable, Extensible Toolkit for Scientific Computation (don't ask me why the 'c' in 'Computation' is lower case). For Underworld we're using v2.3.3, which can be downloaded as a tarball. Put this somewhere sensible (e.g., /usr/local/src/PETSC, untar (tar xvf petsc-2.3.3.tar.gz) and then discover to your horror by looking through the configure.py file that it will not be happy if you try to install from a sensible location.


module load openmpi/1.0.1
module load intel/11.0

Edit out the configure file that restricts installation to non-root users.


# # Check for sudo
# if os.getuid() == 0:
# print '================================================================================='
# print ' *** Do not run configure as root, or using sudo. ***'
# print ' *** Use the --with-sudo=sudo option to have ***'
# print ' *** installs of external packages done with sudo ***'
# print ' *** use only with --prefix= when installing in ***'
# print ' *** system directories ***'
# print '================================================================================='
# sys.exit(3)

You need to set the openmpi directory and set mpi-compilers to auto-detect. I won't mention how many experiments it took to get this right, but I think you can tell by the configure command.


./config/configure.py --with-shared=1 --with-debugging=0 --download-f-blas-lapack=yes --prefix=`pwd` --configModules=PETSc.Configure --optionsModule=PETSc.compilerOptions -with-mpi-dir=/usr/local/openmpi-1.0.1/ --with-mpi-compilers=0

Got all that?

Set the architecture and directory variables


PETSC_ARCH=linux-gnu-c-opt; export PETSC_ARCH
PETSC_DIR=/usr/local/petsc/2.3.3-p8; export PETSC_DIR

Now edit out the test in the makefile which prohibits installation as root


# @if [ "`whoami`" = "root" ]; then \
# echo "********************************************************************"; \
# echo " Do not run configure as root, or using sudo.";\
# echo " That should be reserved for installation";\
# echo "********************************************************************"; \
# exit 1; fi


make all test

..

Number of Newton iterations = 2
lid velocity = 0.0016, prandtl # = 1, grashof # = 1
Number of Newton iterations = 2
Fortran example src/snes/examples/tutorials/ex5f run successfully with 1 MPI process
Completed test examples

Wow, it worked. That wasn't expected.

Make a module directory and file in /usr/local/Modules/modulefiles/petsc and that part is done.