The Washburn University
High-Performance Academic
Computing Environment

On this page:

Types of Users

At this time there are three ways you can use the HiPACE system:

  • Using a Web Browser (appropriate for many users)
  • Using a Command-Line interface through Putty (for compiling and running non-graphical programs)
  • Using an X-Windows Emulator (necessary for running graphical (GUI) programs on HiPACE)
Most HiPACE users will be web-based users. If you are a web-based user, you will not need an account on the HiPACE system itself. You will be able to access the resources you need via a web browser connected to the Internet. Many of the programs available on the HiPACE system can be downloaded from the web site and actually run locally on your PC.

If you need a command-line interface or a graphical user interface (GUI) on the HiPACE system (to compile and run non-graphical or GUI programs that run on the HiPACE system), please contact one of the HiPACE system administrators:

David Bainum - david.bainum@washburn.edu,
Bruce Mechtly - bruce.mechtly@washburn.edu or
Steve Black - steve.black@washburn.edu.

Connecting to HiPACE

Connecting from a UNIX/Linux System

Assuming your system has X-Windows running all you need to do is run the following command:
  ssh -X -p2222 -l<username> hipace.washburn.edu
where you replace <username> with your WUAD username. You are now logged into the HiPACE Gateway. To work on Master or Monster you can simply ssh to that node:
  ssh -X master
  ssh -X monster
The following expect script can expedite the login process:
set username "<your username here>"
set password "<your password here>"
spawn ssh -X -oPort=2222 -l$username hipace.washburn.edu
expect "password: "
send "$password\r"
expect -exact "-sh-3.2$"
send "ssh -X master\r"

Make sure that expect is installed on your local system. The following is a variation that prompts the user for a password instead of storing it in a script:

Note: The string "-sh-3.2$" must match the command line prompt that you see when you log into hipace.washburn.edu manually.

set username "<your username here>"
stty -echo
send_user "Password: "
expect_user -re "(.*)\n"
set password $expect_out(1,string)
stty echo
spawn ssh -X -oPort=2222 -l$username hipace.washburn.edu
expect "password: "
send "$password\r"
expect -exact "-sh-3.2$"
send "ssh -X master\r"

Connecting from a Windows System

If you don't need to run GUI programs you can use a program called "PuTTY". It is a windows executable that can be downloaded for free here. While you're at it get "PSFTP" too.

If you need to run GUI programs then you'll need a windows-based X-Server. There is a good free one called "Cygwin". Once Cygwin is installed you can open a command window and use the ssh commands in the section above.

Changing Your HiPACE Password

You can change your password on HiPACE by typing the command "passwd" at the command line. Your password must be at least 7 characters long and must not be a dictionary word.

Using Basic Linux Commands

If you are using Linux on the command-line and you want to learn more about UNIX and Linux, follow this link to a UNIX tutorial that may be of some help. UNIX and Linux are nearly 100% compatable. Skip the parts about AIX. That part of the tutorial is somewhat out of date.

Click here for a summary of UNIX/Linux commands.

Most Linux users need to know how to use the "vi" text editor. Here is a link to a vi tutorial that will help you use vi to edit text files.

Moving Files To and From HiPACE
From a UNIX/Linux system simply use the sftp command. Keep in mind that the ssh port is 2222 and not 22. Here's the syntax:
  sftp -oPort=2222 <username>@hipace.washburn.edu

Your home directories for master, monster, and cuda1 are NFS mounted on hipace.washburn.edu as /master/home/"username", /monster/home/"username", and /cuda1/home/"username". Thus, files that need to be copied to or from your home directory on any of these machines can be done by copying to or from these directories.

From a Windows environment use "PSFTP" or any of a number of FTP programs. Just make sure they support Secure FTP(SFTP) and change the port to 2222.

Acknowledging Use of HiPACE Resources and/or Personnel in Publications
  1. A publication that is based in whole or in part on computations performed using HiPACE systems, including but not limited to hardware, storage, networking and/or software, should incorporate the following text into the Acknowledgements section of the publication:

    [Some of] The computing for this project was performed using the High Performance Academic Computing Environment (HiPACE) at Washburn University (WU).

  2. If any HiPACE staff member(s) assisted with the work in any way, then for each HiPACE staff member that was involved in the work:

    1. If the publication includes a substantial amount of text about the work that the HiPACE staff member contributed to, and if the HiPACE staff member did a substantial amount of development or optimization of software, and/or they contributed significantly to the writing of the publication, then that staff member should be included as a co-author on that publication, with author order to be negotiated among the authors.

      NOTE: This requirement can be waived for tenure track (but not yet tenured) faculty if the faculty member has a compelling tenure-related interest in, for example, producing single-author publications.

    2. If the conditions above don't apply, then the HiPACE staff member should be acknowledged by name and job title in the Acknowledgements section of the paper.
Programming Languages on HiPACE


You can compile java programs on the command line as follows:

  javac ClassName.java
To run a java program use the following command:
  java ClassName


You can compile C programs with the following command:

  gcc -o execfile.exe sourcefile.c
The result is an executable named "execfile.exe". To run the executable, simply type its name on the command line. On some nodes you'll need to type a "./" in front of the executable's name or it won't find it.

If your program uses math functions like sqrt and sin, you should add the "-lm" option to the gcc command (BTW that's a lower case L not a 1).


Note:The g95 compiler is currently only available on the "master" node.

You can compile FORTRAN programs with the following command:

  g95 -o execfile.exe sourcefile.f
The result is an executable named "execfile.exe". To run the executable, simply type its name on the command line. On some nodes you'll need to type a "./" in front of the executable's name or it won't find it.

Distributed Programming on the HiPACE Cluster
The Distributed Shell - dsh

Perhaps the easiest way to distribute a programming project is to use dsh. It will run linux commands on all nodes or on particular compute nodes. If your batch programs use command-line arguments you can pass different parameters to each program so that every compute node is working on a different part of the problem. We'll give an example shortly.

To run a command on all 34 compute nodes use the following command:

   dsh -a command
To run a command on a particular node (node 5 for example) use the command:
   dsh -w cnode5 command
Here's a simple example of how you can distribute a program that calculates data at various time intervals. This works if each time interval can be calculated independently of the other intervals. Let's say the program is called "simulation.exe" and we pass it a time value and a filename where the result will be stored. Create a file called "batch1.bash" with the following contents:
  dsh -w cnode1 simulation.exe 0.1 data01.txt
  dsh -w cnode2 simulation.exe 0.2 data02.txt
  dsh -w cnode3 simulation.exe 0.3 data03.txt
  dsh -w cnode4 simulation.exe 0.4 data04.txt
      . . .
The first line is a linux standard for a bash shell script. After the script is saved you need to give yourself execute access to it. Run the following command:
  chmod u+x batch1.bash
You can run the batch program on the command line like this:
If you have a lot of batch programs to run it is probably worth taking the time to write a program that generates the batch shell script as its output.

MPI - The Message Passing Interface

Using MPI you can write programs that share data between the various programs while they run on different compute nodes. Click on this link to see a great MPI Resource Page.

One important difference between MPI programs and non-MPI programs is how one compiles and runs the programs. Before compiling you should load the appropriate module for the MPI environment. For C programs type the following command:

  module load mpich/enet-gcc
You can see what modules are available with the command "module avail".

To compile a C program that has MPI calls in it, use the command:

  mpicc -o programname.exe programname.c
It will create an executable called "programname.exe".

As with C compiling with the gcc compiler, if you use math functions like sqrt, sin, cos, etc you should use the "-lm" option with mpicc.

To run the executable produced by mpicc use the command
  mpirun -np 10 programname.exe
where the -np option tells it to distribute the program over 10 nodes. It is a good idea to write your code in such a way that it will work no matter how many nodes you run it on. This will make it more flexible in the future when you may have to run it on fewer nodes than you wish due to down time or heavy load. However, sometimes a program needs to have an even number of nodes or even a power of 2.

A popular way to write MPI programs consists of writing one program that does different things depending on which compute node it runs on. The program uses calls to "MPI_Comm_size" and "MPI_Comm_rank" to determine how many nodes are being used in the problem and which particular node this program is running on. For example, if there are 10 nodes working on the problem, then "size" will be 10 and "rank" will be a number between 0 and 9, depending on the node.

Within the program we use an if to determine what to do on this particular node. In the following C example, the node of rank 0 opens a file and waits for data from the other nodes which all solve the problem for a different time value:

     . . .
    //open the data file and wait for data from other nodes
      float time=i*0.1;
         . . .
     . . .
In this example, if there are 10 nodes working on the problem (size=10) the node with rank=1 will work on times 0.1, 1.0, 1.9, etc. The node with rank=2 will work on times 0.2, 1.1, 2.0, etc.

Data can be passed between the nodes using MPI_Send and MPI_Recv. In the example above the node with rank=0 will call MPI_Recv in a for loop where we specify which compute node we are receiving data from (which is incremented in a round-robin fashion so the data can be written to the file in order). Compute nodes of rank=1 through 9 will use MPI_Send to send their results to the node of rank 0.

Notice that the for loop that assigns time values to work on ensures that the time values are processed in an efficient way. The time values 0.1 through 0.9 are all being done concurrently and values 1.0 through 1.8 are done next (presumably concurrently, unless some nodes fall behind). Since the node with rank=0 is blocked waiting for the results in order of time value, it is important to get the time values processed somewhat in order. The calls to MPI_Send and MPI_Recv will synchronize the nodes to a degree, so it is important to proceed in a way that results in the shortest blocking time.

For the coding details see this MPI Tutorial.