• Setup
• How to profile code
  • Look for “slow” sections
• Making improvements
• Single node parallel
  • Multiple cpus, 1 computer
• Multi node parallel
  • Multiple cpus, multiple computers

Get presentation materials

git clone https://github.com/uschpc/workshop-hpc-python.git

• Easiest to do from conda environment
• From Discovery/Endeavour cluster initialize conda with

module purge
module load conda
eval "$(conda shell.bash hook)"

• Initialize environment

conda env create -f environment.yml

1. Profile code to understand the execution time and memory use of each part
2. Identify bottlenecks (i.e., parts of code that take the most time)
3. Try to improve performance of bottlenecks by modifying code
4. Benchmark alternative code to identify best alternative

Eventually you need to stop

• Aim for fast enough code
  
• Compute time is less expensive than human time

This snippet is not great

def write_data(x,y,n,t):
    filename=("output/%s%05d" %(output,i))

    i_max=len(x)
    j_max=len(y)
    data=np.zeros((i_max,j_max))

    for i in range(i_max):
        for j in range(j_max):
            data[i,j]=generate_data(x[i],y[j],nFiles,fileID)

    np.savetxt(filename,data)

where generate_data() is defined as:

def generate_data(x,y,n,t):
    value = (x/10)**2*(y/10) + (y/10)**2*(x/10)+t/n
    value = value %1 # Keep number between 0 and 1
    return value

• n - number files to be generated
• t - which file to generate

• Loops in Python tend to be slow
• Apply same operation to each element
• Packages like numpy often have built-in vectorized versions
• Often written in C/C++/Fortran for performance
• Use built in functions when possible

numpy.meshgrid(*xi, copy=True, sparse=False, indexing='xy')

Return coordinate matrices from coordinate vectors. Make N-D coordinate arrays for vectorized evaluations of N-D scalar/vector fields over N-D grids, given one-dimensional coordinate arrays x1, x2,…, xn.

def write_data(X,Y,n,t):
    filename=("output/%s%05d" %(output,i))

    Z = generate_data(X,Y,nFiles,i)
    np.savetxt(filename,Z)

Where

x = np.arange(x_origin-size/2,x_origin+size/2,1)
y = np.arange(y_origin-size/2,y_origin+size/2,1)
X,Y = np.meshgrid(x,y)

• See example under examples/write_vectorized.py

python3 -m cProfile -s tottime examples/write_vectorized.py -n 100 write_vectorized > write_vectorized

Original write.py ~4.8s

New write_vectorized.py ~ 1.4s

• Our example has the line

    np.savetxt(filename,data)

• Generate many small files in plain, text
• Sacrifice readibility for performance?

    np.save(filename,data)

• We save time by writing less data, in binary
• numpy.savetxt
• numpy.save

write_vectorized.py ~1.4s

examples/write_vectorized_binary.py ~0.46s

• Performance is pretty good BUT
• ~40% of time is WAITING on opening files
• Keep in mind for later
• Assume we have fine tuned program
• Assume it’s worth our time to make it faster
• What next?

• Generally parallel means concurrent execution of tasks
• Depends on the needs of your program
• Doing other tasks while waiting for IO to finish
• Splitting up workload across “workers” (cpus/compute nodes)

• Some computations are not worth parallelizing
• Some costs to parallelizing (overhead):
  • changing code
  • spawning child processes
  • communications
• Speedup not proportional to number of cores (Amdahl’s law)
• Optimal number of cores
  • depends on specific computations
  • experiment to find
• Different issues pop up as you scale

• Currently script writes data files in sequence
• It may be possible to write multiple files at the same time

• Multiprocessing is a built-in package
• Create pool of processes
• Typical example, apply a function over multiple inputs

def f(x):
    return x*x

if __name__ == '__main__':
    with Pool(5) as p:
        print(p.map(f, [1, 2, 3]))

• Since we have multiple arguments

with mp.Pool(processes=nWorkers) as pool:
    for i in range(0,nFiles):
        pool.apply(write_data,args=(X,Y,output,nFiles,i,))

• Also add -w option for nWorkers
• See example under examples/write_multiprocessing.py

• Compute nodes have different configurations
  • number of cores
  • amount of memory
• On CARC systems:
  • Discovery Resource Overview
  • Enter sinfo2 in shell to see node types
  • 1 logical CPU = 1 core = 1 thread (--cpus-per-task)

#!/bin/bash

#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=8
#SBATCH --mem=16GB
#SBATCH --time=1:00:00
#SBATCH --account=<account_id>

module purge
module load conda
eval "$(conda shell.bash hook)"
conda activate hpc-python

python3 /path/to/script.py

• We can launch our script with something like python3 write_multiprocessing.py -w 3 -n 100
• Check example job script
  • examples/job-scripts/multicore.job

• Let’s use Viztracer to profile
• viztracer -o multiprocessing.json examples/write_multiprocessing.py -w 3 -n 100
• vizviewer multiprocessig.json
• use ‘wasd’ to navigate

• For 1 worker, ~6.7 s

• For 3 workers, ~7.6 s

• Despite more workers, no speedup
• There is communication overhead
• This line is problematic:

pool.apply(write_data,args=(X,Y,output,nFiles,i))

• During communication, variables are “pickled” or serialized
• We pass in X,Y,output,nFiles even though they never change

• For 1 worker, ~5.4 s

• For 3 workers, ~2.8 s

• examples/write_multiprocessing.py 3 workers, ~7.6 s

• examples/write_multiprocessing_queue.py 3 workers, ~2.8 s

• Workers communicate fairly quickly now
• Uneven distribution of work
• Program ends when LAST worker completes

• Larger files 500x500, 1K files
• As problem size increases, start up overhead is less important
• Workload is more even

• mpi4py allows you to add mpi functionality to your scripts
• Send messages across compute nodes
• Easier to scale up
  • Assuming it’s worth communication overhead
• Easier to read?

• For all other workers, receive data into local buffer

else:
    local_data=np.empty(int(nFiles/n_chunks),dtype='i')
    comm.Recv([local_data,MPI.INT],source=0, tag=77)

• Once data is distributed, everyone should process it

for i in local_data:
    write_data(X,Y,output,nFiles,i)

#!/bin/bash

#SBATCH --ntasks=16
#SBATCH --cpus-per-task=1
#SBATCH --mem-per-cpu=3GB
#SBATCH --time=1:00:00
#SBATCH --account=<account_id>

module purge
module load gcc/8.3.0
module load openblas/0.3.8
module load openmpi/4.0.2
module load pmix/3.1.3
module load python/3.7.6

srun --mpi=pmix_v2 python3 /path/to/script.py

• We can launch our script with something like srun --mpi=pmix_v2 python3 write_mpi.py -n 100
• Check example job script
  • examples/job-scripts/mpi.job

• In some cases file operations are a bottleneck
• We saw in one example ~ 40% of time was waiting to open a file
• It’s best to write one large file
• Libraries like hdf5, h5py allow multiple processes to write to same file
• Parallelization not needed for speedup

• HDF5 provides a way to store complicated datasets
• Instead of putting metadata in filename, HDF5 can manage it for you
• We will use very basic HDF5 features.
• We will save all of our files into 1 HDF5 file.

• We can access HDF5 functionality from h5py python library
• In examples/write_vectorized_binary.py we can modifying write_data()
• See examples/write_hdf5.py

def write_data(X,Y,output,nFiles,i,hf):

    filename=("output/%s%05d" %(output,i))
    Z = generate_data(X,Y,nFiles,i)

    hf.create_dataset(filename,data=Z)

• Where hf = h5py.File('output/data.h5', 'w')

examples/write_vectorized_binary.py ~5.46s (1000 writes)

examples/write_hdf5.py ~2.54s (1000 writes)

• Using h5py is always faster
• Performance benefit goes down
• Operating system optimizations?

• It’s a bit faster
• Additional code not much more complicated
• Environment is much more complicated
• Probably not worth the effort?

• multiprocesing
• mpi4py
  
• h5py
• hdf5
• cProfile
• snakeviz
• viztracer

• Questions?
  • carc-support@usc.edu
• Workshop schedule