Skip to content

Add recipe for tuning distributed calculations - #68

Open
RizzoHou wants to merge 1 commit into
dftbplus:mainfrom
RizzoHou:parallel-tuning-recipe
Open

Add recipe for tuning distributed calculations#68
RizzoHou wants to merge 1 commit into
dftbplus:mainfrom
RizzoHou:parallel-tuning-recipe

Conversation

@RizzoHou

Copy link
Copy Markdown

The parallel/ chapter currently covers how many processors to use, but the
"Distributed memory parallelism" part of benchmarking.rst is a placeholder,
and introduction.rst notes that thread affinity settings are "beyond the
scope of this tutorial". This adds a page covering that ground: the settings
that decide how well a given number of processors is actually used.

New page parallel/tuning.rst, covering

  • reading the internal timers first, so that effort goes where the time is,
  • solver choice, including why a linear scaling solver is not automatically
    faster below its crossover size, and why ELPA's autotuner can cost more
    than it saves for a short SCC cycle,
  • splitting node cores between MPI ranks and OpenMP threads, and the
    UseOmpThreads safety net,
  • process and thread pinning, with two silent traps: setting
    OMP_NUM_THREADS inline rather than exporting it before the launcher, so
    that ranks get pinned to a single core while the code still believes it
    has several threads; and asynchronous progress threads being spawned per
    rank rather than per node, which deadlocks any layout that already fills
    every core,
  • Parallel { Blacs { BlockSize } }, which is an input-file-only experiment,
  • ELPA's own kernel selection environment variables,
  • forcing MPI collective algorithms, and checking the fabric selection,
  • a scanning method, including measuring the run-to-run scatter first and
    checking that the SCC iteration count and total energy are unchanged before
    believing a timing difference.

The numbers quoted are measurements on a 16,000 atom SiC supercell over four
nodes of 2 x 28-core Xeon 8280 with Intel MPI and MKL, made while preparing an
ISC student cluster competition entry. They are presented as indications of the
size of each effect rather than as recommended values, since every one of them
depends on machine, libraries and problem.

Two small edits to existing files: a label on benchmarking.rst so the new
page can link to it, and a pointer to the new page from the placeholder
section. The topic list in that placeholder is left alone, since the remaining
items (scaling examples, Groups, latency) are not covered here.

Built locally with Sphinx, no new warnings.

Documents the settings that determine how efficiently a given number of
processors is used for large self-consistent calculations: solver choice,
the split between MPI ranks and OpenMP threads, process and thread
pinning, the BLACS block size, ELPA kernel selection and the choice of
MPI collective algorithms.

Also describes a method for scanning settings, including the checks that
distinguish a genuine speed-up from a changed calculation, and the traps
that silently cost performance without changing results.

The measurements quoted are for a 16,000 atom SiC supercell on four Xeon
8280 nodes with Intel MPI and MKL, and are given to show the size of the
effects rather than as recommended values.

Signed-off-by: Yuze Hou <rizzohou@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant