Add recipe for tuning distributed calculations - #68
Open
RizzoHou wants to merge 1 commit into
Open
Conversation
Documents the settings that determine how efficiently a given number of processors is used for large self-consistent calculations: solver choice, the split between MPI ranks and OpenMP threads, process and thread pinning, the BLACS block size, ELPA kernel selection and the choice of MPI collective algorithms. Also describes a method for scanning settings, including the checks that distinguish a genuine speed-up from a changed calculation, and the traps that silently cost performance without changing results. The measurements quoted are for a 16,000 atom SiC supercell on four Xeon 8280 nodes with Intel MPI and MKL, and are given to show the size of the effects rather than as recommended values. Signed-off-by: Yuze Hou <rizzohou@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The
parallel/chapter currently covers how many processors to use, but the"Distributed memory parallelism" part of
benchmarking.rstis a placeholder,and
introduction.rstnotes that thread affinity settings are "beyond thescope of this tutorial". This adds a page covering that ground: the settings
that decide how well a given number of processors is actually used.
New page
parallel/tuning.rst, coveringfaster below its crossover size, and why ELPA's autotuner can cost more
than it saves for a short SCC cycle,
UseOmpThreadssafety net,OMP_NUM_THREADSinline rather than exporting it before the launcher, sothat ranks get pinned to a single core while the code still believes it
has several threads; and asynchronous progress threads being spawned per
rank rather than per node, which deadlocks any layout that already fills
every core,
Parallel { Blacs { BlockSize } }, which is an input-file-only experiment,checking that the SCC iteration count and total energy are unchanged before
believing a timing difference.
The numbers quoted are measurements on a 16,000 atom SiC supercell over four
nodes of 2 x 28-core Xeon 8280 with Intel MPI and MKL, made while preparing an
ISC student cluster competition entry. They are presented as indications of the
size of each effect rather than as recommended values, since every one of them
depends on machine, libraries and problem.
Two small edits to existing files: a label on
benchmarking.rstso the newpage can link to it, and a pointer to the new page from the placeholder
section. The topic list in that placeholder is left alone, since the remaining
items (scaling examples,
Groups, latency) are not covered here.Built locally with Sphinx, no new warnings.