OOM (huge memory usage) during release compilation with include_str of 100MB

[oberien]

When compiling a project of mine, my PC with 32GB of RAM went OOM. I reduced the problem to the following code:

use std::collections::HashMap;

static TESTDATA: &str = include_str!("../testdata");

fn parse_testdata() -> HashMap<(), ()> {
    let mut map = HashMap::new();
    for line in TESTDATA.lines() {
    }
    map
}

pub fn do_stuff() {
    let (tx, _) = std::sync::mpsc::channel();
    let testdata = parse_testdata();
    tx.send(vec![0]).unwrap();
}

fn main() {
    do_stuff();
}

Compiling this code with /usr/bin/time -v cargo build --release shows a maximum resident RAM usage of 16GB if testdata is generated with python -c "print('foo bar baz qux 1337 42\n' * 4_500_000)" > testdata (103MB). In debug mode, only 1.3GB RAM are used. It doesn't matter if edition is 2018 or not.

Here are some different testdata sizes with their compilation memory usage:

python generation	size	RAM	comment
'foo bar baz qux 1337 42\n' * 4_500_000	103MB	16GB	file structure similar to my real-world use-case
'foo bar baz qux 1337 42 ' * 4_500_000	103MB	16GB	newlines don't matter
'a' * 100*1024*1024	100MB	15.7GB	we can reduce this to simple file size
'a' * 80*1024*1024	80MB	14GB
'a' * 50*1024*1024	50MB	8GB
'a' * 25*1024*1024	25MB	3.9GB
'a' * 10*1024*1024	10MB	1.9GB
'a' * 1*1024*1024	1MB	256MB

If anything in the code is changed, the problem gets slightly better. Here is the RAM usage after some small changes for 100MB (for comparison, include_str just with a hello-world requires 670MB):

• use a Vec instead of the HashMap: 2.5GB
• create the HashMap at the end of of the parse_testdata function: 6GB
• inline the parse_testdata function manually: 6GB
• make parse_testdata not return anything and remove the HashMap from the function: 2.5GB
• send () instead of vec![0]: 6GB
• remove the channel creation and sending into it: 4.4GB
• don't call do_stuff: 672MB

Meta

uname -a: Linux 5.5.4-arch1-1-vfio #1 SMP PREEMPT Wed, 19 Feb 2020 15:49:02 +0000 x86_64 GNU/Linux (Archlinux)

Tested on (always testing with 100MB):

• current stable: rustc 1.42.0 (b8cedc004 2020-03-09): 16GB
• current nightly: rustc 1.43.0-nightly (45ebd5808 2020-03-15): 16GB
• rustc 1.41.1 (f3e1a954d 2020-02-24): 16GB
• rustc 1.41.0 (5e1a79984 2020-01-27): 16GB
• rustc 1.40.0 (73528e339 2019-12-16): 2.5GB
• rustc 1.39.0 (4560ea788 2019-11-04): 2.5GB
• rustc 1.38.0 (625451e37 2019-09-23): 6GB
• rustc 1.35.0 (3c235d560 2019-05-20): 6GB
• rustc 1.34.0 (91856ed52 2019-04-10): 6GB
• rustc 1.32.0 (9fda7c223 2019-01-16): 8GB
• rustc 1.30.0 (da5f414c2 2018-10-24): 5GB

[main--]

Automated testing:

Running HashMap<(), ()> tests
Testing 'foo bar baz qux 1337 42\n' * 1_000_000 ... file=23MiB ... ram=4,6GiB
Testing 'foo bar baz qux 1337 42 ' * 1_000_000 ... file=23MiB ... ram=4,6GiB
Testing 'a' * 25*1024*1024 ... file=25MiB ... ram=4,8GiB
Testing 'a' * 10*1024*1024 ... file=10MiB ... ram=2,3GiB
Testing 'a' * 1*1024*1024 ... file=1,0MiB ... ram=286MiB

Running Vec<()> tests
Testing 'foo bar baz qux 1337 42\n' * 1_000_000 ... file=23MiB ... ram=1,2GiB
Testing 'foo bar baz qux 1337 42 ' * 1_000_000 ... file=23MiB ... ram=1,1GiB
Testing 'a' * 25*1024*1024 ... file=25MiB ... ram=1,2GiB
Testing 'a' * 10*1024*1024 ... file=10MiB ... ram=592MiB
Testing 'a' * 1*1024*1024 ... file=1,0MiB ... ram=163MiB

rustc 1.42.0 (b8cedc004 2020-03-09)

run_test() {
    echo -n "Testing $1 ... "
    file="$(mktemp)"
    outfile="$(mktemp)"
    python -c "print($1, end='')" > "$file"
    echo -n "file=$(numfmt --to=iec-i --suffix=B $(stat --printf="%s" "$file")) ... "

    kb="$(/usr/bin/time -v rustc -A warnings -O -o "$outfile" /dev/stdin <<EOT 2>&1 | grep 'Maximum resident set size' | cut -d ':' -f 2
use std::collections::HashMap;

static TESTDATA: &str = include_str!("$file");

fn parse_testdata() -> $outtype {
    let mut map = Default::default();
    for line in TESTDATA.lines() {
    }
    map
}

pub fn do_stuff() {
    let (tx, _) = std::sync::mpsc::channel();
    let testdata = parse_testdata();
    tx.send(vec![0]).unwrap();
}

fn main() {
    do_stuff();
}
EOT
)"


    echo "ram=$(numfmt --to=iec-i --suffix=B --from-unit 1024 $kb)"
    rm "$file" "$outfile"
}

run_all() {
    echo "Running $outtype tests"
    run_test "'foo bar baz qux 1337 42\n' * 1_000_000"
    run_test "'foo bar baz qux 1337 42 ' * 1_000_000"
    run_test "'a' * 25*1024*1024"
    run_test "'a' * 10*1024*1024"
    run_test "'a' * 1*1024*1024"
    echo
}

outtype="HashMap<(), ()>" run_all
outtype="Vec<()>" run_all
rustc --version

[jonas-schievink]

Duplicate of #52380

[oberien]

I'm not sure, if #52380 is the same bug. Reproducing the case in #52380 with the 100MB file only results in "only" 670MB RAM being used ("for comparison, include_str just with a hello-world requires 670MB"). Compiling a plain hello world takes 115MB of RAM. While this means that including a file increases the RAM usage by roughly 5 times that file's size, it's nowhere near the factor of 160 times in my example.

[jonas-schievink]

I'm pretty sure they're caused by the same thing though. This one does look like a pretty hefty regression though, so reopening.

[spastorino]

@rustbot ping cleanup, would be nice to find why it regressed so bad between 1.40 and 1.41.