split dwarf doesn't work with crate dependencies

[davidtwco]

As reported in #77117 (comment) (cc @philipc), Split DWARF currently fails when compiling a crate with a dependency.

To reproduce this, compile backtrace-rs with the following command:

$ RUSTFLAGS="-Z split-dwarf=split" cargo +nightly --verbose build

[davidtwco]

As of writing, here's what I've figured out - using backtrace-rs as an example, as that was the crate which was used when reporting the bug:

Compiling backtrace-rs with RUSTFLAGS="-Z split-dwarf=split" cargo +nightly --verbose build produces the following output:

  Compiling autocfg v1.0.1
   Compiling libc v0.2.82
   Compiling gimli v0.23.0
   Compiling adler v0.2.3
   Compiling cfg-if v1.0.0
   Compiling rustc-demangle v0.1.18
   Compiling object v0.22.0
     Running `rustc --crate-name autocfg /home/david/.cargo/registry/src/github.com-1ecc6299db9ec823/autocfg-1.0.1/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type lib --emit=dep-info,metadata,link -Cembed-bitcode=no -C debuginfo=2 -C metadata=f40b69de65cfa16a -C extra-filename=-f40b69de65cfa16a --out-dir /home/david/Projects/rust/backtrace-rs/target/debug/deps -L dependency=/home/david/Projects/rust/backtrace-rs/target/debug/deps --cap-lints allow -Z split-dwarf=split`
     Running `rustc --crate-name build_script_build /home/david/.cargo/registry/src/github.com-1ecc6299db9ec823/libc-0.2.82/build.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type bin --emit=dep-info,link -Cembed-bitcode=no -C debuginfo=2 -C metadata=299960e7a1692464 -C extra-filename=-299960e7a1692464 --out-dir /home/david/Projects/rust/backtrace-rs/target/debug/build/libc-299960e7a1692464 -L dependency=/home/david/Projects/rust/backtrace-rs/target/debug/deps --cap-lints allow -Z split-dwarf=split`
     Running `rustc --crate-name gimli --edition=2018 /home/david/.cargo/registry/src/github.com-1ecc6299db9ec823/gimli-0.23.0/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -Cembed-bitcode=no -C debuginfo=2 --cfg 'feature="read"' -C metadata=1558b68d5687196a -C extra-filename=-1558b68d5687196a --out-dir /home/david/Projects/rust/backtrace-rs/target/debug/deps -L dependency=/home/david/Projects/rust/backtrace-rs/target/debug/deps --cap-lints allow -Z split-dwarf=split`
     Running `rustc --crate-name cfg_if --edition=2018 /home/david/.cargo/registry/src/github.com-1ecc6299db9ec823/cfg-if-1.0.0/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -Cembed-bitcode=no -C debuginfo=2 -C metadata=ce59338d636be0e5 -C extra-filename=-ce59338d636be0e5 --out-dir /home/david/Projects/rust/backtrace-rs/target/debug/deps -L dependency=/home/david/Projects/rust/backtrace-rs/target/debug/deps --cap-lints allow -Z split-dwarf=split`
     Running `rustc --crate-name adler /home/david/.cargo/registry/src/github.com-1ecc6299db9ec823/adler-0.2.3/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -Cembed-bitcode=no -C debuginfo=2 -C metadata=b66c5323933d2b31 -C extra-filename=-b66c5323933d2b31 --out-dir /home/david/Projects/rust/backtrace-rs/target/debug/deps -L dependency=/home/david/Projects/rust/backtrace-rs/target/debug/deps --cap-lints allow -Z split-dwarf=split`
     Running `rustc --crate-name rustc_demangle /home/david/.cargo/registry/src/github.com-1ecc6299db9ec823/rustc-demangle-0.1.18/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -Cembed-bitcode=no -C debuginfo=2 -C metadata=882a963021f29db3 -C extra-filename=-882a963021f29db3 --out-dir /home/david/Projects/rust/backtrace-rs/target/debug/deps -L dependency=/home/david/Projects/rust/backtrace-rs/target/debug/deps --cap-lints allow -Z split-dwarf=split`
     Running `rustc --crate-name object --edition=2018 /home/david/.cargo/registry/src/github.com-1ecc6299db9ec823/object-0.22.0/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -Cembed-bitcode=no -C debuginfo=2 --cfg 'feature="archive"' --cfg 'feature="coff"' --cfg 'feature="elf"' --cfg 'feature="macho"' --cfg 'feature="pe"' --cfg 'feature="read_core"' --cfg 'feature="unaligned"' -C metadata=fdd8b8c7c909b622 -C extra-filename=-fdd8b8c7c909b622 --out-dir /home/david/Projects/rust/backtrace-rs/target/debug/deps -L dependency=/home/david/Projects/rust/backtrace-rs/target/debug/deps --cap-lints allow -Z split-dwarf=split`
     Running `/home/david/Projects/rust/backtrace-rs/target/debug/build/libc-299960e7a1692464/build-script-build`
   Compiling miniz_oxide v0.4.3
     Running `rustc --crate-name build_script_build --edition=2018 /home/david/.cargo/registry/src/github.com-1ecc6299db9ec823/miniz_oxide-0.4.3/build.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type bin --emit=dep-info,link -Cembed-bitcode=no -C debuginfo=2 -C metadata=6f3f48770204354e -C extra-filename=-6f3f48770204354e --out-dir /home/david/Projects/rust/backtrace-rs/target/debug/build/miniz_oxide-6f3f48770204354e -L dependency=/home/david/Projects/rust/backtrace-rs/target/debug/deps --extern autocfg=/home/david/Projects/rust/backtrace-rs/target/debug/deps/libautocfg-f40b69de65cfa16a.rlib --cap-lints allow -Z split-dwarf=split`
     Running `rustc --crate-name libc /home/david/.cargo/registry/src/github.com-1ecc6299db9ec823/libc-0.2.82/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -Cembed-bitcode=no -C debuginfo=2 -C metadata=55acfc5d67d2fc7b -C extra-filename=-55acfc5d67d2fc7b --out-dir /home/david/Projects/rust/backtrace-rs/target/debug/deps -L dependency=/home/david/Projects/rust/backtrace-rs/target/debug/deps --cap-lints allow -Z split-dwarf=split --cfg freebsd11 --cfg libc_priv_mod_use --cfg libc_union --cfg libc_const_size_of --cfg libc_align --cfg libc_core_cvoid --cfg libc_packedN --cfg libc_cfg_target_vendor`
error: linking dwarf objects with `rust-llvm-dwp` failed: exit code: 1
  |
  = note: "rust-llvm-dwp" "-e" "/home/david/Projects/rust/backtrace-rs/target/debug/build/miniz_oxide-6f3f48770204354e/build_script_build-6f3f48770204354e" "-o" "/home/david/Projects/rust/backtrace-rs/target/debug/build/miniz_oxide-6f3f48770204354e/build_script_build-6f3f48770204354e.dwp"
  = note:
  = note: error: No such file or directory


error: aborting due to previous error

     Running `/home/david/Projects/rust/backtrace-rs/target/debug/build/miniz_oxide-6f3f48770204354e/build-script-build`
^C⏎ 

llvm-dwp expects to be able to find all of the dwo files referenced by the linked build_script_build-6f3f48770204354e binary. Using readelf -wi, we can inspect the DW_AT_GNU_dwo_name attributes to find the expected filenames (prepending DW_AT_comp_dir for the complete path). rustc should have kept DWARF object files around at this point in compilation, so "No such file or directory" error suggests a failure in that mechanism. However, there are DWARF object files referenced from previous compilations:

  Compilation Unit @ offset 0x2259:
   Length:        0x30 (32-bit)
   Version:       4
   Abbrev Offset: 0x447
   Pointer Size:  8
 <0><2264>: Abbrev Number: 1 (DW_TAG_compile_unit)
    <2265>   DW_AT_stmt_list   : 0xa76
    <2269>   DW_AT_comp_dir    : (indirect string, offset: 0x16b6): /home/david/Projects/rust/backtrace-rs/target/debug/deps
    <226d>   DW_AT_GNU_dwo_name: (indirect string, offset: 0x16ef): autocfg-f40b69de65cfa16a.autocfg.4pya73nn-cgu.0.rcgu.dwo
    <2271>   DW_AT_GNU_dwo_id  : 0x532a2a4a633d9dea
    <2279>   DW_AT_GNU_ranges_base: 0xf0
    <227d>   DW_AT_low_pc      : 0x0
    <2285>   DW_AT_ranges      : 0x4f0
    <2289>   DW_AT_GNU_addr_base: 0x70

During earlier compilation of autocfg, the DWARF object file referenced would have been created, but then deleted at the end of the compilation. This explains why adding -Csave-temps to the RUSTFLAGS environment variable (as suggested in the initial report) fixes the issue. If we added -Csave-temps to only the failing compiler invocation then that would be insufficient to resolve the issue.

autocfg's compilation does not produce a DWARF package because rustc will only produce one when it is producing an output binary. Even if a DWARF package had been produced, it would have been insufficient to resolve the problem, as llvm-dwp only loads DWARF object files (DWARF package files are only consumed by the debugger, and DW_AT_GNU_dwo_name is never actually modified to point to the package file, debuggers just know to look for sibling .dwp files of binaries):

https://github.com/llvm/llvm-project/blob/5cf2696317afb0631a4a09414ae40a4e226a905e/llvm/tools/llvm-dwp/llvm-dwp.cpp#L510-L537

Therefore, it is necessary that DWARF object files from dependencies still exist during the compilation of dependent crates (in ./target/{debug,release}/deps, the output directory of the dependency's compilation).

---

I've come up with a some potential approaches which would resolve this issue, and I'd appreciate feedback:

Adding a DWARF object output type

Unlike other output files of the compiler, DWARF objects are not requested with the --emit flag:

rust/compiler/rustc_session/src/config.rs

Lines 241 to 252 in 4275ef6

	#[derive(Clone, Copy, PartialEq, Eq, Hash, Debug, PartialOrd, Ord)]
	#[derive(Encodable, Decodable)]
	pub enum OutputType {
	Bitcode,
	Assembly,
	LlvmAssembly,
	Mir,
	Metadata,
	Object,
	Exe,
	DepInfo,
	}

Instead, when Split DWARF is enabled: DWARF object files are produced whenever an object file is emitted (we produce DWARF objects alongside object files in the same LLVMRustWriteOutputFile invocation, by providing a buffer as an additional argument to LLVM's addPassesToEmitFile); DWARF package files are created whenever a OutputType::Exe is requested; and DWARF object files are removed whenever we would remove object files.

We could choose to add a OutputType::DwarfObject, which would allow Cargo to pass --emit=dwarf-object,dep-info,metadata,link to the invocation of rustc for dependencies. When --emit=dwarf-object is provided, DWARF object files will not be removed at the end of the compiler invocation. Necessarily when --emit=dwarf-object is requested, object files would have to be created, but these would be removed if only DWARF objects were explicitly requested.

Through this mechanism, we could ensure that the DWARF object files of dependencies continue to exist after the compilation session and thus would be available for dependent compilations.

Adding DWARF objects to rlibs

Alternatively, we could add DWARF object files to rlib files during linking, and then in the dependent compilation, before invoking llvm-dwp, read the DWARF object files from the rlib and write them to the expected location.

Adding DWARF objects to the rlib would require appending the following snippet..

    for dwo in codegen_results.modules.iter().filter_map(|m| m.dwarf_object.as_ref()) {
        ab.add_file(dwo);
    }

..after this part of the compiler:

rust/compiler/rustc_codegen_ssa/src/back/link.rs

Lines 302 to 314 in 4275ef6

	fn link_rlib<'a, B: ArchiveBuilder<'a>>(
	sess: &'a Session,
	codegen_results: &CodegenResults,
	flavor: RlibFlavor,
	out_filename: &Path,
	tmpdir: &MaybeTempDir,
	) -> B {
	info!("preparing rlib to {:?}", out_filename);
	let mut ab = <B as ArchiveBuilder>::new(sess, out_filename, None);
	for obj in codegen_results.modules.iter().filter_map(|m| m.object.as_ref()) {
	ab.add_file(obj);
	}

Reading DWARF objects from the rlib would require similar code to the following function, but with a different value for METADATA_FILENAME (instead, this would be the filename of the DWARF object which should be retrieved):

rust/compiler/rustc_codegen_llvm/src/metadata.rs

Lines 21 to 41 in 4275ef6

	fn get_rlib_metadata(&self, _: &Target, filename: &Path) -> Result<MetadataRef, String> {
	// Use ArchiveRO for speed here, it's backed by LLVM and uses mmap
	// internally to read the file. We also avoid even using a memcpy by
	// just keeping the archive along while the metadata is in use.
	let archive =
	ArchiveRO::open(filename).map(|ar| OwningRef::new(Box::new(ar))).map_err(|e| {
	debug!("llvm didn't like `{}`: {}", filename.display(), e);
	format!("failed to read rlib metadata in '{}': {}", filename.display(), e)
	})?;
	let buf: OwningRef<_, [u8]> = archive.try_map(|ar| {
	ar.iter()
	.filter_map(|s| s.ok())
	.find(|sect| sect.name() == Some(METADATA_FILENAME))
	.map(|s| s.data())
	.ok_or_else(|| {
	debug!("didn't find '{}' in the archive", METADATA_FILENAME);
	format!("failed to read rlib metadata: '{}'", filename.display())
	})
	})?;
	Ok(rustc_erase_owner!(buf))
	}

(The snippet above is from rustc_codegen_llvm, rustc_codegen_cranelift has a similar function)

If we don't already, we would need to record the codegen modules produced for the dependency. This would be required to know which sections to load from the rlib and write out for the llvm-dwp invocation to succeed.

This is quite a bit more complicated than the first solution, but limits the required changes to only rustc.

Always keep DWARF objects for library crates

Similar to "Adding a DWARF object output type" described above, but instead of having Cargo explicitly request that DWARF objects are kept, infer that a crate is a dependency by checking the crate type is a library and that an rlib is also being produced.

---

I'd appreciate any feedback on which of the above solutions is desirable, or if anyone has any alternatives that I haven't thought of yet.

[bjorn3]

Alternatively, we could add DWARF object files to rlib files during linking, and then in the dependent compilation, before invoking llvm-dwp, read the DWARF object files from the rlib and write them to the expected location.

I was under the impression that this already happend. At least my FCP proposal for -Csplit-dwarf was made with this assumption.

[davidtwco]

Alternatively, we could add DWARF object files to rlib files during linking, and then in the dependent compilation, before invoking llvm-dwp, read the DWARF object files from the rlib and write them to the expected location.

I was under the impression that this already happend. At least my FCP proposal for -Csplit-dwarf was made with this assumption.

I assumed that the -Csplit-debuginfo proposal wouldn't enable Split DWARF initially, only -Zrun-dsymutil, since it had only been in nightly for a short time.

[bjorn3]

-Csplit-debuginfo will be gated behind -Zunstable-options for all targets other than macOS, but it is supposed to work for both Linux and Windows.

[apiraino]

Nominated for discussion for next week's compiler meeting (Zulip discussion)