ls-apis should use package-manifest.toml to figure out which version of related repos to use

[davepacheco]

(depends on #10217)

This change causes ls-apis to parse package-manifest.toml to figure out what commits of related repos (like Crucible, Dendrite, etc.) will actually be deployed (from the current Omicron workspace). It then uses this information to choose the correct clone of the repo to use for its analysis.

There are a few other changes here I had to make (like bringing lldpd-client's deps in sync). I will probably split these out into separate PRs. I will likely wait for R19 because I'm not sure what the impact of updating these components is.

Background

ls-apis needs access to checked-out repos for Omicron as well as related components like Dendrite, LLDP, Crucible, Propolis, etc. It wants the versions of these repos that get deployed on real systems (based on the Omicron workspace that it's running in), since the goal is to analyze the runtime API dependencies between these components. It could create its own clones of these repos, but instead, it leverages the fact that just running cargo metadata in Omicron requires having downloaded copies of all of these repos already. How does ls-apis find these copies? It uses Cargo to locate a package that's known to be in that repo. Generally, it picks the package of a client that Omicron already from that repo, like dpd-client to find Dendrite.

But it's not quite so simple: Omicron can reference multiple versions of a given repo. More specifically: Omicron may reference dpd-client from multiple versions of Dendrite. This happens with dpd-client specifically:

$ cargo tree -e normal -i dpd-client
error: There are multiple `dpd-client` packages in your project, and the specification `dpd-client` is ambiguous.
Please re-run this command with one of the following specifications:
  git+https://github.com/oxidecomputer/dendrite?branch=main#dpd-client@0.1.0
  git+https://github.com/oxidecomputer/dendrite?rev=44a949c9bedf4fcd4d280337fa1965b4293c88d1#dpd-client@0.1.0
  git+https://github.com/oxidecomputer/dendrite?rev=cc8e02a0800034c431c8cf96b889ea638da3d194#dpd-client@0.1.0

$ cargo tree -e normal -i git+https://github.com/oxidecomputer/dendrite?branch=main#dpd-client@0.1.0
dpd-client v0.1.0 (https://github.com/oxidecomputer/dendrite?branch=main#f20f786e)
└── lldpd-common v0.1.0 (https://github.com/oxidecomputer/lldp?rev=c3305fd1a7ea7aba31f3834757a6b931e4f59fe6#c3305fd1)
    └── lldpd-client v0.1.0 (https://github.com/oxidecomputer/lldp?rev=c3305fd1a7ea7aba31f3834757a6b931e4f59fe6#c3305fd1)
        └── omicron-nexus v0.1.0 (/home/dap/omicron-review/nexus)
            └── omicron-dev v0.1.0 (/home/dap/omicron-review/dev-tools/omicron-dev)

$ cargo tree -e normal -i git+https://github.com/oxidecomputer/dendrite?rev=44a949c9bedf4fcd4d280337fa1965b4293c88d1#dpd-client@0.1.0
dpd-client v0.1.0 (https://github.com/oxidecomputer/dendrite?rev=44a949c9bedf4fcd4d280337fa1965b4293c88d1#44a949c9)
├── nexus-test-utils v0.1.0 (/home/dap/omicron-review/nexus/test-utils)
│   └── omicron-dev v0.1.0 (/home/dap/omicron-review/dev-tools/omicron-dev)
├── omicron-nexus v0.1.0 (/home/dap/omicron-review/nexus)
│   └── omicron-dev v0.1.0 (/home/dap/omicron-review/dev-tools/omicron-dev)
├── wicket-common v0.1.0 (/home/dap/omicron-review/wicket-common)
│   ├── wicket v0.1.0 (/home/dap/omicron-review/wicket)
│   │   └── wicket-dbg v0.1.0 (/home/dap/omicron-review/wicket-dbg)
│   ├── wicketd v0.1.0 (/home/dap/omicron-review/wicketd)
│   ├── wicketd-api v0.1.0 (/home/dap/omicron-review/wicketd-api)
│   │   ├── omicron-dropshot-apis v0.1.0 (/home/dap/omicron-review/dev-tools/dropshot-apis)
│   │   └── wicketd v0.1.0 (/home/dap/omicron-review/wicketd)
│   └── wicketd-client v0.1.0 (/home/dap/omicron-review/clients/wicketd-client)
│       ├── wicket v0.1.0 (/home/dap/omicron-review/wicket) (*)
│       └── wicketd v0.1.0 (/home/dap/omicron-review/wicketd)
└── wicketd v0.1.0 (/home/dap/omicron-review/wicketd)

$ cargo tree -e normal -i git+https://github.com/oxidecomputer/dendrite?rev=cc8e02a0800034c431c8cf96b889ea638da3d194#dpd-client@0.1.0
dpd-client v0.1.0 (https://github.com/oxidecomputer/dendrite?rev=cc8e02a0800034c431c8cf96b889ea638da3d194#cc8e02a0)
└── omicron-sled-agent v0.1.0 (/home/dap/omicron-review/sled-agent)
    ├── end-to-end-tests v0.1.0 (/home/dap/omicron-review/end-to-end-tests)
    └── nexus-test-utils v0.1.0 (/home/dap/omicron-review/nexus/test-utils)
        └── omicron-dev v0.1.0 (/home/dap/omicron-review/dev-tools/omicron-dev)

This is almost certainly not great. But it shouldn't cause ls-apis to break. Right now if this happens, ls-apis picks one of these arbitrarily, which can cause it to analyze the wrong version of our software and draw wrong conclusions. This is the real cause of #10214.

Again: we want ls-apis to be looking at the version of these things that gets deployed. How can it know which one it is? The authoritative version is the one in package-manifest.toml. Hence the solution here: parse that file, find the commit being used there, and choose the version of the package that corresponds to that commit.

Other notes

This is still a little cheesy in a few ways:

• to determine if it's the right commit, we do a string comparison on the "source", which is basically a git URL
• it's still using this sort of goofy heuristic (find a known package referenced by Omicron but contained in the other repo) -- just to be able to re-use Cargo's clone of the repo. It's pretty nice to not have to manage a separate set of clones (and make sure they're correct, have no local changes, etc.), but it's kind of a weird assumption to make and it would break if we ever had a repo we cared about where Omicron doesn't reference one of its packages.

but I think it's a meaningful improvement.

One other note: this will break in the future if:

• Omicron has no reference at all to a package in the other repo (the case above -- this would already have broken before)
• Omicron has no reference to a package in the other repo at the same Git commit as the one in package-manifest.toml. This would be unlikely, since we usually move all of our deps forward at the same time. But there's a notable exception today: one of the dpd-client paths above is deliberately fixed to an old version for upgrade-related reasons. This works out fine though because there's another reference to dpd-client that is the right version.
• package-manifest.toml references the same repo multiple times with different commits. Again, this seems unlikely, and the problem is deeper than it looks. That means we have two different versions of something deployed and ls-apis needs to analyze both?

TODO

• split out changes mentioned above (e.g., lldpd-client deps)

[davepacheco]

This appears to have been totally superfluous after #7907, which added "dendrite" to the block above instead.

[davepacheco]

Most of our other packages have specific Git commits in Cargo.toml and that seems appropriate so I've done that here. In practice, this also updates this dependency. I now see that was a breaking change so there's more work here to be done. I'll split this into a separate PR.

[davepacheco]

This updates lldp to latest. I will split this into a separate PR (along with the lldpd-client dep change). I'm not sure what the implications are so I'll likely wait til the next release.