Anytopo

[ziyuezzy]

Hi Devs,

This is a pull request, previously mentioned in issue #2584

There are documentations of this work in
"sst-elements-src/src/sst/elements/merlin/interfaces/endpointNIC/documentations/README.md", and
"sst-elements-src/src/sst/elements/merlin/topology/anytopo_ultility/README.md"

In a few words, this branch extends the Merlin module in SST-element, such that any network topology can be built with an input networkx graph. Another key feature is to support source routing (the endpoint NIC determines the routed paths). And a few popular HPC network topologies (dragonfly, slimfly, polarfly, jellyfish) are already defined in
"sst-elements-src/src/sst/elements/merlin/topology/anytopo_ultility/" dir, together with some tests in "sst-elements-src/src/sst/elements/merlin/topology/anytopo_ultility/tests".

If you would like to accept this pull requests, maybe we can move these tests to the main merlin test dir?

It worth mention that the reorderedlinkcontrol has been modified to fit in the new 'ExtendedRequest' framework, this has been included in tests as well, see EndpointNIC(use_reorderLinkControl=True ... in the tests.

I believe that this pull request will have the following contribution to Merlin:

• supporting more network topologies (any topology from input graph)
• supporting source routing, with is increasingly interesting in HPC network traffic engineering
• Adds a framework of EndpointNIC that allows different NIC/smartNIC functionalities to be plugged in the packet-processing pipeline. For now source routing is implemented through this framework.

Please let me know if there is any feedback/comments/questions for the code.

Best regards,
Z.

[sst-autotester]

Status Flag 'Pre-Test Inspection' - - This Pull Request Requires Inspection... The code must be inspected by a member of the Team before Testing/Merging
NO INSPECTION HAS BEEN PERFORMED ON THIS PULL REQUEST! - This PR must be inspected by setting label 'AT: PRE-TEST INSPECTED'.

[feldergast]

In order to merge the PR, the tests will need to be part of the main merlin testing. Go ahead and move them there and we can review the code and release the PR for testing.

[ziyuezzy]

Before I add new tests, I want to make sure the current tests all pass.
I don't really understand why the 'macos-14 / macos-15' are failing here?

When I run 'sst-test-elements' locally (ubuntu 22, python3.8), all tests pass (see screenshot below).

For example the test "test_EmberSweep_*" all passed locally, but failed in the github action...
Does anyone have an idea what is wrong there?

[feldergast]

There are some known issues with the GitHub macOS runners and how they interact with SST. You can safely ignore the results of the GitHub macOS builders for now. They are experimental and the official testing is done on other machines.

[feldergast]

Looks like that those issues were actually fixed a week or so ago. I dug a little deeper and it looks like you introduced a dependance on a third-party library that isn't installed on those systems (NetworkX). You'll need to do a check for that and not load it if it's not there (i.e. anytopo won't be available if NetworkX isn't there. The polarfly and polarstar topos have a similar issue with a different library. Give me a bit to look through the code and I can send some pointers on the best way to do this.

[feldergast]

This looks like the code segment that is failing when networks isn't installed. It looks like this code is executing when the module is loaded, so the ImportError below ends up failing in the module loading. So, this causes every simulation that imports sst.merlin to fail, not just those that use topoany. Probably need to set a variable if the modules are missing and fail when trying to use the topology instead of at module load time.

[feldergast]

Looks like polarstar and polarfly depend on the same third-party Python libraries as anytopo, so we already have builders set up with those libraries, so anytopo should test on those and be skipped on all the ones without those libraries.

[sst-autotester]

Status Flag 'Pre-Test Inspection' - - This Pull Request Requires Inspection... The code must be inspected by a member of the Team before Testing/Merging
NO INSPECTION HAS BEEN PERFORMED ON THIS PULL REQUEST! - This PR must be inspected by setting label 'AT: PRE-TEST INSPECTED'.

[ziyuezzy]

Looks like polarstar and polarfly depend on the same third-party Python libraries as anytopo, so we already have builders set up with those libraries, so anytopo should test on those and be skipped on all the ones without those libraries.

Ok that's good news. Thanks for checking.
Will the github action automatically skip or continue with the tests, depending on whether or not networkx is installed? Or Should I add some indications in the test files?

Another question is that some of my tests require extra python libraries, for example I imported the library "galois" for generating the 'Slimfly; topology. This import is not built with SST, but only is required when running the test.
Is there any solution there?

[feldergast]

You'll need to add an @unittest.skipIf similar to what the polar* topology tests do so that they don't try to execute if the libraries aren't found. Note that the testsuite_Default_merlin.py file also tries to load the required libraries so that they can check if they are in sys.modules.

[sst-autotester]

Status Flag 'Pre-Test Inspection' - - This Pull Request Requires Inspection... The code must be inspected by a member of the Team before Testing/Merging
NO INSPECTION HAS BEEN PERFORMED ON THIS PULL REQUEST! - This PR must be inspected by setting label 'AT: PRE-TEST INSPECTED'.

[ziyuezzy]

I have added new tests for Anytopo, and all tests passed. So I think this PR is ready for review.

[feldergast]

Sorry, with the SC conference last week, Thanksgiving this week and vacation next week, it will be a bit before I can dive deep into this PR. I took a quick glance through it and didn't see anything that immediately stood out as an issue. I'll revisit this the week of December 8th and do a complete review.