Consider changing definition of SCT test so that baseline comparisons are done on the single-column case

[billsacks]

Issue Type

Infrastructure Update

Issue Description

My understanding of CAM's SCT test is that it runs two cases: case 1 is a global run that creates an iop file (not sure what that is...); case 2 is a single-column run that uses the iop file generated by case 1. It uses the SystemTestsCompareTwo infrastructure to facilitate doing these two cases in a single test, though if I'm reading things correctly, there is no actual comparison of the two cases (which are probably not directly comparable anyway, since one is a global case and the other a single-column case).

From what I can find (maybe wrong), this appears to be the only testing done of CAM's single-column capabilities. If true, then this seems like an important test.

However, what I found in running this test is that baseline comparisons are just done for the global case, so the single-column case isn't covered by baseline comparisons. This is because of how SystemTestsCompareTwo is set up: the assumption there is that the two cases should give identical results, and so it shouldn't matter which case is used for baseline comparisons - so it (arbitrarily, I think) chooses case1 for the baseline generation / comparison.

If I'm right that this is the main / only way that CAM's single-column mode is tested, then it seems good to have baseline comparisons done of the single-column run. I'm not sure of the best way to accomplish this, but here are a few ideas:

(1) Rewrite the SCT test to not use SystemTestsCompareTwo. Instead follow examples like the ERI test or CTSM's SSP test that similarly do a sequence of tests that depend on each other, where the last one is in the main case directory and so is the one used for baselines. I'm not sure where you'd put the code that's currently in SCT's _component_compare_test, but since that appears not to actually be doing any comparison, I'd probably put that in the run phase and call it a run failure if the test there fails.

(2) If it's acceptable to use the iop files from a different CAM tag, and these don't need to change very often, then I'd suggest splitting this test into two separate tests, each of which does a single run: a test that generates these iop files to make sure that generation works correctly, and a test that uses pre-generated iop files, stored in the inputdata repository, to exercise the single-column functionality. Each of these tests would then be a simple single-case test. I think this is a good way to go if it's safe to point to relatively-stable versions of the iop forcings for the single-column test; I'm not sure whether that's the case or not.

(3) Change SystemTestsCompareTwo so that baseline generation / comparison is done from case2 rather than case1. Looking back at the implementation of SystemTestsCompareTwo, I think this might be tricky... but it actually feels like a better thing to do in general. My initial thought is that this could be done by using the main case directory for case2 and creating an explicit case1 directory. This would require care to make sure this is done correctly and doesn't break any of the tests built on this infrastructure. This may also require naming case2 with "base" if this comment is still true (I'm not sure if it is) (if so, this isn't ideal from an understandability perspective and also because I think it would require changes to many / all tests that are built on SystemTestsCompareTwo):

https://github.com/ESMCI/cime/blob/240989d6ae3d236894cda07993e0f64a20af1623/CIME/SystemTests/system_tests_compare_two.py#L117-L122

(4) There are probably other solutions I'm not thinking of.

All of these solutions feel somewhat involved - though (2) seems easiest if that's acceptable. I'll leave it to the CAM group to determine if this is worth the effort.

Will this change answers?

No

Will you be implementing this yourself?

No

[billsacks]

Hmmm, based on a comment from @jtruesdal -

The test you identified will make a bit for bit comparison with a column in a global run.

Maybe the compare-two really is important here? I can't see where this comparison is done in the test, but maybe there's some magic in the iop stuff that I don't understand?

If such a comparison is done, then my idea (2) isn't feasible. (1) could be possible, though with additional work (e.g., the ERI test does this... but that's also quite a complex test script).

[jtruesdal]

A prognostic time step for a 3d model consists of running the physics parameterization in the vertical over every column to produce a physics tendency followed by the dynamics parameterization to determine the tendency across columns. The physics and dynamics tendencies are added to an initial state to create an updated state and complete the time step. Rinse and Repeat. The single column model runs CAM physics in a single vertical column and since it has no horizontal dimension receives the dynamic horizontal advective tendencies from an IOP file that has derived those state tendencies during a meteorologic field campaign. Typically the IOP also contains serial snapshots of the observed state. IOP stands for Intensive Observational Period and an IOP data file is one of the products produced by some field programs to allow modeling groups to test their physics parameterizations in a single column framework. Concentrating on the physics in a single column while using specified dynamics tendencies, simplifies the study of convective and microphysics parameterization by decoupling them from dynamic feedbacks.

In CAM we have an option to produce a model IOP file that mimics the official IOP files produced by the field campaigns. It captures the dynamic tendencies every time step for all the prognosed variables as well as the final updated state for every column. We can then make a separate single column model run using the IOP file from the 3d run. SCAM starts with the exact same state as the original 3d run, runs through the exact same physics code and finally reads the IOP file to apply the exact same dynamics tendency that the 3d model captured for that column. The upshot is that SCAM can reproduce the same updated state in its column as the 3d model prognosed. Since the IOP file also contains the updated (observed) state from the 3d run at each time step, SCAM can verify that its prognosed T and Q variables are identical to Tobs and Qobs output during the 3d model. This same procedure is used when running an SCT case. The sct.py driver will run the 3d model to produce an IOP and then run SCAM using that IOP. It checks the TDIFF and QDIFF variables from the SCAM run to make sure they are 0 and Passes the BFB test. Unlike the other comparisons which look at the various history files from two runs, the SCAM history file contains the proof of a bit for bit run in its TDIFF and QDIFF variables.

[jtruesdal]

@billsacks making sure that the SCAM column is identical to the same column in a 3d run is our test for correctness.

As for adding a new baseline test, we haven't seen a need in the past. It always been good enough to assume if SCAM is BFB with CAM then by the transitive property of testing the SCAM baseline state is already tested via the CAM baseline tests. I don't think we a separate SCAM baseline test.

[billsacks]

@jtruesdal - thanks for these detailed explanations! This makes more sense to me now. The transitive property you mention makes sense. So it seems like the main (only??) thing missing from the current test in terms of baseline comparisons is if problems arise somewhere between the internals of CAM (which are captured in the current comparison) and the outputting to file. I opened this issue because that output-to-file piece is what got changed in #1623 - so I was surprised when my baselines passed when I expected them to fail. (Specifically, I fixed the dimensions of variables on the cpl hist files in that PR.) But maybe this is a rare enough situation that the status quo is fine.

I'll go ahead and close this as a wontfix, because I'm now satisfied that this is less of an issue than I originally thought it was.