Disallow RPM upload with the same NVR [RHELDST-37824]

[rbikar]

This change introduces a functionality that will raise a fatal error
when upload of an RPM with identical cdn_path of already present unit in pulp
is attempted.

The RPMs in question differ only with their checksums (and signing keys),
further association to live repositories and publish will cause breakage
of repository.

Duplicates check is by default disabled, and can be enabled by setting
PUBTOOLS_PULP_ALLOW_DUPLICATE_UNITS env var.

This should be a workaround until proper suport for such scenario is
implemented.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 99.80%. Comparing base (0066a0d) to head (ffd2d44).
⚠️ Report is 2 commits behind head on master.

Additional details and impacted files

@@           Coverage Diff           @@
##           master     #403   +/-   ##
=======================================
  Coverage   99.80%   99.80%           
=======================================
  Files          54       54           
  Lines        3018     3030   +12     
=======================================
+ Hits         3012     3024   +12     
  Misses          6        6           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[MichalHaluza]

Has this been tested on production-like data? AFAIK, we used to have the same RPMs signed with different GPG keys for different products in the past. This was (and still is) perfectly valid as long as the RPMs with the same NVRA but different signature aren't associated to the same repo. Checking at the time of upload may break these cases.

[MichalHaluza]

Leftover from debugging

[rbikar]

Has this been tested on production-like data? AFAIK, we used to have the same RPMs signed with different GPG keys for different products in the past. This was (and still is) perfectly valid as long as the RPMs with the same NVRA but different signature aren't associated to the same repo. Checking at the time of upload may break these cases.

Haven't done any big tests on production data, but the case you mentioned will be disabled with current solution.
I just wanted workflow to fail as soon as possible.

I can move this check to Associate phase probably where I'll do the same or similar query but directly to dest repo ,
maybe (even better) doing query only to dest repo would be sufficient even in the Upload phase. Now we do generic content query.

With new query we will enable upload of NVR duplicates unless the same NVR is present in dest repo. Sounds good?

[MichalHaluza]

Yes, checking only the dest repo would be preferred, however can we actually do that? Are the dest repos already known during the upload step? (Note that the same RPM can be uploaded to multiple repos during a signle push, and in RHEL it oftentimes is - mainline and dot)

[rbikar]

Yes, checking only the dest repo would be preferred, however can we actually do that? Are the dest repos already known during the upload step? (Note that the same RPM can be uploaded to multiple repos during a signle push, and in RHEL it oftentimes is - mainline and dot)

Yep, dest repos are known after Query pulp step.

[rbikar]

Implementation update - now the error is raised only in case when RPM would create a duplicate in destination repository.

This allows to upload of RPMs with same NVR and different gpg signing keys as long as RPMs target different repositories. Although I think this may also lead to problems with re-signed RPMs with the same path to origin even when targeting different repos. We would probably need to check cdn_path value or the gpg signing key.

[rbikar]

After consultation with @MichalHaluza this should the correct solution:

• do not allow upload or association of RPM unit that has identical cdn_path (origin) as any unit in pulp.