Expose `content_ids` on the repository version API

[acheng-01]

Is your feature request related to a problem? Please describe.
There is no way to retrieve a repository version's content unit IDs over the REST API. Clients that want to compute the difference between two repository versions must instead issue content filter queries (e.g. filtering content by repository_version), which can be expensive on large repositories and put significant load on the database.

Describe the solution you'd like
Expose the existing RepositoryVersion.content_ids array on the repository version detail endpoint as an opt-in field. To avoid bloating normal responses, it is only returned when the request includes the content_ids=true query parameter; otherwise the field is null. Clients can then fetch each version's content_ids and diff the sets directly, so the cost scales with the size of the change rather than the size of the repository

Describe alternatives you've considered

• Filtering content by repository_version and diffing the results — this is the current approach and is the performance problem we're trying to avoid.
• Always returning content_ids — rejected because the array can be very large and would unnecessarily inflate every repository version response

Additional context
content_ids is already stored and maintained on the model (added/backfilled in prior migrations and kept current by the add/remove content hooks), so this is a purely additive serializer change with no schema migration or backfill required

[gerrod3]

What is your need for diffing the contend_ids on the client side? Have you tried using the ?repository_version_added= query? Using it with the ?include= filter can reduce the result to just the prn or href of the content in the repository version. I would expect this to be performant -- and if it isn't we should focus on making it so. I'm not sure if we want to expose content_ids since it's an implementation detail.

[acheng-01]

We're not diffing adjacent versions unfortunately. Our use case is having to purge the cache of stale content between two publications from our CDN, hence the diffing. However, our publications are decoupled from version creation, so those two versions are often several increments apart. If I'm understanding correctly, ?repository_version_added= / ?repository_version_removed= resolve to RepositoryVersion.added() / .removed() with no base, so they only give the single-step diff against the immediate predecessor. This means our diffing strategy would miss content changed in the realized (published) versions.

For some context, we are currently running q of repository_version = x AND NOT repository_version = y in the entire content table for content of a specific type. As you can imagine, this has been a very expensive query - one that has recently more frequently been giving us db resource consumption issues.

That said, I did feel iffy about exposing content_ids as well. I do have a PR open to conditionally expose that, but I'd be happy to drop it. One alternative perhaps is to expose a base_version parameter on those filters? For example, ?repository_version_added=<v>&base_version=<v0>. I'm also open to other ideas.

Thanks for the feedback!

[mdellweg]

• Always returning content_ids — rejected because the array can be very large and would unnecessarily inflate every repository version response

To me this even being optional is an invitation to ddos the server. (May even lead to OOM, because content_ids can be up in the 100 000.)

Can you try the "q"-filter on the "content_list" operation?
Like: [...]?q='repository_version=<prn1> AND NOT repository_version=<prn2>' (didn't test this myself, but the api-docs suggest it should be possible.)

https://pulpproject.org/pulpcore/docs/user/guides/use-complex-filters/

[acheng-01]

Thanks, that makes sense.

To clarify, we are already doing what you're suggesting. It is possible, but I believe the query in the back is eating up all the resources in the db and preventing all of our other operations from being able to connect. I'm happy to be wrong, but I think the actual translated SQL of that query becomes:

(SELECT ... FROM rpm_package WHERE content_ptr_id IN (<current version ids>))
INTERSECT
(
  (SELECT ... FROM rpm_package)                              -- FULL TABLE, no WHERE
  EXCEPT
  (SELECT ... FROM rpm_package WHERE content_ptr_id IN (<previous version ids>))
)
ORDER BY col1 LIMIT 100 OFFSET 100

That middle SELECT...EXCEPT...SELECT is running through the entire list of rpm packages in the table. In addition, the specific operations INTERSECT and EXCEPT (translated from AND and NOT respectively) force full sort-or-hash + dedup over those large inputs. This is why we are now trying to move away from using the q filter.

What do you think about my idea of exposing a base_version field into ContentFilter? The RepositoryVersion model's added() and removed() methods already support diffing the current repo version's content from a specified previous version that isn't adjacent. The implementation would look something like the changes I've made in this draft PR: #7844

I think it should be relatively less invasive to the db than exposing content_ids and diffing it ourselves.

[gerrod3]

The base_version filter sounds like a good idea. Have you tested the database query performance? I assume the generated SQL would look like SELECT ... FROM content.pk in (<content_ids>) AND NOT content.pk in (<base_content_ids>). If that's the case, I don't know if Postgres handles the query well with scaling sizes of content_ids. I wonder if it would be better to do the diff in python first before handing the final content_ids to the database. We would love some performance metrics if you have a large system that you can test.

[acheng-01]

Hmm, so you're right about the generated SQL. I wonder if we can address the problem you mentioned by switching the current inline filter query into an unnest subquery when base_version is supplied. It looks like get_content already has this implementation to work around a 65,535-parameter limit.

I had copilot test both scenarios using a seeded dataset of 200k content units, and here are the results. Version pair here means repo version pairs base and current. Each pairing has a 1k content unit diff between them.

Version Pair	Inlined list (current added())	Unnest subquery	Speedup
50k units / 50k units	1,471 ms	116 ms	~12.7x
199k units / 199k units	6,802 ms	521 ms	~13x

I've added the refactor into my draft PR linked in a previous comment for reference.