Flow profiles by johnbilt · Pull Request #130 · bbc/tams

johnbilt · 2025-05-20T10:50:55Z

Details

Add the content of flow profiles to aid with the creation of flows in stores where there are house formats being used. Also significantly simplifies matching flows across edit by reference workflows

Jira Issue (if relevant)

Jira URL:

Related PRs

If merged before #115, add the new endpoints to the auth logic listings in that PR. If merged after, add that logic in this PR

Submitter PR Checks

(tick as appropriate)

PR completes task/fixes bug
API version has been incremented if necessary
ADR status has been updated, and ADR implementation has been recorded
Documentation updated (README, etc.)
PR added to Jira Issue (if relevant)
Follow-up stories added to Jira

Reviewer PR Checks

(tick as appropriate)

PR completes task/fixes bug
Design makes sense, and fits with our current code base
Code is easy to follow
PR size is sensible
Commit history is sensible and tidy

Info on PRs

The checks above are guidelines. They don't all have to be ticked, but they should all have been considered.

himslm01

A few readability improvements.

Updated wording Co-authored-by: James Sandford <james.sandford@bbc.co.uk>

… flow-profiles

samdbmg

Looks good to me overall: a good approach of maintaining flexibility while constraining in a way to make it easier to use.

I've inlined a handful of suggestions, and a couple of nits

samdbmg · 2025-07-31T15:31:44Z

I've also kicked off the CI for you, so you should be able to see if it passes validation

GeorginaShippey · 2025-08-05T17:12:10Z

+{
+  "type": "object",
+  "description": "Describes the technical characteristics of a Flow (common properties to all Flows, imported by type-specific specifications)",
+  "title": "Flow Core",


NIT: Would be good to have the title updated as it keeps the rendered version more readable, there are currently two schemas rendered out as Flow Core

Co-authored-by: Sam Mesterton-Gibbons <sam@samn.co.uk>

… flow-profiles

Co-authored-by: Georgina Shippey <9370167+GeorginaShippey@users.noreply.github.com>

… flow-profiles

danjhd

Also this may be a nit and BBC R&D can guide but my feeling is that when referring to TAMS entity types like Flow(s). Source(s) or Profile(s) we should always capitalise these words to help clarify?

danjhd · 2025-12-09T11:19:36Z

+
+When using the second option on submission of the create flow request, the store will then read the metadata for a given profile and use this to populate the metadata for the flow.
+This de-normalisation of the technical parameters means that on the read side the flows created via both mechanisms have the same technical parameters available.
+Additionally if the decision is made to change a standard profile in the future then this does not affect the existing flows as the parameters have already been replicated and should remain unchanged to reflect the media actually being stored.


Above you mention, quite rightly i believe, that Flows should be immutable. Yet here you talk about a use case where a standard profile could be changed. This conflicts. I don't believe we should allow this use case. As you say above if the profile needs to change a new one with a new Id should be created and used.

Good point - we should remove this as not required. Will re-work this section anyway based on another comment

danjhd · 2025-12-09T11:19:39Z

+1. Specify all the technical characteristics of the flow including the wrapper, codec and essence parameters alongside the non-technical parameters such as label and description
+2. Provide just the technical profile for the flow and non-technical parameters required
+
+When using the second option on submission of the create flow request, the store will then read the metadata for a given profile and use this to populate the metadata for the flow.


It feels here like we are making a decision for the implementation? As long as the GET request to the Flow returns all the metadata it should not matter how the implementation does this. i.e. it might not populate them it might just keep the reference to the profile id and "hydrate" on request? The important point is that when GETting Flows a client should be able to see all the metadata regardless of how it was created.

Re-worked section to hopefully including this

danjhd · 2025-12-09T11:20:46Z

+
+A profile should be treated as immutable, so once created it cannot be updated.
+Updating a profile would cause mismatches with flows which have been already created using that profile and so breaks the model for profiles.
+To update a profile a new one with a new ID should be created.


Do we (or should we) have an option to "archive" an "old" Profile that should no longer be used but is in use already and therefore cannot easily be deleted?

Currently this is not in scope of this PR. We do need to think about old profiles, however we should also consider the same approach for the storage backend endpoint under the services section also. So previously proposed that we handle both of these at the same time in a future PR

For completeness, one option is to add fine-grained auth to profiles (and storage backends) to allow their use to be restricted as required.

danjhd · 2025-12-09T11:23:36Z

+Additionally if the decision is made to change a standard profile in the future then this does not affect the existing flows as the parameters have already been replicated and should remain unchanged to reflect the media actually being stored.
+
+Flows that have been created from a profile will include the parameter indicating which profile they were created from.
+This differentiates them from the flows that have been created with the technical characteristics directly.


Do we need to differentiate between flows that were created using a Profile and a Flow created using exactly the same Metadata as an existing Profile?
If we didn't then perhaps the store should detect this at creation time and set the Profile Id to match even if not supplied?
OR it could reject the request saying that this Flow "metadata" matches an existing Profile and therefore the profile should be used?
Just thinking out loud here.

This probably needs the outcome of the greedy match discussion to complete. If we're not going to implement greedy match at any level then the only way the profile_id field should be populated is if it's provided at creation.

If we want to do match on creation and populate the profile field then yes we may want to indicate this. However the question is what use is this in the actual TAMS metadata or whether it should be captured in an observability layer for reporting/tracking.

I would probably stay away from rejecting requests where they exactly match a profile. I would either match on creation and populate the field or leave creators the choice to use profiles or not as they wish.

danjhd · 2025-12-09T11:26:30Z

+The edit process could then process each input source in turn read the available flows and match them to the destination flows using the profile tag easily.
+If the source content has additional non standard flows then these could be ignored.
+
+From a flow created from a profile, it is possible to query via both profile ID and also the individual parameter of the flow that have been inheritied from the profile.


We should define the behaviour if these conflict. I.e. You have a profile A that defines codec as X. You query for Flows that use Profile A using the new query parameter but also use the existing codec query parameter and set that to Y. What should the behaviour be?

Re-worked section to hopefully including this

j616 · 2026-05-13T16:21:22Z

+For workflows where replication of the same content formats are happening on a regular basis then it is recommended that the same Profile is loaded into both stores using the same UUID.
+This will mean than when Flows are replicated between the stores then the Profile identifier will continue to link to the metadata.
+
+If the Profile does not exist within the destination store then the Profile ID should continue to be preserved.


Do we need to describe how this interacts with the Profiles endpoint? Should the Profile be created if it doesn't exist? If not, how does it interact with the creation of a Profile with a matching ID? I think the paragraph below covers some of this. It might help if that is moved above this para?

j616 · 2026-05-13T16:27:13Z

+If the Profile does not exist within the destination store then the Profile ID should continue to be preserved.
+This will continue to allow the matching of content within the store by Profile ID, plus the Profile can be added later and will link to the existing content.
+
+For workflows including more than two organisations it is recommended that one organisation takes responsibility for owning and publishing the Profiles.


Should this be one organisation takes responsibility for owning and publishing a given Profile or shared Profiles? I read this as all Profiles, which I don't think is practical/possible.

j616 · 2026-05-13T16:32:20Z

+- Good: Enables brownfield adoption — existing Flows that match a Profile's technical characteristics become discoverable
+- Good: Simplifies edit-by-reference workflows across mixed content where some Flows were created with Profiles and others were not
+- Good: Makes Profiles useful as a query template even in stores where Profile-based creation has not been universally adopted
+- Bad: Requires clear definition of what constitutes a "match"


I might be missing something. I had assumed that all parameters specified in the Profile would have to match exactly in Flows. Or is it items like tags and other dictionaries where behaviour might have to differ?

j616 · 2026-05-13T16:33:41Z

+        It is up to the store implementation whether this is normalised on the flow creation or read-only.
+
+        When creating a flow using a Profile it is not possible to override any specific fields as this would invalidate the link to the Profile.  
+        Supplying a Profile ID and technical metadata should result in a 400 validation error


Suggested change

Supplying a Profile ID and technical metadata should result in a 400 validation error

Supplying a Profile ID and technical metadata defined within the specified Profile should result in a 400 validation error

j616 · 2026-05-13T16:35:58Z

        "204":
          description: No content. The Flow has been updated.
        "400":
          description: Bad request. Invalid Flow JSON.


Do we need to add conflicting Profiles, and invalid Profile IDs to this return code?

j616 · 2026-05-13T16:42:54Z

+      description: |
+        Create a new Profile to be used when creating flows with the matching metadata.  
+
+        It is not possible to update an existing Profile as this would mean it no longer matches flows that have already been created using it.


I wonder if normative language would be useful here?

Suggested change

It is not possible to update an existing Profile as this would mean it no longer matches flows that have already been created using it.

Services MUST reject requests that would update an existing Profile as this would mean it no longer matches flows that have already been created using it.

j616 · 2026-05-13T16:46:07Z

+[
+  {
+    "id": "329b98d6-adeb-418c-8eaf-eca14edc1706",
+    "label": "1080p50 TS Video 50Mpbs",


NIT: We recently added diverse formats to examples throughout the spec to help with the misconception that TAMS is TS-centric. It might be useful to add diverse formats to the profile examples too?

j616 · 2026-05-13T16:55:59Z

+          ],
+          "properties": {
+              "profile_id": {
+                  "description": "Profile identifier that was used to create the flow.  When supplying a profile_id no technical metadata should also be provided.  Doing so will result in a 400 validation error",


The Profile schema seems to allow for partial technical metadata. I think I'd expected Flows to be able to add additional metadata not defined in the Profile. As it stands, I think the main places where metadata might diverge is captured in flow-common.json, so maybe thats ok? Though I don't think we've previously defined what is "technical metadata" and what is "non-technical metadata" anywhere. I feel like we might need to either generalise the approach or specify the distinction.

johnbilt added 2 commits May 20, 2025 11:47

Flow profiles draft app note

0b0ab0a

Formatting improvements

b1150d3

johnbilt requested a review from a team as a code owner May 20, 2025 10:50

himslm01 reviewed Jun 3, 2025

View reviewed changes

Incorporating changes from Mark

d8680fe

j616 reviewed Jun 4, 2025

View reviewed changes

Comment thread docs/appnotes/00xx-using-flow-profiles.md Outdated

Comment thread docs/appnotes/00xx-using-flow-profiles.md Outdated

Comment thread docs/appnotes/00xx-using-flow-profiles.md Outdated

Comment thread docs/appnotes/00xx-using-flow-profiles.md Outdated

johnbilt and others added 9 commits June 4, 2025 17:26

Update docs/appnotes/00xx-using-flow-profiles.md

2a686d6

Updated wording Co-authored-by: James Sandford <james.sandford@bbc.co.uk>

Combining changes

c7a9347

Merge branch 'flow-profiles' of https://github.com/johnbilt/tams into…

3bb8bd0

… flow-profiles

Merge branch 'bbc:main' into flow-profiles

117b99e

Initial profile examples

1e71dbd

Merge branch 'bbc:main' into flow-profiles

3f1ccb7

Merge branch 'bbc:main' into flow-profiles

e89aa37

First set of profile schema changes

6770a5d

Merge branch 'flow-profiles' of https://github.com/johnbilt/tams into…

f741890

… flow-profiles

samdbmg reviewed Jul 31, 2025

View reviewed changes

GeorginaShippey reviewed Aug 5, 2025

View reviewed changes

j616 mentioned this pull request Aug 7, 2025

appnote: Add appnote covering Authorisation #115

Closed

11 tasks

johnbilt and others added 11 commits August 15, 2025 13:48

Update docs/appnotes/00xx-using-flow-profiles.md

d3a8c22

Co-authored-by: Sam Mesterton-Gibbons <sam@samn.co.uk>

Update api/schemas/flow-put.json

17e44f3

Co-authored-by: Sam Mesterton-Gibbons <sam@samn.co.uk>

Various updates from feedback

3d4f1f7

Merge branch 'flow-profiles' of https://github.com/johnbilt/tams into…

b6fc547

… flow-profiles

Update api/TimeAddressableMediaStore.yaml

f7899bb

Co-authored-by: Georgina Shippey <9370167+GeorginaShippey@users.noreply.github.com>

Update api/TimeAddressableMediaStore.yaml

4211448

Co-authored-by: Georgina Shippey <9370167+GeorginaShippey@users.noreply.github.com>

Correct example file name

d43ee0a

Merge branch 'flow-profiles' of https://github.com/johnbilt/tams into…

7b4a58e

… flow-profiles

Profile query clarification

e5655fe

Remove profile urn and add wording on multi-store workflows

49ee246

Updated app note number

7b51eb8

johnbilt and others added 5 commits November 20, 2025 11:13

Merge remote-tracking branch 'forRepoRequest' into flow-profiles

929f7ed

Merge branch 'bbc:main' into flow-profiles

64946ae

Formatting and consistency with recent changes

bb1dba0

Fix lint issue

dc4992d

Fixes for validation errors

7491aa4

danjhd reviewed Dec 9, 2025

View reviewed changes

johnbilt and others added 5 commits January 6, 2026 11:06

Merge branch 'bbc:main' into flow-profiles

4b159d0

Typos and spelling corrections from Dan

01de27b

Encorporating feedback from Dan

a270b49

Capitalisation

dd53166

Merge remote-tracking branch 'forRepoRequest/main' into flow-profiles

b1b873c

samdbmg linked an issue Apr 29, 2026 that may be closed by this pull request

Flow Profiles #190

Open

johnbilt added 8 commits May 12, 2026 11:32

Add comment on greedy match

980dd16

Merge remote-tracking branch 'forRepoRequest/main' into flow-profiles

233221a

Remove duplicate sections from merge

7af38cd

Correct app note numbering

ed1a948

General wording improvements

c9f13cd

Flow Profiles adr added

d3c4668

Lint fixes

27b7924

Fix validation fails

5437988

j616 reviewed May 13, 2026

View reviewed changes

	Supplying a Profile ID and technical metadata should result in a 400 validation error
	Supplying a Profile ID and technical metadata defined within the specified Profile should result in a 400 validation error

	It is not possible to update an existing Profile as this would mean it no longer matches flows that have already been created using it.
	Services MUST reject requests that would update an existing Profile as this would mean it no longer matches flows that have already been created using it.

Conversation

johnbilt commented May 20, 2025 • edited by j616 Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Details

Jira Issue (if relevant)

Related PRs

Submitter PR Checks

Reviewer PR Checks

Info on PRs

Uh oh!

himslm01 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

samdbmg left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

samdbmg commented Jul 31, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

danjhd left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

johnbilt commented May 20, 2025 •

edited by j616

Loading