Expose recent committed ledger prefixes

CHANGELOG.md

@@ -16,6 +16,7 @@ and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.
 ### Added
 
 - C++ endpoints can now use `ccf::endpoints::Endpoint::add_openapi_response<Out>()` to document additional HTTP responses in their generated OpenAPI schema without changing the endpoint's primary success response (#8115).
+- Ledger chunk download clients can opt in to immutable `.committed_prefix` resources containing recent committed entries that are not yet available in canonical `.committed` files. (#8214)
 - New `ledger.max_transaction_size` node configuration option (default `32MB`), which caps the total serialised size of transactions written to the ledger. The limit covers the whole ledger entry: the fixed 8-byte ledger entry header, the ledger encryption header, public domain size field, public domain and encrypted private domain. It is checked before a transaction is applied, so an oversized transaction is now rejected with `413 Payload Too Large` and error code `TransactionTooLarge`, and subsequent transactions are unaffected, where previously an excessively large transaction could terminate the node. Reserved internal signature transactions are exempt because they must fill their reserved ledger version. The limit applies only to newly serialised non-reserved transactions; deserialising existing entries (including during recovery), historical queries and snapshots are unaffected, so entries written under a larger or unset limit remain readable. It must be smaller than `memory.max_msg_size` by at least the ring-buffer range response overhead, which is validated at node startup and by `--check` (#7992).
 
 ### Changed

doc/operations/configuration.rst

@@ -71,7 +71,7 @@ The `enabled_operator_features` configuration field allows enabling or disabling
 Currently supported features are:
 
 1. 'SnapshotRead': gates access to endpoints used to fetch snapshots directly from nodes (:http:GET:`/node/snapshot`, :http:HEAD:`/node/snapshot`, :http:GET:`/node/snapshot/{snapshot_name}` and :http:HEAD:`/node/snapshot/{snapshot_name}`).
-2. 'LedgerChunkRead': gates access to endpoints used to retrieve ledger chunks (:http:GET:`/node/ledger_chunk`, :http:HEAD:`/node/ledger_chunk`, :http:GET:`/node/ledger_chunk/{chunk_name}` and :http:HEAD:`/node/ledger_chunk/{chunk_name}`).
+2. 'LedgerChunkRead': gates access to endpoints used to retrieve ledger chunks (:http:GET:`/node/ledger_chunk`, :http:HEAD:`/node/ledger_chunk`, :http:GET:`/node/ledger_chunk/{chunk_name}`, :http:HEAD:`/node/ledger_chunk/{chunk_name}`, :http:GET:`/node/ledger_chunk/committed_prefix/{chunk_name}` and :http:HEAD:`/node/ledger_chunk/committed_prefix/{chunk_name}`).
 3. 'SnapshotCreate': gates access to the operator endpoint used to create a snapshot on the next signature transaction (:http:POST:`/node/snapshot:create`).
 
 Since these operations may require disk IO and produce large responses, these features should not be enabled on interfaces with public access, and instead restricted to interfaces with local connectivity for node-to-node and operator access.
@@ -100,4 +100,3 @@ A rolling upgrade from ``Dual`` to ``CoseOnly`` is a two-step process:
 1. **CoseAllowDualJoin.** Deploy a binary that returns ``CoseAllowDualJoin``. Replace nodes one at a time. During this phase, new nodes running the old ``Dual`` binary can still join as replacements.
 
 2. **CoseOnly.** Once all nodes are upgraded, deploy a binary that returns ``CoseOnly``. Replace nodes again. After this, ``Dual`` joiners are rejected.
-

doc/operations/ledger_snapshot.rst

@@ -114,10 +114,33 @@ These endpoints allow downloading a specific ledger chunk by name, where `<chunk
 They support the HTTP `Range` header for partial downloads, and the `HEAD` method for clients to query metadata such as the total size without downloading the full chunk.
 They also populate the `x-ms-ccf-ledger-chunk-name` response header with the name of the chunk being served.
 
+Committed Prefixes
+^^^^^^^^^^^^^^^^^^
+
+By default, the ledger chunk locator endpoints expose only canonical ``.committed`` files. A client that also needs recent committed entries from an in-progress file can opt in with ``include_committed_prefix=true``:
+
+.. code-block:: http
+
+    GET /node/ledger_chunk?since=101&include_committed_prefix=true HTTP/1.1
+
+The node still prefers a canonical ``.committed`` file when one covers the requested sequence number. Otherwise, if the sequence number is locally available and committed, it returns a ``307 Temporary Redirect`` to a resource such as:
+
+.. code-block:: text
+
+    /node/ledger_chunk/committed_prefix/ledger_101-140.committed_prefix
+
+The ``.committed_prefix`` suffix distinguishes this synthetic resource from a canonical physical ledger file. The response contains a normal completed ledger representation - header, unchanged transaction bytes, and positions table - but only for the selected committed range. It can be read directly with :py:class:`ccf.ledger.LedgerChunk`.
+
+Both the temporary redirect and the committed-prefix response include ``Cache-Control: no-store``. The response also includes ``x-ms-ccf-ledger-chunk-kind: committed-prefix``. Clients must not archive, install, or use this resource for recovery as though it were a canonical ``.committed`` file. In particular, committed-prefix files are ignored by committed-only ledger directory discovery.
+
+The ``307`` redirect intentionally differs from the ``308 Permanent Redirect`` used for canonical ``.committed`` files. A physical chunk's name and range are final, while the committed-prefix range selected for the same ``since`` value may grow as the commit watermark advances. The exact ``.committed_prefix`` URL is immutable once returned, so it remains suitable for retrying or resuming that specific download.
+
+More than one contiguous committed prefix can be available. For example, physical chunking may have produced unsuffixed files ``ledger_101-150`` and ``ledger_151-200`` before the commit watermark catches up to 200. The API preserves these physical boundaries and can expose ``ledger_101-150.committed_prefix`` followed by ``ledger_151-200.committed_prefix``.
+
 Want-Repr-Digest and Repr-Digest
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
-These endpoints also support the ``Want-Repr-Digest`` request header (`RFC 9530 <https://www.rfc-editor.org/rfc/rfc9530>`_).
+The canonical and committed-prefix chunk endpoints also support the ``Want-Repr-Digest`` request header (`RFC 9530 <https://www.rfc-editor.org/rfc/rfc9530>`_).
 When set, the response will include a ``Repr-Digest`` header containing the digest of the full representation of the file.
 Supported algorithms are ``sha-256``, ``sha-384``, and ``sha-512``. If the header contains only unsupported or invalid algorithms, the server defaults to ``sha-256`` (as permitted by `RFC 9530 Appendix C.2 <https://www.rfc-editor.org/rfc/rfc9530#appendix-C.2>`_).
 For example, a client sending ``Want-Repr-Digest: sha-256=1`` will receive a header such as ``Repr-Digest: sha-256=:AEGPTgUMw5e96wxZuDtpfm23RBU3nFwtgY5fw4NYORo=:`` in the response.

doc/schemas/node_openapi.json

@@ -919,7 +919,7 @@
   "info": {
     "description": "This API provides public, uncredentialed access to service and node state.",
     "title": "CCF Public Node API",
-    "version": "5.0.6"
+    "version": "5.0.7"
   },
   "openapi": "3.0.0",
   "paths": {
@@ -1192,7 +1192,7 @@
     },
     "/node/ledger_chunk": {
       "get": {
-        "description": "Redirect to the corresponding /node/ledger_chunk/{chunk_name} endpoint for the ledger chunk including the sequence number specified in the 'since' query parameter.",
+        "description": "Redirect to the corresponding /node/ledger_chunk/{chunk_name} endpoint for the ledger chunk including the sequence number specified in the 'since' query parameter. If 'include_committed_prefix' is true and no committed file is available, this may temporarily redirect to a synthetic committed-prefix resource.",
         "operationId": "GetNodeLedgerChunk",
         "parameters": [
           {
@@ -1202,9 +1202,20 @@
             "schema": {
               "$ref": "#/components/schemas/uint64"
             }
+          },
+          {
+            "in": "query",
+            "name": "include_committed_prefix",
+            "required": false,
+            "schema": {
+              "$ref": "#/components/schemas/boolean"
+            }
           }
         ],
         "responses": {
+          "307": {
+            "description": "Redirect to a temporary committed ledger prefix."
+          },
           "308": {
             "description": "Redirect to the selected ledger chunk."
           },
@@ -1221,7 +1232,7 @@
         }
       },
       "head": {
-        "description": "Redirect to the corresponding /node/ledger_chunk/{chunk_name} endpoint for the ledger chunk including the sequence number specified in the 'since' query parameter.",
+        "description": "Redirect to the corresponding /node/ledger_chunk/{chunk_name} endpoint for the ledger chunk including the sequence number specified in the 'since' query parameter. If 'include_committed_prefix' is true and no committed file is available, this may temporarily redirect to a synthetic committed-prefix resource.",
         "operationId": "HeadNodeLedgerChunk",
         "parameters": [
           {
@@ -1231,9 +1242,20 @@
             "schema": {
               "$ref": "#/components/schemas/uint64"
             }
+          },
+          {
+            "in": "query",
+            "name": "include_committed_prefix",
+            "required": false,
+            "schema": {
+              "$ref": "#/components/schemas/boolean"
+            }
           }
         ],
         "responses": {
+          "307": {
+            "description": "Redirect to a temporary committed ledger prefix."
+          },
           "308": {
             "description": "Redirect to the selected ledger chunk."
           },
@@ -1250,6 +1272,82 @@
         }
       }
     },
+    "/node/ledger_chunk/committed_prefix/{chunk_name}": {
+      "get": {
+        "description": "Download a synthetic chunk containing only committed ledger entries. Supports HTTP Range and digest headers. The resource is not a canonical .committed ledger file and must not be used for recovery.",
+        "operationId": "GetNodeLedgerChunkCommittedPrefixChunkName",
+        "responses": {
+          "200": {
+            "content": {
+              "application/octet-stream": {
+                "schema": {
+                  "$ref": "#/components/schemas/Binary"
+                }
+              }
+            },
+            "description": "The requested committed ledger prefix."
+          },
+          "206": {
+            "content": {
+              "application/octet-stream": {
+                "schema": {
+                  "$ref": "#/components/schemas/Binary"
+                }
+              }
+            },
+            "description": "The requested byte range of the committed ledger prefix."
+          },
+          "304": {
+            "description": "The requested committed ledger prefix has not changed."
+          },
+          "404": {
+            "description": "The requested committed ledger prefix is not available."
+          },
+          "default": {
+            "$ref": "#/components/responses/default"
+          }
+        },
+        "summary": "Download committed ledger prefix",
+        "x-ccf-forwarding": {
+          "$ref": "#/components/x-ccf-forwarding/never"
+        }
+      },
+      "head": {
+        "description": "Metadata about a synthetic chunk containing only committed ledger entries. The resource is not a canonical .committed ledger file.",
+        "operationId": "HeadNodeLedgerChunkCommittedPrefixChunkName",
+        "responses": {
+          "200": {
+            "description": "Metadata for the requested committed ledger prefix."
+          },
+          "206": {
+            "description": "Metadata for the requested committed ledger prefix range."
+          },
+          "304": {
+            "description": "The requested committed ledger prefix has not changed."
+          },
+          "404": {
+            "description": "The requested committed ledger prefix is not available."
+          },
+          "default": {
+            "$ref": "#/components/responses/default"
+          }
+        },
+        "summary": "Committed ledger prefix metadata",
+        "x-ccf-forwarding": {
+          "$ref": "#/components/x-ccf-forwarding/never"
+        }
+      },
+      "parameters": [
+        {
+          "in": "path",
+          "name": "chunk_name",
+          "required": true,
+          "schema": {
+            "type": "string"
+          }
+        }
+      ]
+    },
     "/node/ledger_chunk/{chunk_name}": {
       "get": {
         "description": "Download a specific ledger chunk by name. Supports HTTP Range header for partial downloads.",

include/ccf/http_consts.h

@@ -33,6 +33,8 @@ namespace ccf
       static constexpr auto CCF_SNAPSHOT_NAME = "x-ms-ccf-snapshot-name";
       static constexpr auto CCF_LEDGER_CHUNK_NAME =
         "x-ms-ccf-ledger-chunk-name";
+      static constexpr auto CCF_LEDGER_CHUNK_KIND =
+        "x-ms-ccf-ledger-chunk-kind";
     }
 
     namespace headervalues::contenttype

python/src/ccf/ledger.py

@@ -58,6 +58,7 @@ def __getattr__(name: str):
 SERVICE_INFO_TABLE_NAME = "public:ccf.gov.service.info"
 
 COMMITTED_FILE_SUFFIX = ".committed"
+COMMITTED_PREFIX_FILE_SUFFIX = ".committed_prefix"
 RECOVERY_FILE_SUFFIX = ".recovery"
 IGNORED_FILE_SUFFIX = ".ignored"
 
@@ -157,13 +158,29 @@ def unpack_array(buf, fmt):
 
 
 def range_from_filename(filename: str) -> tuple[int, int | None]:
-    elements = (
-        os.path.basename(filename)
-        .replace(COMMITTED_FILE_SUFFIX, "")
-        .replace(RECOVERY_FILE_SUFFIX, "")
-        .replace("ledger_", "")
-        .split("-")
-    )
+    basename = os.path.basename(filename)
+    is_recovery = basename.endswith(RECOVERY_FILE_SUFFIX)
+    basename = basename.removesuffix(RECOVERY_FILE_SUFFIX)
+    if basename.endswith(COMMITTED_PREFIX_FILE_SUFFIX):
+        if is_recovery:
+            raise ValueError(f"Could not read seqno range from ledger file {filename}")
+
+        range_str = basename.removesuffix(COMMITTED_PREFIX_FILE_SUFFIX)
+        if not range_str.startswith("ledger_"):
+            raise ValueError(f"Could not read seqno range from ledger file {filename}")
+
+        elements = range_str[len("ledger_") :].split("-")
+        if (
+            len(elements) != 2
+            or not all(element.isascii() and element.isdigit() for element in elements)
+            or int(elements[0]) == 0
+            or int(elements[1]) < int(elements[0])
+        ):
+            raise ValueError(f"Could not read seqno range from ledger file {filename}")
+        return (int(elements[0]), int(elements[1]))
+
+    basename = basename.removesuffix(COMMITTED_FILE_SUFFIX)
+    elements = basename.replace("ledger_", "").split("-")
     if len(elements) == 2:
         return (int(elements[0]), int(elements[1]))
     elif len(elements) == 1:

python/tests/test_ledger.py

@@ -0,0 +1,34 @@
+# Copyright (c) Microsoft Corporation. All rights reserved.
+# Licensed under the Apache 2.0 License.
+
+"""Unit tests for CCF ledger filename handling."""
+
+import ccf.ledger
+import pytest
+
+
+def test_committed_prefix_filename_range():
+    """Committed-prefix names expose their closed sequence-number range."""
+    assert ccf.ledger.range_from_filename("ledger_42-100.committed_prefix") == (42, 100)
+
+
+def test_committed_prefix_is_not_canonical_committed_file():
+    """Committed prefixes stay excluded from committed-only discovery."""
+    assert not ccf.ledger.is_ledger_chunk_committed("ledger_42-100.committed_prefix")
+
+
+@pytest.mark.parametrize(
+    "filename",
+    [
+        "ledger_0-100.committed_prefix",
+        "ledger_42-41.committed_prefix",
+        "ledger_42.committed_prefix",
+        "ledger_ledger_42-100.committed_prefix",
+        "ledger_42x-100.committed_prefix",
+        "ledger_42-100.committed_prefix.recovery",
+    ],
+)
+def test_committed_prefix_filename_rejects_invalid_ranges(filename: str):
+    """Committed-prefix names must contain one valid, closed range."""
+    with pytest.raises(ValueError):
+        ccf.ledger.range_from_filename(filename)