feat(platform)!: required document fields via contract updates (requiredSince)

book/src/data-model/data-contracts.md

@@ -217,6 +217,32 @@ This intermediate format is important because serialization versions and code st
 
 There is also `versioned_limit_deserialize`, which imposes a size limit and always performs full validation -- this is used for data coming from untrusted sources (anything not from Drive's own storage).
 
+## Evolving a Contract: Adding Required Fields
+
+Contract updates are deliberately conservative: existing documents must stay valid and their stored bytes must stay readable, so most schema changes that would break either are rejected. Historically that froze the `required` set of a document type in both directions — requiredness is baked into the document wire format (required properties serialize raw, optional ones carry a presence flag), so changing it would desynchronize every stored document's bytes from the schema used to read them.
+
+From protocol v14, an update **may add a brand-new required property** by annotating it with `requiredSince` equal to the contract version the update creates:
+
+```json
+"properties": {
+  "newField": { "type": "string", "maxLength": 63, "position": 4, "requiredSince": 3 }
+},
+"required": ["existingField", "newField"]
+```
+
+The rules, enforced by consensus (`DataContractInvalidRequiredFieldsUpdateError`, code 10276, on violation):
+
+- The annotation must name **exactly the new contract version** — requiredness can be neither pre-scheduled for a future version nor backdated.
+- Only **brand-new properties** can become required. Promoting an existing (optional) property is still rejected, as is removing anything from `required` or touching an existing `requiredSince` annotation.
+- On contract **creation**, `requiredSince` may only be `1`.
+- Annotations sit on **top-level properties** listed in `required`; nested properties cannot carry them.
+
+What happens to data:
+
+- **Existing documents are grandfathered.** Each document carries a *contract version stamp* recording the contract version its bytes conform to (see [Document Serialization](../serialization/document-serialization.md)); a document stamped below a property's `requiredSince` may omit that property and still reads, transfers, and deletes normally.
+- **New writes are held to the new schema.** Creates must supply the property; replaces re-supply full content, so replacing a grandfathered document requires the new property and re-stamps the document at the current version — lazy migration, one document at a time.
+- **Indexes are unaffected** because index additions on update remain banned — a newly added required field cannot be indexed retroactively (there is no backfill).
+
 ## Rules and Guidelines
 
 **Do:**

book/src/data-model/documents.md

@@ -37,6 +37,7 @@ pub struct DocumentV0 {
     pub updated_at_core_block_height: Option<CoreBlockHeight>,
     pub transferred_at_core_block_height: Option<CoreBlockHeight>,
     pub creator_id: Option<Identifier>,
+    pub contract_version: Option<u32>,
 }
 ```
 
@@ -54,6 +55,8 @@ Let us walk through the key fields:
 
 - **`creator_id`**: The original creator of the document. This differs from `owner_id` when a document has been transferred to a new owner.
 
+- **`contract_version`**: The data contract version this document's bytes conform to — the *contract version stamp* (protocol v14+, document serialization format 3). Drive assigns it whenever document content is supplied (create and replace) and preserves it through transfers and purchases. `None` means the document was serialized before format 3, which predates every `requiredSince` annotation. The stamp resolves per-property byte layouts when a document type gains required properties through contract updates — see the [Document Serialization](../serialization/document-serialization.md) chapter.
+
 ## Document ID Generation
 
 Document IDs are not random -- they are derived deterministically. From `packages/rs-dpp/src/document/generate_document_id.rs`:
@@ -99,6 +102,7 @@ pub trait DocumentV0Getters {
     fn created_at_block_height(&self) -> Option<u64>;
     fn updated_at_block_height(&self) -> Option<u64>;
     fn creator_id(&self) -> Option<Identifier>;
+    fn contract_version(&self) -> Option<u32>;
     // ... and more
 }
 ```

book/src/error-handling/error-codes.md

@@ -50,7 +50,7 @@ Error codes are organized into ranges that correspond to error categories and su
 |-------|----------|----------|
 | 10000-10099 | Versioning | `UnsupportedVersionError` (10000), `ProtocolVersionParsingError` (10001), `IncompatibleProtocolVersionError` (10004) |
 | 10100-10199 | Structure | `JsonSchemaCompilationError` (10100), `InvalidIdentifierError` (10102), `ValueError` (10103) |
-| 10200-10275 | Data Contract | `DataContractMaxDepthExceedError` (10200), `DuplicateIndexError` (10201), `InvalidDataContractIdError` (10204) |
+| 10200-10276 | Data Contract | `DataContractMaxDepthExceedError` (10200), `DuplicateIndexError` (10201), `InvalidDataContractIdError` (10204), `DataContractInvalidRequiredFieldsUpdateError` (10276) |
 | 10350-10359 | Groups | `GroupPositionDoesNotExistError` (10350), `GroupExceedsMaxMembersError` (10354) |
 | 10400-10418 | Documents | `DataContractNotPresentError` (10400), `DuplicateDocumentTransitionsWithIdsError` (10401) |
 | 10450-10460 | Tokens | `InvalidTokenIdError` (10450), `TokenTransferToOurselfError` (10456) |

book/src/serialization/document-serialization.md

@@ -11,7 +11,10 @@ Every serialized document follows this layout:
 ```text
 ┌──────────────────────┐
 │  Serialization       │  varint (1-2 bytes)
-│  Version             │  Currently: 0, 1, or 2
+│  Version             │  Currently: 0, 1, 2, or 3
+├──────────────────────┤
+│  Contract version    │  V3 only: varint
+│  stamp               │  (0 = unstamped)
 ├──────────────────────┤
 │  $id                 │  32 bytes
 ├──────────────────────┤
@@ -44,8 +47,9 @@ The first bytes of a serialized document are a **varint** encoding the serializa
 | 0 | Original format. All integers encoded as **i64** (8 bytes big-endian) regardless of their schema type. |
 | 1 | Integers encoded at their **native size** (u8 = 1 byte, u16 = 2 bytes, u32 = 4 bytes, etc.). Otherwise identical to v0. |
 | 2 | Same as v1, but adds **`$creatorId`** field after `$ownerId` for document types that support transfers or trading. |
+| 3 | Same as v2, but adds a **contract version stamp** varint immediately after the version varint (protocol v14+). The stamp selects each property's layout when the document type carries `requiredSince` annotations — see below. |
 
-The varint encoding uses the [`integer-encoding`](https://docs.rs/integer-encoding) crate's `VarInt` format. For values 0, 1, and 2, the varint is a single byte: `0x00`, `0x01`, or `0x02`.
+The varint encoding uses the [`integer-encoding`](https://docs.rs/integer-encoding) crate's `VarInt` format. For values 0 through 3, the varint is a single byte: `0x00`, `0x01`, `0x02`, or `0x03`.
 
 ```rust
 // Serialization version is written first
@@ -60,12 +64,36 @@ match serialized_version {
     0 => DocumentV0::from_bytes_v0(serialized_document, document_type, platform_version),
     1 => DocumentV0::from_bytes_v1(serialized_document, document_type, platform_version),
     2 => DocumentV0::from_bytes_v2(serialized_document, document_type, platform_version),
+    3 => DocumentV0::from_bytes_v3(serialized_document, document_type, platform_version),
     _ => Err(/* unknown version */),
 }
 ```
 
 Note: version 0 has a fallback — if deserialization as v0 (all i64) fails, it retries as v1 (native integer types). This handles edge cases from protocol versions 1–8 where the version byte was 0 but non-i64 integer types may have been used.
 
+## The contract version stamp (v3)
+
+Serialization version 3 (the default from protocol v14) writes one extra varint immediately after the version varint: the **contract version stamp** — the version of the data contract the document's bytes conform to. A value of `0` means *unstamped*: the document was originally serialized before format 3 existed and has merely been rewritten in the new envelope (for example by a transfer).
+
+The stamp exists because contract updates may add new **required** properties from a specific contract version onward, using the `requiredSince` schema keyword:
+
+```json
+"properties": {
+  "newField": { "type": "string", "maxLength": 63, "position": 4, "requiredSince": 3 }
+},
+"required": ["existingField", "newField"]
+```
+
+Requiredness is baked into the wire format — a required property serializes raw while an optional one carries a presence flag — so a property whose requiredness varies by contract version needs the stamp to resolve its layout. The rule, per property:
+
+> A property is encoded as **required** (no presence flag) if it is listed in `required` **and** either it has no `requiredSince` annotation, or the document's stamp is **at or above** the annotation. Otherwise it is encoded as optional (presence-flagged).
+
+An unstamped document (`0`) predates every `requiredSince` annotation, so only unconditionally required properties count as required for it. This means the **latest contract alone** reconstructs the byte layout of every document ever stored — no historical contract lookups are needed.
+
+The stamp is **platform-assigned**: Drive sets it to the current contract version whenever document content is supplied (create and replace), and preserves it untouched through server-side rewrites that do not re-supply content (transfer and purchase). A document created before a contract update therefore keeps its old stamp — and may legitimately omit properties the newest schema requires — until a replace re-supplies its content and re-stamps it. Clients can also use the stamp as a staleness signal: a document stamped above the client's cached contract version means the contract needs refetching.
+
+Formats 0–2 have no stamp; documents read from them deserialize with `contract_version = None`, equivalent to a `0` stamp.
+
 ## Field-by-field breakdown
 
 ### `$id` (32 bytes)
@@ -128,7 +156,7 @@ If the document type's `trade_mode` allows seller-set pricing:
 
 Properties are serialized **in schema position order** — each property in the data contract schema has a `position` field, and `document_type.properties()` returns an `IndexMap` sorted by that position. This is *not* alphabetical order.
 
-Each property is encoded based on its type and whether it is required:
+Each property is encoded based on its type and whether it is required. In serialization version 3, "required" means *required at the document's contract version stamp* (see above); in versions 0–2 — and for every property without a `requiredSince` annotation — it is simply whether the property is listed in `required`.
 
 **Required fields**: The value is written directly with no prefix byte.
 
@@ -255,3 +283,5 @@ See `packages/rs-scripts/README.md` for full usage details.
 5. **Optional fields have a presence byte.** If you forget to read the `0x00`/`0x01` prefix for optional fields, every subsequent field will be shifted by one byte.
 
 6. **ByteArray encoding depends on size constraints.** Fixed-size byte arrays (where `minItems == maxItems` in the schema) have no length prefix. Variable-size byte arrays have a varint length prefix. Check the schema to know which encoding is used.
+
+7. **In version 3, the same document type can produce different property layouts.** A property annotated with `requiredSince` is presence-flagged in documents stamped below the annotation and raw in documents stamped at or above it. Two version-3 documents of the same type may therefore differ in layout — always read the stamp varint and resolve each property's requiredness against it before decoding the properties section.

packages/rs-dpp/schema/meta_schemas/document/v3/document-meta.json

@@ -1,7 +1,7 @@
 {
   "$schema": "https://json-schema.org/draft/2020-12/schema",
   "$id": "https://github.com/dashpay/platform/blob/master/packages/rs-dpp/schema/meta_schemas/document/v1/document-meta.json",
-  "$comment": "EDITABLE UNTIL THE RELEASE CARRYING PROTOCOL V14 SHIPS — FROZEN AFTER. This v3 document meta-schema activates with protocol v14 (CONTRACT_VERSIONS_V6). It is v2 plus the ranked index keywords (rankedCountable, rankedSummable, rankedAverageable) and the refersTo reference keyword on identifier properties, and admits every v14+ contract written to disk. v2 stays in place for protocol v13, where those keys still fail an index entry's `additionalProperties: false`. Once the release carrying protocol v14 ships, mutating it would change historical validation results and break consensus replay. After release, any new top-level property or rule MUST go in a newer meta-schema version (v4+). The $id above deliberately still names the v1 path: v1, v2 and v3 all share that identity, and it is the exact string `enrich_with_base_schema` injects as every PV12+ document schema's `$schema`, so bumping it here would be a wire-visible change rather than a documentation fix.",
+  "$comment": "EDITABLE UNTIL THE RELEASE CARRYING PROTOCOL V14 SHIPS — FROZEN AFTER. This v3 document meta-schema activates with protocol v14 (CONTRACT_VERSIONS_V6). It is v2 plus the ranked index keywords (rankedCountable, rankedSummable, rankedAverageable), the refersTo reference keyword on identifier properties, and the requiredSince property keyword (the contract version a property is required from), and admits every v14+ contract written to disk. v2 stays in place for protocol v13, where those keys still fail an index entry's `additionalProperties: false`. Once the release carrying protocol v14 ships, mutating it would change historical validation results and break consensus replay. After release, any new top-level property or rule MUST go in a newer meta-schema version (v4+). The $id above deliberately still names the v1 path: v1, v2 and v3 all share that identity, and it is the exact string `enrich_with_base_schema` injects as every PV12+ document schema's `$schema`, so bumping it here would be a wire-visible change rather than a documentation fix.",
   "type": "object",
   "$defs": {
     "documentProperties": {
@@ -224,6 +224,11 @@
         "position": {
           "type": "integer",
           "minimum": 0
+        },
+        "requiredSince": {
+          "type": "integer",
+          "minimum": 1,
+          "maximum": 4294967295
         }
       },
       "dependentSchemas": {

packages/rs-dpp/src/data_contract/document_type/class_methods/apply_required_since/mod.rs

@@ -0,0 +1,40 @@
+use std::collections::BTreeMap;
+
+use platform_value::Value;
+use platform_version::version::PlatformVersion;
+
+use crate::data_contract::errors::DataContractError;
+
+mod v0;
+
+/// Parses the `requiredSince` keyword: the contract version from which the
+/// property is required. Only meaningful on top-level required properties —
+/// the document wire format encodes a required property without a presence
+/// flag, so requiredness that varies by contract version must be resolvable
+/// per property from the current schema alone (see the per-document contract
+/// version stamp in document serialization format 3).
+///
+/// Versioned on `apply_required_since` in the platform version's document
+/// type schema versions. `None` selects the behavior of the versions that
+/// predate the keyword: it is ignored entirely, so their parses stay
+/// byte-for-byte identical to what they always produced.
+pub(crate) fn apply_required_since(
+    inner_properties: &BTreeMap<String, &Value>,
+    is_required: bool,
+    is_top_level: bool,
+    platform_version: &PlatformVersion,
+) -> Result<Option<u32>, DataContractError> {
+    match platform_version
+        .dpp
+        .contract_versions
+        .document_type_versions
+        .schema
+        .apply_required_since
+    {
+        None => Ok(None),
+        Some(0) => v0::apply_required_since_v0(inner_properties, is_required, is_top_level),
+        Some(version) => Err(DataContractError::Unsupported(format!(
+            "apply_required_since version {version} is not supported"
+        ))),
+    }
+}

packages/rs-dpp/src/data_contract/document_type/class_methods/apply_required_since/v0/mod.rs

@@ -0,0 +1,43 @@
+use std::collections::BTreeMap;
+
+use platform_value::Value;
+
+use crate::data_contract::document_type::property_names;
+use crate::data_contract::errors::DataContractError;
+
+/// Generation 0 parse rules: the keyword is admitted only on a top-level
+/// property listed in `required`, and its value is a contract version of at
+/// least 1 fitting in a u32.
+pub(super) fn apply_required_since_v0(
+    inner_properties: &BTreeMap<String, &Value>,
+    is_required: bool,
+    is_top_level: bool,
+) -> Result<Option<u32>, DataContractError> {
+    let Some(required_since_value) = inner_properties.get(property_names::REQUIRED_SINCE) else {
+        return Ok(None);
+    };
+
+    if !is_top_level {
+        return Err(DataContractError::InvalidContractStructure(
+            "requiredSince is only allowed on top-level properties".to_string(),
+        ));
+    }
+
+    if !is_required {
+        return Err(DataContractError::InvalidContractStructure(
+            "requiredSince is only allowed on properties listed in required".to_string(),
+        ));
+    }
+
+    let required_since: u32 = required_since_value
+        .to_integer()
+        .map_err(|e| DataContractError::ValueWrongType(e.to_string()))?;
+
+    if required_since == 0 {
+        return Err(DataContractError::InvalidContractStructure(
+            "requiredSince must be a contract version of at least 1".to_string(),
+        ));
+    }
+
+    Ok(Some(required_since))
+}

packages/rs-dpp/src/data_contract/document_type/class_methods/mod.rs

@@ -1,10 +1,12 @@
+use crate::consensus::basic::data_contract::DataContractInvalidRequiredFieldsUpdateError;
 #[cfg(feature = "validation")]
 use crate::consensus::basic::BasicError;
 #[cfg(feature = "validation")]
 use crate::consensus::ConsensusError;
 use crate::data_contract::errors::DataContractError;
 use crate::ProtocolError;
 
+pub(crate) mod apply_required_since;
 mod create_document_types_from_document_schemas;
 mod should_use_creator_id;
 mod system_properties;
@@ -26,6 +28,27 @@ pub(crate) fn consensus_or_protocol_data_contract_error(
     }
 }
 
+#[inline]
+pub(crate) fn consensus_or_protocol_required_fields_error(
+    error: DataContractInvalidRequiredFieldsUpdateError,
+) -> ProtocolError {
+    #[cfg(feature = "validation")]
+    {
+        ProtocolError::ConsensusError(
+            ConsensusError::BasicError(BasicError::DataContractInvalidRequiredFieldsUpdateError(
+                error,
+            ))
+            .into(),
+        )
+    }
+    #[cfg(not(feature = "validation"))]
+    {
+        ProtocolError::DataContractError(DataContractError::InvalidContractStructure(
+            error.to_string(),
+        ))
+    }
+}
+
 #[inline]
 pub(crate) fn consensus_or_protocol_value_error(
     platform_value_error: platform_value::Error,

packages/rs-dpp/src/data_contract/document_type/class_methods/try_from_schema/common/mod.rs

@@ -745,6 +745,7 @@ fn parse_document_properties(
             &mut document_properties,
             &required_fields,
             &transient_fields,
+            true,
             property_key,
             property_value,
             root_schema,