feat: add JSON format versioning support for converter#88
Closed
Conversation
Introduce JSONVersion type and constants to track input format versions. This prepares for versioned parsing when the JSON format changes.
DetectJSONVersion analyzes JSON structure to determine which parser version should handle the data. Currently detects V1 by required fields.
Introduce Parser interface for version-specific JSON parsing. Registry pattern allows adding new format versions without modifying existing code.
Move JSON parsing logic to dedicated ParserV1 implementing Parser interface. This preserves existing behavior while enabling future format versions.
Replace hardcoded parseJSONData with parseJSONDataVersioned that detects the input format version and delegates to the appropriate parser. Falls back to V1 for unrecognized formats.
Track which JSON format version was used for each recording. Enables future re-conversion when format changes.
Clients can now check which JSON format version was used to create each recording.
Document how to add new format versions when the OCAP JSON schema changes.
Merging this branch will increase overall coverage
Coverage by fileChanged files (no unit tests)
Please note that the "Total", "Covered", and "Missed" counts above refer to code statements instead of lines of code. The value in brackets refers to the test coverage of that file in the old version of the code. Changed unit test files
|
Member
Author
|
Closing in favor of a more comprehensive end-to-end versioning approach. See new plan in docs/plans/2026-01-30-end-to-end-versioning.md |
fank
added a commit
that referenced
this pull request
Jan 31, 2026
Comprehensive versioning strategy covering: - JSON input format versioning - Protobuf/FlatBuffers schema versioning - Versioned writers and readers - Database schema_version tracking - API schemaVersion in responses - Frontend versioned loaders Replaces partial implementation from closed PR #88.
fank
added a commit
that referenced
this pull request
Jan 31, 2026
* docs: add end-to-end format versioning plan Comprehensive versioning strategy covering: - JSON input format versioning - Protobuf/FlatBuffers schema versioning - Versioned writers and readers - Database schema_version tracking - API schemaVersion in responses - Frontend versioned loaders Replaces partial implementation from closed PR #88. * feat(storage): add unified version types for end-to-end versioning * refactor(schemas): move protobuf schema to v1 directory Move protobuf schema files to versioned directory structure: - pkg/schemas/protobuf/ocap.proto -> pkg/schemas/protobuf/v1/ocap.proto - Update package from `ocap` to `ocap.v1` - Update go_package to `github.com/OCAP2/web/pkg/schemas/protobuf/v1` - Regenerate Go code with protoc Part of end-to-end versioning system (Task 2.1). * refactor(schemas): move flatbuffers schema to v1 directory - Create pkg/schemas/flatbuffers/v1/ directory structure - Move ocap.fbs to v1/ and update namespace from ocap.fb to ocap.v1.fb - Regenerate FlatBuffers Go code in v1/generated/ - Delete old fbs and generated/ from pkg/schemas/flatbuffers/ Note: Build will fail until imports are updated in Task 2.3 * refactor(storage): update imports to use versioned schema packages Update all Go files that import protobuf and flatbuffers schemas to use the new v1 versioned paths. Changes pb -> pbv1 and fb -> fbv1 aliases. * feat(storage): add Parser interface with schema-agnostic types Add ParseResult intermediate representation for JSON parsing that is schema-agnostic. This allows different input JSON versions to be parsed by different Parser implementations and then written to any output format. - ParseResult: intermediate data structure with manifest, entities, events, markers, times, and position data - Parser interface: Version() and Parse() methods - Parser registry: RegisterParser/GetParser for version-specific parsers - Reuses existing EntityDef and Event types from engine.go * feat(storage): add ParserV1 for JSON v1 format Implement ParserV1 that parses the current Arma 3 JSON format into schema-agnostic types (EntityDef, Event, MarkerDef, TimeSample, EntityPositionData). This is part of the versioned parser/writer architecture that enables future format evolution. * feat(storage): add Writer interface with version prefix helpers Add Writer interface as the counterpart to Parser for writing ParseResult to specific formats (protobuf/flatbuffers) with schema versioning. - Writer interface with Version(), Format(), WriteManifest(), WriteChunks() - Writer registry with RegisterWriter() and GetWriter() - WriteVersionPrefix() writes 4-byte little-endian version prefix - ReadVersionPrefix() reads version prefix from files - Comprehensive tests for registry and version prefix round-trip * feat(storage): add ProtobufWriterV1 for v1 protobuf format Implement Writer interface for protobuf v1 format that: - Writes 4-byte version prefix before protobuf data - Converts schema-agnostic ParseResult to protobuf messages - Writes manifest.pb with version prefix - Writes chunk files with version prefix * feat(storage): add FlatBuffersWriterV1 for v1 flatbuffers format Implements the Writer interface for FlatBuffers v1 format with 4-byte version prefix. Supports writing manifest with entities, events, markers, and time samples, plus chunked frame data with entity states. * feat(storage): add version prefix support to protobuf engine Update ProtobufEngine to handle files with optional 4-byte version prefix. This provides backward compatibility with legacy files while supporting the new versioned format written by ProtobufWriterV1. The version detection uses a heuristic: if bytes 2-4 are all zero, the first 4 bytes are interpreted as a little-endian version number. Otherwise, the file is treated as legacy format without version prefix. * feat(storage): add version prefix support to flatbuffers engine Add readVersionedData method to FlatBuffersEngine for handling files with 4-byte version prefix. This provides backward compatibility with both versioned and legacy FlatBuffers files. Key changes: - Add readVersionedData method using FlatBuffers-specific heuristic - Update GetManifest and GetChunk to use versioned data reading - Add tests for versioned and legacy file handling The heuristic distinguishes version prefixes from FlatBuffers root offsets by checking: first byte < 16 AND bytes 2-4 are zero (version prefix) vs first byte >= 16 (FlatBuffers root offset). * refactor: use versioned parser/writer in converter Refactor converter.go to use the versioned parser and writer components: - Use DetectJSONInputVersion() to detect input JSON version - Use GetParser() to get the appropriate versioned parser - Use GetWriter() to get the appropriate versioned writer - Add format parameter to Convert() for output format selection - Remove duplicated parsing logic (now in parser_v1.go) - Remove duplicated writing logic (now in writer_protobuf_v1.go/writer_flatbuffers_v1.go) - Update protobuf.go and flatbuffers.go Convert methods to pass format - Remove unused helper functions from flatbuffers.go - Update tests to handle 4-byte version prefix in output files * feat: complete end-to-end versioning implementation - Add schema_version column to database (migration v4) - Add SchemaVersion to FormatInfo API response - Add UpdateSchemaVersion to OperationRepo interface - Create frontend versioned loader system (LoaderV1, LoaderRegistry) - Update ocap.js to use versioned loaders from registry - Update ChunkManager to use versioned loader for chunk decoding - Mark versioning plan as implemented * Revert "docs: add end-to-end format versioning plan" This reverts commit e61e5c2. * feat: end-to-end format versioning Implements comprehensive versioning throughout the entire data pipeline, allowing format changes to evolve independently while maintaining backwards compatibility. Version flow: JSON Input → Parser → Writer → .pb/.fb → Reader → API → UI Loader Backend: - Add SchemaVersion/JSONInputVersion types with detection - Move schemas to versioned directories (protobuf/v1, flatbuffers/v1) - Add Parser interface with registry (ParserV1 implementation) - Add Writer interface with registry (ProtobufWriterV1, FlatBuffersWriterV1) - Add 4-byte version prefix to binary files - Add schema_version column to database (migration v4) - Add schemaVersion to FormatInfo API response Frontend: - Add LoaderRegistry for versioned loaders - Add LoaderV1 wrapping existing decoders - Update ocap.js and ChunkManager to use versioned loaders * fix: CLI convert command now preserves .json in output path The convertSingleFile function was stripping both .gz and .json extensions, but database filenames include .json. This caused a mismatch where converted files couldn't be found by the API. Now only strips .gz to match the database filename format. * feat: CLI convert updates database when operation exists When using `convert --input`, the CLI now: - Looks up operation by filename in database - Updates mission duration from manifest - Updates storage format, conversion status, and schema version Also adds GetByFilename method to RepoOperation. * refactor: CLI uses worker.ConvertOne for DB operations When converting a file that exists in the database, the CLI now uses the same worker.ConvertOne() method as the background worker, ensuring identical behavior for both paths. Standalone conversions (no DB entry) still work without DB updates. * test: add tests for GetByFilename and UpdateSchemaVersion * test: add test for CLI convert with database entry * test: add tests for protobuf ChunkCount and readVersionedData error paths * test: add SchemaVersion assertions to handler tests
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Changes
Storage layer (
internal/storage/):version.go- JSONVersion type, constants, and DetectJSONVersion()parser.go- Parser interface and registry (RegisterParser, GetParser)parser_v1.go- V1 parser implementing Parser interfaceconverter.go- Refactored to use versioned parser registryVERSIONING.md- Documentation for adding new versionsServer layer (
internal/server/):operation.go- Addedjson_format_versioncolumn (migration v4)handler.go- FormatInfo now includes JSONFormatVersionHow it works
When the JSON format changes in the future:
JSONVersionV2)DetectJSONVersion()to recognize new formatparser_v2.go)Test plan