[Rust] Improve Object API codegen: idiomatic types, Eq/Hash derives, String shadowing fix#8931
Open
pittengermdp wants to merge 2665 commits intogoogle:masterfrom
Open
[Rust] Improve Object API codegen: idiomatic types, Eq/Hash derives, String shadowing fix#8931pittengermdp wants to merge 2665 commits intogoogle:masterfrom
pittengermdp wants to merge 2665 commits intogoogle:masterfrom
Conversation
replace `main` with `master` that we are still using
… leading to an incorrect count (google#8451) Removes all the unneeded keyword (mutating) from verifier Adds tests to verify depth
* CNAME: add custom domain * `flatc.md`: Add more documentation
* `quick_start.md`: Add quick start guide * `annotation.md`: Add section on annotating flatbuffers
* Mention uint8 as an alias to ubyte as it's referenced in the note below * Remove an unfinished sentence.
jtdavis777
added
the
waiting-for-update
Add ordered-float dependency to the Rust flatbuffers crate and modify idl_gen_rust.cpp to emit OrderedFloat<f32>/OrderedFloat<f64> for all float fields in generated *T types. This allows generated types with float fields to derive Hash and Eq. Changes: - rust/flatbuffers: add ordered-float dep, re-export ordered_float - idl_gen_rust.cpp: ObjectFloatType() helper, table/struct pack/unpack wrapping, vector-of-float handling, array-of-float handling, TypeIsHashable/UnionIsHashable always return true
pittengermdp
force-pushed
the
rust-object-api-improvements
branch
from
819b32e to
65e4333
Compare
github-actions
bot
added
golang
java
c#
javascript
grpc
typescript
swift
documentation
…API, union-of-unions
Phase 0 — Rust reflection verifier hardening:
- Replace vec![false; buffer.len()] DoS vector with HashSet<usize> capped at max_tables
- Make verify_with_options return HashMap instead of taking &mut ref
- Add Array and Vector64 base type support to reflection verifier
Phase 1a — Go and TypeScript verifier runtimes:
- go/verifier.go: full verifier with CheckTable, CheckString, CheckVector, depth/table limits
- ts/verifier.ts: full verifier with DataView-based reads, TextDecoder UTF-8 validation
- idl_gen_go.cpp: generate VerifyRootAs*/verify* per table with cross-namespace support
- idl_gen_ts.cpp: generate verify*/verifyRootAs* per table with cross-file imports
Phase 1b — Object API improvements:
- Go: omitempty JSON struct tags on *T types
- TypeScript: discriminated union types, pack() required field validation, clone(), equals()
- Rust: fix serde derives on struct *T types, add union discriminant debug_assert in pack()
Phase 2 — Reflection runtimes:
- go/reflect.go: LoadReflectionSchema, ObjectByName, FieldByName, GetField{String,Int,Float}
- ts/reflection.ts: standalone module with loadSchema, getField* functions
Phase 3 — Conformance and testing:
- tests/conformance/: cross-language corpus with Rust/Go/TS runners and compare.py
- tests/codegen_golden/: golden-file codegen tests with check_golden.sh
- Rust fuzz target for reflection verifier
- Go fuzz target and benchmarks for verifier
Union-of-unions:
- idl_parser.cpp: allow union members that are other unions via compile-time flattening
- Inner union members expanded into outer union discriminant numbering
- tests/union_of_union_test/: Go round-trip tests for flattened unions
Version bump to 25.12.20 across all generated headers.
Comprehensive API documentation for downstream consumers: - VTS (Rust): rustdoc on SafeBuffer, verify_with_options, all pub functions in lib.rs - LiftCloud (Go): godoc on Verifier, ReflectionSchema, all GetField* accessors - liftNet_frontend (TypeScript): TSDoc on Verifier class, ReflectionSchema, all exports Includes usage examples, error handling patterns, parameter descriptions, and cross-references between related types. No code logic changes.
Previously --bfbs-gen-embed only worked for C++. Now all three LN2 languages
can embed binary schemas directly in generated code for runtime reflection
without external file I/O.
Rust: generates {name}_bfbs_generated.rs with SCREAMING_SNAKE_CASE const
pub const MONSTER_BFBS: &[u8] = &[...];
pub fn monster_bfbs() -> &'static [u8] { MONSTER_BFBS }
Go: generates {Type}BinarySchema.go with exported var and accessor
var MonsterBfbsData = []byte{...}
func MonsterBinarySchemaBytes() []byte { return MonsterBfbsData }
TypeScript: generates {name}-bfbs.ts with exported Uint8Array
export const bfbsData: Uint8Array = new Uint8Array([...]);
All three use BufferToHexText from the serialized parser builder data.
Existing C++ --bfbs-gen-embed codegen is unchanged.
…eatures Phase 1: Union Match (Go), optional scalars (Go), self-describing envelope Phase 2: Selective unpack, reflection Pack/Unpack from JSON, buffer diffing Phase 3: Chunked builder allocator (Rust ARM, gated on profiling) Council: Systems Architect, Go Lead, Rust Lead, TS Lead, Security Engineer Key decisions: envelope pattern (not wire format), SHA-256, transport-enforced
Runtime reflection needed for CDO version detection, generic Pack/Unpack, structured logging, legacy format detection, and protocol negotiation. Schema leakage mitigated by access control, not stripping.
Rust: 8 doctests in reflection crate - FlatbufferError: fully runnable, tests Display + pattern matching - SafeBuffer::new, get_root, get_field_integer, get_field_string, get_field_table, get_field_float: compile-checked no_run examples showing LN2-style field access patterns Go: 7 Example functions in go/example_test.go - NewVerifier, Verifier_CheckTable, VerificationError, CheckAlignment, PushDepth: self-contained with hand-crafted buffers, output-verified - LoadReflectionSchema, GetFieldString: use monster_test.bfbs and monsterdata_go_wire.mon test fixtures, output-verified All examples serve as both documentation and regression tests.
Feature 1.1 — Union Match (Go):
Generated Match() method on Go union *T types with typed callbacks.
One callback per variant, type-safe dispatch, nil-safe.
Files: src/idl_gen_go.cpp (GenUnionMatch)
Feature 1.2 — Optional Scalars (Go):
Already implemented upstream — verified *int32/*float64 pointer accessors
exist for fields marked optional in the schema.
Feature 1.3 — Self-Describing Envelope:
go/envelope.go: EnvelopeWrap/Unwrap with SHA-256 schema fingerprint
ts/envelope.ts: Same API using Web Crypto SubtleCrypto
Format: "LN2E" magic + version + algo + 32-byte hash + payload
Feature 2.1 — Selective Unpack:
Go: UnpackFields(...string) on generated table types (idl_gen_go.cpp)
TS: unpackFields(...fields: string[]) on generated classes (idl_gen_ts.cpp)
Only materializes named fields, skips rest.
Feature 2.2 — Reflection Pack/Unpack from JSON:
go/reflect_json.go: ReflectUnpack (FlatBuffer → map[string]any)
ReflectPack (JSON → FlatBuffer)
ts/reflection-json.ts: reflectUnpack/reflectPack
Recursive schema walk handling all base types including unions.
Feature 2.3 — Buffer Diffing:
go/reflect_diff.go: DiffBuffers returns []FieldDelta with path/kind/values
ts/reflection-diff.ts: diffBuffers returns FieldDelta[]
Recursive field comparison including nested tables and vectors.
Feature 3.1 — Chunked Builder (Rust proof-of-concept):
rust/reflection/src/chunked_builder.rs: segmented allocator for ARM
8 unit tests passing. Documented as PoC pending ARM profiling.
Version bumped to 25.12.22 across all generated headers.
…omprehensive tests
Reflection API now covers 100% of reflection.fbs tables and fields:
New types (Go + TS):
- ReflectionEnum: name, is_union, values, underlying_type, attributes, docs
- ReflectionEnumVal: name, value, union_type, attributes, docs
- ReflectionKeyValue: key, value
New methods on existing types:
- Schema: Enums(), EnumByName(), EnumValueName(), FileIdent(), FileExt()
- Object: MinAlign(), Attributes(), Documentation(), DeclarationFile()
- Field: ID(), Deprecated(), Key(), Optional(), Padding(), Offset64(),
Attributes(), Documentation()
- Type: FixedLength(), BaseSize(), ElementSize()
Test suite (1,815 lines across 5 files):
- reflect_unit_test.go: every public method on every type
- reflect_integration_test.go: end-to-end schema + data workflows
- reflect_bench_test.go: 8 benchmarks for hot-path operations
- reflect_fuzz_test.go: 3 fuzz targets (schema loading, field reads)
- reflect_property_test.go: 9 schema invariant checks
SafeBuffer schema accessors: - file_ident(), file_ext(), enums(), enum_by_name(), enum_value_name() - objects(), object_by_name() SafeTable new methods: - get_field_bool() — boolean field access - get_field_union() -> Option<(u8, SafeTable)> — union discriminant + table - get_any_field() -> FieldValue — dynamic typed access for logging/inspection - object() — returns the schema Object for this table - fields() — returns all field definitions FieldValue enum: - Bool, Integer, Float, String, Table, Struct, VecInteger, VecFloat, VecString, Union, Absent variants - Display impl for structured logging output SafeStruct: get_field_bool() Tests: 145 unit tests + 23 doc tests passing (up from 113 + 8) Rustdoc: 677 lines of documentation with examples
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
String,Vec,Box) instead of fully-qualifiedalloc::paths in generated Rust Object API code, withuse alloc::*imports added forno_stdcompatibilityEqandHashon Object API types (tables, structs, unions, bitflags) when no floating-point fields are present in the type hierarchyStringtype shadowing: when a schema defines a table namedString, generated code now emitsstd::string::Stringfor stdlib string fields (wires up existing unusedNamespaceShadowsString/ObjectStringTypehelpers)#[allow(unused_imports)]to suppress warnings on generated importsDeserializeimpl for enums to use fully-qualifiedstd::string::StringTest plan
flattests)