feat: add complex type support to native Parquet writer

[andygrove]

Summary

• Enables support for complex types (arrays, maps, structs) in Comet's native Parquet writer
• Removes the blocking check that previously prevented complex types
• Adds comprehensive test coverage for complex types

Changes

• Remove complex type blocking check in CometDataWritingCommand.scala
• Add 12 new tests for complex types in CometParquetWriterSuite.scala:
  • Basic complex types (array, struct, map)
  • Nested complex types (array of structs, struct containing array, map with struct values, deeply nested)
  • Nullable complex types with nulls at various nesting levels
  • Complex types containing decimal and temporal types
  • Empty arrays and maps
  • Fuzz testing with randomly generated complex type schemas
• Update documentation to reflect complex type support

Test plan

• Tests verify round-trip compatibility (write with Comet, read with Spark/Comet)
• Fuzz testing with randomly generated schemas

🤖 Generated with Claude Code

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 59.63%. Comparing base (f09f8af) to head (af1d474).
⚠️ Report is 855 commits behind head on main.

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #3214      +/-   ##
============================================
+ Coverage     56.12%   59.63%   +3.50%     
- Complexity      976     1416     +440     
============================================
  Files           119      170      +51     
  Lines         11743    15700    +3957     
  Branches       2251     2595     +344     
============================================
+ Hits           6591     9362    +2771     
- Misses         4012     5021    +1009     
- Partials       1140     1317     +177     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[andygrove]

With these changes, we can run the PySpark repartition benchmark fully native, and it shows an almost 2x speedup compared to Spark (and also ~2x compared to Comet when writes are disabled and Comet does the columnar-to-row transition).

[andygrove]

With these changes, we can run the PySpark repartition benchmark fully native, and it shows an almost 2x speedup compared to Spark (and also ~2x compared to Comet when writes are disabled and Comet does the columnar-to-row transition).

@comphead ☝️