Enable nested field statistics with inferred metrics limit

[jkasireddy]

Closes #2699

Rationale for this change

PyIceberg currently downgrades nested primitive fields to counts, which prevents lower and upper bound statistics from being collected for nested fields.

This change enables statistics collection for nested primitive fields and adds support for write.metadata.metrics.max-inferred-column-defaults, following the behavior of the Iceberg Java implementation.

When no explicit default metrics mode is configured, inferred metrics are limited to the configured number of fields. Primitive fields at the current struct level are prioritized before descending into nested fields. Explicit per-column metrics configuration continues to take precedence over the inferred limit.

Are these changes tested?

Yes.

Added regression tests covering:

• statistics collection for nested primitive fields
• write.metadata.metrics.max-inferred-column-defaults
• prioritization of top-level primitive fields before nested fields
• explicit per-column overrides beyond the inferred limit
• explicit default metrics mode behavior
• zero and negative inferred-column limits
• lower and upper bounds for nested struct, list, and map primitive fields

Local test results:

uv run python -m pytest tests/io/test_pyarrow_stats.py -q
24 passed

[jkasireddy]

Hi @Fokko
when you have a chance, would you mind taking a look at this? This addresses #2699 by enabling nested-field statistics and adding support for write.metadata.metrics.max-inferred-column-defaults, with behavior aligned to the Java implementation. I’ve added regression coverage for nested fields and the inferred metrics limit. Happy to make any changes based on your feedback. Thanks!

[greenlaw]

If write.metadata.metrics.default is explicitly set (say, to 'full'), wouldn't we still want write.metadata.metrics.max-inferred-column-defaults to apply? in fact, isn't that when limiting is most important?

[jkasireddy]

@greenlaw
Good question. I had considered that as well, but I intentionally made an explicitly configured write.metadata.metrics.default bypass write.metadata.metrics.max-inferred-column-defaults to match the current Java MetricsConfig behavior.

In the Java implementation, max-inferred-column-defaults is applied only when there is no explicitly configured default. If write.metadata.metrics.default=full is set, that is treated as an explicit user choice and the mode applies to all columns. Explicit per-column overrides similarly take precedence over the inferred limit.

Since the property is specifically max-inferred-column-defaults, I interpreted the limit as protecting the automatically inferred default behavior rather than overriding an explicit global configuration.

That said, I agree that limiting an explicit full default could also be useful for controlling manifest size. If we want PyIceberg to behave differently here, I’m happy to adjust it, but that would diverge from the current Java behavior. Would you prefer that we preserve Java parity for this PR?