fix(client): scale connection budget by distinct endpoint count

[viraatc]

Summary

The client's auto max_connections clamp needlessly throttles throughput when multiple endpoints are configured.

The ephemeral-port limit is per (source IP, destination) pair — the TCP 4-tuple (src_ip, src_port, dst_ip, dst_port) only needs to be unique, so the kernel reuses local ports across distinct destinations and each endpoint gets its own ~ephemeral-range budget. But the auto clamp set max_connections to a single pair's budget (available_ports), so listing N frontends capped total concurrency at one endpoint's worth — even though workers are already round-robined across endpoints (worker.py).

Change

Scale the clamp by the number of distinct (host, port) endpoints:

port_budget = available_ports * max(1, distinct_endpoints)

• Single endpoint: unchanged.
• Duplicate endpoints (same host:port, even different paths): don't inflate the budget — the 4-tuple ignores path.
• Explicit --max-connections is validated against the scaled budget.

No new config or CLI — the multiplier comes from the endpoint_urls you already specify.

Validation

• 5 unit tests: scaling by distinct endpoints, duplicates don't inflate, single-endpoint unchanged, explicit-budget within/exceeding.
• OS-level, through the real ConnectionPool (one pool per destination, ephemeral range shrunk to 1000 ports in a netns): 1 endpoint → 999 simultaneous connections; 5 endpoints → 4975 (995 each — independent per-pair budgets), confirming the port limit is per source-destination pair and each endpoint contributes its own budget.

Real-hardware verification (GB200, lyris, 2-node cross-node)

Exhausted the full ephemeral range (32768–60999 = 28,232 ports/destination) from a client node to 3 distinct destination ports on a separate server node, via the tool's real connect path (create_connection, no source bind → autobind-at-connect):

destination (server port)	connections before EADDRNOTAVAIL
:50001	28,227
:50002	28,228
:50003	28,228
total	84,683 = 3.00× the single-destination range

• Source-port overlap across destinations: 28,227 — essentially every ephemeral port is reused across all 3 endpoints, directly confirming the 4-tuple reuse this PR relies on.
• Idle pool memory: 13 bytes/connection (1.1 MB for all 84,683 sockets) — a scaled idle pool is effectively free; resident memory tracks in-flight data, not pool size.
• fd ceiling: the GB200 compute node's RLIMIT_NOFILE hard cap is 131,072, so beyond ~4 endpoints the fd limit (not ephemeral ports) becomes the binding constraint — relevant when sizing very large --max-connections.

🤖 Generated with Claude Code

[github-actions]

MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅

[gemini-code-assist]

Code Review

This pull request scales the maximum connections budget based on the number of distinct endpoints configured, preventing unnecessary concurrency throttling when multiple endpoints are used. It also introduces comprehensive unit tests to verify this scaling behavior. The review feedback identifies potential edge cases in the endpoint parsing logic, specifically regarding missing URL schemes and default ports (e.g., HTTP vs. HTTPS), which could lead to incorrect budget calculations, and suggests a more robust normalization approach.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[gemini-code-assist]

The current parsing logic for distinct endpoints can lead to incorrect budget calculations in several scenarios:

1. Missing Schemes: If an endpoint URL is specified without a scheme (e.g., '10.0.0.1:8000'), urlparse parses it as a path, resulting in (None, None) for the host and port. This collapses all such endpoints into a single budget.
2. Default Ports: Endpoints like 'http://example.com' and 'https://example.com' are distinct destinations (ports 80 and 443), but both return None for the port, collapsing them into one. Conversely, 'http://example.com' and 'http://example.com:80' are the same destination but will be counted as two distinct endpoints.

We can make this more robust by normalizing the scheme and resolving default ports based on the scheme.

            distinct_endpoints = set()
            for u in self.endpoint_urls:
                parsed = urlparse(u)
                if not parsed.scheme:
                    parsed = urlparse(f"http://{u}")
                host = parsed.hostname
                port = parsed.port or (443 if parsed.scheme == "https" else 80)
                distinct_endpoints.add((host, port))
            distinct_endpoints_count = len(distinct_endpoints)
            port_budget = available_ports * max(1, distinct_endpoints_count)

            if self.max_connections == -1:
                object.__setattr__(self, "max_connections", port_budget)
            elif self.max_connections > 0:
                if self.max_connections > port_budget:
                    raise RuntimeError(
                        f"--max-connections ({self.max_connections}) exceeds the ephemeral "
                        f"port budget ({port_budget} = {available_ports} ports x "
                        f"{max(1, distinct_endpoints_count)} distinct endpoint(s)). Reduce "
                        f"--max-connections, add endpoints, or raise the system port range."
                    )

[nv-alicheng]

This is true if we don't enforce 'http://' being in the YAML 'endpoint list' @viraatc