Memory metrics ignore container memory limits (no cgroup awareness)

[vdusek]

get_memory_info() in src/crawlee/_utils/system.py takes the system totals from psutil.virtual_memory(), which reads /proc/meminfo. That file is not namespaced, so inside a container it reports the host's memory rather than the cgroup limit:

vm = psutil.virtual_memory()

return MemoryInfo(
    total_size=ByteSize(vm.total),
    current_size=ByteSize(current_size_bytes),
    system_wide_used_size=ByteSize(vm.total - vm.available),
)

Consequences for a crawler in a memory-limited container:

• MemoryInfo.total_size is the host's total RAM.
• Snapshotter derives the memory budget from it - with memory_mbytes unset, max_memory_size = total_size * available_memory_ratio (src/crawlee/_autoscaling/snapshotter.py:122-128). So the autoscaler can scale past the container limit and get OOM-killed instead of throttling.
• system_wide_used_size reflects the host's usage, including processes unrelated to the crawler.

Crawlee for JS handles this - getMemoryInfoV2() reads the cgroup files directly (packages/utils/src/internals/systemInfoV2/memory-info.ts), with cgroups v1/v2 detection, the max sentinel for "unlimited", and a separate AWS Lambda path:

TOTAL: v1 /sys/fs/cgroup/memory/memory.limit_in_bytes, v2 /sys/fs/cgroup/memory.max
USED:  v1 /sys/fs/cgroup/memory/memory.usage_in_bytes,  v2 /sys/fs/cgroup/memory.current

Suggested direction: mirror the JS behavior - detect containerization, read the cgroup limit when present, and fall back to psutil.virtual_memory() otherwise. Worth deciding whether it should be automatic or opt-in via Configuration (JS gates it behind a systemInfoV2 flag plus a containerized option).

Notes:

• Not a regression - crawlee-python has never had cgroup awareness (grep -rn cgroup src/ finds nothing).
• Setting memory_mbytes (or CRAWLEE_MEMORY_MBYTES) is the current workaround, since it bypasses total_size entirely.
• Out of scope for fix(system): do not fail memory snapshots on inaccessible processes #2078, which hardens the per-process side of the same function against unreadable metrics. The container-limit side is untouched by it.

✍️ Drafted by Claude Code

[vdusek]

Hi @Mantisus, assigning you this in case you would like to inspect it.