AWS Lambda - Promises in Queue continue to process after Memory Exhausting on logical/corrupt Memory with all zombine promises

[wesleyolis]

Version

Node.js 16.x

Platform

Good Day,

This is a strangle one, you have to work with aws on this one and potentially security teams, because opens the door creative hacking and miscellaneous opportunities for code execution post running out of memory.

Case: 178411688600105

It looks like this was a memory crash in which the lambda function basically doesn’t terminate what is in the queue.
That malloc returns wrong pointer after it runs out of memory and we process random shit with the zombie promises that are still to complete.
Alternative someone didn’t do a null check for memory and assume that there would be enough memory, hence then continued process on random memory.
However, here the malloc is already virtualized, so it depends what
did the OS hosting system return after out of memory was trigger for follow up mallocs, as those promises should have failed and no
one should have been able to steal there memory they have allocated already, unless virtualized system release it and we malloc something
already on the reversed and populated, as long we didn’t attempt to write to it.

I guess to ensure that the os correctly returns an address for failed mallocs through all the virtualization and that no could break
out the virtualization at all and read other peoples memory, hopefully I can only think that the memory reservation for dynamically memory
must have been rest and the zombie functions continued to execute, not getting an error and other functions writing data into the memory.

So either no one checked for failure of memory alloc in NodeJS or library with direct memory management, work the virtualization stack has a race condition that it reset the dynamic memory pointer before hand
back to zero and then all remaining code continues to execute, that is in the queue given priority over it being nuked, so then have data written over old memory heap of the program
and well it continuous to execute with corruption and until some check triggers it to crash.

Maybe if the os when triggers out of memory, that all further calls from the same program get a special address of memory that when written to are no ops, which at least prevent the corruption of the current state still executing for security reasons and the fact no one is going to know what bugs exist in libraries, that wouldn't allow a process continue process with corrupt data, which then make it to some other system or storage or network pipe.
Depends on if the os, blocks io or all other persistence to protect them from logical corruption upon memory exhausting, as say 20 plus promises in the query could actually finish processing correct, but on corrupt data.

Logs:

INFO Retrieve operator for ip: 105.8.6.82
INFO Retrieve operator for ip: netdoc:
ERROR Uncaught Exception {"errorType":"SyntaxError","errorMessage":"Unexpected token < in JSON at position 0","stack":["SyntaxError: Unexpected token < in JSON at position 0"," at )"," at processTicksAndRejections (node:internal/process/task_queues:83:21)"]}

const { createHash } = require('crypto')
const { promisify } = require('util')
const AWS = require('aws-sdk')

const ipRangeCheck = require('ip-range-check')
const redis = require('redis')
const https = require('https')

const { getSecret } = require('secrets.js')

 "redis": "^3.1.1"
    ,"ip-range-check": "^0.2.0"

If a Node.js process runs out of memory, there are a few different scenarios:

Normal JavaScript heap out of memory

You'll typically see:

FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory
This is a fatal V8 error.
The entire Node.js process terminates immediately.
No pending promises continue executing because the process no longer exists.
Memory corruption (native code)
JavaScript itself cannot corrupt memory.
Memory corruption usually comes from:
native addons (N-API, node-gyp modules)
bugs in Node.js itself (rare)
C/C++ libraries
In this case, the process may:
crash with a segmentation fault
behave unpredictably
continue running with corrupted state
If it continues running, queued promises may still execute, but there is no guarantee they are operating on valid state. Any behavior is undefined.

Application-level corruption
Sometimes people describe "corrupt memory" when they actually mean inconsistent application state:

const queue = [];

processTask().then(() => {
    // object was partially modified
});

Here the process is still alive, so promises keep executing even though some objects are in an invalid state. This is logical corruption rather than memory corruption.

If the process is under heavy memory pressure

Node may spend a long time garbage collecting before eventually failing.

During this period:

promises already in the event loop can continue to execute,
new allocations may fail,
latency increases dramatically,
eventually the process usually aborts with an out-of-memory fatal error.
If you're seeing promises execute after an OOM

That usually indicates one of these situations:

the process hasn't actually hit the fatal heap limit yet;
an individual promise rejected due to allocation failure while the process remained alive;
the OOM occurred in a worker thread rather than the main process;
you're observing multiple Node.js processes, and a different process continued handling the queued work.
Questions to narrow it down
What is the exact error message? (JavaScript heap out of memory, ENOMEM, Segmentation fault, etc.)
Are you using a promise queue library (such as p-queue, BullMQ, Bee-Queue, or something custom)?
Is this a single Node.js process, or are you using cluster, worker_threads, or a process manager like PM2?
Are you using any native modules?

Subsystem

AWS Lambda

What steps will reproduce the bug?

You are going to have to try reproduce and out of memory with network io in all the requests.
const output = await Promise.all(
event.records.map(async (record) => {
.. code redis and then external http lookup
}

How often does it reproduce? Is there a required condition?

Reproduce for 29 lambda executions of batches, try pushing near the file descriptor limits maybe if nessary.

What is the expected behavior? Why is that the expected behavior?

logical state/memory corruption, should be prevented by the design of the system, once memory has been exhausted and tailing promises should complete with a guarantee of no state corruption or fail with out of memory exceptions. Thus not propagating there logical state corruption any further than the current micro block of code.
Should be contained.

After an out of memory event is triggered, any further memory writes should be prevented or result in no actual change to memory, if a program attempts them, in case someone forgets to check for failed memory allocation request, thus by design the hosting/encapsulating system is protecting the data integrity at an os/software design level then, Free form human error.

What do you see instead?

Continuous to execute code on corrupt or logical state corrupted memory and propagating data corruption onto other further down steam

Additional information

Contact AWS for the support case with the lambda team.

[trivikr]

Node.js 16 reached End-of-Life (EOL) on September 11, 2023

Is this reproducible in supported versions of node, i.e. 22, 24 or 26?