Implement PEP 492: async/await support by mikasoukhov · Pull Request #2004 · IronLanguages/ironpython3

mikasoukhov · 2026-02-24T08:54:32Z

Tokenizer/Parser: async def, async for, async with, await keywords
AST nodes: AwaitExpression, AsyncForStatement, AsyncWithStatement
Runtime: PythonCoroutine, CoroutineWrapper types
Code generation: coroutines reuse generator state machine via yield from desugaring (await → yield from expr.await())
Fix GeneratorRewriter VisitExtension to reduce one level at a time, preventing "must be reducible node" with DebugInfoRemovalExpression

Verified against CPython 3.14: 20/20 comparison tests identical.

- Tokenizer/Parser: async def, async for, async with, await keywords - AST nodes: AwaitExpression, AsyncForStatement, AsyncWithStatement - Runtime: PythonCoroutine, CoroutineWrapper types - Code generation: coroutines reuse generator state machine via yield from desugaring (await → yield from expr.__await__()) - Fix GeneratorRewriter VisitExtension to reduce one level at a time, preventing "must be reducible node" with DebugInfoRemovalExpression Verified against CPython 3.14: 20/20 comparison tests identical.

Copilot

Pull request overview

This PR adds core language, AST, runtime, and codegen support for Python’s PEP 492 async/await features in IronPython, including coroutine objects and async control-flow constructs.

Changes:

Extend tokenizer/parser and AST to support await, async for, and async with.
Introduce coroutine runtime types (PythonCoroutine, CoroutineWrapper) and propagate coroutine flags through FunctionCode/codegen.
Update generator rewriting to support coroutine wrapping and fix extension-node reduction behavior.

Reviewed changes

Copilot reviewed 16 out of 16 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
src/core/IronPython/Runtime/Operations/PythonOps.cs	Adds coroutine construction helpers and passes coroutine intent into generator transformation.
src/core/IronPython/Runtime/FunctionCode.cs	Treats coroutine functions like generators for rewriting, threading an `isCoroutine` flag.
src/core/IronPython/Runtime/FunctionAttributes.cs	Introduces `FunctionAttributes.Coroutine`.
src/core/IronPython/Runtime/Coroutine.cs	Adds `PythonCoroutine` and `CoroutineWrapper` runtime types.
src/core/IronPython/Modules/_ast.cs	Adds `_ast` nodes/conversion for `Await`, `AsyncFor`, `AsyncWith`.
src/core/IronPython/Compiler/Tokenizer.cs	Tokenizes `await` as a keyword.
src/core/IronPython/Compiler/TokenKind.Generated.cs	Adds `KeywordAwait` and updates keyword range.
src/core/IronPython/Compiler/Parser.cs	Parses `await`, `async for`, `async with`.
src/core/IronPython/Compiler/GeneratorRewriter.cs	Wraps coroutine generators and adjusts extension-node reduction strategy.
src/core/IronPython/Compiler/Ast/PythonWalker.Generated.cs	Adds walker hooks for new AST nodes.
src/core/IronPython/Compiler/Ast/PythonNameBinder.cs	Binds new nodes into scope/binding passes.
src/core/IronPython/Compiler/Ast/FunctionDefinition.cs	Marks async functions as generator-like and sets coroutine flags for codegen.
src/core/IronPython/Compiler/Ast/AwaitExpression.cs	Implements `await` via `yield from expr.__await__()` desugaring.
src/core/IronPython/Compiler/Ast/AsyncWithStatement.cs	Implements `async with` via desugaring.
src/core/IronPython/Compiler/Ast/AsyncForStatement.cs	Implements `async for` via desugaring.
src/core/IronPython/Compiler/Ast/AstMethods.cs	Adds a cached `MakeCoroutine` MethodInfo.

Comments suppressed due to low confidence (6)

src/core/IronPython/Compiler/Ast/AstMethods.cs:82

AstMethods.MakeCoroutine is added but appears unused (no references found in the repo). If codegen no longer calls it, consider removing it to avoid dead API surface; otherwise, update the coroutine codegen to use this cached MethodInfo instead of repeated reflection lookups.

        public static readonly MethodInfo GeneratorCheckThrowableAndReturnSendValue = GetMethod((Func<object, object>)PythonOps.GeneratorCheckThrowableAndReturnSendValue);
        public static readonly MethodInfo MakeCoroutine = GetMethod((Func<PythonFunction, MutableTuple, object, PythonCoroutine>)PythonOps.MakeCoroutine);

src/core/IronPython/Runtime/Operations/PythonOps.cs:3205

MakeCoroutineWrapper currently returns a PythonCoroutine, not a CoroutineWrapper, which makes the name misleading (and MakeCoroutine appears unused). Consider renaming to something like MakeCoroutineFromGenerator (and returning PythonCoroutine directly instead of object) or wiring codegen to use MakeCoroutine and dropping the extra wrapper method to reduce confusion.

        public static PythonCoroutine MakeCoroutine(PythonFunction function, MutableTuple data, object generatorCode) {
            return new PythonCoroutine(MakeGenerator(function, data, generatorCode));
        }

        public static object MakeCoroutineWrapper(PythonGenerator generator) {
            return new PythonCoroutine(generator);
        }

src/core/IronPython/Compiler/Parser.cs:1996

There are no existing test cases in tests/ exercising async def / await / async for / async with (searching the suite finds no async def). Given the amount of new parsing + desugaring behavior introduced here, please add targeted tests (e.g., precedence like await a ** b, unary like await -x, async with exception suppression, and async for finalization) to prevent regressions.

        // power: ['await'] atom trailer* ['**' factor]
        private Expression ParsePower() {
            if (MaybeEat(TokenKind.KeywordAwait)) {
                return ParseAwaitExpression();
            }
            Expression ret = ParseAtom();
            ret = AddTrailers(ret);
            if (MaybeEat(TokenKind.Power)) {
                var start = ret.StartIndex;
                ret = new BinaryExpression(PythonOperator.Power, ret, ParseFactor());
                ret.SetLoc(_globalParent, start, GetEnd());
            }
            return ret;
        }

        // await_expr: 'await' unary_expr (essentially power level)
        private Expression ParseAwaitExpression() {
            FunctionDefinition current = CurrentFunction;
            if (current == null || !current.IsAsync) {
                ReportSyntaxError("'await' outside async function");
            }

            if (current != null) {
                current.IsGenerator = true;
                current.GeneratorStop = GeneratorStop;
            }

            var start = GetStart();

            // Parse the awaitable expression at the unary level
            Expression expr = ParsePower();

            var ret = new AwaitExpression(expr);
            ret.SetLoc(_globalParent, start, GetEnd());
            return ret;
        }

src/core/IronPython/Compiler/Parser.cs:1500

ParseAsyncForStmt declares var start = GetStart(); but never uses it. This will generate an unused-local warning and can be removed (or used for SetLoc if that was the intent).

            Eat(TokenKind.KeywordFor);
            var start = GetStart();

src/core/IronPython/Compiler/Ast/AsyncWithStatement.cs:110

Reduce() assumes _desugared was built during Walk() and will null-deref if Reduce() is called before a walk pass. Either build _desugared lazily in Reduce() as well (when Parent/IndexSpan are available) or throw a clearer exception explaining the required initialization order.

        public override MSAst.Expression Reduce() {
            return _desugared!.Reduce();
        }

src/core/IronPython/Compiler/Ast/AsyncForStatement.cs:131

Reduce() assumes _desugared was built during Walk() and will null-deref if Reduce() is called before Walk(). Consider building _desugared lazily in Reduce() too (or failing with a clearer message) to avoid fragile initialization ordering.

        public override MSAst.Expression Reduce() {
            return _desugared!.Reduce();
        }

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-02-24T09:04:59Z

src/core/IronPython/Modules/_ast.cs

+
+            internal override Statement Revert() {
+                Statement statement = RevertStmts(this.body);
+                foreach (withitem item in items) {


AsyncWith.Revert() nests multiple context managers in the wrong order: iterating items forward makes the last item become the outermost AsyncWithStatement, reversing evaluation/exit order compared to CPython. Iterate items in reverse (like ParseWithStmt) so the first item remains outermost.

Suggested change

foreach (withitem item in items) {

for (int i = items.Count - 1; i >= 0; i--) {

withitem item = (withitem)items[i];

Copilot · 2026-02-24T09:04:59Z

src/core/IronPython/Compiler/Parser.cs

+            var withItem = ParseWithItem();
+            var header = GetEnd();
+            Statement body = ParseSuite();
+            AsyncWithStatement ret = new AsyncWithStatement(withItem.ContextManager, withItem.Variable, body);
+            ret.HeaderIndex = header;
+            ret.SetLoc(_globalParent, asyncStart, GetEnd());
+            return ret;


async with currently parses only a single with_item and ignores the comma-separated form (async with a as x, b as y:). with_stmt already supports multiple items by looping on commas and nesting; async with should mirror that logic (parse all items and create nested AsyncWithStatement nodes in reverse order).

Suggested change

var withItem = ParseWithItem();

var header = GetEnd();

Statement body = ParseSuite();

AsyncWithStatement ret = new AsyncWithStatement(withItem.ContextManager, withItem.Variable, body);

ret.HeaderIndex = header;

ret.SetLoc(_globalParent, asyncStart, GetEnd());

return ret;

var withItems = new List<WithItem>();

withItems.Add(ParseWithItem());

while (MaybeEat(TokenKind.Comma)) {

withItems.Add(ParseWithItem());

}

var header = GetEnd();

Statement body = ParseSuite();

// Build nested AsyncWithStatement nodes in reverse order of withItems

Statement currentBody = body;

AsyncWithStatement outermost = null;

for (int i = withItems.Count - 1; i >= 0; i--) {

var withItem = withItems[i];

var asyncWith = new AsyncWithStatement(withItem.ContextManager, withItem.Variable, currentBody);

if (i == 0) {

// Preserve existing header and location behavior on the outermost node

asyncWith.HeaderIndex = header;

asyncWith.SetLoc(_globalParent, asyncStart, GetEnd());

outermost = asyncWith;

}

currentBody = asyncWith;

}

return outermost;

Copilot · 2026-02-24T09:04:59Z

src/core/IronPython/Compiler/Ast/AsyncWithStatement.cs

+            // await mgr.__aexit__(None, None, None)
+            var aexitAttr = new MemberExpression(MakeName("__asyncwith_mgr"), "__aexit__") { Parent = parent };
+            aexitAttr.IndexSpan = span;
+            var none1 = new ConstantExpression(null) { Parent = parent }; none1.IndexSpan = span;
+            var none2 = new ConstantExpression(null) { Parent = parent }; none2.IndexSpan = span;
+            var none3 = new ConstantExpression(null) { Parent = parent }; none3.IndexSpan = span;
+            var aexitCallNormal = new CallExpression(aexitAttr,
+                new Expression[] { none1, none2, none3 }, null) { Parent = parent };
+            aexitCallNormal.IndexSpan = span;
+            var awaitExitNormal = new AwaitExpression(aexitCallNormal);
+
+            // try/finally: await __aexit__ on normal exit
+            var finallyExprStmt = new ExpressionStatement(awaitExitNormal) { Parent = parent };
+            finallyExprStmt.IndexSpan = span;
+            var tryFinally = new TryStatement(bodyStmt, null, null, finallyExprStmt) { Parent = parent };
+            tryFinally.IndexSpan = span;


The current desugaring always calls await mgr.__aexit__(None, None, None) in a finally block, which is not equivalent to CPython semantics for async with: on exceptions it must pass exception details to __aexit__ and must respect its truthy return value to decide whether to suppress or re-raise. This needs a try/except/finally structure analogous to WithStatement.Reduce(), but using await for __aenter__/__aexit__.

Copilot · 2026-02-24T09:05:00Z

src/core/IronPython/Compiler/Parser.cs

+            // Parse the awaitable expression at the unary level
+            Expression expr = ParsePower();


ParseAwaitExpression parses the operand via ParsePower(), which prevents valid syntax like await -x (operand starts with a unary operator) and also gives await the wrong precedence relative to ** (it will parse await a ** b as (await a) ** b instead of await (a ** b)). Parse the operand at the unary/factor level (e.g., ParseFactor() / unary_expr) rather than ParsePower() so unary ops and exponentiation bind correctly.

Suggested change

// Parse the awaitable expression at the unary level

Expression expr = ParsePower();

// Parse the awaitable expression at the unary/factor level so unary ops and '**' bind correctly

Expression expr = ParseFactor();

Copilot · 2026-02-24T09:05:00Z

src/core/IronPython/Compiler/Ast/AsyncForStatement.cs

+            // while __running: try/except/else
+            var whileStmt = new WhileStatement(MakeName(runningName), tryExcept, Else);
+            whileStmt.SetLoc(GlobalParent, span.Start, span.End, span.End);
+            whileStmt.Parent = parent;
+
+            var suite = WithSpan(new SuiteStatement(new Statement[] { assignIter, assignRunning, whileStmt }) { Parent = parent });
+            return suite;


The async for desugaring doesn't perform iterator finalization on early loop exit (e.g., break, return, or exception). In CPython, async for ensures aclose() is awaited for async generators / async iterators that provide it, to avoid leaking resources. Consider wrapping the loop in a try/finally that conditionally awaits __asyncfor_iter?.aclose() when the loop is exited prematurely.

mikasoukhov · 2026-02-24T09:08:54Z

@mikasoukhov please read the following Contributor License Agreement(CLA). If you agree with the CLA, please reply with the following information.
@dotnet-policy-service agree [company="{your company}"]
Options:

(default - no company specified) I have sole ownership of intellectual property rights to my Submissions and I am not making Submissions in the course of work for my employer.
@dotnet-policy-service agree
(when company given) I am making Submissions in the course of work for my employer (or my employer has intellectual property rights in my Submissions by contract or applicable law). I have permission from my employer to make Submissions and enter into this Agreement on behalf of my employer. By signing below, the defined term “You” includes me and my employer.
@dotnet-policy-service agree company="Microsoft"
Contributor License Agreement

@dotnet-policy-service agree [company="StockSharp"]

23 tests covering async def, await, async with, async for, coroutine properties, __await__ protocol, custom awaitables, break/continue/else, nested loops, and combined patterns.

…numerable, CancelledError - Add TaskAwaitable/ValueTaskAwaitable wrappers enabling `await` on Task, Task<T>, ValueTask and ValueTask<T> from Python async code - Add AsyncEnumerableWrapper enabling `async for` over IAsyncEnumerable<T> - Map OperationCanceledException to new CancelledError Python exception - Add __await__, __aiter__, __anext__ resolvers in PythonTypeInfo - Add bridge methods in InstanceOps for the resolver pattern - ValueTask/IAsyncEnumerable support gated behind #if NET (requires .NET Core) - Handle Task<VoidTaskResult> (internal type arg) by falling back to non-generic TaskAwaitable via IsVisible check

- Add 'await' keyword to generate_ops.py kwlist - Add CancelledError factory-only exception to generate_exceptions.py - Regenerate TokenKind, Tokenizer, PythonWalker, PythonNameBinder - Fix CancelledError placement in ToPythonHelper to match generator order

Add CancelledError to exception_hierarchy.txt so test_pep352 test_inheritance accounts for the new builtin exception. Isolate test_async in a separate process to prevent it from loading IronPythonTest assembly which causes duplicate SpecialName GetBoundMember on XmlElement in test_attrinjector.

slozier

Thanks this is great! It will probably take me a while to go over this properly but it definitely seems like a good start. So far I've taken a few notes but nothing blocking that can't be fixed later - will write them up when I have a bit more time.

It seems like CLA service is not happy. Maybe because of the brackets ([ ]) around company in the reply?

slozier · 2026-02-25T03:55:54Z

tests/IronPython.Tests/Cases/CommonCases.cs

                return -1;
            } finally {
                m?.ReleaseMutex();
+                CleanupTempFiles(testcase);


Some test are running concurrently and this temp file cleanup breaks them.

The runtime type of async Task<T> is often a subclass like AsyncStateMachineBox<TResult, TStateMachine>, not Task<T> itself. Walk up the BaseType chain to find Task<T> so that await on real async .NET operations (e.g. HttpClient.GetStringAsync) correctly returns the result instead of None.

BCSharp · 2026-02-25T04:58:21Z

Thanks for the submission. async/await is one of the major blocking issues to upgrade IronPython's StdLib to newer versions of Python.

I think the CLA bot requires the reply on its own on one line, without any additional text like quoting a reply and no brackets around the company name.

I too will need more time to digest the submission, right now my main question is how does this async/await implementation interoperate with C# async/await? Can I await .NET awaitable objects from IronPython and can C# methods await IronPython async methods?

Instead of blocking the thread with GetAwaiter().GetResult(), TaskAwaitable.__next__ now yields the Task back to the runner when it's not yet completed. The runner can then wait on the Task and resume the coroutine, enabling true concurrency between coroutines.

mikasoukhov · 2026-02-27T09:30:43Z

@mikasoukhov please read the following Contributor License Agreement(CLA). If you agree with the CLA, please reply with the following information.
@dotnet-policy-service agree [company="{your company}"]
Options:

(default - no company specified) I have sole ownership of intellectual property rights to my Submissions and I am not making Submissions in the course of work for my employer.
@dotnet-policy-service agree
(when company given) I am making Submissions in the course of work for my employer (or my employer has intellectual property rights in my Submissions by contract or applicable law). I have permission from my employer to make Submissions and enter into this Agreement on behalf of my employer. By signing below, the defined term “You” includes me and my employer.
@dotnet-policy-service agree company="Microsoft"
Contributor License Agreement

@dotnet-policy-service agree company="StockSharp"

Allows C# code to directly await IronPython coroutines: object result = await coroutine;

mikasoukhov · 2026-02-27T10:50:03Z

I too will need more time to digest the submission, right now my main question is how does this async/await implementation interoperate with C# async/await? Can I await .NET awaitable objects from IronPython and can C# methods await IronPython async methods?

IronPython -> .NET: Yes. await Task, Task, ValueTask, ValueTask and async for over IAsyncEnumerable all work. OperationCanceledException maps to CancelledError. Awaiting is non-blocking - the Task is yielded to the runner instead of blocking with GetResult().

C# -> IronPython: Yes. PythonCoroutine implements GetAwaiter(), so C# can directly await it:

var coro = (PythonCoroutine)engine.Execute("foo()");
object result = await coro;

Copilot AI review requested due to automatic review settings February 24, 2026 08:54

Copilot started reviewing on behalf of mikasoukhov February 24, 2026 08:54 View session

Copilot AI reviewed Feb 24, 2026

View reviewed changes

mikasoukhov added 5 commits February 24, 2026 12:15

Add tests for PEP 492 async/await

a82d993

23 tests covering async def, await, async with, async for, coroutine properties, __await__ protocol, custom awaitables, break/continue/else, nested loops, and combined patterns.

Clean up @test_*_tmp artifacts after each test run

826dfeb

slozier reviewed Feb 25, 2026

View reviewed changes

Add PythonCoroutine.AsTask() and GetAwaiter() for C# async interop

dfcace5

Allows C# code to directly await IronPython coroutines: object result = await coroutine;

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement PEP 492: async/await support#2004

Implement PEP 492: async/await support#2004
mikasoukhov wants to merge 9 commits intoIronLanguages:mainfrom
StockSharp:main

mikasoukhov commented Feb 24, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Feb 24, 2026

Uh oh!

Copilot AI Feb 24, 2026

Uh oh!

Copilot AI Feb 24, 2026

Uh oh!

Copilot AI Feb 24, 2026

Uh oh!

Copilot AI Feb 24, 2026

Uh oh!

mikasoukhov commented Feb 24, 2026

Uh oh!

slozier left a comment

Uh oh!

slozier Feb 25, 2026

Uh oh!

BCSharp commented Feb 25, 2026

Uh oh!

mikasoukhov commented Feb 27, 2026

Uh oh!

mikasoukhov commented Feb 27, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

	foreach (withitem item in items) {
	for (int i = items.Count - 1; i >= 0; i--) {
	withitem item = (withitem)items[i];

-            var withItem = ParseWithItem();
-            var header = GetEnd();
-            Statement body = ParseSuite();
-            AsyncWithStatement ret = new AsyncWithStatement(withItem.ContextManager, withItem.Variable, body);
-            ret.HeaderIndex = header;
-            ret.SetLoc(_globalParent, asyncStart, GetEnd());
-            return ret;
+            var withItems = new List<WithItem>();
+            withItems.Add(ParseWithItem());
+            while (MaybeEat(TokenKind.Comma)) {
+                withItems.Add(ParseWithItem());
+            }
+            var header = GetEnd();
+            Statement body = ParseSuite();
+            // Build nested AsyncWithStatement nodes in reverse order of withItems
+            Statement currentBody = body;
+            AsyncWithStatement outermost = null;
+            for (int i = withItems.Count - 1; i >= 0; i--) {
+                var withItem = withItems[i];
+                var asyncWith = new AsyncWithStatement(withItem.ContextManager, withItem.Variable, currentBody);
+                if (i == 0) {
+                    // Preserve existing header and location behavior on the outermost node
+                    asyncWith.HeaderIndex = header;
+                    asyncWith.SetLoc(_globalParent, asyncStart, GetEnd());
+                    outermost = asyncWith;
+                }
+                currentBody = asyncWith;
+            }
+            return outermost;

		// Parse the awaitable expression at the unary level
		Expression expr = ParsePower();

Conversation

mikasoukhov commented Feb 24, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Feb 24, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 24, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 24, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 24, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 24, 2026

Choose a reason for hiding this comment

Uh oh!

mikasoukhov commented Feb 24, 2026

Uh oh!

slozier left a comment

Choose a reason for hiding this comment

Uh oh!

slozier Feb 25, 2026

Choose a reason for hiding this comment

Uh oh!

BCSharp commented Feb 25, 2026

Uh oh!

mikasoukhov commented Feb 27, 2026

Uh oh!

mikasoukhov commented Feb 27, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants