Skip to content

kedro-datasets: update dependency range to support PySpark 4.x #1221

@michal-mmm

Description

@michal-mmm

Description

The dependency constraint currently blocks PySpark 4.0.0 and above:

spark-base = ["pyspark>=2.2, <4.0"]

However, PySpark 4.0.0 was released earlier this year. The current upper bound prevents users from using the latest version of PySpark.

Context

Upgrading to PySpark 4.0.0 would allow users to take advantage of new features, performance improvements, and bug fixes introduced in Spark 4.

Possible Implementation

Relax the dependency constraint to:

spark-base = ["pyspark>=2.2, <5.0"]

Metadata

Metadata

Labels

CommunityIssue/PR opened by the open-source community

Type

No type

Projects

Status

Done

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions