Skip to content

fix(dataproc): add mising fields in yarnApplication proto#16868

Closed
blcksrx wants to merge 1 commit intogoogleapis:mainfrom
blcksrx:blcksrx/dataproc
Closed

fix(dataproc): add mising fields in yarnApplication proto#16868
blcksrx wants to merge 1 commit intogoogleapis:mainfrom
blcksrx:blcksrx/dataproc

Conversation

@blcksrx
Copy link
Copy Markdown

@blcksrx blcksrx commented Apr 29, 2026

Two fields in YarnApplicationProto is missing:

  1. vcore_sconds
  2. memory_mb_seconds

This Pr, is a fix for missing these fields.

Reference:
https://docs.cloud.google.com/dataproc/docs/reference/rpc/google.cloud.dataproc.v1#google.cloud.dataproc.v1.YarnApplication

Thank you for opening a Pull Request! Before submitting your PR, there are a few things you can do to make sure it goes smoothly:

  • Make sure to open an issue as a bug/issue before writing your code! That way we can discuss the change, evaluate designs, and agree on the general idea
  • Ensure the tests and linter pass
  • Code coverage does not decrease (if any source code was changed)
  • Appropriate docs were updated (if necessary)

Fixes #<issue_number_goes_here> 🦕

@blcksrx blcksrx requested a review from a team as a code owner April 29, 2026 13:05
Copy link
Copy Markdown
Contributor

@gemini-code-assist gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request adds vcore_seconds and memory_mb_seconds fields to the YarnApplication message in the Dataproc V1 library. Feedback was provided to remove the optional=True parameter from these fields to ensure consistency with the backend proto definition and existing fields. Additionally, the docstrings should be updated to remove the 'Optional.' prefix and use more descriptive unit names like 'megabyte-seconds' for clarity.

Comment on lines +1201 to +1210
vcore_seconds: int = proto.Field(
proto.INT64,
number=5,
optional=True,
)
memory_mb_seconds: int = proto.Field(
proto.INT64,
number=6,
optional=True,
)
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

The optional=True parameter should be removed. According to the official Dataproc V1 YarnApplication proto definition, these fields are standard int64 fields and do not use the optional keyword. Adding optional=True in proto-plus enables explicit presence tracking, which is inconsistent with the existing fields in this message (like field 4) and the backend proto definition.

    vcore_seconds: int = proto.Field(
        proto.INT64,
        number=5,
    )
    memory_mb_seconds: int = proto.Field(
        proto.INT64,
        number=6,
    )

Comment on lines +1143 to +1146
vcore_seconds (int):
Optional. The cumulative CPU time consumed by the application for a job, measured in vcore-seconds.
memory_mb_seconds (int):
Optional. The cumulative memory usage of the application for a job, measured in mb-seconds.
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The docstring for these fields should be updated for consistency and accuracy. Specifically:

  1. The "Optional." prefix is not standard for message fields in this library and should be removed, as these fields are not marked as optional in the official Dataproc proto.
  2. The unit for memory_mb_seconds should be "megabyte-seconds" to match the official documentation and provide better clarity than "mb-seconds".
  3. Note that the progress field (field 4) appears to be missing from the docstring between tracking_url and vcore_seconds.
Suggested change
vcore_seconds (int):
Optional. The cumulative CPU time consumed by the application for a job, measured in vcore-seconds.
memory_mb_seconds (int):
Optional. The cumulative memory usage of the application for a job, measured in mb-seconds.
vcore_seconds (int):
The cumulative CPU time consumed by the application for a job, measured in vcore-seconds.
memory_mb_seconds (int):
The cumulative memory usage of the application for a job, measured in megabyte-seconds.

@blcksrx blcksrx changed the title fix YarnApplication proto fix(dataproc): add mising fields in yarnApplication proto Apr 29, 2026
@parthea parthea self-assigned this Apr 29, 2026
@parthea
Copy link
Copy Markdown
Contributor

parthea commented Apr 29, 2026

Regrettably, this PR can't be merged. The code is automatically generated from the *.proto files which live in https://github.com/googleapis/googleapis/blob/master/google/cloud/dataproc/v1/. I've reached out to the team that maintains these files via an internal issue tracker. (Googlers see b/507808022). I'm going to close this PR but please feel free to open an issue if you'd like to request a status update from the team.

@parthea parthea closed this Apr 29, 2026
@blcksrx
Copy link
Copy Markdown
Author

blcksrx commented Apr 29, 2026

@parthea Thank you very much for your answer, is it possible to contribute to the mentioned project to speed up the process deilivery?

@parthea
Copy link
Copy Markdown
Contributor

parthea commented Apr 29, 2026

@blcksrx, Regrettably, this change needs to be made in an internal code base by the team that maintains this product. You can reach them directly using the email address here. If you've filed a Google Cloud Support issue for this, your support contact can also reach the team via internal issue b/507808022.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants