feat: OpenAI realtime api

[vladimir-a-sap]

Context

AI/ai-sdk-java-backlog#386.

Adds OpenAI realtime API support

Feature scope:

• Implemented destination selection, connection and authorization logic
• Added heartbeat functionality to correctly manage conversation connections without unintended TCP resets
• Implemented SDK Realtime API operations
  • Text to Speech
  • Speech to Speech
  • Speech to Text (requires additional support on the core/orchestration end)
• Added sample app interactive examples
  • Text to Speech
  • Speech to Speech
  • Speech to Text (a blocker to be resolved on the BE side)
• Added smoke test
  • Text to Speech
  • Speech to Speech
  • Speech to Text (requires additional support on the core/orchestration end)

Definition of Done

• Functionality scope stated & covered
• Tests cover the scope above
• Error handling created / updated & covered by the tests above
• Aligned changes with the JavaScript SDK
• Documentation updated
• Release notes updated

[CharlesDuboisSAP]

What do we do about the previous realtime PR? Is it still valid? Why is it abandonned?

• #870

[vladimir-a-sap]

What do we do about the previous realtime PR? Is it still valid? Why is it abandonned?

• feat: PoC OpenAI Realtime API #870

My current understanding is that the mentioned PR cannot be completed to a SDK realtime solution. Here is the context and outcomes of my research why I think so:

There are three ways to implement realtime API with OpenAPI:

• WebRTC
• Websockets
• SIP (telephony)

The previous PR implements SIP - it is a way for client and server to communicate about a connection in order to pass data, but not to establish connection or pass data in fact. The data layer should be implemented separately from the SIP protocol.

OpenAPI SIP protocol by design works with client-side web hooks as a mean to pass data and accept incoming connections (calls). Outgoing calls (client initiated) are not seem to be possible at all using solely SIP API.

We cannot expect SDK environment to have public http endpoints accessible from the internet to expose hooks. AI Core does not implement hook functionality either AFAIK.

Given that WebRTC is difficult to support on the BE and, AFAIK, our backend only supports Websockets at the moment, Websockets seem to be the only option we have.

[Jonas-Isr]

The design is very cool! Thanks a lot for the detailed investigation and work :) I am reviewing this as if it would be the beta release already because I think without too much effort we turn this into a releasable state.

The two points below (and many of my comments) concern code conventions in the team.

General points:

• Please add @Beta to all newly introduced public API. This way we have a grace period to change the API if we overlooked something in the beginning. We periodically re-evaluate and remove these annotations later on.
• Please don’t put hand written classes into any of the model/ packages. These are only for generated code. If possible, put the SpeechOutputParam interface and the 2 classes implementing it into the open package.

[vladimir-a-sap]

The design is very cool! Thanks a lot for the detailed investigation and work :) I am reviewing this as if it would be the beta release already because I think without too much effort we turn this into a releasable state.

The two points below (and many of my comments) concern code conventions in the team.

General points:

• Please add @Beta to all newly introduced public API. This way we have a grace period to change the API if we overlooked something in the beginning. We periodically re-evaluate and remove these annotations later on.
• Please don’t put hand written classes into any of the model/ packages. These are only for generated code. If possible, put the SpeechOutputParam interface and the 2 classes implementing it into the open package.

addressed both these points in addition to your comments in the PR

[vladimir-a-sap]

What do we do about the previous realtime PR? Is it still valid? Why is it abandonned?

• feat: PoC OpenAI Realtime API #870

Discussed with @n-o-u-r-h-a-n and @rpanackal.

The initial implementation was SIP based and contains only http communication endpoints (and no audio processing implementation nor a means how this processing can be implemented). SIP-based approach requires a public http hook endpoint, which we cannot expect our SDK users to implement.

Hence we made the decision to close the previous PR, that was done after discussion. We may reopen and update it later if we decide to support SIP.