Document use of disableHostPrefixInjection for 2.18.x

[henricook]

2.18.x introduced virtual bucket addressing as standard

i.e.
Before 2.18.x: I connected to bucket foo on mys3stub.mynetwork.local
But now: the SDK changes my configured hostname on the fly and tries to connect to foo.mys3stub.mynetwork.local.

I don't want to to have to setup DNS for every bucket name, I'd rather have the SDK connect the old way, without it modifying the hostname I configured. It's unclear how to achieve this.

I can't find any documentation on how to use disableHostPrefixInjection which I think from reading the code will stop this behaviour happening. Can it be documented? (and can anyone tell me the recommended way to use it?)

UPDATE: On further testing this doesn't seem to be the right option - any documentation you can provide on how to go back to having the bucket name in the path would be great

UPDATE 2: I've found something that works - is it right?

.forcePathStyle(true) - code example is Scala

  implicit val clientAsync: S3AsyncClient =
    S3AsyncClient
      .builder()
      .credentialsProvider(credentialsProvider)
      .region(region)
      .forcePathStyle(true)
      .endpointOverride(s3Uri)
      .build()

Links

NA (not sure where it's best to put it)

[yasminetalby]

Hello @henricook,

Thank you very much for your submission.

I apologize for any inconvenience experienced while upgrading to v2.18.x

v2.18.0 introduces the following change:

When including an endpoint override, the SDK will use virtual bucket addressing as long as the bucket is a valid DNS label. This is different from the previous behavior where this only happens if the endpoint override begins with 's3'. see changelog

As a side-note, if your bucket is not a valid DNS label , the behavior would be to not use virtual bucket addressing.

Using .forcePathStyle(true) forces the client to use path-style addressing for buckets (see) and would be the proper workaround for your use case.

For reference here is the AWS S3 User Guide documentation on Virtual hosting of buckets.

Let me know if you have any further questions or if you would have ideas of potential documentation improvements or documentation request. Your feedback is extremely valuable to our team!

Sincerely,

Yasmine

[henricook]

Thanks @yasminetalby for confirming that's the best approach.

Picking up on your point As a side-note, if your bucket is not a valid DNS label , the behavior would be to not use virtual bucket addressing.

I was thrown by behaviour where my s3 host was 'localhost' because it seems by default *.localhost resolves on machines. So my bucket is called foo and host is localhost and foo.localhost seems to resolve (despite me not explicitly configuring it) yet my s3 host/stub wasn't configured to support it. Maybe a special case to support this is called for (but obviously that's not a documentation task).

w.r.t. this issue all I could recommend is adding a specific example in a walkthrough - but I acknowledge this method is contained in the docs despite taking me quite a while googling to find it.

Thanks again for your speedy response

[yasminetalby]

Hello @henricook,
Thank you very much for your response and for providing further detailed on your use case.
Your feedback is very helpful for the team! I will look into potential documentation improvement and have shared your current use case and experience.

Thank you again for your feedback!

Sincerely,

Yasmine

[Ivan-Fn]

I suggest that SDK error message would give a hint on what is incorrect.
Current error message diverts from root cause.
software.amazon.awssdk.core.exception.SdkClientException: Received an UnknownHostException when attempting to interact with a service. See cause for the exact endpoint that is failing to resolve. If this is happening on an endpoint that previously worked, there may be a network connectivity issue or your DNS cache could be storing endpoints for too long.

In my usecase I had S3 on localstack for testing.

[yasminetalby]

Hello @LuckySpb ,

Thank you very much for your feedback and for providing your use case.
I have added a backlog item to provide better documentation support for the new SDK behavior. Your comment is very helpful to us!

I will update this ticket with all changes made.

Sincerely,

Yasmine

[MugheesQasim]

Different Name for the method:

I am getting the same exception and I am using AmazonS3ClientBuilder for building the client. After searching the documentation, I could not find .forcePathStyle(true). AmazonS3ClientBuilder inherits the method enablePathStyleAccess() from AmazonS3Builder. I used this method and it solved the issue. For anyone who is still confused after reading the previous comments, just use this method to enable the path style access for the client.

Tech Stack Used:

JDK: 18
aws-java-sdk-s3: 1.11.318
localstack: 1.4.0