You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@polaris.apache.org by Dmitri Bourlatchkov <di...@apache.org> on 2025/06/25 04:23:23 UTC
[DISCUSS] Adding endpoint to AwsStorageConfigInfo
Hi All,
I propose to add an `endpoint` optional parameter to AwsStorageConfigInfo
in PR [1913].
The main idea is to support non-AWS S3 implementations for [1530].
Existing classes related to supporting S3 Polaris are coded to the AWS SDK,
which supports setting STS endpoints. Therefore, it seems natural to allow
users to optionally define a specific endpoint for their catalogs.
This change is backward-compatible with existing clients and deployed
catalogs.
When an endpoint is defined it will be used for both STS and S3 requests
inside Polaris and will be used to populate the "s3.endpoint" properties in
REST Catalog clients.
Thoughts?
[1530] https://github.com/apache/polaris/issues/1530
[1913] https://github.com/apache/polaris/pull/1913
Thanks,
Dmitri.
Re: [DISCUSS] Adding endpoint to AwsStorageConfigInfo
Posted by Dmitri Bourlatchkov <di...@apache.org>.
Heads up: PR [1913] got approved. If there are no more comments, I'm going
to merge tomorrow (EDT).
[1913] https://github.com/apache/polaris/pull/1913
Cheers,
Dmitri.
On Fri, Jun 27, 2025 at 8:47 PM Dmitri Bourlatchkov <di...@apache.org>
wrote:
> I added a separate STS endpoint property to
> https://github.com/apache/polaris/pull/1913
>
> Cheers,
> Dmitri.
>
> On Wed, Jun 25, 2025 at 7:59 PM Dmitri Bourlatchkov <di...@apache.org>
> wrote:
>
>> Thanks for the feedback, Adnan!
>>
>> I was initially thinking of deferring separating STS and S3 endpoints to
>> a later PR, but given interest, I'll add it to the current PR to reduce the
>> number of API changes over time.
>>
>> Cheers,
>> Dmitri.
>>
>> On Wed, Jun 25, 2025 at 1:28 PM Adnan Hemani
>> <ad...@snowflake.com.invalid> wrote:
>>
>>> +1 on overall idea
>>>
>>> Comment however on keeping the same endpoint for STS and S3 - there may
>>> be
>>> a customer use case (I have seen this before for similar use cases) where
>>> the S3 bucket is not in the same region as the STS endpoint that has been
>>> exposed to the Polaris server. In that case, you really need two separate
>>> endpoints (example below). But I don't think this necessarily blocks this
>>> idea from being merged today and then enriched later.
>>>
>>> Example:
>>> * Polaris is deployed in us-west-1 in a EC2 instance that is within a
>>> private subnet, which is given access only to STS' us-west-1 regional
>>> endpoint through AWS configurations.
>>> * User has S3 data in both us-west-1 and us-east-1.
>>> * Under this proposal, user can now access data in us-west-1, but when
>>> trying to access data in us-east-1, STS will not respond due to no
>>> network
>>> routing tables for the us-east-1 regional endpoint.
>>> * Important thing to remember is that STS token generated at any regional
>>> endpoint can be used at any other regional endpoint. So, in theory, user
>>> could still use us-west-1 STS to generate credentials to use for S3 data
>>> in
>>> us-east-1 and it would've been ok - but this proposal does not allow for
>>> this.
>>>
>>> Best,
>>> Adnan Hemani
>>>
>>> On Tue, Jun 24, 2025 at 9:23 PM Dmitri Bourlatchkov <di...@apache.org>
>>> wrote:
>>>
>>> > Hi All,
>>> >
>>> > I propose to add an `endpoint` optional parameter to
>>> AwsStorageConfigInfo
>>> > in PR [1913].
>>> >
>>> > The main idea is to support non-AWS S3 implementations for [1530].
>>> >
>>> > Existing classes related to supporting S3 Polaris are coded to the AWS
>>> SDK,
>>> > which supports setting STS endpoints. Therefore, it seems natural to
>>> allow
>>> > users to optionally define a specific endpoint for their catalogs.
>>> >
>>> > This change is backward-compatible with existing clients and deployed
>>> > catalogs.
>>> >
>>> > When an endpoint is defined it will be used for both STS and S3
>>> requests
>>> > inside Polaris and will be used to populate the "s3.endpoint"
>>> properties in
>>> > REST Catalog clients.
>>> >
>>> > Thoughts?
>>> >
>>> > [1530] https://github.com/apache/polaris/issues/1530
>>> > [1913] https://github.com/apache/polaris/pull/1913
>>> >
>>> > Thanks,
>>> > Dmitri.
>>> >
>>>
>>
Re: [DISCUSS] Adding endpoint to AwsStorageConfigInfo
Posted by Dmitri Bourlatchkov <di...@apache.org>.
I added a separate STS endpoint property to
https://github.com/apache/polaris/pull/1913
Cheers,
Dmitri.
On Wed, Jun 25, 2025 at 7:59 PM Dmitri Bourlatchkov <di...@apache.org>
wrote:
> Thanks for the feedback, Adnan!
>
> I was initially thinking of deferring separating STS and S3 endpoints to a
> later PR, but given interest, I'll add it to the current PR to reduce the
> number of API changes over time.
>
> Cheers,
> Dmitri.
>
> On Wed, Jun 25, 2025 at 1:28 PM Adnan Hemani
> <ad...@snowflake.com.invalid> wrote:
>
>> +1 on overall idea
>>
>> Comment however on keeping the same endpoint for STS and S3 - there may be
>> a customer use case (I have seen this before for similar use cases) where
>> the S3 bucket is not in the same region as the STS endpoint that has been
>> exposed to the Polaris server. In that case, you really need two separate
>> endpoints (example below). But I don't think this necessarily blocks this
>> idea from being merged today and then enriched later.
>>
>> Example:
>> * Polaris is deployed in us-west-1 in a EC2 instance that is within a
>> private subnet, which is given access only to STS' us-west-1 regional
>> endpoint through AWS configurations.
>> * User has S3 data in both us-west-1 and us-east-1.
>> * Under this proposal, user can now access data in us-west-1, but when
>> trying to access data in us-east-1, STS will not respond due to no network
>> routing tables for the us-east-1 regional endpoint.
>> * Important thing to remember is that STS token generated at any regional
>> endpoint can be used at any other regional endpoint. So, in theory, user
>> could still use us-west-1 STS to generate credentials to use for S3 data
>> in
>> us-east-1 and it would've been ok - but this proposal does not allow for
>> this.
>>
>> Best,
>> Adnan Hemani
>>
>> On Tue, Jun 24, 2025 at 9:23 PM Dmitri Bourlatchkov <di...@apache.org>
>> wrote:
>>
>> > Hi All,
>> >
>> > I propose to add an `endpoint` optional parameter to
>> AwsStorageConfigInfo
>> > in PR [1913].
>> >
>> > The main idea is to support non-AWS S3 implementations for [1530].
>> >
>> > Existing classes related to supporting S3 Polaris are coded to the AWS
>> SDK,
>> > which supports setting STS endpoints. Therefore, it seems natural to
>> allow
>> > users to optionally define a specific endpoint for their catalogs.
>> >
>> > This change is backward-compatible with existing clients and deployed
>> > catalogs.
>> >
>> > When an endpoint is defined it will be used for both STS and S3 requests
>> > inside Polaris and will be used to populate the "s3.endpoint"
>> properties in
>> > REST Catalog clients.
>> >
>> > Thoughts?
>> >
>> > [1530] https://github.com/apache/polaris/issues/1530
>> > [1913] https://github.com/apache/polaris/pull/1913
>> >
>> > Thanks,
>> > Dmitri.
>> >
>>
>
Re: [DISCUSS] Adding endpoint to AwsStorageConfigInfo
Posted by Dmitri Bourlatchkov <di...@apache.org>.
Thanks for the feedback, Adnan!
I was initially thinking of deferring separating STS and S3 endpoints to a
later PR, but given interest, I'll add it to the current PR to reduce the
number of API changes over time.
Cheers,
Dmitri.
On Wed, Jun 25, 2025 at 1:28 PM Adnan Hemani
<ad...@snowflake.com.invalid> wrote:
> +1 on overall idea
>
> Comment however on keeping the same endpoint for STS and S3 - there may be
> a customer use case (I have seen this before for similar use cases) where
> the S3 bucket is not in the same region as the STS endpoint that has been
> exposed to the Polaris server. In that case, you really need two separate
> endpoints (example below). But I don't think this necessarily blocks this
> idea from being merged today and then enriched later.
>
> Example:
> * Polaris is deployed in us-west-1 in a EC2 instance that is within a
> private subnet, which is given access only to STS' us-west-1 regional
> endpoint through AWS configurations.
> * User has S3 data in both us-west-1 and us-east-1.
> * Under this proposal, user can now access data in us-west-1, but when
> trying to access data in us-east-1, STS will not respond due to no network
> routing tables for the us-east-1 regional endpoint.
> * Important thing to remember is that STS token generated at any regional
> endpoint can be used at any other regional endpoint. So, in theory, user
> could still use us-west-1 STS to generate credentials to use for S3 data in
> us-east-1 and it would've been ok - but this proposal does not allow for
> this.
>
> Best,
> Adnan Hemani
>
> On Tue, Jun 24, 2025 at 9:23 PM Dmitri Bourlatchkov <di...@apache.org>
> wrote:
>
> > Hi All,
> >
> > I propose to add an `endpoint` optional parameter to AwsStorageConfigInfo
> > in PR [1913].
> >
> > The main idea is to support non-AWS S3 implementations for [1530].
> >
> > Existing classes related to supporting S3 Polaris are coded to the AWS
> SDK,
> > which supports setting STS endpoints. Therefore, it seems natural to
> allow
> > users to optionally define a specific endpoint for their catalogs.
> >
> > This change is backward-compatible with existing clients and deployed
> > catalogs.
> >
> > When an endpoint is defined it will be used for both STS and S3 requests
> > inside Polaris and will be used to populate the "s3.endpoint" properties
> in
> > REST Catalog clients.
> >
> > Thoughts?
> >
> > [1530] https://github.com/apache/polaris/issues/1530
> > [1913] https://github.com/apache/polaris/pull/1913
> >
> > Thanks,
> > Dmitri.
> >
>
Re: [DISCUSS] Adding endpoint to AwsStorageConfigInfo
Posted by Adnan Hemani <ad...@snowflake.com.INVALID>.
+1 on overall idea
Comment however on keeping the same endpoint for STS and S3 - there may be
a customer use case (I have seen this before for similar use cases) where
the S3 bucket is not in the same region as the STS endpoint that has been
exposed to the Polaris server. In that case, you really need two separate
endpoints (example below). But I don't think this necessarily blocks this
idea from being merged today and then enriched later.
Example:
* Polaris is deployed in us-west-1 in a EC2 instance that is within a
private subnet, which is given access only to STS' us-west-1 regional
endpoint through AWS configurations.
* User has S3 data in both us-west-1 and us-east-1.
* Under this proposal, user can now access data in us-west-1, but when
trying to access data in us-east-1, STS will not respond due to no network
routing tables for the us-east-1 regional endpoint.
* Important thing to remember is that STS token generated at any regional
endpoint can be used at any other regional endpoint. So, in theory, user
could still use us-west-1 STS to generate credentials to use for S3 data in
us-east-1 and it would've been ok - but this proposal does not allow for
this.
Best,
Adnan Hemani
On Tue, Jun 24, 2025 at 9:23 PM Dmitri Bourlatchkov <di...@apache.org>
wrote:
> Hi All,
>
> I propose to add an `endpoint` optional parameter to AwsStorageConfigInfo
> in PR [1913].
>
> The main idea is to support non-AWS S3 implementations for [1530].
>
> Existing classes related to supporting S3 Polaris are coded to the AWS SDK,
> which supports setting STS endpoints. Therefore, it seems natural to allow
> users to optionally define a specific endpoint for their catalogs.
>
> This change is backward-compatible with existing clients and deployed
> catalogs.
>
> When an endpoint is defined it will be used for both STS and S3 requests
> inside Polaris and will be used to populate the "s3.endpoint" properties in
> REST Catalog clients.
>
> Thoughts?
>
> [1530] https://github.com/apache/polaris/issues/1530
> [1913] https://github.com/apache/polaris/pull/1913
>
> Thanks,
> Dmitri.
>
Re: [DISCUSS] Adding endpoint to AwsStorageConfigInfo
Posted by Eric Maynard <er...@gmail.com>.
+1, I think this is a good idea.
If there is a similar concept for the other storage types, could we
consider adding that too?
*
https://learn.microsoft.com/en-us/azure/storage/common/storage-private-endpoints
--EM
On Tue, Jun 24, 2025 at 9:23 PM Dmitri Bourlatchkov <di...@apache.org>
wrote:
> Hi All,
>
> I propose to add an `endpoint` optional parameter to AwsStorageConfigInfo
> in PR [1913].
>
> The main idea is to support non-AWS S3 implementations for [1530].
>
> Existing classes related to supporting S3 Polaris are coded to the AWS SDK,
> which supports setting STS endpoints. Therefore, it seems natural to allow
> users to optionally define a specific endpoint for their catalogs.
>
> This change is backward-compatible with existing clients and deployed
> catalogs.
>
> When an endpoint is defined it will be used for both STS and S3 requests
> inside Polaris and will be used to populate the "s3.endpoint" properties in
> REST Catalog clients.
>
> Thoughts?
>
> [1530] https://github.com/apache/polaris/issues/1530
> [1913] https://github.com/apache/polaris/pull/1913
>
> Thanks,
> Dmitri.
>