You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@arrow.apache.org by Will Jones <wi...@gmail.com> on 2022/05/24 22:40:24 UTC
DISCUSS: Make bucket creation opt-in in C++ S3FileSystem?
Hello Arrow devs,
I've written a PR for the C++ S3FileSystem that adds an option
"allow_create_buckets" which when false will error if attempting to create
or delete a bucket [1]. I've made the default value for this setting false,
though made it easy to override in URI, during construction, or after
construction. However, this is a breaking API change, so want to check:
Does this change seem reasonable to others?
My intention is to make creating and deleting buckets opt-in, rather than
opt-out. In my experience working with S3 in an industry setting, it's rare
to want to create buckets through an API that doesn't allow setting IAM
access policies or resource (cost-tracking) tags. And it seems likewise
undesirable to be able to accidentally delete the bucket using the
filesystem. There are definitely use-cases (like integration testing) where
creating and deleting buckets are useful, I just think it's slightly safer
for users to opt-in to that functionality.
Thanks,
Will Jones
[1] https://github.com/apache/arrow/pull/13206
Re: DISCUSS: Make bucket creation opt-in in C++ S3FileSystem?
Posted by Micah Kornfield <em...@gmail.com>.
+1, this sounds reasonable to me.
On Sun, May 29, 2022 at 9:26 AM Antoine Pitrou <an...@python.org> wrote:
> +1 as well
>
>
> Le 25/05/2022 à 01:25, Weston Pace a écrit :
> > +1
> >
> > I think opt-in is the right way to go here.
> >
> > On Tue, May 24, 2022 at 12:40 PM Will Jones <wi...@gmail.com>
> wrote:
> >>
> >> Hello Arrow devs,
> >>
> >> I've written a PR for the C++ S3FileSystem that adds an option
> >> "allow_create_buckets" which when false will error if attempting to
> create
> >> or delete a bucket [1]. I've made the default value for this setting
> false,
> >> though made it easy to override in URI, during construction, or after
> >> construction. However, this is a breaking API change, so want to check:
> >> Does this change seem reasonable to others?
> >>
> >> My intention is to make creating and deleting buckets opt-in, rather
> than
> >> opt-out. In my experience working with S3 in an industry setting, it's
> rare
> >> to want to create buckets through an API that doesn't allow setting IAM
> >> access policies or resource (cost-tracking) tags. And it seems likewise
> >> undesirable to be able to accidentally delete the bucket using the
> >> filesystem. There are definitely use-cases (like integration testing)
> where
> >> creating and deleting buckets are useful, I just think it's slightly
> safer
> >> for users to opt-in to that functionality.
> >>
> >> Thanks,
> >>
> >> Will Jones
> >>
> >> [1] https://github.com/apache/arrow/pull/13206
>
Re: DISCUSS: Make bucket creation opt-in in C++ S3FileSystem?
Posted by Antoine Pitrou <an...@python.org>.
+1 as well
Le 25/05/2022 à 01:25, Weston Pace a écrit :
> +1
>
> I think opt-in is the right way to go here.
>
> On Tue, May 24, 2022 at 12:40 PM Will Jones <wi...@gmail.com> wrote:
>>
>> Hello Arrow devs,
>>
>> I've written a PR for the C++ S3FileSystem that adds an option
>> "allow_create_buckets" which when false will error if attempting to create
>> or delete a bucket [1]. I've made the default value for this setting false,
>> though made it easy to override in URI, during construction, or after
>> construction. However, this is a breaking API change, so want to check:
>> Does this change seem reasonable to others?
>>
>> My intention is to make creating and deleting buckets opt-in, rather than
>> opt-out. In my experience working with S3 in an industry setting, it's rare
>> to want to create buckets through an API that doesn't allow setting IAM
>> access policies or resource (cost-tracking) tags. And it seems likewise
>> undesirable to be able to accidentally delete the bucket using the
>> filesystem. There are definitely use-cases (like integration testing) where
>> creating and deleting buckets are useful, I just think it's slightly safer
>> for users to opt-in to that functionality.
>>
>> Thanks,
>>
>> Will Jones
>>
>> [1] https://github.com/apache/arrow/pull/13206
Re: DISCUSS: Make bucket creation opt-in in C++ S3FileSystem?
Posted by Weston Pace <we...@gmail.com>.
+1
I think opt-in is the right way to go here.
On Tue, May 24, 2022 at 12:40 PM Will Jones <wi...@gmail.com> wrote:
>
> Hello Arrow devs,
>
> I've written a PR for the C++ S3FileSystem that adds an option
> "allow_create_buckets" which when false will error if attempting to create
> or delete a bucket [1]. I've made the default value for this setting false,
> though made it easy to override in URI, during construction, or after
> construction. However, this is a breaking API change, so want to check:
> Does this change seem reasonable to others?
>
> My intention is to make creating and deleting buckets opt-in, rather than
> opt-out. In my experience working with S3 in an industry setting, it's rare
> to want to create buckets through an API that doesn't allow setting IAM
> access policies or resource (cost-tracking) tags. And it seems likewise
> undesirable to be able to accidentally delete the bucket using the
> filesystem. There are definitely use-cases (like integration testing) where
> creating and deleting buckets are useful, I just think it's slightly safer
> for users to opt-in to that functionality.
>
> Thanks,
>
> Will Jones
>
> [1] https://github.com/apache/arrow/pull/13206