You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@beam.apache.org by Juan Carlos Garcia <jc...@gmail.com> on 2019/02/20 10:11:08 UTC

FileIO with GCP - KMS support

Hi Folks,

Is there any discussion going on regarding the support to writes to GCP
bucket protected with KMS ?

Thanks and regards,
-- 

JC

Re: FileIO with GCP - KMS support

Posted by Juan Carlos Garcia <jc...@gmail.com>.

Hi Udi,

Thanks for that, looking forward for the 2.11 release.!!!!

JC

Udi Meiri <eh...@google.com> schrieb am Mi., 20. Feb. 2019, 22:26:

> Hi Juan!
>
> I've recently replaced GCS copy with rewrite:
> https://github.com/apache/beam/pull/7682
> It should be available in the next Beam release (2.11).
>
> On Wed, Feb 20, 2019 at 7:43 AM Juan Carlos Garcia <jc...@gmail.com>
> wrote:
>
>> Sorry, i hit send before verifying the right name of the method:
>>
>> The correct method name is: *enqueueCopy*
>>
>> On Wed, Feb 20, 2019 at 4:39 PM Juan Carlos Garcia <jc...@gmail.com>
>> wrote:
>>
>>> For anyone interested on the same while waiting for KMS support, just
>>> place the class on your own project
>>> *org.apache.beam.sdk.util.GcsUtil *
>>>
>>> Look / modify the *enqueCopy *method and replace the *storageClient.objects().copy(....)
>>> *with a *storageClient.objects().rewrite(....) , add the corresponding
>>> callback *and it should works as expected.
>>>
>>> Cheers!
>>>
>>>
>>> On Wed, Feb 20, 2019 at 11:11 AM Juan Carlos Garcia <jc...@gmail.com>
>>> wrote:
>>>
>>>> Hi Folks,
>>>>
>>>> Is there any discussion going on regarding the support to writes to GCP
>>>> bucket protected with KMS ?
>>>>
>>>> Thanks and regards,
>>>> --
>>>>
>>>> JC
>>>>
>>>>
>>>
>>> --
>>>
>>> JC
>>>
>>>
>>
>> --
>>
>> JC
>>
>>

Re: FileIO with GCP - KMS support

Posted by Udi Meiri <eh...@google.com>.

Hi Juan!

I've recently replaced GCS copy with rewrite:
https://github.com/apache/beam/pull/7682
It should be available in the next Beam release (2.11).

On Wed, Feb 20, 2019 at 7:43 AM Juan Carlos Garcia <jc...@gmail.com>
wrote:

> Sorry, i hit send before verifying the right name of the method:
>
> The correct method name is: *enqueueCopy*
>
> On Wed, Feb 20, 2019 at 4:39 PM Juan Carlos Garcia <jc...@gmail.com>
> wrote:
>
>> For anyone interested on the same while waiting for KMS support, just
>> place the class on your own project
>> *org.apache.beam.sdk.util.GcsUtil *
>>
>> Look / modify the *enqueCopy *method and replace the *storageClient.objects().copy(....)
>> *with a *storageClient.objects().rewrite(....) , add the corresponding
>> callback *and it should works as expected.
>>
>> Cheers!
>>
>>
>> On Wed, Feb 20, 2019 at 11:11 AM Juan Carlos Garcia <jc...@gmail.com>
>> wrote:
>>
>>> Hi Folks,
>>>
>>> Is there any discussion going on regarding the support to writes to GCP
>>> bucket protected with KMS ?
>>>
>>> Thanks and regards,
>>> --
>>>
>>> JC
>>>
>>>
>>
>> --
>>
>> JC
>>
>>
>
> --
>
> JC
>
>

Re: FileIO with GCP - KMS support

Posted by Juan Carlos Garcia <jc...@gmail.com>.

Sorry, i hit send before verifying the right name of the method:

The correct method name is: *enqueueCopy*

On Wed, Feb 20, 2019 at 4:39 PM Juan Carlos Garcia <jc...@gmail.com>
wrote:

> For anyone interested on the same while waiting for KMS support, just
> place the class on your own project
> *org.apache.beam.sdk.util.GcsUtil *
>
> Look / modify the *enqueCopy *method and replace the *storageClient.objects().copy(....)
> *with a *storageClient.objects().rewrite(....) , add the corresponding
> callback *and it should works as expected.
>
> Cheers!
>
>
> On Wed, Feb 20, 2019 at 11:11 AM Juan Carlos Garcia <jc...@gmail.com>
> wrote:
>
>> Hi Folks,
>>
>> Is there any discussion going on regarding the support to writes to GCP
>> bucket protected with KMS ?
>>
>> Thanks and regards,
>> --
>>
>> JC
>>
>>
>
> --
>
> JC
>
>

-- 

JC

Re: FileIO with GCP - KMS support

Posted by Juan Carlos Garcia <jc...@gmail.com>.

For anyone interested on the same while waiting for KMS support, just place
the class on your own project
*org.apache.beam.sdk.util.GcsUtil *

Look / modify the *enqueCopy *method and replace the
*storageClient.objects().copy(....)
*with a *storageClient.objects().rewrite(....) , add the corresponding
callback *and it should works as expected.

Cheers!

On Wed, Feb 20, 2019 at 11:11 AM Juan Carlos Garcia <jc...@gmail.com>
wrote:

> Hi Folks,
>
> Is there any discussion going on regarding the support to writes to GCP
> bucket protected with KMS ?
>
> Thanks and regards,
> --
>
> JC
>
>

-- 

JC