Re: Parquet Sync - Meeting notes

[Zoltan Ivanfi <zi...@cloudera.com.INVALID>]

Hi,

I would like to add the following to the notes for topic "1. key signing":
- Zoltan brought up the question of whether and how PMC-s from the US
could remotely sign the keys of committers/PMC-s located in Europe.
- Julien and Ryan commented that for the purpose of signing releases
it is not really necessary for the signer's key to be in the web of
trust as long as it is in the central KEYS file (especially if the
signers participate in Parquet Syncs discussing the RC, thereby
implicitly confirming their ownership of that signature).

Br,

Zoltan

On Tue, Apr 30, 2019 at 10:25 PM Xinli shang <sh...@uber.com> wrote:
>
> Hi all,
>
> This is to follow up of the meeting notes below. I created Jira ticket PARQUET-1396 and the design can be found here.  The recorded video in Hadoop Contributor Meetup can also help reading the design. Please share your feedback by commenting on the design doc.
>
> On top of Gidon’s change, we introduced a plugin/interface to Parquet to activate encryption and build up encryption properties. Currently, we implement its schema driven implementation, but it can be implemented in another way too. I will send out the design soon.
>
>
> Xinli
>
> On Tue, Apr 30, 2019 at 12:30 PM Xinli shang <sh...@uber.com> wrote:
>>
>> 4/30/2019
>>
>>
>> Attendee:
>>
>> Zoltan and Several other folks(Cloudera)
>>
>> Brian (SaS?)
>>
>> Ryan Blue(Netflix)
>>
>> Julien(WeWorks)
>>
>> Wes McKinney(Ursa Labs)
>>
>> Gidon Gershinsky(IBM)
>>
>> Steven(?)
>>
>> Anikt(?)
>>
>> Deepak(?)
>>
>> Xinli Shang(Uber)
>>
>>
>> Topics:
>>
>> Key signing issue
>>
>> Zoltan/Julien/Ryan:
>>
>> We already have email exchange of this issue.
>>
>> In the past, it is done in person. But it is OK to sign each other via video conference. We can do a video session of signing keys.
>>
>> It is painful to do this every release
>>
>>
>> Column Encryption
>>
>> Gidon:
>>
>> C++ version progress well. It is pretty much done.
>>
>> Wait for Parquet-1.11.0 release to send out code review
>>
>> Found issues in Java. Worked around it. Will talk to Java community.
>>
>> Xinli:
>>
>> On top of Gidon’s change, we introduced a plugin/interface to Parquet to activate encryption and build up encryption properties. Currently, we implement its schema driven implementation, but it can be implemented in another way too. I will send out the design soon.
>>
>> Gidon:
>>
>> Overall we took a bottom-up approach. We might need another layer on top of these to make the adoption easier.
>>
>> Ryan:
>>
>> Different companies can have a different implementation. It is good to have a plugin mode.
>>
>> Brian: Question of the key metadata, KMS.
>>
>> Currently, Parquet designs it as a byte array. Depending on the implementation, it can be used to record the KMS/Key Metadata.
>>
>> Parquet-1.11.0 Release Validation
>>
>> Ryan
>>
>> Validate the write path of column index - We need to test the calculation is correct; Validation is independent. Ryan will take this task.
>>
>> Brian:
>>
>> Can help some testing in Summer if needed.
>>
>> Steven:
>>
>> What is the test strategy, any fuzzing test?
>>
>> Ryan:
>>
>> We have some random test but not reliable. Inside Netflix, we have stable fuzzing test. May need to port some to Parquet.
>>
>> Xinli:
>>
>> We have run a lot of regression test on Parquet-1.11.0. We add encryption code on top of 1.11.0 and run a lot of tests. No new feature test of 1.110 but existing features tests are so far so good. Let us know if you want us to add some more tests into our test suite.
>>
>>
>> Remove old Parquet modules
>>
>> Ryan
>>
>> We should remove those old modules if they are not needed
>>
>> Hive module - Seems not used
>>
>> Scrooge module - if it is only used by one company, we might not want to maintain it
>>
>> Does anybody still use parquet-tools instead of parquet-cli? Maybe we can mark it as deprecated.
>>
>> Open a Jira ticket for it.
>>
>> Julien
>>
>> Twitter may use it. Julien will check with Twitter.
>>
>> We should communicate widely.
>>
>>
>> --
>> Xinli Shang (Uber)
>
>
>
> --
> Xinli Shang