You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@drill.apache.org by Parth Chandra <pa...@apache.org> on 2017/06/22 22:20:14 UTC

[DISCUSS] Drill 1.11.0 release

Hello Drillers,

  It's been three months since the last release and it is time to do the
next one. Does anyone want to volunteer to be the release manager? I would
encourage our new committers to try their hand at it :)

  While we wait for volunteers, if there are any issues on which work is in
progress, that you feel we *must* include in the release, please post in
reply to this thread.

Thanks

Parth

Re: [DISCUSS] Drill 1.11.0 release

Posted by Charles Givre <cg...@gmail.com>.

Hi @arina
I’m going to try to get the crypto/hash functions submitted in the next day or so if that’s ok.
—C

> On Jun 29, 2017, at 03:26, Arina Yelchiyeva <ar...@gmail.com> wrote:
> 
> Hi all,
> 
> here is the list of Jira we are waiting before making final cut off:
> 1. DRILL-5601  - Paul
> 2. DRILL-3640  - Kunal
> 3. DRILL-5420 - Kunal
> 
> Please let me know if there are any others Jiras we should wait for.
> 
> @Charles,
> you have mentioned that you want to include some UDFs in Drill 1.11, do you
> have any ETA for the PR?
> 
> @Paul / Parth
> I see there is active CR for pcap format PR, do we want to include it in
> this release as well?
> 
> Kind regards
> Arina
> 
> 
> On Thu, Jun 29, 2017 at 10:05 AM, Arina Yelchiyeva <
> arina.yelchiyeva@gmail.com> wrote:
> 
>> Parth, thanks a lot the instructions.
>> 
>> 
>> 
>> On Wed, Jun 28, 2017 at 8:24 PM, Parth Chandra <pa...@apache.org> wrote:
>> 
>>> FWIW, I put release instructions culled from gists written by previous
>>> release managers into an updated gist here [1]
>>> 
>>> HTH
>>> 
>>> Parth
>>> 
>>> [1] https://github.com/parthchandra/drill/wiki/Drill-Release-Process
>>> <https://github.com/parthchandra/drill/wiki/Drill-Release-Process>
>>> 
>>> 
>>> 
>>> On Tue, Jun 27, 2017 at 5:24 PM, Paul Rogers <pr...@mapr.com> wrote:
>>> 
>>>> Hi Arina,
>>>> 
>>>> I have two projects in progress: one is a candidate for 1.11, the other
>>> is
>>>> not.
>>>> 
>>>> The “managed external sort” is the good candidate for 1.11: The PR for
>>>> DRILL-5601 is being reviewed. Getting that in will allow us to enable
>>> the
>>>> “new” external sort by default.
>>>> 
>>>> The other project is to address Drill’s memory fragmentation issue with
>>>> vectors over 16 MB in size. Two PRs have been open for a while. They are
>>>> the first of several. Each takes a while to review. So, the project does
>>>> not fit 1.11. The work can go into the next release instead.
>>>> 
>>>> Thanks,
>>>> 
>>>> - Paul
>>>> 
>>>> 
>>>> 
>>>> 
>>> 
>> 
>>

Re: [DISCUSS] Drill 1.11.0 release

Posted by Arina Yelchiyeva <ar...@gmail.com>.

I have merged all pending Jiras (including blockers) and will start building
RC0 shortly.

*Note for the committers:*
*until the release is not over and Drill version is not changed to
1.12.0-SNAPSHOT, please do not push any changes into Drill master. *

Kind regards
Arina

On Wed, Jul 19, 2017 at 12:15 PM, Arina Yelchiyeva <
arina.yelchiyeva@gmail.com> wrote:

> Thanks, Boaz!
>
> Unfortunately, I still see the failures connected to DRILL-5669.
> I'll share the details.
>
> Kind regards
> Arina
>
> On Wed, Jul 19, 2017 at 2:00 AM, Boaz Ben-Zvi <bb...@mapr.com> wrote:
>
>> Hi Arina,
>>
>>      The three Jira’s are ready (+1, squashed, rebased, etc.):
>>
>> • DRILL-5616: Add memory checks, plus minor metrics changes
>>     (Commit ID: c7ad56b7e075e84761896c9ce660b18b09d9e49a )
>> • DRILL-5665: Add the option planner.force_2phase_aggr to override small
>> inputs
>>     (Commit ID: 1917ec084bdef74962370fbe5b253172c634c86e )
>> • DRILL-5669: Add a configurable option for minimum memory allocation to
>> buffered ops
>>     (Commit ID: 4d7a3fe3e7ae52db57b0fd1f0619c6d1d64308c8 )
>>
>>   Thanks,
>>
>>          Boaz
>>
>> On 7/18/17, 3:04 AM, "Arina Yelchiyeva" <ar...@gmail.com>
>> wrote:
>>
>>     Hi Parth,
>>
>>     no problem, I did not start preparing the release since there is one
>>     blocker has left:
>>
>>     *DRILL-5669 - Boaz Ben-Zvi (fix will be ready in a couple of days)*
>>
>>
>>     Kind regards
>>     Arina
>>
>>     On Mon, Jul 17, 2017 at 11:26 PM, Parth Chandra <pa...@apache.org>
>> wrote:
>>
>>     > Hi Arina,
>>     >
>>     >   I just realized that I did not push the C++ client fix (
>> DRILL-5659) to
>>     > master. I have just done so.
>>     >
>>     >   Sorry if I messed things up for the release, but we should get
>> this fix
>>     > in.
>>     >
>>     > Parth
>>     >
>>     > On Fri, Jul 14, 2017 at 8:15 AM, Paul Rogers <pr...@mapr.com>
>> wrote:
>>     >
>>     > > I looked into one NPE on CSV. Turns out it is already fixed by
>> previous
>>     > > 1.11 PRs, so we’re good.
>>     > >
>>     > > There is another reported by a user on the user list, but I’ve
>> not heard
>>     > > back about getting sample data. Let’s not hold the release for
>> that.
>>     > >
>>     > > - Paul
>>     > >
>>     > > > On Jul 14, 2017, at 2:38 AM, Arina Yelchiyeva <
>>     > > arina.yelchiyeva@gmail.com> wrote:
>>     > > >
>>     > > > *Blockers*
>>     > > > DRILL-5659 - Parth (done)
>>     > > > DRILL-5669 - Boaz Ben-Zvi (fix will be ready in a couple of
>> days)
>>     > > >
>>     > > > *Possible blockers*
>>     > > > NPE on CSV - Paul can you please provide an update?
>>     > > >
>>     > > > Kind regards
>>     > > > Arina
>>     > > >
>>     > > > On Thu, Jul 13, 2017 at 2:46 PM, Arina Yelchiyeva <
>>     > > > arina.yelchiyeva@gmail.com> wrote:
>>     > > >
>>     > > >> *Blockers*
>>     > > >> DRILL-5659 - problem is identified, Parth is working on the
>> fix.
>>     > > >>
>>     > > >> *Not blockers*
>>     > > >> DRILL-5660 - Vitalii
>>     > > >> DRILL-5468 - Jinfeng
>>     > > >>
>>     > > >> *Possible blockers*
>>     > > >> DRILL-5669 - Boaz Ben-Zvi / Paul
>>     > > >> NPE on CSV - Paul
>>     > > >>
>>     > > >> Kind regards
>>     > > >> Arina
>>     > > >>
>>     > > >>
>>     > > >> On Thu, Jul 13, 2017 at 1:28 AM, Paul Rogers <progers@mapr.com
>> >
>>     > wrote:
>>     > > >>
>>     > > >>> I withdraw my suggestion to fix DRILL-5660. Was a big hassle
>> for us
>>     > to
>>     > > >>> track down the incompatibility the issue internally, but
>> production
>>     > > users
>>     > > >>> probably don’t attempt to use one build of Drill with
>> metadata files
>>     > > >>> created from another. We can worry about the issue later only
>> if a
>>     > user
>>     > > >>> actually runs into it.
>>     > > >>>
>>     > > >>> As suggested earlier, let’s just document this obscure case
>> in a
>>     > > release
>>     > > >>> note.
>>     > > >>>
>>     > > >>> - Paul
>>     > > >>>
>>     > > >>>> On Jul 12, 2017, at 3:04 PM, Jinfeng Ni <jn...@apache.org>
>> wrote:
>>     > > >>>>
>>     > > >>>> Upgrade from 1.10 to 1.11 is irreversible only when parquet
>> metadata
>>     > > >>> cache
>>     > > >>>> file is involved.
>>     > > >>>>
>>     > > >>>> I might be wrong, but I thought that if someone is running on
>>     > 1.10.0,
>>     > > >>> and
>>     > > >>>> he wants to go back to 1.0.0, where the first version of
>> parquet
>>     > > >>> meatadata
>>     > > >>>> cache was introduced, can we guarantee that Drillbit would
>> work
>>     > > >>> correctly?
>>     > > >>>> I'm trying to understand what DRILL-5660 asks for is a new
>>     > > requirement,
>>     > > >>> or
>>     > > >>>> an requirement being there starting from Drill 1.0.0?
>>     > > >>>>
>>     > > >>>> Regression failures are normally regarded as release blocker.
>>     > > >>>>
>>     > > >>>>
>>     > > >>>>
>>     > > >>>> On Wed, Jul 12, 2017 at 2:54 PM, Paul Rogers <
>> progers@mapr.com>
>>     > > wrote:
>>     > > >>>>
>>     > > >>>>> If we don’t do DRILL-5660 for 1.11 then we don’t need to do
>> it at
>>     > > all,
>>     > > >>> so
>>     > > >>>>> that is a time savings. We could handle this via a release
>> note
>>     > item
>>     > > >>> that
>>     > > >>>>> simply states that an upgrade from 1.10 to 1.11 is
>> irreversible: we
>>     > > do
>>     > > >>> not
>>     > > >>>>> support the ability to fall back to the old version.
>> Everyone OK
>>     > with
>>     > > >>> that?
>>     > > >>>>>
>>     > > >>>>> We are seeing some stress test regression failures that,
>> I’m told,
>>     > > will
>>     > > >>>>> show up as JIRA entries shortly.
>>     > > >>>>>
>>     > > >>>>> - Paul
>>     > > >>>>>
>>     > > >>>>>> On Jul 12, 2017, at 11:05 AM, Jinfeng Ni <jn...@apache.org>
>> wrote:
>>     > > >>>>>>
>>     > > >>>>>> I put some analysis in DRILL-5468.
>>     > > >>>>>>
>>     > > >>>>>> Also, IMHO, DRILL-5660 is not a release blocker.
>>     > > >>>>>>
>>     > > >>>>>>
>>     > > >>>>>>
>>     > > >>>>>> On Tue, Jul 11, 2017 at 11:27 AM, Parth Chandra <
>>     > parthc@apache.org>
>>     > > >>>>> wrote:
>>     > > >>>>>>
>>     > > >>>>>>> Re DRILL-5634:
>>     > > >>>>>>> From the Apache commons web page [1] -
>>     > > >>>>>>>
>>     > > >>>>>>> Please note that Apache Commons Crypto doesn't implement
>> the
>>     > > >>>>> cryptographic
>>     > > >>>>>>> algorithm such as AES directly.* It wraps to Openssl or
>> JCE which
>>     > > >>>>> implement
>>     > > >>>>>>> the algorithms.*
>>     > > >>>>>>>
>>     > > >>>>>>>
>>     > > >>>>>>> Since we are cleared for using Openssl and JCE, we are OK
>> to
>>     > > include
>>     > > >>> the
>>     > > >>>>>>> AES code in the UDFs.
>>     > > >>>>>>>
>>     > > >>>>>>>
>>     > > >>>>>>> Parth
>>     > > >>>>>>>
>>     > > >>>>>>> [1] https://commons.apache.org/proper/commons-crypto/
>>     > > >>>>>>>
>>     > > >>>>>>> On Tue, Jul 11, 2017 at 10:56 AM, Arina Yelchiyeva <
>>     > > >>>>>>> arina.yelchiyeva@gmail.com> wrote:
>>     > > >>>>>>>
>>     > > >>>>>>>> During today hangouts we agreed on approximate cut-off
>> date - 14
>>     > > >>> July,
>>     > > >>>>>>> 2017
>>     > > >>>>>>>> (date may be shifted due to the blockers).
>>     > > >>>>>>>> I'll start preparing test RC on Thursday to solve any
>> release
>>     > > >>>>> preparation
>>     > > >>>>>>>> issues beforehand.
>>     > > >>>>>>>>
>>     > > >>>>>>>> *Blockers*
>>     > > >>>>>>>> DRILL-5660 - Vitalii
>>     > > >>>>>>>>
>>     > > >>>>>>>> *Possible blockers*
>>     > > >>>>>>>> DRILL-5468 - Jinfeng
>>     > > >>>>>>>> NPE on CSV source - Paul
>>     > > >>>>>>>> DRILL-5659 - Parth
>>     > > >>>>>>>>
>>     > > >>>>>>>> *Will be included if done before the cut-off date*
>>     > > >>>>>>>> DRILL-5616 / DRILL-5665 - Boaz
>>     > > >>>>>>>> DRILL-5634 - Parth will take a look at Apache commons
>>     > > implementation
>>     > > >>>>> and
>>     > > >>>>>>>> update.
>>     > > >>>>>>>>
>>     > > >>>>>>>> Kind regards
>>     > > >>>>>>>> Arina
>>     > > >>>>>>>>
>>     > > >>>>>>>> On Tue, Jul 11, 2017 at 3:14 PM, Charles Givre <
>>     > cgivre@gmail.com>
>>     > > >>>>> wrote:
>>     > > >>>>>>>>
>>     > > >>>>>>>>> Hi Arina,
>>     > > >>>>>>>>> I can’t make the hangout, but we have a few options.
>>     > Basically,
>>     > > >>> since
>>     > > >>>>>>>> all
>>     > > >>>>>>>>> the hashing algorithms have no export restrictions, we
>> could:
>>     > > >>>>>>>>> 1.  Remove the AES encrypt/decrypt until I can figure
>> out what
>>     > we
>>     > > >>> have
>>     > > >>>>>>> to
>>     > > >>>>>>>>> do (and do it)
>>     > > >>>>>>>>> 2.  I can try to research it today or tomorrow to
>> figure out
>>     > > >>> exactly
>>     > > >>>>>>> what
>>     > > >>>>>>>>> we have to do to include AES.
>>     > > >>>>>>>>> 3.  Remove the entire package
>>     > > >>>>>>>>>
>>     > > >>>>>>>>> I’d like to see the hashing functions at least included
>> because
>>     > > >>> we’ve
>>     > > >>>>>>> had
>>     > > >>>>>>>>> a lot of requests for that functionality.   I also
>> don’t want
>>     > > this
>>     > > >>> to
>>     > > >>>>>>>> hold
>>     > > >>>>>>>>> things up, so please let me know your preference.
>> Regardless,
>>     > > I’m
>>     > > >>>>>>> going
>>     > > >>>>>>>> to
>>     > > >>>>>>>>> start trying to figure out the restrictions so we’ll
>> know for
>>     > the
>>     > > >>>>>>> future.
>>     > > >>>>>>>>>
>>     > > >>>>>>>>> Best,
>>     > > >>>>>>>>> — C
>>     > > >>>>>>>>>
>>     > > >>>>>>>>>
>>     > > >>>>>>>>>
>>     > > >>>>>>>>>> On Jul 11, 2017, at 05:58, Arina Yelchiyeva <
>>     > > >>>>>>>> arina.yelchiyeva@gmail.com>
>>     > > >>>>>>>>> wrote:
>>     > > >>>>>>>>>>
>>     > > >>>>>>>>>> @Jinfeng,
>>     > > >>>>>>>>>> Let's discuss today on Drill hangout final cut-off
>> date.
>>     > > >>>>>>>>>>
>>     > > >>>>>>>>>> @Charles
>>     > > >>>>>>>>>> Since there were questions about AES in DRILL-5634,
>> should we
>>     > > >>> exclude
>>     > > >>>>>>>> it
>>     > > >>>>>>>>>> from the candidates for 1.11.0 release?
>>     > > >>>>>>>>>>
>>     > > >>>>>>>>>> @Boaz
>>     > > >>>>>>>>>> Do we have any ETA when DRILL-5616 and DRILL-5665 will
>> pass
>>     > code
>>     > > >>>>>>>> review?
>>     > > >>>>>>>>>>
>>     > > >>>>>>>>>> Also I believe we should include DRILL-5660 where we
>> need to
>>     > > bump
>>     > > >>> up
>>     > > >>>>>>>>>> parquet metadata version.
>>     > > >>>>>>>>>> @Vitalii
>>     > > >>>>>>>>>> Do you have any ETA for the fix?
>>     > > >>>>>>>>>>
>>     > > >>>>>>>>>> Kind regards
>>     > > >>>>>>>>>> Arina
>>     > > >>>>>>>>>>
>>     > > >>>>>>>>>> On Tue, Jul 11, 2017 at 6:02 AM, Boaz Ben-Zvi <
>>     > > bben-zvi@mapr.com>
>>     > > >>>>>>>> wrote:
>>     > > >>>>>>>>>>
>>     > > >>>>>>>>>>> Hi Arina,
>>     > > >>>>>>>>>>>
>>     > > >>>>>>>>>>>  Two PR have just been submitted, relating to the
>> Hash Aggr
>>     > > >>>>>>> Spill:
>>     > > >>>>>>>>>>> DRILL-5616 and DRILL-5665 .
>>     > > >>>>>>>>>>> Can these be considered for 1.11 as well ?
>>     > > >>>>>>>>>>>
>>     > > >>>>>>>>>>>  Thanks,
>>     > > >>>>>>>>>>>
>>     > > >>>>>>>>>>>            Boaz
>>     > > >>>>>>>>>>>
>>     > > >>>>>>>>>>> On 7/10/17, 1:49 PM, "Paul Rogers" <pr...@mapr.com>
>> wrote:
>>     > > >>>>>>>>>>>
>>     > > >>>>>>>>>>> DRILL-5601 is no longer a blocker for 1.11; we’ll
>> defer that
>>     > > >>>>>>> until
>>     > > >>>>>>>>>>> after the release to allow others more time to work
>> on other
>>     > > >>>>>>> features
>>     > > >>>>>>>>>>> rather than doing code reviews.
>>     > > >>>>>>>>>>>
>>     > > >>>>>>>>>>> - Paul
>>     > > >>>>>>>>>>>
>>     > > >>>>>>>>>>>> On Jul 10, 2017, at 1:43 PM, Jinfeng Ni <
>> jni@apache.org>
>>     > > wrote:
>>     > > >>>>>>>>>>>>
>>     > > >>>>>>>>>>>> Hi Arina,
>>     > > >>>>>>>>>>>>
>>     > > >>>>>>>>>>>> It has been a while since we discussed 1.11.0
>> release.  Any
>>     > > >>> update
>>     > > >>>>>>>>>>> on the
>>     > > >>>>>>>>>>>> new release plan?  Do we have a cut-off date ?
>>     > > >>>>>>>>>>>>
>>     > > >>>>>>>>>>>> In the past, I believe Drill release is a
>> time-bounded, not
>>     > > >>> feature
>>     > > >>>>>>>>>>>> bounded. If certain fixes/features could not make it
>> within
>>     > a
>>     > > >>>>>>> certain
>>     > > >>>>>>>>>>>> time-frame, we had better defer them into future
>> release,
>>     > > unless
>>     > > >>>>>>>>>>> those
>>     > > >>>>>>>>>>>> fixes/features are regarded as blocking issues for a
>> new
>>     > > >>> release.
>>     > > >>>>>>>>>>>>
>>     > > >>>>>>>>>>>>
>>     > > >>>>>>>>>>>>
>>     > > >>>>>>>>>>>>
>>     > > >>>>>>>>>>>>
>>     > > >>>>>>>>>>>>
>>     > > >>>>>>>>>>>>
>>     > > >>>>>>>>>>>> On Tue, Jul 4, 2017 at 5:14 AM, Arina Yelchiyeva <
>>     > > >>>>>>>>>>> arina.yelchiyeva@gmail.com
>>     > > >>>>>>>>>>>>> wrote:
>>     > > >>>>>>>>>>>>
>>     > > >>>>>>>>>>>>> Latest update:
>>     > > >>>>>>>>>>>>>
>>     > > >>>>>>>>>>>>> We wait for the following Jiras before final cut
>> off.
>>     > > >>>>>>>>>>>>> 1. DRILL-5601  - Paul
>>     > > >>>>>>>>>>>>> 2. DRILL-5420 - Kunal
>>     > > >>>>>>>>>>>>> 3. DRILL-5634 - Charles
>>     > > >>>>>>>>>>>>>
>>     > > >>>>>>>>>>>>> Kind regards
>>     > > >>>>>>>>>>>>> Arina
>>     > > >>>>>>>>>>>>>
>>     > > >>>>>>>>>>>>> On Fri, Jun 30, 2017 at 9:29 PM, Kunal Khatua <
>>     > > >>> kkhatua@mapr.com>
>>     > > >>>>>>>>>>> wrote:
>>     > > >>>>>>>>>>>>>
>>     > > >>>>>>>>>>>>>> Parth's already +1'd the process with a minor
>> update. So,
>>     > > >>> we're
>>     > > >>>>>>>>>>> good to
>>     > > >>>>>>>>>>>>> go
>>     > > >>>>>>>>>>>>>> on DRILL-5420. I'll change the label on that to
>>     > > >>> ready-to-commit.
>>     > > >>>>>>>>>>>>>>
>>     > > >>>>>>>>>>>>>> <http://www.mapr.com/>
>>     > > >>>>>>>>>>>>>>
>>     > > >>>>>>>>>>>>>> ________________________________
>>     > > >>>>>>>>>>>>>> From: Parth Chandra <pa...@apache.org>
>>     > > >>>>>>>>>>>>>> Sent: Thursday, June 29, 2017 9:44:39 AM
>>     > > >>>>>>>>>>>>>> To: dev@drill.apache.org
>>     > > >>>>>>>>>>>>>> Subject: Re: [DISCUSS] Drill 1.11.0 release
>>     > > >>>>>>>>>>>>>>
>>     > > >>>>>>>>>>>>>> I'm Ok with the pcap format plugin if Pauls
>> comments are
>>     > > also
>>     > > >>>>>>>>>>> addressed.
>>     > > >>>>>>>>>>>>> If
>>     > > >>>>>>>>>>>>>> the changes are done before your cutoff
>> deadline,we can
>>     > > merge
>>     > > >>> it
>>     > > >>>>>>>>>>> in.
>>     > > >>>>>>>>>>>>>>
>>     > > >>>>>>>>>>>>>> On Thu, Jun 29, 2017 at 12:26 AM, Arina Yelchiyeva
>> <
>>     > > >>>>>>>>>>>>>> arina.yelchiyeva@gmail.com> wrote:
>>     > > >>>>>>>>>>>>>>
>>     > > >>>>>>>>>>>>>>> Hi all,
>>     > > >>>>>>>>>>>>>>>
>>     > > >>>>>>>>>>>>>>> here is the list of Jira we are waiting before
>> making
>>     > final
>>     > > >>> cut
>>     > > >>>>>>>>>>> off:
>>     > > >>>>>>>>>>>>>>> 1. DRILL-5601  - Paul
>>     > > >>>>>>>>>>>>>>> 2. DRILL-3640  - Kunal
>>     > > >>>>>>>>>>>>>>> 3. DRILL-5420 - Kunal
>>     > > >>>>>>>>>>>>>>>
>>     > > >>>>>>>>>>>>>>> Please let me know if there are any others Jiras
>> we
>>     > should
>>     > > >>> wait
>>     > > >>>>>>>>>>> for.
>>     > > >>>>>>>>>>>>>>>
>>     > > >>>>>>>>>>>>>>> @Charles,
>>     > > >>>>>>>>>>>>>>> you have mentioned that you want to include some
>> UDFs in
>>     > > >>> Drill
>>     > > >>>>>>>>>>> 1.11, do
>>     > > >>>>>>>>>>>>>> you
>>     > > >>>>>>>>>>>>>>> have any ETA for the PR?
>>     > > >>>>>>>>>>>>>>>
>>     > > >>>>>>>>>>>>>>> @Paul / Parth
>>     > > >>>>>>>>>>>>>>> I see there is active CR for pcap format PR, do
>> we want
>>     > to
>>     > > >>>>>>>>>>> include it
>>     > > >>>>>>>>>>>>> in
>>     > > >>>>>>>>>>>>>>> this release as well?
>>     > > >>>>>>>>>>>>>>>
>>     > > >>>>>>>>>>>>>>> Kind regards
>>     > > >>>>>>>>>>>>>>> Arina
>>     > > >>>>>>>>>>>>>>>
>>     > > >>>>>>>>>>>>>>>
>>     > > >>>>>>>>>>>>>>> On Thu, Jun 29, 2017 at 10:05 AM, Arina
>> Yelchiyeva <
>>     > > >>>>>>>>>>>>>>> arina.yelchiyeva@gmail.com> wrote:
>>     > > >>>>>>>>>>>>>>>
>>     > > >>>>>>>>>>>>>>>> Parth, thanks a lot the instructions.
>>     > > >>>>>>>>>>>>>>>>
>>     > > >>>>>>>>>>>>>>>>
>>     > > >>>>>>>>>>>>>>>>
>>     > > >>>>>>>>>>>>>>>> On Wed, Jun 28, 2017 at 8:24 PM, Parth Chandra <
>>     > > >>>>>>>>>>> parthc@apache.org>
>>     > > >>>>>>>>>>>>>>> wrote:
>>     > > >>>>>>>>>>>>>>>>
>>     > > >>>>>>>>>>>>>>>>> FWIW, I put release instructions culled from
>> gists
>>     > > written
>>     > > >>> by
>>     > > >>>>>>>>>>>>> previous
>>     > > >>>>>>>>>>>>>>>>> release managers into an updated gist here [1]
>>     > > >>>>>>>>>>>>>>>>>
>>     > > >>>>>>>>>>>>>>>>> HTH
>>     > > >>>>>>>>>>>>>>>>>
>>     > > >>>>>>>>>>>>>>>>> Parth
>>     > > >>>>>>>>>>>>>>>>>
>>     > > >>>>>>>>>>>>>>>>> [1] https://github.com/parthchandr
>> a/drill/wiki/Drill-
>>     > > >>>>>>>>>>>>> Release-Process
>>     > > >>>>>>>>>>>>>>>>> <https://github.com/parthchand
>> ra/drill/wiki/Drill-
>>     > > >>>>>>>>>>> Release-Process>
>>     > > >>>>>>>>>>>>>>>>>
>>     > > >>>>>>>>>>>>>>>>>
>>     > > >>>>>>>>>>>>>>>>>
>>     > > >>>>>>>>>>>>>>>>> On Tue, Jun 27, 2017 at 5:24 PM, Paul Rogers <
>>     > > >>>>>>> progers@mapr.com>
>>     > > >>>>>>>>>>>>>> wrote:
>>     > > >>>>>>>>>>>>>>>>>
>>     > > >>>>>>>>>>>>>>>>>> Hi Arina,
>>     > > >>>>>>>>>>>>>>>>>>
>>     > > >>>>>>>>>>>>>>>>>> I have two projects in progress: one is a
>> candidate
>>     > for
>>     > > >>> 1.11,
>>     > > >>>>>>>>>>> the
>>     > > >>>>>>>>>>>>>>> other
>>     > > >>>>>>>>>>>>>>>>> is
>>     > > >>>>>>>>>>>>>>>>>> not.
>>     > > >>>>>>>>>>>>>>>>>>
>>     > > >>>>>>>>>>>>>>>>>> The “managed external sort” is the good
>> candidate for
>>     > > >>> 1.11:
>>     > > >>>>>>>>>>> The PR
>>     > > >>>>>>>>>>>>>> for
>>     > > >>>>>>>>>>>>>>>>>> DRILL-5601 is being reviewed. Getting that in
>> will
>>     > allow
>>     > > >>> us
>>     > > >>>>>>> to
>>     > > >>>>>>>>>>>>>> enable
>>     > > >>>>>>>>>>>>>>>>> the
>>     > > >>>>>>>>>>>>>>>>>> “new” external sort by default.
>>     > > >>>>>>>>>>>>>>>>>>
>>     > > >>>>>>>>>>>>>>>>>> The other project is to address Drill’s memory
>>     > > >>> fragmentation
>>     > > >>>>>>>>>>> issue
>>     > > >>>>>>>>>>>>>>> with
>>     > > >>>>>>>>>>>>>>>>>> vectors over 16 MB in size. Two PRs have been
>> open
>>     > for a
>>     > > >>>>>>> while.
>>     > > >>>>>>>>>>>>> They
>>     > > >>>>>>>>>>>>>>> are
>>     > > >>>>>>>>>>>>>>>>>> the first of several. Each takes a while to
>> review.
>>     > So,
>>     > > >>> the
>>     > > >>>>>>>>>>>>> project
>>     > > >>>>>>>>>>>>>>> does
>>     > > >>>>>>>>>>>>>>>>>> not fit 1.11. The work can go into the next
>> release
>>     > > >>> instead.
>>     > > >>>>>>>>>>>>>>>>>>
>>     > > >>>>>>>>>>>>>>>>>> Thanks,
>>     > > >>>>>>>>>>>>>>>>>>
>>     > > >>>>>>>>>>>>>>>>>> - Paul
>>     > > >>>>>>>>>>>>>>>>>>
>>     > > >>>>>>>>>>>>>>>>>>
>>     > > >>>>>>>>>>>>>>>>>>
>>     > > >>>>>>>>>>>>>>>>>>
>>     > > >>>>>>>>>>>>>>>>>
>>     > > >>>>>>>>>>>>>>>>
>>     > > >>>>>>>>>>>>>>>>
>>     > > >>>>>>>>>>>>>>>
>>     > > >>>>>>>>>>>>>>
>>     > > >>>>>>>>>>>>>
>>     > > >>>>>>>>>>>
>>     > > >>>>>>>>>>>
>>     > > >>>>>>>>>>>
>>     > > >>>>>>>>>>>
>>     > > >>>>>>>>>
>>     > > >>>>>>>>>
>>     > > >>>>>>>>
>>     > > >>>>>>>
>>     > > >>>>>
>>     > > >>>>>
>>     > > >>>
>>     > > >>>
>>     > > >>
>>     > >
>>     > >
>>     >
>>
>>
>>
>

Re: [DISCUSS] Drill 1.11.0 release

Posted by Arina Yelchiyeva <ar...@gmail.com>.

Thanks, Boaz!

Unfortunately, I still see the failures connected to DRILL-5669.
I'll share the details.

Kind regards
Arina

On Wed, Jul 19, 2017 at 2:00 AM, Boaz Ben-Zvi <bb...@mapr.com> wrote:

> Hi Arina,
>
>      The three Jira’s are ready (+1, squashed, rebased, etc.):
>
> • DRILL-5616: Add memory checks, plus minor metrics changes
>     (Commit ID: c7ad56b7e075e84761896c9ce660b18b09d9e49a )
> • DRILL-5665: Add the option planner.force_2phase_aggr to override small
> inputs
>     (Commit ID: 1917ec084bdef74962370fbe5b253172c634c86e )
> • DRILL-5669: Add a configurable option for minimum memory allocation to
> buffered ops
>     (Commit ID: 4d7a3fe3e7ae52db57b0fd1f0619c6d1d64308c8 )
>
>   Thanks,
>
>          Boaz
>
> On 7/18/17, 3:04 AM, "Arina Yelchiyeva" <ar...@gmail.com>
> wrote:
>
>     Hi Parth,
>
>     no problem, I did not start preparing the release since there is one
>     blocker has left:
>
>     *DRILL-5669 - Boaz Ben-Zvi (fix will be ready in a couple of days)*
>
>
>     Kind regards
>     Arina
>
>     On Mon, Jul 17, 2017 at 11:26 PM, Parth Chandra <pa...@apache.org>
> wrote:
>
>     > Hi Arina,
>     >
>     >   I just realized that I did not push the C++ client fix (
> DRILL-5659) to
>     > master. I have just done so.
>     >
>     >   Sorry if I messed things up for the release, but we should get
> this fix
>     > in.
>     >
>     > Parth
>     >
>     > On Fri, Jul 14, 2017 at 8:15 AM, Paul Rogers <pr...@mapr.com>
> wrote:
>     >
>     > > I looked into one NPE on CSV. Turns out it is already fixed by
> previous
>     > > 1.11 PRs, so we’re good.
>     > >
>     > > There is another reported by a user on the user list, but I’ve not
> heard
>     > > back about getting sample data. Let’s not hold the release for
> that.
>     > >
>     > > - Paul
>     > >
>     > > > On Jul 14, 2017, at 2:38 AM, Arina Yelchiyeva <
>     > > arina.yelchiyeva@gmail.com> wrote:
>     > > >
>     > > > *Blockers*
>     > > > DRILL-5659 - Parth (done)
>     > > > DRILL-5669 - Boaz Ben-Zvi (fix will be ready in a couple of days)
>     > > >
>     > > > *Possible blockers*
>     > > > NPE on CSV - Paul can you please provide an update?
>     > > >
>     > > > Kind regards
>     > > > Arina
>     > > >
>     > > > On Thu, Jul 13, 2017 at 2:46 PM, Arina Yelchiyeva <
>     > > > arina.yelchiyeva@gmail.com> wrote:
>     > > >
>     > > >> *Blockers*
>     > > >> DRILL-5659 - problem is identified, Parth is working on the fix.
>     > > >>
>     > > >> *Not blockers*
>     > > >> DRILL-5660 - Vitalii
>     > > >> DRILL-5468 - Jinfeng
>     > > >>
>     > > >> *Possible blockers*
>     > > >> DRILL-5669 - Boaz Ben-Zvi / Paul
>     > > >> NPE on CSV - Paul
>     > > >>
>     > > >> Kind regards
>     > > >> Arina
>     > > >>
>     > > >>
>     > > >> On Thu, Jul 13, 2017 at 1:28 AM, Paul Rogers <pr...@mapr.com>
>     > wrote:
>     > > >>
>     > > >>> I withdraw my suggestion to fix DRILL-5660. Was a big hassle
> for us
>     > to
>     > > >>> track down the incompatibility the issue internally, but
> production
>     > > users
>     > > >>> probably don’t attempt to use one build of Drill with metadata
> files
>     > > >>> created from another. We can worry about the issue later only
> if a
>     > user
>     > > >>> actually runs into it.
>     > > >>>
>     > > >>> As suggested earlier, let’s just document this obscure case in
> a
>     > > release
>     > > >>> note.
>     > > >>>
>     > > >>> - Paul
>     > > >>>
>     > > >>>> On Jul 12, 2017, at 3:04 PM, Jinfeng Ni <jn...@apache.org>
> wrote:
>     > > >>>>
>     > > >>>> Upgrade from 1.10 to 1.11 is irreversible only when parquet
> metadata
>     > > >>> cache
>     > > >>>> file is involved.
>     > > >>>>
>     > > >>>> I might be wrong, but I thought that if someone is running on
>     > 1.10.0,
>     > > >>> and
>     > > >>>> he wants to go back to 1.0.0, where the first version of
> parquet
>     > > >>> meatadata
>     > > >>>> cache was introduced, can we guarantee that Drillbit would
> work
>     > > >>> correctly?
>     > > >>>> I'm trying to understand what DRILL-5660 asks for is a new
>     > > requirement,
>     > > >>> or
>     > > >>>> an requirement being there starting from Drill 1.0.0?
>     > > >>>>
>     > > >>>> Regression failures are normally regarded as release blocker.
>     > > >>>>
>     > > >>>>
>     > > >>>>
>     > > >>>> On Wed, Jul 12, 2017 at 2:54 PM, Paul Rogers <
> progers@mapr.com>
>     > > wrote:
>     > > >>>>
>     > > >>>>> If we don’t do DRILL-5660 for 1.11 then we don’t need to do
> it at
>     > > all,
>     > > >>> so
>     > > >>>>> that is a time savings. We could handle this via a release
> note
>     > item
>     > > >>> that
>     > > >>>>> simply states that an upgrade from 1.10 to 1.11 is
> irreversible: we
>     > > do
>     > > >>> not
>     > > >>>>> support the ability to fall back to the old version.
> Everyone OK
>     > with
>     > > >>> that?
>     > > >>>>>
>     > > >>>>> We are seeing some stress test regression failures that, I’m
> told,
>     > > will
>     > > >>>>> show up as JIRA entries shortly.
>     > > >>>>>
>     > > >>>>> - Paul
>     > > >>>>>
>     > > >>>>>> On Jul 12, 2017, at 11:05 AM, Jinfeng Ni <jn...@apache.org>
> wrote:
>     > > >>>>>>
>     > > >>>>>> I put some analysis in DRILL-5468.
>     > > >>>>>>
>     > > >>>>>> Also, IMHO, DRILL-5660 is not a release blocker.
>     > > >>>>>>
>     > > >>>>>>
>     > > >>>>>>
>     > > >>>>>> On Tue, Jul 11, 2017 at 11:27 AM, Parth Chandra <
>     > parthc@apache.org>
>     > > >>>>> wrote:
>     > > >>>>>>
>     > > >>>>>>> Re DRILL-5634:
>     > > >>>>>>> From the Apache commons web page [1] -
>     > > >>>>>>>
>     > > >>>>>>> Please note that Apache Commons Crypto doesn't implement
> the
>     > > >>>>> cryptographic
>     > > >>>>>>> algorithm such as AES directly.* It wraps to Openssl or
> JCE which
>     > > >>>>> implement
>     > > >>>>>>> the algorithms.*
>     > > >>>>>>>
>     > > >>>>>>>
>     > > >>>>>>> Since we are cleared for using Openssl and JCE, we are OK
> to
>     > > include
>     > > >>> the
>     > > >>>>>>> AES code in the UDFs.
>     > > >>>>>>>
>     > > >>>>>>>
>     > > >>>>>>> Parth
>     > > >>>>>>>
>     > > >>>>>>> [1] https://commons.apache.org/proper/commons-crypto/
>     > > >>>>>>>
>     > > >>>>>>> On Tue, Jul 11, 2017 at 10:56 AM, Arina Yelchiyeva <
>     > > >>>>>>> arina.yelchiyeva@gmail.com> wrote:
>     > > >>>>>>>
>     > > >>>>>>>> During today hangouts we agreed on approximate cut-off
> date - 14
>     > > >>> July,
>     > > >>>>>>> 2017
>     > > >>>>>>>> (date may be shifted due to the blockers).
>     > > >>>>>>>> I'll start preparing test RC on Thursday to solve any
> release
>     > > >>>>> preparation
>     > > >>>>>>>> issues beforehand.
>     > > >>>>>>>>
>     > > >>>>>>>> *Blockers*
>     > > >>>>>>>> DRILL-5660 - Vitalii
>     > > >>>>>>>>
>     > > >>>>>>>> *Possible blockers*
>     > > >>>>>>>> DRILL-5468 - Jinfeng
>     > > >>>>>>>> NPE on CSV source - Paul
>     > > >>>>>>>> DRILL-5659 - Parth
>     > > >>>>>>>>
>     > > >>>>>>>> *Will be included if done before the cut-off date*
>     > > >>>>>>>> DRILL-5616 / DRILL-5665 - Boaz
>     > > >>>>>>>> DRILL-5634 - Parth will take a look at Apache commons
>     > > implementation
>     > > >>>>> and
>     > > >>>>>>>> update.
>     > > >>>>>>>>
>     > > >>>>>>>> Kind regards
>     > > >>>>>>>> Arina
>     > > >>>>>>>>
>     > > >>>>>>>> On Tue, Jul 11, 2017 at 3:14 PM, Charles Givre <
>     > cgivre@gmail.com>
>     > > >>>>> wrote:
>     > > >>>>>>>>
>     > > >>>>>>>>> Hi Arina,
>     > > >>>>>>>>> I can’t make the hangout, but we have a few options.
>     > Basically,
>     > > >>> since
>     > > >>>>>>>> all
>     > > >>>>>>>>> the hashing algorithms have no export restrictions, we
> could:
>     > > >>>>>>>>> 1.  Remove the AES encrypt/decrypt until I can figure
> out what
>     > we
>     > > >>> have
>     > > >>>>>>> to
>     > > >>>>>>>>> do (and do it)
>     > > >>>>>>>>> 2.  I can try to research it today or tomorrow to figure
> out
>     > > >>> exactly
>     > > >>>>>>> what
>     > > >>>>>>>>> we have to do to include AES.
>     > > >>>>>>>>> 3.  Remove the entire package
>     > > >>>>>>>>>
>     > > >>>>>>>>> I’d like to see the hashing functions at least included
> because
>     > > >>> we’ve
>     > > >>>>>>> had
>     > > >>>>>>>>> a lot of requests for that functionality.   I also don’t
> want
>     > > this
>     > > >>> to
>     > > >>>>>>>> hold
>     > > >>>>>>>>> things up, so please let me know your preference.
> Regardless,
>     > > I’m
>     > > >>>>>>> going
>     > > >>>>>>>> to
>     > > >>>>>>>>> start trying to figure out the restrictions so we’ll
> know for
>     > the
>     > > >>>>>>> future.
>     > > >>>>>>>>>
>     > > >>>>>>>>> Best,
>     > > >>>>>>>>> — C
>     > > >>>>>>>>>
>     > > >>>>>>>>>
>     > > >>>>>>>>>
>     > > >>>>>>>>>> On Jul 11, 2017, at 05:58, Arina Yelchiyeva <
>     > > >>>>>>>> arina.yelchiyeva@gmail.com>
>     > > >>>>>>>>> wrote:
>     > > >>>>>>>>>>
>     > > >>>>>>>>>> @Jinfeng,
>     > > >>>>>>>>>> Let's discuss today on Drill hangout final cut-off date.
>     > > >>>>>>>>>>
>     > > >>>>>>>>>> @Charles
>     > > >>>>>>>>>> Since there were questions about AES in DRILL-5634,
> should we
>     > > >>> exclude
>     > > >>>>>>>> it
>     > > >>>>>>>>>> from the candidates for 1.11.0 release?
>     > > >>>>>>>>>>
>     > > >>>>>>>>>> @Boaz
>     > > >>>>>>>>>> Do we have any ETA when DRILL-5616 and DRILL-5665 will
> pass
>     > code
>     > > >>>>>>>> review?
>     > > >>>>>>>>>>
>     > > >>>>>>>>>> Also I believe we should include DRILL-5660 where we
> need to
>     > > bump
>     > > >>> up
>     > > >>>>>>>>>> parquet metadata version.
>     > > >>>>>>>>>> @Vitalii
>     > > >>>>>>>>>> Do you have any ETA for the fix?
>     > > >>>>>>>>>>
>     > > >>>>>>>>>> Kind regards
>     > > >>>>>>>>>> Arina
>     > > >>>>>>>>>>
>     > > >>>>>>>>>> On Tue, Jul 11, 2017 at 6:02 AM, Boaz Ben-Zvi <
>     > > bben-zvi@mapr.com>
>     > > >>>>>>>> wrote:
>     > > >>>>>>>>>>
>     > > >>>>>>>>>>> Hi Arina,
>     > > >>>>>>>>>>>
>     > > >>>>>>>>>>>  Two PR have just been submitted, relating to the Hash
> Aggr
>     > > >>>>>>> Spill:
>     > > >>>>>>>>>>> DRILL-5616 and DRILL-5665 .
>     > > >>>>>>>>>>> Can these be considered for 1.11 as well ?
>     > > >>>>>>>>>>>
>     > > >>>>>>>>>>>  Thanks,
>     > > >>>>>>>>>>>
>     > > >>>>>>>>>>>            Boaz
>     > > >>>>>>>>>>>
>     > > >>>>>>>>>>> On 7/10/17, 1:49 PM, "Paul Rogers" <pr...@mapr.com>
> wrote:
>     > > >>>>>>>>>>>
>     > > >>>>>>>>>>> DRILL-5601 is no longer a blocker for 1.11; we’ll
> defer that
>     > > >>>>>>> until
>     > > >>>>>>>>>>> after the release to allow others more time to work on
> other
>     > > >>>>>>> features
>     > > >>>>>>>>>>> rather than doing code reviews.
>     > > >>>>>>>>>>>
>     > > >>>>>>>>>>> - Paul
>     > > >>>>>>>>>>>
>     > > >>>>>>>>>>>> On Jul 10, 2017, at 1:43 PM, Jinfeng Ni <
> jni@apache.org>
>     > > wrote:
>     > > >>>>>>>>>>>>
>     > > >>>>>>>>>>>> Hi Arina,
>     > > >>>>>>>>>>>>
>     > > >>>>>>>>>>>> It has been a while since we discussed 1.11.0
> release.  Any
>     > > >>> update
>     > > >>>>>>>>>>> on the
>     > > >>>>>>>>>>>> new release plan?  Do we have a cut-off date ?
>     > > >>>>>>>>>>>>
>     > > >>>>>>>>>>>> In the past, I believe Drill release is a
> time-bounded, not
>     > > >>> feature
>     > > >>>>>>>>>>>> bounded. If certain fixes/features could not make it
> within
>     > a
>     > > >>>>>>> certain
>     > > >>>>>>>>>>>> time-frame, we had better defer them into future
> release,
>     > > unless
>     > > >>>>>>>>>>> those
>     > > >>>>>>>>>>>> fixes/features are regarded as blocking issues for a
> new
>     > > >>> release.
>     > > >>>>>>>>>>>>
>     > > >>>>>>>>>>>>
>     > > >>>>>>>>>>>>
>     > > >>>>>>>>>>>>
>     > > >>>>>>>>>>>>
>     > > >>>>>>>>>>>>
>     > > >>>>>>>>>>>>
>     > > >>>>>>>>>>>> On Tue, Jul 4, 2017 at 5:14 AM, Arina Yelchiyeva <
>     > > >>>>>>>>>>> arina.yelchiyeva@gmail.com
>     > > >>>>>>>>>>>>> wrote:
>     > > >>>>>>>>>>>>
>     > > >>>>>>>>>>>>> Latest update:
>     > > >>>>>>>>>>>>>
>     > > >>>>>>>>>>>>> We wait for the following Jiras before final cut off.
>     > > >>>>>>>>>>>>> 1. DRILL-5601  - Paul
>     > > >>>>>>>>>>>>> 2. DRILL-5420 - Kunal
>     > > >>>>>>>>>>>>> 3. DRILL-5634 - Charles
>     > > >>>>>>>>>>>>>
>     > > >>>>>>>>>>>>> Kind regards
>     > > >>>>>>>>>>>>> Arina
>     > > >>>>>>>>>>>>>
>     > > >>>>>>>>>>>>> On Fri, Jun 30, 2017 at 9:29 PM, Kunal Khatua <
>     > > >>> kkhatua@mapr.com>
>     > > >>>>>>>>>>> wrote:
>     > > >>>>>>>>>>>>>
>     > > >>>>>>>>>>>>>> Parth's already +1'd the process with a minor
> update. So,
>     > > >>> we're
>     > > >>>>>>>>>>> good to
>     > > >>>>>>>>>>>>> go
>     > > >>>>>>>>>>>>>> on DRILL-5420. I'll change the label on that to
>     > > >>> ready-to-commit.
>     > > >>>>>>>>>>>>>>
>     > > >>>>>>>>>>>>>> <http://www.mapr.com/>
>     > > >>>>>>>>>>>>>>
>     > > >>>>>>>>>>>>>> ________________________________
>     > > >>>>>>>>>>>>>> From: Parth Chandra <pa...@apache.org>
>     > > >>>>>>>>>>>>>> Sent: Thursday, June 29, 2017 9:44:39 AM
>     > > >>>>>>>>>>>>>> To: dev@drill.apache.org
>     > > >>>>>>>>>>>>>> Subject: Re: [DISCUSS] Drill 1.11.0 release
>     > > >>>>>>>>>>>>>>
>     > > >>>>>>>>>>>>>> I'm Ok with the pcap format plugin if Pauls
> comments are
>     > > also
>     > > >>>>>>>>>>> addressed.
>     > > >>>>>>>>>>>>> If
>     > > >>>>>>>>>>>>>> the changes are done before your cutoff deadline,we
> can
>     > > merge
>     > > >>> it
>     > > >>>>>>>>>>> in.
>     > > >>>>>>>>>>>>>>
>     > > >>>>>>>>>>>>>> On Thu, Jun 29, 2017 at 12:26 AM, Arina Yelchiyeva <
>     > > >>>>>>>>>>>>>> arina.yelchiyeva@gmail.com> wrote:
>     > > >>>>>>>>>>>>>>
>     > > >>>>>>>>>>>>>>> Hi all,
>     > > >>>>>>>>>>>>>>>
>     > > >>>>>>>>>>>>>>> here is the list of Jira we are waiting before
> making
>     > final
>     > > >>> cut
>     > > >>>>>>>>>>> off:
>     > > >>>>>>>>>>>>>>> 1. DRILL-5601  - Paul
>     > > >>>>>>>>>>>>>>> 2. DRILL-3640  - Kunal
>     > > >>>>>>>>>>>>>>> 3. DRILL-5420 - Kunal
>     > > >>>>>>>>>>>>>>>
>     > > >>>>>>>>>>>>>>> Please let me know if there are any others Jiras we
>     > should
>     > > >>> wait
>     > > >>>>>>>>>>> for.
>     > > >>>>>>>>>>>>>>>
>     > > >>>>>>>>>>>>>>> @Charles,
>     > > >>>>>>>>>>>>>>> you have mentioned that you want to include some
> UDFs in
>     > > >>> Drill
>     > > >>>>>>>>>>> 1.11, do
>     > > >>>>>>>>>>>>>> you
>     > > >>>>>>>>>>>>>>> have any ETA for the PR?
>     > > >>>>>>>>>>>>>>>
>     > > >>>>>>>>>>>>>>> @Paul / Parth
>     > > >>>>>>>>>>>>>>> I see there is active CR for pcap format PR, do we
> want
>     > to
>     > > >>>>>>>>>>> include it
>     > > >>>>>>>>>>>>> in
>     > > >>>>>>>>>>>>>>> this release as well?
>     > > >>>>>>>>>>>>>>>
>     > > >>>>>>>>>>>>>>> Kind regards
>     > > >>>>>>>>>>>>>>> Arina
>     > > >>>>>>>>>>>>>>>
>     > > >>>>>>>>>>>>>>>
>     > > >>>>>>>>>>>>>>> On Thu, Jun 29, 2017 at 10:05 AM, Arina Yelchiyeva
> <
>     > > >>>>>>>>>>>>>>> arina.yelchiyeva@gmail.com> wrote:
>     > > >>>>>>>>>>>>>>>
>     > > >>>>>>>>>>>>>>>> Parth, thanks a lot the instructions.
>     > > >>>>>>>>>>>>>>>>
>     > > >>>>>>>>>>>>>>>>
>     > > >>>>>>>>>>>>>>>>
>     > > >>>>>>>>>>>>>>>> On Wed, Jun 28, 2017 at 8:24 PM, Parth Chandra <
>     > > >>>>>>>>>>> parthc@apache.org>
>     > > >>>>>>>>>>>>>>> wrote:
>     > > >>>>>>>>>>>>>>>>
>     > > >>>>>>>>>>>>>>>>> FWIW, I put release instructions culled from
> gists
>     > > written
>     > > >>> by
>     > > >>>>>>>>>>>>> previous
>     > > >>>>>>>>>>>>>>>>> release managers into an updated gist here [1]
>     > > >>>>>>>>>>>>>>>>>
>     > > >>>>>>>>>>>>>>>>> HTH
>     > > >>>>>>>>>>>>>>>>>
>     > > >>>>>>>>>>>>>>>>> Parth
>     > > >>>>>>>>>>>>>>>>>
>     > > >>>>>>>>>>>>>>>>> [1] https://github.com/
> parthchandra/drill/wiki/Drill-
>     > > >>>>>>>>>>>>> Release-Process
>     > > >>>>>>>>>>>>>>>>> <https://github.com/
> parthchandra/drill/wiki/Drill-
>     > > >>>>>>>>>>> Release-Process>
>     > > >>>>>>>>>>>>>>>>>
>     > > >>>>>>>>>>>>>>>>>
>     > > >>>>>>>>>>>>>>>>>
>     > > >>>>>>>>>>>>>>>>> On Tue, Jun 27, 2017 at 5:24 PM, Paul Rogers <
>     > > >>>>>>> progers@mapr.com>
>     > > >>>>>>>>>>>>>> wrote:
>     > > >>>>>>>>>>>>>>>>>
>     > > >>>>>>>>>>>>>>>>>> Hi Arina,
>     > > >>>>>>>>>>>>>>>>>>
>     > > >>>>>>>>>>>>>>>>>> I have two projects in progress: one is a
> candidate
>     > for
>     > > >>> 1.11,
>     > > >>>>>>>>>>> the
>     > > >>>>>>>>>>>>>>> other
>     > > >>>>>>>>>>>>>>>>> is
>     > > >>>>>>>>>>>>>>>>>> not.
>     > > >>>>>>>>>>>>>>>>>>
>     > > >>>>>>>>>>>>>>>>>> The “managed external sort” is the good
> candidate for
>     > > >>> 1.11:
>     > > >>>>>>>>>>> The PR
>     > > >>>>>>>>>>>>>> for
>     > > >>>>>>>>>>>>>>>>>> DRILL-5601 is being reviewed. Getting that in
> will
>     > allow
>     > > >>> us
>     > > >>>>>>> to
>     > > >>>>>>>>>>>>>> enable
>     > > >>>>>>>>>>>>>>>>> the
>     > > >>>>>>>>>>>>>>>>>> “new” external sort by default.
>     > > >>>>>>>>>>>>>>>>>>
>     > > >>>>>>>>>>>>>>>>>> The other project is to address Drill’s memory
>     > > >>> fragmentation
>     > > >>>>>>>>>>> issue
>     > > >>>>>>>>>>>>>>> with
>     > > >>>>>>>>>>>>>>>>>> vectors over 16 MB in size. Two PRs have been
> open
>     > for a
>     > > >>>>>>> while.
>     > > >>>>>>>>>>>>> They
>     > > >>>>>>>>>>>>>>> are
>     > > >>>>>>>>>>>>>>>>>> the first of several. Each takes a while to
> review.
>     > So,
>     > > >>> the
>     > > >>>>>>>>>>>>> project
>     > > >>>>>>>>>>>>>>> does
>     > > >>>>>>>>>>>>>>>>>> not fit 1.11. The work can go into the next
> release
>     > > >>> instead.
>     > > >>>>>>>>>>>>>>>>>>
>     > > >>>>>>>>>>>>>>>>>> Thanks,
>     > > >>>>>>>>>>>>>>>>>>
>     > > >>>>>>>>>>>>>>>>>> - Paul
>     > > >>>>>>>>>>>>>>>>>>
>     > > >>>>>>>>>>>>>>>>>>
>     > > >>>>>>>>>>>>>>>>>>
>     > > >>>>>>>>>>>>>>>>>>
>     > > >>>>>>>>>>>>>>>>>
>     > > >>>>>>>>>>>>>>>>
>     > > >>>>>>>>>>>>>>>>
>     > > >>>>>>>>>>>>>>>
>     > > >>>>>>>>>>>>>>
>     > > >>>>>>>>>>>>>
>     > > >>>>>>>>>>>
>     > > >>>>>>>>>>>
>     > > >>>>>>>>>>>
>     > > >>>>>>>>>>>
>     > > >>>>>>>>>
>     > > >>>>>>>>>
>     > > >>>>>>>>
>     > > >>>>>>>
>     > > >>>>>
>     > > >>>>>
>     > > >>>
>     > > >>>
>     > > >>
>     > >
>     > >
>     >
>
>
>

Re: [DISCUSS] Drill 1.11.0 release

Posted by Boaz Ben-Zvi <bb...@mapr.com>.

Hi Arina,

     The three Jira’s are ready (+1, squashed, rebased, etc.):

• DRILL-5616: Add memory checks, plus minor metrics changes
    (Commit ID: c7ad56b7e075e84761896c9ce660b18b09d9e49a )
• DRILL-5665: Add the option planner.force_2phase_aggr to override small inputs
    (Commit ID: 1917ec084bdef74962370fbe5b253172c634c86e )
• DRILL-5669: Add a configurable option for minimum memory allocation to buffered ops
    (Commit ID: 4d7a3fe3e7ae52db57b0fd1f0619c6d1d64308c8 )

  Thanks,

         Boaz 

On 7/18/17, 3:04 AM, "Arina Yelchiyeva" <ar...@gmail.com> wrote:

    Hi Parth,
    
    no problem, I did not start preparing the release since there is one
    blocker has left:
    
    *DRILL-5669 - Boaz Ben-Zvi (fix will be ready in a couple of days)*
    
    
    Kind regards
    Arina
    
    On Mon, Jul 17, 2017 at 11:26 PM, Parth Chandra <pa...@apache.org> wrote:
    
    > Hi Arina,
    >
    >   I just realized that I did not push the C++ client fix ( DRILL-5659) to
    > master. I have just done so.
    >
    >   Sorry if I messed things up for the release, but we should get this fix
    > in.
    >
    > Parth
    >
    > On Fri, Jul 14, 2017 at 8:15 AM, Paul Rogers <pr...@mapr.com> wrote:
    >
    > > I looked into one NPE on CSV. Turns out it is already fixed by previous
    > > 1.11 PRs, so we’re good.
    > >
    > > There is another reported by a user on the user list, but I’ve not heard
    > > back about getting sample data. Let’s not hold the release for that.
    > >
    > > - Paul
    > >
    > > > On Jul 14, 2017, at 2:38 AM, Arina Yelchiyeva <
    > > arina.yelchiyeva@gmail.com> wrote:
    > > >
    > > > *Blockers*
    > > > DRILL-5659 - Parth (done)
    > > > DRILL-5669 - Boaz Ben-Zvi (fix will be ready in a couple of days)
    > > >
    > > > *Possible blockers*
    > > > NPE on CSV - Paul can you please provide an update?
    > > >
    > > > Kind regards
    > > > Arina
    > > >
    > > > On Thu, Jul 13, 2017 at 2:46 PM, Arina Yelchiyeva <
    > > > arina.yelchiyeva@gmail.com> wrote:
    > > >
    > > >> *Blockers*
    > > >> DRILL-5659 - problem is identified, Parth is working on the fix.
    > > >>
    > > >> *Not blockers*
    > > >> DRILL-5660 - Vitalii
    > > >> DRILL-5468 - Jinfeng
    > > >>
    > > >> *Possible blockers*
    > > >> DRILL-5669 - Boaz Ben-Zvi / Paul
    > > >> NPE on CSV - Paul
    > > >>
    > > >> Kind regards
    > > >> Arina
    > > >>
    > > >>
    > > >> On Thu, Jul 13, 2017 at 1:28 AM, Paul Rogers <pr...@mapr.com>
    > wrote:
    > > >>
    > > >>> I withdraw my suggestion to fix DRILL-5660. Was a big hassle for us
    > to
    > > >>> track down the incompatibility the issue internally, but production
    > > users
    > > >>> probably don’t attempt to use one build of Drill with metadata files
    > > >>> created from another. We can worry about the issue later only if a
    > user
    > > >>> actually runs into it.
    > > >>>
    > > >>> As suggested earlier, let’s just document this obscure case in a
    > > release
    > > >>> note.
    > > >>>
    > > >>> - Paul
    > > >>>
    > > >>>> On Jul 12, 2017, at 3:04 PM, Jinfeng Ni <jn...@apache.org> wrote:
    > > >>>>
    > > >>>> Upgrade from 1.10 to 1.11 is irreversible only when parquet metadata
    > > >>> cache
    > > >>>> file is involved.
    > > >>>>
    > > >>>> I might be wrong, but I thought that if someone is running on
    > 1.10.0,
    > > >>> and
    > > >>>> he wants to go back to 1.0.0, where the first version of parquet
    > > >>> meatadata
    > > >>>> cache was introduced, can we guarantee that Drillbit would work
    > > >>> correctly?
    > > >>>> I'm trying to understand what DRILL-5660 asks for is a new
    > > requirement,
    > > >>> or
    > > >>>> an requirement being there starting from Drill 1.0.0?
    > > >>>>
    > > >>>> Regression failures are normally regarded as release blocker.
    > > >>>>
    > > >>>>
    > > >>>>
    > > >>>> On Wed, Jul 12, 2017 at 2:54 PM, Paul Rogers <pr...@mapr.com>
    > > wrote:
    > > >>>>
    > > >>>>> If we don’t do DRILL-5660 for 1.11 then we don’t need to do it at
    > > all,
    > > >>> so
    > > >>>>> that is a time savings. We could handle this via a release note
    > item
    > > >>> that
    > > >>>>> simply states that an upgrade from 1.10 to 1.11 is irreversible: we
    > > do
    > > >>> not
    > > >>>>> support the ability to fall back to the old version. Everyone OK
    > with
    > > >>> that?
    > > >>>>>
    > > >>>>> We are seeing some stress test regression failures that, I’m told,
    > > will
    > > >>>>> show up as JIRA entries shortly.
    > > >>>>>
    > > >>>>> - Paul
    > > >>>>>
    > > >>>>>> On Jul 12, 2017, at 11:05 AM, Jinfeng Ni <jn...@apache.org> wrote:
    > > >>>>>>
    > > >>>>>> I put some analysis in DRILL-5468.
    > > >>>>>>
    > > >>>>>> Also, IMHO, DRILL-5660 is not a release blocker.
    > > >>>>>>
    > > >>>>>>
    > > >>>>>>
    > > >>>>>> On Tue, Jul 11, 2017 at 11:27 AM, Parth Chandra <
    > parthc@apache.org>
    > > >>>>> wrote:
    > > >>>>>>
    > > >>>>>>> Re DRILL-5634:
    > > >>>>>>> From the Apache commons web page [1] -
    > > >>>>>>>
    > > >>>>>>> Please note that Apache Commons Crypto doesn't implement the
    > > >>>>> cryptographic
    > > >>>>>>> algorithm such as AES directly.* It wraps to Openssl or JCE which
    > > >>>>> implement
    > > >>>>>>> the algorithms.*
    > > >>>>>>>
    > > >>>>>>>
    > > >>>>>>> Since we are cleared for using Openssl and JCE, we are OK to
    > > include
    > > >>> the
    > > >>>>>>> AES code in the UDFs.
    > > >>>>>>>
    > > >>>>>>>
    > > >>>>>>> Parth
    > > >>>>>>>
    > > >>>>>>> [1] https://commons.apache.org/proper/commons-crypto/
    > > >>>>>>>
    > > >>>>>>> On Tue, Jul 11, 2017 at 10:56 AM, Arina Yelchiyeva <
    > > >>>>>>> arina.yelchiyeva@gmail.com> wrote:
    > > >>>>>>>
    > > >>>>>>>> During today hangouts we agreed on approximate cut-off date - 14
    > > >>> July,
    > > >>>>>>> 2017
    > > >>>>>>>> (date may be shifted due to the blockers).
    > > >>>>>>>> I'll start preparing test RC on Thursday to solve any release
    > > >>>>> preparation
    > > >>>>>>>> issues beforehand.
    > > >>>>>>>>
    > > >>>>>>>> *Blockers*
    > > >>>>>>>> DRILL-5660 - Vitalii
    > > >>>>>>>>
    > > >>>>>>>> *Possible blockers*
    > > >>>>>>>> DRILL-5468 - Jinfeng
    > > >>>>>>>> NPE on CSV source - Paul
    > > >>>>>>>> DRILL-5659 - Parth
    > > >>>>>>>>
    > > >>>>>>>> *Will be included if done before the cut-off date*
    > > >>>>>>>> DRILL-5616 / DRILL-5665 - Boaz
    > > >>>>>>>> DRILL-5634 - Parth will take a look at Apache commons
    > > implementation
    > > >>>>> and
    > > >>>>>>>> update.
    > > >>>>>>>>
    > > >>>>>>>> Kind regards
    > > >>>>>>>> Arina
    > > >>>>>>>>
    > > >>>>>>>> On Tue, Jul 11, 2017 at 3:14 PM, Charles Givre <
    > cgivre@gmail.com>
    > > >>>>> wrote:
    > > >>>>>>>>
    > > >>>>>>>>> Hi Arina,
    > > >>>>>>>>> I can’t make the hangout, but we have a few options.
    > Basically,
    > > >>> since
    > > >>>>>>>> all
    > > >>>>>>>>> the hashing algorithms have no export restrictions, we could:
    > > >>>>>>>>> 1.  Remove the AES encrypt/decrypt until I can figure out what
    > we
    > > >>> have
    > > >>>>>>> to
    > > >>>>>>>>> do (and do it)
    > > >>>>>>>>> 2.  I can try to research it today or tomorrow to figure out
    > > >>> exactly
    > > >>>>>>> what
    > > >>>>>>>>> we have to do to include AES.
    > > >>>>>>>>> 3.  Remove the entire package
    > > >>>>>>>>>
    > > >>>>>>>>> I’d like to see the hashing functions at least included because
    > > >>> we’ve
    > > >>>>>>> had
    > > >>>>>>>>> a lot of requests for that functionality.   I also don’t want
    > > this
    > > >>> to
    > > >>>>>>>> hold
    > > >>>>>>>>> things up, so please let me know your preference.  Regardless,
    > > I’m
    > > >>>>>>> going
    > > >>>>>>>> to
    > > >>>>>>>>> start trying to figure out the restrictions so we’ll know for
    > the
    > > >>>>>>> future.
    > > >>>>>>>>>
    > > >>>>>>>>> Best,
    > > >>>>>>>>> — C
    > > >>>>>>>>>
    > > >>>>>>>>>
    > > >>>>>>>>>
    > > >>>>>>>>>> On Jul 11, 2017, at 05:58, Arina Yelchiyeva <
    > > >>>>>>>> arina.yelchiyeva@gmail.com>
    > > >>>>>>>>> wrote:
    > > >>>>>>>>>>
    > > >>>>>>>>>> @Jinfeng,
    > > >>>>>>>>>> Let's discuss today on Drill hangout final cut-off date.
    > > >>>>>>>>>>
    > > >>>>>>>>>> @Charles
    > > >>>>>>>>>> Since there were questions about AES in DRILL-5634, should we
    > > >>> exclude
    > > >>>>>>>> it
    > > >>>>>>>>>> from the candidates for 1.11.0 release?
    > > >>>>>>>>>>
    > > >>>>>>>>>> @Boaz
    > > >>>>>>>>>> Do we have any ETA when DRILL-5616 and DRILL-5665 will pass
    > code
    > > >>>>>>>> review?
    > > >>>>>>>>>>
    > > >>>>>>>>>> Also I believe we should include DRILL-5660 where we need to
    > > bump
    > > >>> up
    > > >>>>>>>>>> parquet metadata version.
    > > >>>>>>>>>> @Vitalii
    > > >>>>>>>>>> Do you have any ETA for the fix?
    > > >>>>>>>>>>
    > > >>>>>>>>>> Kind regards
    > > >>>>>>>>>> Arina
    > > >>>>>>>>>>
    > > >>>>>>>>>> On Tue, Jul 11, 2017 at 6:02 AM, Boaz Ben-Zvi <
    > > bben-zvi@mapr.com>
    > > >>>>>>>> wrote:
    > > >>>>>>>>>>
    > > >>>>>>>>>>> Hi Arina,
    > > >>>>>>>>>>>
    > > >>>>>>>>>>>  Two PR have just been submitted, relating to the Hash Aggr
    > > >>>>>>> Spill:
    > > >>>>>>>>>>> DRILL-5616 and DRILL-5665 .
    > > >>>>>>>>>>> Can these be considered for 1.11 as well ?
    > > >>>>>>>>>>>
    > > >>>>>>>>>>>  Thanks,
    > > >>>>>>>>>>>
    > > >>>>>>>>>>>            Boaz
    > > >>>>>>>>>>>
    > > >>>>>>>>>>> On 7/10/17, 1:49 PM, "Paul Rogers" <pr...@mapr.com> wrote:
    > > >>>>>>>>>>>
    > > >>>>>>>>>>> DRILL-5601 is no longer a blocker for 1.11; we’ll defer that
    > > >>>>>>> until
    > > >>>>>>>>>>> after the release to allow others more time to work on other
    > > >>>>>>> features
    > > >>>>>>>>>>> rather than doing code reviews.
    > > >>>>>>>>>>>
    > > >>>>>>>>>>> - Paul
    > > >>>>>>>>>>>
    > > >>>>>>>>>>>> On Jul 10, 2017, at 1:43 PM, Jinfeng Ni <jn...@apache.org>
    > > wrote:
    > > >>>>>>>>>>>>
    > > >>>>>>>>>>>> Hi Arina,
    > > >>>>>>>>>>>>
    > > >>>>>>>>>>>> It has been a while since we discussed 1.11.0 release.  Any
    > > >>> update
    > > >>>>>>>>>>> on the
    > > >>>>>>>>>>>> new release plan?  Do we have a cut-off date ?
    > > >>>>>>>>>>>>
    > > >>>>>>>>>>>> In the past, I believe Drill release is a time-bounded, not
    > > >>> feature
    > > >>>>>>>>>>>> bounded. If certain fixes/features could not make it within
    > a
    > > >>>>>>> certain
    > > >>>>>>>>>>>> time-frame, we had better defer them into future release,
    > > unless
    > > >>>>>>>>>>> those
    > > >>>>>>>>>>>> fixes/features are regarded as blocking issues for a new
    > > >>> release.
    > > >>>>>>>>>>>>
    > > >>>>>>>>>>>>
    > > >>>>>>>>>>>>
    > > >>>>>>>>>>>>
    > > >>>>>>>>>>>>
    > > >>>>>>>>>>>>
    > > >>>>>>>>>>>>
    > > >>>>>>>>>>>> On Tue, Jul 4, 2017 at 5:14 AM, Arina Yelchiyeva <
    > > >>>>>>>>>>> arina.yelchiyeva@gmail.com
    > > >>>>>>>>>>>>> wrote:
    > > >>>>>>>>>>>>
    > > >>>>>>>>>>>>> Latest update:
    > > >>>>>>>>>>>>>
    > > >>>>>>>>>>>>> We wait for the following Jiras before final cut off.
    > > >>>>>>>>>>>>> 1. DRILL-5601  - Paul
    > > >>>>>>>>>>>>> 2. DRILL-5420 - Kunal
    > > >>>>>>>>>>>>> 3. DRILL-5634 - Charles
    > > >>>>>>>>>>>>>
    > > >>>>>>>>>>>>> Kind regards
    > > >>>>>>>>>>>>> Arina
    > > >>>>>>>>>>>>>
    > > >>>>>>>>>>>>> On Fri, Jun 30, 2017 at 9:29 PM, Kunal Khatua <
    > > >>> kkhatua@mapr.com>
    > > >>>>>>>>>>> wrote:
    > > >>>>>>>>>>>>>
    > > >>>>>>>>>>>>>> Parth's already +1'd the process with a minor update. So,
    > > >>> we're
    > > >>>>>>>>>>> good to
    > > >>>>>>>>>>>>> go
    > > >>>>>>>>>>>>>> on DRILL-5420. I'll change the label on that to
    > > >>> ready-to-commit.
    > > >>>>>>>>>>>>>>
    > > >>>>>>>>>>>>>> <http://www.mapr.com/>
    > > >>>>>>>>>>>>>>
    > > >>>>>>>>>>>>>> ________________________________
    > > >>>>>>>>>>>>>> From: Parth Chandra <pa...@apache.org>
    > > >>>>>>>>>>>>>> Sent: Thursday, June 29, 2017 9:44:39 AM
    > > >>>>>>>>>>>>>> To: dev@drill.apache.org
    > > >>>>>>>>>>>>>> Subject: Re: [DISCUSS] Drill 1.11.0 release
    > > >>>>>>>>>>>>>>
    > > >>>>>>>>>>>>>> I'm Ok with the pcap format plugin if Pauls comments are
    > > also
    > > >>>>>>>>>>> addressed.
    > > >>>>>>>>>>>>> If
    > > >>>>>>>>>>>>>> the changes are done before your cutoff deadline,we can
    > > merge
    > > >>> it
    > > >>>>>>>>>>> in.
    > > >>>>>>>>>>>>>>
    > > >>>>>>>>>>>>>> On Thu, Jun 29, 2017 at 12:26 AM, Arina Yelchiyeva <
    > > >>>>>>>>>>>>>> arina.yelchiyeva@gmail.com> wrote:
    > > >>>>>>>>>>>>>>
    > > >>>>>>>>>>>>>>> Hi all,
    > > >>>>>>>>>>>>>>>
    > > >>>>>>>>>>>>>>> here is the list of Jira we are waiting before making
    > final
    > > >>> cut
    > > >>>>>>>>>>> off:
    > > >>>>>>>>>>>>>>> 1. DRILL-5601  - Paul
    > > >>>>>>>>>>>>>>> 2. DRILL-3640  - Kunal
    > > >>>>>>>>>>>>>>> 3. DRILL-5420 - Kunal
    > > >>>>>>>>>>>>>>>
    > > >>>>>>>>>>>>>>> Please let me know if there are any others Jiras we
    > should
    > > >>> wait
    > > >>>>>>>>>>> for.
    > > >>>>>>>>>>>>>>>
    > > >>>>>>>>>>>>>>> @Charles,
    > > >>>>>>>>>>>>>>> you have mentioned that you want to include some UDFs in
    > > >>> Drill
    > > >>>>>>>>>>> 1.11, do
    > > >>>>>>>>>>>>>> you
    > > >>>>>>>>>>>>>>> have any ETA for the PR?
    > > >>>>>>>>>>>>>>>
    > > >>>>>>>>>>>>>>> @Paul / Parth
    > > >>>>>>>>>>>>>>> I see there is active CR for pcap format PR, do we want
    > to
    > > >>>>>>>>>>> include it
    > > >>>>>>>>>>>>> in
    > > >>>>>>>>>>>>>>> this release as well?
    > > >>>>>>>>>>>>>>>
    > > >>>>>>>>>>>>>>> Kind regards
    > > >>>>>>>>>>>>>>> Arina
    > > >>>>>>>>>>>>>>>
    > > >>>>>>>>>>>>>>>
    > > >>>>>>>>>>>>>>> On Thu, Jun 29, 2017 at 10:05 AM, Arina Yelchiyeva <
    > > >>>>>>>>>>>>>>> arina.yelchiyeva@gmail.com> wrote:
    > > >>>>>>>>>>>>>>>
    > > >>>>>>>>>>>>>>>> Parth, thanks a lot the instructions.
    > > >>>>>>>>>>>>>>>>
    > > >>>>>>>>>>>>>>>>
    > > >>>>>>>>>>>>>>>>
    > > >>>>>>>>>>>>>>>> On Wed, Jun 28, 2017 at 8:24 PM, Parth Chandra <
    > > >>>>>>>>>>> parthc@apache.org>
    > > >>>>>>>>>>>>>>> wrote:
    > > >>>>>>>>>>>>>>>>
    > > >>>>>>>>>>>>>>>>> FWIW, I put release instructions culled from gists
    > > written
    > > >>> by
    > > >>>>>>>>>>>>> previous
    > > >>>>>>>>>>>>>>>>> release managers into an updated gist here [1]
    > > >>>>>>>>>>>>>>>>>
    > > >>>>>>>>>>>>>>>>> HTH
    > > >>>>>>>>>>>>>>>>>
    > > >>>>>>>>>>>>>>>>> Parth
    > > >>>>>>>>>>>>>>>>>
    > > >>>>>>>>>>>>>>>>> [1] https://github.com/parthchandra/drill/wiki/Drill-
    > > >>>>>>>>>>>>> Release-Process
    > > >>>>>>>>>>>>>>>>> <https://github.com/parthchandra/drill/wiki/Drill-
    > > >>>>>>>>>>> Release-Process>
    > > >>>>>>>>>>>>>>>>>
    > > >>>>>>>>>>>>>>>>>
    > > >>>>>>>>>>>>>>>>>
    > > >>>>>>>>>>>>>>>>> On Tue, Jun 27, 2017 at 5:24 PM, Paul Rogers <
    > > >>>>>>> progers@mapr.com>
    > > >>>>>>>>>>>>>> wrote:
    > > >>>>>>>>>>>>>>>>>
    > > >>>>>>>>>>>>>>>>>> Hi Arina,
    > > >>>>>>>>>>>>>>>>>>
    > > >>>>>>>>>>>>>>>>>> I have two projects in progress: one is a candidate
    > for
    > > >>> 1.11,
    > > >>>>>>>>>>> the
    > > >>>>>>>>>>>>>>> other
    > > >>>>>>>>>>>>>>>>> is
    > > >>>>>>>>>>>>>>>>>> not.
    > > >>>>>>>>>>>>>>>>>>
    > > >>>>>>>>>>>>>>>>>> The “managed external sort” is the good candidate for
    > > >>> 1.11:
    > > >>>>>>>>>>> The PR
    > > >>>>>>>>>>>>>> for
    > > >>>>>>>>>>>>>>>>>> DRILL-5601 is being reviewed. Getting that in will
    > allow
    > > >>> us
    > > >>>>>>> to
    > > >>>>>>>>>>>>>> enable
    > > >>>>>>>>>>>>>>>>> the
    > > >>>>>>>>>>>>>>>>>> “new” external sort by default.
    > > >>>>>>>>>>>>>>>>>>
    > > >>>>>>>>>>>>>>>>>> The other project is to address Drill’s memory
    > > >>> fragmentation
    > > >>>>>>>>>>> issue
    > > >>>>>>>>>>>>>>> with
    > > >>>>>>>>>>>>>>>>>> vectors over 16 MB in size. Two PRs have been open
    > for a
    > > >>>>>>> while.
    > > >>>>>>>>>>>>> They
    > > >>>>>>>>>>>>>>> are
    > > >>>>>>>>>>>>>>>>>> the first of several. Each takes a while to review.
    > So,
    > > >>> the
    > > >>>>>>>>>>>>> project
    > > >>>>>>>>>>>>>>> does
    > > >>>>>>>>>>>>>>>>>> not fit 1.11. The work can go into the next release
    > > >>> instead.
    > > >>>>>>>>>>>>>>>>>>
    > > >>>>>>>>>>>>>>>>>> Thanks,
    > > >>>>>>>>>>>>>>>>>>
    > > >>>>>>>>>>>>>>>>>> - Paul
    > > >>>>>>>>>>>>>>>>>>
    > > >>>>>>>>>>>>>>>>>>
    > > >>>>>>>>>>>>>>>>>>
    > > >>>>>>>>>>>>>>>>>>
    > > >>>>>>>>>>>>>>>>>
    > > >>>>>>>>>>>>>>>>
    > > >>>>>>>>>>>>>>>>
    > > >>>>>>>>>>>>>>>
    > > >>>>>>>>>>>>>>
    > > >>>>>>>>>>>>>
    > > >>>>>>>>>>>
    > > >>>>>>>>>>>
    > > >>>>>>>>>>>
    > > >>>>>>>>>>>
    > > >>>>>>>>>
    > > >>>>>>>>>
    > > >>>>>>>>
    > > >>>>>>>
    > > >>>>>
    > > >>>>>
    > > >>>
    > > >>>
    > > >>
    > >
    > >
    >

Re: [DISCUSS] Drill 1.11.0 release

Posted by Arina Yelchiyeva <ar...@gmail.com>.

Hi Parth,

no problem, I did not start preparing the release since there is one
blocker has left:

*DRILL-5669 - Boaz Ben-Zvi (fix will be ready in a couple of days)*


Kind regards
Arina

On Mon, Jul 17, 2017 at 11:26 PM, Parth Chandra <pa...@apache.org> wrote:

> Hi Arina,
>
>   I just realized that I did not push the C++ client fix ( DRILL-5659) to
> master. I have just done so.
>
>   Sorry if I messed things up for the release, but we should get this fix
> in.
>
> Parth
>
> On Fri, Jul 14, 2017 at 8:15 AM, Paul Rogers <pr...@mapr.com> wrote:
>
> > I looked into one NPE on CSV. Turns out it is already fixed by previous
> > 1.11 PRs, so we’re good.
> >
> > There is another reported by a user on the user list, but I’ve not heard
> > back about getting sample data. Let’s not hold the release for that.
> >
> > - Paul
> >
> > > On Jul 14, 2017, at 2:38 AM, Arina Yelchiyeva <
> > arina.yelchiyeva@gmail.com> wrote:
> > >
> > > *Blockers*
> > > DRILL-5659 - Parth (done)
> > > DRILL-5669 - Boaz Ben-Zvi (fix will be ready in a couple of days)
> > >
> > > *Possible blockers*
> > > NPE on CSV - Paul can you please provide an update?
> > >
> > > Kind regards
> > > Arina
> > >
> > > On Thu, Jul 13, 2017 at 2:46 PM, Arina Yelchiyeva <
> > > arina.yelchiyeva@gmail.com> wrote:
> > >
> > >> *Blockers*
> > >> DRILL-5659 - problem is identified, Parth is working on the fix.
> > >>
> > >> *Not blockers*
> > >> DRILL-5660 - Vitalii
> > >> DRILL-5468 - Jinfeng
> > >>
> > >> *Possible blockers*
> > >> DRILL-5669 - Boaz Ben-Zvi / Paul
> > >> NPE on CSV - Paul
> > >>
> > >> Kind regards
> > >> Arina
> > >>
> > >>
> > >> On Thu, Jul 13, 2017 at 1:28 AM, Paul Rogers <pr...@mapr.com>
> wrote:
> > >>
> > >>> I withdraw my suggestion to fix DRILL-5660. Was a big hassle for us
> to
> > >>> track down the incompatibility the issue internally, but production
> > users
> > >>> probably don’t attempt to use one build of Drill with metadata files
> > >>> created from another. We can worry about the issue later only if a
> user
> > >>> actually runs into it.
> > >>>
> > >>> As suggested earlier, let’s just document this obscure case in a
> > release
> > >>> note.
> > >>>
> > >>> - Paul
> > >>>
> > >>>> On Jul 12, 2017, at 3:04 PM, Jinfeng Ni <jn...@apache.org> wrote:
> > >>>>
> > >>>> Upgrade from 1.10 to 1.11 is irreversible only when parquet metadata
> > >>> cache
> > >>>> file is involved.
> > >>>>
> > >>>> I might be wrong, but I thought that if someone is running on
> 1.10.0,
> > >>> and
> > >>>> he wants to go back to 1.0.0, where the first version of parquet
> > >>> meatadata
> > >>>> cache was introduced, can we guarantee that Drillbit would work
> > >>> correctly?
> > >>>> I'm trying to understand what DRILL-5660 asks for is a new
> > requirement,
> > >>> or
> > >>>> an requirement being there starting from Drill 1.0.0?
> > >>>>
> > >>>> Regression failures are normally regarded as release blocker.
> > >>>>
> > >>>>
> > >>>>
> > >>>> On Wed, Jul 12, 2017 at 2:54 PM, Paul Rogers <pr...@mapr.com>
> > wrote:
> > >>>>
> > >>>>> If we don’t do DRILL-5660 for 1.11 then we don’t need to do it at
> > all,
> > >>> so
> > >>>>> that is a time savings. We could handle this via a release note
> item
> > >>> that
> > >>>>> simply states that an upgrade from 1.10 to 1.11 is irreversible: we
> > do
> > >>> not
> > >>>>> support the ability to fall back to the old version. Everyone OK
> with
> > >>> that?
> > >>>>>
> > >>>>> We are seeing some stress test regression failures that, I’m told,
> > will
> > >>>>> show up as JIRA entries shortly.
> > >>>>>
> > >>>>> - Paul
> > >>>>>
> > >>>>>> On Jul 12, 2017, at 11:05 AM, Jinfeng Ni <jn...@apache.org> wrote:
> > >>>>>>
> > >>>>>> I put some analysis in DRILL-5468.
> > >>>>>>
> > >>>>>> Also, IMHO, DRILL-5660 is not a release blocker.
> > >>>>>>
> > >>>>>>
> > >>>>>>
> > >>>>>> On Tue, Jul 11, 2017 at 11:27 AM, Parth Chandra <
> parthc@apache.org>
> > >>>>> wrote:
> > >>>>>>
> > >>>>>>> Re DRILL-5634:
> > >>>>>>> From the Apache commons web page [1] -
> > >>>>>>>
> > >>>>>>> Please note that Apache Commons Crypto doesn't implement the
> > >>>>> cryptographic
> > >>>>>>> algorithm such as AES directly.* It wraps to Openssl or JCE which
> > >>>>> implement
> > >>>>>>> the algorithms.*
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> Since we are cleared for using Openssl and JCE, we are OK to
> > include
> > >>> the
> > >>>>>>> AES code in the UDFs.
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> Parth
> > >>>>>>>
> > >>>>>>> [1] https://commons.apache.org/proper/commons-crypto/
> > >>>>>>>
> > >>>>>>> On Tue, Jul 11, 2017 at 10:56 AM, Arina Yelchiyeva <
> > >>>>>>> arina.yelchiyeva@gmail.com> wrote:
> > >>>>>>>
> > >>>>>>>> During today hangouts we agreed on approximate cut-off date - 14
> > >>> July,
> > >>>>>>> 2017
> > >>>>>>>> (date may be shifted due to the blockers).
> > >>>>>>>> I'll start preparing test RC on Thursday to solve any release
> > >>>>> preparation
> > >>>>>>>> issues beforehand.
> > >>>>>>>>
> > >>>>>>>> *Blockers*
> > >>>>>>>> DRILL-5660 - Vitalii
> > >>>>>>>>
> > >>>>>>>> *Possible blockers*
> > >>>>>>>> DRILL-5468 - Jinfeng
> > >>>>>>>> NPE on CSV source - Paul
> > >>>>>>>> DRILL-5659 - Parth
> > >>>>>>>>
> > >>>>>>>> *Will be included if done before the cut-off date*
> > >>>>>>>> DRILL-5616 / DRILL-5665 - Boaz
> > >>>>>>>> DRILL-5634 - Parth will take a look at Apache commons
> > implementation
> > >>>>> and
> > >>>>>>>> update.
> > >>>>>>>>
> > >>>>>>>> Kind regards
> > >>>>>>>> Arina
> > >>>>>>>>
> > >>>>>>>> On Tue, Jul 11, 2017 at 3:14 PM, Charles Givre <
> cgivre@gmail.com>
> > >>>>> wrote:
> > >>>>>>>>
> > >>>>>>>>> Hi Arina,
> > >>>>>>>>> I can’t make the hangout, but we have a few options.
> Basically,
> > >>> since
> > >>>>>>>> all
> > >>>>>>>>> the hashing algorithms have no export restrictions, we could:
> > >>>>>>>>> 1.  Remove the AES encrypt/decrypt until I can figure out what
> we
> > >>> have
> > >>>>>>> to
> > >>>>>>>>> do (and do it)
> > >>>>>>>>> 2.  I can try to research it today or tomorrow to figure out
> > >>> exactly
> > >>>>>>> what
> > >>>>>>>>> we have to do to include AES.
> > >>>>>>>>> 3.  Remove the entire package
> > >>>>>>>>>
> > >>>>>>>>> I’d like to see the hashing functions at least included because
> > >>> we’ve
> > >>>>>>> had
> > >>>>>>>>> a lot of requests for that functionality.   I also don’t want
> > this
> > >>> to
> > >>>>>>>> hold
> > >>>>>>>>> things up, so please let me know your preference.  Regardless,
> > I’m
> > >>>>>>> going
> > >>>>>>>> to
> > >>>>>>>>> start trying to figure out the restrictions so we’ll know for
> the
> > >>>>>>> future.
> > >>>>>>>>>
> > >>>>>>>>> Best,
> > >>>>>>>>> — C
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>>> On Jul 11, 2017, at 05:58, Arina Yelchiyeva <
> > >>>>>>>> arina.yelchiyeva@gmail.com>
> > >>>>>>>>> wrote:
> > >>>>>>>>>>
> > >>>>>>>>>> @Jinfeng,
> > >>>>>>>>>> Let's discuss today on Drill hangout final cut-off date.
> > >>>>>>>>>>
> > >>>>>>>>>> @Charles
> > >>>>>>>>>> Since there were questions about AES in DRILL-5634, should we
> > >>> exclude
> > >>>>>>>> it
> > >>>>>>>>>> from the candidates for 1.11.0 release?
> > >>>>>>>>>>
> > >>>>>>>>>> @Boaz
> > >>>>>>>>>> Do we have any ETA when DRILL-5616 and DRILL-5665 will pass
> code
> > >>>>>>>> review?
> > >>>>>>>>>>
> > >>>>>>>>>> Also I believe we should include DRILL-5660 where we need to
> > bump
> > >>> up
> > >>>>>>>>>> parquet metadata version.
> > >>>>>>>>>> @Vitalii
> > >>>>>>>>>> Do you have any ETA for the fix?
> > >>>>>>>>>>
> > >>>>>>>>>> Kind regards
> > >>>>>>>>>> Arina
> > >>>>>>>>>>
> > >>>>>>>>>> On Tue, Jul 11, 2017 at 6:02 AM, Boaz Ben-Zvi <
> > bben-zvi@mapr.com>
> > >>>>>>>> wrote:
> > >>>>>>>>>>
> > >>>>>>>>>>> Hi Arina,
> > >>>>>>>>>>>
> > >>>>>>>>>>>  Two PR have just been submitted, relating to the Hash Aggr
> > >>>>>>> Spill:
> > >>>>>>>>>>> DRILL-5616 and DRILL-5665 .
> > >>>>>>>>>>> Can these be considered for 1.11 as well ?
> > >>>>>>>>>>>
> > >>>>>>>>>>>  Thanks,
> > >>>>>>>>>>>
> > >>>>>>>>>>>            Boaz
> > >>>>>>>>>>>
> > >>>>>>>>>>> On 7/10/17, 1:49 PM, "Paul Rogers" <pr...@mapr.com> wrote:
> > >>>>>>>>>>>
> > >>>>>>>>>>> DRILL-5601 is no longer a blocker for 1.11; we’ll defer that
> > >>>>>>> until
> > >>>>>>>>>>> after the release to allow others more time to work on other
> > >>>>>>> features
> > >>>>>>>>>>> rather than doing code reviews.
> > >>>>>>>>>>>
> > >>>>>>>>>>> - Paul
> > >>>>>>>>>>>
> > >>>>>>>>>>>> On Jul 10, 2017, at 1:43 PM, Jinfeng Ni <jn...@apache.org>
> > wrote:
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Hi Arina,
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> It has been a while since we discussed 1.11.0 release.  Any
> > >>> update
> > >>>>>>>>>>> on the
> > >>>>>>>>>>>> new release plan?  Do we have a cut-off date ?
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> In the past, I believe Drill release is a time-bounded, not
> > >>> feature
> > >>>>>>>>>>>> bounded. If certain fixes/features could not make it within
> a
> > >>>>>>> certain
> > >>>>>>>>>>>> time-frame, we had better defer them into future release,
> > unless
> > >>>>>>>>>>> those
> > >>>>>>>>>>>> fixes/features are regarded as blocking issues for a new
> > >>> release.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> On Tue, Jul 4, 2017 at 5:14 AM, Arina Yelchiyeva <
> > >>>>>>>>>>> arina.yelchiyeva@gmail.com
> > >>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>> Latest update:
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> We wait for the following Jiras before final cut off.
> > >>>>>>>>>>>>> 1. DRILL-5601  - Paul
> > >>>>>>>>>>>>> 2. DRILL-5420 - Kunal
> > >>>>>>>>>>>>> 3. DRILL-5634 - Charles
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Kind regards
> > >>>>>>>>>>>>> Arina
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> On Fri, Jun 30, 2017 at 9:29 PM, Kunal Khatua <
> > >>> kkhatua@mapr.com>
> > >>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Parth's already +1'd the process with a minor update. So,
> > >>> we're
> > >>>>>>>>>>> good to
> > >>>>>>>>>>>>> go
> > >>>>>>>>>>>>>> on DRILL-5420. I'll change the label on that to
> > >>> ready-to-commit.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> <http://www.mapr.com/>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> ________________________________
> > >>>>>>>>>>>>>> From: Parth Chandra <pa...@apache.org>
> > >>>>>>>>>>>>>> Sent: Thursday, June 29, 2017 9:44:39 AM
> > >>>>>>>>>>>>>> To: dev@drill.apache.org
> > >>>>>>>>>>>>>> Subject: Re: [DISCUSS] Drill 1.11.0 release
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> I'm Ok with the pcap format plugin if Pauls comments are
> > also
> > >>>>>>>>>>> addressed.
> > >>>>>>>>>>>>> If
> > >>>>>>>>>>>>>> the changes are done before your cutoff deadline,we can
> > merge
> > >>> it
> > >>>>>>>>>>> in.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> On Thu, Jun 29, 2017 at 12:26 AM, Arina Yelchiyeva <
> > >>>>>>>>>>>>>> arina.yelchiyeva@gmail.com> wrote:
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Hi all,
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> here is the list of Jira we are waiting before making
> final
> > >>> cut
> > >>>>>>>>>>> off:
> > >>>>>>>>>>>>>>> 1. DRILL-5601  - Paul
> > >>>>>>>>>>>>>>> 2. DRILL-3640  - Kunal
> > >>>>>>>>>>>>>>> 3. DRILL-5420 - Kunal
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Please let me know if there are any others Jiras we
> should
> > >>> wait
> > >>>>>>>>>>> for.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> @Charles,
> > >>>>>>>>>>>>>>> you have mentioned that you want to include some UDFs in
> > >>> Drill
> > >>>>>>>>>>> 1.11, do
> > >>>>>>>>>>>>>> you
> > >>>>>>>>>>>>>>> have any ETA for the PR?
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> @Paul / Parth
> > >>>>>>>>>>>>>>> I see there is active CR for pcap format PR, do we want
> to
> > >>>>>>>>>>> include it
> > >>>>>>>>>>>>> in
> > >>>>>>>>>>>>>>> this release as well?
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Kind regards
> > >>>>>>>>>>>>>>> Arina
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> On Thu, Jun 29, 2017 at 10:05 AM, Arina Yelchiyeva <
> > >>>>>>>>>>>>>>> arina.yelchiyeva@gmail.com> wrote:
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> Parth, thanks a lot the instructions.
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> On Wed, Jun 28, 2017 at 8:24 PM, Parth Chandra <
> > >>>>>>>>>>> parthc@apache.org>
> > >>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> FWIW, I put release instructions culled from gists
> > written
> > >>> by
> > >>>>>>>>>>>>> previous
> > >>>>>>>>>>>>>>>>> release managers into an updated gist here [1]
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> HTH
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> Parth
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> [1] https://github.com/parthchandra/drill/wiki/Drill-
> > >>>>>>>>>>>>> Release-Process
> > >>>>>>>>>>>>>>>>> <https://github.com/parthchandra/drill/wiki/Drill-
> > >>>>>>>>>>> Release-Process>
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> On Tue, Jun 27, 2017 at 5:24 PM, Paul Rogers <
> > >>>>>>> progers@mapr.com>
> > >>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> Hi Arina,
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> I have two projects in progress: one is a candidate
> for
> > >>> 1.11,
> > >>>>>>>>>>> the
> > >>>>>>>>>>>>>>> other
> > >>>>>>>>>>>>>>>>> is
> > >>>>>>>>>>>>>>>>>> not.
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> The “managed external sort” is the good candidate for
> > >>> 1.11:
> > >>>>>>>>>>> The PR
> > >>>>>>>>>>>>>> for
> > >>>>>>>>>>>>>>>>>> DRILL-5601 is being reviewed. Getting that in will
> allow
> > >>> us
> > >>>>>>> to
> > >>>>>>>>>>>>>> enable
> > >>>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>> “new” external sort by default.
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> The other project is to address Drill’s memory
> > >>> fragmentation
> > >>>>>>>>>>> issue
> > >>>>>>>>>>>>>>> with
> > >>>>>>>>>>>>>>>>>> vectors over 16 MB in size. Two PRs have been open
> for a
> > >>>>>>> while.
> > >>>>>>>>>>>>> They
> > >>>>>>>>>>>>>>> are
> > >>>>>>>>>>>>>>>>>> the first of several. Each takes a while to review.
> So,
> > >>> the
> > >>>>>>>>>>>>> project
> > >>>>>>>>>>>>>>> does
> > >>>>>>>>>>>>>>>>>> not fit 1.11. The work can go into the next release
> > >>> instead.
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> Thanks,
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> - Paul
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>
> > >>>>>>>
> > >>>>>
> > >>>>>
> > >>>
> > >>>
> > >>
> >
> >
>

Re: [DISCUSS] Drill 1.11.0 release

Posted by Parth Chandra <pa...@apache.org>.

Hi Arina,

  I just realized that I did not push the C++ client fix ( DRILL-5659) to
master. I have just done so.

  Sorry if I messed things up for the release, but we should get this fix
in.

Parth

On Fri, Jul 14, 2017 at 8:15 AM, Paul Rogers <pr...@mapr.com> wrote:

> I looked into one NPE on CSV. Turns out it is already fixed by previous
> 1.11 PRs, so we’re good.
>
> There is another reported by a user on the user list, but I’ve not heard
> back about getting sample data. Let’s not hold the release for that.
>
> - Paul
>
> > On Jul 14, 2017, at 2:38 AM, Arina Yelchiyeva <
> arina.yelchiyeva@gmail.com> wrote:
> >
> > *Blockers*
> > DRILL-5659 - Parth (done)
> > DRILL-5669 - Boaz Ben-Zvi (fix will be ready in a couple of days)
> >
> > *Possible blockers*
> > NPE on CSV - Paul can you please provide an update?
> >
> > Kind regards
> > Arina
> >
> > On Thu, Jul 13, 2017 at 2:46 PM, Arina Yelchiyeva <
> > arina.yelchiyeva@gmail.com> wrote:
> >
> >> *Blockers*
> >> DRILL-5659 - problem is identified, Parth is working on the fix.
> >>
> >> *Not blockers*
> >> DRILL-5660 - Vitalii
> >> DRILL-5468 - Jinfeng
> >>
> >> *Possible blockers*
> >> DRILL-5669 - Boaz Ben-Zvi / Paul
> >> NPE on CSV - Paul
> >>
> >> Kind regards
> >> Arina
> >>
> >>
> >> On Thu, Jul 13, 2017 at 1:28 AM, Paul Rogers <pr...@mapr.com> wrote:
> >>
> >>> I withdraw my suggestion to fix DRILL-5660. Was a big hassle for us to
> >>> track down the incompatibility the issue internally, but production
> users
> >>> probably don’t attempt to use one build of Drill with metadata files
> >>> created from another. We can worry about the issue later only if a user
> >>> actually runs into it.
> >>>
> >>> As suggested earlier, let’s just document this obscure case in a
> release
> >>> note.
> >>>
> >>> - Paul
> >>>
> >>>> On Jul 12, 2017, at 3:04 PM, Jinfeng Ni <jn...@apache.org> wrote:
> >>>>
> >>>> Upgrade from 1.10 to 1.11 is irreversible only when parquet metadata
> >>> cache
> >>>> file is involved.
> >>>>
> >>>> I might be wrong, but I thought that if someone is running on 1.10.0,
> >>> and
> >>>> he wants to go back to 1.0.0, where the first version of parquet
> >>> meatadata
> >>>> cache was introduced, can we guarantee that Drillbit would work
> >>> correctly?
> >>>> I'm trying to understand what DRILL-5660 asks for is a new
> requirement,
> >>> or
> >>>> an requirement being there starting from Drill 1.0.0?
> >>>>
> >>>> Regression failures are normally regarded as release blocker.
> >>>>
> >>>>
> >>>>
> >>>> On Wed, Jul 12, 2017 at 2:54 PM, Paul Rogers <pr...@mapr.com>
> wrote:
> >>>>
> >>>>> If we don’t do DRILL-5660 for 1.11 then we don’t need to do it at
> all,
> >>> so
> >>>>> that is a time savings. We could handle this via a release note item
> >>> that
> >>>>> simply states that an upgrade from 1.10 to 1.11 is irreversible: we
> do
> >>> not
> >>>>> support the ability to fall back to the old version. Everyone OK with
> >>> that?
> >>>>>
> >>>>> We are seeing some stress test regression failures that, I’m told,
> will
> >>>>> show up as JIRA entries shortly.
> >>>>>
> >>>>> - Paul
> >>>>>
> >>>>>> On Jul 12, 2017, at 11:05 AM, Jinfeng Ni <jn...@apache.org> wrote:
> >>>>>>
> >>>>>> I put some analysis in DRILL-5468.
> >>>>>>
> >>>>>> Also, IMHO, DRILL-5660 is not a release blocker.
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> On Tue, Jul 11, 2017 at 11:27 AM, Parth Chandra <pa...@apache.org>
> >>>>> wrote:
> >>>>>>
> >>>>>>> Re DRILL-5634:
> >>>>>>> From the Apache commons web page [1] -
> >>>>>>>
> >>>>>>> Please note that Apache Commons Crypto doesn't implement the
> >>>>> cryptographic
> >>>>>>> algorithm such as AES directly.* It wraps to Openssl or JCE which
> >>>>> implement
> >>>>>>> the algorithms.*
> >>>>>>>
> >>>>>>>
> >>>>>>> Since we are cleared for using Openssl and JCE, we are OK to
> include
> >>> the
> >>>>>>> AES code in the UDFs.
> >>>>>>>
> >>>>>>>
> >>>>>>> Parth
> >>>>>>>
> >>>>>>> [1] https://commons.apache.org/proper/commons-crypto/
> >>>>>>>
> >>>>>>> On Tue, Jul 11, 2017 at 10:56 AM, Arina Yelchiyeva <
> >>>>>>> arina.yelchiyeva@gmail.com> wrote:
> >>>>>>>
> >>>>>>>> During today hangouts we agreed on approximate cut-off date - 14
> >>> July,
> >>>>>>> 2017
> >>>>>>>> (date may be shifted due to the blockers).
> >>>>>>>> I'll start preparing test RC on Thursday to solve any release
> >>>>> preparation
> >>>>>>>> issues beforehand.
> >>>>>>>>
> >>>>>>>> *Blockers*
> >>>>>>>> DRILL-5660 - Vitalii
> >>>>>>>>
> >>>>>>>> *Possible blockers*
> >>>>>>>> DRILL-5468 - Jinfeng
> >>>>>>>> NPE on CSV source - Paul
> >>>>>>>> DRILL-5659 - Parth
> >>>>>>>>
> >>>>>>>> *Will be included if done before the cut-off date*
> >>>>>>>> DRILL-5616 / DRILL-5665 - Boaz
> >>>>>>>> DRILL-5634 - Parth will take a look at Apache commons
> implementation
> >>>>> and
> >>>>>>>> update.
> >>>>>>>>
> >>>>>>>> Kind regards
> >>>>>>>> Arina
> >>>>>>>>
> >>>>>>>> On Tue, Jul 11, 2017 at 3:14 PM, Charles Givre <cg...@gmail.com>
> >>>>> wrote:
> >>>>>>>>
> >>>>>>>>> Hi Arina,
> >>>>>>>>> I can’t make the hangout, but we have a few options.  Basically,
> >>> since
> >>>>>>>> all
> >>>>>>>>> the hashing algorithms have no export restrictions, we could:
> >>>>>>>>> 1.  Remove the AES encrypt/decrypt until I can figure out what we
> >>> have
> >>>>>>> to
> >>>>>>>>> do (and do it)
> >>>>>>>>> 2.  I can try to research it today or tomorrow to figure out
> >>> exactly
> >>>>>>> what
> >>>>>>>>> we have to do to include AES.
> >>>>>>>>> 3.  Remove the entire package
> >>>>>>>>>
> >>>>>>>>> I’d like to see the hashing functions at least included because
> >>> we’ve
> >>>>>>> had
> >>>>>>>>> a lot of requests for that functionality.   I also don’t want
> this
> >>> to
> >>>>>>>> hold
> >>>>>>>>> things up, so please let me know your preference.  Regardless,
> I’m
> >>>>>>> going
> >>>>>>>> to
> >>>>>>>>> start trying to figure out the restrictions so we’ll know for the
> >>>>>>> future.
> >>>>>>>>>
> >>>>>>>>> Best,
> >>>>>>>>> — C
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>> On Jul 11, 2017, at 05:58, Arina Yelchiyeva <
> >>>>>>>> arina.yelchiyeva@gmail.com>
> >>>>>>>>> wrote:
> >>>>>>>>>>
> >>>>>>>>>> @Jinfeng,
> >>>>>>>>>> Let's discuss today on Drill hangout final cut-off date.
> >>>>>>>>>>
> >>>>>>>>>> @Charles
> >>>>>>>>>> Since there were questions about AES in DRILL-5634, should we
> >>> exclude
> >>>>>>>> it
> >>>>>>>>>> from the candidates for 1.11.0 release?
> >>>>>>>>>>
> >>>>>>>>>> @Boaz
> >>>>>>>>>> Do we have any ETA when DRILL-5616 and DRILL-5665 will pass code
> >>>>>>>> review?
> >>>>>>>>>>
> >>>>>>>>>> Also I believe we should include DRILL-5660 where we need to
> bump
> >>> up
> >>>>>>>>>> parquet metadata version.
> >>>>>>>>>> @Vitalii
> >>>>>>>>>> Do you have any ETA for the fix?
> >>>>>>>>>>
> >>>>>>>>>> Kind regards
> >>>>>>>>>> Arina
> >>>>>>>>>>
> >>>>>>>>>> On Tue, Jul 11, 2017 at 6:02 AM, Boaz Ben-Zvi <
> bben-zvi@mapr.com>
> >>>>>>>> wrote:
> >>>>>>>>>>
> >>>>>>>>>>> Hi Arina,
> >>>>>>>>>>>
> >>>>>>>>>>>  Two PR have just been submitted, relating to the Hash Aggr
> >>>>>>> Spill:
> >>>>>>>>>>> DRILL-5616 and DRILL-5665 .
> >>>>>>>>>>> Can these be considered for 1.11 as well ?
> >>>>>>>>>>>
> >>>>>>>>>>>  Thanks,
> >>>>>>>>>>>
> >>>>>>>>>>>            Boaz
> >>>>>>>>>>>
> >>>>>>>>>>> On 7/10/17, 1:49 PM, "Paul Rogers" <pr...@mapr.com> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>> DRILL-5601 is no longer a blocker for 1.11; we’ll defer that
> >>>>>>> until
> >>>>>>>>>>> after the release to allow others more time to work on other
> >>>>>>> features
> >>>>>>>>>>> rather than doing code reviews.
> >>>>>>>>>>>
> >>>>>>>>>>> - Paul
> >>>>>>>>>>>
> >>>>>>>>>>>> On Jul 10, 2017, at 1:43 PM, Jinfeng Ni <jn...@apache.org>
> wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>> Hi Arina,
> >>>>>>>>>>>>
> >>>>>>>>>>>> It has been a while since we discussed 1.11.0 release.  Any
> >>> update
> >>>>>>>>>>> on the
> >>>>>>>>>>>> new release plan?  Do we have a cut-off date ?
> >>>>>>>>>>>>
> >>>>>>>>>>>> In the past, I believe Drill release is a time-bounded, not
> >>> feature
> >>>>>>>>>>>> bounded. If certain fixes/features could not make it within a
> >>>>>>> certain
> >>>>>>>>>>>> time-frame, we had better defer them into future release,
> unless
> >>>>>>>>>>> those
> >>>>>>>>>>>> fixes/features are regarded as blocking issues for a new
> >>> release.
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> On Tue, Jul 4, 2017 at 5:14 AM, Arina Yelchiyeva <
> >>>>>>>>>>> arina.yelchiyeva@gmail.com
> >>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>>> Latest update:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> We wait for the following Jiras before final cut off.
> >>>>>>>>>>>>> 1. DRILL-5601  - Paul
> >>>>>>>>>>>>> 2. DRILL-5420 - Kunal
> >>>>>>>>>>>>> 3. DRILL-5634 - Charles
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Kind regards
> >>>>>>>>>>>>> Arina
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> On Fri, Jun 30, 2017 at 9:29 PM, Kunal Khatua <
> >>> kkhatua@mapr.com>
> >>>>>>>>>>> wrote:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> Parth's already +1'd the process with a minor update. So,
> >>> we're
> >>>>>>>>>>> good to
> >>>>>>>>>>>>> go
> >>>>>>>>>>>>>> on DRILL-5420. I'll change the label on that to
> >>> ready-to-commit.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> <http://www.mapr.com/>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> ________________________________
> >>>>>>>>>>>>>> From: Parth Chandra <pa...@apache.org>
> >>>>>>>>>>>>>> Sent: Thursday, June 29, 2017 9:44:39 AM
> >>>>>>>>>>>>>> To: dev@drill.apache.org
> >>>>>>>>>>>>>> Subject: Re: [DISCUSS] Drill 1.11.0 release
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> I'm Ok with the pcap format plugin if Pauls comments are
> also
> >>>>>>>>>>> addressed.
> >>>>>>>>>>>>> If
> >>>>>>>>>>>>>> the changes are done before your cutoff deadline,we can
> merge
> >>> it
> >>>>>>>>>>> in.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> On Thu, Jun 29, 2017 at 12:26 AM, Arina Yelchiyeva <
> >>>>>>>>>>>>>> arina.yelchiyeva@gmail.com> wrote:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Hi all,
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> here is the list of Jira we are waiting before making final
> >>> cut
> >>>>>>>>>>> off:
> >>>>>>>>>>>>>>> 1. DRILL-5601  - Paul
> >>>>>>>>>>>>>>> 2. DRILL-3640  - Kunal
> >>>>>>>>>>>>>>> 3. DRILL-5420 - Kunal
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Please let me know if there are any others Jiras we should
> >>> wait
> >>>>>>>>>>> for.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> @Charles,
> >>>>>>>>>>>>>>> you have mentioned that you want to include some UDFs in
> >>> Drill
> >>>>>>>>>>> 1.11, do
> >>>>>>>>>>>>>> you
> >>>>>>>>>>>>>>> have any ETA for the PR?
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> @Paul / Parth
> >>>>>>>>>>>>>>> I see there is active CR for pcap format PR, do we want to
> >>>>>>>>>>> include it
> >>>>>>>>>>>>> in
> >>>>>>>>>>>>>>> this release as well?
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Kind regards
> >>>>>>>>>>>>>>> Arina
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> On Thu, Jun 29, 2017 at 10:05 AM, Arina Yelchiyeva <
> >>>>>>>>>>>>>>> arina.yelchiyeva@gmail.com> wrote:
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Parth, thanks a lot the instructions.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> On Wed, Jun 28, 2017 at 8:24 PM, Parth Chandra <
> >>>>>>>>>>> parthc@apache.org>
> >>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> FWIW, I put release instructions culled from gists
> written
> >>> by
> >>>>>>>>>>>>> previous
> >>>>>>>>>>>>>>>>> release managers into an updated gist here [1]
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> HTH
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Parth
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> [1] https://github.com/parthchandra/drill/wiki/Drill-
> >>>>>>>>>>>>> Release-Process
> >>>>>>>>>>>>>>>>> <https://github.com/parthchandra/drill/wiki/Drill-
> >>>>>>>>>>> Release-Process>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> On Tue, Jun 27, 2017 at 5:24 PM, Paul Rogers <
> >>>>>>> progers@mapr.com>
> >>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Hi Arina,
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> I have two projects in progress: one is a candidate for
> >>> 1.11,
> >>>>>>>>>>> the
> >>>>>>>>>>>>>>> other
> >>>>>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>>>> not.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> The “managed external sort” is the good candidate for
> >>> 1.11:
> >>>>>>>>>>> The PR
> >>>>>>>>>>>>>> for
> >>>>>>>>>>>>>>>>>> DRILL-5601 is being reviewed. Getting that in will allow
> >>> us
> >>>>>>> to
> >>>>>>>>>>>>>> enable
> >>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>> “new” external sort by default.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> The other project is to address Drill’s memory
> >>> fragmentation
> >>>>>>>>>>> issue
> >>>>>>>>>>>>>>> with
> >>>>>>>>>>>>>>>>>> vectors over 16 MB in size. Two PRs have been open for a
> >>>>>>> while.
> >>>>>>>>>>>>> They
> >>>>>>>>>>>>>>> are
> >>>>>>>>>>>>>>>>>> the first of several. Each takes a while to review. So,
> >>> the
> >>>>>>>>>>>>> project
> >>>>>>>>>>>>>>> does
> >>>>>>>>>>>>>>>>>> not fit 1.11. The work can go into the next release
> >>> instead.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> - Paul
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>
> >>>>>
> >>>
> >>>
> >>
>
>

Re: [DISCUSS] Drill 1.11.0 release

Posted by Paul Rogers <pr...@mapr.com>.

I looked into one NPE on CSV. Turns out it is already fixed by previous 1.11 PRs, so we’re good.

There is another reported by a user on the user list, but I’ve not heard back about getting sample data. Let’s not hold the release for that.

- Paul

> On Jul 14, 2017, at 2:38 AM, Arina Yelchiyeva <ar...@gmail.com> wrote:
> 
> *Blockers*
> DRILL-5659 - Parth (done)
> DRILL-5669 - Boaz Ben-Zvi (fix will be ready in a couple of days)
> 
> *Possible blockers*
> NPE on CSV - Paul can you please provide an update?
> 
> Kind regards
> Arina
> 
> On Thu, Jul 13, 2017 at 2:46 PM, Arina Yelchiyeva <
> arina.yelchiyeva@gmail.com> wrote:
> 
>> *Blockers*
>> DRILL-5659 - problem is identified, Parth is working on the fix.
>> 
>> *Not blockers*
>> DRILL-5660 - Vitalii
>> DRILL-5468 - Jinfeng
>> 
>> *Possible blockers*
>> DRILL-5669 - Boaz Ben-Zvi / Paul
>> NPE on CSV - Paul
>> 
>> Kind regards
>> Arina
>> 
>> 
>> On Thu, Jul 13, 2017 at 1:28 AM, Paul Rogers <pr...@mapr.com> wrote:
>> 
>>> I withdraw my suggestion to fix DRILL-5660. Was a big hassle for us to
>>> track down the incompatibility the issue internally, but production users
>>> probably don’t attempt to use one build of Drill with metadata files
>>> created from another. We can worry about the issue later only if a user
>>> actually runs into it.
>>> 
>>> As suggested earlier, let’s just document this obscure case in a release
>>> note.
>>> 
>>> - Paul
>>> 
>>>> On Jul 12, 2017, at 3:04 PM, Jinfeng Ni <jn...@apache.org> wrote:
>>>> 
>>>> Upgrade from 1.10 to 1.11 is irreversible only when parquet metadata
>>> cache
>>>> file is involved.
>>>> 
>>>> I might be wrong, but I thought that if someone is running on 1.10.0,
>>> and
>>>> he wants to go back to 1.0.0, where the first version of parquet
>>> meatadata
>>>> cache was introduced, can we guarantee that Drillbit would work
>>> correctly?
>>>> I'm trying to understand what DRILL-5660 asks for is a new requirement,
>>> or
>>>> an requirement being there starting from Drill 1.0.0?
>>>> 
>>>> Regression failures are normally regarded as release blocker.
>>>> 
>>>> 
>>>> 
>>>> On Wed, Jul 12, 2017 at 2:54 PM, Paul Rogers <pr...@mapr.com> wrote:
>>>> 
>>>>> If we don’t do DRILL-5660 for 1.11 then we don’t need to do it at all,
>>> so
>>>>> that is a time savings. We could handle this via a release note item
>>> that
>>>>> simply states that an upgrade from 1.10 to 1.11 is irreversible: we do
>>> not
>>>>> support the ability to fall back to the old version. Everyone OK with
>>> that?
>>>>> 
>>>>> We are seeing some stress test regression failures that, I’m told, will
>>>>> show up as JIRA entries shortly.
>>>>> 
>>>>> - Paul
>>>>> 
>>>>>> On Jul 12, 2017, at 11:05 AM, Jinfeng Ni <jn...@apache.org> wrote:
>>>>>> 
>>>>>> I put some analysis in DRILL-5468.
>>>>>> 
>>>>>> Also, IMHO, DRILL-5660 is not a release blocker.
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> On Tue, Jul 11, 2017 at 11:27 AM, Parth Chandra <pa...@apache.org>
>>>>> wrote:
>>>>>> 
>>>>>>> Re DRILL-5634:
>>>>>>> From the Apache commons web page [1] -
>>>>>>> 
>>>>>>> Please note that Apache Commons Crypto doesn't implement the
>>>>> cryptographic
>>>>>>> algorithm such as AES directly.* It wraps to Openssl or JCE which
>>>>> implement
>>>>>>> the algorithms.*
>>>>>>> 
>>>>>>> 
>>>>>>> Since we are cleared for using Openssl and JCE, we are OK to include
>>> the
>>>>>>> AES code in the UDFs.
>>>>>>> 
>>>>>>> 
>>>>>>> Parth
>>>>>>> 
>>>>>>> [1] https://commons.apache.org/proper/commons-crypto/
>>>>>>> 
>>>>>>> On Tue, Jul 11, 2017 at 10:56 AM, Arina Yelchiyeva <
>>>>>>> arina.yelchiyeva@gmail.com> wrote:
>>>>>>> 
>>>>>>>> During today hangouts we agreed on approximate cut-off date - 14
>>> July,
>>>>>>> 2017
>>>>>>>> (date may be shifted due to the blockers).
>>>>>>>> I'll start preparing test RC on Thursday to solve any release
>>>>> preparation
>>>>>>>> issues beforehand.
>>>>>>>> 
>>>>>>>> *Blockers*
>>>>>>>> DRILL-5660 - Vitalii
>>>>>>>> 
>>>>>>>> *Possible blockers*
>>>>>>>> DRILL-5468 - Jinfeng
>>>>>>>> NPE on CSV source - Paul
>>>>>>>> DRILL-5659 - Parth
>>>>>>>> 
>>>>>>>> *Will be included if done before the cut-off date*
>>>>>>>> DRILL-5616 / DRILL-5665 - Boaz
>>>>>>>> DRILL-5634 - Parth will take a look at Apache commons implementation
>>>>> and
>>>>>>>> update.
>>>>>>>> 
>>>>>>>> Kind regards
>>>>>>>> Arina
>>>>>>>> 
>>>>>>>> On Tue, Jul 11, 2017 at 3:14 PM, Charles Givre <cg...@gmail.com>
>>>>> wrote:
>>>>>>>> 
>>>>>>>>> Hi Arina,
>>>>>>>>> I can’t make the hangout, but we have a few options.  Basically,
>>> since
>>>>>>>> all
>>>>>>>>> the hashing algorithms have no export restrictions, we could:
>>>>>>>>> 1.  Remove the AES encrypt/decrypt until I can figure out what we
>>> have
>>>>>>> to
>>>>>>>>> do (and do it)
>>>>>>>>> 2.  I can try to research it today or tomorrow to figure out
>>> exactly
>>>>>>> what
>>>>>>>>> we have to do to include AES.
>>>>>>>>> 3.  Remove the entire package
>>>>>>>>> 
>>>>>>>>> I’d like to see the hashing functions at least included because
>>> we’ve
>>>>>>> had
>>>>>>>>> a lot of requests for that functionality.   I also don’t want this
>>> to
>>>>>>>> hold
>>>>>>>>> things up, so please let me know your preference.  Regardless, I’m
>>>>>>> going
>>>>>>>> to
>>>>>>>>> start trying to figure out the restrictions so we’ll know for the
>>>>>>> future.
>>>>>>>>> 
>>>>>>>>> Best,
>>>>>>>>> — C
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>>> On Jul 11, 2017, at 05:58, Arina Yelchiyeva <
>>>>>>>> arina.yelchiyeva@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>> 
>>>>>>>>>> @Jinfeng,
>>>>>>>>>> Let's discuss today on Drill hangout final cut-off date.
>>>>>>>>>> 
>>>>>>>>>> @Charles
>>>>>>>>>> Since there were questions about AES in DRILL-5634, should we
>>> exclude
>>>>>>>> it
>>>>>>>>>> from the candidates for 1.11.0 release?
>>>>>>>>>> 
>>>>>>>>>> @Boaz
>>>>>>>>>> Do we have any ETA when DRILL-5616 and DRILL-5665 will pass code
>>>>>>>> review?
>>>>>>>>>> 
>>>>>>>>>> Also I believe we should include DRILL-5660 where we need to bump
>>> up
>>>>>>>>>> parquet metadata version.
>>>>>>>>>> @Vitalii
>>>>>>>>>> Do you have any ETA for the fix?
>>>>>>>>>> 
>>>>>>>>>> Kind regards
>>>>>>>>>> Arina
>>>>>>>>>> 
>>>>>>>>>> On Tue, Jul 11, 2017 at 6:02 AM, Boaz Ben-Zvi <bb...@mapr.com>
>>>>>>>> wrote:
>>>>>>>>>> 
>>>>>>>>>>> Hi Arina,
>>>>>>>>>>> 
>>>>>>>>>>>  Two PR have just been submitted, relating to the Hash Aggr
>>>>>>> Spill:
>>>>>>>>>>> DRILL-5616 and DRILL-5665 .
>>>>>>>>>>> Can these be considered for 1.11 as well ?
>>>>>>>>>>> 
>>>>>>>>>>>  Thanks,
>>>>>>>>>>> 
>>>>>>>>>>>            Boaz
>>>>>>>>>>> 
>>>>>>>>>>> On 7/10/17, 1:49 PM, "Paul Rogers" <pr...@mapr.com> wrote:
>>>>>>>>>>> 
>>>>>>>>>>> DRILL-5601 is no longer a blocker for 1.11; we’ll defer that
>>>>>>> until
>>>>>>>>>>> after the release to allow others more time to work on other
>>>>>>> features
>>>>>>>>>>> rather than doing code reviews.
>>>>>>>>>>> 
>>>>>>>>>>> - Paul
>>>>>>>>>>> 
>>>>>>>>>>>> On Jul 10, 2017, at 1:43 PM, Jinfeng Ni <jn...@apache.org> wrote:
>>>>>>>>>>>> 
>>>>>>>>>>>> Hi Arina,
>>>>>>>>>>>> 
>>>>>>>>>>>> It has been a while since we discussed 1.11.0 release.  Any
>>> update
>>>>>>>>>>> on the
>>>>>>>>>>>> new release plan?  Do we have a cut-off date ?
>>>>>>>>>>>> 
>>>>>>>>>>>> In the past, I believe Drill release is a time-bounded, not
>>> feature
>>>>>>>>>>>> bounded. If certain fixes/features could not make it within a
>>>>>>> certain
>>>>>>>>>>>> time-frame, we had better defer them into future release, unless
>>>>>>>>>>> those
>>>>>>>>>>>> fixes/features are regarded as blocking issues for a new
>>> release.
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> On Tue, Jul 4, 2017 at 5:14 AM, Arina Yelchiyeva <
>>>>>>>>>>> arina.yelchiyeva@gmail.com
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>> 
>>>>>>>>>>>>> Latest update:
>>>>>>>>>>>>> 
>>>>>>>>>>>>> We wait for the following Jiras before final cut off.
>>>>>>>>>>>>> 1. DRILL-5601  - Paul
>>>>>>>>>>>>> 2. DRILL-5420 - Kunal
>>>>>>>>>>>>> 3. DRILL-5634 - Charles
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Kind regards
>>>>>>>>>>>>> Arina
>>>>>>>>>>>>> 
>>>>>>>>>>>>> On Fri, Jun 30, 2017 at 9:29 PM, Kunal Khatua <
>>> kkhatua@mapr.com>
>>>>>>>>>>> wrote:
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Parth's already +1'd the process with a minor update. So,
>>> we're
>>>>>>>>>>> good to
>>>>>>>>>>>>> go
>>>>>>>>>>>>>> on DRILL-5420. I'll change the label on that to
>>> ready-to-commit.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> <http://www.mapr.com/>
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> ________________________________
>>>>>>>>>>>>>> From: Parth Chandra <pa...@apache.org>
>>>>>>>>>>>>>> Sent: Thursday, June 29, 2017 9:44:39 AM
>>>>>>>>>>>>>> To: dev@drill.apache.org
>>>>>>>>>>>>>> Subject: Re: [DISCUSS] Drill 1.11.0 release
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> I'm Ok with the pcap format plugin if Pauls comments are also
>>>>>>>>>>> addressed.
>>>>>>>>>>>>> If
>>>>>>>>>>>>>> the changes are done before your cutoff deadline,we can merge
>>> it
>>>>>>>>>>> in.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> On Thu, Jun 29, 2017 at 12:26 AM, Arina Yelchiyeva <
>>>>>>>>>>>>>> arina.yelchiyeva@gmail.com> wrote:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Hi all,
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> here is the list of Jira we are waiting before making final
>>> cut
>>>>>>>>>>> off:
>>>>>>>>>>>>>>> 1. DRILL-5601  - Paul
>>>>>>>>>>>>>>> 2. DRILL-3640  - Kunal
>>>>>>>>>>>>>>> 3. DRILL-5420 - Kunal
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Please let me know if there are any others Jiras we should
>>> wait
>>>>>>>>>>> for.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> @Charles,
>>>>>>>>>>>>>>> you have mentioned that you want to include some UDFs in
>>> Drill
>>>>>>>>>>> 1.11, do
>>>>>>>>>>>>>> you
>>>>>>>>>>>>>>> have any ETA for the PR?
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> @Paul / Parth
>>>>>>>>>>>>>>> I see there is active CR for pcap format PR, do we want to
>>>>>>>>>>> include it
>>>>>>>>>>>>> in
>>>>>>>>>>>>>>> this release as well?
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Kind regards
>>>>>>>>>>>>>>> Arina
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> On Thu, Jun 29, 2017 at 10:05 AM, Arina Yelchiyeva <
>>>>>>>>>>>>>>> arina.yelchiyeva@gmail.com> wrote:
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Parth, thanks a lot the instructions.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> On Wed, Jun 28, 2017 at 8:24 PM, Parth Chandra <
>>>>>>>>>>> parthc@apache.org>
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> FWIW, I put release instructions culled from gists written
>>> by
>>>>>>>>>>>>> previous
>>>>>>>>>>>>>>>>> release managers into an updated gist here [1]
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> HTH
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Parth
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> [1] https://github.com/parthchandra/drill/wiki/Drill-
>>>>>>>>>>>>> Release-Process
>>>>>>>>>>>>>>>>> <https://github.com/parthchandra/drill/wiki/Drill-
>>>>>>>>>>> Release-Process>
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> On Tue, Jun 27, 2017 at 5:24 PM, Paul Rogers <
>>>>>>> progers@mapr.com>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Hi Arina,
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> I have two projects in progress: one is a candidate for
>>> 1.11,
>>>>>>>>>>> the
>>>>>>>>>>>>>>> other
>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>> not.
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> The “managed external sort” is the good candidate for
>>> 1.11:
>>>>>>>>>>> The PR
>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>> DRILL-5601 is being reviewed. Getting that in will allow
>>> us
>>>>>>> to
>>>>>>>>>>>>>> enable
>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>> “new” external sort by default.
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> The other project is to address Drill’s memory
>>> fragmentation
>>>>>>>>>>> issue
>>>>>>>>>>>>>>> with
>>>>>>>>>>>>>>>>>> vectors over 16 MB in size. Two PRs have been open for a
>>>>>>> while.
>>>>>>>>>>>>> They
>>>>>>>>>>>>>>> are
>>>>>>>>>>>>>>>>>> the first of several. Each takes a while to review. So,
>>> the
>>>>>>>>>>>>> project
>>>>>>>>>>>>>>> does
>>>>>>>>>>>>>>>>>> not fit 1.11. The work can go into the next release
>>> instead.
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> - Paul
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>> 
>>>>> 
>>> 
>>> 
>>

Re: [DISCUSS] Drill 1.11.0 release

Posted by Arina Yelchiyeva <ar...@gmail.com>.

*Blockers*
DRILL-5659 - Parth (done)
DRILL-5669 - Boaz Ben-Zvi (fix will be ready in a couple of days)

*Possible blockers*
NPE on CSV - Paul can you please provide an update?

Kind regards
Arina

On Thu, Jul 13, 2017 at 2:46 PM, Arina Yelchiyeva <
arina.yelchiyeva@gmail.com> wrote:

> *Blockers*
> DRILL-5659 - problem is identified, Parth is working on the fix.
>
> *Not blockers*
> DRILL-5660 - Vitalii
> DRILL-5468 - Jinfeng
>
> *Possible blockers*
> DRILL-5669 - Boaz Ben-Zvi / Paul
> NPE on CSV - Paul
>
> Kind regards
> Arina
>
>
> On Thu, Jul 13, 2017 at 1:28 AM, Paul Rogers <pr...@mapr.com> wrote:
>
>> I withdraw my suggestion to fix DRILL-5660. Was a big hassle for us to
>> track down the incompatibility the issue internally, but production users
>> probably don’t attempt to use one build of Drill with metadata files
>> created from another. We can worry about the issue later only if a user
>> actually runs into it.
>>
>> As suggested earlier, let’s just document this obscure case in a release
>> note.
>>
>> - Paul
>>
>> > On Jul 12, 2017, at 3:04 PM, Jinfeng Ni <jn...@apache.org> wrote:
>> >
>> > Upgrade from 1.10 to 1.11 is irreversible only when parquet metadata
>> cache
>> > file is involved.
>> >
>> > I might be wrong, but I thought that if someone is running on 1.10.0,
>> and
>> > he wants to go back to 1.0.0, where the first version of parquet
>> meatadata
>> > cache was introduced, can we guarantee that Drillbit would work
>> correctly?
>> > I'm trying to understand what DRILL-5660 asks for is a new requirement,
>> or
>> > an requirement being there starting from Drill 1.0.0?
>> >
>> > Regression failures are normally regarded as release blocker.
>> >
>> >
>> >
>> > On Wed, Jul 12, 2017 at 2:54 PM, Paul Rogers <pr...@mapr.com> wrote:
>> >
>> >> If we don’t do DRILL-5660 for 1.11 then we don’t need to do it at all,
>> so
>> >> that is a time savings. We could handle this via a release note item
>> that
>> >> simply states that an upgrade from 1.10 to 1.11 is irreversible: we do
>> not
>> >> support the ability to fall back to the old version. Everyone OK with
>> that?
>> >>
>> >> We are seeing some stress test regression failures that, I’m told, will
>> >> show up as JIRA entries shortly.
>> >>
>> >> - Paul
>> >>
>> >>> On Jul 12, 2017, at 11:05 AM, Jinfeng Ni <jn...@apache.org> wrote:
>> >>>
>> >>> I put some analysis in DRILL-5468.
>> >>>
>> >>> Also, IMHO, DRILL-5660 is not a release blocker.
>> >>>
>> >>>
>> >>>
>> >>> On Tue, Jul 11, 2017 at 11:27 AM, Parth Chandra <pa...@apache.org>
>> >> wrote:
>> >>>
>> >>>> Re DRILL-5634:
>> >>>> From the Apache commons web page [1] -
>> >>>>
>> >>>> Please note that Apache Commons Crypto doesn't implement the
>> >> cryptographic
>> >>>> algorithm such as AES directly.* It wraps to Openssl or JCE which
>> >> implement
>> >>>> the algorithms.*
>> >>>>
>> >>>>
>> >>>> Since we are cleared for using Openssl and JCE, we are OK to include
>> the
>> >>>> AES code in the UDFs.
>> >>>>
>> >>>>
>> >>>> Parth
>> >>>>
>> >>>> [1] https://commons.apache.org/proper/commons-crypto/
>> >>>>
>> >>>> On Tue, Jul 11, 2017 at 10:56 AM, Arina Yelchiyeva <
>> >>>> arina.yelchiyeva@gmail.com> wrote:
>> >>>>
>> >>>>> During today hangouts we agreed on approximate cut-off date - 14
>> July,
>> >>>> 2017
>> >>>>> (date may be shifted due to the blockers).
>> >>>>> I'll start preparing test RC on Thursday to solve any release
>> >> preparation
>> >>>>> issues beforehand.
>> >>>>>
>> >>>>> *Blockers*
>> >>>>> DRILL-5660 - Vitalii
>> >>>>>
>> >>>>> *Possible blockers*
>> >>>>> DRILL-5468 - Jinfeng
>> >>>>> NPE on CSV source - Paul
>> >>>>> DRILL-5659 - Parth
>> >>>>>
>> >>>>> *Will be included if done before the cut-off date*
>> >>>>> DRILL-5616 / DRILL-5665 - Boaz
>> >>>>> DRILL-5634 - Parth will take a look at Apache commons implementation
>> >> and
>> >>>>> update.
>> >>>>>
>> >>>>> Kind regards
>> >>>>> Arina
>> >>>>>
>> >>>>> On Tue, Jul 11, 2017 at 3:14 PM, Charles Givre <cg...@gmail.com>
>> >> wrote:
>> >>>>>
>> >>>>>> Hi Arina,
>> >>>>>> I can’t make the hangout, but we have a few options.  Basically,
>> since
>> >>>>> all
>> >>>>>> the hashing algorithms have no export restrictions, we could:
>> >>>>>> 1.  Remove the AES encrypt/decrypt until I can figure out what we
>> have
>> >>>> to
>> >>>>>> do (and do it)
>> >>>>>> 2.  I can try to research it today or tomorrow to figure out
>> exactly
>> >>>> what
>> >>>>>> we have to do to include AES.
>> >>>>>> 3.  Remove the entire package
>> >>>>>>
>> >>>>>> I’d like to see the hashing functions at least included because
>> we’ve
>> >>>> had
>> >>>>>> a lot of requests for that functionality.   I also don’t want this
>> to
>> >>>>> hold
>> >>>>>> things up, so please let me know your preference.  Regardless, I’m
>> >>>> going
>> >>>>> to
>> >>>>>> start trying to figure out the restrictions so we’ll know for the
>> >>>> future.
>> >>>>>>
>> >>>>>> Best,
>> >>>>>> — C
>> >>>>>>
>> >>>>>>
>> >>>>>>
>> >>>>>>> On Jul 11, 2017, at 05:58, Arina Yelchiyeva <
>> >>>>> arina.yelchiyeva@gmail.com>
>> >>>>>> wrote:
>> >>>>>>>
>> >>>>>>> @Jinfeng,
>> >>>>>>> Let's discuss today on Drill hangout final cut-off date.
>> >>>>>>>
>> >>>>>>> @Charles
>> >>>>>>> Since there were questions about AES in DRILL-5634, should we
>> exclude
>> >>>>> it
>> >>>>>>> from the candidates for 1.11.0 release?
>> >>>>>>>
>> >>>>>>> @Boaz
>> >>>>>>> Do we have any ETA when DRILL-5616 and DRILL-5665 will pass code
>> >>>>> review?
>> >>>>>>>
>> >>>>>>> Also I believe we should include DRILL-5660 where we need to bump
>> up
>> >>>>>>> parquet metadata version.
>> >>>>>>> @Vitalii
>> >>>>>>> Do you have any ETA for the fix?
>> >>>>>>>
>> >>>>>>> Kind regards
>> >>>>>>> Arina
>> >>>>>>>
>> >>>>>>> On Tue, Jul 11, 2017 at 6:02 AM, Boaz Ben-Zvi <bb...@mapr.com>
>> >>>>> wrote:
>> >>>>>>>
>> >>>>>>>> Hi Arina,
>> >>>>>>>>
>> >>>>>>>>   Two PR have just been submitted, relating to the Hash Aggr
>> >>>> Spill:
>> >>>>>>>> DRILL-5616 and DRILL-5665 .
>> >>>>>>>> Can these be considered for 1.11 as well ?
>> >>>>>>>>
>> >>>>>>>>   Thanks,
>> >>>>>>>>
>> >>>>>>>>             Boaz
>> >>>>>>>>
>> >>>>>>>> On 7/10/17, 1:49 PM, "Paul Rogers" <pr...@mapr.com> wrote:
>> >>>>>>>>
>> >>>>>>>>  DRILL-5601 is no longer a blocker for 1.11; we’ll defer that
>> >>>> until
>> >>>>>>>> after the release to allow others more time to work on other
>> >>>> features
>> >>>>>>>> rather than doing code reviews.
>> >>>>>>>>
>> >>>>>>>>  - Paul
>> >>>>>>>>
>> >>>>>>>>> On Jul 10, 2017, at 1:43 PM, Jinfeng Ni <jn...@apache.org> wrote:
>> >>>>>>>>>
>> >>>>>>>>> Hi Arina,
>> >>>>>>>>>
>> >>>>>>>>> It has been a while since we discussed 1.11.0 release.  Any
>> update
>> >>>>>>>> on the
>> >>>>>>>>> new release plan?  Do we have a cut-off date ?
>> >>>>>>>>>
>> >>>>>>>>> In the past, I believe Drill release is a time-bounded, not
>> feature
>> >>>>>>>>> bounded. If certain fixes/features could not make it within a
>> >>>> certain
>> >>>>>>>>> time-frame, we had better defer them into future release, unless
>> >>>>>>>> those
>> >>>>>>>>> fixes/features are regarded as blocking issues for a new
>> release.
>> >>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>> On Tue, Jul 4, 2017 at 5:14 AM, Arina Yelchiyeva <
>> >>>>>>>> arina.yelchiyeva@gmail.com
>> >>>>>>>>>> wrote:
>> >>>>>>>>>
>> >>>>>>>>>> Latest update:
>> >>>>>>>>>>
>> >>>>>>>>>> We wait for the following Jiras before final cut off.
>> >>>>>>>>>> 1. DRILL-5601  - Paul
>> >>>>>>>>>> 2. DRILL-5420 - Kunal
>> >>>>>>>>>> 3. DRILL-5634 - Charles
>> >>>>>>>>>>
>> >>>>>>>>>> Kind regards
>> >>>>>>>>>> Arina
>> >>>>>>>>>>
>> >>>>>>>>>> On Fri, Jun 30, 2017 at 9:29 PM, Kunal Khatua <
>> kkhatua@mapr.com>
>> >>>>>>>> wrote:
>> >>>>>>>>>>
>> >>>>>>>>>>> Parth's already +1'd the process with a minor update. So,
>> we're
>> >>>>>>>> good to
>> >>>>>>>>>> go
>> >>>>>>>>>>> on DRILL-5420. I'll change the label on that to
>> ready-to-commit.
>> >>>>>>>>>>>
>> >>>>>>>>>>> <http://www.mapr.com/>
>> >>>>>>>>>>>
>> >>>>>>>>>>> ________________________________
>> >>>>>>>>>>> From: Parth Chandra <pa...@apache.org>
>> >>>>>>>>>>> Sent: Thursday, June 29, 2017 9:44:39 AM
>> >>>>>>>>>>> To: dev@drill.apache.org
>> >>>>>>>>>>> Subject: Re: [DISCUSS] Drill 1.11.0 release
>> >>>>>>>>>>>
>> >>>>>>>>>>> I'm Ok with the pcap format plugin if Pauls comments are also
>> >>>>>>>> addressed.
>> >>>>>>>>>> If
>> >>>>>>>>>>> the changes are done before your cutoff deadline,we can merge
>> it
>> >>>>>>>> in.
>> >>>>>>>>>>>
>> >>>>>>>>>>> On Thu, Jun 29, 2017 at 12:26 AM, Arina Yelchiyeva <
>> >>>>>>>>>>> arina.yelchiyeva@gmail.com> wrote:
>> >>>>>>>>>>>
>> >>>>>>>>>>>> Hi all,
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> here is the list of Jira we are waiting before making final
>> cut
>> >>>>>>>> off:
>> >>>>>>>>>>>> 1. DRILL-5601  - Paul
>> >>>>>>>>>>>> 2. DRILL-3640  - Kunal
>> >>>>>>>>>>>> 3. DRILL-5420 - Kunal
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> Please let me know if there are any others Jiras we should
>> wait
>> >>>>>>>> for.
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> @Charles,
>> >>>>>>>>>>>> you have mentioned that you want to include some UDFs in
>> Drill
>> >>>>>>>> 1.11, do
>> >>>>>>>>>>> you
>> >>>>>>>>>>>> have any ETA for the PR?
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> @Paul / Parth
>> >>>>>>>>>>>> I see there is active CR for pcap format PR, do we want to
>> >>>>>>>> include it
>> >>>>>>>>>> in
>> >>>>>>>>>>>> this release as well?
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> Kind regards
>> >>>>>>>>>>>> Arina
>> >>>>>>>>>>>>
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> On Thu, Jun 29, 2017 at 10:05 AM, Arina Yelchiyeva <
>> >>>>>>>>>>>> arina.yelchiyeva@gmail.com> wrote:
>> >>>>>>>>>>>>
>> >>>>>>>>>>>>> Parth, thanks a lot the instructions.
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> On Wed, Jun 28, 2017 at 8:24 PM, Parth Chandra <
>> >>>>>>>> parthc@apache.org>
>> >>>>>>>>>>>> wrote:
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>>> FWIW, I put release instructions culled from gists written
>> by
>> >>>>>>>>>> previous
>> >>>>>>>>>>>>>> release managers into an updated gist here [1]
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>> HTH
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>> Parth
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>> [1] https://github.com/parthchandra/drill/wiki/Drill-
>> >>>>>>>>>> Release-Process
>> >>>>>>>>>>>>>> <https://github.com/parthchandra/drill/wiki/Drill-
>> >>>>>>>> Release-Process>
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>> On Tue, Jun 27, 2017 at 5:24 PM, Paul Rogers <
>> >>>> progers@mapr.com>
>> >>>>>>>>>>> wrote:
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>> Hi Arina,
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>> I have two projects in progress: one is a candidate for
>> 1.11,
>> >>>>>>>> the
>> >>>>>>>>>>>> other
>> >>>>>>>>>>>>>> is
>> >>>>>>>>>>>>>>> not.
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>> The “managed external sort” is the good candidate for
>> 1.11:
>> >>>>>>>> The PR
>> >>>>>>>>>>> for
>> >>>>>>>>>>>>>>> DRILL-5601 is being reviewed. Getting that in will allow
>> us
>> >>>> to
>> >>>>>>>>>>> enable
>> >>>>>>>>>>>>>> the
>> >>>>>>>>>>>>>>> “new” external sort by default.
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>> The other project is to address Drill’s memory
>> fragmentation
>> >>>>>>>> issue
>> >>>>>>>>>>>> with
>> >>>>>>>>>>>>>>> vectors over 16 MB in size. Two PRs have been open for a
>> >>>> while.
>> >>>>>>>>>> They
>> >>>>>>>>>>>> are
>> >>>>>>>>>>>>>>> the first of several. Each takes a while to review. So,
>> the
>> >>>>>>>>>> project
>> >>>>>>>>>>>> does
>> >>>>>>>>>>>>>>> not fit 1.11. The work can go into the next release
>> instead.
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>> Thanks,
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>> - Paul
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>
>> >>>>>>
>> >>>>>
>> >>>>
>> >>
>> >>
>>
>>
>

Re: [DISCUSS] Drill 1.11.0 release

Posted by Arina Yelchiyeva <ar...@gmail.com>.

*Blockers*
DRILL-5659 - problem is identified, Parth is working on the fix.

*Not blockers*
DRILL-5660 - Vitalii
DRILL-5468 - Jinfeng

*Possible blockers*
DRILL-5669 - Boaz Ben-Zvi / Paul
NPE on CSV - Paul

Kind regards
Arina


On Thu, Jul 13, 2017 at 1:28 AM, Paul Rogers <pr...@mapr.com> wrote:

> I withdraw my suggestion to fix DRILL-5660. Was a big hassle for us to
> track down the incompatibility the issue internally, but production users
> probably don’t attempt to use one build of Drill with metadata files
> created from another. We can worry about the issue later only if a user
> actually runs into it.
>
> As suggested earlier, let’s just document this obscure case in a release
> note.
>
> - Paul
>
> > On Jul 12, 2017, at 3:04 PM, Jinfeng Ni <jn...@apache.org> wrote:
> >
> > Upgrade from 1.10 to 1.11 is irreversible only when parquet metadata
> cache
> > file is involved.
> >
> > I might be wrong, but I thought that if someone is running on 1.10.0, and
> > he wants to go back to 1.0.0, where the first version of parquet
> meatadata
> > cache was introduced, can we guarantee that Drillbit would work
> correctly?
> > I'm trying to understand what DRILL-5660 asks for is a new requirement,
> or
> > an requirement being there starting from Drill 1.0.0?
> >
> > Regression failures are normally regarded as release blocker.
> >
> >
> >
> > On Wed, Jul 12, 2017 at 2:54 PM, Paul Rogers <pr...@mapr.com> wrote:
> >
> >> If we don’t do DRILL-5660 for 1.11 then we don’t need to do it at all,
> so
> >> that is a time savings. We could handle this via a release note item
> that
> >> simply states that an upgrade from 1.10 to 1.11 is irreversible: we do
> not
> >> support the ability to fall back to the old version. Everyone OK with
> that?
> >>
> >> We are seeing some stress test regression failures that, I’m told, will
> >> show up as JIRA entries shortly.
> >>
> >> - Paul
> >>
> >>> On Jul 12, 2017, at 11:05 AM, Jinfeng Ni <jn...@apache.org> wrote:
> >>>
> >>> I put some analysis in DRILL-5468.
> >>>
> >>> Also, IMHO, DRILL-5660 is not a release blocker.
> >>>
> >>>
> >>>
> >>> On Tue, Jul 11, 2017 at 11:27 AM, Parth Chandra <pa...@apache.org>
> >> wrote:
> >>>
> >>>> Re DRILL-5634:
> >>>> From the Apache commons web page [1] -
> >>>>
> >>>> Please note that Apache Commons Crypto doesn't implement the
> >> cryptographic
> >>>> algorithm such as AES directly.* It wraps to Openssl or JCE which
> >> implement
> >>>> the algorithms.*
> >>>>
> >>>>
> >>>> Since we are cleared for using Openssl and JCE, we are OK to include
> the
> >>>> AES code in the UDFs.
> >>>>
> >>>>
> >>>> Parth
> >>>>
> >>>> [1] https://commons.apache.org/proper/commons-crypto/
> >>>>
> >>>> On Tue, Jul 11, 2017 at 10:56 AM, Arina Yelchiyeva <
> >>>> arina.yelchiyeva@gmail.com> wrote:
> >>>>
> >>>>> During today hangouts we agreed on approximate cut-off date - 14
> July,
> >>>> 2017
> >>>>> (date may be shifted due to the blockers).
> >>>>> I'll start preparing test RC on Thursday to solve any release
> >> preparation
> >>>>> issues beforehand.
> >>>>>
> >>>>> *Blockers*
> >>>>> DRILL-5660 - Vitalii
> >>>>>
> >>>>> *Possible blockers*
> >>>>> DRILL-5468 - Jinfeng
> >>>>> NPE on CSV source - Paul
> >>>>> DRILL-5659 - Parth
> >>>>>
> >>>>> *Will be included if done before the cut-off date*
> >>>>> DRILL-5616 / DRILL-5665 - Boaz
> >>>>> DRILL-5634 - Parth will take a look at Apache commons implementation
> >> and
> >>>>> update.
> >>>>>
> >>>>> Kind regards
> >>>>> Arina
> >>>>>
> >>>>> On Tue, Jul 11, 2017 at 3:14 PM, Charles Givre <cg...@gmail.com>
> >> wrote:
> >>>>>
> >>>>>> Hi Arina,
> >>>>>> I can’t make the hangout, but we have a few options.  Basically,
> since
> >>>>> all
> >>>>>> the hashing algorithms have no export restrictions, we could:
> >>>>>> 1.  Remove the AES encrypt/decrypt until I can figure out what we
> have
> >>>> to
> >>>>>> do (and do it)
> >>>>>> 2.  I can try to research it today or tomorrow to figure out exactly
> >>>> what
> >>>>>> we have to do to include AES.
> >>>>>> 3.  Remove the entire package
> >>>>>>
> >>>>>> I’d like to see the hashing functions at least included because
> we’ve
> >>>> had
> >>>>>> a lot of requests for that functionality.   I also don’t want this
> to
> >>>>> hold
> >>>>>> things up, so please let me know your preference.  Regardless, I’m
> >>>> going
> >>>>> to
> >>>>>> start trying to figure out the restrictions so we’ll know for the
> >>>> future.
> >>>>>>
> >>>>>> Best,
> >>>>>> — C
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>> On Jul 11, 2017, at 05:58, Arina Yelchiyeva <
> >>>>> arina.yelchiyeva@gmail.com>
> >>>>>> wrote:
> >>>>>>>
> >>>>>>> @Jinfeng,
> >>>>>>> Let's discuss today on Drill hangout final cut-off date.
> >>>>>>>
> >>>>>>> @Charles
> >>>>>>> Since there were questions about AES in DRILL-5634, should we
> exclude
> >>>>> it
> >>>>>>> from the candidates for 1.11.0 release?
> >>>>>>>
> >>>>>>> @Boaz
> >>>>>>> Do we have any ETA when DRILL-5616 and DRILL-5665 will pass code
> >>>>> review?
> >>>>>>>
> >>>>>>> Also I believe we should include DRILL-5660 where we need to bump
> up
> >>>>>>> parquet metadata version.
> >>>>>>> @Vitalii
> >>>>>>> Do you have any ETA for the fix?
> >>>>>>>
> >>>>>>> Kind regards
> >>>>>>> Arina
> >>>>>>>
> >>>>>>> On Tue, Jul 11, 2017 at 6:02 AM, Boaz Ben-Zvi <bb...@mapr.com>
> >>>>> wrote:
> >>>>>>>
> >>>>>>>> Hi Arina,
> >>>>>>>>
> >>>>>>>>   Two PR have just been submitted, relating to the Hash Aggr
> >>>> Spill:
> >>>>>>>> DRILL-5616 and DRILL-5665 .
> >>>>>>>> Can these be considered for 1.11 as well ?
> >>>>>>>>
> >>>>>>>>   Thanks,
> >>>>>>>>
> >>>>>>>>             Boaz
> >>>>>>>>
> >>>>>>>> On 7/10/17, 1:49 PM, "Paul Rogers" <pr...@mapr.com> wrote:
> >>>>>>>>
> >>>>>>>>  DRILL-5601 is no longer a blocker for 1.11; we’ll defer that
> >>>> until
> >>>>>>>> after the release to allow others more time to work on other
> >>>> features
> >>>>>>>> rather than doing code reviews.
> >>>>>>>>
> >>>>>>>>  - Paul
> >>>>>>>>
> >>>>>>>>> On Jul 10, 2017, at 1:43 PM, Jinfeng Ni <jn...@apache.org> wrote:
> >>>>>>>>>
> >>>>>>>>> Hi Arina,
> >>>>>>>>>
> >>>>>>>>> It has been a while since we discussed 1.11.0 release.  Any
> update
> >>>>>>>> on the
> >>>>>>>>> new release plan?  Do we have a cut-off date ?
> >>>>>>>>>
> >>>>>>>>> In the past, I believe Drill release is a time-bounded, not
> feature
> >>>>>>>>> bounded. If certain fixes/features could not make it within a
> >>>> certain
> >>>>>>>>> time-frame, we had better defer them into future release, unless
> >>>>>>>> those
> >>>>>>>>> fixes/features are regarded as blocking issues for a new release.
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> On Tue, Jul 4, 2017 at 5:14 AM, Arina Yelchiyeva <
> >>>>>>>> arina.yelchiyeva@gmail.com
> >>>>>>>>>> wrote:
> >>>>>>>>>
> >>>>>>>>>> Latest update:
> >>>>>>>>>>
> >>>>>>>>>> We wait for the following Jiras before final cut off.
> >>>>>>>>>> 1. DRILL-5601  - Paul
> >>>>>>>>>> 2. DRILL-5420 - Kunal
> >>>>>>>>>> 3. DRILL-5634 - Charles
> >>>>>>>>>>
> >>>>>>>>>> Kind regards
> >>>>>>>>>> Arina
> >>>>>>>>>>
> >>>>>>>>>> On Fri, Jun 30, 2017 at 9:29 PM, Kunal Khatua <kkhatua@mapr.com
> >
> >>>>>>>> wrote:
> >>>>>>>>>>
> >>>>>>>>>>> Parth's already +1'd the process with a minor update. So, we're
> >>>>>>>> good to
> >>>>>>>>>> go
> >>>>>>>>>>> on DRILL-5420. I'll change the label on that to
> ready-to-commit.
> >>>>>>>>>>>
> >>>>>>>>>>> <http://www.mapr.com/>
> >>>>>>>>>>>
> >>>>>>>>>>> ________________________________
> >>>>>>>>>>> From: Parth Chandra <pa...@apache.org>
> >>>>>>>>>>> Sent: Thursday, June 29, 2017 9:44:39 AM
> >>>>>>>>>>> To: dev@drill.apache.org
> >>>>>>>>>>> Subject: Re: [DISCUSS] Drill 1.11.0 release
> >>>>>>>>>>>
> >>>>>>>>>>> I'm Ok with the pcap format plugin if Pauls comments are also
> >>>>>>>> addressed.
> >>>>>>>>>> If
> >>>>>>>>>>> the changes are done before your cutoff deadline,we can merge
> it
> >>>>>>>> in.
> >>>>>>>>>>>
> >>>>>>>>>>> On Thu, Jun 29, 2017 at 12:26 AM, Arina Yelchiyeva <
> >>>>>>>>>>> arina.yelchiyeva@gmail.com> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>>> Hi all,
> >>>>>>>>>>>>
> >>>>>>>>>>>> here is the list of Jira we are waiting before making final
> cut
> >>>>>>>> off:
> >>>>>>>>>>>> 1. DRILL-5601  - Paul
> >>>>>>>>>>>> 2. DRILL-3640  - Kunal
> >>>>>>>>>>>> 3. DRILL-5420 - Kunal
> >>>>>>>>>>>>
> >>>>>>>>>>>> Please let me know if there are any others Jiras we should
> wait
> >>>>>>>> for.
> >>>>>>>>>>>>
> >>>>>>>>>>>> @Charles,
> >>>>>>>>>>>> you have mentioned that you want to include some UDFs in Drill
> >>>>>>>> 1.11, do
> >>>>>>>>>>> you
> >>>>>>>>>>>> have any ETA for the PR?
> >>>>>>>>>>>>
> >>>>>>>>>>>> @Paul / Parth
> >>>>>>>>>>>> I see there is active CR for pcap format PR, do we want to
> >>>>>>>> include it
> >>>>>>>>>> in
> >>>>>>>>>>>> this release as well?
> >>>>>>>>>>>>
> >>>>>>>>>>>> Kind regards
> >>>>>>>>>>>> Arina
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> On Thu, Jun 29, 2017 at 10:05 AM, Arina Yelchiyeva <
> >>>>>>>>>>>> arina.yelchiyeva@gmail.com> wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>>> Parth, thanks a lot the instructions.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> On Wed, Jun 28, 2017 at 8:24 PM, Parth Chandra <
> >>>>>>>> parthc@apache.org>
> >>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> FWIW, I put release instructions culled from gists written
> by
> >>>>>>>>>> previous
> >>>>>>>>>>>>>> release managers into an updated gist here [1]
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> HTH
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Parth
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> [1] https://github.com/parthchandra/drill/wiki/Drill-
> >>>>>>>>>> Release-Process
> >>>>>>>>>>>>>> <https://github.com/parthchandra/drill/wiki/Drill-
> >>>>>>>> Release-Process>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> On Tue, Jun 27, 2017 at 5:24 PM, Paul Rogers <
> >>>> progers@mapr.com>
> >>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Hi Arina,
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> I have two projects in progress: one is a candidate for
> 1.11,
> >>>>>>>> the
> >>>>>>>>>>>> other
> >>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>> not.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> The “managed external sort” is the good candidate for 1.11:
> >>>>>>>> The PR
> >>>>>>>>>>> for
> >>>>>>>>>>>>>>> DRILL-5601 is being reviewed. Getting that in will allow us
> >>>> to
> >>>>>>>>>>> enable
> >>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>> “new” external sort by default.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> The other project is to address Drill’s memory
> fragmentation
> >>>>>>>> issue
> >>>>>>>>>>>> with
> >>>>>>>>>>>>>>> vectors over 16 MB in size. Two PRs have been open for a
> >>>> while.
> >>>>>>>>>> They
> >>>>>>>>>>>> are
> >>>>>>>>>>>>>>> the first of several. Each takes a while to review. So, the
> >>>>>>>>>> project
> >>>>>>>>>>>> does
> >>>>>>>>>>>>>>> not fit 1.11. The work can go into the next release
> instead.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> - Paul
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>
> >>
>
>

Re: [DISCUSS] Drill 1.11.0 release

Posted by Paul Rogers <pr...@mapr.com>.

I withdraw my suggestion to fix DRILL-5660. Was a big hassle for us to track down the incompatibility the issue internally, but production users probably don’t attempt to use one build of Drill with metadata files created from another. We can worry about the issue later only if a user actually runs into it.

As suggested earlier, let’s just document this obscure case in a release note. 

- Paul

> On Jul 12, 2017, at 3:04 PM, Jinfeng Ni <jn...@apache.org> wrote:
> 
> Upgrade from 1.10 to 1.11 is irreversible only when parquet metadata cache
> file is involved.
> 
> I might be wrong, but I thought that if someone is running on 1.10.0, and
> he wants to go back to 1.0.0, where the first version of parquet meatadata
> cache was introduced, can we guarantee that Drillbit would work correctly?
> I'm trying to understand what DRILL-5660 asks for is a new requirement, or
> an requirement being there starting from Drill 1.0.0?
> 
> Regression failures are normally regarded as release blocker.
> 
> 
> 
> On Wed, Jul 12, 2017 at 2:54 PM, Paul Rogers <pr...@mapr.com> wrote:
> 
>> If we don’t do DRILL-5660 for 1.11 then we don’t need to do it at all, so
>> that is a time savings. We could handle this via a release note item that
>> simply states that an upgrade from 1.10 to 1.11 is irreversible: we do not
>> support the ability to fall back to the old version. Everyone OK with that?
>> 
>> We are seeing some stress test regression failures that, I’m told, will
>> show up as JIRA entries shortly.
>> 
>> - Paul
>> 
>>> On Jul 12, 2017, at 11:05 AM, Jinfeng Ni <jn...@apache.org> wrote:
>>> 
>>> I put some analysis in DRILL-5468.
>>> 
>>> Also, IMHO, DRILL-5660 is not a release blocker.
>>> 
>>> 
>>> 
>>> On Tue, Jul 11, 2017 at 11:27 AM, Parth Chandra <pa...@apache.org>
>> wrote:
>>> 
>>>> Re DRILL-5634:
>>>> From the Apache commons web page [1] -
>>>> 
>>>> Please note that Apache Commons Crypto doesn't implement the
>> cryptographic
>>>> algorithm such as AES directly.* It wraps to Openssl or JCE which
>> implement
>>>> the algorithms.*
>>>> 
>>>> 
>>>> Since we are cleared for using Openssl and JCE, we are OK to include the
>>>> AES code in the UDFs.
>>>> 
>>>> 
>>>> Parth
>>>> 
>>>> [1] https://commons.apache.org/proper/commons-crypto/
>>>> 
>>>> On Tue, Jul 11, 2017 at 10:56 AM, Arina Yelchiyeva <
>>>> arina.yelchiyeva@gmail.com> wrote:
>>>> 
>>>>> During today hangouts we agreed on approximate cut-off date - 14 July,
>>>> 2017
>>>>> (date may be shifted due to the blockers).
>>>>> I'll start preparing test RC on Thursday to solve any release
>> preparation
>>>>> issues beforehand.
>>>>> 
>>>>> *Blockers*
>>>>> DRILL-5660 - Vitalii
>>>>> 
>>>>> *Possible blockers*
>>>>> DRILL-5468 - Jinfeng
>>>>> NPE on CSV source - Paul
>>>>> DRILL-5659 - Parth
>>>>> 
>>>>> *Will be included if done before the cut-off date*
>>>>> DRILL-5616 / DRILL-5665 - Boaz
>>>>> DRILL-5634 - Parth will take a look at Apache commons implementation
>> and
>>>>> update.
>>>>> 
>>>>> Kind regards
>>>>> Arina
>>>>> 
>>>>> On Tue, Jul 11, 2017 at 3:14 PM, Charles Givre <cg...@gmail.com>
>> wrote:
>>>>> 
>>>>>> Hi Arina,
>>>>>> I can’t make the hangout, but we have a few options.  Basically, since
>>>>> all
>>>>>> the hashing algorithms have no export restrictions, we could:
>>>>>> 1.  Remove the AES encrypt/decrypt until I can figure out what we have
>>>> to
>>>>>> do (and do it)
>>>>>> 2.  I can try to research it today or tomorrow to figure out exactly
>>>> what
>>>>>> we have to do to include AES.
>>>>>> 3.  Remove the entire package
>>>>>> 
>>>>>> I’d like to see the hashing functions at least included because we’ve
>>>> had
>>>>>> a lot of requests for that functionality.   I also don’t want this to
>>>>> hold
>>>>>> things up, so please let me know your preference.  Regardless, I’m
>>>> going
>>>>> to
>>>>>> start trying to figure out the restrictions so we’ll know for the
>>>> future.
>>>>>> 
>>>>>> Best,
>>>>>> — C
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>>> On Jul 11, 2017, at 05:58, Arina Yelchiyeva <
>>>>> arina.yelchiyeva@gmail.com>
>>>>>> wrote:
>>>>>>> 
>>>>>>> @Jinfeng,
>>>>>>> Let's discuss today on Drill hangout final cut-off date.
>>>>>>> 
>>>>>>> @Charles
>>>>>>> Since there were questions about AES in DRILL-5634, should we exclude
>>>>> it
>>>>>>> from the candidates for 1.11.0 release?
>>>>>>> 
>>>>>>> @Boaz
>>>>>>> Do we have any ETA when DRILL-5616 and DRILL-5665 will pass code
>>>>> review?
>>>>>>> 
>>>>>>> Also I believe we should include DRILL-5660 where we need to bump up
>>>>>>> parquet metadata version.
>>>>>>> @Vitalii
>>>>>>> Do you have any ETA for the fix?
>>>>>>> 
>>>>>>> Kind regards
>>>>>>> Arina
>>>>>>> 
>>>>>>> On Tue, Jul 11, 2017 at 6:02 AM, Boaz Ben-Zvi <bb...@mapr.com>
>>>>> wrote:
>>>>>>> 
>>>>>>>> Hi Arina,
>>>>>>>> 
>>>>>>>>   Two PR have just been submitted, relating to the Hash Aggr
>>>> Spill:
>>>>>>>> DRILL-5616 and DRILL-5665 .
>>>>>>>> Can these be considered for 1.11 as well ?
>>>>>>>> 
>>>>>>>>   Thanks,
>>>>>>>> 
>>>>>>>>             Boaz
>>>>>>>> 
>>>>>>>> On 7/10/17, 1:49 PM, "Paul Rogers" <pr...@mapr.com> wrote:
>>>>>>>> 
>>>>>>>>  DRILL-5601 is no longer a blocker for 1.11; we’ll defer that
>>>> until
>>>>>>>> after the release to allow others more time to work on other
>>>> features
>>>>>>>> rather than doing code reviews.
>>>>>>>> 
>>>>>>>>  - Paul
>>>>>>>> 
>>>>>>>>> On Jul 10, 2017, at 1:43 PM, Jinfeng Ni <jn...@apache.org> wrote:
>>>>>>>>> 
>>>>>>>>> Hi Arina,
>>>>>>>>> 
>>>>>>>>> It has been a while since we discussed 1.11.0 release.  Any update
>>>>>>>> on the
>>>>>>>>> new release plan?  Do we have a cut-off date ?
>>>>>>>>> 
>>>>>>>>> In the past, I believe Drill release is a time-bounded, not feature
>>>>>>>>> bounded. If certain fixes/features could not make it within a
>>>> certain
>>>>>>>>> time-frame, we had better defer them into future release, unless
>>>>>>>> those
>>>>>>>>> fixes/features are regarded as blocking issues for a new release.
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> On Tue, Jul 4, 2017 at 5:14 AM, Arina Yelchiyeva <
>>>>>>>> arina.yelchiyeva@gmail.com
>>>>>>>>>> wrote:
>>>>>>>>> 
>>>>>>>>>> Latest update:
>>>>>>>>>> 
>>>>>>>>>> We wait for the following Jiras before final cut off.
>>>>>>>>>> 1. DRILL-5601  - Paul
>>>>>>>>>> 2. DRILL-5420 - Kunal
>>>>>>>>>> 3. DRILL-5634 - Charles
>>>>>>>>>> 
>>>>>>>>>> Kind regards
>>>>>>>>>> Arina
>>>>>>>>>> 
>>>>>>>>>> On Fri, Jun 30, 2017 at 9:29 PM, Kunal Khatua <kk...@mapr.com>
>>>>>>>> wrote:
>>>>>>>>>> 
>>>>>>>>>>> Parth's already +1'd the process with a minor update. So, we're
>>>>>>>> good to
>>>>>>>>>> go
>>>>>>>>>>> on DRILL-5420. I'll change the label on that to ready-to-commit.
>>>>>>>>>>> 
>>>>>>>>>>> <http://www.mapr.com/>
>>>>>>>>>>> 
>>>>>>>>>>> ________________________________
>>>>>>>>>>> From: Parth Chandra <pa...@apache.org>
>>>>>>>>>>> Sent: Thursday, June 29, 2017 9:44:39 AM
>>>>>>>>>>> To: dev@drill.apache.org
>>>>>>>>>>> Subject: Re: [DISCUSS] Drill 1.11.0 release
>>>>>>>>>>> 
>>>>>>>>>>> I'm Ok with the pcap format plugin if Pauls comments are also
>>>>>>>> addressed.
>>>>>>>>>> If
>>>>>>>>>>> the changes are done before your cutoff deadline,we can merge it
>>>>>>>> in.
>>>>>>>>>>> 
>>>>>>>>>>> On Thu, Jun 29, 2017 at 12:26 AM, Arina Yelchiyeva <
>>>>>>>>>>> arina.yelchiyeva@gmail.com> wrote:
>>>>>>>>>>> 
>>>>>>>>>>>> Hi all,
>>>>>>>>>>>> 
>>>>>>>>>>>> here is the list of Jira we are waiting before making final cut
>>>>>>>> off:
>>>>>>>>>>>> 1. DRILL-5601  - Paul
>>>>>>>>>>>> 2. DRILL-3640  - Kunal
>>>>>>>>>>>> 3. DRILL-5420 - Kunal
>>>>>>>>>>>> 
>>>>>>>>>>>> Please let me know if there are any others Jiras we should wait
>>>>>>>> for.
>>>>>>>>>>>> 
>>>>>>>>>>>> @Charles,
>>>>>>>>>>>> you have mentioned that you want to include some UDFs in Drill
>>>>>>>> 1.11, do
>>>>>>>>>>> you
>>>>>>>>>>>> have any ETA for the PR?
>>>>>>>>>>>> 
>>>>>>>>>>>> @Paul / Parth
>>>>>>>>>>>> I see there is active CR for pcap format PR, do we want to
>>>>>>>> include it
>>>>>>>>>> in
>>>>>>>>>>>> this release as well?
>>>>>>>>>>>> 
>>>>>>>>>>>> Kind regards
>>>>>>>>>>>> Arina
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> On Thu, Jun 29, 2017 at 10:05 AM, Arina Yelchiyeva <
>>>>>>>>>>>> arina.yelchiyeva@gmail.com> wrote:
>>>>>>>>>>>> 
>>>>>>>>>>>>> Parth, thanks a lot the instructions.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> On Wed, Jun 28, 2017 at 8:24 PM, Parth Chandra <
>>>>>>>> parthc@apache.org>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> FWIW, I put release instructions culled from gists written by
>>>>>>>>>> previous
>>>>>>>>>>>>>> release managers into an updated gist here [1]
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> HTH
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Parth
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> [1] https://github.com/parthchandra/drill/wiki/Drill-
>>>>>>>>>> Release-Process
>>>>>>>>>>>>>> <https://github.com/parthchandra/drill/wiki/Drill-
>>>>>>>> Release-Process>
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> On Tue, Jun 27, 2017 at 5:24 PM, Paul Rogers <
>>>> progers@mapr.com>
>>>>>>>>>>> wrote:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Hi Arina,
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> I have two projects in progress: one is a candidate for 1.11,
>>>>>>>> the
>>>>>>>>>>>> other
>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>> not.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> The “managed external sort” is the good candidate for 1.11:
>>>>>>>> The PR
>>>>>>>>>>> for
>>>>>>>>>>>>>>> DRILL-5601 is being reviewed. Getting that in will allow us
>>>> to
>>>>>>>>>>> enable
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>> “new” external sort by default.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> The other project is to address Drill’s memory fragmentation
>>>>>>>> issue
>>>>>>>>>>>> with
>>>>>>>>>>>>>>> vectors over 16 MB in size. Two PRs have been open for a
>>>> while.
>>>>>>>>>> They
>>>>>>>>>>>> are
>>>>>>>>>>>>>>> the first of several. Each takes a while to review. So, the
>>>>>>>>>> project
>>>>>>>>>>>> does
>>>>>>>>>>>>>>> not fit 1.11. The work can go into the next release instead.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> - Paul
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>> 
>>

Re: [DISCUSS] Drill 1.11.0 release

Posted by Jinfeng Ni <jn...@apache.org>.

Upgrade from 1.10 to 1.11 is irreversible only when parquet metadata cache
file is involved.

I might be wrong, but I thought that if someone is running on 1.10.0, and
he wants to go back to 1.0.0, where the first version of parquet meatadata
cache was introduced, can we guarantee that Drillbit would work correctly?
I'm trying to understand what DRILL-5660 asks for is a new requirement, or
an requirement being there starting from Drill 1.0.0?

Regression failures are normally regarded as release blocker.



On Wed, Jul 12, 2017 at 2:54 PM, Paul Rogers <pr...@mapr.com> wrote:

> If we don’t do DRILL-5660 for 1.11 then we don’t need to do it at all, so
> that is a time savings. We could handle this via a release note item that
> simply states that an upgrade from 1.10 to 1.11 is irreversible: we do not
> support the ability to fall back to the old version. Everyone OK with that?
>
> We are seeing some stress test regression failures that, I’m told, will
> show up as JIRA entries shortly.
>
> - Paul
>
> > On Jul 12, 2017, at 11:05 AM, Jinfeng Ni <jn...@apache.org> wrote:
> >
> > I put some analysis in DRILL-5468.
> >
> > Also, IMHO, DRILL-5660 is not a release blocker.
> >
> >
> >
> > On Tue, Jul 11, 2017 at 11:27 AM, Parth Chandra <pa...@apache.org>
> wrote:
> >
> >> Re DRILL-5634:
> >> From the Apache commons web page [1] -
> >>
> >> Please note that Apache Commons Crypto doesn't implement the
> cryptographic
> >> algorithm such as AES directly.* It wraps to Openssl or JCE which
> implement
> >> the algorithms.*
> >>
> >>
> >> Since we are cleared for using Openssl and JCE, we are OK to include the
> >> AES code in the UDFs.
> >>
> >>
> >> Parth
> >>
> >> [1] https://commons.apache.org/proper/commons-crypto/
> >>
> >> On Tue, Jul 11, 2017 at 10:56 AM, Arina Yelchiyeva <
> >> arina.yelchiyeva@gmail.com> wrote:
> >>
> >>> During today hangouts we agreed on approximate cut-off date - 14 July,
> >> 2017
> >>> (date may be shifted due to the blockers).
> >>> I'll start preparing test RC on Thursday to solve any release
> preparation
> >>> issues beforehand.
> >>>
> >>> *Blockers*
> >>> DRILL-5660 - Vitalii
> >>>
> >>> *Possible blockers*
> >>> DRILL-5468 - Jinfeng
> >>> NPE on CSV source - Paul
> >>> DRILL-5659 - Parth
> >>>
> >>> *Will be included if done before the cut-off date*
> >>> DRILL-5616 / DRILL-5665 - Boaz
> >>> DRILL-5634 - Parth will take a look at Apache commons implementation
> and
> >>> update.
> >>>
> >>> Kind regards
> >>> Arina
> >>>
> >>> On Tue, Jul 11, 2017 at 3:14 PM, Charles Givre <cg...@gmail.com>
> wrote:
> >>>
> >>>> Hi Arina,
> >>>> I can’t make the hangout, but we have a few options.  Basically, since
> >>> all
> >>>> the hashing algorithms have no export restrictions, we could:
> >>>> 1.  Remove the AES encrypt/decrypt until I can figure out what we have
> >> to
> >>>> do (and do it)
> >>>> 2.  I can try to research it today or tomorrow to figure out exactly
> >> what
> >>>> we have to do to include AES.
> >>>> 3.  Remove the entire package
> >>>>
> >>>> I’d like to see the hashing functions at least included because we’ve
> >> had
> >>>> a lot of requests for that functionality.   I also don’t want this to
> >>> hold
> >>>> things up, so please let me know your preference.  Regardless, I’m
> >> going
> >>> to
> >>>> start trying to figure out the restrictions so we’ll know for the
> >> future.
> >>>>
> >>>> Best,
> >>>> — C
> >>>>
> >>>>
> >>>>
> >>>>> On Jul 11, 2017, at 05:58, Arina Yelchiyeva <
> >>> arina.yelchiyeva@gmail.com>
> >>>> wrote:
> >>>>>
> >>>>> @Jinfeng,
> >>>>> Let's discuss today on Drill hangout final cut-off date.
> >>>>>
> >>>>> @Charles
> >>>>> Since there were questions about AES in DRILL-5634, should we exclude
> >>> it
> >>>>> from the candidates for 1.11.0 release?
> >>>>>
> >>>>> @Boaz
> >>>>> Do we have any ETA when DRILL-5616 and DRILL-5665 will pass code
> >>> review?
> >>>>>
> >>>>> Also I believe we should include DRILL-5660 where we need to bump up
> >>>>> parquet metadata version.
> >>>>> @Vitalii
> >>>>> Do you have any ETA for the fix?
> >>>>>
> >>>>> Kind regards
> >>>>> Arina
> >>>>>
> >>>>> On Tue, Jul 11, 2017 at 6:02 AM, Boaz Ben-Zvi <bb...@mapr.com>
> >>> wrote:
> >>>>>
> >>>>>> Hi Arina,
> >>>>>>
> >>>>>>    Two PR have just been submitted, relating to the Hash Aggr
> >> Spill:
> >>>>>> DRILL-5616 and DRILL-5665 .
> >>>>>> Can these be considered for 1.11 as well ?
> >>>>>>
> >>>>>>    Thanks,
> >>>>>>
> >>>>>>              Boaz
> >>>>>>
> >>>>>> On 7/10/17, 1:49 PM, "Paul Rogers" <pr...@mapr.com> wrote:
> >>>>>>
> >>>>>>   DRILL-5601 is no longer a blocker for 1.11; we’ll defer that
> >> until
> >>>>>> after the release to allow others more time to work on other
> >> features
> >>>>>> rather than doing code reviews.
> >>>>>>
> >>>>>>   - Paul
> >>>>>>
> >>>>>>> On Jul 10, 2017, at 1:43 PM, Jinfeng Ni <jn...@apache.org> wrote:
> >>>>>>>
> >>>>>>> Hi Arina,
> >>>>>>>
> >>>>>>> It has been a while since we discussed 1.11.0 release.  Any update
> >>>>>> on the
> >>>>>>> new release plan?  Do we have a cut-off date ?
> >>>>>>>
> >>>>>>> In the past, I believe Drill release is a time-bounded, not feature
> >>>>>>> bounded. If certain fixes/features could not make it within a
> >> certain
> >>>>>>> time-frame, we had better defer them into future release, unless
> >>>>>> those
> >>>>>>> fixes/features are regarded as blocking issues for a new release.
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> On Tue, Jul 4, 2017 at 5:14 AM, Arina Yelchiyeva <
> >>>>>> arina.yelchiyeva@gmail.com
> >>>>>>>> wrote:
> >>>>>>>
> >>>>>>>> Latest update:
> >>>>>>>>
> >>>>>>>> We wait for the following Jiras before final cut off.
> >>>>>>>> 1. DRILL-5601  - Paul
> >>>>>>>> 2. DRILL-5420 - Kunal
> >>>>>>>> 3. DRILL-5634 - Charles
> >>>>>>>>
> >>>>>>>> Kind regards
> >>>>>>>> Arina
> >>>>>>>>
> >>>>>>>> On Fri, Jun 30, 2017 at 9:29 PM, Kunal Khatua <kk...@mapr.com>
> >>>>>> wrote:
> >>>>>>>>
> >>>>>>>>> Parth's already +1'd the process with a minor update. So, we're
> >>>>>> good to
> >>>>>>>> go
> >>>>>>>>> on DRILL-5420. I'll change the label on that to ready-to-commit.
> >>>>>>>>>
> >>>>>>>>> <http://www.mapr.com/>
> >>>>>>>>>
> >>>>>>>>> ________________________________
> >>>>>>>>> From: Parth Chandra <pa...@apache.org>
> >>>>>>>>> Sent: Thursday, June 29, 2017 9:44:39 AM
> >>>>>>>>> To: dev@drill.apache.org
> >>>>>>>>> Subject: Re: [DISCUSS] Drill 1.11.0 release
> >>>>>>>>>
> >>>>>>>>> I'm Ok with the pcap format plugin if Pauls comments are also
> >>>>>> addressed.
> >>>>>>>> If
> >>>>>>>>> the changes are done before your cutoff deadline,we can merge it
> >>>>>> in.
> >>>>>>>>>
> >>>>>>>>> On Thu, Jun 29, 2017 at 12:26 AM, Arina Yelchiyeva <
> >>>>>>>>> arina.yelchiyeva@gmail.com> wrote:
> >>>>>>>>>
> >>>>>>>>>> Hi all,
> >>>>>>>>>>
> >>>>>>>>>> here is the list of Jira we are waiting before making final cut
> >>>>>> off:
> >>>>>>>>>> 1. DRILL-5601  - Paul
> >>>>>>>>>> 2. DRILL-3640  - Kunal
> >>>>>>>>>> 3. DRILL-5420 - Kunal
> >>>>>>>>>>
> >>>>>>>>>> Please let me know if there are any others Jiras we should wait
> >>>>>> for.
> >>>>>>>>>>
> >>>>>>>>>> @Charles,
> >>>>>>>>>> you have mentioned that you want to include some UDFs in Drill
> >>>>>> 1.11, do
> >>>>>>>>> you
> >>>>>>>>>> have any ETA for the PR?
> >>>>>>>>>>
> >>>>>>>>>> @Paul / Parth
> >>>>>>>>>> I see there is active CR for pcap format PR, do we want to
> >>>>>> include it
> >>>>>>>> in
> >>>>>>>>>> this release as well?
> >>>>>>>>>>
> >>>>>>>>>> Kind regards
> >>>>>>>>>> Arina
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> On Thu, Jun 29, 2017 at 10:05 AM, Arina Yelchiyeva <
> >>>>>>>>>> arina.yelchiyeva@gmail.com> wrote:
> >>>>>>>>>>
> >>>>>>>>>>> Parth, thanks a lot the instructions.
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> On Wed, Jun 28, 2017 at 8:24 PM, Parth Chandra <
> >>>>>> parthc@apache.org>
> >>>>>>>>>> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>>> FWIW, I put release instructions culled from gists written by
> >>>>>>>> previous
> >>>>>>>>>>>> release managers into an updated gist here [1]
> >>>>>>>>>>>>
> >>>>>>>>>>>> HTH
> >>>>>>>>>>>>
> >>>>>>>>>>>> Parth
> >>>>>>>>>>>>
> >>>>>>>>>>>> [1] https://github.com/parthchandra/drill/wiki/Drill-
> >>>>>>>> Release-Process
> >>>>>>>>>>>> <https://github.com/parthchandra/drill/wiki/Drill-
> >>>>>> Release-Process>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> On Tue, Jun 27, 2017 at 5:24 PM, Paul Rogers <
> >> progers@mapr.com>
> >>>>>>>>> wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>>> Hi Arina,
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> I have two projects in progress: one is a candidate for 1.11,
> >>>>>> the
> >>>>>>>>>> other
> >>>>>>>>>>>> is
> >>>>>>>>>>>>> not.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> The “managed external sort” is the good candidate for 1.11:
> >>>>>> The PR
> >>>>>>>>> for
> >>>>>>>>>>>>> DRILL-5601 is being reviewed. Getting that in will allow us
> >> to
> >>>>>>>>> enable
> >>>>>>>>>>>> the
> >>>>>>>>>>>>> “new” external sort by default.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> The other project is to address Drill’s memory fragmentation
> >>>>>> issue
> >>>>>>>>>> with
> >>>>>>>>>>>>> vectors over 16 MB in size. Two PRs have been open for a
> >> while.
> >>>>>>>> They
> >>>>>>>>>> are
> >>>>>>>>>>>>> the first of several. Each takes a while to review. So, the
> >>>>>>>> project
> >>>>>>>>>> does
> >>>>>>>>>>>>> not fit 1.11. The work can go into the next release instead.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> - Paul
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>
> >>>>
> >>>
> >>
>
>

Re: [DISCUSS] Drill 1.11.0 release

Posted by Paul Rogers <pr...@mapr.com>.

If we don’t do DRILL-5660 for 1.11 then we don’t need to do it at all, so that is a time savings. We could handle this via a release note item that simply states that an upgrade from 1.10 to 1.11 is irreversible: we do not support the ability to fall back to the old version. Everyone OK with that?

We are seeing some stress test regression failures that, I’m told, will show up as JIRA entries shortly.

- Paul

> On Jul 12, 2017, at 11:05 AM, Jinfeng Ni <jn...@apache.org> wrote:
> 
> I put some analysis in DRILL-5468.
> 
> Also, IMHO, DRILL-5660 is not a release blocker.
> 
> 
> 
> On Tue, Jul 11, 2017 at 11:27 AM, Parth Chandra <pa...@apache.org> wrote:
> 
>> Re DRILL-5634:
>> From the Apache commons web page [1] -
>> 
>> Please note that Apache Commons Crypto doesn't implement the cryptographic
>> algorithm such as AES directly.* It wraps to Openssl or JCE which implement
>> the algorithms.*
>> 
>> 
>> Since we are cleared for using Openssl and JCE, we are OK to include the
>> AES code in the UDFs.
>> 
>> 
>> Parth
>> 
>> [1] https://commons.apache.org/proper/commons-crypto/
>> 
>> On Tue, Jul 11, 2017 at 10:56 AM, Arina Yelchiyeva <
>> arina.yelchiyeva@gmail.com> wrote:
>> 
>>> During today hangouts we agreed on approximate cut-off date - 14 July,
>> 2017
>>> (date may be shifted due to the blockers).
>>> I'll start preparing test RC on Thursday to solve any release preparation
>>> issues beforehand.
>>> 
>>> *Blockers*
>>> DRILL-5660 - Vitalii
>>> 
>>> *Possible blockers*
>>> DRILL-5468 - Jinfeng
>>> NPE on CSV source - Paul
>>> DRILL-5659 - Parth
>>> 
>>> *Will be included if done before the cut-off date*
>>> DRILL-5616 / DRILL-5665 - Boaz
>>> DRILL-5634 - Parth will take a look at Apache commons implementation and
>>> update.
>>> 
>>> Kind regards
>>> Arina
>>> 
>>> On Tue, Jul 11, 2017 at 3:14 PM, Charles Givre <cg...@gmail.com> wrote:
>>> 
>>>> Hi Arina,
>>>> I can’t make the hangout, but we have a few options.  Basically, since
>>> all
>>>> the hashing algorithms have no export restrictions, we could:
>>>> 1.  Remove the AES encrypt/decrypt until I can figure out what we have
>> to
>>>> do (and do it)
>>>> 2.  I can try to research it today or tomorrow to figure out exactly
>> what
>>>> we have to do to include AES.
>>>> 3.  Remove the entire package
>>>> 
>>>> I’d like to see the hashing functions at least included because we’ve
>> had
>>>> a lot of requests for that functionality.   I also don’t want this to
>>> hold
>>>> things up, so please let me know your preference.  Regardless, I’m
>> going
>>> to
>>>> start trying to figure out the restrictions so we’ll know for the
>> future.
>>>> 
>>>> Best,
>>>> — C
>>>> 
>>>> 
>>>> 
>>>>> On Jul 11, 2017, at 05:58, Arina Yelchiyeva <
>>> arina.yelchiyeva@gmail.com>
>>>> wrote:
>>>>> 
>>>>> @Jinfeng,
>>>>> Let's discuss today on Drill hangout final cut-off date.
>>>>> 
>>>>> @Charles
>>>>> Since there were questions about AES in DRILL-5634, should we exclude
>>> it
>>>>> from the candidates for 1.11.0 release?
>>>>> 
>>>>> @Boaz
>>>>> Do we have any ETA when DRILL-5616 and DRILL-5665 will pass code
>>> review?
>>>>> 
>>>>> Also I believe we should include DRILL-5660 where we need to bump up
>>>>> parquet metadata version.
>>>>> @Vitalii
>>>>> Do you have any ETA for the fix?
>>>>> 
>>>>> Kind regards
>>>>> Arina
>>>>> 
>>>>> On Tue, Jul 11, 2017 at 6:02 AM, Boaz Ben-Zvi <bb...@mapr.com>
>>> wrote:
>>>>> 
>>>>>> Hi Arina,
>>>>>> 
>>>>>>    Two PR have just been submitted, relating to the Hash Aggr
>> Spill:
>>>>>> DRILL-5616 and DRILL-5665 .
>>>>>> Can these be considered for 1.11 as well ?
>>>>>> 
>>>>>>    Thanks,
>>>>>> 
>>>>>>              Boaz
>>>>>> 
>>>>>> On 7/10/17, 1:49 PM, "Paul Rogers" <pr...@mapr.com> wrote:
>>>>>> 
>>>>>>   DRILL-5601 is no longer a blocker for 1.11; we’ll defer that
>> until
>>>>>> after the release to allow others more time to work on other
>> features
>>>>>> rather than doing code reviews.
>>>>>> 
>>>>>>   - Paul
>>>>>> 
>>>>>>> On Jul 10, 2017, at 1:43 PM, Jinfeng Ni <jn...@apache.org> wrote:
>>>>>>> 
>>>>>>> Hi Arina,
>>>>>>> 
>>>>>>> It has been a while since we discussed 1.11.0 release.  Any update
>>>>>> on the
>>>>>>> new release plan?  Do we have a cut-off date ?
>>>>>>> 
>>>>>>> In the past, I believe Drill release is a time-bounded, not feature
>>>>>>> bounded. If certain fixes/features could not make it within a
>> certain
>>>>>>> time-frame, we had better defer them into future release, unless
>>>>>> those
>>>>>>> fixes/features are regarded as blocking issues for a new release.
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> On Tue, Jul 4, 2017 at 5:14 AM, Arina Yelchiyeva <
>>>>>> arina.yelchiyeva@gmail.com
>>>>>>>> wrote:
>>>>>>> 
>>>>>>>> Latest update:
>>>>>>>> 
>>>>>>>> We wait for the following Jiras before final cut off.
>>>>>>>> 1. DRILL-5601  - Paul
>>>>>>>> 2. DRILL-5420 - Kunal
>>>>>>>> 3. DRILL-5634 - Charles
>>>>>>>> 
>>>>>>>> Kind regards
>>>>>>>> Arina
>>>>>>>> 
>>>>>>>> On Fri, Jun 30, 2017 at 9:29 PM, Kunal Khatua <kk...@mapr.com>
>>>>>> wrote:
>>>>>>>> 
>>>>>>>>> Parth's already +1'd the process with a minor update. So, we're
>>>>>> good to
>>>>>>>> go
>>>>>>>>> on DRILL-5420. I'll change the label on that to ready-to-commit.
>>>>>>>>> 
>>>>>>>>> <http://www.mapr.com/>
>>>>>>>>> 
>>>>>>>>> ________________________________
>>>>>>>>> From: Parth Chandra <pa...@apache.org>
>>>>>>>>> Sent: Thursday, June 29, 2017 9:44:39 AM
>>>>>>>>> To: dev@drill.apache.org
>>>>>>>>> Subject: Re: [DISCUSS] Drill 1.11.0 release
>>>>>>>>> 
>>>>>>>>> I'm Ok with the pcap format plugin if Pauls comments are also
>>>>>> addressed.
>>>>>>>> If
>>>>>>>>> the changes are done before your cutoff deadline,we can merge it
>>>>>> in.
>>>>>>>>> 
>>>>>>>>> On Thu, Jun 29, 2017 at 12:26 AM, Arina Yelchiyeva <
>>>>>>>>> arina.yelchiyeva@gmail.com> wrote:
>>>>>>>>> 
>>>>>>>>>> Hi all,
>>>>>>>>>> 
>>>>>>>>>> here is the list of Jira we are waiting before making final cut
>>>>>> off:
>>>>>>>>>> 1. DRILL-5601  - Paul
>>>>>>>>>> 2. DRILL-3640  - Kunal
>>>>>>>>>> 3. DRILL-5420 - Kunal
>>>>>>>>>> 
>>>>>>>>>> Please let me know if there are any others Jiras we should wait
>>>>>> for.
>>>>>>>>>> 
>>>>>>>>>> @Charles,
>>>>>>>>>> you have mentioned that you want to include some UDFs in Drill
>>>>>> 1.11, do
>>>>>>>>> you
>>>>>>>>>> have any ETA for the PR?
>>>>>>>>>> 
>>>>>>>>>> @Paul / Parth
>>>>>>>>>> I see there is active CR for pcap format PR, do we want to
>>>>>> include it
>>>>>>>> in
>>>>>>>>>> this release as well?
>>>>>>>>>> 
>>>>>>>>>> Kind regards
>>>>>>>>>> Arina
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> On Thu, Jun 29, 2017 at 10:05 AM, Arina Yelchiyeva <
>>>>>>>>>> arina.yelchiyeva@gmail.com> wrote:
>>>>>>>>>> 
>>>>>>>>>>> Parth, thanks a lot the instructions.
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> On Wed, Jun 28, 2017 at 8:24 PM, Parth Chandra <
>>>>>> parthc@apache.org>
>>>>>>>>>> wrote:
>>>>>>>>>>> 
>>>>>>>>>>>> FWIW, I put release instructions culled from gists written by
>>>>>>>> previous
>>>>>>>>>>>> release managers into an updated gist here [1]
>>>>>>>>>>>> 
>>>>>>>>>>>> HTH
>>>>>>>>>>>> 
>>>>>>>>>>>> Parth
>>>>>>>>>>>> 
>>>>>>>>>>>> [1] https://github.com/parthchandra/drill/wiki/Drill-
>>>>>>>> Release-Process
>>>>>>>>>>>> <https://github.com/parthchandra/drill/wiki/Drill-
>>>>>> Release-Process>
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> On Tue, Jun 27, 2017 at 5:24 PM, Paul Rogers <
>> progers@mapr.com>
>>>>>>>>> wrote:
>>>>>>>>>>>> 
>>>>>>>>>>>>> Hi Arina,
>>>>>>>>>>>>> 
>>>>>>>>>>>>> I have two projects in progress: one is a candidate for 1.11,
>>>>>> the
>>>>>>>>>> other
>>>>>>>>>>>> is
>>>>>>>>>>>>> not.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> The “managed external sort” is the good candidate for 1.11:
>>>>>> The PR
>>>>>>>>> for
>>>>>>>>>>>>> DRILL-5601 is being reviewed. Getting that in will allow us
>> to
>>>>>>>>> enable
>>>>>>>>>>>> the
>>>>>>>>>>>>> “new” external sort by default.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> The other project is to address Drill’s memory fragmentation
>>>>>> issue
>>>>>>>>>> with
>>>>>>>>>>>>> vectors over 16 MB in size. Two PRs have been open for a
>> while.
>>>>>>>> They
>>>>>>>>>> are
>>>>>>>>>>>>> the first of several. Each takes a while to review. So, the
>>>>>>>> project
>>>>>>>>>> does
>>>>>>>>>>>>> not fit 1.11. The work can go into the next release instead.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>> 
>>>>>>>>>>>>> - Paul
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>> 
>>>> 
>>> 
>>

Re: [DISCUSS] Drill 1.11.0 release

Posted by Jinfeng Ni <jn...@apache.org>.

I put some analysis in DRILL-5468.

Also, IMHO, DRILL-5660 is not a release blocker.



On Tue, Jul 11, 2017 at 11:27 AM, Parth Chandra <pa...@apache.org> wrote:

> Re DRILL-5634:
> From the Apache commons web page [1] -
>
> Please note that Apache Commons Crypto doesn't implement the cryptographic
> algorithm such as AES directly.* It wraps to Openssl or JCE which implement
> the algorithms.*
>
>
> Since we are cleared for using Openssl and JCE, we are OK to include the
> AES code in the UDFs.
>
>
> Parth
>
> [1] https://commons.apache.org/proper/commons-crypto/
>
> On Tue, Jul 11, 2017 at 10:56 AM, Arina Yelchiyeva <
> arina.yelchiyeva@gmail.com> wrote:
>
> > During today hangouts we agreed on approximate cut-off date - 14 July,
> 2017
> > (date may be shifted due to the blockers).
> > I'll start preparing test RC on Thursday to solve any release preparation
> > issues beforehand.
> >
> > *Blockers*
> > DRILL-5660 - Vitalii
> >
> > *Possible blockers*
> > DRILL-5468 - Jinfeng
> > NPE on CSV source - Paul
> > DRILL-5659 - Parth
> >
> > *Will be included if done before the cut-off date*
> > DRILL-5616 / DRILL-5665 - Boaz
> > DRILL-5634 - Parth will take a look at Apache commons implementation and
> > update.
> >
> > Kind regards
> > Arina
> >
> > On Tue, Jul 11, 2017 at 3:14 PM, Charles Givre <cg...@gmail.com> wrote:
> >
> > > Hi Arina,
> > > I can’t make the hangout, but we have a few options.  Basically, since
> > all
> > > the hashing algorithms have no export restrictions, we could:
> > > 1.  Remove the AES encrypt/decrypt until I can figure out what we have
> to
> > > do (and do it)
> > > 2.  I can try to research it today or tomorrow to figure out exactly
> what
> > > we have to do to include AES.
> > > 3.  Remove the entire package
> > >
> > > I’d like to see the hashing functions at least included because we’ve
> had
> > > a lot of requests for that functionality.   I also don’t want this to
> > hold
> > > things up, so please let me know your preference.  Regardless, I’m
> going
> > to
> > > start trying to figure out the restrictions so we’ll know for the
> future.
> > >
> > > Best,
> > > — C
> > >
> > >
> > >
> > > > On Jul 11, 2017, at 05:58, Arina Yelchiyeva <
> > arina.yelchiyeva@gmail.com>
> > > wrote:
> > > >
> > > > @Jinfeng,
> > > > Let's discuss today on Drill hangout final cut-off date.
> > > >
> > > > @Charles
> > > > Since there were questions about AES in DRILL-5634, should we exclude
> > it
> > > > from the candidates for 1.11.0 release?
> > > >
> > > > @Boaz
> > > > Do we have any ETA when DRILL-5616 and DRILL-5665 will pass code
> > review?
> > > >
> > > > Also I believe we should include DRILL-5660 where we need to bump up
> > > > parquet metadata version.
> > > > @Vitalii
> > > > Do you have any ETA for the fix?
> > > >
> > > > Kind regards
> > > > Arina
> > > >
> > > > On Tue, Jul 11, 2017 at 6:02 AM, Boaz Ben-Zvi <bb...@mapr.com>
> > wrote:
> > > >
> > > >> Hi Arina,
> > > >>
> > > >>     Two PR have just been submitted, relating to the Hash Aggr
> Spill:
> > > >> DRILL-5616 and DRILL-5665 .
> > > >> Can these be considered for 1.11 as well ?
> > > >>
> > > >>     Thanks,
> > > >>
> > > >>               Boaz
> > > >>
> > > >> On 7/10/17, 1:49 PM, "Paul Rogers" <pr...@mapr.com> wrote:
> > > >>
> > > >>    DRILL-5601 is no longer a blocker for 1.11; we’ll defer that
> until
> > > >> after the release to allow others more time to work on other
> features
> > > >> rather than doing code reviews.
> > > >>
> > > >>    - Paul
> > > >>
> > > >>> On Jul 10, 2017, at 1:43 PM, Jinfeng Ni <jn...@apache.org> wrote:
> > > >>>
> > > >>> Hi Arina,
> > > >>>
> > > >>> It has been a while since we discussed 1.11.0 release.  Any update
> > > >> on the
> > > >>> new release plan?  Do we have a cut-off date ?
> > > >>>
> > > >>> In the past, I believe Drill release is a time-bounded, not feature
> > > >>> bounded. If certain fixes/features could not make it within a
> certain
> > > >>> time-frame, we had better defer them into future release, unless
> > > >> those
> > > >>> fixes/features are regarded as blocking issues for a new release.
> > > >>>
> > > >>>
> > > >>>
> > > >>>
> > > >>>
> > > >>>
> > > >>>
> > > >>> On Tue, Jul 4, 2017 at 5:14 AM, Arina Yelchiyeva <
> > > >> arina.yelchiyeva@gmail.com
> > > >>>> wrote:
> > > >>>
> > > >>>> Latest update:
> > > >>>>
> > > >>>> We wait for the following Jiras before final cut off.
> > > >>>> 1. DRILL-5601  - Paul
> > > >>>> 2. DRILL-5420 - Kunal
> > > >>>> 3. DRILL-5634 - Charles
> > > >>>>
> > > >>>> Kind regards
> > > >>>> Arina
> > > >>>>
> > > >>>> On Fri, Jun 30, 2017 at 9:29 PM, Kunal Khatua <kk...@mapr.com>
> > > >> wrote:
> > > >>>>
> > > >>>>> Parth's already +1'd the process with a minor update. So, we're
> > > >> good to
> > > >>>> go
> > > >>>>> on DRILL-5420. I'll change the label on that to ready-to-commit.
> > > >>>>>
> > > >>>>> <http://www.mapr.com/>
> > > >>>>>
> > > >>>>> ________________________________
> > > >>>>> From: Parth Chandra <pa...@apache.org>
> > > >>>>> Sent: Thursday, June 29, 2017 9:44:39 AM
> > > >>>>> To: dev@drill.apache.org
> > > >>>>> Subject: Re: [DISCUSS] Drill 1.11.0 release
> > > >>>>>
> > > >>>>> I'm Ok with the pcap format plugin if Pauls comments are also
> > > >> addressed.
> > > >>>> If
> > > >>>>> the changes are done before your cutoff deadline,we can merge it
> > > >> in.
> > > >>>>>
> > > >>>>> On Thu, Jun 29, 2017 at 12:26 AM, Arina Yelchiyeva <
> > > >>>>> arina.yelchiyeva@gmail.com> wrote:
> > > >>>>>
> > > >>>>>> Hi all,
> > > >>>>>>
> > > >>>>>> here is the list of Jira we are waiting before making final cut
> > > >> off:
> > > >>>>>> 1. DRILL-5601  - Paul
> > > >>>>>> 2. DRILL-3640  - Kunal
> > > >>>>>> 3. DRILL-5420 - Kunal
> > > >>>>>>
> > > >>>>>> Please let me know if there are any others Jiras we should wait
> > > >> for.
> > > >>>>>>
> > > >>>>>> @Charles,
> > > >>>>>> you have mentioned that you want to include some UDFs in Drill
> > > >> 1.11, do
> > > >>>>> you
> > > >>>>>> have any ETA for the PR?
> > > >>>>>>
> > > >>>>>> @Paul / Parth
> > > >>>>>> I see there is active CR for pcap format PR, do we want to
> > > >> include it
> > > >>>> in
> > > >>>>>> this release as well?
> > > >>>>>>
> > > >>>>>> Kind regards
> > > >>>>>> Arina
> > > >>>>>>
> > > >>>>>>
> > > >>>>>> On Thu, Jun 29, 2017 at 10:05 AM, Arina Yelchiyeva <
> > > >>>>>> arina.yelchiyeva@gmail.com> wrote:
> > > >>>>>>
> > > >>>>>>> Parth, thanks a lot the instructions.
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>> On Wed, Jun 28, 2017 at 8:24 PM, Parth Chandra <
> > > >> parthc@apache.org>
> > > >>>>>> wrote:
> > > >>>>>>>
> > > >>>>>>>> FWIW, I put release instructions culled from gists written by
> > > >>>> previous
> > > >>>>>>>> release managers into an updated gist here [1]
> > > >>>>>>>>
> > > >>>>>>>> HTH
> > > >>>>>>>>
> > > >>>>>>>> Parth
> > > >>>>>>>>
> > > >>>>>>>> [1] https://github.com/parthchandra/drill/wiki/Drill-
> > > >>>> Release-Process
> > > >>>>>>>> <https://github.com/parthchandra/drill/wiki/Drill-
> > > >> Release-Process>
> > > >>>>>>>>
> > > >>>>>>>>
> > > >>>>>>>>
> > > >>>>>>>> On Tue, Jun 27, 2017 at 5:24 PM, Paul Rogers <
> progers@mapr.com>
> > > >>>>> wrote:
> > > >>>>>>>>
> > > >>>>>>>>> Hi Arina,
> > > >>>>>>>>>
> > > >>>>>>>>> I have two projects in progress: one is a candidate for 1.11,
> > > >> the
> > > >>>>>> other
> > > >>>>>>>> is
> > > >>>>>>>>> not.
> > > >>>>>>>>>
> > > >>>>>>>>> The “managed external sort” is the good candidate for 1.11:
> > > >> The PR
> > > >>>>> for
> > > >>>>>>>>> DRILL-5601 is being reviewed. Getting that in will allow us
> to
> > > >>>>> enable
> > > >>>>>>>> the
> > > >>>>>>>>> “new” external sort by default.
> > > >>>>>>>>>
> > > >>>>>>>>> The other project is to address Drill’s memory fragmentation
> > > >> issue
> > > >>>>>> with
> > > >>>>>>>>> vectors over 16 MB in size. Two PRs have been open for a
> while.
> > > >>>> They
> > > >>>>>> are
> > > >>>>>>>>> the first of several. Each takes a while to review. So, the
> > > >>>> project
> > > >>>>>> does
> > > >>>>>>>>> not fit 1.11. The work can go into the next release instead.
> > > >>>>>>>>>
> > > >>>>>>>>> Thanks,
> > > >>>>>>>>>
> > > >>>>>>>>> - Paul
> > > >>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>>
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>
> > > >>>>>
> > > >>>>
> > > >>
> > > >>
> > > >>
> > > >>
> > >
> > >
> >
>

Re: [DISCUSS] Drill 1.11.0 release

Posted by Parth Chandra <pa...@apache.org>.

Re DRILL-5634:
From the Apache commons web page [1] -

Please note that Apache Commons Crypto doesn't implement the cryptographic
algorithm such as AES directly.* It wraps to Openssl or JCE which implement
the algorithms.*


Since we are cleared for using Openssl and JCE, we are OK to include the
AES code in the UDFs.


Parth

[1] https://commons.apache.org/proper/commons-crypto/

On Tue, Jul 11, 2017 at 10:56 AM, Arina Yelchiyeva <
arina.yelchiyeva@gmail.com> wrote:

> During today hangouts we agreed on approximate cut-off date - 14 July, 2017
> (date may be shifted due to the blockers).
> I'll start preparing test RC on Thursday to solve any release preparation
> issues beforehand.
>
> *Blockers*
> DRILL-5660 - Vitalii
>
> *Possible blockers*
> DRILL-5468 - Jinfeng
> NPE on CSV source - Paul
> DRILL-5659 - Parth
>
> *Will be included if done before the cut-off date*
> DRILL-5616 / DRILL-5665 - Boaz
> DRILL-5634 - Parth will take a look at Apache commons implementation and
> update.
>
> Kind regards
> Arina
>
> On Tue, Jul 11, 2017 at 3:14 PM, Charles Givre <cg...@gmail.com> wrote:
>
> > Hi Arina,
> > I can’t make the hangout, but we have a few options.  Basically, since
> all
> > the hashing algorithms have no export restrictions, we could:
> > 1.  Remove the AES encrypt/decrypt until I can figure out what we have to
> > do (and do it)
> > 2.  I can try to research it today or tomorrow to figure out exactly what
> > we have to do to include AES.
> > 3.  Remove the entire package
> >
> > I’d like to see the hashing functions at least included because we’ve had
> > a lot of requests for that functionality.   I also don’t want this to
> hold
> > things up, so please let me know your preference.  Regardless, I’m going
> to
> > start trying to figure out the restrictions so we’ll know for the future.
> >
> > Best,
> > — C
> >
> >
> >
> > > On Jul 11, 2017, at 05:58, Arina Yelchiyeva <
> arina.yelchiyeva@gmail.com>
> > wrote:
> > >
> > > @Jinfeng,
> > > Let's discuss today on Drill hangout final cut-off date.
> > >
> > > @Charles
> > > Since there were questions about AES in DRILL-5634, should we exclude
> it
> > > from the candidates for 1.11.0 release?
> > >
> > > @Boaz
> > > Do we have any ETA when DRILL-5616 and DRILL-5665 will pass code
> review?
> > >
> > > Also I believe we should include DRILL-5660 where we need to bump up
> > > parquet metadata version.
> > > @Vitalii
> > > Do you have any ETA for the fix?
> > >
> > > Kind regards
> > > Arina
> > >
> > > On Tue, Jul 11, 2017 at 6:02 AM, Boaz Ben-Zvi <bb...@mapr.com>
> wrote:
> > >
> > >> Hi Arina,
> > >>
> > >>     Two PR have just been submitted, relating to the Hash Aggr Spill:
> > >> DRILL-5616 and DRILL-5665 .
> > >> Can these be considered for 1.11 as well ?
> > >>
> > >>     Thanks,
> > >>
> > >>               Boaz
> > >>
> > >> On 7/10/17, 1:49 PM, "Paul Rogers" <pr...@mapr.com> wrote:
> > >>
> > >>    DRILL-5601 is no longer a blocker for 1.11; we’ll defer that until
> > >> after the release to allow others more time to work on other features
> > >> rather than doing code reviews.
> > >>
> > >>    - Paul
> > >>
> > >>> On Jul 10, 2017, at 1:43 PM, Jinfeng Ni <jn...@apache.org> wrote:
> > >>>
> > >>> Hi Arina,
> > >>>
> > >>> It has been a while since we discussed 1.11.0 release.  Any update
> > >> on the
> > >>> new release plan?  Do we have a cut-off date ?
> > >>>
> > >>> In the past, I believe Drill release is a time-bounded, not feature
> > >>> bounded. If certain fixes/features could not make it within a certain
> > >>> time-frame, we had better defer them into future release, unless
> > >> those
> > >>> fixes/features are regarded as blocking issues for a new release.
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>> On Tue, Jul 4, 2017 at 5:14 AM, Arina Yelchiyeva <
> > >> arina.yelchiyeva@gmail.com
> > >>>> wrote:
> > >>>
> > >>>> Latest update:
> > >>>>
> > >>>> We wait for the following Jiras before final cut off.
> > >>>> 1. DRILL-5601  - Paul
> > >>>> 2. DRILL-5420 - Kunal
> > >>>> 3. DRILL-5634 - Charles
> > >>>>
> > >>>> Kind regards
> > >>>> Arina
> > >>>>
> > >>>> On Fri, Jun 30, 2017 at 9:29 PM, Kunal Khatua <kk...@mapr.com>
> > >> wrote:
> > >>>>
> > >>>>> Parth's already +1'd the process with a minor update. So, we're
> > >> good to
> > >>>> go
> > >>>>> on DRILL-5420. I'll change the label on that to ready-to-commit.
> > >>>>>
> > >>>>> <http://www.mapr.com/>
> > >>>>>
> > >>>>> ________________________________
> > >>>>> From: Parth Chandra <pa...@apache.org>
> > >>>>> Sent: Thursday, June 29, 2017 9:44:39 AM
> > >>>>> To: dev@drill.apache.org
> > >>>>> Subject: Re: [DISCUSS] Drill 1.11.0 release
> > >>>>>
> > >>>>> I'm Ok with the pcap format plugin if Pauls comments are also
> > >> addressed.
> > >>>> If
> > >>>>> the changes are done before your cutoff deadline,we can merge it
> > >> in.
> > >>>>>
> > >>>>> On Thu, Jun 29, 2017 at 12:26 AM, Arina Yelchiyeva <
> > >>>>> arina.yelchiyeva@gmail.com> wrote:
> > >>>>>
> > >>>>>> Hi all,
> > >>>>>>
> > >>>>>> here is the list of Jira we are waiting before making final cut
> > >> off:
> > >>>>>> 1. DRILL-5601  - Paul
> > >>>>>> 2. DRILL-3640  - Kunal
> > >>>>>> 3. DRILL-5420 - Kunal
> > >>>>>>
> > >>>>>> Please let me know if there are any others Jiras we should wait
> > >> for.
> > >>>>>>
> > >>>>>> @Charles,
> > >>>>>> you have mentioned that you want to include some UDFs in Drill
> > >> 1.11, do
> > >>>>> you
> > >>>>>> have any ETA for the PR?
> > >>>>>>
> > >>>>>> @Paul / Parth
> > >>>>>> I see there is active CR for pcap format PR, do we want to
> > >> include it
> > >>>> in
> > >>>>>> this release as well?
> > >>>>>>
> > >>>>>> Kind regards
> > >>>>>> Arina
> > >>>>>>
> > >>>>>>
> > >>>>>> On Thu, Jun 29, 2017 at 10:05 AM, Arina Yelchiyeva <
> > >>>>>> arina.yelchiyeva@gmail.com> wrote:
> > >>>>>>
> > >>>>>>> Parth, thanks a lot the instructions.
> > >>>>>>>
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> On Wed, Jun 28, 2017 at 8:24 PM, Parth Chandra <
> > >> parthc@apache.org>
> > >>>>>> wrote:
> > >>>>>>>
> > >>>>>>>> FWIW, I put release instructions culled from gists written by
> > >>>> previous
> > >>>>>>>> release managers into an updated gist here [1]
> > >>>>>>>>
> > >>>>>>>> HTH
> > >>>>>>>>
> > >>>>>>>> Parth
> > >>>>>>>>
> > >>>>>>>> [1] https://github.com/parthchandra/drill/wiki/Drill-
> > >>>> Release-Process
> > >>>>>>>> <https://github.com/parthchandra/drill/wiki/Drill-
> > >> Release-Process>
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>> On Tue, Jun 27, 2017 at 5:24 PM, Paul Rogers <pr...@mapr.com>
> > >>>>> wrote:
> > >>>>>>>>
> > >>>>>>>>> Hi Arina,
> > >>>>>>>>>
> > >>>>>>>>> I have two projects in progress: one is a candidate for 1.11,
> > >> the
> > >>>>>> other
> > >>>>>>>> is
> > >>>>>>>>> not.
> > >>>>>>>>>
> > >>>>>>>>> The “managed external sort” is the good candidate for 1.11:
> > >> The PR
> > >>>>> for
> > >>>>>>>>> DRILL-5601 is being reviewed. Getting that in will allow us to
> > >>>>> enable
> > >>>>>>>> the
> > >>>>>>>>> “new” external sort by default.
> > >>>>>>>>>
> > >>>>>>>>> The other project is to address Drill’s memory fragmentation
> > >> issue
> > >>>>>> with
> > >>>>>>>>> vectors over 16 MB in size. Two PRs have been open for a while.
> > >>>> They
> > >>>>>> are
> > >>>>>>>>> the first of several. Each takes a while to review. So, the
> > >>>> project
> > >>>>>> does
> > >>>>>>>>> not fit 1.11. The work can go into the next release instead.
> > >>>>>>>>>
> > >>>>>>>>> Thanks,
> > >>>>>>>>>
> > >>>>>>>>> - Paul
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>
> > >>>>>>>
> > >>>>>>>
> > >>>>>>
> > >>>>>
> > >>>>
> > >>
> > >>
> > >>
> > >>
> >
> >
>

Re: [DISCUSS] Drill 1.11.0 release

Posted by Arina Yelchiyeva <ar...@gmail.com>.

During today hangouts we agreed on approximate cut-off date - 14 July, 2017
(date may be shifted due to the blockers).
I'll start preparing test RC on Thursday to solve any release preparation
issues beforehand.

*Blockers*
DRILL-5660 - Vitalii

*Possible blockers*
DRILL-5468 - Jinfeng
NPE on CSV source - Paul
DRILL-5659 - Parth

*Will be included if done before the cut-off date*
DRILL-5616 / DRILL-5665 - Boaz
DRILL-5634 - Parth will take a look at Apache commons implementation and
update.

Kind regards
Arina

On Tue, Jul 11, 2017 at 3:14 PM, Charles Givre <cg...@gmail.com> wrote:

> Hi Arina,
> I can’t make the hangout, but we have a few options.  Basically, since all
> the hashing algorithms have no export restrictions, we could:
> 1.  Remove the AES encrypt/decrypt until I can figure out what we have to
> do (and do it)
> 2.  I can try to research it today or tomorrow to figure out exactly what
> we have to do to include AES.
> 3.  Remove the entire package
>
> I’d like to see the hashing functions at least included because we’ve had
> a lot of requests for that functionality.   I also don’t want this to hold
> things up, so please let me know your preference.  Regardless, I’m going to
> start trying to figure out the restrictions so we’ll know for the future.
>
> Best,
> — C
>
>
>
> > On Jul 11, 2017, at 05:58, Arina Yelchiyeva <ar...@gmail.com>
> wrote:
> >
> > @Jinfeng,
> > Let's discuss today on Drill hangout final cut-off date.
> >
> > @Charles
> > Since there were questions about AES in DRILL-5634, should we exclude it
> > from the candidates for 1.11.0 release?
> >
> > @Boaz
> > Do we have any ETA when DRILL-5616 and DRILL-5665 will pass code review?
> >
> > Also I believe we should include DRILL-5660 where we need to bump up
> > parquet metadata version.
> > @Vitalii
> > Do you have any ETA for the fix?
> >
> > Kind regards
> > Arina
> >
> > On Tue, Jul 11, 2017 at 6:02 AM, Boaz Ben-Zvi <bb...@mapr.com> wrote:
> >
> >> Hi Arina,
> >>
> >>     Two PR have just been submitted, relating to the Hash Aggr Spill:
> >> DRILL-5616 and DRILL-5665 .
> >> Can these be considered for 1.11 as well ?
> >>
> >>     Thanks,
> >>
> >>               Boaz
> >>
> >> On 7/10/17, 1:49 PM, "Paul Rogers" <pr...@mapr.com> wrote:
> >>
> >>    DRILL-5601 is no longer a blocker for 1.11; we’ll defer that until
> >> after the release to allow others more time to work on other features
> >> rather than doing code reviews.
> >>
> >>    - Paul
> >>
> >>> On Jul 10, 2017, at 1:43 PM, Jinfeng Ni <jn...@apache.org> wrote:
> >>>
> >>> Hi Arina,
> >>>
> >>> It has been a while since we discussed 1.11.0 release.  Any update
> >> on the
> >>> new release plan?  Do we have a cut-off date ?
> >>>
> >>> In the past, I believe Drill release is a time-bounded, not feature
> >>> bounded. If certain fixes/features could not make it within a certain
> >>> time-frame, we had better defer them into future release, unless
> >> those
> >>> fixes/features are regarded as blocking issues for a new release.
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>> On Tue, Jul 4, 2017 at 5:14 AM, Arina Yelchiyeva <
> >> arina.yelchiyeva@gmail.com
> >>>> wrote:
> >>>
> >>>> Latest update:
> >>>>
> >>>> We wait for the following Jiras before final cut off.
> >>>> 1. DRILL-5601  - Paul
> >>>> 2. DRILL-5420 - Kunal
> >>>> 3. DRILL-5634 - Charles
> >>>>
> >>>> Kind regards
> >>>> Arina
> >>>>
> >>>> On Fri, Jun 30, 2017 at 9:29 PM, Kunal Khatua <kk...@mapr.com>
> >> wrote:
> >>>>
> >>>>> Parth's already +1'd the process with a minor update. So, we're
> >> good to
> >>>> go
> >>>>> on DRILL-5420. I'll change the label on that to ready-to-commit.
> >>>>>
> >>>>> <http://www.mapr.com/>
> >>>>>
> >>>>> ________________________________
> >>>>> From: Parth Chandra <pa...@apache.org>
> >>>>> Sent: Thursday, June 29, 2017 9:44:39 AM
> >>>>> To: dev@drill.apache.org
> >>>>> Subject: Re: [DISCUSS] Drill 1.11.0 release
> >>>>>
> >>>>> I'm Ok with the pcap format plugin if Pauls comments are also
> >> addressed.
> >>>> If
> >>>>> the changes are done before your cutoff deadline,we can merge it
> >> in.
> >>>>>
> >>>>> On Thu, Jun 29, 2017 at 12:26 AM, Arina Yelchiyeva <
> >>>>> arina.yelchiyeva@gmail.com> wrote:
> >>>>>
> >>>>>> Hi all,
> >>>>>>
> >>>>>> here is the list of Jira we are waiting before making final cut
> >> off:
> >>>>>> 1. DRILL-5601  - Paul
> >>>>>> 2. DRILL-3640  - Kunal
> >>>>>> 3. DRILL-5420 - Kunal
> >>>>>>
> >>>>>> Please let me know if there are any others Jiras we should wait
> >> for.
> >>>>>>
> >>>>>> @Charles,
> >>>>>> you have mentioned that you want to include some UDFs in Drill
> >> 1.11, do
> >>>>> you
> >>>>>> have any ETA for the PR?
> >>>>>>
> >>>>>> @Paul / Parth
> >>>>>> I see there is active CR for pcap format PR, do we want to
> >> include it
> >>>> in
> >>>>>> this release as well?
> >>>>>>
> >>>>>> Kind regards
> >>>>>> Arina
> >>>>>>
> >>>>>>
> >>>>>> On Thu, Jun 29, 2017 at 10:05 AM, Arina Yelchiyeva <
> >>>>>> arina.yelchiyeva@gmail.com> wrote:
> >>>>>>
> >>>>>>> Parth, thanks a lot the instructions.
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> On Wed, Jun 28, 2017 at 8:24 PM, Parth Chandra <
> >> parthc@apache.org>
> >>>>>> wrote:
> >>>>>>>
> >>>>>>>> FWIW, I put release instructions culled from gists written by
> >>>> previous
> >>>>>>>> release managers into an updated gist here [1]
> >>>>>>>>
> >>>>>>>> HTH
> >>>>>>>>
> >>>>>>>> Parth
> >>>>>>>>
> >>>>>>>> [1] https://github.com/parthchandra/drill/wiki/Drill-
> >>>> Release-Process
> >>>>>>>> <https://github.com/parthchandra/drill/wiki/Drill-
> >> Release-Process>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> On Tue, Jun 27, 2017 at 5:24 PM, Paul Rogers <pr...@mapr.com>
> >>>>> wrote:
> >>>>>>>>
> >>>>>>>>> Hi Arina,
> >>>>>>>>>
> >>>>>>>>> I have two projects in progress: one is a candidate for 1.11,
> >> the
> >>>>>> other
> >>>>>>>> is
> >>>>>>>>> not.
> >>>>>>>>>
> >>>>>>>>> The “managed external sort” is the good candidate for 1.11:
> >> The PR
> >>>>> for
> >>>>>>>>> DRILL-5601 is being reviewed. Getting that in will allow us to
> >>>>> enable
> >>>>>>>> the
> >>>>>>>>> “new” external sort by default.
> >>>>>>>>>
> >>>>>>>>> The other project is to address Drill’s memory fragmentation
> >> issue
> >>>>>> with
> >>>>>>>>> vectors over 16 MB in size. Two PRs have been open for a while.
> >>>> They
> >>>>>> are
> >>>>>>>>> the first of several. Each takes a while to review. So, the
> >>>> project
> >>>>>> does
> >>>>>>>>> not fit 1.11. The work can go into the next release instead.
> >>>>>>>>>
> >>>>>>>>> Thanks,
> >>>>>>>>>
> >>>>>>>>> - Paul
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>
> >>
> >>
> >>
>
>

Re: [DISCUSS] Drill 1.11.0 release

Posted by Charles Givre <cg...@gmail.com>.

Hi Arina, 
I can’t make the hangout, but we have a few options.  Basically, since all the hashing algorithms have no export restrictions, we could:
1.  Remove the AES encrypt/decrypt until I can figure out what we have to do (and do it)
2.  I can try to research it today or tomorrow to figure out exactly what we have to do to include AES.  
3.  Remove the entire package

I’d like to see the hashing functions at least included because we’ve had a lot of requests for that functionality.   I also don’t want this to hold things up, so please let me know your preference.  Regardless, I’m going to start trying to figure out the restrictions so we’ll know for the future. 

Best,
— C



> On Jul 11, 2017, at 05:58, Arina Yelchiyeva <ar...@gmail.com> wrote:
> 
> @Jinfeng,
> Let's discuss today on Drill hangout final cut-off date.
> 
> @Charles
> Since there were questions about AES in DRILL-5634, should we exclude it
> from the candidates for 1.11.0 release?
> 
> @Boaz
> Do we have any ETA when DRILL-5616 and DRILL-5665 will pass code review?
> 
> Also I believe we should include DRILL-5660 where we need to bump up
> parquet metadata version.
> @Vitalii
> Do you have any ETA for the fix?
> 
> Kind regards
> Arina
> 
> On Tue, Jul 11, 2017 at 6:02 AM, Boaz Ben-Zvi <bb...@mapr.com> wrote:
> 
>> Hi Arina,
>> 
>>     Two PR have just been submitted, relating to the Hash Aggr Spill:
>> DRILL-5616 and DRILL-5665 .
>> Can these be considered for 1.11 as well ?
>> 
>>     Thanks,
>> 
>>               Boaz
>> 
>> On 7/10/17, 1:49 PM, "Paul Rogers" <pr...@mapr.com> wrote:
>> 
>>    DRILL-5601 is no longer a blocker for 1.11; we’ll defer that until
>> after the release to allow others more time to work on other features
>> rather than doing code reviews.
>> 
>>    - Paul
>> 
>>> On Jul 10, 2017, at 1:43 PM, Jinfeng Ni <jn...@apache.org> wrote:
>>> 
>>> Hi Arina,
>>> 
>>> It has been a while since we discussed 1.11.0 release.  Any update
>> on the
>>> new release plan?  Do we have a cut-off date ?
>>> 
>>> In the past, I believe Drill release is a time-bounded, not feature
>>> bounded. If certain fixes/features could not make it within a certain
>>> time-frame, we had better defer them into future release, unless
>> those
>>> fixes/features are regarded as blocking issues for a new release.
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> On Tue, Jul 4, 2017 at 5:14 AM, Arina Yelchiyeva <
>> arina.yelchiyeva@gmail.com
>>>> wrote:
>>> 
>>>> Latest update:
>>>> 
>>>> We wait for the following Jiras before final cut off.
>>>> 1. DRILL-5601  - Paul
>>>> 2. DRILL-5420 - Kunal
>>>> 3. DRILL-5634 - Charles
>>>> 
>>>> Kind regards
>>>> Arina
>>>> 
>>>> On Fri, Jun 30, 2017 at 9:29 PM, Kunal Khatua <kk...@mapr.com>
>> wrote:
>>>> 
>>>>> Parth's already +1'd the process with a minor update. So, we're
>> good to
>>>> go
>>>>> on DRILL-5420. I'll change the label on that to ready-to-commit.
>>>>> 
>>>>> <http://www.mapr.com/>
>>>>> 
>>>>> ________________________________
>>>>> From: Parth Chandra <pa...@apache.org>
>>>>> Sent: Thursday, June 29, 2017 9:44:39 AM
>>>>> To: dev@drill.apache.org
>>>>> Subject: Re: [DISCUSS] Drill 1.11.0 release
>>>>> 
>>>>> I'm Ok with the pcap format plugin if Pauls comments are also
>> addressed.
>>>> If
>>>>> the changes are done before your cutoff deadline,we can merge it
>> in.
>>>>> 
>>>>> On Thu, Jun 29, 2017 at 12:26 AM, Arina Yelchiyeva <
>>>>> arina.yelchiyeva@gmail.com> wrote:
>>>>> 
>>>>>> Hi all,
>>>>>> 
>>>>>> here is the list of Jira we are waiting before making final cut
>> off:
>>>>>> 1. DRILL-5601  - Paul
>>>>>> 2. DRILL-3640  - Kunal
>>>>>> 3. DRILL-5420 - Kunal
>>>>>> 
>>>>>> Please let me know if there are any others Jiras we should wait
>> for.
>>>>>> 
>>>>>> @Charles,
>>>>>> you have mentioned that you want to include some UDFs in Drill
>> 1.11, do
>>>>> you
>>>>>> have any ETA for the PR?
>>>>>> 
>>>>>> @Paul / Parth
>>>>>> I see there is active CR for pcap format PR, do we want to
>> include it
>>>> in
>>>>>> this release as well?
>>>>>> 
>>>>>> Kind regards
>>>>>> Arina
>>>>>> 
>>>>>> 
>>>>>> On Thu, Jun 29, 2017 at 10:05 AM, Arina Yelchiyeva <
>>>>>> arina.yelchiyeva@gmail.com> wrote:
>>>>>> 
>>>>>>> Parth, thanks a lot the instructions.
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> On Wed, Jun 28, 2017 at 8:24 PM, Parth Chandra <
>> parthc@apache.org>
>>>>>> wrote:
>>>>>>> 
>>>>>>>> FWIW, I put release instructions culled from gists written by
>>>> previous
>>>>>>>> release managers into an updated gist here [1]
>>>>>>>> 
>>>>>>>> HTH
>>>>>>>> 
>>>>>>>> Parth
>>>>>>>> 
>>>>>>>> [1] https://github.com/parthchandra/drill/wiki/Drill-
>>>> Release-Process
>>>>>>>> <https://github.com/parthchandra/drill/wiki/Drill-
>> Release-Process>
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> On Tue, Jun 27, 2017 at 5:24 PM, Paul Rogers <pr...@mapr.com>
>>>>> wrote:
>>>>>>>> 
>>>>>>>>> Hi Arina,
>>>>>>>>> 
>>>>>>>>> I have two projects in progress: one is a candidate for 1.11,
>> the
>>>>>> other
>>>>>>>> is
>>>>>>>>> not.
>>>>>>>>> 
>>>>>>>>> The “managed external sort” is the good candidate for 1.11:
>> The PR
>>>>> for
>>>>>>>>> DRILL-5601 is being reviewed. Getting that in will allow us to
>>>>> enable
>>>>>>>> the
>>>>>>>>> “new” external sort by default.
>>>>>>>>> 
>>>>>>>>> The other project is to address Drill’s memory fragmentation
>> issue
>>>>>> with
>>>>>>>>> vectors over 16 MB in size. Two PRs have been open for a while.
>>>> They
>>>>>> are
>>>>>>>>> the first of several. Each takes a while to review. So, the
>>>> project
>>>>>> does
>>>>>>>>> not fit 1.11. The work can go into the next release instead.
>>>>>>>>> 
>>>>>>>>> Thanks,
>>>>>>>>> 
>>>>>>>>> - Paul
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>> 
>> 
>> 
>>

Re: [DISCUSS] Drill 1.11.0 release

Posted by Arina Yelchiyeva <ar...@gmail.com>.

@Jinfeng,
Let's discuss today on Drill hangout final cut-off date.

@Charles
Since there were questions about AES in DRILL-5634, should we exclude it
from the candidates for 1.11.0 release?

@Boaz
Do we have any ETA when DRILL-5616 and DRILL-5665 will pass code review?

Also I believe we should include DRILL-5660 where we need to bump up
parquet metadata version.
@Vitalii
Do you have any ETA for the fix?

Kind regards
Arina

On Tue, Jul 11, 2017 at 6:02 AM, Boaz Ben-Zvi <bb...@mapr.com> wrote:

>  Hi Arina,
>
>      Two PR have just been submitted, relating to the Hash Aggr Spill:
> DRILL-5616 and DRILL-5665 .
> Can these be considered for 1.11 as well ?
>
>      Thanks,
>
>                Boaz
>
> On 7/10/17, 1:49 PM, "Paul Rogers" <pr...@mapr.com> wrote:
>
>     DRILL-5601 is no longer a blocker for 1.11; we’ll defer that until
> after the release to allow others more time to work on other features
> rather than doing code reviews.
>
>     - Paul
>
>     > On Jul 10, 2017, at 1:43 PM, Jinfeng Ni <jn...@apache.org> wrote:
>     >
>     > Hi Arina,
>     >
>     > It has been a while since we discussed 1.11.0 release.  Any update
> on the
>     > new release plan?  Do we have a cut-off date ?
>     >
>     > In the past, I believe Drill release is a time-bounded, not feature
>     > bounded. If certain fixes/features could not make it within a certain
>     > time-frame, we had better defer them into future release, unless
> those
>     > fixes/features are regarded as blocking issues for a new release.
>     >
>     >
>     >
>     >
>     >
>     >
>     >
>     > On Tue, Jul 4, 2017 at 5:14 AM, Arina Yelchiyeva <
> arina.yelchiyeva@gmail.com
>     >> wrote:
>     >
>     >> Latest update:
>     >>
>     >> We wait for the following Jiras before final cut off.
>     >> 1. DRILL-5601  - Paul
>     >> 2. DRILL-5420 - Kunal
>     >> 3. DRILL-5634 - Charles
>     >>
>     >> Kind regards
>     >> Arina
>     >>
>     >> On Fri, Jun 30, 2017 at 9:29 PM, Kunal Khatua <kk...@mapr.com>
> wrote:
>     >>
>     >>> Parth's already +1'd the process with a minor update. So, we're
> good to
>     >> go
>     >>> on DRILL-5420. I'll change the label on that to ready-to-commit.
>     >>>
>     >>> <http://www.mapr.com/>
>     >>>
>     >>> ________________________________
>     >>> From: Parth Chandra <pa...@apache.org>
>     >>> Sent: Thursday, June 29, 2017 9:44:39 AM
>     >>> To: dev@drill.apache.org
>     >>> Subject: Re: [DISCUSS] Drill 1.11.0 release
>     >>>
>     >>> I'm Ok with the pcap format plugin if Pauls comments are also
> addressed.
>     >> If
>     >>> the changes are done before your cutoff deadline,we can merge it
> in.
>     >>>
>     >>> On Thu, Jun 29, 2017 at 12:26 AM, Arina Yelchiyeva <
>     >>> arina.yelchiyeva@gmail.com> wrote:
>     >>>
>     >>>> Hi all,
>     >>>>
>     >>>> here is the list of Jira we are waiting before making final cut
> off:
>     >>>> 1. DRILL-5601  - Paul
>     >>>> 2. DRILL-3640  - Kunal
>     >>>> 3. DRILL-5420 - Kunal
>     >>>>
>     >>>> Please let me know if there are any others Jiras we should wait
> for.
>     >>>>
>     >>>> @Charles,
>     >>>> you have mentioned that you want to include some UDFs in Drill
> 1.11, do
>     >>> you
>     >>>> have any ETA for the PR?
>     >>>>
>     >>>> @Paul / Parth
>     >>>> I see there is active CR for pcap format PR, do we want to
> include it
>     >> in
>     >>>> this release as well?
>     >>>>
>     >>>> Kind regards
>     >>>> Arina
>     >>>>
>     >>>>
>     >>>> On Thu, Jun 29, 2017 at 10:05 AM, Arina Yelchiyeva <
>     >>>> arina.yelchiyeva@gmail.com> wrote:
>     >>>>
>     >>>>> Parth, thanks a lot the instructions.
>     >>>>>
>     >>>>>
>     >>>>>
>     >>>>> On Wed, Jun 28, 2017 at 8:24 PM, Parth Chandra <
> parthc@apache.org>
>     >>>> wrote:
>     >>>>>
>     >>>>>> FWIW, I put release instructions culled from gists written by
>     >> previous
>     >>>>>> release managers into an updated gist here [1]
>     >>>>>>
>     >>>>>> HTH
>     >>>>>>
>     >>>>>> Parth
>     >>>>>>
>     >>>>>> [1] https://github.com/parthchandra/drill/wiki/Drill-
>     >> Release-Process
>     >>>>>> <https://github.com/parthchandra/drill/wiki/Drill-
> Release-Process>
>     >>>>>>
>     >>>>>>
>     >>>>>>
>     >>>>>> On Tue, Jun 27, 2017 at 5:24 PM, Paul Rogers <pr...@mapr.com>
>     >>> wrote:
>     >>>>>>
>     >>>>>>> Hi Arina,
>     >>>>>>>
>     >>>>>>> I have two projects in progress: one is a candidate for 1.11,
> the
>     >>>> other
>     >>>>>> is
>     >>>>>>> not.
>     >>>>>>>
>     >>>>>>> The “managed external sort” is the good candidate for 1.11:
> The PR
>     >>> for
>     >>>>>>> DRILL-5601 is being reviewed. Getting that in will allow us to
>     >>> enable
>     >>>>>> the
>     >>>>>>> “new” external sort by default.
>     >>>>>>>
>     >>>>>>> The other project is to address Drill’s memory fragmentation
> issue
>     >>>> with
>     >>>>>>> vectors over 16 MB in size. Two PRs have been open for a while.
>     >> They
>     >>>> are
>     >>>>>>> the first of several. Each takes a while to review. So, the
>     >> project
>     >>>> does
>     >>>>>>> not fit 1.11. The work can go into the next release instead.
>     >>>>>>>
>     >>>>>>> Thanks,
>     >>>>>>>
>     >>>>>>> - Paul
>     >>>>>>>
>     >>>>>>>
>     >>>>>>>
>     >>>>>>>
>     >>>>>>
>     >>>>>
>     >>>>>
>     >>>>
>     >>>
>     >>
>
>
>
>

Re: [DISCUSS] Drill 1.11.0 release

Posted by Boaz Ben-Zvi <bb...@mapr.com>.

 Hi Arina,

     Two PR have just been submitted, relating to the Hash Aggr Spill: DRILL-5616 and DRILL-5665 .
Can these be considered for 1.11 as well ?

     Thanks,

               Boaz 

On 7/10/17, 1:49 PM, "Paul Rogers" <pr...@mapr.com> wrote:

    DRILL-5601 is no longer a blocker for 1.11; we’ll defer that until after the release to allow others more time to work on other features rather than doing code reviews.
    
    - Paul
    
    > On Jul 10, 2017, at 1:43 PM, Jinfeng Ni <jn...@apache.org> wrote:
    > 
    > Hi Arina,
    > 
    > It has been a while since we discussed 1.11.0 release.  Any update on the
    > new release plan?  Do we have a cut-off date ?
    > 
    > In the past, I believe Drill release is a time-bounded, not feature
    > bounded. If certain fixes/features could not make it within a certain
    > time-frame, we had better defer them into future release, unless those
    > fixes/features are regarded as blocking issues for a new release.
    > 
    > 
    > 
    > 
    > 
    > 
    > 
    > On Tue, Jul 4, 2017 at 5:14 AM, Arina Yelchiyeva <arina.yelchiyeva@gmail.com
    >> wrote:
    > 
    >> Latest update:
    >> 
    >> We wait for the following Jiras before final cut off.
    >> 1. DRILL-5601  - Paul
    >> 2. DRILL-5420 - Kunal
    >> 3. DRILL-5634 - Charles
    >> 
    >> Kind regards
    >> Arina
    >> 
    >> On Fri, Jun 30, 2017 at 9:29 PM, Kunal Khatua <kk...@mapr.com> wrote:
    >> 
    >>> Parth's already +1'd the process with a minor update. So, we're good to
    >> go
    >>> on DRILL-5420. I'll change the label on that to ready-to-commit.
    >>> 
    >>> <http://www.mapr.com/>
    >>> 
    >>> ________________________________
    >>> From: Parth Chandra <pa...@apache.org>
    >>> Sent: Thursday, June 29, 2017 9:44:39 AM
    >>> To: dev@drill.apache.org
    >>> Subject: Re: [DISCUSS] Drill 1.11.0 release
    >>> 
    >>> I'm Ok with the pcap format plugin if Pauls comments are also addressed.
    >> If
    >>> the changes are done before your cutoff deadline,we can merge it in.
    >>> 
    >>> On Thu, Jun 29, 2017 at 12:26 AM, Arina Yelchiyeva <
    >>> arina.yelchiyeva@gmail.com> wrote:
    >>> 
    >>>> Hi all,
    >>>> 
    >>>> here is the list of Jira we are waiting before making final cut off:
    >>>> 1. DRILL-5601  - Paul
    >>>> 2. DRILL-3640  - Kunal
    >>>> 3. DRILL-5420 - Kunal
    >>>> 
    >>>> Please let me know if there are any others Jiras we should wait for.
    >>>> 
    >>>> @Charles,
    >>>> you have mentioned that you want to include some UDFs in Drill 1.11, do
    >>> you
    >>>> have any ETA for the PR?
    >>>> 
    >>>> @Paul / Parth
    >>>> I see there is active CR for pcap format PR, do we want to include it
    >> in
    >>>> this release as well?
    >>>> 
    >>>> Kind regards
    >>>> Arina
    >>>> 
    >>>> 
    >>>> On Thu, Jun 29, 2017 at 10:05 AM, Arina Yelchiyeva <
    >>>> arina.yelchiyeva@gmail.com> wrote:
    >>>> 
    >>>>> Parth, thanks a lot the instructions.
    >>>>> 
    >>>>> 
    >>>>> 
    >>>>> On Wed, Jun 28, 2017 at 8:24 PM, Parth Chandra <pa...@apache.org>
    >>>> wrote:
    >>>>> 
    >>>>>> FWIW, I put release instructions culled from gists written by
    >> previous
    >>>>>> release managers into an updated gist here [1]
    >>>>>> 
    >>>>>> HTH
    >>>>>> 
    >>>>>> Parth
    >>>>>> 
    >>>>>> [1] https://github.com/parthchandra/drill/wiki/Drill-
    >> Release-Process
    >>>>>> <https://github.com/parthchandra/drill/wiki/Drill-Release-Process>
    >>>>>> 
    >>>>>> 
    >>>>>> 
    >>>>>> On Tue, Jun 27, 2017 at 5:24 PM, Paul Rogers <pr...@mapr.com>
    >>> wrote:
    >>>>>> 
    >>>>>>> Hi Arina,
    >>>>>>> 
    >>>>>>> I have two projects in progress: one is a candidate for 1.11, the
    >>>> other
    >>>>>> is
    >>>>>>> not.
    >>>>>>> 
    >>>>>>> The “managed external sort” is the good candidate for 1.11: The PR
    >>> for
    >>>>>>> DRILL-5601 is being reviewed. Getting that in will allow us to
    >>> enable
    >>>>>> the
    >>>>>>> “new” external sort by default.
    >>>>>>> 
    >>>>>>> The other project is to address Drill’s memory fragmentation issue
    >>>> with
    >>>>>>> vectors over 16 MB in size. Two PRs have been open for a while.
    >> They
    >>>> are
    >>>>>>> the first of several. Each takes a while to review. So, the
    >> project
    >>>> does
    >>>>>>> not fit 1.11. The work can go into the next release instead.
    >>>>>>> 
    >>>>>>> Thanks,
    >>>>>>> 
    >>>>>>> - Paul
    >>>>>>> 
    >>>>>>> 
    >>>>>>> 
    >>>>>>> 
    >>>>>> 
    >>>>> 
    >>>>> 
    >>>> 
    >>> 
    >>

Re: [DISCUSS] Drill 1.11.0 release

Posted by Paul Rogers <pr...@mapr.com>.

DRILL-5601 is no longer a blocker for 1.11; we’ll defer that until after the release to allow others more time to work on other features rather than doing code reviews.

- Paul

> On Jul 10, 2017, at 1:43 PM, Jinfeng Ni <jn...@apache.org> wrote:
> 
> Hi Arina,
> 
> It has been a while since we discussed 1.11.0 release.  Any update on the
> new release plan?  Do we have a cut-off date ?
> 
> In the past, I believe Drill release is a time-bounded, not feature
> bounded. If certain fixes/features could not make it within a certain
> time-frame, we had better defer them into future release, unless those
> fixes/features are regarded as blocking issues for a new release.
> 
> 
> 
> 
> 
> 
> 
> On Tue, Jul 4, 2017 at 5:14 AM, Arina Yelchiyeva <arina.yelchiyeva@gmail.com
>> wrote:
> 
>> Latest update:
>> 
>> We wait for the following Jiras before final cut off.
>> 1. DRILL-5601  - Paul
>> 2. DRILL-5420 - Kunal
>> 3. DRILL-5634 - Charles
>> 
>> Kind regards
>> Arina
>> 
>> On Fri, Jun 30, 2017 at 9:29 PM, Kunal Khatua <kk...@mapr.com> wrote:
>> 
>>> Parth's already +1'd the process with a minor update. So, we're good to
>> go
>>> on DRILL-5420. I'll change the label on that to ready-to-commit.
>>> 
>>> <http://www.mapr.com/>
>>> 
>>> ________________________________
>>> From: Parth Chandra <pa...@apache.org>
>>> Sent: Thursday, June 29, 2017 9:44:39 AM
>>> To: dev@drill.apache.org
>>> Subject: Re: [DISCUSS] Drill 1.11.0 release
>>> 
>>> I'm Ok with the pcap format plugin if Pauls comments are also addressed.
>> If
>>> the changes are done before your cutoff deadline,we can merge it in.
>>> 
>>> On Thu, Jun 29, 2017 at 12:26 AM, Arina Yelchiyeva <
>>> arina.yelchiyeva@gmail.com> wrote:
>>> 
>>>> Hi all,
>>>> 
>>>> here is the list of Jira we are waiting before making final cut off:
>>>> 1. DRILL-5601  - Paul
>>>> 2. DRILL-3640  - Kunal
>>>> 3. DRILL-5420 - Kunal
>>>> 
>>>> Please let me know if there are any others Jiras we should wait for.
>>>> 
>>>> @Charles,
>>>> you have mentioned that you want to include some UDFs in Drill 1.11, do
>>> you
>>>> have any ETA for the PR?
>>>> 
>>>> @Paul / Parth
>>>> I see there is active CR for pcap format PR, do we want to include it
>> in
>>>> this release as well?
>>>> 
>>>> Kind regards
>>>> Arina
>>>> 
>>>> 
>>>> On Thu, Jun 29, 2017 at 10:05 AM, Arina Yelchiyeva <
>>>> arina.yelchiyeva@gmail.com> wrote:
>>>> 
>>>>> Parth, thanks a lot the instructions.
>>>>> 
>>>>> 
>>>>> 
>>>>> On Wed, Jun 28, 2017 at 8:24 PM, Parth Chandra <pa...@apache.org>
>>>> wrote:
>>>>> 
>>>>>> FWIW, I put release instructions culled from gists written by
>> previous
>>>>>> release managers into an updated gist here [1]
>>>>>> 
>>>>>> HTH
>>>>>> 
>>>>>> Parth
>>>>>> 
>>>>>> [1] https://github.com/parthchandra/drill/wiki/Drill-
>> Release-Process
>>>>>> <https://github.com/parthchandra/drill/wiki/Drill-Release-Process>
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> On Tue, Jun 27, 2017 at 5:24 PM, Paul Rogers <pr...@mapr.com>
>>> wrote:
>>>>>> 
>>>>>>> Hi Arina,
>>>>>>> 
>>>>>>> I have two projects in progress: one is a candidate for 1.11, the
>>>> other
>>>>>> is
>>>>>>> not.
>>>>>>> 
>>>>>>> The “managed external sort” is the good candidate for 1.11: The PR
>>> for
>>>>>>> DRILL-5601 is being reviewed. Getting that in will allow us to
>>> enable
>>>>>> the
>>>>>>> “new” external sort by default.
>>>>>>> 
>>>>>>> The other project is to address Drill’s memory fragmentation issue
>>>> with
>>>>>>> vectors over 16 MB in size. Two PRs have been open for a while.
>> They
>>>> are
>>>>>>> the first of several. Each takes a while to review. So, the
>> project
>>>> does
>>>>>>> not fit 1.11. The work can go into the next release instead.
>>>>>>> 
>>>>>>> Thanks,
>>>>>>> 
>>>>>>> - Paul
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>> 
>>>>> 
>>>> 
>>> 
>>

Re: [DISCUSS] Drill 1.11.0 release

Posted by Jinfeng Ni <jn...@apache.org>.

Hi Arina,

It has been a while since we discussed 1.11.0 release.  Any update on the
new release plan?  Do we have a cut-off date ?

In the past, I believe Drill release is a time-bounded, not feature
bounded. If certain fixes/features could not make it within a certain
time-frame, we had better defer them into future release, unless those
fixes/features are regarded as blocking issues for a new release.







On Tue, Jul 4, 2017 at 5:14 AM, Arina Yelchiyeva <arina.yelchiyeva@gmail.com
> wrote:

> Latest update:
>
> We wait for the following Jiras before final cut off.
> 1. DRILL-5601  - Paul
> 2. DRILL-5420 - Kunal
> 3. DRILL-5634 - Charles
>
> Kind regards
> Arina
>
> On Fri, Jun 30, 2017 at 9:29 PM, Kunal Khatua <kk...@mapr.com> wrote:
>
> > Parth's already +1'd the process with a minor update. So, we're good to
> go
> > on DRILL-5420. I'll change the label on that to ready-to-commit.
> >
> > <http://www.mapr.com/>
> >
> > ________________________________
> > From: Parth Chandra <pa...@apache.org>
> > Sent: Thursday, June 29, 2017 9:44:39 AM
> > To: dev@drill.apache.org
> > Subject: Re: [DISCUSS] Drill 1.11.0 release
> >
> > I'm Ok with the pcap format plugin if Pauls comments are also addressed.
> If
> > the changes are done before your cutoff deadline,we can merge it in.
> >
> > On Thu, Jun 29, 2017 at 12:26 AM, Arina Yelchiyeva <
> > arina.yelchiyeva@gmail.com> wrote:
> >
> > > Hi all,
> > >
> > > here is the list of Jira we are waiting before making final cut off:
> > > 1. DRILL-5601  - Paul
> > > 2. DRILL-3640  - Kunal
> > > 3. DRILL-5420 - Kunal
> > >
> > > Please let me know if there are any others Jiras we should wait for.
> > >
> > > @Charles,
> > > you have mentioned that you want to include some UDFs in Drill 1.11, do
> > you
> > > have any ETA for the PR?
> > >
> > > @Paul / Parth
> > > I see there is active CR for pcap format PR, do we want to include it
> in
> > > this release as well?
> > >
> > > Kind regards
> > > Arina
> > >
> > >
> > > On Thu, Jun 29, 2017 at 10:05 AM, Arina Yelchiyeva <
> > > arina.yelchiyeva@gmail.com> wrote:
> > >
> > > > Parth, thanks a lot the instructions.
> > > >
> > > >
> > > >
> > > > On Wed, Jun 28, 2017 at 8:24 PM, Parth Chandra <pa...@apache.org>
> > > wrote:
> > > >
> > > >> FWIW, I put release instructions culled from gists written by
> previous
> > > >> release managers into an updated gist here [1]
> > > >>
> > > >> HTH
> > > >>
> > > >> Parth
> > > >>
> > > >> [1] https://github.com/parthchandra/drill/wiki/Drill-
> Release-Process
> > > >> <https://github.com/parthchandra/drill/wiki/Drill-Release-Process>
> > > >>
> > > >>
> > > >>
> > > >> On Tue, Jun 27, 2017 at 5:24 PM, Paul Rogers <pr...@mapr.com>
> > wrote:
> > > >>
> > > >> > Hi Arina,
> > > >> >
> > > >> > I have two projects in progress: one is a candidate for 1.11, the
> > > other
> > > >> is
> > > >> > not.
> > > >> >
> > > >> > The “managed external sort” is the good candidate for 1.11: The PR
> > for
> > > >> > DRILL-5601 is being reviewed. Getting that in will allow us to
> > enable
> > > >> the
> > > >> > “new” external sort by default.
> > > >> >
> > > >> > The other project is to address Drill’s memory fragmentation issue
> > > with
> > > >> > vectors over 16 MB in size. Two PRs have been open for a while.
> They
> > > are
> > > >> > the first of several. Each takes a while to review. So, the
> project
> > > does
> > > >> > not fit 1.11. The work can go into the next release instead.
> > > >> >
> > > >> > Thanks,
> > > >> >
> > > >> > - Paul
> > > >> >
> > > >> >
> > > >> >
> > > >> >
> > > >>
> > > >
> > > >
> > >
> >
>

Re: [DISCUSS] Drill 1.11.0 release

Posted by Arina Yelchiyeva <ar...@gmail.com>.

Latest update:

We wait for the following Jiras before final cut off.
1. DRILL-5601  - Paul
2. DRILL-5420 - Kunal
3. DRILL-5634 - Charles

Kind regards
Arina

On Fri, Jun 30, 2017 at 9:29 PM, Kunal Khatua <kk...@mapr.com> wrote:

> Parth's already +1'd the process with a minor update. So, we're good to go
> on DRILL-5420. I'll change the label on that to ready-to-commit.
>
> <http://www.mapr.com/>
>
> ________________________________
> From: Parth Chandra <pa...@apache.org>
> Sent: Thursday, June 29, 2017 9:44:39 AM
> To: dev@drill.apache.org
> Subject: Re: [DISCUSS] Drill 1.11.0 release
>
> I'm Ok with the pcap format plugin if Pauls comments are also addressed. If
> the changes are done before your cutoff deadline,we can merge it in.
>
> On Thu, Jun 29, 2017 at 12:26 AM, Arina Yelchiyeva <
> arina.yelchiyeva@gmail.com> wrote:
>
> > Hi all,
> >
> > here is the list of Jira we are waiting before making final cut off:
> > 1. DRILL-5601  - Paul
> > 2. DRILL-3640  - Kunal
> > 3. DRILL-5420 - Kunal
> >
> > Please let me know if there are any others Jiras we should wait for.
> >
> > @Charles,
> > you have mentioned that you want to include some UDFs in Drill 1.11, do
> you
> > have any ETA for the PR?
> >
> > @Paul / Parth
> > I see there is active CR for pcap format PR, do we want to include it in
> > this release as well?
> >
> > Kind regards
> > Arina
> >
> >
> > On Thu, Jun 29, 2017 at 10:05 AM, Arina Yelchiyeva <
> > arina.yelchiyeva@gmail.com> wrote:
> >
> > > Parth, thanks a lot the instructions.
> > >
> > >
> > >
> > > On Wed, Jun 28, 2017 at 8:24 PM, Parth Chandra <pa...@apache.org>
> > wrote:
> > >
> > >> FWIW, I put release instructions culled from gists written by previous
> > >> release managers into an updated gist here [1]
> > >>
> > >> HTH
> > >>
> > >> Parth
> > >>
> > >> [1] https://github.com/parthchandra/drill/wiki/Drill-Release-Process
> > >> <https://github.com/parthchandra/drill/wiki/Drill-Release-Process>
> > >>
> > >>
> > >>
> > >> On Tue, Jun 27, 2017 at 5:24 PM, Paul Rogers <pr...@mapr.com>
> wrote:
> > >>
> > >> > Hi Arina,
> > >> >
> > >> > I have two projects in progress: one is a candidate for 1.11, the
> > other
> > >> is
> > >> > not.
> > >> >
> > >> > The “managed external sort” is the good candidate for 1.11: The PR
> for
> > >> > DRILL-5601 is being reviewed. Getting that in will allow us to
> enable
> > >> the
> > >> > “new” external sort by default.
> > >> >
> > >> > The other project is to address Drill’s memory fragmentation issue
> > with
> > >> > vectors over 16 MB in size. Two PRs have been open for a while. They
> > are
> > >> > the first of several. Each takes a while to review. So, the project
> > does
> > >> > not fit 1.11. The work can go into the next release instead.
> > >> >
> > >> > Thanks,
> > >> >
> > >> > - Paul
> > >> >
> > >> >
> > >> >
> > >> >
> > >>
> > >
> > >
> >
>

Re: [DISCUSS] Drill 1.11.0 release

Posted by Kunal Khatua <kk...@mapr.com>.

Parth's already +1'd the process with a minor update. So, we're good to go on DRILL-5420. I'll change the label on that to ready-to-commit.

<http://www.mapr.com/>

________________________________
From: Parth Chandra <pa...@apache.org>
Sent: Thursday, June 29, 2017 9:44:39 AM
To: dev@drill.apache.org
Subject: Re: [DISCUSS] Drill 1.11.0 release

I'm Ok with the pcap format plugin if Pauls comments are also addressed. If
the changes are done before your cutoff deadline,we can merge it in.

On Thu, Jun 29, 2017 at 12:26 AM, Arina Yelchiyeva <
arina.yelchiyeva@gmail.com> wrote:

> Hi all,
>
> here is the list of Jira we are waiting before making final cut off:
> 1. DRILL-5601  - Paul
> 2. DRILL-3640  - Kunal
> 3. DRILL-5420 - Kunal
>
> Please let me know if there are any others Jiras we should wait for.
>
> @Charles,
> you have mentioned that you want to include some UDFs in Drill 1.11, do you
> have any ETA for the PR?
>
> @Paul / Parth
> I see there is active CR for pcap format PR, do we want to include it in
> this release as well?
>
> Kind regards
> Arina
>
>
> On Thu, Jun 29, 2017 at 10:05 AM, Arina Yelchiyeva <
> arina.yelchiyeva@gmail.com> wrote:
>
> > Parth, thanks a lot the instructions.
> >
> >
> >
> > On Wed, Jun 28, 2017 at 8:24 PM, Parth Chandra <pa...@apache.org>
> wrote:
> >
> >> FWIW, I put release instructions culled from gists written by previous
> >> release managers into an updated gist here [1]
> >>
> >> HTH
> >>
> >> Parth
> >>
> >> [1] https://github.com/parthchandra/drill/wiki/Drill-Release-Process
> >> <https://github.com/parthchandra/drill/wiki/Drill-Release-Process>
> >>
> >>
> >>
> >> On Tue, Jun 27, 2017 at 5:24 PM, Paul Rogers <pr...@mapr.com> wrote:
> >>
> >> > Hi Arina,
> >> >
> >> > I have two projects in progress: one is a candidate for 1.11, the
> other
> >> is
> >> > not.
> >> >
> >> > The “managed external sort” is the good candidate for 1.11: The PR for
> >> > DRILL-5601 is being reviewed. Getting that in will allow us to enable
> >> the
> >> > “new” external sort by default.
> >> >
> >> > The other project is to address Drill’s memory fragmentation issue
> with
> >> > vectors over 16 MB in size. Two PRs have been open for a while. They
> are
> >> > the first of several. Each takes a while to review. So, the project
> does
> >> > not fit 1.11. The work can go into the next release instead.
> >> >
> >> > Thanks,
> >> >
> >> > - Paul
> >> >
> >> >
> >> >
> >> >
> >>
> >
> >
>

Re: [DISCUSS] Drill 1.11.0 release

Posted by Parth Chandra <pa...@apache.org>.

I'm Ok with the pcap format plugin if Pauls comments are also addressed. If
the changes are done before your cutoff deadline,we can merge it in.

On Thu, Jun 29, 2017 at 12:26 AM, Arina Yelchiyeva <
arina.yelchiyeva@gmail.com> wrote:

> Hi all,
>
> here is the list of Jira we are waiting before making final cut off:
> 1. DRILL-5601  - Paul
> 2. DRILL-3640  - Kunal
> 3. DRILL-5420 - Kunal
>
> Please let me know if there are any others Jiras we should wait for.
>
> @Charles,
> you have mentioned that you want to include some UDFs in Drill 1.11, do you
> have any ETA for the PR?
>
> @Paul / Parth
> I see there is active CR for pcap format PR, do we want to include it in
> this release as well?
>
> Kind regards
> Arina
>
>
> On Thu, Jun 29, 2017 at 10:05 AM, Arina Yelchiyeva <
> arina.yelchiyeva@gmail.com> wrote:
>
> > Parth, thanks a lot the instructions.
> >
> >
> >
> > On Wed, Jun 28, 2017 at 8:24 PM, Parth Chandra <pa...@apache.org>
> wrote:
> >
> >> FWIW, I put release instructions culled from gists written by previous
> >> release managers into an updated gist here [1]
> >>
> >> HTH
> >>
> >> Parth
> >>
> >> [1] https://github.com/parthchandra/drill/wiki/Drill-Release-Process
> >> <https://github.com/parthchandra/drill/wiki/Drill-Release-Process>
> >>
> >>
> >>
> >> On Tue, Jun 27, 2017 at 5:24 PM, Paul Rogers <pr...@mapr.com> wrote:
> >>
> >> > Hi Arina,
> >> >
> >> > I have two projects in progress: one is a candidate for 1.11, the
> other
> >> is
> >> > not.
> >> >
> >> > The “managed external sort” is the good candidate for 1.11: The PR for
> >> > DRILL-5601 is being reviewed. Getting that in will allow us to enable
> >> the
> >> > “new” external sort by default.
> >> >
> >> > The other project is to address Drill’s memory fragmentation issue
> with
> >> > vectors over 16 MB in size. Two PRs have been open for a while. They
> are
> >> > the first of several. Each takes a while to review. So, the project
> does
> >> > not fit 1.11. The work can go into the next release instead.
> >> >
> >> > Thanks,
> >> >
> >> > - Paul
> >> >
> >> >
> >> >
> >> >
> >>
> >
> >
>

Re: [DISCUSS] Drill 1.11.0 release

Posted by Arina Yelchiyeva <ar...@gmail.com>.

Hi all,

here is the list of Jira we are waiting before making final cut off:
1. DRILL-5601  - Paul
2. DRILL-3640  - Kunal
3. DRILL-5420 - Kunal

Please let me know if there are any others Jiras we should wait for.

@Charles,
you have mentioned that you want to include some UDFs in Drill 1.11, do you
have any ETA for the PR?

@Paul / Parth
I see there is active CR for pcap format PR, do we want to include it in
this release as well?

Kind regards
Arina


On Thu, Jun 29, 2017 at 10:05 AM, Arina Yelchiyeva <
arina.yelchiyeva@gmail.com> wrote:

> Parth, thanks a lot the instructions.
>
>
>
> On Wed, Jun 28, 2017 at 8:24 PM, Parth Chandra <pa...@apache.org> wrote:
>
>> FWIW, I put release instructions culled from gists written by previous
>> release managers into an updated gist here [1]
>>
>> HTH
>>
>> Parth
>>
>> [1] https://github.com/parthchandra/drill/wiki/Drill-Release-Process
>> <https://github.com/parthchandra/drill/wiki/Drill-Release-Process>
>>
>>
>>
>> On Tue, Jun 27, 2017 at 5:24 PM, Paul Rogers <pr...@mapr.com> wrote:
>>
>> > Hi Arina,
>> >
>> > I have two projects in progress: one is a candidate for 1.11, the other
>> is
>> > not.
>> >
>> > The “managed external sort” is the good candidate for 1.11: The PR for
>> > DRILL-5601 is being reviewed. Getting that in will allow us to enable
>> the
>> > “new” external sort by default.
>> >
>> > The other project is to address Drill’s memory fragmentation issue with
>> > vectors over 16 MB in size. Two PRs have been open for a while. They are
>> > the first of several. Each takes a while to review. So, the project does
>> > not fit 1.11. The work can go into the next release instead.
>> >
>> > Thanks,
>> >
>> > - Paul
>> >
>> >
>> >
>> >
>>
>
>

Re: [DISCUSS] Drill 1.11.0 release

Posted by Arina Yelchiyeva <ar...@gmail.com>.

Parth, thanks a lot the instructions.



On Wed, Jun 28, 2017 at 8:24 PM, Parth Chandra <pa...@apache.org> wrote:

> FWIW, I put release instructions culled from gists written by previous
> release managers into an updated gist here [1]
>
> HTH
>
> Parth
>
> [1] https://github.com/parthchandra/drill/wiki/Drill-Release-Process
> <https://github.com/parthchandra/drill/wiki/Drill-Release-Process>
>
>
>
> On Tue, Jun 27, 2017 at 5:24 PM, Paul Rogers <pr...@mapr.com> wrote:
>
> > Hi Arina,
> >
> > I have two projects in progress: one is a candidate for 1.11, the other
> is
> > not.
> >
> > The “managed external sort” is the good candidate for 1.11: The PR for
> > DRILL-5601 is being reviewed. Getting that in will allow us to enable the
> > “new” external sort by default.
> >
> > The other project is to address Drill’s memory fragmentation issue with
> > vectors over 16 MB in size. Two PRs have been open for a while. They are
> > the first of several. Each takes a while to review. So, the project does
> > not fit 1.11. The work can go into the next release instead.
> >
> > Thanks,
> >
> > - Paul
> >
> >
> >
> >
>

Re: [DISCUSS] Drill 1.11.0 release

Posted by Parth Chandra <pa...@apache.org>.

FWIW, I put release instructions culled from gists written by previous
release managers into an updated gist here [1]

HTH

Parth

[1] https://github.com/parthchandra/drill/wiki/Drill-Release-Process
<https://github.com/parthchandra/drill/wiki/Drill-Release-Process>



On Tue, Jun 27, 2017 at 5:24 PM, Paul Rogers <pr...@mapr.com> wrote:

> Hi Arina,
>
> I have two projects in progress: one is a candidate for 1.11, the other is
> not.
>
> The “managed external sort” is the good candidate for 1.11: The PR for
> DRILL-5601 is being reviewed. Getting that in will allow us to enable the
> “new” external sort by default.
>
> The other project is to address Drill’s memory fragmentation issue with
> vectors over 16 MB in size. Two PRs have been open for a while. They are
> the first of several. Each takes a while to review. So, the project does
> not fit 1.11. The work can go into the next release instead.
>
> Thanks,
>
> - Paul
>
>
>
>

Re: [DISCUSS] Drill 1.11.0 release

Posted by Paul Rogers <pr...@mapr.com>.

Hi Arina,

I have two projects in progress: one is a candidate for 1.11, the other is not.

The “managed external sort” is the good candidate for 1.11: The PR for DRILL-5601 is being reviewed. Getting that in will allow us to enable the “new” external sort by default.

The other project is to address Drill’s memory fragmentation issue with vectors over 16 MB in size. Two PRs have been open for a while. They are the first of several. Each takes a while to review. So, the project does not fit 1.11. The work can go into the next release instead.

Thanks,

- Paul

Re: [DISCUSS] Drill 1.11.0 release

Posted by Arina Yelchiyeva <ar...@gmail.com>.

Hi Charles,

You create Drill Jira for this purpose but please add doc-impact label so
when your changes will be committed, functions are documented.
Drill stores UDFs in org.apache.drill.exec.expr.fn.impl package, you may
add your UDFs there. Please make sure your code complies with Drill
standards. Please add java doc describing general purpose of each function.
Regarding the unit tests, please take a look at String functions as example
[1, 2].

[1]
https://github.com/apache/drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/impl/StringFunctions.java

[2]
https://github.com/apache/drill/blob/master/exec/java-exec/src/test/java/org/apache/drill/exec/expr/fn/impl/TestStringFunctions.java

Kind regards
Arina


On Mon, Jun 26, 2017 at 6:10 PM, Charles Givre <cg...@gmail.com> wrote:

> Hi Arina,
> I have some UDFs which I’d love to include in version 1.11 which are
> available (and documented) here:
> https://github.com/cgivre/drill-crypto-functions <
> https://github.com/cgivre/drill-crypto-functions>
> and
> https://github.com/cgivre/drill-network-functions <
> https://github.com/cgivre/drill-network-functions>
>
> I’m not quite sure how to write unit tests for Drill UDFs so if someone
> could send me an example, I’ll do that and submit.  The crypto functions
> (MD5, SHA1) seem to be a fairly common request and the networking functions
> are a nice compliment for the PCAP functionality that Ted submitted.
>
> Thanks,
> — C
>
> > On Jun 23, 2017, at 07:45, Arina Yelchiyeva <ar...@gmail.com>
> wrote:
> >
> > If there are no objections, I'd like to volunteer.
> >
> > Kind regards
> > Arina
> >
> > On Fri, Jun 23, 2017 at 1:20 AM, Parth Chandra <pa...@apache.org>
> wrote:
> >
> >> Hello Drillers,
> >>
> >>  It's been three months since the last release and it is time to do the
> >> next one. Does anyone want to volunteer to be the release manager? I
> would
> >> encourage our new committers to try their hand at it :)
> >>
> >>  While we wait for volunteers, if there are any issues on which work is
> in
> >> progress, that you feel we *must* include in the release, please post in
> >> reply to this thread.
> >>
> >> Thanks
> >>
> >> Parth
> >>
>
>

Re: [DISCUSS] Drill 1.11.0 release

Posted by Charles Givre <cg...@gmail.com>.

Hi Arina, 
I have some UDFs which I’d love to include in version 1.11 which are available (and documented) here:
https://github.com/cgivre/drill-crypto-functions <https://github.com/cgivre/drill-crypto-functions>
and 
https://github.com/cgivre/drill-network-functions <https://github.com/cgivre/drill-network-functions>

I’m not quite sure how to write unit tests for Drill UDFs so if someone could send me an example, I’ll do that and submit.  The crypto functions (MD5, SHA1) seem to be a fairly common request and the networking functions are a nice compliment for the PCAP functionality that Ted submitted.

Thanks,
— C

> On Jun 23, 2017, at 07:45, Arina Yelchiyeva <ar...@gmail.com> wrote:
> 
> If there are no objections, I'd like to volunteer.
> 
> Kind regards
> Arina
> 
> On Fri, Jun 23, 2017 at 1:20 AM, Parth Chandra <pa...@apache.org> wrote:
> 
>> Hello Drillers,
>> 
>>  It's been three months since the last release and it is time to do the
>> next one. Does anyone want to volunteer to be the release manager? I would
>> encourage our new committers to try their hand at it :)
>> 
>>  While we wait for volunteers, if there are any issues on which work is in
>> progress, that you feel we *must* include in the release, please post in
>> reply to this thread.
>> 
>> Thanks
>> 
>> Parth
>>

Re: [DISCUSS] Drill 1.11.0 release

Posted by Parth Chandra <pa...@apache.org>.

Thanks Arina for volunteering to do the release. Please continue to use
this thread for the discussion or start a new one, whichever is convenient.

Parth

On Fri, Jun 23, 2017 at 10:52 AM, Jinfeng Ni <jn...@apache.org> wrote:

> +1  do Drill 1.11 release soon, and Arina as the new release manager.
>
>
>
> On Fri, Jun 23, 2017 at 9:49 AM, Aman Sinha <am...@apache.org> wrote:
>
> > Sounds good, Arina and thanks for volunteering!   I think it will be a
> good
> > experience for you doing the release.  Reach out to past release managers
> > for any help.  In particular, make sure to use Parth's handy scripts that
> > should make the job lot easier.
> >
> > -Aman
> >
> > On Fri, Jun 23, 2017 at 4:45 AM, Arina Yelchiyeva <
> > arina.yelchiyeva@gmail.com> wrote:
> >
> > > If there are no objections, I'd like to volunteer.
> > >
> > > Kind regards
> > > Arina
> > >
> > > On Fri, Jun 23, 2017 at 1:20 AM, Parth Chandra <pa...@apache.org>
> > wrote:
> > >
> > > > Hello Drillers,
> > > >
> > > >   It's been three months since the last release and it is time to do
> > the
> > > > next one. Does anyone want to volunteer to be the release manager? I
> > > would
> > > > encourage our new committers to try their hand at it :)
> > > >
> > > >   While we wait for volunteers, if there are any issues on which work
> > is
> > > in
> > > > progress, that you feel we *must* include in the release, please post
> > in
> > > > reply to this thread.
> > > >
> > > > Thanks
> > > >
> > > > Parth
> > > >
> > >
> >
>

Re: [DISCUSS] Drill 1.11.0 release

Posted by Jinfeng Ni <jn...@apache.org>.

+1  do Drill 1.11 release soon, and Arina as the new release manager.



On Fri, Jun 23, 2017 at 9:49 AM, Aman Sinha <am...@apache.org> wrote:

> Sounds good, Arina and thanks for volunteering!   I think it will be a good
> experience for you doing the release.  Reach out to past release managers
> for any help.  In particular, make sure to use Parth's handy scripts that
> should make the job lot easier.
>
> -Aman
>
> On Fri, Jun 23, 2017 at 4:45 AM, Arina Yelchiyeva <
> arina.yelchiyeva@gmail.com> wrote:
>
> > If there are no objections, I'd like to volunteer.
> >
> > Kind regards
> > Arina
> >
> > On Fri, Jun 23, 2017 at 1:20 AM, Parth Chandra <pa...@apache.org>
> wrote:
> >
> > > Hello Drillers,
> > >
> > >   It's been three months since the last release and it is time to do
> the
> > > next one. Does anyone want to volunteer to be the release manager? I
> > would
> > > encourage our new committers to try their hand at it :)
> > >
> > >   While we wait for volunteers, if there are any issues on which work
> is
> > in
> > > progress, that you feel we *must* include in the release, please post
> in
> > > reply to this thread.
> > >
> > > Thanks
> > >
> > > Parth
> > >
> >
>

Re: [DISCUSS] Drill 1.11.0 release

Posted by Aman Sinha <am...@apache.org>.

Sounds good, Arina and thanks for volunteering!   I think it will be a good
experience for you doing the release.  Reach out to past release managers
for any help.  In particular, make sure to use Parth's handy scripts that
should make the job lot easier.

-Aman

On Fri, Jun 23, 2017 at 4:45 AM, Arina Yelchiyeva <
arina.yelchiyeva@gmail.com> wrote:

> If there are no objections, I'd like to volunteer.
>
> Kind regards
> Arina
>
> On Fri, Jun 23, 2017 at 1:20 AM, Parth Chandra <pa...@apache.org> wrote:
>
> > Hello Drillers,
> >
> >   It's been three months since the last release and it is time to do the
> > next one. Does anyone want to volunteer to be the release manager? I
> would
> > encourage our new committers to try their hand at it :)
> >
> >   While we wait for volunteers, if there are any issues on which work is
> in
> > progress, that you feel we *must* include in the release, please post in
> > reply to this thread.
> >
> > Thanks
> >
> > Parth
> >
>

Re: [DISCUSS] Drill 1.11.0 release

Posted by Arina Yelchiyeva <ar...@gmail.com>.

If there are no objections, I'd like to volunteer.

Kind regards
Arina

On Fri, Jun 23, 2017 at 1:20 AM, Parth Chandra <pa...@apache.org> wrote:

> Hello Drillers,
>
>   It's been three months since the last release and it is time to do the
> next one. Does anyone want to volunteer to be the release manager? I would
> encourage our new committers to try their hand at it :)
>
>   While we wait for volunteers, if there are any issues on which work is in
> progress, that you feel we *must* include in the release, please post in
> reply to this thread.
>
> Thanks
>
> Parth
>