You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@drill.apache.org by Boaz Ben-Zvi <bo...@mapr.com> on 2018/07/03 03:01:47 UTC
Re: [DISCUSS] 1.14.0 release
Let's try to make progress on the 1.14 release, aiming for a Release
Candidate towards the end of this week (a little ambitious, with the
July 4th and people on vacations).
Current Status of the previously requested Jiras:
==============================
In Progress - DRILL-6104: Generic Logfile Format Plugin
PR - DRILL-6422: Update Guava to 23.0 and shade it
PR - DRILL-5999 (DRILL-6516): Support for EMIT outcome in Streaming Agg
Ready2Commit: DRILL-5977: predicate pushdown support kafkaMsgOffset
Ready2Commit: DRILL-6519: Add String Distance and Phonetic Functions
Ready2Commit: DRILL-6577: Change Hash-Join default to not fallback (into
pre-1.14 unlimited memory)
Committed: DRILL-6353: Upgrade Parquet MR dependencies
Committed: DRILL-6310: limit batch size for hash aggregate
===============================
And there are few more open or in a PR state.
Lets try and most of these ready by the end of the week.
Boaz
Re: [DISCUSS] 1.14.0 release
Posted by Boaz Ben-Zvi <bo...@apache.org>.
That cracked blocker is DRILL-6453 (not 6475 ...), sorry ..
Boaz
On 7/19/18 2:32 AM, Boaz Ben-Zvi wrote:
> Hi Charles,
>
> I merged the DRILL-6104 work into the Apache master; thanks for
> your useful contribution, it will be included in 1.14.
>
> We may have just cracked the blocker DRILL-6475 , so we hopefully
> could start the RC process tomorrow.
>
> Thanks,
>
> Boaz
>
>
> On 7/18/18 8:42 PM, Charles Givre wrote:
>> HI Boaz,
>> DRILL-6104 is ready to release. Do you think we’ll have an RC this
>> week?
>> Thanks,
>> — C
>>
>>> On Jul 2, 2018, at 23:01, Boaz Ben-Zvi <bo...@mapr.com> wrote:
>>>
>>> Let's try to make progress on the 1.14 release, aiming for a
>>> Release Candidate towards the end of this week (a little ambitious,
>>> with the July 4th and people on vacations).
>>>
>>> Current Status of the previously requested Jiras:
>>>
>>> ==============================
>>>
>>> In Progress - DRILL-6104: Generic Logfile Format Plugin
>>>
>>> PR - DRILL-6422: Update Guava to 23.0 and shade it
>>>
>>> PR - DRILL-5999 (DRILL-6516): Support for EMIT outcome in Streaming Agg
>>>
>>> Ready2Commit: DRILL-5977: predicate pushdown support kafkaMsgOffset
>>>
>>> Ready2Commit: DRILL-6519: Add String Distance and Phonetic Functions
>>>
>>> Ready2Commit: DRILL-6577: Change Hash-Join default to not fallback
>>> (into pre-1.14 unlimited memory)
>>>
>>> Committed: DRILL-6353: Upgrade Parquet MR dependencies
>>>
>>> Committed: DRILL-6310: limit batch size for hash aggregate
>>>
>>> ===============================
>>>
>>> And there are few more open or in a PR state.
>>>
>>> Lets try and most of these ready by the end of the week.
>>>
>>> Boaz
>>>
>>>
>
Re: [DISCUSS] 1.14.0 release
Posted by Boaz Ben-Zvi <bo...@apache.org>.
Hi Charles,
I merged the DRILL-6104 work into the Apache master; thanks for
your useful contribution, it will be included in 1.14.
We may have just cracked the blocker DRILL-6475 , so we hopefully could
start the RC process tomorrow.
Thanks,
Boaz
On 7/18/18 8:42 PM, Charles Givre wrote:
> HI Boaz,
> DRILL-6104 is ready to release. Do you think we’ll have an RC this week?
> Thanks,
> — C
>
>> On Jul 2, 2018, at 23:01, Boaz Ben-Zvi <bo...@mapr.com> wrote:
>>
>> Let's try to make progress on the 1.14 release, aiming for a Release Candidate towards the end of this week (a little ambitious, with the July 4th and people on vacations).
>>
>> Current Status of the previously requested Jiras:
>>
>> ==============================
>>
>> In Progress - DRILL-6104: Generic Logfile Format Plugin
>>
>> PR - DRILL-6422: Update Guava to 23.0 and shade it
>>
>> PR - DRILL-5999 (DRILL-6516): Support for EMIT outcome in Streaming Agg
>>
>> Ready2Commit: DRILL-5977: predicate pushdown support kafkaMsgOffset
>>
>> Ready2Commit: DRILL-6519: Add String Distance and Phonetic Functions
>>
>> Ready2Commit: DRILL-6577: Change Hash-Join default to not fallback (into pre-1.14 unlimited memory)
>>
>> Committed: DRILL-6353: Upgrade Parquet MR dependencies
>>
>> Committed: DRILL-6310: limit batch size for hash aggregate
>>
>> ===============================
>>
>> And there are few more open or in a PR state.
>>
>> Lets try and most of these ready by the end of the week.
>>
>> Boaz
>>
>>
Re: [DISCUSS] 1.14.0 release
Posted by Charles Givre <cg...@gmail.com>.
HI Boaz,
DRILL-6104 is ready to release. Do you think we’ll have an RC this week?
Thanks,
— C
> On Jul 2, 2018, at 23:01, Boaz Ben-Zvi <bo...@mapr.com> wrote:
>
> Let's try to make progress on the 1.14 release, aiming for a Release Candidate towards the end of this week (a little ambitious, with the July 4th and people on vacations).
>
> Current Status of the previously requested Jiras:
>
> ==============================
>
> In Progress - DRILL-6104: Generic Logfile Format Plugin
>
> PR - DRILL-6422: Update Guava to 23.0 and shade it
>
> PR - DRILL-5999 (DRILL-6516): Support for EMIT outcome in Streaming Agg
>
> Ready2Commit: DRILL-5977: predicate pushdown support kafkaMsgOffset
>
> Ready2Commit: DRILL-6519: Add String Distance and Phonetic Functions
>
> Ready2Commit: DRILL-6577: Change Hash-Join default to not fallback (into pre-1.14 unlimited memory)
>
> Committed: DRILL-6353: Upgrade Parquet MR dependencies
>
> Committed: DRILL-6310: limit batch size for hash aggregate
>
> ===============================
>
> And there are few more open or in a PR state.
>
> Lets try and most of these ready by the end of the week.
>
> Boaz
>
>
Re: [DISCUSS] 1.14.0 release
Posted by Kunal Khatua <ku...@apache.org>.
I have 2 fixes (minor UX and a trivial SysTable patch) that are needed for 1.14.0
DRILL-6583 (already has a +1)
DRILL-6588 (pending review)
~Kunal
On 7/10/2018 12:02:36 AM, Arina Yelchiyeva <ar...@gmail.com> wrote:
Hi Boaz,
as far as I understand you either move this feature to 1.15 or wait with
building RC until it's finished. Adding changes later means, you'll have to
build one more RC candidate and conduct one more vote.
I think making people voting twice instead of once, it's not a good thing :)
Kind regards,
Arina
On Tue, Jul 10, 2018 at 8:07 AM Boaz Ben-Zvi wrote:
> Hi Charles,
>
> The main reason for rushing a Release Candidate is that we could
> give it enough testing.
>
> Given that DRILL-6104 is a separate feature, with almost no impact on
> the current code, then it seems low risk to add it a few days later.
>
> Anyone has an objection ?
>
> Boaz
>
> On 7/9/18 9:54 PM, Charles Givre wrote:
> > Hi Boaz,
> > I’m traveling at the moment, but I can have DRILL-6104 back in Paul’s
> hands by the end of the week.
> > —C
> >
> >> On Jul 10, 2018, at 00:53, Boaz Ben-Zvi wrote:
> >>
> >> We are making progress towards 1.14.
> >>
> >> Let's aim for a Release Candidate branch off on Thursday (July 12) !!!
> >>
> >> Below are the unfinished cases; can most be completed and checked in by
> 7/12 ?
> >>
> >> (( Relevant people:
> >>
> >> Abhishek, Arina, Boaz, Charles, Hanumath, Jean-Blas, Karthik,
> Kunal,
> >>
> >> Parth, Paul, Salim, Sorabh, Tim, Vitalii, Vlad, Volodymyr ))
> >>
> >> ======================
> >>
> >> Open/blocker - DRILL-6453 + DRILL-6517:
> >> Two issues - Parquet Scanner (?) not setting container's record num
> (to zero), and a hang following this failure.
> >> Currently testing a fix / workaround ((Boaz))
> >>
> >> In Progress - DRILL-6104: Generic Logfile Format Plugin ((Charles +
> Paul -- can you be done by 7/12 ?))
> >>
> >> PR - DRILL-5999 (DRILL-6516): Support for EMIT outcome in Streaming Agg
> ((Parth + Boaz reviewing))
> >>
> >> Open - DRILL-6542: Index out of bounds ((Sorabh))
> >>
> >> Open - DRILL-6475: Unnest Null fieldId pointer ((Hanumath))
> >>
> >> ==== The following PRs are still waiting for reviews ====
> >>
> >> DRILL-6583: UI usability issue ((Kunal / Sorabh))
> >>
> >> DRILL-6579: Add sanity checks to the Parquet Reader ((Salim / Vlad +
> Boaz))
> >>
> >> DRILL-6578: handle query cancellation in Parquet Reader ((Salim / Vlad
> + Boaz))
> >>
> >> DRILL-6560: Allow options for controlling the batch size per operator
> ((Salim / Karthik))
> >>
> >> DRILL-6559: Travis timing out ((Vitalii / Tim))
> >>
> >> DRILL-6496: VectorUtil.showVectorAccessibleContent does not log vector
> content ((Tim / Volodymyr))
> >>
> >> DRILL-6410: Memory Leak in Parquet Reader during cancellation ((Vlad /
> Parth))
> >>
> >> DRILL-6346: Create an Official Drill Docker Container ((Abhishek / Tim))
> >>
> >> DRILL-6179: Added pcapng-format support ((Vlad / Paul))
> >>
> >> DRILL-5796: Filter pruning for multi rowgroup parquet file ((Jean-Blas
> / Arina))
> >>
> >> DRILL-5365: FileNotFoundException when reading a parquet file ((Tim /
> Vitalii))
> >>
> >> ==================================
> >>
> >> Thanks,
> >>
> >> Boaz
> >>
> >> On 7/6/18 2:51 PM, Pritesh Maker wrote:
> >>> Here is the release 1.14 dashboard (
> https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_secure_Dashboard.jspa-3FselectPageId-3D12332463&d=DwIGaQ&c=cskdkSMqhcnjZxdQVpwTXg&r=7lXQnf0aC8VQ0iMXwVgNHw&m=V0b4-BeuIMwRczzbiSXCgL7Z7f8lrmLBGH1vnSHLjB4&s=HRQU6Q4umbONtN4EqY3ryggJNEyCOghAzICypRJOels&e=
> ) and agile board (
> https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_secure_RapidBoard.jspa-3FrapidView-3D185&d=DwIGaQ&c=cskdkSMqhcnjZxdQVpwTXg&r=7lXQnf0aC8VQ0iMXwVgNHw&m=V0b4-BeuIMwRczzbiSXCgL7Z7f8lrmLBGH1vnSHLjB4&s=GKxSMl97YRnHJu-AL-A-vvRe5SXqw7vdDPDzMzj-Cj4&e=
> )
> >>>
> >>> I believe Volodymyr is targeting DRILL-6422 (Guava update) for 1.15
> release so it shouldn't be blocking the release. So overall, we have 2 open
> bugs, 2 in progress bugs (+2 doc issues), and 12 in review (+1 ready to
> commit).
> >>>
> >>> If the reviewable commits won't be ready soon, can the developers
> please remove the 1.14 fix version for these issues.
> >>>
> >>> Pritesh
> >>>
> >>>
> >>>
> >>>
> >>> On 7/6/18, 11:54 AM, "Boaz Ben-Zvi"
> boaz@mapr.com> wrote:
> >>>
> >>> Current status: There's a blocker, and some work in progress
> that will
> >>> stretch into next week.
> >>> Current detail:
> >>> ==============================
> >>> Open/blocker - DRILL-6453 + DRILL-6517: Two issues - Parquet
> Scanner not setting record num (to zero), and a hang following this failure.
> >>> In Progress - DRILL-6104: Generic Logfile Format Plugin
> >>> PR - DRILL-6422: Update Guava to 23.0 and shade it
> >>> PR - DRILL-5999 (DRILL-6516): Support for EMIT outcome in
> Streaming Agg (I'm reviewing)
> >>> Ready2Commit: DRILL-6519: Add String Distance and
> Phonetic Functions (Arina gave it a +1 ; is it "Ready-To-Commit" or waiting
> for more reviews ?)
> >>> Committed: DRILL-6570: Mentioned as a blocker by Kunal
> (I just merge #1354; the Jira was (mistakenly ?) marked "Resolved" so it
> missed the batch commit).
> >>> Committed: DRILL-5977: predicate pushdown support
> kafkaMsgOffset
> >>> Committed: DRILL-6577: Change Hash-Join default to not
> fallback (into
> >>> pre-1.14 unlimited memory)
> >>> Committed: DRILL-6353: Upgrade Parquet MR dependencies
> >>> Committed: DRILL-6310: limit batch size for hash aggregate
> >>> ===============================
> >>> Thanks,
> >>> Boaz
> >>> On 7/2/18 9:51 PM, Khurram Faraaz wrote:
> >>> > Do we plan to fix this one too, because this is a regression
> from Apache
> >>> > Drill 1.13.0.
> >>> >
> https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_DRILL-2D6453&d=DwIBaQ&c=cskdkSMqhcnjZxdQVpwTXg&r=EqulKDxxEDCX6zbp1AZAa1-iAPQGgCioAqgDp7DE2BU&m=9HpIZKDh_DBcx9yXCI2TtUWum8vRhdQgmw_3ljJQi1M&s=w-1C-HnlUAGRHvDFUusZ78WZrHahDka2r-F-cNb-pZA&e=
> >>> >
> >>> > On Mon, Jul 2, 2018 at 9:33 PM, Kunal Khatua
> wrote:
> >>> >
> >>> >> DRILL-6570 seems like a must-have (release blocker, IMHO).
> >>> >> On 7/2/2018 8:02:00 PM, Boaz Ben-Zvi wrote:
> >>> >> Let's try to make progress on the 1.14 release, aiming for a
> Release
> >>> >> Candidate towards the end of this week (a little ambitious,
> with the
> >>> >> July 4th and people on vacations).
> >>> >>
> >>> >> Current Status of the previously requested Jiras:
> >>> >>
> >>> >> ==============================
> >>> >>
> >>> >> In Progress - DRILL-6104: Generic Logfile Format Plugin
> >>> >>
> >>> >> PR - DRILL-6422: Update Guava to 23.0 and shade it
> >>> >>
> >>> >> PR - DRILL-5999 (DRILL-6516): Support for EMIT outcome in
> Streaming Agg
> >>> >>
> >>> >> Ready2Commit: DRILL-5977: predicate pushdown support
> kafkaMsgOffset
> >>> >>
> >>> >> Ready2Commit: DRILL-6519: Add String Distance and Phonetic
> Functions
> >>> >>
> >>> >> Ready2Commit: DRILL-6577: Change Hash-Join default to not
> fallback (into
> >>> >> pre-1.14 unlimited memory)
> >>> >>
> >>> >> Committed: DRILL-6353: Upgrade Parquet MR dependencies
> >>> >>
> >>> >> Committed: DRILL-6310: limit batch size for hash aggregate
> >>> >>
> >>> >> ===============================
> >>> >>
> >>> >> And there are few more open or in a PR state.
> >>> >>
> >>> >> Lets try and most of these ready by the end of the week.
> >>> >>
> >>> >> Boaz
> >>> >>
> >>> >>
> >>> >>
> >>>
>
>
Re: [DISCUSS] 1.14.0 release
Posted by Arina Yelchiyeva <ar...@gmail.com>.
Hi Boaz,
as far as I understand you either move this feature to 1.15 or wait with
building RC until it's finished. Adding changes later means, you'll have to
build one more RC candidate and conduct one more vote.
I think making people voting twice instead of once, it's not a good thing :)
Kind regards,
Arina
On Tue, Jul 10, 2018 at 8:07 AM Boaz Ben-Zvi <bo...@mapr.com> wrote:
> Hi Charles,
>
> The main reason for rushing a Release Candidate is that we could
> give it enough testing.
>
> Given that DRILL-6104 is a separate feature, with almost no impact on
> the current code, then it seems low risk to add it a few days later.
>
> Anyone has an objection ?
>
> Boaz
>
> On 7/9/18 9:54 PM, Charles Givre wrote:
> > Hi Boaz,
> > I’m traveling at the moment, but I can have DRILL-6104 back in Paul’s
> hands by the end of the week.
> > —C
> >
> >> On Jul 10, 2018, at 00:53, Boaz Ben-Zvi <bo...@mapr.com> wrote:
> >>
> >> We are making progress towards 1.14.
> >>
> >> Let's aim for a Release Candidate branch off on Thursday (July 12) !!!
> >>
> >> Below are the unfinished cases; can most be completed and checked in by
> 7/12 ?
> >>
> >> (( Relevant people:
> >>
> >> Abhishek, Arina, Boaz, Charles, Hanumath, Jean-Blas, Karthik,
> Kunal,
> >>
> >> Parth, Paul, Salim, Sorabh, Tim, Vitalii, Vlad, Volodymyr ))
> >>
> >> ======================
> >>
> >> Open/blocker - DRILL-6453 + DRILL-6517:
> >> Two issues - Parquet Scanner (?) not setting container's record num
> (to zero), and a hang following this failure.
> >> Currently testing a fix / workaround ((Boaz))
> >>
> >> In Progress - DRILL-6104: Generic Logfile Format Plugin ((Charles +
> Paul -- can you be done by 7/12 ?))
> >>
> >> PR - DRILL-5999 (DRILL-6516): Support for EMIT outcome in Streaming Agg
> ((Parth + Boaz reviewing))
> >>
> >> Open - DRILL-6542: Index out of bounds ((Sorabh))
> >>
> >> Open - DRILL-6475: Unnest Null fieldId pointer ((Hanumath))
> >>
> >> ==== The following PRs are still waiting for reviews ====
> >>
> >> DRILL-6583: UI usability issue ((Kunal / Sorabh))
> >>
> >> DRILL-6579: Add sanity checks to the Parquet Reader ((Salim / Vlad +
> Boaz))
> >>
> >> DRILL-6578: handle query cancellation in Parquet Reader ((Salim / Vlad
> + Boaz))
> >>
> >> DRILL-6560: Allow options for controlling the batch size per operator
> ((Salim / Karthik))
> >>
> >> DRILL-6559: Travis timing out ((Vitalii / Tim))
> >>
> >> DRILL-6496: VectorUtil.showVectorAccessibleContent does not log vector
> content ((Tim / Volodymyr))
> >>
> >> DRILL-6410: Memory Leak in Parquet Reader during cancellation ((Vlad /
> Parth))
> >>
> >> DRILL-6346: Create an Official Drill Docker Container ((Abhishek / Tim))
> >>
> >> DRILL-6179: Added pcapng-format support ((Vlad / Paul))
> >>
> >> DRILL-5796: Filter pruning for multi rowgroup parquet file ((Jean-Blas
> / Arina))
> >>
> >> DRILL-5365: FileNotFoundException when reading a parquet file ((Tim /
> Vitalii))
> >>
> >> ==================================
> >>
> >> Thanks,
> >>
> >> Boaz
> >>
> >> On 7/6/18 2:51 PM, Pritesh Maker wrote:
> >>> Here is the release 1.14 dashboard (
> https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_secure_Dashboard.jspa-3FselectPageId-3D12332463&d=DwIGaQ&c=cskdkSMqhcnjZxdQVpwTXg&r=7lXQnf0aC8VQ0iMXwVgNHw&m=V0b4-BeuIMwRczzbiSXCgL7Z7f8lrmLBGH1vnSHLjB4&s=HRQU6Q4umbONtN4EqY3ryggJNEyCOghAzICypRJOels&e=
> ) and agile board (
> https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_secure_RapidBoard.jspa-3FrapidView-3D185&d=DwIGaQ&c=cskdkSMqhcnjZxdQVpwTXg&r=7lXQnf0aC8VQ0iMXwVgNHw&m=V0b4-BeuIMwRczzbiSXCgL7Z7f8lrmLBGH1vnSHLjB4&s=GKxSMl97YRnHJu-AL-A-vvRe5SXqw7vdDPDzMzj-Cj4&e=
> )
> >>>
> >>> I believe Volodymyr is targeting DRILL-6422 (Guava update) for 1.15
> release so it shouldn't be blocking the release. So overall, we have 2 open
> bugs, 2 in progress bugs (+2 doc issues), and 12 in review (+1 ready to
> commit).
> >>>
> >>> If the reviewable commits won't be ready soon, can the developers
> please remove the 1.14 fix version for these issues.
> >>>
> >>> Pritesh
> >>>
> >>>
> >>>
> >>>
> >>> On 7/6/18, 11:54 AM, "Boaz Ben-Zvi" <bben-zvi@mapr.com on behalf of
> boaz@mapr.com> wrote:
> >>>
> >>> Current status: There's a blocker, and some work in progress
> that will
> >>> stretch into next week.
> >>> Current detail:
> >>> ==============================
> >>> Open/blocker - DRILL-6453 + DRILL-6517: Two issues - Parquet
> Scanner not setting record num (to zero), and a hang following this failure.
> >>> In Progress - DRILL-6104: Generic Logfile Format Plugin
> >>> PR - DRILL-6422: Update Guava to 23.0 and shade it
> >>> PR - DRILL-5999 (DRILL-6516): Support for EMIT outcome in
> Streaming Agg (I'm reviewing)
> >>> Ready2Commit: DRILL-6519: Add String Distance and
> Phonetic Functions (Arina gave it a +1 ; is it "Ready-To-Commit" or waiting
> for more reviews ?)
> >>> Committed: DRILL-6570: Mentioned as a blocker by Kunal
> (I just merge #1354; the Jira was (mistakenly ?) marked "Resolved" so it
> missed the batch commit).
> >>> Committed: DRILL-5977: predicate pushdown support
> kafkaMsgOffset
> >>> Committed: DRILL-6577: Change Hash-Join default to not
> fallback (into
> >>> pre-1.14 unlimited memory)
> >>> Committed: DRILL-6353: Upgrade Parquet MR dependencies
> >>> Committed: DRILL-6310: limit batch size for hash aggregate
> >>> ===============================
> >>> Thanks,
> >>> Boaz
> >>> On 7/2/18 9:51 PM, Khurram Faraaz wrote:
> >>> > Do we plan to fix this one too, because this is a regression
> from Apache
> >>> > Drill 1.13.0.
> >>> >
> https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_DRILL-2D6453&d=DwIBaQ&c=cskdkSMqhcnjZxdQVpwTXg&r=EqulKDxxEDCX6zbp1AZAa1-iAPQGgCioAqgDp7DE2BU&m=9HpIZKDh_DBcx9yXCI2TtUWum8vRhdQgmw_3ljJQi1M&s=w-1C-HnlUAGRHvDFUusZ78WZrHahDka2r-F-cNb-pZA&e=
> >>> >
> >>> > On Mon, Jul 2, 2018 at 9:33 PM, Kunal Khatua <ku...@apache.org>
> wrote:
> >>> >
> >>> >> DRILL-6570 seems like a must-have (release blocker, IMHO).
> >>> >> On 7/2/2018 8:02:00 PM, Boaz Ben-Zvi <bo...@mapr.com> wrote:
> >>> >> Let's try to make progress on the 1.14 release, aiming for a
> Release
> >>> >> Candidate towards the end of this week (a little ambitious,
> with the
> >>> >> July 4th and people on vacations).
> >>> >>
> >>> >> Current Status of the previously requested Jiras:
> >>> >>
> >>> >> ==============================
> >>> >>
> >>> >> In Progress - DRILL-6104: Generic Logfile Format Plugin
> >>> >>
> >>> >> PR - DRILL-6422: Update Guava to 23.0 and shade it
> >>> >>
> >>> >> PR - DRILL-5999 (DRILL-6516): Support for EMIT outcome in
> Streaming Agg
> >>> >>
> >>> >> Ready2Commit: DRILL-5977: predicate pushdown support
> kafkaMsgOffset
> >>> >>
> >>> >> Ready2Commit: DRILL-6519: Add String Distance and Phonetic
> Functions
> >>> >>
> >>> >> Ready2Commit: DRILL-6577: Change Hash-Join default to not
> fallback (into
> >>> >> pre-1.14 unlimited memory)
> >>> >>
> >>> >> Committed: DRILL-6353: Upgrade Parquet MR dependencies
> >>> >>
> >>> >> Committed: DRILL-6310: limit batch size for hash aggregate
> >>> >>
> >>> >> ===============================
> >>> >>
> >>> >> And there are few more open or in a PR state.
> >>> >>
> >>> >> Lets try and most of these ready by the end of the week.
> >>> >>
> >>> >> Boaz
> >>> >>
> >>> >>
> >>> >>
> >>>
>
>
Re: [DISCUSS] 1.14.0 release
Posted by Boaz Ben-Zvi <bo...@mapr.com>.
Hi Charles,
The main reason for rushing a Release Candidate is that we could
give it enough testing.
Given that DRILL-6104 is a separate feature, with almost no impact on
the current code, then it seems low risk to add it a few days later.
Anyone has an objection ?
Boaz
On 7/9/18 9:54 PM, Charles Givre wrote:
> Hi Boaz,
> I’m traveling at the moment, but I can have DRILL-6104 back in Paul’s hands by the end of the week.
> —C
>
>> On Jul 10, 2018, at 00:53, Boaz Ben-Zvi <bo...@mapr.com> wrote:
>>
>> We are making progress towards 1.14.
>>
>> Let's aim for a Release Candidate branch off on Thursday (July 12) !!!
>>
>> Below are the unfinished cases; can most be completed and checked in by 7/12 ?
>>
>> (( Relevant people:
>>
>> Abhishek, Arina, Boaz, Charles, Hanumath, Jean-Blas, Karthik, Kunal,
>>
>> Parth, Paul, Salim, Sorabh, Tim, Vitalii, Vlad, Volodymyr ))
>>
>> ======================
>>
>> Open/blocker - DRILL-6453 + DRILL-6517:
>> Two issues - Parquet Scanner (?) not setting container's record num (to zero), and a hang following this failure.
>> Currently testing a fix / workaround ((Boaz))
>>
>> In Progress - DRILL-6104: Generic Logfile Format Plugin ((Charles + Paul -- can you be done by 7/12 ?))
>>
>> PR - DRILL-5999 (DRILL-6516): Support for EMIT outcome in Streaming Agg ((Parth + Boaz reviewing))
>>
>> Open - DRILL-6542: Index out of bounds ((Sorabh))
>>
>> Open - DRILL-6475: Unnest Null fieldId pointer ((Hanumath))
>>
>> ==== The following PRs are still waiting for reviews ====
>>
>> DRILL-6583: UI usability issue ((Kunal / Sorabh))
>>
>> DRILL-6579: Add sanity checks to the Parquet Reader ((Salim / Vlad + Boaz))
>>
>> DRILL-6578: handle query cancellation in Parquet Reader ((Salim / Vlad + Boaz))
>>
>> DRILL-6560: Allow options for controlling the batch size per operator ((Salim / Karthik))
>>
>> DRILL-6559: Travis timing out ((Vitalii / Tim))
>>
>> DRILL-6496: VectorUtil.showVectorAccessibleContent does not log vector content ((Tim / Volodymyr))
>>
>> DRILL-6410: Memory Leak in Parquet Reader during cancellation ((Vlad / Parth))
>>
>> DRILL-6346: Create an Official Drill Docker Container ((Abhishek / Tim))
>>
>> DRILL-6179: Added pcapng-format support ((Vlad / Paul))
>>
>> DRILL-5796: Filter pruning for multi rowgroup parquet file ((Jean-Blas / Arina))
>>
>> DRILL-5365: FileNotFoundException when reading a parquet file ((Tim / Vitalii))
>>
>> ==================================
>>
>> Thanks,
>>
>> Boaz
>>
>> On 7/6/18 2:51 PM, Pritesh Maker wrote:
>>> Here is the release 1.14 dashboard (https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_secure_Dashboard.jspa-3FselectPageId-3D12332463&d=DwIGaQ&c=cskdkSMqhcnjZxdQVpwTXg&r=7lXQnf0aC8VQ0iMXwVgNHw&m=V0b4-BeuIMwRczzbiSXCgL7Z7f8lrmLBGH1vnSHLjB4&s=HRQU6Q4umbONtN4EqY3ryggJNEyCOghAzICypRJOels&e= ) and agile board (https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_secure_RapidBoard.jspa-3FrapidView-3D185&d=DwIGaQ&c=cskdkSMqhcnjZxdQVpwTXg&r=7lXQnf0aC8VQ0iMXwVgNHw&m=V0b4-BeuIMwRczzbiSXCgL7Z7f8lrmLBGH1vnSHLjB4&s=GKxSMl97YRnHJu-AL-A-vvRe5SXqw7vdDPDzMzj-Cj4&e=)
>>>
>>> I believe Volodymyr is targeting DRILL-6422 (Guava update) for 1.15 release so it shouldn't be blocking the release. So overall, we have 2 open bugs, 2 in progress bugs (+2 doc issues), and 12 in review (+1 ready to commit).
>>>
>>> If the reviewable commits won't be ready soon, can the developers please remove the 1.14 fix version for these issues.
>>>
>>> Pritesh
>>>
>>>
>>>
>>>
>>> On 7/6/18, 11:54 AM, "Boaz Ben-Zvi" <bben-zvi@mapr.com on behalf of boaz@mapr.com> wrote:
>>>
>>> Current status: There's a blocker, and some work in progress that will
>>> stretch into next week.
>>> Current detail:
>>> ==============================
>>> Open/blocker - DRILL-6453 + DRILL-6517: Two issues - Parquet Scanner not setting record num (to zero), and a hang following this failure.
>>> In Progress - DRILL-6104: Generic Logfile Format Plugin
>>> PR - DRILL-6422: Update Guava to 23.0 and shade it
>>> PR - DRILL-5999 (DRILL-6516): Support for EMIT outcome in Streaming Agg (I'm reviewing)
>>> Ready2Commit: DRILL-6519: Add String Distance and Phonetic Functions (Arina gave it a +1 ; is it "Ready-To-Commit" or waiting for more reviews ?)
>>> Committed: DRILL-6570: Mentioned as a blocker by Kunal (I just merge #1354; the Jira was (mistakenly ?) marked "Resolved" so it missed the batch commit).
>>> Committed: DRILL-5977: predicate pushdown support kafkaMsgOffset
>>> Committed: DRILL-6577: Change Hash-Join default to not fallback (into
>>> pre-1.14 unlimited memory)
>>> Committed: DRILL-6353: Upgrade Parquet MR dependencies
>>> Committed: DRILL-6310: limit batch size for hash aggregate
>>> ===============================
>>> Thanks,
>>> Boaz
>>> On 7/2/18 9:51 PM, Khurram Faraaz wrote:
>>> > Do we plan to fix this one too, because this is a regression from Apache
>>> > Drill 1.13.0.
>>> > https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_DRILL-2D6453&d=DwIBaQ&c=cskdkSMqhcnjZxdQVpwTXg&r=EqulKDxxEDCX6zbp1AZAa1-iAPQGgCioAqgDp7DE2BU&m=9HpIZKDh_DBcx9yXCI2TtUWum8vRhdQgmw_3ljJQi1M&s=w-1C-HnlUAGRHvDFUusZ78WZrHahDka2r-F-cNb-pZA&e=
>>> >
>>> > On Mon, Jul 2, 2018 at 9:33 PM, Kunal Khatua <ku...@apache.org> wrote:
>>> >
>>> >> DRILL-6570 seems like a must-have (release blocker, IMHO).
>>> >> On 7/2/2018 8:02:00 PM, Boaz Ben-Zvi <bo...@mapr.com> wrote:
>>> >> Let's try to make progress on the 1.14 release, aiming for a Release
>>> >> Candidate towards the end of this week (a little ambitious, with the
>>> >> July 4th and people on vacations).
>>> >>
>>> >> Current Status of the previously requested Jiras:
>>> >>
>>> >> ==============================
>>> >>
>>> >> In Progress - DRILL-6104: Generic Logfile Format Plugin
>>> >>
>>> >> PR - DRILL-6422: Update Guava to 23.0 and shade it
>>> >>
>>> >> PR - DRILL-5999 (DRILL-6516): Support for EMIT outcome in Streaming Agg
>>> >>
>>> >> Ready2Commit: DRILL-5977: predicate pushdown support kafkaMsgOffset
>>> >>
>>> >> Ready2Commit: DRILL-6519: Add String Distance and Phonetic Functions
>>> >>
>>> >> Ready2Commit: DRILL-6577: Change Hash-Join default to not fallback (into
>>> >> pre-1.14 unlimited memory)
>>> >>
>>> >> Committed: DRILL-6353: Upgrade Parquet MR dependencies
>>> >>
>>> >> Committed: DRILL-6310: limit batch size for hash aggregate
>>> >>
>>> >> ===============================
>>> >>
>>> >> And there are few more open or in a PR state.
>>> >>
>>> >> Lets try and most of these ready by the end of the week.
>>> >>
>>> >> Boaz
>>> >>
>>> >>
>>> >>
>>>
Re: [DISCUSS] 1.14.0 release
Posted by Charles Givre <cg...@gmail.com>.
Hi Boaz,
I’m traveling at the moment, but I can have DRILL-6104 back in Paul’s hands by the end of the week.
—C
> On Jul 10, 2018, at 00:53, Boaz Ben-Zvi <bo...@mapr.com> wrote:
>
> We are making progress towards 1.14.
>
> Let's aim for a Release Candidate branch off on Thursday (July 12) !!!
>
> Below are the unfinished cases; can most be completed and checked in by 7/12 ?
>
> (( Relevant people:
>
> Abhishek, Arina, Boaz, Charles, Hanumath, Jean-Blas, Karthik, Kunal,
>
> Parth, Paul, Salim, Sorabh, Tim, Vitalii, Vlad, Volodymyr ))
>
> ======================
>
> Open/blocker - DRILL-6453 + DRILL-6517:
> Two issues - Parquet Scanner (?) not setting container's record num (to zero), and a hang following this failure.
> Currently testing a fix / workaround ((Boaz))
>
> In Progress - DRILL-6104: Generic Logfile Format Plugin ((Charles + Paul -- can you be done by 7/12 ?))
>
> PR - DRILL-5999 (DRILL-6516): Support for EMIT outcome in Streaming Agg ((Parth + Boaz reviewing))
>
> Open - DRILL-6542: Index out of bounds ((Sorabh))
>
> Open - DRILL-6475: Unnest Null fieldId pointer ((Hanumath))
>
> ==== The following PRs are still waiting for reviews ====
>
> DRILL-6583: UI usability issue ((Kunal / Sorabh))
>
> DRILL-6579: Add sanity checks to the Parquet Reader ((Salim / Vlad + Boaz))
>
> DRILL-6578: handle query cancellation in Parquet Reader ((Salim / Vlad + Boaz))
>
> DRILL-6560: Allow options for controlling the batch size per operator ((Salim / Karthik))
>
> DRILL-6559: Travis timing out ((Vitalii / Tim))
>
> DRILL-6496: VectorUtil.showVectorAccessibleContent does not log vector content ((Tim / Volodymyr))
>
> DRILL-6410: Memory Leak in Parquet Reader during cancellation ((Vlad / Parth))
>
> DRILL-6346: Create an Official Drill Docker Container ((Abhishek / Tim))
>
> DRILL-6179: Added pcapng-format support ((Vlad / Paul))
>
> DRILL-5796: Filter pruning for multi rowgroup parquet file ((Jean-Blas / Arina))
>
> DRILL-5365: FileNotFoundException when reading a parquet file ((Tim / Vitalii))
>
> ==================================
>
> Thanks,
>
> Boaz
>
> On 7/6/18 2:51 PM, Pritesh Maker wrote:
>> Here is the release 1.14 dashboard (https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_secure_Dashboard.jspa-3FselectPageId-3D12332463&d=DwIGaQ&c=cskdkSMqhcnjZxdQVpwTXg&r=7lXQnf0aC8VQ0iMXwVgNHw&m=V0b4-BeuIMwRczzbiSXCgL7Z7f8lrmLBGH1vnSHLjB4&s=HRQU6Q4umbONtN4EqY3ryggJNEyCOghAzICypRJOels&e= ) and agile board (https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_secure_RapidBoard.jspa-3FrapidView-3D185&d=DwIGaQ&c=cskdkSMqhcnjZxdQVpwTXg&r=7lXQnf0aC8VQ0iMXwVgNHw&m=V0b4-BeuIMwRczzbiSXCgL7Z7f8lrmLBGH1vnSHLjB4&s=GKxSMl97YRnHJu-AL-A-vvRe5SXqw7vdDPDzMzj-Cj4&e=)
>>
>> I believe Volodymyr is targeting DRILL-6422 (Guava update) for 1.15 release so it shouldn't be blocking the release. So overall, we have 2 open bugs, 2 in progress bugs (+2 doc issues), and 12 in review (+1 ready to commit).
>>
>> If the reviewable commits won't be ready soon, can the developers please remove the 1.14 fix version for these issues.
>>
>> Pritesh
>>
>>
>>
>>
>> On 7/6/18, 11:54 AM, "Boaz Ben-Zvi" <bben-zvi@mapr.com on behalf of boaz@mapr.com> wrote:
>>
>> Current status: There's a blocker, and some work in progress that will
>> stretch into next week.
>> Current detail:
>> ==============================
>> Open/blocker - DRILL-6453 + DRILL-6517: Two issues - Parquet Scanner not setting record num (to zero), and a hang following this failure.
>> In Progress - DRILL-6104: Generic Logfile Format Plugin
>> PR - DRILL-6422: Update Guava to 23.0 and shade it
>> PR - DRILL-5999 (DRILL-6516): Support for EMIT outcome in Streaming Agg (I'm reviewing)
>> Ready2Commit: DRILL-6519: Add String Distance and Phonetic Functions (Arina gave it a +1 ; is it "Ready-To-Commit" or waiting for more reviews ?)
>> Committed: DRILL-6570: Mentioned as a blocker by Kunal (I just merge #1354; the Jira was (mistakenly ?) marked "Resolved" so it missed the batch commit).
>> Committed: DRILL-5977: predicate pushdown support kafkaMsgOffset
>> Committed: DRILL-6577: Change Hash-Join default to not fallback (into
>> pre-1.14 unlimited memory)
>> Committed: DRILL-6353: Upgrade Parquet MR dependencies
>> Committed: DRILL-6310: limit batch size for hash aggregate
>> ===============================
>> Thanks,
>> Boaz
>> On 7/2/18 9:51 PM, Khurram Faraaz wrote:
>> > Do we plan to fix this one too, because this is a regression from Apache
>> > Drill 1.13.0.
>> > https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_DRILL-2D6453&d=DwIBaQ&c=cskdkSMqhcnjZxdQVpwTXg&r=EqulKDxxEDCX6zbp1AZAa1-iAPQGgCioAqgDp7DE2BU&m=9HpIZKDh_DBcx9yXCI2TtUWum8vRhdQgmw_3ljJQi1M&s=w-1C-HnlUAGRHvDFUusZ78WZrHahDka2r-F-cNb-pZA&e=
>> >
>> > On Mon, Jul 2, 2018 at 9:33 PM, Kunal Khatua <ku...@apache.org> wrote:
>> >
>> >> DRILL-6570 seems like a must-have (release blocker, IMHO).
>> >> On 7/2/2018 8:02:00 PM, Boaz Ben-Zvi <bo...@mapr.com> wrote:
>> >> Let's try to make progress on the 1.14 release, aiming for a Release
>> >> Candidate towards the end of this week (a little ambitious, with the
>> >> July 4th and people on vacations).
>> >>
>> >> Current Status of the previously requested Jiras:
>> >>
>> >> ==============================
>> >>
>> >> In Progress - DRILL-6104: Generic Logfile Format Plugin
>> >>
>> >> PR - DRILL-6422: Update Guava to 23.0 and shade it
>> >>
>> >> PR - DRILL-5999 (DRILL-6516): Support for EMIT outcome in Streaming Agg
>> >>
>> >> Ready2Commit: DRILL-5977: predicate pushdown support kafkaMsgOffset
>> >>
>> >> Ready2Commit: DRILL-6519: Add String Distance and Phonetic Functions
>> >>
>> >> Ready2Commit: DRILL-6577: Change Hash-Join default to not fallback (into
>> >> pre-1.14 unlimited memory)
>> >>
>> >> Committed: DRILL-6353: Upgrade Parquet MR dependencies
>> >>
>> >> Committed: DRILL-6310: limit batch size for hash aggregate
>> >>
>> >> ===============================
>> >>
>> >> And there are few more open or in a PR state.
>> >>
>> >> Lets try and most of these ready by the end of the week.
>> >>
>> >> Boaz
>> >>
>> >>
>> >>
>>
>
Re: [DISCUSS] 1.14.0 release
Posted by Timothy Farkas <tf...@mapr.com>.
Hi Boaz,
I looked at DRILL-6606 and have updated the ticket, I should have a fix
monday. It looks like a minor logical error.
I'm not clear on why you suspect DRILL-6453 is cause by batch sniffing,
perhaps we can discuss offline.
Thanks,
Tim
On Fri, Jul 13, 2018 at 2:01 PM, Arina Yelchiyeva <
arina.yelchiyeva@gmail.com> wrote:
> Two more regressions:
> https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.
> apache.org_jira_browse_DRILL-2D6603&d=DwIBaQ&c=cskdkSMqhcnjZxdQVpwTXg&r=
> 4eQVr8zB8ZBff-yxTimdOQ&m=F7Ih5Ah_SfOS5fZXFApt88iMe3Vd-
> Jq1XDvxPN6b3y4&s=FqDR26vK2kVG-P69NcqoqNxRrdHKZvBCWamYRftPPYg&e=
> https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.
> apache.org_jira_browse_DRILL-2D6605&d=DwIBaQ&c=cskdkSMqhcnjZxdQVpwTXg&r=
> 4eQVr8zB8ZBff-yxTimdOQ&m=F7Ih5Ah_SfOS5fZXFApt88iMe3Vd-Jq1XDvxPN6b3y4&s=
> KzhJcDP4F8FIA7h4GiQE8wg_CSae0CAI0yEMslff52o&e=
>
> Kind regards,
> Arina
>
> On Fri, Jul 13, 2018 at 11:25 PM Sorabh Hamirwasia <sh...@mapr.com>
> wrote:
>
> > Hi Boaz,
> > Couple of updates.
> >
> > *Merged In:*
> > DRILL-6542: (May be Ready2Commit soon) IndexOutOfBounds exception for
> > multilevel lateral ((Sorabh / Parth))
> >
> > *In Review:*
> >
> >
> > *DRILL-6475: Query with UNNEST causes a Null Pointer . (( Hanumath ))*
> > Thanks,
> > Sorabh
> >
> > On Fri, Jul 13, 2018 at 1:17 PM, Parth Chandra <pa...@apache.org>
> wrote:
> >
> > > Our (unwritten) rule has been that a commit cannot even go in unless
> unit
> > > _and_ regression tests pass.
> > > Releases are stricter, all tests, longevity tests, UI, are required to
> > > pass. In addition, any performance regression needs to be discussed.
> > >
> > > So far we have not made any exceptions, but that is not to say we
> cannot.
> > >
> > > On Fri, Jul 13, 2018 at 1:03 PM, Vlad Rozov <vr...@apache.org> wrote:
> > >
> > > > My 2 cents:
> > > >
> > > > From Apache point of view it is OK to do a release even if unit tests
> > do
> > > > not pass at all or there is a large number of regression introduced.
> > > Apache
> > > > release is a source release and as long as it compiles and does not
> > have
> > > > license issues, it is up to community (PMC) to decide on any other
> > > criteria
> > > > for a release.
> > > >
> > > > The issue in DRILL-6453 is not limited to a large number of hash
> joins.
> > > It
> > > > should be possible to reproduce it even with a single hash join as
> long
> > > as
> > > > left and right sides are getting batches from one(many) to many
> > exchanges
> > > > (broadcast or hash partitioner senders).
> > > >
> > > > Thank you,
> > > >
> > > > Vlad
> > > >
> > > >
> > > > On 7/13/18 08:41, Aman Sinha wrote:
> > > >
> > > >> I would say we have to take a measured approach to this and decide
> on
> > a
> > > >> case-by-case which issue is a show stopper.
> > > >> While of course we have to make every effort to avoid regression, we
> > > >> cannot
> > > >> claim that a particular release will not cause any regression.
> > > >> I believe there are 10000+ passing tests, so that should provide a
> > > level
> > > >> of confidence. The TPC-DS 72 is a 10 table join which in the
> hadoop
> > > >> world
> > > >> of
> > > >> denormalized schemas is not relatively common. The main question is
> > > does
> > > >> the issue reproduce with fewer joins having the same type of
> > > distribution
> > > >> plan ?
> > > >>
> > > >>
> > > >> Aman
> > > >>
> > > >> On Fri, Jul 13, 2018 at 7:36 AM Arina Yelchiyeva <
> > > >> arina.yelchiyeva@gmail.com>
> > > >> wrote:
> > > >>
> > > >> We cannot release with existing regressions, especially taking into
> > > >>> account
> > > >>> the there are not minor issues.
> > > >>> As far as I understand reverting is not an option since hash join
> > spill
> > > >>> feature are extended into several commits + subsequent fixes.
> > > >>> I guess we need to consider postponing the release until issues are
> > > >>> resolved.
> > > >>>
> > > >>> Kind regards,
> > > >>> Arina
> > > >>>
> > > >>> On Fri, Jul 13, 2018 at 5:14 PM Boaz Ben-Zvi <bo...@apache.org>
> > wrote:
> > > >>>
> > > >>> (Guessing ...) It is possible that the root cause for DRILL-6606 is
> > > >>>> similar to that in DRILL-6453 -- that is the new "early sniffing"
> > in
> > > >>>> the
> > > >>>> Hash-Join, which repeatedly invokes next() on the two "children"
> of
> > > the
> > > >>>> join *during schema discovery* until non-empty data is returned
> (or
> > > >>>> NONE,
> > > >>>> STOP, etc). Last night Salim, Vlad and I briefly discussed
> > > >>>> alternatives,
> > > >>>> like postponing the "sniffing" to a later time (beginning of the
> > build
> > > >>>>
> > > >>> for
> > > >>>
> > > >>>> the right child, and beginning of the probe for the left child).
> > > >>>>
> > > >>>> However this would require some work time. So what should we do
> > about
> > > >>>>
> > > >>> 1.14
> > > >>>
> > > >>>> ?
> > > >>>>
> > > >>>> Thanks,
> > > >>>>
> > > >>>> Boaz
> > > >>>>
> > > >>>> On Fri, Jul 13, 2018 at 3:46 AM, Arina Yelchiyeva <
> > > >>>> arina.yelchiyeva@gmail.com> wrote:
> > > >>>>
> > > >>>> During implementing late limit 0 optimization, Bohdan has found
> one
> > > more
> > > >>>>> regression after Hash Join spill to disk.
> > > >>>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.
> > > apache.org_jira_browse_DRILL-2D6606&d=DwIBaQ&c=
> cskdkSMqhcnjZxdQVpwTXg&r=
> > > gRpEl0WzXE3EMrwj0KFbZXGXRyadOthF2jlYxvhTlQg&m=TGqnVoxNweQMiHTgP4J-
> > > rOnguFThVKShnQqHE_CmySI&s=aHUGrClgE_9UsRpRlNM95TbW91ivkqGdF1hV0EDc
> 3xU&e=
> > > >>>>> <
> > > >>>>>
> > > >>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.
> > > >>> apache.org_jira_browse_DRILL-2D6606&d=DwMFaQ&c=cskdkSMqhcnjZ
> > > >>> xdQVpwTXg&r=7lXQnf0aC8VQ0iMXwVgNHw&m=OHnyHeZpNk3hcwkG-JoQG6E
> > > >>> 90tKdoS47J1rv5x-hJzw&s=wm5zpJf9K2zYzrqRB1LqLpKcvmBK5y
> 6XC0ZUqVmSjko&e=
> > > >>>
> > > >>>> Boaz please take a look.
> > > >>>>>
> > > >>>>> Kind regards,
> > > >>>>> Arina
> > > >>>>>
> > > >>>>>
> > > >>>>
> > > >
> > >
> >
>
Re: [DISCUSS] 1.14.0 release
Posted by Arina Yelchiyeva <ar...@gmail.com>.
Two more regressions:
https://issues.apache.org/jira/browse/DRILL-6603
https://issues.apache.org/jira/browse/DRILL-6605
Kind regards,
Arina
On Fri, Jul 13, 2018 at 11:25 PM Sorabh Hamirwasia <sh...@mapr.com>
wrote:
> Hi Boaz,
> Couple of updates.
>
> *Merged In:*
> DRILL-6542: (May be Ready2Commit soon) IndexOutOfBounds exception for
> multilevel lateral ((Sorabh / Parth))
>
> *In Review:*
>
>
> *DRILL-6475: Query with UNNEST causes a Null Pointer . (( Hanumath ))*
> Thanks,
> Sorabh
>
> On Fri, Jul 13, 2018 at 1:17 PM, Parth Chandra <pa...@apache.org> wrote:
>
> > Our (unwritten) rule has been that a commit cannot even go in unless unit
> > _and_ regression tests pass.
> > Releases are stricter, all tests, longevity tests, UI, are required to
> > pass. In addition, any performance regression needs to be discussed.
> >
> > So far we have not made any exceptions, but that is not to say we cannot.
> >
> > On Fri, Jul 13, 2018 at 1:03 PM, Vlad Rozov <vr...@apache.org> wrote:
> >
> > > My 2 cents:
> > >
> > > From Apache point of view it is OK to do a release even if unit tests
> do
> > > not pass at all or there is a large number of regression introduced.
> > Apache
> > > release is a source release and as long as it compiles and does not
> have
> > > license issues, it is up to community (PMC) to decide on any other
> > criteria
> > > for a release.
> > >
> > > The issue in DRILL-6453 is not limited to a large number of hash joins.
> > It
> > > should be possible to reproduce it even with a single hash join as long
> > as
> > > left and right sides are getting batches from one(many) to many
> exchanges
> > > (broadcast or hash partitioner senders).
> > >
> > > Thank you,
> > >
> > > Vlad
> > >
> > >
> > > On 7/13/18 08:41, Aman Sinha wrote:
> > >
> > >> I would say we have to take a measured approach to this and decide on
> a
> > >> case-by-case which issue is a show stopper.
> > >> While of course we have to make every effort to avoid regression, we
> > >> cannot
> > >> claim that a particular release will not cause any regression.
> > >> I believe there are 10000+ passing tests, so that should provide a
> > level
> > >> of confidence. The TPC-DS 72 is a 10 table join which in the hadoop
> > >> world
> > >> of
> > >> denormalized schemas is not relatively common. The main question is
> > does
> > >> the issue reproduce with fewer joins having the same type of
> > distribution
> > >> plan ?
> > >>
> > >>
> > >> Aman
> > >>
> > >> On Fri, Jul 13, 2018 at 7:36 AM Arina Yelchiyeva <
> > >> arina.yelchiyeva@gmail.com>
> > >> wrote:
> > >>
> > >> We cannot release with existing regressions, especially taking into
> > >>> account
> > >>> the there are not minor issues.
> > >>> As far as I understand reverting is not an option since hash join
> spill
> > >>> feature are extended into several commits + subsequent fixes.
> > >>> I guess we need to consider postponing the release until issues are
> > >>> resolved.
> > >>>
> > >>> Kind regards,
> > >>> Arina
> > >>>
> > >>> On Fri, Jul 13, 2018 at 5:14 PM Boaz Ben-Zvi <bo...@apache.org>
> wrote:
> > >>>
> > >>> (Guessing ...) It is possible that the root cause for DRILL-6606 is
> > >>>> similar to that in DRILL-6453 -- that is the new "early sniffing"
> in
> > >>>> the
> > >>>> Hash-Join, which repeatedly invokes next() on the two "children" of
> > the
> > >>>> join *during schema discovery* until non-empty data is returned (or
> > >>>> NONE,
> > >>>> STOP, etc). Last night Salim, Vlad and I briefly discussed
> > >>>> alternatives,
> > >>>> like postponing the "sniffing" to a later time (beginning of the
> build
> > >>>>
> > >>> for
> > >>>
> > >>>> the right child, and beginning of the probe for the left child).
> > >>>>
> > >>>> However this would require some work time. So what should we do
> about
> > >>>>
> > >>> 1.14
> > >>>
> > >>>> ?
> > >>>>
> > >>>> Thanks,
> > >>>>
> > >>>> Boaz
> > >>>>
> > >>>> On Fri, Jul 13, 2018 at 3:46 AM, Arina Yelchiyeva <
> > >>>> arina.yelchiyeva@gmail.com> wrote:
> > >>>>
> > >>>> During implementing late limit 0 optimization, Bohdan has found one
> > more
> > >>>>> regression after Hash Join spill to disk.
> > >>>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.
> > apache.org_jira_browse_DRILL-2D6606&d=DwIBaQ&c=cskdkSMqhcnjZxdQVpwTXg&r=
> > gRpEl0WzXE3EMrwj0KFbZXGXRyadOthF2jlYxvhTlQg&m=TGqnVoxNweQMiHTgP4J-
> > rOnguFThVKShnQqHE_CmySI&s=aHUGrClgE_9UsRpRlNM95TbW91ivkqGdF1hV0EDc3xU&e=
> > >>>>> <
> > >>>>>
> > >>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.
> > >>> apache.org_jira_browse_DRILL-2D6606&d=DwMFaQ&c=cskdkSMqhcnjZ
> > >>> xdQVpwTXg&r=7lXQnf0aC8VQ0iMXwVgNHw&m=OHnyHeZpNk3hcwkG-JoQG6E
> > >>> 90tKdoS47J1rv5x-hJzw&s=wm5zpJf9K2zYzrqRB1LqLpKcvmBK5y6XC0ZUqVmSjko&e=
> > >>>
> > >>>> Boaz please take a look.
> > >>>>>
> > >>>>> Kind regards,
> > >>>>> Arina
> > >>>>>
> > >>>>>
> > >>>>
> > >
> >
>
Re: [DISCUSS] 1.14.0 release
Posted by Sorabh Hamirwasia <sh...@mapr.com>.
Hi Boaz,
Couple of updates.
*Merged In:*
DRILL-6542: (May be Ready2Commit soon) IndexOutOfBounds exception for
multilevel lateral ((Sorabh / Parth))
*In Review:*
*DRILL-6475: Query with UNNEST causes a Null Pointer . (( Hanumath ))*
Thanks,
Sorabh
On Fri, Jul 13, 2018 at 1:17 PM, Parth Chandra <pa...@apache.org> wrote:
> Our (unwritten) rule has been that a commit cannot even go in unless unit
> _and_ regression tests pass.
> Releases are stricter, all tests, longevity tests, UI, are required to
> pass. In addition, any performance regression needs to be discussed.
>
> So far we have not made any exceptions, but that is not to say we cannot.
>
> On Fri, Jul 13, 2018 at 1:03 PM, Vlad Rozov <vr...@apache.org> wrote:
>
> > My 2 cents:
> >
> > From Apache point of view it is OK to do a release even if unit tests do
> > not pass at all or there is a large number of regression introduced.
> Apache
> > release is a source release and as long as it compiles and does not have
> > license issues, it is up to community (PMC) to decide on any other
> criteria
> > for a release.
> >
> > The issue in DRILL-6453 is not limited to a large number of hash joins.
> It
> > should be possible to reproduce it even with a single hash join as long
> as
> > left and right sides are getting batches from one(many) to many exchanges
> > (broadcast or hash partitioner senders).
> >
> > Thank you,
> >
> > Vlad
> >
> >
> > On 7/13/18 08:41, Aman Sinha wrote:
> >
> >> I would say we have to take a measured approach to this and decide on a
> >> case-by-case which issue is a show stopper.
> >> While of course we have to make every effort to avoid regression, we
> >> cannot
> >> claim that a particular release will not cause any regression.
> >> I believe there are 10000+ passing tests, so that should provide a
> level
> >> of confidence. The TPC-DS 72 is a 10 table join which in the hadoop
> >> world
> >> of
> >> denormalized schemas is not relatively common. The main question is
> does
> >> the issue reproduce with fewer joins having the same type of
> distribution
> >> plan ?
> >>
> >>
> >> Aman
> >>
> >> On Fri, Jul 13, 2018 at 7:36 AM Arina Yelchiyeva <
> >> arina.yelchiyeva@gmail.com>
> >> wrote:
> >>
> >> We cannot release with existing regressions, especially taking into
> >>> account
> >>> the there are not minor issues.
> >>> As far as I understand reverting is not an option since hash join spill
> >>> feature are extended into several commits + subsequent fixes.
> >>> I guess we need to consider postponing the release until issues are
> >>> resolved.
> >>>
> >>> Kind regards,
> >>> Arina
> >>>
> >>> On Fri, Jul 13, 2018 at 5:14 PM Boaz Ben-Zvi <bo...@apache.org> wrote:
> >>>
> >>> (Guessing ...) It is possible that the root cause for DRILL-6606 is
> >>>> similar to that in DRILL-6453 -- that is the new "early sniffing" in
> >>>> the
> >>>> Hash-Join, which repeatedly invokes next() on the two "children" of
> the
> >>>> join *during schema discovery* until non-empty data is returned (or
> >>>> NONE,
> >>>> STOP, etc). Last night Salim, Vlad and I briefly discussed
> >>>> alternatives,
> >>>> like postponing the "sniffing" to a later time (beginning of the build
> >>>>
> >>> for
> >>>
> >>>> the right child, and beginning of the probe for the left child).
> >>>>
> >>>> However this would require some work time. So what should we do about
> >>>>
> >>> 1.14
> >>>
> >>>> ?
> >>>>
> >>>> Thanks,
> >>>>
> >>>> Boaz
> >>>>
> >>>> On Fri, Jul 13, 2018 at 3:46 AM, Arina Yelchiyeva <
> >>>> arina.yelchiyeva@gmail.com> wrote:
> >>>>
> >>>> During implementing late limit 0 optimization, Bohdan has found one
> more
> >>>>> regression after Hash Join spill to disk.
> >>>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.
> apache.org_jira_browse_DRILL-2D6606&d=DwIBaQ&c=cskdkSMqhcnjZxdQVpwTXg&r=
> gRpEl0WzXE3EMrwj0KFbZXGXRyadOthF2jlYxvhTlQg&m=TGqnVoxNweQMiHTgP4J-
> rOnguFThVKShnQqHE_CmySI&s=aHUGrClgE_9UsRpRlNM95TbW91ivkqGdF1hV0EDc3xU&e=
> >>>>> <
> >>>>>
> >>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.
> >>> apache.org_jira_browse_DRILL-2D6606&d=DwMFaQ&c=cskdkSMqhcnjZ
> >>> xdQVpwTXg&r=7lXQnf0aC8VQ0iMXwVgNHw&m=OHnyHeZpNk3hcwkG-JoQG6E
> >>> 90tKdoS47J1rv5x-hJzw&s=wm5zpJf9K2zYzrqRB1LqLpKcvmBK5y6XC0ZUqVmSjko&e=
> >>>
> >>>> Boaz please take a look.
> >>>>>
> >>>>> Kind regards,
> >>>>> Arina
> >>>>>
> >>>>>
> >>>>
> >
>
Re: [DISCUSS] 1.14.0 release
Posted by Parth Chandra <pa...@apache.org>.
Our (unwritten) rule has been that a commit cannot even go in unless unit
_and_ regression tests pass.
Releases are stricter, all tests, longevity tests, UI, are required to
pass. In addition, any performance regression needs to be discussed.
So far we have not made any exceptions, but that is not to say we cannot.
On Fri, Jul 13, 2018 at 1:03 PM, Vlad Rozov <vr...@apache.org> wrote:
> My 2 cents:
>
> From Apache point of view it is OK to do a release even if unit tests do
> not pass at all or there is a large number of regression introduced. Apache
> release is a source release and as long as it compiles and does not have
> license issues, it is up to community (PMC) to decide on any other criteria
> for a release.
>
> The issue in DRILL-6453 is not limited to a large number of hash joins. It
> should be possible to reproduce it even with a single hash join as long as
> left and right sides are getting batches from one(many) to many exchanges
> (broadcast or hash partitioner senders).
>
> Thank you,
>
> Vlad
>
>
> On 7/13/18 08:41, Aman Sinha wrote:
>
>> I would say we have to take a measured approach to this and decide on a
>> case-by-case which issue is a show stopper.
>> While of course we have to make every effort to avoid regression, we
>> cannot
>> claim that a particular release will not cause any regression.
>> I believe there are 10000+ passing tests, so that should provide a level
>> of confidence. The TPC-DS 72 is a 10 table join which in the hadoop
>> world
>> of
>> denormalized schemas is not relatively common. The main question is does
>> the issue reproduce with fewer joins having the same type of distribution
>> plan ?
>>
>>
>> Aman
>>
>> On Fri, Jul 13, 2018 at 7:36 AM Arina Yelchiyeva <
>> arina.yelchiyeva@gmail.com>
>> wrote:
>>
>> We cannot release with existing regressions, especially taking into
>>> account
>>> the there are not minor issues.
>>> As far as I understand reverting is not an option since hash join spill
>>> feature are extended into several commits + subsequent fixes.
>>> I guess we need to consider postponing the release until issues are
>>> resolved.
>>>
>>> Kind regards,
>>> Arina
>>>
>>> On Fri, Jul 13, 2018 at 5:14 PM Boaz Ben-Zvi <bo...@apache.org> wrote:
>>>
>>> (Guessing ...) It is possible that the root cause for DRILL-6606 is
>>>> similar to that in DRILL-6453 -- that is the new "early sniffing" in
>>>> the
>>>> Hash-Join, which repeatedly invokes next() on the two "children" of the
>>>> join *during schema discovery* until non-empty data is returned (or
>>>> NONE,
>>>> STOP, etc). Last night Salim, Vlad and I briefly discussed
>>>> alternatives,
>>>> like postponing the "sniffing" to a later time (beginning of the build
>>>>
>>> for
>>>
>>>> the right child, and beginning of the probe for the left child).
>>>>
>>>> However this would require some work time. So what should we do about
>>>>
>>> 1.14
>>>
>>>> ?
>>>>
>>>> Thanks,
>>>>
>>>> Boaz
>>>>
>>>> On Fri, Jul 13, 2018 at 3:46 AM, Arina Yelchiyeva <
>>>> arina.yelchiyeva@gmail.com> wrote:
>>>>
>>>> During implementing late limit 0 optimization, Bohdan has found one more
>>>>> regression after Hash Join spill to disk.
>>>>> https://issues.apache.org/jira/browse/DRILL-6606
>>>>> <
>>>>>
>>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.
>>> apache.org_jira_browse_DRILL-2D6606&d=DwMFaQ&c=cskdkSMqhcnjZ
>>> xdQVpwTXg&r=7lXQnf0aC8VQ0iMXwVgNHw&m=OHnyHeZpNk3hcwkG-JoQG6E
>>> 90tKdoS47J1rv5x-hJzw&s=wm5zpJf9K2zYzrqRB1LqLpKcvmBK5y6XC0ZUqVmSjko&e=
>>>
>>>> Boaz please take a look.
>>>>>
>>>>> Kind regards,
>>>>> Arina
>>>>>
>>>>>
>>>>
>
Re: [DISCUSS] 1.14.0 release
Posted by Vlad Rozov <vr...@apache.org>.
My 2 cents:
From Apache point of view it is OK to do a release even if unit tests
do not pass at all or there is a large number of regression introduced.
Apache release is a source release and as long as it compiles and does
not have license issues, it is up to community (PMC) to decide on any
other criteria for a release.
The issue in DRILL-6453 is not limited to a large number of hash joins.
It should be possible to reproduce it even with a single hash join as
long as left and right sides are getting batches from one(many) to many
exchanges (broadcast or hash partitioner senders).
Thank you,
Vlad
On 7/13/18 08:41, Aman Sinha wrote:
> I would say we have to take a measured approach to this and decide on a
> case-by-case which issue is a show stopper.
> While of course we have to make every effort to avoid regression, we cannot
> claim that a particular release will not cause any regression.
> I believe there are 10000+ passing tests, so that should provide a level
> of confidence. The TPC-DS 72 is a 10 table join which in the hadoop world
> of
> denormalized schemas is not relatively common. The main question is does
> the issue reproduce with fewer joins having the same type of distribution
> plan ?
>
>
> Aman
>
> On Fri, Jul 13, 2018 at 7:36 AM Arina Yelchiyeva <ar...@gmail.com>
> wrote:
>
>> We cannot release with existing regressions, especially taking into account
>> the there are not minor issues.
>> As far as I understand reverting is not an option since hash join spill
>> feature are extended into several commits + subsequent fixes.
>> I guess we need to consider postponing the release until issues are
>> resolved.
>>
>> Kind regards,
>> Arina
>>
>> On Fri, Jul 13, 2018 at 5:14 PM Boaz Ben-Zvi <bo...@apache.org> wrote:
>>
>>> (Guessing ...) It is possible that the root cause for DRILL-6606 is
>>> similar to that in DRILL-6453 -- that is the new "early sniffing" in the
>>> Hash-Join, which repeatedly invokes next() on the two "children" of the
>>> join *during schema discovery* until non-empty data is returned (or NONE,
>>> STOP, etc). Last night Salim, Vlad and I briefly discussed alternatives,
>>> like postponing the "sniffing" to a later time (beginning of the build
>> for
>>> the right child, and beginning of the probe for the left child).
>>>
>>> However this would require some work time. So what should we do about
>> 1.14
>>> ?
>>>
>>> Thanks,
>>>
>>> Boaz
>>>
>>> On Fri, Jul 13, 2018 at 3:46 AM, Arina Yelchiyeva <
>>> arina.yelchiyeva@gmail.com> wrote:
>>>
>>>> During implementing late limit 0 optimization, Bohdan has found one more
>>>> regression after Hash Join spill to disk.
>>>> https://issues.apache.org/jira/browse/DRILL-6606
>>>> <
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_DRILL-2D6606&d=DwMFaQ&c=cskdkSMqhcnjZxdQVpwTXg&r=7lXQnf0aC8VQ0iMXwVgNHw&m=OHnyHeZpNk3hcwkG-JoQG6E90tKdoS47J1rv5x-hJzw&s=wm5zpJf9K2zYzrqRB1LqLpKcvmBK5y6XC0ZUqVmSjko&e=
>>>> Boaz please take a look.
>>>>
>>>> Kind regards,
>>>> Arina
>>>>
>>>
Re: [DISCUSS] 1.14.0 release
Posted by Aman Sinha <am...@apache.org>.
I would say we have to take a measured approach to this and decide on a
case-by-case which issue is a show stopper.
While of course we have to make every effort to avoid regression, we cannot
claim that a particular release will not cause any regression.
I believe there are 10000+ passing tests, so that should provide a level
of confidence. The TPC-DS 72 is a 10 table join which in the hadoop world
of
denormalized schemas is not relatively common. The main question is does
the issue reproduce with fewer joins having the same type of distribution
plan ?
Aman
On Fri, Jul 13, 2018 at 7:36 AM Arina Yelchiyeva <ar...@gmail.com>
wrote:
> We cannot release with existing regressions, especially taking into account
> the there are not minor issues.
> As far as I understand reverting is not an option since hash join spill
> feature are extended into several commits + subsequent fixes.
> I guess we need to consider postponing the release until issues are
> resolved.
>
> Kind regards,
> Arina
>
> On Fri, Jul 13, 2018 at 5:14 PM Boaz Ben-Zvi <bo...@apache.org> wrote:
>
> > (Guessing ...) It is possible that the root cause for DRILL-6606 is
> > similar to that in DRILL-6453 -- that is the new "early sniffing" in the
> > Hash-Join, which repeatedly invokes next() on the two "children" of the
> > join *during schema discovery* until non-empty data is returned (or NONE,
> > STOP, etc). Last night Salim, Vlad and I briefly discussed alternatives,
> > like postponing the "sniffing" to a later time (beginning of the build
> for
> > the right child, and beginning of the probe for the left child).
> >
> > However this would require some work time. So what should we do about
> 1.14
> > ?
> >
> > Thanks,
> >
> > Boaz
> >
> > On Fri, Jul 13, 2018 at 3:46 AM, Arina Yelchiyeva <
> > arina.yelchiyeva@gmail.com> wrote:
> >
> >> During implementing late limit 0 optimization, Bohdan has found one more
> >> regression after Hash Join spill to disk.
> >> https://issues.apache.org/jira/browse/DRILL-6606
> >> <
> https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_DRILL-2D6606&d=DwMFaQ&c=cskdkSMqhcnjZxdQVpwTXg&r=7lXQnf0aC8VQ0iMXwVgNHw&m=OHnyHeZpNk3hcwkG-JoQG6E90tKdoS47J1rv5x-hJzw&s=wm5zpJf9K2zYzrqRB1LqLpKcvmBK5y6XC0ZUqVmSjko&e=
> >
> >> Boaz please take a look.
> >>
> >> Kind regards,
> >> Arina
> >>
> >
> >
>
Re: [DISCUSS] 1.14.0 release
Posted by Arina Yelchiyeva <ar...@gmail.com>.
We cannot release with existing regressions, especially taking into account
the there are not minor issues.
As far as I understand reverting is not an option since hash join spill
feature are extended into several commits + subsequent fixes.
I guess we need to consider postponing the release until issues are
resolved.
Kind regards,
Arina
On Fri, Jul 13, 2018 at 5:14 PM Boaz Ben-Zvi <bo...@apache.org> wrote:
> (Guessing ...) It is possible that the root cause for DRILL-6606 is
> similar to that in DRILL-6453 -- that is the new "early sniffing" in the
> Hash-Join, which repeatedly invokes next() on the two "children" of the
> join *during schema discovery* until non-empty data is returned (or NONE,
> STOP, etc). Last night Salim, Vlad and I briefly discussed alternatives,
> like postponing the "sniffing" to a later time (beginning of the build for
> the right child, and beginning of the probe for the left child).
>
> However this would require some work time. So what should we do about 1.14
> ?
>
> Thanks,
>
> Boaz
>
> On Fri, Jul 13, 2018 at 3:46 AM, Arina Yelchiyeva <
> arina.yelchiyeva@gmail.com> wrote:
>
>> During implementing late limit 0 optimization, Bohdan has found one more
>> regression after Hash Join spill to disk.
>> https://issues.apache.org/jira/browse/DRILL-6606
>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_DRILL-2D6606&d=DwMFaQ&c=cskdkSMqhcnjZxdQVpwTXg&r=7lXQnf0aC8VQ0iMXwVgNHw&m=OHnyHeZpNk3hcwkG-JoQG6E90tKdoS47J1rv5x-hJzw&s=wm5zpJf9K2zYzrqRB1LqLpKcvmBK5y6XC0ZUqVmSjko&e=>
>> Boaz please take a look.
>>
>> Kind regards,
>> Arina
>>
>
>
Re: [DISCUSS] 1.14.0 release
Posted by Boaz Ben-Zvi <bo...@apache.org>.
(Guessing ...) It is possible that the root cause for DRILL-6606 is similar
to that in DRILL-6453 -- that is the new "early sniffing" in the
Hash-Join, which repeatedly invokes next() on the two "children" of the
join *during schema discovery* until non-empty data is returned (or NONE,
STOP, etc). Last night Salim, Vlad and I briefly discussed alternatives,
like postponing the "sniffing" to a later time (beginning of the build for
the right child, and beginning of the probe for the left child).
However this would require some work time. So what should we do about 1.14 ?
Thanks,
Boaz
On Fri, Jul 13, 2018 at 3:46 AM, Arina Yelchiyeva <
arina.yelchiyeva@gmail.com> wrote:
> During implementing late limit 0 optimization, Bohdan has found one more
> regression after Hash Join spill to disk.
> https://issues.apache.org/jira/browse/DRILL-6606
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_DRILL-2D6606&d=DwMFaQ&c=cskdkSMqhcnjZxdQVpwTXg&r=7lXQnf0aC8VQ0iMXwVgNHw&m=OHnyHeZpNk3hcwkG-JoQG6E90tKdoS47J1rv5x-hJzw&s=wm5zpJf9K2zYzrqRB1LqLpKcvmBK5y6XC0ZUqVmSjko&e=>
> Boaz please take a look.
>
> Kind regards,
> Arina
>
Re: [DISCUSS] 1.14.0 release
Posted by Arina Yelchiyeva <ar...@gmail.com>.
During implementing late limit 0 optimization, Bohdan has found one more
regression after Hash Join spill to disk.
https://issues.apache.org/jira/browse/DRILL-6606
Boaz please take a look.
Kind regards,
Arina
On Fri, Jul 13, 2018 at 4:34 AM Boaz Ben-Zvi <bo...@mapr.com> wrote:
> We are getting close to a Release Candidate, though some issues are
> still pending, and we need to make decisions soon.
>
> Soliciting opinions -- which of the following issues should be
> considered a RELEASE BLOCKER for 1.14:
>
> ===== OPEN ======
>
> OPEN - DRILL-6453 : TPCDS query 72 is Hanging (on a cluster) (( Boaz,
> Salim ))
>
> We still do not have a lead on the cause, nor a work around to make
> this query run.
>
> OPEN - DRILL-6475: Query with UNNEST causes a Null Pointer . (( Hanumath
> ))
>
> OPEN - DRILL-5495: convert_from causes ArrayIndexOutOfBounds exception.
> (( Vitalii ))
>
> ==== In Review =======
>
> DRILL-6589: Push Transitive Closure generated predicates past aggregates
> / projects ((Gautam / Vitalii))
>
> DRILL-6588: System table columns incorrectly marked as non-nullable
> ((Kunal / Aman))
>
> DRILL-6542: (May be Ready2Commit soon) IndexOutOfBounds exception for
> multilevel lateral ((Sorabh / Parth))
>
> DRILL-6517: (May be Ready2Commit soon) IllegalState exception in
> Hash-Join ((Boaz / Padma, Tim))
>
> DRILL-6496: VectorUtil.showVectorAccessibleContent does not log vector
> content ((Tim / Volodymyr))
>
> DRILL-6410: Memory Leak in Parquet Reader during cancellation ((Vlad /
> Parth))
>
> DRILL-6179: Added pcapng-format support ((Vlad / Paul))
>
> DRILL-5796: Filter pruning for multi rowgroup parquet file ((Jean-Blas /
> Arina))
>
> DRILL-5365: FileNotFoundException when reading a parquet file ((Tim /
> Vitalii))
>
> ======================
>
> Thanks,
>
> -- Boaz
> p.s.
> There's a batch commit in process now with some of the PRs listed in
> the prior email.
>
> On 7/9/18 9:53 PM, Boaz Ben-Zvi wrote:
> > We are making progress towards 1.14.
> >
> > Let's aim for a Release Candidate branch off on Thursday (July 12) !!!
> >
> > Below are the unfinished cases; can most be completed and checked in
> > by 7/12 ?
> >
> > (( Relevant people:
> >
> > Abhishek, Arina, Boaz, Charles, Hanumath, Jean-Blas, Karthik, Kunal,
> >
> > Parth, Paul, Salim, Sorabh, Tim, Vitalii, Vlad, Volodymyr ))
> >
> > ======================
> >
> > Open/blocker - DRILL-6453 + DRILL-6517:
> > Two issues - Parquet Scanner (?) not setting container's record num
> > (to zero), and a hang following this failure.
> > Currently testing a fix / workaround ((Boaz))
> >
> > In Progress - DRILL-6104: Generic Logfile Format Plugin ((Charles +
> > Paul -- can you be done by 7/12 ?))
> >
> > PR - DRILL-5999 (DRILL-6516): Support for EMIT outcome in Streaming
> > Agg ((Parth + Boaz reviewing))
> >
> > Open - DRILL-6542: Index out of bounds ((Sorabh))
> >
> > Open - DRILL-6475: Unnest Null fieldId pointer ((Hanumath))
> >
> > ==== The following PRs are still waiting for reviews ====
> >
> > DRILL-6583: UI usability issue ((Kunal / Sorabh))
> >
> > DRILL-6579: Add sanity checks to the Parquet Reader ((Salim / Vlad +
> > Boaz))
> >
> > DRILL-6578: handle query cancellation in Parquet Reader ((Salim / Vlad
> > + Boaz))
> >
> > DRILL-6560: Allow options for controlling the batch size per operator
> > ((Salim / Karthik))
> >
> > DRILL-6559: Travis timing out ((Vitalii / Tim))
> >
> > DRILL-6496: VectorUtil.showVectorAccessibleContent does not log vector
> > content ((Tim / Volodymyr))
> >
> > DRILL-6410: Memory Leak in Parquet Reader during cancellation ((Vlad /
> > Parth))
> >
> > DRILL-6346: Create an Official Drill Docker Container ((Abhishek / Tim))
> >
> > DRILL-6179: Added pcapng-format support ((Vlad / Paul))
> >
> > DRILL-5796: Filter pruning for multi rowgroup parquet file ((Jean-Blas
> > / Arina))
> >
> > DRILL-5365: FileNotFoundException when reading a parquet file ((Tim /
> > Vitalii))
> >
> > ==================================
> >
> > Thanks,
> >
> > Boaz
> >
> > On 7/6/18 2:51 PM, Pritesh Maker wrote:
> >> Here is the release 1.14 dashboard
> >> (
> https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_secure_Dashboard.jspa-3FselectPageId-3D12332463&d=DwIGaQ&c=cskdkSMqhcnjZxdQVpwTXg&r=7lXQnf0aC8VQ0iMXwVgNHw&m=V0b4-BeuIMwRczzbiSXCgL7Z7f8lrmLBGH1vnSHLjB4&s=HRQU6Q4umbONtN4EqY3ryggJNEyCOghAzICypRJOels&e=
> >> ) and agile board
> >> (
> https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_secure_RapidBoard.jspa-3FrapidView-3D185&d=DwIGaQ&c=cskdkSMqhcnjZxdQVpwTXg&r=7lXQnf0aC8VQ0iMXwVgNHw&m=V0b4-BeuIMwRczzbiSXCgL7Z7f8lrmLBGH1vnSHLjB4&s=GKxSMl97YRnHJu-AL-A-vvRe5SXqw7vdDPDzMzj-Cj4&e=
> )
> >>
> >> I believe Volodymyr is targeting DRILL-6422 (Guava update) for 1.15
> >> release so it shouldn't be blocking the release. So overall, we have
> >> 2 open bugs, 2 in progress bugs (+2 doc issues), and 12 in review (+1
> >> ready to commit).
> >>
> >> If the reviewable commits won't be ready soon, can the developers
> >> please remove the 1.14 fix version for these issues.
> >>
> >> Pritesh
> >>
> >>
> >>
> >>
> >> On 7/6/18, 11:54 AM, "Boaz Ben-Zvi" <bben-zvi@mapr.com on behalf of
> >> boaz@mapr.com> wrote:
> >>
> >> Current status: There's a blocker, and some work in progress
> >> that will
> >> stretch into next week.
> >> Current detail:
> >> ==============================
> >> Open/blocker - DRILL-6453 + DRILL-6517: Two issues -
> >> Parquet Scanner not setting record num (to zero), and a hang
> >> following this failure.
> >> In Progress - DRILL-6104: Generic Logfile Format Plugin
> >> PR - DRILL-6422: Update Guava to 23.0 and shade it
> >> PR - DRILL-5999 (DRILL-6516): Support for EMIT outcome in
> >> Streaming Agg (I'm reviewing)
> >> Ready2Commit: DRILL-6519: Add String Distance and
> >> Phonetic Functions (Arina gave it a +1 ; is it "Ready-To-Commit" or
> >> waiting for more reviews ?)
> >> Committed: DRILL-6570: Mentioned as a blocker by Kunal
> >> (I just merge #1354; the Jira was (mistakenly ?) marked "Resolved" so
> >> it missed the batch commit).
> >> Committed: DRILL-5977: predicate pushdown support
> >> kafkaMsgOffset
> >> Committed: DRILL-6577: Change Hash-Join default to not
> >> fallback (into
> >> pre-1.14 unlimited memory)
> >> Committed: DRILL-6353: Upgrade Parquet MR dependencies
> >> Committed: DRILL-6310: limit batch size for hash aggregate
> >> ===============================
> >> Thanks,
> >> Boaz
> >> On 7/2/18 9:51 PM, Khurram Faraaz wrote:
> >> > Do we plan to fix this one too, because this is a regression
> >> from Apache
> >> > Drill 1.13.0.
> >> >
> >>
> https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_DRILL-2D6453&d=DwIBaQ&c=cskdkSMqhcnjZxdQVpwTXg&r=EqulKDxxEDCX6zbp1AZAa1-iAPQGgCioAqgDp7DE2BU&m=9HpIZKDh_DBcx9yXCI2TtUWum8vRhdQgmw_3ljJQi1M&s=w-1C-HnlUAGRHvDFUusZ78WZrHahDka2r-F-cNb-pZA&e=
> >> >
> >> > On Mon, Jul 2, 2018 at 9:33 PM, Kunal Khatua
> >> <ku...@apache.org> wrote:
> >> >
> >> >> DRILL-6570 seems like a must-have (release blocker, IMHO).
> >> >> On 7/2/2018 8:02:00 PM, Boaz Ben-Zvi <bo...@mapr.com> wrote:
> >> >> Let's try to make progress on the 1.14 release, aiming for a
> >> Release
> >> >> Candidate towards the end of this week (a little ambitious,
> >> with the
> >> >> July 4th and people on vacations).
> >> >>
> >> >> Current Status of the previously requested Jiras:
> >> >>
> >> >> ==============================
> >> >>
> >> >> In Progress - DRILL-6104: Generic Logfile Format Plugin
> >> >>
> >> >> PR - DRILL-6422: Update Guava to 23.0 and shade it
> >> >>
> >> >> PR - DRILL-5999 (DRILL-6516): Support for EMIT outcome in
> >> Streaming Agg
> >> >>
> >> >> Ready2Commit: DRILL-5977: predicate pushdown support
> >> kafkaMsgOffset
> >> >>
> >> >> Ready2Commit: DRILL-6519: Add String Distance and Phonetic
> >> Functions
> >> >>
> >> >> Ready2Commit: DRILL-6577: Change Hash-Join default to not
> >> fallback (into
> >> >> pre-1.14 unlimited memory)
> >> >>
> >> >> Committed: DRILL-6353: Upgrade Parquet MR dependencies
> >> >>
> >> >> Committed: DRILL-6310: limit batch size for hash aggregate
> >> >>
> >> >> ===============================
> >> >>
> >> >> And there are few more open or in a PR state.
> >> >>
> >> >> Lets try and most of these ready by the end of the week.
> >> >>
> >> >> Boaz
> >> >>
> >> >>
> >> >>
> >
>
>
Re: [DISCUSS] 1.14.0 release
Posted by Boaz Ben-Zvi <bo...@mapr.com>.
We are getting close to a Release Candidate, though some issues are
still pending, and we need to make decisions soon.
Soliciting opinions -- which of the following issues should be
considered a RELEASE BLOCKER for 1.14:
===== OPEN ======
OPEN - DRILL-6453 : TPCDS query 72 is Hanging (on a cluster) (( Boaz,
Salim ))
We still do not have a lead on the cause, nor a work around to make
this query run.
OPEN - DRILL-6475: Query with UNNEST causes a Null Pointer . (( Hanumath ))
OPEN - DRILL-5495: convert_from causes ArrayIndexOutOfBounds exception.
(( Vitalii ))
==== In Review =======
DRILL-6589: Push Transitive Closure generated predicates past aggregates
/ projects ((Gautam / Vitalii))
DRILL-6588: System table columns incorrectly marked as non-nullable
((Kunal / Aman))
DRILL-6542: (May be Ready2Commit soon) IndexOutOfBounds exception for
multilevel lateral ((Sorabh / Parth))
DRILL-6517: (May be Ready2Commit soon) IllegalState exception in
Hash-Join ((Boaz / Padma, Tim))
DRILL-6496: VectorUtil.showVectorAccessibleContent does not log vector
content ((Tim / Volodymyr))
DRILL-6410: Memory Leak in Parquet Reader during cancellation ((Vlad /
Parth))
DRILL-6179: Added pcapng-format support ((Vlad / Paul))
DRILL-5796: Filter pruning for multi rowgroup parquet file ((Jean-Blas /
Arina))
DRILL-5365: FileNotFoundException when reading a parquet file ((Tim /
Vitalii))
======================
Thanks,
-- Boaz
p.s.
There's a batch commit in process now with some of the PRs listed in
the prior email.
On 7/9/18 9:53 PM, Boaz Ben-Zvi wrote:
> We are making progress towards 1.14.
>
> Let's aim for a Release Candidate branch off on Thursday (July 12) !!!
>
> Below are the unfinished cases; can most be completed and checked in
> by 7/12 ?
>
> (( Relevant people:
>
> Abhishek, Arina, Boaz, Charles, Hanumath, Jean-Blas, Karthik, Kunal,
>
> Parth, Paul, Salim, Sorabh, Tim, Vitalii, Vlad, Volodymyr ))
>
> ======================
>
> Open/blocker - DRILL-6453 + DRILL-6517:
> Two issues - Parquet Scanner (?) not setting container's record num
> (to zero), and a hang following this failure.
> Currently testing a fix / workaround ((Boaz))
>
> In Progress - DRILL-6104: Generic Logfile Format Plugin ((Charles +
> Paul -- can you be done by 7/12 ?))
>
> PR - DRILL-5999 (DRILL-6516): Support for EMIT outcome in Streaming
> Agg ((Parth + Boaz reviewing))
>
> Open - DRILL-6542: Index out of bounds ((Sorabh))
>
> Open - DRILL-6475: Unnest Null fieldId pointer ((Hanumath))
>
> ==== The following PRs are still waiting for reviews ====
>
> DRILL-6583: UI usability issue ((Kunal / Sorabh))
>
> DRILL-6579: Add sanity checks to the Parquet Reader ((Salim / Vlad +
> Boaz))
>
> DRILL-6578: handle query cancellation in Parquet Reader ((Salim / Vlad
> + Boaz))
>
> DRILL-6560: Allow options for controlling the batch size per operator
> ((Salim / Karthik))
>
> DRILL-6559: Travis timing out ((Vitalii / Tim))
>
> DRILL-6496: VectorUtil.showVectorAccessibleContent does not log vector
> content ((Tim / Volodymyr))
>
> DRILL-6410: Memory Leak in Parquet Reader during cancellation ((Vlad /
> Parth))
>
> DRILL-6346: Create an Official Drill Docker Container ((Abhishek / Tim))
>
> DRILL-6179: Added pcapng-format support ((Vlad / Paul))
>
> DRILL-5796: Filter pruning for multi rowgroup parquet file ((Jean-Blas
> / Arina))
>
> DRILL-5365: FileNotFoundException when reading a parquet file ((Tim /
> Vitalii))
>
> ==================================
>
> Thanks,
>
> Boaz
>
> On 7/6/18 2:51 PM, Pritesh Maker wrote:
>> Here is the release 1.14 dashboard
>> (https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_secure_Dashboard.jspa-3FselectPageId-3D12332463&d=DwIGaQ&c=cskdkSMqhcnjZxdQVpwTXg&r=7lXQnf0aC8VQ0iMXwVgNHw&m=V0b4-BeuIMwRczzbiSXCgL7Z7f8lrmLBGH1vnSHLjB4&s=HRQU6Q4umbONtN4EqY3ryggJNEyCOghAzICypRJOels&e=
>> ) and agile board
>> (https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_secure_RapidBoard.jspa-3FrapidView-3D185&d=DwIGaQ&c=cskdkSMqhcnjZxdQVpwTXg&r=7lXQnf0aC8VQ0iMXwVgNHw&m=V0b4-BeuIMwRczzbiSXCgL7Z7f8lrmLBGH1vnSHLjB4&s=GKxSMl97YRnHJu-AL-A-vvRe5SXqw7vdDPDzMzj-Cj4&e=)
>>
>> I believe Volodymyr is targeting DRILL-6422 (Guava update) for 1.15
>> release so it shouldn't be blocking the release. So overall, we have
>> 2 open bugs, 2 in progress bugs (+2 doc issues), and 12 in review (+1
>> ready to commit).
>>
>> If the reviewable commits won't be ready soon, can the developers
>> please remove the 1.14 fix version for these issues.
>>
>> Pritesh
>>
>>
>>
>>
>> On 7/6/18, 11:54 AM, "Boaz Ben-Zvi" <bben-zvi@mapr.com on behalf of
>> boaz@mapr.com> wrote:
>>
>> Current status: There's a blocker, and some work in progress
>> that will
>> stretch into next week.
>> Current detail:
>> ==============================
>> Open/blocker - DRILL-6453 + DRILL-6517: Two issues -
>> Parquet Scanner not setting record num (to zero), and a hang
>> following this failure.
>> In Progress - DRILL-6104: Generic Logfile Format Plugin
>> PR - DRILL-6422: Update Guava to 23.0 and shade it
>> PR - DRILL-5999 (DRILL-6516): Support for EMIT outcome in
>> Streaming Agg (I'm reviewing)
>> Ready2Commit: DRILL-6519: Add String Distance and
>> Phonetic Functions (Arina gave it a +1 ; is it "Ready-To-Commit" or
>> waiting for more reviews ?)
>> Committed: DRILL-6570: Mentioned as a blocker by Kunal
>> (I just merge #1354; the Jira was (mistakenly ?) marked "Resolved" so
>> it missed the batch commit).
>> Committed: DRILL-5977: predicate pushdown support
>> kafkaMsgOffset
>> Committed: DRILL-6577: Change Hash-Join default to not
>> fallback (into
>> pre-1.14 unlimited memory)
>> Committed: DRILL-6353: Upgrade Parquet MR dependencies
>> Committed: DRILL-6310: limit batch size for hash aggregate
>> ===============================
>> Thanks,
>> Boaz
>> On 7/2/18 9:51 PM, Khurram Faraaz wrote:
>> > Do we plan to fix this one too, because this is a regression
>> from Apache
>> > Drill 1.13.0.
>> >
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_DRILL-2D6453&d=DwIBaQ&c=cskdkSMqhcnjZxdQVpwTXg&r=EqulKDxxEDCX6zbp1AZAa1-iAPQGgCioAqgDp7DE2BU&m=9HpIZKDh_DBcx9yXCI2TtUWum8vRhdQgmw_3ljJQi1M&s=w-1C-HnlUAGRHvDFUusZ78WZrHahDka2r-F-cNb-pZA&e=
>> >
>> > On Mon, Jul 2, 2018 at 9:33 PM, Kunal Khatua
>> <ku...@apache.org> wrote:
>> >
>> >> DRILL-6570 seems like a must-have (release blocker, IMHO).
>> >> On 7/2/2018 8:02:00 PM, Boaz Ben-Zvi <bo...@mapr.com> wrote:
>> >> Let's try to make progress on the 1.14 release, aiming for a
>> Release
>> >> Candidate towards the end of this week (a little ambitious,
>> with the
>> >> July 4th and people on vacations).
>> >>
>> >> Current Status of the previously requested Jiras:
>> >>
>> >> ==============================
>> >>
>> >> In Progress - DRILL-6104: Generic Logfile Format Plugin
>> >>
>> >> PR - DRILL-6422: Update Guava to 23.0 and shade it
>> >>
>> >> PR - DRILL-5999 (DRILL-6516): Support for EMIT outcome in
>> Streaming Agg
>> >>
>> >> Ready2Commit: DRILL-5977: predicate pushdown support
>> kafkaMsgOffset
>> >>
>> >> Ready2Commit: DRILL-6519: Add String Distance and Phonetic
>> Functions
>> >>
>> >> Ready2Commit: DRILL-6577: Change Hash-Join default to not
>> fallback (into
>> >> pre-1.14 unlimited memory)
>> >>
>> >> Committed: DRILL-6353: Upgrade Parquet MR dependencies
>> >>
>> >> Committed: DRILL-6310: limit batch size for hash aggregate
>> >>
>> >> ===============================
>> >>
>> >> And there are few more open or in a PR state.
>> >>
>> >> Lets try and most of these ready by the end of the week.
>> >>
>> >> Boaz
>> >>
>> >>
>> >>
>
Re: [DISCUSS] 1.14.0 release
Posted by Boaz Ben-Zvi <bo...@mapr.com>.
We are making progress towards 1.14.
Let's aim for a Release Candidate branch off on Thursday (July 12) !!!
Below are the unfinished cases; can most be completed and checked in by
7/12 ?
(( Relevant people:
Abhishek, Arina, Boaz, Charles, Hanumath, Jean-Blas, Karthik, Kunal,
Parth, Paul, Salim, Sorabh, Tim, Vitalii, Vlad, Volodymyr ))
======================
Open/blocker - DRILL-6453 + DRILL-6517:
Two issues - Parquet Scanner (?) not setting container's record num (to zero), and a hang following this failure.
Currently testing a fix / workaround ((Boaz))
In Progress - DRILL-6104: Generic Logfile Format Plugin ((Charles + Paul -- can you be done by 7/12 ?))
PR - DRILL-5999 (DRILL-6516): Support for EMIT outcome in Streaming Agg ((Parth + Boaz reviewing))
Open - DRILL-6542: Index out of bounds ((Sorabh))
Open - DRILL-6475: Unnest Null fieldId pointer ((Hanumath))
==== The following PRs are still waiting for reviews ====
DRILL-6583: UI usability issue ((Kunal / Sorabh))
DRILL-6579: Add sanity checks to the Parquet Reader ((Salim / Vlad + Boaz))
DRILL-6578: handle query cancellation in Parquet Reader ((Salim / Vlad + Boaz))
DRILL-6560: Allow options for controlling the batch size per operator ((Salim / Karthik))
DRILL-6559: Travis timing out ((Vitalii / Tim))
DRILL-6496: VectorUtil.showVectorAccessibleContent does not log vector content ((Tim / Volodymyr))
DRILL-6410: Memory Leak in Parquet Reader during cancellation ((Vlad / Parth))
DRILL-6346: Create an Official Drill Docker Container ((Abhishek / Tim))
DRILL-6179: Added pcapng-format support ((Vlad / Paul))
DRILL-5796: Filter pruning for multi rowgroup parquet file ((Jean-Blas / Arina))
DRILL-5365: FileNotFoundException when reading a parquet file ((Tim / Vitalii))
==================================
Thanks,
Boaz
On 7/6/18 2:51 PM, Pritesh Maker wrote:
> Here is the release 1.14 dashboard (https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_secure_Dashboard.jspa-3FselectPageId-3D12332463&d=DwIGaQ&c=cskdkSMqhcnjZxdQVpwTXg&r=7lXQnf0aC8VQ0iMXwVgNHw&m=V0b4-BeuIMwRczzbiSXCgL7Z7f8lrmLBGH1vnSHLjB4&s=HRQU6Q4umbONtN4EqY3ryggJNEyCOghAzICypRJOels&e= ) and agile board (https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_secure_RapidBoard.jspa-3FrapidView-3D185&d=DwIGaQ&c=cskdkSMqhcnjZxdQVpwTXg&r=7lXQnf0aC8VQ0iMXwVgNHw&m=V0b4-BeuIMwRczzbiSXCgL7Z7f8lrmLBGH1vnSHLjB4&s=GKxSMl97YRnHJu-AL-A-vvRe5SXqw7vdDPDzMzj-Cj4&e=)
>
> I believe Volodymyr is targeting DRILL-6422 (Guava update) for 1.15 release so it shouldn't be blocking the release. So overall, we have 2 open bugs, 2 in progress bugs (+2 doc issues), and 12 in review (+1 ready to commit).
>
> If the reviewable commits won't be ready soon, can the developers please remove the 1.14 fix version for these issues.
>
> Pritesh
>
>
>
>
> On 7/6/18, 11:54 AM, "Boaz Ben-Zvi" <bben-zvi@mapr.com on behalf of boaz@mapr.com> wrote:
>
> Current status: There's a blocker, and some work in progress that will
> stretch into next week.
>
> Current detail:
>
> ==============================
>
> Open/blocker - DRILL-6453 + DRILL-6517: Two issues - Parquet Scanner not setting record num (to zero), and a hang following this failure.
>
> In Progress - DRILL-6104: Generic Logfile Format Plugin
>
> PR - DRILL-6422: Update Guava to 23.0 and shade it
>
> PR - DRILL-5999 (DRILL-6516): Support for EMIT outcome in Streaming Agg (I'm reviewing)
>
>
> Ready2Commit: DRILL-6519: Add String Distance and Phonetic Functions (Arina gave it a +1 ; is it "Ready-To-Commit" or waiting for more reviews ?)
>
>
> Committed: DRILL-6570: Mentioned as a blocker by Kunal (I just merge #1354; the Jira was (mistakenly ?) marked "Resolved" so it missed the batch commit).
>
> Committed: DRILL-5977: predicate pushdown support kafkaMsgOffset
>
> Committed: DRILL-6577: Change Hash-Join default to not fallback (into
> pre-1.14 unlimited memory)
>
> Committed: DRILL-6353: Upgrade Parquet MR dependencies
>
> Committed: DRILL-6310: limit batch size for hash aggregate
>
> ===============================
>
> Thanks,
>
> Boaz
>
> On 7/2/18 9:51 PM, Khurram Faraaz wrote:
> > Do we plan to fix this one too, because this is a regression from Apache
> > Drill 1.13.0.
> > https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_DRILL-2D6453&d=DwIBaQ&c=cskdkSMqhcnjZxdQVpwTXg&r=EqulKDxxEDCX6zbp1AZAa1-iAPQGgCioAqgDp7DE2BU&m=9HpIZKDh_DBcx9yXCI2TtUWum8vRhdQgmw_3ljJQi1M&s=w-1C-HnlUAGRHvDFUusZ78WZrHahDka2r-F-cNb-pZA&e=
> >
> > On Mon, Jul 2, 2018 at 9:33 PM, Kunal Khatua <ku...@apache.org> wrote:
> >
> >> DRILL-6570 seems like a must-have (release blocker, IMHO).
> >> On 7/2/2018 8:02:00 PM, Boaz Ben-Zvi <bo...@mapr.com> wrote:
> >> Let's try to make progress on the 1.14 release, aiming for a Release
> >> Candidate towards the end of this week (a little ambitious, with the
> >> July 4th and people on vacations).
> >>
> >> Current Status of the previously requested Jiras:
> >>
> >> ==============================
> >>
> >> In Progress - DRILL-6104: Generic Logfile Format Plugin
> >>
> >> PR - DRILL-6422: Update Guava to 23.0 and shade it
> >>
> >> PR - DRILL-5999 (DRILL-6516): Support for EMIT outcome in Streaming Agg
> >>
> >> Ready2Commit: DRILL-5977: predicate pushdown support kafkaMsgOffset
> >>
> >> Ready2Commit: DRILL-6519: Add String Distance and Phonetic Functions
> >>
> >> Ready2Commit: DRILL-6577: Change Hash-Join default to not fallback (into
> >> pre-1.14 unlimited memory)
> >>
> >> Committed: DRILL-6353: Upgrade Parquet MR dependencies
> >>
> >> Committed: DRILL-6310: limit batch size for hash aggregate
> >>
> >> ===============================
> >>
> >> And there are few more open or in a PR state.
> >>
> >> Lets try and most of these ready by the end of the week.
> >>
> >> Boaz
> >>
> >>
> >>
>
>
Re: [DISCUSS] 1.14.0 release
Posted by Padma Penumarthy <pe...@gmail.com>.
If possible, please include PR 1363 in this release so we can complete our
batch sizing work (except for exchange operators)
DRILL-6549: batch sizing for nested loop join.
Thanks
Padma
On Fri, Jul 6, 2018 at 2:51 PM, Pritesh Maker <pm...@mapr.com> wrote:
> Here is the release 1.14 dashboard (https://issues.apache.org/
> jira/secure/Dashboard.jspa?selectPageId=12332463 ) and agile board (
> https://issues.apache.org/jira/secure/RapidBoard.jspa?rapidView=185)
>
> I believe Volodymyr is targeting DRILL-6422 (Guava update) for 1.15
> release so it shouldn't be blocking the release. So overall, we have 2 open
> bugs, 2 in progress bugs (+2 doc issues), and 12 in review (+1 ready to
> commit).
>
> If the reviewable commits won't be ready soon, can the developers please
> remove the 1.14 fix version for these issues.
>
> Pritesh
>
>
>
>
> On 7/6/18, 11:54 AM, "Boaz Ben-Zvi" <bben-zvi@mapr.com on behalf of
> boaz@mapr.com> wrote:
>
> Current status: There's a blocker, and some work in progress that
> will
> stretch into next week.
>
> Current detail:
>
> ==============================
>
> Open/blocker - DRILL-6453 + DRILL-6517: Two issues - Parquet Scanner
> not setting record num (to zero), and a hang following this failure.
>
> In Progress - DRILL-6104: Generic Logfile Format Plugin
>
> PR - DRILL-6422: Update Guava to 23.0 and shade it
>
> PR - DRILL-5999 (DRILL-6516): Support for EMIT outcome in Streaming
> Agg (I'm reviewing)
>
>
> Ready2Commit: DRILL-6519: Add String Distance and Phonetic Functions
> (Arina gave it a +1 ; is it "Ready-To-Commit" or waiting for more reviews ?)
>
>
> Committed: DRILL-6570: Mentioned as a blocker by Kunal (I just merge
> #1354; the Jira was (mistakenly ?) marked "Resolved" so it missed the batch
> commit).
>
> Committed: DRILL-5977: predicate pushdown support kafkaMsgOffset
>
> Committed: DRILL-6577: Change Hash-Join default to not fallback (into
> pre-1.14 unlimited memory)
>
> Committed: DRILL-6353: Upgrade Parquet MR dependencies
>
> Committed: DRILL-6310: limit batch size for hash aggregate
>
> ===============================
>
> Thanks,
>
> Boaz
>
> On 7/2/18 9:51 PM, Khurram Faraaz wrote:
> > Do we plan to fix this one too, because this is a regression from
> Apache
> > Drill 1.13.0.
> > https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.
> apache.org_jira_browse_DRILL-2D6453&d=DwIBaQ&c=cskdkSMqhcnjZxdQVpwTXg&r=
> EqulKDxxEDCX6zbp1AZAa1-iAPQGgCioAqgDp7DE2BU&m=9HpIZKDh_
> DBcx9yXCI2TtUWum8vRhdQgmw_3ljJQi1M&s=w-1C-HnlUAGRHvDFUusZ78WZrHahDka2r-
> F-cNb-pZA&e=
> >
> > On Mon, Jul 2, 2018 at 9:33 PM, Kunal Khatua <ku...@apache.org>
> wrote:
> >
> >> DRILL-6570 seems like a must-have (release blocker, IMHO).
> >> On 7/2/2018 8:02:00 PM, Boaz Ben-Zvi <bo...@mapr.com> wrote:
> >> Let's try to make progress on the 1.14 release, aiming for a Release
> >> Candidate towards the end of this week (a little ambitious, with the
> >> July 4th and people on vacations).
> >>
> >> Current Status of the previously requested Jiras:
> >>
> >> ==============================
> >>
> >> In Progress - DRILL-6104: Generic Logfile Format Plugin
> >>
> >> PR - DRILL-6422: Update Guava to 23.0 and shade it
> >>
> >> PR - DRILL-5999 (DRILL-6516): Support for EMIT outcome in Streaming
> Agg
> >>
> >> Ready2Commit: DRILL-5977: predicate pushdown support kafkaMsgOffset
> >>
> >> Ready2Commit: DRILL-6519: Add String Distance and Phonetic Functions
> >>
> >> Ready2Commit: DRILL-6577: Change Hash-Join default to not fallback
> (into
> >> pre-1.14 unlimited memory)
> >>
> >> Committed: DRILL-6353: Upgrade Parquet MR dependencies
> >>
> >> Committed: DRILL-6310: limit batch size for hash aggregate
> >>
> >> ===============================
> >>
> >> And there are few more open or in a PR state.
> >>
> >> Lets try and most of these ready by the end of the week.
> >>
> >> Boaz
> >>
> >>
> >>
>
>
>
Re: [DISCUSS] 1.14.0 release
Posted by Pritesh Maker <pm...@mapr.com>.
Here is the release 1.14 dashboard (https://issues.apache.org/jira/secure/Dashboard.jspa?selectPageId=12332463 ) and agile board (https://issues.apache.org/jira/secure/RapidBoard.jspa?rapidView=185)
I believe Volodymyr is targeting DRILL-6422 (Guava update) for 1.15 release so it shouldn't be blocking the release. So overall, we have 2 open bugs, 2 in progress bugs (+2 doc issues), and 12 in review (+1 ready to commit).
If the reviewable commits won't be ready soon, can the developers please remove the 1.14 fix version for these issues.
Pritesh
On 7/6/18, 11:54 AM, "Boaz Ben-Zvi" <bben-zvi@mapr.com on behalf of boaz@mapr.com> wrote:
Current status: There's a blocker, and some work in progress that will
stretch into next week.
Current detail:
==============================
Open/blocker - DRILL-6453 + DRILL-6517: Two issues - Parquet Scanner not setting record num (to zero), and a hang following this failure.
In Progress - DRILL-6104: Generic Logfile Format Plugin
PR - DRILL-6422: Update Guava to 23.0 and shade it
PR - DRILL-5999 (DRILL-6516): Support for EMIT outcome in Streaming Agg (I'm reviewing)
Ready2Commit: DRILL-6519: Add String Distance and Phonetic Functions (Arina gave it a +1 ; is it "Ready-To-Commit" or waiting for more reviews ?)
Committed: DRILL-6570: Mentioned as a blocker by Kunal (I just merge #1354; the Jira was (mistakenly ?) marked "Resolved" so it missed the batch commit).
Committed: DRILL-5977: predicate pushdown support kafkaMsgOffset
Committed: DRILL-6577: Change Hash-Join default to not fallback (into
pre-1.14 unlimited memory)
Committed: DRILL-6353: Upgrade Parquet MR dependencies
Committed: DRILL-6310: limit batch size for hash aggregate
===============================
Thanks,
Boaz
On 7/2/18 9:51 PM, Khurram Faraaz wrote:
> Do we plan to fix this one too, because this is a regression from Apache
> Drill 1.13.0.
> https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_DRILL-2D6453&d=DwIBaQ&c=cskdkSMqhcnjZxdQVpwTXg&r=EqulKDxxEDCX6zbp1AZAa1-iAPQGgCioAqgDp7DE2BU&m=9HpIZKDh_DBcx9yXCI2TtUWum8vRhdQgmw_3ljJQi1M&s=w-1C-HnlUAGRHvDFUusZ78WZrHahDka2r-F-cNb-pZA&e=
>
> On Mon, Jul 2, 2018 at 9:33 PM, Kunal Khatua <ku...@apache.org> wrote:
>
>> DRILL-6570 seems like a must-have (release blocker, IMHO).
>> On 7/2/2018 8:02:00 PM, Boaz Ben-Zvi <bo...@mapr.com> wrote:
>> Let's try to make progress on the 1.14 release, aiming for a Release
>> Candidate towards the end of this week (a little ambitious, with the
>> July 4th and people on vacations).
>>
>> Current Status of the previously requested Jiras:
>>
>> ==============================
>>
>> In Progress - DRILL-6104: Generic Logfile Format Plugin
>>
>> PR - DRILL-6422: Update Guava to 23.0 and shade it
>>
>> PR - DRILL-5999 (DRILL-6516): Support for EMIT outcome in Streaming Agg
>>
>> Ready2Commit: DRILL-5977: predicate pushdown support kafkaMsgOffset
>>
>> Ready2Commit: DRILL-6519: Add String Distance and Phonetic Functions
>>
>> Ready2Commit: DRILL-6577: Change Hash-Join default to not fallback (into
>> pre-1.14 unlimited memory)
>>
>> Committed: DRILL-6353: Upgrade Parquet MR dependencies
>>
>> Committed: DRILL-6310: limit batch size for hash aggregate
>>
>> ===============================
>>
>> And there are few more open or in a PR state.
>>
>> Lets try and most of these ready by the end of the week.
>>
>> Boaz
>>
>>
>>
Re: [DISCUSS] 1.14.0 release
Posted by Charles Givre <cg...@gmail.com>.
Hi Boaz,
@Paul-Rogers said he would review DRILL-6104 over the weekend, so hopefully this will be ready to go, and I made a mistake in the unit tests for DRILL-6519, but fixed it this week, so pending one final check it should be good to go.
> On Jul 6, 2018, at 14:54, Boaz Ben-Zvi <bo...@mapr.com> wrote:
>
> Current status: There's a blocker, and some work in progress that will stretch into next week.
>
> Current detail:
>
> ==============================
>
> Open/blocker - DRILL-6453 + DRILL-6517: Two issues - Parquet Scanner not setting record num (to zero), and a hang following this failure.
>
> In Progress - DRILL-6104: Generic Logfile Format Plugin
>
> PR - DRILL-6422: Update Guava to 23.0 and shade it
>
> PR - DRILL-5999 (DRILL-6516): Support for EMIT outcome in Streaming Agg (I'm reviewing)
>
>
> Ready2Commit: DRILL-6519: Add String Distance and Phonetic Functions (Arina gave it a +1 ; is it "Ready-To-Commit" or waiting for more reviews ?)
>
>
> Committed: DRILL-6570: Mentioned as a blocker by Kunal (I just merge #1354; the Jira was (mistakenly ?) marked "Resolved" so it missed the batch commit).
>
> Committed: DRILL-5977: predicate pushdown support kafkaMsgOffset
>
> Committed: DRILL-6577: Change Hash-Join default to not fallback (into
> pre-1.14 unlimited memory)
>
> Committed: DRILL-6353: Upgrade Parquet MR dependencies
>
> Committed: DRILL-6310: limit batch size for hash aggregate
>
> ===============================
>
> Thanks,
>
> Boaz
>
> On 7/2/18 9:51 PM, Khurram Faraaz wrote:
>> Do we plan to fix this one too, because this is a regression from Apache
>> Drill 1.13.0.
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_DRILL-2D6453&d=DwIBaQ&c=cskdkSMqhcnjZxdQVpwTXg&r=EqulKDxxEDCX6zbp1AZAa1-iAPQGgCioAqgDp7DE2BU&m=9HpIZKDh_DBcx9yXCI2TtUWum8vRhdQgmw_3ljJQi1M&s=w-1C-HnlUAGRHvDFUusZ78WZrHahDka2r-F-cNb-pZA&e=
>>
>> On Mon, Jul 2, 2018 at 9:33 PM, Kunal Khatua <ku...@apache.org> wrote:
>>
>>> DRILL-6570 seems like a must-have (release blocker, IMHO).
>>> On 7/2/2018 8:02:00 PM, Boaz Ben-Zvi <bo...@mapr.com> wrote:
>>> Let's try to make progress on the 1.14 release, aiming for a Release
>>> Candidate towards the end of this week (a little ambitious, with the
>>> July 4th and people on vacations).
>>>
>>> Current Status of the previously requested Jiras:
>>>
>>> ==============================
>>>
>>> In Progress - DRILL-6104: Generic Logfile Format Plugin
>>>
>>> PR - DRILL-6422: Update Guava to 23.0 and shade it
>>>
>>> PR - DRILL-5999 (DRILL-6516): Support for EMIT outcome in Streaming Agg
>>>
>>> Ready2Commit: DRILL-5977: predicate pushdown support kafkaMsgOffset
>>>
>>> Ready2Commit: DRILL-6519: Add String Distance and Phonetic Functions
>>>
>>> Ready2Commit: DRILL-6577: Change Hash-Join default to not fallback (into
>>> pre-1.14 unlimited memory)
>>>
>>> Committed: DRILL-6353: Upgrade Parquet MR dependencies
>>>
>>> Committed: DRILL-6310: limit batch size for hash aggregate
>>>
>>> ===============================
>>>
>>> And there are few more open or in a PR state.
>>>
>>> Lets try and most of these ready by the end of the week.
>>>
>>> Boaz
>>>
>>>
>>>
>
Re: [DISCUSS] 1.14.0 release
Posted by Boaz Ben-Zvi <bo...@mapr.com>.
Current status: There's a blocker, and some work in progress that will
stretch into next week.
Current detail:
==============================
Open/blocker - DRILL-6453 + DRILL-6517: Two issues - Parquet Scanner not setting record num (to zero), and a hang following this failure.
In Progress - DRILL-6104: Generic Logfile Format Plugin
PR - DRILL-6422: Update Guava to 23.0 and shade it
PR - DRILL-5999 (DRILL-6516): Support for EMIT outcome in Streaming Agg (I'm reviewing)
Ready2Commit: DRILL-6519: Add String Distance and Phonetic Functions (Arina gave it a +1 ; is it "Ready-To-Commit" or waiting for more reviews ?)
Committed: DRILL-6570: Mentioned as a blocker by Kunal (I just merge #1354; the Jira was (mistakenly ?) marked "Resolved" so it missed the batch commit).
Committed: DRILL-5977: predicate pushdown support kafkaMsgOffset
Committed: DRILL-6577: Change Hash-Join default to not fallback (into
pre-1.14 unlimited memory)
Committed: DRILL-6353: Upgrade Parquet MR dependencies
Committed: DRILL-6310: limit batch size for hash aggregate
===============================
Thanks,
Boaz
On 7/2/18 9:51 PM, Khurram Faraaz wrote:
> Do we plan to fix this one too, because this is a regression from Apache
> Drill 1.13.0.
> https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_DRILL-2D6453&d=DwIBaQ&c=cskdkSMqhcnjZxdQVpwTXg&r=EqulKDxxEDCX6zbp1AZAa1-iAPQGgCioAqgDp7DE2BU&m=9HpIZKDh_DBcx9yXCI2TtUWum8vRhdQgmw_3ljJQi1M&s=w-1C-HnlUAGRHvDFUusZ78WZrHahDka2r-F-cNb-pZA&e=
>
> On Mon, Jul 2, 2018 at 9:33 PM, Kunal Khatua <ku...@apache.org> wrote:
>
>> DRILL-6570 seems like a must-have (release blocker, IMHO).
>> On 7/2/2018 8:02:00 PM, Boaz Ben-Zvi <bo...@mapr.com> wrote:
>> Let's try to make progress on the 1.14 release, aiming for a Release
>> Candidate towards the end of this week (a little ambitious, with the
>> July 4th and people on vacations).
>>
>> Current Status of the previously requested Jiras:
>>
>> ==============================
>>
>> In Progress - DRILL-6104: Generic Logfile Format Plugin
>>
>> PR - DRILL-6422: Update Guava to 23.0 and shade it
>>
>> PR - DRILL-5999 (DRILL-6516): Support for EMIT outcome in Streaming Agg
>>
>> Ready2Commit: DRILL-5977: predicate pushdown support kafkaMsgOffset
>>
>> Ready2Commit: DRILL-6519: Add String Distance and Phonetic Functions
>>
>> Ready2Commit: DRILL-6577: Change Hash-Join default to not fallback (into
>> pre-1.14 unlimited memory)
>>
>> Committed: DRILL-6353: Upgrade Parquet MR dependencies
>>
>> Committed: DRILL-6310: limit batch size for hash aggregate
>>
>> ===============================
>>
>> And there are few more open or in a PR state.
>>
>> Lets try and most of these ready by the end of the week.
>>
>> Boaz
>>
>>
>>
Re: [DISCUSS] 1.14.0 release
Posted by Khurram Faraaz <kf...@mapr.com>.
Do we plan to fix this one too, because this is a regression from Apache
Drill 1.13.0.
https://issues.apache.org/jira/browse/DRILL-6453
On Mon, Jul 2, 2018 at 9:33 PM, Kunal Khatua <ku...@apache.org> wrote:
> DRILL-6570 seems like a must-have (release blocker, IMHO).
> On 7/2/2018 8:02:00 PM, Boaz Ben-Zvi <bo...@mapr.com> wrote:
> Let's try to make progress on the 1.14 release, aiming for a Release
> Candidate towards the end of this week (a little ambitious, with the
> July 4th and people on vacations).
>
> Current Status of the previously requested Jiras:
>
> ==============================
>
> In Progress - DRILL-6104: Generic Logfile Format Plugin
>
> PR - DRILL-6422: Update Guava to 23.0 and shade it
>
> PR - DRILL-5999 (DRILL-6516): Support for EMIT outcome in Streaming Agg
>
> Ready2Commit: DRILL-5977: predicate pushdown support kafkaMsgOffset
>
> Ready2Commit: DRILL-6519: Add String Distance and Phonetic Functions
>
> Ready2Commit: DRILL-6577: Change Hash-Join default to not fallback (into
> pre-1.14 unlimited memory)
>
> Committed: DRILL-6353: Upgrade Parquet MR dependencies
>
> Committed: DRILL-6310: limit batch size for hash aggregate
>
> ===============================
>
> And there are few more open or in a PR state.
>
> Lets try and most of these ready by the end of the week.
>
> Boaz
>
>
>
Re: [DISCUSS] 1.14.0 release
Posted by Kunal Khatua <ku...@apache.org>.
DRILL-6570 seems like a must-have (release blocker, IMHO).
On 7/2/2018 8:02:00 PM, Boaz Ben-Zvi <bo...@mapr.com> wrote:
Let's try to make progress on the 1.14 release, aiming for a Release
Candidate towards the end of this week (a little ambitious, with the
July 4th and people on vacations).
Current Status of the previously requested Jiras:
==============================
In Progress - DRILL-6104: Generic Logfile Format Plugin
PR - DRILL-6422: Update Guava to 23.0 and shade it
PR - DRILL-5999 (DRILL-6516): Support for EMIT outcome in Streaming Agg
Ready2Commit: DRILL-5977: predicate pushdown support kafkaMsgOffset
Ready2Commit: DRILL-6519: Add String Distance and Phonetic Functions
Ready2Commit: DRILL-6577: Change Hash-Join default to not fallback (into
pre-1.14 unlimited memory)
Committed: DRILL-6353: Upgrade Parquet MR dependencies
Committed: DRILL-6310: limit batch size for hash aggregate
===============================
And there are few more open or in a PR state.
Lets try and most of these ready by the end of the week.
Boaz