You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Holden Karau <ho...@pigscanfly.ca> on 2017/03/20 22:12:35 UTC
Outstanding Spark 2.1.1 issues
Hi Spark Developers!
As we start working on the Spark 2.1.1 release I've been looking at our
outstanding issues still targeted for it. I've tried to break it down by
component so that people in charge of each component can take a quick look
and see if any of these things can/should be re-targeted to 2.2 or 2.1.2 &
the overall list is pretty short (only 9 items - 5 if we only look at
explicitly tagged) :)
If your working on something for Spark 2.1.1 and it doesn't show up in this
list please speak up now :) We have a lot of issues (including "in
progress") that are listed as impacting 2.1.0, but they aren't targeted for
2.1.1 - if there is something you are working in their which should be
targeted for 2.1.1 please let us know so it doesn't slip through the cracks.
The query string I used for looking at the 2.1.1 open issues is:
((affectedVersion = 2.1.1 AND cf[12310320] is Empty) OR fixVersion = 2.1.1
OR cf[12310320] = "2.1.1") AND project = spark AND resolution = Unresolved
ORDER BY priority DESC
None of the open issues appear to be a regression from 2.1.0, but those
seem more likely to show up during the RC process (thanks in advance to
everyone testing their workloads :)) & generally none of them seem to be
(Note: the cfs are for Target Version/s field)
Critical Issues:
SQL:
SPARK-19690 <https://issues.apache.org/jira/browse/SPARK-19690> - Join a
streaming DataFrame with a batch DataFrame may not work - PR
https://github.com/apache/spark/pull/17052 (review in progress by zsxwing,
currently failing Jenkins)*
Major Issues:
SQL:
SPARK-19035 <https://issues.apache.org/jira/browse/SPARK-19035> - rand()
function in case when cause failed - no outstanding PR (consensus on JIRA
seems to be leaning towards it being a real issue but not necessarily
everyone agrees just yet - maybe we should slip this?)*
Deploy:
SPARK-19522 <https://issues.apache.org/jira/browse/SPARK-19522>
- --executor-memory flag doesn't work in local-cluster mode -
https://github.com/apache/spark/pull/16975 (review in progress by vanzin,
but PR currently stalled waiting on response) *
Core:
SPARK-20025 <https://issues.apache.org/jira/browse/SPARK-20025> - Driver
fail over will not work, if SPARK_LOCAL* env is set. -
https://github.com/apache/spark/pull/17357 (waiting on review) *
PySpark:
SPARK-19955 <https://issues.apache.org/jira/browse/SPARK-19955> - Update
run-tests to support conda [ Part of Dropping 2.6 support -- which we
shouldn't do in a minor release -- but also fixes pip installability tests
to run in Jenkins ]- PR failing Jenkins (I need to poke this some more,
but seems like 2.7 support works but some other issues. Maybe slip to 2.2?)
Minor issues:
Tests:
SPARK-19612 <https://issues.apache.org/jira/browse/SPARK-19612> - Tests
failing with timeout - No PR per-se but it seems unrelated to the 2.1.1
release. It's not targetted for 2.1.1 but listed as affecting 2.1.1 - I'd
consider explicitly targeting this for 2.2?
PySpark:
SPARK-19570 <https://issues.apache.org/jira/browse/SPARK-19570> - Allow
to disable hive in pyspark shell - https://github.com/apache/
spark/pull/16906 PR exists but its difficult to add automated tests for
this (although if SPARK-19955
<https://issues.apache.org/jira/browse/SPARK-19955> gets in would make
testing this easier) - no reviewers yet. Possible re-target?*
Structured Streaming:
SPARK-19613 <https://issues.apache.org/jira/browse/SPARK-19613> - Flaky
test: StateStoreRDDSuite.versioning and immutability - It's not targetted
for 2.1.1 but listed as affecting 2.1.1 - I'd consider explicitly targeting
this for 2.2?
ML:
SPARK-19759 <https://issues.apache.org/jira/browse/SPARK-19759>
- ALSModel.predict on Dataframes : potential optimization by not using
blas - No PR consider re-targeting unless someone has a PR waiting in the
wings?
Explicitly targeted issues are marked with a *, the remaining issues are
listed as impacting 2.1.1 and don't have a specific target version set.
Since 2.1.1 continues the 2.1.0 branch, looking at 2.1.0 shows 1 open
blocker in SQL( SPARK-19983
<https://issues.apache.org/jira/browse/SPARK-19983> ),
Query string is:
affectedVersion = 2.1.0 AND cf[12310320] is EMPTY AND project = spark AND
resolution = Unresolved AND priority = targetPriority
Continuing on for unresolved 2.1.0 issues in Major there are 163 (76 of
them in progress), 65 Minor (26 in progress), and 9 trivial (6 in progress).
I'll be going through the 2.1.0 major issues with open PRs that impact the
PySpark component and seeing if any of them should be targeted for 2.1.1,
if anyone from the other components wants to take a look through we might
find some easy wins to be merged.
Cheers,
Holden :)
--
Cell : 425-233-8271
Twitter: https://twitter.com/holdenkarau
Re: Outstanding Spark 2.1.1 issues
Posted by Daniel Siegmann <ds...@securityscorecard.io>.
Any chance of back-porting
SPARK-14536 <https://issues.apache.org/jira/browse/SPARK-14536> - NPE in
JDBCRDD when array column contains nulls (postgresql)
It just adds a null check - just a simple bug fix - so it really belongs in
Spark 2.1.x.
On Mon, Mar 20, 2017 at 6:12 PM, Holden Karau <ho...@pigscanfly.ca> wrote:
> Hi Spark Developers!
>
> As we start working on the Spark 2.1.1 release I've been looking at our
> outstanding issues still targeted for it. I've tried to break it down by
> component so that people in charge of each component can take a quick look
> and see if any of these things can/should be re-targeted to 2.2 or 2.1.2 &
> the overall list is pretty short (only 9 items - 5 if we only look at
> explicitly tagged) :)
>
> If your working on something for Spark 2.1.1 and it doesn't show up in
> this list please speak up now :) We have a lot of issues (including "in
> progress") that are listed as impacting 2.1.0, but they aren't targeted for
> 2.1.1 - if there is something you are working in their which should be
> targeted for 2.1.1 please let us know so it doesn't slip through the cracks.
>
> The query string I used for looking at the 2.1.1 open issues is:
>
> ((affectedVersion = 2.1.1 AND cf[12310320] is Empty) OR fixVersion = 2.1.1
> OR cf[12310320] = "2.1.1") AND project = spark AND resolution = Unresolved
> ORDER BY priority DESC
>
> None of the open issues appear to be a regression from 2.1.0, but those
> seem more likely to show up during the RC process (thanks in advance to
> everyone testing their workloads :)) & generally none of them seem to be
>
> (Note: the cfs are for Target Version/s field)
>
> Critical Issues:
> SQL:
> SPARK-19690 <https://issues.apache.org/jira/browse/SPARK-19690> - Join
> a streaming DataFrame with a batch DataFrame may not work - PR
> https://github.com/apache/spark/pull/17052 (review in progress by
> zsxwing, currently failing Jenkins)*
>
> Major Issues:
> SQL:
> SPARK-19035 <https://issues.apache.org/jira/browse/SPARK-19035> - rand()
> function in case when cause failed - no outstanding PR (consensus on JIRA
> seems to be leaning towards it being a real issue but not necessarily
> everyone agrees just yet - maybe we should slip this?)*
> Deploy:
> SPARK-19522 <https://issues.apache.org/jira/browse/SPARK-19522>
> - --executor-memory flag doesn't work in local-cluster mode -
> https://github.com/apache/spark/pull/16975 (review in progress by vanzin,
> but PR currently stalled waiting on response) *
> Core:
> SPARK-20025 <https://issues.apache.org/jira/browse/SPARK-20025> - Driver
> fail over will not work, if SPARK_LOCAL* env is set. -
> https://github.com/apache/spark/pull/17357 (waiting on review) *
> PySpark:
> SPARK-19955 <https://issues.apache.org/jira/browse/SPARK-19955> - Update
> run-tests to support conda [ Part of Dropping 2.6 support -- which we
> shouldn't do in a minor release -- but also fixes pip installability tests
> to run in Jenkins ]- PR failing Jenkins (I need to poke this some more,
> but seems like 2.7 support works but some other issues. Maybe slip to 2.2?)
>
> Minor issues:
> Tests:
> SPARK-19612 <https://issues.apache.org/jira/browse/SPARK-19612> - Tests
> failing with timeout - No PR per-se but it seems unrelated to the 2.1.1
> release. It's not targetted for 2.1.1 but listed as affecting 2.1.1 - I'd
> consider explicitly targeting this for 2.2?
> PySpark:
> SPARK-19570 <https://issues.apache.org/jira/browse/SPARK-19570> - Allow
> to disable hive in pyspark shell - https://github.com/apache/sp
> ark/pull/16906 PR exists but its difficult to add automated tests for
> this (although if SPARK-19955
> <https://issues.apache.org/jira/browse/SPARK-19955> gets in would make
> testing this easier) - no reviewers yet. Possible re-target?*
> Structured Streaming:
> SPARK-19613 <https://issues.apache.org/jira/browse/SPARK-19613> - Flaky
> test: StateStoreRDDSuite.versioning and immutability - It's not targetted
> for 2.1.1 but listed as affecting 2.1.1 - I'd consider explicitly targeting
> this for 2.2?
> ML:
> SPARK-19759 <https://issues.apache.org/jira/browse/SPARK-19759>
> - ALSModel.predict on Dataframes : potential optimization by not using
> blas - No PR consider re-targeting unless someone has a PR waiting in the
> wings?
>
> Explicitly targeted issues are marked with a *, the remaining issues are
> listed as impacting 2.1.1 and don't have a specific target version set.
>
> Since 2.1.1 continues the 2.1.0 branch, looking at 2.1.0 shows 1 open
> blocker in SQL( SPARK-19983
> <https://issues.apache.org/jira/browse/SPARK-19983> ),
>
> Query string is:
>
> affectedVersion = 2.1.0 AND cf[12310320] is EMPTY AND project = spark AND
> resolution = Unresolved AND priority = targetPriority
>
> Continuing on for unresolved 2.1.0 issues in Major there are 163 (76 of
> them in progress), 65 Minor (26 in progress), and 9 trivial (6 in progress).
>
> I'll be going through the 2.1.0 major issues with open PRs that impact the
> PySpark component and seeing if any of them should be targeted for 2.1.1,
> if anyone from the other components wants to take a look through we might
> find some easy wins to be merged.
>
> Cheers,
>
> Holden :)
>
> --
> Cell : 425-233-8271 <(425)%20233-8271>
> Twitter: https://twitter.com/holdenkarau
>
Re: Outstanding Spark 2.1.1 issues
Posted by Holden Karau <ho...@pigscanfly.ca>.
Hi All,
Just circling back to see if there is anything blocking the RC that isn't
being tracked in JIRA?
The current in progress list from ((affectedVersion = 2.1.1 AND
cf[12310320] is Empty) OR fixVersion = 2.1.1 OR cf[12310320] = "2.1.1") AND
project = spark AND resolution = Unresolved ORDER BY priority DESC is only
4 elements:
1. SPARK-19690 <https://issues.apache.org/jira/browse/SPARK-19690> - Join
a streaming DataFrame with a batch DataFrame may not work (PR
https://github.com/apache/spark/pull/17052
<https://github.com/apache/spark/pull/17052> ) - some discussion around
re-targeting exists on the PR
2.
1.
1. SPARK-19522 <https://issues.apache.org/jira/browse/SPARK-19522>
- --executor-memory flag doesn't work in local-cluster mode (PR
https://github.com/apache/spark/pull/16975
<https://github.com/apache/spark/pull/16975>
2. SPARK-19035 <https://issues.apache.org/jira/browse/SPARK-19035>
-rand() function in case when cause failed - no PR exists and it
isn't a blocker so I'd suggest we consider re-targetting
1. SPARK-19759
<https://issues.apache.org/jira/browse/SPARK-19759> -
ALSModel.predict
on Dataframes : potential optimization by not using
blas - not explicitly
targeted but I'd suggest targeting for 2.3 if people agree
Cheers,
Holden :)
On Tue, Mar 28, 2017 at 2:07 PM, Xiao Li <ga...@gmail.com> wrote:
> Hi, Michael,
>
> Since Daniel Siegmann asked for a bug fix backport in the previous email,
> I just merged https://issues.apache.org/jira/browse/SPARK-14536 into
> Spark 2.1 branch.
>
> If this JIRA is not part of Spark 2.1.1 release, could you help me correct
> the fix version from 2.1.1. to the next release number.
>
> Thanks,
>
> Xiao
>
> 2017-03-28 8:33 GMT-07:00 Michael Armbrust <mi...@databricks.com>:
>
>> We just fixed the build yesterday. I'll kick off a new RC today.
>>
>> On Tue, Mar 28, 2017 at 8:04 AM, Asher Krim <ak...@hubspot.com> wrote:
>>
>>> Hey Michael,
>>> any update on this? We're itching for a 2.1.1 release (specifically
>>> SPARK-14804 which is currently blocking us)
>>>
>>> Thanks,
>>> Asher Krim
>>> Senior Software Engineer
>>>
>>> On Wed, Mar 22, 2017 at 7:44 PM, Michael Armbrust <
>>> michael@databricks.com> wrote:
>>>
>>>> An update: I cut the tag for RC1 last night. Currently fighting with
>>>> the release process. Will post RC1 once I get it working.
>>>>
>>>> On Tue, Mar 21, 2017 at 2:16 PM, Nick Pentreath <
>>>> nick.pentreath@gmail.com> wrote:
>>>>
>>>>> As for SPARK-19759 <https://issues.apache.org/jira/browse/SPARK-19759>,
>>>>> I don't think that needs to be targeted for 2.1.1 so we don't need to worry
>>>>> about it
>>>>>
>>>>>
>>>>> On Tue, 21 Mar 2017 at 13:49 Holden Karau <ho...@pigscanfly.ca>
>>>>> wrote:
>>>>>
>>>>>> I agree with Michael, I think we've got some outstanding issues but
>>>>>> none of them seem like regression from 2.1 so we should be good to start
>>>>>> the RC process.
>>>>>>
>>>>>> On Tue, Mar 21, 2017 at 1:41 PM, Michael Armbrust <
>>>>>> michael@databricks.com> wrote:
>>>>>>
>>>>>> Please speak up if I'm wrong, but none of these seem like critical
>>>>>> regressions from 2.1. As such I'll start the RC process later today.
>>>>>>
>>>>>> On Mon, Mar 20, 2017 at 9:52 PM, Holden Karau <ho...@pigscanfly.ca>
>>>>>> wrote:
>>>>>>
>>>>>> I'm not super sure it should be a blocker for 2.1.1 -- is it a
>>>>>> regression? Maybe we can get TDs input on it?
>>>>>>
>>>>>> On Mon, Mar 20, 2017 at 8:48 PM Nan Zhu <zh...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>> I think https://issues.apache.org/jira/browse/SPARK-19280 should be
>>>>>> a blocker
>>>>>>
>>>>>> Best,
>>>>>>
>>>>>> Nan
>>>>>>
>>>>>> On Mon, Mar 20, 2017 at 8:18 PM, Felix Cheung <
>>>>>> felixcheung_m@hotmail.com> wrote:
>>>>>>
>>>>>> I've been scrubbing R and think we are tracking 2 issues
>>>>>>
>>>>>> https://issues.apache.org/jira/browse/SPARK-19237
>>>>>>
>>>>>> https://issues.apache.org/jira/browse/SPARK-19925
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> ------------------------------
>>>>>> *From:* holden.karau@gmail.com <ho...@gmail.com> on behalf of
>>>>>> Holden Karau <ho...@pigscanfly.ca>
>>>>>> *Sent:* Monday, March 20, 2017 3:12:35 PM
>>>>>> *To:* dev@spark.apache.org
>>>>>> *Subject:* Outstanding Spark 2.1.1 issues
>>>>>>
>>>>>> Hi Spark Developers!
>>>>>>
>>>>>> As we start working on the Spark 2.1.1 release I've been looking at
>>>>>> our outstanding issues still targeted for it. I've tried to break it down
>>>>>> by component so that people in charge of each component can take a quick
>>>>>> look and see if any of these things can/should be re-targeted to 2.2 or
>>>>>> 2.1.2 & the overall list is pretty short (only 9 items - 5 if we only look
>>>>>> at explicitly tagged) :)
>>>>>>
>>>>>> If your working on something for Spark 2.1.1 and it doesn't show up
>>>>>> in this list please speak up now :) We have a lot of issues (including "in
>>>>>> progress") that are listed as impacting 2.1.0, but they aren't targeted for
>>>>>> 2.1.1 - if there is something you are working in their which should be
>>>>>> targeted for 2.1.1 please let us know so it doesn't slip through the cracks.
>>>>>>
>>>>>> The query string I used for looking at the 2.1.1 open issues is:
>>>>>>
>>>>>> ((affectedVersion = 2.1.1 AND cf[12310320] is Empty) OR fixVersion =
>>>>>> 2.1.1 OR cf[12310320] = "2.1.1") AND project = spark AND resolution =
>>>>>> Unresolved ORDER BY priority DESC
>>>>>>
>>>>>> None of the open issues appear to be a regression from 2.1.0, but
>>>>>> those seem more likely to show up during the RC process (thanks in advance
>>>>>> to everyone testing their workloads :)) & generally none of them seem to be
>>>>>>
>>>>>> (Note: the cfs are for Target Version/s field)
>>>>>>
>>>>>> Critical Issues:
>>>>>> SQL:
>>>>>> SPARK-19690 <https://issues.apache.org/jira/browse/SPARK-19690> - Join
>>>>>> a streaming DataFrame with a batch DataFrame may not work - PR
>>>>>> https://github.com/apache/spark/pull/17052 (review in progress by
>>>>>> zsxwing, currently failing Jenkins)*
>>>>>>
>>>>>> Major Issues:
>>>>>> SQL:
>>>>>> SPARK-19035 <https://issues.apache.org/jira/browse/SPARK-19035> - rand()
>>>>>> function in case when cause failed - no outstanding PR (consensus on JIRA
>>>>>> seems to be leaning towards it being a real issue but not necessarily
>>>>>> everyone agrees just yet - maybe we should slip this?)*
>>>>>> Deploy:
>>>>>> SPARK-19522 <https://issues.apache.org/jira/browse/SPARK-19522>
>>>>>> - --executor-memory flag doesn't work in local-cluster mode -
>>>>>> https://github.com/apache/spark/pull/16975 (review in progress by
>>>>>> vanzin, but PR currently stalled waiting on response) *
>>>>>> Core:
>>>>>> SPARK-20025 <https://issues.apache.org/jira/browse/SPARK-20025> - Driver
>>>>>> fail over will not work, if SPARK_LOCAL* env is set. -
>>>>>> https://github.com/apache/spark/pull/17357 (waiting on review) *
>>>>>> PySpark:
>>>>>> SPARK-19955 <https://issues.apache.org/jira/browse/SPARK-19955> -
>>>>>> Update run-tests to support conda [ Part of Dropping 2.6 support -- which
>>>>>> we shouldn't do in a minor release -- but also fixes pip installability
>>>>>> tests to run in Jenkins ]- PR failing Jenkins (I need to poke this some
>>>>>> more, but seems like 2.7 support works but some other issues. Maybe slip to
>>>>>> 2.2?)
>>>>>>
>>>>>> Minor issues:
>>>>>> Tests:
>>>>>> SPARK-19612 <https://issues.apache.org/jira/browse/SPARK-19612> - Tests
>>>>>> failing with timeout - No PR per-se but it seems unrelated to the 2.1.1
>>>>>> release. It's not targetted for 2.1.1 but listed as affecting 2.1.1 - I'd
>>>>>> consider explicitly targeting this for 2.2?
>>>>>> PySpark:
>>>>>> SPARK-19570 <https://issues.apache.org/jira/browse/SPARK-19570> - Allow
>>>>>> to disable hive in pyspark shell - https://github.com/apache/sp
>>>>>> ark/pull/16906 PR exists but its difficult to add automated tests
>>>>>> for this (although if SPARK-19955
>>>>>> <https://issues.apache.org/jira/browse/SPARK-19955> gets in would
>>>>>> make testing this easier) - no reviewers yet. Possible re-target?*
>>>>>> Structured Streaming:
>>>>>> SPARK-19613 <https://issues.apache.org/jira/browse/SPARK-19613> - Flaky
>>>>>> test: StateStoreRDDSuite.versioning and immutability - It's not targetted
>>>>>> for 2.1.1 but listed as affecting 2.1.1 - I'd consider explicitly targeting
>>>>>> this for 2.2?
>>>>>> ML:
>>>>>> SPARK-19759 <https://issues.apache.org/jira/browse/SPARK-19759>
>>>>>> - ALSModel.predict on Dataframes : potential optimization by not
>>>>>> using blas - No PR consider re-targeting unless someone has a PR waiting in
>>>>>> the wings?
>>>>>>
>>>>>> Explicitly targeted issues are marked with a *, the remaining issues
>>>>>> are listed as impacting 2.1.1 and don't have a specific target version set.
>>>>>>
>>>>>> Since 2.1.1 continues the 2.1.0 branch, looking at 2.1.0 shows 1 open
>>>>>> blocker in SQL( SPARK-19983
>>>>>> <https://issues.apache.org/jira/browse/SPARK-19983> ),
>>>>>>
>>>>>> Query string is:
>>>>>>
>>>>>> affectedVersion = 2.1.0 AND cf[12310320] is EMPTY AND project = spark
>>>>>> AND resolution = Unresolved AND priority = targetPriority
>>>>>>
>>>>>> Continuing on for unresolved 2.1.0 issues in Major there are 163 (76
>>>>>> of them in progress), 65 Minor (26 in progress), and 9 trivial (6 in
>>>>>> progress).
>>>>>>
>>>>>> I'll be going through the 2.1.0 major issues with open PRs that
>>>>>> impact the PySpark component and seeing if any of them should be targeted
>>>>>> for 2.1.1, if anyone from the other components wants to take a look through
>>>>>> we might find some easy wins to be merged.
>>>>>>
>>>>>> Cheers,
>>>>>>
>>>>>> Holden :)
>>>>>>
>>>>>> --
>>>>>> Cell : 425-233-8271 <(425)%20233-8271>
>>>>>> Twitter: https://twitter.com/holdenkarau
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Cell : 425-233-8271 <(425)%20233-8271>
>>>>>> Twitter: https://twitter.com/holdenkarau
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Cell : 425-233-8271 <(425)%20233-8271>
>>>>>> Twitter: https://twitter.com/holdenkarau
>>>>>>
>>>>>
>>>>
>>>
>>
>
--
Cell : 425-233-8271
Twitter: https://twitter.com/holdenkarau
Re: Outstanding Spark 2.1.1 issues
Posted by Xiao Li <ga...@gmail.com>.
Hi, Michael,
Since Daniel Siegmann asked for a bug fix backport in the previous email, I
just merged https://issues.apache.org/jira/browse/SPARK-14536 into Spark
2.1 branch.
If this JIRA is not part of Spark 2.1.1 release, could you help me correct
the fix version from 2.1.1. to the next release number.
Thanks,
Xiao
2017-03-28 8:33 GMT-07:00 Michael Armbrust <mi...@databricks.com>:
> We just fixed the build yesterday. I'll kick off a new RC today.
>
> On Tue, Mar 28, 2017 at 8:04 AM, Asher Krim <ak...@hubspot.com> wrote:
>
>> Hey Michael,
>> any update on this? We're itching for a 2.1.1 release (specifically
>> SPARK-14804 which is currently blocking us)
>>
>> Thanks,
>> Asher Krim
>> Senior Software Engineer
>>
>> On Wed, Mar 22, 2017 at 7:44 PM, Michael Armbrust <michael@databricks.com
>> > wrote:
>>
>>> An update: I cut the tag for RC1 last night. Currently fighting with
>>> the release process. Will post RC1 once I get it working.
>>>
>>> On Tue, Mar 21, 2017 at 2:16 PM, Nick Pentreath <
>>> nick.pentreath@gmail.com> wrote:
>>>
>>>> As for SPARK-19759 <https://issues.apache.org/jira/browse/SPARK-19759>,
>>>> I don't think that needs to be targeted for 2.1.1 so we don't need to worry
>>>> about it
>>>>
>>>>
>>>> On Tue, 21 Mar 2017 at 13:49 Holden Karau <ho...@pigscanfly.ca> wrote:
>>>>
>>>>> I agree with Michael, I think we've got some outstanding issues but
>>>>> none of them seem like regression from 2.1 so we should be good to start
>>>>> the RC process.
>>>>>
>>>>> On Tue, Mar 21, 2017 at 1:41 PM, Michael Armbrust <
>>>>> michael@databricks.com> wrote:
>>>>>
>>>>> Please speak up if I'm wrong, but none of these seem like critical
>>>>> regressions from 2.1. As such I'll start the RC process later today.
>>>>>
>>>>> On Mon, Mar 20, 2017 at 9:52 PM, Holden Karau <ho...@pigscanfly.ca>
>>>>> wrote:
>>>>>
>>>>> I'm not super sure it should be a blocker for 2.1.1 -- is it a
>>>>> regression? Maybe we can get TDs input on it?
>>>>>
>>>>> On Mon, Mar 20, 2017 at 8:48 PM Nan Zhu <zh...@gmail.com>
>>>>> wrote:
>>>>>
>>>>> I think https://issues.apache.org/jira/browse/SPARK-19280 should be a
>>>>> blocker
>>>>>
>>>>> Best,
>>>>>
>>>>> Nan
>>>>>
>>>>> On Mon, Mar 20, 2017 at 8:18 PM, Felix Cheung <
>>>>> felixcheung_m@hotmail.com> wrote:
>>>>>
>>>>> I've been scrubbing R and think we are tracking 2 issues
>>>>>
>>>>> https://issues.apache.org/jira/browse/SPARK-19237
>>>>>
>>>>> https://issues.apache.org/jira/browse/SPARK-19925
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> ------------------------------
>>>>> *From:* holden.karau@gmail.com <ho...@gmail.com> on behalf of
>>>>> Holden Karau <ho...@pigscanfly.ca>
>>>>> *Sent:* Monday, March 20, 2017 3:12:35 PM
>>>>> *To:* dev@spark.apache.org
>>>>> *Subject:* Outstanding Spark 2.1.1 issues
>>>>>
>>>>> Hi Spark Developers!
>>>>>
>>>>> As we start working on the Spark 2.1.1 release I've been looking at
>>>>> our outstanding issues still targeted for it. I've tried to break it down
>>>>> by component so that people in charge of each component can take a quick
>>>>> look and see if any of these things can/should be re-targeted to 2.2 or
>>>>> 2.1.2 & the overall list is pretty short (only 9 items - 5 if we only look
>>>>> at explicitly tagged) :)
>>>>>
>>>>> If your working on something for Spark 2.1.1 and it doesn't show up in
>>>>> this list please speak up now :) We have a lot of issues (including "in
>>>>> progress") that are listed as impacting 2.1.0, but they aren't targeted for
>>>>> 2.1.1 - if there is something you are working in their which should be
>>>>> targeted for 2.1.1 please let us know so it doesn't slip through the cracks.
>>>>>
>>>>> The query string I used for looking at the 2.1.1 open issues is:
>>>>>
>>>>> ((affectedVersion = 2.1.1 AND cf[12310320] is Empty) OR fixVersion =
>>>>> 2.1.1 OR cf[12310320] = "2.1.1") AND project = spark AND resolution =
>>>>> Unresolved ORDER BY priority DESC
>>>>>
>>>>> None of the open issues appear to be a regression from 2.1.0, but
>>>>> those seem more likely to show up during the RC process (thanks in advance
>>>>> to everyone testing their workloads :)) & generally none of them seem to be
>>>>>
>>>>> (Note: the cfs are for Target Version/s field)
>>>>>
>>>>> Critical Issues:
>>>>> SQL:
>>>>> SPARK-19690 <https://issues.apache.org/jira/browse/SPARK-19690> - Join
>>>>> a streaming DataFrame with a batch DataFrame may not work - PR
>>>>> https://github.com/apache/spark/pull/17052 (review in progress by
>>>>> zsxwing, currently failing Jenkins)*
>>>>>
>>>>> Major Issues:
>>>>> SQL:
>>>>> SPARK-19035 <https://issues.apache.org/jira/browse/SPARK-19035> - rand()
>>>>> function in case when cause failed - no outstanding PR (consensus on JIRA
>>>>> seems to be leaning towards it being a real issue but not necessarily
>>>>> everyone agrees just yet - maybe we should slip this?)*
>>>>> Deploy:
>>>>> SPARK-19522 <https://issues.apache.org/jira/browse/SPARK-19522>
>>>>> - --executor-memory flag doesn't work in local-cluster mode -
>>>>> https://github.com/apache/spark/pull/16975 (review in progress by
>>>>> vanzin, but PR currently stalled waiting on response) *
>>>>> Core:
>>>>> SPARK-20025 <https://issues.apache.org/jira/browse/SPARK-20025> - Driver
>>>>> fail over will not work, if SPARK_LOCAL* env is set. -
>>>>> https://github.com/apache/spark/pull/17357 (waiting on review) *
>>>>> PySpark:
>>>>> SPARK-19955 <https://issues.apache.org/jira/browse/SPARK-19955> -
>>>>> Update run-tests to support conda [ Part of Dropping 2.6 support -- which
>>>>> we shouldn't do in a minor release -- but also fixes pip installability
>>>>> tests to run in Jenkins ]- PR failing Jenkins (I need to poke this some
>>>>> more, but seems like 2.7 support works but some other issues. Maybe slip to
>>>>> 2.2?)
>>>>>
>>>>> Minor issues:
>>>>> Tests:
>>>>> SPARK-19612 <https://issues.apache.org/jira/browse/SPARK-19612> - Tests
>>>>> failing with timeout - No PR per-se but it seems unrelated to the 2.1.1
>>>>> release. It's not targetted for 2.1.1 but listed as affecting 2.1.1 - I'd
>>>>> consider explicitly targeting this for 2.2?
>>>>> PySpark:
>>>>> SPARK-19570 <https://issues.apache.org/jira/browse/SPARK-19570> - Allow
>>>>> to disable hive in pyspark shell - https://github.com/apache/sp
>>>>> ark/pull/16906 PR exists but its difficult to add automated tests for
>>>>> this (although if SPARK-19955
>>>>> <https://issues.apache.org/jira/browse/SPARK-19955> gets in would
>>>>> make testing this easier) - no reviewers yet. Possible re-target?*
>>>>> Structured Streaming:
>>>>> SPARK-19613 <https://issues.apache.org/jira/browse/SPARK-19613> - Flaky
>>>>> test: StateStoreRDDSuite.versioning and immutability - It's not targetted
>>>>> for 2.1.1 but listed as affecting 2.1.1 - I'd consider explicitly targeting
>>>>> this for 2.2?
>>>>> ML:
>>>>> SPARK-19759 <https://issues.apache.org/jira/browse/SPARK-19759>
>>>>> - ALSModel.predict on Dataframes : potential optimization by not
>>>>> using blas - No PR consider re-targeting unless someone has a PR waiting in
>>>>> the wings?
>>>>>
>>>>> Explicitly targeted issues are marked with a *, the remaining issues
>>>>> are listed as impacting 2.1.1 and don't have a specific target version set.
>>>>>
>>>>> Since 2.1.1 continues the 2.1.0 branch, looking at 2.1.0 shows 1 open
>>>>> blocker in SQL( SPARK-19983
>>>>> <https://issues.apache.org/jira/browse/SPARK-19983> ),
>>>>>
>>>>> Query string is:
>>>>>
>>>>> affectedVersion = 2.1.0 AND cf[12310320] is EMPTY AND project = spark
>>>>> AND resolution = Unresolved AND priority = targetPriority
>>>>>
>>>>> Continuing on for unresolved 2.1.0 issues in Major there are 163 (76
>>>>> of them in progress), 65 Minor (26 in progress), and 9 trivial (6 in
>>>>> progress).
>>>>>
>>>>> I'll be going through the 2.1.0 major issues with open PRs that impact
>>>>> the PySpark component and seeing if any of them should be targeted for
>>>>> 2.1.1, if anyone from the other components wants to take a look through we
>>>>> might find some easy wins to be merged.
>>>>>
>>>>> Cheers,
>>>>>
>>>>> Holden :)
>>>>>
>>>>> --
>>>>> Cell : 425-233-8271 <(425)%20233-8271>
>>>>> Twitter: https://twitter.com/holdenkarau
>>>>>
>>>>>
>>>>> --
>>>>> Cell : 425-233-8271 <(425)%20233-8271>
>>>>> Twitter: https://twitter.com/holdenkarau
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Cell : 425-233-8271 <(425)%20233-8271>
>>>>> Twitter: https://twitter.com/holdenkarau
>>>>>
>>>>
>>>
>>
>
Re: Outstanding Spark 2.1.1 issues
Posted by Michael Armbrust <mi...@databricks.com>.
We just fixed the build yesterday. I'll kick off a new RC today.
On Tue, Mar 28, 2017 at 8:04 AM, Asher Krim <ak...@hubspot.com> wrote:
> Hey Michael,
> any update on this? We're itching for a 2.1.1 release (specifically
> SPARK-14804 which is currently blocking us)
>
> Thanks,
> Asher Krim
> Senior Software Engineer
>
> On Wed, Mar 22, 2017 at 7:44 PM, Michael Armbrust <mi...@databricks.com>
> wrote:
>
>> An update: I cut the tag for RC1 last night. Currently fighting with the
>> release process. Will post RC1 once I get it working.
>>
>> On Tue, Mar 21, 2017 at 2:16 PM, Nick Pentreath <nick.pentreath@gmail.com
>> > wrote:
>>
>>> As for SPARK-19759 <https://issues.apache.org/jira/browse/SPARK-19759>,
>>> I don't think that needs to be targeted for 2.1.1 so we don't need to worry
>>> about it
>>>
>>>
>>> On Tue, 21 Mar 2017 at 13:49 Holden Karau <ho...@pigscanfly.ca> wrote:
>>>
>>>> I agree with Michael, I think we've got some outstanding issues but
>>>> none of them seem like regression from 2.1 so we should be good to start
>>>> the RC process.
>>>>
>>>> On Tue, Mar 21, 2017 at 1:41 PM, Michael Armbrust <
>>>> michael@databricks.com> wrote:
>>>>
>>>> Please speak up if I'm wrong, but none of these seem like critical
>>>> regressions from 2.1. As such I'll start the RC process later today.
>>>>
>>>> On Mon, Mar 20, 2017 at 9:52 PM, Holden Karau <ho...@pigscanfly.ca>
>>>> wrote:
>>>>
>>>> I'm not super sure it should be a blocker for 2.1.1 -- is it a
>>>> regression? Maybe we can get TDs input on it?
>>>>
>>>> On Mon, Mar 20, 2017 at 8:48 PM Nan Zhu <zh...@gmail.com> wrote:
>>>>
>>>> I think https://issues.apache.org/jira/browse/SPARK-19280 should be a
>>>> blocker
>>>>
>>>> Best,
>>>>
>>>> Nan
>>>>
>>>> On Mon, Mar 20, 2017 at 8:18 PM, Felix Cheung <
>>>> felixcheung_m@hotmail.com> wrote:
>>>>
>>>> I've been scrubbing R and think we are tracking 2 issues
>>>>
>>>> https://issues.apache.org/jira/browse/SPARK-19237
>>>>
>>>> https://issues.apache.org/jira/browse/SPARK-19925
>>>>
>>>>
>>>>
>>>>
>>>> ------------------------------
>>>> *From:* holden.karau@gmail.com <ho...@gmail.com> on behalf of
>>>> Holden Karau <ho...@pigscanfly.ca>
>>>> *Sent:* Monday, March 20, 2017 3:12:35 PM
>>>> *To:* dev@spark.apache.org
>>>> *Subject:* Outstanding Spark 2.1.1 issues
>>>>
>>>> Hi Spark Developers!
>>>>
>>>> As we start working on the Spark 2.1.1 release I've been looking at our
>>>> outstanding issues still targeted for it. I've tried to break it down by
>>>> component so that people in charge of each component can take a quick look
>>>> and see if any of these things can/should be re-targeted to 2.2 or 2.1.2 &
>>>> the overall list is pretty short (only 9 items - 5 if we only look at
>>>> explicitly tagged) :)
>>>>
>>>> If your working on something for Spark 2.1.1 and it doesn't show up in
>>>> this list please speak up now :) We have a lot of issues (including "in
>>>> progress") that are listed as impacting 2.1.0, but they aren't targeted for
>>>> 2.1.1 - if there is something you are working in their which should be
>>>> targeted for 2.1.1 please let us know so it doesn't slip through the cracks.
>>>>
>>>> The query string I used for looking at the 2.1.1 open issues is:
>>>>
>>>> ((affectedVersion = 2.1.1 AND cf[12310320] is Empty) OR fixVersion =
>>>> 2.1.1 OR cf[12310320] = "2.1.1") AND project = spark AND resolution =
>>>> Unresolved ORDER BY priority DESC
>>>>
>>>> None of the open issues appear to be a regression from 2.1.0, but those
>>>> seem more likely to show up during the RC process (thanks in advance to
>>>> everyone testing their workloads :)) & generally none of them seem to be
>>>>
>>>> (Note: the cfs are for Target Version/s field)
>>>>
>>>> Critical Issues:
>>>> SQL:
>>>> SPARK-19690 <https://issues.apache.org/jira/browse/SPARK-19690> - Join
>>>> a streaming DataFrame with a batch DataFrame may not work - PR
>>>> https://github.com/apache/spark/pull/17052 (review in progress by
>>>> zsxwing, currently failing Jenkins)*
>>>>
>>>> Major Issues:
>>>> SQL:
>>>> SPARK-19035 <https://issues.apache.org/jira/browse/SPARK-19035> - rand()
>>>> function in case when cause failed - no outstanding PR (consensus on JIRA
>>>> seems to be leaning towards it being a real issue but not necessarily
>>>> everyone agrees just yet - maybe we should slip this?)*
>>>> Deploy:
>>>> SPARK-19522 <https://issues.apache.org/jira/browse/SPARK-19522>
>>>> - --executor-memory flag doesn't work in local-cluster mode -
>>>> https://github.com/apache/spark/pull/16975 (review in progress by
>>>> vanzin, but PR currently stalled waiting on response) *
>>>> Core:
>>>> SPARK-20025 <https://issues.apache.org/jira/browse/SPARK-20025> - Driver
>>>> fail over will not work, if SPARK_LOCAL* env is set. -
>>>> https://github.com/apache/spark/pull/17357 (waiting on review) *
>>>> PySpark:
>>>> SPARK-19955 <https://issues.apache.org/jira/browse/SPARK-19955> -
>>>> Update run-tests to support conda [ Part of Dropping 2.6 support -- which
>>>> we shouldn't do in a minor release -- but also fixes pip installability
>>>> tests to run in Jenkins ]- PR failing Jenkins (I need to poke this some
>>>> more, but seems like 2.7 support works but some other issues. Maybe slip to
>>>> 2.2?)
>>>>
>>>> Minor issues:
>>>> Tests:
>>>> SPARK-19612 <https://issues.apache.org/jira/browse/SPARK-19612> - Tests
>>>> failing with timeout - No PR per-se but it seems unrelated to the 2.1.1
>>>> release. It's not targetted for 2.1.1 but listed as affecting 2.1.1 - I'd
>>>> consider explicitly targeting this for 2.2?
>>>> PySpark:
>>>> SPARK-19570 <https://issues.apache.org/jira/browse/SPARK-19570> - Allow
>>>> to disable hive in pyspark shell - https://github.com/apache/sp
>>>> ark/pull/16906 PR exists but its difficult to add automated tests for
>>>> this (although if SPARK-19955
>>>> <https://issues.apache.org/jira/browse/SPARK-19955> gets in would make
>>>> testing this easier) - no reviewers yet. Possible re-target?*
>>>> Structured Streaming:
>>>> SPARK-19613 <https://issues.apache.org/jira/browse/SPARK-19613> - Flaky
>>>> test: StateStoreRDDSuite.versioning and immutability - It's not targetted
>>>> for 2.1.1 but listed as affecting 2.1.1 - I'd consider explicitly targeting
>>>> this for 2.2?
>>>> ML:
>>>> SPARK-19759 <https://issues.apache.org/jira/browse/SPARK-19759>
>>>> - ALSModel.predict on Dataframes : potential optimization by not
>>>> using blas - No PR consider re-targeting unless someone has a PR waiting in
>>>> the wings?
>>>>
>>>> Explicitly targeted issues are marked with a *, the remaining issues
>>>> are listed as impacting 2.1.1 and don't have a specific target version set.
>>>>
>>>> Since 2.1.1 continues the 2.1.0 branch, looking at 2.1.0 shows 1 open
>>>> blocker in SQL( SPARK-19983
>>>> <https://issues.apache.org/jira/browse/SPARK-19983> ),
>>>>
>>>> Query string is:
>>>>
>>>> affectedVersion = 2.1.0 AND cf[12310320] is EMPTY AND project = spark
>>>> AND resolution = Unresolved AND priority = targetPriority
>>>>
>>>> Continuing on for unresolved 2.1.0 issues in Major there are 163 (76 of
>>>> them in progress), 65 Minor (26 in progress), and 9 trivial (6 in progress).
>>>>
>>>> I'll be going through the 2.1.0 major issues with open PRs that impact
>>>> the PySpark component and seeing if any of them should be targeted for
>>>> 2.1.1, if anyone from the other components wants to take a look through we
>>>> might find some easy wins to be merged.
>>>>
>>>> Cheers,
>>>>
>>>> Holden :)
>>>>
>>>> --
>>>> Cell : 425-233-8271 <(425)%20233-8271>
>>>> Twitter: https://twitter.com/holdenkarau
>>>>
>>>>
>>>> --
>>>> Cell : 425-233-8271 <(425)%20233-8271>
>>>> Twitter: https://twitter.com/holdenkarau
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Cell : 425-233-8271 <(425)%20233-8271>
>>>> Twitter: https://twitter.com/holdenkarau
>>>>
>>>
>>
>
Re: Outstanding Spark 2.1.1 issues
Posted by Asher Krim <ak...@hubspot.com>.
Hey Michael,
any update on this? We're itching for a 2.1.1 release (specifically
SPARK-14804 which is currently blocking us)
Thanks,
Asher Krim
Senior Software Engineer
On Wed, Mar 22, 2017 at 7:44 PM, Michael Armbrust <mi...@databricks.com>
wrote:
> An update: I cut the tag for RC1 last night. Currently fighting with the
> release process. Will post RC1 once I get it working.
>
> On Tue, Mar 21, 2017 at 2:16 PM, Nick Pentreath <ni...@gmail.com>
> wrote:
>
>> As for SPARK-19759 <https://issues.apache.org/jira/browse/SPARK-19759>,
>> I don't think that needs to be targeted for 2.1.1 so we don't need to worry
>> about it
>>
>>
>> On Tue, 21 Mar 2017 at 13:49 Holden Karau <ho...@pigscanfly.ca> wrote:
>>
>>> I agree with Michael, I think we've got some outstanding issues but none
>>> of them seem like regression from 2.1 so we should be good to start the RC
>>> process.
>>>
>>> On Tue, Mar 21, 2017 at 1:41 PM, Michael Armbrust <
>>> michael@databricks.com> wrote:
>>>
>>> Please speak up if I'm wrong, but none of these seem like critical
>>> regressions from 2.1. As such I'll start the RC process later today.
>>>
>>> On Mon, Mar 20, 2017 at 9:52 PM, Holden Karau <ho...@pigscanfly.ca>
>>> wrote:
>>>
>>> I'm not super sure it should be a blocker for 2.1.1 -- is it a
>>> regression? Maybe we can get TDs input on it?
>>>
>>> On Mon, Mar 20, 2017 at 8:48 PM Nan Zhu <zh...@gmail.com> wrote:
>>>
>>> I think https://issues.apache.org/jira/browse/SPARK-19280 should be a
>>> blocker
>>>
>>> Best,
>>>
>>> Nan
>>>
>>> On Mon, Mar 20, 2017 at 8:18 PM, Felix Cheung <felixcheung_m@hotmail.com
>>> > wrote:
>>>
>>> I've been scrubbing R and think we are tracking 2 issues
>>>
>>> https://issues.apache.org/jira/browse/SPARK-19237
>>>
>>> https://issues.apache.org/jira/browse/SPARK-19925
>>>
>>>
>>>
>>>
>>> ------------------------------
>>> *From:* holden.karau@gmail.com <ho...@gmail.com> on behalf of
>>> Holden Karau <ho...@pigscanfly.ca>
>>> *Sent:* Monday, March 20, 2017 3:12:35 PM
>>> *To:* dev@spark.apache.org
>>> *Subject:* Outstanding Spark 2.1.1 issues
>>>
>>> Hi Spark Developers!
>>>
>>> As we start working on the Spark 2.1.1 release I've been looking at our
>>> outstanding issues still targeted for it. I've tried to break it down by
>>> component so that people in charge of each component can take a quick look
>>> and see if any of these things can/should be re-targeted to 2.2 or 2.1.2 &
>>> the overall list is pretty short (only 9 items - 5 if we only look at
>>> explicitly tagged) :)
>>>
>>> If your working on something for Spark 2.1.1 and it doesn't show up in
>>> this list please speak up now :) We have a lot of issues (including "in
>>> progress") that are listed as impacting 2.1.0, but they aren't targeted for
>>> 2.1.1 - if there is something you are working in their which should be
>>> targeted for 2.1.1 please let us know so it doesn't slip through the cracks.
>>>
>>> The query string I used for looking at the 2.1.1 open issues is:
>>>
>>> ((affectedVersion = 2.1.1 AND cf[12310320] is Empty) OR fixVersion =
>>> 2.1.1 OR cf[12310320] = "2.1.1") AND project = spark AND resolution =
>>> Unresolved ORDER BY priority DESC
>>>
>>> None of the open issues appear to be a regression from 2.1.0, but those
>>> seem more likely to show up during the RC process (thanks in advance to
>>> everyone testing their workloads :)) & generally none of them seem to be
>>>
>>> (Note: the cfs are for Target Version/s field)
>>>
>>> Critical Issues:
>>> SQL:
>>> SPARK-19690 <https://issues.apache.org/jira/browse/SPARK-19690> - Join
>>> a streaming DataFrame with a batch DataFrame may not work - PR
>>> https://github.com/apache/spark/pull/17052 (review in progress by
>>> zsxwing, currently failing Jenkins)*
>>>
>>> Major Issues:
>>> SQL:
>>> SPARK-19035 <https://issues.apache.org/jira/browse/SPARK-19035> - rand()
>>> function in case when cause failed - no outstanding PR (consensus on JIRA
>>> seems to be leaning towards it being a real issue but not necessarily
>>> everyone agrees just yet - maybe we should slip this?)*
>>> Deploy:
>>> SPARK-19522 <https://issues.apache.org/jira/browse/SPARK-19522>
>>> - --executor-memory flag doesn't work in local-cluster mode -
>>> https://github.com/apache/spark/pull/16975 (review in progress by
>>> vanzin, but PR currently stalled waiting on response) *
>>> Core:
>>> SPARK-20025 <https://issues.apache.org/jira/browse/SPARK-20025> - Driver
>>> fail over will not work, if SPARK_LOCAL* env is set. -
>>> https://github.com/apache/spark/pull/17357 (waiting on review) *
>>> PySpark:
>>> SPARK-19955 <https://issues.apache.org/jira/browse/SPARK-19955> -
>>> Update run-tests to support conda [ Part of Dropping 2.6 support -- which
>>> we shouldn't do in a minor release -- but also fixes pip installability
>>> tests to run in Jenkins ]- PR failing Jenkins (I need to poke this some
>>> more, but seems like 2.7 support works but some other issues. Maybe slip to
>>> 2.2?)
>>>
>>> Minor issues:
>>> Tests:
>>> SPARK-19612 <https://issues.apache.org/jira/browse/SPARK-19612> - Tests
>>> failing with timeout - No PR per-se but it seems unrelated to the 2.1.1
>>> release. It's not targetted for 2.1.1 but listed as affecting 2.1.1 - I'd
>>> consider explicitly targeting this for 2.2?
>>> PySpark:
>>> SPARK-19570 <https://issues.apache.org/jira/browse/SPARK-19570> - Allow
>>> to disable hive in pyspark shell - https://github.com/apache/sp
>>> ark/pull/16906 PR exists but its difficult to add automated tests for
>>> this (although if SPARK-19955
>>> <https://issues.apache.org/jira/browse/SPARK-19955> gets in would make
>>> testing this easier) - no reviewers yet. Possible re-target?*
>>> Structured Streaming:
>>> SPARK-19613 <https://issues.apache.org/jira/browse/SPARK-19613> - Flaky
>>> test: StateStoreRDDSuite.versioning and immutability - It's not targetted
>>> for 2.1.1 but listed as affecting 2.1.1 - I'd consider explicitly targeting
>>> this for 2.2?
>>> ML:
>>> SPARK-19759 <https://issues.apache.org/jira/browse/SPARK-19759>
>>> - ALSModel.predict on Dataframes : potential optimization by not using
>>> blas - No PR consider re-targeting unless someone has a PR waiting in the
>>> wings?
>>>
>>> Explicitly targeted issues are marked with a *, the remaining issues are
>>> listed as impacting 2.1.1 and don't have a specific target version set.
>>>
>>> Since 2.1.1 continues the 2.1.0 branch, looking at 2.1.0 shows 1 open
>>> blocker in SQL( SPARK-19983
>>> <https://issues.apache.org/jira/browse/SPARK-19983> ),
>>>
>>> Query string is:
>>>
>>> affectedVersion = 2.1.0 AND cf[12310320] is EMPTY AND project = spark
>>> AND resolution = Unresolved AND priority = targetPriority
>>>
>>> Continuing on for unresolved 2.1.0 issues in Major there are 163 (76 of
>>> them in progress), 65 Minor (26 in progress), and 9 trivial (6 in progress).
>>>
>>> I'll be going through the 2.1.0 major issues with open PRs that impact
>>> the PySpark component and seeing if any of them should be targeted for
>>> 2.1.1, if anyone from the other components wants to take a look through we
>>> might find some easy wins to be merged.
>>>
>>> Cheers,
>>>
>>> Holden :)
>>>
>>> --
>>> Cell : 425-233-8271 <(425)%20233-8271>
>>> Twitter: https://twitter.com/holdenkarau
>>>
>>>
>>> --
>>> Cell : 425-233-8271 <(425)%20233-8271>
>>> Twitter: https://twitter.com/holdenkarau
>>>
>>>
>>>
>>>
>>>
>>> --
>>> Cell : 425-233-8271 <(425)%20233-8271>
>>> Twitter: https://twitter.com/holdenkarau
>>>
>>
>
Re: Outstanding Spark 2.1.1 issues
Posted by Michael Armbrust <mi...@databricks.com>.
An update: I cut the tag for RC1 last night. Currently fighting with the
release process. Will post RC1 once I get it working.
On Tue, Mar 21, 2017 at 2:16 PM, Nick Pentreath <ni...@gmail.com>
wrote:
> As for SPARK-19759 <https://issues.apache.org/jira/browse/SPARK-19759>, I
> don't think that needs to be targeted for 2.1.1 so we don't need to worry
> about it
>
>
> On Tue, 21 Mar 2017 at 13:49 Holden Karau <ho...@pigscanfly.ca> wrote:
>
>> I agree with Michael, I think we've got some outstanding issues but none
>> of them seem like regression from 2.1 so we should be good to start the RC
>> process.
>>
>> On Tue, Mar 21, 2017 at 1:41 PM, Michael Armbrust <michael@databricks.com
>> > wrote:
>>
>> Please speak up if I'm wrong, but none of these seem like critical
>> regressions from 2.1. As such I'll start the RC process later today.
>>
>> On Mon, Mar 20, 2017 at 9:52 PM, Holden Karau <ho...@pigscanfly.ca>
>> wrote:
>>
>> I'm not super sure it should be a blocker for 2.1.1 -- is it a
>> regression? Maybe we can get TDs input on it?
>>
>> On Mon, Mar 20, 2017 at 8:48 PM Nan Zhu <zh...@gmail.com> wrote:
>>
>> I think https://issues.apache.org/jira/browse/SPARK-19280 should be a
>> blocker
>>
>> Best,
>>
>> Nan
>>
>> On Mon, Mar 20, 2017 at 8:18 PM, Felix Cheung <fe...@hotmail.com>
>> wrote:
>>
>> I've been scrubbing R and think we are tracking 2 issues
>>
>> https://issues.apache.org/jira/browse/SPARK-19237
>>
>> https://issues.apache.org/jira/browse/SPARK-19925
>>
>>
>>
>>
>> ------------------------------
>> *From:* holden.karau@gmail.com <ho...@gmail.com> on behalf of
>> Holden Karau <ho...@pigscanfly.ca>
>> *Sent:* Monday, March 20, 2017 3:12:35 PM
>> *To:* dev@spark.apache.org
>> *Subject:* Outstanding Spark 2.1.1 issues
>>
>> Hi Spark Developers!
>>
>> As we start working on the Spark 2.1.1 release I've been looking at our
>> outstanding issues still targeted for it. I've tried to break it down by
>> component so that people in charge of each component can take a quick look
>> and see if any of these things can/should be re-targeted to 2.2 or 2.1.2 &
>> the overall list is pretty short (only 9 items - 5 if we only look at
>> explicitly tagged) :)
>>
>> If your working on something for Spark 2.1.1 and it doesn't show up in
>> this list please speak up now :) We have a lot of issues (including "in
>> progress") that are listed as impacting 2.1.0, but they aren't targeted for
>> 2.1.1 - if there is something you are working in their which should be
>> targeted for 2.1.1 please let us know so it doesn't slip through the cracks.
>>
>> The query string I used for looking at the 2.1.1 open issues is:
>>
>> ((affectedVersion = 2.1.1 AND cf[12310320] is Empty) OR fixVersion =
>> 2.1.1 OR cf[12310320] = "2.1.1") AND project = spark AND resolution =
>> Unresolved ORDER BY priority DESC
>>
>> None of the open issues appear to be a regression from 2.1.0, but those
>> seem more likely to show up during the RC process (thanks in advance to
>> everyone testing their workloads :)) & generally none of them seem to be
>>
>> (Note: the cfs are for Target Version/s field)
>>
>> Critical Issues:
>> SQL:
>> SPARK-19690 <https://issues.apache.org/jira/browse/SPARK-19690> - Join
>> a streaming DataFrame with a batch DataFrame may not work - PR
>> https://github.com/apache/spark/pull/17052 (review in progress by
>> zsxwing, currently failing Jenkins)*
>>
>> Major Issues:
>> SQL:
>> SPARK-19035 <https://issues.apache.org/jira/browse/SPARK-19035> - rand()
>> function in case when cause failed - no outstanding PR (consensus on JIRA
>> seems to be leaning towards it being a real issue but not necessarily
>> everyone agrees just yet - maybe we should slip this?)*
>> Deploy:
>> SPARK-19522 <https://issues.apache.org/jira/browse/SPARK-19522>
>> - --executor-memory flag doesn't work in local-cluster mode -
>> https://github.com/apache/spark/pull/16975 (review in progress by
>> vanzin, but PR currently stalled waiting on response) *
>> Core:
>> SPARK-20025 <https://issues.apache.org/jira/browse/SPARK-20025> - Driver
>> fail over will not work, if SPARK_LOCAL* env is set. -
>> https://github.com/apache/spark/pull/17357 (waiting on review) *
>> PySpark:
>> SPARK-19955 <https://issues.apache.org/jira/browse/SPARK-19955> -
>> Update run-tests to support conda [ Part of Dropping 2.6 support -- which
>> we shouldn't do in a minor release -- but also fixes pip installability
>> tests to run in Jenkins ]- PR failing Jenkins (I need to poke this some
>> more, but seems like 2.7 support works but some other issues. Maybe slip to
>> 2.2?)
>>
>> Minor issues:
>> Tests:
>> SPARK-19612 <https://issues.apache.org/jira/browse/SPARK-19612> - Tests
>> failing with timeout - No PR per-se but it seems unrelated to the 2.1.1
>> release. It's not targetted for 2.1.1 but listed as affecting 2.1.1 - I'd
>> consider explicitly targeting this for 2.2?
>> PySpark:
>> SPARK-19570 <https://issues.apache.org/jira/browse/SPARK-19570> - Allow
>> to disable hive in pyspark shell - https://github.com/apache/
>> spark/pull/16906 PR exists but its difficult to add automated tests for
>> this (although if SPARK-19955
>> <https://issues.apache.org/jira/browse/SPARK-19955> gets in would make
>> testing this easier) - no reviewers yet. Possible re-target?*
>> Structured Streaming:
>> SPARK-19613 <https://issues.apache.org/jira/browse/SPARK-19613> - Flaky
>> test: StateStoreRDDSuite.versioning and immutability - It's not targetted
>> for 2.1.1 but listed as affecting 2.1.1 - I'd consider explicitly targeting
>> this for 2.2?
>> ML:
>> SPARK-19759 <https://issues.apache.org/jira/browse/SPARK-19759>
>> - ALSModel.predict on Dataframes : potential optimization by not using
>> blas - No PR consider re-targeting unless someone has a PR waiting in the
>> wings?
>>
>> Explicitly targeted issues are marked with a *, the remaining issues are
>> listed as impacting 2.1.1 and don't have a specific target version set.
>>
>> Since 2.1.1 continues the 2.1.0 branch, looking at 2.1.0 shows 1 open
>> blocker in SQL( SPARK-19983
>> <https://issues.apache.org/jira/browse/SPARK-19983> ),
>>
>> Query string is:
>>
>> affectedVersion = 2.1.0 AND cf[12310320] is EMPTY AND project = spark AND
>> resolution = Unresolved AND priority = targetPriority
>>
>> Continuing on for unresolved 2.1.0 issues in Major there are 163 (76 of
>> them in progress), 65 Minor (26 in progress), and 9 trivial (6 in progress).
>>
>> I'll be going through the 2.1.0 major issues with open PRs that impact
>> the PySpark component and seeing if any of them should be targeted for
>> 2.1.1, if anyone from the other components wants to take a look through we
>> might find some easy wins to be merged.
>>
>> Cheers,
>>
>> Holden :)
>>
>> --
>> Cell : 425-233-8271 <(425)%20233-8271>
>> Twitter: https://twitter.com/holdenkarau
>>
>>
>> --
>> Cell : 425-233-8271 <(425)%20233-8271>
>> Twitter: https://twitter.com/holdenkarau
>>
>>
>>
>>
>>
>> --
>> Cell : 425-233-8271 <(425)%20233-8271>
>> Twitter: https://twitter.com/holdenkarau
>>
>
Re: Outstanding Spark 2.1.1 issues
Posted by Nick Pentreath <ni...@gmail.com>.
As for SPARK-19759 <https://issues.apache.org/jira/browse/SPARK-19759>, I
don't think that needs to be targeted for 2.1.1 so we don't need to worry
about it
On Tue, 21 Mar 2017 at 13:49 Holden Karau <ho...@pigscanfly.ca> wrote:
> I agree with Michael, I think we've got some outstanding issues but none
> of them seem like regression from 2.1 so we should be good to start the RC
> process.
>
> On Tue, Mar 21, 2017 at 1:41 PM, Michael Armbrust <mi...@databricks.com>
> wrote:
>
> Please speak up if I'm wrong, but none of these seem like critical
> regressions from 2.1. As such I'll start the RC process later today.
>
> On Mon, Mar 20, 2017 at 9:52 PM, Holden Karau <ho...@pigscanfly.ca>
> wrote:
>
> I'm not super sure it should be a blocker for 2.1.1 -- is it a regression?
> Maybe we can get TDs input on it?
>
> On Mon, Mar 20, 2017 at 8:48 PM Nan Zhu <zh...@gmail.com> wrote:
>
> I think https://issues.apache.org/jira/browse/SPARK-19280 should be a
> blocker
>
> Best,
>
> Nan
>
> On Mon, Mar 20, 2017 at 8:18 PM, Felix Cheung <fe...@hotmail.com>
> wrote:
>
> I've been scrubbing R and think we are tracking 2 issues
>
> https://issues.apache.org/jira/browse/SPARK-19237
>
> https://issues.apache.org/jira/browse/SPARK-19925
>
>
>
>
> ------------------------------
> *From:* holden.karau@gmail.com <ho...@gmail.com> on behalf of
> Holden Karau <ho...@pigscanfly.ca>
> *Sent:* Monday, March 20, 2017 3:12:35 PM
> *To:* dev@spark.apache.org
> *Subject:* Outstanding Spark 2.1.1 issues
>
> Hi Spark Developers!
>
> As we start working on the Spark 2.1.1 release I've been looking at our
> outstanding issues still targeted for it. I've tried to break it down by
> component so that people in charge of each component can take a quick look
> and see if any of these things can/should be re-targeted to 2.2 or 2.1.2 &
> the overall list is pretty short (only 9 items - 5 if we only look at
> explicitly tagged) :)
>
> If your working on something for Spark 2.1.1 and it doesn't show up in
> this list please speak up now :) We have a lot of issues (including "in
> progress") that are listed as impacting 2.1.0, but they aren't targeted for
> 2.1.1 - if there is something you are working in their which should be
> targeted for 2.1.1 please let us know so it doesn't slip through the cracks.
>
> The query string I used for looking at the 2.1.1 open issues is:
>
> ((affectedVersion = 2.1.1 AND cf[12310320] is Empty) OR fixVersion = 2.1.1
> OR cf[12310320] = "2.1.1") AND project = spark AND resolution = Unresolved
> ORDER BY priority DESC
>
> None of the open issues appear to be a regression from 2.1.0, but those
> seem more likely to show up during the RC process (thanks in advance to
> everyone testing their workloads :)) & generally none of them seem to be
>
> (Note: the cfs are for Target Version/s field)
>
> Critical Issues:
> SQL:
> SPARK-19690 <https://issues.apache.org/jira/browse/SPARK-19690> - Join
> a streaming DataFrame with a batch DataFrame may not work - PR
> https://github.com/apache/spark/pull/17052 (review in progress by
> zsxwing, currently failing Jenkins)*
>
> Major Issues:
> SQL:
> SPARK-19035 <https://issues.apache.org/jira/browse/SPARK-19035> - rand()
> function in case when cause failed - no outstanding PR (consensus on JIRA
> seems to be leaning towards it being a real issue but not necessarily
> everyone agrees just yet - maybe we should slip this?)*
> Deploy:
> SPARK-19522 <https://issues.apache.org/jira/browse/SPARK-19522> - --executor-memory
> flag doesn't work in local-cluster mode -
> https://github.com/apache/spark/pull/16975 (review in progress by vanzin,
> but PR currently stalled waiting on response) *
> Core:
> SPARK-20025 <https://issues.apache.org/jira/browse/SPARK-20025> - Driver
> fail over will not work, if SPARK_LOCAL* env is set. -
> https://github.com/apache/spark/pull/17357 (waiting on review) *
> PySpark:
> SPARK-19955 <https://issues.apache.org/jira/browse/SPARK-19955> - Update
> run-tests to support conda [ Part of Dropping 2.6 support -- which we
> shouldn't do in a minor release -- but also fixes pip installability tests
> to run in Jenkins ]- PR failing Jenkins (I need to poke this some more,
> but seems like 2.7 support works but some other issues. Maybe slip to 2.2?)
>
> Minor issues:
> Tests:
> SPARK-19612 <https://issues.apache.org/jira/browse/SPARK-19612> - Tests
> failing with timeout - No PR per-se but it seems unrelated to the 2.1.1
> release. It's not targetted for 2.1.1 but listed as affecting 2.1.1 - I'd
> consider explicitly targeting this for 2.2?
> PySpark:
> SPARK-19570 <https://issues.apache.org/jira/browse/SPARK-19570> - Allow
> to disable hive in pyspark shell -
> https://github.com/apache/spark/pull/16906 PR exists but its difficult to
> add automated tests for this (although if SPARK-19955
> <https://issues.apache.org/jira/browse/SPARK-19955> gets in would make
> testing this easier) - no reviewers yet. Possible re-target?*
> Structured Streaming:
> SPARK-19613 <https://issues.apache.org/jira/browse/SPARK-19613> - Flaky
> test: StateStoreRDDSuite.versioning and immutability - It's not targetted
> for 2.1.1 but listed as affecting 2.1.1 - I'd consider explicitly targeting
> this for 2.2?
> ML:
> SPARK-19759 <https://issues.apache.org/jira/browse/SPARK-19759> - ALSModel.predict
> on Dataframes : potential optimization by not using blas - No PR consider
> re-targeting unless someone has a PR waiting in the wings?
>
> Explicitly targeted issues are marked with a *, the remaining issues are
> listed as impacting 2.1.1 and don't have a specific target version set.
>
> Since 2.1.1 continues the 2.1.0 branch, looking at 2.1.0 shows 1 open
> blocker in SQL( SPARK-19983
> <https://issues.apache.org/jira/browse/SPARK-19983> ),
>
> Query string is:
>
> affectedVersion = 2.1.0 AND cf[12310320] is EMPTY AND project = spark AND
> resolution = Unresolved AND priority = targetPriority
>
> Continuing on for unresolved 2.1.0 issues in Major there are 163 (76 of
> them in progress), 65 Minor (26 in progress), and 9 trivial (6 in progress).
>
> I'll be going through the 2.1.0 major issues with open PRs that impact the
> PySpark component and seeing if any of them should be targeted for 2.1.1,
> if anyone from the other components wants to take a look through we might
> find some easy wins to be merged.
>
> Cheers,
>
> Holden :)
>
> --
> Cell : 425-233-8271 <(425)%20233-8271>
> Twitter: https://twitter.com/holdenkarau
>
>
> --
> Cell : 425-233-8271 <(425)%20233-8271>
> Twitter: https://twitter.com/holdenkarau
>
>
>
>
>
> --
> Cell : 425-233-8271 <(425)%20233-8271>
> Twitter: https://twitter.com/holdenkarau
>
Re: Outstanding Spark 2.1.1 issues
Posted by Holden Karau <ho...@pigscanfly.ca>.
I agree with Michael, I think we've got some outstanding issues but none of
them seem like regression from 2.1 so we should be good to start the RC
process.
On Tue, Mar 21, 2017 at 1:41 PM, Michael Armbrust <mi...@databricks.com>
wrote:
> Please speak up if I'm wrong, but none of these seem like critical
> regressions from 2.1. As such I'll start the RC process later today.
>
> On Mon, Mar 20, 2017 at 9:52 PM, Holden Karau <ho...@pigscanfly.ca>
> wrote:
>
>> I'm not super sure it should be a blocker for 2.1.1 -- is it a
>> regression? Maybe we can get TDs input on it?
>>
>> On Mon, Mar 20, 2017 at 8:48 PM Nan Zhu <zh...@gmail.com> wrote:
>>
>>> I think https://issues.apache.org/jira/browse/SPARK-19280 should be a
>>> blocker
>>>
>>> Best,
>>>
>>> Nan
>>>
>>> On Mon, Mar 20, 2017 at 8:18 PM, Felix Cheung <felixcheung_m@hotmail.com
>>> > wrote:
>>>
>>> I've been scrubbing R and think we are tracking 2 issues
>>>
>>> https://issues.apache.org/jira/browse/SPARK-19237
>>>
>>> https://issues.apache.org/jira/browse/SPARK-19925
>>>
>>>
>>>
>>>
>>> ------------------------------
>>> *From:* holden.karau@gmail.com <ho...@gmail.com> on behalf of
>>> Holden Karau <ho...@pigscanfly.ca>
>>> *Sent:* Monday, March 20, 2017 3:12:35 PM
>>> *To:* dev@spark.apache.org
>>> *Subject:* Outstanding Spark 2.1.1 issues
>>>
>>> Hi Spark Developers!
>>>
>>> As we start working on the Spark 2.1.1 release I've been looking at our
>>> outstanding issues still targeted for it. I've tried to break it down by
>>> component so that people in charge of each component can take a quick look
>>> and see if any of these things can/should be re-targeted to 2.2 or 2.1.2 &
>>> the overall list is pretty short (only 9 items - 5 if we only look at
>>> explicitly tagged) :)
>>>
>>> If your working on something for Spark 2.1.1 and it doesn't show up in
>>> this list please speak up now :) We have a lot of issues (including "in
>>> progress") that are listed as impacting 2.1.0, but they aren't targeted for
>>> 2.1.1 - if there is something you are working in their which should be
>>> targeted for 2.1.1 please let us know so it doesn't slip through the cracks.
>>>
>>> The query string I used for looking at the 2.1.1 open issues is:
>>>
>>> ((affectedVersion = 2.1.1 AND cf[12310320] is Empty) OR fixVersion =
>>> 2.1.1 OR cf[12310320] = "2.1.1") AND project = spark AND resolution =
>>> Unresolved ORDER BY priority DESC
>>>
>>> None of the open issues appear to be a regression from 2.1.0, but those
>>> seem more likely to show up during the RC process (thanks in advance to
>>> everyone testing their workloads :)) & generally none of them seem to be
>>>
>>> (Note: the cfs are for Target Version/s field)
>>>
>>> Critical Issues:
>>> SQL:
>>> SPARK-19690 <https://issues.apache.org/jira/browse/SPARK-19690> - Join
>>> a streaming DataFrame with a batch DataFrame may not work - PR
>>> https://github.com/apache/spark/pull/17052 (review in progress by
>>> zsxwing, currently failing Jenkins)*
>>>
>>> Major Issues:
>>> SQL:
>>> SPARK-19035 <https://issues.apache.org/jira/browse/SPARK-19035> - rand()
>>> function in case when cause failed - no outstanding PR (consensus on JIRA
>>> seems to be leaning towards it being a real issue but not necessarily
>>> everyone agrees just yet - maybe we should slip this?)*
>>> Deploy:
>>> SPARK-19522 <https://issues.apache.org/jira/browse/SPARK-19522>
>>> - --executor-memory flag doesn't work in local-cluster mode -
>>> https://github.com/apache/spark/pull/16975 (review in progress by
>>> vanzin, but PR currently stalled waiting on response) *
>>> Core:
>>> SPARK-20025 <https://issues.apache.org/jira/browse/SPARK-20025> - Driver
>>> fail over will not work, if SPARK_LOCAL* env is set. -
>>> https://github.com/apache/spark/pull/17357 (waiting on review) *
>>> PySpark:
>>> SPARK-19955 <https://issues.apache.org/jira/browse/SPARK-19955> -
>>> Update run-tests to support conda [ Part of Dropping 2.6 support -- which
>>> we shouldn't do in a minor release -- but also fixes pip installability
>>> tests to run in Jenkins ]- PR failing Jenkins (I need to poke this some
>>> more, but seems like 2.7 support works but some other issues. Maybe slip to
>>> 2.2?)
>>>
>>> Minor issues:
>>> Tests:
>>> SPARK-19612 <https://issues.apache.org/jira/browse/SPARK-19612> - Tests
>>> failing with timeout - No PR per-se but it seems unrelated to the 2.1.1
>>> release. It's not targetted for 2.1.1 but listed as affecting 2.1.1 - I'd
>>> consider explicitly targeting this for 2.2?
>>> PySpark:
>>> SPARK-19570 <https://issues.apache.org/jira/browse/SPARK-19570> - Allow
>>> to disable hive in pyspark shell - https://github.com/apache/sp
>>> ark/pull/16906 PR exists but its difficult to add automated tests for
>>> this (although if SPARK-19955
>>> <https://issues.apache.org/jira/browse/SPARK-19955> gets in would make
>>> testing this easier) - no reviewers yet. Possible re-target?*
>>> Structured Streaming:
>>> SPARK-19613 <https://issues.apache.org/jira/browse/SPARK-19613> - Flaky
>>> test: StateStoreRDDSuite.versioning and immutability - It's not targetted
>>> for 2.1.1 but listed as affecting 2.1.1 - I'd consider explicitly targeting
>>> this for 2.2?
>>> ML:
>>> SPARK-19759 <https://issues.apache.org/jira/browse/SPARK-19759>
>>> - ALSModel.predict on Dataframes : potential optimization by not using
>>> blas - No PR consider re-targeting unless someone has a PR waiting in the
>>> wings?
>>>
>>> Explicitly targeted issues are marked with a *, the remaining issues are
>>> listed as impacting 2.1.1 and don't have a specific target version set.
>>>
>>> Since 2.1.1 continues the 2.1.0 branch, looking at 2.1.0 shows 1 open
>>> blocker in SQL( SPARK-19983
>>> <https://issues.apache.org/jira/browse/SPARK-19983> ),
>>>
>>> Query string is:
>>>
>>> affectedVersion = 2.1.0 AND cf[12310320] is EMPTY AND project = spark
>>> AND resolution = Unresolved AND priority = targetPriority
>>>
>>> Continuing on for unresolved 2.1.0 issues in Major there are 163 (76 of
>>> them in progress), 65 Minor (26 in progress), and 9 trivial (6 in progress).
>>>
>>> I'll be going through the 2.1.0 major issues with open PRs that impact
>>> the PySpark component and seeing if any of them should be targeted for
>>> 2.1.1, if anyone from the other components wants to take a look through we
>>> might find some easy wins to be merged.
>>>
>>> Cheers,
>>>
>>> Holden :)
>>>
>>> --
>>> Cell : 425-233-8271 <(425)%20233-8271>
>>> Twitter: https://twitter.com/holdenkarau
>>>
>>>
>>> --
>> Cell : 425-233-8271 <(425)%20233-8271>
>> Twitter: https://twitter.com/holdenkarau
>>
>
>
--
Cell : 425-233-8271
Twitter: https://twitter.com/holdenkarau
Re: Outstanding Spark 2.1.1 issues
Posted by Michael Armbrust <mi...@databricks.com>.
Please speak up if I'm wrong, but none of these seem like critical
regressions from 2.1. As such I'll start the RC process later today.
On Mon, Mar 20, 2017 at 9:52 PM, Holden Karau <ho...@pigscanfly.ca> wrote:
> I'm not super sure it should be a blocker for 2.1.1 -- is it a regression?
> Maybe we can get TDs input on it?
>
> On Mon, Mar 20, 2017 at 8:48 PM Nan Zhu <zh...@gmail.com> wrote:
>
>> I think https://issues.apache.org/jira/browse/SPARK-19280 should be a
>> blocker
>>
>> Best,
>>
>> Nan
>>
>> On Mon, Mar 20, 2017 at 8:18 PM, Felix Cheung <fe...@hotmail.com>
>> wrote:
>>
>> I've been scrubbing R and think we are tracking 2 issues
>>
>> https://issues.apache.org/jira/browse/SPARK-19237
>>
>> https://issues.apache.org/jira/browse/SPARK-19925
>>
>>
>>
>>
>> ------------------------------
>> *From:* holden.karau@gmail.com <ho...@gmail.com> on behalf of
>> Holden Karau <ho...@pigscanfly.ca>
>> *Sent:* Monday, March 20, 2017 3:12:35 PM
>> *To:* dev@spark.apache.org
>> *Subject:* Outstanding Spark 2.1.1 issues
>>
>> Hi Spark Developers!
>>
>> As we start working on the Spark 2.1.1 release I've been looking at our
>> outstanding issues still targeted for it. I've tried to break it down by
>> component so that people in charge of each component can take a quick look
>> and see if any of these things can/should be re-targeted to 2.2 or 2.1.2 &
>> the overall list is pretty short (only 9 items - 5 if we only look at
>> explicitly tagged) :)
>>
>> If your working on something for Spark 2.1.1 and it doesn't show up in
>> this list please speak up now :) We have a lot of issues (including "in
>> progress") that are listed as impacting 2.1.0, but they aren't targeted for
>> 2.1.1 - if there is something you are working in their which should be
>> targeted for 2.1.1 please let us know so it doesn't slip through the cracks.
>>
>> The query string I used for looking at the 2.1.1 open issues is:
>>
>> ((affectedVersion = 2.1.1 AND cf[12310320] is Empty) OR fixVersion =
>> 2.1.1 OR cf[12310320] = "2.1.1") AND project = spark AND resolution =
>> Unresolved ORDER BY priority DESC
>>
>> None of the open issues appear to be a regression from 2.1.0, but those
>> seem more likely to show up during the RC process (thanks in advance to
>> everyone testing their workloads :)) & generally none of them seem to be
>>
>> (Note: the cfs are for Target Version/s field)
>>
>> Critical Issues:
>> SQL:
>> SPARK-19690 <https://issues.apache.org/jira/browse/SPARK-19690> - Join
>> a streaming DataFrame with a batch DataFrame may not work - PR
>> https://github.com/apache/spark/pull/17052 (review in progress by
>> zsxwing, currently failing Jenkins)*
>>
>> Major Issues:
>> SQL:
>> SPARK-19035 <https://issues.apache.org/jira/browse/SPARK-19035> - rand()
>> function in case when cause failed - no outstanding PR (consensus on JIRA
>> seems to be leaning towards it being a real issue but not necessarily
>> everyone agrees just yet - maybe we should slip this?)*
>> Deploy:
>> SPARK-19522 <https://issues.apache.org/jira/browse/SPARK-19522>
>> - --executor-memory flag doesn't work in local-cluster mode -
>> https://github.com/apache/spark/pull/16975 (review in progress by
>> vanzin, but PR currently stalled waiting on response) *
>> Core:
>> SPARK-20025 <https://issues.apache.org/jira/browse/SPARK-20025> - Driver
>> fail over will not work, if SPARK_LOCAL* env is set. -
>> https://github.com/apache/spark/pull/17357 (waiting on review) *
>> PySpark:
>> SPARK-19955 <https://issues.apache.org/jira/browse/SPARK-19955> -
>> Update run-tests to support conda [ Part of Dropping 2.6 support -- which
>> we shouldn't do in a minor release -- but also fixes pip installability
>> tests to run in Jenkins ]- PR failing Jenkins (I need to poke this some
>> more, but seems like 2.7 support works but some other issues. Maybe slip to
>> 2.2?)
>>
>> Minor issues:
>> Tests:
>> SPARK-19612 <https://issues.apache.org/jira/browse/SPARK-19612> - Tests
>> failing with timeout - No PR per-se but it seems unrelated to the 2.1.1
>> release. It's not targetted for 2.1.1 but listed as affecting 2.1.1 - I'd
>> consider explicitly targeting this for 2.2?
>> PySpark:
>> SPARK-19570 <https://issues.apache.org/jira/browse/SPARK-19570> - Allow
>> to disable hive in pyspark shell - https://github.com/apache/
>> spark/pull/16906 PR exists but its difficult to add automated tests for
>> this (although if SPARK-19955
>> <https://issues.apache.org/jira/browse/SPARK-19955> gets in would make
>> testing this easier) - no reviewers yet. Possible re-target?*
>> Structured Streaming:
>> SPARK-19613 <https://issues.apache.org/jira/browse/SPARK-19613> - Flaky
>> test: StateStoreRDDSuite.versioning and immutability - It's not targetted
>> for 2.1.1 but listed as affecting 2.1.1 - I'd consider explicitly targeting
>> this for 2.2?
>> ML:
>> SPARK-19759 <https://issues.apache.org/jira/browse/SPARK-19759>
>> - ALSModel.predict on Dataframes : potential optimization by not using
>> blas - No PR consider re-targeting unless someone has a PR waiting in the
>> wings?
>>
>> Explicitly targeted issues are marked with a *, the remaining issues are
>> listed as impacting 2.1.1 and don't have a specific target version set.
>>
>> Since 2.1.1 continues the 2.1.0 branch, looking at 2.1.0 shows 1 open
>> blocker in SQL( SPARK-19983
>> <https://issues.apache.org/jira/browse/SPARK-19983> ),
>>
>> Query string is:
>>
>> affectedVersion = 2.1.0 AND cf[12310320] is EMPTY AND project = spark AND
>> resolution = Unresolved AND priority = targetPriority
>>
>> Continuing on for unresolved 2.1.0 issues in Major there are 163 (76 of
>> them in progress), 65 Minor (26 in progress), and 9 trivial (6 in progress).
>>
>> I'll be going through the 2.1.0 major issues with open PRs that impact
>> the PySpark component and seeing if any of them should be targeted for
>> 2.1.1, if anyone from the other components wants to take a look through we
>> might find some easy wins to be merged.
>>
>> Cheers,
>>
>> Holden :)
>>
>> --
>> Cell : 425-233-8271 <(425)%20233-8271>
>> Twitter: https://twitter.com/holdenkarau
>>
>>
>> --
> Cell : 425-233-8271 <(425)%20233-8271>
> Twitter: https://twitter.com/holdenkarau
>
Re: Outstanding Spark 2.1.1 issues
Posted by Holden Karau <ho...@pigscanfly.ca>.
I'm not super sure it should be a blocker for 2.1.1 -- is it a regression?
Maybe we can get TDs input on it?
On Mon, Mar 20, 2017 at 8:48 PM Nan Zhu <zh...@gmail.com> wrote:
> I think https://issues.apache.org/jira/browse/SPARK-19280 should be a
> blocker
>
> Best,
>
> Nan
>
> On Mon, Mar 20, 2017 at 8:18 PM, Felix Cheung <fe...@hotmail.com>
> wrote:
>
> I've been scrubbing R and think we are tracking 2 issues
>
> https://issues.apache.org/jira/browse/SPARK-19237
>
> https://issues.apache.org/jira/browse/SPARK-19925
>
>
>
>
> ------------------------------
> *From:* holden.karau@gmail.com <ho...@gmail.com> on behalf of
> Holden Karau <ho...@pigscanfly.ca>
> *Sent:* Monday, March 20, 2017 3:12:35 PM
> *To:* dev@spark.apache.org
> *Subject:* Outstanding Spark 2.1.1 issues
>
> Hi Spark Developers!
>
> As we start working on the Spark 2.1.1 release I've been looking at our
> outstanding issues still targeted for it. I've tried to break it down by
> component so that people in charge of each component can take a quick look
> and see if any of these things can/should be re-targeted to 2.2 or 2.1.2 &
> the overall list is pretty short (only 9 items - 5 if we only look at
> explicitly tagged) :)
>
> If your working on something for Spark 2.1.1 and it doesn't show up in
> this list please speak up now :) We have a lot of issues (including "in
> progress") that are listed as impacting 2.1.0, but they aren't targeted for
> 2.1.1 - if there is something you are working in their which should be
> targeted for 2.1.1 please let us know so it doesn't slip through the cracks.
>
> The query string I used for looking at the 2.1.1 open issues is:
>
> ((affectedVersion = 2.1.1 AND cf[12310320] is Empty) OR fixVersion = 2.1.1
> OR cf[12310320] = "2.1.1") AND project = spark AND resolution = Unresolved
> ORDER BY priority DESC
>
> None of the open issues appear to be a regression from 2.1.0, but those
> seem more likely to show up during the RC process (thanks in advance to
> everyone testing their workloads :)) & generally none of them seem to be
>
> (Note: the cfs are for Target Version/s field)
>
> Critical Issues:
> SQL:
> SPARK-19690 <https://issues.apache.org/jira/browse/SPARK-19690> - Join
> a streaming DataFrame with a batch DataFrame may not work - PR
> https://github.com/apache/spark/pull/17052 (review in progress by
> zsxwing, currently failing Jenkins)*
>
> Major Issues:
> SQL:
> SPARK-19035 <https://issues.apache.org/jira/browse/SPARK-19035> - rand()
> function in case when cause failed - no outstanding PR (consensus on JIRA
> seems to be leaning towards it being a real issue but not necessarily
> everyone agrees just yet - maybe we should slip this?)*
> Deploy:
> SPARK-19522 <https://issues.apache.org/jira/browse/SPARK-19522> - --executor-memory
> flag doesn't work in local-cluster mode -
> https://github.com/apache/spark/pull/16975 (review in progress by vanzin,
> but PR currently stalled waiting on response) *
> Core:
> SPARK-20025 <https://issues.apache.org/jira/browse/SPARK-20025> - Driver
> fail over will not work, if SPARK_LOCAL* env is set. -
> https://github.com/apache/spark/pull/17357 (waiting on review) *
> PySpark:
> SPARK-19955 <https://issues.apache.org/jira/browse/SPARK-19955> - Update
> run-tests to support conda [ Part of Dropping 2.6 support -- which we
> shouldn't do in a minor release -- but also fixes pip installability tests
> to run in Jenkins ]- PR failing Jenkins (I need to poke this some more,
> but seems like 2.7 support works but some other issues. Maybe slip to 2.2?)
>
> Minor issues:
> Tests:
> SPARK-19612 <https://issues.apache.org/jira/browse/SPARK-19612> - Tests
> failing with timeout - No PR per-se but it seems unrelated to the 2.1.1
> release. It's not targetted for 2.1.1 but listed as affecting 2.1.1 - I'd
> consider explicitly targeting this for 2.2?
> PySpark:
> SPARK-19570 <https://issues.apache.org/jira/browse/SPARK-19570> - Allow
> to disable hive in pyspark shell -
> https://github.com/apache/spark/pull/16906 PR exists but its difficult to
> add automated tests for this (although if SPARK-19955
> <https://issues.apache.org/jira/browse/SPARK-19955> gets in would make
> testing this easier) - no reviewers yet. Possible re-target?*
> Structured Streaming:
> SPARK-19613 <https://issues.apache.org/jira/browse/SPARK-19613> - Flaky
> test: StateStoreRDDSuite.versioning and immutability - It's not targetted
> for 2.1.1 but listed as affecting 2.1.1 - I'd consider explicitly targeting
> this for 2.2?
> ML:
> SPARK-19759 <https://issues.apache.org/jira/browse/SPARK-19759> - ALSModel.predict
> on Dataframes : potential optimization by not using blas - No PR consider
> re-targeting unless someone has a PR waiting in the wings?
>
> Explicitly targeted issues are marked with a *, the remaining issues are
> listed as impacting 2.1.1 and don't have a specific target version set.
>
> Since 2.1.1 continues the 2.1.0 branch, looking at 2.1.0 shows 1 open
> blocker in SQL( SPARK-19983
> <https://issues.apache.org/jira/browse/SPARK-19983> ),
>
> Query string is:
>
> affectedVersion = 2.1.0 AND cf[12310320] is EMPTY AND project = spark AND
> resolution = Unresolved AND priority = targetPriority
>
> Continuing on for unresolved 2.1.0 issues in Major there are 163 (76 of
> them in progress), 65 Minor (26 in progress), and 9 trivial (6 in progress).
>
> I'll be going through the 2.1.0 major issues with open PRs that impact the
> PySpark component and seeing if any of them should be targeted for 2.1.1,
> if anyone from the other components wants to take a look through we might
> find some easy wins to be merged.
>
> Cheers,
>
> Holden :)
>
> --
> Cell : 425-233-8271 <(425)%20233-8271>
> Twitter: https://twitter.com/holdenkarau
>
>
> --
Cell : 425-233-8271
Twitter: https://twitter.com/holdenkarau
Re: Outstanding Spark 2.1.1 issues
Posted by Nan Zhu <zh...@gmail.com>.
I think https://issues.apache.org/jira/browse/SPARK-19280 should be a
blocker
Best,
Nan
On Mon, Mar 20, 2017 at 8:18 PM, Felix Cheung <fe...@hotmail.com>
wrote:
> I've been scrubbing R and think we are tracking 2 issues
>
> https://issues.apache.org/jira/browse/SPARK-19237
>
> https://issues.apache.org/jira/browse/SPARK-19925
>
>
>
>
> ------------------------------
> *From:* holden.karau@gmail.com <ho...@gmail.com> on behalf of
> Holden Karau <ho...@pigscanfly.ca>
> *Sent:* Monday, March 20, 2017 3:12:35 PM
> *To:* dev@spark.apache.org
> *Subject:* Outstanding Spark 2.1.1 issues
>
> Hi Spark Developers!
>
> As we start working on the Spark 2.1.1 release I've been looking at our
> outstanding issues still targeted for it. I've tried to break it down by
> component so that people in charge of each component can take a quick look
> and see if any of these things can/should be re-targeted to 2.2 or 2.1.2 &
> the overall list is pretty short (only 9 items - 5 if we only look at
> explicitly tagged) :)
>
> If your working on something for Spark 2.1.1 and it doesn't show up in
> this list please speak up now :) We have a lot of issues (including "in
> progress") that are listed as impacting 2.1.0, but they aren't targeted for
> 2.1.1 - if there is something you are working in their which should be
> targeted for 2.1.1 please let us know so it doesn't slip through the cracks.
>
> The query string I used for looking at the 2.1.1 open issues is:
>
> ((affectedVersion = 2.1.1 AND cf[12310320] is Empty) OR fixVersion = 2.1.1
> OR cf[12310320] = "2.1.1") AND project = spark AND resolution = Unresolved
> ORDER BY priority DESC
>
> None of the open issues appear to be a regression from 2.1.0, but those
> seem more likely to show up during the RC process (thanks in advance to
> everyone testing their workloads :)) & generally none of them seem to be
>
> (Note: the cfs are for Target Version/s field)
>
> Critical Issues:
> SQL:
> SPARK-19690 <https://issues.apache.org/jira/browse/SPARK-19690> - Join
> a streaming DataFrame with a batch DataFrame may not work - PR
> https://github.com/apache/spark/pull/17052 (review in progress by
> zsxwing, currently failing Jenkins)*
>
> Major Issues:
> SQL:
> SPARK-19035 <https://issues.apache.org/jira/browse/SPARK-19035> - rand()
> function in case when cause failed - no outstanding PR (consensus on JIRA
> seems to be leaning towards it being a real issue but not necessarily
> everyone agrees just yet - maybe we should slip this?)*
> Deploy:
> SPARK-19522 <https://issues.apache.org/jira/browse/SPARK-19522>
> - --executor-memory flag doesn't work in local-cluster mode -
> https://github.com/apache/spark/pull/16975 (review in progress by vanzin,
> but PR currently stalled waiting on response) *
> Core:
> SPARK-20025 <https://issues.apache.org/jira/browse/SPARK-20025> - Driver
> fail over will not work, if SPARK_LOCAL* env is set. -
> https://github.com/apache/spark/pull/17357 (waiting on review) *
> PySpark:
> SPARK-19955 <https://issues.apache.org/jira/browse/SPARK-19955> - Update
> run-tests to support conda [ Part of Dropping 2.6 support -- which we
> shouldn't do in a minor release -- but also fixes pip installability tests
> to run in Jenkins ]- PR failing Jenkins (I need to poke this some more,
> but seems like 2.7 support works but some other issues. Maybe slip to 2.2?)
>
> Minor issues:
> Tests:
> SPARK-19612 <https://issues.apache.org/jira/browse/SPARK-19612> - Tests
> failing with timeout - No PR per-se but it seems unrelated to the 2.1.1
> release. It's not targetted for 2.1.1 but listed as affecting 2.1.1 - I'd
> consider explicitly targeting this for 2.2?
> PySpark:
> SPARK-19570 <https://issues.apache.org/jira/browse/SPARK-19570> - Allow
> to disable hive in pyspark shell - https://github.com/apache/sp
> ark/pull/16906 PR exists but its difficult to add automated tests for
> this (although if SPARK-19955
> <https://issues.apache.org/jira/browse/SPARK-19955> gets in would make
> testing this easier) - no reviewers yet. Possible re-target?*
> Structured Streaming:
> SPARK-19613 <https://issues.apache.org/jira/browse/SPARK-19613> - Flaky
> test: StateStoreRDDSuite.versioning and immutability - It's not targetted
> for 2.1.1 but listed as affecting 2.1.1 - I'd consider explicitly targeting
> this for 2.2?
> ML:
> SPARK-19759 <https://issues.apache.org/jira/browse/SPARK-19759>
> - ALSModel.predict on Dataframes : potential optimization by not using
> blas - No PR consider re-targeting unless someone has a PR waiting in the
> wings?
>
> Explicitly targeted issues are marked with a *, the remaining issues are
> listed as impacting 2.1.1 and don't have a specific target version set.
>
> Since 2.1.1 continues the 2.1.0 branch, looking at 2.1.0 shows 1 open
> blocker in SQL( SPARK-19983
> <https://issues.apache.org/jira/browse/SPARK-19983> ),
>
> Query string is:
>
> affectedVersion = 2.1.0 AND cf[12310320] is EMPTY AND project = spark AND
> resolution = Unresolved AND priority = targetPriority
>
> Continuing on for unresolved 2.1.0 issues in Major there are 163 (76 of
> them in progress), 65 Minor (26 in progress), and 9 trivial (6 in progress).
>
> I'll be going through the 2.1.0 major issues with open PRs that impact the
> PySpark component and seeing if any of them should be targeted for 2.1.1,
> if anyone from the other components wants to take a look through we might
> find some easy wins to be merged.
>
> Cheers,
>
> Holden :)
>
> --
> Cell : 425-233-8271 <(425)%20233-8271>
> Twitter: https://twitter.com/holdenkarau
>
Re: Outstanding Spark 2.1.1 issues
Posted by Felix Cheung <fe...@hotmail.com>.
I've been scrubbing R and think we are tracking 2 issues
https://issues.apache.org/jira/browse/SPARK-19237
https://issues.apache.org/jira/browse/SPARK-19925
________________________________
From: holden.karau@gmail.com <ho...@gmail.com> on behalf of Holden Karau <ho...@pigscanfly.ca>
Sent: Monday, March 20, 2017 3:12:35 PM
To: dev@spark.apache.org
Subject: Outstanding Spark 2.1.1 issues
Hi Spark Developers!
As we start working on the Spark 2.1.1 release I've been looking at our outstanding issues still targeted for it. I've tried to break it down by component so that people in charge of each component can take a quick look and see if any of these things can/should be re-targeted to 2.2 or 2.1.2 & the overall list is pretty short (only 9 items - 5 if we only look at explicitly tagged) :)
If your working on something for Spark 2.1.1 and it doesn't show up in this list please speak up now :) We have a lot of issues (including "in progress") that are listed as impacting 2.1.0, but they aren't targeted for 2.1.1 - if there is something you are working in their which should be targeted for 2.1.1 please let us know so it doesn't slip through the cracks.
The query string I used for looking at the 2.1.1 open issues is:
((affectedVersion = 2.1.1 AND cf[12310320] is Empty) OR fixVersion = 2.1.1 OR cf[12310320] = "2.1.1") AND project = spark AND resolution = Unresolved ORDER BY priority DESC
None of the open issues appear to be a regression from 2.1.0, but those seem more likely to show up during the RC process (thanks in advance to everyone testing their workloads :)) & generally none of them seem to be
(Note: the cfs are for Target Version/s field)
Critical Issues:
SQL:
SPARK-19690<https://issues.apache.org/jira/browse/SPARK-19690> - Join a streaming DataFrame with a batch DataFrame may not work - PR https://github.com/apache/spark/pull/17052 (review in progress by zsxwing, currently failing Jenkins)*
Major Issues:
SQL:
SPARK-19035<https://issues.apache.org/jira/browse/SPARK-19035> - rand() function in case when cause failed - no outstanding PR (consensus on JIRA seems to be leaning towards it being a real issue but not necessarily everyone agrees just yet - maybe we should slip this?)*
Deploy:
SPARK-19522<https://issues.apache.org/jira/browse/SPARK-19522> - --executor-memory flag doesn't work in local-cluster mode - https://github.com/apache/spark/pull/16975 (review in progress by vanzin, but PR currently stalled waiting on response) *
Core:
SPARK-20025<https://issues.apache.org/jira/browse/SPARK-20025> - Driver fail over will not work, if SPARK_LOCAL* env is set. - https://github.com/apache/spark/pull/17357 (waiting on review) *
PySpark:
SPARK-19955<https://issues.apache.org/jira/browse/SPARK-19955> - Update run-tests to support conda [ Part of Dropping 2.6 support -- which we shouldn't do in a minor release -- but also fixes pip installability tests to run in Jenkins ]- PR failing Jenkins (I need to poke this some more, but seems like 2.7 support works but some other issues. Maybe slip to 2.2?)
Minor issues:
Tests:
SPARK-19612<https://issues.apache.org/jira/browse/SPARK-19612> - Tests failing with timeout - No PR per-se but it seems unrelated to the 2.1.1 release. It's not targetted for 2.1.1 but listed as affecting 2.1.1 - I'd consider explicitly targeting this for 2.2?
PySpark:
SPARK-19570<https://issues.apache.org/jira/browse/SPARK-19570> - Allow to disable hive in pyspark shell - https://github.com/apache/spark/pull/16906 PR exists but its difficult to add automated tests for this (although if SPARK-19955<https://issues.apache.org/jira/browse/SPARK-19955> gets in would make testing this easier) - no reviewers yet. Possible re-target?*
Structured Streaming:
SPARK-19613<https://issues.apache.org/jira/browse/SPARK-19613> - Flaky test: StateStoreRDDSuite.versioning and immutability - It's not targetted for 2.1.1 but listed as affecting 2.1.1 - I'd consider explicitly targeting this for 2.2?
ML:
SPARK-19759<https://issues.apache.org/jira/browse/SPARK-19759> - ALSModel.predict on Dataframes : potential optimization by not using blas - No PR consider re-targeting unless someone has a PR waiting in the wings?
Explicitly targeted issues are marked with a *, the remaining issues are listed as impacting 2.1.1 and don't have a specific target version set.
Since 2.1.1 continues the 2.1.0 branch, looking at 2.1.0 shows 1 open blocker in SQL( SPARK-19983<https://issues.apache.org/jira/browse/SPARK-19983> ),
Query string is:
affectedVersion = 2.1.0 AND cf[12310320] is EMPTY AND project = spark AND resolution = Unresolved AND priority = targetPriority
Continuing on for unresolved 2.1.0 issues in Major there are 163 (76 of them in progress), 65 Minor (26 in progress), and 9 trivial (6 in progress).
I'll be going through the 2.1.0 major issues with open PRs that impact the PySpark component and seeing if any of them should be targeted for 2.1.1, if anyone from the other components wants to take a look through we might find some easy wins to be merged.
Cheers,
Holden :)
--
Cell : 425-233-8271
Twitter: https://twitter.com/holdenkarau