You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by DB Tsai <d_...@apple.com.INVALID> on 2019/02/11 21:17:07 UTC
Time to cut an Apache 2.4.1 release?
Hello all,
I am preparing to cut a new Apache 2.4.1 release as there are many bugs and correctness issues fixed in branch-2.4.
The list of addressed issues are https://issues.apache.org/jira/browse/SPARK-26583?jql=project%20%3D%20SPARK%20AND%20fixVersion%20%3D%202.4.1%20order%20by%20updated%20DESC
Let me know if you have any concern or any PR you would like to get in.
Thanks!
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
Re: Time to cut an Apache 2.4.1 release?
Posted by Reynold Xin <rx...@databricks.com>.
"In addition, please note that new ORC library also contains bug fixes, too." → of course. nobody ships software with only regressions and no features or bug fixes.
On Thu, Feb 14, 2019 at 4:26 PM, Dongjoon Hyun < dongjoon.hyun@gmail.com > wrote:
>
> We have a configuration for this specific case, 'spark.sql.orc.impl'.
>
>
> It was 'hive' by default in 2.3, and 'native' by default 2.4.
>
>
> For the specific Hive cluster, they can set the configuration back to
> 'hive' if they need Spark 2.3 behavior.
>
>
> In addition, please note that new ORC library also contains bug fixes,
> too.
>
>
> Bests,
> Dongjoon.
>
>
>
>
>
>
>
> On Thu, Feb 14, 2019 at 15:30 Reynold Xin < rxin@ databricks. com (
> rxin@databricks.com ) > wrote:
>
>
>> I think in this case we could go either way, but this should be a lesson
>> for the community. A lot of the new people always get confused by
>> pushbacks on dependency upgrades from some of the old timers. The reason
>> "old timers" often push back against dep upgrades is because often new
>> versions of deps haven't gone through a lot of battle testing and for a
>> variety of reasons they might break downstream jobs, be it issues with the
>> dep themselves, or other libraries and implicit assumptions present.
>>
>>
>>
>>
>>
>>
>>
>> On Thu, Feb 14, 2019 at 3:03 PM, DB Tsai < dbtsai@ dbtsai. com. invalid (
>> dbtsai@dbtsai.com.invalid ) > wrote:
>>
>>> Although the bug is in Hive side, this ORC upgrade may break current
>>> running Hive deployment. I agree with Xiao that we might have to revert
>>> ORC upgrade in 2.4 release.
>>>
>>> On Thu, Feb 14, 2019 at 2:15 PM Xiao Li < lixiao@ databricks. com (
>>> lixiao@databricks.com ) > wrote:
>>>
>>>
>>>> In general, for the maintenance releases, we normally avoid upgrading the
>>>> library, if the upgrade is not a must. Parquet upgrade is required for
>>>> fixing the correctness bugs, but ORC upgrades is nice to have but not
>>>> required, I think.
>>>>
>>>>
>>>> We also need to evaluate the impact of the bug of Hive and see whether we
>>>> should do the revert. Spark 2.4 is the last 2.x releases. We need to be
>>>> more conservative, I think.
>>>>
>>>>
>>>> Cheers,
>>>>
>>>>
>>>> Xiao
>>>>
>>>> On Thu, Feb 14, 2019 at 1:40 PM Sean Owen < srowen@ gmail. com (
>>>> srowen@gmail.com ) > wrote:
>>>>
>>>>
>>>>> (That may be so, but it may still be correct to revert a change in Spark
>>>>> if necessary to not be exposed to it in the short term. I have no idea
>>>>> whether that's the right thing here or not, just answering the point about
>>>>> why we'd care about a bug in another project. Also, not clear which Hive
>>>>> releases would fix it? we'd have to get on a newer version to get fixes to
>>>>> a recent version.)
>>>>>
>>>>> On Thu, Feb 14, 2019 at 2:44 PM Dongjoon Hyun < dongjoon. hyun@ gmail. com
>>>>> ( dongjoon.hyun@gmail.com ) > wrote:
>>>>>
>>>>>
>>>>>> Hi, Xiao.
>>>>>>
>>>>>>
>>>>>> Isn't it more natural for Hive community to fix the bug in the old
>>>>>> release.
>>>>>>
>>>>>>
>>>>>> For me, that would be an ideal solution for the whole echo system.
>>>>>>
>>>>>>
>>>>>> Apache ORC project is released independently, too.
>>>>>>
>>>>>>
>>>>>> Bests,
>>>>>> Dongjoon.
>>>>>>
>>>>>> On Thu, Feb 14, 2019 at 09:54 Xiao Li < lixiao@ databricks. com (
>>>>>> lixiao@databricks.com ) > wrote:
>>>>>>
>>>>>>
>>>>>>> Hi, Darcy,
>>>>>>>
>>>>>>>
>>>>>>> Could you show us the Hive JIRA? If the impact is big, we can revert the
>>>>>>> ORC package upgrade. To improve the adoption of Spark 2.4, I think the
>>>>>>> revert is reasonable since the library upgrade of ORC is not a must based
>>>>>>> on my understanding.
>>>>>>>
>>>>>>>
>>>>>>> Cheers,
>>>>>>>
>>>>>>>
>>>>>>> Xiao
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>>
>>>>
>>>>
>>>
>>>
>>
>>
>>> --
>>> - DB Sent from my iPhone
>>>
>>
>>
>
>
Re: Time to cut an Apache 2.4.1 release?
Posted by Dongjoon Hyun <do...@gmail.com>.
We have a configuration for this specific case, 'spark.sql.orc.impl'.
It was 'hive' by default in 2.3, and 'native' by default 2.4.
For the specific Hive cluster, they can set the configuration back to
'hive' if they need Spark 2.3 behavior.
In addition, please note that new ORC library also contains bug fixes, too.
Bests,
Dongjoon.
On Thu, Feb 14, 2019 at 15:30 Reynold Xin <rx...@databricks.com> wrote:
> I think in this case we could go either way, but this should be a lesson
> for the community. A lot of the new people always get confused by pushbacks
> on dependency upgrades from some of the old timers. The reason "old timers"
> often push back against dep upgrades is because often new versions of deps
> haven't gone through a lot of battle testing and for a variety of reasons
> they might break downstream jobs, be it issues with the dep themselves, or
> other libraries and implicit assumptions present.
>
>
>
>
> On Thu, Feb 14, 2019 at 3:03 PM, DB Tsai <db...@dbtsai.com.invalid>
> wrote:
>
>> Although the bug is in Hive side, this ORC upgrade may break current
>> running Hive deployment. I agree with Xiao that we might have to revert ORC
>> upgrade in 2.4 release.
>>
>> On Thu, Feb 14, 2019 at 2:15 PM Xiao Li <li...@databricks.com> wrote:
>>
>> In general, for the maintenance releases, we normally avoid upgrading the
>> library, if the upgrade is not a must. Parquet upgrade is required for
>> fixing the correctness bugs, but ORC upgrades is nice to have but not
>> required, I think.
>>
>> We also need to evaluate the impact of the bug of Hive and see whether we
>> should do the revert. Spark 2.4 is the last 2.x releases. We need to be
>> more conservative, I think.
>>
>> Cheers,
>>
>> Xiao
>>
>> On Thu, Feb 14, 2019 at 1:40 PM Sean Owen <sr...@gmail.com> wrote:
>>
>> (That may be so, but it may still be correct to revert a change in Spark
>> if necessary to not be exposed to it in the short term. I have no idea
>> whether that's the right thing here or not, just answering the point about
>> why we'd care about a bug in another project. Also, not clear which Hive
>> releases would fix it? we'd have to get on a newer version to get fixes to
>> a recent version.)
>>
>> On Thu, Feb 14, 2019 at 2:44 PM Dongjoon Hyun <do...@gmail.com>
>> wrote:
>>
>> Hi, Xiao.
>>
>> Isn't it more natural for Hive community to fix the bug in the old
>> release.
>>
>> For me, that would be an ideal solution for the whole echo system.
>>
>> Apache ORC project is released independently, too.
>>
>> Bests,
>> Dongjoon.
>>
>> On Thu, Feb 14, 2019 at 09:54 Xiao Li <li...@databricks.com> wrote:
>>
>> Hi, Darcy,
>>
>> Could you show us the Hive JIRA? If the impact is big, we can revert the
>> ORC package upgrade. To improve the adoption of Spark 2.4, I think the
>> revert is reasonable since the library upgrade of ORC is not a must based
>> on my understanding.
>>
>> Cheers,
>>
>> Xiao
>>
>>
>>
>> --
>>
>> --
>> - DB Sent from my iPhone
>>
>
>
Re: Time to cut an Apache 2.4.1 release?
Posted by Reynold Xin <rx...@databricks.com>.
I think in this case we could go either way, but this should be a lesson for the community. A lot of the new people always get confused by pushbacks on dependency upgrades from some of the old timers. The reason "old timers" often push back against dep upgrades is because often new versions of deps haven't gone through a lot of battle testing and for a variety of reasons they might break downstream jobs, be it issues with the dep themselves, or other libraries and implicit assumptions present.
On Thu, Feb 14, 2019 at 3:03 PM, DB Tsai < dbtsai@dbtsai.com.invalid > wrote:
>
> Although the bug is in Hive side, this ORC upgrade may break current
> running Hive deployment. I agree with Xiao that we might have to revert
> ORC upgrade in 2.4 release.
>
> On Thu, Feb 14, 2019 at 2:15 PM Xiao Li < lixiao@ databricks. com (
> lixiao@databricks.com ) > wrote:
>
>
>> In general, for the maintenance releases, we normally avoid upgrading the
>> library, if the upgrade is not a must. Parquet upgrade is required for
>> fixing the correctness bugs, but ORC upgrades is nice to have but not
>> required, I think.
>>
>>
>> We also need to evaluate the impact of the bug of Hive and see whether we
>> should do the revert. Spark 2.4 is the last 2.x releases. We need to be
>> more conservative, I think.
>>
>>
>> Cheers,
>>
>>
>> Xiao
>>
>> On Thu, Feb 14, 2019 at 1:40 PM Sean Owen < srowen@ gmail. com (
>> srowen@gmail.com ) > wrote:
>>
>>
>>> (That may be so, but it may still be correct to revert a change in Spark
>>> if necessary to not be exposed to it in the short term. I have no idea
>>> whether that's the right thing here or not, just answering the point about
>>> why we'd care about a bug in another project. Also, not clear which Hive
>>> releases would fix it? we'd have to get on a newer version to get fixes to
>>> a recent version.)
>>>
>>> On Thu, Feb 14, 2019 at 2:44 PM Dongjoon Hyun < dongjoon. hyun@ gmail. com
>>> ( dongjoon.hyun@gmail.com ) > wrote:
>>>
>>>
>>>> Hi, Xiao.
>>>>
>>>>
>>>> Isn't it more natural for Hive community to fix the bug in the old
>>>> release.
>>>>
>>>>
>>>> For me, that would be an ideal solution for the whole echo system.
>>>>
>>>>
>>>> Apache ORC project is released independently, too.
>>>>
>>>>
>>>> Bests,
>>>> Dongjoon.
>>>>
>>>> On Thu, Feb 14, 2019 at 09:54 Xiao Li < lixiao@ databricks. com (
>>>> lixiao@databricks.com ) > wrote:
>>>>
>>>>
>>>>> Hi, Darcy,
>>>>>
>>>>>
>>>>> Could you show us the Hive JIRA? If the impact is big, we can revert the
>>>>> ORC package upgrade. To improve the adoption of Spark 2.4, I think the
>>>>> revert is reasonable since the library upgrade of ORC is not a must based
>>>>> on my understanding.
>>>>>
>>>>>
>>>>> Cheers,
>>>>>
>>>>>
>>>>> Xiao
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>
>>>
>>
>>
>>
>>
>> --
>>
>>
>>
>
> --
> - DB Sent from my iPhone
>
Re: Time to cut an Apache 2.4.1 release?
Posted by DB Tsai <db...@dbtsai.com.INVALID>.
Although the bug is in Hive side, this ORC upgrade may break current
running Hive deployment. I agree with Xiao that we might have to revert ORC
upgrade in 2.4 release.
On Thu, Feb 14, 2019 at 2:15 PM Xiao Li <li...@databricks.com> wrote:
> In general, for the maintenance releases, we normally avoid upgrading the
> library, if the upgrade is not a must. Parquet upgrade is required for
> fixing the correctness bugs, but ORC upgrades is nice to have but not
> required, I think.
>
> We also need to evaluate the impact of the bug of Hive and see whether we
> should do the revert. Spark 2.4 is the last 2.x releases. We need to be
> more conservative, I think.
>
> Cheers,
>
> Xiao
>
> On Thu, Feb 14, 2019 at 1:40 PM Sean Owen <sr...@gmail.com> wrote:
>
>> (That may be so, but it may still be correct to revert a change in Spark
>> if necessary to not be exposed to it in the short term. I have no idea
>> whether that's the right thing here or not, just answering the point about
>> why we'd care about a bug in another project. Also, not clear which Hive
>> releases would fix it? we'd have to get on a newer version to get fixes to
>> a recent version.)
>>
>> On Thu, Feb 14, 2019 at 2:44 PM Dongjoon Hyun <do...@gmail.com>
>> wrote:
>>
>>> Hi, Xiao.
>>>
>>> Isn't it more natural for Hive community to fix the bug in the old
>>> release.
>>>
>>> For me, that would be an ideal solution for the whole echo system.
>>>
>>> Apache ORC project is released independently, too.
>>>
>>> Bests,
>>> Dongjoon.
>>>
>>> On Thu, Feb 14, 2019 at 09:54 Xiao Li <li...@databricks.com> wrote:
>>>
>>>> Hi, Darcy,
>>>>
>>>> Could you show us the Hive JIRA? If the impact is big, we can revert
>>>> the ORC package upgrade. To improve the adoption of Spark 2.4, I think the
>>>> revert is reasonable since the library upgrade of ORC is not a must based
>>>> on my understanding.
>>>>
>>>> Cheers,
>>>>
>>>> Xiao
>>>>
>>>>
>
> --
> [image:
> https://databricks.com/sparkaisummit/north-america?utm_source=email&utm_medium=signature]
>
--
- DB Sent from my iPhone
Re: Time to cut an Apache 2.4.1 release?
Posted by Xiao Li <li...@databricks.com>.
In general, for the maintenance releases, we normally avoid upgrading the
library, if the upgrade is not a must. Parquet upgrade is required for
fixing the correctness bugs, but ORC upgrades is nice to have but not
required, I think.
We also need to evaluate the impact of the bug of Hive and see whether we
should do the revert. Spark 2.4 is the last 2.x releases. We need to be
more conservative, I think.
Cheers,
Xiao
On Thu, Feb 14, 2019 at 1:40 PM Sean Owen <sr...@gmail.com> wrote:
> (That may be so, but it may still be correct to revert a change in Spark
> if necessary to not be exposed to it in the short term. I have no idea
> whether that's the right thing here or not, just answering the point about
> why we'd care about a bug in another project. Also, not clear which Hive
> releases would fix it? we'd have to get on a newer version to get fixes to
> a recent version.)
>
> On Thu, Feb 14, 2019 at 2:44 PM Dongjoon Hyun <do...@gmail.com>
> wrote:
>
>> Hi, Xiao.
>>
>> Isn't it more natural for Hive community to fix the bug in the old
>> release.
>>
>> For me, that would be an ideal solution for the whole echo system.
>>
>> Apache ORC project is released independently, too.
>>
>> Bests,
>> Dongjoon.
>>
>> On Thu, Feb 14, 2019 at 09:54 Xiao Li <li...@databricks.com> wrote:
>>
>>> Hi, Darcy,
>>>
>>> Could you show us the Hive JIRA? If the impact is big, we can revert the
>>> ORC package upgrade. To improve the adoption of Spark 2.4, I think the
>>> revert is reasonable since the library upgrade of ORC is not a must based
>>> on my understanding.
>>>
>>> Cheers,
>>>
>>> Xiao
>>>
>>>
--
[image:
https://databricks.com/sparkaisummit/north-america?utm_source=email&utm_medium=signature]
Re: Time to cut an Apache 2.4.1 release?
Posted by Sean Owen <sr...@gmail.com>.
(That may be so, but it may still be correct to revert a change in Spark if
necessary to not be exposed to it in the short term. I have no idea whether
that's the right thing here or not, just answering the point about why we'd
care about a bug in another project. Also, not clear which Hive releases
would fix it? we'd have to get on a newer version to get fixes to a recent
version.)
On Thu, Feb 14, 2019 at 2:44 PM Dongjoon Hyun <do...@gmail.com>
wrote:
> Hi, Xiao.
>
> Isn't it more natural for Hive community to fix the bug in the old release.
>
> For me, that would be an ideal solution for the whole echo system.
>
> Apache ORC project is released independently, too.
>
> Bests,
> Dongjoon.
>
> On Thu, Feb 14, 2019 at 09:54 Xiao Li <li...@databricks.com> wrote:
>
>> Hi, Darcy,
>>
>> Could you show us the Hive JIRA? If the impact is big, we can revert the
>> ORC package upgrade. To improve the adoption of Spark 2.4, I think the
>> revert is reasonable since the library upgrade of ORC is not a must based
>> on my understanding.
>>
>> Cheers,
>>
>> Xiao
>>
>>
Re: Time to cut an Apache 2.4.1 release?
Posted by Dongjoon Hyun <do...@gmail.com>.
Hi, Xiao.
Isn't it more natural for Hive community to fix the bug in the old release.
For me, that would be an ideal solution for the whole echo system.
Apache ORC project is released independently, too.
Bests,
Dongjoon.
On Thu, Feb 14, 2019 at 09:54 Xiao Li <li...@databricks.com> wrote:
> Hi, Darcy,
>
> Could you show us the Hive JIRA? If the impact is big, we can revert the
> ORC package upgrade. To improve the adoption of Spark 2.4, I think the
> revert is reasonable since the library upgrade of ORC is not a must based
> on my understanding.
>
> Cheers,
>
> Xiao
>
> On Thu, Feb 14, 2019 at 9:13 AM Darcy Shen <sa...@zoho.com.invalid>
> wrote:
>
>>
>> Well, it is not a bug for Spark 2.4 but a bug for Hive 2.1.1 . My
>> colleague will report it on the Spark JIRA later.
>>
>> Presto works fine when reading the ORC table created by Spark 2.4.
>>
>> We've decided to fix it in Hive 2.1.1 . Since Hive 2.1.1 is widely used,
>> I suggest that we should keep a good interop with "legacy" hive. Otherwise,
>> newbies will complain about it and decide not to use Spark SQL for CTAS
>> scenarios. And for now, there are several cases Apache Hive does better
>> than Spark SQL.
>>
>> We are on the way replacing Hive 2.1.1 with Spark 2.4.x (SQL) and won't
>> consider upgrading Hive.
>>
>> ---- On Thu, 14 Feb 2019 21:36:08 +0800 *Wenchen Fan
>> <cloud0fan@gmail.com <cl...@gmail.com>>* wrote ----
>>
>> Do you know which bug ORC 1.5.2 introduced? Or is it because Hive uses a
>> legacy version of ORC which has a bug?
>>
>> On Thu, Feb 14, 2019 at 2:35 PM Darcy Shen <sa...@zoho.com.invalid>
>> wrote:
>>
>>
>>>
>>> We found that ORC table created by Spark 2.4 failed to be read by Hive
>>> 2.1.1.
>>>
>>>
>>> spark-sql -e 'CREATE TABLE tmp.orcTable2 USING orc AS SELECT * FROM
>>> tmp.orcTable1 limit 10;'
>>>
>>> hive -e 'select * from tmp.orcTable2'
>>>
>>> The ERROR messages by Hive:
>>>
>>> Failed with exception java.io.IOException:java.lang.RuntimeException:
>>> ORC split generation failed with exception:
>>> java.lang.ArrayIndexOutOfBoundsException: 6
>>>
>>> And Spark 2.3.2 (or below) works fine.
>>>
>>> I think we should git revert [SPARK-24576][BUILD] Upgrade Apache ORC to
>>> 1.5.2 by Dongjoon Hyun
>>>
>>>
>>> ---- On Tue, 12 Feb 2019 16:56:09 +0800 *Dongjin Lee
>>> <dongjin@apache.org <do...@apache.org>>* wrote ----
>>>
>>> > SPARK-23539 is a non-trivial improvement, so probably would not be
>>> back-ported to 2.4.x.
>>>
>>> Got it. It seems reasonable.
>>>
>>> Committers:
>>>
>>> Please don't omit SPARK-23539 from 2.5.0. Kafka community needs this
>>> feature.
>>>
>>> Thanks,
>>> Dongjin
>>>
>>> On Tue, Feb 12, 2019 at 1:50 PM Takeshi Yamamuro <li...@gmail.com>
>>> wrote:
>>>
>>>
>>>
>>> --
>>> *Dongjin Lee*
>>>
>>> *A hitchhiker in the mathematical world.*
>>>
>>> *github: <http://goog_969573159/>github.com/dongjinleekr
>>> <https://github.com/dongjinleekr>*
>>> linkedin: kr.linkedin.com/in/dongjinleekr
>>> speakerdeck: speakerdeck.com/dongjin
>>>
>>>> +1, too.
>>>> branch-2.4 accumulates too many commits..:
>>>>
>>>> https://github.com/apache/spark/compare/0a4c03f7d084f1d2aa48673b99f3b9496893ce8d...af3c7111efd22907976fc8bbd7810fe3cfd92092
>>>>
>>>> On Tue, Feb 12, 2019 at 12:36 PM Dongjoon Hyun <do...@apache.org>
>>>> wrote:
>>>>
>>>>> Thank you, DB.
>>>>>
>>>>> +1, Yes. It's time for preparing 2.4.1 release.
>>>>>
>>>>> Bests,
>>>>> Dongjoon.
>>>>>
>>>>> On 2019/02/12 03:16:05, Sean Owen <sr...@gmail.com> wrote:
>>>>> > I support a 2.4.1 release now, yes.
>>>>> >
>>>>> > SPARK-23539 is a non-trivial improvement, so probably would not be
>>>>> > back-ported to 2.4.x.SPARK-26154 does look like a bug whose fix could
>>>>> > be back-ported, but that's a big change. I wouldn't hold up 2.4.1 for
>>>>> > it, but it could go in if otherwise ready.
>>>>> >
>>>>> >
>>>>> > On Mon, Feb 11, 2019 at 5:20 PM Dongjin Lee <do...@apache.org>
>>>>> wrote:
>>>>> > >
>>>>> > > Hi DB,
>>>>> > >
>>>>> > > Could you add SPARK-23539[^1] into 2.4.1? I opened the PR[^2] a
>>>>> little bit ago, but it has not included in 2.3.0 nor get enough review.
>>>>> > >
>>>>> > > Thanks,
>>>>> > > Dongjin
>>>>> > >
>>>>> > > [^1]: https://issues.apache.org/jira/browse/SPARK-23539
>>>>> > > [^2]: https://github.com/apache/spark/pull/22282
>>>>> > >
>>>>> > > On Tue, Feb 12, 2019 at 6:28 AM Jungtaek Lim <ka...@gmail.com>
>>>>> wrote:
>>>>> > >>
>>>>> > >> Given SPARK-26154 [1] is a correctness issue and PR [2] is
>>>>> submitted, I hope it can be reviewed and included within Spark 2.4.1 -
>>>>> otherwise it will be a long-live correctness issue.
>>>>> > >>
>>>>> > >> Thanks,
>>>>> > >> Jungtaek Lim (HeartSaVioR)
>>>>> > >>
>>>>> > >> 1. https://issues.apache.org/jira/browse/SPARK-26154
>>>>> > >> 2. https://github.com/apache/spark/pull/23634
>>>>> > >>
>>>>> > >>
>>>>> > >> 2019년 2월 12일 (화) 오전 6:17, DB Tsai <d_...@apple.com.invalid>님이
>>>>> 작성:
>>>>> > >>>
>>>>> > >>> Hello all,
>>>>> > >>>
>>>>> > >>> I am preparing to cut a new Apache 2.4.1 release as there are
>>>>> many bugs and correctness issues fixed in branch-2.4.
>>>>> > >>>
>>>>> > >>> The list of addressed issues are
>>>>> https://issues.apache.org/jira/browse/SPARK-26583?jql=project%20%3D%20SPARK%20AND%20fixVersion%20%3D%202.4.1%20order%20by%20updated%20DESC
>>>>> > >>>
>>>>> > >>> Let me know if you have any concern or any PR you would like to
>>>>> get in.
>>>>> > >>>
>>>>> > >>> Thanks!
>>>>> > >>>
>>>>> > >>>
>>>>> ---------------------------------------------------------------------
>>>>> > >>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>>> > >>>
>>>>> > >
>>>>> > >
>>>>> > > --
>>>>> > > Dongjin Lee
>>>>> > >
>>>>> > > A hitchhiker in the mathematical world.
>>>>> > >
>>>>> > > github: github.com/dongjinleekr
>>>>> > > linkedin: kr.linkedin.com/in/dongjinleekr
>>>>> > > speakerdeck: speakerdeck.com/dongjin
>>>>> >
>>>>> > ---------------------------------------------------------------------
>>>>> > To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>>> >
>>>>> >
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>>>
>>>>>
>>>>
>>>> --
>>>> ---
>>>> Takeshi Yamamuro
>>>>
>>>
>>>
>>
>>
>
> --
> [image:
> https://databricks.com/sparkaisummit/north-america?utm_source=email&utm_medium=signature]
>
Re: Time to cut an Apache 2.4.1 release?
Posted by Xiao Li <ga...@gmail.com>.
Hi, Haibo and Darcy,
Thanks! The Spark community is unable to help based on the latest info you
provided. Could you please submit a PR to help update the doc?
https://spark.apache.org/docs/latest/sql-data-sources-orc.html
Thanks,
Xiao
Haibo <ha...@163.com> 于2019年2月15日周五 上午3:40写道：
> Hi All,
>
> Sorry for all, I found spark2.3.2 used hive orc writer in my environment
> before, and spark 2.3.2(orc 1.4.4) has the same issue.
>
> I think hive 2.1.1's orc reader which implemented by hive self is
> non-forward compatible caused this issue instead of orc new feature.
>
> Firstly, orc have 2 'Version'(file format version), hive 2.1.1 and spark
> 2.3.2, 2.4.0 use the same Version valued 0.12.
>
> Sencondly, orc of hive 2.1.1 has 4 'WriterVersion' but spark 2.4 (orc
> 1.5.2) has 6. Orc writer will write the latest WriteVersion value into orc
> file. so hive 2.1.1 orc reader cant reader file created by spark 2.4 and
> throws exception with index out of range.
>
> Comments about Version and 'WriterVersion' show below:
> hive 2.1.1 OrcFile.java:
> https://github.com/apache/hive/blob/rel/release-2.1.1/orc/src/java/org/apache/orc/OrcFile.java
> orc 1.5.2 (spark 2.4) OrcFile.java:
> https://github.com/apache/orc/blob/rel/release-1.5.2/java/core/src/java/org/apache/orc/OrcFile.java
>
> On 02/15/2019 12:25，Wenchen Fan<cl...@gmail.com> <cl...@gmail.com>
> wrote：
>
> Hi Haibo,
>
> Did you verify that reverting SPARK-24576 works for you? If the problem
> was caused by a new feature in ORC 1.5, then reverting the upgrade doesn't
> help, we need to switch back to the hive orc reader.
>
> On Fri, Feb 15, 2019 at 10:50 AM Xiao Li <li...@databricks.com> wrote:
>
>> Hi, Haibo,
>>
>> Originally, I thought this was caused by the fixes in ORC 1.5.2.
>>
>> Could you please share your JIRA and description in Apache Hive? We need
>> to evaluate the impact.
>>
>> Thanks!
>>
>> Xiao
>>
>>
>>
>>
>> On Thu, Feb 14, 2019 at 6:45 PM 海博 <ha...@163.com> wrote:
>>
>>> Hi, Xiao.
>>> I'm Darcy's colleague who found this issue.
>>>
>>> I compared the underlying file of table which created by spark 2.4 (orc
>>> 1.5.2) and spark 2.3.2 (orc 1.4.4), and found that the format of ORC had
>>> changed.
>>> I checked the orc release log and found this new feature(
>>> https://issues.apache.org/jira/browse/ORC-305) of ORC 1.5 change orc
>>> file format but I was not sure whether they were caused by this.
>>>
>>> Secondly, the full stack trace by hive 2.1.1 show following:
>>>
>>> Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 6
>>> at org.apache.orc.OrcFile$WriterVersion.from(OrcFile.java:145)
>>> at org.apache.orc.impl.OrcTail.getWriterVersion(OrcTail.java:74)
>>> at org.apache.orc.impl.ReaderImpl.<init>(ReaderImpl.java:385)
>>> at org.apache.orc.OrcFile.createReader(OrcFile.java:222)
>>> at org.apache.orc.tools.FileDump.getReader(FileDump.java:255)
>>> at org.apache.orc.tools.FileDump.printMetaDataImpl(FileDump.java:328)
>>> at org.apache.orc.tools.FileDump.printMetaData(FileDump.java:307)
>>> at org.apache.orc.tools.FileDump.main(FileDump.java:154)
>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>> at java.lang.reflect.Method.invoke(Method.java:498)
>>> at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
>>> at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
>>>
>>> On 02/15/2019 01:53，Xiao Li<li...@databricks.com>
>>> <li...@databricks.com> wrote：
>>>
>>> Hi, Darcy,
>>>
>>> Could you show us the Hive JIRA? If the impact is big, we can revert the
>>> ORC package upgrade. To improve the adoption of Spark 2.4, I think the
>>> revert is reasonable since the library upgrade of ORC is not a must based
>>> on my understanding.
>>>
>>> Cheers,
>>>
>>> Xiao
>>>
>>> On Thu, Feb 14, 2019 at 9:13 AM Darcy Shen <sa...@zoho.com.invalid>
>>> wrote:
>>>
>>>>
>>>> Well, it is not a bug for Spark 2.4 but a bug for Hive 2.1.1 . My
>>>> colleague will report it on the Spark JIRA later.
>>>>
>>>> Presto works fine when reading the ORC table created by Spark 2.4.
>>>>
>>>> We've decided to fix it in Hive 2.1.1 . Since Hive 2.1.1 is widely
>>>> used, I suggest that we should keep a good interop with "legacy" hive.
>>>> Otherwise, newbies will complain about it and decide not to use Spark SQL
>>>> for CTAS scenarios. And for now, there are several cases Apache Hive does
>>>> better than Spark SQL.
>>>>
>>>> We are on the way replacing Hive 2.1.1 with Spark 2.4.x (SQL) and won't
>>>> consider upgrading Hive.
>>>>
>>>> ---- On Thu, 14 Feb 2019 21:36:08 +0800 *Wenchen Fan
>>>> <cloud0fan@gmail.com <cl...@gmail.com>>* wrote ----
>>>>
>>>> Do you know which bug ORC 1.5.2 introduced? Or is it because Hive uses
>>>> a legacy version of ORC which has a bug?
>>>>
>>>> On Thu, Feb 14, 2019 at 2:35 PM Darcy Shen <sa...@zoho.com.invalid>
>>>> wrote:
>>>>
>>>>
>>>>>
>>>>> We found that ORC table created by Spark 2.4 failed to be read by Hive
>>>>> 2.1.1.
>>>>>
>>>>>
>>>>> spark-sql -e 'CREATE TABLE tmp.orcTable2 USING orc AS SELECT * FROM
>>>>> tmp.orcTable1 limit 10;'
>>>>>
>>>>> hive -e 'select * from tmp.orcTable2'
>>>>>
>>>>> The ERROR messages by Hive:
>>>>>
>>>>> Failed with exception java.io.IOException:java.lang.RuntimeException:
>>>>> ORC split generation failed with exception:
>>>>> java.lang.ArrayIndexOutOfBoundsException: 6
>>>>>
>>>>> And Spark 2.3.2 (or below) works fine.
>>>>>
>>>>> I think we should git revert [SPARK-24576][BUILD] Upgrade Apache ORC
>>>>> to 1.5.2 by Dongjoon Hyun
>>>>>
>>>>>
>>>>> ---- On Tue, 12 Feb 2019 16:56:09 +0800 *Dongjin Lee
>>>>> <dongjin@apache.org <do...@apache.org>>* wrote ----
>>>>>
>>>>> > SPARK-23539 is a non-trivial improvement, so probably would not be
>>>>> back-ported to 2.4.x.
>>>>>
>>>>> Got it. It seems reasonable.
>>>>>
>>>>> Committers:
>>>>>
>>>>> Please don't omit SPARK-23539 from 2.5.0. Kafka community needs this
>>>>> feature.
>>>>>
>>>>> Thanks,
>>>>> Dongjin
>>>>>
>>>>> On Tue, Feb 12, 2019 at 1:50 PM Takeshi Yamamuro <
>>>>> linguin.m.s@gmail.com> wrote:
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> *Dongjin Lee*
>>>>>
>>>>> *A hitchhiker in the mathematical world.*
>>>>>
>>>>> *github: <http://goog_969573159/>github.com/dongjinleekr
>>>>> <https://github.com/dongjinleekr>*
>>>>> linkedin: kr.linkedin.com/in/dongjinleekr
>>>>> speakerdeck: speakerdeck.com/dongjin
>>>>>
>>>>>> +1, too.
>>>>>> branch-2.4 accumulates too many commits..:
>>>>>>
>>>>>> https://github.com/apache/spark/compare/0a4c03f7d084f1d2aa48673b99f3b9496893ce8d...af3c7111efd22907976fc8bbd7810fe3cfd92092
>>>>>>
>>>>>> On Tue, Feb 12, 2019 at 12:36 PM Dongjoon Hyun <do...@apache.org>
>>>>>> wrote:
>>>>>>
>>>>>>> Thank you, DB.
>>>>>>>
>>>>>>> +1, Yes. It's time for preparing 2.4.1 release.
>>>>>>>
>>>>>>> Bests,
>>>>>>> Dongjoon.
>>>>>>>
>>>>>>> On 2019/02/12 03:16:05, Sean Owen <sr...@gmail.com> wrote:
>>>>>>> > I support a 2.4.1 release now, yes.
>>>>>>> >
>>>>>>> > SPARK-23539 is a non-trivial improvement, so probably would not be
>>>>>>> > back-ported to 2.4.x.SPARK-26154 does look like a bug whose fix
>>>>>>> could
>>>>>>> > be back-ported, but that's a big change. I wouldn't hold up 2.4.1
>>>>>>> for
>>>>>>> > it, but it could go in if otherwise ready.
>>>>>>> >
>>>>>>> >
>>>>>>> > On Mon, Feb 11, 2019 at 5:20 PM Dongjin Lee <do...@apache.org>
>>>>>>> wrote:
>>>>>>> > >
>>>>>>> > > Hi DB,
>>>>>>> > >
>>>>>>> > > Could you add SPARK-23539[^1] into 2.4.1? I opened the PR[^2] a
>>>>>>> little bit ago, but it has not included in 2.3.0 nor get enough review.
>>>>>>> > >
>>>>>>> > > Thanks,
>>>>>>> > > Dongjin
>>>>>>> > >
>>>>>>> > > [^1]: https://issues.apache.org/jira/browse/SPARK-23539
>>>>>>> > > [^2]: https://github.com/apache/spark/pull/22282
>>>>>>> > >
>>>>>>> > > On Tue, Feb 12, 2019 at 6:28 AM Jungtaek Lim <ka...@gmail.com>
>>>>>>> wrote:
>>>>>>> > >>
>>>>>>> > >> Given SPARK-26154 [1] is a correctness issue and PR [2] is
>>>>>>> submitted, I hope it can be reviewed and included within Spark 2.4.1 -
>>>>>>> otherwise it will be a long-live correctness issue.
>>>>>>> > >>
>>>>>>> > >> Thanks,
>>>>>>> > >> Jungtaek Lim (HeartSaVioR)
>>>>>>> > >>
>>>>>>> > >> 1. https://issues.apache.org/jira/browse/SPARK-26154
>>>>>>> > >> 2. https://github.com/apache/spark/pull/23634
>>>>>>> > >>
>>>>>>> > >>
>>>>>>> > >> 2019년 2월 12일 (화) 오전 6:17, DB Tsai <d_...@apple.com.invalid>님이
>>>>>>> 작성:
>>>>>>> > >>>
>>>>>>> > >>> Hello all,
>>>>>>> > >>>
>>>>>>> > >>> I am preparing to cut a new Apache 2.4.1 release as there are
>>>>>>> many bugs and correctness issues fixed in branch-2.4.
>>>>>>> > >>>
>>>>>>> > >>> The list of addressed issues are
>>>>>>> https://issues.apache.org/jira/browse/SPARK-26583?jql=project%20%3D%20SPARK%20AND%20fixVersion%20%3D%202.4.1%20order%20by%20updated%20DESC
>>>>>>> > >>>
>>>>>>> > >>> Let me know if you have any concern or any PR you would like
>>>>>>> to get in.
>>>>>>> > >>>
>>>>>>> > >>> Thanks!
>>>>>>> > >>>
>>>>>>> > >>>
>>>>>>> ---------------------------------------------------------------------
>>>>>>> > >>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>>>>> > >>>
>>>>>>> > >
>>>>>>> > >
>>>>>>> > > --
>>>>>>> > > Dongjin Lee
>>>>>>> > >
>>>>>>> > > A hitchhiker in the mathematical world.
>>>>>>> > >
>>>>>>> > > github: github.com/dongjinleekr
>>>>>>> > > linkedin: kr.linkedin.com/in/dongjinleekr
>>>>>>> > > speakerdeck: speakerdeck.com/dongjin
>>>>>>> >
>>>>>>> >
>>>>>>> ---------------------------------------------------------------------
>>>>>>> > To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>>>>> >
>>>>>>> >
>>>>>>>
>>>>>>> ---------------------------------------------------------------------
>>>>>>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> --
>>>>>> ---
>>>>>> Takeshi Yamamuro
>>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>
>>> --
>>> [image:
>>> https://databricks.com/sparkaisummit/north-america?utm_source=email&utm_medium=signature]
>>>
>>>
>>
>> --
>> [image:
>> https://databricks.com/sparkaisummit/north-america?utm_source=email&utm_medium=signature]
>>
>
Re: Time to cut an Apache 2.4.1 release?
Posted by Haibo <ha...@163.com>.
Hi All,
Sorry for all, I found spark2.3.2 used hive orc writer in my environment before, and spark 2.3.2(orc 1.4.4) has the same issue.
I think hive 2.1.1's orc reader which implemented by hive self is non-forward compatible caused this issue instead of orc new feature.
Firstly, orc have 2 'Version'(file format version), hive 2.1.1 and spark 2.3.2, 2.4.0 use the same Version valued 0.12.
Sencondly, orc of hive 2.1.1 has 4 'WriterVersion' but spark 2.4 (orc 1.5.2) has 6. Orc writer will write the latest WriteVersion value into orc file. so hive 2.1.1 orc reader cant reader file created by spark 2.4 and throws exception with index out of range.
Comments about Version and 'WriterVersion' show below:
hive 2.1.1 OrcFile.java: https://github.com/apache/hive/blob/rel/release-2.1.1/orc/src/java/org/apache/orc/OrcFile.java
orc 1.5.2 (spark 2.4) OrcFile.java: https://github.com/apache/orc/blob/rel/release-1.5.2/java/core/src/java/org/apache/orc/OrcFile.java
On 02/15/2019 12:25，Wenchen Fan<cl...@gmail.com> wrote：
Hi Haibo,
Did you verify that reverting SPARK-24576 works for you? If the problem was caused by a new feature in ORC 1.5, then reverting the upgrade doesn't help, we need to switch back to the hive orc reader.
On Fri, Feb 15, 2019 at 10:50 AM Xiao Li <li...@databricks.com> wrote:
Hi, Haibo,
Originally, I thought this was caused by the fixes in ORC 1.5.2.
Could you please share your JIRA and description in Apache Hive? We need to evaluate the impact.
Thanks!
Xiao
On Thu, Feb 14, 2019 at 6:45 PM 海博 <ha...@163.com> wrote:
Hi, Xiao.
I'm Darcy's colleague who found this issue.
I compared the underlying file of table which created by spark 2.4 (orc 1.5.2) and spark 2.3.2 (orc 1.4.4), and found that the format of ORC had changed.
I checked the orc release log and found this new feature(https://issues.apache.org/jira/browse/ORC-305) of ORC 1.5 change orc file format but I was not sure whether they were caused by this.
Secondly, the full stack trace by hive 2.1.1 show following:
Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 6
at org.apache.orc.OrcFile$WriterVersion.from(OrcFile.java:145)
at org.apache.orc.impl.OrcTail.getWriterVersion(OrcTail.java:74)
at org.apache.orc.impl.ReaderImpl.<init>(ReaderImpl.java:385)
at org.apache.orc.OrcFile.createReader(OrcFile.java:222)
at org.apache.orc.tools.FileDump.getReader(FileDump.java:255)
at org.apache.orc.tools.FileDump.printMetaDataImpl(FileDump.java:328)
at org.apache.orc.tools.FileDump.printMetaData(FileDump.java:307)
at org.apache.orc.tools.FileDump.main(FileDump.java:154)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
On 02/15/2019 01:53，Xiao Li<li...@databricks.com> wrote：
Hi, Darcy,
Could you show us the Hive JIRA? If the impact is big, we can revert the ORC package upgrade. To improve the adoption of Spark 2.4, I think the revert is reasonable since the library upgrade of ORC is not a must based on my understanding.
Cheers,
Xiao
On Thu, Feb 14, 2019 at 9:13 AM Darcy Shen <sa...@zoho.com.invalid> wrote:
Well, it is not a bug for Spark 2.4 but a bug for Hive 2.1.1 . My colleague will report it on the Spark JIRA later.
Presto works fine when reading the ORC table created by Spark 2.4.
We've decided to fix it in Hive 2.1.1 . Since Hive 2.1.1 is widely used, I suggest that we should keep a good interop with "legacy" hive. Otherwise, newbies will complain about it and decide not to use Spark SQL for CTAS scenarios. And for now, there are several cases Apache Hive does better than Spark SQL.
We are on the way replacing Hive 2.1.1 with Spark 2.4.x (SQL) and won't consider upgrading Hive.
---- On Thu, 14 Feb 2019 21:36:08 +0800 Wenchen Fan <cl...@gmail.com> wrote ----
Do you know which bug ORC 1.5.2 introduced? Or is it because Hive uses a legacy version of ORC which has a bug?
On Thu, Feb 14, 2019 at 2:35 PM Darcy Shen <sa...@zoho.com.invalid> wrote:
We found that ORC table created by Spark 2.4 failed to be read by Hive 2.1.1.
spark-sql -e 'CREATE TABLE tmp.orcTable2 USING orc AS SELECT * FROM tmp.orcTable1 limit 10;'
hive -e 'select * from tmp.orcTable2'
The ERROR messages by Hive:
Failed with exception java.io.IOException:java.lang.RuntimeException: ORC split generation failed with exception: java.lang.ArrayIndexOutOfBoundsException: 6
And Spark 2.3.2 (or below) works fine.
I think we should git revert [SPARK-24576][BUILD] Upgrade Apache ORC to 1.5.2 by Dongjoon Hyun
---- On Tue, 12 Feb 2019 16:56:09 +0800 Dongjin Lee <do...@apache.org> wrote ----
> SPARK-23539 is a non-trivial improvement, so probably would not be back-ported to 2.4.x.
Got it. It seems reasonable.
Committers:
Please don't omit SPARK-23539 from 2.5.0. Kafka community needs this feature.
Thanks,
Dongjin
On Tue, Feb 12, 2019 at 1:50 PM Takeshi Yamamuro <li...@gmail.com> wrote:
--
Dongjin Lee
A hitchhiker in the mathematical world.
github: github.com/dongjinleekr
linkedin: kr.linkedin.com/in/dongjinleekr
speakerdeck: speakerdeck.com/dongjin
+1, too.
branch-2.4 accumulates too many commits..:
https://github.com/apache/spark/compare/0a4c03f7d084f1d2aa48673b99f3b9496893ce8d...af3c7111efd22907976fc8bbd7810fe3cfd92092
On Tue, Feb 12, 2019 at 12:36 PM Dongjoon Hyun <do...@apache.org> wrote:
Thank you, DB.
+1, Yes. It's time for preparing 2.4.1 release.
Bests,
Dongjoon.
On 2019/02/12 03:16:05, Sean Owen <sr...@gmail.com> wrote:
> I support a 2.4.1 release now, yes.
>
> SPARK-23539 is a non-trivial improvement, so probably would not be
> back-ported to 2.4.x.SPARK-26154 does look like a bug whose fix could
> be back-ported, but that's a big change. I wouldn't hold up 2.4.1 for
> it, but it could go in if otherwise ready.
>
>
> On Mon, Feb 11, 2019 at 5:20 PM Dongjin Lee <do...@apache.org> wrote:
> >
> > Hi DB,
> >
> > Could you add SPARK-23539[^1] into 2.4.1? I opened the PR[^2] a little bit ago, but it has not included in 2.3.0 nor get enough review.
> >
> > Thanks,
> > Dongjin
> >
> > [^1]: https://issues.apache.org/jira/browse/SPARK-23539
> > [^2]: https://github.com/apache/spark/pull/22282
> >
> > On Tue, Feb 12, 2019 at 6:28 AM Jungtaek Lim <ka...@gmail.com> wrote:
> >>
> >> Given SPARK-26154 [1] is a correctness issue and PR [2] is submitted, I hope it can be reviewed and included within Spark 2.4.1 - otherwise it will be a long-live correctness issue.
> >>
> >> Thanks,
> >> Jungtaek Lim (HeartSaVioR)
> >>
> >> 1. https://issues.apache.org/jira/browse/SPARK-26154
> >> 2. https://github.com/apache/spark/pull/23634
> >>
> >>
> >> 2019년 2월 12일 (화) 오전 6:17, DB Tsai <d_...@apple.com.invalid>님이 작성:
> >>>
> >>> Hello all,
> >>>
> >>> I am preparing to cut a new Apache 2.4.1 release as there are many bugs and correctness issues fixed in branch-2.4.
> >>>
> >>> The list of addressed issues are https://issues.apache.org/jira/browse/SPARK-26583?jql=project%20%3D%20SPARK%20AND%20fixVersion%20%3D%202.4.1%20order%20by%20updated%20DESC
> >>>
> >>> Let me know if you have any concern or any PR you would like to get in.
> >>>
> >>> Thanks!
> >>>
> >>> ---------------------------------------------------------------------
> >>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
> >>>
> >
> >
> > --
> > Dongjin Lee
> >
> > A hitchhiker in the mathematical world.
> >
> > github: github.com/dongjinleekr
> > linkedin: kr.linkedin.com/in/dongjinleekr
> > speakerdeck: speakerdeck.com/dongjin
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
--
---
Takeshi Yamamuro
--
--
Re: Time to cut an Apache 2.4.1 release?
Posted by Wenchen Fan <cl...@gmail.com>.
Hi Haibo,
Did you verify that reverting SPARK-24576 works for you? If the problem was
caused by a new feature in ORC 1.5, then reverting the upgrade doesn't
help, we need to switch back to the hive orc reader.
On Fri, Feb 15, 2019 at 10:50 AM Xiao Li <li...@databricks.com> wrote:
> Hi, Haibo,
>
> Originally, I thought this was caused by the fixes in ORC 1.5.2.
>
> Could you please share your JIRA and description in Apache Hive? We need
> to evaluate the impact.
>
> Thanks!
>
> Xiao
>
>
>
>
> On Thu, Feb 14, 2019 at 6:45 PM 海博 <ha...@163.com> wrote:
>
>> Hi, Xiao.
>> I'm Darcy's colleague who found this issue.
>>
>> I compared the underlying file of table which created by spark 2.4 (orc
>> 1.5.2) and spark 2.3.2 (orc 1.4.4), and found that the format of ORC had
>> changed.
>> I checked the orc release log and found this new feature(
>> https://issues.apache.org/jira/browse/ORC-305) of ORC 1.5 change orc
>> file format but I was not sure whether they were caused by this.
>>
>> Secondly, the full stack trace by hive 2.1.1 show following:
>>
>> Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 6
>> at org.apache.orc.OrcFile$WriterVersion.from(OrcFile.java:145)
>> at org.apache.orc.impl.OrcTail.getWriterVersion(OrcTail.java:74)
>> at org.apache.orc.impl.ReaderImpl.<init>(ReaderImpl.java:385)
>> at org.apache.orc.OrcFile.createReader(OrcFile.java:222)
>> at org.apache.orc.tools.FileDump.getReader(FileDump.java:255)
>> at org.apache.orc.tools.FileDump.printMetaDataImpl(FileDump.java:328)
>> at org.apache.orc.tools.FileDump.printMetaData(FileDump.java:307)
>> at org.apache.orc.tools.FileDump.main(FileDump.java:154)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:498)
>> at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
>> at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
>>
>> On 02/15/2019 01:53，Xiao Li<li...@databricks.com>
>> <li...@databricks.com> wrote：
>>
>> Hi, Darcy,
>>
>> Could you show us the Hive JIRA? If the impact is big, we can revert the
>> ORC package upgrade. To improve the adoption of Spark 2.4, I think the
>> revert is reasonable since the library upgrade of ORC is not a must based
>> on my understanding.
>>
>> Cheers,
>>
>> Xiao
>>
>> On Thu, Feb 14, 2019 at 9:13 AM Darcy Shen <sa...@zoho.com.invalid>
>> wrote:
>>
>>>
>>> Well, it is not a bug for Spark 2.4 but a bug for Hive 2.1.1 . My
>>> colleague will report it on the Spark JIRA later.
>>>
>>> Presto works fine when reading the ORC table created by Spark 2.4.
>>>
>>> We've decided to fix it in Hive 2.1.1 . Since Hive 2.1.1 is widely used,
>>> I suggest that we should keep a good interop with "legacy" hive. Otherwise,
>>> newbies will complain about it and decide not to use Spark SQL for CTAS
>>> scenarios. And for now, there are several cases Apache Hive does better
>>> than Spark SQL.
>>>
>>> We are on the way replacing Hive 2.1.1 with Spark 2.4.x (SQL) and won't
>>> consider upgrading Hive.
>>>
>>> ---- On Thu, 14 Feb 2019 21:36:08 +0800 *Wenchen Fan
>>> <cloud0fan@gmail.com <cl...@gmail.com>>* wrote ----
>>>
>>> Do you know which bug ORC 1.5.2 introduced? Or is it because Hive uses a
>>> legacy version of ORC which has a bug?
>>>
>>> On Thu, Feb 14, 2019 at 2:35 PM Darcy Shen <sa...@zoho.com.invalid>
>>> wrote:
>>>
>>>
>>>>
>>>> We found that ORC table created by Spark 2.4 failed to be read by Hive
>>>> 2.1.1.
>>>>
>>>>
>>>> spark-sql -e 'CREATE TABLE tmp.orcTable2 USING orc AS SELECT * FROM
>>>> tmp.orcTable1 limit 10;'
>>>>
>>>> hive -e 'select * from tmp.orcTable2'
>>>>
>>>> The ERROR messages by Hive:
>>>>
>>>> Failed with exception java.io.IOException:java.lang.RuntimeException:
>>>> ORC split generation failed with exception:
>>>> java.lang.ArrayIndexOutOfBoundsException: 6
>>>>
>>>> And Spark 2.3.2 (or below) works fine.
>>>>
>>>> I think we should git revert [SPARK-24576][BUILD] Upgrade Apache ORC to
>>>> 1.5.2 by Dongjoon Hyun
>>>>
>>>>
>>>> ---- On Tue, 12 Feb 2019 16:56:09 +0800 *Dongjin Lee
>>>> <dongjin@apache.org <do...@apache.org>>* wrote ----
>>>>
>>>> > SPARK-23539 is a non-trivial improvement, so probably would not be
>>>> back-ported to 2.4.x.
>>>>
>>>> Got it. It seems reasonable.
>>>>
>>>> Committers:
>>>>
>>>> Please don't omit SPARK-23539 from 2.5.0. Kafka community needs this
>>>> feature.
>>>>
>>>> Thanks,
>>>> Dongjin
>>>>
>>>> On Tue, Feb 12, 2019 at 1:50 PM Takeshi Yamamuro <li...@gmail.com>
>>>> wrote:
>>>>
>>>>
>>>>
>>>> --
>>>> *Dongjin Lee*
>>>>
>>>> *A hitchhiker in the mathematical world.*
>>>>
>>>> *github: <http://goog_969573159/>github.com/dongjinleekr
>>>> <https://github.com/dongjinleekr>*
>>>> linkedin: kr.linkedin.com/in/dongjinleekr
>>>> speakerdeck: speakerdeck.com/dongjin
>>>>
>>>>> +1, too.
>>>>> branch-2.4 accumulates too many commits..:
>>>>>
>>>>> https://github.com/apache/spark/compare/0a4c03f7d084f1d2aa48673b99f3b9496893ce8d...af3c7111efd22907976fc8bbd7810fe3cfd92092
>>>>>
>>>>> On Tue, Feb 12, 2019 at 12:36 PM Dongjoon Hyun <do...@apache.org>
>>>>> wrote:
>>>>>
>>>>>> Thank you, DB.
>>>>>>
>>>>>> +1, Yes. It's time for preparing 2.4.1 release.
>>>>>>
>>>>>> Bests,
>>>>>> Dongjoon.
>>>>>>
>>>>>> On 2019/02/12 03:16:05, Sean Owen <sr...@gmail.com> wrote:
>>>>>> > I support a 2.4.1 release now, yes.
>>>>>> >
>>>>>> > SPARK-23539 is a non-trivial improvement, so probably would not be
>>>>>> > back-ported to 2.4.x.SPARK-26154 does look like a bug whose fix
>>>>>> could
>>>>>> > be back-ported, but that's a big change. I wouldn't hold up 2.4.1
>>>>>> for
>>>>>> > it, but it could go in if otherwise ready.
>>>>>> >
>>>>>> >
>>>>>> > On Mon, Feb 11, 2019 at 5:20 PM Dongjin Lee <do...@apache.org>
>>>>>> wrote:
>>>>>> > >
>>>>>> > > Hi DB,
>>>>>> > >
>>>>>> > > Could you add SPARK-23539[^1] into 2.4.1? I opened the PR[^2] a
>>>>>> little bit ago, but it has not included in 2.3.0 nor get enough review.
>>>>>> > >
>>>>>> > > Thanks,
>>>>>> > > Dongjin
>>>>>> > >
>>>>>> > > [^1]: https://issues.apache.org/jira/browse/SPARK-23539
>>>>>> > > [^2]: https://github.com/apache/spark/pull/22282
>>>>>> > >
>>>>>> > > On Tue, Feb 12, 2019 at 6:28 AM Jungtaek Lim <ka...@gmail.com>
>>>>>> wrote:
>>>>>> > >>
>>>>>> > >> Given SPARK-26154 [1] is a correctness issue and PR [2] is
>>>>>> submitted, I hope it can be reviewed and included within Spark 2.4.1 -
>>>>>> otherwise it will be a long-live correctness issue.
>>>>>> > >>
>>>>>> > >> Thanks,
>>>>>> > >> Jungtaek Lim (HeartSaVioR)
>>>>>> > >>
>>>>>> > >> 1. https://issues.apache.org/jira/browse/SPARK-26154
>>>>>> > >> 2. https://github.com/apache/spark/pull/23634
>>>>>> > >>
>>>>>> > >>
>>>>>> > >> 2019년 2월 12일 (화) 오전 6:17, DB Tsai <d_...@apple.com.invalid>님이
>>>>>> 작성:
>>>>>> > >>>
>>>>>> > >>> Hello all,
>>>>>> > >>>
>>>>>> > >>> I am preparing to cut a new Apache 2.4.1 release as there are
>>>>>> many bugs and correctness issues fixed in branch-2.4.
>>>>>> > >>>
>>>>>> > >>> The list of addressed issues are
>>>>>> https://issues.apache.org/jira/browse/SPARK-26583?jql=project%20%3D%20SPARK%20AND%20fixVersion%20%3D%202.4.1%20order%20by%20updated%20DESC
>>>>>> > >>>
>>>>>> > >>> Let me know if you have any concern or any PR you would like to
>>>>>> get in.
>>>>>> > >>>
>>>>>> > >>> Thanks!
>>>>>> > >>>
>>>>>> > >>>
>>>>>> ---------------------------------------------------------------------
>>>>>> > >>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>>>> > >>>
>>>>>> > >
>>>>>> > >
>>>>>> > > --
>>>>>> > > Dongjin Lee
>>>>>> > >
>>>>>> > > A hitchhiker in the mathematical world.
>>>>>> > >
>>>>>> > > github: github.com/dongjinleekr
>>>>>> > > linkedin: kr.linkedin.com/in/dongjinleekr
>>>>>> > > speakerdeck: speakerdeck.com/dongjin
>>>>>> >
>>>>>> >
>>>>>> ---------------------------------------------------------------------
>>>>>> > To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>>>> >
>>>>>> >
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>>>>
>>>>>>
>>>>>
>>>>> --
>>>>> ---
>>>>> Takeshi Yamamuro
>>>>>
>>>>
>>>>
>>>
>>>
>>
>> --
>> [image:
>> https://databricks.com/sparkaisummit/north-america?utm_source=email&utm_medium=signature]
>>
>>
>
> --
> [image:
> https://databricks.com/sparkaisummit/north-america?utm_source=email&utm_medium=signature]
>
Re: Time to cut an Apache 2.4.1 release?
Posted by Xiao Li <li...@databricks.com>.
Hi, Haibo,
Originally, I thought this was caused by the fixes in ORC 1.5.2.
Could you please share your JIRA and description in Apache Hive? We need to
evaluate the impact.
Thanks!
Xiao
On Thu, Feb 14, 2019 at 6:45 PM 海博 <ha...@163.com> wrote:
> Hi, Xiao.
> I'm Darcy's colleague who found this issue.
>
> I compared the underlying file of table which created by spark 2.4 (orc
> 1.5.2) and spark 2.3.2 (orc 1.4.4), and found that the format of ORC had
> changed.
> I checked the orc release log and found this new feature(
> https://issues.apache.org/jira/browse/ORC-305) of ORC 1.5 change orc file
> format but I was not sure whether they were caused by this.
>
> Secondly, the full stack trace by hive 2.1.1 show following:
>
> Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 6
> at org.apache.orc.OrcFile$WriterVersion.from(OrcFile.java:145)
> at org.apache.orc.impl.OrcTail.getWriterVersion(OrcTail.java:74)
> at org.apache.orc.impl.ReaderImpl.<init>(ReaderImpl.java:385)
> at org.apache.orc.OrcFile.createReader(OrcFile.java:222)
> at org.apache.orc.tools.FileDump.getReader(FileDump.java:255)
> at org.apache.orc.tools.FileDump.printMetaDataImpl(FileDump.java:328)
> at org.apache.orc.tools.FileDump.printMetaData(FileDump.java:307)
> at org.apache.orc.tools.FileDump.main(FileDump.java:154)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
>
> On 02/15/2019 01:53，Xiao Li<li...@databricks.com> <li...@databricks.com>
> wrote：
>
> Hi, Darcy,
>
> Could you show us the Hive JIRA? If the impact is big, we can revert the
> ORC package upgrade. To improve the adoption of Spark 2.4, I think the
> revert is reasonable since the library upgrade of ORC is not a must based
> on my understanding.
>
> Cheers,
>
> Xiao
>
> On Thu, Feb 14, 2019 at 9:13 AM Darcy Shen <sa...@zoho.com.invalid>
> wrote:
>
>>
>> Well, it is not a bug for Spark 2.4 but a bug for Hive 2.1.1 . My
>> colleague will report it on the Spark JIRA later.
>>
>> Presto works fine when reading the ORC table created by Spark 2.4.
>>
>> We've decided to fix it in Hive 2.1.1 . Since Hive 2.1.1 is widely used,
>> I suggest that we should keep a good interop with "legacy" hive. Otherwise,
>> newbies will complain about it and decide not to use Spark SQL for CTAS
>> scenarios. And for now, there are several cases Apache Hive does better
>> than Spark SQL.
>>
>> We are on the way replacing Hive 2.1.1 with Spark 2.4.x (SQL) and won't
>> consider upgrading Hive.
>>
>> ---- On Thu, 14 Feb 2019 21:36:08 +0800 *Wenchen Fan
>> <cloud0fan@gmail.com <cl...@gmail.com>>* wrote ----
>>
>> Do you know which bug ORC 1.5.2 introduced? Or is it because Hive uses a
>> legacy version of ORC which has a bug?
>>
>> On Thu, Feb 14, 2019 at 2:35 PM Darcy Shen <sa...@zoho.com.invalid>
>> wrote:
>>
>>
>>>
>>> We found that ORC table created by Spark 2.4 failed to be read by Hive
>>> 2.1.1.
>>>
>>>
>>> spark-sql -e 'CREATE TABLE tmp.orcTable2 USING orc AS SELECT * FROM
>>> tmp.orcTable1 limit 10;'
>>>
>>> hive -e 'select * from tmp.orcTable2'
>>>
>>> The ERROR messages by Hive:
>>>
>>> Failed with exception java.io.IOException:java.lang.RuntimeException:
>>> ORC split generation failed with exception:
>>> java.lang.ArrayIndexOutOfBoundsException: 6
>>>
>>> And Spark 2.3.2 (or below) works fine.
>>>
>>> I think we should git revert [SPARK-24576][BUILD] Upgrade Apache ORC to
>>> 1.5.2 by Dongjoon Hyun
>>>
>>>
>>> ---- On Tue, 12 Feb 2019 16:56:09 +0800 *Dongjin Lee
>>> <dongjin@apache.org <do...@apache.org>>* wrote ----
>>>
>>> > SPARK-23539 is a non-trivial improvement, so probably would not be
>>> back-ported to 2.4.x.
>>>
>>> Got it. It seems reasonable.
>>>
>>> Committers:
>>>
>>> Please don't omit SPARK-23539 from 2.5.0. Kafka community needs this
>>> feature.
>>>
>>> Thanks,
>>> Dongjin
>>>
>>> On Tue, Feb 12, 2019 at 1:50 PM Takeshi Yamamuro <li...@gmail.com>
>>> wrote:
>>>
>>>
>>>
>>> --
>>> *Dongjin Lee*
>>>
>>> *A hitchhiker in the mathematical world.*
>>>
>>> *github: <http://goog_969573159/>github.com/dongjinleekr
>>> <https://github.com/dongjinleekr>*
>>> linkedin: kr.linkedin.com/in/dongjinleekr
>>> speakerdeck: speakerdeck.com/dongjin
>>>
>>>> +1, too.
>>>> branch-2.4 accumulates too many commits..:
>>>>
>>>> https://github.com/apache/spark/compare/0a4c03f7d084f1d2aa48673b99f3b9496893ce8d...af3c7111efd22907976fc8bbd7810fe3cfd92092
>>>>
>>>> On Tue, Feb 12, 2019 at 12:36 PM Dongjoon Hyun <do...@apache.org>
>>>> wrote:
>>>>
>>>>> Thank you, DB.
>>>>>
>>>>> +1, Yes. It's time for preparing 2.4.1 release.
>>>>>
>>>>> Bests,
>>>>> Dongjoon.
>>>>>
>>>>> On 2019/02/12 03:16:05, Sean Owen <sr...@gmail.com> wrote:
>>>>> > I support a 2.4.1 release now, yes.
>>>>> >
>>>>> > SPARK-23539 is a non-trivial improvement, so probably would not be
>>>>> > back-ported to 2.4.x.SPARK-26154 does look like a bug whose fix could
>>>>> > be back-ported, but that's a big change. I wouldn't hold up 2.4.1 for
>>>>> > it, but it could go in if otherwise ready.
>>>>> >
>>>>> >
>>>>> > On Mon, Feb 11, 2019 at 5:20 PM Dongjin Lee <do...@apache.org>
>>>>> wrote:
>>>>> > >
>>>>> > > Hi DB,
>>>>> > >
>>>>> > > Could you add SPARK-23539[^1] into 2.4.1? I opened the PR[^2] a
>>>>> little bit ago, but it has not included in 2.3.0 nor get enough review.
>>>>> > >
>>>>> > > Thanks,
>>>>> > > Dongjin
>>>>> > >
>>>>> > > [^1]: https://issues.apache.org/jira/browse/SPARK-23539
>>>>> > > [^2]: https://github.com/apache/spark/pull/22282
>>>>> > >
>>>>> > > On Tue, Feb 12, 2019 at 6:28 AM Jungtaek Lim <ka...@gmail.com>
>>>>> wrote:
>>>>> > >>
>>>>> > >> Given SPARK-26154 [1] is a correctness issue and PR [2] is
>>>>> submitted, I hope it can be reviewed and included within Spark 2.4.1 -
>>>>> otherwise it will be a long-live correctness issue.
>>>>> > >>
>>>>> > >> Thanks,
>>>>> > >> Jungtaek Lim (HeartSaVioR)
>>>>> > >>
>>>>> > >> 1. https://issues.apache.org/jira/browse/SPARK-26154
>>>>> > >> 2. https://github.com/apache/spark/pull/23634
>>>>> > >>
>>>>> > >>
>>>>> > >> 2019년 2월 12일 (화) 오전 6:17, DB Tsai <d_...@apple.com.invalid>님이
>>>>> 작성:
>>>>> > >>>
>>>>> > >>> Hello all,
>>>>> > >>>
>>>>> > >>> I am preparing to cut a new Apache 2.4.1 release as there are
>>>>> many bugs and correctness issues fixed in branch-2.4.
>>>>> > >>>
>>>>> > >>> The list of addressed issues are
>>>>> https://issues.apache.org/jira/browse/SPARK-26583?jql=project%20%3D%20SPARK%20AND%20fixVersion%20%3D%202.4.1%20order%20by%20updated%20DESC
>>>>> > >>>
>>>>> > >>> Let me know if you have any concern or any PR you would like to
>>>>> get in.
>>>>> > >>>
>>>>> > >>> Thanks!
>>>>> > >>>
>>>>> > >>>
>>>>> ---------------------------------------------------------------------
>>>>> > >>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>>> > >>>
>>>>> > >
>>>>> > >
>>>>> > > --
>>>>> > > Dongjin Lee
>>>>> > >
>>>>> > > A hitchhiker in the mathematical world.
>>>>> > >
>>>>> > > github: github.com/dongjinleekr
>>>>> > > linkedin: kr.linkedin.com/in/dongjinleekr
>>>>> > > speakerdeck: speakerdeck.com/dongjin
>>>>> >
>>>>> > ---------------------------------------------------------------------
>>>>> > To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>>> >
>>>>> >
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>>>
>>>>>
>>>>
>>>> --
>>>> ---
>>>> Takeshi Yamamuro
>>>>
>>>
>>>
>>
>>
>
> --
> [image:
> https://databricks.com/sparkaisummit/north-america?utm_source=email&utm_medium=signature]
>
>
--
[image:
https://databricks.com/sparkaisummit/north-america?utm_source=email&utm_medium=signature]
Re: Time to cut an Apache 2.4.1 release?
Posted by Felix Cheung <fe...@hotmail.com>.
I’d rather agree with Mridul in this case.
In fact, Hive 2.1.1 is quite “old”
8 December 2016 : release 2.1.1 available
________________________________
From: Mridul Muralidharan <mr...@gmail.com>
Sent: Friday, February 15, 2019 12:02 AM
To: 海博
Cc: Xiao Li; Wenchen Fan; Spark; Dongjoon Hyun; Darcy Shen; DB Tsai; Sean Owen
Subject: Re: Time to cut an Apache 2.4.1 release?
An on-disk format change is problematic w.r.t rolling back the version.
Since there is a on-disk data change, if we revert to the older ORC version, will that cause issues for users who have been using 2.4.0 with the newer format ?
As in, will the earlier ORC version be able to read files created with the newer ORC version (coming from 2.4.0 release) ?
If it cannot, then reverting to older ORC will not help - and we will need to move to a bug fixed ORC version.
Regards,
Mridul
On Thu, Feb 14, 2019 at 6:45 PM 海博 <ha...@163.com>> wrote:
Hi, Xiao.
I'm Darcy's colleague who found this issue.
I compared the underlying file of table which created by spark 2.4 (orc 1.5.2) and spark 2.3.2 (orc 1.4.4), and found that the format of ORC had changed.
I checked the orc release log and found this new feature(https://issues.apache.org/jira/browse/ORC-305) of ORC 1.5 change orc file format but I was not sure whether they were caused by this.
Secondly, the full stack trace by hive 2.1.1 show following:
Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 6
at org.apache.orc.OrcFile$WriterVersion.from(OrcFile.java:145)
at org.apache.orc.impl.OrcTail.getWriterVersion(OrcTail.java:74)
at org.apache.orc.impl.ReaderImpl.<init>(ReaderImpl.java:385)
at org.apache.orc.OrcFile.createReader(OrcFile.java:222)
at org.apache.orc.tools.FileDump.getReader(FileDump.java:255)
at org.apache.orc.tools.FileDump.printMetaDataImpl(FileDump.java:328)
at org.apache.orc.tools.FileDump.printMetaData(FileDump.java:307)
at org.apache.orc.tools.FileDump.main(FileDump.java:154)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
On 02/15/2019 01:53，Xiao Li<li...@databricks.com> wrote：
Hi, Darcy,
Could you show us the Hive JIRA? If the impact is big, we can revert the ORC package upgrade. To improve the adoption of Spark 2.4, I think the revert is reasonable since the library upgrade of ORC is not a must based on my understanding.
Cheers,
Xiao
On Thu, Feb 14, 2019 at 9:13 AM Darcy Shen <sa...@zoho.com.invalid>> wrote:
Well, it is not a bug for Spark 2.4 but a bug for Hive 2.1.1 . My colleague will report it on the Spark JIRA later.
Presto works fine when reading the ORC table created by Spark 2.4.
We've decided to fix it in Hive 2.1.1 . Since Hive 2.1.1 is widely used, I suggest that we should keep a good interop with "legacy" hive. Otherwise, newbies will complain about it and decide not to use Spark SQL for CTAS scenarios. And for now, there are several cases Apache Hive does better than Spark SQL.
We are on the way replacing Hive 2.1.1 with Spark 2.4.x (SQL) and won't consider upgrading Hive.
---- On Thu, 14 Feb 2019 21:36:08 +0800 Wenchen Fan <cl...@gmail.com>> wrote ----
Do you know which bug ORC 1.5.2 introduced? Or is it because Hive uses a legacy version of ORC which has a bug?
On Thu, Feb 14, 2019 at 2:35 PM Darcy Shen <sa...@zoho.com.invalid>> wrote:
We found that ORC table created by Spark 2.4 failed to be read by Hive 2.1.1.
spark-sql -e 'CREATE TABLE tmp.orcTable2 USING orc AS SELECT * FROM tmp.orcTable1 limit 10;'
hive -e 'select * from tmp.orcTable2'
The ERROR messages by Hive:
Failed with exception java.io.IOException:java.lang.RuntimeException: ORC split generation failed with exception: java.lang.ArrayIndexOutOfBoundsException: 6
And Spark 2.3.2 (or below) works fine.
I think we should git revert [SPARK-24576][BUILD] Upgrade Apache ORC to 1.5.2 by Dongjoon Hyun
---- On Tue, 12 Feb 2019 16:56:09 +0800 Dongjin Lee <do...@apache.org>> wrote ----
> SPARK-23539 is a non-trivial improvement, so probably would not be back-ported to 2.4.x.
Got it. It seems reasonable.
Committers:
Please don't omit SPARK-23539 from 2.5.0. Kafka community needs this feature.
Thanks,
Dongjin
On Tue, Feb 12, 2019 at 1:50 PM Takeshi Yamamuro <li...@gmail.com>> wrote:
--
Dongjin Lee
A hitchhiker in the mathematical world.
github: <http://goog_969573159/> github.com/dongjinleekr<https://github.com/dongjinleekr>
linkedin: kr.linkedin.com/in/dongjinleekr<https://kr.linkedin.com/in/dongjinleekr>
speakerdeck: speakerdeck.com/dongjin<https://speakerdeck.com/dongjin>
+1, too.
branch-2.4 accumulates too many commits..:
https://github.com/apache/spark/compare/0a4c03f7d084f1d2aa48673b99f3b9496893ce8d...af3c7111efd22907976fc8bbd7810fe3cfd92092
On Tue, Feb 12, 2019 at 12:36 PM Dongjoon Hyun <do...@apache.org>> wrote:
Thank you, DB.
+1, Yes. It's time for preparing 2.4.1 release.
Bests,
Dongjoon.
On 2019/02/12 03:16:05, Sean Owen <sr...@gmail.com>> wrote:
> I support a 2.4.1 release now, yes.
>
> SPARK-23539 is a non-trivial improvement, so probably would not be
> back-ported to 2.4.x.SPARK-26154 does look like a bug whose fix could
> be back-ported, but that's a big change. I wouldn't hold up 2.4.1 for
> it, but it could go in if otherwise ready.
>
>
> On Mon, Feb 11, 2019 at 5:20 PM Dongjin Lee <do...@apache.org>> wrote:
> >
> > Hi DB,
> >
> > Could you add SPARK-23539[^1] into 2.4.1? I opened the PR[^2] a little bit ago, but it has not included in 2.3.0 nor get enough review.
> >
> > Thanks,
> > Dongjin
> >
> > [^1]: https://issues.apache.org/jira/browse/SPARK-23539
> > [^2]: https://github.com/apache/spark/pull/22282
> >
> > On Tue, Feb 12, 2019 at 6:28 AM Jungtaek Lim <ka...@gmail.com>> wrote:
> >>
> >> Given SPARK-26154 [1] is a correctness issue and PR [2] is submitted, I hope it can be reviewed and included within Spark 2.4.1 - otherwise it will be a long-live correctness issue.
> >>
> >> Thanks,
> >> Jungtaek Lim (HeartSaVioR)
> >>
> >> 1. https://issues.apache.org/jira/browse/SPARK-26154
> >> 2. https://github.com/apache/spark/pull/23634
> >>
> >>
> >> 2019년 2월 12일 (화) 오전 6:17, DB Tsai <d_...@apple.com.invalid>>님이 작성:
> >>>
> >>> Hello all,
> >>>
> >>> I am preparing to cut a new Apache 2.4.1 release as there are many bugs and correctness issues fixed in branch-2.4.
> >>>
> >>> The list of addressed issues are https://issues.apache.org/jira/browse/SPARK-26583?jql=project%20%3D%20SPARK%20AND%20fixVersion%20%3D%202.4.1%20order%20by%20updated%20DESC
> >>>
> >>> Let me know if you have any concern or any PR you would like to get in.
> >>>
> >>> Thanks!
> >>>
> >>> ---------------------------------------------------------------------
> >>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org<ma...@spark.apache.org>
> >>>
> >
> >
> > --
> > Dongjin Lee
> >
> > A hitchhiker in the mathematical world.
> >
> > github: github.com/dongjinleekr<http://github.com/dongjinleekr>
> > linkedin: kr.linkedin.com/in/dongjinleekr<http://kr.linkedin.com/in/dongjinleekr>
> > speakerdeck: speakerdeck.com/dongjin<http://speakerdeck.com/dongjin>
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org<ma...@spark.apache.org>
>
>
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org<ma...@spark.apache.org>
--
---
Takeshi Yamamuro
--
[https://databricks.com/sparkaisummit/north-america?utm_source=email&utm_medium=signature]
Re: Time to cut an Apache 2.4.1 release?
Posted by Mridul Muralidharan <mr...@gmail.com>.
An on-disk format change is problematic w.r.t rolling back the version.
Since there is a on-disk data change, if we revert to the older ORC
version, will that cause issues for users who have been using 2.4.0 with
the newer format ?
As in, will the earlier ORC version be able to read files created with the
newer ORC version (coming from 2.4.0 release) ?
If it cannot, then reverting to older ORC will not help - and we will need
to move to a bug fixed ORC version.
Regards,
Mridul
On Thu, Feb 14, 2019 at 6:45 PM 海博 <ha...@163.com> wrote:
> Hi, Xiao.
> I'm Darcy's colleague who found this issue.
>
> I compared the underlying file of table which created by spark 2.4 (orc
> 1.5.2) and spark 2.3.2 (orc 1.4.4), and found that the format of ORC had
> changed.
> I checked the orc release log and found this new feature(
> https://issues.apache.org/jira/browse/ORC-305) of ORC 1.5 change orc file
> format but I was not sure whether they were caused by this.
>
> Secondly, the full stack trace by hive 2.1.1 show following:
>
> Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 6
> at org.apache.orc.OrcFile$WriterVersion.from(OrcFile.java:145)
> at org.apache.orc.impl.OrcTail.getWriterVersion(OrcTail.java:74)
> at org.apache.orc.impl.ReaderImpl.<init>(ReaderImpl.java:385)
> at org.apache.orc.OrcFile.createReader(OrcFile.java:222)
> at org.apache.orc.tools.FileDump.getReader(FileDump.java:255)
> at org.apache.orc.tools.FileDump.printMetaDataImpl(FileDump.java:328)
> at org.apache.orc.tools.FileDump.printMetaData(FileDump.java:307)
> at org.apache.orc.tools.FileDump.main(FileDump.java:154)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
>
> On 02/15/2019 01:53，Xiao Li<li...@databricks.com> <li...@databricks.com>
> wrote：
>
> Hi, Darcy,
>
> Could you show us the Hive JIRA? If the impact is big, we can revert the
> ORC package upgrade. To improve the adoption of Spark 2.4, I think the
> revert is reasonable since the library upgrade of ORC is not a must based
> on my understanding.
>
> Cheers,
>
> Xiao
>
> On Thu, Feb 14, 2019 at 9:13 AM Darcy Shen <sa...@zoho.com.invalid>
> wrote:
>
>>
>> Well, it is not a bug for Spark 2.4 but a bug for Hive 2.1.1 . My
>> colleague will report it on the Spark JIRA later.
>>
>> Presto works fine when reading the ORC table created by Spark 2.4.
>>
>> We've decided to fix it in Hive 2.1.1 . Since Hive 2.1.1 is widely used,
>> I suggest that we should keep a good interop with "legacy" hive. Otherwise,
>> newbies will complain about it and decide not to use Spark SQL for CTAS
>> scenarios. And for now, there are several cases Apache Hive does better
>> than Spark SQL.
>>
>> We are on the way replacing Hive 2.1.1 with Spark 2.4.x (SQL) and won't
>> consider upgrading Hive.
>>
>> ---- On Thu, 14 Feb 2019 21:36:08 +0800 *Wenchen Fan
>> <cloud0fan@gmail.com <cl...@gmail.com>>* wrote ----
>>
>> Do you know which bug ORC 1.5.2 introduced? Or is it because Hive uses a
>> legacy version of ORC which has a bug?
>>
>> On Thu, Feb 14, 2019 at 2:35 PM Darcy Shen <sa...@zoho.com.invalid>
>> wrote:
>>
>>
>>>
>>> We found that ORC table created by Spark 2.4 failed to be read by Hive
>>> 2.1.1.
>>>
>>>
>>> spark-sql -e 'CREATE TABLE tmp.orcTable2 USING orc AS SELECT * FROM
>>> tmp.orcTable1 limit 10;'
>>>
>>> hive -e 'select * from tmp.orcTable2'
>>>
>>> The ERROR messages by Hive:
>>>
>>> Failed with exception java.io.IOException:java.lang.RuntimeException:
>>> ORC split generation failed with exception:
>>> java.lang.ArrayIndexOutOfBoundsException: 6
>>>
>>> And Spark 2.3.2 (or below) works fine.
>>>
>>> I think we should git revert [SPARK-24576][BUILD] Upgrade Apache ORC to
>>> 1.5.2 by Dongjoon Hyun
>>>
>>>
>>> ---- On Tue, 12 Feb 2019 16:56:09 +0800 *Dongjin Lee
>>> <dongjin@apache.org <do...@apache.org>>* wrote ----
>>>
>>> > SPARK-23539 is a non-trivial improvement, so probably would not be
>>> back-ported to 2.4.x.
>>>
>>> Got it. It seems reasonable.
>>>
>>> Committers:
>>>
>>> Please don't omit SPARK-23539 from 2.5.0. Kafka community needs this
>>> feature.
>>>
>>> Thanks,
>>> Dongjin
>>>
>>> On Tue, Feb 12, 2019 at 1:50 PM Takeshi Yamamuro <li...@gmail.com>
>>> wrote:
>>>
>>>
>>>
>>> --
>>> *Dongjin Lee*
>>>
>>> *A hitchhiker in the mathematical world.*
>>>
>>> *github: <http://goog_969573159/>github.com/dongjinleekr
>>> <https://github.com/dongjinleekr>*
>>> linkedin: kr.linkedin.com/in/dongjinleekr
>>> speakerdeck: speakerdeck.com/dongjin
>>>
>>>> +1, too.
>>>> branch-2.4 accumulates too many commits..:
>>>>
>>>> https://github.com/apache/spark/compare/0a4c03f7d084f1d2aa48673b99f3b9496893ce8d...af3c7111efd22907976fc8bbd7810fe3cfd92092
>>>>
>>>> On Tue, Feb 12, 2019 at 12:36 PM Dongjoon Hyun <do...@apache.org>
>>>> wrote:
>>>>
>>>>> Thank you, DB.
>>>>>
>>>>> +1, Yes. It's time for preparing 2.4.1 release.
>>>>>
>>>>> Bests,
>>>>> Dongjoon.
>>>>>
>>>>> On 2019/02/12 03:16:05, Sean Owen <sr...@gmail.com> wrote:
>>>>> > I support a 2.4.1 release now, yes.
>>>>> >
>>>>> > SPARK-23539 is a non-trivial improvement, so probably would not be
>>>>> > back-ported to 2.4.x.SPARK-26154 does look like a bug whose fix could
>>>>> > be back-ported, but that's a big change. I wouldn't hold up 2.4.1 for
>>>>> > it, but it could go in if otherwise ready.
>>>>> >
>>>>> >
>>>>> > On Mon, Feb 11, 2019 at 5:20 PM Dongjin Lee <do...@apache.org>
>>>>> wrote:
>>>>> > >
>>>>> > > Hi DB,
>>>>> > >
>>>>> > > Could you add SPARK-23539[^1] into 2.4.1? I opened the PR[^2] a
>>>>> little bit ago, but it has not included in 2.3.0 nor get enough review.
>>>>> > >
>>>>> > > Thanks,
>>>>> > > Dongjin
>>>>> > >
>>>>> > > [^1]: https://issues.apache.org/jira/browse/SPARK-23539
>>>>> > > [^2]: https://github.com/apache/spark/pull/22282
>>>>> > >
>>>>> > > On Tue, Feb 12, 2019 at 6:28 AM Jungtaek Lim <ka...@gmail.com>
>>>>> wrote:
>>>>> > >>
>>>>> > >> Given SPARK-26154 [1] is a correctness issue and PR [2] is
>>>>> submitted, I hope it can be reviewed and included within Spark 2.4.1 -
>>>>> otherwise it will be a long-live correctness issue.
>>>>> > >>
>>>>> > >> Thanks,
>>>>> > >> Jungtaek Lim (HeartSaVioR)
>>>>> > >>
>>>>> > >> 1. https://issues.apache.org/jira/browse/SPARK-26154
>>>>> > >> 2. https://github.com/apache/spark/pull/23634
>>>>> > >>
>>>>> > >>
>>>>> > >> 2019년 2월 12일 (화) 오전 6:17, DB Tsai <d_...@apple.com.invalid>님이
>>>>> 작성:
>>>>> > >>>
>>>>> > >>> Hello all,
>>>>> > >>>
>>>>> > >>> I am preparing to cut a new Apache 2.4.1 release as there are
>>>>> many bugs and correctness issues fixed in branch-2.4.
>>>>> > >>>
>>>>> > >>> The list of addressed issues are
>>>>> https://issues.apache.org/jira/browse/SPARK-26583?jql=project%20%3D%20SPARK%20AND%20fixVersion%20%3D%202.4.1%20order%20by%20updated%20DESC
>>>>> > >>>
>>>>> > >>> Let me know if you have any concern or any PR you would like to
>>>>> get in.
>>>>> > >>>
>>>>> > >>> Thanks!
>>>>> > >>>
>>>>> > >>>
>>>>> ---------------------------------------------------------------------
>>>>> > >>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>>> > >>>
>>>>> > >
>>>>> > >
>>>>> > > --
>>>>> > > Dongjin Lee
>>>>> > >
>>>>> > > A hitchhiker in the mathematical world.
>>>>> > >
>>>>> > > github: github.com/dongjinleekr
>>>>> > > linkedin: kr.linkedin.com/in/dongjinleekr
>>>>> > > speakerdeck: speakerdeck.com/dongjin
>>>>> >
>>>>> > ---------------------------------------------------------------------
>>>>> > To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>>> >
>>>>> >
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>>>
>>>>>
>>>>
>>>> --
>>>> ---
>>>> Takeshi Yamamuro
>>>>
>>>
>>>
>>
>>
>
> --
> [image:
> https://databricks.com/sparkaisummit/north-america?utm_source=email&utm_medium=signature]
>
>
Re: Time to cut an Apache 2.4.1 release?
Posted by 海博 <ha...@163.com>.
Hi, Xiao.
I'm Darcy's colleague who found this issue.
I compared the underlying file of table which created by spark 2.4 (orc 1.5.2) and spark 2.3.2 (orc 1.4.4), and found that the format of ORC had changed.
I checked the orc release log and found this new feature(https://issues.apache.org/jira/browse/ORC-305) of ORC 1.5 change orc file format but I was not sure whether they were caused by this.
Secondly, the full stack trace by hive 2.1.1 show following:
Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 6
at org.apache.orc.OrcFile$WriterVersion.from(OrcFile.java:145)
at org.apache.orc.impl.OrcTail.getWriterVersion(OrcTail.java:74)
at org.apache.orc.impl.ReaderImpl.<init>(ReaderImpl.java:385)
at org.apache.orc.OrcFile.createReader(OrcFile.java:222)
at org.apache.orc.tools.FileDump.getReader(FileDump.java:255)
at org.apache.orc.tools.FileDump.printMetaDataImpl(FileDump.java:328)
at org.apache.orc.tools.FileDump.printMetaData(FileDump.java:307)
at org.apache.orc.tools.FileDump.main(FileDump.java:154)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
On 02/15/2019 01:53，Xiao Li<li...@databricks.com> wrote：
Hi, Darcy,
Could you show us the Hive JIRA? If the impact is big, we can revert the ORC package upgrade. To improve the adoption of Spark 2.4, I think the revert is reasonable since the library upgrade of ORC is not a must based on my understanding.
Cheers,
Xiao
On Thu, Feb 14, 2019 at 9:13 AM Darcy Shen <sa...@zoho.com.invalid> wrote:
Well, it is not a bug for Spark 2.4 but a bug for Hive 2.1.1 . My colleague will report it on the Spark JIRA later.
Presto works fine when reading the ORC table created by Spark 2.4.
We've decided to fix it in Hive 2.1.1 . Since Hive 2.1.1 is widely used, I suggest that we should keep a good interop with "legacy" hive. Otherwise, newbies will complain about it and decide not to use Spark SQL for CTAS scenarios. And for now, there are several cases Apache Hive does better than Spark SQL.
We are on the way replacing Hive 2.1.1 with Spark 2.4.x (SQL) and won't consider upgrading Hive.
---- On Thu, 14 Feb 2019 21:36:08 +0800 Wenchen Fan <cl...@gmail.com> wrote ----
Do you know which bug ORC 1.5.2 introduced? Or is it because Hive uses a legacy version of ORC which has a bug?
On Thu, Feb 14, 2019 at 2:35 PM Darcy Shen <sa...@zoho.com.invalid> wrote:
We found that ORC table created by Spark 2.4 failed to be read by Hive 2.1.1.
spark-sql -e 'CREATE TABLE tmp.orcTable2 USING orc AS SELECT * FROM tmp.orcTable1 limit 10;'
hive -e 'select * from tmp.orcTable2'
The ERROR messages by Hive:
Failed with exception java.io.IOException:java.lang.RuntimeException: ORC split generation failed with exception: java.lang.ArrayIndexOutOfBoundsException: 6
And Spark 2.3.2 (or below) works fine.
I think we should git revert [SPARK-24576][BUILD] Upgrade Apache ORC to 1.5.2 by Dongjoon Hyun
---- On Tue, 12 Feb 2019 16:56:09 +0800 Dongjin Lee <do...@apache.org> wrote ----
> SPARK-23539 is a non-trivial improvement, so probably would not be back-ported to 2.4.x.
Got it. It seems reasonable.
Committers:
Please don't omit SPARK-23539 from 2.5.0. Kafka community needs this feature.
Thanks,
Dongjin
On Tue, Feb 12, 2019 at 1:50 PM Takeshi Yamamuro <li...@gmail.com> wrote:
--
Dongjin Lee
A hitchhiker in the mathematical world.
github: github.com/dongjinleekr
linkedin: kr.linkedin.com/in/dongjinleekr
speakerdeck: speakerdeck.com/dongjin
+1, too.
branch-2.4 accumulates too many commits..:
https://github.com/apache/spark/compare/0a4c03f7d084f1d2aa48673b99f3b9496893ce8d...af3c7111efd22907976fc8bbd7810fe3cfd92092
On Tue, Feb 12, 2019 at 12:36 PM Dongjoon Hyun <do...@apache.org> wrote:
Thank you, DB.
+1, Yes. It's time for preparing 2.4.1 release.
Bests,
Dongjoon.
On 2019/02/12 03:16:05, Sean Owen <sr...@gmail.com> wrote:
> I support a 2.4.1 release now, yes.
>
> SPARK-23539 is a non-trivial improvement, so probably would not be
> back-ported to 2.4.x.SPARK-26154 does look like a bug whose fix could
> be back-ported, but that's a big change. I wouldn't hold up 2.4.1 for
> it, but it could go in if otherwise ready.
>
>
> On Mon, Feb 11, 2019 at 5:20 PM Dongjin Lee <do...@apache.org> wrote:
> >
> > Hi DB,
> >
> > Could you add SPARK-23539[^1] into 2.4.1? I opened the PR[^2] a little bit ago, but it has not included in 2.3.0 nor get enough review.
> >
> > Thanks,
> > Dongjin
> >
> > [^1]: https://issues.apache.org/jira/browse/SPARK-23539
> > [^2]: https://github.com/apache/spark/pull/22282
> >
> > On Tue, Feb 12, 2019 at 6:28 AM Jungtaek Lim <ka...@gmail.com> wrote:
> >>
> >> Given SPARK-26154 [1] is a correctness issue and PR [2] is submitted, I hope it can be reviewed and included within Spark 2.4.1 - otherwise it will be a long-live correctness issue.
> >>
> >> Thanks,
> >> Jungtaek Lim (HeartSaVioR)
> >>
> >> 1. https://issues.apache.org/jira/browse/SPARK-26154
> >> 2. https://github.com/apache/spark/pull/23634
> >>
> >>
> >> 2019년 2월 12일 (화) 오전 6:17, DB Tsai <d_...@apple.com.invalid>님이 작성:
> >>>
> >>> Hello all,
> >>>
> >>> I am preparing to cut a new Apache 2.4.1 release as there are many bugs and correctness issues fixed in branch-2.4.
> >>>
> >>> The list of addressed issues are https://issues.apache.org/jira/browse/SPARK-26583?jql=project%20%3D%20SPARK%20AND%20fixVersion%20%3D%202.4.1%20order%20by%20updated%20DESC
> >>>
> >>> Let me know if you have any concern or any PR you would like to get in.
> >>>
> >>> Thanks!
> >>>
> >>> ---------------------------------------------------------------------
> >>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
> >>>
> >
> >
> > --
> > Dongjin Lee
> >
> > A hitchhiker in the mathematical world.
> >
> > github: github.com/dongjinleekr
> > linkedin: kr.linkedin.com/in/dongjinleekr
> > speakerdeck: speakerdeck.com/dongjin
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
--
---
Takeshi Yamamuro
--
Re: Time to cut an Apache 2.4.1 release?
Posted by Xiao Li <li...@databricks.com>.
Hi, Darcy,
Could you show us the Hive JIRA? If the impact is big, we can revert the
ORC package upgrade. To improve the adoption of Spark 2.4, I think the
revert is reasonable since the library upgrade of ORC is not a must based
on my understanding.
Cheers,
Xiao
On Thu, Feb 14, 2019 at 9:13 AM Darcy Shen <sa...@zoho.com.invalid> wrote:
>
> Well, it is not a bug for Spark 2.4 but a bug for Hive 2.1.1 . My
> colleague will report it on the Spark JIRA later.
>
> Presto works fine when reading the ORC table created by Spark 2.4.
>
> We've decided to fix it in Hive 2.1.1 . Since Hive 2.1.1 is widely used, I
> suggest that we should keep a good interop with "legacy" hive. Otherwise,
> newbies will complain about it and decide not to use Spark SQL for CTAS
> scenarios. And for now, there are several cases Apache Hive does better
> than Spark SQL.
>
> We are on the way replacing Hive 2.1.1 with Spark 2.4.x (SQL) and won't
> consider upgrading Hive.
>
> ---- On Thu, 14 Feb 2019 21:36:08 +0800 *Wenchen Fan <cloud0fan@gmail.com
> <cl...@gmail.com>>* wrote ----
>
> Do you know which bug ORC 1.5.2 introduced? Or is it because Hive uses a
> legacy version of ORC which has a bug?
>
> On Thu, Feb 14, 2019 at 2:35 PM Darcy Shen <sa...@zoho.com.invalid>
> wrote:
>
>
>>
>> We found that ORC table created by Spark 2.4 failed to be read by Hive
>> 2.1.1.
>>
>>
>> spark-sql -e 'CREATE TABLE tmp.orcTable2 USING orc AS SELECT * FROM
>> tmp.orcTable1 limit 10;'
>>
>> hive -e 'select * from tmp.orcTable2'
>>
>> The ERROR messages by Hive:
>>
>> Failed with exception java.io.IOException:java.lang.RuntimeException: ORC
>> split generation failed with exception:
>> java.lang.ArrayIndexOutOfBoundsException: 6
>>
>> And Spark 2.3.2 (or below) works fine.
>>
>> I think we should git revert [SPARK-24576][BUILD] Upgrade Apache ORC to
>> 1.5.2 by Dongjoon Hyun
>>
>>
>> ---- On Tue, 12 Feb 2019 16:56:09 +0800 *Dongjin Lee <dongjin@apache.org
>> <do...@apache.org>>* wrote ----
>>
>> > SPARK-23539 is a non-trivial improvement, so probably would not be
>> back-ported to 2.4.x.
>>
>> Got it. It seems reasonable.
>>
>> Committers:
>>
>> Please don't omit SPARK-23539 from 2.5.0. Kafka community needs this
>> feature.
>>
>> Thanks,
>> Dongjin
>>
>> On Tue, Feb 12, 2019 at 1:50 PM Takeshi Yamamuro <li...@gmail.com>
>> wrote:
>>
>>
>>
>> --
>> *Dongjin Lee*
>>
>> *A hitchhiker in the mathematical world.*
>>
>> *github: <http://goog_969573159/>github.com/dongjinleekr
>> <https://github.com/dongjinleekr>*
>> linkedin: kr.linkedin.com/in/dongjinleekr
>> speakerdeck: speakerdeck.com/dongjin
>>
>>> +1, too.
>>> branch-2.4 accumulates too many commits..:
>>>
>>> https://github.com/apache/spark/compare/0a4c03f7d084f1d2aa48673b99f3b9496893ce8d...af3c7111efd22907976fc8bbd7810fe3cfd92092
>>>
>>> On Tue, Feb 12, 2019 at 12:36 PM Dongjoon Hyun <do...@apache.org>
>>> wrote:
>>>
>>>> Thank you, DB.
>>>>
>>>> +1, Yes. It's time for preparing 2.4.1 release.
>>>>
>>>> Bests,
>>>> Dongjoon.
>>>>
>>>> On 2019/02/12 03:16:05, Sean Owen <sr...@gmail.com> wrote:
>>>> > I support a 2.4.1 release now, yes.
>>>> >
>>>> > SPARK-23539 is a non-trivial improvement, so probably would not be
>>>> > back-ported to 2.4.x.SPARK-26154 does look like a bug whose fix could
>>>> > be back-ported, but that's a big change. I wouldn't hold up 2.4.1 for
>>>> > it, but it could go in if otherwise ready.
>>>> >
>>>> >
>>>> > On Mon, Feb 11, 2019 at 5:20 PM Dongjin Lee <do...@apache.org>
>>>> wrote:
>>>> > >
>>>> > > Hi DB,
>>>> > >
>>>> > > Could you add SPARK-23539[^1] into 2.4.1? I opened the PR[^2] a
>>>> little bit ago, but it has not included in 2.3.0 nor get enough review.
>>>> > >
>>>> > > Thanks,
>>>> > > Dongjin
>>>> > >
>>>> > > [^1]: https://issues.apache.org/jira/browse/SPARK-23539
>>>> > > [^2]: https://github.com/apache/spark/pull/22282
>>>> > >
>>>> > > On Tue, Feb 12, 2019 at 6:28 AM Jungtaek Lim <ka...@gmail.com>
>>>> wrote:
>>>> > >>
>>>> > >> Given SPARK-26154 [1] is a correctness issue and PR [2] is
>>>> submitted, I hope it can be reviewed and included within Spark 2.4.1 -
>>>> otherwise it will be a long-live correctness issue.
>>>> > >>
>>>> > >> Thanks,
>>>> > >> Jungtaek Lim (HeartSaVioR)
>>>> > >>
>>>> > >> 1. https://issues.apache.org/jira/browse/SPARK-26154
>>>> > >> 2. https://github.com/apache/spark/pull/23634
>>>> > >>
>>>> > >>
>>>> > >> 2019년 2월 12일 (화) 오전 6:17, DB Tsai <d_...@apple.com.invalid>님이 작성:
>>>> > >>>
>>>> > >>> Hello all,
>>>> > >>>
>>>> > >>> I am preparing to cut a new Apache 2.4.1 release as there are
>>>> many bugs and correctness issues fixed in branch-2.4.
>>>> > >>>
>>>> > >>> The list of addressed issues are
>>>> https://issues.apache.org/jira/browse/SPARK-26583?jql=project%20%3D%20SPARK%20AND%20fixVersion%20%3D%202.4.1%20order%20by%20updated%20DESC
>>>> > >>>
>>>> > >>> Let me know if you have any concern or any PR you would like to
>>>> get in.
>>>> > >>>
>>>> > >>> Thanks!
>>>> > >>>
>>>> > >>>
>>>> ---------------------------------------------------------------------
>>>> > >>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>> > >>>
>>>> > >
>>>> > >
>>>> > > --
>>>> > > Dongjin Lee
>>>> > >
>>>> > > A hitchhiker in the mathematical world.
>>>> > >
>>>> > > github: github.com/dongjinleekr
>>>> > > linkedin: kr.linkedin.com/in/dongjinleekr
>>>> > > speakerdeck: speakerdeck.com/dongjin
>>>> >
>>>> > ---------------------------------------------------------------------
>>>> > To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>> >
>>>> >
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>>
>>>>
>>>
>>> --
>>> ---
>>> Takeshi Yamamuro
>>>
>>
>>
>
>
--
[image:
https://databricks.com/sparkaisummit/north-america?utm_source=email&utm_medium=signature]
Re: Time to cut an Apache 2.4.1 release?
Posted by Darcy Shen <sa...@zoho.com.INVALID>.
Well, it is not a bug for Spark 2.4 but a bug for Hive 2.1.1 . My colleague will report it on the Spark JIRA later.
Presto works fine when reading the ORC table created by Spark 2.4.
We've decided to fix it in Hive 2.1.1 . Since Hive 2.1.1 is widely used, I suggest that we should keep a good interop with "legacy" hive. Otherwise, newbies will complain about it and decide not to use Spark SQL for CTAS scenarios. And for now, there are several cases Apache Hive does better than Spark SQL.
We are on the way replacing Hive 2.1.1 with Spark 2.4.x (SQL) and won't consider upgrading Hive.
---- On Thu, 14 Feb 2019 21:36:08 +0800 Wenchen Fan <cl...@gmail.com> wrote ----
Do you know which bug ORC 1.5.2 introduced? Or is it because Hive uses a legacy version of ORC which has a bug?
On Thu, Feb 14, 2019 at 2:35 PM Darcy Shen <ma...@zoho.com.invalid> wrote:
We found that ORC table created by Spark 2.4 failed to be read by Hive 2.1.1.
spark-sql -e 'CREATE TABLE tmp.orcTable2 USING orc AS SELECT * FROM tmp.orcTable1 limit 10;'
hive -e 'select * from tmp.orcTable2'
The ERROR messages by Hive:
Failed with exception java.io.IOException:java.lang.RuntimeException: ORC split generation failed with exception: java.lang.ArrayIndexOutOfBoundsException: 6
And Spark 2.3.2 (or below) works fine.
I think we should git revert [SPARK-24576][BUILD] Upgrade Apache ORC to 1.5.2 by Dongjoon Hyun
---- On Tue, 12 Feb 2019 16:56:09 +0800 Dongjin Lee <ma...@apache.org> wrote ----
> SPARK-23539 is a non-trivial improvement, so probably would not be back-ported to 2.4.x.
Got it. It seems reasonable.
Committers:
Please don't omit SPARK-23539 from 2.5.0. Kafka community needs this feature.
Thanks,
Dongjin
On Tue, Feb 12, 2019 at 1:50 PM Takeshi Yamamuro <ma...@gmail.com> wrote:
--
Dongjin Lee
A hitchhiker in the mathematical world.
github:http://goog_969573159/https://github.com/dongjinleekr
linkedin: https://kr.linkedin.com/in/dongjinleekr
speakerdeck: https://speakerdeck.com/dongjin
+1, too.
branch-2.4 accumulates too many commits..:
https://github.com/apache/spark/compare/0a4c03f7d084f1d2aa48673b99f3b9496893ce8d...af3c7111efd22907976fc8bbd7810fe3cfd92092
On Tue, Feb 12, 2019 at 12:36 PM Dongjoon Hyun <ma...@apache.org> wrote:
Thank you, DB.
+1, Yes. It's time for preparing 2.4.1 release.
Bests,
Dongjoon.
On 2019/02/12 03:16:05, Sean Owen <ma...@gmail.com> wrote:
> I support a 2.4.1 release now, yes.
>
> SPARK-23539 is a non-trivial improvement, so probably would not be
> back-ported to 2.4.x.SPARK-26154 does look like a bug whose fix could
> be back-ported, but that's a big change. I wouldn't hold up 2.4.1 for
> it, but it could go in if otherwise ready.
>
>
> On Mon, Feb 11, 2019 at 5:20 PM Dongjin Lee <ma...@apache.org> wrote:
> >
> > Hi DB,
> >
> > Could you add SPARK-23539[^1] into 2.4.1? I opened the PR[^2] a little bit ago, but it has not included in 2.3.0 nor get enough review.
> >
> > Thanks,
> > Dongjin
> >
> > [^1]: https://issues.apache.org/jira/browse/SPARK-23539
> > [^2]: https://github.com/apache/spark/pull/22282
> >
> > On Tue, Feb 12, 2019 at 6:28 AM Jungtaek Lim <ma...@gmail.com> wrote:
> >>
> >> Given SPARK-26154 [1] is a correctness issue and PR [2] is submitted, I hope it can be reviewed and included within Spark 2.4.1 - otherwise it will be a long-live correctness issue.
> >>
> >> Thanks,
> >> Jungtaek Lim (HeartSaVioR)
> >>
> >> 1. https://issues.apache.org/jira/browse/SPARK-26154
> >> 2. https://github.com/apache/spark/pull/23634
> >>
> >>
> >> 2019년 2월 12일 (화) 오전 6:17, DB Tsai <ma...@apple.com.invalid>님이 작성:
> >>>
> >>> Hello all,
> >>>
> >>> I am preparing to cut a new Apache 2.4.1 release as there are many bugs and correctness issues fixed in branch-2.4.
> >>>
> >>> The list of addressed issues are https://issues.apache.org/jira/browse/SPARK-26583?jql=project%20%3D%20SPARK%20AND%20fixVersion%20%3D%202.4.1%20order%20by%20updated%20DESC
> >>>
> >>> Let me know if you have any concern or any PR you would like to get in.
> >>>
> >>> Thanks!
> >>>
> >>> ---------------------------------------------------------------------
> >>> To unsubscribe e-mail: mailto:dev-unsubscribe@spark.apache.org
> >>>
> >
> >
> > --
> > Dongjin Lee
> >
> > A hitchhiker in the mathematical world.
> >
> > github: http://github.com/dongjinleekr
> > linkedin: http://kr.linkedin.com/in/dongjinleekr
> > speakerdeck: http://speakerdeck.com/dongjin
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: mailto:dev-unsubscribe@spark.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe e-mail: mailto:dev-unsubscribe@spark.apache.org
--
---
Takeshi Yamamuro
Re: Time to cut an Apache 2.4.1 release?
Posted by Wenchen Fan <cl...@gmail.com>.
Do you know which bug ORC 1.5.2 introduced? Or is it because Hive uses a
legacy version of ORC which has a bug?
On Thu, Feb 14, 2019 at 2:35 PM Darcy Shen <sa...@zoho.com.invalid> wrote:
>
> We found that ORC table created by Spark 2.4 failed to be read by Hive
> 2.1.1.
>
>
> spark-sql -e 'CREATE TABLE tmp.orcTable2 USING orc AS SELECT * FROM
> tmp.orcTable1 limit 10;'
>
> hive -e 'select * from tmp.orcTable2'
>
> The ERROR messages by Hive:
>
> Failed with exception java.io.IOException:java.lang.RuntimeException: ORC
> split generation failed with exception:
> java.lang.ArrayIndexOutOfBoundsException: 6
>
> And Spark 2.3.2 (or below) works fine.
>
> I think we should git revert [SPARK-24576][BUILD] Upgrade Apache ORC to
> 1.5.2 by Dongjoon Hyun
>
>
> ---- On Tue, 12 Feb 2019 16:56:09 +0800 *Dongjin Lee <dongjin@apache.org
> <do...@apache.org>>* wrote ----
>
> > SPARK-23539 is a non-trivial improvement, so probably would not be
> back-ported to 2.4.x.
>
> Got it. It seems reasonable.
>
> Committers:
>
> Please don't omit SPARK-23539 from 2.5.0. Kafka community needs this
> feature.
>
> Thanks,
> Dongjin
>
> On Tue, Feb 12, 2019 at 1:50 PM Takeshi Yamamuro <li...@gmail.com>
> wrote:
>
>
>
> --
>
> *Dongjin Lee*
>
>
> *A hitchhiker in the mathematical world.*
>
>
>
>
> *github: <http://goog_969573159/>github.com/dongjinleekr
> <https://github.com/dongjinleekr>linkedin: kr.linkedin.com/in/dongjinleekr
> <https://kr.linkedin.com/in/dongjinleekr>speakerdeck: speakerdeck.com/dongjin
> <https://speakerdeck.com/dongjin>*
>
>> +1, too.
>> branch-2.4 accumulates too many commits..:
>>
>> https://github.com/apache/spark/compare/0a4c03f7d084f1d2aa48673b99f3b9496893ce8d...af3c7111efd22907976fc8bbd7810fe3cfd92092
>>
>> On Tue, Feb 12, 2019 at 12:36 PM Dongjoon Hyun <do...@apache.org>
>> wrote:
>>
>>> Thank you, DB.
>>>
>>> +1, Yes. It's time for preparing 2.4.1 release.
>>>
>>> Bests,
>>> Dongjoon.
>>>
>>> On 2019/02/12 03:16:05, Sean Owen <sr...@gmail.com> wrote:
>>> > I support a 2.4.1 release now, yes.
>>> >
>>> > SPARK-23539 is a non-trivial improvement, so probably would not be
>>> > back-ported to 2.4.x.SPARK-26154 does look like a bug whose fix could
>>> > be back-ported, but that's a big change. I wouldn't hold up 2.4.1 for
>>> > it, but it could go in if otherwise ready.
>>> >
>>> >
>>> > On Mon, Feb 11, 2019 at 5:20 PM Dongjin Lee <do...@apache.org>
>>> wrote:
>>> > >
>>> > > Hi DB,
>>> > >
>>> > > Could you add SPARK-23539[^1] into 2.4.1? I opened the PR[^2] a
>>> little bit ago, but it has not included in 2.3.0 nor get enough review.
>>> > >
>>> > > Thanks,
>>> > > Dongjin
>>> > >
>>> > > [^1]: https://issues.apache.org/jira/browse/SPARK-23539
>>> > > [^2]: https://github.com/apache/spark/pull/22282
>>> > >
>>> > > On Tue, Feb 12, 2019 at 6:28 AM Jungtaek Lim <ka...@gmail.com>
>>> wrote:
>>> > >>
>>> > >> Given SPARK-26154 [1] is a correctness issue and PR [2] is
>>> submitted, I hope it can be reviewed and included within Spark 2.4.1 -
>>> otherwise it will be a long-live correctness issue.
>>> > >>
>>> > >> Thanks,
>>> > >> Jungtaek Lim (HeartSaVioR)
>>> > >>
>>> > >> 1. https://issues.apache.org/jira/browse/SPARK-26154
>>> > >> 2. https://github.com/apache/spark/pull/23634
>>> > >>
>>> > >>
>>> > >> 2019년 2월 12일 (화) 오전 6:17, DB Tsai <d_...@apple.com.invalid>님이 작성:
>>> > >>>
>>> > >>> Hello all,
>>> > >>>
>>> > >>> I am preparing to cut a new Apache 2.4.1 release as there are many
>>> bugs and correctness issues fixed in branch-2.4.
>>> > >>>
>>> > >>> The list of addressed issues are
>>> https://issues.apache.org/jira/browse/SPARK-26583?jql=project%20%3D%20SPARK%20AND%20fixVersion%20%3D%202.4.1%20order%20by%20updated%20DESC
>>> > >>>
>>> > >>> Let me know if you have any concern or any PR you would like to
>>> get in.
>>> > >>>
>>> > >>> Thanks!
>>> > >>>
>>> > >>>
>>> ---------------------------------------------------------------------
>>> > >>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>> > >>>
>>> > >
>>> > >
>>> > > --
>>> > > Dongjin Lee
>>> > >
>>> > > A hitchhiker in the mathematical world.
>>> > >
>>> > > github: github.com/dongjinleekr
>>> > > linkedin: kr.linkedin.com/in/dongjinleekr
>>> > > speakerdeck: speakerdeck.com/dongjin
>>> >
>>> > ---------------------------------------------------------------------
>>> > To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>> >
>>> >
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>
>>>
>>
>> --
>> ---
>> Takeshi Yamamuro
>>
>
>
>
Re: Time to cut an Apache 2.4.1 release?
Posted by Darcy Shen <sa...@zoho.com.INVALID>.
We found that ORC table created by Spark 2.4 failed to be read by Hive 2.1.1.
spark-sql -e 'CREATE TABLE tmp.orcTable2 USING orc AS SELECT * FROM tmp.orcTable1 limit 10;'
hive -e 'select * from tmp.orcTable2'
The ERROR messages by Hive:
Failed with exception java.io.IOException:java.lang.RuntimeException: ORC split generation failed with exception: java.lang.ArrayIndexOutOfBoundsException: 6
And Spark 2.3.2 (or below) works fine.
I think we should git revert [SPARK-24576][BUILD] Upgrade Apache ORC to 1.5.2 by Dongjoon Hyun
---- On Tue, 12 Feb 2019 16:56:09 +0800 Dongjin Lee <do...@apache.org> wrote ----
> SPARK-23539 is a non-trivial improvement, so probably would not be back-ported to 2.4.x.
Got it. It seems reasonable.
Committers:
Please don't omit SPARK-23539 from 2.5.0. Kafka community needs this feature.
Thanks,
Dongjin
On Tue, Feb 12, 2019 at 1:50 PM Takeshi Yamamuro <ma...@gmail.com> wrote:
--
Dongjin Lee
A hitchhiker in the mathematical world.
github:http://goog_969573159/https://github.com/dongjinleekr
linkedin: https://kr.linkedin.com/in/dongjinleekr
speakerdeck: https://speakerdeck.com/dongjin
+1, too.
branch-2.4 accumulates too many commits..:
https://github.com/apache/spark/compare/0a4c03f7d084f1d2aa48673b99f3b9496893ce8d...af3c7111efd22907976fc8bbd7810fe3cfd92092
On Tue, Feb 12, 2019 at 12:36 PM Dongjoon Hyun <ma...@apache.org> wrote:
Thank you, DB.
+1, Yes. It's time for preparing 2.4.1 release.
Bests,
Dongjoon.
On 2019/02/12 03:16:05, Sean Owen <ma...@gmail.com> wrote:
> I support a 2.4.1 release now, yes.
>
> SPARK-23539 is a non-trivial improvement, so probably would not be
> back-ported to 2.4.x.SPARK-26154 does look like a bug whose fix could
> be back-ported, but that's a big change. I wouldn't hold up 2.4.1 for
> it, but it could go in if otherwise ready.
>
>
> On Mon, Feb 11, 2019 at 5:20 PM Dongjin Lee <ma...@apache.org> wrote:
> >
> > Hi DB,
> >
> > Could you add SPARK-23539[^1] into 2.4.1? I opened the PR[^2] a little bit ago, but it has not included in 2.3.0 nor get enough review.
> >
> > Thanks,
> > Dongjin
> >
> > [^1]: https://issues.apache.org/jira/browse/SPARK-23539
> > [^2]: https://github.com/apache/spark/pull/22282
> >
> > On Tue, Feb 12, 2019 at 6:28 AM Jungtaek Lim <ma...@gmail.com> wrote:
> >>
> >> Given SPARK-26154 [1] is a correctness issue and PR [2] is submitted, I hope it can be reviewed and included within Spark 2.4.1 - otherwise it will be a long-live correctness issue.
> >>
> >> Thanks,
> >> Jungtaek Lim (HeartSaVioR)
> >>
> >> 1. https://issues.apache.org/jira/browse/SPARK-26154
> >> 2. https://github.com/apache/spark/pull/23634
> >>
> >>
> >> 2019년 2월 12일 (화) 오전 6:17, DB Tsai <ma...@apple.com.invalid>님이 작성:
> >>>
> >>> Hello all,
> >>>
> >>> I am preparing to cut a new Apache 2.4.1 release as there are many bugs and correctness issues fixed in branch-2.4.
> >>>
> >>> The list of addressed issues are https://issues.apache.org/jira/browse/SPARK-26583?jql=project%20%3D%20SPARK%20AND%20fixVersion%20%3D%202.4.1%20order%20by%20updated%20DESC
> >>>
> >>> Let me know if you have any concern or any PR you would like to get in.
> >>>
> >>> Thanks!
> >>>
> >>> ---------------------------------------------------------------------
> >>> To unsubscribe e-mail: mailto:dev-unsubscribe@spark.apache.org
> >>>
> >
> >
> > --
> > Dongjin Lee
> >
> > A hitchhiker in the mathematical world.
> >
> > github: http://github.com/dongjinleekr
> > linkedin: http://kr.linkedin.com/in/dongjinleekr
> > speakerdeck: http://speakerdeck.com/dongjin
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: mailto:dev-unsubscribe@spark.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe e-mail: mailto:dev-unsubscribe@spark.apache.org
--
---
Takeshi Yamamuro
Re: Time to cut an Apache 2.4.1 release?
Posted by Nan Zhu <zh...@gmail.com>.
just filed a JIRA in https://issues.apache.org/jira/browse/SPARK-26862
'
this issue only happens in 2.4.0 but not in 2.3.2
anyone would help to look into that?
On Tue, Feb 12, 2019 at 10:41 AM DB Tsai <d_...@apple.com.invalid> wrote:
> Great. I'll prepare the release for voting. Thanks!
>
> DB Tsai | Siri Open Source Technologies [not a contribution] |
> Apple, Inc
>
> > On Feb 12, 2019, at 4:11 AM, Wenchen Fan <cl...@gmail.com> wrote:
> >
> > +1 for 2.4.1
> >
> > On Tue, Feb 12, 2019 at 7:55 PM Hyukjin Kwon <gu...@gmail.com>
> wrote:
> > +1 for 2.4.1
> >
> > 2019년 2월 12일 (화) 오후 4:56, Dongjin Lee <do...@apache.org>님이 작성:
> > > SPARK-23539 is a non-trivial improvement, so probably would not be
> back-ported to 2.4.x.
> >
> > Got it. It seems reasonable.
> >
> > Committers:
> >
> > Please don't omit SPARK-23539 from 2.5.0. Kafka community needs this
> feature.
> >
> > Thanks,
> > Dongjin
> >
> > On Tue, Feb 12, 2019 at 1:50 PM Takeshi Yamamuro <li...@gmail.com>
> wrote:
> > +1, too.
> > branch-2.4 accumulates too many commits..:
> >
> https://github.com/apache/spark/compare/0a4c03f7d084f1d2aa48673b99f3b9496893ce8d...af3c7111efd22907976fc8bbd7810fe3cfd92092
> >
> > On Tue, Feb 12, 2019 at 12:36 PM Dongjoon Hyun <do...@apache.org>
> wrote:
> > Thank you, DB.
> >
> > +1, Yes. It's time for preparing 2.4.1 release.
> >
> > Bests,
> > Dongjoon.
> >
> > On 2019/02/12 03:16:05, Sean Owen <sr...@gmail.com> wrote:
> > > I support a 2.4.1 release now, yes.
> > >
> > > SPARK-23539 is a non-trivial improvement, so probably would not be
> > > back-ported to 2.4.x.SPARK-26154 does look like a bug whose fix could
> > > be back-ported, but that's a big change. I wouldn't hold up 2.4.1 for
> > > it, but it could go in if otherwise ready.
> > >
> > >
> > > On Mon, Feb 11, 2019 at 5:20 PM Dongjin Lee <do...@apache.org>
> wrote:
> > > >
> > > > Hi DB,
> > > >
> > > > Could you add SPARK-23539[^1] into 2.4.1? I opened the PR[^2] a
> little bit ago, but it has not included in 2.3.0 nor get enough review.
> > > >
> > > > Thanks,
> > > > Dongjin
> > > >
> > > > [^1]: https://issues.apache.org/jira/browse/SPARK-23539
> > > > [^2]: https://github.com/apache/spark/pull/22282
> > > >
> > > > On Tue, Feb 12, 2019 at 6:28 AM Jungtaek Lim <ka...@gmail.com>
> wrote:
> > > >>
> > > >> Given SPARK-26154 [1] is a correctness issue and PR [2] is
> submitted, I hope it can be reviewed and included within Spark 2.4.1 -
> otherwise it will be a long-live correctness issue.
> > > >>
> > > >> Thanks,
> > > >> Jungtaek Lim (HeartSaVioR)
> > > >>
> > > >> 1. https://issues.apache.org/jira/browse/SPARK-26154
> > > >> 2. https://github.com/apache/spark/pull/23634
> > > >>
> > > >>
> > > >> 2019년 2월 12일 (화) 오전 6:17, DB Tsai <d_...@apple.com.invalid>님이 작성:
> > > >>>
> > > >>> Hello all,
> > > >>>
> > > >>> I am preparing to cut a new Apache 2.4.1 release as there are many
> bugs and correctness issues fixed in branch-2.4.
> > > >>>
> > > >>> The list of addressed issues are
> https://issues.apache.org/jira/browse/SPARK-26583?jql=project%20%3D%20SPARK%20AND%20fixVersion%20%3D%202.4.1%20order%20by%20updated%20DESC
> > > >>>
> > > >>> Let me know if you have any concern or any PR you would like to
> get in.
> > > >>>
> > > >>> Thanks!
> > > >>>
> > > >>>
> ---------------------------------------------------------------------
> > > >>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
> > > >>>
> > > >
> > > >
> > > > --
> > > > Dongjin Lee
> > > >
> > > > A hitchhiker in the mathematical world.
> > > >
> > > > github: github.com/dongjinleekr
> > > > linkedin: kr.linkedin.com/in/dongjinleekr
> > > > speakerdeck: speakerdeck.com/dongjin
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
> > >
> > >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
> >
> >
> >
> > --
> > ---
> > Takeshi Yamamuro
> >
> >
> > --
> > Dongjin Lee
> >
> > A hitchhiker in the mathematical world.
> >
> > github: github.com/dongjinleekr
> > linkedin: kr.linkedin.com/in/dongjinleekr
> > speakerdeck: speakerdeck.com/dongjin
>
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>
>
Re: Time to cut an Apache 2.4.1 release?
Posted by DB Tsai <d_...@apple.com.INVALID>.
Great. I'll prepare the release for voting. Thanks!
DB Tsai | Siri Open Source Technologies [not a contribution] | Apple, Inc
> On Feb 12, 2019, at 4:11 AM, Wenchen Fan <cl...@gmail.com> wrote:
>
> +1 for 2.4.1
>
> On Tue, Feb 12, 2019 at 7:55 PM Hyukjin Kwon <gu...@gmail.com> wrote:
> +1 for 2.4.1
>
> 2019년 2월 12일 (화) 오후 4:56, Dongjin Lee <do...@apache.org>님이 작성:
> > SPARK-23539 is a non-trivial improvement, so probably would not be back-ported to 2.4.x.
>
> Got it. It seems reasonable.
>
> Committers:
>
> Please don't omit SPARK-23539 from 2.5.0. Kafka community needs this feature.
>
> Thanks,
> Dongjin
>
> On Tue, Feb 12, 2019 at 1:50 PM Takeshi Yamamuro <li...@gmail.com> wrote:
> +1, too.
> branch-2.4 accumulates too many commits..:
> https://github.com/apache/spark/compare/0a4c03f7d084f1d2aa48673b99f3b9496893ce8d...af3c7111efd22907976fc8bbd7810fe3cfd92092
>
> On Tue, Feb 12, 2019 at 12:36 PM Dongjoon Hyun <do...@apache.org> wrote:
> Thank you, DB.
>
> +1, Yes. It's time for preparing 2.4.1 release.
>
> Bests,
> Dongjoon.
>
> On 2019/02/12 03:16:05, Sean Owen <sr...@gmail.com> wrote:
> > I support a 2.4.1 release now, yes.
> >
> > SPARK-23539 is a non-trivial improvement, so probably would not be
> > back-ported to 2.4.x.SPARK-26154 does look like a bug whose fix could
> > be back-ported, but that's a big change. I wouldn't hold up 2.4.1 for
> > it, but it could go in if otherwise ready.
> >
> >
> > On Mon, Feb 11, 2019 at 5:20 PM Dongjin Lee <do...@apache.org> wrote:
> > >
> > > Hi DB,
> > >
> > > Could you add SPARK-23539[^1] into 2.4.1? I opened the PR[^2] a little bit ago, but it has not included in 2.3.0 nor get enough review.
> > >
> > > Thanks,
> > > Dongjin
> > >
> > > [^1]: https://issues.apache.org/jira/browse/SPARK-23539
> > > [^2]: https://github.com/apache/spark/pull/22282
> > >
> > > On Tue, Feb 12, 2019 at 6:28 AM Jungtaek Lim <ka...@gmail.com> wrote:
> > >>
> > >> Given SPARK-26154 [1] is a correctness issue and PR [2] is submitted, I hope it can be reviewed and included within Spark 2.4.1 - otherwise it will be a long-live correctness issue.
> > >>
> > >> Thanks,
> > >> Jungtaek Lim (HeartSaVioR)
> > >>
> > >> 1. https://issues.apache.org/jira/browse/SPARK-26154
> > >> 2. https://github.com/apache/spark/pull/23634
> > >>
> > >>
> > >> 2019년 2월 12일 (화) 오전 6:17, DB Tsai <d_...@apple.com.invalid>님이 작성:
> > >>>
> > >>> Hello all,
> > >>>
> > >>> I am preparing to cut a new Apache 2.4.1 release as there are many bugs and correctness issues fixed in branch-2.4.
> > >>>
> > >>> The list of addressed issues are https://issues.apache.org/jira/browse/SPARK-26583?jql=project%20%3D%20SPARK%20AND%20fixVersion%20%3D%202.4.1%20order%20by%20updated%20DESC
> > >>>
> > >>> Let me know if you have any concern or any PR you would like to get in.
> > >>>
> > >>> Thanks!
> > >>>
> > >>> ---------------------------------------------------------------------
> > >>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
> > >>>
> > >
> > >
> > > --
> > > Dongjin Lee
> > >
> > > A hitchhiker in the mathematical world.
> > >
> > > github: github.com/dongjinleekr
> > > linkedin: kr.linkedin.com/in/dongjinleekr
> > > speakerdeck: speakerdeck.com/dongjin
> >
> > ---------------------------------------------------------------------
> > To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
> >
> >
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>
>
>
> --
> ---
> Takeshi Yamamuro
>
>
> --
> Dongjin Lee
>
> A hitchhiker in the mathematical world.
>
> github: github.com/dongjinleekr
> linkedin: kr.linkedin.com/in/dongjinleekr
> speakerdeck: speakerdeck.com/dongjin
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
Re: Time to cut an Apache 2.4.1 release?
Posted by Wenchen Fan <cl...@gmail.com>.
+1 for 2.4.1
On Tue, Feb 12, 2019 at 7:55 PM Hyukjin Kwon <gu...@gmail.com> wrote:
> +1 for 2.4.1
>
> 2019년 2월 12일 (화) 오후 4:56, Dongjin Lee <do...@apache.org>님이 작성:
>
>> > SPARK-23539 is a non-trivial improvement, so probably would not be
>> back-ported to 2.4.x.
>>
>> Got it. It seems reasonable.
>>
>> Committers:
>>
>> Please don't omit SPARK-23539 from 2.5.0. Kafka community needs this
>> feature.
>>
>> Thanks,
>> Dongjin
>>
>> On Tue, Feb 12, 2019 at 1:50 PM Takeshi Yamamuro <li...@gmail.com>
>> wrote:
>>
>>> +1, too.
>>> branch-2.4 accumulates too many commits..:
>>>
>>> https://github.com/apache/spark/compare/0a4c03f7d084f1d2aa48673b99f3b9496893ce8d...af3c7111efd22907976fc8bbd7810fe3cfd92092
>>>
>>> On Tue, Feb 12, 2019 at 12:36 PM Dongjoon Hyun <do...@apache.org>
>>> wrote:
>>>
>>>> Thank you, DB.
>>>>
>>>> +1, Yes. It's time for preparing 2.4.1 release.
>>>>
>>>> Bests,
>>>> Dongjoon.
>>>>
>>>> On 2019/02/12 03:16:05, Sean Owen <sr...@gmail.com> wrote:
>>>> > I support a 2.4.1 release now, yes.
>>>> >
>>>> > SPARK-23539 is a non-trivial improvement, so probably would not be
>>>> > back-ported to 2.4.x.SPARK-26154 does look like a bug whose fix could
>>>> > be back-ported, but that's a big change. I wouldn't hold up 2.4.1 for
>>>> > it, but it could go in if otherwise ready.
>>>> >
>>>> >
>>>> > On Mon, Feb 11, 2019 at 5:20 PM Dongjin Lee <do...@apache.org>
>>>> wrote:
>>>> > >
>>>> > > Hi DB,
>>>> > >
>>>> > > Could you add SPARK-23539[^1] into 2.4.1? I opened the PR[^2] a
>>>> little bit ago, but it has not included in 2.3.0 nor get enough review.
>>>> > >
>>>> > > Thanks,
>>>> > > Dongjin
>>>> > >
>>>> > > [^1]: https://issues.apache.org/jira/browse/SPARK-23539
>>>> > > [^2]: https://github.com/apache/spark/pull/22282
>>>> > >
>>>> > > On Tue, Feb 12, 2019 at 6:28 AM Jungtaek Lim <ka...@gmail.com>
>>>> wrote:
>>>> > >>
>>>> > >> Given SPARK-26154 [1] is a correctness issue and PR [2] is
>>>> submitted, I hope it can be reviewed and included within Spark 2.4.1 -
>>>> otherwise it will be a long-live correctness issue.
>>>> > >>
>>>> > >> Thanks,
>>>> > >> Jungtaek Lim (HeartSaVioR)
>>>> > >>
>>>> > >> 1. https://issues.apache.org/jira/browse/SPARK-26154
>>>> > >> 2. https://github.com/apache/spark/pull/23634
>>>> > >>
>>>> > >>
>>>> > >> 2019년 2월 12일 (화) 오전 6:17, DB Tsai <d_...@apple.com.invalid>님이 작성:
>>>> > >>>
>>>> > >>> Hello all,
>>>> > >>>
>>>> > >>> I am preparing to cut a new Apache 2.4.1 release as there are
>>>> many bugs and correctness issues fixed in branch-2.4.
>>>> > >>>
>>>> > >>> The list of addressed issues are
>>>> https://issues.apache.org/jira/browse/SPARK-26583?jql=project%20%3D%20SPARK%20AND%20fixVersion%20%3D%202.4.1%20order%20by%20updated%20DESC
>>>> > >>>
>>>> > >>> Let me know if you have any concern or any PR you would like to
>>>> get in.
>>>> > >>>
>>>> > >>> Thanks!
>>>> > >>>
>>>> > >>>
>>>> ---------------------------------------------------------------------
>>>> > >>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>> > >>>
>>>> > >
>>>> > >
>>>> > > --
>>>> > > Dongjin Lee
>>>> > >
>>>> > > A hitchhiker in the mathematical world.
>>>> > >
>>>> > > github: github.com/dongjinleekr
>>>> > > linkedin: kr.linkedin.com/in/dongjinleekr
>>>> > > speakerdeck: speakerdeck.com/dongjin
>>>> >
>>>> > ---------------------------------------------------------------------
>>>> > To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>> >
>>>> >
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>>
>>>>
>>>
>>> --
>>> ---
>>> Takeshi Yamamuro
>>>
>>
>>
>> --
>> *Dongjin Lee*
>>
>> *A hitchhiker in the mathematical world.*
>> *github: <http://goog_969573159/>github.com/dongjinleekr
>> <https://github.com/dongjinleekr>linkedin: kr.linkedin.com/in/dongjinleekr
>> <https://kr.linkedin.com/in/dongjinleekr>speakerdeck: speakerdeck.com/dongjin
>> <https://speakerdeck.com/dongjin>*
>>
>
Re: Time to cut an Apache 2.4.1 release?
Posted by Hyukjin Kwon <gu...@gmail.com>.
+1 for 2.4.1
2019년 2월 12일 (화) 오후 4:56, Dongjin Lee <do...@apache.org>님이 작성:
> > SPARK-23539 is a non-trivial improvement, so probably would not be
> back-ported to 2.4.x.
>
> Got it. It seems reasonable.
>
> Committers:
>
> Please don't omit SPARK-23539 from 2.5.0. Kafka community needs this
> feature.
>
> Thanks,
> Dongjin
>
> On Tue, Feb 12, 2019 at 1:50 PM Takeshi Yamamuro <li...@gmail.com>
> wrote:
>
>> +1, too.
>> branch-2.4 accumulates too many commits..:
>>
>> https://github.com/apache/spark/compare/0a4c03f7d084f1d2aa48673b99f3b9496893ce8d...af3c7111efd22907976fc8bbd7810fe3cfd92092
>>
>> On Tue, Feb 12, 2019 at 12:36 PM Dongjoon Hyun <do...@apache.org>
>> wrote:
>>
>>> Thank you, DB.
>>>
>>> +1, Yes. It's time for preparing 2.4.1 release.
>>>
>>> Bests,
>>> Dongjoon.
>>>
>>> On 2019/02/12 03:16:05, Sean Owen <sr...@gmail.com> wrote:
>>> > I support a 2.4.1 release now, yes.
>>> >
>>> > SPARK-23539 is a non-trivial improvement, so probably would not be
>>> > back-ported to 2.4.x.SPARK-26154 does look like a bug whose fix could
>>> > be back-ported, but that's a big change. I wouldn't hold up 2.4.1 for
>>> > it, but it could go in if otherwise ready.
>>> >
>>> >
>>> > On Mon, Feb 11, 2019 at 5:20 PM Dongjin Lee <do...@apache.org>
>>> wrote:
>>> > >
>>> > > Hi DB,
>>> > >
>>> > > Could you add SPARK-23539[^1] into 2.4.1? I opened the PR[^2] a
>>> little bit ago, but it has not included in 2.3.0 nor get enough review.
>>> > >
>>> > > Thanks,
>>> > > Dongjin
>>> > >
>>> > > [^1]: https://issues.apache.org/jira/browse/SPARK-23539
>>> > > [^2]: https://github.com/apache/spark/pull/22282
>>> > >
>>> > > On Tue, Feb 12, 2019 at 6:28 AM Jungtaek Lim <ka...@gmail.com>
>>> wrote:
>>> > >>
>>> > >> Given SPARK-26154 [1] is a correctness issue and PR [2] is
>>> submitted, I hope it can be reviewed and included within Spark 2.4.1 -
>>> otherwise it will be a long-live correctness issue.
>>> > >>
>>> > >> Thanks,
>>> > >> Jungtaek Lim (HeartSaVioR)
>>> > >>
>>> > >> 1. https://issues.apache.org/jira/browse/SPARK-26154
>>> > >> 2. https://github.com/apache/spark/pull/23634
>>> > >>
>>> > >>
>>> > >> 2019년 2월 12일 (화) 오전 6:17, DB Tsai <d_...@apple.com.invalid>님이 작성:
>>> > >>>
>>> > >>> Hello all,
>>> > >>>
>>> > >>> I am preparing to cut a new Apache 2.4.1 release as there are many
>>> bugs and correctness issues fixed in branch-2.4.
>>> > >>>
>>> > >>> The list of addressed issues are
>>> https://issues.apache.org/jira/browse/SPARK-26583?jql=project%20%3D%20SPARK%20AND%20fixVersion%20%3D%202.4.1%20order%20by%20updated%20DESC
>>> > >>>
>>> > >>> Let me know if you have any concern or any PR you would like to
>>> get in.
>>> > >>>
>>> > >>> Thanks!
>>> > >>>
>>> > >>>
>>> ---------------------------------------------------------------------
>>> > >>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>> > >>>
>>> > >
>>> > >
>>> > > --
>>> > > Dongjin Lee
>>> > >
>>> > > A hitchhiker in the mathematical world.
>>> > >
>>> > > github: github.com/dongjinleekr
>>> > > linkedin: kr.linkedin.com/in/dongjinleekr
>>> > > speakerdeck: speakerdeck.com/dongjin
>>> >
>>> > ---------------------------------------------------------------------
>>> > To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>> >
>>> >
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>
>>>
>>
>> --
>> ---
>> Takeshi Yamamuro
>>
>
>
> --
> *Dongjin Lee*
>
> *A hitchhiker in the mathematical world.*
> *github: <http://goog_969573159/>github.com/dongjinleekr
> <https://github.com/dongjinleekr>linkedin: kr.linkedin.com/in/dongjinleekr
> <https://kr.linkedin.com/in/dongjinleekr>speakerdeck: speakerdeck.com/dongjin
> <https://speakerdeck.com/dongjin>*
>
Re: Time to cut an Apache 2.4.1 release?
Posted by Dongjin Lee <do...@apache.org>.
> SPARK-23539 is a non-trivial improvement, so probably would not be
back-ported to 2.4.x.
Got it. It seems reasonable.
Committers:
Please don't omit SPARK-23539 from 2.5.0. Kafka community needs this
feature.
Thanks,
Dongjin
On Tue, Feb 12, 2019 at 1:50 PM Takeshi Yamamuro <li...@gmail.com>
wrote:
> +1, too.
> branch-2.4 accumulates too many commits..:
>
> https://github.com/apache/spark/compare/0a4c03f7d084f1d2aa48673b99f3b9496893ce8d...af3c7111efd22907976fc8bbd7810fe3cfd92092
>
> On Tue, Feb 12, 2019 at 12:36 PM Dongjoon Hyun <do...@apache.org>
> wrote:
>
>> Thank you, DB.
>>
>> +1, Yes. It's time for preparing 2.4.1 release.
>>
>> Bests,
>> Dongjoon.
>>
>> On 2019/02/12 03:16:05, Sean Owen <sr...@gmail.com> wrote:
>> > I support a 2.4.1 release now, yes.
>> >
>> > SPARK-23539 is a non-trivial improvement, so probably would not be
>> > back-ported to 2.4.x.SPARK-26154 does look like a bug whose fix could
>> > be back-ported, but that's a big change. I wouldn't hold up 2.4.1 for
>> > it, but it could go in if otherwise ready.
>> >
>> >
>> > On Mon, Feb 11, 2019 at 5:20 PM Dongjin Lee <do...@apache.org> wrote:
>> > >
>> > > Hi DB,
>> > >
>> > > Could you add SPARK-23539[^1] into 2.4.1? I opened the PR[^2] a
>> little bit ago, but it has not included in 2.3.0 nor get enough review.
>> > >
>> > > Thanks,
>> > > Dongjin
>> > >
>> > > [^1]: https://issues.apache.org/jira/browse/SPARK-23539
>> > > [^2]: https://github.com/apache/spark/pull/22282
>> > >
>> > > On Tue, Feb 12, 2019 at 6:28 AM Jungtaek Lim <ka...@gmail.com>
>> wrote:
>> > >>
>> > >> Given SPARK-26154 [1] is a correctness issue and PR [2] is
>> submitted, I hope it can be reviewed and included within Spark 2.4.1 -
>> otherwise it will be a long-live correctness issue.
>> > >>
>> > >> Thanks,
>> > >> Jungtaek Lim (HeartSaVioR)
>> > >>
>> > >> 1. https://issues.apache.org/jira/browse/SPARK-26154
>> > >> 2. https://github.com/apache/spark/pull/23634
>> > >>
>> > >>
>> > >> 2019년 2월 12일 (화) 오전 6:17, DB Tsai <d_...@apple.com.invalid>님이 작성:
>> > >>>
>> > >>> Hello all,
>> > >>>
>> > >>> I am preparing to cut a new Apache 2.4.1 release as there are many
>> bugs and correctness issues fixed in branch-2.4.
>> > >>>
>> > >>> The list of addressed issues are
>> https://issues.apache.org/jira/browse/SPARK-26583?jql=project%20%3D%20SPARK%20AND%20fixVersion%20%3D%202.4.1%20order%20by%20updated%20DESC
>> > >>>
>> > >>> Let me know if you have any concern or any PR you would like to get
>> in.
>> > >>>
>> > >>> Thanks!
>> > >>>
>> > >>>
>> ---------------------------------------------------------------------
>> > >>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>> > >>>
>> > >
>> > >
>> > > --
>> > > Dongjin Lee
>> > >
>> > > A hitchhiker in the mathematical world.
>> > >
>> > > github: github.com/dongjinleekr
>> > > linkedin: kr.linkedin.com/in/dongjinleekr
>> > > speakerdeck: speakerdeck.com/dongjin
>> >
>> > ---------------------------------------------------------------------
>> > To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>> >
>> >
>>
>> ---------------------------------------------------------------------
>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>
>>
>
> --
> ---
> Takeshi Yamamuro
>
--
*Dongjin Lee*
*A hitchhiker in the mathematical world.*
*github: <http://goog_969573159/>github.com/dongjinleekr
<https://github.com/dongjinleekr>linkedin: kr.linkedin.com/in/dongjinleekr
<https://kr.linkedin.com/in/dongjinleekr>speakerdeck: speakerdeck.com/dongjin
<https://speakerdeck.com/dongjin>*
Re: Time to cut an Apache 2.4.1 release?
Posted by Takeshi Yamamuro <li...@gmail.com>.
+1, too.
branch-2.4 accumulates too many commits..:
https://github.com/apache/spark/compare/0a4c03f7d084f1d2aa48673b99f3b9496893ce8d...af3c7111efd22907976fc8bbd7810fe3cfd92092
On Tue, Feb 12, 2019 at 12:36 PM Dongjoon Hyun <do...@apache.org> wrote:
> Thank you, DB.
>
> +1, Yes. It's time for preparing 2.4.1 release.
>
> Bests,
> Dongjoon.
>
> On 2019/02/12 03:16:05, Sean Owen <sr...@gmail.com> wrote:
> > I support a 2.4.1 release now, yes.
> >
> > SPARK-23539 is a non-trivial improvement, so probably would not be
> > back-ported to 2.4.x.SPARK-26154 does look like a bug whose fix could
> > be back-ported, but that's a big change. I wouldn't hold up 2.4.1 for
> > it, but it could go in if otherwise ready.
> >
> >
> > On Mon, Feb 11, 2019 at 5:20 PM Dongjin Lee <do...@apache.org> wrote:
> > >
> > > Hi DB,
> > >
> > > Could you add SPARK-23539[^1] into 2.4.1? I opened the PR[^2] a little
> bit ago, but it has not included in 2.3.0 nor get enough review.
> > >
> > > Thanks,
> > > Dongjin
> > >
> > > [^1]: https://issues.apache.org/jira/browse/SPARK-23539
> > > [^2]: https://github.com/apache/spark/pull/22282
> > >
> > > On Tue, Feb 12, 2019 at 6:28 AM Jungtaek Lim <ka...@gmail.com>
> wrote:
> > >>
> > >> Given SPARK-26154 [1] is a correctness issue and PR [2] is submitted,
> I hope it can be reviewed and included within Spark 2.4.1 - otherwise it
> will be a long-live correctness issue.
> > >>
> > >> Thanks,
> > >> Jungtaek Lim (HeartSaVioR)
> > >>
> > >> 1. https://issues.apache.org/jira/browse/SPARK-26154
> > >> 2. https://github.com/apache/spark/pull/23634
> > >>
> > >>
> > >> 2019년 2월 12일 (화) 오전 6:17, DB Tsai <d_...@apple.com.invalid>님이 작성:
> > >>>
> > >>> Hello all,
> > >>>
> > >>> I am preparing to cut a new Apache 2.4.1 release as there are many
> bugs and correctness issues fixed in branch-2.4.
> > >>>
> > >>> The list of addressed issues are
> https://issues.apache.org/jira/browse/SPARK-26583?jql=project%20%3D%20SPARK%20AND%20fixVersion%20%3D%202.4.1%20order%20by%20updated%20DESC
> > >>>
> > >>> Let me know if you have any concern or any PR you would like to get
> in.
> > >>>
> > >>> Thanks!
> > >>>
> > >>> ---------------------------------------------------------------------
> > >>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
> > >>>
> > >
> > >
> > > --
> > > Dongjin Lee
> > >
> > > A hitchhiker in the mathematical world.
> > >
> > > github: github.com/dongjinleekr
> > > linkedin: kr.linkedin.com/in/dongjinleekr
> > > speakerdeck: speakerdeck.com/dongjin
> >
> > ---------------------------------------------------------------------
> > To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
> >
> >
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>
>
--
---
Takeshi Yamamuro
Re: Time to cut an Apache 2.4.1 release?
Posted by Dongjoon Hyun <do...@apache.org>.
Thank you, DB.
+1, Yes. It's time for preparing 2.4.1 release.
Bests,
Dongjoon.
On 2019/02/12 03:16:05, Sean Owen <sr...@gmail.com> wrote:
> I support a 2.4.1 release now, yes.
>
> SPARK-23539 is a non-trivial improvement, so probably would not be
> back-ported to 2.4.x.SPARK-26154 does look like a bug whose fix could
> be back-ported, but that's a big change. I wouldn't hold up 2.4.1 for
> it, but it could go in if otherwise ready.
>
>
> On Mon, Feb 11, 2019 at 5:20 PM Dongjin Lee <do...@apache.org> wrote:
> >
> > Hi DB,
> >
> > Could you add SPARK-23539[^1] into 2.4.1? I opened the PR[^2] a little bit ago, but it has not included in 2.3.0 nor get enough review.
> >
> > Thanks,
> > Dongjin
> >
> > [^1]: https://issues.apache.org/jira/browse/SPARK-23539
> > [^2]: https://github.com/apache/spark/pull/22282
> >
> > On Tue, Feb 12, 2019 at 6:28 AM Jungtaek Lim <ka...@gmail.com> wrote:
> >>
> >> Given SPARK-26154 [1] is a correctness issue and PR [2] is submitted, I hope it can be reviewed and included within Spark 2.4.1 - otherwise it will be a long-live correctness issue.
> >>
> >> Thanks,
> >> Jungtaek Lim (HeartSaVioR)
> >>
> >> 1. https://issues.apache.org/jira/browse/SPARK-26154
> >> 2. https://github.com/apache/spark/pull/23634
> >>
> >>
> >> 2019년 2월 12일 (화) 오전 6:17, DB Tsai <d_...@apple.com.invalid>님이 작성:
> >>>
> >>> Hello all,
> >>>
> >>> I am preparing to cut a new Apache 2.4.1 release as there are many bugs and correctness issues fixed in branch-2.4.
> >>>
> >>> The list of addressed issues are https://issues.apache.org/jira/browse/SPARK-26583?jql=project%20%3D%20SPARK%20AND%20fixVersion%20%3D%202.4.1%20order%20by%20updated%20DESC
> >>>
> >>> Let me know if you have any concern or any PR you would like to get in.
> >>>
> >>> Thanks!
> >>>
> >>> ---------------------------------------------------------------------
> >>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
> >>>
> >
> >
> > --
> > Dongjin Lee
> >
> > A hitchhiker in the mathematical world.
> >
> > github: github.com/dongjinleekr
> > linkedin: kr.linkedin.com/in/dongjinleekr
> > speakerdeck: speakerdeck.com/dongjin
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
Re: Time to cut an Apache 2.4.1 release?
Posted by Sean Owen <sr...@gmail.com>.
I support a 2.4.1 release now, yes.
SPARK-23539 is a non-trivial improvement, so probably would not be
back-ported to 2.4.x.SPARK-26154 does look like a bug whose fix could
be back-ported, but that's a big change. I wouldn't hold up 2.4.1 for
it, but it could go in if otherwise ready.
On Mon, Feb 11, 2019 at 5:20 PM Dongjin Lee <do...@apache.org> wrote:
>
> Hi DB,
>
> Could you add SPARK-23539[^1] into 2.4.1? I opened the PR[^2] a little bit ago, but it has not included in 2.3.0 nor get enough review.
>
> Thanks,
> Dongjin
>
> [^1]: https://issues.apache.org/jira/browse/SPARK-23539
> [^2]: https://github.com/apache/spark/pull/22282
>
> On Tue, Feb 12, 2019 at 6:28 AM Jungtaek Lim <ka...@gmail.com> wrote:
>>
>> Given SPARK-26154 [1] is a correctness issue and PR [2] is submitted, I hope it can be reviewed and included within Spark 2.4.1 - otherwise it will be a long-live correctness issue.
>>
>> Thanks,
>> Jungtaek Lim (HeartSaVioR)
>>
>> 1. https://issues.apache.org/jira/browse/SPARK-26154
>> 2. https://github.com/apache/spark/pull/23634
>>
>>
>> 2019년 2월 12일 (화) 오전 6:17, DB Tsai <d_...@apple.com.invalid>님이 작성:
>>>
>>> Hello all,
>>>
>>> I am preparing to cut a new Apache 2.4.1 release as there are many bugs and correctness issues fixed in branch-2.4.
>>>
>>> The list of addressed issues are https://issues.apache.org/jira/browse/SPARK-26583?jql=project%20%3D%20SPARK%20AND%20fixVersion%20%3D%202.4.1%20order%20by%20updated%20DESC
>>>
>>> Let me know if you have any concern or any PR you would like to get in.
>>>
>>> Thanks!
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>
>
>
> --
> Dongjin Lee
>
> A hitchhiker in the mathematical world.
>
> github: github.com/dongjinleekr
> linkedin: kr.linkedin.com/in/dongjinleekr
> speakerdeck: speakerdeck.com/dongjin
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
Re: Time to cut an Apache 2.4.1 release?
Posted by Dongjin Lee <do...@apache.org>.
Hi DB,
Could you add SPARK-23539[^1] into 2.4.1? I opened the PR[^2] a little bit
ago, but it has not included in 2.3.0 nor get enough review.
Thanks,
Dongjin
[^1]: https://issues.apache.org/jira/browse/SPARK-23539
[^2]: https://github.com/apache/spark/pull/22282
On Tue, Feb 12, 2019 at 6:28 AM Jungtaek Lim <ka...@gmail.com> wrote:
> Given SPARK-26154 [1] is a correctness issue and PR [2] is submitted, I
> hope it can be reviewed and included within Spark 2.4.1 - otherwise it will
> be a long-live correctness issue.
>
> Thanks,
> Jungtaek Lim (HeartSaVioR)
>
> 1. https://issues.apache.org/jira/browse/SPARK-26154
> 2. https://github.com/apache/spark/pull/23634
>
>
> 2019년 2월 12일 (화) 오전 6:17, DB Tsai <d_...@apple.com.invalid>님이 작성:
>
>> Hello all,
>>
>> I am preparing to cut a new Apache 2.4.1 release as there are many bugs
>> and correctness issues fixed in branch-2.4.
>>
>> The list of addressed issues are
>> https://issues.apache.org/jira/browse/SPARK-26583?jql=project%20%3D%20SPARK%20AND%20fixVersion%20%3D%202.4.1%20order%20by%20updated%20DESC
>>
>> Let me know if you have any concern or any PR you would like to get in.
>>
>> Thanks!
>>
>> ---------------------------------------------------------------------
>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>
>>
--
*Dongjin Lee*
*A hitchhiker in the mathematical world.*
*github: <http://goog_969573159/>github.com/dongjinleekr
<https://github.com/dongjinleekr>linkedin: kr.linkedin.com/in/dongjinleekr
<https://kr.linkedin.com/in/dongjinleekr>speakerdeck: speakerdeck.com/dongjin
<https://speakerdeck.com/dongjin>*
Re: Time to cut an Apache 2.4.1 release?
Posted by Jungtaek Lim <ka...@gmail.com>.
Given SPARK-26154 [1] is a correctness issue and PR [2] is submitted, I
hope it can be reviewed and included within Spark 2.4.1 - otherwise it will
be a long-live correctness issue.
Thanks,
Jungtaek Lim (HeartSaVioR)
1. https://issues.apache.org/jira/browse/SPARK-26154
2. https://github.com/apache/spark/pull/23634
2019년 2월 12일 (화) 오전 6:17, DB Tsai <d_...@apple.com.invalid>님이 작성:
> Hello all,
>
> I am preparing to cut a new Apache 2.4.1 release as there are many bugs
> and correctness issues fixed in branch-2.4.
>
> The list of addressed issues are
> https://issues.apache.org/jira/browse/SPARK-26583?jql=project%20%3D%20SPARK%20AND%20fixVersion%20%3D%202.4.1%20order%20by%20updated%20DESC
>
> Let me know if you have any concern or any PR you would like to get in.
>
> Thanks!
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>
>