You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by Nandor Kollar <nk...@cloudera.com> on 2017/05/24 16:21:55 UTC
Review Request 59530: PIG-5157 Upgrade to Spark 2.0
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59530/
-----------------------------------------------------------
Review request for pig, liyun zhang, Rohini Palaniswamy, and Adam Szita.
Repository: pig-git
Description
-------
Upgrade to Spark 2.1 API using shims.
Diffs
-----
build.xml 4040fcec8f88d448ed7442461fbf0dea8cd1136e
ivy/libraries.properties a0eb00acd2df42324540df4a9d762c64c608a6d3
ivy/spark1.properties PRE-CREATION
ivy/spark2.properties PRE-CREATION
shims/src/spark16/org/apache/pig/backend/hadoop/executionengine/spark/JobMetricsListener.java PRE-CREATION
shims/src/spark16/org/apache/pig/backend/hadoop/executionengine/spark/converter/AbstractFlatMapFunction.java PRE-CREATION
shims/src/spark16/org/apache/pig/backend/hadoop/executionengine/spark/converter/AbstractLimitConverter.java PRE-CREATION
shims/src/spark16/org/apache/pig/backend/hadoop/executionengine/spark/converter/AbstractPairFlatMapFunction.java PRE-CREATION
shims/src/spark16/org/apache/pig/backend/hadoop/executionengine/spark/converter/SkewedJoinUtil.java PRE-CREATION
shims/src/spark16/org/apache/pig/tools/pigstats/spark/SparkJobStats.java PRE-CREATION
shims/src/spark21/org/apache/pig/backend/hadoop/executionengine/spark/JobMetricsListener.java PRE-CREATION
shims/src/spark21/org/apache/pig/backend/hadoop/executionengine/spark/converter/AbstractFlatMapFunction.java PRE-CREATION
shims/src/spark21/org/apache/pig/backend/hadoop/executionengine/spark/converter/AbstractLimitConverter.java PRE-CREATION
shims/src/spark21/org/apache/pig/backend/hadoop/executionengine/spark/converter/AbstractPairFlatMapFunction.java PRE-CREATION
shims/src/spark21/org/apache/pig/backend/hadoop/executionengine/spark/converter/SkewedJoinUtil.java PRE-CREATION
shims/src/spark21/org/apache/pig/tools/pigstats/spark/SparkJobStats.java PRE-CREATION
src/org/apache/pig/backend/hadoop/executionengine/spark/JobMetricsListener.java f81341233447203abc4800cc7b22a4f419e10262
src/org/apache/pig/backend/hadoop/executionengine/spark/SparkLauncher.java c6351e01a48f297ea2e432401ffd65c4f27f8078
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/CollectedGroupConverter.java 83311dfa5bb25209a5366c2db7e8d483c31d94cd
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/FRJoinConverter.java 382258e7ff9105aa397c5a2888df0c11e9562ec9
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/ForEachConverter.java b58415e7e18ca4cf1331beef06e9214600a51424
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/GlobalRearrangeConverter.java f571b808839c2de9415a3e8e4b229a7f4b2eebd7
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/LimitConverter.java fe1b54c8f128661d7d19c276d3bb2de7874d3086
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/MergeCogroupConverter.java adf78ecab0da10d3b1a7fdde8af2b42dd899810f
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/MergeJoinConverter.java d1c43b1e06adc4c9fe45a83b81103333402e3756
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/PoissonSampleConverter.java e003bbd95763b2d189ff9ec540c89abe52592420
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SecondaryKeySortUtil.java 00d29b44848546ed16dde2baa8c61b36939971b2
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SkewedJoinConverter.java c55ba3145495a53d69db2dd56434dcc9b3bf8ed5
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SortConverter.java baabfa090323e3bef087e259ce19df2e4c34dd63
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SparkSampleSortConverter.java 3166fdc31745c013380492e089c83f3e853a3e6e
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/StreamConverter.java 3a50d485cfd54b9f3b9c1a982e6c30497a4c85fc
src/org/apache/pig/tools/pigstats/spark/SparkJobStats.java c8cc03194b223d2ee181d73c6b651a6872cac6b6
Diff: https://reviews.apache.org/r/59530/diff/1/
Testing
-------
Thanks,
Nandor Kollar
Re: Review Request 59530: PIG-5157 Upgrade to Spark 2.0
Posted by Adam Szita <sz...@cloudera.com>.
> On May 24, 2017, 9:21 p.m., Rohini Palaniswamy wrote:
> > build.xml
> > Lines 251 (patched)
> > <https://reviews.apache.org/r/59530/diff/1/?file=1731615#file1731615line252>
> >
> > Can just load one property file for spark2, leaving the spark 1.6 version in libraries.properties
AFAIK property values in ant are immutable, so once we set e.g. spark.version=1.6.1 by loading the default property file and then we load the other propery file for spark2 I think it will stay unchanged and we cannot override it for anything else e.g. spark.version=2.1.1
Also this way it looks clearer I think.
- Adam
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59530/#review175998
-----------------------------------------------------------
On May 24, 2017, 4:21 p.m., Nandor Kollar wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/59530/
> -----------------------------------------------------------
>
> (Updated May 24, 2017, 4:21 p.m.)
>
>
> Review request for pig, liyun zhang, Rohini Palaniswamy, and Adam Szita.
>
>
> Repository: pig-git
>
>
> Description
> -------
>
> Upgrade to Spark 2.1 API using shims.
>
>
> Diffs
> -----
>
> build.xml 4040fcec8f88d448ed7442461fbf0dea8cd1136e
> ivy/libraries.properties a0eb00acd2df42324540df4a9d762c64c608a6d3
> ivy/spark1.properties PRE-CREATION
> ivy/spark2.properties PRE-CREATION
> shims/src/spark16/org/apache/pig/backend/hadoop/executionengine/spark/JobMetricsListener.java PRE-CREATION
> shims/src/spark16/org/apache/pig/backend/hadoop/executionengine/spark/converter/AbstractFlatMapFunction.java PRE-CREATION
> shims/src/spark16/org/apache/pig/backend/hadoop/executionengine/spark/converter/AbstractLimitConverter.java PRE-CREATION
> shims/src/spark16/org/apache/pig/backend/hadoop/executionengine/spark/converter/AbstractPairFlatMapFunction.java PRE-CREATION
> shims/src/spark16/org/apache/pig/backend/hadoop/executionengine/spark/converter/SkewedJoinUtil.java PRE-CREATION
> shims/src/spark16/org/apache/pig/tools/pigstats/spark/SparkJobStats.java PRE-CREATION
> shims/src/spark21/org/apache/pig/backend/hadoop/executionengine/spark/JobMetricsListener.java PRE-CREATION
> shims/src/spark21/org/apache/pig/backend/hadoop/executionengine/spark/converter/AbstractFlatMapFunction.java PRE-CREATION
> shims/src/spark21/org/apache/pig/backend/hadoop/executionengine/spark/converter/AbstractLimitConverter.java PRE-CREATION
> shims/src/spark21/org/apache/pig/backend/hadoop/executionengine/spark/converter/AbstractPairFlatMapFunction.java PRE-CREATION
> shims/src/spark21/org/apache/pig/backend/hadoop/executionengine/spark/converter/SkewedJoinUtil.java PRE-CREATION
> shims/src/spark21/org/apache/pig/tools/pigstats/spark/SparkJobStats.java PRE-CREATION
> src/org/apache/pig/backend/hadoop/executionengine/spark/JobMetricsListener.java f81341233447203abc4800cc7b22a4f419e10262
> src/org/apache/pig/backend/hadoop/executionengine/spark/SparkLauncher.java c6351e01a48f297ea2e432401ffd65c4f27f8078
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/CollectedGroupConverter.java 83311dfa5bb25209a5366c2db7e8d483c31d94cd
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/FRJoinConverter.java 382258e7ff9105aa397c5a2888df0c11e9562ec9
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/ForEachConverter.java b58415e7e18ca4cf1331beef06e9214600a51424
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/GlobalRearrangeConverter.java f571b808839c2de9415a3e8e4b229a7f4b2eebd7
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/LimitConverter.java fe1b54c8f128661d7d19c276d3bb2de7874d3086
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/MergeCogroupConverter.java adf78ecab0da10d3b1a7fdde8af2b42dd899810f
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/MergeJoinConverter.java d1c43b1e06adc4c9fe45a83b81103333402e3756
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/PoissonSampleConverter.java e003bbd95763b2d189ff9ec540c89abe52592420
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SecondaryKeySortUtil.java 00d29b44848546ed16dde2baa8c61b36939971b2
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SkewedJoinConverter.java c55ba3145495a53d69db2dd56434dcc9b3bf8ed5
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SortConverter.java baabfa090323e3bef087e259ce19df2e4c34dd63
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SparkSampleSortConverter.java 3166fdc31745c013380492e089c83f3e853a3e6e
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/StreamConverter.java 3a50d485cfd54b9f3b9c1a982e6c30497a4c85fc
> src/org/apache/pig/tools/pigstats/spark/SparkJobStats.java c8cc03194b223d2ee181d73c6b651a6872cac6b6
>
>
> Diff: https://reviews.apache.org/r/59530/diff/1/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Nandor Kollar
>
>
Re: Review Request 59530: PIG-5157 Upgrade to Spark 2.0
Posted by Rohini Palaniswamy <ro...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59530/#review175998
-----------------------------------------------------------
build.xml
Lines 243 (patched)
<https://reviews.apache.org/r/59530/#comment249342>
Please have sparkversion as 1 and 2 and the shims directory names as well.
build.xml
Lines 251 (patched)
<https://reviews.apache.org/r/59530/#comment249344>
Can just load one property file for spark2, leaving the spark 1.6 version in libraries.properties
- Rohini Palaniswamy
On May 24, 2017, 4:21 p.m., Nandor Kollar wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/59530/
> -----------------------------------------------------------
>
> (Updated May 24, 2017, 4:21 p.m.)
>
>
> Review request for pig, liyun zhang, Rohini Palaniswamy, and Adam Szita.
>
>
> Repository: pig-git
>
>
> Description
> -------
>
> Upgrade to Spark 2.1 API using shims.
>
>
> Diffs
> -----
>
> build.xml 4040fcec8f88d448ed7442461fbf0dea8cd1136e
> ivy/libraries.properties a0eb00acd2df42324540df4a9d762c64c608a6d3
> ivy/spark1.properties PRE-CREATION
> ivy/spark2.properties PRE-CREATION
> shims/src/spark16/org/apache/pig/backend/hadoop/executionengine/spark/JobMetricsListener.java PRE-CREATION
> shims/src/spark16/org/apache/pig/backend/hadoop/executionengine/spark/converter/AbstractFlatMapFunction.java PRE-CREATION
> shims/src/spark16/org/apache/pig/backend/hadoop/executionengine/spark/converter/AbstractLimitConverter.java PRE-CREATION
> shims/src/spark16/org/apache/pig/backend/hadoop/executionengine/spark/converter/AbstractPairFlatMapFunction.java PRE-CREATION
> shims/src/spark16/org/apache/pig/backend/hadoop/executionengine/spark/converter/SkewedJoinUtil.java PRE-CREATION
> shims/src/spark16/org/apache/pig/tools/pigstats/spark/SparkJobStats.java PRE-CREATION
> shims/src/spark21/org/apache/pig/backend/hadoop/executionengine/spark/JobMetricsListener.java PRE-CREATION
> shims/src/spark21/org/apache/pig/backend/hadoop/executionengine/spark/converter/AbstractFlatMapFunction.java PRE-CREATION
> shims/src/spark21/org/apache/pig/backend/hadoop/executionengine/spark/converter/AbstractLimitConverter.java PRE-CREATION
> shims/src/spark21/org/apache/pig/backend/hadoop/executionengine/spark/converter/AbstractPairFlatMapFunction.java PRE-CREATION
> shims/src/spark21/org/apache/pig/backend/hadoop/executionengine/spark/converter/SkewedJoinUtil.java PRE-CREATION
> shims/src/spark21/org/apache/pig/tools/pigstats/spark/SparkJobStats.java PRE-CREATION
> src/org/apache/pig/backend/hadoop/executionengine/spark/JobMetricsListener.java f81341233447203abc4800cc7b22a4f419e10262
> src/org/apache/pig/backend/hadoop/executionengine/spark/SparkLauncher.java c6351e01a48f297ea2e432401ffd65c4f27f8078
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/CollectedGroupConverter.java 83311dfa5bb25209a5366c2db7e8d483c31d94cd
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/FRJoinConverter.java 382258e7ff9105aa397c5a2888df0c11e9562ec9
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/ForEachConverter.java b58415e7e18ca4cf1331beef06e9214600a51424
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/GlobalRearrangeConverter.java f571b808839c2de9415a3e8e4b229a7f4b2eebd7
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/LimitConverter.java fe1b54c8f128661d7d19c276d3bb2de7874d3086
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/MergeCogroupConverter.java adf78ecab0da10d3b1a7fdde8af2b42dd899810f
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/MergeJoinConverter.java d1c43b1e06adc4c9fe45a83b81103333402e3756
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/PoissonSampleConverter.java e003bbd95763b2d189ff9ec540c89abe52592420
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SecondaryKeySortUtil.java 00d29b44848546ed16dde2baa8c61b36939971b2
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SkewedJoinConverter.java c55ba3145495a53d69db2dd56434dcc9b3bf8ed5
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SortConverter.java baabfa090323e3bef087e259ce19df2e4c34dd63
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SparkSampleSortConverter.java 3166fdc31745c013380492e089c83f3e853a3e6e
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/StreamConverter.java 3a50d485cfd54b9f3b9c1a982e6c30497a4c85fc
> src/org/apache/pig/tools/pigstats/spark/SparkJobStats.java c8cc03194b223d2ee181d73c6b651a6872cac6b6
>
>
> Diff: https://reviews.apache.org/r/59530/diff/1/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Nandor Kollar
>
>
Re: Review Request 59530: PIG-5157 Upgrade to Spark 2.0
Posted by Nandor Kollar <nk...@cloudera.com>.
> On June 13, 2017, 7:33 a.m., kelly zhang wrote:
> > src/org/apache/pig/backend/hadoop/executionengine/spark/JobMetricsListener.java
> > Line 82 (original), 83 (patched)
> > <https://reviews.apache.org/r/59530/diff/4/?file=1749068#file1749068line83>
> >
> > I guess in spark2, the comment in onStageCompleted(SparkListenerStageCompleted stageCompleted) method should be uncommented, the code can be used in spark2.
> > but these code can not be used in spark1.Maybe you need create JobMetricsListener1, JobMetricsListener2
Indeed, there should be two separate JobMetricsListener: one for Spark 1.x and one for Spark 2.x. Thanks for catching this, will do this and update the review.
- Nandor
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59530/#review177699
-----------------------------------------------------------
On June 12, 2017, 9:20 p.m., Nandor Kollar wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/59530/
> -----------------------------------------------------------
>
> (Updated June 12, 2017, 9:20 p.m.)
>
>
> Review request for pig, liyun zhang, Rohini Palaniswamy, and Adam Szita.
>
>
> Repository: pig-git
>
>
> Description
> -------
>
> Upgrade to Spark 2.1 API using shims.
>
>
> Diffs
> -----
>
> build.xml 4040fcec8f88d448ed7442461fbf0dea8cd1136e
> ivy.xml 971724380f086d214ce62c7ab7879b08b6926802
> ivy/libraries.properties a0eb00acd2df42324540df4a9d762c64c608a6d3
> src/org/apache/pig/backend/hadoop/executionengine/spark/FlatMapFunctionAdapter.java PRE-CREATION
> src/org/apache/pig/backend/hadoop/executionengine/spark/JobMetricsListener.java f81341233447203abc4800cc7b22a4f419e10262
> src/org/apache/pig/backend/hadoop/executionengine/spark/PairFlatMapFunctionAdapter.java PRE-CREATION
> src/org/apache/pig/backend/hadoop/executionengine/spark/SparkLauncher.java c6351e01a48f297ea2e432401ffd65c4f27f8078
> src/org/apache/pig/backend/hadoop/executionengine/spark/SparkShim.java PRE-CREATION
> src/org/apache/pig/backend/hadoop/executionengine/spark/SparkShim1.java PRE-CREATION
> src/org/apache/pig/backend/hadoop/executionengine/spark/SparkShim2.java PRE-CREATION
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/CollectedGroupConverter.java 83311dfa5bb25209a5366c2db7e8d483c31d94cd
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/FRJoinConverter.java 382258e7ff9105aa397c5a2888df0c11e9562ec9
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/ForEachConverter.java b58415e7e18ca4cf1331beef06e9214600a51424
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/GlobalRearrangeConverter.java f571b808839c2de9415a3e8e4b229a7f4b2eebd7
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/LimitConverter.java fe1b54c8f128661d7d19c276d3bb2de7874d3086
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/MergeCogroupConverter.java adf78ecab0da10d3b1a7fdde8af2b42dd899810f
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/MergeJoinConverter.java d1c43b1e06adc4c9fe45a83b81103333402e3756
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/PoissonSampleConverter.java e003bbd95763b2d189ff9ec540c89abe52592420
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SecondaryKeySortUtil.java 00d29b44848546ed16dde2baa8c61b36939971b2
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SkewedJoinConverter.java c55ba3145495a53d69db2dd56434dcc9b3bf8ed5
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SortConverter.java baabfa090323e3bef087e259ce19df2e4c34dd63
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SparkSampleSortConverter.java 3166fdc31745c013380492e089c83f3e853a3e6e
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/StreamConverter.java 3a50d485cfd54b9f3b9c1a982e6c30497a4c85fc
> src/org/apache/pig/tools/pigstats/spark/SparkJobStats.java c8cc03194b223d2ee181d73c6b651a6872cac6b6
> src/org/apache/pig/tools/pigstats/spark/SparkJobStats1.java PRE-CREATION
> src/org/apache/pig/tools/pigstats/spark/SparkJobStats2.java PRE-CREATION
> src/org/apache/pig/tools/pigstats/spark/SparkPigStats.java 61ccbcc9fd723f6e2e578a8476230c42d5587dfe
> test/org/apache/pig/test/TestPigRunner.java 1b1b11b0ccaca2cddeb3ad8ddc659a1c8a75722c
>
>
> Diff: https://reviews.apache.org/r/59530/diff/4/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Nandor Kollar
>
>
RE: Review Request 59530: PIG-5157 Upgrade to Spark 2.0
Posted by "Zhang, Liyun" <li...@intel.com>.
No need to add null checks. Thanks for explanation. Please update the patch with latest trunk code and this makes me easy to test the code.
Best Regards
Kelly Zhang/Zhang,Liyun
-----Original Message-----
From: Nandor Kollar [mailto:noreply@reviews.apache.org] On Behalf Of Nandor Kollar
Sent: Tuesday, June 13, 2017 3:52 PM
To: Rohini Palaniswamy <ro...@gmail.com>; Zhang, Liyun <li...@intel.com>; Adam Szita <sz...@cloudera.com>
Cc: Zhang, Liyun <li...@intel.com>; pig <de...@pig.apache.org>; Nandor Kollar <nk...@cloudera.com>
Subject: Re: Review Request 59530: PIG-5157 Upgrade to Spark 2.0
> On June 13, 2017, 7:33 a.m., kelly zhang wrote:
> > src/org/apache/pig/tools/pigstats/spark/SparkJobStats1.java
> > Lines 60-63 (patched)
> > <https://reviews.apache.org/r/59530/diff/4/?file=1749088#file1749088line60>
> >
> > why inputMetricExists,outputMetricExist,shuffleReadMetricExist,shuffleWriteMetricExist are deleted in SparkJobStats2.java?
These metrics are no longer Optional, but have an initialized value. For example this is the shuffleWriteMetrics: val shuffleWriteMetrics: ShuffleWriteMetrics = new ShuffleWriteMetrics(); I don't think we need to check for the existence of these metrics, they are initialized all the time, but if you think it is better to check for null before using them, I can add null checks. for it.
- Nandor
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59530/#review177699
-----------------------------------------------------------
On June 12, 2017, 9:20 p.m., Nandor Kollar wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/59530/
> -----------------------------------------------------------
>
> (Updated June 12, 2017, 9:20 p.m.)
>
>
> Review request for pig, liyun zhang, Rohini Palaniswamy, and Adam Szita.
>
>
> Repository: pig-git
>
>
> Description
> -------
>
> Upgrade to Spark 2.1 API using shims.
>
>
> Diffs
> -----
>
> build.xml 4040fcec8f88d448ed7442461fbf0dea8cd1136e
> ivy.xml 971724380f086d214ce62c7ab7879b08b6926802
> ivy/libraries.properties a0eb00acd2df42324540df4a9d762c64c608a6d3
> src/org/apache/pig/backend/hadoop/executionengine/spark/FlatMapFunctionAdapter.java PRE-CREATION
> src/org/apache/pig/backend/hadoop/executionengine/spark/JobMetricsListener.java f81341233447203abc4800cc7b22a4f419e10262
> src/org/apache/pig/backend/hadoop/executionengine/spark/PairFlatMapFunctionAdapter.java PRE-CREATION
> src/org/apache/pig/backend/hadoop/executionengine/spark/SparkLauncher.java c6351e01a48f297ea2e432401ffd65c4f27f8078
> src/org/apache/pig/backend/hadoop/executionengine/spark/SparkShim.java PRE-CREATION
> src/org/apache/pig/backend/hadoop/executionengine/spark/SparkShim1.java PRE-CREATION
> src/org/apache/pig/backend/hadoop/executionengine/spark/SparkShim2.java PRE-CREATION
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/CollectedGroupConverter.java 83311dfa5bb25209a5366c2db7e8d483c31d94cd
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/FRJoinConverter.java 382258e7ff9105aa397c5a2888df0c11e9562ec9
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/ForEachConverter.java b58415e7e18ca4cf1331beef06e9214600a51424
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/GlobalRearrangeConverter.java f571b808839c2de9415a3e8e4b229a7f4b2eebd7
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/LimitConverter.java fe1b54c8f128661d7d19c276d3bb2de7874d3086
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/MergeCogroupConverter.java adf78ecab0da10d3b1a7fdde8af2b42dd899810f
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/MergeJoinConverter.java d1c43b1e06adc4c9fe45a83b81103333402e3756
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/PoissonSampleConverter.java e003bbd95763b2d189ff9ec540c89abe52592420
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SecondaryKeySortUtil.java 00d29b44848546ed16dde2baa8c61b36939971b2
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SkewedJoinConverter.java c55ba3145495a53d69db2dd56434dcc9b3bf8ed5
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SortConverter.java baabfa090323e3bef087e259ce19df2e4c34dd63
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SparkSampleSortConverter.java 3166fdc31745c013380492e089c83f3e853a3e6e
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/StreamConverter.java 3a50d485cfd54b9f3b9c1a982e6c30497a4c85fc
> src/org/apache/pig/tools/pigstats/spark/SparkJobStats.java c8cc03194b223d2ee181d73c6b651a6872cac6b6
> src/org/apache/pig/tools/pigstats/spark/SparkJobStats1.java PRE-CREATION
> src/org/apache/pig/tools/pigstats/spark/SparkJobStats2.java PRE-CREATION
> src/org/apache/pig/tools/pigstats/spark/SparkPigStats.java 61ccbcc9fd723f6e2e578a8476230c42d5587dfe
> test/org/apache/pig/test/TestPigRunner.java 1b1b11b0ccaca2cddeb3ad8ddc659a1c8a75722c
>
>
> Diff: https://reviews.apache.org/r/59530/diff/4/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Nandor Kollar
>
>
Re: Review Request 59530: PIG-5157 Upgrade to Spark 2.0
Posted by Nandor Kollar <nk...@cloudera.com>.
> On June 13, 2017, 7:33 a.m., kelly zhang wrote:
> > src/org/apache/pig/tools/pigstats/spark/SparkJobStats1.java
> > Lines 60-63 (patched)
> > <https://reviews.apache.org/r/59530/diff/4/?file=1749088#file1749088line60>
> >
> > why inputMetricExists,outputMetricExist,shuffleReadMetricExist,shuffleWriteMetricExist are deleted in SparkJobStats2.java?
These metrics are no longer Optional, but have an initialized value. For example this is the shuffleWriteMetrics: val shuffleWriteMetrics: ShuffleWriteMetrics = new ShuffleWriteMetrics(); I don't think we need to check for the existence of these metrics, they are initialized all the time, but if you think it is better to check for null before using them, I can add null checks. for it.
- Nandor
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59530/#review177699
-----------------------------------------------------------
On June 12, 2017, 9:20 p.m., Nandor Kollar wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/59530/
> -----------------------------------------------------------
>
> (Updated June 12, 2017, 9:20 p.m.)
>
>
> Review request for pig, liyun zhang, Rohini Palaniswamy, and Adam Szita.
>
>
> Repository: pig-git
>
>
> Description
> -------
>
> Upgrade to Spark 2.1 API using shims.
>
>
> Diffs
> -----
>
> build.xml 4040fcec8f88d448ed7442461fbf0dea8cd1136e
> ivy.xml 971724380f086d214ce62c7ab7879b08b6926802
> ivy/libraries.properties a0eb00acd2df42324540df4a9d762c64c608a6d3
> src/org/apache/pig/backend/hadoop/executionengine/spark/FlatMapFunctionAdapter.java PRE-CREATION
> src/org/apache/pig/backend/hadoop/executionengine/spark/JobMetricsListener.java f81341233447203abc4800cc7b22a4f419e10262
> src/org/apache/pig/backend/hadoop/executionengine/spark/PairFlatMapFunctionAdapter.java PRE-CREATION
> src/org/apache/pig/backend/hadoop/executionengine/spark/SparkLauncher.java c6351e01a48f297ea2e432401ffd65c4f27f8078
> src/org/apache/pig/backend/hadoop/executionengine/spark/SparkShim.java PRE-CREATION
> src/org/apache/pig/backend/hadoop/executionengine/spark/SparkShim1.java PRE-CREATION
> src/org/apache/pig/backend/hadoop/executionengine/spark/SparkShim2.java PRE-CREATION
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/CollectedGroupConverter.java 83311dfa5bb25209a5366c2db7e8d483c31d94cd
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/FRJoinConverter.java 382258e7ff9105aa397c5a2888df0c11e9562ec9
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/ForEachConverter.java b58415e7e18ca4cf1331beef06e9214600a51424
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/GlobalRearrangeConverter.java f571b808839c2de9415a3e8e4b229a7f4b2eebd7
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/LimitConverter.java fe1b54c8f128661d7d19c276d3bb2de7874d3086
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/MergeCogroupConverter.java adf78ecab0da10d3b1a7fdde8af2b42dd899810f
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/MergeJoinConverter.java d1c43b1e06adc4c9fe45a83b81103333402e3756
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/PoissonSampleConverter.java e003bbd95763b2d189ff9ec540c89abe52592420
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SecondaryKeySortUtil.java 00d29b44848546ed16dde2baa8c61b36939971b2
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SkewedJoinConverter.java c55ba3145495a53d69db2dd56434dcc9b3bf8ed5
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SortConverter.java baabfa090323e3bef087e259ce19df2e4c34dd63
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SparkSampleSortConverter.java 3166fdc31745c013380492e089c83f3e853a3e6e
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/StreamConverter.java 3a50d485cfd54b9f3b9c1a982e6c30497a4c85fc
> src/org/apache/pig/tools/pigstats/spark/SparkJobStats.java c8cc03194b223d2ee181d73c6b651a6872cac6b6
> src/org/apache/pig/tools/pigstats/spark/SparkJobStats1.java PRE-CREATION
> src/org/apache/pig/tools/pigstats/spark/SparkJobStats2.java PRE-CREATION
> src/org/apache/pig/tools/pigstats/spark/SparkPigStats.java 61ccbcc9fd723f6e2e578a8476230c42d5587dfe
> test/org/apache/pig/test/TestPigRunner.java 1b1b11b0ccaca2cddeb3ad8ddc659a1c8a75722c
>
>
> Diff: https://reviews.apache.org/r/59530/diff/4/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Nandor Kollar
>
>
Re: Review Request 59530: PIG-5157 Upgrade to Spark 2.0
Posted by kelly zhang <li...@intel.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59530/#review177699
-----------------------------------------------------------
src/org/apache/pig/backend/hadoop/executionengine/spark/JobMetricsListener.java
Line 82 (original), 83 (patched)
<https://reviews.apache.org/r/59530/#comment251429>
I guess in spark2, the comment in onStageCompleted(SparkListenerStageCompleted stageCompleted) method should be uncommented, the code can be used in spark2.
but these code can not be used in spark1.Maybe you need create JobMetricsListener1, JobMetricsListener2
src/org/apache/pig/tools/pigstats/spark/SparkJobStats1.java
Lines 60-63 (patched)
<https://reviews.apache.org/r/59530/#comment251427>
why inputMetricExists,outputMetricExist,shuffleReadMetricExist,shuffleWriteMetricExist are deleted in SparkJobStats2.java?
- kelly zhang
On June 12, 2017, 9:20 p.m., Nandor Kollar wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/59530/
> -----------------------------------------------------------
>
> (Updated June 12, 2017, 9:20 p.m.)
>
>
> Review request for pig, liyun zhang, Rohini Palaniswamy, and Adam Szita.
>
>
> Repository: pig-git
>
>
> Description
> -------
>
> Upgrade to Spark 2.1 API using shims.
>
>
> Diffs
> -----
>
> build.xml 4040fcec8f88d448ed7442461fbf0dea8cd1136e
> ivy.xml 971724380f086d214ce62c7ab7879b08b6926802
> ivy/libraries.properties a0eb00acd2df42324540df4a9d762c64c608a6d3
> src/org/apache/pig/backend/hadoop/executionengine/spark/FlatMapFunctionAdapter.java PRE-CREATION
> src/org/apache/pig/backend/hadoop/executionengine/spark/JobMetricsListener.java f81341233447203abc4800cc7b22a4f419e10262
> src/org/apache/pig/backend/hadoop/executionengine/spark/PairFlatMapFunctionAdapter.java PRE-CREATION
> src/org/apache/pig/backend/hadoop/executionengine/spark/SparkLauncher.java c6351e01a48f297ea2e432401ffd65c4f27f8078
> src/org/apache/pig/backend/hadoop/executionengine/spark/SparkShim.java PRE-CREATION
> src/org/apache/pig/backend/hadoop/executionengine/spark/SparkShim1.java PRE-CREATION
> src/org/apache/pig/backend/hadoop/executionengine/spark/SparkShim2.java PRE-CREATION
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/CollectedGroupConverter.java 83311dfa5bb25209a5366c2db7e8d483c31d94cd
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/FRJoinConverter.java 382258e7ff9105aa397c5a2888df0c11e9562ec9
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/ForEachConverter.java b58415e7e18ca4cf1331beef06e9214600a51424
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/GlobalRearrangeConverter.java f571b808839c2de9415a3e8e4b229a7f4b2eebd7
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/LimitConverter.java fe1b54c8f128661d7d19c276d3bb2de7874d3086
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/MergeCogroupConverter.java adf78ecab0da10d3b1a7fdde8af2b42dd899810f
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/MergeJoinConverter.java d1c43b1e06adc4c9fe45a83b81103333402e3756
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/PoissonSampleConverter.java e003bbd95763b2d189ff9ec540c89abe52592420
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SecondaryKeySortUtil.java 00d29b44848546ed16dde2baa8c61b36939971b2
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SkewedJoinConverter.java c55ba3145495a53d69db2dd56434dcc9b3bf8ed5
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SortConverter.java baabfa090323e3bef087e259ce19df2e4c34dd63
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SparkSampleSortConverter.java 3166fdc31745c013380492e089c83f3e853a3e6e
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/StreamConverter.java 3a50d485cfd54b9f3b9c1a982e6c30497a4c85fc
> src/org/apache/pig/tools/pigstats/spark/SparkJobStats.java c8cc03194b223d2ee181d73c6b651a6872cac6b6
> src/org/apache/pig/tools/pigstats/spark/SparkJobStats1.java PRE-CREATION
> src/org/apache/pig/tools/pigstats/spark/SparkJobStats2.java PRE-CREATION
> src/org/apache/pig/tools/pigstats/spark/SparkPigStats.java 61ccbcc9fd723f6e2e578a8476230c42d5587dfe
> test/org/apache/pig/test/TestPigRunner.java 1b1b11b0ccaca2cddeb3ad8ddc659a1c8a75722c
>
>
> Diff: https://reviews.apache.org/r/59530/diff/4/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Nandor Kollar
>
>
Re: Review Request 59530: PIG-5157 Upgrade to Spark 2.0
Posted by Nandor Kollar <nk...@cloudera.com>.
> On June 20, 2017, 8:43 p.m., Rohini Palaniswamy wrote:
> > build.xml
> > Line 795 (original), 795 (patched)
> > <https://reviews.apache.org/r/59530/diff/5-6/?file=1750170#file1750170line795>
> >
> > Instead of changing all references of jar to jar-simple, you can just compile Spark 1 at the end so that dependencies for test remain same as doing jar-simple. HBase2 can be later added to the Spark2 compilation and HBase1 to Spark1.
> >
> >
> > <target name="jar-core" depends="compile,ivy-buildJar" description="Create only pig core jar">
> > <buildJar svnString="${svn.revision}" outputFile="${output.jarfile.core}" includedJars="core.dependencies.jar"/>
> > </target>
> >
> > <target name="jar" description="Create pig jar with Spark 1 and 2">
> > <echo>Compiling against Spark 2</echo>
> > <antcall target="clean" inheritRefs="true" inheritall="true"/>
> > <propertyreset name="sparkversion" value="2"/>
> > <propertyreset name="src.exclude.dir" value="**/Spark1*.java" />
> > <antcall target="jar-core" inheritRefs="true" inheritall="true"/>
> > <move file="${output.jarfile.core}" tofile="${basedir}/_pig-shims.jar"/>
> >
> > <echo>Compiling against Spark 1</echo>
> > <antcall target="clean" inheritRefs="true" inheritall="true"/>
> > <propertyreset name="sparkversion" value="1"/>
> > <propertyreset name="src.exclude.dir" value="**/Spark2*.java" />
> > <antcall target="jar-simple" inheritRefs="true" inheritall="true"/>
> > <jar update="yes" jarfile="${output.jarfile.core}">
> > <zipfileset src="${basedir}/_pig-shims.jar" includes="**/Spark2*.class"/>
> > </jar>
> > <jar update="yes" jarfile="${output.jarfile.backcompat-core-h2}">
> > <zipfileset src="${basedir}/_pig-shims.jar" includes="**/Spark2*.class"/>
> > </jar>
> > <jar update="yes" jarfile="${output.jarfile.withouthadoop-h2}">
> > <zipfileset src="${basedir}/_pig-shims.jar" includes="**/Spark2*.class"/>
> > </jar>
> > <delete file="${basedir}/_pig-shims.jar"/>
> > </target>
Thanks Rohini, make sense. Modified the targets accordingly, could you please have a look at the latest patch?
- Nandor
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59530/#review178421
-----------------------------------------------------------
On June 21, 2017, 9:06 p.m., Nandor Kollar wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/59530/
> -----------------------------------------------------------
>
> (Updated June 21, 2017, 9:06 p.m.)
>
>
> Review request for pig, liyun zhang, Rohini Palaniswamy, and Adam Szita.
>
>
> Repository: pig-git
>
>
> Description
> -------
>
> Upgrade to Spark 2.1 API using shims.
>
>
> Diffs
> -----
>
> build.xml bba2b52d9354ab909ad26f969480806f6d91911c
> ivy.xml 3f2c94373ba9455bbb6a3c96bfd61fc6cfaab588
> ivy/libraries.properties c2aed45a3244dfd108a255c7308a7dcb0dabd3b5
> src/org/apache/pig/backend/hadoop/executionengine/spark/FlatMapFunctionAdapter.java PRE-CREATION
> src/org/apache/pig/backend/hadoop/executionengine/spark/JobMetricsListener.java f81341233447203abc4800cc7b22a4f419e10262
> src/org/apache/pig/backend/hadoop/executionengine/spark/PairFlatMapFunctionAdapter.java PRE-CREATION
> src/org/apache/pig/backend/hadoop/executionengine/spark/Spark1Shims.java PRE-CREATION
> src/org/apache/pig/backend/hadoop/executionengine/spark/Spark2Shims.java PRE-CREATION
> src/org/apache/pig/backend/hadoop/executionengine/spark/SparkLauncher.java 237fd9431a16226234d91059088f91aab346b83c
> src/org/apache/pig/backend/hadoop/executionengine/spark/SparkShims.java PRE-CREATION
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/CollectedGroupConverter.java 83311dfa5bb25209a5366c2db7e8d483c31d94cd
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/FRJoinConverter.java 382258e7ff9105aa397c5a2888df0c11e9562ec9
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/ForEachConverter.java b58415e7e18ca4cf1331beef06e9214600a51424
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/GlobalRearrangeConverter.java 130c8b9a747b176ce2b649ca6d5260527595fb76
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/LimitConverter.java fe1b54c8f128661d7d19c276d3bb2de7874d3086
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/MergeCogroupConverter.java adf78ecab0da10d3b1a7fdde8af2b42dd899810f
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/MergeJoinConverter.java d1c43b1e06adc4c9fe45a83b81103333402e3756
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/PoissonSampleConverter.java e003bbd95763b2d189ff9ec540c89abe52592420
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SecondaryKeySortUtil.java 00d29b44848546ed16dde2baa8c61b36939971b2
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SkewedJoinConverter.java c55ba3145495a53d69db2dd56434dcc9b3bf8ed5
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SortConverter.java baabfa090323e3bef087e259ce19df2e4c34dd63
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SparkSampleSortConverter.java 3166fdc31745c013380492e089c83f3e853a3e6e
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/StreamConverter.java 3a50d485cfd54b9f3b9c1a982e6c30497a4c85fc
> src/org/apache/pig/tools/pigstats/spark/Spark1JobStats.java PRE-CREATION
> src/org/apache/pig/tools/pigstats/spark/Spark2JobStats.java PRE-CREATION
> src/org/apache/pig/tools/pigstats/spark/SparkJobStats.java c8cc03194b223d2ee181d73c6b651a6872cac6b6
> src/org/apache/pig/tools/pigstats/spark/SparkPigStats.java 61ccbcc9fd723f6e2e578a8476230c42d5587dfe
> test/org/apache/pig/test/TestPigRunner.java ec08417f2f71deec514cab5cfb9d2f99520ad641
>
>
> Diff: https://reviews.apache.org/r/59530/diff/7/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Nandor Kollar
>
>
Re: Review Request 59530: PIG-5157 Upgrade to Spark 2.0
Posted by Rohini Palaniswamy <ro...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59530/#review178421
-----------------------------------------------------------
build.xml
Line 795 (original), 795 (patched)
<https://reviews.apache.org/r/59530/#comment252369>
Instead of changing all references of jar to jar-simple, you can just compile Spark 1 at the end so that dependencies for test remain same as doing jar-simple. HBase2 can be later added to the Spark2 compilation and HBase1 to Spark1.
<target name="jar-core" depends="compile,ivy-buildJar" description="Create only pig core jar">
<buildJar svnString="${svn.revision}" outputFile="${output.jarfile.core}" includedJars="core.dependencies.jar"/>
</target>
<target name="jar" description="Create pig jar with Spark 1 and 2">
<echo>Compiling against Spark 2</echo>
<antcall target="clean" inheritRefs="true" inheritall="true"/>
<propertyreset name="sparkversion" value="2"/>
<propertyreset name="src.exclude.dir" value="**/Spark1*.java" />
<antcall target="jar-core" inheritRefs="true" inheritall="true"/>
<move file="${output.jarfile.core}" tofile="${basedir}/_pig-shims.jar"/>
<echo>Compiling against Spark 1</echo>
<antcall target="clean" inheritRefs="true" inheritall="true"/>
<propertyreset name="sparkversion" value="1"/>
<propertyreset name="src.exclude.dir" value="**/Spark2*.java" />
<antcall target="jar-simple" inheritRefs="true" inheritall="true"/>
<jar update="yes" jarfile="${output.jarfile.core}">
<zipfileset src="${basedir}/_pig-shims.jar" includes="**/Spark2*.class"/>
</jar>
<jar update="yes" jarfile="${output.jarfile.backcompat-core-h2}">
<zipfileset src="${basedir}/_pig-shims.jar" includes="**/Spark2*.class"/>
</jar>
<jar update="yes" jarfile="${output.jarfile.withouthadoop-h2}">
<zipfileset src="${basedir}/_pig-shims.jar" includes="**/Spark2*.class"/>
</jar>
<delete file="${basedir}/_pig-shims.jar"/>
</target>
- Rohini Palaniswamy
On June 19, 2017, 9:41 a.m., Nandor Kollar wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/59530/
> -----------------------------------------------------------
>
> (Updated June 19, 2017, 9:41 a.m.)
>
>
> Review request for pig, liyun zhang, Rohini Palaniswamy, and Adam Szita.
>
>
> Repository: pig-git
>
>
> Description
> -------
>
> Upgrade to Spark 2.1 API using shims.
>
>
> Diffs
> -----
>
> build.xml bba2b52d9354ab909ad26f969480806f6d91911c
> ivy.xml 3f2c94373ba9455bbb6a3c96bfd61fc6cfaab588
> ivy/libraries.properties c2aed45a3244dfd108a255c7308a7dcb0dabd3b5
> src/org/apache/pig/backend/hadoop/executionengine/spark/FlatMapFunctionAdapter.java PRE-CREATION
> src/org/apache/pig/backend/hadoop/executionengine/spark/JobMetricsListener.java f81341233447203abc4800cc7b22a4f419e10262
> src/org/apache/pig/backend/hadoop/executionengine/spark/PairFlatMapFunctionAdapter.java PRE-CREATION
> src/org/apache/pig/backend/hadoop/executionengine/spark/Spark1Shims.java PRE-CREATION
> src/org/apache/pig/backend/hadoop/executionengine/spark/Spark2Shims.java PRE-CREATION
> src/org/apache/pig/backend/hadoop/executionengine/spark/SparkLauncher.java 237fd9431a16226234d91059088f91aab346b83c
> src/org/apache/pig/backend/hadoop/executionengine/spark/SparkShims.java PRE-CREATION
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/CollectedGroupConverter.java 83311dfa5bb25209a5366c2db7e8d483c31d94cd
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/FRJoinConverter.java 382258e7ff9105aa397c5a2888df0c11e9562ec9
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/ForEachConverter.java b58415e7e18ca4cf1331beef06e9214600a51424
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/GlobalRearrangeConverter.java 130c8b9a747b176ce2b649ca6d5260527595fb76
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/LimitConverter.java fe1b54c8f128661d7d19c276d3bb2de7874d3086
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/MergeCogroupConverter.java adf78ecab0da10d3b1a7fdde8af2b42dd899810f
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/MergeJoinConverter.java d1c43b1e06adc4c9fe45a83b81103333402e3756
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/PoissonSampleConverter.java e003bbd95763b2d189ff9ec540c89abe52592420
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SecondaryKeySortUtil.java 00d29b44848546ed16dde2baa8c61b36939971b2
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SkewedJoinConverter.java c55ba3145495a53d69db2dd56434dcc9b3bf8ed5
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SortConverter.java baabfa090323e3bef087e259ce19df2e4c34dd63
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SparkSampleSortConverter.java 3166fdc31745c013380492e089c83f3e853a3e6e
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/StreamConverter.java 3a50d485cfd54b9f3b9c1a982e6c30497a4c85fc
> src/org/apache/pig/tools/pigstats/spark/Spark1JobStats.java PRE-CREATION
> src/org/apache/pig/tools/pigstats/spark/Spark2JobStats.java PRE-CREATION
> src/org/apache/pig/tools/pigstats/spark/SparkJobStats.java c8cc03194b223d2ee181d73c6b651a6872cac6b6
> src/org/apache/pig/tools/pigstats/spark/SparkPigStats.java 61ccbcc9fd723f6e2e578a8476230c42d5587dfe
> test/org/apache/pig/test/TestPigRunner.java ec08417f2f71deec514cab5cfb9d2f99520ad641
>
>
> Diff: https://reviews.apache.org/r/59530/diff/6/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Nandor Kollar
>
>
Re: Review Request 59530: PIG-5157 Upgrade to Spark 2.0
Posted by kelly zhang <li...@intel.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59530/#review180657
-----------------------------------------------------------
Fix it then Ship it
src/org/apache/pig/backend/hadoop/executionengine/spark/JobMetricsListener.java
Line 32 (original), 32 (patched)
<https://reviews.apache.org/r/59530/#comment255850>
There are two JobMetricsListener.
org.apache.pig.backend.hadoop.executionengine.spark.Spark1Shims.JobMetricsListener
org.apache.pig.backend.hadoop.executionengine.spark.JobMetricsListener
Can you rename to make it more understandable?
- kelly zhang
On June 28, 2017, 8:55 a.m., Nandor Kollar wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/59530/
> -----------------------------------------------------------
>
> (Updated June 28, 2017, 8:55 a.m.)
>
>
> Review request for pig, liyun zhang, Rohini Palaniswamy, and Adam Szita.
>
>
> Repository: pig-git
>
>
> Description
> -------
>
> Upgrade to Spark 2.1 API using shims.
>
>
> Diffs
> -----
>
> build.xml d274af9be3805f5167f11afed9d85f19e72916d8
> ivy.xml 3f2c94373ba9455bbb6a3c96bfd61fc6cfaab588
> ivy/libraries.properties c2aed45a3244dfd108a255c7308a7dcb0dabd3b5
> src/org/apache/pig/backend/hadoop/executionengine/spark/FlatMapFunctionAdapter.java PRE-CREATION
> src/org/apache/pig/backend/hadoop/executionengine/spark/JobMetricsListener.java f81341233447203abc4800cc7b22a4f419e10262
> src/org/apache/pig/backend/hadoop/executionengine/spark/PairFlatMapFunctionAdapter.java PRE-CREATION
> src/org/apache/pig/backend/hadoop/executionengine/spark/Spark1Shims.java PRE-CREATION
> src/org/apache/pig/backend/hadoop/executionengine/spark/Spark2Shims.java PRE-CREATION
> src/org/apache/pig/backend/hadoop/executionengine/spark/SparkLauncher.java 237fd9431a16226234d91059088f91aab346b83c
> src/org/apache/pig/backend/hadoop/executionengine/spark/SparkShims.java PRE-CREATION
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/CollectedGroupConverter.java 83311dfa5bb25209a5366c2db7e8d483c31d94cd
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/FRJoinConverter.java 382258e7ff9105aa397c5a2888df0c11e9562ec9
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/ForEachConverter.java b58415e7e18ca4cf1331beef06e9214600a51424
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/GlobalRearrangeConverter.java 130c8b9a747b176ce2b649ca6d5260527595fb76
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/LimitConverter.java fe1b54c8f128661d7d19c276d3bb2de7874d3086
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/MergeCogroupConverter.java adf78ecab0da10d3b1a7fdde8af2b42dd899810f
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/MergeJoinConverter.java d1c43b1e06adc4c9fe45a83b81103333402e3756
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/PoissonSampleConverter.java e003bbd95763b2d189ff9ec540c89abe52592420
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SecondaryKeySortUtil.java 00d29b44848546ed16dde2baa8c61b36939971b2
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SkewedJoinConverter.java c55ba3145495a53d69db2dd56434dcc9b3bf8ed5
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SortConverter.java baabfa090323e3bef087e259ce19df2e4c34dd63
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SparkSampleSortConverter.java 3166fdc31745c013380492e089c83f3e853a3e6e
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/StreamConverter.java 3a50d485cfd54b9f3b9c1a982e6c30497a4c85fc
> src/org/apache/pig/tools/pigstats/spark/Spark1JobStats.java PRE-CREATION
> src/org/apache/pig/tools/pigstats/spark/Spark2JobStats.java PRE-CREATION
> src/org/apache/pig/tools/pigstats/spark/SparkJobStats.java c8cc03194b223d2ee181d73c6b651a6872cac6b6
> src/org/apache/pig/tools/pigstats/spark/SparkPigStats.java 61ccbcc9fd723f6e2e578a8476230c42d5587dfe
> test/org/apache/pig/test/TestPigRunner.java ec08417f2f71deec514cab5cfb9d2f99520ad641
>
>
> Diff: https://reviews.apache.org/r/59530/diff/13/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Nandor Kollar
>
>
Re: Review Request 59530: PIG-5157 Upgrade to Spark 2.0
Posted by Nandor Kollar <nk...@cloudera.com>.
> On July 18, 2017, 7:16 a.m., kelly zhang wrote:
> > src/org/apache/pig/backend/hadoop/executionengine/spark/Spark1Shims.java
> > Line 133 (original), 133 (patched)
> > <https://reviews.apache.org/r/59530/diff/13-14/?file=1766113#file1766113line133>
> >
> > How about change org.apache.pig.backend.hadoop.executionengine.spark.JobMetricsListener to org.apache.pig.backend.hadoop.executionengine.spark.JobStatisticCollector?
> > and rename JobMetricsListener#allJobMetrics to JobStatisticCollector#allJobStatistics?
> >
> >
> > Just suggestion to make the code more better.
Ok, makes sense, renamed the classes and the private field according to you recommendation.
- Nandor
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59530/#review180799
-----------------------------------------------------------
On July 18, 2017, 11:38 a.m., Nandor Kollar wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/59530/
> -----------------------------------------------------------
>
> (Updated July 18, 2017, 11:38 a.m.)
>
>
> Review request for pig, liyun zhang, Rohini Palaniswamy, and Adam Szita.
>
>
> Repository: pig-git
>
>
> Description
> -------
>
> Upgrade to Spark 2.1 API using shims.
>
>
> Diffs
> -----
>
> build.xml d274af9be3805f5167f11afed9d85f19e72916d8
> ivy.xml 3f2c94373ba9455bbb6a3c96bfd61fc6cfaab588
> ivy/libraries.properties c2aed45a3244dfd108a255c7308a7dcb0dabd3b5
> src/org/apache/pig/backend/hadoop/executionengine/spark/FlatMapFunctionAdapter.java PRE-CREATION
> src/org/apache/pig/backend/hadoop/executionengine/spark/JobGraphBuilder.java 5eac045dc6b7b7f3e06cf177e97d19e7d7f43010
> src/org/apache/pig/backend/hadoop/executionengine/spark/JobMetricsListener.java f81341233447203abc4800cc7b22a4f419e10262
> src/org/apache/pig/backend/hadoop/executionengine/spark/JobStatisticCollector.java PRE-CREATION
> src/org/apache/pig/backend/hadoop/executionengine/spark/PairFlatMapFunctionAdapter.java PRE-CREATION
> src/org/apache/pig/backend/hadoop/executionengine/spark/Spark1Shims.java PRE-CREATION
> src/org/apache/pig/backend/hadoop/executionengine/spark/Spark2Shims.java PRE-CREATION
> src/org/apache/pig/backend/hadoop/executionengine/spark/SparkLauncher.java 237fd9431a16226234d91059088f91aab346b83c
> src/org/apache/pig/backend/hadoop/executionengine/spark/SparkShims.java PRE-CREATION
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/CollectedGroupConverter.java 83311dfa5bb25209a5366c2db7e8d483c31d94cd
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/FRJoinConverter.java 382258e7ff9105aa397c5a2888df0c11e9562ec9
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/ForEachConverter.java b58415e7e18ca4cf1331beef06e9214600a51424
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/GlobalRearrangeConverter.java 130c8b9a747b176ce2b649ca6d5260527595fb76
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/LimitConverter.java fe1b54c8f128661d7d19c276d3bb2de7874d3086
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/MergeCogroupConverter.java adf78ecab0da10d3b1a7fdde8af2b42dd899810f
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/MergeJoinConverter.java d1c43b1e06adc4c9fe45a83b81103333402e3756
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/PoissonSampleConverter.java e003bbd95763b2d189ff9ec540c89abe52592420
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SecondaryKeySortUtil.java 00d29b44848546ed16dde2baa8c61b36939971b2
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SkewedJoinConverter.java c55ba3145495a53d69db2dd56434dcc9b3bf8ed5
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SortConverter.java baabfa090323e3bef087e259ce19df2e4c34dd63
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SparkSampleSortConverter.java 3166fdc31745c013380492e089c83f3e853a3e6e
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/StreamConverter.java 3a50d485cfd54b9f3b9c1a982e6c30497a4c85fc
> src/org/apache/pig/tools/pigstats/spark/Spark1JobStats.java PRE-CREATION
> src/org/apache/pig/tools/pigstats/spark/Spark2JobStats.java PRE-CREATION
> src/org/apache/pig/tools/pigstats/spark/SparkJobStats.java c8cc03194b223d2ee181d73c6b651a6872cac6b6
> src/org/apache/pig/tools/pigstats/spark/SparkPigStats.java 61ccbcc9fd723f6e2e578a8476230c42d5587dfe
> src/org/apache/pig/tools/pigstats/spark/SparkStatsUtil.java 1541264c4088fbe4770e706fd8fc53907fa83796
> test/org/apache/pig/test/TestPigRunner.java ec08417f2f71deec514cab5cfb9d2f99520ad641
>
>
> Diff: https://reviews.apache.org/r/59530/diff/15/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Nandor Kollar
>
>
Re: Review Request 59530: PIG-5157 Upgrade to Spark 2.0
Posted by kelly zhang <li...@intel.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59530/#review180799
-----------------------------------------------------------
src/org/apache/pig/backend/hadoop/executionengine/spark/Spark1Shims.java
Line 133 (original), 133 (patched)
<https://reviews.apache.org/r/59530/#comment256056>
How about change org.apache.pig.backend.hadoop.executionengine.spark.JobMetricsListener to org.apache.pig.backend.hadoop.executionengine.spark.JobStatisticCollector?
and rename JobMetricsListener#allJobMetrics to JobStatisticCollector#allJobStatistics?
Just suggestion to make the code more better.
- kelly zhang
On July 17, 2017, 3:01 p.m., Nandor Kollar wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/59530/
> -----------------------------------------------------------
>
> (Updated July 17, 2017, 3:01 p.m.)
>
>
> Review request for pig, liyun zhang, Rohini Palaniswamy, and Adam Szita.
>
>
> Repository: pig-git
>
>
> Description
> -------
>
> Upgrade to Spark 2.1 API using shims.
>
>
> Diffs
> -----
>
> build.xml d274af9be3805f5167f11afed9d85f19e72916d8
> ivy.xml 3f2c94373ba9455bbb6a3c96bfd61fc6cfaab588
> ivy/libraries.properties c2aed45a3244dfd108a255c7308a7dcb0dabd3b5
> src/org/apache/pig/backend/hadoop/executionengine/spark/FlatMapFunctionAdapter.java PRE-CREATION
> src/org/apache/pig/backend/hadoop/executionengine/spark/JobMetricsListener.java f81341233447203abc4800cc7b22a4f419e10262
> src/org/apache/pig/backend/hadoop/executionengine/spark/PairFlatMapFunctionAdapter.java PRE-CREATION
> src/org/apache/pig/backend/hadoop/executionengine/spark/Spark1Shims.java PRE-CREATION
> src/org/apache/pig/backend/hadoop/executionengine/spark/Spark2Shims.java PRE-CREATION
> src/org/apache/pig/backend/hadoop/executionengine/spark/SparkLauncher.java 237fd9431a16226234d91059088f91aab346b83c
> src/org/apache/pig/backend/hadoop/executionengine/spark/SparkShims.java PRE-CREATION
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/CollectedGroupConverter.java 83311dfa5bb25209a5366c2db7e8d483c31d94cd
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/FRJoinConverter.java 382258e7ff9105aa397c5a2888df0c11e9562ec9
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/ForEachConverter.java b58415e7e18ca4cf1331beef06e9214600a51424
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/GlobalRearrangeConverter.java 130c8b9a747b176ce2b649ca6d5260527595fb76
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/LimitConverter.java fe1b54c8f128661d7d19c276d3bb2de7874d3086
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/MergeCogroupConverter.java adf78ecab0da10d3b1a7fdde8af2b42dd899810f
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/MergeJoinConverter.java d1c43b1e06adc4c9fe45a83b81103333402e3756
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/PoissonSampleConverter.java e003bbd95763b2d189ff9ec540c89abe52592420
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SecondaryKeySortUtil.java 00d29b44848546ed16dde2baa8c61b36939971b2
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SkewedJoinConverter.java c55ba3145495a53d69db2dd56434dcc9b3bf8ed5
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SortConverter.java baabfa090323e3bef087e259ce19df2e4c34dd63
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SparkSampleSortConverter.java 3166fdc31745c013380492e089c83f3e853a3e6e
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/StreamConverter.java 3a50d485cfd54b9f3b9c1a982e6c30497a4c85fc
> src/org/apache/pig/tools/pigstats/spark/Spark1JobStats.java PRE-CREATION
> src/org/apache/pig/tools/pigstats/spark/Spark2JobStats.java PRE-CREATION
> src/org/apache/pig/tools/pigstats/spark/SparkJobStats.java c8cc03194b223d2ee181d73c6b651a6872cac6b6
> src/org/apache/pig/tools/pigstats/spark/SparkPigStats.java 61ccbcc9fd723f6e2e578a8476230c42d5587dfe
> test/org/apache/pig/test/TestPigRunner.java ec08417f2f71deec514cab5cfb9d2f99520ad641
>
>
> Diff: https://reviews.apache.org/r/59530/diff/14/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Nandor Kollar
>
>
Re: Review Request 59530: PIG-5157 Upgrade to Spark 2.0
Posted by kelly zhang <li...@intel.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59530/#review180895
-----------------------------------------------------------
Ship it!
Ship It!
- kelly zhang
On July 18, 2017, 11:38 a.m., Nandor Kollar wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/59530/
> -----------------------------------------------------------
>
> (Updated July 18, 2017, 11:38 a.m.)
>
>
> Review request for pig, liyun zhang, Rohini Palaniswamy, and Adam Szita.
>
>
> Repository: pig-git
>
>
> Description
> -------
>
> Upgrade to Spark 2.1 API using shims.
>
>
> Diffs
> -----
>
> build.xml d274af9be3805f5167f11afed9d85f19e72916d8
> ivy.xml 3f2c94373ba9455bbb6a3c96bfd61fc6cfaab588
> ivy/libraries.properties c2aed45a3244dfd108a255c7308a7dcb0dabd3b5
> src/org/apache/pig/backend/hadoop/executionengine/spark/FlatMapFunctionAdapter.java PRE-CREATION
> src/org/apache/pig/backend/hadoop/executionengine/spark/JobGraphBuilder.java 5eac045dc6b7b7f3e06cf177e97d19e7d7f43010
> src/org/apache/pig/backend/hadoop/executionengine/spark/JobMetricsListener.java f81341233447203abc4800cc7b22a4f419e10262
> src/org/apache/pig/backend/hadoop/executionengine/spark/JobStatisticCollector.java PRE-CREATION
> src/org/apache/pig/backend/hadoop/executionengine/spark/PairFlatMapFunctionAdapter.java PRE-CREATION
> src/org/apache/pig/backend/hadoop/executionengine/spark/Spark1Shims.java PRE-CREATION
> src/org/apache/pig/backend/hadoop/executionengine/spark/Spark2Shims.java PRE-CREATION
> src/org/apache/pig/backend/hadoop/executionengine/spark/SparkLauncher.java 237fd9431a16226234d91059088f91aab346b83c
> src/org/apache/pig/backend/hadoop/executionengine/spark/SparkShims.java PRE-CREATION
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/CollectedGroupConverter.java 83311dfa5bb25209a5366c2db7e8d483c31d94cd
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/FRJoinConverter.java 382258e7ff9105aa397c5a2888df0c11e9562ec9
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/ForEachConverter.java b58415e7e18ca4cf1331beef06e9214600a51424
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/GlobalRearrangeConverter.java 130c8b9a747b176ce2b649ca6d5260527595fb76
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/LimitConverter.java fe1b54c8f128661d7d19c276d3bb2de7874d3086
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/MergeCogroupConverter.java adf78ecab0da10d3b1a7fdde8af2b42dd899810f
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/MergeJoinConverter.java d1c43b1e06adc4c9fe45a83b81103333402e3756
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/PoissonSampleConverter.java e003bbd95763b2d189ff9ec540c89abe52592420
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SecondaryKeySortUtil.java 00d29b44848546ed16dde2baa8c61b36939971b2
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SkewedJoinConverter.java c55ba3145495a53d69db2dd56434dcc9b3bf8ed5
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SortConverter.java baabfa090323e3bef087e259ce19df2e4c34dd63
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SparkSampleSortConverter.java 3166fdc31745c013380492e089c83f3e853a3e6e
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/StreamConverter.java 3a50d485cfd54b9f3b9c1a982e6c30497a4c85fc
> src/org/apache/pig/tools/pigstats/spark/Spark1JobStats.java PRE-CREATION
> src/org/apache/pig/tools/pigstats/spark/Spark2JobStats.java PRE-CREATION
> src/org/apache/pig/tools/pigstats/spark/SparkJobStats.java c8cc03194b223d2ee181d73c6b651a6872cac6b6
> src/org/apache/pig/tools/pigstats/spark/SparkPigStats.java 61ccbcc9fd723f6e2e578a8476230c42d5587dfe
> src/org/apache/pig/tools/pigstats/spark/SparkStatsUtil.java 1541264c4088fbe4770e706fd8fc53907fa83796
> test/org/apache/pig/test/TestPigRunner.java ec08417f2f71deec514cab5cfb9d2f99520ad641
>
>
> Diff: https://reviews.apache.org/r/59530/diff/15/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Nandor Kollar
>
>
Re: Review Request 59530: PIG-5157 Upgrade to Spark 2.0
Posted by Nandor Kollar <nk...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59530/
-----------------------------------------------------------
(Updated July 18, 2017, 11:38 a.m.)
Review request for pig, liyun zhang, Rohini Palaniswamy, and Adam Szita.
Repository: pig-git
Description
-------
Upgrade to Spark 2.1 API using shims.
Diffs (updated)
-----
build.xml d274af9be3805f5167f11afed9d85f19e72916d8
ivy.xml 3f2c94373ba9455bbb6a3c96bfd61fc6cfaab588
ivy/libraries.properties c2aed45a3244dfd108a255c7308a7dcb0dabd3b5
src/org/apache/pig/backend/hadoop/executionengine/spark/FlatMapFunctionAdapter.java PRE-CREATION
src/org/apache/pig/backend/hadoop/executionengine/spark/JobGraphBuilder.java 5eac045dc6b7b7f3e06cf177e97d19e7d7f43010
src/org/apache/pig/backend/hadoop/executionengine/spark/JobMetricsListener.java f81341233447203abc4800cc7b22a4f419e10262
src/org/apache/pig/backend/hadoop/executionengine/spark/JobStatisticCollector.java PRE-CREATION
src/org/apache/pig/backend/hadoop/executionengine/spark/PairFlatMapFunctionAdapter.java PRE-CREATION
src/org/apache/pig/backend/hadoop/executionengine/spark/Spark1Shims.java PRE-CREATION
src/org/apache/pig/backend/hadoop/executionengine/spark/Spark2Shims.java PRE-CREATION
src/org/apache/pig/backend/hadoop/executionengine/spark/SparkLauncher.java 237fd9431a16226234d91059088f91aab346b83c
src/org/apache/pig/backend/hadoop/executionengine/spark/SparkShims.java PRE-CREATION
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/CollectedGroupConverter.java 83311dfa5bb25209a5366c2db7e8d483c31d94cd
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/FRJoinConverter.java 382258e7ff9105aa397c5a2888df0c11e9562ec9
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/ForEachConverter.java b58415e7e18ca4cf1331beef06e9214600a51424
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/GlobalRearrangeConverter.java 130c8b9a747b176ce2b649ca6d5260527595fb76
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/LimitConverter.java fe1b54c8f128661d7d19c276d3bb2de7874d3086
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/MergeCogroupConverter.java adf78ecab0da10d3b1a7fdde8af2b42dd899810f
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/MergeJoinConverter.java d1c43b1e06adc4c9fe45a83b81103333402e3756
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/PoissonSampleConverter.java e003bbd95763b2d189ff9ec540c89abe52592420
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SecondaryKeySortUtil.java 00d29b44848546ed16dde2baa8c61b36939971b2
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SkewedJoinConverter.java c55ba3145495a53d69db2dd56434dcc9b3bf8ed5
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SortConverter.java baabfa090323e3bef087e259ce19df2e4c34dd63
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SparkSampleSortConverter.java 3166fdc31745c013380492e089c83f3e853a3e6e
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/StreamConverter.java 3a50d485cfd54b9f3b9c1a982e6c30497a4c85fc
src/org/apache/pig/tools/pigstats/spark/Spark1JobStats.java PRE-CREATION
src/org/apache/pig/tools/pigstats/spark/Spark2JobStats.java PRE-CREATION
src/org/apache/pig/tools/pigstats/spark/SparkJobStats.java c8cc03194b223d2ee181d73c6b651a6872cac6b6
src/org/apache/pig/tools/pigstats/spark/SparkPigStats.java 61ccbcc9fd723f6e2e578a8476230c42d5587dfe
src/org/apache/pig/tools/pigstats/spark/SparkStatsUtil.java 1541264c4088fbe4770e706fd8fc53907fa83796
test/org/apache/pig/test/TestPigRunner.java ec08417f2f71deec514cab5cfb9d2f99520ad641
Diff: https://reviews.apache.org/r/59530/diff/15/
Changes: https://reviews.apache.org/r/59530/diff/14-15/
Testing
-------
Thanks,
Nandor Kollar
Re: Review Request 59530: PIG-5157 Upgrade to Spark 2.0
Posted by Nandor Kollar <nk...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59530/
-----------------------------------------------------------
(Updated July 17, 2017, 3:01 p.m.)
Review request for pig, liyun zhang, Rohini Palaniswamy, and Adam Szita.
Changes
-------
rename JobMetricsListener classes
Repository: pig-git
Description
-------
Upgrade to Spark 2.1 API using shims.
Diffs (updated)
-----
build.xml d274af9be3805f5167f11afed9d85f19e72916d8
ivy.xml 3f2c94373ba9455bbb6a3c96bfd61fc6cfaab588
ivy/libraries.properties c2aed45a3244dfd108a255c7308a7dcb0dabd3b5
src/org/apache/pig/backend/hadoop/executionengine/spark/FlatMapFunctionAdapter.java PRE-CREATION
src/org/apache/pig/backend/hadoop/executionengine/spark/JobMetricsListener.java f81341233447203abc4800cc7b22a4f419e10262
src/org/apache/pig/backend/hadoop/executionengine/spark/PairFlatMapFunctionAdapter.java PRE-CREATION
src/org/apache/pig/backend/hadoop/executionengine/spark/Spark1Shims.java PRE-CREATION
src/org/apache/pig/backend/hadoop/executionengine/spark/Spark2Shims.java PRE-CREATION
src/org/apache/pig/backend/hadoop/executionengine/spark/SparkLauncher.java 237fd9431a16226234d91059088f91aab346b83c
src/org/apache/pig/backend/hadoop/executionengine/spark/SparkShims.java PRE-CREATION
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/CollectedGroupConverter.java 83311dfa5bb25209a5366c2db7e8d483c31d94cd
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/FRJoinConverter.java 382258e7ff9105aa397c5a2888df0c11e9562ec9
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/ForEachConverter.java b58415e7e18ca4cf1331beef06e9214600a51424
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/GlobalRearrangeConverter.java 130c8b9a747b176ce2b649ca6d5260527595fb76
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/LimitConverter.java fe1b54c8f128661d7d19c276d3bb2de7874d3086
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/MergeCogroupConverter.java adf78ecab0da10d3b1a7fdde8af2b42dd899810f
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/MergeJoinConverter.java d1c43b1e06adc4c9fe45a83b81103333402e3756
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/PoissonSampleConverter.java e003bbd95763b2d189ff9ec540c89abe52592420
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SecondaryKeySortUtil.java 00d29b44848546ed16dde2baa8c61b36939971b2
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SkewedJoinConverter.java c55ba3145495a53d69db2dd56434dcc9b3bf8ed5
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SortConverter.java baabfa090323e3bef087e259ce19df2e4c34dd63
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SparkSampleSortConverter.java 3166fdc31745c013380492e089c83f3e853a3e6e
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/StreamConverter.java 3a50d485cfd54b9f3b9c1a982e6c30497a4c85fc
src/org/apache/pig/tools/pigstats/spark/Spark1JobStats.java PRE-CREATION
src/org/apache/pig/tools/pigstats/spark/Spark2JobStats.java PRE-CREATION
src/org/apache/pig/tools/pigstats/spark/SparkJobStats.java c8cc03194b223d2ee181d73c6b651a6872cac6b6
src/org/apache/pig/tools/pigstats/spark/SparkPigStats.java 61ccbcc9fd723f6e2e578a8476230c42d5587dfe
test/org/apache/pig/test/TestPigRunner.java ec08417f2f71deec514cab5cfb9d2f99520ad641
Diff: https://reviews.apache.org/r/59530/diff/14/
Changes: https://reviews.apache.org/r/59530/diff/13-14/
Testing
-------
Thanks,
Nandor Kollar
Re: Review Request 59530: PIG-5157 Upgrade to Spark 2.0
Posted by Nandor Kollar <nk...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59530/
-----------------------------------------------------------
(Updated June 28, 2017, 8:55 a.m.)
Review request for pig, liyun zhang, Rohini Palaniswamy, and Adam Szita.
Changes
-------
e2e tests were hanging on Spark2 due to accidentally deleted onJobEnd method from Spark2 shim
Repository: pig-git
Description
-------
Upgrade to Spark 2.1 API using shims.
Diffs (updated)
-----
build.xml d274af9be3805f5167f11afed9d85f19e72916d8
ivy.xml 3f2c94373ba9455bbb6a3c96bfd61fc6cfaab588
ivy/libraries.properties c2aed45a3244dfd108a255c7308a7dcb0dabd3b5
src/org/apache/pig/backend/hadoop/executionengine/spark/FlatMapFunctionAdapter.java PRE-CREATION
src/org/apache/pig/backend/hadoop/executionengine/spark/JobMetricsListener.java f81341233447203abc4800cc7b22a4f419e10262
src/org/apache/pig/backend/hadoop/executionengine/spark/PairFlatMapFunctionAdapter.java PRE-CREATION
src/org/apache/pig/backend/hadoop/executionengine/spark/Spark1Shims.java PRE-CREATION
src/org/apache/pig/backend/hadoop/executionengine/spark/Spark2Shims.java PRE-CREATION
src/org/apache/pig/backend/hadoop/executionengine/spark/SparkLauncher.java 237fd9431a16226234d91059088f91aab346b83c
src/org/apache/pig/backend/hadoop/executionengine/spark/SparkShims.java PRE-CREATION
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/CollectedGroupConverter.java 83311dfa5bb25209a5366c2db7e8d483c31d94cd
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/FRJoinConverter.java 382258e7ff9105aa397c5a2888df0c11e9562ec9
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/ForEachConverter.java b58415e7e18ca4cf1331beef06e9214600a51424
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/GlobalRearrangeConverter.java 130c8b9a747b176ce2b649ca6d5260527595fb76
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/LimitConverter.java fe1b54c8f128661d7d19c276d3bb2de7874d3086
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/MergeCogroupConverter.java adf78ecab0da10d3b1a7fdde8af2b42dd899810f
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/MergeJoinConverter.java d1c43b1e06adc4c9fe45a83b81103333402e3756
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/PoissonSampleConverter.java e003bbd95763b2d189ff9ec540c89abe52592420
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SecondaryKeySortUtil.java 00d29b44848546ed16dde2baa8c61b36939971b2
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SkewedJoinConverter.java c55ba3145495a53d69db2dd56434dcc9b3bf8ed5
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SortConverter.java baabfa090323e3bef087e259ce19df2e4c34dd63
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SparkSampleSortConverter.java 3166fdc31745c013380492e089c83f3e853a3e6e
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/StreamConverter.java 3a50d485cfd54b9f3b9c1a982e6c30497a4c85fc
src/org/apache/pig/tools/pigstats/spark/Spark1JobStats.java PRE-CREATION
src/org/apache/pig/tools/pigstats/spark/Spark2JobStats.java PRE-CREATION
src/org/apache/pig/tools/pigstats/spark/SparkJobStats.java c8cc03194b223d2ee181d73c6b651a6872cac6b6
src/org/apache/pig/tools/pigstats/spark/SparkPigStats.java 61ccbcc9fd723f6e2e578a8476230c42d5587dfe
test/org/apache/pig/test/TestPigRunner.java ec08417f2f71deec514cab5cfb9d2f99520ad641
Diff: https://reviews.apache.org/r/59530/diff/13/
Changes: https://reviews.apache.org/r/59530/diff/12-13/
Testing
-------
Thanks,
Nandor Kollar
Re: Review Request 59530: PIG-5157 Upgrade to Spark 2.0
Posted by Nandor Kollar <nk...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59530/
-----------------------------------------------------------
(Updated June 27, 2017, 3:33 p.m.)
Review request for pig, liyun zhang, Rohini Palaniswamy, and Adam Szita.
Repository: pig-git
Description
-------
Upgrade to Spark 2.1 API using shims.
Diffs (updated)
-----
build.xml d274af9be3805f5167f11afed9d85f19e72916d8
ivy.xml 3f2c94373ba9455bbb6a3c96bfd61fc6cfaab588
ivy/libraries.properties c2aed45a3244dfd108a255c7308a7dcb0dabd3b5
src/org/apache/pig/backend/hadoop/executionengine/spark/FlatMapFunctionAdapter.java PRE-CREATION
src/org/apache/pig/backend/hadoop/executionengine/spark/JobMetricsListener.java f81341233447203abc4800cc7b22a4f419e10262
src/org/apache/pig/backend/hadoop/executionengine/spark/PairFlatMapFunctionAdapter.java PRE-CREATION
src/org/apache/pig/backend/hadoop/executionengine/spark/Spark1Shims.java PRE-CREATION
src/org/apache/pig/backend/hadoop/executionengine/spark/Spark2Shims.java PRE-CREATION
src/org/apache/pig/backend/hadoop/executionengine/spark/SparkLauncher.java 237fd9431a16226234d91059088f91aab346b83c
src/org/apache/pig/backend/hadoop/executionengine/spark/SparkShims.java PRE-CREATION
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/CollectedGroupConverter.java 83311dfa5bb25209a5366c2db7e8d483c31d94cd
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/FRJoinConverter.java 382258e7ff9105aa397c5a2888df0c11e9562ec9
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/ForEachConverter.java b58415e7e18ca4cf1331beef06e9214600a51424
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/GlobalRearrangeConverter.java 130c8b9a747b176ce2b649ca6d5260527595fb76
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/LimitConverter.java fe1b54c8f128661d7d19c276d3bb2de7874d3086
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/MergeCogroupConverter.java adf78ecab0da10d3b1a7fdde8af2b42dd899810f
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/MergeJoinConverter.java d1c43b1e06adc4c9fe45a83b81103333402e3756
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/PoissonSampleConverter.java e003bbd95763b2d189ff9ec540c89abe52592420
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SecondaryKeySortUtil.java 00d29b44848546ed16dde2baa8c61b36939971b2
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SkewedJoinConverter.java c55ba3145495a53d69db2dd56434dcc9b3bf8ed5
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SortConverter.java baabfa090323e3bef087e259ce19df2e4c34dd63
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SparkSampleSortConverter.java 3166fdc31745c013380492e089c83f3e853a3e6e
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/StreamConverter.java 3a50d485cfd54b9f3b9c1a982e6c30497a4c85fc
src/org/apache/pig/tools/pigstats/spark/Spark1JobStats.java PRE-CREATION
src/org/apache/pig/tools/pigstats/spark/Spark2JobStats.java PRE-CREATION
src/org/apache/pig/tools/pigstats/spark/SparkJobStats.java c8cc03194b223d2ee181d73c6b651a6872cac6b6
src/org/apache/pig/tools/pigstats/spark/SparkPigStats.java 61ccbcc9fd723f6e2e578a8476230c42d5587dfe
test/org/apache/pig/test/TestPigRunner.java ec08417f2f71deec514cab5cfb9d2f99520ad641
Diff: https://reviews.apache.org/r/59530/diff/12/
Changes: https://reviews.apache.org/r/59530/diff/11-12/
Testing
-------
Thanks,
Nandor Kollar
Re: Review Request 59530: PIG-5157 Upgrade to Spark 2.0
Posted by Nandor Kollar <nk...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59530/
-----------------------------------------------------------
(Updated June 23, 2017, 12:18 p.m.)
Review request for pig, liyun zhang, Rohini Palaniswamy, and Adam Szita.
Changes
-------
Commit tests broken for Spark, function adapters have to extend Serializable (like FlatMapFunction and PairFlatMapFunction).
Repository: pig-git
Description
-------
Upgrade to Spark 2.1 API using shims.
Diffs (updated)
-----
build.xml bba2b52d9354ab909ad26f969480806f6d91911c
ivy.xml 3f2c94373ba9455bbb6a3c96bfd61fc6cfaab588
ivy/libraries.properties c2aed45a3244dfd108a255c7308a7dcb0dabd3b5
src/org/apache/pig/backend/hadoop/executionengine/spark/FlatMapFunctionAdapter.java PRE-CREATION
src/org/apache/pig/backend/hadoop/executionengine/spark/JobMetricsListener.java f81341233447203abc4800cc7b22a4f419e10262
src/org/apache/pig/backend/hadoop/executionengine/spark/PairFlatMapFunctionAdapter.java PRE-CREATION
src/org/apache/pig/backend/hadoop/executionengine/spark/Spark1Shims.java PRE-CREATION
src/org/apache/pig/backend/hadoop/executionengine/spark/Spark2Shims.java PRE-CREATION
src/org/apache/pig/backend/hadoop/executionengine/spark/SparkLauncher.java 237fd9431a16226234d91059088f91aab346b83c
src/org/apache/pig/backend/hadoop/executionengine/spark/SparkShims.java PRE-CREATION
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/CollectedGroupConverter.java 83311dfa5bb25209a5366c2db7e8d483c31d94cd
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/FRJoinConverter.java 382258e7ff9105aa397c5a2888df0c11e9562ec9
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/ForEachConverter.java b58415e7e18ca4cf1331beef06e9214600a51424
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/GlobalRearrangeConverter.java 130c8b9a747b176ce2b649ca6d5260527595fb76
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/LimitConverter.java fe1b54c8f128661d7d19c276d3bb2de7874d3086
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/MergeCogroupConverter.java adf78ecab0da10d3b1a7fdde8af2b42dd899810f
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/MergeJoinConverter.java d1c43b1e06adc4c9fe45a83b81103333402e3756
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/PoissonSampleConverter.java e003bbd95763b2d189ff9ec540c89abe52592420
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SecondaryKeySortUtil.java 00d29b44848546ed16dde2baa8c61b36939971b2
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SkewedJoinConverter.java c55ba3145495a53d69db2dd56434dcc9b3bf8ed5
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SortConverter.java baabfa090323e3bef087e259ce19df2e4c34dd63
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SparkSampleSortConverter.java 3166fdc31745c013380492e089c83f3e853a3e6e
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/StreamConverter.java 3a50d485cfd54b9f3b9c1a982e6c30497a4c85fc
src/org/apache/pig/tools/pigstats/spark/Spark1JobStats.java PRE-CREATION
src/org/apache/pig/tools/pigstats/spark/Spark2JobStats.java PRE-CREATION
src/org/apache/pig/tools/pigstats/spark/SparkJobStats.java c8cc03194b223d2ee181d73c6b651a6872cac6b6
src/org/apache/pig/tools/pigstats/spark/SparkPigStats.java 61ccbcc9fd723f6e2e578a8476230c42d5587dfe
test/org/apache/pig/test/TestPigRunner.java ec08417f2f71deec514cab5cfb9d2f99520ad641
Diff: https://reviews.apache.org/r/59530/diff/11/
Changes: https://reviews.apache.org/r/59530/diff/10-11/
Testing
-------
Thanks,
Nandor Kollar
Re: Review Request 59530: PIG-5157 Upgrade to Spark 2.0
Posted by Nandor Kollar <nk...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59530/
-----------------------------------------------------------
(Updated June 23, 2017, 9:49 a.m.)
Review request for pig, liyun zhang, Rohini Palaniswamy, and Adam Szita.
Changes
-------
Synchronize on the correct object in JobMetricsListener, no need to change compile conf
Repository: pig-git
Description
-------
Upgrade to Spark 2.1 API using shims.
Diffs (updated)
-----
build.xml bba2b52d9354ab909ad26f969480806f6d91911c
ivy.xml 3f2c94373ba9455bbb6a3c96bfd61fc6cfaab588
ivy/libraries.properties c2aed45a3244dfd108a255c7308a7dcb0dabd3b5
src/org/apache/pig/backend/hadoop/executionengine/spark/FlatMapFunctionAdapter.java PRE-CREATION
src/org/apache/pig/backend/hadoop/executionengine/spark/JobMetricsListener.java f81341233447203abc4800cc7b22a4f419e10262
src/org/apache/pig/backend/hadoop/executionengine/spark/PairFlatMapFunctionAdapter.java PRE-CREATION
src/org/apache/pig/backend/hadoop/executionengine/spark/Spark1Shims.java PRE-CREATION
src/org/apache/pig/backend/hadoop/executionengine/spark/Spark2Shims.java PRE-CREATION
src/org/apache/pig/backend/hadoop/executionengine/spark/SparkLauncher.java 237fd9431a16226234d91059088f91aab346b83c
src/org/apache/pig/backend/hadoop/executionengine/spark/SparkShims.java PRE-CREATION
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/CollectedGroupConverter.java 83311dfa5bb25209a5366c2db7e8d483c31d94cd
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/FRJoinConverter.java 382258e7ff9105aa397c5a2888df0c11e9562ec9
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/ForEachConverter.java b58415e7e18ca4cf1331beef06e9214600a51424
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/GlobalRearrangeConverter.java 130c8b9a747b176ce2b649ca6d5260527595fb76
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/LimitConverter.java fe1b54c8f128661d7d19c276d3bb2de7874d3086
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/MergeCogroupConverter.java adf78ecab0da10d3b1a7fdde8af2b42dd899810f
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/MergeJoinConverter.java d1c43b1e06adc4c9fe45a83b81103333402e3756
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/PoissonSampleConverter.java e003bbd95763b2d189ff9ec540c89abe52592420
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SecondaryKeySortUtil.java 00d29b44848546ed16dde2baa8c61b36939971b2
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SkewedJoinConverter.java c55ba3145495a53d69db2dd56434dcc9b3bf8ed5
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SortConverter.java baabfa090323e3bef087e259ce19df2e4c34dd63
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SparkSampleSortConverter.java 3166fdc31745c013380492e089c83f3e853a3e6e
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/StreamConverter.java 3a50d485cfd54b9f3b9c1a982e6c30497a4c85fc
src/org/apache/pig/tools/pigstats/spark/Spark1JobStats.java PRE-CREATION
src/org/apache/pig/tools/pigstats/spark/Spark2JobStats.java PRE-CREATION
src/org/apache/pig/tools/pigstats/spark/SparkJobStats.java c8cc03194b223d2ee181d73c6b651a6872cac6b6
src/org/apache/pig/tools/pigstats/spark/SparkPigStats.java 61ccbcc9fd723f6e2e578a8476230c42d5587dfe
test/org/apache/pig/test/TestPigRunner.java ec08417f2f71deec514cab5cfb9d2f99520ad641
Diff: https://reviews.apache.org/r/59530/diff/10/
Changes: https://reviews.apache.org/r/59530/diff/9-10/
Testing
-------
Thanks,
Nandor Kollar
Re: Review Request 59530: PIG-5157 Upgrade to Spark 2.0
Posted by Rohini Palaniswamy <ro...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59530/#review178716
-----------------------------------------------------------
Ship it!
Ship It!
- Rohini Palaniswamy
On June 22, 2017, 2:38 p.m., Nandor Kollar wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/59530/
> -----------------------------------------------------------
>
> (Updated June 22, 2017, 2:38 p.m.)
>
>
> Review request for pig, liyun zhang, Rohini Palaniswamy, and Adam Szita.
>
>
> Repository: pig-git
>
>
> Description
> -------
>
> Upgrade to Spark 2.1 API using shims.
>
>
> Diffs
> -----
>
> build.xml bba2b52d9354ab909ad26f969480806f6d91911c
> ivy.xml 3f2c94373ba9455bbb6a3c96bfd61fc6cfaab588
> ivy/libraries.properties c2aed45a3244dfd108a255c7308a7dcb0dabd3b5
> src/org/apache/pig/backend/hadoop/executionengine/spark/FlatMapFunctionAdapter.java PRE-CREATION
> src/org/apache/pig/backend/hadoop/executionengine/spark/JobMetricsListener.java f81341233447203abc4800cc7b22a4f419e10262
> src/org/apache/pig/backend/hadoop/executionengine/spark/PairFlatMapFunctionAdapter.java PRE-CREATION
> src/org/apache/pig/backend/hadoop/executionengine/spark/Spark1Shims.java PRE-CREATION
> src/org/apache/pig/backend/hadoop/executionengine/spark/Spark2Shims.java PRE-CREATION
> src/org/apache/pig/backend/hadoop/executionengine/spark/SparkLauncher.java 237fd9431a16226234d91059088f91aab346b83c
> src/org/apache/pig/backend/hadoop/executionengine/spark/SparkShims.java PRE-CREATION
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/CollectedGroupConverter.java 83311dfa5bb25209a5366c2db7e8d483c31d94cd
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/FRJoinConverter.java 382258e7ff9105aa397c5a2888df0c11e9562ec9
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/ForEachConverter.java b58415e7e18ca4cf1331beef06e9214600a51424
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/GlobalRearrangeConverter.java 130c8b9a747b176ce2b649ca6d5260527595fb76
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/LimitConverter.java fe1b54c8f128661d7d19c276d3bb2de7874d3086
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/MergeCogroupConverter.java adf78ecab0da10d3b1a7fdde8af2b42dd899810f
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/MergeJoinConverter.java d1c43b1e06adc4c9fe45a83b81103333402e3756
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/PoissonSampleConverter.java e003bbd95763b2d189ff9ec540c89abe52592420
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SecondaryKeySortUtil.java 00d29b44848546ed16dde2baa8c61b36939971b2
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SkewedJoinConverter.java c55ba3145495a53d69db2dd56434dcc9b3bf8ed5
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SortConverter.java baabfa090323e3bef087e259ce19df2e4c34dd63
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SparkSampleSortConverter.java 3166fdc31745c013380492e089c83f3e853a3e6e
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/StreamConverter.java 3a50d485cfd54b9f3b9c1a982e6c30497a4c85fc
> src/org/apache/pig/tools/pigstats/spark/Spark1JobStats.java PRE-CREATION
> src/org/apache/pig/tools/pigstats/spark/Spark2JobStats.java PRE-CREATION
> src/org/apache/pig/tools/pigstats/spark/SparkJobStats.java c8cc03194b223d2ee181d73c6b651a6872cac6b6
> src/org/apache/pig/tools/pigstats/spark/SparkPigStats.java 61ccbcc9fd723f6e2e578a8476230c42d5587dfe
> test/org/apache/pig/test/TestPigRunner.java ec08417f2f71deec514cab5cfb9d2f99520ad641
>
>
> Diff: https://reviews.apache.org/r/59530/diff/9/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Nandor Kollar
>
>
Re: Review Request 59530: PIG-5157 Upgrade to Spark 2.0
Posted by Nandor Kollar <nk...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59530/
-----------------------------------------------------------
(Updated June 22, 2017, 2:38 p.m.)
Review request for pig, liyun zhang, Rohini Palaniswamy, and Adam Szita.
Changes
-------
fix serialization problem for SparkShims
Repository: pig-git
Description
-------
Upgrade to Spark 2.1 API using shims.
Diffs (updated)
-----
build.xml bba2b52d9354ab909ad26f969480806f6d91911c
ivy.xml 3f2c94373ba9455bbb6a3c96bfd61fc6cfaab588
ivy/libraries.properties c2aed45a3244dfd108a255c7308a7dcb0dabd3b5
src/org/apache/pig/backend/hadoop/executionengine/spark/FlatMapFunctionAdapter.java PRE-CREATION
src/org/apache/pig/backend/hadoop/executionengine/spark/JobMetricsListener.java f81341233447203abc4800cc7b22a4f419e10262
src/org/apache/pig/backend/hadoop/executionengine/spark/PairFlatMapFunctionAdapter.java PRE-CREATION
src/org/apache/pig/backend/hadoop/executionengine/spark/Spark1Shims.java PRE-CREATION
src/org/apache/pig/backend/hadoop/executionengine/spark/Spark2Shims.java PRE-CREATION
src/org/apache/pig/backend/hadoop/executionengine/spark/SparkLauncher.java 237fd9431a16226234d91059088f91aab346b83c
src/org/apache/pig/backend/hadoop/executionengine/spark/SparkShims.java PRE-CREATION
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/CollectedGroupConverter.java 83311dfa5bb25209a5366c2db7e8d483c31d94cd
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/FRJoinConverter.java 382258e7ff9105aa397c5a2888df0c11e9562ec9
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/ForEachConverter.java b58415e7e18ca4cf1331beef06e9214600a51424
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/GlobalRearrangeConverter.java 130c8b9a747b176ce2b649ca6d5260527595fb76
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/LimitConverter.java fe1b54c8f128661d7d19c276d3bb2de7874d3086
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/MergeCogroupConverter.java adf78ecab0da10d3b1a7fdde8af2b42dd899810f
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/MergeJoinConverter.java d1c43b1e06adc4c9fe45a83b81103333402e3756
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/PoissonSampleConverter.java e003bbd95763b2d189ff9ec540c89abe52592420
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SecondaryKeySortUtil.java 00d29b44848546ed16dde2baa8c61b36939971b2
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SkewedJoinConverter.java c55ba3145495a53d69db2dd56434dcc9b3bf8ed5
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SortConverter.java baabfa090323e3bef087e259ce19df2e4c34dd63
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SparkSampleSortConverter.java 3166fdc31745c013380492e089c83f3e853a3e6e
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/StreamConverter.java 3a50d485cfd54b9f3b9c1a982e6c30497a4c85fc
src/org/apache/pig/tools/pigstats/spark/Spark1JobStats.java PRE-CREATION
src/org/apache/pig/tools/pigstats/spark/Spark2JobStats.java PRE-CREATION
src/org/apache/pig/tools/pigstats/spark/SparkJobStats.java c8cc03194b223d2ee181d73c6b651a6872cac6b6
src/org/apache/pig/tools/pigstats/spark/SparkPigStats.java 61ccbcc9fd723f6e2e578a8476230c42d5587dfe
test/org/apache/pig/test/TestPigRunner.java ec08417f2f71deec514cab5cfb9d2f99520ad641
Diff: https://reviews.apache.org/r/59530/diff/9/
Changes: https://reviews.apache.org/r/59530/diff/8-9/
Testing
-------
Thanks,
Nandor Kollar
Re: Review Request 59530: PIG-5157 Upgrade to Spark 2.0
Posted by Nandor Kollar <nk...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59530/
-----------------------------------------------------------
(Updated June 22, 2017, 2:07 p.m.)
Review request for pig, liyun zhang, Rohini Palaniswamy, and Adam Szita.
Changes
-------
Fix for java.lang.ClassNotFoundException: org.apache.spark.scheduler.SparkListenerInterface
Repository: pig-git
Description
-------
Upgrade to Spark 2.1 API using shims.
Diffs (updated)
-----
build.xml bba2b52d9354ab909ad26f969480806f6d91911c
ivy.xml 3f2c94373ba9455bbb6a3c96bfd61fc6cfaab588
ivy/libraries.properties c2aed45a3244dfd108a255c7308a7dcb0dabd3b5
src/org/apache/pig/backend/hadoop/executionengine/spark/FlatMapFunctionAdapter.java PRE-CREATION
src/org/apache/pig/backend/hadoop/executionengine/spark/JobMetricsListener.java f81341233447203abc4800cc7b22a4f419e10262
src/org/apache/pig/backend/hadoop/executionengine/spark/PairFlatMapFunctionAdapter.java PRE-CREATION
src/org/apache/pig/backend/hadoop/executionengine/spark/Spark1Shims.java PRE-CREATION
src/org/apache/pig/backend/hadoop/executionengine/spark/Spark2Shims.java PRE-CREATION
src/org/apache/pig/backend/hadoop/executionengine/spark/SparkLauncher.java 237fd9431a16226234d91059088f91aab346b83c
src/org/apache/pig/backend/hadoop/executionengine/spark/SparkShims.java PRE-CREATION
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/CollectedGroupConverter.java 83311dfa5bb25209a5366c2db7e8d483c31d94cd
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/FRJoinConverter.java 382258e7ff9105aa397c5a2888df0c11e9562ec9
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/ForEachConverter.java b58415e7e18ca4cf1331beef06e9214600a51424
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/GlobalRearrangeConverter.java 130c8b9a747b176ce2b649ca6d5260527595fb76
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/LimitConverter.java fe1b54c8f128661d7d19c276d3bb2de7874d3086
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/MergeCogroupConverter.java adf78ecab0da10d3b1a7fdde8af2b42dd899810f
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/MergeJoinConverter.java d1c43b1e06adc4c9fe45a83b81103333402e3756
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/PoissonSampleConverter.java e003bbd95763b2d189ff9ec540c89abe52592420
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SecondaryKeySortUtil.java 00d29b44848546ed16dde2baa8c61b36939971b2
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SkewedJoinConverter.java c55ba3145495a53d69db2dd56434dcc9b3bf8ed5
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SortConverter.java baabfa090323e3bef087e259ce19df2e4c34dd63
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SparkSampleSortConverter.java 3166fdc31745c013380492e089c83f3e853a3e6e
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/StreamConverter.java 3a50d485cfd54b9f3b9c1a982e6c30497a4c85fc
src/org/apache/pig/tools/pigstats/spark/Spark1JobStats.java PRE-CREATION
src/org/apache/pig/tools/pigstats/spark/Spark2JobStats.java PRE-CREATION
src/org/apache/pig/tools/pigstats/spark/SparkJobStats.java c8cc03194b223d2ee181d73c6b651a6872cac6b6
src/org/apache/pig/tools/pigstats/spark/SparkPigStats.java 61ccbcc9fd723f6e2e578a8476230c42d5587dfe
test/org/apache/pig/test/TestPigRunner.java ec08417f2f71deec514cab5cfb9d2f99520ad641
Diff: https://reviews.apache.org/r/59530/diff/8/
Changes: https://reviews.apache.org/r/59530/diff/7-8/
Testing
-------
Thanks,
Nandor Kollar
Re: Review Request 59530: PIG-5157 Upgrade to Spark 2.0
Posted by Nandor Kollar <nk...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59530/
-----------------------------------------------------------
(Updated June 21, 2017, 9:06 p.m.)
Review request for pig, liyun zhang, Rohini Palaniswamy, and Adam Szita.
Changes
-------
Address Rohini's latest comment: no need to change references for jar to jar-simple, and no need to delete build artifact
Repository: pig-git
Description
-------
Upgrade to Spark 2.1 API using shims.
Diffs (updated)
-----
build.xml bba2b52d9354ab909ad26f969480806f6d91911c
ivy.xml 3f2c94373ba9455bbb6a3c96bfd61fc6cfaab588
ivy/libraries.properties c2aed45a3244dfd108a255c7308a7dcb0dabd3b5
src/org/apache/pig/backend/hadoop/executionengine/spark/FlatMapFunctionAdapter.java PRE-CREATION
src/org/apache/pig/backend/hadoop/executionengine/spark/JobMetricsListener.java f81341233447203abc4800cc7b22a4f419e10262
src/org/apache/pig/backend/hadoop/executionengine/spark/PairFlatMapFunctionAdapter.java PRE-CREATION
src/org/apache/pig/backend/hadoop/executionengine/spark/Spark1Shims.java PRE-CREATION
src/org/apache/pig/backend/hadoop/executionengine/spark/Spark2Shims.java PRE-CREATION
src/org/apache/pig/backend/hadoop/executionengine/spark/SparkLauncher.java 237fd9431a16226234d91059088f91aab346b83c
src/org/apache/pig/backend/hadoop/executionengine/spark/SparkShims.java PRE-CREATION
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/CollectedGroupConverter.java 83311dfa5bb25209a5366c2db7e8d483c31d94cd
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/FRJoinConverter.java 382258e7ff9105aa397c5a2888df0c11e9562ec9
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/ForEachConverter.java b58415e7e18ca4cf1331beef06e9214600a51424
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/GlobalRearrangeConverter.java 130c8b9a747b176ce2b649ca6d5260527595fb76
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/LimitConverter.java fe1b54c8f128661d7d19c276d3bb2de7874d3086
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/MergeCogroupConverter.java adf78ecab0da10d3b1a7fdde8af2b42dd899810f
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/MergeJoinConverter.java d1c43b1e06adc4c9fe45a83b81103333402e3756
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/PoissonSampleConverter.java e003bbd95763b2d189ff9ec540c89abe52592420
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SecondaryKeySortUtil.java 00d29b44848546ed16dde2baa8c61b36939971b2
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SkewedJoinConverter.java c55ba3145495a53d69db2dd56434dcc9b3bf8ed5
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SortConverter.java baabfa090323e3bef087e259ce19df2e4c34dd63
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SparkSampleSortConverter.java 3166fdc31745c013380492e089c83f3e853a3e6e
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/StreamConverter.java 3a50d485cfd54b9f3b9c1a982e6c30497a4c85fc
src/org/apache/pig/tools/pigstats/spark/Spark1JobStats.java PRE-CREATION
src/org/apache/pig/tools/pigstats/spark/Spark2JobStats.java PRE-CREATION
src/org/apache/pig/tools/pigstats/spark/SparkJobStats.java c8cc03194b223d2ee181d73c6b651a6872cac6b6
src/org/apache/pig/tools/pigstats/spark/SparkPigStats.java 61ccbcc9fd723f6e2e578a8476230c42d5587dfe
test/org/apache/pig/test/TestPigRunner.java ec08417f2f71deec514cab5cfb9d2f99520ad641
Diff: https://reviews.apache.org/r/59530/diff/7/
Changes: https://reviews.apache.org/r/59530/diff/6-7/
Testing
-------
Thanks,
Nandor Kollar
Re: Review Request 59530: PIG-5157 Upgrade to Spark 2.0
Posted by Adam Szita <sz...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59530/#review178236
-----------------------------------------------------------
Ship it!
Ship It!
- Adam Szita
On June 19, 2017, 9:41 a.m., Nandor Kollar wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/59530/
> -----------------------------------------------------------
>
> (Updated June 19, 2017, 9:41 a.m.)
>
>
> Review request for pig, liyun zhang, Rohini Palaniswamy, and Adam Szita.
>
>
> Repository: pig-git
>
>
> Description
> -------
>
> Upgrade to Spark 2.1 API using shims.
>
>
> Diffs
> -----
>
> build.xml bba2b52d9354ab909ad26f969480806f6d91911c
> ivy.xml 3f2c94373ba9455bbb6a3c96bfd61fc6cfaab588
> ivy/libraries.properties c2aed45a3244dfd108a255c7308a7dcb0dabd3b5
> src/org/apache/pig/backend/hadoop/executionengine/spark/FlatMapFunctionAdapter.java PRE-CREATION
> src/org/apache/pig/backend/hadoop/executionengine/spark/JobMetricsListener.java f81341233447203abc4800cc7b22a4f419e10262
> src/org/apache/pig/backend/hadoop/executionengine/spark/PairFlatMapFunctionAdapter.java PRE-CREATION
> src/org/apache/pig/backend/hadoop/executionengine/spark/Spark1Shims.java PRE-CREATION
> src/org/apache/pig/backend/hadoop/executionengine/spark/Spark2Shims.java PRE-CREATION
> src/org/apache/pig/backend/hadoop/executionengine/spark/SparkLauncher.java 237fd9431a16226234d91059088f91aab346b83c
> src/org/apache/pig/backend/hadoop/executionengine/spark/SparkShims.java PRE-CREATION
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/CollectedGroupConverter.java 83311dfa5bb25209a5366c2db7e8d483c31d94cd
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/FRJoinConverter.java 382258e7ff9105aa397c5a2888df0c11e9562ec9
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/ForEachConverter.java b58415e7e18ca4cf1331beef06e9214600a51424
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/GlobalRearrangeConverter.java 130c8b9a747b176ce2b649ca6d5260527595fb76
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/LimitConverter.java fe1b54c8f128661d7d19c276d3bb2de7874d3086
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/MergeCogroupConverter.java adf78ecab0da10d3b1a7fdde8af2b42dd899810f
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/MergeJoinConverter.java d1c43b1e06adc4c9fe45a83b81103333402e3756
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/PoissonSampleConverter.java e003bbd95763b2d189ff9ec540c89abe52592420
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SecondaryKeySortUtil.java 00d29b44848546ed16dde2baa8c61b36939971b2
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SkewedJoinConverter.java c55ba3145495a53d69db2dd56434dcc9b3bf8ed5
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SortConverter.java baabfa090323e3bef087e259ce19df2e4c34dd63
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SparkSampleSortConverter.java 3166fdc31745c013380492e089c83f3e853a3e6e
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/StreamConverter.java 3a50d485cfd54b9f3b9c1a982e6c30497a4c85fc
> src/org/apache/pig/tools/pigstats/spark/Spark1JobStats.java PRE-CREATION
> src/org/apache/pig/tools/pigstats/spark/Spark2JobStats.java PRE-CREATION
> src/org/apache/pig/tools/pigstats/spark/SparkJobStats.java c8cc03194b223d2ee181d73c6b651a6872cac6b6
> src/org/apache/pig/tools/pigstats/spark/SparkPigStats.java 61ccbcc9fd723f6e2e578a8476230c42d5587dfe
> test/org/apache/pig/test/TestPigRunner.java ec08417f2f71deec514cab5cfb9d2f99520ad641
>
>
> Diff: https://reviews.apache.org/r/59530/diff/6/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Nandor Kollar
>
>
Re: Review Request 59530: PIG-5157 Upgrade to Spark 2.0
Posted by Nandor Kollar <nk...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59530/
-----------------------------------------------------------
(Updated June 19, 2017, 9:41 a.m.)
Review request for pig, liyun zhang, Rohini Palaniswamy, and Adam Szita.
Changes
-------
Address comments from latest review
Repository: pig-git
Description
-------
Upgrade to Spark 2.1 API using shims.
Diffs (updated)
-----
build.xml bba2b52d9354ab909ad26f969480806f6d91911c
ivy.xml 3f2c94373ba9455bbb6a3c96bfd61fc6cfaab588
ivy/libraries.properties c2aed45a3244dfd108a255c7308a7dcb0dabd3b5
src/org/apache/pig/backend/hadoop/executionengine/spark/FlatMapFunctionAdapter.java PRE-CREATION
src/org/apache/pig/backend/hadoop/executionengine/spark/JobMetricsListener.java f81341233447203abc4800cc7b22a4f419e10262
src/org/apache/pig/backend/hadoop/executionengine/spark/PairFlatMapFunctionAdapter.java PRE-CREATION
src/org/apache/pig/backend/hadoop/executionengine/spark/Spark1Shims.java PRE-CREATION
src/org/apache/pig/backend/hadoop/executionengine/spark/Spark2Shims.java PRE-CREATION
src/org/apache/pig/backend/hadoop/executionengine/spark/SparkLauncher.java 237fd9431a16226234d91059088f91aab346b83c
src/org/apache/pig/backend/hadoop/executionengine/spark/SparkShims.java PRE-CREATION
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/CollectedGroupConverter.java 83311dfa5bb25209a5366c2db7e8d483c31d94cd
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/FRJoinConverter.java 382258e7ff9105aa397c5a2888df0c11e9562ec9
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/ForEachConverter.java b58415e7e18ca4cf1331beef06e9214600a51424
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/GlobalRearrangeConverter.java 130c8b9a747b176ce2b649ca6d5260527595fb76
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/LimitConverter.java fe1b54c8f128661d7d19c276d3bb2de7874d3086
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/MergeCogroupConverter.java adf78ecab0da10d3b1a7fdde8af2b42dd899810f
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/MergeJoinConverter.java d1c43b1e06adc4c9fe45a83b81103333402e3756
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/PoissonSampleConverter.java e003bbd95763b2d189ff9ec540c89abe52592420
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SecondaryKeySortUtil.java 00d29b44848546ed16dde2baa8c61b36939971b2
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SkewedJoinConverter.java c55ba3145495a53d69db2dd56434dcc9b3bf8ed5
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SortConverter.java baabfa090323e3bef087e259ce19df2e4c34dd63
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SparkSampleSortConverter.java 3166fdc31745c013380492e089c83f3e853a3e6e
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/StreamConverter.java 3a50d485cfd54b9f3b9c1a982e6c30497a4c85fc
src/org/apache/pig/tools/pigstats/spark/Spark1JobStats.java PRE-CREATION
src/org/apache/pig/tools/pigstats/spark/Spark2JobStats.java PRE-CREATION
src/org/apache/pig/tools/pigstats/spark/SparkJobStats.java c8cc03194b223d2ee181d73c6b651a6872cac6b6
src/org/apache/pig/tools/pigstats/spark/SparkPigStats.java 61ccbcc9fd723f6e2e578a8476230c42d5587dfe
test/org/apache/pig/test/TestPigRunner.java ec08417f2f71deec514cab5cfb9d2f99520ad641
Diff: https://reviews.apache.org/r/59530/diff/6/
Changes: https://reviews.apache.org/r/59530/diff/5-6/
Testing
-------
Thanks,
Nandor Kollar
Re: Review Request 59530: PIG-5157 Upgrade to Spark 2.0
Posted by Nandor Kollar <nk...@cloudera.com>.
> On June 15, 2017, 10:18 a.m., Adam Szita wrote:
> > build.xml
> > Lines 795 (patched)
> > <https://reviews.apache.org/r/59530/diff/5/?file=1750170#file1750170line796>
> >
> > This will leave the jar inside build directory having Spark2 classes only. We should fix that, so there is no discrepancy between pig jars generated.
Deleting the jar from build directory as discussed offline.
- Nandor
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59530/#review177990
-----------------------------------------------------------
On June 19, 2017, 9:41 a.m., Nandor Kollar wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/59530/
> -----------------------------------------------------------
>
> (Updated June 19, 2017, 9:41 a.m.)
>
>
> Review request for pig, liyun zhang, Rohini Palaniswamy, and Adam Szita.
>
>
> Repository: pig-git
>
>
> Description
> -------
>
> Upgrade to Spark 2.1 API using shims.
>
>
> Diffs
> -----
>
> build.xml bba2b52d9354ab909ad26f969480806f6d91911c
> ivy.xml 3f2c94373ba9455bbb6a3c96bfd61fc6cfaab588
> ivy/libraries.properties c2aed45a3244dfd108a255c7308a7dcb0dabd3b5
> src/org/apache/pig/backend/hadoop/executionengine/spark/FlatMapFunctionAdapter.java PRE-CREATION
> src/org/apache/pig/backend/hadoop/executionengine/spark/JobMetricsListener.java f81341233447203abc4800cc7b22a4f419e10262
> src/org/apache/pig/backend/hadoop/executionengine/spark/PairFlatMapFunctionAdapter.java PRE-CREATION
> src/org/apache/pig/backend/hadoop/executionengine/spark/Spark1Shims.java PRE-CREATION
> src/org/apache/pig/backend/hadoop/executionengine/spark/Spark2Shims.java PRE-CREATION
> src/org/apache/pig/backend/hadoop/executionengine/spark/SparkLauncher.java 237fd9431a16226234d91059088f91aab346b83c
> src/org/apache/pig/backend/hadoop/executionengine/spark/SparkShims.java PRE-CREATION
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/CollectedGroupConverter.java 83311dfa5bb25209a5366c2db7e8d483c31d94cd
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/FRJoinConverter.java 382258e7ff9105aa397c5a2888df0c11e9562ec9
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/ForEachConverter.java b58415e7e18ca4cf1331beef06e9214600a51424
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/GlobalRearrangeConverter.java 130c8b9a747b176ce2b649ca6d5260527595fb76
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/LimitConverter.java fe1b54c8f128661d7d19c276d3bb2de7874d3086
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/MergeCogroupConverter.java adf78ecab0da10d3b1a7fdde8af2b42dd899810f
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/MergeJoinConverter.java d1c43b1e06adc4c9fe45a83b81103333402e3756
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/PoissonSampleConverter.java e003bbd95763b2d189ff9ec540c89abe52592420
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SecondaryKeySortUtil.java 00d29b44848546ed16dde2baa8c61b36939971b2
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SkewedJoinConverter.java c55ba3145495a53d69db2dd56434dcc9b3bf8ed5
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SortConverter.java baabfa090323e3bef087e259ce19df2e4c34dd63
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SparkSampleSortConverter.java 3166fdc31745c013380492e089c83f3e853a3e6e
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/StreamConverter.java 3a50d485cfd54b9f3b9c1a982e6c30497a4c85fc
> src/org/apache/pig/tools/pigstats/spark/Spark1JobStats.java PRE-CREATION
> src/org/apache/pig/tools/pigstats/spark/Spark2JobStats.java PRE-CREATION
> src/org/apache/pig/tools/pigstats/spark/SparkJobStats.java c8cc03194b223d2ee181d73c6b651a6872cac6b6
> src/org/apache/pig/tools/pigstats/spark/SparkPigStats.java 61ccbcc9fd723f6e2e578a8476230c42d5587dfe
> test/org/apache/pig/test/TestPigRunner.java ec08417f2f71deec514cab5cfb9d2f99520ad641
>
>
> Diff: https://reviews.apache.org/r/59530/diff/6/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Nandor Kollar
>
>
Re: Review Request 59530: PIG-5157 Upgrade to Spark 2.0
Posted by Adam Szita <sz...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59530/#review177990
-----------------------------------------------------------
build.xml
Lines 245 (patched)
<https://reviews.apache.org/r/59530/#comment251720>
Should we call this src.exclude.pattern instead of dir?
build.xml
Lines 795 (patched)
<https://reviews.apache.org/r/59530/#comment251721>
This will leave the jar inside build directory having Spark2 classes only. We should fix that, so there is no discrepancy between pig jars generated.
- Adam Szita
On June 14, 2017, 9:13 a.m., Nandor Kollar wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/59530/
> -----------------------------------------------------------
>
> (Updated June 14, 2017, 9:13 a.m.)
>
>
> Review request for pig, liyun zhang, Rohini Palaniswamy, and Adam Szita.
>
>
> Repository: pig-git
>
>
> Description
> -------
>
> Upgrade to Spark 2.1 API using shims.
>
>
> Diffs
> -----
>
> build.xml bba2b52d9354ab909ad26f969480806f6d91911c
> ivy.xml 3f2c94373ba9455bbb6a3c96bfd61fc6cfaab588
> ivy/libraries.properties c2aed45a3244dfd108a255c7308a7dcb0dabd3b5
> src/org/apache/pig/backend/hadoop/executionengine/spark/FlatMapFunctionAdapter.java PRE-CREATION
> src/org/apache/pig/backend/hadoop/executionengine/spark/JobMetricsListener.java f81341233447203abc4800cc7b22a4f419e10262
> src/org/apache/pig/backend/hadoop/executionengine/spark/PairFlatMapFunctionAdapter.java PRE-CREATION
> src/org/apache/pig/backend/hadoop/executionengine/spark/SparkLauncher.java 237fd9431a16226234d91059088f91aab346b83c
> src/org/apache/pig/backend/hadoop/executionengine/spark/SparkShim.java PRE-CREATION
> src/org/apache/pig/backend/hadoop/executionengine/spark/SparkShim1.java PRE-CREATION
> src/org/apache/pig/backend/hadoop/executionengine/spark/SparkShim2.java PRE-CREATION
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/CollectedGroupConverter.java 83311dfa5bb25209a5366c2db7e8d483c31d94cd
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/FRJoinConverter.java 382258e7ff9105aa397c5a2888df0c11e9562ec9
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/ForEachConverter.java b58415e7e18ca4cf1331beef06e9214600a51424
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/GlobalRearrangeConverter.java 130c8b9a747b176ce2b649ca6d5260527595fb76
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/LimitConverter.java fe1b54c8f128661d7d19c276d3bb2de7874d3086
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/MergeCogroupConverter.java adf78ecab0da10d3b1a7fdde8af2b42dd899810f
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/MergeJoinConverter.java d1c43b1e06adc4c9fe45a83b81103333402e3756
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/PoissonSampleConverter.java e003bbd95763b2d189ff9ec540c89abe52592420
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SecondaryKeySortUtil.java 00d29b44848546ed16dde2baa8c61b36939971b2
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SkewedJoinConverter.java c55ba3145495a53d69db2dd56434dcc9b3bf8ed5
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SortConverter.java baabfa090323e3bef087e259ce19df2e4c34dd63
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SparkSampleSortConverter.java 3166fdc31745c013380492e089c83f3e853a3e6e
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/StreamConverter.java 3a50d485cfd54b9f3b9c1a982e6c30497a4c85fc
> src/org/apache/pig/tools/pigstats/spark/SparkJobStats.java c8cc03194b223d2ee181d73c6b651a6872cac6b6
> src/org/apache/pig/tools/pigstats/spark/SparkJobStats1.java PRE-CREATION
> src/org/apache/pig/tools/pigstats/spark/SparkJobStats2.java PRE-CREATION
> src/org/apache/pig/tools/pigstats/spark/SparkPigStats.java 61ccbcc9fd723f6e2e578a8476230c42d5587dfe
> test/org/apache/pig/test/TestPigRunner.java ec08417f2f71deec514cab5cfb9d2f99520ad641
>
>
> Diff: https://reviews.apache.org/r/59530/diff/5/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Nandor Kollar
>
>
Re: Review Request 59530: PIG-5157 Upgrade to Spark 2.0
Posted by Adam Szita <sz...@cloudera.com>.
> On June 16, 2017, 9:33 p.m., Rohini Palaniswamy wrote:
> > build.xml
> > Lines 795 (patched)
> > <https://reviews.apache.org/r/59530/diff/2-5/?file=1732429#file1732429line806>
> >
> > Can you rename the current "jar" target to "jar-simple" and actually make this one part of the jar target? HBase and Hadoop3 can be added to it in the future. This way anyone who builds with "ant clean jar" as today gets a pig jar usable across all HBase/Spark/Hadoop3 versions.
> > Developers like us can use the jar-simple target to save time.
>
> Nandor Kollar wrote:
> Renamed targets as requested, should the default remain jar, or it should be jar-simple instead?
I believe the default should stay "jar" for the same reasons expressed above
- Adam
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59530/#review178126
-----------------------------------------------------------
On June 19, 2017, 9:41 a.m., Nandor Kollar wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/59530/
> -----------------------------------------------------------
>
> (Updated June 19, 2017, 9:41 a.m.)
>
>
> Review request for pig, liyun zhang, Rohini Palaniswamy, and Adam Szita.
>
>
> Repository: pig-git
>
>
> Description
> -------
>
> Upgrade to Spark 2.1 API using shims.
>
>
> Diffs
> -----
>
> build.xml bba2b52d9354ab909ad26f969480806f6d91911c
> ivy.xml 3f2c94373ba9455bbb6a3c96bfd61fc6cfaab588
> ivy/libraries.properties c2aed45a3244dfd108a255c7308a7dcb0dabd3b5
> src/org/apache/pig/backend/hadoop/executionengine/spark/FlatMapFunctionAdapter.java PRE-CREATION
> src/org/apache/pig/backend/hadoop/executionengine/spark/JobMetricsListener.java f81341233447203abc4800cc7b22a4f419e10262
> src/org/apache/pig/backend/hadoop/executionengine/spark/PairFlatMapFunctionAdapter.java PRE-CREATION
> src/org/apache/pig/backend/hadoop/executionengine/spark/Spark1Shims.java PRE-CREATION
> src/org/apache/pig/backend/hadoop/executionengine/spark/Spark2Shims.java PRE-CREATION
> src/org/apache/pig/backend/hadoop/executionengine/spark/SparkLauncher.java 237fd9431a16226234d91059088f91aab346b83c
> src/org/apache/pig/backend/hadoop/executionengine/spark/SparkShims.java PRE-CREATION
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/CollectedGroupConverter.java 83311dfa5bb25209a5366c2db7e8d483c31d94cd
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/FRJoinConverter.java 382258e7ff9105aa397c5a2888df0c11e9562ec9
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/ForEachConverter.java b58415e7e18ca4cf1331beef06e9214600a51424
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/GlobalRearrangeConverter.java 130c8b9a747b176ce2b649ca6d5260527595fb76
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/LimitConverter.java fe1b54c8f128661d7d19c276d3bb2de7874d3086
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/MergeCogroupConverter.java adf78ecab0da10d3b1a7fdde8af2b42dd899810f
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/MergeJoinConverter.java d1c43b1e06adc4c9fe45a83b81103333402e3756
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/PoissonSampleConverter.java e003bbd95763b2d189ff9ec540c89abe52592420
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SecondaryKeySortUtil.java 00d29b44848546ed16dde2baa8c61b36939971b2
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SkewedJoinConverter.java c55ba3145495a53d69db2dd56434dcc9b3bf8ed5
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SortConverter.java baabfa090323e3bef087e259ce19df2e4c34dd63
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SparkSampleSortConverter.java 3166fdc31745c013380492e089c83f3e853a3e6e
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/StreamConverter.java 3a50d485cfd54b9f3b9c1a982e6c30497a4c85fc
> src/org/apache/pig/tools/pigstats/spark/Spark1JobStats.java PRE-CREATION
> src/org/apache/pig/tools/pigstats/spark/Spark2JobStats.java PRE-CREATION
> src/org/apache/pig/tools/pigstats/spark/SparkJobStats.java c8cc03194b223d2ee181d73c6b651a6872cac6b6
> src/org/apache/pig/tools/pigstats/spark/SparkPigStats.java 61ccbcc9fd723f6e2e578a8476230c42d5587dfe
> test/org/apache/pig/test/TestPigRunner.java ec08417f2f71deec514cab5cfb9d2f99520ad641
>
>
> Diff: https://reviews.apache.org/r/59530/diff/6/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Nandor Kollar
>
>
Re: Review Request 59530: PIG-5157 Upgrade to Spark 2.0
Posted by Nandor Kollar <nk...@cloudera.com>.
> On June 16, 2017, 9:33 p.m., Rohini Palaniswamy wrote:
> > build.xml
> > Lines 795 (patched)
> > <https://reviews.apache.org/r/59530/diff/2-5/?file=1732429#file1732429line806>
> >
> > Can you rename the current "jar" target to "jar-simple" and actually make this one part of the jar target? HBase and Hadoop3 can be added to it in the future. This way anyone who builds with "ant clean jar" as today gets a pig jar usable across all HBase/Spark/Hadoop3 versions.
> > Developers like us can use the jar-simple target to save time.
Renamed targets as requested, should the default remain jar, or it should be jar-simple instead?
- Nandor
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59530/#review178126
-----------------------------------------------------------
On June 19, 2017, 9:41 a.m., Nandor Kollar wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/59530/
> -----------------------------------------------------------
>
> (Updated June 19, 2017, 9:41 a.m.)
>
>
> Review request for pig, liyun zhang, Rohini Palaniswamy, and Adam Szita.
>
>
> Repository: pig-git
>
>
> Description
> -------
>
> Upgrade to Spark 2.1 API using shims.
>
>
> Diffs
> -----
>
> build.xml bba2b52d9354ab909ad26f969480806f6d91911c
> ivy.xml 3f2c94373ba9455bbb6a3c96bfd61fc6cfaab588
> ivy/libraries.properties c2aed45a3244dfd108a255c7308a7dcb0dabd3b5
> src/org/apache/pig/backend/hadoop/executionengine/spark/FlatMapFunctionAdapter.java PRE-CREATION
> src/org/apache/pig/backend/hadoop/executionengine/spark/JobMetricsListener.java f81341233447203abc4800cc7b22a4f419e10262
> src/org/apache/pig/backend/hadoop/executionengine/spark/PairFlatMapFunctionAdapter.java PRE-CREATION
> src/org/apache/pig/backend/hadoop/executionengine/spark/Spark1Shims.java PRE-CREATION
> src/org/apache/pig/backend/hadoop/executionengine/spark/Spark2Shims.java PRE-CREATION
> src/org/apache/pig/backend/hadoop/executionengine/spark/SparkLauncher.java 237fd9431a16226234d91059088f91aab346b83c
> src/org/apache/pig/backend/hadoop/executionengine/spark/SparkShims.java PRE-CREATION
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/CollectedGroupConverter.java 83311dfa5bb25209a5366c2db7e8d483c31d94cd
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/FRJoinConverter.java 382258e7ff9105aa397c5a2888df0c11e9562ec9
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/ForEachConverter.java b58415e7e18ca4cf1331beef06e9214600a51424
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/GlobalRearrangeConverter.java 130c8b9a747b176ce2b649ca6d5260527595fb76
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/LimitConverter.java fe1b54c8f128661d7d19c276d3bb2de7874d3086
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/MergeCogroupConverter.java adf78ecab0da10d3b1a7fdde8af2b42dd899810f
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/MergeJoinConverter.java d1c43b1e06adc4c9fe45a83b81103333402e3756
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/PoissonSampleConverter.java e003bbd95763b2d189ff9ec540c89abe52592420
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SecondaryKeySortUtil.java 00d29b44848546ed16dde2baa8c61b36939971b2
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SkewedJoinConverter.java c55ba3145495a53d69db2dd56434dcc9b3bf8ed5
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SortConverter.java baabfa090323e3bef087e259ce19df2e4c34dd63
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SparkSampleSortConverter.java 3166fdc31745c013380492e089c83f3e853a3e6e
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/StreamConverter.java 3a50d485cfd54b9f3b9c1a982e6c30497a4c85fc
> src/org/apache/pig/tools/pigstats/spark/Spark1JobStats.java PRE-CREATION
> src/org/apache/pig/tools/pigstats/spark/Spark2JobStats.java PRE-CREATION
> src/org/apache/pig/tools/pigstats/spark/SparkJobStats.java c8cc03194b223d2ee181d73c6b651a6872cac6b6
> src/org/apache/pig/tools/pigstats/spark/SparkPigStats.java 61ccbcc9fd723f6e2e578a8476230c42d5587dfe
> test/org/apache/pig/test/TestPigRunner.java ec08417f2f71deec514cab5cfb9d2f99520ad641
>
>
> Diff: https://reviews.apache.org/r/59530/diff/6/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Nandor Kollar
>
>
Re: Review Request 59530: PIG-5157 Upgrade to Spark 2.0
Posted by Rohini Palaniswamy <ro...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59530/#review178126
-----------------------------------------------------------
Looks good. Just minor changes for class names and ant target.
build.xml
Lines 795 (patched)
<https://reviews.apache.org/r/59530/#comment251955>
Can you rename the current "jar" target to "jar-simple" and actually make this one part of the jar target? HBase and Hadoop3 can be added to it in the future. This way anyone who builds with "ant clean jar" as today gets a pig jar usable across all HBase/Spark/Hadoop3 versions.
Developers like us can use the jar-simple target to save time.
src/org/apache/pig/backend/hadoop/executionengine/spark/SparkShim.java
Lines 38 (patched)
<https://reviews.apache.org/r/59530/#comment251956>
SparkShims
src/org/apache/pig/backend/hadoop/executionengine/spark/SparkShim.java
Lines 68 (patched)
<https://reviews.apache.org/r/59530/#comment251959>
getInstance(). get() makes you think it is a ThreadLocal variable.
src/org/apache/pig/backend/hadoop/executionengine/spark/SparkShim1.java
Lines 59 (patched)
<https://reviews.apache.org/r/59530/#comment251957>
Spark1Shims
src/org/apache/pig/backend/hadoop/executionengine/spark/SparkShim2.java
Lines 46 (patched)
<https://reviews.apache.org/r/59530/#comment251958>
Spark2Shims
src/org/apache/pig/tools/pigstats/spark/SparkJobStats1.java
Lines 32 (patched)
<https://reviews.apache.org/r/59530/#comment251960>
Spark1JobStats
src/org/apache/pig/tools/pigstats/spark/SparkJobStats2.java
Lines 31 (patched)
<https://reviews.apache.org/r/59530/#comment251961>
Spark2JobStats
- Rohini Palaniswamy
On June 14, 2017, 9:13 a.m., Nandor Kollar wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/59530/
> -----------------------------------------------------------
>
> (Updated June 14, 2017, 9:13 a.m.)
>
>
> Review request for pig, liyun zhang, Rohini Palaniswamy, and Adam Szita.
>
>
> Repository: pig-git
>
>
> Description
> -------
>
> Upgrade to Spark 2.1 API using shims.
>
>
> Diffs
> -----
>
> build.xml bba2b52d9354ab909ad26f969480806f6d91911c
> ivy.xml 3f2c94373ba9455bbb6a3c96bfd61fc6cfaab588
> ivy/libraries.properties c2aed45a3244dfd108a255c7308a7dcb0dabd3b5
> src/org/apache/pig/backend/hadoop/executionengine/spark/FlatMapFunctionAdapter.java PRE-CREATION
> src/org/apache/pig/backend/hadoop/executionengine/spark/JobMetricsListener.java f81341233447203abc4800cc7b22a4f419e10262
> src/org/apache/pig/backend/hadoop/executionengine/spark/PairFlatMapFunctionAdapter.java PRE-CREATION
> src/org/apache/pig/backend/hadoop/executionengine/spark/SparkLauncher.java 237fd9431a16226234d91059088f91aab346b83c
> src/org/apache/pig/backend/hadoop/executionengine/spark/SparkShim.java PRE-CREATION
> src/org/apache/pig/backend/hadoop/executionengine/spark/SparkShim1.java PRE-CREATION
> src/org/apache/pig/backend/hadoop/executionengine/spark/SparkShim2.java PRE-CREATION
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/CollectedGroupConverter.java 83311dfa5bb25209a5366c2db7e8d483c31d94cd
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/FRJoinConverter.java 382258e7ff9105aa397c5a2888df0c11e9562ec9
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/ForEachConverter.java b58415e7e18ca4cf1331beef06e9214600a51424
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/GlobalRearrangeConverter.java 130c8b9a747b176ce2b649ca6d5260527595fb76
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/LimitConverter.java fe1b54c8f128661d7d19c276d3bb2de7874d3086
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/MergeCogroupConverter.java adf78ecab0da10d3b1a7fdde8af2b42dd899810f
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/MergeJoinConverter.java d1c43b1e06adc4c9fe45a83b81103333402e3756
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/PoissonSampleConverter.java e003bbd95763b2d189ff9ec540c89abe52592420
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SecondaryKeySortUtil.java 00d29b44848546ed16dde2baa8c61b36939971b2
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SkewedJoinConverter.java c55ba3145495a53d69db2dd56434dcc9b3bf8ed5
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SortConverter.java baabfa090323e3bef087e259ce19df2e4c34dd63
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SparkSampleSortConverter.java 3166fdc31745c013380492e089c83f3e853a3e6e
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/StreamConverter.java 3a50d485cfd54b9f3b9c1a982e6c30497a4c85fc
> src/org/apache/pig/tools/pigstats/spark/SparkJobStats.java c8cc03194b223d2ee181d73c6b651a6872cac6b6
> src/org/apache/pig/tools/pigstats/spark/SparkJobStats1.java PRE-CREATION
> src/org/apache/pig/tools/pigstats/spark/SparkJobStats2.java PRE-CREATION
> src/org/apache/pig/tools/pigstats/spark/SparkPigStats.java 61ccbcc9fd723f6e2e578a8476230c42d5587dfe
> test/org/apache/pig/test/TestPigRunner.java ec08417f2f71deec514cab5cfb9d2f99520ad641
>
>
> Diff: https://reviews.apache.org/r/59530/diff/5/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Nandor Kollar
>
>
Re: Review Request 59530: PIG-5157 Upgrade to Spark 2.0
Posted by Nandor Kollar <nk...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59530/
-----------------------------------------------------------
(Updated June 14, 2017, 9:13 a.m.)
Review request for pig, liyun zhang, Rohini Palaniswamy, and Adam Szita.
Changes
-------
updated JobMetricsListener for Spark 1.x and 2.x, patch created on top of the latest trunk codebase
Repository: pig-git
Description
-------
Upgrade to Spark 2.1 API using shims.
Diffs (updated)
-----
build.xml bba2b52d9354ab909ad26f969480806f6d91911c
ivy.xml 3f2c94373ba9455bbb6a3c96bfd61fc6cfaab588
ivy/libraries.properties c2aed45a3244dfd108a255c7308a7dcb0dabd3b5
src/org/apache/pig/backend/hadoop/executionengine/spark/FlatMapFunctionAdapter.java PRE-CREATION
src/org/apache/pig/backend/hadoop/executionengine/spark/JobMetricsListener.java f81341233447203abc4800cc7b22a4f419e10262
src/org/apache/pig/backend/hadoop/executionengine/spark/PairFlatMapFunctionAdapter.java PRE-CREATION
src/org/apache/pig/backend/hadoop/executionengine/spark/SparkLauncher.java 237fd9431a16226234d91059088f91aab346b83c
src/org/apache/pig/backend/hadoop/executionengine/spark/SparkShim.java PRE-CREATION
src/org/apache/pig/backend/hadoop/executionengine/spark/SparkShim1.java PRE-CREATION
src/org/apache/pig/backend/hadoop/executionengine/spark/SparkShim2.java PRE-CREATION
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/CollectedGroupConverter.java 83311dfa5bb25209a5366c2db7e8d483c31d94cd
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/FRJoinConverter.java 382258e7ff9105aa397c5a2888df0c11e9562ec9
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/ForEachConverter.java b58415e7e18ca4cf1331beef06e9214600a51424
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/GlobalRearrangeConverter.java 130c8b9a747b176ce2b649ca6d5260527595fb76
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/LimitConverter.java fe1b54c8f128661d7d19c276d3bb2de7874d3086
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/MergeCogroupConverter.java adf78ecab0da10d3b1a7fdde8af2b42dd899810f
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/MergeJoinConverter.java d1c43b1e06adc4c9fe45a83b81103333402e3756
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/PoissonSampleConverter.java e003bbd95763b2d189ff9ec540c89abe52592420
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SecondaryKeySortUtil.java 00d29b44848546ed16dde2baa8c61b36939971b2
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SkewedJoinConverter.java c55ba3145495a53d69db2dd56434dcc9b3bf8ed5
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SortConverter.java baabfa090323e3bef087e259ce19df2e4c34dd63
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SparkSampleSortConverter.java 3166fdc31745c013380492e089c83f3e853a3e6e
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/StreamConverter.java 3a50d485cfd54b9f3b9c1a982e6c30497a4c85fc
src/org/apache/pig/tools/pigstats/spark/SparkJobStats.java c8cc03194b223d2ee181d73c6b651a6872cac6b6
src/org/apache/pig/tools/pigstats/spark/SparkJobStats1.java PRE-CREATION
src/org/apache/pig/tools/pigstats/spark/SparkJobStats2.java PRE-CREATION
src/org/apache/pig/tools/pigstats/spark/SparkPigStats.java 61ccbcc9fd723f6e2e578a8476230c42d5587dfe
test/org/apache/pig/test/TestPigRunner.java ec08417f2f71deec514cab5cfb9d2f99520ad641
Diff: https://reviews.apache.org/r/59530/diff/5/
Changes: https://reviews.apache.org/r/59530/diff/4-5/
Testing
-------
Thanks,
Nandor Kollar
Re: Review Request 59530: PIG-5157 Upgrade to Spark 2.0
Posted by Nandor Kollar <nk...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59530/
-----------------------------------------------------------
(Updated June 12, 2017, 9:20 p.m.)
Review request for pig, liyun zhang, Rohini Palaniswamy, and Adam Szita.
Changes
-------
Assemble jar for both Spark 1.x and 2.x
Repository: pig-git
Description
-------
Upgrade to Spark 2.1 API using shims.
Diffs (updated)
-----
build.xml 4040fcec8f88d448ed7442461fbf0dea8cd1136e
ivy.xml 971724380f086d214ce62c7ab7879b08b6926802
ivy/libraries.properties a0eb00acd2df42324540df4a9d762c64c608a6d3
src/org/apache/pig/backend/hadoop/executionengine/spark/FlatMapFunctionAdapter.java PRE-CREATION
src/org/apache/pig/backend/hadoop/executionengine/spark/JobMetricsListener.java f81341233447203abc4800cc7b22a4f419e10262
src/org/apache/pig/backend/hadoop/executionengine/spark/PairFlatMapFunctionAdapter.java PRE-CREATION
src/org/apache/pig/backend/hadoop/executionengine/spark/SparkLauncher.java c6351e01a48f297ea2e432401ffd65c4f27f8078
src/org/apache/pig/backend/hadoop/executionengine/spark/SparkShim.java PRE-CREATION
src/org/apache/pig/backend/hadoop/executionengine/spark/SparkShim1.java PRE-CREATION
src/org/apache/pig/backend/hadoop/executionengine/spark/SparkShim2.java PRE-CREATION
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/CollectedGroupConverter.java 83311dfa5bb25209a5366c2db7e8d483c31d94cd
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/FRJoinConverter.java 382258e7ff9105aa397c5a2888df0c11e9562ec9
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/ForEachConverter.java b58415e7e18ca4cf1331beef06e9214600a51424
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/GlobalRearrangeConverter.java f571b808839c2de9415a3e8e4b229a7f4b2eebd7
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/LimitConverter.java fe1b54c8f128661d7d19c276d3bb2de7874d3086
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/MergeCogroupConverter.java adf78ecab0da10d3b1a7fdde8af2b42dd899810f
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/MergeJoinConverter.java d1c43b1e06adc4c9fe45a83b81103333402e3756
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/PoissonSampleConverter.java e003bbd95763b2d189ff9ec540c89abe52592420
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SecondaryKeySortUtil.java 00d29b44848546ed16dde2baa8c61b36939971b2
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SkewedJoinConverter.java c55ba3145495a53d69db2dd56434dcc9b3bf8ed5
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SortConverter.java baabfa090323e3bef087e259ce19df2e4c34dd63
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SparkSampleSortConverter.java 3166fdc31745c013380492e089c83f3e853a3e6e
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/StreamConverter.java 3a50d485cfd54b9f3b9c1a982e6c30497a4c85fc
src/org/apache/pig/tools/pigstats/spark/SparkJobStats.java c8cc03194b223d2ee181d73c6b651a6872cac6b6
src/org/apache/pig/tools/pigstats/spark/SparkJobStats1.java PRE-CREATION
src/org/apache/pig/tools/pigstats/spark/SparkJobStats2.java PRE-CREATION
src/org/apache/pig/tools/pigstats/spark/SparkPigStats.java 61ccbcc9fd723f6e2e578a8476230c42d5587dfe
test/org/apache/pig/test/TestPigRunner.java 1b1b11b0ccaca2cddeb3ad8ddc659a1c8a75722c
Diff: https://reviews.apache.org/r/59530/diff/4/
Changes: https://reviews.apache.org/r/59530/diff/3-4/
Testing
-------
Thanks,
Nandor Kollar
Re: Review Request 59530: PIG-5157 Upgrade to Spark 2.0
Posted by Nandor Kollar <nk...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59530/
-----------------------------------------------------------
(Updated June 2, 2017, 3:20 p.m.)
Review request for pig, liyun zhang, Rohini Palaniswamy, and Adam Szita.
Changes
-------
fix for ClassCastException in skew join revealed while executing e2e tests
Repository: pig-git
Description
-------
Upgrade to Spark 2.1 API using shims.
Diffs (updated)
-----
build.xml 4040fcec8f88d448ed7442461fbf0dea8cd1136e
ivy/spark2.properties PRE-CREATION
shims/src/spark1/org/apache/pig/backend/hadoop/executionengine/spark/JobMetricsListener.java PRE-CREATION
shims/src/spark1/org/apache/pig/backend/hadoop/executionengine/spark/converter/AbstractFlatMapFunction.java PRE-CREATION
shims/src/spark1/org/apache/pig/backend/hadoop/executionengine/spark/converter/AbstractLimitConverter.java PRE-CREATION
shims/src/spark1/org/apache/pig/backend/hadoop/executionengine/spark/converter/AbstractPairFlatMapFunction.java PRE-CREATION
shims/src/spark1/org/apache/pig/backend/hadoop/executionengine/spark/converter/SkewedJoinUtil.java PRE-CREATION
shims/src/spark1/org/apache/pig/tools/pigstats/spark/SparkJobStats.java PRE-CREATION
shims/src/spark2/org/apache/pig/backend/hadoop/executionengine/spark/JobMetricsListener.java PRE-CREATION
shims/src/spark2/org/apache/pig/backend/hadoop/executionengine/spark/converter/AbstractFlatMapFunction.java PRE-CREATION
shims/src/spark2/org/apache/pig/backend/hadoop/executionengine/spark/converter/AbstractLimitConverter.java PRE-CREATION
shims/src/spark2/org/apache/pig/backend/hadoop/executionengine/spark/converter/AbstractPairFlatMapFunction.java PRE-CREATION
shims/src/spark2/org/apache/pig/backend/hadoop/executionengine/spark/converter/SkewedJoinUtil.java PRE-CREATION
shims/src/spark2/org/apache/pig/tools/pigstats/spark/SparkJobStats.java PRE-CREATION
src/org/apache/pig/backend/hadoop/executionengine/spark/JobMetricsListener.java f81341233447203abc4800cc7b22a4f419e10262
src/org/apache/pig/backend/hadoop/executionengine/spark/SparkLauncher.java c6351e01a48f297ea2e432401ffd65c4f27f8078
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/CollectedGroupConverter.java 83311dfa5bb25209a5366c2db7e8d483c31d94cd
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/FRJoinConverter.java 382258e7ff9105aa397c5a2888df0c11e9562ec9
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/ForEachConverter.java b58415e7e18ca4cf1331beef06e9214600a51424
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/GlobalRearrangeConverter.java f571b808839c2de9415a3e8e4b229a7f4b2eebd7
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/LimitConverter.java fe1b54c8f128661d7d19c276d3bb2de7874d3086
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/MergeCogroupConverter.java adf78ecab0da10d3b1a7fdde8af2b42dd899810f
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/MergeJoinConverter.java d1c43b1e06adc4c9fe45a83b81103333402e3756
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/PoissonSampleConverter.java e003bbd95763b2d189ff9ec540c89abe52592420
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SecondaryKeySortUtil.java 00d29b44848546ed16dde2baa8c61b36939971b2
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SkewedJoinConverter.java c55ba3145495a53d69db2dd56434dcc9b3bf8ed5
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SortConverter.java baabfa090323e3bef087e259ce19df2e4c34dd63
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SparkSampleSortConverter.java 3166fdc31745c013380492e089c83f3e853a3e6e
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/StreamConverter.java 3a50d485cfd54b9f3b9c1a982e6c30497a4c85fc
src/org/apache/pig/tools/pigstats/spark/SparkJobStats.java c8cc03194b223d2ee181d73c6b651a6872cac6b6
Diff: https://reviews.apache.org/r/59530/diff/3/
Changes: https://reviews.apache.org/r/59530/diff/2-3/
Testing
-------
Thanks,
Nandor Kollar
Re: Review Request 59530: PIG-5157 Upgrade to Spark 2.0
Posted by Rohini Palaniswamy <ro...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59530/#review176232
-----------------------------------------------------------
Ship it!
Ship It!
- Rohini Palaniswamy
On May 25, 2017, 7:13 a.m., Nandor Kollar wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/59530/
> -----------------------------------------------------------
>
> (Updated May 25, 2017, 7:13 a.m.)
>
>
> Review request for pig, liyun zhang, Rohini Palaniswamy, and Adam Szita.
>
>
> Repository: pig-git
>
>
> Description
> -------
>
> Upgrade to Spark 2.1 API using shims.
>
>
> Diffs
> -----
>
> build.xml 4040fce
> ivy/spark2.properties PRE-CREATION
> shims/src/spark1/org/apache/pig/backend/hadoop/executionengine/spark/JobMetricsListener.java PRE-CREATION
> shims/src/spark1/org/apache/pig/backend/hadoop/executionengine/spark/converter/AbstractFlatMapFunction.java PRE-CREATION
> shims/src/spark1/org/apache/pig/backend/hadoop/executionengine/spark/converter/AbstractLimitConverter.java PRE-CREATION
> shims/src/spark1/org/apache/pig/backend/hadoop/executionengine/spark/converter/AbstractPairFlatMapFunction.java PRE-CREATION
> shims/src/spark1/org/apache/pig/backend/hadoop/executionengine/spark/converter/SkewedJoinUtil.java PRE-CREATION
> shims/src/spark1/org/apache/pig/tools/pigstats/spark/SparkJobStats.java PRE-CREATION
> shims/src/spark2/org/apache/pig/backend/hadoop/executionengine/spark/JobMetricsListener.java PRE-CREATION
> shims/src/spark2/org/apache/pig/backend/hadoop/executionengine/spark/converter/AbstractFlatMapFunction.java PRE-CREATION
> shims/src/spark2/org/apache/pig/backend/hadoop/executionengine/spark/converter/AbstractLimitConverter.java PRE-CREATION
> shims/src/spark2/org/apache/pig/backend/hadoop/executionengine/spark/converter/AbstractPairFlatMapFunction.java PRE-CREATION
> shims/src/spark2/org/apache/pig/backend/hadoop/executionengine/spark/converter/SkewedJoinUtil.java PRE-CREATION
> shims/src/spark2/org/apache/pig/tools/pigstats/spark/SparkJobStats.java PRE-CREATION
> src/org/apache/pig/backend/hadoop/executionengine/spark/JobMetricsListener.java f813412
> src/org/apache/pig/backend/hadoop/executionengine/spark/SparkLauncher.java c6351e0
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/CollectedGroupConverter.java 83311df
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/FRJoinConverter.java 382258e
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/ForEachConverter.java b58415e
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/GlobalRearrangeConverter.java f571b80
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/LimitConverter.java fe1b54c
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/MergeCogroupConverter.java adf78ec
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/MergeJoinConverter.java d1c43b1
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/PoissonSampleConverter.java e003bbd
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SecondaryKeySortUtil.java 00d29b4
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SkewedJoinConverter.java c55ba31
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SortConverter.java baabfa0
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SparkSampleSortConverter.java 3166fdc
> src/org/apache/pig/backend/hadoop/executionengine/spark/converter/StreamConverter.java 3a50d48
> src/org/apache/pig/tools/pigstats/spark/SparkJobStats.java c8cc031
>
>
> Diff: https://reviews.apache.org/r/59530/diff/2/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Nandor Kollar
>
>
Re: Review Request 59530: PIG-5157 Upgrade to Spark 2.0
Posted by Nandor Kollar <nk...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59530/
-----------------------------------------------------------
(Updated May 25, 2017, 7:13 a.m.)
Review request for pig, liyun zhang, Rohini Palaniswamy, and Adam Szita.
Repository: pig-git
Description
-------
Upgrade to Spark 2.1 API using shims.
Diffs (updated)
-----
build.xml 4040fce
ivy/spark2.properties PRE-CREATION
shims/src/spark1/org/apache/pig/backend/hadoop/executionengine/spark/JobMetricsListener.java PRE-CREATION
shims/src/spark1/org/apache/pig/backend/hadoop/executionengine/spark/converter/AbstractFlatMapFunction.java PRE-CREATION
shims/src/spark1/org/apache/pig/backend/hadoop/executionengine/spark/converter/AbstractLimitConverter.java PRE-CREATION
shims/src/spark1/org/apache/pig/backend/hadoop/executionengine/spark/converter/AbstractPairFlatMapFunction.java PRE-CREATION
shims/src/spark1/org/apache/pig/backend/hadoop/executionengine/spark/converter/SkewedJoinUtil.java PRE-CREATION
shims/src/spark1/org/apache/pig/tools/pigstats/spark/SparkJobStats.java PRE-CREATION
shims/src/spark2/org/apache/pig/backend/hadoop/executionengine/spark/JobMetricsListener.java PRE-CREATION
shims/src/spark2/org/apache/pig/backend/hadoop/executionengine/spark/converter/AbstractFlatMapFunction.java PRE-CREATION
shims/src/spark2/org/apache/pig/backend/hadoop/executionengine/spark/converter/AbstractLimitConverter.java PRE-CREATION
shims/src/spark2/org/apache/pig/backend/hadoop/executionengine/spark/converter/AbstractPairFlatMapFunction.java PRE-CREATION
shims/src/spark2/org/apache/pig/backend/hadoop/executionengine/spark/converter/SkewedJoinUtil.java PRE-CREATION
shims/src/spark2/org/apache/pig/tools/pigstats/spark/SparkJobStats.java PRE-CREATION
src/org/apache/pig/backend/hadoop/executionengine/spark/JobMetricsListener.java f813412
src/org/apache/pig/backend/hadoop/executionengine/spark/SparkLauncher.java c6351e0
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/CollectedGroupConverter.java 83311df
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/FRJoinConverter.java 382258e
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/ForEachConverter.java b58415e
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/GlobalRearrangeConverter.java f571b80
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/LimitConverter.java fe1b54c
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/MergeCogroupConverter.java adf78ec
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/MergeJoinConverter.java d1c43b1
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/PoissonSampleConverter.java e003bbd
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SecondaryKeySortUtil.java 00d29b4
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SkewedJoinConverter.java c55ba31
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SortConverter.java baabfa0
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/SparkSampleSortConverter.java 3166fdc
src/org/apache/pig/backend/hadoop/executionengine/spark/converter/StreamConverter.java 3a50d48
src/org/apache/pig/tools/pigstats/spark/SparkJobStats.java c8cc031
Diff: https://reviews.apache.org/r/59530/diff/2/
Changes: https://reviews.apache.org/r/59530/diff/1-2/
Testing
-------
Thanks,
Nandor Kollar