You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by Praveen R <pr...@sigmoidanalytics.com> on 2015/02/02 07:32:26 UTC
Re: Review Request 30262: PIG-4393 : Add stats and error reporting
for Spark
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30262/#review70542
-----------------------------------------------------------
@Mohit: Did you have any chance to run the whole unit tests, would like to know if we are having any additional unit tests breaking than current 129 failures. I see you are having 2 new tests passing with the patch, but it would be easy for the merge if the patch doesn't cause any additional failures.
- Praveen R
On Jan. 30, 2015, 1:39 p.m., Mohit Sabharwal wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/30262/
> -----------------------------------------------------------
>
> (Updated Jan. 30, 2015, 1:39 p.m.)
>
>
> Review request for pig, liyun zhang and Praveen R.
>
>
> Bugs: PIG-4393
> https://issues.apache.org/jira/browse/PIG-4393
>
>
> Repository: pig-git
>
>
> Description
> -------
>
> PIG-4393 : Add stats and error reporting for Spark
>
> After Pig submits a job to Spark cluster, we need to report job progress, spark specific stats and any error logs back to the user.
>
> (1) It adds getting back status of basic success/failure for each Spark job.
> (2) It adds logging of Spark specific stats in log file. Essentially, registers a job metrics listener with spark context and collects spark task level metrics and aggregates.
> (3) It also re-factors code to correctly populate PigStats, which is used by most unit tests. This should fix a bunch of unit tests.
>
> TODO items in a follow-up patch:
> - Add #records to OutputStats for each job.
> - Though StatsReportListener prints spark job progress in the logs, we also probably need to implement PigProgressNotificationListener for spark.
>
>
> Diffs
> -----
>
> src/org/apache/pig/backend/hadoop/executionengine/spark/JobMetricsListener.java PRE-CREATION
> src/org/apache/pig/backend/hadoop/executionengine/spark/SparkExecutionEngine.java db152b5003ce6e79b001b2624010b91cc0f921d8
> src/org/apache/pig/backend/hadoop/executionengine/spark/SparkLauncher.java b15994d525250bbb26f7b7126dae619b9da363c8
> src/org/apache/pig/tools/pigstats/SparkStats.java fd45dd4f0be415dd48d9fb7381c57c861bbbf7ce
> src/org/apache/pig/tools/pigstats/spark/SparkJobStats.java PRE-CREATION
> src/org/apache/pig/tools/pigstats/spark/SparkPigStats.java PRE-CREATION
> src/org/apache/pig/tools/pigstats/spark/SparkStatsUtil.java PRE-CREATION
>
> Diff: https://reviews.apache.org/r/30262/diff/
>
>
> Testing
> -------
>
> Tested with unit tests, at least some unit tests that were failing
> eariler due to lack of stats, like TestToolsPigServer and TestSplitStore
> now pass.
>
>
> Example of Spark Job metrics that appear in logs:
>
> 2015-01-29 23:06:42,520 [main] INFO org.apache.pig.tools.pigstats.spark.SparkPigStats - Spark Job [0] Metrics
> 2015-01-29 23:06:42,520 [main] INFO org.apache.pig.tools.pigstats.spark.SparkPigStats - EexcutorDeserializeTime : 74
> 2015-01-29 23:06:42,520 [main] INFO org.apache.pig.tools.pigstats.spark.SparkPigStats - ExecutorRunTime : 538
> 2015-01-29 23:06:42,520 [main] INFO org.apache.pig.tools.pigstats.spark.SparkPigStats - ResultSize : 2535
> 2015-01-29 23:06:42,520 [main] INFO org.apache.pig.tools.pigstats.spark.SparkPigStats - JvmGCTime : 0
> 2015-01-29 23:06:42,520 [main] INFO org.apache.pig.tools.pigstats.spark.SparkPigStats - ResultSerializationTime : 1
> 2015-01-29 23:06:42,520 [main] INFO org.apache.pig.tools.pigstats.spark.SparkPigStats - MemoryBytesSpilled : 0
> 2015-01-29 23:06:42,520 [main] INFO org.apache.pig.tools.pigstats.spark.SparkPigStats - DiskBytesSpilled : 0
> 2015-01-29 23:06:42,520 [main] INFO org.apache.pig.tools.pigstats.spark.SparkPigStats - RemoteBlocksFetched : 0
> 2015-01-29 23:06:42,520 [main] INFO org.apache.pig.tools.pigstats.spark.SparkPigStats - LocalBlocksFetched : 2
> 2015-01-29 23:06:42,521 [main] INFO org.apache.pig.tools.pigstats.spark.SparkPigStats - TotalBlocksFetched : 2
> 2015-01-29 23:06:42,521 [main] INFO org.apache.pig.tools.pigstats.spark.SparkPigStats - FetchWaitTime : 0
> 2015-01-29 23:06:42,521 [main] INFO org.apache.pig.tools.pigstats.spark.SparkPigStats - RemoteBytesRead : 0
> 2015-01-29 23:06:42,521 [main] INFO org.apache.pig.tools.pigstats.spark.SparkPigStats - ShuffleBytesWritten : 918
> 2015-01-29 23:06:42,521 [main] INFO org.apache.pig.tools.pigstats.spark.SparkPigStats - ShuffleWriteTime : 67000
>
>
> Thanks,
>
> Mohit Sabharwal
>
>
Re: Review Request 30262: PIG-4393 : Add stats and error reporting
for Spark
Posted by Mohit Sabharwal <mo...@cloudera.com>.
> On Feb. 2, 2015, 6:32 a.m., Praveen R wrote:
> > @Mohit: Did you have any chance to run the whole unit tests, would like to know if we are having any additional unit tests breaking than current 129 failures. I see you are having 2 new tests passing with the patch, but it would be easy for the merge if the patch doesn't cause any additional failures.
I did and compared to last Jenkins run (https://builds.apache.org/job/Pig-spark/lastCompletedBuild/testReport/), this patch fixes 2 tests (TestStoreInstances and TestToolsPigServer)
It was causing additional failures for TestStreaming, but I have now fixed that by updating the patch.
- Mohit
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30262/#review70542
-----------------------------------------------------------
On Feb. 2, 2015, 8:47 p.m., Mohit Sabharwal wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/30262/
> -----------------------------------------------------------
>
> (Updated Feb. 2, 2015, 8:47 p.m.)
>
>
> Review request for pig, liyun zhang and Praveen R.
>
>
> Bugs: PIG-4393
> https://issues.apache.org/jira/browse/PIG-4393
>
>
> Repository: pig-git
>
>
> Description
> -------
>
> PIG-4393 : Add stats and error reporting for Spark
>
> After Pig submits a job to Spark cluster, we need to report job progress, spark specific stats and any error logs back to the user.
>
> (1) It adds getting back status of basic success/failure for each Spark job.
> (2) It adds logging of Spark specific stats in log file. Essentially, registers a job metrics listener with spark context and collects spark task level metrics and aggregates.
> (3) It also re-factors code to correctly populate PigStats, which is used by most unit tests. This should fix a bunch of unit tests.
>
> TODO items in a follow-up patch:
> - Add #records to OutputStats for each job.
> - Though StatsReportListener prints spark job progress in the logs, we also probably need to implement PigProgressNotificationListener for spark.
>
>
> Diffs
> -----
>
> src/org/apache/pig/backend/hadoop/executionengine/spark/JobMetricsListener.java PRE-CREATION
> src/org/apache/pig/backend/hadoop/executionengine/spark/SparkExecutionEngine.java db152b5
> src/org/apache/pig/backend/hadoop/executionengine/spark/SparkLauncher.java b15994d
> src/org/apache/pig/tools/pigstats/SparkStats.java fd45dd4
> src/org/apache/pig/tools/pigstats/spark/SparkJobStats.java PRE-CREATION
> src/org/apache/pig/tools/pigstats/spark/SparkPigStats.java PRE-CREATION
> src/org/apache/pig/tools/pigstats/spark/SparkStatsUtil.java PRE-CREATION
>
> Diff: https://reviews.apache.org/r/30262/diff/
>
>
> Testing
> -------
>
> Tested with unit tests:
> Compared to last Jenkins unit test run for the branch (baseline), two unit tests TestToolsPigServer and TestStoreInstances are fixed.
> Baseline: https://builds.apache.org/job/Pig-spark/lastCompletedBuild/testReport/
>
>
> Example of Spark Job metrics that appear in logs:
>
> 2015-01-29 23:06:42,520 [main] INFO org.apache.pig.tools.pigstats.spark.SparkPigStats - Spark Job [0] Metrics
> 2015-01-29 23:06:42,520 [main] INFO org.apache.pig.tools.pigstats.spark.SparkPigStats - EexcutorDeserializeTime : 74
> 2015-01-29 23:06:42,520 [main] INFO org.apache.pig.tools.pigstats.spark.SparkPigStats - ExecutorRunTime : 538
> 2015-01-29 23:06:42,520 [main] INFO org.apache.pig.tools.pigstats.spark.SparkPigStats - ResultSize : 2535
> 2015-01-29 23:06:42,520 [main] INFO org.apache.pig.tools.pigstats.spark.SparkPigStats - JvmGCTime : 0
> 2015-01-29 23:06:42,520 [main] INFO org.apache.pig.tools.pigstats.spark.SparkPigStats - ResultSerializationTime : 1
> 2015-01-29 23:06:42,520 [main] INFO org.apache.pig.tools.pigstats.spark.SparkPigStats - MemoryBytesSpilled : 0
> 2015-01-29 23:06:42,520 [main] INFO org.apache.pig.tools.pigstats.spark.SparkPigStats - DiskBytesSpilled : 0
> 2015-01-29 23:06:42,520 [main] INFO org.apache.pig.tools.pigstats.spark.SparkPigStats - RemoteBlocksFetched : 0
> 2015-01-29 23:06:42,520 [main] INFO org.apache.pig.tools.pigstats.spark.SparkPigStats - LocalBlocksFetched : 2
> 2015-01-29 23:06:42,521 [main] INFO org.apache.pig.tools.pigstats.spark.SparkPigStats - TotalBlocksFetched : 2
> 2015-01-29 23:06:42,521 [main] INFO org.apache.pig.tools.pigstats.spark.SparkPigStats - FetchWaitTime : 0
> 2015-01-29 23:06:42,521 [main] INFO org.apache.pig.tools.pigstats.spark.SparkPigStats - RemoteBytesRead : 0
> 2015-01-29 23:06:42,521 [main] INFO org.apache.pig.tools.pigstats.spark.SparkPigStats - ShuffleBytesWritten : 918
> 2015-01-29 23:06:42,521 [main] INFO org.apache.pig.tools.pigstats.spark.SparkPigStats - ShuffleWriteTime : 67000
>
>
> Thanks,
>
> Mohit Sabharwal
>
>