You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@kudu.apache.org by "Hao Hao (Code Review)" <ge...@cloudera.org> on 2017/04/27 07:43:03 UTC
[kudu-CR] mapreduce: add support for fault tolerant scanner
Hao Hao has uploaded a new change for review.
http://gerrit.cloudera.org:8080/6745
Change subject: mapreduce: add support for fault tolerant scanner
......................................................................
mapreduce: add support for fault tolerant scanner
This adds support to use fault tolerant scanner in mapreduce job.
By default non fault tolerant scanner is used. To turn on fault
tolerant scanner, use job config:
'kudu.mapreduce.input.fault.tolerant.scan'.
Change-Id: Ibc39472e2733bab4e00e73658f8a7619153bd7c6
---
M java/kudu-mapreduce/src/main/java/org/apache/kudu/mapreduce/KuduTableInputFormat.java
M java/kudu-mapreduce/src/main/java/org/apache/kudu/mapreduce/KuduTableMapReduceUtil.java
M java/kudu-mapreduce/src/test/java/org/apache/kudu/mapreduce/ITInputFormatJob.java
3 files changed, 23 insertions(+), 2 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/45/6745/1
--
To view, visit http://gerrit.cloudera.org:8080/6745
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: Ibc39472e2733bab4e00e73658f8a7619153bd7c6
Gerrit-PatchSet: 1
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Hao Hao <ha...@cloudera.com>
[kudu-CR] mapreduce: add support for fault tolerant scanner
Posted by "Jean-Daniel Cryans (Code Review)" <ge...@cloudera.org>.
Jean-Daniel Cryans has submitted this change and it was merged.
Change subject: mapreduce: add support for fault tolerant scanner
......................................................................
mapreduce: add support for fault tolerant scanner
This adds support to use fault tolerant scanner in mapreduce job.
By default non fault tolerant scanner is used. To turn on fault
tolerant scanner, use job config:
'kudu.mapreduce.input.fault.tolerant.scan'.
Change-Id: Ibc39472e2733bab4e00e73658f8a7619153bd7c6
Reviewed-on: http://gerrit.cloudera.org:8080/6745
Reviewed-by: Jean-Daniel Cryans <jd...@apache.org>
Tested-by: Kudu Jenkins
---
M java/kudu-mapreduce/src/main/java/org/apache/kudu/mapreduce/KuduTableInputFormat.java
M java/kudu-mapreduce/src/main/java/org/apache/kudu/mapreduce/KuduTableMapReduceUtil.java
M java/kudu-mapreduce/src/test/java/org/apache/kudu/mapreduce/ITInputFormatJob.java
3 files changed, 25 insertions(+), 3 deletions(-)
Approvals:
Jean-Daniel Cryans: Looks good to me, approved
Kudu Jenkins: Verified
--
To view, visit http://gerrit.cloudera.org:8080/6745
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: Ibc39472e2733bab4e00e73658f8a7619153bd7c6
Gerrit-PatchSet: 7
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Dan Burkert <da...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Jean-Daniel Cryans <jd...@apache.org>
Gerrit-Reviewer: Kudu Jenkins
[kudu-CR] mapreduce: add support for fault tolerant scanner
Posted by "Jean-Daniel Cryans (Code Review)" <ge...@cloudera.org>.
Jean-Daniel Cryans has posted comments on this change.
Change subject: mapreduce: add support for fault tolerant scanner
......................................................................
Patch Set 2:
The test failure is related to the fault tolerant scanner patch:
01:09:34.954 [INFO - kudu-tserver:64040] (MiniKuduCluster.java:568) W0428 01:09:34.954133 17141 connection.cc:380] Connection torn down before Call kudu.tserver.TabletServerService.Write from 127.0.0.1:35554 (ReqId={client: 21809ecd299e454fa1a5ce2dcc9d4faf, seq_no=16347, attempt_no=1}) could send its response
01:09:34.954 [ERROR - Thread-10] (ITClient.java:134) Got error while row counting
org.apache.kudu.client.RecoverableException: [Peer 7605e33bc78748a08bd01815dd8aff56] Connection disconnected
at org.apache.kudu.client.TabletClient.cleanup(TabletClient.java:657)
at org.apache.kudu.client.TabletClient.channelDisconnected(TabletClient.java:610)
at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:102)
at org.apache.kudu.client.TabletClient.handleUpstream(TabletClient.java:603)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
What this means is that the connection got closed before the scan was sent. I think this should be a retryable exception even for non-fault tolerant scanners? Maybe it would make more sense to make ITClient use fault tolerant scanners too.
--
To view, visit http://gerrit.cloudera.org:8080/6745
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibc39472e2733bab4e00e73658f8a7619153bd7c6
Gerrit-PatchSet: 2
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Jean-Daniel Cryans <jd...@apache.org>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-HasComments: No
[kudu-CR] mapreduce: add support for fault tolerant scanner
Posted by "Hao Hao (Code Review)" <ge...@cloudera.org>.
Hao Hao has posted comments on this change.
Change subject: mapreduce: add support for fault tolerant scanner
......................................................................
Patch Set 2:
Agree. I just uploaded a patch to improve ITClient test.
--
To view, visit http://gerrit.cloudera.org:8080/6745
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibc39472e2733bab4e00e73658f8a7619153bd7c6
Gerrit-PatchSet: 2
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Jean-Daniel Cryans <jd...@apache.org>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-HasComments: No
[kudu-CR] mapreduce: add support for fault tolerant scanner
Posted by "Hao Hao (Code Review)" <ge...@cloudera.org>.
Hao Hao has posted comments on this change.
Change subject: mapreduce: add support for fault tolerant scanner
......................................................................
Patch Set 6:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/6745/5/java/kudu-mapreduce/src/main/java/org/apache/kudu/mapreduce/KuduTableInputFormat.java
File java/kudu-mapreduce/src/main/java/org/apache/kudu/mapreduce/KuduTableInputFormat.java:
Line 91: /**
> nit: if you do multi-line, start writing on the second line.
Done
http://gerrit.cloudera.org:8080/#/c/6745/5/java/kudu-mapreduce/src/main/java/org/apache/kudu/mapreduce/KuduTableMapReduceUtil.java
File java/kudu-mapreduce/src/main/java/org/apache/kudu/mapreduce/KuduTableMapReduceUtil.java:
PS5, Line 216:
> nit: no period here
Done
--
To view, visit http://gerrit.cloudera.org:8080/6745
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibc39472e2733bab4e00e73658f8a7619153bd7c6
Gerrit-PatchSet: 6
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Dan Burkert <da...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Jean-Daniel Cryans <jd...@apache.org>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-HasComments: Yes
[kudu-CR] mapreduce: add support for fault tolerant scanner
Posted by "Jean-Daniel Cryans (Code Review)" <ge...@cloudera.org>.
Jean-Daniel Cryans has posted comments on this change.
Change subject: mapreduce: add support for fault tolerant scanner
......................................................................
Patch Set 5:
(2 comments)
some nits and then we're good to go
http://gerrit.cloudera.org:8080/#/c/6745/5/java/kudu-mapreduce/src/main/java/org/apache/kudu/mapreduce/KuduTableInputFormat.java
File java/kudu-mapreduce/src/main/java/org/apache/kudu/mapreduce/KuduTableInputFormat.java:
Line 91: /** Job parameter that specifies if the scanner should be fault tolerant
nit: if you do multi-line, start writing on the second line.
http://gerrit.cloudera.org:8080/#/c/6745/5/java/kudu-mapreduce/src/main/java/org/apache/kudu/mapreduce/KuduTableMapReduceUtil.java
File java/kudu-mapreduce/src/main/java/org/apache/kudu/mapreduce/KuduTableMapReduceUtil.java:
PS5, Line 216: .
nit: no period here
--
To view, visit http://gerrit.cloudera.org:8080/6745
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibc39472e2733bab4e00e73658f8a7619153bd7c6
Gerrit-PatchSet: 5
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Jean-Daniel Cryans <jd...@apache.org>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-HasComments: Yes
[kudu-CR] mapreduce: add support for fault tolerant scanner
Posted by "Jean-Daniel Cryans (Code Review)" <ge...@cloudera.org>.
Jean-Daniel Cryans has posted comments on this change.
Change subject: mapreduce: add support for fault tolerant scanner
......................................................................
Patch Set 6: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/6745
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibc39472e2733bab4e00e73658f8a7619153bd7c6
Gerrit-PatchSet: 6
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Dan Burkert <da...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Jean-Daniel Cryans <jd...@apache.org>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-HasComments: No
[kudu-CR] mapreduce: add support for fault tolerant scanner
Posted by "Hao Hao (Code Review)" <ge...@cloudera.org>.
Hello Kudu Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/6745
to look at the new patch set (#6).
Change subject: mapreduce: add support for fault tolerant scanner
......................................................................
mapreduce: add support for fault tolerant scanner
This adds support to use fault tolerant scanner in mapreduce job.
By default non fault tolerant scanner is used. To turn on fault
tolerant scanner, use job config:
'kudu.mapreduce.input.fault.tolerant.scan'.
Change-Id: Ibc39472e2733bab4e00e73658f8a7619153bd7c6
---
M java/kudu-mapreduce/src/main/java/org/apache/kudu/mapreduce/KuduTableInputFormat.java
M java/kudu-mapreduce/src/main/java/org/apache/kudu/mapreduce/KuduTableMapReduceUtil.java
M java/kudu-mapreduce/src/test/java/org/apache/kudu/mapreduce/ITInputFormatJob.java
3 files changed, 25 insertions(+), 3 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/45/6745/6
--
To view, visit http://gerrit.cloudera.org:8080/6745
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ibc39472e2733bab4e00e73658f8a7619153bd7c6
Gerrit-PatchSet: 6
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Jean-Daniel Cryans <jd...@apache.org>
Gerrit-Reviewer: Kudu Jenkins