You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@kudu.apache.org by "Andrew Wong (Code Review)" <ge...@cloudera.org> on 2018/07/17 18:12:50 UTC
[kudu-CR] KUDU-2507: use FBM for KuduMiniCluster
Andrew Wong has uploaded this change for review. ( http://gerrit.cloudera.org:8080/10961
Change subject: KUDU-2507: use FBM for KuduMiniCluster
......................................................................
KUDU-2507: use FBM for KuduMiniCluster
The log block manager isn't supported on some systems, even though it's
the default in some cases, i.e. on Linux machines that do not support
hole punching. This was causing many Java tests to fail in such
environments. This patch addresses the issue in Java tests by changing
the BaseKuduTest's KuduMiniCluster to use the file block manager.
Change-Id: I8c2622d7049dacf7693e7f830fa022dd6c7d7274
---
M java/kudu-client/src/test/java/org/apache/kudu/client/BaseKuduTest.java
1 file changed, 6 insertions(+), 1 deletion(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/61/10961/1
--
To view, visit http://gerrit.cloudera.org:8080/10961
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I8c2622d7049dacf7693e7f830fa022dd6c7d7274
Gerrit-Change-Number: 10961
Gerrit-PatchSet: 1
Gerrit-Owner: Andrew Wong <aw...@cloudera.com>
[kudu-CR] KUDU-2507: use FBM for KuduMiniCluster
Posted by "Andrew Wong (Code Review)" <ge...@cloudera.org>.
Andrew Wong has posted comments on this change. ( http://gerrit.cloudera.org:8080/10961 )
Change subject: KUDU-2507: use FBM for KuduMiniCluster
......................................................................
Patch Set 1:
I'm going to test this on one such environment.
--
To view, visit http://gerrit.cloudera.org:8080/10961
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8c2622d7049dacf7693e7f830fa022dd6c7d7274
Gerrit-Change-Number: 10961
Gerrit-PatchSet: 1
Gerrit-Owner: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Comment-Date: Tue, 17 Jul 2018 18:15:41 +0000
Gerrit-HasComments: No
[kudu-CR] KUDU-2507: use FBM for KuduMiniCluster
Posted by "Andrew Wong (Code Review)" <ge...@cloudera.org>.
Andrew Wong has posted comments on this change. ( http://gerrit.cloudera.org:8080/10961 )
Change subject: KUDU-2507: use FBM for KuduMiniCluster
......................................................................
Patch Set 1:
> Patch Set 1:
>
> > I agree with the sentiment, but at the same time, the point of Java tests isn't to test against the log block manager, but to test against some Kudu cluster.
>
> That's true. But, I'm afraid if we eventually make the minicluster more consumable by external projects, people may write some integration tests that actually insert non-trivial amounts of data into it (eg to test a query engine) at which point the scalability/perf issues of the FBM may actually be problematic.
Ah, fair point.
Adar actually helped get to the bottom of the underlying reason we were seeing these failures, which is that the KuduMiniCluster isn't honoring the TEST_TMPDIR environment variable, as we do in kudu_tools_test.cc. As such, the KMCs were being directed to some /tmp directory instead of the expected drive, which should have been one that supports hole punching. I pointed Grant at the issue, since he's familiar with the KMCs and the general area.
--
To view, visit http://gerrit.cloudera.org:8080/10961
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8c2622d7049dacf7693e7f830fa022dd6c7d7274
Gerrit-Change-Number: 10961
Gerrit-PatchSet: 1
Gerrit-Owner: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Comment-Date: Tue, 17 Jul 2018 20:23:09 +0000
Gerrit-HasComments: No
[kudu-CR] KUDU-2507: use FBM for KuduMiniCluster
Posted by "Todd Lipcon (Code Review)" <ge...@cloudera.org>.
Todd Lipcon has posted comments on this change. ( http://gerrit.cloudera.org:8080/10961 )
Change subject: KUDU-2507: use FBM for KuduMiniCluster
......................................................................
Patch Set 1:
> I agree with the sentiment, but at the same time, the point of Java tests isn't to test against the log block manager, but to test against some Kudu cluster.
That's true. But, I'm afraid if we eventually make the minicluster more consumable by external projects, people may write some integration tests that actually insert non-trivial amounts of data into it (eg to test a query engine) at which point the scalability/perf issues of the FBM may actually be problematic.
--
To view, visit http://gerrit.cloudera.org:8080/10961
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8c2622d7049dacf7693e7f830fa022dd6c7d7274
Gerrit-Change-Number: 10961
Gerrit-PatchSet: 1
Gerrit-Owner: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Comment-Date: Tue, 17 Jul 2018 18:53:03 +0000
Gerrit-HasComments: No
[kudu-CR] KUDU-2507: use FBM for KuduMiniCluster
Posted by "Grant Henke (Code Review)" <ge...@cloudera.org>.
Grant Henke has posted comments on this change. ( http://gerrit.cloudera.org:8080/10961 )
Change subject: KUDU-2507: use FBM for KuduMiniCluster
......................................................................
Patch Set 1:
I don't think we need this now that a TEST_TMPDIR fix is in https://github.com/apache/kudu/commits/c2cecb0
--
To view, visit http://gerrit.cloudera.org:8080/10961
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8c2622d7049dacf7693e7f830fa022dd6c7d7274
Gerrit-Change-Number: 10961
Gerrit-PatchSet: 1
Gerrit-Owner: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Comment-Date: Mon, 30 Jul 2018 20:09:55 +0000
Gerrit-HasComments: No
[kudu-CR] KUDU-2507: use FBM for KuduMiniCluster
Posted by "Andrew Wong (Code Review)" <ge...@cloudera.org>.
Andrew Wong has posted comments on this change. ( http://gerrit.cloudera.org:8080/10961 )
Change subject: KUDU-2507: use FBM for KuduMiniCluster
......................................................................
Patch Set 1:
> Patch Set 1:
>
> hrm, I'm not sure I agree with this change. Shouldn't we use our most common supported configuration for our tests?
>
> Maybe we are better off with some sort of Java system property that users can use if they want to be able to run the Java tests some unsupported environment?
I agree with the sentiment, but at the same time, the point of Java tests isn't to test against the log block manager, but to test against some Kudu cluster.
A system property sounds fine too though.
--
To view, visit http://gerrit.cloudera.org:8080/10961
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8c2622d7049dacf7693e7f830fa022dd6c7d7274
Gerrit-Change-Number: 10961
Gerrit-PatchSet: 1
Gerrit-Owner: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Comment-Date: Tue, 17 Jul 2018 18:43:41 +0000
Gerrit-HasComments: No
[kudu-CR] KUDU-2507: use FBM for KuduMiniCluster
Posted by "Todd Lipcon (Code Review)" <ge...@cloudera.org>.
Todd Lipcon has posted comments on this change. ( http://gerrit.cloudera.org:8080/10961 )
Change subject: KUDU-2507: use FBM for KuduMiniCluster
......................................................................
Patch Set 1:
hrm, I'm not sure I agree with this change. Shouldn't we use our most common supported configuration for our tests?
Maybe we are better off with some sort of Java system property that users can use if they want to be able to run the Java tests some unsupported environment?
--
To view, visit http://gerrit.cloudera.org:8080/10961
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8c2622d7049dacf7693e7f830fa022dd6c7d7274
Gerrit-Change-Number: 10961
Gerrit-PatchSet: 1
Gerrit-Owner: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Comment-Date: Tue, 17 Jul 2018 18:31:45 +0000
Gerrit-HasComments: No
[kudu-CR] KUDU-2507: use FBM for KuduMiniCluster
Posted by "Andrew Wong (Code Review)" <ge...@cloudera.org>.
Andrew Wong has abandoned this change. ( http://gerrit.cloudera.org:8080/10961 )
Change subject: KUDU-2507: use FBM for KuduMiniCluster
......................................................................
Abandoned
Issue fixed via other patch
--
To view, visit http://gerrit.cloudera.org:8080/10961
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: abandon
Gerrit-Change-Id: I8c2622d7049dacf7693e7f830fa022dd6c7d7274
Gerrit-Change-Number: 10961
Gerrit-PatchSet: 1
Gerrit-Owner: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>