You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@kudu.apache.org by "Adar Dembo (Code Review)" <ge...@cloudera.org> on 2019/05/15 22:55:05 UTC

[kudu-CR] external mini cluster: raise start process timeout to 60s 

Hello Will Berkeley,

I'd like you to do a code review. Please visit

    http://gerrit.cloudera.org:8080/13348

to review the following change.


Change subject: external_mini_cluster: raise start process timeout to 60s
......................................................................

external_mini_cluster: raise start process timeout to 60s

After commit 7a6df6aa1 changed SystemNtp::Init to wait using ntp_adjtime(),
a few test failures showed up with the following:

  14:01:46.947 [DEBUG - main] (MiniKuduCluster.java:165) Response: error {
    code: TIMED_OUT
    message: "failed to start masters: Unable to start Master at index 0: Timed out after 30.000ss waiting for process (/tmp/dist-test-taskOjnAIG/build/tsan/bin/kudu-master) to write info file (/tmp/dist-test-taskOjnAIG/test-tmp/mini-kudu-cluster9136683209203726578/master-0/data/info.pb)"
  }

This was because the kernel's clock lost synchronization for more than 30s,
at which point the minicluster gave up on the process even though
SystemNtp::Init was prepared to keep waiting.

Here's a bandaid: extend the minicluster's process starting timeout to 60s,
so that it matches the default value of --ntp_initial_sync_wait_secs. We
could tightly couple the two values, but I'm not sure that's worth doing.

Change-Id: Ib4172e5b45a325eab9c8387bcf25c106d31b3328
---
M src/kudu/mini-cluster/external_mini_cluster.cc
M src/kudu/mini-cluster/external_mini_cluster.h
2 files changed, 3 insertions(+), 3 deletions(-)



  git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/48/13348/1
-- 
To view, visit http://gerrit.cloudera.org:8080/13348
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: Ib4172e5b45a325eab9c8387bcf25c106d31b3328
Gerrit-Change-Number: 13348
Gerrit-PatchSet: 1
Gerrit-Owner: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Will Berkeley <wd...@gmail.com>

[kudu-CR] external mini cluster: raise start process timeout to 60s

Posted by "Adar Dembo (Code Review)" <ge...@cloudera.org>.
Adar Dembo has removed Kudu Jenkins from this change.  ( http://gerrit.cloudera.org:8080/13348 )

Change subject: external_mini_cluster: raise start process timeout to 60s
......................................................................


Removed reviewer Kudu Jenkins with the following votes:

* Verified-1 by Kudu Jenkins (120)
-- 
To view, visit http://gerrit.cloudera.org:8080/13348
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: deleteReviewer
Gerrit-Change-Id: Ib4172e5b45a325eab9c8387bcf25c106d31b3328
Gerrit-Change-Number: 13348
Gerrit-PatchSet: 2
Gerrit-Owner: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Will Berkeley <wd...@gmail.com>

[kudu-CR] external mini cluster: raise start process timeout to 60s

Posted by "Will Berkeley (Code Review)" <ge...@cloudera.org>.
Will Berkeley has posted comments on this change. ( http://gerrit.cloudera.org:8080/13348 )

Change subject: external_mini_cluster: raise start process timeout to 60s
......................................................................


Patch Set 2: Code-Review+2


-- 
To view, visit http://gerrit.cloudera.org:8080/13348
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib4172e5b45a325eab9c8387bcf25c106d31b3328
Gerrit-Change-Number: 13348
Gerrit-PatchSet: 2
Gerrit-Owner: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Will Berkeley <wd...@gmail.com>
Gerrit-Comment-Date: Fri, 17 May 2019 22:06:31 +0000
Gerrit-HasComments: No

[kudu-CR] external mini cluster: raise start process timeout to 60s

Posted by "Adar Dembo (Code Review)" <ge...@cloudera.org>.
Adar Dembo has posted comments on this change. ( http://gerrit.cloudera.org:8080/13348 )

Change subject: external_mini_cluster: raise start process timeout to 60s
......................................................................


Patch Set 2: Verified+1

Overriding Jenkins, unrelated test failure in a flume test.


-- 
To view, visit http://gerrit.cloudera.org:8080/13348
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib4172e5b45a325eab9c8387bcf25c106d31b3328
Gerrit-Change-Number: 13348
Gerrit-PatchSet: 2
Gerrit-Owner: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Will Berkeley <wd...@gmail.com>
Gerrit-Comment-Date: Thu, 16 May 2019 00:03:14 +0000
Gerrit-HasComments: No

[kudu-CR] external mini cluster: raise start process timeout to 60s

Posted by "Adar Dembo (Code Review)" <ge...@cloudera.org>.
Adar Dembo has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/13348 )

Change subject: external_mini_cluster: raise start process timeout to 60s
......................................................................

external_mini_cluster: raise start process timeout to 60s

After commit 7a6df6aa1 changed SystemNtp::Init to wait using ntp_adjtime(),
a few test failures showed up with the following:

  14:01:46.947 [DEBUG - main] (MiniKuduCluster.java:165) Response: error {
    code: TIMED_OUT
    message: "failed to start masters: Unable to start Master at index 0: Timed out after 30.000ss waiting for process (/tmp/dist-test-taskOjnAIG/build/tsan/bin/kudu-master) to write info file (/tmp/dist-test-taskOjnAIG/test-tmp/mini-kudu-cluster9136683209203726578/master-0/data/info.pb)"
  }

This was because the kernel's clock lost synchronization for more than 30s,
at which point the minicluster gave up on the process even though
SystemNtp::Init was prepared to keep waiting.

Here's a bandaid: extend the minicluster's process starting timeout to 60s,
so that it matches the default value of --ntp_initial_sync_wait_secs. We
could tightly couple the two values, but I'm not sure that's worth doing.

Change-Id: Ib4172e5b45a325eab9c8387bcf25c106d31b3328
Reviewed-on: http://gerrit.cloudera.org:8080/13348
Tested-by: Adar Dembo <ad...@cloudera.com>
Reviewed-by: Will Berkeley <wd...@gmail.com>
---
M src/kudu/mini-cluster/external_mini_cluster.cc
M src/kudu/mini-cluster/external_mini_cluster.h
2 files changed, 3 insertions(+), 4 deletions(-)

Approvals:
  Adar Dembo: Verified
  Will Berkeley: Looks good to me, approved

-- 
To view, visit http://gerrit.cloudera.org:8080/13348
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: Ib4172e5b45a325eab9c8387bcf25c106d31b3328
Gerrit-Change-Number: 13348
Gerrit-PatchSet: 3
Gerrit-Owner: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Will Berkeley <wd...@gmail.com>

[kudu-CR] external mini cluster: raise start process timeout to 60s

Posted by "Adar Dembo (Code Review)" <ge...@cloudera.org>.
Hello Will Berkeley, Kudu Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/13348

to look at the new patch set (#2).

Change subject: external_mini_cluster: raise start process timeout to 60s
......................................................................

external_mini_cluster: raise start process timeout to 60s

After commit 7a6df6aa1 changed SystemNtp::Init to wait using ntp_adjtime(),
a few test failures showed up with the following:

  14:01:46.947 [DEBUG - main] (MiniKuduCluster.java:165) Response: error {
    code: TIMED_OUT
    message: "failed to start masters: Unable to start Master at index 0: Timed out after 30.000ss waiting for process (/tmp/dist-test-taskOjnAIG/build/tsan/bin/kudu-master) to write info file (/tmp/dist-test-taskOjnAIG/test-tmp/mini-kudu-cluster9136683209203726578/master-0/data/info.pb)"
  }

This was because the kernel's clock lost synchronization for more than 30s,
at which point the minicluster gave up on the process even though
SystemNtp::Init was prepared to keep waiting.

Here's a bandaid: extend the minicluster's process starting timeout to 60s,
so that it matches the default value of --ntp_initial_sync_wait_secs. We
could tightly couple the two values, but I'm not sure that's worth doing.

Change-Id: Ib4172e5b45a325eab9c8387bcf25c106d31b3328
---
M src/kudu/mini-cluster/external_mini_cluster.cc
M src/kudu/mini-cluster/external_mini_cluster.h
2 files changed, 3 insertions(+), 4 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/48/13348/2
-- 
To view, visit http://gerrit.cloudera.org:8080/13348
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ib4172e5b45a325eab9c8387bcf25c106d31b3328
Gerrit-Change-Number: 13348
Gerrit-PatchSet: 2
Gerrit-Owner: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Will Berkeley <wd...@gmail.com>