You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Lars Volker (Code Review)" <ge...@cloudera.org> on 2017/01/10 20:01:04 UTC
[Impala-ASF-CR] IMPALA-4722: Disable log caching in test scratch disk
Lars Volker has uploaded a new change for review.
http://gerrit.cloudera.org:8080/5669
Change subject: IMPALA-4722: Disable log caching in test_scratch_disk
......................................................................
IMPALA-4722: Disable log caching in test_scratch_disk
test_scratch_disk fails sporadically when trying to assert the presence
of log messages. This is probably caused by log caching, since after
such failures the log files do contains the lines in question.
I manually tested this by running the tests repeatedly for 2 days (10k
runs).
To make future diagnosis of similar problems easier, this change also
adds more output to assert_impalad_log_contains().
Change-Id: I9f21284338ee7b4374aca249b6556282b0148389
---
M tests/common/custom_cluster_test_suite.py
M tests/custom_cluster/test_scratch_disk.py
2 files changed, 9 insertions(+), 6 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/69/5669/1
--
To view, visit http://gerrit.cloudera.org:8080/5669
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: I9f21284338ee7b4374aca249b6556282b0148389
Gerrit-PatchSet: 1
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Lars Volker <lv...@cloudera.com>
[Impala-ASF-CR] IMPALA-4722: Disable log caching in test scratch disk
Posted by "Dan Hecht (Code Review)" <ge...@cloudera.org>.
Dan Hecht has posted comments on this change.
Change subject: IMPALA-4722: Disable log caching in test_scratch_disk
......................................................................
Patch Set 2:
> > Okay
>
> Do you think this needs more explanation in the breakpad tests?
I guess you already say "running processes" in that comment you added, so I think that's enough.
--
To view, visit http://gerrit.cloudera.org:8080/5669
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: I9f21284338ee7b4374aca249b6556282b0148389
Gerrit-PatchSet: 2
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Lars Volker <lv...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Lars Volker <lv...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-4722: Disable log caching in test scratch disk
Posted by "Lars Volker (Code Review)" <ge...@cloudera.org>.
Lars Volker has posted comments on this change.
Change subject: IMPALA-4722: Disable log caching in test_scratch_disk
......................................................................
Patch Set 2:
> Okay
Do you think this needs more explanation in the breakpad tests?
--
To view, visit http://gerrit.cloudera.org:8080/5669
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: I9f21284338ee7b4374aca249b6556282b0148389
Gerrit-PatchSet: 2
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Lars Volker <lv...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Lars Volker <lv...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-4722: Disable log caching in test scratch disk
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has submitted this change and it was merged.
Change subject: IMPALA-4722: Disable log caching in test_scratch_disk
......................................................................
IMPALA-4722: Disable log caching in test_scratch_disk
test_scratch_disk fails sporadically when trying to assert the presence
of log messages. This is probably caused by log caching, since after
such failures the log files do contains the lines in question.
I manually tested this by running the tests repeatedly for 2 days (10k
runs).
To make future diagnosis of similar problems easier, this change also
adds more output to assert_impalad_log_contains().
Change-Id: I9f21284338ee7b4374aca249b6556282b0148389
Reviewed-on: http://gerrit.cloudera.org:8080/5669
Reviewed-by: Tim Armstrong <ta...@cloudera.com>
Tested-by: Impala Public Jenkins
---
M tests/common/custom_cluster_test_suite.py
M tests/custom_cluster/test_scratch_disk.py
2 files changed, 15 insertions(+), 9 deletions(-)
Approvals:
Impala Public Jenkins: Verified
Tim Armstrong: Looks good to me, approved
--
To view, visit http://gerrit.cloudera.org:8080/5669
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: I9f21284338ee7b4374aca249b6556282b0148389
Gerrit-PatchSet: 3
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Lars Volker <lv...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Lars Volker <lv...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
[Impala-ASF-CR] IMPALA-4722: Disable log caching in test scratch disk
Posted by "Lars Volker (Code Review)" <ge...@cloudera.org>.
Lars Volker has posted comments on this change.
Change subject: IMPALA-4722: Disable log caching in test_scratch_disk
......................................................................
Patch Set 1:
(2 comments)
Thanks for the review, please see PS2.
http://gerrit.cloudera.org:8080/#/c/5669/1/tests/common/custom_cluster_test_suite.py
File tests/common/custom_cluster_test_suite.py:
PS1, Line 142: \
> nit: the line continuation shouldn't be necessary inside parentheses. Might
Done
http://gerrit.cloudera.org:8080/#/c/5669/1/tests/custom_cluster/test_scratch_disk.py
File tests/custom_cluster/test_scratch_disk.py:
PS1, Line 86: -logbuflevel=-1
> We should document the importance of this somewhere. Maybe in the comment f
Done
--
To view, visit http://gerrit.cloudera.org:8080/5669
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: I9f21284338ee7b4374aca249b6556282b0148389
Gerrit-PatchSet: 1
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Lars Volker <lv...@cloudera.com>
Gerrit-Reviewer: Lars Volker <lv...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-4722: Disable log caching in test scratch disk
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change.
Change subject: IMPALA-4722: Disable log caching in test_scratch_disk
......................................................................
Patch Set 2:
Build started: http://jenkins.impala.io:8080/job/gerrit-verify-dryrun/179/
--
To view, visit http://gerrit.cloudera.org:8080/5669
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: I9f21284338ee7b4374aca249b6556282b0148389
Gerrit-PatchSet: 2
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Lars Volker <lv...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Lars Volker <lv...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-4722: Disable log caching in test scratch disk
Posted by "Tim Armstrong (Code Review)" <ge...@cloudera.org>.
Tim Armstrong has posted comments on this change.
Change subject: IMPALA-4722: Disable log caching in test_scratch_disk
......................................................................
Patch Set 2: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/5669
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: I9f21284338ee7b4374aca249b6556282b0148389
Gerrit-PatchSet: 2
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Lars Volker <lv...@cloudera.com>
Gerrit-Reviewer: Lars Volker <lv...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-4722: Disable log caching in test scratch disk
Posted by "Dan Hecht (Code Review)" <ge...@cloudera.org>.
Dan Hecht has posted comments on this change.
Change subject: IMPALA-4722: Disable log caching in test_scratch_disk
......................................................................
Patch Set 2:
Should we add this to tests/custom_cluster/test_breakpad.py where we also look for logs?
--
To view, visit http://gerrit.cloudera.org:8080/5669
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: I9f21284338ee7b4374aca249b6556282b0148389
Gerrit-PatchSet: 2
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Lars Volker <lv...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Lars Volker <lv...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-4722: Disable log caching in test scratch disk
Posted by "Lars Volker (Code Review)" <ge...@cloudera.org>.
Lars Volker has uploaded a new patch set (#2).
Change subject: IMPALA-4722: Disable log caching in test_scratch_disk
......................................................................
IMPALA-4722: Disable log caching in test_scratch_disk
test_scratch_disk fails sporadically when trying to assert the presence
of log messages. This is probably caused by log caching, since after
such failures the log files do contains the lines in question.
I manually tested this by running the tests repeatedly for 2 days (10k
runs).
To make future diagnosis of similar problems easier, this change also
adds more output to assert_impalad_log_contains().
Change-Id: I9f21284338ee7b4374aca249b6556282b0148389
---
M tests/common/custom_cluster_test_suite.py
M tests/custom_cluster/test_scratch_disk.py
2 files changed, 15 insertions(+), 9 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/69/5669/2
--
To view, visit http://gerrit.cloudera.org:8080/5669
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I9f21284338ee7b4374aca249b6556282b0148389
Gerrit-PatchSet: 2
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Lars Volker <lv...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
[Impala-ASF-CR] IMPALA-4722: Disable log caching in test scratch disk
Posted by "Tim Armstrong (Code Review)" <ge...@cloudera.org>.
Tim Armstrong has posted comments on this change.
Change subject: IMPALA-4722: Disable log caching in test_scratch_disk
......................................................................
Patch Set 1:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/5669/1/tests/common/custom_cluster_test_suite.py
File tests/common/custom_cluster_test_suite.py:
PS1, Line 142: \
nit: the line continuation shouldn't be necessary inside parentheses. Might as well fix it while we're here.
http://gerrit.cloudera.org:8080/#/c/5669/1/tests/custom_cluster/test_scratch_disk.py
File tests/custom_cluster/test_scratch_disk.py:
PS1, Line 86: -logbuflevel=-1
We should document the importance of this somewhere. Maybe in the comment for assert_impalad_log_contains()?
--
To view, visit http://gerrit.cloudera.org:8080/5669
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: I9f21284338ee7b4374aca249b6556282b0148389
Gerrit-PatchSet: 1
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Lars Volker <lv...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-4722: Disable log caching in test scratch disk
Posted by "Lars Volker (Code Review)" <ge...@cloudera.org>.
Lars Volker has posted comments on this change.
Change subject: IMPALA-4722: Disable log caching in test_scratch_disk
......................................................................
Patch Set 2:
> Should we add this to tests/custom_cluster/test_breakpad.py where
> we also look for logs?
I don't think we need this for the breakpad tests. The tests that actually kill the cluster trigger a flush of the log files before the process exits. Also, Breakpad forks to write a minidump and then prints its log messages to stdout and stderr before exiting, which are redirected into the log files. Therefore it seems that the messages show up in the logs, once the forks have written the minidumps and terminated (for which we wait in the test).
--
To view, visit http://gerrit.cloudera.org:8080/5669
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: I9f21284338ee7b4374aca249b6556282b0148389
Gerrit-PatchSet: 2
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Lars Volker <lv...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Lars Volker <lv...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-4722: Disable log caching in test scratch disk
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change.
Change subject: IMPALA-4722: Disable log caching in test_scratch_disk
......................................................................
Patch Set 2: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/5669
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: I9f21284338ee7b4374aca249b6556282b0148389
Gerrit-PatchSet: 2
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Lars Volker <lv...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Lars Volker <lv...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-4722: Disable log caching in test scratch disk
Posted by "Dan Hecht (Code Review)" <ge...@cloudera.org>.
Dan Hecht has posted comments on this change.
Change subject: IMPALA-4722: Disable log caching in test_scratch_disk
......................................................................
Patch Set 2:
Okay
--
To view, visit http://gerrit.cloudera.org:8080/5669
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: I9f21284338ee7b4374aca249b6556282b0148389
Gerrit-PatchSet: 2
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Lars Volker <lv...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Lars Volker <lv...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-HasComments: No