You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Fang-Yu Rao (Code Review)" <ge...@cloudera.org> on 2021/06/29 23:17:17 UTC

[Impala-ASF-CR] IMPALA-10774: Collect the output when starting Hive Metastore or HiveServer2 

Fang-Yu Rao has uploaded this change for review. ( http://gerrit.cloudera.org:8080/17642


Change subject: IMPALA-10774: Collect the output when starting Hive Metastore or HiveServer2
......................................................................

IMPALA-10774: Collect the output when starting Hive Metastore or HiveServer2

Before this patch, when we started the Hive Metastore or the HiveServer2
via run-hive-server.sh, hive-metastore.out and hive-server2.out would be
generated, respectively. The output of executing the hive command was
redirected to the corresponding files mentioned above. However, since
">" was used to redirect the output, the previously generated files
would be overwritten every time during the restart and thus some
important error message may be lost.

To collect more information regarding the restart, this patch appends
the newly produced output to hive-metastore.out and hive-server2.out
during the restart, which makes it easier for developers to troubleshoot
issues related to the restart.

Change-Id: I2efdbcf886e2d32ccf8c7eef038360884e44f216
---
M testdata/bin/run-hive-server.sh
1 file changed, 4 insertions(+), 2 deletions(-)



  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/42/17642/1
-- 
To view, visit http://gerrit.cloudera.org:8080/17642
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I2efdbcf886e2d32ccf8c7eef038360884e44f216
Gerrit-Change-Number: 17642
Gerrit-PatchSet: 1
Gerrit-Owner: Fang-Yu Rao <fa...@cloudera.com>

[Impala-ASF-CR] IMPALA-10774: Collect the output when starting Hive Metastore or HiveServer2

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17642 )

Change subject: IMPALA-10774: Collect the output when starting Hive Metastore or HiveServer2
......................................................................


Patch Set 1:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/9027/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/17642
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2efdbcf886e2d32ccf8c7eef038360884e44f216
Gerrit-Change-Number: 17642
Gerrit-PatchSet: 1
Gerrit-Owner: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Tue, 29 Jun 2021 23:41:36 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10774: Collect the output when starting Hive Metastore or HiveServer2

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/17642 )

Change subject: IMPALA-10774: Collect the output when starting Hive Metastore or HiveServer2
......................................................................

IMPALA-10774: Collect the output when starting Hive Metastore or HiveServer2

Before this patch, when we started the Hive Metastore or the HiveServer2
via run-hive-server.sh, hive-metastore.out and hive-server2.out would be
generated, respectively. The output of executing the hive command was
redirected to the corresponding files mentioned above. However, since
">" was used to redirect the output, the previously generated files
would be overwritten every time during the restart and thus some
important error message may be lost.

To collect more information regarding the restart, this patch appends
the newly produced output to hive-metastore.out and hive-server2.out
during the restart, which makes it easier for developers to troubleshoot
issues related to the restart.

Change-Id: I2efdbcf886e2d32ccf8c7eef038360884e44f216
Reviewed-on: http://gerrit.cloudera.org:8080/17642
Reviewed-by: Quanlong Huang <hu...@gmail.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
M testdata/bin/run-hive-server.sh
1 file changed, 2 insertions(+), 2 deletions(-)

Approvals:
  Quanlong Huang: Looks good to me, approved
  Impala Public Jenkins: Verified

-- 
To view, visit http://gerrit.cloudera.org:8080/17642
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I2efdbcf886e2d32ccf8c7eef038360884e44f216
Gerrit-Change-Number: 17642
Gerrit-PatchSet: 3
Gerrit-Owner: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>

[Impala-ASF-CR] IMPALA-10774: Collect the output when starting Hive Metastore or HiveServer2

Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/17642 )

Change subject: IMPALA-10774: Collect the output when starting Hive Metastore or HiveServer2
......................................................................


Patch Set 2: Code-Review+2


-- 
To view, visit http://gerrit.cloudera.org:8080/17642
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2efdbcf886e2d32ccf8c7eef038360884e44f216
Gerrit-Change-Number: 17642
Gerrit-PatchSet: 2
Gerrit-Owner: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Thu, 01 Jul 2021 06:07:19 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10774: Collect the output when starting Hive Metastore or HiveServer2

Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/17642 )

Change subject: IMPALA-10774: Collect the output when starting Hive Metastore or HiveServer2
......................................................................


Patch Set 1:

(2 comments)

http://gerrit.cloudera.org:8080/#/c/17642/1/testdata/bin/run-hive-server.sh
File testdata/bin/run-hive-server.sh:

http://gerrit.cloudera.org:8080/#/c/17642/1/testdata/bin/run-hive-server.sh@135
PS1, Line 135:       --service metastore -p $HIVE_METASTORE_PORT >> ${LOGDIR}/hive-metastore.out \
             :       2>> ${LOGDIR}/hive-metastore.err 2>&1 &
Most of these outputs don't have associated timestamps. I can only see one line has timestamp:

 2021-06-25 18:14:12: Starting Hive Metastore Server

I think it'd be better to keep them in the same file (i.e. hive-metastore.out) so we can match them. So I prefer to removing "2>> ${LOGDIR}/hive-metastore.err".

BTW, "2>&1" seems to conflicts with "2>> ${LOGDIR}/hive-metastore.err".


http://gerrit.cloudera.org:8080/#/c/17642/1/testdata/bin/run-hive-server.sh@153
PS1, Line 153:       2>> ${LOGDIR}/hive-server2.err 2>&1 &
Same comment as above.



-- 
To view, visit http://gerrit.cloudera.org:8080/17642
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2efdbcf886e2d32ccf8c7eef038360884e44f216
Gerrit-Change-Number: 17642
Gerrit-PatchSet: 1
Gerrit-Owner: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Wed, 30 Jun 2021 01:39:23 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-10774: Collect the output when starting Hive Metastore or HiveServer2

Posted by "Fang-Yu Rao (Code Review)" <ge...@cloudera.org>.
Fang-Yu Rao has posted comments on this change. ( http://gerrit.cloudera.org:8080/17642 )

Change subject: IMPALA-10774: Collect the output when starting Hive Metastore or HiveServer2
......................................................................


Patch Set 2:

(2 comments)

Hi all, I have addressed Quanlong's comments in the previous iteration. Let me know if there is any other suggestion. Thank you very much for the help!

http://gerrit.cloudera.org:8080/#/c/17642/1/testdata/bin/run-hive-server.sh
File testdata/bin/run-hive-server.sh:

http://gerrit.cloudera.org:8080/#/c/17642/1/testdata/bin/run-hive-server.sh@135
PS1, Line 135:       --service metastore -p $HIVE_METASTORE_PORT >> ${LOGDIR}/hive-metastore.out 2>&1 &
             : 
> Most of these outputs don't have associated timestamps. I can only see one 
Thanks Quanlong! I agree with you. I will remove "2>> ${LOGDIR}/hive-metastore.err" in the next iteration.

As for those 2 redirection operators, I think you are correct. According to https://stackoverflow.com/questions/20081653/two-redirection-operators-in-one-command, only the last one (the rightmost one) will be used.


http://gerrit.cloudera.org:8080/#/c/17642/1/testdata/bin/run-hive-server.sh@153
PS1, Line 153:   # Wait for the HiveServer2 service to come up because callers of this script
> Same comment as above.
Done



-- 
To view, visit http://gerrit.cloudera.org:8080/17642
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2efdbcf886e2d32ccf8c7eef038360884e44f216
Gerrit-Change-Number: 17642
Gerrit-PatchSet: 2
Gerrit-Owner: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Wed, 30 Jun 2021 22:11:56 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-10774: Collect the output when starting Hive Metastore or HiveServer2

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17642 )

Change subject: IMPALA-10774: Collect the output when starting Hive Metastore or HiveServer2
......................................................................


Patch Set 2:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/7274/ DRY_RUN=false


-- 
To view, visit http://gerrit.cloudera.org:8080/17642
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2efdbcf886e2d32ccf8c7eef038360884e44f216
Gerrit-Change-Number: 17642
Gerrit-PatchSet: 2
Gerrit-Owner: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Mon, 05 Jul 2021 12:50:21 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10774: Collect the output when starting Hive Metastore or HiveServer2

Posted by "Fang-Yu Rao (Code Review)" <ge...@cloudera.org>.
Fang-Yu Rao has posted comments on this change. ( http://gerrit.cloudera.org:8080/17642 )

Change subject: IMPALA-10774: Collect the output when starting Hive Metastore or HiveServer2
......................................................................


Patch Set 1:

Hi all, please review the patch that aims to collect more information during the restart of the Hive Metastore or HiveServer2. Let me know if you have any comment or suggestion. Thank you very much for the help!


-- 
To view, visit http://gerrit.cloudera.org:8080/17642
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2efdbcf886e2d32ccf8c7eef038360884e44f216
Gerrit-Change-Number: 17642
Gerrit-PatchSet: 1
Gerrit-Owner: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Tue, 29 Jun 2021 23:19:29 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10774: Collect the output when starting Hive Metastore or HiveServer2

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17642 )

Change subject: IMPALA-10774: Collect the output when starting Hive Metastore or HiveServer2
......................................................................


Patch Set 2:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/9028/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/17642
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2efdbcf886e2d32ccf8c7eef038360884e44f216
Gerrit-Change-Number: 17642
Gerrit-PatchSet: 2
Gerrit-Owner: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Wed, 30 Jun 2021 22:34:15 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10774: Collect the output when starting Hive Metastore or HiveServer2

Posted by "Fang-Yu Rao (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, Aman Sinha, Vihang Karajgaonkar, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/17642

to look at the new patch set (#2).

Change subject: IMPALA-10774: Collect the output when starting Hive Metastore or HiveServer2
......................................................................

IMPALA-10774: Collect the output when starting Hive Metastore or HiveServer2

Before this patch, when we started the Hive Metastore or the HiveServer2
via run-hive-server.sh, hive-metastore.out and hive-server2.out would be
generated, respectively. The output of executing the hive command was
redirected to the corresponding files mentioned above. However, since
">" was used to redirect the output, the previously generated files
would be overwritten every time during the restart and thus some
important error message may be lost.

To collect more information regarding the restart, this patch appends
the newly produced output to hive-metastore.out and hive-server2.out
during the restart, which makes it easier for developers to troubleshoot
issues related to the restart.

Change-Id: I2efdbcf886e2d32ccf8c7eef038360884e44f216
---
M testdata/bin/run-hive-server.sh
1 file changed, 2 insertions(+), 2 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/42/17642/2
-- 
To view, visit http://gerrit.cloudera.org:8080/17642
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I2efdbcf886e2d32ccf8c7eef038360884e44f216
Gerrit-Change-Number: 17642
Gerrit-PatchSet: 2
Gerrit-Owner: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>

[Impala-ASF-CR] IMPALA-10774: Collect the output when starting Hive Metastore or HiveServer2

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17642 )

Change subject: IMPALA-10774: Collect the output when starting Hive Metastore or HiveServer2
......................................................................


Patch Set 2: Verified+1


-- 
To view, visit http://gerrit.cloudera.org:8080/17642
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2efdbcf886e2d32ccf8c7eef038360884e44f216
Gerrit-Change-Number: 17642
Gerrit-PatchSet: 2
Gerrit-Owner: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Mon, 05 Jul 2021 18:44:59 +0000
Gerrit-HasComments: No