You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by Aihua Xu <ax...@cloudera.com> on 2017/03/30 16:54:23 UTC
Review Request 58085: HIVE-16061: Some of console output is not
printed to the beeline console
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58085/
-----------------------------------------------------------
Review request for hive.
Repository: hive-git
Description
-------
HIVE-16061: Some of console output is not printed to the beeline console
Diffs
-----
common/src/java/org/apache/hadoop/hive/common/LogUtils.java 01b2e7c2e0568eebde6af7fe9c1e359d7ec5e7e8
ql/src/java/org/apache/hadoop/hive/ql/Driver.java d981119d3f6eb8fba66bf7c16aee838280d1c969
ql/src/java/org/apache/hadoop/hive/ql/exec/TaskRunner.java a596e92d8d67e7a96d8164de086c4f2eca0b0403
ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecDriver.java 1945163a0e2cfce53ee75c742143367ad23f97ed
ql/src/java/org/apache/hadoop/hive/ql/exec/mr/MapredLocalTask.java 591ea973f97913cad4cd7e96dd1ffc6a301b4bb1
ql/src/java/org/apache/hadoop/hive/ql/log/LogDivertAppender.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/parse/ColumnStatsSemanticAnalyzer.java 08d0544c49e6d2c2bbe473e8dfed4f38c1606ca7
ql/src/java/org/apache/hadoop/hive/ql/session/OperationLog.java 18216f25444a0b7867cbbf0ae0a5046a686b9e64
service/src/java/org/apache/hive/service/cli/operation/HiveCommandOperation.java 8f08c2e2ad0bb73494b2cd23359af5594997ebcf
service/src/java/org/apache/hive/service/cli/operation/LogDivertAppender.java eaf1acbcfeb687eeebf1b3f7eed2099241fc46a2
service/src/java/org/apache/hive/service/cli/operation/Operation.java 11a820fae6bb6a5815e0efb113ec73a399b0b5bd
service/src/java/org/apache/hive/service/cli/operation/OperationManager.java 3f8f68e31fde0c8c578a1052b1f98b7a86a55089
service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java f41092ed15ac8b88de72222e436abd80bdb8639b
Diff: https://reviews.apache.org/r/58085/diff/1/
Testing
-------
Test is done locally. You can see the beeline output now.
0: jdbc:hive2://localhost:10000> select t1.key from src t1 join src t2 on t1.key=t2.key limit 10;
INFO : Compiling command(queryId=axu_20170330125216_0de3bbf7-60f5-476d-b7eb-9861891d2961): select t1.key from src t1 join src t2 on t1.key=t2.key limit 10
INFO : Semantic Analysis Completed
INFO : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:t1.key, type:string, comment:null)], properties:null)
INFO : Completed compiling command(queryId=axu_20170330125216_0de3bbf7-60f5-476d-b7eb-9861891d2961); Time taken: 0.219 seconds
INFO : Concurrency mode is disabled, not creating a lock manager
INFO : Executing command(queryId=axu_20170330125216_0de3bbf7-60f5-476d-b7eb-9861891d2961): select t1.key from src t1 join src t2 on t1.key=t2.key limit 10
WARN : Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.
INFO : Query ID = axu_20170330125216_0de3bbf7-60f5-476d-b7eb-9861891d2961
INFO : Total jobs = 1
INFO : Starting task [Stage-4:MAPREDLOCAL] in serial mode
INFO : Execution completed successfully
INFO : MapredLocal task succeeded
INFO : Launching Job 1 out of 1
INFO : Starting task [Stage-3:MAPRED] in serial mode
INFO : Number of reduce tasks is set to 0 since there's no reduce operator
INFO : Starting Job = job_local1894165710_0002, Tracking URL = http://localhost:8080/
INFO : Kill Command = /Users/axu/Documents/workspaces/tools/hadoop/hadoop-2.6.0/bin/hadoop job -kill job_local1894165710_0002
INFO : Hadoop job information for Stage-3: number of mappers: 0; number of reducers: 0
INFO : 2017-03-30 12:52:21,788 Stage-3 map = 0%, reduce = 0%
ERROR : Ended Job = job_local1894165710_0002 with errors
ERROR : FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
INFO : MapReduce Jobs Launched:
INFO : Stage-Stage-3: HDFS Read: 0 HDFS Write: 0 FAIL
INFO : Total MapReduce CPU Time Spent: 0 msec
Thanks,
Aihua Xu
Re: Review Request 58085: HIVE-16061: Some of console output is not
printed to the beeline console
Posted by Aihua Xu <ax...@cloudera.com>.
> On March 30, 2017, 8:41 p.m., Aihua Xu wrote:
> > common/src/java/org/apache/hadoop/hive/common/LogUtils.java
> > Lines 52 (patched)
> > <https://reviews.apache.org/r/58085/diff/1/?file=1681072#file1681072line52>
> >
> > Any suggestions?
> >
> > Seems that is what it does. Can't think of anything else.
>
> Peter Vary wrote:
> // Constants of the key strings for the log4j MDC.
>
> Maybe?
They are the same thing. Actually Log4J uses ThreadContext. See https://logging.apache.org/log4j/2.x/manual/thread-context.html.
> On March 30, 2017, 8:41 p.m., Aihua Xu wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/Driver.java
> > Lines 1376-1380 (original)
> > <https://reviews.apache.org/r/58085/diff/1/?file=1681073#file1681073line1376>
> >
> > We were using OperationLog incorrectly before. This log should not be for direct write, but we should be able to write to it through LOG.info/debug(). As you can see, there is already LOG.debug("Waiting to acquire compile lock: " + command); above.
> >
> > With this patch, it will write this message at DEBUG level.
> >
> > The loglevels in operationLog (execution, verbose) are actually to control which classes are allowed to output logs. It's very confusion though.
>
> Peter Vary wrote:
> When we used OperationLog for direct write, we might thought that there are some messages we would like to show the user regardless of the log settings. Like the MR deprecation warning. I am not saying that it was good, merely stating that it was the case, and we will not be able to archive this after this change :D
>
>
> I was thinking about HiveCLI/BeeLine compatibility.
> HiveCLI uses SessionState._console, to print out information to the user. There are some cases when we use _console.getInfoStream() to write directly to the HiveCLI console. Also there are some places where we use SessionState.out to print a message to the HiveCLI user. I am not sure how can print these messages to the BeeLine user, when we do not want to show the other log messages of the given class to him.
If you want to print any message, you should still use LOG.info() which will print to console, file, etc which are configured. We are using those improperly, probably because we only had HiveCLI originally but later we add HiveServer2. So consoles in 2 HiveCLI and HiveServer2 mean different things.
I will have followup jiras to 1) make sure the logs are logged at the right level, 2) cleanup the console output of HiveCLI and Beeline.
Regarding some messages like "MR deprecation warning", I think it still makes sense not to show if the user disable it (like run in silense). It's just a warning like other warnings.
- Aihua
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58085/#review170638
-----------------------------------------------------------
On March 30, 2017, 9:01 p.m., Aihua Xu wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/58085/
> -----------------------------------------------------------
>
> (Updated March 30, 2017, 9:01 p.m.)
>
>
> Review request for hive.
>
>
> Bugs: hive-16061
> https://issues.apache.org/jira/browse/hive-16061
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> HIVE-16061: Some of console output is not printed to the beeline console
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/common/LogUtils.java 01b2e7c
> itests/hive-unit/src/test/java/org/apache/hive/service/cli/operation/TestOperationLoggingLayout.java e344e0f
> ql/src/java/org/apache/hadoop/hive/ql/Driver.java d981119
> ql/src/java/org/apache/hadoop/hive/ql/exec/TaskRunner.java a596e92
> ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecDriver.java 1945163
> ql/src/java/org/apache/hadoop/hive/ql/exec/mr/MapredLocalTask.java 591ea97
> ql/src/java/org/apache/hadoop/hive/ql/log/LogDivertAppender.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/parse/ColumnStatsSemanticAnalyzer.java 08d0544
> ql/src/java/org/apache/hadoop/hive/ql/session/OperationLog.java 18216f2
> ql/src/test/results/clientpositive/beeline/drop_with_concurrency.q.out 993329e
> ql/src/test/results/clientpositive/beeline/escape_comments.q.out 2a05e53
> service/src/java/org/apache/hive/service/cli/operation/HiveCommandOperation.java 8f08c2e
> service/src/java/org/apache/hive/service/cli/operation/LogDivertAppender.java eaf1acb
> service/src/java/org/apache/hive/service/cli/operation/Operation.java 11a820f
> service/src/java/org/apache/hive/service/cli/operation/OperationManager.java 3f8f68e
> service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java f41092e
>
>
> Diff: https://reviews.apache.org/r/58085/diff/2/
>
>
> Testing
> -------
>
> Test is done locally. You can see the beeline output now.
>
> 0: jdbc:hive2://localhost:10000> select t1.key from src t1 join src t2 on t1.key=t2.key limit 10;
> INFO : Compiling command(queryId=axu_20170330125216_0de3bbf7-60f5-476d-b7eb-9861891d2961): select t1.key from src t1 join src t2 on t1.key=t2.key limit 10
> INFO : Semantic Analysis Completed
> INFO : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:t1.key, type:string, comment:null)], properties:null)
> INFO : Completed compiling command(queryId=axu_20170330125216_0de3bbf7-60f5-476d-b7eb-9861891d2961); Time taken: 0.219 seconds
> INFO : Concurrency mode is disabled, not creating a lock manager
> INFO : Executing command(queryId=axu_20170330125216_0de3bbf7-60f5-476d-b7eb-9861891d2961): select t1.key from src t1 join src t2 on t1.key=t2.key limit 10
> WARN : Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.
> INFO : Query ID = axu_20170330125216_0de3bbf7-60f5-476d-b7eb-9861891d2961
> INFO : Total jobs = 1
> INFO : Starting task [Stage-4:MAPREDLOCAL] in serial mode
> INFO : Execution completed successfully
> INFO : MapredLocal task succeeded
> INFO : Launching Job 1 out of 1
> INFO : Starting task [Stage-3:MAPRED] in serial mode
> INFO : Number of reduce tasks is set to 0 since there's no reduce operator
> INFO : Starting Job = job_local1894165710_0002, Tracking URL = http://localhost:8080/
> INFO : Kill Command = /Users/axu/Documents/workspaces/tools/hadoop/hadoop-2.6.0/bin/hadoop job -kill job_local1894165710_0002
> INFO : Hadoop job information for Stage-3: number of mappers: 0; number of reducers: 0
> INFO : 2017-03-30 12:52:21,788 Stage-3 map = 0%, reduce = 0%
> ERROR : Ended Job = job_local1894165710_0002 with errors
> ERROR : FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
> INFO : MapReduce Jobs Launched:
> INFO : Stage-Stage-3: HDFS Read: 0 HDFS Write: 0 FAIL
> INFO : Total MapReduce CPU Time Spent: 0 msec
>
>
> Thanks,
>
> Aihua Xu
>
>
Re: Review Request 58085: HIVE-16061: Some of console output is not
printed to the beeline console
Posted by Peter Vary <pv...@cloudera.com>.
> On March 30, 2017, 8:41 p.m., Aihua Xu wrote:
> > common/src/java/org/apache/hadoop/hive/common/LogUtils.java
> > Lines 52 (patched)
> > <https://reviews.apache.org/r/58085/diff/1/?file=1681072#file1681072line52>
> >
> > Any suggestions?
> >
> > Seems that is what it does. Can't think of anything else.
// Constants of the key strings for the log4j MDC.
Maybe?
> On March 30, 2017, 8:41 p.m., Aihua Xu wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/Driver.java
> > Lines 1376-1380 (original)
> > <https://reviews.apache.org/r/58085/diff/1/?file=1681073#file1681073line1376>
> >
> > We were using OperationLog incorrectly before. This log should not be for direct write, but we should be able to write to it through LOG.info/debug(). As you can see, there is already LOG.debug("Waiting to acquire compile lock: " + command); above.
> >
> > With this patch, it will write this message at DEBUG level.
> >
> > The loglevels in operationLog (execution, verbose) are actually to control which classes are allowed to output logs. It's very confusion though.
When we used OperationLog for direct write, we might thought that there are some messages we would like to show the user regardless of the log settings. Like the MR deprecation warning. I am not saying that it was good, merely stating that it was the case, and we will not be able to archive this after this change :D
I was thinking about HiveCLI/BeeLine compatibility.
HiveCLI uses SessionState._console, to print out information to the user. There are some cases when we use _console.getInfoStream() to write directly to the HiveCLI console. Also there are some places where we use SessionState.out to print a message to the HiveCLI user. I am not sure how can print these messages to the BeeLine user, when we do not want to show the other log messages of the given class to him.
> On March 30, 2017, 8:41 p.m., Aihua Xu wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/log/LogDivertAppender.java
> > Lines 209 (patched)
> > <https://reviews.apache.org/r/58085/diff/1/?file=1681077#file1681077line209>
> >
> > Not really. HIVE_SERVER2_LOGGING_OPERATION_LOG_LOCATION is used to configure logLocation here (the base location). So it's still used.
Yeah, you are right. I missed that :)
- Peter
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58085/#review170638
-----------------------------------------------------------
On March 30, 2017, 9:01 p.m., Aihua Xu wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/58085/
> -----------------------------------------------------------
>
> (Updated March 30, 2017, 9:01 p.m.)
>
>
> Review request for hive.
>
>
> Bugs: hive-16061
> https://issues.apache.org/jira/browse/hive-16061
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> HIVE-16061: Some of console output is not printed to the beeline console
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/common/LogUtils.java 01b2e7c
> itests/hive-unit/src/test/java/org/apache/hive/service/cli/operation/TestOperationLoggingLayout.java e344e0f
> ql/src/java/org/apache/hadoop/hive/ql/Driver.java d981119
> ql/src/java/org/apache/hadoop/hive/ql/exec/TaskRunner.java a596e92
> ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecDriver.java 1945163
> ql/src/java/org/apache/hadoop/hive/ql/exec/mr/MapredLocalTask.java 591ea97
> ql/src/java/org/apache/hadoop/hive/ql/log/LogDivertAppender.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/parse/ColumnStatsSemanticAnalyzer.java 08d0544
> ql/src/java/org/apache/hadoop/hive/ql/session/OperationLog.java 18216f2
> ql/src/test/results/clientpositive/beeline/drop_with_concurrency.q.out 993329e
> ql/src/test/results/clientpositive/beeline/escape_comments.q.out 2a05e53
> service/src/java/org/apache/hive/service/cli/operation/HiveCommandOperation.java 8f08c2e
> service/src/java/org/apache/hive/service/cli/operation/LogDivertAppender.java eaf1acb
> service/src/java/org/apache/hive/service/cli/operation/Operation.java 11a820f
> service/src/java/org/apache/hive/service/cli/operation/OperationManager.java 3f8f68e
> service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java f41092e
>
>
> Diff: https://reviews.apache.org/r/58085/diff/2/
>
>
> Testing
> -------
>
> Test is done locally. You can see the beeline output now.
>
> 0: jdbc:hive2://localhost:10000> select t1.key from src t1 join src t2 on t1.key=t2.key limit 10;
> INFO : Compiling command(queryId=axu_20170330125216_0de3bbf7-60f5-476d-b7eb-9861891d2961): select t1.key from src t1 join src t2 on t1.key=t2.key limit 10
> INFO : Semantic Analysis Completed
> INFO : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:t1.key, type:string, comment:null)], properties:null)
> INFO : Completed compiling command(queryId=axu_20170330125216_0de3bbf7-60f5-476d-b7eb-9861891d2961); Time taken: 0.219 seconds
> INFO : Concurrency mode is disabled, not creating a lock manager
> INFO : Executing command(queryId=axu_20170330125216_0de3bbf7-60f5-476d-b7eb-9861891d2961): select t1.key from src t1 join src t2 on t1.key=t2.key limit 10
> WARN : Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.
> INFO : Query ID = axu_20170330125216_0de3bbf7-60f5-476d-b7eb-9861891d2961
> INFO : Total jobs = 1
> INFO : Starting task [Stage-4:MAPREDLOCAL] in serial mode
> INFO : Execution completed successfully
> INFO : MapredLocal task succeeded
> INFO : Launching Job 1 out of 1
> INFO : Starting task [Stage-3:MAPRED] in serial mode
> INFO : Number of reduce tasks is set to 0 since there's no reduce operator
> INFO : Starting Job = job_local1894165710_0002, Tracking URL = http://localhost:8080/
> INFO : Kill Command = /Users/axu/Documents/workspaces/tools/hadoop/hadoop-2.6.0/bin/hadoop job -kill job_local1894165710_0002
> INFO : Hadoop job information for Stage-3: number of mappers: 0; number of reducers: 0
> INFO : 2017-03-30 12:52:21,788 Stage-3 map = 0%, reduce = 0%
> ERROR : Ended Job = job_local1894165710_0002 with errors
> ERROR : FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
> INFO : MapReduce Jobs Launched:
> INFO : Stage-Stage-3: HDFS Read: 0 HDFS Write: 0 FAIL
> INFO : Total MapReduce CPU Time Spent: 0 msec
>
>
> Thanks,
>
> Aihua Xu
>
>
Re: Review Request 58085: HIVE-16061: Some of console output is not
printed to the beeline console
Posted by Peter Vary <pv...@cloudera.com>.
> On March 30, 2017, 8:41 p.m., Aihua Xu wrote:
> > common/src/java/org/apache/hadoop/hive/common/LogUtils.java
> > Lines 52 (patched)
> > <https://reviews.apache.org/r/58085/diff/1/?file=1681072#file1681072line52>
> >
> > Any suggestions?
> >
> > Seems that is what it does. Can't think of anything else.
>
> Peter Vary wrote:
> // Constants of the key strings for the log4j MDC.
>
> Maybe?
>
> Aihua Xu wrote:
> They are the same thing. Actually Log4J uses ThreadContext. See https://logging.apache.org/log4j/2.x/manual/thread-context.html.
Thanks for the clarification!
> On March 30, 2017, 8:41 p.m., Aihua Xu wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/Driver.java
> > Lines 1376-1380 (original)
> > <https://reviews.apache.org/r/58085/diff/1/?file=1681073#file1681073line1376>
> >
> > We were using OperationLog incorrectly before. This log should not be for direct write, but we should be able to write to it through LOG.info/debug(). As you can see, there is already LOG.debug("Waiting to acquire compile lock: " + command); above.
> >
> > With this patch, it will write this message at DEBUG level.
> >
> > The loglevels in operationLog (execution, verbose) are actually to control which classes are allowed to output logs. It's very confusion though.
>
> Peter Vary wrote:
> When we used OperationLog for direct write, we might thought that there are some messages we would like to show the user regardless of the log settings. Like the MR deprecation warning. I am not saying that it was good, merely stating that it was the case, and we will not be able to archive this after this change :D
>
>
> I was thinking about HiveCLI/BeeLine compatibility.
> HiveCLI uses SessionState._console, to print out information to the user. There are some cases when we use _console.getInfoStream() to write directly to the HiveCLI console. Also there are some places where we use SessionState.out to print a message to the HiveCLI user. I am not sure how can print these messages to the BeeLine user, when we do not want to show the other log messages of the given class to him.
>
> Aihua Xu wrote:
> If you want to print any message, you should still use LOG.info() which will print to console, file, etc which are configured. We are using those improperly, probably because we only had HiveCLI originally but later we add HiveServer2. So consoles in 2 HiveCLI and HiveServer2 mean different things.
> I will have followup jiras to 1) make sure the logs are logged at the right level, 2) cleanup the console output of HiveCLI and Beeline.
>
> Regarding some messages like "MR deprecation warning", I think it still makes sense not to show if the user disable it (like run in silense). It's just a warning like other warnings.
I agree with you, I just wanted to make sure, that we do it intentionally. :)
- Peter
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58085/#review170638
-----------------------------------------------------------
On March 30, 2017, 9:01 p.m., Aihua Xu wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/58085/
> -----------------------------------------------------------
>
> (Updated March 30, 2017, 9:01 p.m.)
>
>
> Review request for hive.
>
>
> Bugs: hive-16061
> https://issues.apache.org/jira/browse/hive-16061
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> HIVE-16061: Some of console output is not printed to the beeline console
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/common/LogUtils.java 01b2e7c
> itests/hive-unit/src/test/java/org/apache/hive/service/cli/operation/TestOperationLoggingLayout.java e344e0f
> ql/src/java/org/apache/hadoop/hive/ql/Driver.java d981119
> ql/src/java/org/apache/hadoop/hive/ql/exec/TaskRunner.java a596e92
> ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecDriver.java 1945163
> ql/src/java/org/apache/hadoop/hive/ql/exec/mr/MapredLocalTask.java 591ea97
> ql/src/java/org/apache/hadoop/hive/ql/log/LogDivertAppender.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/parse/ColumnStatsSemanticAnalyzer.java 08d0544
> ql/src/java/org/apache/hadoop/hive/ql/session/OperationLog.java 18216f2
> ql/src/test/results/clientpositive/beeline/drop_with_concurrency.q.out 993329e
> ql/src/test/results/clientpositive/beeline/escape_comments.q.out 2a05e53
> service/src/java/org/apache/hive/service/cli/operation/HiveCommandOperation.java 8f08c2e
> service/src/java/org/apache/hive/service/cli/operation/LogDivertAppender.java eaf1acb
> service/src/java/org/apache/hive/service/cli/operation/Operation.java 11a820f
> service/src/java/org/apache/hive/service/cli/operation/OperationManager.java 3f8f68e
> service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java f41092e
>
>
> Diff: https://reviews.apache.org/r/58085/diff/2/
>
>
> Testing
> -------
>
> Test is done locally. You can see the beeline output now.
>
> 0: jdbc:hive2://localhost:10000> select t1.key from src t1 join src t2 on t1.key=t2.key limit 10;
> INFO : Compiling command(queryId=axu_20170330125216_0de3bbf7-60f5-476d-b7eb-9861891d2961): select t1.key from src t1 join src t2 on t1.key=t2.key limit 10
> INFO : Semantic Analysis Completed
> INFO : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:t1.key, type:string, comment:null)], properties:null)
> INFO : Completed compiling command(queryId=axu_20170330125216_0de3bbf7-60f5-476d-b7eb-9861891d2961); Time taken: 0.219 seconds
> INFO : Concurrency mode is disabled, not creating a lock manager
> INFO : Executing command(queryId=axu_20170330125216_0de3bbf7-60f5-476d-b7eb-9861891d2961): select t1.key from src t1 join src t2 on t1.key=t2.key limit 10
> WARN : Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.
> INFO : Query ID = axu_20170330125216_0de3bbf7-60f5-476d-b7eb-9861891d2961
> INFO : Total jobs = 1
> INFO : Starting task [Stage-4:MAPREDLOCAL] in serial mode
> INFO : Execution completed successfully
> INFO : MapredLocal task succeeded
> INFO : Launching Job 1 out of 1
> INFO : Starting task [Stage-3:MAPRED] in serial mode
> INFO : Number of reduce tasks is set to 0 since there's no reduce operator
> INFO : Starting Job = job_local1894165710_0002, Tracking URL = http://localhost:8080/
> INFO : Kill Command = /Users/axu/Documents/workspaces/tools/hadoop/hadoop-2.6.0/bin/hadoop job -kill job_local1894165710_0002
> INFO : Hadoop job information for Stage-3: number of mappers: 0; number of reducers: 0
> INFO : 2017-03-30 12:52:21,788 Stage-3 map = 0%, reduce = 0%
> ERROR : Ended Job = job_local1894165710_0002 with errors
> ERROR : FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
> INFO : MapReduce Jobs Launched:
> INFO : Stage-Stage-3: HDFS Read: 0 HDFS Write: 0 FAIL
> INFO : Total MapReduce CPU Time Spent: 0 msec
>
>
> Thanks,
>
> Aihua Xu
>
>
Re: Review Request 58085: HIVE-16061: Some of console output is not
printed to the beeline console
Posted by Aihua Xu <ax...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58085/#review170638
-----------------------------------------------------------
common/src/java/org/apache/hadoop/hive/common/LogUtils.java
Lines 52 (patched)
<https://reviews.apache.org/r/58085/#comment243510>
Any suggestions?
Seems that is what it does. Can't think of anything else.
ql/src/java/org/apache/hadoop/hive/ql/Driver.java
Lines 1376-1380 (original)
<https://reviews.apache.org/r/58085/#comment243512>
We were using OperationLog incorrectly before. This log should not be for direct write, but we should be able to write to it through LOG.info/debug(). As you can see, there is already LOG.debug("Waiting to acquire compile lock: " + command); above.
With this patch, it will write this message at DEBUG level.
The loglevels in operationLog (execution, verbose) are actually to control which classes are allowed to output logs. It's very confusion though.
ql/src/java/org/apache/hadoop/hive/ql/log/LogDivertAppender.java
Lines 209 (patched)
<https://reviews.apache.org/r/58085/#comment243513>
Not really. HIVE_SERVER2_LOGGING_OPERATION_LOG_LOCATION is used to configure logLocation here (the base location). So it's still used.
service/src/java/org/apache/hive/service/cli/operation/Operation.java
Lines 219-252 (original)
<https://reviews.apache.org/r/58085/#comment243514>
Right. The file will be created by the routing appender. We just need to read from such file and output to the beeline.
service/src/java/org/apache/hive/service/cli/operation/Operation.java
Line 295 (original), 246 (patched)
<https://reviews.apache.org/r/58085/#comment243516>
Yeah. I was debating that since it's just one line. I will do that then.
service/src/java/org/apache/hive/service/cli/operation/OperationManager.java
Line 78 (original), 74 (patched)
<https://reviews.apache.org/r/58085/#comment243518>
You are right. We need call the register always.
- Aihua Xu
On March 30, 2017, 4:54 p.m., Aihua Xu wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/58085/
> -----------------------------------------------------------
>
> (Updated March 30, 2017, 4:54 p.m.)
>
>
> Review request for hive.
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> HIVE-16061: Some of console output is not printed to the beeline console
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/common/LogUtils.java 01b2e7c2e0568eebde6af7fe9c1e359d7ec5e7e8
> ql/src/java/org/apache/hadoop/hive/ql/Driver.java d981119d3f6eb8fba66bf7c16aee838280d1c969
> ql/src/java/org/apache/hadoop/hive/ql/exec/TaskRunner.java a596e92d8d67e7a96d8164de086c4f2eca0b0403
> ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecDriver.java 1945163a0e2cfce53ee75c742143367ad23f97ed
> ql/src/java/org/apache/hadoop/hive/ql/exec/mr/MapredLocalTask.java 591ea973f97913cad4cd7e96dd1ffc6a301b4bb1
> ql/src/java/org/apache/hadoop/hive/ql/log/LogDivertAppender.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/parse/ColumnStatsSemanticAnalyzer.java 08d0544c49e6d2c2bbe473e8dfed4f38c1606ca7
> ql/src/java/org/apache/hadoop/hive/ql/session/OperationLog.java 18216f25444a0b7867cbbf0ae0a5046a686b9e64
> service/src/java/org/apache/hive/service/cli/operation/HiveCommandOperation.java 8f08c2e2ad0bb73494b2cd23359af5594997ebcf
> service/src/java/org/apache/hive/service/cli/operation/LogDivertAppender.java eaf1acbcfeb687eeebf1b3f7eed2099241fc46a2
> service/src/java/org/apache/hive/service/cli/operation/Operation.java 11a820fae6bb6a5815e0efb113ec73a399b0b5bd
> service/src/java/org/apache/hive/service/cli/operation/OperationManager.java 3f8f68e31fde0c8c578a1052b1f98b7a86a55089
> service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java f41092ed15ac8b88de72222e436abd80bdb8639b
>
>
> Diff: https://reviews.apache.org/r/58085/diff/1/
>
>
> Testing
> -------
>
> Test is done locally. You can see the beeline output now.
>
> 0: jdbc:hive2://localhost:10000> select t1.key from src t1 join src t2 on t1.key=t2.key limit 10;
> INFO : Compiling command(queryId=axu_20170330125216_0de3bbf7-60f5-476d-b7eb-9861891d2961): select t1.key from src t1 join src t2 on t1.key=t2.key limit 10
> INFO : Semantic Analysis Completed
> INFO : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:t1.key, type:string, comment:null)], properties:null)
> INFO : Completed compiling command(queryId=axu_20170330125216_0de3bbf7-60f5-476d-b7eb-9861891d2961); Time taken: 0.219 seconds
> INFO : Concurrency mode is disabled, not creating a lock manager
> INFO : Executing command(queryId=axu_20170330125216_0de3bbf7-60f5-476d-b7eb-9861891d2961): select t1.key from src t1 join src t2 on t1.key=t2.key limit 10
> WARN : Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.
> INFO : Query ID = axu_20170330125216_0de3bbf7-60f5-476d-b7eb-9861891d2961
> INFO : Total jobs = 1
> INFO : Starting task [Stage-4:MAPREDLOCAL] in serial mode
> INFO : Execution completed successfully
> INFO : MapredLocal task succeeded
> INFO : Launching Job 1 out of 1
> INFO : Starting task [Stage-3:MAPRED] in serial mode
> INFO : Number of reduce tasks is set to 0 since there's no reduce operator
> INFO : Starting Job = job_local1894165710_0002, Tracking URL = http://localhost:8080/
> INFO : Kill Command = /Users/axu/Documents/workspaces/tools/hadoop/hadoop-2.6.0/bin/hadoop job -kill job_local1894165710_0002
> INFO : Hadoop job information for Stage-3: number of mappers: 0; number of reducers: 0
> INFO : 2017-03-30 12:52:21,788 Stage-3 map = 0%, reduce = 0%
> ERROR : Ended Job = job_local1894165710_0002 with errors
> ERROR : FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
> INFO : MapReduce Jobs Launched:
> INFO : Stage-Stage-3: HDFS Read: 0 HDFS Write: 0 FAIL
> INFO : Total MapReduce CPU Time Spent: 0 msec
>
>
> Thanks,
>
> Aihua Xu
>
>
Re: Review Request 58085: HIVE-16061: Some of console output is not
printed to the beeline console
Posted by Peter Vary <pv...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58085/#review170607
-----------------------------------------------------------
Thanks Aihua for the patch!
It would be good to clear up this part of the code!
Overall looks good, just some questions, nits.
Sorry for the duplications, I was not able to remove them. When I tried it I created another one :(
Thanks,
Peter
common/src/java/org/apache/hadoop/hive/common/LogUtils.java
Lines 52 (patched)
<https://reviews.apache.org/r/58085/#comment243486>
Maybe we should rephrase the comment?
ql/src/java/org/apache/hadoop/hive/ql/Driver.java
Lines 1376-1380 (original)
<https://reviews.apache.org/r/58085/#comment243479>
With removing the these explicit writeOperationLog operations we remove the possibility to configure/fire these messages independently of the log4j log level (INFO/DEBUG/TRACE...), and only dependent of the Operation log level (EXECUTION/PERFORMANCE/...).
Until this point it was possible to keep the log4j loglevel at ERROR, still if the operation log level was EXECUTE, we got this message.
It might be intentional, or an accepted change, or I might missing here something. It is not easy to understand this whole OperationLog stuff :)
ql/src/java/org/apache/hadoop/hive/ql/log/LogDivertAppender.java
Lines 209 (patched)
<https://reviews.apache.org/r/58085/#comment243481>
We might be able to remove the HIVE_SERVER2_LOGGING_OPERATION_LOG_LOCATION from HiveConf, since we do not use it any more.
Am I right?
ql/src/java/org/apache/hadoop/hive/ql/log/LogDivertAppender.java
Lines 209 (patched)
<https://reviews.apache.org/r/58085/#comment243482>
We might be able to remove the HIVE_SERVER2_LOGGING_OPERATION_LOG_LOCATION from HiveConf, since we do not use it any more.
Am I right?
ql/src/java/org/apache/hadoop/hive/ql/log/LogDivertAppender.java
Lines 209 (patched)
<https://reviews.apache.org/r/58085/#comment243484>
We might be able to remove the HIVE_SERVER2_LOGGING_OPERATION_LOG_LOCATION from HiveConf, since we do not use it any more.
Am I right?
ql/src/java/org/apache/hadoop/hive/ql/log/LogDivertAppender.java
Lines 209 (patched)
<https://reviews.apache.org/r/58085/#comment243491>
We might be able to remove the HIVE_SERVER2_LOGGING_OPERATION_LOG_LOCATION from HiveConf, since we do not use it any more.
Am I right?
service/src/java/org/apache/hive/service/cli/operation/Operation.java
Lines 219-252 (original)
<https://reviews.apache.org/r/58085/#comment243474>
We do not want this nice sophisticated log file creation? We belive that the operation handle is created, and the appropriate error messages are printed by log4j?
service/src/java/org/apache/hive/service/cli/operation/Operation.java
Line 295 (original), 246 (patched)
<https://reviews.apache.org/r/58085/#comment243476>
I think it might be better to have a LogUtils.unregisterLoggingContext. What do you think?
service/src/java/org/apache/hive/service/cli/operation/OperationManager.java
Line 78 (original), 74 (patched)
<https://reviews.apache.org/r/58085/#comment243487>
What happens, when the operation log is turned off by default, but later for some session we turn it on?
It seems to me, that the init is called, but we might have some unexpected results :) Like from now on we will have the routingAppender, and every session will start to log in EXECUTE level, since that is the default HIVE_SERVER2_LOGGING_OPERATION_LEVEL.
Just a guess, I did not have time to try it out :)
- Peter Vary
On March 30, 2017, 4:54 p.m., Aihua Xu wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/58085/
> -----------------------------------------------------------
>
> (Updated March 30, 2017, 4:54 p.m.)
>
>
> Review request for hive.
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> HIVE-16061: Some of console output is not printed to the beeline console
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/common/LogUtils.java 01b2e7c2e0568eebde6af7fe9c1e359d7ec5e7e8
> ql/src/java/org/apache/hadoop/hive/ql/Driver.java d981119d3f6eb8fba66bf7c16aee838280d1c969
> ql/src/java/org/apache/hadoop/hive/ql/exec/TaskRunner.java a596e92d8d67e7a96d8164de086c4f2eca0b0403
> ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecDriver.java 1945163a0e2cfce53ee75c742143367ad23f97ed
> ql/src/java/org/apache/hadoop/hive/ql/exec/mr/MapredLocalTask.java 591ea973f97913cad4cd7e96dd1ffc6a301b4bb1
> ql/src/java/org/apache/hadoop/hive/ql/log/LogDivertAppender.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/parse/ColumnStatsSemanticAnalyzer.java 08d0544c49e6d2c2bbe473e8dfed4f38c1606ca7
> ql/src/java/org/apache/hadoop/hive/ql/session/OperationLog.java 18216f25444a0b7867cbbf0ae0a5046a686b9e64
> service/src/java/org/apache/hive/service/cli/operation/HiveCommandOperation.java 8f08c2e2ad0bb73494b2cd23359af5594997ebcf
> service/src/java/org/apache/hive/service/cli/operation/LogDivertAppender.java eaf1acbcfeb687eeebf1b3f7eed2099241fc46a2
> service/src/java/org/apache/hive/service/cli/operation/Operation.java 11a820fae6bb6a5815e0efb113ec73a399b0b5bd
> service/src/java/org/apache/hive/service/cli/operation/OperationManager.java 3f8f68e31fde0c8c578a1052b1f98b7a86a55089
> service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java f41092ed15ac8b88de72222e436abd80bdb8639b
>
>
> Diff: https://reviews.apache.org/r/58085/diff/1/
>
>
> Testing
> -------
>
> Test is done locally. You can see the beeline output now.
>
> 0: jdbc:hive2://localhost:10000> select t1.key from src t1 join src t2 on t1.key=t2.key limit 10;
> INFO : Compiling command(queryId=axu_20170330125216_0de3bbf7-60f5-476d-b7eb-9861891d2961): select t1.key from src t1 join src t2 on t1.key=t2.key limit 10
> INFO : Semantic Analysis Completed
> INFO : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:t1.key, type:string, comment:null)], properties:null)
> INFO : Completed compiling command(queryId=axu_20170330125216_0de3bbf7-60f5-476d-b7eb-9861891d2961); Time taken: 0.219 seconds
> INFO : Concurrency mode is disabled, not creating a lock manager
> INFO : Executing command(queryId=axu_20170330125216_0de3bbf7-60f5-476d-b7eb-9861891d2961): select t1.key from src t1 join src t2 on t1.key=t2.key limit 10
> WARN : Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.
> INFO : Query ID = axu_20170330125216_0de3bbf7-60f5-476d-b7eb-9861891d2961
> INFO : Total jobs = 1
> INFO : Starting task [Stage-4:MAPREDLOCAL] in serial mode
> INFO : Execution completed successfully
> INFO : MapredLocal task succeeded
> INFO : Launching Job 1 out of 1
> INFO : Starting task [Stage-3:MAPRED] in serial mode
> INFO : Number of reduce tasks is set to 0 since there's no reduce operator
> INFO : Starting Job = job_local1894165710_0002, Tracking URL = http://localhost:8080/
> INFO : Kill Command = /Users/axu/Documents/workspaces/tools/hadoop/hadoop-2.6.0/bin/hadoop job -kill job_local1894165710_0002
> INFO : Hadoop job information for Stage-3: number of mappers: 0; number of reducers: 0
> INFO : 2017-03-30 12:52:21,788 Stage-3 map = 0%, reduce = 0%
> ERROR : Ended Job = job_local1894165710_0002 with errors
> ERROR : FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
> INFO : MapReduce Jobs Launched:
> INFO : Stage-Stage-3: HDFS Read: 0 HDFS Write: 0 FAIL
> INFO : Total MapReduce CPU Time Spent: 0 msec
>
>
> Thanks,
>
> Aihua Xu
>
>
Re: Review Request 58085: HIVE-16061: Some of console output is not
printed to the beeline console
Posted by Peter Vary <pv...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58085/#review170723
-----------------------------------------------------------
Ship it!
Thanks for the patch Aihua!
LGTM (non-binding)
- Peter Vary
On March 30, 2017, 9:01 p.m., Aihua Xu wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/58085/
> -----------------------------------------------------------
>
> (Updated March 30, 2017, 9:01 p.m.)
>
>
> Review request for hive.
>
>
> Bugs: hive-16061
> https://issues.apache.org/jira/browse/hive-16061
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> HIVE-16061: Some of console output is not printed to the beeline console
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/common/LogUtils.java 01b2e7c
> itests/hive-unit/src/test/java/org/apache/hive/service/cli/operation/TestOperationLoggingLayout.java e344e0f
> ql/src/java/org/apache/hadoop/hive/ql/Driver.java d981119
> ql/src/java/org/apache/hadoop/hive/ql/exec/TaskRunner.java a596e92
> ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecDriver.java 1945163
> ql/src/java/org/apache/hadoop/hive/ql/exec/mr/MapredLocalTask.java 591ea97
> ql/src/java/org/apache/hadoop/hive/ql/log/LogDivertAppender.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/parse/ColumnStatsSemanticAnalyzer.java 08d0544
> ql/src/java/org/apache/hadoop/hive/ql/session/OperationLog.java 18216f2
> ql/src/test/results/clientpositive/beeline/drop_with_concurrency.q.out 993329e
> ql/src/test/results/clientpositive/beeline/escape_comments.q.out 2a05e53
> service/src/java/org/apache/hive/service/cli/operation/HiveCommandOperation.java 8f08c2e
> service/src/java/org/apache/hive/service/cli/operation/LogDivertAppender.java eaf1acb
> service/src/java/org/apache/hive/service/cli/operation/Operation.java 11a820f
> service/src/java/org/apache/hive/service/cli/operation/OperationManager.java 3f8f68e
> service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java f41092e
>
>
> Diff: https://reviews.apache.org/r/58085/diff/2/
>
>
> Testing
> -------
>
> Test is done locally. You can see the beeline output now.
>
> 0: jdbc:hive2://localhost:10000> select t1.key from src t1 join src t2 on t1.key=t2.key limit 10;
> INFO : Compiling command(queryId=axu_20170330125216_0de3bbf7-60f5-476d-b7eb-9861891d2961): select t1.key from src t1 join src t2 on t1.key=t2.key limit 10
> INFO : Semantic Analysis Completed
> INFO : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:t1.key, type:string, comment:null)], properties:null)
> INFO : Completed compiling command(queryId=axu_20170330125216_0de3bbf7-60f5-476d-b7eb-9861891d2961); Time taken: 0.219 seconds
> INFO : Concurrency mode is disabled, not creating a lock manager
> INFO : Executing command(queryId=axu_20170330125216_0de3bbf7-60f5-476d-b7eb-9861891d2961): select t1.key from src t1 join src t2 on t1.key=t2.key limit 10
> WARN : Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.
> INFO : Query ID = axu_20170330125216_0de3bbf7-60f5-476d-b7eb-9861891d2961
> INFO : Total jobs = 1
> INFO : Starting task [Stage-4:MAPREDLOCAL] in serial mode
> INFO : Execution completed successfully
> INFO : MapredLocal task succeeded
> INFO : Launching Job 1 out of 1
> INFO : Starting task [Stage-3:MAPRED] in serial mode
> INFO : Number of reduce tasks is set to 0 since there's no reduce operator
> INFO : Starting Job = job_local1894165710_0002, Tracking URL = http://localhost:8080/
> INFO : Kill Command = /Users/axu/Documents/workspaces/tools/hadoop/hadoop-2.6.0/bin/hadoop job -kill job_local1894165710_0002
> INFO : Hadoop job information for Stage-3: number of mappers: 0; number of reducers: 0
> INFO : 2017-03-30 12:52:21,788 Stage-3 map = 0%, reduce = 0%
> ERROR : Ended Job = job_local1894165710_0002 with errors
> ERROR : FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
> INFO : MapReduce Jobs Launched:
> INFO : Stage-Stage-3: HDFS Read: 0 HDFS Write: 0 FAIL
> INFO : Total MapReduce CPU Time Spent: 0 msec
>
>
> Thanks,
>
> Aihua Xu
>
>
Re: Review Request 58085: HIVE-16061: Some of console output is not
printed to the beeline console
Posted by Aihua Xu <ax...@cloudera.com>.
> On March 30, 2017, 9:18 p.m., Vihang Karajgaonkar wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/log/LogDivertAppender.java
> > Lines 80-81 (patched)
> > <https://reviews.apache.org/r/58085/diff/2/?file=1681182#file1681182line80>
> >
> > Why not use JobSubmitter.class.getName()? Same with Job.class.getName(), SessionState and SparkJobMonitor
I carried over this code from previous implementation. This is to avoid unnecessary project dependency.
> On March 30, 2017, 9:18 p.m., Vihang Karajgaonkar wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/log/LogDivertAppender.java
> > Lines 120 (patched)
> > <https://reviews.apache.org/r/58085/diff/2/?file=1681182#file1681182line120>
> >
> > isnt this already done at line 107?
We are setting logging mode to a new value so needs to reevalulate if it's verbose.
if (currentLoggingMode != loggingMode) {
loggingMode = currentLoggingMode;
excludeMatches = (loggingMode == OperationLog.LoggingLevel.VERBOSE);
setCurrentNamePattern(loggingMode);
}
- Aihua
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58085/#review170641
-----------------------------------------------------------
On March 30, 2017, 9:01 p.m., Aihua Xu wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/58085/
> -----------------------------------------------------------
>
> (Updated March 30, 2017, 9:01 p.m.)
>
>
> Review request for hive.
>
>
> Bugs: hive-16061
> https://issues.apache.org/jira/browse/hive-16061
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> HIVE-16061: Some of console output is not printed to the beeline console
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/common/LogUtils.java 01b2e7c
> itests/hive-unit/src/test/java/org/apache/hive/service/cli/operation/TestOperationLoggingLayout.java e344e0f
> ql/src/java/org/apache/hadoop/hive/ql/Driver.java d981119
> ql/src/java/org/apache/hadoop/hive/ql/exec/TaskRunner.java a596e92
> ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecDriver.java 1945163
> ql/src/java/org/apache/hadoop/hive/ql/exec/mr/MapredLocalTask.java 591ea97
> ql/src/java/org/apache/hadoop/hive/ql/log/LogDivertAppender.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/parse/ColumnStatsSemanticAnalyzer.java 08d0544
> ql/src/java/org/apache/hadoop/hive/ql/session/OperationLog.java 18216f2
> ql/src/test/results/clientpositive/beeline/drop_with_concurrency.q.out 993329e
> ql/src/test/results/clientpositive/beeline/escape_comments.q.out 2a05e53
> service/src/java/org/apache/hive/service/cli/operation/HiveCommandOperation.java 8f08c2e
> service/src/java/org/apache/hive/service/cli/operation/LogDivertAppender.java eaf1acb
> service/src/java/org/apache/hive/service/cli/operation/Operation.java 11a820f
> service/src/java/org/apache/hive/service/cli/operation/OperationManager.java 3f8f68e
> service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java f41092e
>
>
> Diff: https://reviews.apache.org/r/58085/diff/2/
>
>
> Testing
> -------
>
> Test is done locally. You can see the beeline output now.
>
> 0: jdbc:hive2://localhost:10000> select t1.key from src t1 join src t2 on t1.key=t2.key limit 10;
> INFO : Compiling command(queryId=axu_20170330125216_0de3bbf7-60f5-476d-b7eb-9861891d2961): select t1.key from src t1 join src t2 on t1.key=t2.key limit 10
> INFO : Semantic Analysis Completed
> INFO : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:t1.key, type:string, comment:null)], properties:null)
> INFO : Completed compiling command(queryId=axu_20170330125216_0de3bbf7-60f5-476d-b7eb-9861891d2961); Time taken: 0.219 seconds
> INFO : Concurrency mode is disabled, not creating a lock manager
> INFO : Executing command(queryId=axu_20170330125216_0de3bbf7-60f5-476d-b7eb-9861891d2961): select t1.key from src t1 join src t2 on t1.key=t2.key limit 10
> WARN : Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.
> INFO : Query ID = axu_20170330125216_0de3bbf7-60f5-476d-b7eb-9861891d2961
> INFO : Total jobs = 1
> INFO : Starting task [Stage-4:MAPREDLOCAL] in serial mode
> INFO : Execution completed successfully
> INFO : MapredLocal task succeeded
> INFO : Launching Job 1 out of 1
> INFO : Starting task [Stage-3:MAPRED] in serial mode
> INFO : Number of reduce tasks is set to 0 since there's no reduce operator
> INFO : Starting Job = job_local1894165710_0002, Tracking URL = http://localhost:8080/
> INFO : Kill Command = /Users/axu/Documents/workspaces/tools/hadoop/hadoop-2.6.0/bin/hadoop job -kill job_local1894165710_0002
> INFO : Hadoop job information for Stage-3: number of mappers: 0; number of reducers: 0
> INFO : 2017-03-30 12:52:21,788 Stage-3 map = 0%, reduce = 0%
> ERROR : Ended Job = job_local1894165710_0002 with errors
> ERROR : FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
> INFO : MapReduce Jobs Launched:
> INFO : Stage-Stage-3: HDFS Read: 0 HDFS Write: 0 FAIL
> INFO : Total MapReduce CPU Time Spent: 0 msec
>
>
> Thanks,
>
> Aihua Xu
>
>
Re: Review Request 58085: HIVE-16061: Some of console output is not
printed to the beeline console
Posted by Vihang Karajgaonkar <vi...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58085/#review170641
-----------------------------------------------------------
ql/src/java/org/apache/hadoop/hive/ql/log/LogDivertAppender.java
Lines 80-81 (patched)
<https://reviews.apache.org/r/58085/#comment243522>
Why not use JobSubmitter.class.getName()? Same with Job.class.getName(), SessionState and SparkJobMonitor
ql/src/java/org/apache/hadoop/hive/ql/log/LogDivertAppender.java
Lines 120 (patched)
<https://reviews.apache.org/r/58085/#comment243523>
isnt this already done at line 107?
- Vihang Karajgaonkar
On March 30, 2017, 9:01 p.m., Aihua Xu wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/58085/
> -----------------------------------------------------------
>
> (Updated March 30, 2017, 9:01 p.m.)
>
>
> Review request for hive.
>
>
> Bugs: hive-16061
> https://issues.apache.org/jira/browse/hive-16061
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> HIVE-16061: Some of console output is not printed to the beeline console
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/common/LogUtils.java 01b2e7c
> itests/hive-unit/src/test/java/org/apache/hive/service/cli/operation/TestOperationLoggingLayout.java e344e0f
> ql/src/java/org/apache/hadoop/hive/ql/Driver.java d981119
> ql/src/java/org/apache/hadoop/hive/ql/exec/TaskRunner.java a596e92
> ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecDriver.java 1945163
> ql/src/java/org/apache/hadoop/hive/ql/exec/mr/MapredLocalTask.java 591ea97
> ql/src/java/org/apache/hadoop/hive/ql/log/LogDivertAppender.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/parse/ColumnStatsSemanticAnalyzer.java 08d0544
> ql/src/java/org/apache/hadoop/hive/ql/session/OperationLog.java 18216f2
> ql/src/test/results/clientpositive/beeline/drop_with_concurrency.q.out 993329e
> ql/src/test/results/clientpositive/beeline/escape_comments.q.out 2a05e53
> service/src/java/org/apache/hive/service/cli/operation/HiveCommandOperation.java 8f08c2e
> service/src/java/org/apache/hive/service/cli/operation/LogDivertAppender.java eaf1acb
> service/src/java/org/apache/hive/service/cli/operation/Operation.java 11a820f
> service/src/java/org/apache/hive/service/cli/operation/OperationManager.java 3f8f68e
> service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java f41092e
>
>
> Diff: https://reviews.apache.org/r/58085/diff/2/
>
>
> Testing
> -------
>
> Test is done locally. You can see the beeline output now.
>
> 0: jdbc:hive2://localhost:10000> select t1.key from src t1 join src t2 on t1.key=t2.key limit 10;
> INFO : Compiling command(queryId=axu_20170330125216_0de3bbf7-60f5-476d-b7eb-9861891d2961): select t1.key from src t1 join src t2 on t1.key=t2.key limit 10
> INFO : Semantic Analysis Completed
> INFO : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:t1.key, type:string, comment:null)], properties:null)
> INFO : Completed compiling command(queryId=axu_20170330125216_0de3bbf7-60f5-476d-b7eb-9861891d2961); Time taken: 0.219 seconds
> INFO : Concurrency mode is disabled, not creating a lock manager
> INFO : Executing command(queryId=axu_20170330125216_0de3bbf7-60f5-476d-b7eb-9861891d2961): select t1.key from src t1 join src t2 on t1.key=t2.key limit 10
> WARN : Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.
> INFO : Query ID = axu_20170330125216_0de3bbf7-60f5-476d-b7eb-9861891d2961
> INFO : Total jobs = 1
> INFO : Starting task [Stage-4:MAPREDLOCAL] in serial mode
> INFO : Execution completed successfully
> INFO : MapredLocal task succeeded
> INFO : Launching Job 1 out of 1
> INFO : Starting task [Stage-3:MAPRED] in serial mode
> INFO : Number of reduce tasks is set to 0 since there's no reduce operator
> INFO : Starting Job = job_local1894165710_0002, Tracking URL = http://localhost:8080/
> INFO : Kill Command = /Users/axu/Documents/workspaces/tools/hadoop/hadoop-2.6.0/bin/hadoop job -kill job_local1894165710_0002
> INFO : Hadoop job information for Stage-3: number of mappers: 0; number of reducers: 0
> INFO : 2017-03-30 12:52:21,788 Stage-3 map = 0%, reduce = 0%
> ERROR : Ended Job = job_local1894165710_0002 with errors
> ERROR : FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
> INFO : MapReduce Jobs Launched:
> INFO : Stage-Stage-3: HDFS Read: 0 HDFS Write: 0 FAIL
> INFO : Total MapReduce CPU Time Spent: 0 msec
>
>
> Thanks,
>
> Aihua Xu
>
>
Re: Review Request 58085: HIVE-16061: Some of console output is not
printed to the beeline console
Posted by Aihua Xu <ax...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58085/
-----------------------------------------------------------
(Updated March 30, 2017, 9:01 p.m.)
Review request for hive.
Changes
-------
Address the comments and fix the unit test failures.
Bugs: hive-16061
https://issues.apache.org/jira/browse/hive-16061
Repository: hive-git
Description
-------
HIVE-16061: Some of console output is not printed to the beeline console
Diffs (updated)
-----
common/src/java/org/apache/hadoop/hive/common/LogUtils.java 01b2e7c
itests/hive-unit/src/test/java/org/apache/hive/service/cli/operation/TestOperationLoggingLayout.java e344e0f
ql/src/java/org/apache/hadoop/hive/ql/Driver.java d981119
ql/src/java/org/apache/hadoop/hive/ql/exec/TaskRunner.java a596e92
ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecDriver.java 1945163
ql/src/java/org/apache/hadoop/hive/ql/exec/mr/MapredLocalTask.java 591ea97
ql/src/java/org/apache/hadoop/hive/ql/log/LogDivertAppender.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/parse/ColumnStatsSemanticAnalyzer.java 08d0544
ql/src/java/org/apache/hadoop/hive/ql/session/OperationLog.java 18216f2
ql/src/test/results/clientpositive/beeline/drop_with_concurrency.q.out 993329e
ql/src/test/results/clientpositive/beeline/escape_comments.q.out 2a05e53
service/src/java/org/apache/hive/service/cli/operation/HiveCommandOperation.java 8f08c2e
service/src/java/org/apache/hive/service/cli/operation/LogDivertAppender.java eaf1acb
service/src/java/org/apache/hive/service/cli/operation/Operation.java 11a820f
service/src/java/org/apache/hive/service/cli/operation/OperationManager.java 3f8f68e
service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java f41092e
Diff: https://reviews.apache.org/r/58085/diff/2/
Changes: https://reviews.apache.org/r/58085/diff/1-2/
Testing
-------
Test is done locally. You can see the beeline output now.
0: jdbc:hive2://localhost:10000> select t1.key from src t1 join src t2 on t1.key=t2.key limit 10;
INFO : Compiling command(queryId=axu_20170330125216_0de3bbf7-60f5-476d-b7eb-9861891d2961): select t1.key from src t1 join src t2 on t1.key=t2.key limit 10
INFO : Semantic Analysis Completed
INFO : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:t1.key, type:string, comment:null)], properties:null)
INFO : Completed compiling command(queryId=axu_20170330125216_0de3bbf7-60f5-476d-b7eb-9861891d2961); Time taken: 0.219 seconds
INFO : Concurrency mode is disabled, not creating a lock manager
INFO : Executing command(queryId=axu_20170330125216_0de3bbf7-60f5-476d-b7eb-9861891d2961): select t1.key from src t1 join src t2 on t1.key=t2.key limit 10
WARN : Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.
INFO : Query ID = axu_20170330125216_0de3bbf7-60f5-476d-b7eb-9861891d2961
INFO : Total jobs = 1
INFO : Starting task [Stage-4:MAPREDLOCAL] in serial mode
INFO : Execution completed successfully
INFO : MapredLocal task succeeded
INFO : Launching Job 1 out of 1
INFO : Starting task [Stage-3:MAPRED] in serial mode
INFO : Number of reduce tasks is set to 0 since there's no reduce operator
INFO : Starting Job = job_local1894165710_0002, Tracking URL = http://localhost:8080/
INFO : Kill Command = /Users/axu/Documents/workspaces/tools/hadoop/hadoop-2.6.0/bin/hadoop job -kill job_local1894165710_0002
INFO : Hadoop job information for Stage-3: number of mappers: 0; number of reducers: 0
INFO : 2017-03-30 12:52:21,788 Stage-3 map = 0%, reduce = 0%
ERROR : Ended Job = job_local1894165710_0002 with errors
ERROR : FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
INFO : MapReduce Jobs Launched:
INFO : Stage-Stage-3: HDFS Read: 0 HDFS Write: 0 FAIL
INFO : Total MapReduce CPU Time Spent: 0 msec
Thanks,
Aihua Xu