You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@logging.apache.org by Gary Gregory <ga...@gmail.com> on 2017/07/21 01:06:34 UTC
Fwd: Review Request 61010: HIVE-17128 Operation Logging leaks file
descriptors as the log4j Appender is never closed
In the Apache Hive message below, you'll see mention of using reflection to
implement something I thought we already supported: closing files file that
are dynamically created.
Any thoughts on how set get Hive to use Log4j in a safer manner?
Gary
---------- Forwarded message ----------
From: Aihua Xu <ax...@cloudera.com>
Date: Thu, Jul 20, 2017 at 4:59 PM
Subject: Re: Review Request 61010: HIVE-17128 Operation Logging leaks file
descriptors as the log4j Appender is never closed
To: hive <de...@hive.apache.org>, Aihua Xu <ax...@cloudera.com>, Andrew Sherman
<as...@cloudera.com>
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61010/#review181083
-----------------------------------------------------------
service/src/java/org/apache/hive/service/cli/operation/Operation.java
Lines 269 (patched)
<https://reviews.apache.org/r/61010/#comment256514>
We only register the Operation log appender once for all the operation
logs at the beginning. Now if you stop the appender here, then the
following queries will not be able to output to the appender any more,
right?
Can you test your patch by: connect to HS2 from one beeline session,
disconnect and reconnect. Then see if you still see output from the beeline
console?
Will we be able to close OutputStream instead? stopQueryAppender()
should be called when HS2 service gets shutdown.
- Aihua Xu
On July 20, 2017, 10:45 p.m., Andrew Sherman wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61010/
> -----------------------------------------------------------
>
> (Updated July 20, 2017, 10:45 p.m.)
>
>
> Review request for hive.
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> Previously HIVE-16061 and HIVE-16400 changed Operation Logging to use the
Log4j2
> RoutingAppender to automatically output the log for each query into an
> individual operation log file. As log4j does not know when a query is
finished
> it keeps the OutputStream in the Appender open even when the query
completes.
> The stream holds a file descriptor and so we leak file descriptors. Note
that we
> are already careful to close any streams reading from the operation log
file.
> To fix this we use a technique described in the comments of LOG4J2-510
which
> uses reflection to close the appender. The test in
TestOperationLoggingLayout is
> extended to check that the Appender is closed.
>
>
> Diffs
> -----
>
> itests/hive-unit/src/test/java/org/apache/hive/service/cli/operation/
TestOperationLoggingLayout.java 1a8337f574bb753e8c3c48a6b477b17700b05256
> ql/src/java/org/apache/hadoop/hive/ql/log/LogDivertAppender.java
e697b545984555414e27bafe92d7f22829a22687
> service/src/java/org/apache/hive/service/cli/operation/Operation.java
8d453d5d9153c2ec86c4adc7a68bd3b5dd249743
>
>
> Diff: https://reviews.apache.org/r/61010/diff/1/
>
>
> Testing
> -------
>
> Hand testing to show leak has gone.
> The test in TestOperationLoggingLayout is extended to check that the
Appender is closed.
>
>
> Thanks,
>
> Andrew Sherman
>
>