You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by "Moore, Douglas" <Do...@thinkbiganalytics.com> on 2015/04/06 18:15:18 UTC
Over-logging by ORC packages
On a cluster recently upgraded to Hive 0.14 (HDP 2.2) we found that Gigabytes and millions more INFO level hive.log entries from ORC packages were being logged.
I feel these log entries should be at the DEBUG level.
Is there an existing bug in Hive or ORC?
Here is one example:
2015-04-06 15:12:43,212 INFO orc.OrcInputFormat (OrcInputFormat.java:setSearchArgument(298)) - ORC pushdown predicate: leaf-0 = (EQUALS company XYZ)
leaf-1 = (EQUALS site DEF)
leaf-2 = (EQUALS table ABC)
expr = (and leaf-0 leaf-1 leaf-2)
To get an acceptable amount of logging that did not fill /tmp we had to add these entries to /etc/hive/conf/hive-log4j.settings:
log4j.logger.org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger=WARN,DRFA
log4j.logger.org.apache.hadoop.hive.ql.io.orc.ReaderImpl=WARN,DRFA
log4j.logger.org.apache.hadoop.hive.ql.io.orc.OrcInputFormat=WARN,DRFA
While I'm on the subject, to operationally harden Hive, I think Hive should use a more aggressive rolling file appender by default, one that can roll hourly or max size, compress the rolled logs…
- Douglas
Re: Over-logging by ORC packages
Posted by "Moore, Douglas" <Do...@thinkbiganalytics.com>.
Owen, we're seeing a millions of those log entries.
There are three, one for each package listed below in the revised hive-log4j.settings., one full example provided below.
Seems to repeat fewer than per-row (that would be billions). Perhaps repeats for each and every partition in a table (1000's to 10'000s).
To reproduce, create a y/m/d partitioned table and do select * from <table> where year=2015 and month=4 and day = 6 limit 1;
- Douglas
From: Owen O'Malley <om...@apache.org>>
Reply-To: <us...@hive.apache.org>>
Date: Mon, 6 Apr 2015 11:13:28 -0700
To: "user@hive.apache.org<ma...@hive.apache.org>" <us...@hive.apache.org>>
Subject: Re: Over-logging by ORC packages
Sorry for the excessive logging. The pushdown logging should only be at the start, is there a particular message that was being repeated per a row?
Thanks,
Owen
On Mon, Apr 6, 2015 at 9:15 AM, Moore, Douglas <Do...@thinkbiganalytics.com>> wrote:
On a cluster recently upgraded to Hive 0.14 (HDP 2.2) we found that Gigabytes and millions more INFO level hive.log entries from ORC packages were being logged.
I feel these log entries should be at the DEBUG level.
Is there an existing bug in Hive or ORC?
Here is one example:
2015-04-06 15:12:43,212 INFO orc.OrcInputFormat (OrcInputFormat.java:setSearchArgument(298)) - ORC pushdown predicate: leaf-0 = (EQUALS company XYZ)
leaf-1 = (EQUALS site DEF)
leaf-2 = (EQUALS table ABC)
expr = (and leaf-0 leaf-1 leaf-2)
To get an acceptable amount of logging that did not fill /tmp we had to add these entries to /etc/hive/conf/hive-log4j.settings:
log4j.logger.org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger=WARN,DRFA
log4j.logger.org.apache.hadoop.hive.ql.io.orc.ReaderImpl=WARN,DRFA
log4j.logger.org.apache.hadoop.hive.ql.io.orc.OrcInputFormat=WARN,DRFA
While I'm on the subject, to operationally harden Hive, I think Hive should use a more aggressive rolling file appender by default, one that can roll hourly or max size, compress the rolled logs…
- Douglas
Re: Over-logging by ORC packages
Posted by Owen O'Malley <om...@apache.org>.
Sorry for the excessive logging. The pushdown logging should only be at the
start, is there a particular message that was being repeated per a row?
Thanks,
Owen
On Mon, Apr 6, 2015 at 9:15 AM, Moore, Douglas <
Douglas.Moore@thinkbiganalytics.com> wrote:
> On a cluster recently upgraded to Hive 0.14 (HDP 2.2) we found that
> Gigabytes and millions more INFO level hive.log entries from ORC packages
> were being logged.
>
> I feel these log entries should be at the DEBUG level.
>
> Is there an existing bug in Hive or ORC?
>
> Here is one example:
>
> 2015-04-06 15:12:43,212 INFO orc.OrcInputFormat
> (OrcInputFormat.java:setSearchArgument(298)) - ORC pushdown predicate:
> leaf-0 = (EQUALS company XYZ)
>
> leaf-1 = (EQUALS site DEF)
>
> leaf-2 = (EQUALS table ABC)
>
> expr = (and leaf-0 leaf-1 leaf-2)
>
>
> To get an acceptable amount of logging that did not fill /tmp we had to
> add these entries to /etc/hive/conf/hive-log4j.settings:
>
> log4j.logger.org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger=WARN,DRFA
>
> log4j.logger.org.apache.hadoop.hive.ql.io.orc.ReaderImpl=WARN,DRFA
>
> log4j.logger.org.apache.hadoop.hive.ql.io.orc.OrcInputFormat=WARN,DRFA
>
>
>
> While I'm on the subject, to operationally harden Hive, I think Hive
> should use a more aggressive rolling file appender by default, one that
> can roll hourly or max size, compress the rolled logs…
>
>
> - Douglas
>