You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@hive.apache.org by Steven Wong <sw...@netflix.com> on 2011/06/02 03:31:55 UTC

Hive logging concurrency

By default, all Hive clients log to the same file called hive.log via DRFA. What I'm seeing is that many log lines are "lost" after hive.log is rolled over to hive.log.YYYY-MM-DD. Is this an issue with DRFA? What do folks do to avoid this problem when using concurrent Hive clients?

Thanks.
Steven

RE: Hive logging concurrency

Posted by Steven Wong <sw...@netflix.com>.

After researching on the web, the lost log lines is apparently an issue with DailyRollingFileAppender that may have been fixed in 1.2.16 (see https://issues.apache.org/bugzilla/show_bug.cgi?id=43374).

On the other hand, DailyRollingFileAppender's documentation recommends using RollingFileAppender to avoid "synchronization issues and data loss."

On the third hand, I really should look into using separate log files for separate Hive clients. Interleaved log lines from concurrent Hive clients make debugging difficult.

From: Steven Wong [mailto:swong@netflix.com]
Sent: Wednesday, June 01, 2011 6:32 PM
To: user@hive.apache.org
Subject: Hive logging concurrency

By default, all Hive clients log to the same file called hive.log via DRFA. What I'm seeing is that many log lines are "lost" after hive.log is rolled over to hive.log.YYYY-MM-DD. Is this an issue with DRFA? What do folks do to avoid this problem when using concurrent Hive clients?

Thanks.
Steven