You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@chukwa.apache.org by "Eric Yang (JIRA)" <ji...@apache.org> on 2015/04/05 03:54:33 UTC

[jira] [Created] (CHUKWA-744) Refactor ETL process for HBaseWriter

Eric Yang created CHUKWA-744:
--------------------------------

             Summary: Refactor ETL process for HBaseWriter
                 Key: CHUKWA-744
                 URL: https://issues.apache.org/jira/browse/CHUKWA-744
             Project: Chukwa
          Issue Type: Task
          Components: Data Processors
    Affects Versions: 0.6.0
            Reporter: Eric Yang
            Assignee: Eric Yang


The current ETL classes are based on Demux MapProcessor and ReduceProcessor.  The processors were designed to pass in archive key embedded in the processor as well as ChunkSaver to preserve chunks that can not be parsed.  This is fine when running map reduce based demux job for processing data.  The short lived task will spill out ChunkSaver into separate file for examination later.  However, the processors can generate memory leaks for long period of time in Chukwa agent because Chunks are saved in ChukwaSaver without clean up.

It would be better to redesign the parser classes with well defined behavior.  If the chunk can not be parsed, it should throw ParseException to upper layer for retry or log to agent log.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)