[jira] Issue Comment Edited: (MAPREDUCE-157) Job History log file format is not friendly for external tools.

["Owen O'Malley (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/MAPREDUCE-157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12742712#action_12742712 ] 

Owen O'Malley edited comment on MAPREDUCE-157 at 8/12/09 9:47 PM:
------------------------------------------------------------------

I'm confused what the goal of using Avro here would be.

Let's review the goals:
  1. Get an easily parseable text format.
  2. Not require excessive amounts of time for logging
     2a. Not require excessive object allocations.

It seems like to use Avro, we'd need to create the Avro objects and then write them out. I'd rather just use a JsonWriter to write the events out to the stream. Of course reading is the reverse. It would be like writing xml files by generating the necessary DOM objects. You can do it (and in fact Configuration is written that way. *sigh*), but it costs a lot of time.

Not having seen the Avro text format, I can't evaluation how much overhead it adds. None of the features of Avro seem compelling in this case, and could easily lead to unfortunate choices.

Furthermore, I don't know if there are any guarantees about the Avro text format's stability. We need stability in this format.

      was (Author: owen.omalley):
    I'm confused what the goal of using Avro here would be.

Let's review the goals:
  1. Get an easily parseable text format.
  2. Not require excessive amounts of time for logging
     2a. Not require excessive object allocations.

It seems like to use Avro, we'd need to create the Avro objects and then write them out. I'd rather just use a JsonWriter to write the events out to the stream. Of course reading is the reverse. I would be like writing xml files by generating the necessary DOM objects. You can do it (and in fact Configuration is written that way. *sigh*), but it costs a lot of time.

Not having seen the Avro text format, I can't evaluation how much overhead it adds. None of the features of Avro seem compelling in this case, and could easily lead to unfortunate choices.

Furthermore, I don't know if there are any guarantees about the Avro text format's stability. We need stability in this format.
  
> Job History log file format is not friendly for external tools.
> ---------------------------------------------------------------
>
>                 Key: MAPREDUCE-157
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-157
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>            Reporter: Owen O'Malley
>            Assignee: Jothi Padmanabhan
>
> Currently, parsing the job history logs with external tools is very difficult because of the format. The most critical problem is that newlines aren't escaped in the strings. That makes using tools like grep, sed, and awk very tricky.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.