You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Owen O'Malley (JIRA)" <ji...@apache.org> on 2009/09/21 23:07:16 UTC
[jira] Created: (MAPREDUCE-1016) Make the format of the Job History
be JSON instead of Avro binary
Make the format of the Job History be JSON instead of Avro binary
-----------------------------------------------------------------
Key: MAPREDUCE-1016
URL: https://issues.apache.org/jira/browse/MAPREDUCE-1016
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Owen O'Malley
Fix For: 0.21.0, 0.22.0
I forgot that one of the features that would be nice is to off load the job history display from the JobTracker. That will be a lot easier, if the job history is stored in JSON. Therefore, I think we should change the storage now to prevent incompatibilities later.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1016) Make the format of the Job History
be JSON instead of Avro binary
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Doug Cutting updated MAPREDUCE-1016:
------------------------------------
Attachment: MAPREDUCE-1016.patch
Here's a new version of this patch that also upgrades to Avro 1.2.0.
In Avro 1.2, generated classes are no longer all nested within an interface, which mandated a systematic update to references to the generated classes.
> Make the format of the Job History be JSON instead of Avro binary
> -----------------------------------------------------------------
>
> Key: MAPREDUCE-1016
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1016
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: Owen O'Malley
> Assignee: Doug Cutting
> Fix For: 0.21.0, 0.22.0
>
> Attachments: MAPREDUCE-1016.patch, MAPREDUCE-1016.patch
>
>
> I forgot that one of the features that would be nice is to off load the job history display from the JobTracker. That will be a lot easier, if the job history is stored in JSON. Therefore, I think we should change the storage now to prevent incompatibilities later.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1016) Make the format of the Job
History be JSON instead of Avro binary
Posted by "Sharad Agarwal (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12763484#action_12763484 ]
Sharad Agarwal commented on MAPREDUCE-1016:
-------------------------------------------
bq. Or we can strip out the schema string and put it in a TFile metadata section.
That would require the parsing of schema string to figure out that the schema matches across the files before merging, no ?
I agree that there is a benefit of making the files self contained. Ideally I would like to remove the redundancy and make it self contained by pointing to a schema file in the similar way as done for schema references in xml files. Can we somehow make it available remotely via url ? svn view url comes to my mind. Does that sound too awkward?
> Make the format of the Job History be JSON instead of Avro binary
> -----------------------------------------------------------------
>
> Key: MAPREDUCE-1016
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1016
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: Owen O'Malley
> Assignee: Doug Cutting
> Fix For: 0.21.0, 0.22.0
>
> Attachments: MAPREDUCE-1016.patch
>
>
> I forgot that one of the features that would be nice is to off load the job history display from the JobTracker. That will be a lot easier, if the job history is stored in JSON. Therefore, I think we should change the storage now to prevent incompatibilities later.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (MAPREDUCE-1016) Make the format of the Job
History be JSON instead of Avro binary
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Owen O'Malley reassigned MAPREDUCE-1016:
----------------------------------------
Assignee: Doug Cutting
> Make the format of the Job History be JSON instead of Avro binary
> -----------------------------------------------------------------
>
> Key: MAPREDUCE-1016
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1016
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: Owen O'Malley
> Assignee: Doug Cutting
> Fix For: 0.21.0, 0.22.0
>
>
> I forgot that one of the features that would be nice is to off load the job history display from the JobTracker. That will be a lot easier, if the job history is stored in JSON. Therefore, I think we should change the storage now to prevent incompatibilities later.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1016) Make the format of the Job History
be JSON instead of Avro binary
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Doug Cutting updated MAPREDUCE-1016:
------------------------------------
Status: Open (was: Patch Available)
> Make the format of the Job History be JSON instead of Avro binary
> -----------------------------------------------------------------
>
> Key: MAPREDUCE-1016
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1016
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: Owen O'Malley
> Assignee: Doug Cutting
> Fix For: 0.21.0, 0.22.0
>
> Attachments: MAPREDUCE-1016.patch, MAPREDUCE-1016.patch
>
>
> I forgot that one of the features that would be nice is to off load the job history display from the JobTracker. That will be a lot easier, if the job history is stored in JSON. Therefore, I think we should change the storage now to prevent incompatibilities later.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1016) Make the format of the Job
History be JSON instead of Avro binary
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12763117#action_12763117 ]
Doug Cutting commented on MAPREDUCE-1016:
-----------------------------------------
Note that if we add or remove a field we should not need to change the version. Even if we use a totally different schema, we could determine that by looking at the schema. So we should really only need to change the version string if we stop using Avro.
Note that even if this turns out to be wrong, we can always add a version number to the version string later. But adding one now seems like belt-and-suspenders: the schema is the version.
> Make the format of the Job History be JSON instead of Avro binary
> -----------------------------------------------------------------
>
> Key: MAPREDUCE-1016
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1016
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: Owen O'Malley
> Assignee: Doug Cutting
> Fix For: 0.21.0, 0.22.0
>
> Attachments: MAPREDUCE-1016.patch
>
>
> I forgot that one of the features that would be nice is to off load the job history display from the JobTracker. That will be a lot easier, if the job history is stored in JSON. Therefore, I think we should change the storage now to prevent incompatibilities later.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1016) Make the format of the Job
History be JSON instead of Avro binary
Posted by "Hong Tang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12763441#action_12763441 ]
Hong Tang commented on MAPREDUCE-1016:
--------------------------------------
No, this is probably harder to manage. What we could do is (not sure if this is close to what Doug said) that we can periodically group a bunch of job history files into a single TFile. Hopefully compression would take care of the redundancy in the schema string. Or we can strip out the schema string and put it in a TFile metadata section.
> Make the format of the Job History be JSON instead of Avro binary
> -----------------------------------------------------------------
>
> Key: MAPREDUCE-1016
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1016
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: Owen O'Malley
> Assignee: Doug Cutting
> Fix For: 0.21.0, 0.22.0
>
> Attachments: MAPREDUCE-1016.patch
>
>
> I forgot that one of the features that would be nice is to off load the job history display from the JobTracker. That will be a lot easier, if the job history is stored in JSON. Therefore, I think we should change the storage now to prevent incompatibilities later.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1016) Make the format of the Job
History be JSON instead of Avro binary
Posted by "Sharad Agarwal (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12763437#action_12763437 ]
Sharad Agarwal commented on MAPREDUCE-1016:
-------------------------------------------
bq. Line 3: "Schema: <schema string>"
Adding event schema to each file may be too verbose, especially for small jobs. We can add the mapreduce release version instead. The event schema will always be corresponding to the mapreduce version.
> Make the format of the Job History be JSON instead of Avro binary
> -----------------------------------------------------------------
>
> Key: MAPREDUCE-1016
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1016
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: Owen O'Malley
> Assignee: Doug Cutting
> Fix For: 0.21.0, 0.22.0
>
> Attachments: MAPREDUCE-1016.patch
>
>
> I forgot that one of the features that would be nice is to off load the job history display from the JobTracker. That will be a lot easier, if the job history is stored in JSON. Therefore, I think we should change the storage now to prevent incompatibilities later.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1016) Make the format of the Job
History be JSON instead of Avro binary
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12769553#action_12769553 ]
Hudson commented on MAPREDUCE-1016:
-----------------------------------
Integrated in Hadoop-Mapreduce-trunk-Commit #96 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/96/])
. Make the job history log format JSON.
> Make the format of the Job History be JSON instead of Avro binary
> -----------------------------------------------------------------
>
> Key: MAPREDUCE-1016
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1016
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: Owen O'Malley
> Assignee: Doug Cutting
> Fix For: 0.21.0
>
> Attachments: MAPREDUCE-1016.patch, MAPREDUCE-1016.patch
>
>
> I forgot that one of the features that would be nice is to off load the job history display from the JobTracker. That will be a lot easier, if the job history is stored in JSON. Therefore, I think we should change the storage now to prevent incompatibilities later.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1016) Make the format of the Job
History be JSON instead of Avro binary
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12758132#action_12758132 ]
Owen O'Malley commented on MAPREDUCE-1016:
------------------------------------------
The use case that I'm considering is that we have an AJAX client that reads the job history via hftp, parses it and displays it to the user. The only work the JobTracker needs to do is give out the URL for getting the job history file. The rest is done in JavaScript on the client's browser. Until we have an Avro library in JavaScript, it will be easier to do if the job history is in JSON.
> Make the format of the Job History be JSON instead of Avro binary
> -----------------------------------------------------------------
>
> Key: MAPREDUCE-1016
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1016
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: Owen O'Malley
> Assignee: Doug Cutting
> Fix For: 0.21.0, 0.22.0
>
>
> I forgot that one of the features that would be nice is to off load the job history display from the JobTracker. That will be a lot easier, if the job history is stored in JSON. Therefore, I think we should change the storage now to prevent incompatibilities later.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1016) Make the format of the Job
History be JSON instead of Avro binary
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12763617#action_12763617 ]
Doug Cutting commented on MAPREDUCE-1016:
-----------------------------------------
> In this jira, comparing with the current code, I am only proposing breaking down the first line (version string) into two lines and add an empty line after the schema (and add a few keywords).
I think that is overkill. The point of the change to the log format was to make it trivial for external tools to process these logs. We don't want to make folks have to write a header processor. Most folks will probably just skip the first two lines and use a json parser. If they want to do some error-checking, they can check that the first line is "Avro-Json". Further, if they want, they can read the schema and use Avro. If we permit arbitrary headers then folks have to skip up until a blank line, which harder, although not much. But if they want to check the format or use Avro then they have to write a header parser, and logic to process the header, which I think is an unneeded imposition.
The magic signature for this format is "Avro-Json\n". The format is:
- first line is magic
- second line is an Avro schema
- subsequent lines are json-encoded instances of that schema
I see no need for a more complex format at this time. The only change from the original Json-based format when Avro was added was changing the magic line to not have a version number and adding a line with the schema, which is, in essence, the version number. If you think we should have a more general, header-based format, please file a separate issue.
> Make the format of the Job History be JSON instead of Avro binary
> -----------------------------------------------------------------
>
> Key: MAPREDUCE-1016
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1016
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: Owen O'Malley
> Assignee: Doug Cutting
> Fix For: 0.21.0, 0.22.0
>
> Attachments: MAPREDUCE-1016.patch
>
>
> I forgot that one of the features that would be nice is to off load the job history display from the JobTracker. That will be a lot easier, if the job history is stored in JSON. Therefore, I think we should change the storage now to prevent incompatibilities later.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1016) Make the format of the Job
History be JSON instead of Avro binary
Posted by "Hong Tang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12763153#action_12763153 ]
Hong Tang commented on MAPREDUCE-1016:
--------------------------------------
Maybe I miss something. I am considering a situation where the JobHistory event schemas are significantly changed and we want to prevent older version of the software from reading job-history logs produced by the newer version of the software. This is consistent with your comments here: https://issues.apache.org/jira/browse/MAPREDUCE-980?focusedCommentId=12757389&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#action_12757389.
Again, this would be part of the JobHistory format specification and I think it would make sense to normalize the format of the version string. Are you saying that maybe we will never need to have a "avro-json-2.0"?
Also, what is your opinion on whether job-history format would be fixed on a specific encoder permanently?
> Make the format of the Job History be JSON instead of Avro binary
> -----------------------------------------------------------------
>
> Key: MAPREDUCE-1016
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1016
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: Owen O'Malley
> Assignee: Doug Cutting
> Fix For: 0.21.0, 0.22.0
>
> Attachments: MAPREDUCE-1016.patch
>
>
> I forgot that one of the features that would be nice is to off load the job history display from the JobTracker. That will be a lot easier, if the job history is stored in JSON. Therefore, I think we should change the storage now to prevent incompatibilities later.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
Re: [jira] Commented: (MAPREDUCE-1016) Make the format of the Job
History be JSON instead of Avro binary
Posted by Milind A Bhandarkar <mi...@yahoo-inc.com>.
My reason for reverting the change to jobhistory in avro is more
straightforward. Text is good. It has always been good. Even average
humans can read text. And jobhistory is something we, as average
humans, read a hundred times a day (and we think that every hadoop
user should read their job history, which contains a lot of
information about their jobs). Along with job configuration.
I have made my dissatisfaction with small-stuff stored as compact
binaries known on a very public forum, facebook, known already.
On Sep 21, 2009, at 20:42, "Jothi Padmanabhan (JIRA)"
<ji...@apache.org> wrote:
>
> [ https://issues.apache.org/jira/browse/MAPREDUCE-1016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12758109#action_12758109
> ]
>
> Jothi Padmanabhan commented on MAPREDUCE-1016:
> ----------------------------------------------
>
> I think storing it in JSON is a good idea too. Nevertheless, I think
> the best way for other history consumers/tools to insulate
> themselves from the underlying changes/incompatibilities would be to
> use the JobHistoryParsing API's. At the least, they will abstract
> out the underlying storage format.
>
>> Make the format of the Job History be JSON instead of Avro binary
>> -----------------------------------------------------------------
>>
>> Key: MAPREDUCE-1016
>> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1016
>> Project: Hadoop Map/Reduce
>> Issue Type: Bug
>> Reporter: Owen O'Malley
>> Assignee: Doug Cutting
>> Fix For: 0.21.0, 0.22.0
>>
>>
>> I forgot that one of the features that would be nice is to off load
>> the job history display from the JobTracker. That will be a lot
>> easier, if the job history is stored in JSON. Therefore, I think we
>> should change the storage now to prevent incompatibilities later.
>
> --
> This message is automatically generated by JIRA.
> -
> You can reply to this email to add a comment to the issue online.
>
[jira] Commented: (MAPREDUCE-1016) Make the format of the Job
History be JSON instead of Avro binary
Posted by "Jothi Padmanabhan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12758109#action_12758109 ]
Jothi Padmanabhan commented on MAPREDUCE-1016:
----------------------------------------------
I think storing it in JSON is a good idea too. Nevertheless, I think the best way for other history consumers/tools to insulate themselves from the underlying changes/incompatibilities would be to use the JobHistoryParsing API's. At the least, they will abstract out the underlying storage format.
> Make the format of the Job History be JSON instead of Avro binary
> -----------------------------------------------------------------
>
> Key: MAPREDUCE-1016
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1016
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: Owen O'Malley
> Assignee: Doug Cutting
> Fix For: 0.21.0, 0.22.0
>
>
> I forgot that one of the features that would be nice is to off load the job history display from the JobTracker. That will be a lot easier, if the job history is stored in JSON. Therefore, I think we should change the storage now to prevent incompatibilities later.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1016) Make the format of the Job History
be JSON instead of Avro binary
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Doug Cutting updated MAPREDUCE-1016:
------------------------------------
Resolution: Fixed
Fix Version/s: (was: 0.22.0)
Status: Resolved (was: Patch Available)
I just committed this.
> Make the format of the Job History be JSON instead of Avro binary
> -----------------------------------------------------------------
>
> Key: MAPREDUCE-1016
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1016
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: Owen O'Malley
> Assignee: Doug Cutting
> Fix For: 0.21.0
>
> Attachments: MAPREDUCE-1016.patch, MAPREDUCE-1016.patch
>
>
> I forgot that one of the features that would be nice is to off load the job history display from the JobTracker. That will be a lot easier, if the job history is stored in JSON. Therefore, I think we should change the storage now to prevent incompatibilities later.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1016) Make the format of the Job History
be JSON instead of Avro binary
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Doug Cutting updated MAPREDUCE-1016:
------------------------------------
Attachment: MAPREDUCE-1016.patch
Here's a patch that implements this.
> Make the format of the Job History be JSON instead of Avro binary
> -----------------------------------------------------------------
>
> Key: MAPREDUCE-1016
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1016
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: Owen O'Malley
> Assignee: Doug Cutting
> Fix For: 0.21.0, 0.22.0
>
> Attachments: MAPREDUCE-1016.patch
>
>
> I forgot that one of the features that would be nice is to off load the job history display from the JobTracker. That will be a lot easier, if the job history is stored in JSON. Therefore, I think we should change the storage now to prevent incompatibilities later.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1016) Make the format of the Job
History be JSON instead of Avro binary
Posted by "Hong Tang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12768898#action_12768898 ]
Hong Tang commented on MAPREDUCE-1016:
--------------------------------------
bq. If you think we should have a more general, header-based format, please file a separate issue.
I created MAPREDUCE-1135 to track my proposal.
> Make the format of the Job History be JSON instead of Avro binary
> -----------------------------------------------------------------
>
> Key: MAPREDUCE-1016
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1016
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: Owen O'Malley
> Assignee: Doug Cutting
> Fix For: 0.21.0, 0.22.0
>
> Attachments: MAPREDUCE-1016.patch, MAPREDUCE-1016.patch
>
>
> I forgot that one of the features that would be nice is to off load the job history display from the JobTracker. That will be a lot easier, if the job history is stored in JSON. Therefore, I think we should change the storage now to prevent incompatibilities later.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1016) Make the format of the Job
History be JSON instead of Avro binary
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12763551#action_12763551 ]
Doug Cutting commented on MAPREDUCE-1016:
-----------------------------------------
> I'd like to propose the following that may make it easier toward future extension without requiring too much special case code to support backward compatibility [ ... ]
I think we're over-engineering this. Can we take this to a separate issue?
> Make the format of the Job History be JSON instead of Avro binary
> -----------------------------------------------------------------
>
> Key: MAPREDUCE-1016
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1016
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: Owen O'Malley
> Assignee: Doug Cutting
> Fix For: 0.21.0, 0.22.0
>
> Attachments: MAPREDUCE-1016.patch
>
>
> I forgot that one of the features that would be nice is to off load the job history display from the JobTracker. That will be a lot easier, if the job history is stored in JSON. Therefore, I think we should change the storage now to prevent incompatibilities later.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1016) Make the format of the Job
History be JSON instead of Avro binary
Posted by "Milind Bhandarkar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12758048#action_12758048 ]
Milind Bhandarkar commented on MAPREDUCE-1016:
----------------------------------------------
Oh Thank you Owen ! Thank you, thank you, thank you !
> Make the format of the Job History be JSON instead of Avro binary
> -----------------------------------------------------------------
>
> Key: MAPREDUCE-1016
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1016
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: Owen O'Malley
> Assignee: Doug Cutting
> Fix For: 0.21.0, 0.22.0
>
>
> I forgot that one of the features that would be nice is to off load the job history display from the JobTracker. That will be a lot easier, if the job history is stored in JSON. Therefore, I think we should change the storage now to prevent incompatibilities later.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1016) Make the format of the Job
History be JSON instead of Avro binary
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12768880#action_12768880 ]
Doug Cutting commented on MAPREDUCE-1016:
-----------------------------------------
Can someone please review this before it goes stale?
> Make the format of the Job History be JSON instead of Avro binary
> -----------------------------------------------------------------
>
> Key: MAPREDUCE-1016
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1016
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: Owen O'Malley
> Assignee: Doug Cutting
> Fix For: 0.21.0, 0.22.0
>
> Attachments: MAPREDUCE-1016.patch, MAPREDUCE-1016.patch
>
>
> I forgot that one of the features that would be nice is to off load the job history display from the JobTracker. That will be a lot easier, if the job history is stored in JSON. Therefore, I think we should change the storage now to prevent incompatibilities later.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1016) Make the format of the Job
History be JSON instead of Avro binary
Posted by "Hong Tang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12763420#action_12763420 ]
Hong Tang commented on MAPREDUCE-1016:
--------------------------------------
I'd like to propose the following that may make it easier toward future extension without requiring too much special case code to support backward compatibility.
Line 1: The version string. We, for now, would require an exact match. And it should be "Avro". In the future, it could be "Avro/2.0".
Line 2: "Encoder: Json" (Or "Encoder: Binary")
Line 3: "Schema: <schema string>"
Line 4: Empty line
Remaining file: encoded history-event objects.
Essentially, the structure of this file mimic the HTTP style:
Line 1: Protocol + Version. For JobHistoryParser to determine whether it can handle the input, and if so, what backend deserialization framework to invoke.
Line 2 until the empty line: Header. These are parameters that are needed to instantiate the deserializer.
Remainder of the file: serialized data.
Does that make sense?
> Make the format of the Job History be JSON instead of Avro binary
> -----------------------------------------------------------------
>
> Key: MAPREDUCE-1016
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1016
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: Owen O'Malley
> Assignee: Doug Cutting
> Fix For: 0.21.0, 0.22.0
>
> Attachments: MAPREDUCE-1016.patch
>
>
> I forgot that one of the features that would be nice is to off load the job history display from the JobTracker. That will be a lot easier, if the job history is stored in JSON. Therefore, I think we should change the storage now to prevent incompatibilities later.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1016) Make the format of the Job
History be JSON instead of Avro binary
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12758827#action_12758827 ]
Doug Cutting commented on MAPREDUCE-1016:
-----------------------------------------
Test failure is unrelated and failing for other tasks.
No new tests are required for this change.
> Make the format of the Job History be JSON instead of Avro binary
> -----------------------------------------------------------------
>
> Key: MAPREDUCE-1016
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1016
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: Owen O'Malley
> Assignee: Doug Cutting
> Fix For: 0.21.0, 0.22.0
>
> Attachments: MAPREDUCE-1016.patch
>
>
> I forgot that one of the features that would be nice is to off load the job history display from the JobTracker. That will be a lot easier, if the job history is stored in JSON. Therefore, I think we should change the storage now to prevent incompatibilities later.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1016) Make the format of the Job History
be JSON instead of Avro binary
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Doug Cutting updated MAPREDUCE-1016:
------------------------------------
Status: Patch Available (was: Open)
> Make the format of the Job History be JSON instead of Avro binary
> -----------------------------------------------------------------
>
> Key: MAPREDUCE-1016
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1016
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: Owen O'Malley
> Assignee: Doug Cutting
> Fix For: 0.21.0, 0.22.0
>
> Attachments: MAPREDUCE-1016.patch
>
>
> I forgot that one of the features that would be nice is to off load the job history display from the JobTracker. That will be a lot easier, if the job history is stored in JSON. Therefore, I think we should change the storage now to prevent incompatibilities later.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1016) Make the format of the Job
History be JSON instead of Avro binary
Posted by "Hong Tang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12763602#action_12763602 ]
Hong Tang commented on MAPREDUCE-1016:
--------------------------------------
bq. I think we're over-engineering this. Can we take this to a separate issue?
Sure. In this jira, comparing with the current code, I am only proposing breaking down the first line (version string) into two lines and add an empty line after the schema (and add a few keywords). This could make the format easily extended to a more elaborated schema in the future without any special code handling backward/forward compatibility.
bq. That would require the parsing of schema string to figure out that the schema matches across the files before merging, no ?
Yes/No. The one stored in TFile would be a default one. If a job history has a different schema than that default, then, for that entry, we can keep the "Schema: " line in the data.
bq. Can we somehow make it available remotely via url ?
One of the principal advantage of Avro versus Protocol Buffer is that it is completely self-contained, This seems a step backwards.
> Make the format of the Job History be JSON instead of Avro binary
> -----------------------------------------------------------------
>
> Key: MAPREDUCE-1016
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1016
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: Owen O'Malley
> Assignee: Doug Cutting
> Fix For: 0.21.0, 0.22.0
>
> Attachments: MAPREDUCE-1016.patch
>
>
> I forgot that one of the features that would be nice is to off load the job history display from the JobTracker. That will be a lot easier, if the job history is stored in JSON. Therefore, I think we should change the storage now to prevent incompatibilities later.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1016) Make the format of the Job
History be JSON instead of Avro binary
Posted by "Sharad Agarwal (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12769132#action_12769132 ]
Sharad Agarwal commented on MAPREDUCE-1016:
-------------------------------------------
Patch looks good. Could you update the eclipse .classpath template as well when you commit?
> Make the format of the Job History be JSON instead of Avro binary
> -----------------------------------------------------------------
>
> Key: MAPREDUCE-1016
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1016
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: Owen O'Malley
> Assignee: Doug Cutting
> Fix For: 0.21.0, 0.22.0
>
> Attachments: MAPREDUCE-1016.patch, MAPREDUCE-1016.patch
>
>
> I forgot that one of the features that would be nice is to off load the job history display from the JobTracker. That will be a lot easier, if the job history is stored in JSON. Therefore, I think we should change the storage now to prevent incompatibilities later.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1016) Make the format of the Job
History be JSON instead of Avro binary
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12758068#action_12758068 ]
Arun C Murthy commented on MAPREDUCE-1016:
------------------------------------------
+1
> Make the format of the Job History be JSON instead of Avro binary
> -----------------------------------------------------------------
>
> Key: MAPREDUCE-1016
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1016
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: Owen O'Malley
> Assignee: Doug Cutting
> Fix For: 0.21.0, 0.22.0
>
>
> I forgot that one of the features that would be nice is to off load the job history display from the JobTracker. That will be a lot easier, if the job history is stored in JSON. Therefore, I think we should change the storage now to prevent incompatibilities later.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1016) Make the format of the Job
History be JSON instead of Avro binary
Posted by "Hong Tang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12762994#action_12762994 ]
Hong Tang commented on MAPREDUCE-1016:
--------------------------------------
I think we are misusing version string here. I suggest the version string be "{serializer}-{encoder}-{version}". E.g. "avro-json-1.0". EventReader should compare only the {serializer} and {version} part of the string to determine whether a job history file matches (or is compatible with) the schema of the HistoryInfo object it is about to create - and it would use the {encoder} string to decide what decoder should be used.
> Make the format of the Job History be JSON instead of Avro binary
> -----------------------------------------------------------------
>
> Key: MAPREDUCE-1016
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1016
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: Owen O'Malley
> Assignee: Doug Cutting
> Fix For: 0.21.0, 0.22.0
>
> Attachments: MAPREDUCE-1016.patch
>
>
> I forgot that one of the features that would be nice is to off load the job history display from the JobTracker. That will be a lot easier, if the job history is stored in JSON. Therefore, I think we should change the storage now to prevent incompatibilities later.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1016) Make the format of the Job
History be JSON instead of Avro binary
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12766811#action_12766811 ]
Hadoop QA commented on MAPREDUCE-1016:
--------------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12422414/MAPREDUCE-1016.patch
against trunk revision 825469.
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/78/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/78/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/78/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/78/console
This message is automatically generated.
> Make the format of the Job History be JSON instead of Avro binary
> -----------------------------------------------------------------
>
> Key: MAPREDUCE-1016
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1016
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: Owen O'Malley
> Assignee: Doug Cutting
> Fix For: 0.21.0, 0.22.0
>
> Attachments: MAPREDUCE-1016.patch, MAPREDUCE-1016.patch
>
>
> I forgot that one of the features that would be nice is to off load the job history display from the JobTracker. That will be a lot easier, if the job history is stored in JSON. Therefore, I think we should change the storage now to prevent incompatibilities later.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1016) Make the format of the Job History
be JSON instead of Avro binary
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Doug Cutting updated MAPREDUCE-1016:
------------------------------------
Status: Patch Available (was: Open)
> Make the format of the Job History be JSON instead of Avro binary
> -----------------------------------------------------------------
>
> Key: MAPREDUCE-1016
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1016
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: Owen O'Malley
> Assignee: Doug Cutting
> Fix For: 0.21.0, 0.22.0
>
> Attachments: MAPREDUCE-1016.patch, MAPREDUCE-1016.patch
>
>
> I forgot that one of the features that would be nice is to off load the job history display from the JobTracker. That will be a lot easier, if the job history is stored in JSON. Therefore, I think we should change the storage now to prevent incompatibilities later.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1016) Make the format of the Job
History be JSON instead of Avro binary
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12758457#action_12758457 ]
Hadoop QA commented on MAPREDUCE-1016:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12420284/MAPREDUCE-1016.patch
against trunk revision 817740.
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed core unit tests.
-1 contrib tests. The patch failed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/123/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/123/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/123/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/123/console
This message is automatically generated.
> Make the format of the Job History be JSON instead of Avro binary
> -----------------------------------------------------------------
>
> Key: MAPREDUCE-1016
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1016
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: Owen O'Malley
> Assignee: Doug Cutting
> Fix For: 0.21.0, 0.22.0
>
> Attachments: MAPREDUCE-1016.patch
>
>
> I forgot that one of the features that would be nice is to off load the job history display from the JobTracker. That will be a lot easier, if the job history is stored in JSON. Therefore, I think we should change the storage now to prevent incompatibilities later.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1016) Make the format of the Job
History be JSON instead of Avro binary
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12763160#action_12763160 ]
Doug Cutting commented on MAPREDUCE-1016:
-----------------------------------------
> Are you saying that maybe we will never need to have a "avro-json-2.0"?
I think it unlikely but not impossible. If Avro incompatibly changed it's schema format then we'd probably change it to "Avro2-json" or somesuch. We'd need to change the version string if we changed the schema in some critical way that's impossible to detect by examining the schema itself, but I have a hard time imagining a change like that, and we could always instead, e.g., change the name or package of the top-level class in the schema to make such a change easy to detect. So it's unlikely but not impossible that we'll need to change the version string, and, if we do, we can append a version number to it. The version string already future-proofs us sufficiently.
> Also, what is your opinion on whether job-history format would be fixed on a specific encoder permanently?
Nothing's permanent. It seems possible we someday might want to permit binary too, so I added the encoding to the version string. But for now I think the proposal is to be json-only.
What seem more probable to me is that we might someday use a standard container format like TFile or Avro's data file. These can be differentiated by use of "magic": Avro data file always begins with 'AVR\0', the json file always begins with "Avro-Json\n", etc. So it's good that the file begins with a fixed string rather than just the schema, but exactly what string it begins with is less critical.
> Make the format of the Job History be JSON instead of Avro binary
> -----------------------------------------------------------------
>
> Key: MAPREDUCE-1016
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1016
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: Owen O'Malley
> Assignee: Doug Cutting
> Fix For: 0.21.0, 0.22.0
>
> Attachments: MAPREDUCE-1016.patch
>
>
> I forgot that one of the features that would be nice is to off load the job history display from the JobTracker. That will be a lot easier, if the job history is stored in JSON. Therefore, I think we should change the storage now to prevent incompatibilities later.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1016) Make the format of the Job
History be JSON instead of Avro binary
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12771506#action_12771506 ]
Hudson commented on MAPREDUCE-1016:
-----------------------------------
Integrated in Hadoop-Mapreduce-trunk #127 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/127/])
> Make the format of the Job History be JSON instead of Avro binary
> -----------------------------------------------------------------
>
> Key: MAPREDUCE-1016
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1016
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: Owen O'Malley
> Assignee: Doug Cutting
> Fix For: 0.21.0
>
> Attachments: MAPREDUCE-1016.patch, MAPREDUCE-1016.patch
>
>
> I forgot that one of the features that would be nice is to off load the job history display from the JobTracker. That will be a lot easier, if the job history is stored in JSON. Therefore, I think we should change the storage now to prevent incompatibilities later.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.