[jira] [Updated] (GIRAPH-648) Allow IO formats to add parameters to Configuration

["Maja Kabiljo (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/GIRAPH-648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Maja Kabiljo updated GIRAPH-648:
--------------------------------

    Attachment: GIRAPH-648.patch

Unfortunately this is not as straightforward as it sounds, because methods from io formats, readers/writers/OutputCommitter have JobContext or TaskAttemptContext as an argument, and in some cases those hold the copy of Configuration, not the original. So I added a way to track which parameters where added to GiraphConfiguration, and wrapped all io related calls to append those parameters to JobContext/TaskAttemptContext before passing control to actual io formats.

Cleaned up HiveGiraphRunner and moved all control to its io formats, I can do similar for HCatalog in a separate patch.

This will also help us do GIRAPH-639 in a cleaner way, and it will actually be possible to mix different kind of input formats (hcat, hive, hbase, or whatever).
                
> Allow IO formats to add parameters to Configuration
> ---------------------------------------------------
>
>                 Key: GIRAPH-648
>                 URL: https://issues.apache.org/jira/browse/GIRAPH-648
>             Project: Giraph
>          Issue Type: Improvement
>            Reporter: Maja Kabiljo
>            Assignee: Maja Kabiljo
>         Attachments: GIRAPH-648.patch
>
>
> Currently we heavily rely on some runners (HCatGiraphRunner and HiveGiraphRunner) to prepare Configuration before application starts, and we have no way of using hcat/hive io without these runners. It would be better and more flexible if io formats would add what's needed for underlying io to Configuration themselves.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira