You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@chukwa.apache.org by "shreyas subramanya (JIRA)" <ji...@apache.org> on 2013/01/16 04:04:12 UTC

[jira] [Commented] (CHUKWA-679) Create WebHDFSWriter

    [ https://issues.apache.org/jira/browse/CHUKWA-679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13554689#comment-13554689 ] 

shreyas subramanya commented on CHUKWA-679:
-------------------------------------------

The idea is to append to text files (named based on chunk metadata) through webhdfs. The append requires 2 post operations - the latter being a url to datanode. Since we expect small updates to the file (like data from log4j appender), we want to cache the redirect url and keep posting to the same datanode until the configured "rotate interval" (like 1 hour). Then we create a new file, get a new datanode url to post to and continue.
                
> Create WebHDFSWriter
> --------------------
>
>                 Key: CHUKWA-679
>                 URL: https://issues.apache.org/jira/browse/CHUKWA-679
>             Project: Chukwa
>          Issue Type: Sub-task
>          Components: Data Processors
>    Affects Versions: 0.5.0, 0.6.0
>         Environment: Java 6
>            Reporter: shreyas subramanya
>   Original Estimate: 10h
>  Remaining Estimate: 10h
>
> Create a pipelineable writer which uses WebHDFS REST api.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira