You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flume.apache.org by "Jarek Jarcec Cecho (JIRA)" <ji...@apache.org> on 2012/07/31 17:59:33 UTC
[jira] [Commented] (FLUME-1391) Use sync() instead of syncFs() in
HDFS Sink to be compatible with hadoop 0.20.2
[ https://issues.apache.org/jira/browse/FLUME-1391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13425869#comment-13425869 ]
Jarek Jarcec Cecho commented on FLUME-1391:
-------------------------------------------
Committed and pushed to our new GIT repository. I'll close this ticket once I'll be able to assign it to Yongkun.
Jarcec
> Use sync() instead of syncFs() in HDFS Sink to be compatible with hadoop 0.20.2
> -------------------------------------------------------------------------------
>
> Key: FLUME-1391
> URL: https://issues.apache.org/jira/browse/FLUME-1391
> Project: Flume
> Issue Type: Improvement
> Components: Sinks+Sources
> Affects Versions: v1.1.0
> Reporter: Yongkun Wang
> Labels: hadoop
> Fix For: v1.3.0
>
> Attachments: HDFSSink-for-hadoop-0.20.2.patch
>
>
> For HDFS sink, the syncFs() is called in HDFSSequenceFile. But syncFs() is not available in legacy hadoop 0.20.2, which may be a widely used version. sync() method is available for all hadoop versions. And syncFs() is also implemented by sync() in hadoop (SequenceFile):
> {code}
> /** create a sync point */
> public void sync() throws IOException {
> if (sync != null && lastSyncPos != out.getPos()) {
> out.writeInt(SYNC_ESCAPE); // mark the start of the sync
> out.write(sync); // write sync
> lastSyncPos = out.getPos(); // update lastSyncPos
> }
> }
> /** flush all currently written data to the file system */
> public void syncFs() throws IOException {
> if (out != null) {
> out.sync(); // flush contents to file system
> }
> }
> {code}
> Therefore, using sync() in HDFSSequenceFile may be better.
> {code}
> @Override
> public void sync() throws IOException {
> //writer.syncFs(); //for hadoop 0.20.205.0+
> writer.sync(); //support hadoop 0.20.2+
> }
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira