You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flume.apache.org by "ASF GitHub Bot (JIRA)" <ji...@apache.org> on 2018/11/28 13:43:00 UTC
[jira] [Commented] (FLUME-3146) Use public API
HdfsDataOutputStream#getCurrentBlockReplication where applicable
[ https://issues.apache.org/jira/browse/FLUME-3146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16701896#comment-16701896 ]
ASF GitHub Bot commented on FLUME-3146:
---------------------------------------
GitHub user majorendre opened a pull request:
https://github.com/apache/flume/pull/250
FLUME-3146 Use public API HdfsDataOutputStream#getCurrentBlockReplica…
…tion where applicable
Took over this issue from Wei-Chiu Chuang. Added a few lines to the tests.
All tests pass, the new feature works.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/majorendre/flume FLUME-3146
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flume/pull/250.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #250
----
commit c41c97dd7d8b4d92ba99e27d404dc2ddc1b3e7ee
Author: Endre Major <em...@...>
Date: 2018-11-27T10:14:06Z
FLUME-3146 Use public API HdfsDataOutputStream#getCurrentBlockReplication where applicable
----
> Use public API HdfsDataOutputStream#getCurrentBlockReplication where applicable
> -------------------------------------------------------------------------------
>
> Key: FLUME-3146
> URL: https://issues.apache.org/jira/browse/FLUME-3146
> Project: Flume
> Issue Type: Bug
> Reporter: Wei-Chiu Chuang
> Assignee: Endre Major
> Priority: Major
> Attachments: FLUME-3146.001.patch
>
>
> AbstractHDFSWrite#reflectGetNumCurrentReplicas uses a private HDFS API, which causes it to emit an confusing error message:
> {noformat}
> 2017-08-08 22:01:10,743 INFO org.apache.flume.sink.hdfs.AbstractHDFSWriter: FileSystem's output stream doesn't support getNumCurrentReplicas; --HDFS-826 not available; fsOut=org.apache.hadoop.crypto.CryptoOutputStream; err=java.lang.NoSuchMethodException: org.apache.hadoop.crypto.CryptoOutputStream.getNumCurrentReplicas()
> 2017-08-08 22:02:11,086 ERROR org.apache.flume.sink.hdfs.AbstractHDFSWriter: Error while trying to hflushOrSync!
> {noformat}
> This error message is emitted even with the most recent version of HDFS that has HDFS-826 for sure.
> Turns out that the relevant code is not valid with HDFS encryption. Instead of using the current private (and deprecated) getNumCurrentReplicas#getNumCurrentReplicas() API, it should use HdfsDataOutputStream#getCurrentBlockReplication, which is a public API and which considers the case of HDFS encryption.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@flume.apache.org
For additional commands, e-mail: issues-help@flume.apache.org