You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Wei-Chiu Chuang (JIRA)" <ji...@apache.org> on 2018/11/13 21:51:00 UTC
[jira] [Comment Edited] (HADOOP-15928) Excessive error logging when
using HDFS in S3 environment
[ https://issues.apache.org/jira/browse/HADOOP-15928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16685777#comment-16685777 ]
Wei-Chiu Chuang edited comment on HADOOP-15928 at 11/13/18 9:50 PM:
--------------------------------------------------------------------
bq. I don't understand what's going on here. Is it that impala is trying to use the hdfs native binding to talk to s3? And that's logging things which aren't relevant?
The summary was a little confusing. Yes Impala uses libhdfs as the native wrapper for hdfs client code to access s3. That UnsupportedOperationException message is printed into stderr every time it tries to read from byte butter, and the stderr could grow up to millions of UnsupportedOperationException messages.
was (Author: jojochuang):
bq. I don't understand what's going on here. Is it that impala is trying to use the hdfs native binding to talk to s3? And that's logging things which aren't relevant?
The summary was a little confusing. Yes Impala uses libhdfs as the native wrapper for hdfs client code. That UnsupportedOperationException message is printed into stderr every time it tries to read from byte butter, and the stderr could grow up to millions of UnsupportedOperationException messages.
> Excessive error logging when using HDFS in S3 environment
> ---------------------------------------------------------
>
> Key: HADOOP-15928
> URL: https://issues.apache.org/jira/browse/HADOOP-15928
> Project: Hadoop Common
> Issue Type: Improvement
> Reporter: Pranay Singh
> Assignee: Pranay Singh
> Priority: Major
> Attachments: HADOOP-15928.001.patch
>
>
> Problem:
> ------------
> There is excessive error logging when Impala uses HDFS in S3 environment, this issue is caused because of defect HADOOP-14603 "S3A input stream to support ByteBufferReadable"
> Excessive error logging results in defect IMPALA-5256: "ERROR log files can get very large". This causes the error log files to be huge.
> The following message is printed repeatedly in the error log:
> UnsupportedOperationException: Byte-buffer read unsupported by input streamjava.lang.UnsupportedOperationException: Byte-buffer read unsupported by input stream
> at org.apache.hadoop.fs.FSDataInputStream.read(FSDataInputStream.java:150)
> Root cause
> ----------------
> After investigating the issue, it appears that the above exception is printed because
> when a file is opened via hdfsOpenFileImpl() calls readDirect() which is hitting this
> exception.
> Fix:
> ----
> Since the hdfs client is not initiating the byte buffered read but is happening in a implicit manner, we should not be generating the error log during open of a file.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org