You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "ASF GitHub Bot (Jira)" <ji...@apache.org> on 2021/05/04 08:22:00 UTC
[jira] [Work logged] (HADOOP-17657) SequeneFile.Writer should
implement StreamCapabilities
[ https://issues.apache.org/jira/browse/HADOOP-17657?focusedWorklogId=592601&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-592601 ]
ASF GitHub Bot logged work on HADOOP-17657:
-------------------------------------------
Author: ASF GitHub Bot
Created on: 04/May/21 08:21
Start Date: 04/May/21 08:21
Worklog Time Spent: 10m
Work Description: jojochuang merged pull request #2949:
URL: https://github.com/apache/hadoop/pull/2949
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Issue Time Tracking
-------------------
Worklog Id: (was: 592601)
Time Spent: 3h 50m (was: 3h 40m)
> SequeneFile.Writer should implement StreamCapabilities
> ------------------------------------------------------
>
> Key: HADOOP-17657
> URL: https://issues.apache.org/jira/browse/HADOOP-17657
> Project: Hadoop Common
> Issue Type: Bug
> Reporter: Kishen Das
> Assignee: Kishen Das
> Priority: Major
> Labels: pull-request-available
> Time Spent: 3h 50m
> Remaining Estimate: 0h
>
> Following exception is thrown whenever we invoke ProtoMessageWriter.hflush on S3 from Tez, which internally calls org.apache.hadoop.io.SequenceFile$Writer.hflush -> org.apache.hadoop.fs.FS DataOutputStream.hflush -> S3ABlockOutputStream.hflush which is not implemented and throws java.lang.UnsupportedOperationException.
> bdffe22d96ae [mdc@18060 class="yarn.YarnUncaughtExceptionHandler" level="ERROR" thread="HistoryEventHandlingThread"] Thread Thread[HistoryEventHandlingThread, 5,main] threw an Exception.^Mjava.lang.UnsupportedOperationException: S3A streams are not Syncable^M at org.apache.hadoop.fs.s3a.S3ABlockOutputStream.hflush(S3ABlockOutputStream.java:657)^M at org.apache.hadoop.fs.FS DataOutputStream.hflush(FSDataOutputStream.java:136)^M at org.apache.hadoop.io.SequenceFile$Writer.hflush(SequenceFile.java:1367)^M at org.apache.tez.dag.history.logging.proto.ProtoMessageWriter.hflush(ProtoMessageWr iter.java:64)^M at org.apache.tez.dag.history.logging.proto.ProtoHistoryLoggingService.finishCurrentDag(ProtoHistoryLoggingService.java:239)^M at org.apache.tez.dag.history.logging.proto.ProtoHistoryLoggingService.han dleEvent(ProtoHistoryLoggingService.java:198)^M at org.apache.tez.dag.history.logging.proto.ProtoHistoryLoggingService.loop(ProtoHistoryLoggingService.java:153)^M at java.lang.Thread.run(Thread.java:748)^M
> In order to fix this issue we should implement StreamCapabilities in SequenceFile.Writer. Also, we should fall back to flush(), if hflush() is not supported.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org