You are viewing a plain text version of this content. The canonical link for it is here.
Posted to yarn-issues@hadoop.apache.org by "shailesh gupta (Jira)" <ji...@apache.org> on 2022/03/22 14:46:00 UTC
[jira] [Updated] (YARN-11093) In fs-support-append false case, timelineserver reads event files in arbitrary order
[ https://issues.apache.org/jira/browse/YARN-11093?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
shailesh gupta updated YARN-11093:
----------------------------------
Summary: In fs-support-append false case, timelineserver reads event files in arbitrary order (was: In fs-support-append false case, server reads event files in arbitrary order)
> In fs-support-append false case, timelineserver reads event files in arbitrary order
> ------------------------------------------------------------------------------------
>
> Key: YARN-11093
> URL: https://issues.apache.org/jira/browse/YARN-11093
> Project: Hadoop YARN
> Issue Type: Bug
> Components: timelineserver
> Affects Versions: 3.3.2
> Reporter: shailesh gupta
> Priority: Critical
>
> In our setup, we are using Hive and Tez and using offline tez-ui mode, where we copy the ATS events files to other place and start tez-ui. To keep the event files small, we are using
> fs-support-append = false, and creating a new file for writing the events. We can see that, in this mode, file name contains [suffix|https://github.com/apache/hadoop/blob/1d5650c4d0acf33f141d593762682b3603523104/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/client/api/impl/FileSystemTimelineWriter.java#L382] as timestamp.
> But at the time of [read|https://github.com/apache/hadoop/blob/1d5650c4d0acf33f141d593762682b3603523104/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timeline-pluginstorage/src/main/java/org/apache/hadoop/yarn/server/timeline/EntityGroupFSTimelineStore.java#L844], we are relying on the FS to provide correct file order. If that fails, we are adding event out of order, leading to event being discarded, or incorrect information.
> Fix could be sorting of the file names, based on suffix if append mode is not used.
> sample file names:
> * summarylog-appattempt_1647348120288_0001_000001_460237
> * entitylog-timelineEntityGroupId_1647348120288_1_dag_1647348120288_0001_1_673147
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org