You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "mingchao zhao (Jira)" <ji...@apache.org> on 2020/03/12 11:07:00 UTC
[jira] [Comment Edited] (HDDS-3155) Improved ozone client flush
implementation to make it faster.
[ https://issues.apache.org/jira/browse/HDDS-3155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17057814#comment-17057814 ]
mingchao zhao edited comment on HDDS-3155 at 3/12/20, 11:06 AM:
----------------------------------------------------------------
Hi [~elek] , [~shashikant] and [~msingh] , very thanks for your discussing this issue. Your Suggestions have helped me a lot.
Here is my previous test result, simulating the operation of MapReduce AM to write Log. Contrast HDFS and ozone:
{code:java}
//
String dst = "o3fs://bucket.hadoop/test4.log";
FileSystem fs = FileSystem.get(URI.create(dst), conf,"root");
FSDataOutputStream fsOut = fs.create(new Path(dst), true);
String d1 = new Date().toString();
for(int i=0;i<8000;i++){
String str = "org.apache.hadoop.mapreduce.jobhistory.TaskAttemporg.apache.hadoop.mapreduce.jobhistory.TaskAttempa";
fsOut.write(str.getBytes());
fsOut.flush();
}
fsOut.close();
{code}
When testing ozone, the previous phenomenon I saw was that a chunk file would be generated each time we performed flush：
!image-2020-03-12-16-48-08-391.png|width=543,height=302!
And when testing hdfs, It generated only one data file.
!image-2020-03-12-17-47-57-770.png|width=552,height=94!
The actual execution time is also quite different. I found that HDFS is called hflush([Refer to the description of this method|https://github.com/apache/hadoop/blob/ac4b556e2d44d3cd10b81c190ecee23e2dd66c10/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java#L573]) when performing flush.
At present, I am not sure if ozone has a similar design. I will look at the implementation of HDDS-2717 first to confirm whether this PR can solve the current problem.
was (Author: micahzhao):
Hi [~elek] [~shashikant] [~msingh] very thanks for your discussing this issue. Your Suggestions have helped me a lot.
Here is my previous test result, simulating the operation of MapReduce AM to write Log. Contrast HDFS and ozone:
{code:java}
//
String dst = "o3fs://bucket.hadoop/test4.log";
FileSystem fs = FileSystem.get(URI.create(dst), conf,"root");
FSDataOutputStream fsOut = fs.create(new Path(dst), true);
String d1 = new Date().toString();
for(int i=0;i<8000;i++){
String str = "org.apache.hadoop.mapreduce.jobhistory.TaskAttemporg.apache.hadoop.mapreduce.jobhistory.TaskAttempa";
fsOut.write(str.getBytes());
fsOut.flush();
}
fsOut.close();
{code}
When testing ozone, the previous phenomenon I saw was that a chunk file would be generated each time we performed flush：
!image-2020-03-12-16-48-08-391.png|width=543,height=302!
And when testing hdfs, It generated only one data file.
!image-2020-03-12-17-47-57-770.png|width=552,height=94!
The actual execution time is also quite different. I found that HDFS is called hflush([Refer to the description of this method|https://github.com/apache/hadoop/blob/ac4b556e2d44d3cd10b81c190ecee23e2dd66c10/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java#L573]) when performing flush.
At present, I am not sure if ozone has a similar design. I will look at the implementation of HDDS-2717 first to confirm whether this PR can solve the current problem.
> Improved ozone client flush implementation to make it faster.
> -------------------------------------------------------------
>
> Key: HDDS-3155
> URL: https://issues.apache.org/jira/browse/HDDS-3155
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
> Reporter: mingchao zhao
> Priority: Major
> Attachments: amlog, image-2020-03-12-16-48-08-391.png, image-2020-03-12-17-47-57-770.png, stdout
>
>
> Background:
> When we execute mapreduce in the ozone, we find that the task will be stuck for a long time after the completion of Map and Reduce. The log is as follows:
> {code:java}
> //Refer to the attachment: stdout
> 20/03/05 14:43:30 INFO mapreduce.Job: map 100% reduce 33%
> 20/03/05 14:43:33 INFO mapreduce.Job: map 100% reduce 100%
> 20/03/05 15:29:52 INFO mapreduce.Job: Job job_1583385253878_0002 completed successfully{code}
> By looking at AM's log(Refer to the amlog for details), we found that the time of over 40 minutes is AM writing a task log into ozone.
> At present, after MR execution, the Task information is recorded into the log on HDFS or ozone by AM. Moreover, the task information is flush to HDFS or ozone one by one ([details|https://github.com/apache/hadoop/blob/a55d6bba71c81c1c4e9d8cd11f55c78f10a548b0/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobHistoryEventHandler.java#L1640]). The problem occurs when the number of task maps is large.
> Currently, each flush operation in ozone generates a new chunk file in real time on the disk. This approach is not very efficient at the moment. For this we can refer to the implementation of HDFS flush. Instead of writing to disk each time flush writes the contents of the buffer to the datanode's OS buffer. In the first place, we need to ensure that this content can be read by other datanodes.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org