[jira] [Updated] (HDDS-3552) OzoneFS is slow compared to HDFS using Spark job

["Prashant Pogde (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HDDS-3552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Prashant Pogde updated HDDS-3552:
---------------------------------
    Target Version/s: 1.2.0

I am managing the 1.1.0 release and we currently have more than 600 issues targeted for 1.1.0. I am moving the target field to 1.2.0. 

If you are actively working on this jira and believe this should be targeted to 1.1.0 release, Please change the target field back to 1.1.0 before Feb 05, 2021. 

> OzoneFS is slow compared to HDFS using Spark job
> ------------------------------------------------
>
>                 Key: HDDS-3552
>                 URL: https://issues.apache.org/jira/browse/HDDS-3552
>             Project: Hadoop Distributed Data Store
>          Issue Type: Improvement
>            Reporter: Marton Elek
>            Assignee: Marton Elek
>            Priority: Major
>              Labels: performance
>
> Reported by "Andrey Mindrin" on the the-asf Slack:
> {quote}
> We have made a few tests to compare OZONE (0.4 and 0.5 on Cloudera Runtime 7.0.3 with 3 nodes) performance with HDFS and OZONE is slower in most cases. For example, Spark application with 18 containers that copies 6 Gb parquet file is about 50% slower on OzoneFS. The only one case shows the same performance - Hive queries over partitioned tables.
>  simple spark code we used:
> {code}
> val file = spark.read.format(format).load(path_input)
> file.write.format(format).save(path_output)
> {code}
> Tested on CSV file with 800 million records, 2 columns and parquet file converted from CSV above. Just copied file from HDFS to HDFS and from Ozone to Ozone. Application time is 1m 14s on HDFS and  1m 51s (+50%) on Ozone (parquet file). Ozone has default settings. (edited) 
> {quote}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org