You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "GeorgeJahad (via GitHub)" <gi...@apache.org> on 2023/12/13 19:45:47 UTC
[PR] HDDS-9910. Slow ozone hdfs performance when lots of config files. [ozone]
GeorgeJahad opened a new pull request, #5787:
URL: https://github.com/apache/ozone/pull/5787
## What changes were proposed in this pull request?
Improving hdfs perf by eliminating redundant config reads, and replacing them with a single one when the fs is initialized.
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-9910
## How was this patch tested?
Current tests still apply.
Rename perf under spark improved by a factor of 5:
```
spark-shell --master local --conf spark.kerberos.access.hadoopFileSystems=ofs://vol1 --conf spark.ui.enabled=false --conf spark.port.maxRetries=64
// The following commands are all run from within the sparkshell
// init the fs
import org.apache.hadoop.fs.Path
val ozone_fs = org.apache.hadoop.fs.FileSystem.get(new java.net.URI("ofs://vol1/bdpe/squirrelfso"), sc.hadoopConfiguration)
ozone_fs.delete(new Path(s"ofs://vol1/bdpe/squirrelfso/file1/part10001"))
ozone_fs.copyFromLocalFile(new Path(s"/etc/hosts"), new Path(s"ofs://vol1/bdpe/squirrelfso/file1/part1"))
// run the test
spark.time(Range.inclusive(1, 10000).foreach(i => ozone_fs.rename(new Path(s"ofs://vol1/bdpe/squirrelfso/file1/part${i.toString()}"), new Path(s"ofs://vol1/bdpe/squirrelfso/file1/part${(i + 1).toString()}"))))
Time taken: 17763 ms
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-9910. Slow ozone hdfs performance when lots of config files. [ozone]
Posted by "GeorgeJahad (via GitHub)" <gi...@apache.org>.
GeorgeJahad commented on PR #5787:
URL: https://github.com/apache/ozone/pull/5787#issuecomment-1855229933
Thanks for the reviews @hemantk-12 @sadanand48 and @adoroszlai !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-9910. Slow ozone hdfs performance when lots of config files. [ozone]
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on PR #5787:
URL: https://github.com/apache/ozone/pull/5787#issuecomment-1854618209
Thanks @GeorgeJahad for the improvement.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-9910. Slow ozone hdfs performance when lots of config files. [ozone]
Posted by "GeorgeJahad (via GitHub)" <gi...@apache.org>.
GeorgeJahad merged PR #5787:
URL: https://github.com/apache/ozone/pull/5787
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-9910. Slow ozone hdfs performance when lots of config files. [ozone]
Posted by "smengcl (via GitHub)" <gi...@apache.org>.
smengcl commented on PR #5787:
URL: https://github.com/apache/ozone/pull/5787#issuecomment-1858926769
Nice catch. Thanks @GeorgeJahad !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org