You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@uniffle.apache.org by GitBox <gi...@apache.org> on 2022/12/20 04:38:30 UTC
[GitHub] [incubator-uniffle] zuston opened a new issue, #440: [Improvement] Reuse HDFS client FSDataOutputStream to avoid extra overhead
zuston opened a new issue, #440:
URL: https://github.com/apache/incubator-uniffle/issues/440
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
### Search before asking
- [X] I have searched in the [issues](https://github.com/apache/incubator-uniffle/issues?q=is%3Aissue) and found no similar issues.
### What would you like to be improved?
In `HdfsShuffleWriteHandler,` it will always create the instance of `HdfsFileWriter` to write data, in which, the FSDataOutputStream will connect or close with remote NN.
There is no need to close every time, we could reuse this to speed up.
### How should we improve?
_No response_
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] zuston commented on issue #440: [Improvement] Reuse HDFS client FSDataOutputStream to avoid extra overhead
Posted by GitBox <gi...@apache.org>.
zuston commented on issue #440:
URL: https://github.com/apache/incubator-uniffle/issues/440#issuecomment-1360758016
> If the data is flushed successfully, but the data is close unsuccessfully, how to process this case?
If it could be read by downstream reducer once flushed, we could accept its close failure. But I'm not sure whether the flushing operation could guarantee the visibility
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] zuston commented on issue #440: [Improvement] Reuse HDFS client FSDataOutputStream to avoid extra overhead
Posted by GitBox <gi...@apache.org>.
zuston commented on issue #440:
URL: https://github.com/apache/incubator-uniffle/issues/440#issuecomment-1359111986
> How to process when we fail to write HDFS?
Invalidate this cache.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] zuston commented on issue #440: [Improvement] Reuse HDFS client FSDataOutputStream to avoid extra overhead
Posted by GitBox <gi...@apache.org>.
zuston commented on issue #440:
URL: https://github.com/apache/incubator-uniffle/issues/440#issuecomment-1359205179
> The cost of flush operation is similar to the operation close.
But this could avoid re-creating with the datanode when flushing events again. Right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] zuston closed issue #440: [Improvement] Reuse HDFS client FSDataOutputStream to avoid extra overhead
Posted by "zuston (via GitHub)" <gi...@apache.org>.
zuston closed issue #440: [Improvement] Reuse HDFS client FSDataOutputStream to avoid extra overhead
URL: https://github.com/apache/incubator-uniffle/issues/440
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
[GitHub] [incubator-uniffle] jerqi commented on issue #440: [Improvement] Reuse HDFS client FSDataOutputStream to avoid extra overhead
Posted by GitBox <gi...@apache.org>.
jerqi commented on issue #440:
URL: https://github.com/apache/incubator-uniffle/issues/440#issuecomment-1359108968
How to process when we fail to write HDFS?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] xianjingfeng commented on issue #440: [Improvement] Reuse HDFS client FSDataOutputStream to avoid extra overhead
Posted by GitBox <gi...@apache.org>.
xianjingfeng commented on issue #440:
URL: https://github.com/apache/incubator-uniffle/issues/440#issuecomment-1361008354
Not every events will flush to hdfs. In our production environment, the probability that flush to hdfs is less than 1 in 1000. So i think if this feature is required, it can be used as an option.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] jerqi commented on issue #440: [Improvement] Reuse HDFS client FSDataOutputStream to avoid extra overhead
Posted by GitBox <gi...@apache.org>.
jerqi commented on issue #440:
URL: https://github.com/apache/incubator-uniffle/issues/440#issuecomment-1360743025
> > The cost of flush operation is similar to the operation close.
>
> But this could avoid re-creating with the datanode when flushing events again. Right?
If the data is flushed successfully, but the data is close unsuccessfully, how to process this case?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] jerqi commented on issue #440: [Improvement] Reuse HDFS client FSDataOutputStream to avoid extra overhead
Posted by GitBox <gi...@apache.org>.
jerqi commented on issue #440:
URL: https://github.com/apache/incubator-uniffle/issues/440#issuecomment-1359203057
> > If you don't close, you can't guarantee that the data is safe. you can't delete the memory data.
>
> The operation of `flush` can't guarantee this?
The cost of flush operation is similar to the operation close.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] zuston commented on issue #440: [Improvement] Reuse HDFS client FSDataOutputStream to avoid extra overhead
Posted by GitBox <gi...@apache.org>.
zuston commented on issue #440:
URL: https://github.com/apache/incubator-uniffle/issues/440#issuecomment-1359197876
> If you don't close, you can't guarantee that the data is safe. you can't delete the memory data.
The operation of `flush` can't guarantee this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] zuston commented on issue #440: [Improvement] Reuse HDFS client FSDataOutputStream to avoid extra overhead
Posted by GitBox <gi...@apache.org>.
zuston commented on issue #440:
URL: https://github.com/apache/incubator-uniffle/issues/440#issuecomment-1361010692
> Not every events will flush to hdfs. In our production environment, the probability that flush to hdfs is less than 1 in 1000. So i think if this feature is required, it can be used as an option.
Yes. Actually, I don't know the real performance improvement for this feature, maybe I won't raise a PR to solve
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org