You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@uniffle.apache.org by GitBox <gi...@apache.org> on 2022/12/07 08:48:12 UTC
[GitHub] [incubator-uniffle] yuyang733 opened a new issue, #391: [FEATURE] Expect to support the filesystem not implementing the append-mode.
yuyang733 opened a new issue, #391:
URL: https://github.com/apache/incubator-uniffle/issues/391
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
### Search before asking
- [X] I have searched in the [issues](https://github.com/apache/incubator-uniffle/issues?q=is%3Aissue) and found no similar issues.
### Describe the feature
Cloud object storage and its corresponding accelerated cache are widely used in storage-computing separation architecture.
But many of them may not implement the append-mode or the write amplification of the append-mode will be tricky.
Is it possible to consider supporting the way of independently storing each block as a data file at the same time?
### Motivation
Cloud object storage and its corresponding accelerated cache are widely used in storage-computing separation architecture.
But many of them may not implement the append-mode or the write amplification of the append-mode will be tricky.
### Describe the solution
The initially envisaged solution is to implement a non-append storage type, such as `COSShuffleWriteHandler`/ `COSShuffleDeleteHandler`, which writes all ShuffleBlocks as a DataFile instead of appending blocks in a dataFile.
### Additional context
_No response_
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] yuyang733 commented on issue #391: [FEATURE] Expect to support the filesystem not implementing the append-mode.
Posted by GitBox <gi...@apache.org>.
yuyang733 commented on issue #391:
URL: https://github.com/apache/incubator-uniffle/issues/391#issuecomment-1340919665
> > Aren't several blocks appended to a data file at present?
>
> Yes. I know it’s better to avoid using the append mode to improve the performance. But if the data layout is changed, do we need a index file? In the local file or HDFS storage type, the index file will maintain the relation of block offset in the single one file.
Great, thanks for reminding me, the index file seems to be not needed anymore.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] zuston commented on issue #391: [FEATURE] Expect to support the filesystem not implementing the append-mode.
Posted by GitBox <gi...@apache.org>.
zuston commented on issue #391:
URL: https://github.com/apache/incubator-uniffle/issues/391#issuecomment-1340961037
> Every data file should have index file.
The performance will not be good, as we have to read all index files in one time for doing split segments.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] zuston commented on issue #391: [FEATURE] Expect to support the filesystem not implementing the append-mode.
Posted by GitBox <gi...@apache.org>.
zuston commented on issue #391:
URL: https://github.com/apache/incubator-uniffle/issues/391#issuecomment-1340917575
> Aren't several blocks appended to a data file at present?
Yes. I know it’s better to avoid using the append mode to improve the performance. But if the data layout is changed, do we need a index file? In the local file or HDFS storage type, the index file will maintain the relation of block offset in the single one file.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] zuston commented on issue #391: [FEATURE] Expect to support the filesystem not implementing the append-mode.
Posted by GitBox <gi...@apache.org>.
zuston commented on issue #391:
URL: https://github.com/apache/incubator-uniffle/issues/391#issuecomment-1383558806
Any update on this? @yuyang733 cc @jerqi @advancedxy
If object store is supported, I will use this to store huge partition to reduce HDFS pressure for iQiyi. And this is an important feature for uniffle
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] zuston commented on issue #391: [FEATURE] Expect to support the filesystem not implementing the append-mode.
Posted by GitBox <gi...@apache.org>.
zuston commented on issue #391:
URL: https://github.com/apache/incubator-uniffle/issues/391#issuecomment-1340923650
I have another question that will we directly use the object store api or to use the hadoop filesystem api to support this?
If it is the former, maybe it’s better to introduce the uniffle dedicated filesystem api to wrap different concrete filesystems, also including the cos.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] zuston commented on issue #391: [FEATURE] Expect to support the filesystem not implementing the append-mode.
Posted by GitBox <gi...@apache.org>.
zuston commented on issue #391:
URL: https://github.com/apache/incubator-uniffle/issues/391#issuecomment-1340914781
Could u help give a data layout picture, which includes the relations of data block and oss file name.
Besides, we should also consider how to support the local_order in object store.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] jerqi commented on issue #391: [FEATURE] Expect to support the filesystem not implementing the append-mode.
Posted by GitBox <gi...@apache.org>.
jerqi commented on issue #391:
URL: https://github.com/apache/incubator-uniffle/issues/391#issuecomment-1340962808
> > Every data file should have index file.
>
> The performance will not be good, as we have to read all index files in one time for doing split segments.
We can read only one index file.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] yuyang733 commented on issue #391: [FEATURE] Expect to support the filesystem not implementing the append-mode.
Posted by GitBox <gi...@apache.org>.
yuyang733 commented on issue #391:
URL: https://github.com/apache/incubator-uniffle/issues/391#issuecomment-1340916605
> Could u help give a data layout picture, which includes the relations of data block and oss file name.
>
> Besides, we should also consider how to support the local_order in object store.
Okay, I will provide a design gram to describe it later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] yuyang733 commented on issue #391: [FEATURE] Expect to support the filesystem not implementing the append-mode.
Posted by GitBox <gi...@apache.org>.
yuyang733 commented on issue #391:
URL: https://github.com/apache/incubator-uniffle/issues/391#issuecomment-1340914709
> Do you mean that one dataFlushEvent could directly write a new single oss file? If so, how to maintain the index file?
Supply a way that supports the non-append storage.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] zuston commented on issue #391: [FEATURE] Expect to support the filesystem not implementing the append-mode.
Posted by GitBox <gi...@apache.org>.
zuston commented on issue #391:
URL: https://github.com/apache/incubator-uniffle/issues/391#issuecomment-1340903857
Do you mean that one dataFlushEvent could directly write a new single oss file? If so, how to maintain the index file?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] advancedxy commented on issue #391: [FEATURE] Expect to support the filesystem not implementing the append-mode.
Posted by GitBox <gi...@apache.org>.
advancedxy commented on issue #391:
URL: https://github.com/apache/incubator-uniffle/issues/391#issuecomment-1341094666
> In fact, Can an abstract storage layer be provided in uniffle, and each manufacturer implements the necessary storage interface?
It may require a detailed design doc to illustrate your idea and proposal.
In fact, I believe uniffle(and other RSS systems also) makes a big assumption of filesystem capability, such as append support, if we want to support new storages and dropping append requirement, we should reconsider the data layout patterns and all the features it required such as read operation, data distribution, etc. cc @zuston and @LuciferYang.
P.S: I think it's nice to have object store as a new storage type supported, we just need to think it throughly, make sure it doesn't introduce too much complexity, and maintain the flexibility to extend more storages.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] LuciferYang commented on issue #391: [FEATURE] Expect to support the filesystem not implementing the append-mode.
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on issue #391:
URL: https://github.com/apache/incubator-uniffle/issues/391#issuecomment-1341982684
Thanks for ping me @advancedxy I am not very familiar with object storage, but I think it is better to design it separately to avoid incompatibility of data layout and negative impact on the performance of the current implementation
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] yuyang733 commented on issue #391: [FEATURE] Expect to support the filesystem not implementing the append-mode.
Posted by GitBox <gi...@apache.org>.
yuyang733 commented on issue #391:
URL: https://github.com/apache/incubator-uniffle/issues/391#issuecomment-1340964715
> I have another question that will we directly use the object store api or to use the hadoop filesystem api to support this?
>
> If it is the former, maybe it’s better to introduce the uniffle dedicated filesystem api to wrap different concrete filesystems, also including the cos.
In fact, Can an abstract storage layer be provided in uniffle, and each manufacturer implements the necessary storage interface?
It does not care whether the concrete class uses the native API (such as S3, etc) or the HCFS file system.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] jerqi commented on issue #391: [FEATURE] Expect to support the filesystem not implementing the append-mode.
Posted by GitBox <gi...@apache.org>.
jerqi commented on issue #391:
URL: https://github.com/apache/incubator-uniffle/issues/391#issuecomment-1340947861
Every data file should have index file.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] yuyang733 commented on issue #391: [FEATURE] Expect to support the filesystem not implementing the append-mode.
Posted by GitBox <gi...@apache.org>.
yuyang733 commented on issue #391:
URL: https://github.com/apache/incubator-uniffle/issues/391#issuecomment-1340913678
> Do you mean that one dataFlushEvent could directly write a new single oss file? If so, how to maintain the index file?
Aren't several blocks appended to a data file at present? I want to write a block into a single data file, so it corresponds to an index file.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] zuston commented on issue #391: [FEATURE] Expect to support the filesystem not implementing the append-mode.
Posted by GitBox <gi...@apache.org>.
zuston commented on issue #391:
URL: https://github.com/apache/incubator-uniffle/issues/391#issuecomment-1340926367
> the index file seems to be not needed anymore.
Emm.. Maybe not. If having no a index structure, it means we don’t a global vision to determine find files we needed in object store, especially for local order. WDYT? @jerqi
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org