You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "MarsKT (via GitHub)" <gi...@apache.org> on 2023/10/11 11:44:33 UTC
[I] How to read data in the order in which files are commited? [iceberg]
MarsKT opened a new issue, #8802:
URL: https://github.com/apache/iceberg/issues/8802
### Query engine
Iceberg java api
### Question
I want the data read from iceberg to be in the same order every time. But I can't find an attribute that would make FileScanTask ordered. Is there a way I can implement it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] How to read data in the order in which files are commited? [iceberg]
Posted by "Zhanxiao-Ma (via GitHub)" <gi...@apache.org>.
Zhanxiao-Ma commented on issue #8802:
URL: https://github.com/apache/iceberg/issues/8802#issuecomment-1760704067
> Sometimes we need to do similar thing in Flink Source, and we ended up creating our own comparator for this which compares Iceberg splits (which are a wrapper above ScanTasks).
I'm sorry, I didn't quite understand this point. Could you please explain it in more detail?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] How to read data in the order in which files are commited? [iceberg]
Posted by "Zhanxiao-Ma (via GitHub)" <gi...@apache.org>.
Zhanxiao-Ma commented on issue #8802:
URL: https://github.com/apache/iceberg/issues/8802#issuecomment-1760700741
> Currently there is no way to order the scan task. The planning side specifically makes sure that even the planning could be done by parallel threads (reading manifests files parallel)
>
> Sometimes we need to do similar thing in Flink Source, and we ended up creating our own comparator for this which compares Iceberg splits (which are a wrapper above ScanTasks).
>
> You can do something similar like this in java code with one serious caveat: For a big table you might not want/able to keep all of the tasks in memory, which is needed for sorting. What we do in flink is limit the number of snapshots to read once.
>
> I hope this helps, Peter
> Currently there is no way to order the scan task. The planning side specifically makes sure that even the planning could be done by parallel threads (reading manifests files parallel)
>
> Sometimes we need to do similar thing in Flink Source, and we ended up creating our own comparator for this which compares Iceberg splits (which are a wrapper above ScanTasks).
>
> You can do something similar like this in java code with one serious caveat: For a big table you might not want/able to keep all of the tasks in memory, which is needed for sorting. What we do in flink is limit the number of snapshots to read once.
>
> I hope this helps, Peter
> Currently there is no way to order the scan task. The planning side specifically makes sure that even the planning could be done by parallel threads (reading manifests files parallel)
>
> Sometimes we need to do similar thing in Flink Source, and we ended up creating our own comparator for this which compares Iceberg splits (which are a wrapper above ScanTasks).
>
> You can do something similar like this in java code with one serious caveat: For a big table you might not want/able to keep all of the tasks in memory, which is needed for sorting. What we do in flink is limit the number of snapshots to read once.
>
> I hope this helps, Peter
>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] How to read data in the order in which files are commited? [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on issue #8802:
URL: https://github.com/apache/iceberg/issues/8802#issuecomment-1757603105
Currently there is no way to order the scan task. The planning side specifically makes sure that even the planning could be done by parallel threads (reading manifests files parallel)
Sometimes we need to do similar thing in Flink Source, and we ended up creating our own comparator for this which compares Iceberg splits (which are a wrapper above ScanTasks).
You can do something similar like this in java code with one serious caveat: For a big table you might not want/able to keep all of the tasks in memory, which is needed for sorting. What we do in flink is limit the number of snapshots to read once.
I hope this helps,
Peter
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] How to read data in the order in which files are commited? [iceberg]
Posted by "Zhanxiao-Ma (via GitHub)" <gi...@apache.org>.
Zhanxiao-Ma closed issue #8802: How to read data in the order in which files are commited?
URL: https://github.com/apache/iceberg/issues/8802
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] How to read data in the order in which files are commited? [iceberg]
Posted by "MarsKT (via GitHub)" <gi...@apache.org>.
MarsKT commented on issue #8802:
URL: https://github.com/apache/iceberg/issues/8802#issuecomment-1760659443
> Thanks @pvary, I have a maybe naive question @MarsKT
>
> > want the data read from iceberg to be in the same order every time.
>
> Can I ask what's driving this need?
The users of a data analytics software that utilizes a data lake as its storage layer desire consistent data ordering when viewing the data.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] How to read data in the order in which files are commited? [iceberg]
Posted by "Zhanxiao-Ma (via GitHub)" <gi...@apache.org>.
Zhanxiao-Ma closed issue #8802: How to read data in the order in which files are commited?
URL: https://github.com/apache/iceberg/issues/8802
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] How to read data in the order in which files are commited? [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on issue #8802:
URL: https://github.com/apache/iceberg/issues/8802#issuecomment-1765938139
> > Sometimes we need to do similar thing in Flink Source, and we ended up creating our own comparator for this which compares Iceberg splits (which are a wrapper above ScanTasks).
>
> I'm sorry, I didn't quite understand this point. Could you please explain it in more detail?
We fetch all the tasks for a given plan here (and covert it to an IcebergSplit using `fromCombinedScanTask`):
https://github.com/apache/iceberg/blob/46cad6ddaeff8104d96defab25206a4ff7e01629/flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/source/FlinkSplitPlanner.java#L73-L82
And we use a [SerializableComparator](https://github.com/apache/iceberg/blob/46cad6ddaeff8104d96defab25206a4ff7e01629/flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/source/split/SplitComparators.java#L33) to order the splits before assigning them:
https://github.com/apache/iceberg/blob/46cad6ddaeff8104d96defab25206a4ff7e01629/flink/v1.17/flink/src/main/java/org/apache/iceberg/flink/source/assigner/DefaultSplitAssigner.java#L44-L46
I hope this helps!
Thanks,
Peter
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] How to read data in the order in which files are commited? [iceberg]
Posted by "amogh-jahagirdar (via GitHub)" <gi...@apache.org>.
amogh-jahagirdar commented on issue #8802:
URL: https://github.com/apache/iceberg/issues/8802#issuecomment-1757829625
Thanks @pvary, I have a maybe naive question @MarsKT
> want the data read from iceberg to be in the same order every time.
Can I ask what's driving this need?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org