You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "Zelin1203 (via GitHub)" <gi...@apache.org> on 2023/10/08 09:06:50 UTC
[I] Can not split files on s3 when use flink to read iceberg on s3 with small split-size [iceberg]
Zelin1203 opened a new issue, #8744:
URL: https://github.com/apache/iceberg/issues/8744
### Query engine
Flink
### Question
I set the split-size smaller than files on s3. Each snapshot has 16 added-files.
When reading data, only 16 splits can be obtained at a time, which is consistent with the number of files on S3.
Can files on s3 be splited?
The query sql is as follows:
select * from realtime_catalog.trancare_realtime_dws.dws_stability_app_crash_10min
/*+ OPTIONS('streaming'='true', 'monitor-interval'='120s', 'start-snapshot-id'='8326101375840043894', 'max-planning-snapshot-count'='1','split-size'='52428','split-file-open-cost'='5242')*/
The snapshot in dws_stability_app_crash_10min is as follows:
"sequence-number" : 13553,
"snapshot-id" : 2706808882911224212,
"parent-snapshot-id" : 4090495094516144724,
"timestamp-ms" : 1696657624094,
"summary" : {
"operation" : "append",
"flink.job-id" : "0d30143259f3b2b2c59167ed530e0d0b",
"flink.max-committed-checkpoint-id" : "217042",
"added-data-files" : "16",
"added-records" : "1031343",
"added-files-size" : "87765325",
"changed-partition-count" : "1",
"total-records" : "1122395594",
"total-files-size" : "95451297590",
"total-data-files" : "29316",
"total-delete-files" : "0",
"total-position-deletes" : "0",
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Can not split files on s3 when use flink to read iceberg on s3 with small split-size [iceberg]
Posted by "Zelin1203 (via GitHub)" <gi...@apache.org>.
Zelin1203 closed issue #8744: Can not split files on s3 when use flink to read iceberg on s3 with small split-size
URL: https://github.com/apache/iceberg/issues/8744
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Can not split files on s3 when use flink to read iceberg on s3 with small split-size [iceberg]
Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on issue #8744:
URL: https://github.com/apache/iceberg/issues/8744#issuecomment-1752029027
One important thing to remember is that Parquet files are not arbitrarily splittable. If there is only a single row group in a parquet file, it cannot be split. Are you sure your files span more than 1 row group? You can always change your row group size to be small to test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Can not split files on s3 when use flink to read iceberg on s3 with small split-size [iceberg]
Posted by "Zelin1203 (via GitHub)" <gi...@apache.org>.
Zelin1203 commented on issue #8744:
URL: https://github.com/apache/iceberg/issues/8744#issuecomment-1752047235
> One important thing to remember is that Parquet files are not arbitrarily splittable. If there is only a single row group in a parquet file, it cannot be split. Are you sure your files span more than 1 row group? You can always change your row group size to be small to test.
Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org