You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "lei-su-awx (via GitHub)" <gi...@apache.org> on 2023/12/12 06:37:42 UTC
[I] [SUPPORT] How to skip some partitions in a table when readStreaming in Spark at the init stage [hudi]
lei-su-awx opened a new issue, #10315:
URL: https://github.com/apache/hudi/issues/10315
**_Tips before filing an issue_**
- Have you gone through our [FAQs](https://hudi.apache.org/learn/faq/)?
- Join the mailing list to engage in conversations and get faster support at dev-subscribe@hudi.apache.org.
- If you have triaged this as a bug, then file an [issue](https://issues.apache.org/jira/projects/HUDI/issues) directly.
**Describe the problem you faced**
I have a table partition by operation type ingestion date(like insert/2023-12-11/, update/2023-12-11/, delete/2023-12-11/), when I read(use spark readStream) this table, I just want to read data under `update` partition. And I found a config 'hoodie.datasource.read.incr.path.glob', then I use this config and value is `update/202*`. But I found spark job init very slow, and found job was stuck
<img width="1534" alt="image" src="https://github.com/apache/hudi/assets/19327659/febba726-b441-4b01-abb0-2e12f8bc62d7">.
But this parquet file is not under `update` partition, it is under `insert` partition, which is very confused.
So I want ask is there a config that can only read the target partition and skip others and also does not read other partitions' data files to get schema.
**Expected behavior**
I want to know is there a config that can only read the target partition and skip others and also does not read other partitions' data files to get schema.
**Environment Description**
* Hudi version : 0.14.0
* Spark version : 3.4.1
* Hive version :
* Hadoop version :
* Storage (HDFS/S3/GCS..) : GCS
* Running on Docker? (yes/no) : no
**Additional context**
I use the below configurations to write to table:
hudi_write_options = {
'hoodie.table.name': hudi_table_name,
'hoodie.datasource.write.partitionpath.field': 'operation_type, ingestion_dt',
'hoodie.datasource.write.operation': 'insert',
'hoodie.datasource.write.table.type': 'MERGE_ON_READ',
'hoodie.parquet.compression.codec': 'zstd',
"hoodie.datasource.write.payload.class": "org.apache.hudi.common.model.DefaultHoodieRecordPayload",
"hoodie.datasource.write.reconcile.schema": True,
"hoodie.metadata.enable": True
}
And I use the configurations to read from table:
read_streaming_hudi_options = {
'maxFilesPerTrigger': 5,
'hoodie.datasource.read.incr.path.glob': 'update/202*',
'hoodie.read.timeline.holes.resolution.policy': 'BLOCK',
'hoodie.datasource.read.file.index.listing.partition-path-prefix.analysis.enabled': False,
'hoodie.file.listing.parallelism': 1000,
'hoodie.metadata.enable': True,
'hoodie.datasource.read.schema.use.end.instanttime': True,
'hoodie.datasource.streaming.startOffset': '20231211000000000'
}
**Stacktrace**
```Add the stacktrace of the error.```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] How to skip some partitions in a table when readStreaming in Spark at the init stage [hudi]
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope closed issue #10315: [SUPPORT] How to skip some partitions in a table when readStreaming in Spark at the init stage
URL: https://github.com/apache/hudi/issues/10315
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] How to skip some partitions in a table when readStreaming in Spark at the init stage [hudi]
Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on issue #10315:
URL: https://github.com/apache/hudi/issues/10315#issuecomment-1933446992
@lei-su-awx Closing this issue then. thanks. Please reopen in case you see issue again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] How to skip some partitions in a table when readStreaming in Spark at the init stage [hudi]
Posted by "lei-su-awx (via GitHub)" <gi...@apache.org>.
lei-su-awx commented on issue #10315:
URL: https://github.com/apache/hudi/issues/10315#issuecomment-1851457169
Hi @danny0405 , do you mean like this:
<img width="1010" alt="image" src="https://github.com/apache/hudi/assets/19327659/946c95b5-77f5-4d34-a315-a284c6b95b37">
I tried this, but also will read other partitions' data file to resolve schema. And I think kind of filter takes effect after source load all partitions files, but I want a config that can tell source that only reads the partition that in my configs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] How to skip some partitions in a table when readStreaming in Spark at the init stage [hudi]
Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on issue #10315:
URL: https://github.com/apache/hudi/issues/10315#issuecomment-1884502236
@lei-su-awx Any updates here? Do you still have this issue?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] How to skip some partitions in a table when readStreaming in Spark at the init stage [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #10315:
URL: https://github.com/apache/hudi/issues/10315#issuecomment-1851443273
Did you try to add filter condition with the partition fields?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] How to skip some partitions in a table when readStreaming in Spark at the init stage [hudi]
Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on issue #10315:
URL: https://github.com/apache/hudi/issues/10315#issuecomment-1871878668
@lei-su-awx Did you see the behaviour I mentioned? Any more updates here please?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] How to skip some partitions in a table when readStreaming in Spark at the init stage [hudi]
Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on issue #10315:
URL: https://github.com/apache/hudi/issues/10315#issuecomment-1862294498
@lei-su-awx You dont need to read from that directory. You should give the parent directory only. According to your `.filer("operation_type 'update')` partition pruning will take place and it will only process that partition data.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] How to skip some partitions in a table when readStreaming in Spark at the init stage [hudi]
Posted by "lei-su-awx (via GitHub)" <gi...@apache.org>.
lei-su-awx commented on issue #10315:
URL: https://github.com/apache/hudi/issues/10315#issuecomment-1884659448
@ad1happy2go sorry for the late, I did not test this anymore because I meet new issues. But I think `.filer("operation_type 'update')` can do what I want, so never mind this issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] How to skip some partitions in a table when readStreaming in Spark at the init stage [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #10315:
URL: https://github.com/apache/hudi/issues/10315#issuecomment-1851775346
> but I want a config that can tell source that only reads the partition that in my configs so I do not need to use filter
That does not follow the common intuition.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] How to skip some partitions in a table when readStreaming in Spark at the init stage [hudi]
Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on issue #10315:
URL: https://github.com/apache/hudi/issues/10315#issuecomment-1852418826
@lei-su-awx If the table is partitioned then it should only read the files under that partition. Are you seeing any behaviour otherwise if it is reading all files?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] How to skip some partitions in a table when readStreaming in Spark at the init stage [hudi]
Posted by "lei-su-awx (via GitHub)" <gi...@apache.org>.
lei-su-awx commented on issue #10315:
URL: https://github.com/apache/hudi/issues/10315#issuecomment-1853172037
@ad1happy2go I tried to only read files under that partition using spark(spark.readStream), but an error was thrown:
no .hoodie file exists in the partition path, and I found only .hoodie folder only exist under the table path.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org