You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2020/12/23 22:38:41 UTC
[GitHub] [hudi] rswagatika opened a new issue #2373: [SUPPORT]
rswagatika opened a new issue #2373:
URL: https://github.com/apache/hudi/issues/2373
**_Tips before filing an issue_**
- Have you gone through our [FAQs](https://cwiki.apache.org/confluence/display/HUDI/FAQ)?
- Join the mailing list to engage in conversations and get faster support at dev-subscribe@hudi.apache.org.
- If you have triaged this as a bug, then file an [issue](https://issues.apache.org/jira/projects/HUDI/issues) directly.
**Describe the problem you faced**
A clear and concise description of the problem.
**To Reproduce**
Steps to reproduce the behavior:
1.
2.
3.
4.
**Expected behavior**
A clear and concise description of what you expected to happen.
**Environment Description**
* Hudi version :
* Spark version :
* Hive version :
* Hadoop version :
* Storage (HDFS/S3/GCS..) :
* Running on Docker? (yes/no) :
**Additional context**
Add any other context about the problem here.
**Stacktrace**
```Add the stacktrace of the error.```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] rswagatika edited a comment on issue #2373: [SUPPORT] - Hudi Read ,Partition Filter is coming empty.
Posted by GitBox <gi...@apache.org>.
rswagatika edited a comment on issue #2373:
URL: https://github.com/apache/hudi/issues/2373#issuecomment-751848219
Hi , I have added the config for hive style partitioning and in my partition I could see the hive style partitons.
<img width="862" alt="Screen Shot 2020-12-28 at 3 04 16 PM" src="https://user-images.githubusercontent.com/76132270/103240306-fe0bd680-491d-11eb-8253-cc9370a166d6.png">
However if I do a filter on read still the Partition Filter is empty in my Physical Plan. My question is does hudi scan all the partitioned folder in this case or does it only look up the partitioned folder I specified in the read.
```
queryHudiRead = spark.read.format("hudi").load("s3://testdataprocessing/Partition_test13/*/*").filter("range_partition == 10 and posting_key > 20")
queryHudiRead.explain(extended=True)
```
== Physical Plan ==
*(1) Project [_hoodie_commit_time#281, _hoodie_commit_seqno#282, _hoodie_record_key#283, _hoodie_partition_path#284, _hoodie_file_name#285, locations#286, posting_key#287L, update_date#288, range_partition#289L]
+- *(1) Filter (((isnotnull(range_partition#289L) && isnotnull(posting_key#287L)) && (range_partition#289L = 10)) && (posting_key#287L > 20))
+- *(1) FileScan parquet [_hoodie_commit_time#281,_hoodie_commit_seqno#282,_hoodie_record_key#283,_hoodie_partition_path#284,_hoodie_file_name#285,locations#286,posting_key#287L,update_date#288,range_partition#289L] Batched: false, Format: Parquet, Location: InMemoryFileIndex[s3://bgt-atlas-emr-devtest/testdataprocessing/WOutHive_Partition_test13/.hoodie..., PartitionFilters: [], PushedFilters: [IsNotNull(range_partition), IsNotNull(posting_key), EqualTo(range_partition,10), GreaterThan(pos..., ReadSchema: struct<_hoodie_commit_time:string,_hoodie_commit_seqno:string,_hoodie_record_key:string,_hoodie_p...
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] rswagatika closed issue #2373: [SUPPORT] - Hudi Read ,Partition Filter is coming empty.
Posted by GitBox <gi...@apache.org>.
rswagatika closed issue #2373:
URL: https://github.com/apache/hudi/issues/2373
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] garyli1019 commented on issue #2373: [SUPPORT] - Hudi Read ,Partition Filter is coming empty.
Posted by GitBox <gi...@apache.org>.
garyli1019 commented on issue #2373:
URL: https://github.com/apache/hudi/issues/2373#issuecomment-752121598
please try`.load("s3://testdataprocessing/Partition_test13")`, spark won't load the partition if you use the blob path. I think the documentation mentioned this as well. if your dataset already has a `range_partition` column then it won't work too. Spark will try to load the partition based on the folder's name.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] rswagatika commented on issue #2373: [SUPPORT] - Hudi Read ,Partition Filter is coming empty.
Posted by GitBox <gi...@apache.org>.
rswagatika commented on issue #2373:
URL: https://github.com/apache/hudi/issues/2373#issuecomment-751488032
My partition does not look like hive style when am using Hudi write.
The below is the screenshot, how my partition looks like.
<img width="416" alt="Screen Shot 2020-12-27 at 11 27 23 AM" src="https://user-images.githubusercontent.com/76132270/103175283-c413d500-4836-11eb-9a7b-262924313fd3.png">
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] rswagatika commented on issue #2373: [SUPPORT] - Hudi Read ,Partition Filter is coming empty.
Posted by GitBox <gi...@apache.org>.
rswagatika commented on issue #2373:
URL: https://github.com/apache/hudi/issues/2373#issuecomment-752171452
@garyli1019 Thank you for your response and help.I was able to see the partition filter removing the /*/* .
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] rswagatika closed issue #2373: [SUPPORT]
Posted by GitBox <gi...@apache.org>.
rswagatika closed issue #2373:
URL: https://github.com/apache/hudi/issues/2373
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] garyli1019 commented on issue #2373: [SUPPORT] - Hudi Read ,Partition Filter is coming empty.
Posted by GitBox <gi...@apache.org>.
garyli1019 commented on issue #2373:
URL: https://github.com/apache/hudi/issues/2373#issuecomment-751554567
Hi, Spark only can recognize hive style partitioning. Please refer to this doc: https://spark.apache.org/docs/latest/sql-data-sources-parquet.html#partition-discovery
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] rswagatika commented on issue #2373: [SUPPORT] - Hudi Read ,Partition Filter is coming empty.
Posted by GitBox <gi...@apache.org>.
rswagatika commented on issue #2373:
URL: https://github.com/apache/hudi/issues/2373#issuecomment-751848219
Hi , I have added the config for hive style partitioning and in my partition I could see the hive style partitons.
<img width="862" alt="Screen Shot 2020-12-28 at 3 04 16 PM" src="https://user-images.githubusercontent.com/76132270/103240306-fe0bd680-491d-11eb-8253-cc9370a166d6.png">
However if I do a filter on read still the Partition Filter is empty in my Physical Plan. My question is does hudi scan all the partitioned folder in this case or does it only look up the partitioned folder I specified in the read.
== Physical Plan ==
*(1) Project [_hoodie_commit_time#281, _hoodie_commit_seqno#282, _hoodie_record_key#283, _hoodie_partition_path#284, _hoodie_file_name#285, locations#286, posting_key#287L, update_date#288, range_partition#289L]
+- *(1) Filter (((isnotnull(range_partition#289L) && isnotnull(posting_key#287L)) && (range_partition#289L = 10)) && (posting_key#287L > 20))
+- *(1) FileScan parquet [_hoodie_commit_time#281,_hoodie_commit_seqno#282,_hoodie_record_key#283,_hoodie_partition_path#284,_hoodie_file_name#285,locations#286,posting_key#287L,update_date#288,range_partition#289L] Batched: false, Format: Parquet, Location: InMemoryFileIndex[s3://bgt-atlas-emr-devtest/testdataprocessing/WOutHive_Partition_test13/.hoodie..., PartitionFilters: [], PushedFilters: [IsNotNull(range_partition), IsNotNull(posting_key), EqualTo(range_partition,10), GreaterThan(pos..., ReadSchema: struct<_hoodie_commit_time:string,_hoodie_commit_seqno:string,_hoodie_record_key:string,_hoodie_p...
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] garyli1019 commented on issue #2373: [SUPPORT] - Hudi Read ,Partition Filter is coming empty.
Posted by GitBox <gi...@apache.org>.
garyli1019 commented on issue #2373:
URL: https://github.com/apache/hudi/issues/2373#issuecomment-751336015
hi, how's your partition naming looks like? is that in hive style with column name prefix like `year=2020`? The partition is based on Spark partition auto-discovery so the naming must be in hive style.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org