You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "michael1991 (via GitHub)" <gi...@apache.org> on 2024/03/06 08:24:18 UTC
[I] [SUPPORT] Spark Read Hudi Tables with WARN Message [hudi]
michael1991 opened a new issue, #10828:
URL: https://github.com/apache/hudi/issues/10828
**Describe the problem you faced**
I'm using Spark 3.3.2 with Hudi v0.14.1 package to read hudi table(0.12.3), then I will get warn message as below:
```log
WARN HoodieFileIndex:367 - Met incompatible issue when converting to hudi data type, rollback to list by prefix directly
```
**Environment Description**
* Hudi version : 0.14.1
* Spark version : 3.3.2
* Hive version :
* Hadoop version :
* Storage (HDFS/S3/GCS..) : GCS
* Running on Docker? (yes/no) : no
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Spark Read Hudi Tables with WARN Message [hudi]
Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on issue #10828:
URL: https://github.com/apache/hudi/issues/10828#issuecomment-1982363625
@michael1991 Ohh, I was checking 0.14.1. Not sure if you will find anything useful in debug logs but you can turn it on like this -
```
spark.sparkContext.setLogLevel("DEBUG")
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Spark Read Hudi Tables with WARN Message [hudi]
Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on issue #10828:
URL: https://github.com/apache/hudi/issues/10828#issuecomment-1980408152
@michael1991 Can you let us know more about when are you getting this issue. This is expected when the partition column datatype can't relate to hudi data type based on below mapping. Can you provide writer configuration and schema of the table.
<img width="950" alt="image" src="https://github.com/apache/hudi/assets/63430370/2ecef57f-1310-4a8e-9bfd-2b1ac98f5d35">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Spark Read Hudi Tables with WARN Message [hudi]
Posted by "michael1991 (via GitHub)" <gi...@apache.org>.
michael1991 commented on issue #10828:
URL: https://github.com/apache/hudi/issues/10828#issuecomment-1982196067
I'm using Spark 3.3.2 with Hudi 0.12.3 to write table a, meanwhile i'm using Spark 3.3.2 with Hudi 0.14.1 to read from table a and write to table b. All tables are COW type, just dataframe.write.mode("append").format("hudi").save(path) and spark.read.format("hudi").load(path).
Or is it possible to enable debug log to find more helpful things?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Spark Read Hudi Tables with WARN Message [hudi]
Posted by "michael1991 (via GitHub)" <gi...@apache.org>.
michael1991 commented on issue #10828:
URL: https://github.com/apache/hudi/issues/10828#issuecomment-2160744749
> @michael1991 Not sure if this one is related to warning.
Hi @ad1happy2go , I resolved this warning message by setting `"hoodie.datasource.write.partitionpath.urlencode" -> "true",`, seems if I enable Hive Style Partitioning, urlencode setting must be set to avoid this warning message.
<img width="1798" alt="image" src="https://github.com/apache/hudi/assets/6918592/a247b5e7-6b10-4d83-8177-06969aacd187">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Spark Read Hudi Tables with WARN Message [hudi]
Posted by "Gatsby-Lee (via GitHub)" <gi...@apache.org>.
Gatsby-Lee commented on issue #10828:
URL: https://github.com/apache/hudi/issues/10828#issuecomment-2715644571
> > [@michael1991](https://github.com/michael1991) Not sure if this one is related to warning.
>
> Hi [@ad1happy2go](https://github.com/ad1happy2go) , I resolved this warning message by setting `"hoodie.datasource.write.partitionpath.urlencode" -> "true",`, seems if I enable Hive Style Partitioning, urlencode setting must be set to avoid this warning message.
Hi, have you noticed any functionality issues other than the "WARN" msg?
I don't explicitly set "hoodie.datasource.write.partitionpath.urlencode=true" ( or even with false ).
I am curious if this is just warning or break any functionalities.
Thank you
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Spark Read Hudi Tables with WARN Message [hudi]
Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on issue #10828:
URL: https://github.com/apache/hudi/issues/10828#issuecomment-1980475279
Can you do newDetails.printSchema and check the datatype of req_date and req_hour
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Spark Read Hudi Tables with WARN Message [hudi]
Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on issue #10828:
URL: https://github.com/apache/hudi/issues/10828#issuecomment-1983184313
@michael1991 Not sure if this one is related to warning.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Spark Read Hudi Tables with WARN Message [hudi]
Posted by "michael1991 (via GitHub)" <gi...@apache.org>.
michael1991 closed issue #10828: [SUPPORT] Spark Read Hudi Tables with WARN Message
URL: https://github.com/apache/hudi/issues/10828
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Spark Read Hudi Tables with WARN Message [hudi]
Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on issue #10828:
URL: https://github.com/apache/hudi/issues/10828#issuecomment-1980644143
@michael1991 Ideally you should not see this warning if both are strings. When are you getting this issue? On Read or Write? Can you provide some reproducible script if possible?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Spark Read Hudi Tables with WARN Message [hudi]
Posted by "michael1991 (via GitHub)" <gi...@apache.org>.
michael1991 commented on issue #10828:
URL: https://github.com/apache/hudi/issues/10828#issuecomment-1980451478
Thanks for your quick response! @ad1happy2go
Spark writer configuration:
```scala
newDetails.write.format("hudi")
.option(RECORDKEY_FIELD.key(), "id")
.option(PRECOMBINE_FIELD.key(), "id")
.option(SCHEMA_EVOLUTION_ENABLED.key(), "true")
.option(DATABASE_NAME.key(), "db_prod")
.option(INDEX_TYPE.key(), "BLOOM")
.option(COMBINE_BEFORE_UPSERT.key(), "false")
.option(INSERT_PARALLELISM_VALUE.key(), "30")
.option(UPSERT_PARALLELISM_VALUE.key(), "200")
.option(CLEANER_COMMITS_RETAINED.key(), "4")
.option(ASYNC_CLEAN.key(), "false")
.option(PARTITIONPATH_FIELD.key(), "req_date,req_hour")
.option(TBL_NAME.key(), TBL_LOG_INCREMENT_DETAILS_NAME)
.option(OPERATION.key(), INSERT_OPERATION_OPT_VAL)
.option(WRITE_PAYLOAD_CLASS_NAME.key(), CUSTOM_PAYLOAD_CLASS)
.option(COPY_ON_WRITE_RECORD_SIZE_ESTIMATE.key(), "64")
.mode(SaveMode.Append).save(TBL_LOG_INCREMENT_DETAILS_PATH)
```
Schema just like: id string, req_date string, req_hour string, etc.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Spark Read Hudi Tables with WARN Message [hudi]
Posted by "michael1991 (via GitHub)" <gi...@apache.org>.
michael1991 commented on issue #10828:
URL: https://github.com/apache/hudi/issues/10828#issuecomment-1980495925
<img width="437" alt="image" src="https://github.com/apache/hudi/assets/6918592/2b66c865-4207-47ef-8416-93dfc6d2d5b8">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Spark Read Hudi Tables with WARN Message [hudi]
Posted by "michael1991 (via GitHub)" <gi...@apache.org>.
michael1991 commented on issue #10828:
URL: https://github.com/apache/hudi/issues/10828#issuecomment-1982621163
I find HoodieFileIndex.scala:367 at branch "release-0.14.1" as below
<img width="619" alt="image" src="https://github.com/apache/hudi/assets/6918592/45dda3db-3b98-4f99-920e-f9f30b1d42de">
Seems using index to filter files from metadata, am i right?
If so, the table is using bloom filter with metadata disabled, would it be the root cause?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Spark Read Hudi Tables with WARN Message [hudi]
Posted by "michael1991 (via GitHub)" <gi...@apache.org>.
michael1991 commented on issue #10828:
URL: https://github.com/apache/hudi/issues/10828#issuecomment-2160641880
I use spark3.5+hudi0.15.0 to write and read same table, this warning message could be seen too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org