You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2021/12/30 08:26:22 UTC
[GitHub] [hudi] taisenki opened a new issue #4477: [SUPPORT]using spark on TimestampBasedKeyGenerator has no result when query by partition column
taisenki opened a new issue #4477:
URL: https://github.com/apache/hudi/issues/4477
hudi table options:
hoodie.datasource.write.table.type=MERGE_ON_READ
hoodie.datasource.write.recordkey.field=id
hoodie.datasource.write.partitionpath.field=data_date
hoodie.datasource.write.precombine.field=ts
hoodie.datasource.write.keygenerator.class=org.apache.hudi.keygen.TimestampBasedKeyGenerator
hoodie.deltastreamer.keygen.timebased.timestamp.type=DATE_STRING
hoodie.deltastreamer.keygen.timebased.output.dateformat=yyyy/MM/dd
hoodie.deltastreamer.keygen.timebased.timezone="GMT+8:00"
hoodie.deltastreamer.keygen.timebased.input.dateformat=yyyy-MM-dd
hoodie.datasource.hive_sync.database=hudi
hoodie.datasource.hive_sync.enable=true
hoodie.datasource.hive_sync.partition_extractor_class=org.apache.hudi.hive.SlashEncodedDayPartitionValueExtractor
when i query table with data_date, there're no result.
like this:
```shell
scala> spark.read.format("hudi").load(tablePath).where("id = '225709893'").select("data_date").where("data_date = '2018-09-23'").show
+---------+
|data_date|
+---------+
+---------+
scala> spark.read.format("hudi").load(tablePath).where("id = '225709893'").select("data_date").show
+----------+
| data_date|
+----------+
|2018-09-23|
|2018-09-22|
|2018-09-19|
|2018-09-21|
|2018-09-25|
|2018-09-20|
|2018-09-24|
+----------+
```
I follow the code, and find the partition use string '2018/09/23' and query patten is '2018-09-23' in HoodieFileIndex#prunePartition.
Is there any way to introduce the hoodie.deltastreamer.keygen.timebased.output.dateformat configuration for query conversion?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] YannByron commented on issue #4477: [SUPPORT]using spark on TimestampBasedKeyGenerator has no result when query by partition column
Posted by GitBox <gi...@apache.org>.
YannByron commented on issue #4477:
URL: https://github.com/apache/hudi/issues/4477#issuecomment-1008254173
@nsivabalan I haven't created a ticket yet. You can create it and assign to me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] YannByron edited a comment on issue #4477: [SUPPORT]using spark on TimestampBasedKeyGenerator has no result when query by partition column
Posted by GitBox <gi...@apache.org>.
YannByron edited a comment on issue #4477:
URL: https://github.com/apache/hudi/issues/4477#issuecomment-1008254173
to https://issues.apache.org/jira/browse/HUDI-3204.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #4477: [SUPPORT]using spark on TimestampBasedKeyGenerator has no result when query by partition column
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #4477:
URL: https://github.com/apache/hudi/issues/4477#issuecomment-1008231035
@YannByron : may I know whats the tracking ticket. If not, can we create one for the issue reported in this github issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] taisenki commented on issue #4477: [SUPPORT]using spark on TimestampBasedKeyGenerator has no result when query by partition column
Posted by GitBox <gi...@apache.org>.
taisenki commented on issue #4477:
URL: https://github.com/apache/hudi/issues/4477#issuecomment-1003868316
> thanks for reporting. for the time being, can you try query using `YYYY/MM/DD` format.
yyyy/MM/DD can find the partition file, but there is no equal data in the table for value store in data_date is 'YYYY-MM-DD'
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #4477: [SUPPORT]using spark on TimestampBasedKeyGenerator has no result when query by partition column
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #4477:
URL: https://github.com/apache/hudi/issues/4477#issuecomment-1003866665
thanks for reporting. for the time being, can you try query using `YYYY/MM/DD` format.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] taisenki commented on issue #4477: [SUPPORT]using spark on TimestampBasedKeyGenerator has no result when query by partition column
Posted by GitBox <gi...@apache.org>.
taisenki commented on issue #4477:
URL: https://github.com/apache/hudi/issues/4477#issuecomment-1004467354
@nsivabalan result like this:
```
scala> spark.read.format("hudi").load(tablePath).where("id = '225709893'").select("data_date").where("data_date = '2018/09/23'").show
22/01/04 09:35:09 WARN lineage.LineageWriter: Lineage directory /var/log/spark/lineage doesn't exist or is not writable. Lineage for this application will be disabled.
22/01/04 09:35:27 WARN util.Utils: Truncated the string representation of a plan since it was too large. This behavior can be adjusted by setting 'spark.debug.maxToStringFields' in SparkEnv.conf.
+---------+
|data_date|
+---------+
+---------+
scala>
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] YannByron commented on issue #4477: [SUPPORT]using spark on TimestampBasedKeyGenerator has no result when query by partition column
Posted by GitBox <gi...@apache.org>.
YannByron commented on issue #4477:
URL: https://github.com/apache/hudi/issues/4477#issuecomment-1004474344
@taisenki @nsivabalan
I found there are different behaviors in cow and mor type. i'll take this up and reply asap.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #4477: [SUPPORT]using spark on TimestampBasedKeyGenerator has no result when query by partition column
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #4477:
URL: https://github.com/apache/hudi/issues/4477#issuecomment-1005812141
thanks Yann for helping out.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #4477: [SUPPORT]using spark on TimestampBasedKeyGenerator has no result when query by partition column
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #4477:
URL: https://github.com/apache/hudi/issues/4477#issuecomment-1008821285
Closing it as we have a tracking jira. Yann will work on putting up a fix towards this end. thanks for reporting!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] YannByron edited a comment on issue #4477: [SUPPORT]using spark on TimestampBasedKeyGenerator has no result when query by partition column
Posted by GitBox <gi...@apache.org>.
YannByron edited a comment on issue #4477:
URL: https://github.com/apache/hudi/issues/4477#issuecomment-1004474344
@taisenki @nsivabalan
I found there are different behaviors in cow and mor type using the same config above. i'll take this up and reply asap.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan closed issue #4477: [SUPPORT]using spark on TimestampBasedKeyGenerator has no result when query by partition column
Posted by GitBox <gi...@apache.org>.
nsivabalan closed issue #4477:
URL: https://github.com/apache/hudi/issues/4477
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] taisenki commented on issue #4477: [SUPPORT]using spark on TimestampBasedKeyGenerator has no result when query by partition column
Posted by GitBox <gi...@apache.org>.
taisenki commented on issue #4477:
URL: https://github.com/apache/hudi/issues/4477#issuecomment-1002923542
My env hudi 0.10.0 , spark 2.4.0-cdh6.3.2
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org