You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "matthiasdg (via GitHub)" <gi...@apache.org> on 2023/05/15 09:46:10 UTC
[GitHub] [hudi] matthiasdg opened a new issue, #8713: [SUPPORT] Incremental queries with Spark through Hive metastore
matthiasdg opened a new issue, #8713:
URL: https://github.com/apache/hudi/issues/8713
**Describe the problem you faced**
Just wondering whether it's possible to do incremental queries using Spark with a Hudi table that was synced to Hive metastore using the HiveSyncTool; i.e., something like`sparkSession.table(tableName).select("*").where(...)` .
Did not find any documentation or FAQ about this.
Encountered things like in `docker/demo/hive-incremental-cow.commands`: `set hoodie.stock_ticks_cow.consume.mode=INCREMENTAL;`, but not sure if that also applies here; is this only for going through HiveServer? (We're not running HiveServer, only metastore). Can I somehow pass these as config parameters to the Spark session (did some experiments, also in Spark SQL, but did not seem to be working).
Also, why is there a `'_hoodie_commit_time' > '${min.commit.time}'` in the example query in the file I mentioned? Shouldn't that already be the case because the query is incremental?
**Environment Description**
* Hudi version : 0.10
* Spark version : 3.1.2
* Hive version : HMS 3.0
* Hadoop version : 3.2
* Storage (HDFS/S3/GCS..) : ADLS Gen 2
* Running on Docker? (yes/no) : Kubernetes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] matthiasdg commented on issue #8713: [SUPPORT] Incremental queries with Spark through Hive metastore
Posted by "matthiasdg (via GitHub)" <gi...@apache.org>.
matthiasdg commented on issue #8713:
URL: https://github.com/apache/hudi/issues/8713#issuecomment-1551439622
@ad1happy2go I have no issue with the incremental query cf. the page you linked. However, that example requires passing in the root path of the table. I was wondering if there is an option like:
`spark.read.option(...).table("name_of_table_in_hive")` -> that does not seem to work. Main reason I'd want this is that I then don't have to care about the path of the table. I can of course get the path from the hive table using `org.apache.spark.sql.catalyst.catalog.SessionCatalog.getTableMetadata` and then pass that in, but was wondering if there is a more direct way.
I understand it's about getting data above a certain instant time, but would have thought this is already fulfilled if you load the table with an incremental query that has a start instant of `${min.commit.time}`. Are there cases where this additional filtering is necessary?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] ad1happy2go commented on issue #8713: [SUPPORT] Incremental queries with Spark through Hive metastore
Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on issue #8713:
URL: https://github.com/apache/hudi/issues/8713#issuecomment-1642074078
Closing out the issue since this PR and also documentation PR https://github.com/apache/hudi/pull/8880 is merged, Closing the issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] matthiasdg commented on issue #8713: [SUPPORT] Incremental queries with Spark through Hive metastore
Posted by "matthiasdg (via GitHub)" <gi...@apache.org>.
matthiasdg commented on issue #8713:
URL: https://github.com/apache/hudi/issues/8713#issuecomment-1556734576
@kazdy think that would be fine yes (when merged should then pbly also make its way to one of the documentation pages on the website)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] ad1happy2go commented on issue #8713: [SUPPORT] Incremental queries with Spark through Hive metastore
Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on issue #8713:
URL: https://github.com/apache/hudi/issues/8713#issuecomment-1551541561
@matthiasdg Thanks for the clarification. Let me try to reproduce too.
This is something we should support, was not aware incremental query options doesn't work with table name directly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] ad1happy2go commented on issue #8713: [SUPPORT] Incremental queries with Spark through Hive metastore
Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on issue #8713:
URL: https://github.com/apache/hudi/issues/8713#issuecomment-1550059773
@matthiasdg You can simply do incremental queries with spark datafame api also for the tables which are synced with hive.
https://hudi.apache.org/docs/querying_data/#spark-incr-query
'_hoodie_commit_time' > '${min.commit.time}' --> It is more like min_instant_time. The instant time above which we want to get data.
Let me know in case it doesn't clarifies your issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] kazdy commented on issue #8713: [SUPPORT] Incremental queries with Spark through Hive metastore
Posted by "kazdy (via GitHub)" <gi...@apache.org>.
kazdy commented on issue #8713:
URL: https://github.com/apache/hudi/issues/8713#issuecomment-1555242388
Hi @matthiasdg would spark table-valued function be sufficient?
Here's a PR implementing this: https://github.com/apache/hudi/pull/8729
```sql
select id, name, price, ts
from hudi_table_changes('mytable', 'latest_state', 'beginInstantTime', 'endInstantTime');
select id, name, price, ts
from hudi_table_changes_by_path('my/table/path', 'latest_state', 'beginInstantTime', 'endInstantTime');
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope closed issue #8713: [SUPPORT] Incremental queries with Spark through Hive metastore
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope closed issue #8713: [SUPPORT] Incremental queries with Spark through Hive metastore
URL: https://github.com/apache/hudi/issues/8713
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org