You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2021/12/23 22:44:15 UTC

[GitHub] [hudi] gunjdesai opened a new issue #4437: Example for CREATE TABLE on TRINO using HUDI [QUESTION]

gunjdesai opened a new issue #4437:
URL: https://github.com/apache/hudi/issues/4437


   I am using `Spark Structured Streaming (3.1.1)` to read data from `Kafka` and use `HUDI (0.8.0)` as the storage system on S3 partitioning the data by date. (no problems with this section)
   
   I am looking to use `Trino (355)` to be able to query that data. As a pre-curser, I've already placed the `hudi-presto-bundle-0.8.0.jar` in `/data/trino/hive/`
   
   I created a table with the following schema
   
   ```
   CREATE TABLE table_new (
     columns, dt
   ) WITH (
     partitioned_by = ARRAY['dt'], 
     external_location = 's3a://bucket/location/',
     format = 'parquet'
   );
   ```
   Even after calling the below function, trino is unable to discover any partitions
   
   ```
   CALL system.sync_partition_metadata('schema', 'table_new', 'ALL')
   ```
   My assessment is that I am unable to create a table under trino using hudi largely due to the fact that I am not able to pass the right values under `WITH` Options. I am also unable to find a create table example under documentation for HUDI.
   
   I would really appreciate if anyone can give me a example for that, or point me to the right direction, if in case I've missed anything.
   
   Really appreciate the help
   
   **Environment Description**
   
   * Hudi version : 0.8.0
   
   * Spark version : 3.1.1
   
   * Hive version : 3.0.0 (Metastore only)
   
   * Hadoop version : 3.2.0
   
   * Storage (HDFS/S3/GCS..) : S3
   
   * Running on Docker? (yes/no) : Yes
   
   This is more of a question than an issue, sorry couldn't apply a label for it.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] gunjdesai commented on issue #4437: [QUESTION] Example for CREATE TABLE on TRINO using HUDI

Posted by GitBox <gi...@apache.org>.

gunjdesai commented on issue #4437:
URL: https://github.com/apache/hudi/issues/4437#issuecomment-1048787676


   Thanks @codope for the detailed description. I was under the assumption that you could directly create & query hudi tables using trino. Hence our setup was of trino using hive-metastore but not hive.
   Thanks to your detailed explanation, I will use the hudi-hive-sync to sync our tables and query them via Hudi and revert back to you on the same.
   In the meanwhile, I will also keep an eye out for HUDI connector for trino and try to integrate it as soon as it comes out. 
   
   Thank you once again for your help. Really appreciate it.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] nsivabalan commented on issue #4437: [QUESTION] Example for CREATE TABLE on TRINO using HUDI

Posted by GitBox <gi...@apache.org>.

nsivabalan commented on issue #4437:
URL: https://github.com/apache/hudi/issues/4437#issuecomment-1054648026


   thanks Sagar for the help. 
   will go ahead and close out the issue. If you come across any other issue, feel free to open a new issue. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] gunjdesai commented on issue #4437: [QUESTION] Example for CREATE TABLE on TRINO using HUDI

Posted by GitBox <gi...@apache.org>.

gunjdesai commented on issue #4437:
URL: https://github.com/apache/hudi/issues/4437#issuecomment-1015192029


   Hi Team, 
   
   any luck with this. I've tried asking this question on the slack channel for trino as well, but haven't got any luck there. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] nsivabalan commented on issue #4437: [QUESTION] Example for CREATE TABLE on TRINO using HUDI

Posted by GitBox <gi...@apache.org>.

nsivabalan commented on issue #4437:
URL: https://github.com/apache/hudi/issues/4437#issuecomment-1001763532


   @codope : Can you take this up please.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] gunjdesai commented on issue #4437: [QUESTION] Example for CREATE TABLE on TRINO using HUDI

Posted by GitBox <gi...@apache.org>.

gunjdesai commented on issue #4437:
URL: https://github.com/apache/hudi/issues/4437#issuecomment-1035939568


   Hi Team, 
   Any luck with this question ? I have actually combed through the slack channel and issues channel for `trino` and haven't had any luck. Would really appreciate any help here.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] nsivabalan commented on issue #4437: [QUESTION] Example for CREATE TABLE on TRINO using HUDI

Posted by GitBox <gi...@apache.org>.

nsivabalan commented on issue #4437:
URL: https://github.com/apache/hudi/issues/4437#issuecomment-1018586019


   @codope gentle ping to assist here.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] nsivabalan commented on issue #4437: [QUESTION] Example for CREATE TABLE on TRINO using HUDI

Posted by GitBox <gi...@apache.org>.

nsivabalan commented on issue #4437:
URL: https://github.com/apache/hudi/issues/4437#issuecomment-1036442624


   @yihua : can you help here please.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] codope commented on issue #4437: [QUESTION] Example for CREATE TABLE on TRINO using HUDI

Posted by GitBox <gi...@apache.org>.

codope commented on issue #4437:
URL: https://github.com/apache/hudi/issues/4437#issuecomment-1047448948

@gunjdesai First of all, apologies for the late response. I missed tracking this issue.

In its current state, we must [sync Hudi table to Hive metastore](https://hudi.apache.org/docs/syncing_metastore) in order to query through Trino. Our integration with Trino is via the hive connector. And that too, only for reads i.e. we cannot create hudi tables using Trino.

After syncing the Hudi table to Hive metastore and placing the hudi-presto-bundle in the hive plugin path, we can run snapshot queries on COW table and read-optimized queries on MOR table right away. You can look at an example in our [docker demo](https://hudi.apache.org/docs/docker_demo). That said, we are working on a [separate Hudi connector](https://github.com/trinodb/trino/pull/10228) to which we'll add the write support after the base PR is merged.

I would love to understand your requirements and help you out with the Hive sync and Trino setup to query Hudi tables. You can also help us by checking out the PR linked above and test for your tables. Feel free to directly ping me on Hudi slack or join our weekly [community syncs](https://hudi.apache.org/community/syncs).

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] nsivabalan closed issue #4437: [QUESTION] Example for CREATE TABLE on TRINO using HUDI

Posted by GitBox <gi...@apache.org>.

nsivabalan closed issue #4437:
URL: https://github.com/apache/hudi/issues/4437


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org