You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "jhchee (via GitHub)" <gi...@apache.org> on 2023/04/24 12:28:00 UTC

[GitHub] [hudi] jhchee opened a new issue, #8561: [SUPPORT] How to pass write options to Spark SQL

jhchee opened a new issue, #8561:
URL: https://github.com/apache/hudi/issues/8561

   **Describe the problem you faced**
   Currently, I'm upserting a table using the Spark MERGE INTO command.
   I wish to increase write performance via bucket index.
   How do I pass write config when executing spark.sql("MERGE INTO xxx")
   
   **To Reproduce**
   
   Steps to reproduce the behavior:
   
   1.
   2.
   3.
   4.
   
   **Expected behavior**
   
   A clear and concise description of what you expected to happen.
   
   **Environment Description**
   
   * Hudi version :
   
   * Spark version :
   
   * Hive version :
   
   * Hadoop version :
   
   * Storage (HDFS/S3/GCS..) :
   
   * Running on Docker? (yes/no) :
   
   
   **Additional context**
   
   Add any other context about the problem here.
   
   **Stacktrace**
   
   ```Add the stacktrace of the error.```
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] jhchee commented on issue #8561: [SUPPORT] How to pass write options to Spark SQL

Posted by "jhchee (via GitHub)" <gi...@apache.org>.
jhchee commented on issue #8561:
URL: https://github.com/apache/hudi/issues/8561#issuecomment-1520075536

   Figure out I can pass write config via SET command
   ```
   spark.sql("SET hoodie.index.type = BUCKET");
   spark.sql("SET hoodie.index.bucket.engine = SIMPLE");
   spark.sql("SET hoodie.storage.layout.partitioner.class = org.apache.hudi.table.action.commit.SparkBucketIndexPartitioner");
   spark.sql("SET hoodie.bucket.index.num.buckets = 10");
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] jhchee closed issue #8561: [SUPPORT] How to pass write options to Spark SQL

Posted by "jhchee (via GitHub)" <gi...@apache.org>.
jhchee closed issue #8561: [SUPPORT] How to pass write options to Spark SQL
URL: https://github.com/apache/hudi/issues/8561


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org