You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@hudi.apache.org by "ksoullpwk (via GitHub)" <gi...@apache.org> on 2024/10/16 13:23:25 UTC

[I] [SUPPORT] Hudi enable metadata table feature when running alter table via spark.sql(query) [hudi]

ksoullpwk opened a new issue, #12114:
URL: https://github.com/apache/hudi/issues/12114

**_Tips before filing an issue_**

- Have you gone through our [FAQs](https://hudi.apache.org/learn/faq/)?
- link is dead

- Join the mailing list to engage in conversations and get faster support at dev-subscribe@hudi.apache.org.

- If you have triaged this as a bug, then file an [issue](https://issues.apache.org/jira/projects/HUDI/issues) directly.

**Describe the problem you faced**

A clear and concise description of the problem.
I have a Hudi table with `"hoodie.metadata.enable"=false`. Then, I try to delete the partition with `spark.sql(s"ALTER TABLE $tableName DROP IF EXISTS PARTITION ($partitionSpec)").collect()`. The metadata table somehow becomes enable.

**To Reproduce**

Steps to reproduce the behavior:

1. Write table with `"hoodie.metadata.enable"=false`.
2. Run some delete partition commands via `spark.sql()`.
3. The `.hoodie/metadata/` somehow has been created.

**Expected behavior**

A clear and concise description of what you expected to happen.
The metadata table should still disable

**Environment Description**

* Hudi version : 0.15.0

* Spark version : 3.3

* Hive version : 2.3.9

* Hadoop version :

* Storage (HDFS/S3/GCS..) : GCS

* Running on Docker? (yes/no) : no

**Additional context**

Add any other context about the problem here.

**Stacktrace**

```Add the stacktrace of the error.```

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [SUPPORT] Hudi enable metadata table feature when running alter table via spark.sql(query) [hudi]

Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.

ad1happy2go commented on issue #12114:
URL: https://github.com/apache/hudi/issues/12114#issuecomment-2419035940

   @ksoullpwk hoodie.metadata.enable is the writer properties and not table property so has to be passed along with writer also. Thats why setting with spark.conf works for you. So we should be good here.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [SUPPORT] Hudi enable metadata table feature when running alter table via spark.sql(query) [hudi]

Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.

ad1happy2go closed issue #12114: [SUPPORT] Hudi enable metadata table feature when running alter table via spark.sql(query)
URL: https://github.com/apache/hudi/issues/12114


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [SUPPORT] Hudi enable metadata table feature when running alter table via spark.sql(query) [hudi]

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.

danny0405 commented on issue #12114:
URL: https://github.com/apache/hudi/issues/12114#issuecomment-2418226171

   Should be a bug, we should comply with the existing table config for metadata table.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [SUPPORT] Hudi enable metadata table feature when running alter table via spark.sql(query) [hudi]

Posted by "ksoullpwk (via GitHub)" <gi...@apache.org>.

ksoullpwk commented on issue #12114:
URL: https://github.com/apache/hudi/issues/12114#issuecomment-2418359460

   My workaround now is manually set the spark conf by running `spark.conf.set("hoodie.metadata.enable", false)` before running `spark.sql(...)`. It seems to be okay now.
   I guess it might cause by there is no `hoodie.metadata.enable` property and they read the default as true.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [SUPPORT] Hudi enable metadata table feature when running alter table via spark.sql(query) [hudi]

Posted by "ksoullpwk (via GitHub)" <gi...@apache.org>.

ksoullpwk commented on issue #12114:
URL: https://github.com/apache/hudi/issues/12114#issuecomment-2421359301

   Do we also need to set it when running `spark.table(tableName)`?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [SUPPORT] Hudi enable metadata table feature when running alter table via spark.sql(query) [hudi]

Posted by "rangareddy (via GitHub)" <gi...@apache.org>.

rangareddy commented on issue #12114:
URL: https://github.com/apache/hudi/issues/12114#issuecomment-2419054167

   To disable metadata while performing any operation with `spark.sql()`, we need to configure it as follows:
   
   ```scala
   spark.sql("SET hoodie.metadata.enable=false")
   ```
   
   **Sample Code:**
   
   ```scala
     // spark.sql("SET hoodie.metadata.enable=false")
   
     spark.sql(
       """
         |CREATE TABLE IF NOT EXISTS t_test (
         |  `id` VARCHAR(20),
         |  `name` VARCHAR(10),
         |  `age` INT,
         |  `ts` Long
         |) USING HUDI TBLPROPERTIES (primaryKey = 'id,name', preCombineField = 'ts')
         | LOCATION '/tmp/hudi/t_test'
       """.stripMargin)
   
     spark.sql("INSERT INTO t_test VALUES (1, \"hello\", 42,  1695159649087)")
   
     spark.sql("SELECT * FROM t_test").show(false)
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org