You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2022/04/07 09:14:14 UTC
[GitHub] [hudi] KnightChess opened a new issue, #5248: [QUESION] Should filter prop "hoodie.datasource.write.operation" when use spark sql create table
KnightChess opened a new issue, #5248:
URL: https://github.com/apache/hudi/issues/5248
when I use spark sql create table and set **hoodie.datasource.write.operation**=upsert.
delete sql （like pr #5215 ）, insert overwrite sql etc will still use **hoodie.datasource.write.operation** to update record, not delete, insert_overwrite etc.
eg:
create a table and set hoodie.datasource.write.operation upsert
when I use sql to delete, the delete operation key will be overwrite by hoodie.datasource.write.operation from table or env, **OPERATION.key -> DataSourceWriteOptions.DELETE_OPERATION_OPT_VAL** will not effect, overwrite to **upsert**
```scala
withSparkConf(sparkSession, hoodieCatalogTable.catalogProperties) {
Map(
"path" -> path,
RECORDKEY_FIELD.key -> hoodieCatalogTable.primaryKeys.mkString(","),
TBL_NAME.key -> tableConfig.getTableName,
HIVE_STYLE_PARTITIONING.key -> tableConfig.getHiveStylePartitioningEnable,
URL_ENCODE_PARTITIONING.key -> tableConfig.getUrlEncodePartitioning,
KEYGENERATOR_CLASS_NAME.key -> classOf[SqlKeyGenerator].getCanonicalName,
SqlKeyGenerator.ORIGIN_KEYGEN_CLASS_NAME -> tableConfig.getKeyGeneratorClassName,
OPERATION.key -> DataSourceWriteOptions.DELETE_OPERATION_OPT_VAL,
PARTITIONPATH_FIELD.key -> tableConfig.getPartitionFieldProp,
HiveSyncConfig.HIVE_SYNC_MODE.key -> HiveSyncMode.HMS.name(),
HiveSyncConfig.HIVE_SUPPORT_TIMESTAMP_TYPE.key -> "true",
HoodieWriteConfig.DELETE_PARALLELISM_VALUE.key -> "200",
SqlKeyGenerator.PARTITION_SCHEMA -> partitionSchema.toDDL
)
}
```
so, when use sql, what about don't write it to hoodie.properties, confine it when sql check, command generated itself in runtime.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] dongkelun commented on issue #5248: [QUESION] Should filter prop "hoodie.datasource.write.operation" when use spark sql create table?
Posted by GitBox <gi...@apache.org>.
dongkelun commented on issue #5248:
URL: https://github.com/apache/hudi/issues/5248#issuecomment-1093602094
Not only `hoodie.datasource.write.operation`, but also other properties such as `hoodie.table.name` has the same problem.
I think we should find all the parameters with similar problems
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] KnightChess commented on issue #5248: [QUESION] Should filter prop "hoodie.datasource.write.operation" when use spark sql create table?
Posted by GitBox <gi...@apache.org>.
KnightChess commented on issue #5248:
URL: https://github.com/apache/hudi/issues/5248#issuecomment-1091660722
@XuQianJin-Stars if I not set this parameter when create table, anything will be ok when use sql to insert, delete or other, because statement will set itself in runtime.
So, I think may be **hoodie.datasource.write.operation** param only be used when we use df to write data, rather than as a table parameter. Sql only use runtime operator.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #5248: [QUESION] Should filter prop "hoodie.datasource.write.operation" when use spark sql create table?
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #5248:
URL: https://github.com/apache/hudi/issues/5248#issuecomment-1100570586
@XuQianJin-Stars : Can you file a tracking jira and follow up on the issue. seems like we need to fix this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua commented on issue #5248: [QUESION] Should filter prop "hoodie.datasource.write.operation" when use spark sql create table?
Posted by GitBox <gi...@apache.org>.
yihua commented on issue #5248:
URL: https://github.com/apache/hudi/issues/5248#issuecomment-1115761056
We’re going to track the fix in HUDI-4001. Closing this issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua closed issue #5248: [QUESION] Should filter prop "hoodie.datasource.write.operation" when use spark sql create table?
Posted by GitBox <gi...@apache.org>.
yihua closed issue #5248: [QUESION] Should filter prop "hoodie.datasource.write.operation" when use spark sql create table?
URL: https://github.com/apache/hudi/issues/5248
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua commented on issue #5248: [QUESION] Should filter prop "hoodie.datasource.write.operation" when use spark sql create table?
Posted by GitBox <gi...@apache.org>.
yihua commented on issue #5248:
URL: https://github.com/apache/hudi/issues/5248#issuecomment-1113706163
I created a Jira ticket to track the fix: HUDI-4001.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] XuQianJin-Stars commented on issue #5248: [QUESION] Should filter prop "hoodie.datasource.write.operation" when use spark sql create table?
Posted by GitBox <gi...@apache.org>.
XuQianJin-Stars commented on issue #5248:
URL: https://github.com/apache/hudi/issues/5248#issuecomment-1091647916
hi @KnightChess The problem of refraction is to sort out those parameters that cannot be covered.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org