You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "wypoon (via GitHub)" <gi...@apache.org> on 2023/06/07 01:03:20 UTC
[GitHub] [iceberg] wypoon opened a new pull request, #7790: Spark 3.4: Allow write modes to be specified in SQLConf
wypoon opened a new pull request, #7790:
URL: https://github.com/apache/iceberg/pull/7790
Currently, the write delete/update/merge mode for a table (copy-on-write or merge-on-read) is specified as a table property in the table. For some use cases, it makes sense to want to use copy-on-write for some Spark jobs writing to a table and merge-on-read for other jobs writing to the same table. Constantly toggling the table property is not a usable workaround.
In this change, we allow the mode to be specified in Spark's SQLConf.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] wypoon commented on pull request #7790: Spark 3.4: Allow write mode (copy-on-write/merge-on-read) to be specified in SQLConf
Posted by "wypoon (via GitHub)" <gi...@apache.org>.
wypoon commented on PR #7790:
URL: https://github.com/apache/iceberg/pull/7790#issuecomment-1601319970
> I think read configs like `read.split.*` should also be configurable in SQL session. @wypoon can you initiate a discussion on dev list?
@manuzhang you can add your voice to the thread I already started on the dev list.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] wypoon commented on pull request #7790: Spark 3.4: Allow write modes to be specified in SQLConf
Posted by "wypoon (via GitHub)" <gi...@apache.org>.
wypoon commented on PR #7790:
URL: https://github.com/apache/iceberg/pull/7790#issuecomment-1581042806
@pan3793 I think it would be on a case-by-case basis depending on the use case for each configuration.
cc: @rdblue @aokolnychyi @RussellSpitzer @szehon-ho
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] manuzhang commented on pull request #7790: Spark 3.4: Allow write mode (copy-on-write/merge-on-read) to be specified in SQLConf
Posted by "manuzhang (via GitHub)" <gi...@apache.org>.
manuzhang commented on PR #7790:
URL: https://github.com/apache/iceberg/pull/7790#issuecomment-1596928127
I think read configs like `read.split.*` should also be configurable in SQL session. @wypoon can you initiate a discussion on dev list?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] pan3793 commented on pull request #7790: Spark 3.4: Allow write modes to be specified in SQLConf
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #7790:
URL: https://github.com/apache/iceberg/pull/7790#issuecomment-1580282286
I opened [another PR](https://github.com/apache/iceberg/pull/7733) also aims to expose some configuration to SQL conf.
Seems that Iceberg only allows a small set of configurations to be overwritten by SQL session configuration. I'm wondering if we expose all configurations to SQL conf.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org