You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2022/07/22 22:01:55 UTC
[GitHub] [iceberg] szehon-ho opened a new issue, #5343: Allow table read/write options to be configured and/or enforced at catalog level using catalog properties
szehon-ho opened a new issue, #5343:
URL: https://github.com/apache/iceberg/issues/5343
#4011 allowed table properties to be set on new tables via catalog properties.
It would be nice to have these propagate to runtime table properties as well for read/write time.
Use case: There are a lot of table properties that should be overriden at runtime, see : https://iceberg.apache.org/docs/latest/configuration/ . For example, a lot of write-distribution-mode defaults to hash which cause unexpected shuffles (see https://github.com/apache/iceberg/issues/5224). Or users would like to override vectorization or read/write split size dynamically. But, in spark, spark.sql() cannot take any options, so users are stuck setting table properties for this.
This could be one way without changing Spark, for Iceberg to be able to override table properties at runtime for cases where there is no other override available.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] zhongyujiang commented on issue #5343: Allow table read/write options to be configured and/or enforced at catalog level using catalog properties
Posted by GitBox <gi...@apache.org>.
zhongyujiang commented on issue #5343:
URL: https://github.com/apache/iceberg/issues/5343#issuecomment-1326986534
@szehon-ho FYI: https://issues.apache.org/jira/browse/SPARK-36680, this might be the link you want if you haven't found it yet.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] szehon-ho commented on issue #5343: Allow table read/write options to be configured and/or enforced at catalog level using catalog properties
Posted by "szehon-ho (via GitHub)" <gi...@apache.org>.
szehon-ho commented on issue #5343:
URL: https://github.com/apache/iceberg/issues/5343#issuecomment-1587749497
Sure, i have checked with Spark folks recently, will follow up with them there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] szehon-ho commented on issue #5343: Allow table read/write options to be configured and/or enforced at catalog level using catalog properties
Posted by GitBox <gi...@apache.org>.
szehon-ho commented on issue #5343:
URL: https://github.com/apache/iceberg/issues/5343#issuecomment-1192981264
I see, I must have missed that discussion, if anyone has any link? If there a some issue or proposal there we could definitely take a look at it on spark side, hoping it will be do-able.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] szehon-ho commented on issue #5343: Allow table read/write options to be configured and/or enforced at catalog level using catalog properties
Posted by GitBox <gi...@apache.org>.
szehon-ho commented on issue #5343:
URL: https://github.com/apache/iceberg/issues/5343#issuecomment-1192978868
@SinghAsDev @RussellSpitzer @rdblue @aokolnychyi @dramaticlly if there's any thoughts on this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] szehon-ho closed issue #5343: Allow table read/write options to be configured and/or enforced at catalog level using catalog properties
Posted by "szehon-ho (via GitHub)" <gi...@apache.org>.
szehon-ho closed issue #5343: Allow table read/write options to be configured and/or enforced at catalog level using catalog properties
URL: https://github.com/apache/iceberg/issues/5343
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on issue #5343: Allow table read/write options to be configured and/or enforced at catalog level using catalog properties
Posted by GitBox <gi...@apache.org>.
rdblue commented on issue #5343:
URL: https://github.com/apache/iceberg/issues/5343#issuecomment-1192980287
I'm reluctant to create a broader work-around for this just because Spark can't pass options. What about adding the options in SQL hints to Spark? I think that was discussed earlier, but I don't know where it went.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] dramaticlly commented on issue #5343: Allow table read/write options to be configured and/or enforced at catalog level using catalog properties
Posted by "dramaticlly (via GitHub)" <gi...@apache.org>.
dramaticlly commented on issue #5343:
URL: https://github.com/apache/iceberg/issues/5343#issuecomment-1587751201
I am not 100% sure but #7732 seems promising
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] bitsondatadev commented on issue #5343: Allow table read/write options to be configured and/or enforced at catalog level using catalog properties
Posted by "bitsondatadev (via GitHub)" <gi...@apache.org>.
bitsondatadev commented on issue #5343:
URL: https://github.com/apache/iceberg/issues/5343#issuecomment-1585929913
@rdblue @szehon-ho can this be closed?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] szehon-ho commented on issue #5343: Allow table read/write options to be configured and/or enforced at catalog level using catalog properties
Posted by GitBox <gi...@apache.org>.
szehon-ho commented on issue #5343:
URL: https://github.com/apache/iceberg/issues/5343#issuecomment-1339829786
Thanks, I guess it did not go far. FWIW I think your proposal makes sense. Will have to see any spark folks can help with this if we can take if further.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org