You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2021/05/28 02:33:44 UTC
[GitHub] [iceberg] jshmchenxi opened a new pull request #2648: Docs: Add cache-enabled to catalog property list
jshmchenxi opened a new pull request #2648:
URL: https://github.com/apache/iceberg/pull/2648
By default the catalog cache is enabled. **In one spark-sql session**, same sql over a changing iceberg table will return the same result even if the table has changed. The reason is that `loadTable` method is cached and we couldn't get latest snapshots of the table. If we turn off the cache, we can always get the latest result within one session.
So I think the `cache-enabled` property should be in the docs as it is an important configuration.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue merged pull request #2648: Docs: Add cache-enabled to catalog property list
Posted by GitBox <gi...@apache.org>.
rdblue merged pull request #2648:
URL: https://github.com/apache/iceberg/pull/2648
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on pull request #2648: Docs: Add cache-enabled to catalog property list
Posted by GitBox <gi...@apache.org>.
rdblue commented on pull request #2648:
URL: https://github.com/apache/iceberg/pull/2648#issuecomment-861684800
Looks great. Thanks @jshmchenxi!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] jshmchenxi edited a comment on pull request #2648: Docs: Add cache-enabled to catalog property list
Posted by GitBox <gi...@apache.org>.
jshmchenxi edited a comment on pull request #2648:
URL: https://github.com/apache/iceberg/pull/2648#issuecomment-855189040
@kbendick Yes, there is also a `cache-enabled` option in [FlinkCatalogFactory](https://github.com/apache/iceberg/blob/master/flink/src/main/java/org/apache/iceberg/flink/FlinkCatalogFactory.java#L129)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on pull request #2648: Docs: Add cache-enabled to catalog property list
Posted by GitBox <gi...@apache.org>.
kbendick commented on pull request #2648:
URL: https://github.com/apache/iceberg/pull/2648#issuecomment-855000885
This PR is related to this one: https://github.com/apache/iceberg/pull/2659
Though I'm not sure if cache-enabled is supported in Flink (I might be wrong).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] jshmchenxi commented on pull request #2648: Docs: Add cache-enabled to catalog property list
Posted by GitBox <gi...@apache.org>.
jshmchenxi commented on pull request #2648:
URL: https://github.com/apache/iceberg/pull/2648#issuecomment-855189040
@kbendick Yes, Flink also has a `cache-enabled` option in [FlinkCatalogFactory](https://github.com/apache/iceberg/blob/master/flink/src/main/java/org/apache/iceberg/flink/FlinkCatalogFactory.java#L129)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] jshmchenxi commented on a change in pull request #2648: Docs: Add cache-enabled to catalog property list
Posted by GitBox <gi...@apache.org>.
jshmchenxi commented on a change in pull request #2648:
URL: https://github.com/apache/iceberg/pull/2648#discussion_r651398171
##########
File path: site/docs/spark-configuration.md
##########
@@ -59,6 +59,7 @@ Both catalogs are configured using properties nested under the catalog name. Com
| spark.sql.catalog._catalog-name_.default-namespace | default | The default current namespace for the catalog |
| spark.sql.catalog._catalog-name_.uri | thrift://host:port | Metastore connect URI; default from `hive-site.xml` |
| spark.sql.catalog._catalog-name_.warehouse | hdfs://nn:8020/warehouse/path | Base path for the warehouse directory |
+| spark.sql.catalog._catalog-name_.warehouse.cache-enabled | `true` or `false` | Whether to enable catalog cache, default value is `true` |
Review comment:
Yes, that was a mistake.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #2648: Docs: Add cache-enabled to catalog property list
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #2648:
URL: https://github.com/apache/iceberg/pull/2648#discussion_r651325778
##########
File path: site/docs/spark-configuration.md
##########
@@ -59,6 +59,7 @@ Both catalogs are configured using properties nested under the catalog name. Com
| spark.sql.catalog._catalog-name_.default-namespace | default | The default current namespace for the catalog |
| spark.sql.catalog._catalog-name_.uri | thrift://host:port | Metastore connect URI; default from `hive-site.xml` |
| spark.sql.catalog._catalog-name_.warehouse | hdfs://nn:8020/warehouse/path | Base path for the warehouse directory |
+| spark.sql.catalog._catalog-name_.warehouse.cache-enabled | `true` or `false` | Whether to enable catalog cache, default value is `true` |
Review comment:
I don't think it should include `warehouse`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org