You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/02/08 12:14:17 UTC
[GitHub] [spark] MaxGekk opened a new pull request #31524: [SPARK-34401][SQL][DOCS] Update docs about altering cached tables/views
MaxGekk opened a new pull request #31524:
URL: https://github.com/apache/spark/pull/31524
### What changes were proposed in this pull request?
Update public docs of SQL commands about altering cached tables/views. For instance:
<img width="869" alt="Screenshot 2021-02-08 at 15 11 48" src="https://user-images.githubusercontent.com/1580697/107217940-fd3b8980-6a1f-11eb-98b9-9b2e3fe7f4ef.png">
### Why are the changes needed?
To inform users about commands behavior in altering cached tables or views.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
By running the command below and manually checking the docs:
```
$ SKIP_API=1 SKIP_SCALADOC=1 SKIP_PYTHONDOC=1 SKIP_RDOC=1 jekyll serve --watch
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on a change in pull request #31524: [SPARK-34401][SQL][DOCS] Update docs about altering cached tables/views
Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #31524:
URL: https://github.com/apache/spark/pull/31524#discussion_r572383191
##########
File path: docs/sql-ref-syntax-ddl-drop-table.md
##########
@@ -26,6 +26,8 @@ if the table is not `EXTERNAL` table. If the table is not present it throws an e
In case of an external table, only the associated metadata information is removed from the metastore database.
+If the table is cached, the command uncaches the table and all its dependants such as views. Thereby the dependants should be cached again explicitly.
Review comment:
nit: dependants -> dependents
##########
File path: docs/sql-ref-syntax-ddl-alter-view.md
##########
@@ -28,6 +28,8 @@ the name of a view to a different name, set and unset the metadata of the view b
Renames the existing view. If the new view name already exists in the source database, a `TableAlreadyExistsException` is thrown. This operation
does not support moving the views across databases.
+If the view is cached, the command clears cached data of the view and all its dependents that refer to it. View's cache will be lazily filled when the next time the view or the dependents are accessed. The command leaves view's dependents as uncached.
Review comment:
Hmm is this correct? should we say "The view's cache will be lazily filled when the next time it is accessed"?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #31524: [SPARK-34401][SQL][DOCS] Update docs about altering cached tables/views
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #31524:
URL: https://github.com/apache/spark/pull/31524#discussion_r577656186
##########
File path: docs/sql-ref-syntax-ddl-alter-table.md
##########
@@ -27,6 +27,10 @@ license: |
`ALTER TABLE RENAME TO` statement changes the table name of an existing table in the database.
+If the table is cached, the commands clear cached data of the table. The cache will be lazily filled when the next time the table is accessed. Additionally:
Review comment:
`the commands clear` -> `the command clears`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31524: [SPARK-34401][SQL][DOCS] Update docs about altering cached tables/views
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31524:
URL: https://github.com/apache/spark/pull/31524#issuecomment-781257027
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39806/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31524: [SPARK-34401][SQL][DOCS] Update docs about altering cached tables/views
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31524:
URL: https://github.com/apache/spark/pull/31524#issuecomment-781261952
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39806/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on a change in pull request #31524: [SPARK-34401][SQL][DOCS] Update docs about altering cached tables/views
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on a change in pull request #31524:
URL: https://github.com/apache/spark/pull/31524#discussion_r572037646
##########
File path: docs/sql-ref-syntax-ddl-alter-table.md
##########
@@ -27,6 +27,9 @@ license: |
`ALTER TABLE RENAME TO` statement changes the table name of an existing table in the database.
+If the table is cached, the commands clear cached data of the table and all its dependents that refer to it. The cache will be lazily filled when the next time the table or the dependents are accessed.
+The partition rename command keeps table dependents as cached but the table rename does not. In this way dependents should be re-cached explicitly after table renaming.
Review comment:
because the table rename command does not do that. It just cache table itself.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on pull request #31524: [SPARK-34401][SQL][DOCS] Update docs about altering cached tables/views
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on pull request #31524:
URL: https://github.com/apache/spark/pull/31524#issuecomment-780088272
Are you ok with the update for docs?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on pull request #31524: [SPARK-34401][SQL][DOCS] Update docs about altering cached tables/views
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on pull request #31524:
URL: https://github.com/apache/spark/pull/31524#issuecomment-775103312
@sunchao @cloud-fan @HyukjinKwon @dongjoon-hyun Could you review this PR, please.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #31524: [SPARK-34401][SQL][DOCS] Update docs about altering cached tables/views
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31524:
URL: https://github.com/apache/spark/pull/31524#issuecomment-781242626
**[Test build #135223 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135223/testReport)** for PR 31524 at commit [`15ccb9e`](https://github.com/apache/spark/commit/15ccb9e19e728bc3cc0fe4e3e609676fc90b6975).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #31524: [SPARK-34401][SQL][DOCS] Update docs about altering cached tables/views
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #31524:
URL: https://github.com/apache/spark/pull/31524#discussion_r579021451
##########
File path: docs/sql-ref-syntax-ddl-drop-table.md
##########
@@ -26,6 +26,8 @@ if the table is not `EXTERNAL` table. If the table is not present it throws an e
In case of an external table, only the associated metadata information is removed from the metastore database.
+If the table is cached, the command uncaches the table and all its dependents.
Review comment:
can you try with dataframe temp view?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31524: [SPARK-34401][SQL][DOCS] Update docs about altering cached tables/views
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31524:
URL: https://github.com/apache/spark/pull/31524#issuecomment-781403516
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135223/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31524: [SPARK-34401][SQL][DOCS] Update docs about altering cached tables/views
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31524:
URL: https://github.com/apache/spark/pull/31524#issuecomment-776959240
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39683/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #31524: [SPARK-34401][SQL][DOCS] Update docs about altering cached tables/views
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #31524:
URL: https://github.com/apache/spark/pull/31524#discussion_r572035577
##########
File path: docs/sql-ref-syntax-ddl-alter-table.md
##########
@@ -27,6 +27,9 @@ license: |
`ALTER TABLE RENAME TO` statement changes the table name of an existing table in the database.
+If the table is cached, the commands clear cached data of the table and all its dependents that refer to it. The cache will be lazily filled when the next time the table or the dependents are accessed.
+The partition rename command keeps table dependents as cached but the table rename does not. In this way dependents should be re-cached explicitly after table renaming.
Review comment:
I don't get it. Won't the cache be filled lazily? Why we need to re-cache explicitly?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31524: [SPARK-34401][SQL][DOCS] Update docs about altering cached tables/views
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31524:
URL: https://github.com/apache/spark/pull/31524#issuecomment-777021574
**[Test build #135101 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135101/testReport)** for PR 31524 at commit [`a3e87d8`](https://github.com/apache/spark/commit/a3e87d885bd90cbbf021ed38635fcd740db2eaa2).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31524: [SPARK-34401][SQL][DOCS] Update docs about altering cached tables/views
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31524:
URL: https://github.com/apache/spark/pull/31524#issuecomment-776959240
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39683/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #31524: [SPARK-34401][SQL][DOCS] Update docs about altering cached tables/views
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #31524:
URL: https://github.com/apache/spark/pull/31524#discussion_r572053653
##########
File path: docs/sql-ref-syntax-ddl-alter-table.md
##########
@@ -27,6 +27,9 @@ license: |
`ALTER TABLE RENAME TO` statement changes the table name of an existing table in the database.
+If the table is cached, the commands clear cached data of the table and all its dependents that refer to it. The cache will be lazily filled when the next time the table or the dependents are accessed.
+The partition rename command keeps table dependents as cached but the table rename does not. In this way dependents should be re-cached explicitly after table renaming.
Review comment:
hmm, shall we fix it?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #31524: [SPARK-34401][SQL][DOCS] Update docs about altering cached tables/views
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #31524:
URL: https://github.com/apache/spark/pull/31524#discussion_r572089747
##########
File path: docs/sql-ref-syntax-ddl-alter-table.md
##########
@@ -27,6 +27,9 @@ license: |
`ALTER TABLE RENAME TO` statement changes the table name of an existing table in the database.
+If the table is cached, the commands clear cached data of the table and all its dependents that refer to it. The cache will be lazily filled when the next time the table or the dependents are accessed.
+The partition rename command keeps table dependents as cached but the table rename does not. In this way dependents should be re-cached explicitly after table renaming.
Review comment:
ah got it now.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #31524: [SPARK-34401][SQL][DOCS] Update docs about altering cached tables/views
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #31524:
URL: https://github.com/apache/spark/pull/31524#discussion_r572207495
##########
File path: docs/sql-ref-syntax-ddl-alter-table.md
##########
@@ -27,6 +27,9 @@ license: |
`ALTER TABLE RENAME TO` statement changes the table name of an existing table in the database.
+If the table is cached, the commands clear cached data of the table and all its dependents that refer to it. The cache will be lazily filled when the next time the table or the dependents are accessed.
Review comment:
yea that's even better!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on a change in pull request #31524: [SPARK-34401][SQL][DOCS] Update docs about altering cached tables/views
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on a change in pull request #31524:
URL: https://github.com/apache/spark/pull/31524#discussion_r572203812
##########
File path: docs/sql-ref-syntax-ddl-alter-table.md
##########
@@ -27,6 +27,9 @@ license: |
`ALTER TABLE RENAME TO` statement changes the table name of an existing table in the database.
+If the table is cached, the commands clear cached data of the table and all its dependents that refer to it. The cache will be lazily filled when the next time the table or the dependents are accessed.
Review comment:
The problem is the doc section mixes 2 different commands. I would split the description completely like:
If the table is cached:
- the table rename command ...
- the partition rename command ...
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31524: [SPARK-34401][SQL][DOCS] Update docs about altering cached tables/views
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31524:
URL: https://github.com/apache/spark/pull/31524#issuecomment-775152968
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39609/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on a change in pull request #31524: [SPARK-34401][SQL][DOCS] Update docs about altering cached tables/views
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on a change in pull request #31524:
URL: https://github.com/apache/spark/pull/31524#discussion_r578561145
##########
File path: docs/sql-ref-syntax-ddl-drop-table.md
##########
@@ -26,6 +26,8 @@ if the table is not `EXTERNAL` table. If the table is not present it throws an e
In case of an external table, only the associated metadata information is removed from the metastore database.
+If the table is cached, the command uncaches the table and all its dependents.
Review comment:
```sql
spark-sql> CREATE TABLE tbl (c0 INT);
spark-sql> INSERT INTO tbl SELECT 0;
spark-sql> CACHE TABLE tbl;
spark-sql> CREATE TEMP VIEW v1 AS SELECT c0 + 1 FROM tbl;
spark-sql> SELECT * FROM v1;
1
spark-sql> CACHE TABLE v1;
spark-sql> SELECT * FROM v1;
1
spark-sql> DROP TABLE tbl;
spark-sql> SELECT * FROM v1;
Error in query: Table or view not found: tbl; line 1 pos 19;
'Project [*]
+- 'SubqueryAlias v1
+- View (`v1`, ['(c0 + 1)])
+- 'Project [upcast('(c0 + 1), IntegerType) AS (c0 + 1)#156]
+- 'Project [unresolvedalias(('c0 + 1), None)]
+- 'UnresolvedRelation [tbl], [], false
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on a change in pull request #31524: [SPARK-34401][SQL][DOCS] Update docs about altering cached tables/views
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on a change in pull request #31524:
URL: https://github.com/apache/spark/pull/31524#discussion_r579646193
##########
File path: docs/sql-ref-syntax-ddl-drop-table.md
##########
@@ -26,6 +26,8 @@ if the table is not `EXTERNAL` table. If the table is not present it throws an e
In case of an external table, only the associated metadata information is removed from the metastore database.
+If the table is cached, the command uncaches the table and all its dependents.
Review comment:
```scala
scala> sql("CREATE TABLE tbl (c0 INT)")
scala> sql("INSERT INTO tbl SELECT 0")
scala> val tbl = spark.table("tbl")
scala> tbl.cache()
scala> tbl.show(false)
+---+
|c0 |
+---+
|0 |
+---+
scala> tbl.select(($"c0" + 1).as("c1")).createOrReplaceTempView("tmp_view0")
scala> val v = spark.table("tmp_view0")
v: org.apache.spark.sql.DataFrame = [c1: int]
scala> v.cache()
res6: v.type = [c1: int]
scala> v.show(false)
+---+
|c1 |
+---+
|1 |
+---+
scala> sql("DROP TABLE tbl")
res8: org.apache.spark.sql.DataFrame = []
scala> v.show(false)
org.apache.hadoop.mapred.InvalidInputException: Input path does not exist: file:/Users/maximgekk/proj/doc-cmd-caching/spark-warehouse/tbl
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on a change in pull request #31524: [SPARK-34401][SQL][DOCS] Update docs about altering cached tables/views
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on a change in pull request #31524:
URL: https://github.com/apache/spark/pull/31524#discussion_r573838580
##########
File path: docs/sql-ref-syntax-ddl-drop-table.md
##########
@@ -26,6 +26,8 @@ if the table is not `EXTERNAL` table. If the table is not present it throws an e
In case of an external table, only the associated metadata information is removed from the metastore database.
+If the table is cached, the command uncaches the table and all its dependants such as views. Thereby the dependants should be cached again explicitly.
Review comment:
Thanks. I ran `grep` over all sources codes, and had found one more place:
https://github.com/apache/spark/blob/0986f16c8d2f0d648d5239958dd4bccac43890c2/sql/core/src/main/scala/org/apache/spark/sql/internal/CatalogImpl.scala#L555
I will fix it in this PR as well.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31524: [SPARK-34401][SQL][DOCS] Update docs about altering cached tables/views
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31524:
URL: https://github.com/apache/spark/pull/31524#issuecomment-775123809
**[Test build #135026 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135026/testReport)** for PR 31524 at commit [`e514ada`](https://github.com/apache/spark/commit/e514adab4b9c76cc9deff1e3b640b65ea26a366d).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31524: [SPARK-34401][SQL][DOCS] Update docs about altering cached tables/views
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31524:
URL: https://github.com/apache/spark/pull/31524#issuecomment-775143615
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135026/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31524: [SPARK-34401][SQL][DOCS] Update docs about altering cached tables/views
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31524:
URL: https://github.com/apache/spark/pull/31524#issuecomment-776881470
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39683/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31524: [SPARK-34401][SQL][DOCS] Update docs about altering cached tables/views
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31524:
URL: https://github.com/apache/spark/pull/31524#issuecomment-777048543
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135101/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #31524: [SPARK-34401][SQL][DOCS] Update docs about altering cached tables/views
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31524:
URL: https://github.com/apache/spark/pull/31524#issuecomment-776838050
**[Test build #135101 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135101/testReport)** for PR 31524 at commit [`a3e87d8`](https://github.com/apache/spark/commit/a3e87d885bd90cbbf021ed38635fcd740db2eaa2).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #31524: [SPARK-34401][SQL][DOCS] Update docs about altering cached tables/views
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #31524:
URL: https://github.com/apache/spark/pull/31524#discussion_r577662094
##########
File path: docs/sql-ref-syntax-ddl-drop-table.md
##########
@@ -26,6 +26,8 @@ if the table is not `EXTERNAL` table. If the table is not present it throws an e
In case of an external table, only the associated metadata information is removed from the metastore database.
+If the table is cached, the command uncaches the table and all its dependents such as views. Thereby the dependents should be cached again explicitly.
Review comment:
let's remove `such as views`. Permanent view is not dependents even if it refers to the table.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31524: [SPARK-34401][SQL][DOCS] Update docs about altering cached tables/views
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31524:
URL: https://github.com/apache/spark/pull/31524#issuecomment-775180753
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39609/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #31524: [SPARK-34401][SQL][DOCS] Update docs about altering cached tables/views
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #31524:
URL: https://github.com/apache/spark/pull/31524#issuecomment-783071133
thanks, merging to master!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31524: [SPARK-34401][SQL][DOCS] Update docs about altering cached tables/views
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31524:
URL: https://github.com/apache/spark/pull/31524#issuecomment-781403516
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135223/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31524: [SPARK-34401][SQL][DOCS] Update docs about altering cached tables/views
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31524:
URL: https://github.com/apache/spark/pull/31524#issuecomment-776838050
**[Test build #135101 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135101/testReport)** for PR 31524 at commit [`a3e87d8`](https://github.com/apache/spark/commit/a3e87d885bd90cbbf021ed38635fcd740db2eaa2).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #31524: [SPARK-34401][SQL][DOCS] Update docs about altering cached tables/views
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31524:
URL: https://github.com/apache/spark/pull/31524#issuecomment-775116106
**[Test build #135026 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135026/testReport)** for PR 31524 at commit [`e514ada`](https://github.com/apache/spark/commit/e514adab4b9c76cc9deff1e3b640b65ea26a366d).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on a change in pull request #31524: [SPARK-34401][SQL][DOCS] Update docs about altering cached tables/views
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on a change in pull request #31524:
URL: https://github.com/apache/spark/pull/31524#discussion_r578283636
##########
File path: docs/sql-ref-syntax-ddl-drop-table.md
##########
@@ -26,6 +26,8 @@ if the table is not `EXTERNAL` table. If the table is not present it throws an e
In case of an external table, only the associated metadata information is removed from the metastore database.
+If the table is cached, the command uncaches the table and all its dependents such as views. Thereby the dependents should be cached again explicitly.
Review comment:
The sentence "Thereby the dependents should be cached again explicitly." is not needed too, I believe.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31524: [SPARK-34401][SQL][DOCS] Update docs about altering cached tables/views
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31524:
URL: https://github.com/apache/spark/pull/31524#issuecomment-781399051
**[Test build #135223 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135223/testReport)** for PR 31524 at commit [`15ccb9e`](https://github.com/apache/spark/commit/15ccb9e19e728bc3cc0fe4e3e609676fc90b6975).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan closed pull request #31524: [SPARK-34401][SQL][DOCS] Update docs about altering cached tables/views
Posted by GitBox <gi...@apache.org>.
cloud-fan closed pull request #31524:
URL: https://github.com/apache/spark/pull/31524
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #31524: [SPARK-34401][SQL][DOCS] Update docs about altering cached tables/views
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #31524:
URL: https://github.com/apache/spark/pull/31524#discussion_r572091159
##########
File path: docs/sql-ref-syntax-ddl-alter-table.md
##########
@@ -27,6 +27,9 @@ license: |
`ALTER TABLE RENAME TO` statement changes the table name of an existing table in the database.
+If the table is cached, the commands clear cached data of the table and all its dependents that refer to it. The cache will be lazily filled when the next time the table or the dependents are accessed.
Review comment:
`The cache` -> `The cache of the table being renamed (not its dependents)`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on a change in pull request #31524: [SPARK-34401][SQL][DOCS] Update docs about altering cached tables/views
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on a change in pull request #31524:
URL: https://github.com/apache/spark/pull/31524#discussion_r572038254
##########
File path: docs/sql-ref-syntax-ddl-alter-table.md
##########
@@ -187,6 +194,8 @@ ALTER TABLE table_identifier [ partition_spec ] SET SERDE serde_class_name
`ALTER TABLE SET` command can also be used for changing the file location and file format for
existing tables.
+If the table is cached, the `ALTER TABLE .. SET LOCATION` command clears cached data of the table and all its dependents that refer to it. The cache will be lazily filled when the next time the table or the dependents are accessed.
Review comment:
- It is not applicable to all commands
- I modified not only `ALTER TABLE`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #31524: [SPARK-34401][SQL][DOCS] Update docs about altering cached tables/views
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #31524:
URL: https://github.com/apache/spark/pull/31524#discussion_r578378202
##########
File path: docs/sql-ref-syntax-ddl-drop-table.md
##########
@@ -26,6 +26,8 @@ if the table is not `EXTERNAL` table. If the table is not present it throws an e
In case of an external table, only the associated metadata information is removed from the metastore database.
+If the table is cached, the command uncaches the table and all its dependents.
Review comment:
Can you double-check the behavior of DROP TABLE? Are the dependents (dataframe temp views) still valid?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on a change in pull request #31524: [SPARK-34401][SQL][DOCS] Update docs about altering cached tables/views
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on a change in pull request #31524:
URL: https://github.com/apache/spark/pull/31524#discussion_r572041366
##########
File path: docs/sql-ref-syntax-ddl-alter-table.md
##########
@@ -27,6 +27,9 @@ license: |
`ALTER TABLE RENAME TO` statement changes the table name of an existing table in the database.
+If the table is cached, the commands clear cached data of the table and all its dependents that refer to it. The cache will be lazily filled when the next time the table or the dependents are accessed.
+The partition rename command keeps table dependents as cached but the table rename does not. In this way dependents should be re-cached explicitly after table renaming.
Review comment:
Look at the implementation https://github.com/apache/spark/blob/2c243c93d9b107ed19fa21002d54deccd17ce50a/sql/core/src/main/scala/org/apache/spark/sql/execution/command/tables.scala#L201-L210
It uncaches tables (and its dependents), and caches ONLY the table back.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on a change in pull request #31524: [SPARK-34401][SQL][DOCS] Update docs about altering cached tables/views
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on a change in pull request #31524:
URL: https://github.com/apache/spark/pull/31524#discussion_r573842447
##########
File path: docs/sql-ref-syntax-ddl-alter-view.md
##########
@@ -28,6 +28,8 @@ the name of a view to a different name, set and unset the metadata of the view b
Renames the existing view. If the new view name already exists in the source database, a `TableAlreadyExistsException` is thrown. This operation
does not support moving the views across databases.
+If the view is cached, the command clears cached data of the view and all its dependents that refer to it. View's cache will be lazily filled when the next time the view or the dependents are accessed. The command leaves view's dependents as uncached.
Review comment:
You are right.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31524: [SPARK-34401][SQL][DOCS] Update docs about altering cached tables/views
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31524:
URL: https://github.com/apache/spark/pull/31524#issuecomment-781261978
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39806/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on a change in pull request #31524: [SPARK-34401][SQL][DOCS] Update docs about altering cached tables/views
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on a change in pull request #31524:
URL: https://github.com/apache/spark/pull/31524#discussion_r572069471
##########
File path: docs/sql-ref-syntax-ddl-alter-table.md
##########
@@ -27,6 +27,9 @@ license: |
`ALTER TABLE RENAME TO` statement changes the table name of an existing table in the database.
+If the table is cached, the commands clear cached data of the table and all its dependents that refer to it. The cache will be lazily filled when the next time the table or the dependents are accessed.
+The partition rename command keeps table dependents as cached but the table rename does not. In this way dependents should be re-cached explicitly after table renaming.
Review comment:
We cannot just replace the code above by `CatalogImpl.refreshTable()` or by `recacheByPlan` because the plan after renaming is different. I guess it includes `CatalogTable` which has different `identifier` after renaming.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31524: [SPARK-34401][SQL][DOCS] Update docs about altering cached tables/views
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31524:
URL: https://github.com/apache/spark/pull/31524#issuecomment-777048543
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135101/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #31524: [SPARK-34401][SQL][DOCS] Update docs about altering cached tables/views
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #31524:
URL: https://github.com/apache/spark/pull/31524#discussion_r572199480
##########
File path: docs/sql-ref-syntax-ddl-alter-table.md
##########
@@ -27,6 +27,9 @@ license: |
`ALTER TABLE RENAME TO` statement changes the table name of an existing table in the database.
+If the table is cached, the commands clear cached data of the table and all its dependents that refer to it. The cache will be lazily filled when the next time the table or the dependents are accessed.
Review comment:
It's better to make it clear at the beginning, instead of correcting it later.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31524: [SPARK-34401][SQL][DOCS] Update docs about altering cached tables/views
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31524:
URL: https://github.com/apache/spark/pull/31524#issuecomment-775116106
**[Test build #135026 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135026/testReport)** for PR 31524 at commit [`e514ada`](https://github.com/apache/spark/commit/e514adab4b9c76cc9deff1e3b640b65ea26a366d).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31524: [SPARK-34401][SQL][DOCS] Update docs about altering cached tables/views
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31524:
URL: https://github.com/apache/spark/pull/31524#issuecomment-775143615
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135026/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on a change in pull request #31524: [SPARK-34401][SQL][DOCS] Update docs about altering cached tables/views
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on a change in pull request #31524:
URL: https://github.com/apache/spark/pull/31524#discussion_r572197463
##########
File path: docs/sql-ref-syntax-ddl-alter-table.md
##########
@@ -27,6 +27,9 @@ license: |
`ALTER TABLE RENAME TO` statement changes the table name of an existing table in the database.
+If the table is cached, the commands clear cached data of the table and all its dependents that refer to it. The cache will be lazily filled when the next time the table or the dependents are accessed.
Review comment:
The next sentence says that: "The partition rename command keeps table dependents as cached but the table rename does not."
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31524: [SPARK-34401][SQL][DOCS] Update docs about altering cached tables/views
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31524:
URL: https://github.com/apache/spark/pull/31524#issuecomment-776940734
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39683/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31524: [SPARK-34401][SQL][DOCS] Update docs about altering cached tables/views
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31524:
URL: https://github.com/apache/spark/pull/31524#issuecomment-781261978
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39806/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31524: [SPARK-34401][SQL][DOCS] Update docs about altering cached tables/views
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31524:
URL: https://github.com/apache/spark/pull/31524#issuecomment-775180753
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39609/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on a change in pull request #31524: [SPARK-34401][SQL][DOCS] Update docs about altering cached tables/views
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on a change in pull request #31524:
URL: https://github.com/apache/spark/pull/31524#discussion_r579646797
##########
File path: docs/sql-ref-syntax-ddl-drop-table.md
##########
@@ -26,6 +26,8 @@ if the table is not `EXTERNAL` table. If the table is not present it throws an e
In case of an external table, only the associated metadata information is removed from the metastore database.
+If the table is cached, the command uncaches the table and all its dependents.
Review comment:
```scala
scala> Seq(0).toDF("c0").write.mode("overwrite").parquet("/Users/maximgekk/tmp/tbl")
scala> sql("CREATE TABLE tbl2 (c0 INT) USING parquet LOCATION '/Users/maximgekk/tmp/tbl'")
res11: org.apache.spark.sql.DataFrame = []
scala> val tbl2 = spark.table("tbl2")
tbl2: org.apache.spark.sql.DataFrame = [c0: int]
scala> tbl2.cache()
res12: tbl2.type = [c0: int]
scala> tbl2.show(false)
+---+
|c0 |
+---+
|0 |
+---+
scala> tbl2.select(($"c0" + 2).as("c1")).createOrReplaceTempView("tmp_view2")
scala> val v2 = spark.table("tmp_view2")
v2: org.apache.spark.sql.DataFrame = [c1: int]
scala> v2.cache()
res15: v2.type = [c1: int]
scala> v2.show(false)
+---+
|c1 |
+---+
|2 |
+---+
scala> sql("DROP TABLE tbl2")
res17: org.apache.spark.sql.DataFrame = []
scala> v2.show(false)
+---+
|c1 |
+---+
|2 |
+---+
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31524: [SPARK-34401][SQL][DOCS] Update docs about altering cached tables/views
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31524:
URL: https://github.com/apache/spark/pull/31524#issuecomment-781242626
**[Test build #135223 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135223/testReport)** for PR 31524 at commit [`15ccb9e`](https://github.com/apache/spark/commit/15ccb9e19e728bc3cc0fe4e3e609676fc90b6975).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31524: [SPARK-34401][SQL][DOCS] Update docs about altering cached tables/views
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31524:
URL: https://github.com/apache/spark/pull/31524#issuecomment-775136139
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39609/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #31524: [SPARK-34401][SQL][DOCS] Update docs about altering cached tables/views
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #31524:
URL: https://github.com/apache/spark/pull/31524#discussion_r572036282
##########
File path: docs/sql-ref-syntax-ddl-alter-table.md
##########
@@ -187,6 +194,8 @@ ALTER TABLE table_identifier [ partition_spec ] SET SERDE serde_class_name
`ALTER TABLE SET` command can also be used for changing the file location and file format for
existing tables.
+If the table is cached, the `ALTER TABLE .. SET LOCATION` command clears cached data of the table and all its dependents that refer to it. The cache will be lazily filled when the next time the table or the dependents are accessed.
Review comment:
We should probably put this in the beginning only once:
```
### Description
`ALTER TABLE` statement changes the schema or properties of a table. If the table is cached ...
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org