You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2019/12/28 16:08:31 UTC
[GitHub] [spark] ayudovin opened a new pull request #27030: [SPARK-30244] -
Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
ayudovin opened a new pull request #27030: [SPARK-30244] - Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030
<!--
Thanks for sending a pull request! Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://spark.apache.org/contributing.html
2. Ensure you have added or run the appropriate tests for your PR: https://spark.apache.org/developer-tools.html
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][SPARK-XXXX] Your PR title ...'.
4. Be sure to keep the PR description updated to reflect all changes.
5. Please write your PR title to summarize what this PR proposes.
6. If possible, provide a concise example to reproduce the issue for a faster review.
-->
### What changes were proposed in this pull request?
<!--
Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue.
If possible, please consider writing useful notes for better and faster reviews in your PR. See the examples below.
1. If you refactor some codes with changing classes, showing the class hierarchy will help reviewers.
2. If you fix some SQL features, you can provide some references of other DBMSes.
3. If there is design documentation, please add the link.
4. If there is a discussion in the mailing list, please add the link.
-->
Add events to CREATE, DROP, RENAME and ALTER partitions in `ExternalCatalogWithListener`.
### Why are the changes needed?
<!--
Please clarify why the changes are needed. For instance,
1. If you propose a new API, clarify the use case for a new API.
2. If you fix a bug, you can clarify why it is a bug.
-->
This changes needs for adding hooks for partitions in `ExternalCatalogWithListener`.
### Does this PR introduce any user-facing change?
<!--
If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible.
If no, write 'No'.
-->
No
### How was this patch tested?
<!--
If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
If tests were not added, please describe why they were not added and/or why it was difficult to add.
-->
This functionality is covered by unit tests.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27030: [SPARK-30244][SQL]
Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#issuecomment-572775097
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/21207/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27030:
[SPARK-30244][SQL] Emit pre/post events for "Partition" methods in
ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#issuecomment-574529036
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27030: [SPARK-30244][SQL]
Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#issuecomment-572841008
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27030:
[SPARK-30244][SQL] Emit pre/post events for "Partition" methods in
ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#issuecomment-573457218
**[Test build #116570 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/116570/testReport)** for PR 27030 at commit [`e1220e1`](https://github.com/apache/spark/commit/e1220e105bc4304365ca3f232f1825b00bd6e623).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #27030:
[SPARK-30244][SQL] Emit pre/post events for "Partition" methods in
ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#discussion_r365496650
##########
File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/catalog/ExternalCatalogEventSuite.scala
##########
@@ -209,4 +209,65 @@ class ExternalCatalogEventSuite extends SparkFunSuite {
catalog.dropFunction("db5", "fn4")
checkEvents(DropFunctionPreEvent("db5", "fn4") :: DropFunctionEvent("db5", "fn4") :: Nil)
}
+
+ testWithCatalog("partition") { (catalog, checkEvents) =>
+ // CREATE
+ val path1 = Files.createTempDirectory("db_")
+ val uri1 = preparePath(path1)
+
+ val dbDefinition = createDbDefinition()
+
+ val storage = CatalogStorageFormat.empty.copy(
+ locationUri = Option(uri1))
Review comment:
Is this table storage pointing db path? Maybe, do you want the following?
```scala
val path1 = Files.createTempDirectory("db_")
val path2 = Files.createTempDirectory(path1, "tbl_")
val uri1 = preparePath(path1)
val uri2 = preparePath(path2)
// CREATE
val dbDefinition = createDbDefinition(uri1)
val storage = CatalogStorageFormat.empty.copy(
locationUri = Option(uri2))
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27030: [SPARK-30244][SQL]
Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#issuecomment-578413185
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22160/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #27030:
[SPARK-30244][SQL] Emit pre/post events for "Partition" methods in
ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#discussion_r364894104
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/events.scala
##########
@@ -202,3 +203,62 @@ case class RenameFunctionEvent(
name: String,
newName: String)
extends FunctionEvent
+
+trait PartitionEvent extends DatabaseEvent {
+ /**
+ * Name of the table that was touched.
+ */
+ val name: String
+}
+
+
+/**
+ * Event fired before a partition is created.
+ */
+case class CreatePartitionPreEvent(database: String, name: String,
+ parts: Seq[CatalogTablePartition]) extends PartitionEvent
Review comment:
Indentation? You had better follow the existing style used in this file.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27030:
[SPARK-30244][SQL] Emit pre/post events for "Partition" methods in
ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#issuecomment-573137709
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #27030: [SPARK-30244][SQL]
Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#issuecomment-577898473
Sorry for being late, @ayudovin . I'll try to review again tonight.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27030:
[SPARK-30244][SQL] Emit pre/post events for "Partition" methods in
ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#issuecomment-573043280
**[Test build #116492 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/116492/testReport)** for PR 27030 at commit [`4d5ba9b`](https://github.com/apache/spark/commit/4d5ba9b712404e3ff2db59a4193abd5a30743506).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27030: [SPARK-30244][SQL] Emit
pre/post events for "Partition" methods in ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#issuecomment-574414700
**[Test build #116730 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/116730/testReport)** for PR 27030 at commit [`43aeb9b`](https://github.com/apache/spark/commit/43aeb9ba12c5dabcaddec3e84a2428ad2ebb09b6).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27030:
[SPARK-30244][SQL] Emit pre/post events for "Partition" methods in
ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#issuecomment-572916416
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/21251/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27030:
[SPARK-30244][SQL] Emit pre/post events for "Partition" methods in
ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#issuecomment-572850861
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #27030:
[SPARK-30244][SQL] Emit pre/post events for "Partition" methods in
ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#discussion_r364895453
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/events.scala
##########
@@ -202,3 +203,62 @@ case class RenameFunctionEvent(
name: String,
newName: String)
extends FunctionEvent
+
+trait PartitionEvent extends DatabaseEvent {
+ /**
+ * Name of the table that was touched.
+ */
+ val name: String
Review comment:
`parts: Seq[CatalogTablePartition]` instead of `val name: String`? Table name is already inherited in `TableEvent`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27030:
[SPARK-30244][SQL] Emit pre/post events for "Partition" methods in
ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#issuecomment-572791530
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/116410/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27030: [SPARK-30244][SQL] Emit
pre/post events for "Partition" methods in ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#issuecomment-572790938
**[Test build #116410 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/116410/testReport)** for PR 27030 at commit [`5c242dc`](https://github.com/apache/spark/commit/5c242dc3700789f4af49f9e3d4b544d3b1d2f9e7).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `trait PartitionEvent extends DatabaseEvent `
* `case class CreatePartitionPreEvent(database: String, name: String,`
* `case class CreatePartitionEvent(database: String, name: String,`
* `case class DropPartitionPreEvent(database: String, name: String,`
* `case class DropPartitionEvent(database: String, name: String,`
* `case class RenamePartitionPreEvent(database: String, name: String,`
* `case class RenamePartitionEvent(database: String, name: String,`
* `case class AlterPartitionPreEvent(database: String, name: String,`
* `case class AlterPartitionEvent(database: String, name: String,`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ayudovin commented on a change in pull request #27030:
[SPARK-30244][SQL] Emit pre/post events for "Partition" methods in
ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
ayudovin commented on a change in pull request #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#discussion_r366020298
##########
File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/catalog/ExternalCatalogEventSuite.scala
##########
@@ -209,4 +209,67 @@ class ExternalCatalogEventSuite extends SparkFunSuite {
catalog.dropFunction("db5", "fn4")
checkEvents(DropFunctionPreEvent("db5", "fn4") :: DropFunctionEvent("db5", "fn4") :: Nil)
}
+
+ testWithCatalog("partition") { (catalog, checkEvents) =>
+ val path1 = Files.createTempDirectory("db_")
+ val path2 = Files.createTempDirectory(path1, "tbl_")
+ val uri1 = preparePath(path1)
+ val uri2 = preparePath(path2)
+
+ // CREATE
+ val dbDefinition = createDbDefinition(uri1)
+
+ val storage = CatalogStorageFormat.empty.copy(
+ locationUri = Option(uri2))
+
+ val tableDefinition = CatalogTable(
+ identifier = TableIdentifier("tbl1", Some("db5")),
+ tableType = CatalogTableType.MANAGED,
+ storage = storage,
+ schema = new StructType().add("id", "long"))
+
+ val partition = CatalogTablePartition(spec = CatalogTypes.emptyTablePartitionSpec,
+ storage = CatalogStorageFormat.empty)
+
+ catalog.createDatabase(dbDefinition, ignoreIfExists = false)
+ checkEvents(CreateDatabasePreEvent("db5") :: CreateDatabaseEvent("db5") :: Nil)
+
+ catalog.createTable(tableDefinition, ignoreIfExists = false)
+ checkEvents(CreateTablePreEvent("db5", "tbl1") :: CreateTableEvent("db5", "tbl1") :: Nil)
+
+ catalog.createPartitions("db5", "tbl1", Seq(partition), ignoreIfExists = false)
+ checkEvents(CreatePartitionPreEvent("db5", "tbl1", Seq(partition)) ::
+ CreatePartitionEvent("db5", "tbl1", Seq(partition)) :: Nil)
+
+ catalog.createPartitions("db5", "tbl1", Seq(partition), ignoreIfExists = true)
+ checkEvents(CreatePartitionPreEvent("db5", "tbl1", Seq(partition)) ::
+ CreatePartitionEvent("db5", "tbl1", Seq(partition)) :: Nil)
+
+ // RENAME
+ val newPartition = CatalogTablePartition(spec = Map("key1" -> "1", "key2" -> "2"),
+ storage = CatalogStorageFormat.empty)
+
+ catalog.renamePartitions("db5", "tbl1", Seq(partition.spec), Seq(newPartition.spec))
Review comment:
now it looks like `key=0` to `key1=1, key2=2`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27030:
[SPARK-30244][SQL] Emit pre/post events for "Partition" methods in
ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#issuecomment-573865410
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/21440/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ayudovin opened a new pull request #27030:
[SPARK-30244][SQL] Emit pre/post events for "Partition" methods in
ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
ayudovin opened a new pull request #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030
<!--
Thanks for sending a pull request! Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://spark.apache.org/contributing.html
2. Ensure you have added or run the appropriate tests for your PR: https://spark.apache.org/developer-tools.html
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][SPARK-XXXX] Your PR title ...'.
4. Be sure to keep the PR description updated to reflect all changes.
5. Please write your PR title to summarize what this PR proposes.
6. If possible, provide a concise example to reproduce the issue for a faster review.
-->
### What changes were proposed in this pull request?
<!--
Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue.
If possible, please consider writing useful notes for better and faster reviews in your PR. See the examples below.
1. If you refactor some codes with changing classes, showing the class hierarchy will help reviewers.
2. If you fix some SQL features, you can provide some references of other DBMSes.
3. If there is design documentation, please add the link.
4. If there is a discussion in the mailing list, please add the link.
-->
Add events to CREATE, DROP, RENAME and ALTER partitions in `ExternalCatalogWithListener`.
### Why are the changes needed?
<!--
Please clarify why the changes are needed. For instance,
1. If you propose a new API, clarify the use case for a new API.
2. If you fix a bug, you can clarify why it is a bug.
-->
This changes needs for adding hooks for partitions in `ExternalCatalogWithListener`.
### Does this PR introduce any user-facing change?
<!--
If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible.
If no, write 'No'.
-->
No
### How was this patch tested?
<!--
If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
If tests were not added, please describe why they were not added and/or why it was difficult to add.
-->
This functionality is covered by unit tests.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27030:
[SPARK-30244][SQL] Emit pre/post events for "Partition" methods in
ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#issuecomment-572692528
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #27030:
[SPARK-30244][SQL] Emit pre/post events for "Partition" methods in
ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#discussion_r370965396
##########
File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/catalog/ExternalCatalogEventSuite.scala
##########
@@ -209,4 +209,67 @@ class ExternalCatalogEventSuite extends SparkFunSuite {
catalog.dropFunction("db5", "fn4")
checkEvents(DropFunctionPreEvent("db5", "fn4") :: DropFunctionEvent("db5", "fn4") :: Nil)
}
+
+ testWithCatalog("partition") { (catalog, checkEvents) =>
+ val path1 = Files.createTempDirectory("db_")
+ val path2 = Files.createTempDirectory(path1, "tbl_")
+ val uri1 = preparePath(path1)
+ val uri2 = preparePath(path2)
+
+ // CREATE
+ val dbDefinition = createDbDefinition(uri1)
+
+ val storage = CatalogStorageFormat.empty.copy(
+ locationUri = Option(uri2))
+
+ val tableDefinition = CatalogTable(
+ identifier = TableIdentifier("tbl1", Some("db5")),
+ tableType = CatalogTableType.MANAGED,
+ storage = storage,
+ schema = new StructType().add("id", "long"))
+
+ val partition = CatalogTablePartition(spec = Map("key1" -> "1", "key2" -> "2"),
+ storage = CatalogStorageFormat.empty)
+
+ catalog.createDatabase(dbDefinition, ignoreIfExists = false)
+ checkEvents(CreateDatabasePreEvent("db5") :: CreateDatabaseEvent("db5") :: Nil)
+
+ catalog.createTable(tableDefinition, ignoreIfExists = false)
+ checkEvents(CreateTablePreEvent("db5", "tbl1") :: CreateTableEvent("db5", "tbl1") :: Nil)
+
+ catalog.createPartitions("db5", "tbl1", Seq(partition), ignoreIfExists = false)
+ checkEvents(CreatePartitionPreEvent("db5", "tbl1", Seq(partition)) ::
+ CreatePartitionEvent("db5", "tbl1", Seq(partition)) :: Nil)
+
+ catalog.createPartitions("db5", "tbl1", Seq(partition), ignoreIfExists = true)
+ checkEvents(CreatePartitionPreEvent("db5", "tbl1", Seq(partition)) ::
+ CreatePartitionEvent("db5", "tbl1", Seq(partition)) :: Nil)
Review comment:
If then, this is not helpful in that case because we cannot distinguish the difference.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27030:
[SPARK-30244][SQL] Emit pre/post events for "Partition" methods in
ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#issuecomment-573479249
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/116570/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27030:
[SPARK-30244][SQL] Emit pre/post events for "Partition" methods in
ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#issuecomment-574542935
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #27030:
[SPARK-30244][SQL] Emit pre/post events for "Partition" methods in
ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#discussion_r366618290
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/events.scala
##########
@@ -202,3 +203,83 @@ case class RenameFunctionEvent(
name: String,
newName: String)
extends FunctionEvent
+
+/**
+ * Event fired when a partition is created, dropped, altered or renamed.
+ */
+trait PartitionEvent extends TableEvent
+
+/**
+ * Event fired before a partition is created.
+ */
+case class CreatePartitionPreEvent(
+ database: String,
+ name: String,
+ parts: Seq[CatalogTablePartition])
+ extends PartitionEvent
+
+/**
+ * Event fired after a partition has been created.
+ */
+case class CreatePartitionEvent(
+ database: String,
+ name: String,
+ parts: Seq[CatalogTablePartition])
+ extends PartitionEvent
+
+/**
+ * Event fired before a partition is dropped.
+ */
+case class DropPartitionPreEvent(
+ database: String,
+ name: String,
+ partSpecs: Seq[TablePartitionSpec])
+ extends PartitionEvent
+
+/**
+ * Event fired after a partition has been dropped.
+ */
+case class DropPartitionEvent(
+ database: String,
+ name: String,
+ partSpecs: Seq[TablePartitionSpec])
+ extends PartitionEvent
+
+/**
+ * Event fired before a function is renamed.
+ */
+case class RenamePartitionPreEvent(
+ database: String,
+ name: String,
+ specs: Seq[TablePartitionSpec],
+ newSpecs: Seq[TablePartitionSpec])
+ extends PartitionEvent
+
+/**
+ * Event fired after a function has been renamed.
+ */
+case class RenamePartitionEvent(
+ database: String,
+ name: String,
+ specs: Seq[TablePartitionSpec],
+ newSpecs: Seq[TablePartitionSpec])
+ extends PartitionEvent
+
+/**
+ * Event fired before a function is altered.
+ */
+case class AlterPartitionPreEvent(
+ database: String,
+ name: String,
+ parts: Seq[CatalogTablePartition])
+ extends PartitionEvent
+
+/**
+ * Event fired after a function has been altered.
Review comment:
`function` -> `partition`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #27030: [SPARK-30244][SQL]
Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#issuecomment-577898553
Retest this please.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27030:
[SPARK-30244][SQL] Emit pre/post events for "Partition" methods in
ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#issuecomment-578413185
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22160/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27030:
[SPARK-30244][SQL] Emit pre/post events for "Partition" methods in
ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#issuecomment-578432913
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/117401/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27030: [SPARK-30244][SQL]
Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#issuecomment-572916416
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/21251/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27030: [SPARK-30244][SQL]
Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#issuecomment-573040845
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/21278/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ayudovin commented on a change in pull request #27030:
[SPARK-30244][SQL] Emit pre/post events for "Partition" methods in
ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
ayudovin commented on a change in pull request #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#discussion_r365105197
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/events.scala
##########
@@ -204,61 +204,64 @@ case class RenameFunctionEvent(
newName: String)
extends FunctionEvent
-trait PartitionEvent extends DatabaseEvent {
- /**
- * Name of the table that was touched.
- */
- val name: String
-}
-
+/**
+ * Event fired when a partition is created, dropped, altered or renamed.
+ */
+trait PartitionEvent extends TableEvent
/**
* Event fired before a partition is created.
*/
-case class CreatePartitionPreEvent(database: String, name: String,
- parts: Seq[CatalogTablePartition]) extends PartitionEvent
+case class CreatePartitionPreEvent(
+ database: String,
+ name: String) extends PartitionEvent
Review comment:
> shall we put `extends PartitionEvent` into a new line
yes, I'll put it into a new line.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #27030:
[SPARK-30244][SQL] Emit pre/post events for "Partition" methods in
ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#discussion_r370469860
##########
File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/catalog/ExternalCatalogEventSuite.scala
##########
@@ -209,4 +209,67 @@ class ExternalCatalogEventSuite extends SparkFunSuite {
catalog.dropFunction("db5", "fn4")
checkEvents(DropFunctionPreEvent("db5", "fn4") :: DropFunctionEvent("db5", "fn4") :: Nil)
}
+
+ testWithCatalog("partition") { (catalog, checkEvents) =>
+ val path1 = Files.createTempDirectory("db_")
+ val path2 = Files.createTempDirectory(path1, "tbl_")
+ val uri1 = preparePath(path1)
+ val uri2 = preparePath(path2)
+
+ // CREATE
+ val dbDefinition = createDbDefinition(uri1)
+
+ val storage = CatalogStorageFormat.empty.copy(
+ locationUri = Option(uri2))
+
+ val tableDefinition = CatalogTable(
+ identifier = TableIdentifier("tbl1", Some("db5")),
+ tableType = CatalogTableType.MANAGED,
+ storage = storage,
+ schema = new StructType().add("id", "long"))
+
+ val partition = CatalogTablePartition(spec = Map("key1" -> "1", "key2" -> "2"),
+ storage = CatalogStorageFormat.empty)
+
+ catalog.createDatabase(dbDefinition, ignoreIfExists = false)
+ checkEvents(CreateDatabasePreEvent("db5") :: CreateDatabaseEvent("db5") :: Nil)
+
+ catalog.createTable(tableDefinition, ignoreIfExists = false)
+ checkEvents(CreateTablePreEvent("db5", "tbl1") :: CreateTableEvent("db5", "tbl1") :: Nil)
+
+ catalog.createPartitions("db5", "tbl1", Seq(partition), ignoreIfExists = false)
+ checkEvents(CreatePartitionPreEvent("db5", "tbl1", Seq(partition)) ::
+ CreatePartitionEvent("db5", "tbl1", Seq(partition)) :: Nil)
+
+ catalog.createPartitions("db5", "tbl1", Seq(partition), ignoreIfExists = true)
+ checkEvents(CreatePartitionPreEvent("db5", "tbl1", Seq(partition)) ::
+ CreatePartitionEvent("db5", "tbl1", Seq(partition)) :: Nil)
+
+ // RENAME
+ val newPartition = CatalogTablePartition(spec = Map("key1" -> "3", "key2" -> "4"),
+ storage = CatalogStorageFormat.empty)
+
+ catalog.renamePartitions("db5", "tbl1", Seq(partition.spec), Seq(newPartition.spec))
+ checkEvents(
+ RenamePartitionPreEvent("db5", "tbl1", Seq(partition.spec), Seq(newPartition.spec)) ::
+ RenamePartitionEvent("db5", "tbl1", Seq(partition.spec), Seq(newPartition.spec)) ::
+ Nil)
+
+ // ALTER
Review comment:
`ALTER nothing`? In this case, I guess no event might be natural.
BTW, if there is no valid SQL query for this, I guess we can remove this test case.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #27030: [SPARK-30244][SQL]
Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#issuecomment-575458737
Please don't forget the above comment.
> Could you give me the real SQL statement for that?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27030: [SPARK-30244][SQL] Emit
pre/post events for "Partition" methods in ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#issuecomment-574471266
**[Test build #116730 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/116730/testReport)** for PR 27030 at commit [`43aeb9b`](https://github.com/apache/spark/commit/43aeb9ba12c5dabcaddec3e84a2428ad2ebb09b6).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27030: [SPARK-30244][SQL]
Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#issuecomment-590669465
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27030: [SPARK-30244][SQL]
Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#issuecomment-573865400
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun edited a comment on issue #27030:
[SPARK-30244][SQL] Emit pre/post events for "Partition" methods in
ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun edited a comment on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#issuecomment-578456861
@ayudovin . The current status is that we are waiting the original author of this framework, @hvanhovell . We need to check whether this is not added intentionally or not. Also, https://github.com/apache/spark/pull/27030#discussion_r370965396 is another concern no this approach because it can be misleading.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27030:
[SPARK-30244][SQL] Emit pre/post events for "Partition" methods in
ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#issuecomment-569429336
Can one of the admins verify this patch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27030:
[SPARK-30244][SQL] Emit pre/post events for "Partition" methods in
ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#issuecomment-572780781
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27030:
[SPARK-30244][SQL] Emit pre/post events for "Partition" methods in
ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#issuecomment-573937133
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27030: [SPARK-30244][SQL]
Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#issuecomment-573137709
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #27030:
[SPARK-30244][SQL] Emit pre/post events for "Partition" methods in
ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#discussion_r364895084
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/events.scala
##########
@@ -202,3 +203,62 @@ case class RenameFunctionEvent(
name: String,
newName: String)
extends FunctionEvent
+
+trait PartitionEvent extends DatabaseEvent {
Review comment:
`Partition` is belong to `Table`. `TableEvent` is better than `DatabaseEvent`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #27030:
[SPARK-30244][SQL] Emit pre/post events for "Partition" methods in
ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#discussion_r364895453
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/events.scala
##########
@@ -202,3 +203,62 @@ case class RenameFunctionEvent(
name: String,
newName: String)
extends FunctionEvent
+
+trait PartitionEvent extends DatabaseEvent {
+ /**
+ * Name of the table that was touched.
+ */
+ val name: String
Review comment:
`parts: Seq[CatalogTablePartition]` instead of `val name: String`? If we extends `TableEvent`, Table name is already inherited in `TableEvent`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #27030:
[SPARK-30244][SQL] Emit pre/post events for "Partition" methods in
ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#discussion_r365629981
##########
File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/catalog/ExternalCatalogEventSuite.scala
##########
@@ -209,4 +209,67 @@ class ExternalCatalogEventSuite extends SparkFunSuite {
catalog.dropFunction("db5", "fn4")
checkEvents(DropFunctionPreEvent("db5", "fn4") :: DropFunctionEvent("db5", "fn4") :: Nil)
}
+
+ testWithCatalog("partition") { (catalog, checkEvents) =>
+ val path1 = Files.createTempDirectory("db_")
+ val path2 = Files.createTempDirectory(path1, "tbl_")
+ val uri1 = preparePath(path1)
+ val uri2 = preparePath(path2)
+
+ // CREATE
+ val dbDefinition = createDbDefinition(uri1)
+
+ val storage = CatalogStorageFormat.empty.copy(
+ locationUri = Option(uri2))
+
+ val tableDefinition = CatalogTable(
+ identifier = TableIdentifier("tbl1", Some("db5")),
+ tableType = CatalogTableType.MANAGED,
+ storage = storage,
+ schema = new StructType().add("id", "long"))
+
+ val partition = CatalogTablePartition(spec = CatalogTypes.emptyTablePartitionSpec,
+ storage = CatalogStorageFormat.empty)
+
+ catalog.createDatabase(dbDefinition, ignoreIfExists = false)
+ checkEvents(CreateDatabasePreEvent("db5") :: CreateDatabaseEvent("db5") :: Nil)
+
+ catalog.createTable(tableDefinition, ignoreIfExists = false)
+ checkEvents(CreateTablePreEvent("db5", "tbl1") :: CreateTableEvent("db5", "tbl1") :: Nil)
+
+ catalog.createPartitions("db5", "tbl1", Seq(partition), ignoreIfExists = false)
Review comment:
Ur, is this a valid test case? What happens when we try to create `CatalogTypes.emptyTablePartitionSpec`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #27030:
[SPARK-30244][SQL] Emit pre/post events for "Partition" methods in
ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#discussion_r364975812
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/events.scala
##########
@@ -202,3 +203,62 @@ case class RenameFunctionEvent(
name: String,
newName: String)
extends FunctionEvent
+
+trait PartitionEvent extends DatabaseEvent {
+ /**
+ * Name of the table that was touched.
+ */
+ val name: String
Review comment:
Got it. I was referring only a partial part of your code. In that case, we don't need anything here in `PartitionEvent`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ayudovin edited a comment on issue #27030:
[SPARK-30244][SQL] Emit pre/post events for "Partition" methods in
ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
ayudovin edited a comment on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#issuecomment-575477020
> Please don't forget the above comment.
>
> > Could you give me the real SQL statement for that?
@dongjoon-hyun, sorry, I have added a comment
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27030:
[SPARK-30244][SQL] Emit pre/post events for "Partition" methods in
ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#issuecomment-584437019
Can one of the admins verify this patch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27030: [SPARK-30244][SQL] Emit
pre/post events for "Partition" methods in ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#issuecomment-577899982
**[Test build #117321 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117321/testReport)** for PR 27030 at commit [`8da2c55`](https://github.com/apache/spark/commit/8da2c555d339e3a0442086dc8725f8b1aa31d0bc).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27030: [SPARK-30244][SQL] Emit
pre/post events for "Partition" methods in ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#issuecomment-578412857
**[Test build #117401 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117401/testReport)** for PR 27030 at commit [`30686b4`](https://github.com/apache/spark/commit/30686b4a19edf7ceb13cd1d86838d4a1dd5c861b).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27030: [SPARK-30244][SQL] Emit
pre/post events for "Partition" methods in ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#issuecomment-573457218
**[Test build #116570 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/116570/testReport)** for PR 27030 at commit [`e1220e1`](https://github.com/apache/spark/commit/e1220e105bc4304365ca3f232f1825b00bd6e623).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ayudovin commented on a change in pull request #27030:
[SPARK-30244][SQL] Emit pre/post events for "Partition" methods in
ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
ayudovin commented on a change in pull request #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#discussion_r364959788
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/events.scala
##########
@@ -202,3 +203,62 @@ case class RenameFunctionEvent(
name: String,
newName: String)
extends FunctionEvent
+
+trait PartitionEvent extends DatabaseEvent {
+ /**
+ * Name of the table that was touched.
+ */
+ val name: String
Review comment:
Does it make sense to store `parts: Seq[TablePartitionSpec]`? `CatalogTablePartition` contains `TablePartitionSpec` inside.
- Create and Alter partition have as parameter `Seq[CatalogTablePartition]`
- Rename and Drop partition have as parameter `Seq[TablePartitionSpec]`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ayudovin commented on a change in pull request #27030:
[SPARK-30244][SQL] Emit pre/post events for "Partition" methods in
ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
ayudovin commented on a change in pull request #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#discussion_r364983163
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/events.scala
##########
@@ -204,61 +204,64 @@ case class RenameFunctionEvent(
newName: String)
extends FunctionEvent
-trait PartitionEvent extends DatabaseEvent {
- /**
- * Name of the table that was touched.
- */
- val name: String
-}
-
+/**
+ * Event fired when a partition is created, dropped, altered or renamed.
+ */
+trait PartitionEvent extends TableEvent
/**
* Event fired before a partition is created.
*/
-case class CreatePartitionPreEvent(database: String, name: String,
- parts: Seq[CatalogTablePartition]) extends PartitionEvent
+case class CreatePartitionPreEvent(
+ database: String,
+ name: String) extends PartitionEvent
Review comment:
Do you mean to create such a event:
```
case class CreatePartitionPreEvent(
database: String,
name: String,
parts: Seq[CatalogTablePartition]) extends PartitionEvent
```
but PartitionEvent will be empty ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27030: [SPARK-30244][SQL]
Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#issuecomment-572692528
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27030: [SPARK-30244][SQL]
Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#issuecomment-577962144
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27030: [SPARK-30244][SQL] Emit
pre/post events for "Partition" methods in ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#issuecomment-572777497
**[Test build #116418 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/116418/testReport)** for PR 27030 at commit [`af04a62`](https://github.com/apache/spark/commit/af04a625698d7f6ea2390d8286bcca8db7fdef6f).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27030: [SPARK-30244][SQL] Emit
pre/post events for "Partition" methods in ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#issuecomment-577961669
**[Test build #117321 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117321/testReport)** for PR 27030 at commit [`8da2c55`](https://github.com/apache/spark/commit/8da2c555d339e3a0442086dc8725f8b1aa31d0bc).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27030:
[SPARK-30244][SQL] Emit pre/post events for "Partition" methods in
ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#issuecomment-574529042
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/21532/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27030: [SPARK-30244][SQL]
Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#issuecomment-574542943
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/116758/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27030: [SPARK-30244][SQL] Emit
pre/post events for "Partition" methods in ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#issuecomment-590669172
**[Test build #118893 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118893/testReport)** for PR 27030 at commit [`30686b4`](https://github.com/apache/spark/commit/30686b4a19edf7ceb13cd1d86838d4a1dd5c861b).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27030:
[SPARK-30244][SQL] Emit pre/post events for "Partition" methods in
ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#issuecomment-574471628
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27030: [SPARK-30244][SQL]
Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#issuecomment-577962154
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/117321/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27030: [SPARK-30244][SQL] Emit
pre/post events for "Partition" methods in ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#issuecomment-574528630
**[Test build #116758 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/116758/testReport)** for PR 27030 at commit [`8da2c55`](https://github.com/apache/spark/commit/8da2c555d339e3a0442086dc8725f8b1aa31d0bc).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #27030: [SPARK-30244][SQL]
Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#issuecomment-578456861
@ayudovin . The current status is that we are waiting the original author of this framework, @hvanhovell . We need to check whether this is not added intentionally or not.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27030: [SPARK-30244][SQL]
Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#issuecomment-574415243
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27030:
[SPARK-30244] - Emit pre/post events for "Partition" methods in
ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27030: [SPARK-30244] - Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#issuecomment-569429252
Can one of the admins verify this patch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27030: [SPARK-30244][SQL]
Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#issuecomment-574415247
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/21505/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ayudovin edited a comment on issue #27030:
[SPARK-30244][SQL] Emit pre/post events for "Partition" methods in
ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
ayudovin edited a comment on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#issuecomment-574527485
> Hi, @ayudovin . Please check the PR content once more carefully. Thanks!
@dongjoon-hyun, I have updated PR, could you please review it again?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #27030:
[SPARK-30244][SQL] Emit pre/post events for "Partition" methods in
ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#discussion_r364980808
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/events.scala
##########
@@ -204,61 +204,64 @@ case class RenameFunctionEvent(
newName: String)
extends FunctionEvent
-trait PartitionEvent extends DatabaseEvent {
- /**
- * Name of the table that was touched.
- */
- val name: String
-}
-
+/**
+ * Event fired when a partition is created, dropped, altered or renamed.
+ */
+trait PartitionEvent extends TableEvent
/**
* Event fired before a partition is created.
*/
-case class CreatePartitionPreEvent(database: String, name: String,
- parts: Seq[CatalogTablePartition]) extends PartitionEvent
+case class CreatePartitionPreEvent(
+ database: String,
+ name: String) extends PartitionEvent
Review comment:
Ur? What I meant was we don't need to have the partition information at `PartitionEvent`. We need partition information in this event, don't we?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27030: [SPARK-30244][SQL]
Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#issuecomment-590736734
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118893/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ayudovin commented on a change in pull request #27030:
[SPARK-30244][SQL] Emit pre/post events for "Partition" methods in
ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
ayudovin commented on a change in pull request #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#discussion_r364983163
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/events.scala
##########
@@ -204,61 +204,64 @@ case class RenameFunctionEvent(
newName: String)
extends FunctionEvent
-trait PartitionEvent extends DatabaseEvent {
- /**
- * Name of the table that was touched.
- */
- val name: String
-}
-
+/**
+ * Event fired when a partition is created, dropped, altered or renamed.
+ */
+trait PartitionEvent extends TableEvent
/**
* Event fired before a partition is created.
*/
-case class CreatePartitionPreEvent(database: String, name: String,
- parts: Seq[CatalogTablePartition]) extends PartitionEvent
+case class CreatePartitionPreEvent(
+ database: String,
+ name: String) extends PartitionEvent
Review comment:
Do you mean to create such a event:
```
case class CreatePartitionPreEvent(
database: String,
name: String,
parts: Seq[CatalogTablePartition]) extends PartitionEvent
```
but PartitionEvent remains empty ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27030: [SPARK-30244][SQL]
Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#issuecomment-572775083
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27030: [SPARK-30244][SQL] Emit
pre/post events for "Partition" methods in ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#issuecomment-573137067
**[Test build #116492 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/116492/testReport)** for PR 27030 at commit [`4d5ba9b`](https://github.com/apache/spark/commit/4d5ba9b712404e3ff2db59a4193abd5a30743506).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #27030:
[SPARK-30244][SQL] Emit pre/post events for "Partition" methods in
ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#discussion_r364981019
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/events.scala
##########
@@ -204,61 +204,64 @@ case class RenameFunctionEvent(
newName: String)
extends FunctionEvent
-trait PartitionEvent extends DatabaseEvent {
- /**
- * Name of the table that was touched.
- */
- val name: String
-}
-
+/**
+ * Event fired when a partition is created, dropped, altered or renamed.
+ */
+trait PartitionEvent extends TableEvent
/**
* Event fired before a partition is created.
*/
-case class CreatePartitionPreEvent(database: String, name: String,
- parts: Seq[CatalogTablePartition]) extends PartitionEvent
+case class CreatePartitionPreEvent(
+ database: String,
+ name: String) extends PartitionEvent
Review comment:
For the indentation, shall we put `extends PartitionEvent` into a new line?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27030:
[SPARK-30244][SQL] Emit pre/post events for "Partition" methods in
ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#issuecomment-590669465
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ayudovin commented on a change in pull request #27030:
[SPARK-30244][SQL] Emit pre/post events for "Partition" methods in
ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
ayudovin commented on a change in pull request #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#discussion_r364959788
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/events.scala
##########
@@ -202,3 +203,62 @@ case class RenameFunctionEvent(
name: String,
newName: String)
extends FunctionEvent
+
+trait PartitionEvent extends DatabaseEvent {
+ /**
+ * Name of the table that was touched.
+ */
+ val name: String
Review comment:
Does it make sense to use `parts: Seq[TablePartitionSpec]`?
`CatalogTablePartition` contains `TablePartitionSpec` inside.
- Create and Alter partition have as parameter `Seq[CatalogTablePartition]`
- Rename and Drop partition have as parameter `Seq[TablePartitionSpec]`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27030: [SPARK-30244][SQL]
Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#issuecomment-590669469
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23642/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27030: [SPARK-30244][SQL]
Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#issuecomment-590736728
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27030: [SPARK-30244][SQL]
Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#issuecomment-573040834
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27030:
[SPARK-30244][SQL] Emit pre/post events for "Partition" methods in
ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#issuecomment-573040834
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27030:
[SPARK-30244][SQL] Emit pre/post events for "Partition" methods in
ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#issuecomment-572850866
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/116419/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #27030:
[SPARK-30244][SQL] Emit pre/post events for "Partition" methods in
ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#discussion_r364895084
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/events.scala
##########
@@ -202,3 +203,62 @@ case class RenameFunctionEvent(
name: String,
newName: String)
extends FunctionEvent
+
+trait PartitionEvent extends DatabaseEvent {
Review comment:
Since `Partition` belongs to `Table`, `TableEvent` is better than `DatabaseEvent`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27030: [SPARK-30244][SQL] Emit
pre/post events for "Partition" methods in ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#issuecomment-572840441
**[Test build #116418 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/116418/testReport)** for PR 27030 at commit [`af04a62`](https://github.com/apache/spark/commit/af04a625698d7f6ea2390d8286bcca8db7fdef6f).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `case class CreatePartitionPreEvent(`
* `case class CreatePartitionEvent(`
* `case class DropPartitionPreEvent(`
* `case class DropPartitionEvent(`
* `case class RenamePartitionPreEvent(`
* `case class RenamePartitionEvent(`
* `case class AlterPartitionPreEvent(`
* `case class AlterPartitionEvent(`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27030: [SPARK-30244][SQL]
Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#issuecomment-584437019
Can one of the admins verify this patch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #27030:
[SPARK-30244][SQL] Emit pre/post events for "Partition" methods in
ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#discussion_r365630139
##########
File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/catalog/ExternalCatalogEventSuite.scala
##########
@@ -209,4 +209,67 @@ class ExternalCatalogEventSuite extends SparkFunSuite {
catalog.dropFunction("db5", "fn4")
checkEvents(DropFunctionPreEvent("db5", "fn4") :: DropFunctionEvent("db5", "fn4") :: Nil)
}
+
+ testWithCatalog("partition") { (catalog, checkEvents) =>
+ val path1 = Files.createTempDirectory("db_")
+ val path2 = Files.createTempDirectory(path1, "tbl_")
+ val uri1 = preparePath(path1)
+ val uri2 = preparePath(path2)
+
+ // CREATE
+ val dbDefinition = createDbDefinition(uri1)
+
+ val storage = CatalogStorageFormat.empty.copy(
+ locationUri = Option(uri2))
+
+ val tableDefinition = CatalogTable(
+ identifier = TableIdentifier("tbl1", Some("db5")),
+ tableType = CatalogTableType.MANAGED,
+ storage = storage,
+ schema = new StructType().add("id", "long"))
+
+ val partition = CatalogTablePartition(spec = CatalogTypes.emptyTablePartitionSpec,
+ storage = CatalogStorageFormat.empty)
+
+ catalog.createDatabase(dbDefinition, ignoreIfExists = false)
+ checkEvents(CreateDatabasePreEvent("db5") :: CreateDatabaseEvent("db5") :: Nil)
+
+ catalog.createTable(tableDefinition, ignoreIfExists = false)
+ checkEvents(CreateTablePreEvent("db5", "tbl1") :: CreateTableEvent("db5", "tbl1") :: Nil)
+
+ catalog.createPartitions("db5", "tbl1", Seq(partition), ignoreIfExists = false)
+ checkEvents(CreatePartitionPreEvent("db5", "tbl1", Seq(partition)) ::
+ CreatePartitionEvent("db5", "tbl1", Seq(partition)) :: Nil)
+
+ catalog.createPartitions("db5", "tbl1", Seq(partition), ignoreIfExists = true)
+ checkEvents(CreatePartitionPreEvent("db5", "tbl1", Seq(partition)) ::
+ CreatePartitionEvent("db5", "tbl1", Seq(partition)) :: Nil)
+
+ // RENAME
+ val newPartition = CatalogTablePartition(spec = Map("key1" -> "1", "key2" -> "2"),
+ storage = CatalogStorageFormat.empty)
+
+ catalog.renamePartitions("db5", "tbl1", Seq(partition.spec), Seq(newPartition.spec))
Review comment:
This also sounds like `empty spec` to `key1=1 , key2=2`. Is this valid?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27030: [SPARK-30244] -
Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27030: [SPARK-30244] - Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#issuecomment-569429336
Can one of the admins verify this patch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27030:
[SPARK-30244][SQL] Emit pre/post events for "Partition" methods in
ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#issuecomment-578413180
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27030:
[SPARK-30244][SQL] Emit pre/post events for "Partition" methods in
ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#issuecomment-572777497
**[Test build #116418 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/116418/testReport)** for PR 27030 at commit [`af04a62`](https://github.com/apache/spark/commit/af04a625698d7f6ea2390d8286bcca8db7fdef6f).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27030: [SPARK-30244][SQL]
Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#issuecomment-574529042
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/21532/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #27030: [SPARK-30244][SQL]
Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#issuecomment-596735588
Hi, @gatorsmile and @hvanhovell .
How do you think about this extention?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27030: [SPARK-30244][SQL]
Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#issuecomment-573937138
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/116661/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27030:
[SPARK-30244][SQL] Emit pre/post events for "Partition" methods in
ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#issuecomment-577962144
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27030: [SPARK-30244][SQL]
Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#issuecomment-572780790
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/21208/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27030:
[SPARK-30244][SQL] Emit pre/post events for "Partition" methods in
ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#issuecomment-578412857
**[Test build #117401 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117401/testReport)** for PR 27030 at commit [`30686b4`](https://github.com/apache/spark/commit/30686b4a19edf7ceb13cd1d86838d4a1dd5c861b).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27030: [SPARK-30244][SQL] Emit
pre/post events for "Partition" methods in ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#issuecomment-578432613
**[Test build #117401 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117401/testReport)** for PR 27030 at commit [`30686b4`](https://github.com/apache/spark/commit/30686b4a19edf7ceb13cd1d86838d4a1dd5c861b).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #27030:
[SPARK-30244][SQL] Emit pre/post events for "Partition" methods in
ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#discussion_r366618223
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/events.scala
##########
@@ -202,3 +203,83 @@ case class RenameFunctionEvent(
name: String,
newName: String)
extends FunctionEvent
+
+/**
+ * Event fired when a partition is created, dropped, altered or renamed.
+ */
+trait PartitionEvent extends TableEvent
+
+/**
+ * Event fired before a partition is created.
+ */
+case class CreatePartitionPreEvent(
+ database: String,
+ name: String,
+ parts: Seq[CatalogTablePartition])
+ extends PartitionEvent
+
+/**
+ * Event fired after a partition has been created.
+ */
+case class CreatePartitionEvent(
+ database: String,
+ name: String,
+ parts: Seq[CatalogTablePartition])
+ extends PartitionEvent
+
+/**
+ * Event fired before a partition is dropped.
+ */
+case class DropPartitionPreEvent(
+ database: String,
+ name: String,
+ partSpecs: Seq[TablePartitionSpec])
+ extends PartitionEvent
+
+/**
+ * Event fired after a partition has been dropped.
+ */
+case class DropPartitionEvent(
+ database: String,
+ name: String,
+ partSpecs: Seq[TablePartitionSpec])
+ extends PartitionEvent
+
+/**
+ * Event fired before a function is renamed.
+ */
+case class RenamePartitionPreEvent(
+ database: String,
+ name: String,
+ specs: Seq[TablePartitionSpec],
+ newSpecs: Seq[TablePartitionSpec])
+ extends PartitionEvent
+
+/**
+ * Event fired after a function has been renamed.
+ */
+case class RenamePartitionEvent(
+ database: String,
+ name: String,
+ specs: Seq[TablePartitionSpec],
+ newSpecs: Seq[TablePartitionSpec])
+ extends PartitionEvent
+
+/**
+ * Event fired before a function is altered.
Review comment:
`function` -> `partition`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27030: [SPARK-30244] -
Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27030: [SPARK-30244] - Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#issuecomment-569429252
Can one of the admins verify this patch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27030: [SPARK-30244][SQL]
Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#issuecomment-572916406
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27030: [SPARK-30244][SQL]
Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#issuecomment-574471628
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27030:
[SPARK-30244][SQL] Emit pre/post events for "Partition" methods in
ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#issuecomment-572841014
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/116418/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27030:
[SPARK-30244][SQL] Emit pre/post events for "Partition" methods in
ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#issuecomment-572780298
**[Test build #116419 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/116419/testReport)** for PR 27030 at commit [`0c5edc2`](https://github.com/apache/spark/commit/0c5edc2314ef9d7c23d759735f44610f90655312).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #27030: [SPARK-30244][SQL]
Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#issuecomment-577987120
Also, I'm worrying the situation where this implementation causes a sabotage on the event message pipeline. Sometimes, the changed number of partition can be huge in the production.
Hi, @hvanhovell . How do you think about this PR? Could you give us some advice?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #27030: [SPARK-30244][SQL]
Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#issuecomment-572690653
ok to test
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27030: [SPARK-30244][SQL]
Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#issuecomment-573457358
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27030: [SPARK-30244][SQL]
Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#issuecomment-574542935
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27030: [SPARK-30244][SQL]
Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#issuecomment-573937133
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ayudovin commented on issue #27030: [SPARK-30244][SQL] Emit
pre/post events for "Partition" methods in ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
ayudovin commented on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#issuecomment-575477020
> Please don't forget the above comment.
>
> > Could you give me the real SQL statement for that?
Sorry, I have added a comment
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27030: [SPARK-30244][SQL]
Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#issuecomment-574471635
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/116730/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ayudovin commented on issue #27030:
[SPARK-30244][SQL][Catalyst] - Emit pre/post events for "Partition" methods
in ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
ayudovin commented on issue #27030: [SPARK-30244][SQL][Catalyst] - Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#issuecomment-572686515
cc @dongjoon-hyun
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27030: [SPARK-30244][SQL] Emit
pre/post events for "Partition" methods in ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#issuecomment-574542866
**[Test build #116758 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/116758/testReport)** for PR 27030 at commit [`8da2c55`](https://github.com/apache/spark/commit/8da2c555d339e3a0442086dc8725f8b1aa31d0bc).
* This patch **fails due to an unknown error code, -9**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27030: [SPARK-30244][SQL]
Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#issuecomment-572919678
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ayudovin commented on a change in pull request #27030:
[SPARK-30244][SQL] Emit pre/post events for "Partition" methods in
ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
ayudovin commented on a change in pull request #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#discussion_r367779692
##########
File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/catalog/ExternalCatalogEventSuite.scala
##########
@@ -209,4 +209,67 @@ class ExternalCatalogEventSuite extends SparkFunSuite {
catalog.dropFunction("db5", "fn4")
checkEvents(DropFunctionPreEvent("db5", "fn4") :: DropFunctionEvent("db5", "fn4") :: Nil)
}
+
+ testWithCatalog("partition") { (catalog, checkEvents) =>
+ val path1 = Files.createTempDirectory("db_")
+ val path2 = Files.createTempDirectory(path1, "tbl_")
+ val uri1 = preparePath(path1)
+ val uri2 = preparePath(path2)
+
+ // CREATE
+ val dbDefinition = createDbDefinition(uri1)
+
+ val storage = CatalogStorageFormat.empty.copy(
+ locationUri = Option(uri2))
+
+ val tableDefinition = CatalogTable(
+ identifier = TableIdentifier("tbl1", Some("db5")),
+ tableType = CatalogTableType.MANAGED,
+ storage = storage,
+ schema = new StructType().add("id", "long"))
+
+ val partition = CatalogTablePartition(spec = CatalogTypes.emptyTablePartitionSpec,
+ storage = CatalogStorageFormat.empty)
+
+ catalog.createDatabase(dbDefinition, ignoreIfExists = false)
+ checkEvents(CreateDatabasePreEvent("db5") :: CreateDatabaseEvent("db5") :: Nil)
+
+ catalog.createTable(tableDefinition, ignoreIfExists = false)
+ checkEvents(CreateTablePreEvent("db5", "tbl1") :: CreateTableEvent("db5", "tbl1") :: Nil)
+
+ catalog.createPartitions("db5", "tbl1", Seq(partition), ignoreIfExists = false)
+ checkEvents(CreatePartitionPreEvent("db5", "tbl1", Seq(partition)) ::
+ CreatePartitionEvent("db5", "tbl1", Seq(partition)) :: Nil)
+
+ catalog.createPartitions("db5", "tbl1", Seq(partition), ignoreIfExists = true)
+ checkEvents(CreatePartitionPreEvent("db5", "tbl1", Seq(partition)) ::
+ CreatePartitionEvent("db5", "tbl1", Seq(partition)) :: Nil)
+
+ // RENAME
+ val newPartition = CatalogTablePartition(spec = Map("key1" -> "1", "key2" -> "2"),
+ storage = CatalogStorageFormat.empty)
+
+ catalog.renamePartitions("db5", "tbl1", Seq(partition.spec), Seq(newPartition.spec))
Review comment:
So, I could not find the real SQL statement for that, that's why I changed this unit tests.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27030: [SPARK-30244][SQL]
Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#issuecomment-572780781
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27030: [SPARK-30244][SQL]
Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#issuecomment-574529036
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ayudovin commented on a change in pull request #27030:
[SPARK-30244][SQL] Emit pre/post events for "Partition" methods in
ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
ayudovin commented on a change in pull request #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#discussion_r364959788
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/events.scala
##########
@@ -202,3 +203,62 @@ case class RenameFunctionEvent(
name: String,
newName: String)
extends FunctionEvent
+
+trait PartitionEvent extends DatabaseEvent {
+ /**
+ * Name of the table that was touched.
+ */
+ val name: String
Review comment:
Does it make sense to use `parts: Seq[TablePartitionSpec]`? `CatalogTablePartition` contains `TablePartitionSpec` inside.
- Create and Alter partition have as parameter `Seq[CatalogTablePartition]`
- Rename and Drop partition have as parameter `Seq[TablePartitionSpec]`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27030: [SPARK-30244][SQL] Emit
pre/post events for "Partition" methods in ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#issuecomment-572919650
**[Test build #116464 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/116464/testReport)** for PR 27030 at commit [`3cb79fe`](https://github.com/apache/spark/commit/3cb79fe63a14ecba62bfc9b449ecf9b4a28b6e10).
* This patch **fails due to an unknown error code, -9**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on issue #27030: [SPARK-30244][SQL]
Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#issuecomment-590667901
WDYT @hvanhovell?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27030:
[SPARK-30244][SQL] Emit pre/post events for "Partition" methods in
ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#issuecomment-577899982
**[Test build #117321 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117321/testReport)** for PR 27030 at commit [`8da2c55`](https://github.com/apache/spark/commit/8da2c555d339e3a0442086dc8725f8b1aa31d0bc).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ayudovin commented on a change in pull request #27030:
[SPARK-30244][SQL] Emit pre/post events for "Partition" methods in
ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
ayudovin commented on a change in pull request #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#discussion_r366019687
##########
File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/catalog/ExternalCatalogEventSuite.scala
##########
@@ -209,4 +209,67 @@ class ExternalCatalogEventSuite extends SparkFunSuite {
catalog.dropFunction("db5", "fn4")
checkEvents(DropFunctionPreEvent("db5", "fn4") :: DropFunctionEvent("db5", "fn4") :: Nil)
}
+
+ testWithCatalog("partition") { (catalog, checkEvents) =>
+ val path1 = Files.createTempDirectory("db_")
+ val path2 = Files.createTempDirectory(path1, "tbl_")
+ val uri1 = preparePath(path1)
+ val uri2 = preparePath(path2)
+
+ // CREATE
+ val dbDefinition = createDbDefinition(uri1)
+
+ val storage = CatalogStorageFormat.empty.copy(
+ locationUri = Option(uri2))
+
+ val tableDefinition = CatalogTable(
+ identifier = TableIdentifier("tbl1", Some("db5")),
+ tableType = CatalogTableType.MANAGED,
+ storage = storage,
+ schema = new StructType().add("id", "long"))
+
+ val partition = CatalogTablePartition(spec = CatalogTypes.emptyTablePartitionSpec,
+ storage = CatalogStorageFormat.empty)
+
+ catalog.createDatabase(dbDefinition, ignoreIfExists = false)
+ checkEvents(CreateDatabasePreEvent("db5") :: CreateDatabaseEvent("db5") :: Nil)
+
+ catalog.createTable(tableDefinition, ignoreIfExists = false)
+ checkEvents(CreateTablePreEvent("db5", "tbl1") :: CreateTableEvent("db5", "tbl1") :: Nil)
+
+ catalog.createPartitions("db5", "tbl1", Seq(partition), ignoreIfExists = false)
Review comment:
agree with you, `CatalogTypes.emptyTablePartitionSpec` is not good option for unit test. I have changed it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27030:
[SPARK-30244][SQL] Emit pre/post events for "Partition" methods in
ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#issuecomment-572919678
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #27030:
[SPARK-30244][SQL] Emit pre/post events for "Partition" methods in
ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#discussion_r370469963
##########
File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/catalog/ExternalCatalogEventSuite.scala
##########
@@ -209,4 +209,67 @@ class ExternalCatalogEventSuite extends SparkFunSuite {
catalog.dropFunction("db5", "fn4")
checkEvents(DropFunctionPreEvent("db5", "fn4") :: DropFunctionEvent("db5", "fn4") :: Nil)
}
+
+ testWithCatalog("partition") { (catalog, checkEvents) =>
+ val path1 = Files.createTempDirectory("db_")
+ val path2 = Files.createTempDirectory(path1, "tbl_")
+ val uri1 = preparePath(path1)
+ val uri2 = preparePath(path2)
+
+ // CREATE
+ val dbDefinition = createDbDefinition(uri1)
+
+ val storage = CatalogStorageFormat.empty.copy(
+ locationUri = Option(uri2))
+
+ val tableDefinition = CatalogTable(
+ identifier = TableIdentifier("tbl1", Some("db5")),
+ tableType = CatalogTableType.MANAGED,
+ storage = storage,
+ schema = new StructType().add("id", "long"))
+
+ val partition = CatalogTablePartition(spec = Map("key1" -> "1", "key2" -> "2"),
+ storage = CatalogStorageFormat.empty)
+
+ catalog.createDatabase(dbDefinition, ignoreIfExists = false)
+ checkEvents(CreateDatabasePreEvent("db5") :: CreateDatabaseEvent("db5") :: Nil)
+
+ catalog.createTable(tableDefinition, ignoreIfExists = false)
+ checkEvents(CreateTablePreEvent("db5", "tbl1") :: CreateTableEvent("db5", "tbl1") :: Nil)
+
+ catalog.createPartitions("db5", "tbl1", Seq(partition), ignoreIfExists = false)
+ checkEvents(CreatePartitionPreEvent("db5", "tbl1", Seq(partition)) ::
+ CreatePartitionEvent("db5", "tbl1", Seq(partition)) :: Nil)
+
+ catalog.createPartitions("db5", "tbl1", Seq(partition), ignoreIfExists = true)
+ checkEvents(CreatePartitionPreEvent("db5", "tbl1", Seq(partition)) ::
+ CreatePartitionEvent("db5", "tbl1", Seq(partition)) :: Nil)
+
+ // RENAME
+ val newPartition = CatalogTablePartition(spec = Map("key1" -> "3", "key2" -> "4"),
+ storage = CatalogStorageFormat.empty)
+
+ catalog.renamePartitions("db5", "tbl1", Seq(partition.spec), Seq(newPartition.spec))
+ checkEvents(
+ RenamePartitionPreEvent("db5", "tbl1", Seq(partition.spec), Seq(newPartition.spec)) ::
+ RenamePartitionEvent("db5", "tbl1", Seq(partition.spec), Seq(newPartition.spec)) ::
+ Nil)
+
+ // ALTER
+ catalog.alterPartitions("db5", "tbl1", Seq.empty)
+ checkEvents(AlterPartitionPreEvent("db5", "tbl1", Seq.empty) ::
+ AlterPartitionEvent("db5", "tbl1", Seq.empty) :: Nil)
+
+ // DROP
+ intercept[AnalysisException] {
+ catalog.dropPartitions("db5", "tbl2", Seq(newPartition.spec),
+ ignoreIfNotExists = false, purge = true, retainData = false)
+ }
Review comment:
It's always a good practice to check the error message. Sometime, `AnalysisException` is changed unexpectedly.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27030: [SPARK-30244][SQL]
Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#issuecomment-577900562
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22081/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27030: [SPARK-30244][SQL]
Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#issuecomment-572850866
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/116419/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27030: [SPARK-30244][SQL]
Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#issuecomment-572850861
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27030: [SPARK-30244][SQL]
Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#issuecomment-573457361
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/21356/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27030:
[SPARK-30244][SQL] Emit pre/post events for "Partition" methods in
ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#issuecomment-590736728
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on issue #27030: [SPARK-30244][SQL]
Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#issuecomment-590667763
ok to test
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27030: [SPARK-30244][SQL] Emit
pre/post events for "Partition" methods in ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#issuecomment-590736007
**[Test build #118893 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118893/testReport)** for PR 27030 at commit [`30686b4`](https://github.com/apache/spark/commit/30686b4a19edf7ceb13cd1d86838d4a1dd5c861b).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27030:
[SPARK-30244][SQL] Emit pre/post events for "Partition" methods in
ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#issuecomment-572915998
**[Test build #116464 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/116464/testReport)** for PR 27030 at commit [`3cb79fe`](https://github.com/apache/spark/commit/3cb79fe63a14ecba62bfc9b449ecf9b4a28b6e10).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27030: [SPARK-30244][SQL]
Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#issuecomment-573479246
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ayudovin commented on issue #27030: [SPARK-30244][SQL] Emit
pre/post events for "Partition" methods in ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
ayudovin commented on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#issuecomment-578516239
> @ayudovin . The current status is that we are waiting the original author of this framework, @hvanhovell . We need to check whether this is not added intentionally or not. Also, [#27030 (comment)](https://github.com/apache/spark/pull/27030#discussion_r370965396) is another concern no this approach because it can be misleading.
I have got, It makes sense
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27030:
[SPARK-30244][SQL] Emit pre/post events for "Partition" methods in
ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#issuecomment-572780790
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/21208/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27030:
[SPARK-30244][SQL] Emit pre/post events for "Partition" methods in
ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#issuecomment-572841008
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27030: [SPARK-30244][SQL]
Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#issuecomment-572791530
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/116410/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27030:
[SPARK-30244][SQL] Emit pre/post events for "Partition" methods in
ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#issuecomment-574471635
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/116730/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #27030:
[SPARK-30244][SQL] Emit pre/post events for "Partition" methods in
ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#discussion_r366619234
##########
File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/catalog/ExternalCatalogEventSuite.scala
##########
@@ -209,4 +209,67 @@ class ExternalCatalogEventSuite extends SparkFunSuite {
catalog.dropFunction("db5", "fn4")
checkEvents(DropFunctionPreEvent("db5", "fn4") :: DropFunctionEvent("db5", "fn4") :: Nil)
}
+
+ testWithCatalog("partition") { (catalog, checkEvents) =>
+ val path1 = Files.createTempDirectory("db_")
+ val path2 = Files.createTempDirectory(path1, "tbl_")
+ val uri1 = preparePath(path1)
+ val uri2 = preparePath(path2)
+
+ // CREATE
+ val dbDefinition = createDbDefinition(uri1)
+
+ val storage = CatalogStorageFormat.empty.copy(
+ locationUri = Option(uri2))
+
+ val tableDefinition = CatalogTable(
+ identifier = TableIdentifier("tbl1", Some("db5")),
+ tableType = CatalogTableType.MANAGED,
+ storage = storage,
+ schema = new StructType().add("id", "long"))
+
+ val partition = CatalogTablePartition(spec = CatalogTypes.emptyTablePartitionSpec,
+ storage = CatalogStorageFormat.empty)
+
+ catalog.createDatabase(dbDefinition, ignoreIfExists = false)
+ checkEvents(CreateDatabasePreEvent("db5") :: CreateDatabaseEvent("db5") :: Nil)
+
+ catalog.createTable(tableDefinition, ignoreIfExists = false)
+ checkEvents(CreateTablePreEvent("db5", "tbl1") :: CreateTableEvent("db5", "tbl1") :: Nil)
+
+ catalog.createPartitions("db5", "tbl1", Seq(partition), ignoreIfExists = false)
+ checkEvents(CreatePartitionPreEvent("db5", "tbl1", Seq(partition)) ::
+ CreatePartitionEvent("db5", "tbl1", Seq(partition)) :: Nil)
+
+ catalog.createPartitions("db5", "tbl1", Seq(partition), ignoreIfExists = true)
+ checkEvents(CreatePartitionPreEvent("db5", "tbl1", Seq(partition)) ::
+ CreatePartitionEvent("db5", "tbl1", Seq(partition)) :: Nil)
+
+ // RENAME
+ val newPartition = CatalogTablePartition(spec = Map("key1" -> "1", "key2" -> "2"),
+ storage = CatalogStorageFormat.empty)
+
+ catalog.renamePartitions("db5", "tbl1", Seq(partition.spec), Seq(newPartition.spec))
Review comment:
Could you give me the real SQL statement for that?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27030: [SPARK-30244][SQL]
Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#issuecomment-577900553
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ayudovin commented on issue #27030: [SPARK-30244][SQL] Emit
pre/post events for "Partition" methods in ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
ayudovin commented on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#issuecomment-577882110
Hi @dongjoon-hyun , Do you have still any suggestions or comments on this pull request?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27030: [SPARK-30244][SQL]
Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#issuecomment-578413180
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ayudovin commented on a change in pull request #27030:
[SPARK-30244][SQL] Emit pre/post events for "Partition" methods in
ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
ayudovin commented on a change in pull request #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#discussion_r366727729
##########
File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/catalog/ExternalCatalogEventSuite.scala
##########
@@ -209,4 +209,67 @@ class ExternalCatalogEventSuite extends SparkFunSuite {
catalog.dropFunction("db5", "fn4")
checkEvents(DropFunctionPreEvent("db5", "fn4") :: DropFunctionEvent("db5", "fn4") :: Nil)
}
+
+ testWithCatalog("partition") { (catalog, checkEvents) =>
+ val path1 = Files.createTempDirectory("db_")
+ val path2 = Files.createTempDirectory(path1, "tbl_")
+ val uri1 = preparePath(path1)
+ val uri2 = preparePath(path2)
+
+ // CREATE
+ val dbDefinition = createDbDefinition(uri1)
+
+ val storage = CatalogStorageFormat.empty.copy(
+ locationUri = Option(uri2))
+
+ val tableDefinition = CatalogTable(
+ identifier = TableIdentifier("tbl1", Some("db5")),
+ tableType = CatalogTableType.MANAGED,
+ storage = storage,
+ schema = new StructType().add("id", "long"))
+
+ val partition = CatalogTablePartition(spec = CatalogTypes.emptyTablePartitionSpec,
+ storage = CatalogStorageFormat.empty)
+
+ catalog.createDatabase(dbDefinition, ignoreIfExists = false)
+ checkEvents(CreateDatabasePreEvent("db5") :: CreateDatabaseEvent("db5") :: Nil)
+
+ catalog.createTable(tableDefinition, ignoreIfExists = false)
+ checkEvents(CreateTablePreEvent("db5", "tbl1") :: CreateTableEvent("db5", "tbl1") :: Nil)
+
+ catalog.createPartitions("db5", "tbl1", Seq(partition), ignoreIfExists = false)
Review comment:
hm, I cannot find an example, that's why I also think it isn't happening in the real world.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27030:
[SPARK-30244][SQL] Emit pre/post events for "Partition" methods in
ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#issuecomment-573457358
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27030: [SPARK-30244][SQL]
Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#issuecomment-573137721
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/116492/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27030: [SPARK-30244][SQL] Emit
pre/post events for "Partition" methods in ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#issuecomment-572691990
**[Test build #116410 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/116410/testReport)** for PR 27030 at commit [`5c242dc`](https://github.com/apache/spark/commit/5c242dc3700789f4af49f9e3d4b544d3b1d2f9e7).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ayudovin commented on issue #27030: [SPARK-30244][SQL] Emit
pre/post events for "Partition" methods in ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
ayudovin commented on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#issuecomment-575476101
> Please don't forget the above comment.
>
> > Could you give me the real SQL statement for that?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27030:
[SPARK-30244][SQL] Emit pre/post events for "Partition" methods in
ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#issuecomment-573937138
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/116661/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27030:
[SPARK-30244][SQL] Emit pre/post events for "Partition" methods in
ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#issuecomment-574415247
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/21505/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27030:
[SPARK-30244][SQL] Emit pre/post events for "Partition" methods in
ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#issuecomment-573040845
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/21278/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27030: [SPARK-30244][SQL] Emit
pre/post events for "Partition" methods in ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#issuecomment-573864821
**[Test build #116661 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/116661/testReport)** for PR 27030 at commit [`43aeb9b`](https://github.com/apache/spark/commit/43aeb9ba12c5dabcaddec3e84a2428ad2ebb09b6).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27030:
[SPARK-30244][SQL] Emit pre/post events for "Partition" methods in
ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#issuecomment-572919690
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/116464/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ayudovin commented on a change in pull request #27030:
[SPARK-30244][SQL] Emit pre/post events for "Partition" methods in
ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
ayudovin commented on a change in pull request #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#discussion_r370936472
##########
File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/catalog/ExternalCatalogEventSuite.scala
##########
@@ -209,4 +209,67 @@ class ExternalCatalogEventSuite extends SparkFunSuite {
catalog.dropFunction("db5", "fn4")
checkEvents(DropFunctionPreEvent("db5", "fn4") :: DropFunctionEvent("db5", "fn4") :: Nil)
}
+
+ testWithCatalog("partition") { (catalog, checkEvents) =>
+ val path1 = Files.createTempDirectory("db_")
+ val path2 = Files.createTempDirectory(path1, "tbl_")
+ val uri1 = preparePath(path1)
+ val uri2 = preparePath(path2)
+
+ // CREATE
+ val dbDefinition = createDbDefinition(uri1)
+
+ val storage = CatalogStorageFormat.empty.copy(
+ locationUri = Option(uri2))
+
+ val tableDefinition = CatalogTable(
+ identifier = TableIdentifier("tbl1", Some("db5")),
+ tableType = CatalogTableType.MANAGED,
+ storage = storage,
+ schema = new StructType().add("id", "long"))
+
+ val partition = CatalogTablePartition(spec = Map("key1" -> "1", "key2" -> "2"),
+ storage = CatalogStorageFormat.empty)
+
+ catalog.createDatabase(dbDefinition, ignoreIfExists = false)
+ checkEvents(CreateDatabasePreEvent("db5") :: CreateDatabaseEvent("db5") :: Nil)
+
+ catalog.createTable(tableDefinition, ignoreIfExists = false)
+ checkEvents(CreateTablePreEvent("db5", "tbl1") :: CreateTableEvent("db5", "tbl1") :: Nil)
+
+ catalog.createPartitions("db5", "tbl1", Seq(partition), ignoreIfExists = false)
+ checkEvents(CreatePartitionPreEvent("db5", "tbl1", Seq(partition)) ::
+ CreatePartitionEvent("db5", "tbl1", Seq(partition)) :: Nil)
+
+ catalog.createPartitions("db5", "tbl1", Seq(partition), ignoreIfExists = true)
+ checkEvents(CreatePartitionPreEvent("db5", "tbl1", Seq(partition)) ::
+ CreatePartitionEvent("db5", "tbl1", Seq(partition)) :: Nil)
Review comment:
I agree with you, it looks incorrect, but we can not check if the partition has been created or not created in case of using `ignoreIfExists=true`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27030:
[SPARK-30244][SQL] Emit pre/post events for "Partition" methods in
ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#issuecomment-577962154
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/117321/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27030:
[SPARK-30244][SQL] Emit pre/post events for "Partition" methods in
ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#issuecomment-574414700
**[Test build #116730 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/116730/testReport)** for PR 27030 at commit [`43aeb9b`](https://github.com/apache/spark/commit/43aeb9ba12c5dabcaddec3e84a2428ad2ebb09b6).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ayudovin commented on issue #27030:
[SPARK-30244][SQL][Catalyst] - Emit pre/post events for "Partition" methods
in ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
ayudovin commented on issue #27030: [SPARK-30244][SQL][Catalyst] - Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#issuecomment-570820515
@hvanhovell, Could you please review this pull request?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #27030:
[SPARK-30244][SQL] Emit pre/post events for "Partition" methods in
ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#discussion_r366618627
##########
File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/catalog/ExternalCatalogEventSuite.scala
##########
@@ -209,4 +209,67 @@ class ExternalCatalogEventSuite extends SparkFunSuite {
catalog.dropFunction("db5", "fn4")
checkEvents(DropFunctionPreEvent("db5", "fn4") :: DropFunctionEvent("db5", "fn4") :: Nil)
}
+
+ testWithCatalog("partition") { (catalog, checkEvents) =>
+ val path1 = Files.createTempDirectory("db_")
+ val path2 = Files.createTempDirectory(path1, "tbl_")
+ val uri1 = preparePath(path1)
+ val uri2 = preparePath(path2)
+
+ // CREATE
+ val dbDefinition = createDbDefinition(uri1)
+
+ val storage = CatalogStorageFormat.empty.copy(
+ locationUri = Option(uri2))
+
+ val tableDefinition = CatalogTable(
+ identifier = TableIdentifier("tbl1", Some("db5")),
+ tableType = CatalogTableType.MANAGED,
+ storage = storage,
+ schema = new StructType().add("id", "long"))
+
+ val partition = CatalogTablePartition(spec = CatalogTypes.emptyTablePartitionSpec,
+ storage = CatalogStorageFormat.empty)
+
+ catalog.createDatabase(dbDefinition, ignoreIfExists = false)
+ checkEvents(CreateDatabasePreEvent("db5") :: CreateDatabaseEvent("db5") :: Nil)
+
+ catalog.createTable(tableDefinition, ignoreIfExists = false)
+ checkEvents(CreateTablePreEvent("db5", "tbl1") :: CreateTableEvent("db5", "tbl1") :: Nil)
+
+ catalog.createPartitions("db5", "tbl1", Seq(partition), ignoreIfExists = false)
Review comment:
Does it happen in the real world? Can you give me an SQL example to create the empty partition?
> Despite this, I think that empty partition creation should also produce the event.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27030:
[SPARK-30244][SQL] Emit pre/post events for "Partition" methods in
ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#issuecomment-573479246
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ayudovin removed a comment on issue #27030:
[SPARK-30244][SQL] Emit pre/post events for "Partition" methods in
ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
ayudovin removed a comment on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#issuecomment-575476101
> Please don't forget the above comment.
>
> > Could you give me the real SQL statement for that?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27030:
[SPARK-30244][SQL] Emit pre/post events for "Partition" methods in
ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#issuecomment-573864821
**[Test build #116661 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/116661/testReport)** for PR 27030 at commit [`43aeb9b`](https://github.com/apache/spark/commit/43aeb9ba12c5dabcaddec3e84a2428ad2ebb09b6).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27030:
[SPARK-30244][SQL] Emit pre/post events for "Partition" methods in
ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#issuecomment-572692536
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/21199/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27030: [SPARK-30244][SQL] Emit
pre/post events for "Partition" methods in ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#issuecomment-572780298
**[Test build #116419 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/116419/testReport)** for PR 27030 at commit [`0c5edc2`](https://github.com/apache/spark/commit/0c5edc2314ef9d7c23d759735f44610f90655312).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27030:
[SPARK-30244][SQL] Emit pre/post events for "Partition" methods in
ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#issuecomment-590669172
**[Test build #118893 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118893/testReport)** for PR 27030 at commit [`30686b4`](https://github.com/apache/spark/commit/30686b4a19edf7ceb13cd1d86838d4a1dd5c861b).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27030:
[SPARK-30244][SQL] Emit pre/post events for "Partition" methods in
ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#issuecomment-574542943
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/116758/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27030:
[SPARK-30244][SQL] Emit pre/post events for "Partition" methods in
ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#issuecomment-572775097
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/21207/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27030:
[SPARK-30244][SQL] Emit pre/post events for "Partition" methods in
ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#issuecomment-572691990
**[Test build #116410 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/116410/testReport)** for PR 27030 at commit [`5c242dc`](https://github.com/apache/spark/commit/5c242dc3700789f4af49f9e3d4b544d3b1d2f9e7).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27030:
[SPARK-30244][SQL] Emit pre/post events for "Partition" methods in
ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#issuecomment-572916406
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27030: [SPARK-30244][SQL]
Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#issuecomment-572791525
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ayudovin commented on a change in pull request #27030:
[SPARK-30244][SQL] Emit pre/post events for "Partition" methods in
ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
ayudovin commented on a change in pull request #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#discussion_r364983163
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/events.scala
##########
@@ -204,61 +204,64 @@ case class RenameFunctionEvent(
newName: String)
extends FunctionEvent
-trait PartitionEvent extends DatabaseEvent {
- /**
- * Name of the table that was touched.
- */
- val name: String
-}
-
+/**
+ * Event fired when a partition is created, dropped, altered or renamed.
+ */
+trait PartitionEvent extends TableEvent
/**
* Event fired before a partition is created.
*/
-case class CreatePartitionPreEvent(database: String, name: String,
- parts: Seq[CatalogTablePartition]) extends PartitionEvent
+case class CreatePartitionPreEvent(
+ database: String,
+ name: String) extends PartitionEvent
Review comment:
Do you mean to create such a way:
```
case class CreatePartitionPreEvent(
database: String,
name: String,
parts: Seq[CatalogTablePartition]) extends PartitionEvent
```
but PartitionEvent remains empty ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27030:
[SPARK-30244][SQL] Emit pre/post events for "Partition" methods in
ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#issuecomment-574528630
**[Test build #116758 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/116758/testReport)** for PR 27030 at commit [`8da2c55`](https://github.com/apache/spark/commit/8da2c555d339e3a0442086dc8725f8b1aa31d0bc).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ayudovin closed pull request #27030: [SPARK-30244][SQL]
Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
ayudovin closed pull request #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27030:
[SPARK-30244][SQL] Emit pre/post events for "Partition" methods in
ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#issuecomment-577900562
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22081/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27030:
[SPARK-30244][SQL] Emit pre/post events for "Partition" methods in
ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#issuecomment-590669469
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23642/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27030: [SPARK-30244][SQL]
Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#issuecomment-572919690
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/116464/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ayudovin commented on a change in pull request #27030:
[SPARK-30244][SQL] Emit pre/post events for "Partition" methods in
ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
ayudovin commented on a change in pull request #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#discussion_r365610473
##########
File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/catalog/ExternalCatalogEventSuite.scala
##########
@@ -209,4 +209,65 @@ class ExternalCatalogEventSuite extends SparkFunSuite {
catalog.dropFunction("db5", "fn4")
checkEvents(DropFunctionPreEvent("db5", "fn4") :: DropFunctionEvent("db5", "fn4") :: Nil)
}
+
+ testWithCatalog("partition") { (catalog, checkEvents) =>
+ // CREATE
+ val path1 = Files.createTempDirectory("db_")
+ val uri1 = preparePath(path1)
+
+ val dbDefinition = createDbDefinition()
+
+ val storage = CatalogStorageFormat.empty.copy(
+ locationUri = Option(uri1))
Review comment:
yes, you are right
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ayudovin commented on issue #27030: [SPARK-30244][SQL] Emit
pre/post events for "Partition" methods in ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
ayudovin commented on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#issuecomment-574527485
> Hi, @ayudovin . Please check the PR content once more carefully. Thanks!
I have updated PR, could you please review it again?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27030:
[SPARK-30244][SQL] Emit pre/post events for "Partition" methods in
ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#issuecomment-574415243
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ayudovin commented on a change in pull request #27030:
[SPARK-30244][SQL] Emit pre/post events for "Partition" methods in
ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
ayudovin commented on a change in pull request #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#discussion_r366021582
##########
File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/catalog/ExternalCatalogEventSuite.scala
##########
@@ -209,4 +209,67 @@ class ExternalCatalogEventSuite extends SparkFunSuite {
catalog.dropFunction("db5", "fn4")
checkEvents(DropFunctionPreEvent("db5", "fn4") :: DropFunctionEvent("db5", "fn4") :: Nil)
}
+
+ testWithCatalog("partition") { (catalog, checkEvents) =>
+ val path1 = Files.createTempDirectory("db_")
+ val path2 = Files.createTempDirectory(path1, "tbl_")
+ val uri1 = preparePath(path1)
+ val uri2 = preparePath(path2)
+
+ // CREATE
+ val dbDefinition = createDbDefinition(uri1)
+
+ val storage = CatalogStorageFormat.empty.copy(
+ locationUri = Option(uri2))
+
+ val tableDefinition = CatalogTable(
+ identifier = TableIdentifier("tbl1", Some("db5")),
+ tableType = CatalogTableType.MANAGED,
+ storage = storage,
+ schema = new StructType().add("id", "long"))
+
+ val partition = CatalogTablePartition(spec = CatalogTypes.emptyTablePartitionSpec,
+ storage = CatalogStorageFormat.empty)
+
+ catalog.createDatabase(dbDefinition, ignoreIfExists = false)
+ checkEvents(CreateDatabasePreEvent("db5") :: CreateDatabaseEvent("db5") :: Nil)
+
+ catalog.createTable(tableDefinition, ignoreIfExists = false)
+ checkEvents(CreateTablePreEvent("db5", "tbl1") :: CreateTableEvent("db5", "tbl1") :: Nil)
+
+ catalog.createPartitions("db5", "tbl1", Seq(partition), ignoreIfExists = false)
Review comment:
Despite this, I think that empty partition creation should also produce the event.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27030: [SPARK-30244][SQL] Emit
pre/post events for "Partition" methods in ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#issuecomment-573043280
**[Test build #116492 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/116492/testReport)** for PR 27030 at commit [`4d5ba9b`](https://github.com/apache/spark/commit/4d5ba9b712404e3ff2db59a4193abd5a30743506).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27030: [SPARK-30244][SQL]
Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#issuecomment-578432911
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #27030:
[SPARK-30244][SQL] Emit pre/post events for "Partition" methods in
ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#discussion_r366618103
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/events.scala
##########
@@ -202,3 +203,83 @@ case class RenameFunctionEvent(
name: String,
newName: String)
extends FunctionEvent
+
+/**
+ * Event fired when a partition is created, dropped, altered or renamed.
+ */
+trait PartitionEvent extends TableEvent
+
+/**
+ * Event fired before a partition is created.
+ */
+case class CreatePartitionPreEvent(
+ database: String,
+ name: String,
+ parts: Seq[CatalogTablePartition])
+ extends PartitionEvent
+
+/**
+ * Event fired after a partition has been created.
+ */
+case class CreatePartitionEvent(
+ database: String,
+ name: String,
+ parts: Seq[CatalogTablePartition])
+ extends PartitionEvent
+
+/**
+ * Event fired before a partition is dropped.
+ */
+case class DropPartitionPreEvent(
+ database: String,
+ name: String,
+ partSpecs: Seq[TablePartitionSpec])
+ extends PartitionEvent
+
+/**
+ * Event fired after a partition has been dropped.
+ */
+case class DropPartitionEvent(
+ database: String,
+ name: String,
+ partSpecs: Seq[TablePartitionSpec])
+ extends PartitionEvent
+
+/**
+ * Event fired before a function is renamed.
Review comment:
`function` -> `partition`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #27030:
[SPARK-30244][SQL] Emit pre/post events for "Partition" methods in
ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#discussion_r370469563
##########
File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/catalog/ExternalCatalogEventSuite.scala
##########
@@ -209,4 +209,67 @@ class ExternalCatalogEventSuite extends SparkFunSuite {
catalog.dropFunction("db5", "fn4")
checkEvents(DropFunctionPreEvent("db5", "fn4") :: DropFunctionEvent("db5", "fn4") :: Nil)
}
+
+ testWithCatalog("partition") { (catalog, checkEvents) =>
+ val path1 = Files.createTempDirectory("db_")
+ val path2 = Files.createTempDirectory(path1, "tbl_")
+ val uri1 = preparePath(path1)
+ val uri2 = preparePath(path2)
+
+ // CREATE
+ val dbDefinition = createDbDefinition(uri1)
+
+ val storage = CatalogStorageFormat.empty.copy(
+ locationUri = Option(uri2))
+
+ val tableDefinition = CatalogTable(
+ identifier = TableIdentifier("tbl1", Some("db5")),
+ tableType = CatalogTableType.MANAGED,
+ storage = storage,
+ schema = new StructType().add("id", "long"))
+
+ val partition = CatalogTablePartition(spec = Map("key1" -> "1", "key2" -> "2"),
+ storage = CatalogStorageFormat.empty)
+
+ catalog.createDatabase(dbDefinition, ignoreIfExists = false)
+ checkEvents(CreateDatabasePreEvent("db5") :: CreateDatabaseEvent("db5") :: Nil)
+
+ catalog.createTable(tableDefinition, ignoreIfExists = false)
+ checkEvents(CreateTablePreEvent("db5", "tbl1") :: CreateTableEvent("db5", "tbl1") :: Nil)
+
+ catalog.createPartitions("db5", "tbl1", Seq(partition), ignoreIfExists = false)
+ checkEvents(CreatePartitionPreEvent("db5", "tbl1", Seq(partition)) ::
+ CreatePartitionEvent("db5", "tbl1", Seq(partition)) :: Nil)
+
+ catalog.createPartitions("db5", "tbl1", Seq(partition), ignoreIfExists = true)
+ checkEvents(CreatePartitionPreEvent("db5", "tbl1", Seq(partition)) ::
+ CreatePartitionEvent("db5", "tbl1", Seq(partition)) :: Nil)
Review comment:
Ur, is this correct? We already created the same `partition` at line 240 and we tried with `ignoreIfExists=true` here. I guess `CreatePartitionEvent` should not occur.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27030:
[SPARK-30244][SQL] Emit pre/post events for "Partition" methods in
ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#issuecomment-573457361
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/21356/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27030:
[SPARK-30244][SQL] Emit pre/post events for "Partition" methods in
ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#issuecomment-577900553
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27030: [SPARK-30244][SQL] Emit
pre/post events for "Partition" methods in ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#issuecomment-573479100
**[Test build #116570 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/116570/testReport)** for PR 27030 at commit [`e1220e1`](https://github.com/apache/spark/commit/e1220e105bc4304365ca3f232f1825b00bd6e623).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27030:
[SPARK-30244][SQL] Emit pre/post events for "Partition" methods in
ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#issuecomment-573865400
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27030: [SPARK-30244][SQL] Emit
pre/post events for "Partition" methods in ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#issuecomment-572915998
**[Test build #116464 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/116464/testReport)** for PR 27030 at commit [`3cb79fe`](https://github.com/apache/spark/commit/3cb79fe63a14ecba62bfc9b449ecf9b4a28b6e10).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27030:
[SPARK-30244][SQL] Emit pre/post events for "Partition" methods in
ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#issuecomment-578432911
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27030:
[SPARK-30244][SQL] Emit pre/post events for "Partition" methods in
ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#issuecomment-572791525
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27030: [SPARK-30244][SQL]
Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#issuecomment-572841014
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/116418/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27030: [SPARK-30244][SQL]
Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#issuecomment-573479249
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/116570/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27030:
[SPARK-30244][SQL] Emit pre/post events for "Partition" methods in
ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#issuecomment-572775083
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27030:
[SPARK-30244][SQL] Emit pre/post events for "Partition" methods in
ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#issuecomment-573137721
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/116492/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #27030: [SPARK-30244][SQL]
Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#issuecomment-574413856
Retest this please.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27030: [SPARK-30244][SQL] Emit
pre/post events for "Partition" methods in ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#issuecomment-572850460
**[Test build #116419 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/116419/testReport)** for PR 27030 at commit [`0c5edc2`](https://github.com/apache/spark/commit/0c5edc2314ef9d7c23d759735f44610f90655312).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27030: [SPARK-30244][SQL]
Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#issuecomment-578432913
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/117401/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #27030:
[SPARK-30244][SQL] Emit pre/post events for "Partition" methods in
ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#discussion_r366618162
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/events.scala
##########
@@ -202,3 +203,83 @@ case class RenameFunctionEvent(
name: String,
newName: String)
extends FunctionEvent
+
+/**
+ * Event fired when a partition is created, dropped, altered or renamed.
+ */
+trait PartitionEvent extends TableEvent
+
+/**
+ * Event fired before a partition is created.
+ */
+case class CreatePartitionPreEvent(
+ database: String,
+ name: String,
+ parts: Seq[CatalogTablePartition])
+ extends PartitionEvent
+
+/**
+ * Event fired after a partition has been created.
+ */
+case class CreatePartitionEvent(
+ database: String,
+ name: String,
+ parts: Seq[CatalogTablePartition])
+ extends PartitionEvent
+
+/**
+ * Event fired before a partition is dropped.
+ */
+case class DropPartitionPreEvent(
+ database: String,
+ name: String,
+ partSpecs: Seq[TablePartitionSpec])
+ extends PartitionEvent
+
+/**
+ * Event fired after a partition has been dropped.
+ */
+case class DropPartitionEvent(
+ database: String,
+ name: String,
+ partSpecs: Seq[TablePartitionSpec])
+ extends PartitionEvent
+
+/**
+ * Event fired before a function is renamed.
+ */
+case class RenamePartitionPreEvent(
+ database: String,
+ name: String,
+ specs: Seq[TablePartitionSpec],
+ newSpecs: Seq[TablePartitionSpec])
+ extends PartitionEvent
+
+/**
+ * Event fired after a function has been renamed.
Review comment:
`function` -> `partition`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27030: [SPARK-30244][SQL]
Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#issuecomment-573865410
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/21440/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27030: [SPARK-30244][SQL] Emit
pre/post events for "Partition" methods in ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#issuecomment-573936933
**[Test build #116661 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/116661/testReport)** for PR 27030 at commit [`43aeb9b`](https://github.com/apache/spark/commit/43aeb9ba12c5dabcaddec3e84a2428ad2ebb09b6).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27030:
[SPARK-30244][SQL] Emit pre/post events for "Partition" methods in
ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#issuecomment-590736734
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118893/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27030: [SPARK-30244][SQL]
Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#issuecomment-572692536
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/21199/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org