You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/08/23 06:26:37 UTC
[GitHub] [spark] HyukjinKwon opened a new pull request #33807: [WIP][SPARK-36559][SQL][PYTHON] Move distributed-sequence index implementation to SQL plan to leverage optimization
HyukjinKwon opened a new pull request #33807:
URL: https://github.com/apache/spark/pull/33807
### What changes were proposed in this pull request?
This PR proposes to move distributed-sequence index implementation to SQL plan to leverage optimizations such as column pruning.
TBD
**Before:**
TBD
**After:**
TBD
### Why are the changes needed?
To leverage optimization of SQL engine and avoid unnecessary shuffle to create default index.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Unittests were added. Also, this PR will test all unittests in pandas API on Spark after switching the default index implementation to `distributed-sequence`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33807: [WIP][SPARK-36559][SQL][PYTHON] Create plans dedicated to distributed-sequence index for optimization
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33807:
URL: https://github.com/apache/spark/pull/33807#issuecomment-903488851
**[Test build #142690 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142690/testReport)** for PR 33807 at commit [`296a6ae`](https://github.com/apache/spark/commit/296a6ae314b5e9878b9e5d6a7ff33c0dc736a10d).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33807: [WIP][SPARK-36559][SQL][PYTHON] Create plans dedicated to distributed-sequence index for optimization
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33807:
URL: https://github.com/apache/spark/pull/33807#issuecomment-904318468
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33807: [SPARK-36559][SQL][PYTHON] Create plans dedicated to distributed-sequence index for optimization
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33807:
URL: https://github.com/apache/spark/pull/33807#issuecomment-904707529
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142723/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33807: [WIP][SPARK-36559][SQL][PYTHON] Create plans dedicated to distributed-sequence index for optimization
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33807:
URL: https://github.com/apache/spark/pull/33807#issuecomment-903784851
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142692/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33807: [WIP][SPARK-36559][SQL][PYTHON] Create plans dedicated to distributed-sequence index for optimization
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33807:
URL: https://github.com/apache/spark/pull/33807#issuecomment-904293532
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47213/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33807: [WIP][SPARK-36559][SQL][PYTHON] Create plans dedicated to distributed-sequence index for optimization
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33807:
URL: https://github.com/apache/spark/pull/33807#issuecomment-904447684
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142714/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33807: [SPARK-36559][SQL][PYTHON] Create plans dedicated to distributed-sequence index for optimization
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33807:
URL: https://github.com/apache/spark/pull/33807#issuecomment-904521111
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47223/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33807: [WIP][SPARK-36559][SQL][PYTHON] Create plans dedicated to distributed-sequence index for optimization
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33807:
URL: https://github.com/apache/spark/pull/33807#issuecomment-904444485
**[Test build #142714 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142714/testReport)** for PR 33807 at commit [`d28b29c`](https://github.com/apache/spark/commit/d28b29ccacc5403720213dc31b81da2bac823213).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `case class AttachDistributedSequence(`
* `case class AttachDistributedSequenceExec(`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33807: [WIP][SPARK-36559][SQL][PYTHON] Create plans dedicated to distributed-sequence index for optimization
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33807:
URL: https://github.com/apache/spark/pull/33807#issuecomment-903781249
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47199/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33807: [WIP][SPARK-36559][SQL][PYTHON] Create plans dedicated to distributed-sequence index for optimization
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33807:
URL: https://github.com/apache/spark/pull/33807#issuecomment-903628638
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47194/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33807: [WIP][SPARK-36559][SQL][PYTHON] Create plans dedicated to distributed-sequence index for optimization
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33807:
URL: https://github.com/apache/spark/pull/33807#issuecomment-903690527
Kubernetes integration test unable to build dist.
exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47198/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33807: [WIP][SPARK-36559][SQL][PYTHON] Create plans dedicated to distributed-sequence index for optimization
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33807:
URL: https://github.com/apache/spark/pull/33807#issuecomment-903837509
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47200/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon removed a comment on pull request #33807: [WIP][SPARK-36559][SQL][PYTHON] Create plans dedicated to distributed-sequence index for optimization
Posted by GitBox <gi...@apache.org>.
HyukjinKwon removed a comment on pull request #33807:
URL: https://github.com/apache/spark/pull/33807#issuecomment-904288426
retest this please
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33807: [WIP][SPARK-36559][SQL][PYTHON] Create plans dedicated to distributed-sequence index for optimization
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33807:
URL: https://github.com/apache/spark/pull/33807#issuecomment-903783007
**[Test build #142692 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142692/testReport)** for PR 33807 at commit [`68e0a23`](https://github.com/apache/spark/commit/68e0a23dc3dfb154b19208ef6c93e1930b905d3f).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `case class AttachDistributedSequence(`
* `case class AttachDistributedSequenceExec(`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33807: [WIP][SPARK-36559][SQL][PYTHON] Create plans dedicated to distributed-sequence index for optimization
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33807:
URL: https://github.com/apache/spark/pull/33807#issuecomment-903488851
**[Test build #142690 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142690/testReport)** for PR 33807 at commit [`296a6ae`](https://github.com/apache/spark/commit/296a6ae314b5e9878b9e5d6a7ff33c0dc736a10d).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #33807: [WIP][SPARK-36559][SQL][PYTHON] Create plans dedicated to distributed-sequence index for optimization
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #33807:
URL: https://github.com/apache/spark/pull/33807#issuecomment-904288426
retest this please
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33807: [WIP][SPARK-36559][SQL][PYTHON] Create plans dedicated to distributed-sequence index for optimization
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33807:
URL: https://github.com/apache/spark/pull/33807#issuecomment-903722178
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47199/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33807: [WIP][SPARK-36559][SQL][PYTHON] Create plans dedicated to distributed-sequence index for optimization
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33807:
URL: https://github.com/apache/spark/pull/33807#issuecomment-903670873
**[Test build #142690 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142690/testReport)** for PR 33807 at commit [`296a6ae`](https://github.com/apache/spark/commit/296a6ae314b5e9878b9e5d6a7ff33c0dc736a10d).
* This patch **fails PySpark unit tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `case class AttachDistributedSequence(`
* `case class AttachDistributedSequenceExec(`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33807: [WIP][SPARK-36559][SQL][PYTHON] Create plans dedicated to distributed-sequence index for optimization
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33807:
URL: https://github.com/apache/spark/pull/33807#issuecomment-904281002
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142713/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33807: [WIP][SPARK-36559][SQL][PYTHON] Create plans dedicated to distributed-sequence index for optimization
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33807:
URL: https://github.com/apache/spark/pull/33807#issuecomment-903965871
**[Test build #142698 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142698/testReport)** for PR 33807 at commit [`14021b4`](https://github.com/apache/spark/commit/14021b48598af2cc5155bed553eb64af8ac49bd8).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33807: [WIP][SPARK-36559][SQL][PYTHON] Create plans dedicated to distributed-sequence index for optimization
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33807:
URL: https://github.com/apache/spark/pull/33807#issuecomment-903784851
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142692/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33807: [WIP][SPARK-36559][SQL][PYTHON] Create plans dedicated to distributed-sequence index for optimization
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33807:
URL: https://github.com/apache/spark/pull/33807#issuecomment-904298819
**[Test build #142714 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142714/testReport)** for PR 33807 at commit [`d28b29c`](https://github.com/apache/spark/commit/d28b29ccacc5403720213dc31b81da2bac823213).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #33807: [SPARK-36559][SQL][PYTHON] Create plans dedicated to distributed-sequence index for optimization
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #33807:
URL: https://github.com/apache/spark/pull/33807#discussion_r694801960
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/python/AttachDistributedSequenceExec.scala
##########
@@ -0,0 +1,62 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.python
+
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.plans.physical._
+import org.apache.spark.sql.execution.{SparkPlan, UnaryExecNode}
+
+/**
+ * A physical plan that adds a new long column with `sequenceAttr` that
+ * increases one by one. This is for 'distributed-sequence' default index
+ * in pandas API on Spark.
+ */
+case class AttachDistributedSequenceExec(
+ sequenceAttr: Attribute,
+ child: SparkPlan)
+ extends UnaryExecNode {
+
+ override def producedAttributes: AttributeSet = AttributeSet(sequenceAttr)
+
+ override val output: Seq[Attribute] = sequenceAttr +: child.output
+
+ override def outputPartitioning: Partitioning = child.outputPartitioning
+
+ override protected def doExecute(): RDD[InternalRow] = {
+ child.execute().map(_.copy())
+ .localCheckpoint() // to avoid execute multiple jobs. zipWithIndex launches a Spark job.
Review comment:
The shuffle will be reused. I think `localCheckpoint` is useful to save computation. e.g. `df.sort(...).withSequenceColumn`, if we don't do `localCheckpoint`, the shuffle is still done only once, but the local sort after shuffle will be done twice.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #33807: [SPARK-36559][SQL][PYTHON] Create plans dedicated to distributed-sequence index for optimization
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #33807:
URL: https://github.com/apache/spark/pull/33807#issuecomment-904497305
cc @ueshin and @cloud-fan can you take a look when you find some time?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #33807: [SPARK-36559][SQL][PYTHON] Create plans dedicated to distributed-sequence index for optimization
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #33807:
URL: https://github.com/apache/spark/pull/33807#discussion_r694796187
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/python/AttachDistributedSequenceExec.scala
##########
@@ -0,0 +1,62 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.python
+
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.plans.physical._
+import org.apache.spark.sql.execution.{SparkPlan, UnaryExecNode}
+
+/**
+ * A physical plan that adds a new long column with `sequenceAttr` that
+ * increases one by one. This is for 'distributed-sequence' default index
+ * in pandas API on Spark.
+ */
+case class AttachDistributedSequenceExec(
+ sequenceAttr: Attribute,
+ child: SparkPlan)
+ extends UnaryExecNode {
+
+ override def producedAttributes: AttributeSet = AttributeSet(sequenceAttr)
+
+ override val output: Seq[Attribute] = sequenceAttr +: child.output
+
+ override def outputPartitioning: Partitioning = child.outputPartitioning
+
+ override protected def doExecute(): RDD[InternalRow] = {
+ child.execute().map(_.copy())
Review comment:
Oh, i forgot to describe it. `localCheckpoint` caches (persists) the data, and it stores each row so it needs to copy. This is actually being done at Dataset.checkpoint: https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala#L679
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33807: [WIP][SPARK-36559][SQL][PYTHON] Create plans dedicated to distributed-sequence index for optimization
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33807:
URL: https://github.com/apache/spark/pull/33807#issuecomment-903689541
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142690/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33807: [SPARK-36559][SQL][PYTHON] Create plans dedicated to distributed-sequence index for optimization
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33807:
URL: https://github.com/apache/spark/pull/33807#issuecomment-904542597
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47223/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33807: [WIP][SPARK-36559][SQL][PYTHON] Create plans dedicated to distributed-sequence index for optimization
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33807:
URL: https://github.com/apache/spark/pull/33807#issuecomment-903628638
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47194/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33807: [WIP][SPARK-36559][SQL][PYTHON] Create plans dedicated to distributed-sequence index for optimization
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33807:
URL: https://github.com/apache/spark/pull/33807#issuecomment-904311236
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47213/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33807: [WIP][SPARK-36559][SQL][PYTHON] Create plans dedicated to distributed-sequence index for optimization
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33807:
URL: https://github.com/apache/spark/pull/33807#issuecomment-903885487
**[Test build #142696 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142696/testReport)** for PR 33807 at commit [`61b2ff6`](https://github.com/apache/spark/commit/61b2ff67a2b7c6ab73e293e9b7ce5e8b78578986).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33807: [WIP][SPARK-36559][SQL][PYTHON] Create plans dedicated to distributed-sequence index for optimization
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33807:
URL: https://github.com/apache/spark/pull/33807#issuecomment-903690905
**[Test build #142697 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142697/testReport)** for PR 33807 at commit [`1f3c962`](https://github.com/apache/spark/commit/1f3c962c64d4554d731b503b8a55193fdeeb3fed).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33807: [WIP][SPARK-36559][SQL][PYTHON] Create plans dedicated to distributed-sequence index for optimization
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33807:
URL: https://github.com/apache/spark/pull/33807#issuecomment-903563911
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47192/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #33807: [SPARK-36559][SQL][PYTHON] Create plans dedicated to distributed-sequence index for optimization
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #33807:
URL: https://github.com/apache/spark/pull/33807#issuecomment-905087639
Yeah, I think it won't happen. Just did a quick double check.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33807: [WIP][SPARK-36559][SQL][PYTHON] Create plans dedicated to distributed-sequence index for optimization
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33807:
URL: https://github.com/apache/spark/pull/33807#issuecomment-903690560
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47198/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33807: [WIP][SPARK-36559][SQL][PYTHON] Create plans dedicated to distributed-sequence index for optimization
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33807:
URL: https://github.com/apache/spark/pull/33807#issuecomment-903934870
**[Test build #142697 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142697/testReport)** for PR 33807 at commit [`1f3c962`](https://github.com/apache/spark/commit/1f3c962c64d4554d731b503b8a55193fdeeb3fed).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33807: [WIP][SPARK-36559][SQL][PYTHON] Create plans dedicated to distributed-sequence index for optimization
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33807:
URL: https://github.com/apache/spark/pull/33807#issuecomment-904318466
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33807: [WIP][SPARK-36559][SQL][PYTHON] Create plans dedicated to distributed-sequence index for optimization
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33807:
URL: https://github.com/apache/spark/pull/33807#issuecomment-903628603
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47194/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33807: [SPARK-36559][SQL][PYTHON] Create plans dedicated to distributed-sequence index for optimization
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33807:
URL: https://github.com/apache/spark/pull/33807#issuecomment-904493394
**[Test build #142723 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142723/testReport)** for PR 33807 at commit [`5d74c35`](https://github.com/apache/spark/commit/5d74c359571b5afeef67f7f19ac612631aaf5144).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #33807: [SPARK-36559][SQL][PYTHON] Create plans dedicated to distributed-sequence index for optimization
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #33807:
URL: https://github.com/apache/spark/pull/33807#discussion_r694797537
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/python/AttachDistributedSequenceExec.scala
##########
@@ -0,0 +1,62 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.python
+
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.plans.physical._
+import org.apache.spark.sql.execution.{SparkPlan, UnaryExecNode}
+
+/**
+ * A physical plan that adds a new long column with `sequenceAttr` that
+ * increases one by one. This is for 'distributed-sequence' default index
+ * in pandas API on Spark.
+ */
+case class AttachDistributedSequenceExec(
+ sequenceAttr: Attribute,
+ child: SparkPlan)
+ extends UnaryExecNode {
+
+ override def producedAttributes: AttributeSet = AttributeSet(sequenceAttr)
+
+ override val output: Seq[Attribute] = sequenceAttr +: child.output
+
+ override def outputPartitioning: Partitioning = child.outputPartitioning
+
+ override protected def doExecute(): RDD[InternalRow] = {
+ child.execute().map(_.copy())
+ .localCheckpoint() // to avoid execute multiple jobs. zipWithIndex launches a Spark job.
Review comment:
I am still not sure if we need to `localCheckPoint` in the middle here ... but let me keep it as is for now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #33807: [SPARK-36559][SQL][PYTHON] Create plans dedicated to distributed-sequence index for optimization
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #33807:
URL: https://github.com/apache/spark/pull/33807#discussion_r694669270
##########
File path: python/pyspark/pandas/config.py
##########
@@ -175,7 +175,7 @@ def validate(self, v: Any) -> None:
Option(
key="compute.default_index_type",
doc=("This sets the default index type: sequence, distributed and distributed-sequence."),
- default="sequence",
+ default="distributed-sequence",
Review comment:
```suggestion
default="sequence",
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33807: [WIP][SPARK-36559][SQL][PYTHON] Create plans dedicated to distributed-sequence index for optimization
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33807:
URL: https://github.com/apache/spark/pull/33807#issuecomment-904278724
**[Test build #142713 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142713/testReport)** for PR 33807 at commit [`d28b29c`](https://github.com/apache/spark/commit/d28b29ccacc5403720213dc31b81da2bac823213).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33807: [WIP][SPARK-36559][SQL][PYTHON] Create plans dedicated to distributed-sequence index for optimization
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33807:
URL: https://github.com/apache/spark/pull/33807#issuecomment-904281002
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142713/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #33807: [SPARK-36559][SQL][PYTHON] Create plans dedicated to distributed-sequence index for optimization
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #33807:
URL: https://github.com/apache/spark/pull/33807#discussion_r694785937
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/python/AttachDistributedSequenceExec.scala
##########
@@ -0,0 +1,62 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.python
+
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.plans.physical._
+import org.apache.spark.sql.execution.{SparkPlan, UnaryExecNode}
+
+/**
+ * A physical plan that adds a new long column with `sequenceAttr` that
+ * increases one by one. This is for 'distributed-sequence' default index
+ * in pandas API on Spark.
+ */
+case class AttachDistributedSequenceExec(
+ sequenceAttr: Attribute,
+ child: SparkPlan)
+ extends UnaryExecNode {
+
+ override def producedAttributes: AttributeSet = AttributeSet(sequenceAttr)
+
+ override val output: Seq[Attribute] = sequenceAttr +: child.output
+
+ override def outputPartitioning: Partitioning = child.outputPartitioning
+
+ override protected def doExecute(): RDD[InternalRow] = {
+ child.execute().map(_.copy())
Review comment:
why do we need to copy the unsafe rows before calling `localCheckpoint`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33807: [WIP][SPARK-36559][SQL][PYTHON] Create plans dedicated to distributed-sequence index for optimization
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33807:
URL: https://github.com/apache/spark/pull/33807#issuecomment-903781249
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47199/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33807: [WIP][SPARK-36559][SQL][PYTHON] Create plans dedicated to distributed-sequence index for optimization
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33807:
URL: https://github.com/apache/spark/pull/33807#issuecomment-903978885
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142698/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33807: [WIP][SPARK-36559][SQL][PYTHON] Create plans dedicated to distributed-sequence index for optimization
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33807:
URL: https://github.com/apache/spark/pull/33807#issuecomment-903937853
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142697/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33807: [WIP][SPARK-36559][SQL][PYTHON] Create plans dedicated to distributed-sequence index for optimization
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33807:
URL: https://github.com/apache/spark/pull/33807#issuecomment-903889487
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142696/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33807: [WIP][SPARK-36559][SQL][PYTHON] Create plans dedicated to distributed-sequence index for optimization
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33807:
URL: https://github.com/apache/spark/pull/33807#issuecomment-903572806
**[Test build #142692 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142692/testReport)** for PR 33807 at commit [`68e0a23`](https://github.com/apache/spark/commit/68e0a23dc3dfb154b19208ef6c93e1930b905d3f).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33807: [WIP][SPARK-36559][SQL][PYTHON] Create plans dedicated to distributed-sequence index for optimization
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33807:
URL: https://github.com/apache/spark/pull/33807#issuecomment-903572806
**[Test build #142692 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142692/testReport)** for PR 33807 at commit [`68e0a23`](https://github.com/apache/spark/commit/68e0a23dc3dfb154b19208ef6c93e1930b905d3f).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33807: [SPARK-36559][SQL][PYTHON] Create plans dedicated to distributed-sequence index for optimization
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33807:
URL: https://github.com/apache/spark/pull/33807#issuecomment-904542629
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47223/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33807: [WIP][SPARK-36559][SQL][PYTHON] Create plans dedicated to distributed-sequence index for optimization
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33807:
URL: https://github.com/apache/spark/pull/33807#issuecomment-904447684
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142714/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33807: [WIP][SPARK-36559][SQL][PYTHON] Create plans dedicated to distributed-sequence index for optimization
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33807:
URL: https://github.com/apache/spark/pull/33807#issuecomment-903731384
**[Test build #142698 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142698/testReport)** for PR 33807 at commit [`14021b4`](https://github.com/apache/spark/commit/14021b48598af2cc5155bed553eb64af8ac49bd8).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33807: [SPARK-36559][SQL][PYTHON] Create plans dedicated to distributed-sequence index for optimization
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33807:
URL: https://github.com/apache/spark/pull/33807#issuecomment-904707529
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142723/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33807: [WIP][SPARK-36559][SQL][PYTHON] Create plans dedicated to distributed-sequence index for optimization
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33807:
URL: https://github.com/apache/spark/pull/33807#issuecomment-903731384
**[Test build #142698 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142698/testReport)** for PR 33807 at commit [`14021b4`](https://github.com/apache/spark/commit/14021b48598af2cc5155bed553eb64af8ac49bd8).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33807: [WIP][SPARK-36559][SQL][PYTHON] Create plans dedicated to distributed-sequence index for optimization
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33807:
URL: https://github.com/apache/spark/pull/33807#issuecomment-904280982
**[Test build #142713 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142713/testReport)** for PR 33807 at commit [`d28b29c`](https://github.com/apache/spark/commit/d28b29ccacc5403720213dc31b81da2bac823213).
* This patch **fails to build**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `case class AttachDistributedSequence(`
* `case class AttachDistributedSequenceExec(`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33807: [WIP][SPARK-36559][SQL][PYTHON] Create plans dedicated to distributed-sequence index for optimization
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33807:
URL: https://github.com/apache/spark/pull/33807#issuecomment-903826690
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47200/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #33807: [SPARK-36559][SQL][PYTHON] Create plans dedicated to distributed-sequence index for optimization
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #33807:
URL: https://github.com/apache/spark/pull/33807#discussion_r694796187
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/python/AttachDistributedSequenceExec.scala
##########
@@ -0,0 +1,62 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.python
+
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.plans.physical._
+import org.apache.spark.sql.execution.{SparkPlan, UnaryExecNode}
+
+/**
+ * A physical plan that adds a new long column with `sequenceAttr` that
+ * increases one by one. This is for 'distributed-sequence' default index
+ * in pandas API on Spark.
+ */
+case class AttachDistributedSequenceExec(
+ sequenceAttr: Attribute,
+ child: SparkPlan)
+ extends UnaryExecNode {
+
+ override def producedAttributes: AttributeSet = AttributeSet(sequenceAttr)
+
+ override val output: Seq[Attribute] = sequenceAttr +: child.output
+
+ override def outputPartitioning: Partitioning = child.outputPartitioning
+
+ override protected def doExecute(): RDD[InternalRow] = {
+ child.execute().map(_.copy())
Review comment:
Oh, i forgot to describe it. `localCheckpoint` caches (persists) the data, and it stores the rows so it needs to copy. This is actually being done at Dataset.checkpoint: https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala#L679
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33807: [WIP][SPARK-36559][SQL][PYTHON] Create plans dedicated to distributed-sequence index for optimization
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33807:
URL: https://github.com/apache/spark/pull/33807#issuecomment-904298819
**[Test build #142714 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142714/testReport)** for PR 33807 at commit [`d28b29c`](https://github.com/apache/spark/commit/d28b29ccacc5403720213dc31b81da2bac823213).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #33807: [SPARK-36559][SQL][PYTHON] Create plans dedicated to distributed-sequence index for optimization
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #33807:
URL: https://github.com/apache/spark/pull/33807#discussion_r694796187
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/python/AttachDistributedSequenceExec.scala
##########
@@ -0,0 +1,62 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.python
+
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.plans.physical._
+import org.apache.spark.sql.execution.{SparkPlan, UnaryExecNode}
+
+/**
+ * A physical plan that adds a new long column with `sequenceAttr` that
+ * increases one by one. This is for 'distributed-sequence' default index
+ * in pandas API on Spark.
+ */
+case class AttachDistributedSequenceExec(
+ sequenceAttr: Attribute,
+ child: SparkPlan)
+ extends UnaryExecNode {
+
+ override def producedAttributes: AttributeSet = AttributeSet(sequenceAttr)
+
+ override val output: Seq[Attribute] = sequenceAttr +: child.output
+
+ override def outputPartitioning: Partitioning = child.outputPartitioning
+
+ override protected def doExecute(): RDD[InternalRow] = {
+ child.execute().map(_.copy())
Review comment:
Oh, i forgot to describe it. `localCheckpoint` caches the data, and it stores each row so it needs to copy. This is actually being done at Dataset.checkpoint: https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala#L679
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33807: [SPARK-36559][SQL][PYTHON] Create plans dedicated to distributed-sequence index for optimization
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33807:
URL: https://github.com/apache/spark/pull/33807#issuecomment-904542629
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47223/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #33807: [SPARK-36559][SQL][PYTHON] Create plans dedicated to distributed-sequence index for optimization
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #33807:
URL: https://github.com/apache/spark/pull/33807#issuecomment-905087944
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33807: [WIP][SPARK-36559][SQL][PYTHON] Create plans dedicated to distributed-sequence index for optimization
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33807:
URL: https://github.com/apache/spark/pull/33807#issuecomment-903776029
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47200/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33807: [WIP][SPARK-36559][SQL][PYTHON] Create plans dedicated to distributed-sequence index for optimization
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33807:
URL: https://github.com/apache/spark/pull/33807#issuecomment-903889487
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142696/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #33807: [SPARK-36559][SQL][PYTHON] Create plans dedicated to distributed-sequence index for optimization
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #33807:
URL: https://github.com/apache/spark/pull/33807#discussion_r694797890
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/python/AttachDistributedSequenceExec.scala
##########
@@ -0,0 +1,62 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.python
+
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.plans.physical._
+import org.apache.spark.sql.execution.{SparkPlan, UnaryExecNode}
+
+/**
+ * A physical plan that adds a new long column with `sequenceAttr` that
+ * increases one by one. This is for 'distributed-sequence' default index
+ * in pandas API on Spark.
+ */
+case class AttachDistributedSequenceExec(
+ sequenceAttr: Attribute,
+ child: SparkPlan)
+ extends UnaryExecNode {
+
+ override def producedAttributes: AttributeSet = AttributeSet(sequenceAttr)
+
+ override val output: Seq[Attribute] = sequenceAttr +: child.output
+
+ override def outputPartitioning: Partitioning = child.outputPartitioning
+
+ override protected def doExecute(): RDD[InternalRow] = {
+ child.execute().map(_.copy())
+ .localCheckpoint() // to avoid execute multiple jobs. zipWithIndex launches a Spark job.
Review comment:
e.g.) if the child RDD has a shuffle, the shuffle will be triggered twice, and this checkpoint is to avoid that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33807: [WIP][SPARK-36559][SQL][PYTHON] Create plans dedicated to distributed-sequence index for optimization
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33807:
URL: https://github.com/apache/spark/pull/33807#issuecomment-903603293
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47194/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #33807: [SPARK-36559][SQL][PYTHON] Create plans dedicated to distributed-sequence index for optimization
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #33807:
URL: https://github.com/apache/spark/pull/33807#discussion_r694799535
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/python/AttachDistributedSequenceExec.scala
##########
@@ -0,0 +1,62 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.python
+
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.plans.physical._
+import org.apache.spark.sql.execution.{SparkPlan, UnaryExecNode}
+
+/**
+ * A physical plan that adds a new long column with `sequenceAttr` that
+ * increases one by one. This is for 'distributed-sequence' default index
+ * in pandas API on Spark.
+ */
+case class AttachDistributedSequenceExec(
Review comment:
We could think about implementing this with an expression (like Python UDF or Window) .. but just decided to do this with plans to avoid making it too much complicated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon closed pull request #33807: [SPARK-36559][SQL][PYTHON] Create plans dedicated to distributed-sequence index for optimization
Posted by GitBox <gi...@apache.org>.
HyukjinKwon closed pull request #33807:
URL: https://github.com/apache/spark/pull/33807
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33807: [WIP][SPARK-36559][SQL][PYTHON] Create plans dedicated to distributed-sequence index for optimization
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33807:
URL: https://github.com/apache/spark/pull/33807#issuecomment-903690905
**[Test build #142697 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142697/testReport)** for PR 33807 at commit [`1f3c962`](https://github.com/apache/spark/commit/1f3c962c64d4554d731b503b8a55193fdeeb3fed).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33807: [SPARK-36559][SQL][PYTHON] Create plans dedicated to distributed-sequence index for optimization
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33807:
URL: https://github.com/apache/spark/pull/33807#issuecomment-904705712
**[Test build #142723 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142723/testReport)** for PR 33807 at commit [`5d74c35`](https://github.com/apache/spark/commit/5d74c359571b5afeef67f7f19ac612631aaf5144).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33807: [WIP][SPARK-36559][SQL][PYTHON] Create plans dedicated to distributed-sequence index for optimization
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33807:
URL: https://github.com/apache/spark/pull/33807#issuecomment-903658275
**[Test build #142696 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142696/testReport)** for PR 33807 at commit [`61b2ff6`](https://github.com/apache/spark/commit/61b2ff67a2b7c6ab73e293e9b7ce5e8b78578986).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33807: [WIP][SPARK-36559][SQL][PYTHON] Create plans dedicated to distributed-sequence index for optimization
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33807:
URL: https://github.com/apache/spark/pull/33807#issuecomment-904313492
Kubernetes integration test unable to build dist.
exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47214/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33807: [WIP][SPARK-36559][SQL][PYTHON] Create plans dedicated to distributed-sequence index for optimization
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33807:
URL: https://github.com/apache/spark/pull/33807#issuecomment-903548780
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47192/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33807: [WIP][SPARK-36559][SQL][PYTHON] Create plans dedicated to distributed-sequence index for optimization
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33807:
URL: https://github.com/apache/spark/pull/33807#issuecomment-904278724
**[Test build #142713 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142713/testReport)** for PR 33807 at commit [`d28b29c`](https://github.com/apache/spark/commit/d28b29ccacc5403720213dc31b81da2bac823213).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33807: [WIP][SPARK-36559][SQL][PYTHON] Create plans dedicated to distributed-sequence index for optimization
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33807:
URL: https://github.com/apache/spark/pull/33807#issuecomment-903563911
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47192/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33807: [WIP][SPARK-36559][SQL][PYTHON] Create plans dedicated to distributed-sequence index for optimization
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33807:
URL: https://github.com/apache/spark/pull/33807#issuecomment-903658275
**[Test build #142696 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142696/testReport)** for PR 33807 at commit [`61b2ff6`](https://github.com/apache/spark/commit/61b2ff67a2b7c6ab73e293e9b7ce5e8b78578986).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33807: [WIP][SPARK-36559][SQL][PYTHON] Create plans dedicated to distributed-sequence index for optimization
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33807:
URL: https://github.com/apache/spark/pull/33807#issuecomment-903937853
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142697/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33807: [WIP][SPARK-36559][SQL][PYTHON] Create plans dedicated to distributed-sequence index for optimization
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33807:
URL: https://github.com/apache/spark/pull/33807#issuecomment-903690560
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47198/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33807: [SPARK-36559][SQL][PYTHON] Create plans dedicated to distributed-sequence index for optimization
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33807:
URL: https://github.com/apache/spark/pull/33807#issuecomment-904493394
**[Test build #142723 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142723/testReport)** for PR 33807 at commit [`5d74c35`](https://github.com/apache/spark/commit/5d74c359571b5afeef67f7f19ac612631aaf5144).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33807: [WIP][SPARK-36559][SQL][PYTHON] Create plans dedicated to distributed-sequence index for optimization
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33807:
URL: https://github.com/apache/spark/pull/33807#issuecomment-903754350
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47199/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33807: [WIP][SPARK-36559][SQL][PYTHON] Create plans dedicated to distributed-sequence index for optimization
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33807:
URL: https://github.com/apache/spark/pull/33807#issuecomment-903513061
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47192/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33807: [WIP][SPARK-36559][SQL][PYTHON] Create plans dedicated to distributed-sequence index for optimization
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33807:
URL: https://github.com/apache/spark/pull/33807#issuecomment-903978885
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142698/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33807: [WIP][SPARK-36559][SQL][PYTHON] Create plans dedicated to distributed-sequence index for optimization
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33807:
URL: https://github.com/apache/spark/pull/33807#issuecomment-903689541
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142690/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33807: [WIP][SPARK-36559][SQL][PYTHON] Create plans dedicated to distributed-sequence index for optimization
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33807:
URL: https://github.com/apache/spark/pull/33807#issuecomment-903837509
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47200/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org