You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "fangchenli (via GitHub)" <gi...@apache.org> on 2026/01/27 22:09:22 UTC

[PR] [SPARK-55228][SQL] Implement Dataset.zipWithIndex in Scala API [spark]

fangchenli opened a new pull request, #54014:
URL: https://github.com/apache/spark/pull/54014

   <!--
   Thanks for sending a pull request!  Here are some tips for you:
     1. If this is your first time, please read our contributor guidelines: https://spark.apache.org/contributing.html
     2. Ensure you have added or run the appropriate tests for your PR: https://spark.apache.org/developer-tools.html
     3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][SPARK-XXXX] Your PR title ...'.
     4. Be sure to keep the PR description updated to reflect all changes.
     5. Please write your PR title to summarize what this PR proposes.
     6. If possible, provide a concise example to reproduce the issue for a faster review.
     7. If you want to add a new configuration, please read the guideline first for naming configurations in
        'core/src/main/scala/org/apache/spark/internal/config/ConfigEntry.scala'.
     8. If you want to add or modify an error type or message, please read the guideline first in
        'common/utils/src/main/resources/error/README.md'.
   -->
   
   ### What changes were proposed in this pull request?
   <!--
   Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue. 
   If possible, please consider writing useful notes for better and faster reviews in your PR. See the examples below.
     1. If you refactor some codes with changing classes, showing the class hierarchy will help reviewers.
     2. If you fix some SQL features, you can provide some references of other DBMSes.
     3. If there is design documentation, please add the link.
     4. If there is a discussion in the mailing list, please add the link.
   -->
   Implement Dataset.zipWithIndex in Scala API
   
   ### Why are the changes needed?
   <!--
   Please clarify why the changes are needed. For instance,
     1. If you propose a new API, clarify the use case for a new API.
     2. If you fix a bug, you can clarify why it is a bug.
   -->
   Align Dataset and RDD api
   
   
   ### Does this PR introduce _any_ user-facing change?
   <!--
   Note that it means *any* user-facing change including all aspects such as new features, bug fixes, or other behavior changes. Documentation-only updates are not considered user-facing changes.
   
   If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible.
   If possible, please also clarify if this is a user-facing change compared to the released Spark versions or within the unreleased branches such as master.
   If no, write 'No'.
   -->
   Yes, this PR added Dataset.zipWithIndex() and Dataset.zipWithIndex(indexColName: String)
   
   
   ### How was this patch tested?
   <!--
   If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
   If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
   If tests were not added, please describe why they were not added and/or why it was difficult to add.
   If benchmark tests were added, please run the benchmarks in GitHub Actions for the consistent environment, and the instructions could accord to: https://spark.apache.org/developer-tools.html#github-workflow-benchmarks.
   -->
   Unittests added
   
   ### Was this patch authored or co-authored using generative AI tooling?
   <!--
   If generative AI tooling has been used in the process of authoring this patch, please include the
   phrase: 'Generated-by: ' followed by the name of the tool and its version.
   If no, write 'No'.
   Please refer to the [ASF Generative Tooling Guidance](https://www.apache.org/legal/generative-tooling.html) for details.
   -->
   Generated-by: Claude Opus 4.5
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-55228][SPARK-55230][SQL][CONNECT] Implement Dataset.zipWithIndex in Scala API [spark]

Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on code in PR #54014:
URL: https://github.com/apache/spark/pull/54014#discussion_r2757339301


##########
sql/core/src/main/scala/org/apache/spark/sql/classic/Dataset.scala:
##########
@@ -1198,6 +1198,11 @@ class Dataset[T] private[sql](
     Except(logicalPlan, other.logicalPlan, isAll = true)
   }
 
+  /** @inheritdoc */
+  def zipWithIndex(indexColName: String): DataFrame = {
+    select(col("*"), Column(DistributedSequenceID()).alias(indexColName))

Review Comment:
   it also works for me.  then we can go ahead with `select(*, distributed_sequence_id.alias(...) )`
   
   then @fangchenli please update the tests and fix the doc failure



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-55228][SPARK-55230][SQL][CONNECT] Implement Dataset.zipWithIndex in Scala API [spark]

Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #54014:
URL: https://github.com/apache/spark/pull/54014#discussion_r2754051637


##########
sql/core/src/main/scala/org/apache/spark/sql/classic/Dataset.scala:
##########
@@ -1198,6 +1198,11 @@ class Dataset[T] private[sql](
     Except(logicalPlan, other.logicalPlan, isAll = true)
   }
 
+  /** @inheritdoc */
+  def zipWithIndex(indexColName: String): DataFrame = {
+    select(col("*"), Column(DistributedSequenceID()).alias(indexColName))

Review Comment:
   that means triggering an eager analysis?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-55228][SQL] Implement Dataset.zipWithIndex in Scala API [spark]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #54014:
URL: https://github.com/apache/spark/pull/54014#issuecomment-3807774447

   ## JIRA Issue Information
   
   === Sub-task SPARK-55228 ===
   Summary: Implement  Dataset.zipWithIndex in Scala API
   Assignee: None
   Status: Open
   Affected: ["4.2.0"]
   
   ---
   *This comment was automatically generated by GitHub Actions*


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-55228][SPARK-55230][SQL] Implement Dataset.zipWithIndex in Scala API [spark]

Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on PR #54014:
URL: https://github.com/apache/spark/pull/54014#issuecomment-3808507211

   Add `SPARK-55230` since it also supports scala client


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-55228][SPARK-55230][SQL][CONNECT] Implement Dataset.zipWithIndex in Scala API [spark]

Posted by "fangchenli (via GitHub)" <gi...@apache.org>.
fangchenli commented on PR #54014:
URL: https://github.com/apache/spark/pull/54014#issuecomment-3849702719

   > @fangchenli the doc build is still failing, do you have try sync your spark fork?
   
   Yes, but it's still falling. And it doesn't seem related to my change. Let me investigate.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-55228][SPARK-55230][SQL][CONNECT] Implement Dataset.zipWithIndex in Scala API [spark]

Posted by "fangchenli (via GitHub)" <gi...@apache.org>.
fangchenli commented on code in PR #54014:
URL: https://github.com/apache/spark/pull/54014#discussion_r2752391815


##########
sql/core/src/main/scala/org/apache/spark/sql/classic/Dataset.scala:
##########
@@ -1198,6 +1198,11 @@ class Dataset[T] private[sql](
     Except(logicalPlan, other.logicalPlan, isAll = true)
   }
 
+  /** @inheritdoc */
+  def zipWithIndex(indexColName: String): DataFrame = {
+    select(col("*"), Column(DistributedSequenceID()).alias(indexColName))

Review Comment:
   > how does it fail for duplicated col names?
   
   The current implementation doesn't fail. If we want it to fail on a duplicate column name, should I add the check eagerly here? I don't see similar checks here in other methods. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-55228][SPARK-55230][SQL][CONNECT] Implement Dataset.zipWithIndex in Scala API [spark]

Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on code in PR #54014:
URL: https://github.com/apache/spark/pull/54014#discussion_r2744104706


##########
sql/connect/common/src/main/scala/org/apache/spark/sql/connect/Dataset.scala:
##########
@@ -703,6 +703,13 @@ class Dataset[T] private[sql] (
     }
   }
 
+  /** @inheritdoc */
+  def zipWithIndex(indexColName: String): DataFrame = {
+    // Note: Column existence check is handled server-side to avoid requiring
+    // a schema fetch which needs a server connection.

Review Comment:
   this check is removed?
   doesn't it cause behavior difference from spark classic?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-55228][SPARK-55230][SQL][CONNECT] Implement Dataset.zipWithIndex in Scala API [spark]

Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #54014:
URL: https://github.com/apache/spark/pull/54014#discussion_r2752362758


##########
sql/api/src/main/scala/org/apache/spark/sql/Dataset.scala:
##########
@@ -2010,6 +2010,33 @@ abstract class Dataset[T] extends Serializable {
    */
   def exceptAll(other: Dataset[T]): Dataset[T]
 
+  /**
+   * Returns a new [[Dataset]] by appending a column containing consecutive 0-based Long indices,
+   * similar to `RDD.zipWithIndex()`.
+   *
+   * The index column is appended as the last column of the resulting [[DataFrame]].
+   *
+   * @group typedrel
+   * @since 4.2.0
+   */
+  def zipWithIndex(): DataFrame = zipWithIndex("index")
+
+  /**
+   * Returns a new [[Dataset]] by appending a column containing consecutive 0-based Long indices,
+   * similar to `RDD.zipWithIndex()`.
+   *
+   * The index column is appended as the last column of the resulting [[DataFrame]].
+   *
+   * @param indexColName
+   *   The name of the index column to append. The dataset must not already contain a column with
+   *   this name.
+   * @throws AnalysisException
+   *   if a column with `indexColName` already exists in the schema.

Review Comment:
   This makes sense to me, and we can relax it later if needed.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-55228][SPARK-55230][SQL][CONNECT] Implement Dataset.zipWithIndex in Scala API [spark]

Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #54014:
URL: https://github.com/apache/spark/pull/54014#discussion_r2752363179


##########
sql/connect/client/jvm/src/test/scala/org/apache/spark/sql/connect/ClientE2ETestSuite.scala:
##########
@@ -772,6 +772,30 @@ class ClientE2ETestSuite
     assert(spark.range(10).count() === 10)
   }
 
+  test("Dataset zipWithIndex") {
+    val df = spark.range(5).repartition(3)
+    val result = df.zipWithIndex()
+    assert(result.columns === Array("id", "index"))
+    assert(result.schema.last.dataType === LongType)
+    val indices = result.collect().map(_.getLong(1)).sorted
+    assert(indices === (0L until 5L).toArray)
+  }
+
+  test("Dataset zipWithIndex with custom column name") {
+    val result = spark.range(3).zipWithIndex("row_num")
+    assert(result.columns === Array("id", "row_num"))
+    val indices = result.collect().map(_.getLong(1)).sorted
+    assert(indices === Array(0L, 1L, 2L))
+  }
+
+  test("Dataset zipWithIndex should throw if column name already exists") {

Review Comment:
   looks good: https://github.com/apache/spark/pull/54014/files#r2752362758



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-55228][SPARK-55230][SQL][CONNECT] Implement Dataset.zipWithIndex in Scala API [spark]

Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on code in PR #54014:
URL: https://github.com/apache/spark/pull/54014#discussion_r2745325345


##########
sql/connect/client/jvm/src/test/scala/org/apache/spark/sql/connect/ClientE2ETestSuite.scala:
##########
@@ -772,6 +772,30 @@ class ClientE2ETestSuite
     assert(spark.range(10).count() === 10)
   }
 
+  test("Dataset zipWithIndex") {
+    val df = spark.range(5).repartition(3)
+    val result = df.zipWithIndex()
+    assert(result.columns === Array("id", "index"))
+    assert(result.schema.last.dataType === LongType)
+    val indices = result.collect().map(_.getLong(1)).sorted
+    assert(indices === (0L until 5L).toArray)
+  }
+
+  test("Dataset zipWithIndex with custom column name") {
+    val result = spark.range(3).zipWithIndex("row_num")
+    assert(result.columns === Array("id", "row_num"))
+    val indices = result.collect().map(_.getLong(1)).sorted
+    assert(indices === Array(0L, 1L, 2L))
+  }
+
+  test("Dataset zipWithIndex should throw if column name already exists") {

Review Comment:
   I think we should clearly define the behavior in case `name` already exists in `df`
   
   ```
   df.zipWithIndex("name")
   ```
   
   should it fail (as in the first commit) or silently replace the old column or keep both columns (existing implementation)?
   
   @cloud-fan @HyukjinKwon @ueshin 
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-55228][SPARK-55230][SQL] Implement Dataset.zipWithIndex in Scala API [spark]

Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on code in PR #54014:
URL: https://github.com/apache/spark/pull/54014#discussion_r2734529433


##########
sql/connect/common/src/main/scala/org/apache/spark/sql/connect/Dataset.scala:
##########
@@ -703,6 +703,16 @@ class Dataset[T] private[sql] (
     }
   }
 
+  /** @inheritdoc */
+  def zipWithIndex(indexColName: String): DataFrame = {

Review Comment:
   since scala client is supported in this PR, we may want to add some tests in 
   
   src/test/scala/org/apache/spark/sql/PlanGenerationTestSuite.scala
   org/apache/spark/sql/connect/ClientE2ETestSuite.scala
   
   but the tests can be added in a followup PR



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-55228][SPARK-55230][SQL][CONNECT] Implement Dataset.zipWithIndex in Scala API [spark]

Posted by "fangchenli (via GitHub)" <gi...@apache.org>.
fangchenli commented on code in PR #54014:
URL: https://github.com/apache/spark/pull/54014#discussion_r2749165155


##########
sql/connect/client/jvm/src/test/scala/org/apache/spark/sql/connect/ClientE2ETestSuite.scala:
##########
@@ -772,6 +772,30 @@ class ClientE2ETestSuite
     assert(spark.range(10).count() === 10)
   }
 
+  test("Dataset zipWithIndex") {
+    val df = spark.range(5).repartition(3)
+    val result = df.zipWithIndex()
+    assert(result.columns === Array("id", "index"))
+    assert(result.schema.last.dataType === LongType)
+    val indices = result.collect().map(_.getLong(1)).sorted
+    assert(indices === (0L until 5L).toArray)
+  }
+
+  test("Dataset zipWithIndex with custom column name") {
+    val result = spark.range(3).zipWithIndex("row_num")
+    assert(result.columns === Array("id", "row_num"))
+    val indices = result.collect().map(_.getLong(1)).sorted
+    assert(indices === Array(0L, 1L, 2L))
+  }
+
+  test("Dataset zipWithIndex should throw if column name already exists") {

Review Comment:
   Thanks for the feedback. I'll wait for more comments. 
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-55228][SPARK-55230][SQL][CONNECT] Implement Dataset.zipWithIndex in Scala API [spark]

Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on PR #54014:
URL: https://github.com/apache/spark/pull/54014#issuecomment-3850822144

   @fangchenli zipWithIndex in org.apache.spark.sql.PlanGenerationTestSuite failed


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-55228][SPARK-55230][SQL][CONNECT] Implement Dataset.zipWithIndex in Scala API [spark]

Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on code in PR #54014:
URL: https://github.com/apache/spark/pull/54014#discussion_r2752721347


##########
sql/core/src/main/scala/org/apache/spark/sql/classic/Dataset.scala:
##########
@@ -1198,6 +1198,11 @@ class Dataset[T] private[sql](
     Except(logicalPlan, other.logicalPlan, isAll = true)
   }
 
+  /** @inheritdoc */
+  def zipWithIndex(indexColName: String): DataFrame = {
+    select(col("*"), Column(DistributedSequenceID()).alias(indexColName))

Review Comment:
   yeah, I think we should eagerly check the column name here.
   Otherwise, we will have to introduce a dedicated logical/protobuf plan for `zipWithIndex` and check it when analysis, which seems be over-complicated for this case.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-55228][SPARK-55230][SQL][CONNECT] Implement Dataset.zipWithIndex in Scala API [spark]

Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #54014:
URL: https://github.com/apache/spark/pull/54014#discussion_r2757881691


##########
sql/core/src/test/scala/org/apache/spark/sql/DatasetSuite.scala:
##########
@@ -2997,6 +2997,36 @@ class DatasetSuite extends QueryTest
     val metrics = observation.get
     assert(metrics.isEmpty)
   }
+
+  test("zipWithIndex should append consecutive 0-based indices") {

Review Comment:
   I think @juliuszsompolski added a mechanism to run client and server in the same JVM, which might be a good way to write unified tests.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-55228][SPARK-55230][SQL][CONNECT] Implement Dataset.zipWithIndex in Scala API [spark]

Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #54014:
URL: https://github.com/apache/spark/pull/54014#discussion_r2752365561


##########
sql/core/src/test/scala/org/apache/spark/sql/DatasetSuite.scala:
##########
@@ -2997,6 +2997,36 @@ class DatasetSuite extends QueryTest
     val metrics = observation.get
     assert(metrics.isEmpty)
   }
+
+  test("zipWithIndex should append consecutive 0-based indices") {

Review Comment:
   we should invest into the scala test framework so that we don't need to duplicate the classic and connect tests. cc @zhengruifeng 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-55228][SPARK-55230][SQL][CONNECT] Implement Dataset.zipWithIndex in Scala API [spark]

Posted by "fangchenli (via GitHub)" <gi...@apache.org>.
fangchenli commented on code in PR #54014:
URL: https://github.com/apache/spark/pull/54014#discussion_r2752391815


##########
sql/core/src/main/scala/org/apache/spark/sql/classic/Dataset.scala:
##########
@@ -1198,6 +1198,11 @@ class Dataset[T] private[sql](
     Except(logicalPlan, other.logicalPlan, isAll = true)
   }
 
+  /** @inheritdoc */
+  def zipWithIndex(indexColName: String): DataFrame = {
+    select(col("*"), Column(DistributedSequenceID()).alias(indexColName))

Review Comment:
   > how does it fail for duplicated col names?
   
   The current implementation doesn't fail. If we want it to fail on a duplicate column name, should I add the check eagerly here? I don't see similar checks in other methods. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-55228][SPARK-55230][SQL][CONNECT] Implement Dataset.zipWithIndex in Scala API [spark]

Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on PR #54014:
URL: https://github.com/apache/spark/pull/54014#issuecomment-3844828777

   @fangchenli the doc build is still failing, do you have try sync your spark fork?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-55228][SPARK-55230][SQL] Implement Dataset.zipWithIndex in Scala API [spark]

Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on PR #54014:
URL: https://github.com/apache/spark/pull/54014#issuecomment-3808676002

   @fangchenli linter and doc build failed, ptal


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-55228][SPARK-55230][SQL][CONNECT] Implement Dataset.zipWithIndex in Scala API [spark]

Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on code in PR #54014:
URL: https://github.com/apache/spark/pull/54014#discussion_r2757348342


##########
sql/core/src/test/scala/org/apache/spark/sql/DatasetSuite.scala:
##########
@@ -2997,6 +2997,36 @@ class DatasetSuite extends QueryTest
     val metrics = observation.get
     assert(metrics.isEmpty)
   }
+
+  test("zipWithIndex should append consecutive 0-based indices") {

Review Comment:
   it have conflicts with exisiting sparktestsuite -> plantest -> querytest frameworks, since conncet tests can only be e2e since the internals utils of logcal plans are not avaiable here.
   It might take some time to investigate



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-55228][SPARK-55230][SQL][CONNECT] Implement Dataset.zipWithIndex in Scala API [spark]

Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #54014:
URL: https://github.com/apache/spark/pull/54014#discussion_r2757869861


##########
sql/api/src/main/scala/org/apache/spark/sql/Dataset.scala:
##########
@@ -2010,6 +2010,35 @@ abstract class Dataset[T] extends Serializable {
    */
   def exceptAll(other: Dataset[T]): Dataset[T]
 
+  /**
+   * Returns a new [[Dataset]] by appending a column containing consecutive 0-based Long indices,
+   * similar to `RDD.zipWithIndex()`.
+   *
+   * The index column is appended as the last column of the resulting [[DataFrame]].
+   *
+   * @group typedrel

Review Comment:
   ```suggestion
      * @group untypedrel
   ```



##########
sql/api/src/main/scala/org/apache/spark/sql/Dataset.scala:
##########
@@ -2010,6 +2010,35 @@ abstract class Dataset[T] extends Serializable {
    */
   def exceptAll(other: Dataset[T]): Dataset[T]
 
+  /**
+   * Returns a new [[Dataset]] by appending a column containing consecutive 0-based Long indices,
+   * similar to `RDD.zipWithIndex()`.
+   *
+   * The index column is appended as the last column of the resulting [[DataFrame]].
+   *
+   * @group typedrel
+   * @since 4.2.0
+   */
+  def zipWithIndex(): DataFrame = zipWithIndex("index")
+
+  /**
+   * Returns a new [[Dataset]] by appending a column containing consecutive 0-based Long indices,
+   * similar to `RDD.zipWithIndex()`.
+   *
+   * The index column is appended as the last column of the resulting [[DataFrame]].
+   *
+   * @note
+   *   If a column with `indexColName` already exists in the schema, the resulting [[DataFrame]]
+   *   will have duplicate column names. Selecting the duplicate column by name will throw
+   *   `AMBIGUOUS_REFERENCE`, and writing the [[DataFrame]] will throw `COLUMN_ALREADY_EXISTS`.
+   *
+   * @param indexColName
+   *   The name of the index column to append.
+   * @group typedrel

Review Comment:
   ```suggestion
      * @group untypedrel
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-55228][SPARK-55230][SQL][CONNECT] Implement Dataset.zipWithIndex in Scala API [spark]

Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #54014:
URL: https://github.com/apache/spark/pull/54014#discussion_r2754063405


##########
sql/core/src/main/scala/org/apache/spark/sql/classic/Dataset.scala:
##########
@@ -1198,6 +1198,11 @@ class Dataset[T] private[sql](
     Except(logicalPlan, other.logicalPlan, isAll = true)
   }
 
+  /** @inheritdoc */
+  def zipWithIndex(indexColName: String): DataFrame = {
+    select(col("*"), Column(DistributedSequenceID()).alias(indexColName))

Review Comment:
   Or we allow appending the index column always, and users will get error if they select the column, or write out the dataframe to tables.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-55228][SQL] Implement Dataset.zipWithIndex in Scala API [spark]

Posted by "ueshin (via GitHub)" <gi...@apache.org>.
ueshin commented on code in PR #54014:
URL: https://github.com/apache/spark/pull/54014#discussion_r2734156824


##########
sql/core/src/test/scala/org/apache/spark/sql/DatasetSuite.scala:
##########
@@ -3012,6 +3012,36 @@ object CustomPathEncoder {
     Seq(realClassDataEnc.fields.head.copy(enc = custStringEnc),
       realClassDataEnc.fields.last)
   )
+
+  test("zipWithIndex should append consecutive 0-based indices") {

Review Comment:
   This is not in a test class but in `CustomPathEncoder`.



##########
sql/core/src/test/scala/org/apache/spark/sql/DatasetSuite.scala:
##########
@@ -3012,6 +3012,36 @@ object CustomPathEncoder {
     Seq(realClassDataEnc.fields.head.copy(enc = custStringEnc),
       realClassDataEnc.fields.last)
   )
+
+  test("zipWithIndex should append consecutive 0-based indices") {
+    val ds = Seq(("a", 1), ("b", 2), ("c", 3), ("d", 4), ("e", 5)).toDS().repartition(3)
+    val result = ds.zipWithIndex()
+
+    // Index column should be the last column
+    assert(result.columns === Array("_1", "_2", "index"))
+    assert(result.schema.last.dataType === LongType)
+
+    // Indices should be consecutive 0-based
+    val indices = result.collect().map(_.getLong(2)).sorted
+    assert(indices === (0L until 5L).toArray)
+  }
+
+  test("zipWithIndex with custom column name") {
+    val ds = Seq(1, 2, 3, 4, 5).toDS()
+    val result = ds.zipWithIndex("row_num")
+
+    assert(result.columns === Array("value", "row_num"))
+    val indices = result.collect().map(_.getLong(1)).sorted
+    assert(indices === (0L until 5L).toArray)
+  }
+
+  test("zipWithIndex should throw if column name already exists") {
+    val ds = Seq(("a", 1), ("b", 2)).toDS().withColumnRenamed("_1", "index")

Review Comment:
   nit: 
   ```suggestion
       val ds = Seq(("a", 1), ("b", 2)).toDF("_1", "index")
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-55228][SPARK-55230][SQL][CONNECT] Implement Dataset.zipWithIndex in Scala API [spark]

Posted by "fangchenli (via GitHub)" <gi...@apache.org>.
fangchenli commented on code in PR #54014:
URL: https://github.com/apache/spark/pull/54014#discussion_r2744849236


##########
sql/connect/common/src/main/scala/org/apache/spark/sql/connect/Dataset.scala:
##########
@@ -703,6 +703,13 @@ class Dataset[T] private[sql] (
     }
   }
 
+  /** @inheritdoc */
+  def zipWithIndex(indexColName: String): DataFrame = {
+    // Note: Column existence check is handled server-side to avoid requiring
+    // a schema fetch which needs a server connection.

Review Comment:
   Classic should not have this check either. Fixed.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-55228][SPARK-55230][SQL][CONNECT] Implement Dataset.zipWithIndex in Scala API [spark]

Posted by "han-nah-dev (via GitHub)" <gi...@apache.org>.
han-nah-dev commented on PR #54014:
URL: https://github.com/apache/spark/pull/54014#issuecomment-3852896364

   Wouldn't it make more sense for this to return a Dataset[(T, Long)]?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-55228][SPARK-55230][SQL][CONNECT] Implement Dataset.zipWithIndex in Scala API [spark]

Posted by "hvanhovell (via GitHub)" <gi...@apache.org>.
hvanhovell commented on PR #54014:
URL: https://github.com/apache/spark/pull/54014#issuecomment-3856003628

   @han-nah-dev it is possible. However it is a bit of work because of the following reasons:
   
   1. You need to use a schema that looks like this `struct<id: Long, value: struct<...>>`. This can get a bit tricky with single column datasets.
   2. You create a custom encoding that allows for a flattened schema.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-55228][SPARK-55230][SQL][CONNECT] Implement Dataset.zipWithIndex in Scala API [spark]

Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #54014:
URL: https://github.com/apache/spark/pull/54014#discussion_r2757874654


##########
sql/connect/common/src/main/scala/org/apache/spark/sql/connect/Dataset.scala:
##########
@@ -703,6 +703,11 @@ class Dataset[T] private[sql] (
     }
   }
 
+  /** @inheritdoc */
+  def zipWithIndex(indexColName: String): DataFrame = {
+    select(col("*"), Column.internalFn("distributed_sequence_id").alias(indexColName))

Review Comment:
   can we put the implementation in the base class? It seems to be the same between connect and classic.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-55228][SPARK-55230][SQL][CONNECT] Implement Dataset.zipWithIndex in Scala API [spark]

Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng closed pull request #54014: [SPARK-55228][SPARK-55230][SQL][CONNECT] Implement Dataset.zipWithIndex in Scala API
URL: https://github.com/apache/spark/pull/54014


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-55228][SPARK-55230][SQL][CONNECT] Implement Dataset.zipWithIndex in Scala API [spark]

Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on PR #54014:
URL: https://github.com/apache/spark/pull/54014#issuecomment-3851793118

   thanks, merged to master


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-55228][SPARK-55230][SQL][CONNECT] Implement Dataset.zipWithIndex in Scala API [spark]

Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #54014:
URL: https://github.com/apache/spark/pull/54014#discussion_r2766463361


##########
sql/api/src/main/scala/org/apache/spark/sql/Dataset.scala:
##########
@@ -2010,6 +2010,37 @@ abstract class Dataset[T] extends Serializable {
    */
   def exceptAll(other: Dataset[T]): Dataset[T]
 
+  /**
+   * Returns a new [[Dataset]] by appending a column containing consecutive 0-based Long indices,
+   * similar to `RDD.zipWithIndex()`.
+   *
+   * The index column is appended as the last column of the resulting [[DataFrame]].
+   *
+   * @group untypedrel
+   * @since 4.2.0
+   */
+  def zipWithIndex(): DataFrame = zipWithIndex("index")
+
+  /**
+   * Returns a new [[Dataset]] by appending a column containing consecutive 0-based Long indices,
+   * similar to `RDD.zipWithIndex()`.
+   *
+   * The index column is appended as the last column of the resulting [[DataFrame]].
+   *
+   * @note
+   *   If a column with `indexColName` already exists in the schema, the resulting [[DataFrame]]

Review Comment:
   ```suggestion
      *   If a column with `indexColName` already exists in the schema, the resulting `DataFrame`
   ```



##########
sql/api/src/main/scala/org/apache/spark/sql/Dataset.scala:
##########
@@ -2010,6 +2010,37 @@ abstract class Dataset[T] extends Serializable {
    */
   def exceptAll(other: Dataset[T]): Dataset[T]
 
+  /**
+   * Returns a new [[Dataset]] by appending a column containing consecutive 0-based Long indices,
+   * similar to `RDD.zipWithIndex()`.
+   *
+   * The index column is appended as the last column of the resulting [[DataFrame]].
+   *
+   * @group untypedrel
+   * @since 4.2.0
+   */
+  def zipWithIndex(): DataFrame = zipWithIndex("index")
+
+  /**
+   * Returns a new [[Dataset]] by appending a column containing consecutive 0-based Long indices,
+   * similar to `RDD.zipWithIndex()`.
+   *
+   * The index column is appended as the last column of the resulting [[DataFrame]].

Review Comment:
   ```suggestion
      * The index column is appended as the last column of the resulting `DataFrame`.
   ```



##########
sql/api/src/main/scala/org/apache/spark/sql/Dataset.scala:
##########
@@ -2010,6 +2010,37 @@ abstract class Dataset[T] extends Serializable {
    */
   def exceptAll(other: Dataset[T]): Dataset[T]
 
+  /**
+   * Returns a new [[Dataset]] by appending a column containing consecutive 0-based Long indices,
+   * similar to `RDD.zipWithIndex()`.
+   *
+   * The index column is appended as the last column of the resulting [[DataFrame]].

Review Comment:
   ```suggestion
      * The index column is appended as the last column of the resulting `DataFrame`.
   ```



##########
sql/api/src/main/scala/org/apache/spark/sql/Dataset.scala:
##########
@@ -2010,6 +2010,37 @@ abstract class Dataset[T] extends Serializable {
    */
   def exceptAll(other: Dataset[T]): Dataset[T]
 
+  /**
+   * Returns a new [[Dataset]] by appending a column containing consecutive 0-based Long indices,
+   * similar to `RDD.zipWithIndex()`.
+   *
+   * The index column is appended as the last column of the resulting [[DataFrame]].
+   *
+   * @group untypedrel
+   * @since 4.2.0
+   */
+  def zipWithIndex(): DataFrame = zipWithIndex("index")
+
+  /**
+   * Returns a new [[Dataset]] by appending a column containing consecutive 0-based Long indices,
+   * similar to `RDD.zipWithIndex()`.
+   *
+   * The index column is appended as the last column of the resulting [[DataFrame]].
+   *
+   * @note
+   *   If a column with `indexColName` already exists in the schema, the resulting [[DataFrame]]
+   *   will have duplicate column names. Selecting the duplicate column by name will throw
+   *   `AMBIGUOUS_REFERENCE`, and writing the [[DataFrame]] will throw `COLUMN_ALREADY_EXISTS`.

Review Comment:
   ```suggestion
      *   `AMBIGUOUS_REFERENCE`, and writing the `DataFrame` will throw `COLUMN_ALREADY_EXISTS`.
   ```



##########
sql/api/src/main/scala/org/apache/spark/sql/Dataset.scala:
##########
@@ -2010,6 +2010,37 @@ abstract class Dataset[T] extends Serializable {
    */
   def exceptAll(other: Dataset[T]): Dataset[T]
 
+  /**
+   * Returns a new [[Dataset]] by appending a column containing consecutive 0-based Long indices,
+   * similar to `RDD.zipWithIndex()`.
+   *
+   * The index column is appended as the last column of the resulting [[DataFrame]].
+   *
+   * @group untypedrel
+   * @since 4.2.0
+   */
+  def zipWithIndex(): DataFrame = zipWithIndex("index")
+
+  /**
+   * Returns a new [[Dataset]] by appending a column containing consecutive 0-based Long indices,

Review Comment:
   ```suggestion
      * Returns a new `Dataset` by appending a column containing consecutive 0-based Long indices,
   ```



##########
sql/api/src/main/scala/org/apache/spark/sql/Dataset.scala:
##########
@@ -2010,6 +2010,37 @@ abstract class Dataset[T] extends Serializable {
    */
   def exceptAll(other: Dataset[T]): Dataset[T]
 
+  /**
+   * Returns a new [[Dataset]] by appending a column containing consecutive 0-based Long indices,

Review Comment:
   ```suggestion
      * Returns a new `Dataset` by appending a column containing consecutive 0-based Long indices,
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-55228][SPARK-55230][SQL][CONNECT] Implement Dataset.zipWithIndex in Scala API [spark]

Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #54014:
URL: https://github.com/apache/spark/pull/54014#issuecomment-3850325481

   Unidoc (that generates Javadoc from Scaladoc) fails with `[[DataFrame]]` and `[[Dataset]]` references IIRC.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-55228][SPARK-55230][SQL][CONNECT] Implement Dataset.zipWithIndex in Scala API [spark]

Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #54014:
URL: https://github.com/apache/spark/pull/54014#discussion_r2752366726


##########
sql/core/src/main/scala/org/apache/spark/sql/classic/Dataset.scala:
##########
@@ -1198,6 +1198,11 @@ class Dataset[T] private[sql](
     Except(logicalPlan, other.logicalPlan, isAll = true)
   }
 
+  /** @inheritdoc */
+  def zipWithIndex(indexColName: String): DataFrame = {
+    select(col("*"), Column(DistributedSequenceID()).alias(indexColName))

Review Comment:
   how does it fail for duplicated col names?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-55228][SPARK-55230][SQL][CONNECT] Implement Dataset.zipWithIndex in Scala API [spark]

Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on code in PR #54014:
URL: https://github.com/apache/spark/pull/54014#discussion_r2745325345


##########
sql/connect/client/jvm/src/test/scala/org/apache/spark/sql/connect/ClientE2ETestSuite.scala:
##########
@@ -772,6 +772,30 @@ class ClientE2ETestSuite
     assert(spark.range(10).count() === 10)
   }
 
+  test("Dataset zipWithIndex") {
+    val df = spark.range(5).repartition(3)
+    val result = df.zipWithIndex()
+    assert(result.columns === Array("id", "index"))
+    assert(result.schema.last.dataType === LongType)
+    val indices = result.collect().map(_.getLong(1)).sorted
+    assert(indices === (0L until 5L).toArray)
+  }
+
+  test("Dataset zipWithIndex with custom column name") {
+    val result = spark.range(3).zipWithIndex("row_num")
+    assert(result.columns === Array("id", "row_num"))
+    val indices = result.collect().map(_.getLong(1)).sorted
+    assert(indices === Array(0L, 1L, 2L))
+  }
+
+  test("Dataset zipWithIndex should throw if column name already exists") {

Review Comment:
   I think we should clearly define the behavior in case `name` already exists in `df`
   
   ```
   df.zipWithIndex("name")
   ```
   
   should it fail (as in the first commit) or silently replace the old column (existing implementation)?
   
   @cloud-fan @HyukjinKwon @ueshin 
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org