You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by cloud-fan <gi...@git.apache.org> on 2018/07/30 17:02:29 UTC

[GitHub] spark pull request #21921: [SPARK-24971][SQL] remove SupportsDeprecatedScanR...

GitHub user cloud-fan opened a pull request:

    https://github.com/apache/spark/pull/21921

    [SPARK-24971][SQL] remove SupportsDeprecatedScanRow

    ## What changes were proposed in this pull request?
    
    This is a follow up of https://github.com/apache/spark/pull/21118 .
    
    In https://github.com/apache/spark/pull/21118 we added `SupportsDeprecatedScanRow`. Ideally data source should produce `InternalRow` instead of `Row` for better performance. We should remove `SupportsDeprecatedScanRow` and encourage data sources to produce `InternalRow`, which is also very easy to build.
    
    ## How was this patch tested?
    
    existing tests.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/cloud-fan/spark row

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/spark/pull/21921.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #21921
    
----
commit d6a93b162b4ced2c9ab33715cfbe8d196e6140e4
Author: Wenchen Fan <we...@...>
Date:   2018-07-30T15:50:45Z

    remove SupportsDeprecatedScanRow

----


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21921: [SPARK-24971][SQL] remove SupportsDeprecatedScanRow

Posted by rdblue <gi...@git.apache.org>.
Github user rdblue commented on the issue:

    https://github.com/apache/spark/pull/21921
  
    @cloud-fan, @gatorsmile, I'm fine with that if it's documented somewhere. I wasn't aware of that convention and no one brought it up the last time I pointed out commits without a committer +1.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21921: [SPARK-24971][SQL] remove SupportsDeprecatedScanRow

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21921
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21921: [SPARK-24971][SQL] remove SupportsDeprecatedScanRow

Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the issue:

    https://github.com/apache/spark/pull/21921
  
    cc @rdblue @rxin @gatorsmile @gengliangwang 


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #21921: [SPARK-24971][SQL] remove SupportsDeprecatedScanR...

Posted by rdblue <gi...@git.apache.org>.
Github user rdblue commented on a diff in the pull request:

    https://github.com/apache/spark/pull/21921#discussion_r206936202
  
    --- Diff: sql/core/src/test/scala/org/apache/spark/sql/sources/v2/DataSourceV2Suite.scala ---
    @@ -121,17 +121,6 @@ class DataSourceV2Suite extends QueryTest with SharedSQLContext {
         }
       }
     
    -  test("unsafe row scan implementation") {
    -    Seq(classOf[UnsafeRowDataSourceV2], classOf[JavaUnsafeRowDataSourceV2]).foreach { cls =>
    --- End diff --
    
    Why remove unsafe tests?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21921: [SPARK-24971][SQL] remove SupportsDeprecatedScanRow

Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the issue:

    https://github.com/apache/spark/pull/21921
  
    addressed in https://github.com/apache/spark/pull/21948#discussion_r206999480


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #21921: [SPARK-24971][SQL] remove SupportsDeprecatedScanR...

Posted by rdblue <gi...@git.apache.org>.
Github user rdblue commented on a diff in the pull request:

    https://github.com/apache/spark/pull/21921#discussion_r206946453
  
    --- Diff: sql/core/src/test/scala/org/apache/spark/sql/sources/v2/DataSourceV2Suite.scala ---
    @@ -416,12 +405,12 @@ class AdvancedDataSourceV2 extends DataSourceV2 with ReadSupport {
           requiredSchema
         }
     
    -    override def planRowInputPartitions(): JList[InputPartition[Row]] = {
    -      val lowerBound = filters.collect {
    +    override def planInputPartitions(): JList[InputPartition[InternalRow]] = {
    +      val lowerBound = filters.collectFirst {
    --- End diff --
    
    Fine by me since it is so small, just wanted to point it out.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21921: [SPARK-24971][SQL] remove SupportsDeprecatedScanRow

Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the issue:

    https://github.com/apache/spark/pull/21921
  
    @rdblue I vaguely remember that, if the PR author himself is a committer, we can merge a PR with one more LGTM from the community and no one objects in several days. I'm sorry if it's not the case.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #21921: [SPARK-24971][SQL] remove SupportsDeprecatedScanR...

Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:

    https://github.com/apache/spark/pull/21921#discussion_r206944348
  
    --- Diff: sql/core/src/test/scala/org/apache/spark/sql/sources/v2/DataSourceV2Suite.scala ---
    @@ -121,17 +121,6 @@ class DataSourceV2Suite extends QueryTest with SharedSQLContext {
         }
       }
     
    -  test("unsafe row scan implementation") {
    -    Seq(classOf[UnsafeRowDataSourceV2], classOf[JavaUnsafeRowDataSourceV2]).foreach { cls =>
    --- End diff --
    
    That's a followup of https://github.com/apache/spark/pull/21118. you removed `SupportsScanUnsafeRow` there and then this test becomes meaningless.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #21921: [SPARK-24971][SQL] remove SupportsDeprecatedScanR...

Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:

    https://github.com/apache/spark/pull/21921#discussion_r206944006
  
    --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/sources/RateStreamMicroBatchReader.scala ---
    @@ -169,7 +170,7 @@ class RateStreamMicroBatchReader(options: DataSourceOptions, checkpointLocation:
         (0 until numPartitions).map { p =>
           new RateStreamMicroBatchInputPartition(
             p, numPartitions, rangeStart, rangeEnd, localStartTimeMs, relativeMsPerValue)
    -        : InputPartition[Row]
    +        : InputPartition[InternalRow]
    --- End diff --
    
    ditto


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21921: [SPARK-24971][SQL] remove SupportsDeprecatedScanRow

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/21921
  
    **[Test build #93796 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93796/testReport)** for PR 21921 at commit [`d6a93b1`](https://github.com/apache/spark/commit/d6a93b162b4ced2c9ab33715cfbe8d196e6140e4).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds the following public classes _(experimental)_:
      * `class RateStreamContinuousReader(options: DataSourceOptions) extends ContinuousReader `
      * `class TextSocketMicroBatchReader(options: DataSourceOptions) extends MicroBatchReader with Logging `


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21921: [SPARK-24971][SQL] remove SupportsDeprecatedScanRow

Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:

    https://github.com/apache/spark/pull/21921
  
    retest this please


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21921: [SPARK-24971][SQL] remove SupportsDeprecatedScanRow

Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:

    https://github.com/apache/spark/pull/21921
  
    @cloud-fan To be save, let us get one more LGTM from the other committer. 


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #21921: [SPARK-24971][SQL] remove SupportsDeprecatedScanR...

Posted by rdblue <gi...@git.apache.org>.
Github user rdblue commented on a diff in the pull request:

    https://github.com/apache/spark/pull/21921#discussion_r206946076
  
    --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/continuous/ContinuousRateStreamSource.scala ---
    @@ -91,7 +90,7 @@ class RateStreamContinuousReader(options: DataSourceOptions)
             i,
             numPartitions,
             perPartitionRate)
    -        .asInstanceOf[InputPartition[Row]]
    +        .asInstanceOf[InputPartition[InternalRow]]
    --- End diff --
    
    I don't think it's a good idea to leave casts. Can you check to see if this can be avoided? I found in #21118 that many of the casts were unnecessary if variables had declared types and it is much better to avoid explicit casts that work around the type system.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21921: [SPARK-24971][SQL] remove SupportsDeprecatedScanRow

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21921
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93887/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #21921: [SPARK-24971][SQL] remove SupportsDeprecatedScanR...

Posted by rdblue <gi...@git.apache.org>.
Github user rdblue commented on a diff in the pull request:

    https://github.com/apache/spark/pull/21921#discussion_r206936422
  
    --- Diff: sql/core/src/test/scala/org/apache/spark/sql/sources/v2/DataSourceV2Suite.scala ---
    @@ -416,12 +405,12 @@ class AdvancedDataSourceV2 extends DataSourceV2 with ReadSupport {
           requiredSchema
         }
     
    -    override def planRowInputPartitions(): JList[InputPartition[Row]] = {
    -      val lowerBound = filters.collect {
    +    override def planInputPartitions(): JList[InputPartition[InternalRow]] = {
    +      val lowerBound = filters.collectFirst {
    --- End diff --
    
    Nit: this is an unrelated change.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21921: [SPARK-24971][SQL] remove SupportsDeprecatedScanRow

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/21921
  
    **[Test build #93796 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93796/testReport)** for PR 21921 at commit [`d6a93b1`](https://github.com/apache/spark/commit/d6a93b162b4ced2c9ab33715cfbe8d196e6140e4).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21921: [SPARK-24971][SQL] remove SupportsDeprecatedScanRow

Posted by jose-torres <gi...@git.apache.org>.
Github user jose-torres commented on the issue:

    https://github.com/apache/spark/pull/21921
  
    lgtm


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #21921: [SPARK-24971][SQL] remove SupportsDeprecatedScanR...

Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:

    https://github.com/apache/spark/pull/21921


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21921: [SPARK-24971][SQL] remove SupportsDeprecatedScanRow

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21921
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21921: [SPARK-24971][SQL] remove SupportsDeprecatedScanRow

Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the issue:

    https://github.com/apache/spark/pull/21921
  
    thanks, merging to master!


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21921: [SPARK-24971][SQL] remove SupportsDeprecatedScanRow

Posted by rdblue <gi...@git.apache.org>.
Github user rdblue commented on the issue:

    https://github.com/apache/spark/pull/21921
  
    This looks fine other than the possibly unnecessary cast.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #21921: [SPARK-24971][SQL] remove SupportsDeprecatedScanR...

Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:

    https://github.com/apache/spark/pull/21921#discussion_r206944875
  
    --- Diff: sql/core/src/test/scala/org/apache/spark/sql/sources/v2/DataSourceV2Suite.scala ---
    @@ -416,12 +405,12 @@ class AdvancedDataSourceV2 extends DataSourceV2 with ReadSupport {
           requiredSchema
         }
     
    -    override def planRowInputPartitions(): JList[InputPartition[Row]] = {
    -      val lowerBound = filters.collect {
    +    override def planInputPartitions(): JList[InputPartition[InternalRow]] = {
    +      val lowerBound = filters.collectFirst {
    --- End diff --
    
    I agree but this is really minor. When I changed the code nearby, the IDE shows a warning for not using `collectFirst` here. Then I went for it.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21921: [SPARK-24971][SQL] remove SupportsDeprecatedScanRow

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/21921
  
    **[Test build #93887 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93887/testReport)** for PR 21921 at commit [`d6a93b1`](https://github.com/apache/spark/commit/d6a93b162b4ced2c9ab33715cfbe8d196e6140e4).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds the following public classes _(experimental)_:
      * `class RateStreamContinuousReader(options: DataSourceOptions) extends ContinuousReader `
      * `class TextSocketMicroBatchReader(options: DataSourceOptions) extends MicroBatchReader with Logging `


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #21921: [SPARK-24971][SQL] remove SupportsDeprecatedScanR...

Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:

    https://github.com/apache/spark/pull/21921#discussion_r206943900
  
    --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/continuous/ContinuousRateStreamSource.scala ---
    @@ -91,7 +90,7 @@ class RateStreamContinuousReader(options: DataSourceOptions)
             i,
             numPartitions,
             perPartitionRate)
    -        .asInstanceOf[InputPartition[Row]]
    +        .asInstanceOf[InputPartition[InternalRow]]
    --- End diff --
    
    I didn't dig into it as the cast was already there. The reason seems to be, `java.util.List` isn't covariant.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #21921: [SPARK-24971][SQL] remove SupportsDeprecatedScanR...

Posted by rdblue <gi...@git.apache.org>.
Github user rdblue commented on a diff in the pull request:

    https://github.com/apache/spark/pull/21921#discussion_r206936701
  
    --- Diff: sql/core/src/test/scala/org/apache/spark/sql/sources/v2/DataSourceV2Suite.scala ---
    @@ -456,57 +445,20 @@ class AdvancedInputPartition(start: Int, end: Int, requiredSchema: StructType)
         current < end
       }
     
    -  override def get(): Row = {
    +  override def get(): InternalRow = {
         val values = requiredSchema.map(_.name).map {
           case "i" => current
           case "j" => -current
         }
    -    Row.fromSeq(values)
    +    InternalRow.fromSeq(values)
       }
     }
     
     
    -class UnsafeRowDataSourceV2 extends DataSourceV2 with ReadSupport {
    --- End diff --
    
    These aren't Row implementations. Why remove them?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21921: [SPARK-24971][SQL] remove SupportsDeprecatedScanRow

Posted by rdblue <gi...@git.apache.org>.
Github user rdblue commented on the issue:

    https://github.com/apache/spark/pull/21921
  
    @cloud-fan, I thought it was a requirement to have a committer +1 before merging. Or is this [list of committers](https://spark.apache.org/committers.html) out of date?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21921: [SPARK-24971][SQL] remove SupportsDeprecatedScanRow

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21921
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93796/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #21921: [SPARK-24971][SQL] remove SupportsDeprecatedScanR...

Posted by rdblue <gi...@git.apache.org>.
Github user rdblue commented on a diff in the pull request:

    https://github.com/apache/spark/pull/21921#discussion_r206946335
  
    --- Diff: sql/core/src/test/scala/org/apache/spark/sql/sources/v2/DataSourceV2Suite.scala ---
    @@ -121,17 +121,6 @@ class DataSourceV2Suite extends QueryTest with SharedSQLContext {
         }
       }
     
    -  test("unsafe row scan implementation") {
    -    Seq(classOf[UnsafeRowDataSourceV2], classOf[JavaUnsafeRowDataSourceV2]).foreach { cls =>
    --- End diff --
    
    Ok.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21921: [SPARK-24971][SQL] remove SupportsDeprecatedScanRow

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21921
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/1495/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21921: [SPARK-24971][SQL] remove SupportsDeprecatedScanRow

Posted by rdblue <gi...@git.apache.org>.
Github user rdblue commented on the issue:

    https://github.com/apache/spark/pull/21921
  
    Yeah, I'd say that if it isn't documented then lets go with the usually RTC conventions.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21921: [SPARK-24971][SQL] remove SupportsDeprecatedScanRow

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/21921
  
    **[Test build #93887 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93887/testReport)** for PR 21921 at commit [`d6a93b1`](https://github.com/apache/spark/commit/d6a93b162b4ced2c9ab33715cfbe8d196e6140e4).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #21921: [SPARK-24971][SQL] remove SupportsDeprecatedScanR...

Posted by rdblue <gi...@git.apache.org>.
Github user rdblue commented on a diff in the pull request:

    https://github.com/apache/spark/pull/21921#discussion_r206935616
  
    --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/sources/RateStreamMicroBatchReader.scala ---
    @@ -169,7 +170,7 @@ class RateStreamMicroBatchReader(options: DataSourceOptions, checkpointLocation:
         (0 until numPartitions).map { p =>
           new RateStreamMicroBatchInputPartition(
             p, numPartitions, rangeStart, rangeEnd, localStartTimeMs, relativeMsPerValue)
    -        : InputPartition[Row]
    +        : InputPartition[InternalRow]
    --- End diff --
    
    Is this needed? Doesn't RateStreamMicroBatchInputPartition implement InputPartition[InternalRow]?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21921: [SPARK-24971][SQL] remove SupportsDeprecatedScanRow

Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:

    https://github.com/apache/spark/pull/21921
  
    It sounds like Github is experiencing a very bad delay. @cloud-fan Could you submit a follow-up PR to address the comments from @rdblue ?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21921: [SPARK-24971][SQL] remove SupportsDeprecatedScanRow

Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:

    https://github.com/apache/spark/pull/21921
  
    @rdblue I do not think it is documented. Let us be more conservative. Collect LGTM from the committers no matter whether the PR author is a committer or not. 


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21921: [SPARK-24971][SQL] remove SupportsDeprecatedScanRow

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21921
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/1567/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21921: [SPARK-24971][SQL] remove SupportsDeprecatedScanRow

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21921
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #21921: [SPARK-24971][SQL] remove SupportsDeprecatedScanR...

Posted by rdblue <gi...@git.apache.org>.
Github user rdblue commented on a diff in the pull request:

    https://github.com/apache/spark/pull/21921#discussion_r206935176
  
    --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/continuous/ContinuousRateStreamSource.scala ---
    @@ -91,7 +90,7 @@ class RateStreamContinuousReader(options: DataSourceOptions)
             i,
             numPartitions,
             perPartitionRate)
    -        .asInstanceOf[InputPartition[Row]]
    +        .asInstanceOf[InputPartition[InternalRow]]
    --- End diff --
    
    Why is this cast necessary?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #21921: [SPARK-24971][SQL] remove SupportsDeprecatedScanR...

Posted by rdblue <gi...@git.apache.org>.
Github user rdblue commented on a diff in the pull request:

    https://github.com/apache/spark/pull/21921#discussion_r206946259
  
    --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/sources/RateStreamMicroBatchReader.scala ---
    @@ -169,7 +170,7 @@ class RateStreamMicroBatchReader(options: DataSourceOptions, checkpointLocation:
         (0 until numPartitions).map { p =>
           new RateStreamMicroBatchInputPartition(
             p, numPartitions, rangeStart, rangeEnd, localStartTimeMs, relativeMsPerValue)
    -        : InputPartition[Row]
    +        : InputPartition[InternalRow]
    --- End diff --
    
    This is fine since it isn't a cast, but it's generally better to check whether these are still necessary after refactoring.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #21921: [SPARK-24971][SQL] remove SupportsDeprecatedScanRow

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21921
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org