You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "jackylee-ch (via GitHub)" <gi...@apache.org> on 2024/05/30 17:09:20 UTC
[PR] [SPARK][SQL][MINOR] Fix: V2Write use the same TaskAttemptId for different task attempts [spark]
jackylee-ch opened a new pull request, #46811:
URL: https://github.com/apache/spark/pull/46811
### What changes were proposed in this pull request?
After #40064 , we always get the same TaskAttemptId for different task attempts which has the same partitionId. This would lead different task attempts write to the same directory.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
GA
### Was this patch authored or co-authored using generative AI tooling?
No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK][SQL][MINOR] Fix: V2Write use the same TaskAttemptId for different task attempts [spark]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on PR #46811:
URL: https://github.com/apache/spark/pull/46811#issuecomment-2141214767
No，[SPARK-42478](https://issues.apache.org/jira/browse/SPARK-42478) is not part of the Spark 4.0 cycle, please use a new jira ticket. @jackylee-ch
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-48484][SQL] Fix: V2Write use the same TaskAttemptId for different task attempts [spark]
Posted by "yaooqinn (via GitHub)" <gi...@apache.org>.
yaooqinn commented on code in PR #46811:
URL: https://github.com/apache/spark/pull/46811#discussion_r1622168695
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/v2/FileWriterFactorySuite.scala:
##########
@@ -0,0 +1,48 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.execution.datasources.v2
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.mapreduce.task.TaskAttemptContextImpl
+import org.mockito.Mockito._
+import org.scalatest.PrivateMethodTester
+
+import org.apache.spark.SparkFunSuite
+import org.apache.spark.internal.io.FileCommitProtocol
+import org.apache.spark.sql.execution.datasources.WriteJobDescription
+import org.apache.spark.util.SerializableConfiguration
+
+class FileWriterFactorySuite extends SparkFunSuite with PrivateMethodTester {
+
+ test("SPARK-48484: V2Write use the different TaskAttemptId for different task attempts") {
Review Comment:
```suggestion
test("SPARK-48484: V2Write uses different TaskAttemptIds for different task attempts") {
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-48484][SQL] Fix: V2Write use the same TaskAttemptId for different task attempts [spark]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on code in PR #46811:
URL: https://github.com/apache/spark/pull/46811#discussion_r1623723019
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/FileWriterFactory.scala:
##########
@@ -38,7 +38,7 @@ case class FileWriterFactory (
@transient private lazy val jobId = SparkHadoopWriterUtils.createJobID(jobTrackerID, 0)
override def createWriter(partitionId: Int, realTaskId: Long): DataWriter[InternalRow] = {
- val taskAttemptContext = createTaskAttemptContext(partitionId)
+ val taskAttemptContext = createTaskAttemptContext(partitionId, realTaskId.toInt & Int.MaxValue)
Review Comment:
@cloud-fan If overflow, `realTaskId.toInt & Int.MaxValue` and `math.abs` are not equal:
```scala
scala> val realTaskId = Long.MaxValue
val realTaskId: Long = 9223372036854775807
scala> val a = realTaskId.toInt
val a: Int = -1
scala> val b = realTaskId.toInt & Int.MaxValue
val b: Int = 2147483647
scala> val c= math.abs(realTaskId.toInt)
val c: Int = 1
```
```scala
scala> val realTaskId = Int.MaxValue.toLong + 1
val realTaskId: Long = 2147483648
scala> val a = realTaskId.toInt
val a: Int = -2147483648
scala> val b = realTaskId.toInt & Int.MaxValue
val b: Int = 0
scala> val c= math.abs(realTaskId.toInt)
val c: Int = -2147483648
scala> val c= math.abs(a)
val c: Int = -2147483648
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-48484][SQL] Fix: V2Write use the same TaskAttemptId for different task attempts [spark]
Posted by "jackylee-ch (via GitHub)" <gi...@apache.org>.
jackylee-ch commented on code in PR #46811:
URL: https://github.com/apache/spark/pull/46811#discussion_r1621905165
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/FileWriterFactory.scala:
##########
@@ -47,9 +47,12 @@ case class FileWriterFactory (
}
}
- private def createTaskAttemptContext(partitionId: Int): TaskAttemptContextImpl = {
+ // VisibleForTesting
+ private[v2] def createTaskAttemptContext(
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-48484][SQL] Fix: V2Write use the same TaskAttemptId for different task attempts [spark]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on code in PR #46811:
URL: https://github.com/apache/spark/pull/46811#discussion_r1623723019
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/FileWriterFactory.scala:
##########
@@ -38,7 +38,7 @@ case class FileWriterFactory (
@transient private lazy val jobId = SparkHadoopWriterUtils.createJobID(jobTrackerID, 0)
override def createWriter(partitionId: Int, realTaskId: Long): DataWriter[InternalRow] = {
- val taskAttemptContext = createTaskAttemptContext(partitionId)
+ val taskAttemptContext = createTaskAttemptContext(partitionId, realTaskId.toInt & Int.MaxValue)
Review Comment:
@cloud-fan If overflow, `realTaskId.toInt & Int.MaxValue` and `math.abs` are not equal:
```scala
scala> val realTaskId = Long.MaxValue
val realTaskId: Long = 9223372036854775807
scala> val a = realTaskId.toInt
val a: Int = -1
scala> val b = realTaskId.toInt & Int.MaxValue
val b: Int = 2147483647
scala> val c= math.abs(realTaskId.toInt)
val c: Int = 1
```
```scala
scala> val realTaskId = Int.MaxValue.toLong + 1
val realTaskId: Long = 2147483648
scala> val a = realTaskId.toInt
val a: Int = -2147483648
scala> val b = realTaskId.toInt & Int.MaxValue
val b: Int = 0
scala> val c= math.abs(realTaskId.toInt)
val c: Int = -2147483648
```
Meanwhile, when an overflow occurs, `math.abs` may still return a negative value, so I suggest we continue using `& Int.MaxValue`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-48484][SQL] Fix: V2Write use the same TaskAttemptId for different task attempts [spark]
Posted by "jackylee-ch (via GitHub)" <gi...@apache.org>.
jackylee-ch commented on code in PR #46811:
URL: https://github.com/apache/spark/pull/46811#discussion_r1621940610
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/v2/FileWriterFactorySuite.scala:
##########
@@ -0,0 +1,47 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.execution.datasources.v2
+
+import org.apache.hadoop.mapreduce.task.TaskAttemptContextImpl
+import org.mockito.Mockito._
+import org.scalatest.PrivateMethodTester
+
+import org.apache.spark.internal.io.FileCommitProtocol
+import org.apache.spark.sql.execution.datasources.WriteJobDescription
+import org.apache.spark.sql.test.SharedSparkSession
+import org.apache.spark.util.SerializableConfiguration
+
+class FileWriterFactorySuite extends SharedSparkSession with PrivateMethodTester {
Review Comment:
We need a `Configuration` here as it will be used in `createTaskAttemptContext`. It's ok to me that we just create a new `Configuration`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK][SQL][MINOR] Fix: V2Write use the same TaskAttemptId for different task attempts [spark]
Posted by "jackylee-ch (via GitHub)" <gi...@apache.org>.
jackylee-ch commented on PR #46811:
URL: https://github.com/apache/spark/pull/46811#issuecomment-2141147987
cc @LuciferYang @yikf
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-48484][SQL] Fix: V2Write use the same TaskAttemptId for different task attempts [spark]
Posted by "ulysses-you (via GitHub)" <gi...@apache.org>.
ulysses-you commented on code in PR #46811:
URL: https://github.com/apache/spark/pull/46811#discussion_r1621835420
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/FileWriterFactory.scala:
##########
@@ -38,7 +38,7 @@ case class FileWriterFactory (
@transient private lazy val jobId = SparkHadoopWriterUtils.createJobID(jobTrackerID, 0)
override def createWriter(partitionId: Int, realTaskId: Long): DataWriter[InternalRow] = {
Review Comment:
Not related to this pr, why not naming it as `taskAttemptId` ? does `realTaskId` can be something else ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-48484][SQL] Fix: V2Write use the same TaskAttemptId for different task attempts [spark]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang closed pull request #46811: [SPARK-48484][SQL] Fix: V2Write use the same TaskAttemptId for different task attempts
URL: https://github.com/apache/spark/pull/46811
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-48484][SQL] Fix: V2Write use the same TaskAttemptId for different task attempts [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #46811:
URL: https://github.com/apache/spark/pull/46811#discussion_r1623742283
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/FileWriterFactory.scala:
##########
@@ -38,7 +38,7 @@ case class FileWriterFactory (
@transient private lazy val jobId = SparkHadoopWriterUtils.createJobID(jobTrackerID, 0)
override def createWriter(partitionId: Int, realTaskId: Long): DataWriter[InternalRow] = {
- val taskAttemptContext = createTaskAttemptContext(partitionId)
+ val taskAttemptContext = createTaskAttemptContext(partitionId, realTaskId.toInt & Int.MaxValue)
Review Comment:
I see, `&` is better here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-48484][SQL] Fix: V2Write use the same TaskAttemptId for different task attempts [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #46811:
URL: https://github.com/apache/spark/pull/46811#discussion_r1623742283
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/FileWriterFactory.scala:
##########
@@ -38,7 +38,7 @@ case class FileWriterFactory (
@transient private lazy val jobId = SparkHadoopWriterUtils.createJobID(jobTrackerID, 0)
override def createWriter(partitionId: Int, realTaskId: Long): DataWriter[InternalRow] = {
- val taskAttemptContext = createTaskAttemptContext(partitionId)
+ val taskAttemptContext = createTaskAttemptContext(partitionId, realTaskId.toInt & Int.MaxValue)
Review Comment:
I see, `&` is better here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-48484][SQL] Fix: V2Write use the same TaskAttemptId for different task attempts [spark]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on code in PR #46811:
URL: https://github.com/apache/spark/pull/46811#discussion_r1621871843
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/FileWriterFactory.scala:
##########
@@ -47,9 +47,12 @@ case class FileWriterFactory (
}
}
- private def createTaskAttemptContext(partitionId: Int): TaskAttemptContextImpl = {
+ // VisibleForTesting
+ private[v2] def createTaskAttemptContext(
Review Comment:
can we use `PrivateMethodTester` in `FileWriterFactorySuite` to avoid expanding the scope of this function
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-48484][SQL] Fix: V2Write use the same TaskAttemptId for different task attempts [spark]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on code in PR #46811:
URL: https://github.com/apache/spark/pull/46811#discussion_r1621927729
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/v2/FileWriterFactorySuite.scala:
##########
@@ -0,0 +1,47 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.execution.datasources.v2
+
+import org.apache.hadoop.mapreduce.task.TaskAttemptContextImpl
+import org.mockito.Mockito._
+import org.scalatest.PrivateMethodTester
+
+import org.apache.spark.internal.io.FileCommitProtocol
+import org.apache.spark.sql.execution.datasources.WriteJobDescription
+import org.apache.spark.sql.test.SharedSparkSession
+import org.apache.spark.util.SerializableConfiguration
+
+class FileWriterFactorySuite extends SharedSparkSession with PrivateMethodTester {
Review Comment:
if we just check `createTaskAttemptContext`, do we really need to inherit from `SharedSparkSession`? Can we just inherit from `SparkFunSuite`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-48484][SQL] Fix: V2Write use the same TaskAttemptId for different task attempts [spark]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on PR #46811:
URL: https://github.com/apache/spark/pull/46811#issuecomment-2142407430
Merged into master/3.5/3.4. thanks @jackylee-ch @yaooqinn @cloud-fan @ulysses-you @yikf
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-48484][SQL] Fix: V2Write use the same TaskAttemptId for different task attempts [spark]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on code in PR #46811:
URL: https://github.com/apache/spark/pull/46811#discussion_r1623723019
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/FileWriterFactory.scala:
##########
@@ -38,7 +38,7 @@ case class FileWriterFactory (
@transient private lazy val jobId = SparkHadoopWriterUtils.createJobID(jobTrackerID, 0)
override def createWriter(partitionId: Int, realTaskId: Long): DataWriter[InternalRow] = {
- val taskAttemptContext = createTaskAttemptContext(partitionId)
+ val taskAttemptContext = createTaskAttemptContext(partitionId, realTaskId.toInt & Int.MaxValue)
Review Comment:
@cloud-fan If overflow `realTaskId.toInt & Int.MaxValue` and `math.abs` are not equal:
```scala
scala> val realTaskId = Long.MaxValue
val realTaskId: Long = 9223372036854775807
scala> val a = realTaskId.toInt
val a: Int = -1
scala> val b = realTaskId.toInt & Int.MaxValue
val b: Int = 2147483647
scala> val c= math.abs(realTaskId.toInt)
val c: Int = 1
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK][SQL][MINOR] Fix: V2Write use the same TaskAttemptId for different task attempts [spark]
Posted by "yaooqinn (via GitHub)" <gi...@apache.org>.
yaooqinn commented on PR #46811:
URL: https://github.com/apache/spark/pull/46811#issuecomment-2141177667
Please remove the [MINOR] tag and file a Jira ticket for this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK][SQL][MINOR] Fix: V2Write use the same TaskAttemptId for different task attempts [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #46811:
URL: https://github.com/apache/spark/pull/46811#discussion_r1621669315
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/FileWriterFactory.scala:
##########
@@ -38,7 +38,7 @@ case class FileWriterFactory (
@transient private lazy val jobId = SparkHadoopWriterUtils.createJobID(jobTrackerID, 0)
override def createWriter(partitionId: Int, realTaskId: Long): DataWriter[InternalRow] = {
- val taskAttemptContext = createTaskAttemptContext(partitionId)
+ val taskAttemptContext = createTaskAttemptContext(partitionId, realTaskId.toInt & Int.MaxValue)
Review Comment:
is it the same as `math.abs`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-48484][SQL] Fix: V2Write use the same TaskAttemptId for different task attempts [spark]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on PR #46811:
URL: https://github.com/apache/spark/pull/46811#issuecomment-2141237934
create [SPARK-48484](https://issues.apache.org/jira/browse/SPARK-48484) for this one @jackylee-ch
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK][SQL][MINOR] Fix: V2Write use the same TaskAttemptId for different task attempts [spark]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on code in PR #46811:
URL: https://github.com/apache/spark/pull/46811#discussion_r1621698631
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/FileWriterFactory.scala:
##########
@@ -38,7 +38,7 @@ case class FileWriterFactory (
@transient private lazy val jobId = SparkHadoopWriterUtils.createJobID(jobTrackerID, 0)
override def createWriter(partitionId: Int, realTaskId: Long): DataWriter[InternalRow] = {
- val taskAttemptContext = createTaskAttemptContext(partitionId)
+ val taskAttemptContext = createTaskAttemptContext(partitionId, realTaskId.toInt & Int.MaxValue)
Review Comment:
https://github.com/apache/spark/blob/80addbbddab73980a0ea0672dac5634465c1997b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/WriteFiles.scala#L97
https://github.com/apache/spark/blob/45ba9224602eb18fe45e339cbb8cf2e8a4924f0b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileFormatWriter.scala#L263
There are at least two other similar cases here, should we unify them as `math.abs`? Of course, this should be another PR. @cloud-fan
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK][SQL][MINOR] Fix: V2Write use the same TaskAttemptId for different task attempts [spark]
Posted by "jackylee-ch (via GitHub)" <gi...@apache.org>.
jackylee-ch commented on code in PR #46811:
URL: https://github.com/apache/spark/pull/46811#discussion_r1621672202
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/FileWriterFactory.scala:
##########
@@ -38,7 +38,7 @@ case class FileWriterFactory (
@transient private lazy val jobId = SparkHadoopWriterUtils.createJobID(jobTrackerID, 0)
override def createWriter(partitionId: Int, realTaskId: Long): DataWriter[InternalRow] = {
- val taskAttemptContext = createTaskAttemptContext(partitionId)
+ val taskAttemptContext = createTaskAttemptContext(partitionId, realTaskId.toInt & Int.MaxValue)
Review Comment:
Yep, it same as `Math.abs(realTaskId.toInt)`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK][SQL][MINOR] Fix: V2Write use the same TaskAttemptId for different task attempts [spark]
Posted by "jackylee-ch (via GitHub)" <gi...@apache.org>.
jackylee-ch commented on PR #46811:
URL: https://github.com/apache/spark/pull/46811#issuecomment-2141191717
> Please remove the [MINOR] tag and file a Jira ticket for this
Sure, I will add a follow up for SPARK-42478 and a suite test for this pr.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-48484][SQL] Fix: V2Write use the same TaskAttemptId for different task attempts [spark]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on code in PR #46811:
URL: https://github.com/apache/spark/pull/46811#discussion_r1623081819
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/FileWriterFactory.scala:
##########
@@ -38,7 +38,7 @@ case class FileWriterFactory (
@transient private lazy val jobId = SparkHadoopWriterUtils.createJobID(jobTrackerID, 0)
override def createWriter(partitionId: Int, realTaskId: Long): DataWriter[InternalRow] = {
- val taskAttemptContext = createTaskAttemptContext(partitionId)
+ val taskAttemptContext = createTaskAttemptContext(partitionId, realTaskId.toInt & Int.MaxValue)
Review Comment:
ok, we can unify the above three cases to `math.abs` in a follow-up.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK][SQL][MINOR] Fix: V2Write use the same TaskAttemptId for different task attempts [spark]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on PR #46811:
URL: https://github.com/apache/spark/pull/46811#issuecomment-2141152363
also cc @yaooqinn @HyukjinKwon @cloud-fan
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-48484][SQL] Fix: V2Write use the same TaskAttemptId for different task attempts [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #46811:
URL: https://github.com/apache/spark/pull/46811#discussion_r1622700068
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/FileWriterFactory.scala:
##########
@@ -38,7 +38,7 @@ case class FileWriterFactory (
@transient private lazy val jobId = SparkHadoopWriterUtils.createJobID(jobTrackerID, 0)
override def createWriter(partitionId: Int, realTaskId: Long): DataWriter[InternalRow] = {
- val taskAttemptContext = createTaskAttemptContext(partitionId)
+ val taskAttemptContext = createTaskAttemptContext(partitionId, realTaskId.toInt & Int.MaxValue)
Review Comment:
I don't think perf matters here, and `math.abs` is definitely more readable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-48484][SQL] Fix: V2Write use the same TaskAttemptId for different task attempts [spark]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on code in PR #46811:
URL: https://github.com/apache/spark/pull/46811#discussion_r1623723019
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/FileWriterFactory.scala:
##########
@@ -38,7 +38,7 @@ case class FileWriterFactory (
@transient private lazy val jobId = SparkHadoopWriterUtils.createJobID(jobTrackerID, 0)
override def createWriter(partitionId: Int, realTaskId: Long): DataWriter[InternalRow] = {
- val taskAttemptContext = createTaskAttemptContext(partitionId)
+ val taskAttemptContext = createTaskAttemptContext(partitionId, realTaskId.toInt & Int.MaxValue)
Review Comment:
@cloud-fan If overflow, `realTaskId.toInt & Int.MaxValue` and `math.abs` are not equal:
```scala
scala> val realTaskId = Long.MaxValue
val realTaskId: Long = 9223372036854775807
scala> val a = realTaskId.toInt
val a: Int = -1
scala> val b = realTaskId.toInt & Int.MaxValue
val b: Int = 2147483647
scala> val c= math.abs(realTaskId.toInt)
val c: Int = 1
```
```scala
scala> val realTaskId = Int.MaxValue.toLong + 1
val realTaskId: Long = 2147483648
scala> val a = realTaskId.toInt
val a: Int = -2147483648
scala> val b = realTaskId.toInt & Int.MaxValue
val b: Int = 0
scala> val c= math.abs(realTaskId.toInt)
val c: Int = -2147483648
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org