You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@uniffle.apache.org by "EnricoMi (via GitHub)" <gi...@apache.org> on 2024/02/08 18:02:49 UTC
[PR] [#1512] feature: Replace taskAttemptId with mapIndex in blockId [incubator-uniffle]
EnricoMi opened a new pull request, #1514:
URL: https://github.com/apache/incubator-uniffle/pull/1514
### What changes were proposed in this pull request?
Use map index (map task or source partition id) in block id instead of task attempt id.
Note: this draft lacks the following things (TODOs):
- MR and TEZ clients are not supported
- The `taskAttemptId` field in the blockId is not renamed to `mapIndex`.
- Using the mapIndex should be configurable
### Why are the changes needed?
The map index has a lower cardinality than the unbound positive long (63 bit) task attempt id. This reduces the practical risk (fails with sufficient number of stages for 100TB dataset and 500k partitions) of overflowing sequence number, partition id or map index below an impractical level in today's terms (supports any number of stages for 1PT dataset with 16m partitions).
Fix: #1512
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Unit and integration tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#1512] feature(Spark): Replace taskAttemptId with mapIndex in blockId [incubator-uniffle]
Posted by "jerqi (via GitHub)" <gi...@apache.org>.
jerqi commented on PR #1514:
URL: https://github.com/apache/incubator-uniffle/pull/1514#issuecomment-1942991425
> @jerqi you are right, if both tasks report their shuffle results (block ids), we would not know how many blocks to read for our partition id and map index. For that to work we need to guarantee that only one task attempt succeeds in registering shuffle results (blockids). We then take that taskAttemptId and those blockIds for the partition id and map index.
>
> This would require a two-phase commit to guard / coordinate the report shuffle result step, which would be a much bigger change: https://github.com/G-Research/incubator-uniffle/pull/4/files
>
> There are only two situations, where this can happen, though: speculative task execution (which is not recommended for production) and (without speculative task execution) a task failure after shuffle results have been reported, which triggers another task attempt.
Spark has already had a commit mechanism. If we set up a new two-phrase commit. Maybe there will be some consistent.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#1512] feature(Spark): Replace taskAttemptId with mapIndex in blockId [incubator-uniffle]
Posted by "EnricoMi (via GitHub)" <gi...@apache.org>.
EnricoMi commented on code in PR #1514:
URL: https://github.com/apache/incubator-uniffle/pull/1514#discussion_r1484062913
##########
integration-test/spark-common/src/test/java/org/apache/uniffle/test/FailingTasksTest.java:
##########
@@ -0,0 +1,89 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.uniffle.test;
+
+import java.util.Iterator;
+import java.util.Map;
+
+import com.google.common.collect.Maps;
+import org.apache.spark.TaskContext;
+import org.apache.spark.shuffle.RssSparkConfig;
+import org.apache.spark.sql.SparkSession;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.Test;
+
+import org.apache.uniffle.coordinator.CoordinatorConf;
+import org.apache.uniffle.server.ShuffleServerConf;
+import org.apache.uniffle.storage.util.StorageType;
+
+public class FailingTasksTest extends SparkIntegrationTestBase {
Review Comment:
Do we need to support legacy behaviour client-side? If not, then this PR should be complete.
I think legacy Spark / MR / TEZ clients work nicely with the proposed server-side changes, but new clients do not work with legacy servers, which should be fine from an cluster upgrade point of view.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#1512] feature(Spark): Replace taskAttemptId with mapIndex in blockId [incubator-uniffle]
Posted by "connorlwilkes (via GitHub)" <gi...@apache.org>.
connorlwilkes commented on PR #1514:
URL: https://github.com/apache/incubator-uniffle/pull/1514#issuecomment-1936200605
π
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#1512] feature(Spark): Replace taskAttemptId with mapIndex in blockId [incubator-uniffle]
Posted by "jerqi (via GitHub)" <gi...@apache.org>.
jerqi commented on PR #1514:
URL: https://github.com/apache/incubator-uniffle/pull/1514#issuecomment-1937988904
One question: We can't judge the data is from the same task. If one map task A have two attempts taskAttempt 1, taskAttempt 2. TaskAttempt 1 can have two blocks, TaskAttempt 2 may have three blocks. Because Spark doesn't guarantee every taskAttempts have the same data. Because some transforms of Spark are indeterministic.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#1512] feature(Spark): Replace taskAttemptId with mapIndex in blockId [incubator-uniffle]
Posted by "EnricoMi (via GitHub)" <gi...@apache.org>.
EnricoMi commented on code in PR #1514:
URL: https://github.com/apache/incubator-uniffle/pull/1514#discussion_r1484362254
##########
storage/src/main/java/org/apache/uniffle/storage/factory/ShuffleHandlerFactory.java:
##########
@@ -126,6 +126,8 @@ private ClientReadHandler getMemoryClientReadHandler(
Roaring64NavigableMap realExceptBlockIds = RssUtils.cloneBitMap(request.getExpectBlockIds());
realExceptBlockIds.xor(request.getProcessBlockIds());
expectTaskIds = RssUtils.generateTaskIdBitMap(realExceptBlockIds, request.getIdHelper());
+ } else {
+ expectTaskIds = request.getExpectTaskIds();
Review Comment:
With `serverInfoList.size() == 1`, the `else` branch is used for TEZ and MR client. I think expected task ids and those task ids in block ids differ structurally for those clients. See related code in #1518.
I think with setting `.expectedTaskIdsBitmapFilterEnable(true)` for those two clients would fix this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#1512] feature(Spark): Replace taskAttemptId with mapIndex in blockId [incubator-uniffle]
Posted by "EnricoMi (via GitHub)" <gi...@apache.org>.
EnricoMi commented on code in PR #1514:
URL: https://github.com/apache/incubator-uniffle/pull/1514#discussion_r1484060818
##########
storage/src/main/java/org/apache/uniffle/storage/factory/ShuffleHandlerFactory.java:
##########
@@ -126,6 +126,8 @@ private ClientReadHandler getMemoryClientReadHandler(
Roaring64NavigableMap realExceptBlockIds = RssUtils.cloneBitMap(request.getExpectBlockIds());
realExceptBlockIds.xor(request.getProcessBlockIds());
expectTaskIds = RssUtils.generateTaskIdBitMap(realExceptBlockIds, request.getIdHelper());
+ } else {
+ expectTaskIds = request.getExpectTaskIds();
Review Comment:
Does this harm legacy Spark / TEZ / MR read clients?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#1512] feature(Spark): Replace taskAttemptId with mapIndex in blockId [incubator-uniffle]
Posted by "EnricoMi (via GitHub)" <gi...@apache.org>.
EnricoMi commented on PR #1514:
URL: https://github.com/apache/incubator-uniffle/pull/1514#issuecomment-1943163836
@jerqi Do you mean the RSS Spark clients or Apache Spark? Can you point me to it? What do you suggest to do next?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#1512] feature(Spark): Replace taskAttemptId with mapIndex in blockId [incubator-uniffle]
Posted by "EnricoMi (via GitHub)" <gi...@apache.org>.
EnricoMi commented on PR #1514:
URL: https://github.com/apache/incubator-uniffle/pull/1514#issuecomment-1953752848
Superseded by #1529..
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#1512] feature(Spark): Replace taskAttemptId with mapIndex in blockId [incubator-uniffle]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #1514:
URL: https://github.com/apache/incubator-uniffle/pull/1514#issuecomment-1934891158
## [Codecov](https://app.codecov.io/gh/apache/incubator-uniffle/pull/1514?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
Attention: `3 lines` in your changes are missing coverage. Please review.
> Comparison is base [(`9471cba`)](https://app.codecov.io/gh/apache/incubator-uniffle/commit/9471cba436d0cec604a780faa01da5d1341fef5a?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) 54.23% compared to head [(`4722d4c`)](https://app.codecov.io/gh/apache/incubator-uniffle/pull/1514?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) 54.69%.
| [Files](https://app.codecov.io/gh/apache/incubator-uniffle/pull/1514?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...uniffle/storage/factory/ShuffleHandlerFactory.java](https://app.codecov.io/gh/apache/incubator-uniffle/pull/1514?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c3RvcmFnZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvdW5pZmZsZS9zdG9yYWdlL2ZhY3RvcnkvU2h1ZmZsZUhhbmRsZXJGYWN0b3J5LmphdmE=) | 0.00% | [2 Missing :warning: ](https://app.codecov.io/gh/apache/incubator-uniffle/pull/1514?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...rg/apache/uniffle/server/buffer/ShuffleBuffer.java](https://app.codecov.io/gh/apache/incubator-uniffle/pull/1514?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c2VydmVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS91bmlmZmxlL3NlcnZlci9idWZmZXIvU2h1ZmZsZUJ1ZmZlci5qYXZh) | 50.00% | [0 Missing and 1 partial :warning: ](https://app.codecov.io/gh/apache/incubator-uniffle/pull/1514?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #1514 +/- ##
============================================
+ Coverage 54.23% 54.69% +0.45%
+ Complexity 2810 2273 -537
============================================
Files 430 353 -77
Lines 24410 15807 -8603
Branches 2082 1451 -631
============================================
- Hits 13238 8645 -4593
+ Misses 10340 6669 -3671
+ Partials 832 493 -339
```
</details>
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/incubator-uniffle/pull/1514?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#1512] feature(Spark): Replace taskAttemptId with mapIndex in blockId [incubator-uniffle]
Posted by "EnricoMi (via GitHub)" <gi...@apache.org>.
EnricoMi commented on PR #1514:
URL: https://github.com/apache/incubator-uniffle/pull/1514#issuecomment-1947836048
> #1529 looks fine, can we add a integration test to cover the case of speculative exection?
See https://github.com/apache/incubator-uniffle/pull/1529#issuecomment-1947835597.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#1512] feature(Spark): Replace taskAttemptId with mapIndex in blockId [incubator-uniffle]
Posted by "EnricoMi (via GitHub)" <gi...@apache.org>.
EnricoMi commented on PR #1514:
URL: https://github.com/apache/incubator-uniffle/pull/1514#issuecomment-1947827148
> AttemptNo will waste some bits. If we increase the bits, the bitmap will occupy more memory.
See https://github.com/apache/incubator-uniffle/pull/1529#issuecomment-1947826787.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#1512] feature(Spark): Replace taskAttemptId with mapIndex in blockId [incubator-uniffle]
Posted by "zuston (via GitHub)" <gi...@apache.org>.
zuston commented on PR #1514:
URL: https://github.com/apache/incubator-uniffle/pull/1514#issuecomment-1947695463
Sorry for the late reply, we are in vocation for Chinese new year.
#1529 looks fine, can we add a integration test to cover the case of speculative exection?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#1512] feature(Spark): Replace taskAttemptId with mapIndex in blockId [incubator-uniffle]
Posted by "jerqi (via GitHub)" <gi...@apache.org>.
jerqi commented on PR #1514:
URL: https://github.com/apache/incubator-uniffle/pull/1514#issuecomment-1945328470
We can use TaskSchedulerImpl#nextTaskId to simplify the taskAttemptId.
1. We use Java reflection to get `TaskSchedulerImpl#nextTaskId` from SparkContext.
2. We record the start taskAttemptId in the ShuffleHandle.
3. We use taskAttemptId minus start taskAttemptId.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#1512] feature(Spark): Replace taskAttemptId with mapIndex in blockId [incubator-uniffle]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #1514:
URL: https://github.com/apache/incubator-uniffle/pull/1514#issuecomment-1935242691
## Test Results
2β284 filesβ β-β3ββ2β284 suitesβ β-β3βββ4h 12m 7s :stopwatch: - 16m 13s
ββ811 tests β-β8ββββ810 :white_check_mark: β-β8βββ1 :zzz: Β±0ββ0 :x: Β±0β
9β077 runsβ β-β9ββ9β064 :white_check_mark: β-β9ββ13 :zzz: Β±0ββ0 :x: Β±0β
Results for commit 4722d4c8.βΒ± Comparison against base commit 9471cba4.
<details>
<summary>This pull request <b>removes</b> 9 and <b>adds</b> 1 tests. <i>Note that renamed tests count towards both.</i></summary>
```
org.apache.uniffle.test.TezCartesianProductTest β cartesianProductTest
org.apache.uniffle.test.TezOrderedWordCountTest β orderedWordCountTest
org.apache.uniffle.test.TezSimpleSessionExampleTest β simpleSessionExampleTest
org.apache.uniffle.test.TezSortMergeJoinTest β sortMergeJoinTest
org.apache.uniffle.test.TezWordCountTest β wordCountTest
org.apache.uniffle.test.TezWordCountWithFailuresTest β wordCountTestWithNodeUnhealthyWhenAvoidRecomputeDisable
org.apache.uniffle.test.TezWordCountWithFailuresTest β wordCountTestWithNodeUnhealthyWhenAvoidRecomputeEnable
org.apache.uniffle.test.TezWordCountWithFailuresTest β wordCountTestWithTaskFailureWhenAvoidRecomputeDisable
org.apache.uniffle.test.TezWordCountWithFailuresTest β wordCountTestWithTaskFailureWhenAvoidRecomputeEnable
```
```
org.apache.uniffle.test.FailingTasksTest β testFailedTasks
```
</details>
[test-results]:data:application/gzip;base64,H4sIABuNxWUC/02MzQ6DIBAGX8Vw7gEWlKUv0wBiQupPg3Bq+u4FI8pxZr+dL5n87Hby7ABQPDqyJx9bHlPQ0W9rNqxnILPK91gWyFil156sPRS91dt/ytclJu3nLO6FC2ELpwlpLU1FpTypNhUdxKXOJq+iaR7cJu22LD5mIEICjMIiZXxAOroBFVcOe2aU5iC0AYqGGyC/PyumD4IQAQAA
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#1512] feature(Spark): Replace taskAttemptId with mapIndex in blockId [incubator-uniffle]
Posted by "EnricoMi (via GitHub)" <gi...@apache.org>.
EnricoMi closed pull request #1514: [#1512] feature(Spark): Replace taskAttemptId with mapIndex in blockId
URL: https://github.com/apache/incubator-uniffle/pull/1514
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#1512] feature(Spark): Replace taskAttemptId with mapIndex in blockId [incubator-uniffle]
Posted by "jerqi (via GitHub)" <gi...@apache.org>.
jerqi commented on PR #1514:
URL: https://github.com/apache/incubator-uniffle/pull/1514#issuecomment-1945329022
> @jerqi Do you mean the RSS Spark clients or Apache Spark? Can you point me to it? What do you suggest to do next?
I mean Apache Spark has own commit mechinanism.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#1512] feature(Spark): Replace taskAttemptId with mapIndex in blockId [incubator-uniffle]
Posted by "EnricoMi (via GitHub)" <gi...@apache.org>.
EnricoMi commented on PR #1514:
URL: https://github.com/apache/incubator-uniffle/pull/1514#issuecomment-1945642831
> We can use TaskSchedulerImpl#nextTaskId to simplify the taskAttemptId.
>
> - We use Java reflection to get TaskSchedulerImpl#nextTaskId from SparkContext.
> - We record the start taskAttemptId in the ShuffleHandle.
> - We use taskAttemptId minus start taskAttemptId.
Offsetting the taskAttemptId would help to reduce the ids for long running Spark jobs, but the offset ids can practically still be arbitrary large, e.g. if there are many consecutive large (with many partitions) stages.
What do you think about using the task id and the attempt no (bit-appended)? That id is unique for a stage and much more constraint than the taskAttemptId or the offset taskAttemptId.
> > @jerqi Do you mean the RSS Spark clients or Apache Spark? Can you point me to it? What do you suggest to do next?
>
> I mean Apache Spark has own commit mechinanism.
Do you refer to [ShuffleMapOutputWriter](https://spark.apache.org/docs/latest/api/java/org/apache/spark/shuffle/api/ShuffleMapOutputWriter.html)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#1512] feature(Spark): Replace taskAttemptId with mapIndex in blockId [incubator-uniffle]
Posted by "jerqi (via GitHub)" <gi...@apache.org>.
jerqi commented on PR #1514:
URL: https://github.com/apache/incubator-uniffle/pull/1514#issuecomment-1947743242
> > We can use TaskSchedulerImpl#nextTaskId to simplify the taskAttemptId.
> >
> > * We use Java reflection to get TaskSchedulerImpl#nextTaskId from SparkContext.
> > * We record the start taskAttemptId in the ShuffleHandle.
> > * We use taskAttemptId minus start taskAttemptId.
>
> Offsetting the taskAttemptId would help to reduce the ids for long running Spark jobs, but the offset ids can practically still be arbitrary large, e.g. if there are many consecutive large (with many partitions) stages.
>
> What do you think about using the task id and the attempt no (bit-appended)? That id is unique for a stage and much more constraint than the taskAttemptId or the offset taskAttemptId: #1529
>
> > > @jerqi Do you mean the RSS Spark clients or Apache Spark? Can you point me to it? What do you suggest to do next?
> >
> >
> > I mean Apache Spark has own commit mechinanism.
>
> Do you refer to [ShuffleMapOutputWriter](https://spark.apache.org/docs/latest/api/java/org/apache/spark/shuffle/api/ShuffleMapOutputWriter.html)?
AttemptNo will waste some bits. If we increase the bits, the bitmap will occupy more memory.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#1512] feature(Spark): Replace taskAttemptId with mapIndex in blockId [incubator-uniffle]
Posted by "EnricoMi (via GitHub)" <gi...@apache.org>.
EnricoMi commented on PR #1514:
URL: https://github.com/apache/incubator-uniffle/pull/1514#issuecomment-1938143098
@jerqi you are right, if both tasks report their shuffle results (block ids), we would not know how many blocks to read for our partition id and map index. For that to work we need to guarantee that only one task attempt succeeds in registering shuffle results (blockids). We then take that taskAttemptId and those blockIds for the partition id and map index.
This would require a two-phase commit to guard / coordinate the report shuffle result step, which would be a much bigger change: https://github.com/G-Research/incubator-uniffle/pull/4/files
There are only two situations, where this can happen, though: speculative task execution (which is not recommended for production) and (without speculative task execution) a task failure after shuffle results have been reported, which triggers another task attempt.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#1512] feature(Spark): Replace taskAttemptId with mapIndex in blockId [incubator-uniffle]
Posted by "zuston (via GitHub)" <gi...@apache.org>.
zuston commented on code in PR #1514:
URL: https://github.com/apache/incubator-uniffle/pull/1514#discussion_r1483766774
##########
integration-test/spark-common/src/test/java/org/apache/uniffle/test/FailingTasksTest.java:
##########
@@ -0,0 +1,89 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.uniffle.test;
+
+import java.util.Iterator;
+import java.util.Map;
+
+import com.google.common.collect.Maps;
+import org.apache.spark.TaskContext;
+import org.apache.spark.shuffle.RssSparkConfig;
+import org.apache.spark.sql.SparkSession;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.Test;
+
+import org.apache.uniffle.coordinator.CoordinatorConf;
+import org.apache.uniffle.server.ShuffleServerConf;
+import org.apache.uniffle.storage.util.StorageType;
+
+public class FailingTasksTest extends SparkIntegrationTestBase {
Review Comment:
This test case looks passed, which means the design looks effective. Please ping me if this is ready for review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org