You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@tez.apache.org by GitBox <gi...@apache.org> on 2022/03/15 19:00:03 UTC
[GitHub] [tez] ramesh0201 opened a new pull request #195: TEZ-4397 Open Tez Input splits asynchronously
ramesh0201 opened a new pull request #195:
URL: https://github.com/apache/tez/pull/195
Tez input splits can be opened asynchronously. This will reduce the amount of time spent for s3 to prepare the connection and opening the object
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@tez.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tez] tez-yetus commented on pull request #195: TEZ-4397 Open Tez Input splits asynchronously
Posted by GitBox <gi...@apache.org>.
tez-yetus commented on pull request #195:
URL: https://github.com/apache/tez/pull/195#issuecomment-1074796258
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 12m 52s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| -1 :x: | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. |
||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 14m 15s | master passed |
| +1 :green_heart: | compile | 0m 31s | master passed with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 0m 28s | master passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 0m 58s | master passed |
| +1 :green_heart: | javadoc | 0m 41s | master passed with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 0m 29s | master passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +0 :ok: | spotbugs | 1m 13s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 1m 11s | master passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 20s | the patch passed |
| +1 :green_heart: | compile | 0m 21s | the patch passed with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 0m 21s | the patch passed |
| +1 :green_heart: | compile | 0m 18s | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 0m 17s | the patch passed |
| -0 :warning: | checkstyle | 0m 12s | tez-mapreduce: The patch generated 6 new + 32 unchanged - 1 fixed = 38 total (was 33) |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | javadoc | 0m 17s | the patch passed with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 0m 15s | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | findbugs | 0m 44s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 22s | tez-mapreduce in the patch passed. |
| +1 :green_heart: | asflicense | 0m 13s | The patch does not generate ASF License warnings. |
| | | 36m 18s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-195/3/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/tez/pull/195 |
| Optional Tests | dupname asflicense javac javadoc unit spotbugs findbugs checkstyle compile |
| uname | Linux 77abc76b3997 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/tez.sh |
| git revision | master / 20873a3a1 |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| checkstyle | https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-195/3/artifact/out/diff-checkstyle-tez-mapreduce.txt |
| Test Results | https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-195/3/testReport/ |
| Max. process+thread count | 233 (vs. ulimit of 5500) |
| modules | C: tez-mapreduce U: tez-mapreduce |
| Console output | https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-195/3/console |
| versions | git=2.25.1 maven=3.6.3 findbugs=3.0.1 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@tez.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tez] tez-yetus commented on pull request #195: TEZ-4397 Open Tez Input splits asynchronously
Posted by GitBox <gi...@apache.org>.
tez-yetus commented on pull request #195:
URL: https://github.com/apache/tez/pull/195#issuecomment-1075755496
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 17m 33s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| -1 :x: | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. |
||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 13m 37s | master passed |
| +1 :green_heart: | compile | 0m 33s | master passed with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 0m 32s | master passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 1m 4s | master passed |
| +1 :green_heart: | javadoc | 0m 41s | master passed with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 0m 29s | master passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +0 :ok: | spotbugs | 1m 12s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 1m 11s | master passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 18s | the patch passed |
| +1 :green_heart: | compile | 0m 18s | the patch passed with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 0m 18s | the patch passed |
| +1 :green_heart: | compile | 0m 17s | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 0m 17s | the patch passed |
| -0 :warning: | checkstyle | 0m 12s | tez-mapreduce: The patch generated 6 new + 32 unchanged - 1 fixed = 38 total (was 33) |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | javadoc | 0m 17s | the patch passed with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 0m 14s | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | findbugs | 0m 42s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 21s | tez-mapreduce in the patch passed. |
| +1 :green_heart: | asflicense | 0m 14s | The patch does not generate ASF License warnings. |
| | | 40m 27s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-195/5/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/tez/pull/195 |
| Optional Tests | dupname asflicense javac javadoc unit spotbugs findbugs checkstyle compile |
| uname | Linux 48a183fc752d 4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17 17:06:04 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/tez.sh |
| git revision | master / 20873a3a1 |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| checkstyle | https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-195/5/artifact/out/diff-checkstyle-tez-mapreduce.txt |
| Test Results | https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-195/5/testReport/ |
| Max. process+thread count | 238 (vs. ulimit of 5500) |
| modules | C: tez-mapreduce U: tez-mapreduce |
| Console output | https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-195/5/console |
| versions | git=2.25.1 maven=3.6.3 findbugs=3.0.1 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@tez.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tez] tez-yetus commented on pull request #195: TEZ-4397 Open Tez Input splits asynchronously
Posted by GitBox <gi...@apache.org>.
tez-yetus commented on pull request #195:
URL: https://github.com/apache/tez/pull/195#issuecomment-1074797256
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 13m 16s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 1s | The patch does not contain any @author tags. |
| -1 :x: | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. |
||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 13m 55s | master passed |
| +1 :green_heart: | compile | 0m 33s | master passed with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 0m 32s | master passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 1m 2s | master passed |
| +1 :green_heart: | javadoc | 0m 41s | master passed with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 0m 29s | master passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +0 :ok: | spotbugs | 1m 11s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 1m 10s | master passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 18s | the patch passed |
| +1 :green_heart: | compile | 0m 19s | the patch passed with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 0m 19s | the patch passed |
| +1 :green_heart: | compile | 0m 17s | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 0m 17s | the patch passed |
| -0 :warning: | checkstyle | 0m 11s | tez-mapreduce: The patch generated 6 new + 32 unchanged - 1 fixed = 38 total (was 33) |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | javadoc | 0m 16s | the patch passed with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 0m 14s | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | findbugs | 0m 40s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 19s | tez-mapreduce in the patch passed. |
| +1 :green_heart: | asflicense | 0m 13s | The patch does not generate ASF License warnings. |
| | | 36m 20s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-195/4/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/tez/pull/195 |
| Optional Tests | dupname asflicense javac javadoc unit spotbugs findbugs checkstyle compile |
| uname | Linux 77a7605771ad 4.15.0-169-generic #177-Ubuntu SMP Thu Feb 3 10:50:38 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/tez.sh |
| git revision | master / 20873a3a1 |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| checkstyle | https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-195/4/artifact/out/diff-checkstyle-tez-mapreduce.txt |
| Test Results | https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-195/4/testReport/ |
| Max. process+thread count | 238 (vs. ulimit of 5500) |
| modules | C: tez-mapreduce U: tez-mapreduce |
| Console output | https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-195/4/console |
| versions | git=2.25.1 maven=3.6.3 findbugs=3.0.1 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@tez.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tez] rbalamohan commented on a change in pull request #195: TEZ-4397 Open Tez Input splits asynchronously
Posted by GitBox <gi...@apache.org>.
rbalamohan commented on a change in pull request #195:
URL: https://github.com/apache/tez/pull/195#discussion_r827593274
##########
File path: tez-mapreduce/src/main/java/org/apache/hadoop/mapred/split/TezGroupedSplitsInputFormat.java
##########
@@ -129,14 +137,49 @@ void initInputFormatFromSplit(TezGroupedSplit split) throws TezException {
int idx = 0;
long progress;
RecordReader<K, V> curReader;
-
+ AtomicInteger initIndex;
+ ExecutorService initReaderExecService;
+ Queue<Future<RecordReader<K,V>>> initedReaders;
+
public TezGroupedSplitsRecordReader(TezGroupedSplit split, JobConf job,
Reporter reporter) throws IOException {
this.groupedSplit = split;
this.job = job;
this.reporter = reporter;
+ this.initIndex = new AtomicInteger(0);
+ int numThreads = conf.getInt(TezSplitGrouper.TEZ_GROUPING_SPLIT_INIT_THREADS,
+ TezSplitGrouper.TEZ_GROUPING_SPLIT_INIT_THREADS_DEFAULT);
+ int numReaders = conf.getInt(TezSplitGrouper.TEZ_GROUPING_SPLIT_INIT_NUM_RECORDREADERS,
+ TezSplitGrouper.TEZ_GROUPING_SPLIT_INIT_NUM_RECORDREADERS_DEFAULT);
+ this.initReaderExecService = Executors.newFixedThreadPool(numThreads,
+ new ThreadFactoryBuilder()
+ .setDaemon(true)
+ .setPriority(Thread.MAX_PRIORITY)
+ .setNameFormat("TEZ-Split-Init-Thread-%d")
+ .build());
+ this.initedReaders = new LinkedList<>();
+ submitInitReaders(numReaders);
initNextRecordReader();
}
+
+ private void submitInitReaders(int numReaders) {
+ for (int i = 0; i < numReaders; i++) {
+ initedReaders.offer(this.initReaderExecService.submit(() -> {
+ try {
+ int index = initIndex.getAndIncrement();
+ if (index >= groupedSplit.wrappedSplits.size()) {
+ return null;
+ }
+ InputSplit s = groupedSplit.wrappedSplits.get(index);
+ RecordReader<K, V> reader = wrappedInputFormat.getRecordReader(s, job, reporter);
+ LOG.debug("Init Thread processed reader number {} initialization", index);
+ return reader;
+ } catch(IOException e) {
Review comment:
If thread is interrupted, it should cancel other pending tasks?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@tez.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tez] tez-yetus commented on pull request #195: TEZ-4397 Open Tez Input splits asynchronously
Posted by GitBox <gi...@apache.org>.
tez-yetus commented on pull request #195:
URL: https://github.com/apache/tez/pull/195#issuecomment-1068393849
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 17m 40s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| -1 :x: | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. |
||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 13m 43s | master passed |
| +1 :green_heart: | compile | 0m 32s | master passed with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 0m 31s | master passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 1m 3s | master passed |
| +1 :green_heart: | javadoc | 0m 40s | master passed with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 0m 28s | master passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +0 :ok: | spotbugs | 1m 14s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 1m 11s | master passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 18s | the patch passed |
| +1 :green_heart: | compile | 0m 19s | the patch passed with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 0m 19s | the patch passed |
| +1 :green_heart: | compile | 0m 18s | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 0m 18s | the patch passed |
| -0 :warning: | checkstyle | 0m 12s | tez-mapreduce: The patch generated 4 new + 33 unchanged - 0 fixed = 37 total (was 33) |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | javadoc | 0m 16s | the patch passed with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 0m 14s | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | findbugs | 0m 42s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 21s | tez-mapreduce in the patch passed. |
| +1 :green_heart: | asflicense | 0m 15s | The patch does not generate ASF License warnings. |
| | | 40m 39s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-195/1/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/tez/pull/195 |
| Optional Tests | dupname asflicense javac javadoc unit spotbugs findbugs checkstyle compile |
| uname | Linux 0f0ec404e815 4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17 17:06:04 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/tez.sh |
| git revision | master / 132ea4c1b |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| checkstyle | https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-195/1/artifact/out/diff-checkstyle-tez-mapreduce.txt |
| Test Results | https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-195/1/testReport/ |
| Max. process+thread count | 238 (vs. ulimit of 5500) |
| modules | C: tez-mapreduce U: tez-mapreduce |
| Console output | https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-195/1/console |
| versions | git=2.25.1 maven=3.6.3 findbugs=3.0.1 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@tez.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tez] rbalamohan commented on a change in pull request #195: TEZ-4397 Open Tez Input splits asynchronously
Posted by GitBox <gi...@apache.org>.
rbalamohan commented on a change in pull request #195:
URL: https://github.com/apache/tez/pull/195#discussion_r827480975
##########
File path: tez-mapreduce/src/main/java/org/apache/hadoop/mapred/split/TezGroupedSplitsInputFormat.java
##########
@@ -193,8 +236,8 @@ protected boolean initNextRecordReader() throws IOException {
// get a record reader for the idx-th chunk
try {
- curReader = wrappedInputFormat.getRecordReader(
- groupedSplit.wrappedSplits.get(idx), job, reporter);
+ curReader = initedReaders.poll().get();
+ submitInitReaders(1);
Review comment:
From the next stage onwards (i.e after the init part), this will start executing in sequential order as it requests for 1 additional reader. It will be good to init next set of readers in parallel.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@tez.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tez] jfsii commented on a change in pull request #195: TEZ-4397 Open Tez Input splits asynchronously
Posted by GitBox <gi...@apache.org>.
jfsii commented on a change in pull request #195:
URL: https://github.com/apache/tez/pull/195#discussion_r829370709
##########
File path: tez-mapreduce/src/main/java/org/apache/hadoop/mapred/split/TezGroupedSplitsInputFormat.java
##########
@@ -129,14 +138,50 @@ void initInputFormatFromSplit(TezGroupedSplit split) throws TezException {
int idx = 0;
long progress;
RecordReader<K, V> curReader;
-
+ AtomicInteger initIndex;
+ int numReaders;
+ ExecutorService initReaderExecService;
+ Queue<Future<RecordReader<K,V>>> initedReaders;
+
public TezGroupedSplitsRecordReader(TezGroupedSplit split, JobConf job,
Reporter reporter) throws IOException {
this.groupedSplit = split;
this.job = job;
this.reporter = reporter;
+ this.initIndex = new AtomicInteger(0);
+ int numThreads = conf.getInt(TezSplitGrouper.TEZ_GROUPING_SPLIT_INIT_THREADS,
+ TezSplitGrouper.TEZ_GROUPING_SPLIT_INIT_THREADS_DEFAULT);
+ this.numReaders = conf.getInt(TezSplitGrouper.TEZ_GROUPING_SPLIT_INIT_NUM_RECORDREADERS,
+ TezSplitGrouper.TEZ_GROUPING_SPLIT_INIT_NUM_RECORDREADERS_DEFAULT);
+ this.initReaderExecService = Executors.newFixedThreadPool(numThreads,
Review comment:
I think we will want to somehow make this static (to share between usages of TezGroupedSplitsRecordReader) or figure out how to call shutdown in a reliable manner. Otherwise I think a long lived process that uses multiple TezGroupedSplitsRecordReaders throughout its life will end up having a large number of unused threads since they will not auto shutdown and thus not garbage collected.
https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/ThreadPoolExecutor.html
See finalization section.
##########
File path: tez-mapreduce/src/main/java/org/apache/hadoop/mapred/split/TezGroupedSplitsInputFormat.java
##########
@@ -193,10 +238,16 @@ protected boolean initNextRecordReader() throws IOException {
// get a record reader for the idx-th chunk
try {
- curReader = wrappedInputFormat.getRecordReader(
- groupedSplit.wrappedSplits.get(idx), job, reporter);
- } catch (Exception e) {
- throw new RuntimeException (e);
+ curReader = initedReaders.poll().get();
+ preInitReaders();
+ } catch (ExecutionException e) {
+ throw new RuntimeException(e);
+ } catch (InterruptedException e) {
+ Thread.currentThread().interrupt();
+ for (Future<RecordReader<K,V>> f : initedReaders) {
Review comment:
probably want to cancel on any exception and thread interrupt only on InterruptedException.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@tez.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tez] abstractdog commented on a change in pull request #195: TEZ-4397 Open Tez Input splits asynchronously
Posted by GitBox <gi...@apache.org>.
abstractdog commented on a change in pull request #195:
URL: https://github.com/apache/tez/pull/195#discussion_r830032549
##########
File path: tez-mapreduce/src/main/java/org/apache/hadoop/mapred/split/TezGroupedSplitsInputFormat.java
##########
@@ -129,14 +138,50 @@ void initInputFormatFromSplit(TezGroupedSplit split) throws TezException {
int idx = 0;
long progress;
RecordReader<K, V> curReader;
-
+ AtomicInteger initIndex;
+ int numReaders;
+ ExecutorService initReaderExecService;
+ Queue<Future<RecordReader<K,V>>> initedReaders;
+
public TezGroupedSplitsRecordReader(TezGroupedSplit split, JobConf job,
Reporter reporter) throws IOException {
this.groupedSplit = split;
this.job = job;
this.reporter = reporter;
+ this.initIndex = new AtomicInteger(0);
+ int numThreads = conf.getInt(TezSplitGrouper.TEZ_GROUPING_SPLIT_INIT_THREADS,
+ TezSplitGrouper.TEZ_GROUPING_SPLIT_INIT_THREADS_DEFAULT);
+ this.numReaders = conf.getInt(TezSplitGrouper.TEZ_GROUPING_SPLIT_INIT_NUM_RECORDREADERS,
+ TezSplitGrouper.TEZ_GROUPING_SPLIT_INIT_NUM_RECORDREADERS_DEFAULT);
+ this.initReaderExecService = Executors.newFixedThreadPool(numThreads,
Review comment:
I'm not sure about the context, but if the goal is to properly shutdown a global resource somehow, it can done by the shutdownhandler, somewhere here: https://github.com/apache/tez/blob/master/tez-dag/src/main/java/org/apache/tez/dag/app/DAGAppMaster.java#L954
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@tez.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tez] ramesh0201 commented on pull request #195: TEZ-4397 Open Tez Input splits asynchronously
Posted by GitBox <gi...@apache.org>.
ramesh0201 commented on pull request #195:
URL: https://github.com/apache/tez/pull/195#issuecomment-1075742304
Thanks @rbalamohan @jfsii and @abstractdog for the reviews.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@tez.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tez] abstractdog commented on a change in pull request #195: TEZ-4397 Open Tez Input splits asynchronously
Posted by GitBox <gi...@apache.org>.
abstractdog commented on a change in pull request #195:
URL: https://github.com/apache/tez/pull/195#discussion_r830032549
##########
File path: tez-mapreduce/src/main/java/org/apache/hadoop/mapred/split/TezGroupedSplitsInputFormat.java
##########
@@ -129,14 +138,50 @@ void initInputFormatFromSplit(TezGroupedSplit split) throws TezException {
int idx = 0;
long progress;
RecordReader<K, V> curReader;
-
+ AtomicInteger initIndex;
+ int numReaders;
+ ExecutorService initReaderExecService;
+ Queue<Future<RecordReader<K,V>>> initedReaders;
+
public TezGroupedSplitsRecordReader(TezGroupedSplit split, JobConf job,
Reporter reporter) throws IOException {
this.groupedSplit = split;
this.job = job;
this.reporter = reporter;
+ this.initIndex = new AtomicInteger(0);
+ int numThreads = conf.getInt(TezSplitGrouper.TEZ_GROUPING_SPLIT_INIT_THREADS,
+ TezSplitGrouper.TEZ_GROUPING_SPLIT_INIT_THREADS_DEFAULT);
+ this.numReaders = conf.getInt(TezSplitGrouper.TEZ_GROUPING_SPLIT_INIT_NUM_RECORDREADERS,
+ TezSplitGrouper.TEZ_GROUPING_SPLIT_INIT_NUM_RECORDREADERS_DEFAULT);
+ this.initReaderExecService = Executors.newFixedThreadPool(numThreads,
Review comment:
I'm not sure about the context, but if the goal is to properly shutdown a global resource somehow, it can done by the shutdownhandler, somewhere here: https://github.com/apache/tez/blob/master/tez-dag/src/main/java/org/apache/tez/dag/app/DAGAppMaster.java#L954
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@tez.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tez] tez-yetus commented on pull request #195: TEZ-4397 Open Tez Input splits asynchronously
Posted by GitBox <gi...@apache.org>.
tez-yetus commented on pull request #195:
URL: https://github.com/apache/tez/pull/195#issuecomment-1070292788
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 10s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| -1 :x: | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. |
||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 13m 37s | master passed |
| +1 :green_heart: | compile | 0m 32s | master passed with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 0m 31s | master passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 1m 2s | master passed |
| +1 :green_heart: | javadoc | 0m 42s | master passed with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 0m 28s | master passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +0 :ok: | spotbugs | 1m 14s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 1m 12s | master passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 19s | the patch passed |
| +1 :green_heart: | compile | 0m 19s | the patch passed with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 0m 19s | the patch passed |
| +1 :green_heart: | compile | 0m 18s | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 0m 18s | the patch passed |
| -0 :warning: | checkstyle | 0m 11s | tez-mapreduce: The patch generated 6 new + 32 unchanged - 1 fixed = 38 total (was 33) |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | javadoc | 0m 16s | the patch passed with JDK Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 0m 14s | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | findbugs | 0m 43s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 1m 22s | tez-mapreduce in the patch passed. |
| +1 :green_heart: | asflicense | 0m 14s | The patch does not generate ASF License warnings. |
| | | 24m 1s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-195/2/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/tez/pull/195 |
| Optional Tests | dupname asflicense javac javadoc unit spotbugs findbugs checkstyle compile |
| uname | Linux 65f01d64be1f 4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17 17:06:04 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/tez.sh |
| git revision | master / 20873a3a1 |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| checkstyle | https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-195/2/artifact/out/diff-checkstyle-tez-mapreduce.txt |
| Test Results | https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-195/2/testReport/ |
| Max. process+thread count | 233 (vs. ulimit of 5500) |
| modules | C: tez-mapreduce U: tez-mapreduce |
| Console output | https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-195/2/console |
| versions | git=2.25.1 maven=3.6.3 findbugs=3.0.1 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@tez.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tez] jfsii commented on a change in pull request #195: TEZ-4397 Open Tez Input splits asynchronously
Posted by GitBox <gi...@apache.org>.
jfsii commented on a change in pull request #195:
URL: https://github.com/apache/tez/pull/195#discussion_r829370709
##########
File path: tez-mapreduce/src/main/java/org/apache/hadoop/mapred/split/TezGroupedSplitsInputFormat.java
##########
@@ -129,14 +138,50 @@ void initInputFormatFromSplit(TezGroupedSplit split) throws TezException {
int idx = 0;
long progress;
RecordReader<K, V> curReader;
-
+ AtomicInteger initIndex;
+ int numReaders;
+ ExecutorService initReaderExecService;
+ Queue<Future<RecordReader<K,V>>> initedReaders;
+
public TezGroupedSplitsRecordReader(TezGroupedSplit split, JobConf job,
Reporter reporter) throws IOException {
this.groupedSplit = split;
this.job = job;
this.reporter = reporter;
+ this.initIndex = new AtomicInteger(0);
+ int numThreads = conf.getInt(TezSplitGrouper.TEZ_GROUPING_SPLIT_INIT_THREADS,
+ TezSplitGrouper.TEZ_GROUPING_SPLIT_INIT_THREADS_DEFAULT);
+ this.numReaders = conf.getInt(TezSplitGrouper.TEZ_GROUPING_SPLIT_INIT_NUM_RECORDREADERS,
+ TezSplitGrouper.TEZ_GROUPING_SPLIT_INIT_NUM_RECORDREADERS_DEFAULT);
+ this.initReaderExecService = Executors.newFixedThreadPool(numThreads,
Review comment:
I think we will want to somehow make this static (to share between usages of TezGroupedSplitsRecordReader) or figure out how to call shutdown in a reliable manner. Otherwise I think a long lived process that uses multiple TezGroupedSplitsRecordReaders throughout its life will end up having a large number of unused threads since they will not auto shutdown and thus not garbage collected.
https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/ThreadPoolExecutor.html
See finalization section.
##########
File path: tez-mapreduce/src/main/java/org/apache/hadoop/mapred/split/TezGroupedSplitsInputFormat.java
##########
@@ -193,10 +238,16 @@ protected boolean initNextRecordReader() throws IOException {
// get a record reader for the idx-th chunk
try {
- curReader = wrappedInputFormat.getRecordReader(
- groupedSplit.wrappedSplits.get(idx), job, reporter);
- } catch (Exception e) {
- throw new RuntimeException (e);
+ curReader = initedReaders.poll().get();
+ preInitReaders();
+ } catch (ExecutionException e) {
+ throw new RuntimeException(e);
+ } catch (InterruptedException e) {
+ Thread.currentThread().interrupt();
+ for (Future<RecordReader<K,V>> f : initedReaders) {
Review comment:
probably want to cancel on any exception and thread interrupt only on InterruptedException.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@tez.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tez] rbalamohan merged pull request #195: TEZ-4397 Open Tez Input splits asynchronously
Posted by GitBox <gi...@apache.org>.
rbalamohan merged pull request #195:
URL: https://github.com/apache/tez/pull/195
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@tez.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tez] rbalamohan commented on pull request #195: TEZ-4397 Open Tez Input splits asynchronously
Posted by GitBox <gi...@apache.org>.
rbalamohan commented on pull request #195:
URL: https://github.com/apache/tez/pull/195#issuecomment-1076202185
Latest patch LGTM. +1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@tez.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tez] rbalamohan commented on a change in pull request #195: TEZ-4397 Open Tez Input splits asynchronously
Posted by GitBox <gi...@apache.org>.
rbalamohan commented on a change in pull request #195:
URL: https://github.com/apache/tez/pull/195#discussion_r830637127
##########
File path: tez-mapreduce/src/main/java/org/apache/hadoop/mapred/split/TezGroupedSplitsInputFormat.java
##########
@@ -129,14 +137,49 @@ void initInputFormatFromSplit(TezGroupedSplit split) throws TezException {
int idx = 0;
long progress;
RecordReader<K, V> curReader;
-
+ AtomicInteger initIndex;
+ ExecutorService initReaderExecService;
+ Queue<Future<RecordReader<K,V>>> initedReaders;
+
public TezGroupedSplitsRecordReader(TezGroupedSplit split, JobConf job,
Reporter reporter) throws IOException {
this.groupedSplit = split;
this.job = job;
this.reporter = reporter;
+ this.initIndex = new AtomicInteger(0);
+ int numThreads = conf.getInt(TezSplitGrouper.TEZ_GROUPING_SPLIT_INIT_THREADS,
+ TezSplitGrouper.TEZ_GROUPING_SPLIT_INIT_THREADS_DEFAULT);
+ int numReaders = conf.getInt(TezSplitGrouper.TEZ_GROUPING_SPLIT_INIT_NUM_RECORDREADERS,
+ TezSplitGrouper.TEZ_GROUPING_SPLIT_INIT_NUM_RECORDREADERS_DEFAULT);
+ this.initReaderExecService = Executors.newFixedThreadPool(numThreads,
+ new ThreadFactoryBuilder()
+ .setDaemon(true)
+ .setPriority(Thread.MAX_PRIORITY)
+ .setNameFormat("TEZ-Split-Init-Thread-%d")
+ .build());
+ this.initedReaders = new LinkedList<>();
+ submitInitReaders(numReaders);
initNextRecordReader();
}
+
+ private void submitInitReaders(int numReaders) {
+ for (int i = 0; i < numReaders; i++) {
+ initedReaders.offer(this.initReaderExecService.submit(() -> {
+ try {
+ int index = initIndex.getAndIncrement();
+ if (index >= groupedSplit.wrappedSplits.size()) {
+ return null;
+ }
+ InputSplit s = groupedSplit.wrappedSplits.get(index);
+ RecordReader<K, V> reader = wrappedInputFormat.getRecordReader(s, job, reporter);
+ LOG.debug("Init Thread processed reader number {} initialization", index);
+ return reader;
+ } catch(IOException e) {
Review comment:
This should be done for all exceptions as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@tez.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org