You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@tinkerpop.apache.org by pauljackson <gi...@git.apache.org> on 2016/10/13 02:48:14 UTC
[GitHub] tinkerpop pull request #457: TINKERPOP-1493 Groovy project doesn't build on ...
GitHub user pauljackson opened a pull request:
https://github.com/apache/tinkerpop/pull/457
TINKERPOP-1493 Groovy project doesn't build on Windows
Removed support for user.dir property as it was being prepended to a
fully qualified path and the second drive letter was making the path
illegal.
Made sure JarFile instances were being closed so that Groovy could
delete the directory without encountering file locked errors.
Exclude Unix scripts from RAT plugin
Tinkergraph integration tests fail but can build when skipping
integration tests. Hadoop fails either way. Could be environmental on my
end.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/pauljackson/tinkerpop tp31
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/tinkerpop/pull/457.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #457
----
commit 47da7536a207610024d0497167ae7e800739d23b
Author: PaulJackson123 <pa...@verizon.net>
Date: 2016-10-13T02:41:17Z
TINKERPOP-1493 Groovy project doesn't build on Windows
Removed support for user.dir property as it was being prepended to a
fully qualified path and the second drive letter was making the path
illegal.
Made sure JarFile instances were being closed so that Groovy could
delete the directory without encountering file locked errors.
Exclude Unix scripts from RAT plugin
Tinkergraph integration tests fail but can build when skipping
integration tests. Hadoop fails either way. Could be environmental on my
end.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] tinkerpop issue #457: TINKERPOP-1493 Groovy project doesn't build on Windows
Posted by spmallette <gi...@git.apache.org>.
Github user spmallette commented on the issue:
https://github.com/apache/tinkerpop/pull/457
Well - this isn't a rebase - it is a merge. See the ugly merge commit here: d49f692f890d1c868f9de674901e14c34e7c20ba
For further explanation:
```text
D:\Data\TEMP\@Other\tinkerpop>git push origin tp31
To https://github.com/pauljackson/tinkerpop.git
! [rejected] tp31 -> tp31 (non-fast-forward)
error: failed to push some refs to 'https://github.com/pauljackson/tinkerpop.git'
hint: Updates were rejected because the tip of your current branch is behind
hint: its remote counterpart. Integrate the remote changes (e.g.
hint: 'git pull ...') before pushing again.
hint: See the 'Note about fast-forwards' in 'git push --help' for details.
```
So you had it right up to here but then probably got rattled by the message above and decided to then do the following:
```text
D:\Data\TEMP\@Other\tinkerpop>git pull
Merge made by the 'recursive' strategy.
```
which will do a `git merge` behind the scenes. Your `git push` command should have been:
```text
git push origin tp31 --force
```
The `--force` is required because you have no re-written the git history for `tp31` (i.e. you moved your commit forward to the head of `tp31`) and the server rejected it - "force" is a way of overriding that safeguard. So - anyway - this could be fixed, but i'm not going to worry about it. If only two commits come over in the merge (i.e. your original work and the merge commit) then I suppose that this is fine to merge.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] tinkerpop issue #457: TINKERPOP-1493 Groovy project doesn't build on Windows
Posted by dkuppitz <gi...@git.apache.org>.
Github user dkuppitz commented on the issue:
https://github.com/apache/tinkerpop/pull/457
I just started a fresh build. Regarding the stacktraces: Wasn't the `mkdirs()` errors related to your outdated Docker version?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] tinkerpop issue #457: TINKERPOP-1493 Groovy project doesn't build on Windows
Posted by pauljackson <gi...@git.apache.org>.
Github user pauljackson commented on the issue:
https://github.com/apache/tinkerpop/pull/457
It didn\u2019t look intuitively correct, but I didn\u2019t know what was normal. What\u2019s the easiest way to make this right, right, right? Just start over and create a new pull request?
Thanks,
-Paul
From: Robert Dale [mailto:notifications@github.com]
Sent: Sunday, November 27, 2016 8:41 AM
To: apache/tinkerpop
Cc: Paul A. Jackson; Mention
Subject: Re: [apache/tinkerpop] TINKERPOP-1493 Groovy project doesn't build on Windows (#457)
I didn't consider that your remote branch was behind. Duh. What should have been done is a forced push git push -f origin tp31 to replace your remote branch with your local branch. Instead git ended up doing a merge and the history here looks a little weird because it pulled in all the changes since your changes. But git does the right thing and it merges cleanly on tp31.
Build passes on Linux. Can't test on windows.
VOTE: +0.5
\u2014
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub<https://github.com/apache/tinkerpop/pull/457#issuecomment-263122753>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AAUcSGfiXS20htPRP2UbTre71Si2zbQQks5rCYhqgaJpZM4KVbcO>.
________________________________
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] tinkerpop issue #457: TINKERPOP-1493 Groovy project doesn't build on Windows
Posted by dkuppitz <gi...@git.apache.org>.
Github user dkuppitz commented on the issue:
https://github.com/apache/tinkerpop/pull/457
All good here, `docker/build.sh -t -i -n` succeeded 2 times in a row for me.
VOTE: +1
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] tinkerpop issue #457: TINKERPOP-1493 Groovy project doesn't build on Windows
Posted by spmallette <gi...@git.apache.org>.
Github user spmallette commented on the issue:
https://github.com/apache/tinkerpop/pull/457
```text
$ docker --version
Docker version 1.12.3, build 6b644ec
$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 14.04.5 LTS
Release: 14.04
Codename: trusty
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] tinkerpop issue #457: TINKERPOP-1493 Groovy project doesn't build on Windows
Posted by spmallette <gi...@git.apache.org>.
Github user spmallette commented on the issue:
https://github.com/apache/tinkerpop/pull/457
i didn't try with just `-t` - pretty sure it is just integration tests though. i found what i think is a fix to my problem though - i removed all my docker images and had them rebuild and i just got my first good build on docker with `-t -i` in a long time. i should be back to running it on this PR pretty soon. i have two separate systems doing builds right now on two different branches. hopefully they will both pass given the rebuilding of docker images.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] tinkerpop issue #457: TINKERPOP-1493 Groovy project doesn't build on Windows
Posted by pauljackson <gi...@git.apache.org>.
Github user pauljackson commented on the issue:
https://github.com/apache/tinkerpop/pull/457
Thank you Robert. This was adequate, I think, to give me some things to Google.
I\u2019ve run these commands from my local clone of my fork. It looks like that was all that was necessary. No new pull request. When I look at the status of the old pull request in the main repo it has incorporated 24 other commits since Oct 12 and the status of the request is that the Travis build is pending. Here\u2019s the commands, if someone would be kind to review and confirm. Thanks. \u2013Paul
D:\Data\TEMP\@Other\tinkerpop>git fetch --all
Fetching origin
Fetching apache
remote: Counting objects: 8094, done.
remote: Compressing objects: 100% (823/823), done.
remote: Total 8094 (delta 1986), reused 1680 (delta 1678), pack-reused 5064
Receiving objects: 100% (8094/8094), 11.66 MiB | 1.53 MiB/s, done.
Resolving deltas: 100% (3030/3030), completed with 560 local objects.
From https://github.com/apache/tinkerpop
* [new branch] TINKERPOP-1211 -> apache/TINKERPOP-1211
* [new branch] TINKERPOP-1235 -> apache/TINKERPOP-1235
* [new branch] TINKERPOP-1292 -> apache/TINKERPOP-1292
* [new branch] TINKERPOP-1363 -> apache/TINKERPOP-1363
* [new branch] TINKERPOP-1372 -> apache/TINKERPOP-1372
+ 302df7c...69877a2 TINKERPOP-1389 -> apache/TINKERPOP-1389 (forced update)
* [new branch] TINKERPOP-1399 -> apache/TINKERPOP-1399
* [new branch] TINKERPOP-1420 -> apache/TINKERPOP-1420
* [new branch] TINKERPOP-1428 -> apache/TINKERPOP-1428
* [new branch] TINKERPOP-1434 -> apache/TINKERPOP-1434
* [new branch] TINKERPOP-1471 -> apache/TINKERPOP-1471
* [new branch] TINKERPOP-1473 -> apache/TINKERPOP-1473
* [new branch] TINKERPOP-1485 -> apache/TINKERPOP-1485
* [new branch] TINKERPOP-1490 -> apache/TINKERPOP-1490
* [new branch] TINKERPOP-1502 -> apache/TINKERPOP-1502
* [new branch] TINKERPOP-1506 -> apache/TINKERPOP-1506
* [new branch] TINKERPOP-1507 -> apache/TINKERPOP-1507
* [new branch] TINKERPOP-1508 -> apache/TINKERPOP-1508
* [new branch] TINKERPOP-1520 -> apache/TINKERPOP-1520
* [new branch] TINKERPOP-1527 -> apache/TINKERPOP-1527
* [new branch] TINKERPOP-1529 -> apache/TINKERPOP-1529
* [new branch] TINKERPOP-1529-variant2 -> apache/TINKERPOP-1529-variant2
* [new branch] TINKERPOP-1530 -> apache/TINKERPOP-1530
* [new branch] TINKERPOP-1534 -> apache/TINKERPOP-1534
* [new branch] TINKERPOP-1538 -> apache/TINKERPOP-1538
* [new branch] TINKERPOP-1538-master -> apache/TINKERPOP-1538-master
* [new branch] TINKERPOP-1538-tp32 -> apache/TINKERPOP-1538-tp32
* [new branch] TINKERPOP-1541 -> apache/TINKERPOP-1541
* [new branch] TINKERPOP-1542 -> apache/TINKERPOP-1542
* [new branch] TINKERPOP-1543 -> apache/TINKERPOP-1543
* [new branch] TINKERPOP-1545 -> apache/TINKERPOP-1545
* [new branch] TINKERPOP-1545-tp32 -> apache/TINKERPOP-1545-tp32
* [new branch] TINKERPOP-1547 -> apache/TINKERPOP-1547
* [new branch] TINKERPOP-1548 -> apache/TINKERPOP-1548
* [new branch] TINKERPOP-1557 -> apache/TINKERPOP-1557
* [new branch] TINKERPOP-1561 -> apache/TINKERPOP-1561
* [new branch] TINKERPOP-1562 -> apache/TINKERPOP-1562
* [new branch] TINKERPOP-1564 -> apache/TINKERPOP-1564
* [new branch] TINKERPOP-919 -> apache/TINKERPOP-919
* [new branch] TINKERPOP-932 -> apache/TINKERPOP-932
* [new branch] example_python_promise -> apache/example_python_promise
227b85c..16180e1 master -> apache/master
4a4c526..4745fe1 tp31 -> apache/tp31
* [new branch] tp32 -> apache/tp32
* [new tag] 3.1.5 -> 3.1.5
* [new tag] 3.2.3 -> 3.2.3
D:\Data\TEMP\@Other\tinkerpop>git checkout tp31
Already on 'tp31'
Your branch is up-to-date with 'origin/tp31'.
D:\Data\TEMP\@Other\tinkerpop>git rebase upstream/tp31
fatal: Needed a single revision
invalid upstream upstream/tp31
D:\Data\TEMP\@Other\tinkerpop>git status
On branch tp31
Your branch is up-to-date with 'origin/tp31'.
nothing to commit, working tree clean
D:\Data\TEMP\@Other\tinkerpop>git remote -v
apache https://github.com/apache/tinkerpop.git (fetch)
apache https://github.com/apache/tinkerpop.git (push)
origin https://github.com/pauljackson/tinkerpop.git (fetch)
origin https://github.com/pauljackson/tinkerpop.git (push)
D:\Data\TEMP\@Other\tinkerpop>git rebase apache/tp31
First, rewinding head to replay your work on top of it...
Applying: TINKERPOP-1493 Groovy project doesn't build on Windows
D:\Data\TEMP\@Other\tinkerpop>git push origin tp31
To https://github.com/pauljackson/tinkerpop.git
! [rejected] tp31 -> tp31 (non-fast-forward)
error: failed to push some refs to 'https://github.com/pauljackson/tinkerpop.git'
hint: Updates were rejected because the tip of your current branch is behind
hint: its remote counterpart. Integrate the remote changes (e.g.
hint: 'git pull ...') before pushing again.
hint: See the 'Note about fast-forwards' in 'git push --help' for details.
D:\Data\TEMP\@Other\tinkerpop>git pull
Merge made by the 'recursive' strategy.
D:\Data\TEMP\@Other\tinkerpop>git push origin tp31
Counting objects: 371, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (174/174), done.
Writing objects: 100% (371/371), 65.98 KiB | 0 bytes/s, done.
Total 371 (delta 174), reused 306 (delta 126)
remote: Resolving deltas: 100% (174/174), completed with 68 local objects.
To https://github.com/pauljackson/tinkerpop.git
47da753..d49f692 tp31 -> tp31
D:\Data\TEMP\@Other\tinkerpop>
From: Robert Dale [mailto:notifications@github.com]
Sent: Wednesday, November 23, 2016 9:57 AM
To: apache/tinkerpop
Cc: Paul A. Jackson; Mention
Subject: Re: [apache/tinkerpop] TINKERPOP-1493 Groovy project doesn't build on Windows (#457)
Either one appears to work cleanly. Just update first - git fetch --all.
Obviously, make sure you're on your branch git checkout tp31.
Then either git rebase upstream/tp31 or git merge upstream/tp31.
Finally, push it back to your repo. git push or git push origin tp31 or however it's named.
\u2014
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub<https://github.com/apache/tinkerpop/pull/457#issuecomment-262536090>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AAUcSBxnAokrl0MzVFnP5hDC5pN3O6sCks5rBFQ8gaJpZM4KVbcO>.
________________________________
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] tinkerpop issue #457: TINKERPOP-1493 Groovy project doesn't build on Windows
Posted by pauljackson <gi...@git.apache.org>.
Github user pauljackson commented on the issue:
https://github.com/apache/tinkerpop/pull/457
Looking at the Hadoop code gives no direct clue (to me), but it does seem to indicate that you are not using the 2.7.2 version, which might explain the inconsistent results. Maybe your local maven repo has the wrong version?
The stack trace ends with:
Caused by: java.io.IOException: Mkdirs failed to create /usr/src/tinkerpop/spark-gremlin/target/test-case-data/SparkHadoopGraphProvider/graph-provider-data/~reducing/_temporary/0/_temporary/attempt_201611051613_15726_r_000000_0 (exists=false, cwd=file:/usr/src/tinkerpop/spark-gremlin)
at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:450)
at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:435)
I grabbed the related source from for version 2.7.2 from here: https://archive.apache.org/dist/hadoop/core/hadoop-2.7.2/
https://archive.apache.org/dist/hadoop/core/hadoop-2.7.2/hadoop-2.7.2-src.tar.gz
hadoop-2.7.2-src/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/ChecksumFileSystem.java
The related code does have a call to create() on line 435 but the exception is thrown on line 449:
@Override
public FSDataOutputStream create(Path f, FsPermission permission,
boolean overwrite, int bufferSize, short replication, long blockSize,
Progressable progress) throws IOException {
return create(f, permission, overwrite, true, bufferSize, // 435
replication, blockSize, progress);
}
private FSDataOutputStream create(Path f, FsPermission permission,
boolean overwrite, boolean createParent, int bufferSize,
short replication, long blockSize,
Progressable progress) throws IOException {
Path parent = f.getParent();
if (parent != null) {
if (!createParent && !exists(parent)) {
throw new FileNotFoundException("Parent directory doesn't exist: "
+ parent);
} else if (!mkdirs(parent)) {
throw new IOException("Mkdirs failed to create " + parent //449
+ " (exists=" + exists(parent) + ", cwd=" + getWorkingDirectory() // 450
+ ")");
}
}
From: stephen mallette [mailto:notifications@github.com]
Sent: Tuesday, November 15, 2016 8:24 AM
To: apache/tinkerpop
Cc: Paul A. Jackson; Author
Subject: Re: [apache/tinkerpop] TINKERPOP-1493 Groovy project doesn't build on Windows (#457)
The stack trace is pasted above.
\u2014
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub<https://github.com/apache/tinkerpop/pull/457#issuecomment-260639858>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AAUcSDXZOcPvgxhDo673VXzssRyskI47ks5q-bJdgaJpZM4KVbcO>.
________________________________
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] tinkerpop issue #457: TINKERPOP-1493 Groovy project doesn't build on Windows
Posted by spmallette <gi...@git.apache.org>.
Github user spmallette commented on the issue:
https://github.com/apache/tinkerpop/pull/457
i'm having a tough time getting this to build with docker. not sure why....keep getting failures that seem unrelated to this PR (dies in spark).
```text
java.lang.IllegalStateException: org.apache.spark.SparkException: Job aborted due to stage failure: Task 1 in stage 4226.0 failed 1 times, most recent failure: Lost task 1.0 in stage 4226.0 (TID 4939, localhost): java.io.IOException: Mkdirs failed to create /usr/src/tinkerpop/spark-gremlin/target/test-case-data/SparkHadoopGraphProvider/graph-provider-data/~reducing/_temporary/0/_temporary/attempt_201610302243_9163_r_000001_0 (exists=false, cwd=file:/usr/src/tinkerpop/spark-gremlin)
at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:450)
at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:435)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:909)
at org.apache.hadoop.io.SequenceFile$Writer.<init>(SequenceFile.java:1135)
at org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:273)
at org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:530)
at org.apache.hadoop.mapreduce.lib.output.SequenceFileOutputFormat.getSequenceWriter(SequenceFileOutputFormat.java:64)
at org.apache.hadoop.mapreduce.lib.output.SequenceFileOutputFormat.getRecordWriter(SequenceFileOutputFormat.java:75)
at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12.apply(PairRDDFunctions.scala:1030)
at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12.apply(PairRDDFunctions.scala:1014)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
at org.apache.spark.scheduler.Task.run(Task.scala:88)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Driver stacktrace:
at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895)
at org.apache.tinkerpop.gremlin.process.computer.traversal.step.map.ComputerResultStep.processNextStart(ComputerResultStep.java:81)
at org.apache.tinkerpop.gremlin.process.traversal.step.util.AbstractStep.next(AbstractStep.java:126)
at org.apache.tinkerpop.gremlin.process.traversal.step.util.AbstractStep.next(AbstractStep.java:37)
at org.apache.tinkerpop.gremlin.process.traversal.util.DefaultTraversal.next(DefaultTraversal.java:157)
at org.apache.tinkerpop.gremlin.process.traversal.strategy.decoration.SubgraphStrategyProcessTest.shouldFilterEdgeCriterion(SubgraphStrategyProcessTest.java:166)
Caused by: org.apache.spark.SparkException: Job aborted due to stage failure: Task 1 in stage 4226.0 failed 1 times, most recent failure: Lost task 1.0 in stage 4226.0 (TID 4939, localhost): java.io.IOException: Mkdirs failed to create /usr/src/tinkerpop/spark-gremlin/target/test-case-data/SparkHadoopGraphProvider/graph-provider-data/~reducing/_temporary/0/_temporary/attempt_201610302243_9163_r_000001_0 (exists=false, cwd=file:/usr/src/tinkerpop/spark-gremlin)
at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:450)
at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:435)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:909)
at org.apache.hadoop.io.SequenceFile$Writer.<init>(SequenceFile.java:1135)
at org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:273)
at org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:530)
at org.apache.hadoop.mapreduce.lib.output.SequenceFileOutputFormat.getSequenceWriter(SequenceFileOutputFormat.java:64)
at org.apache.hadoop.mapreduce.lib.output.SequenceFileOutputFormat.getRecordWriter(SequenceFileOutputFormat.java:75)
at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12.apply(PairRDDFunctions.scala:1030)
at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12.apply(PairRDDFunctions.scala:1014)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
at org.apache.spark.scheduler.Task.run(Task.scala:88)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Driver stacktrace:
at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1283)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1271)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1270)
at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1270)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:697)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:697)
at scala.Option.foreach(Option.scala:236)
at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:697)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1496)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1458)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1447)
at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:567)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:1824)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:1837)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:1914)
at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1.apply$mcV$sp(PairRDDFunctions.scala:1055)
at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1.apply(PairRDDFunctions.scala:998)
at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1.apply(PairRDDFunctions.scala:998)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:147)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:108)
at org.apache.spark.rdd.RDD.withScope(RDD.scala:310)
at org.apache.spark.rdd.PairRDDFunctions.saveAsNewAPIHadoopDataset(PairRDDFunctions.scala:998)
at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopFile$2.apply$mcV$sp(PairRDDFunctions.scala:938)
at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopFile$2.apply(PairRDDFunctions.scala:930)
at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopFile$2.apply(PairRDDFunctions.scala:930)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:147)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:108)
at org.apache.spark.rdd.RDD.withScope(RDD.scala:310)
at org.apache.spark.rdd.PairRDDFunctions.saveAsNewAPIHadoopFile(PairRDDFunctions.scala:930)
at org.apache.spark.api.java.JavaPairRDD.saveAsNewAPIHadoopFile(JavaPairRDD.scala:809)
at org.apache.tinkerpop.gremlin.spark.structure.io.OutputFormatRDD.writeMemoryRDD(OutputFormatRDD.java:65)
at org.apache.tinkerpop.gremlin.spark.process.computer.SparkGraphComputer.lambda$submitWithExecutor$1(SparkGraphComputer.java:271)
at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1590)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: Mkdirs failed to create /usr/src/tinkerpop/spark-gremlin/target/test-case-data/SparkHadoopGraphProvider/graph-provider-data/~reducing/_temporary/0/_temporary/attempt_201610302243_9163_r_000001_0 (exists=false, cwd=file:/usr/src/tinkerpop/spark-gremlin)
at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:450)
at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:435)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:909)
at org.apache.hadoop.io.SequenceFile$Writer.<init>(SequenceFile.java:1135)
at org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:273)
at org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:530)
at org.apache.hadoop.mapreduce.lib.output.SequenceFileOutputFormat.getSequenceWriter(SequenceFileOutputFormat.java:64)
at org.apache.hadoop.mapreduce.lib.output.SequenceFileOutputFormat.getRecordWriter(SequenceFileOutputFormat.java:75)
at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12.apply(PairRDDFunctions.scala:1030)
at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12.apply(PairRDDFunctions.scala:1014)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
at org.apache.spark.scheduler.Task.run(Task.scala:88)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] tinkerpop issue #457: TINKERPOP-1493 Groovy project doesn't build on Windows
Posted by pluradj <gi...@git.apache.org>.
Github user pluradj commented on the issue:
https://github.com/apache/tinkerpop/pull/457
I retested again overnight after rebasing the PR on `tp31` and still having success with Docker build on Ubuntu and Mac. Looking at the code changes, I don't see how they would cause that Spark failure.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] tinkerpop issue #457: TINKERPOP-1493 Groovy project doesn't build on Windows
Posted by dkuppitz <gi...@git.apache.org>.
Github user dkuppitz commented on the issue:
https://github.com/apache/tinkerpop/pull/457
Hmm, that's very unlikely. Check the output of `docker images`. The `TAG` column shows the Hadoop version.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] tinkerpop issue #457: TINKERPOP-1493 Groovy project doesn't build on Windows
Posted by robertdale <gi...@git.apache.org>.
Github user robertdale commented on the issue:
https://github.com/apache/tinkerpop/pull/457
@pauljackson It's all good. It's only the history here in the PR that looks weird. Since it's just the one commit, it'll be right in git history when it's actually merged.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
Re: [GitHub] tinkerpop issue #457: TINKERPOP-1493 Groovy project
doesn't build on Windows
Posted by Robert Dale <ro...@gmail.com>.
It's common these days to be enabled by default. Try: sestatus
Robert Dale
On Tue, Nov 15, 2016 at 10:53 AM, spmallette <gi...@git.apache.org> wrote:
> Github user spmallette commented on the issue:
>
> https://github.com/apache/tinkerpop/pull/457
>
> i didn't think so. I purposely didn't turn anything on that I know of.
>
>
> ---
> If your project is set up for it, you can reply to this email and have your
> reply appear on GitHub as well. If your project does not have this feature
> enabled and wishes so, or if the feature is enabled but not working, please
> contact infrastructure at infrastructure@apache.org or file a JIRA ticket
> with INFRA.
> ---
>
[GitHub] tinkerpop issue #457: TINKERPOP-1493 Groovy project doesn't build on Windows
Posted by spmallette <gi...@git.apache.org>.
Github user spmallette commented on the issue:
https://github.com/apache/tinkerpop/pull/457
i didn't think so. I purposely didn't turn anything on that I know of.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] tinkerpop pull request #457: TINKERPOP-1493 Groovy project doesn't build on ...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/tinkerpop/pull/457
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] tinkerpop issue #457: TINKERPOP-1493 Groovy project doesn't build on Windows
Posted by dkuppitz <gi...@git.apache.org>.
Github user dkuppitz commented on the issue:
https://github.com/apache/tinkerpop/pull/457
Btw. I don't think that this error is related to any of our components. The `mkdirs` problem has been reported in Docker's issue tracker several times and whenever they thought they solved it, someone else came up and ran into this issue again. I bet there's something on your system that interferes with Docker. SELinux has often been mentioned when this error popped up. Do you have any security related stuff to protect your file system?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] tinkerpop issue #457: TINKERPOP-1493 Groovy project doesn't build on Windows
Posted by spmallette <gi...@git.apache.org>.
Github user spmallette commented on the issue:
https://github.com/apache/tinkerpop/pull/457
I remember that we talked about that, but I thought we'd said this version was ok
```text
$ docker --version
Docker version 1.12.3, build 6b644ec
```
no?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] tinkerpop issue #457: TINKERPOP-1493 Groovy project doesn't build on Windows
Posted by spmallette <gi...@git.apache.org>.
Github user spmallette commented on the issue:
https://github.com/apache/tinkerpop/pull/457
good to know they both work. i sorta figured `merge` would be ok in this case as github would just figure out the diff and only include his one commit. when we merged that to our stuff, it wouldn't mess up history as it came from a fork. though i wonder if merge would bring an ugly "merge" commit with it. if it did, then i was wrong and `git rebase` would have been the way to go.
@pauljackson since @robertdale confirmed that `git rebase` is good, perhaps you should go that route to keep our history neat.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] tinkerpop issue #457: TINKERPOP-1493 Groovy project doesn't build on Windows
Posted by spmallette <gi...@git.apache.org>.
Github user spmallette commented on the issue:
https://github.com/apache/tinkerpop/pull/457
Looks like #498 has merged. @pauljackson if you could kindly rebase this on `tp31` for me I can give it a final test.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] tinkerpop issue #457: TINKERPOP-1493 Groovy project doesn't build on Windows
Posted by pluradj <gi...@git.apache.org>.
Github user pluradj commented on the issue:
https://github.com/apache/tinkerpop/pull/457
I didn't encounter any failures with the TinkerGraph integration tests on Windows.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] tinkerpop issue #457: TINKERPOP-1493 Groovy project doesn't build on Windows
Posted by pluradj <gi...@git.apache.org>.
Github user pluradj commented on the issue:
https://github.com/apache/tinkerpop/pull/457
We have enough votes to commit this. I'll handle it today. Thanks everyone.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] tinkerpop issue #457: TINKERPOP-1493 Groovy project doesn't build on Windows
Posted by pauljackson <gi...@git.apache.org>.
Github user pauljackson commented on the issue:
https://github.com/apache/tinkerpop/pull/457
Is there a stacktrace or other output that might help us figure this out? Specific test that fails?
Thanks,
-Paul
From: stephen mallette [mailto:notifications@github.com]
Sent: Saturday, November 12, 2016 6:22 AM
To: apache/tinkerpop
Cc: Paul A. Jackson; Author
Subject: Re: [apache/tinkerpop] TINKERPOP-1493 Groovy project doesn't build on Windows (#457)
I'm still in a position where tp31 is fine on docker but this PR is not. I rely on docker pretty heavily, so until I figure out what's wrong with my environment, I'm hesitant to +1 this though I admit that this change really shouldn't affect spark at all (yet somehow it does).
\u2014
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub<https://github.com/apache/tinkerpop/pull/457#issuecomment-260116393>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AAUcSN3Z2a6jxIhpM6WzjYTYgSCYCkKcks5q9aFugaJpZM4KVbcO>.
________________________________
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] tinkerpop issue #457: TINKERPOP-1493 Groovy project doesn't build on Windows
Posted by spmallette <gi...@git.apache.org>.
Github user spmallette commented on the issue:
https://github.com/apache/tinkerpop/pull/457
well - your `tp31` is now out of date because a number of changes (including the one that i need to test this branch with) have been added to that branch that are not in yours. If you were working in our repository I would say that to get those changes you should rebase which will effectively bring in the new changes and more your commits to the head of all those changes.
But it occurs to me now that you are working from a fork and perhaps you just need to merge `upstream/tp31` (our repo - i assume you called the remote "upstream") into your fork. I don't think you need to rebase your fork.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] tinkerpop issue #457: TINKERPOP-1493 Groovy project doesn't build on Windows
Posted by pluradj <gi...@git.apache.org>.
Github user pluradj commented on the issue:
https://github.com/apache/tinkerpop/pull/457
@spmallette are you hitting that with `docker/build.sh -t` also or only with integration tests?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] tinkerpop issue #457: TINKERPOP-1493 Groovy project doesn't build on Windows
Posted by spmallette <gi...@git.apache.org>.
Github user spmallette commented on the issue:
https://github.com/apache/tinkerpop/pull/457
```text
$ sestatus -v
SELinux status: disabled
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] tinkerpop issue #457: TINKERPOP-1493 Groovy project doesn't build on Windows
Posted by spmallette <gi...@git.apache.org>.
Github user spmallette commented on the issue:
https://github.com/apache/tinkerpop/pull/457
Sorry to say I'm still getting errors.
```text
Results :
Tests in error:
GroupTestV3d0$Traversals>GroupTestV3d0.g_V_group_byXoutE_countX_byXnameX:139->AbstractGremlinTest.printTraversalForm:252 ? IllegalState
SideEffectCapTest$Traversals>SideEffectCapTest.g_V_chooseXlabel_person__age_groupCountXaX__name_groupCountXbXX_capXa_bX:64->AbstractGremlinTest.printTraversalForm:252 ? IllegalState
GroovyHasTest$Traversals>HasTest.g_V_hasXname_markoX:135->get_g_V_hasXname_markoX:54 ? IllegalState
GroovyHasTest$Traversals>HasTest.g_V_hasXage_gt_30X:152->get_g_V_hasXage_gt_30X:95 ? IllegalState
GroovyGroupCountTest$Traversals>GroupCountTest.g_V_filterXfalseX_groupCount:142->get_g_V_hasXnoX_groupCount:59 ? IllegalState
```
You might want to rebase this on the latest from tp31, but i'm not so sure that will help as my problems are still in Spark. Here's a sample of the stacktrace from one of the errors:
```text
g_V_filterXfalseX_groupCount(org.apache.tinkerpop.gremlin.process.traversal.step.sideEffect.GroovyGroupCountTest$Traversals) Time elapsed: 0.215 sec <<< ERROR!
java.lang.IllegalStateException: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 6870.0 failed 1 times, most recent failure: Lost task 0.0 in stage 6870.0 (TID 8212, localhost): java.io.IOException: Mkdirs failed to create /usr/src/tinkerpop/spark-gremlin/target/test-case-data/SparkHadoopGraphProvider/graph-provider-data/~reducing/_temporary/0/_temporary/attempt_201611051613_15726_r_000000_0 (exists=false, cwd=file:/usr/src/tinkerpop/spark-gremlin)
at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:450)
at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:435)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:909)
at org.apache.hadoop.io.SequenceFile$Writer.<init>(SequenceFile.java:1135)
at org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:273)
at org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:530)
at org.apache.hadoop.mapreduce.lib.output.SequenceFileOutputFormat.getSequenceWriter(SequenceFileOutputFormat.java:64)
at org.apache.hadoop.mapreduce.lib.output.SequenceFileOutputFormat.getRecordWriter(SequenceFileOutputFormat.java:75)
at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12.apply(PairRDDFunctions.scala:1030)
at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12.apply(PairRDDFunctions.scala:1014)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
at org.apache.spark.scheduler.Task.run(Task.scala:88)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Driver stacktrace:
at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895)
at org.apache.tinkerpop.gremlin.process.traversal.util.TraversalScriptHelper.compute(TraversalScriptHelper.java:46)
at org.apache.tinkerpop.gremlin.process.traversal.util.TraversalScriptHelper.compute(TraversalScriptHelper.java:57)
at org.codehaus.groovy.vmplugin.v7.IndyInterface.selectMethod(IndyInterface.java:228)
at org.apache.tinkerpop.gremlin.process.traversal.step.sideEffect.GroovyGroupCountTest$Traversals.get_g_V_hasXnoX_groupCount(GroovyGroupCountTest.groovy:59)
Caused by: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 6870.0 failed 1 times, most recent failure: Lost task 0.0 in stage 6870.0 (TID 8212, localhost): java.io.IOException: Mkdirs failed to create /usr/src/tinkerpop/spark-gremlin/target/test-case-data/SparkHadoopGraphProvider/graph-provider-data/~reducing/_temporary/0/_temporary/attempt_201611051613_15726_r_000000_0 (exists=false, cwd=file:/usr/src/tinkerpop/spark-gremlin)
at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:450)
at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:435)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:909)
at org.apache.hadoop.io.SequenceFile$Writer.<init>(SequenceFile.java:1135)
at org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:273)
at org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:530)
at org.apache.hadoop.mapreduce.lib.output.SequenceFileOutputFormat.getSequenceWriter(SequenceFileOutputFormat.java:64)
at org.apache.hadoop.mapreduce.lib.output.SequenceFileOutputFormat.getRecordWriter(SequenceFileOutputFormat.java:75)
at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12.apply(PairRDDFunctions.scala:1030)
at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12.apply(PairRDDFunctions.scala:1014)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
at org.apache.spark.scheduler.Task.run(Task.scala:88)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Driver stacktrace:
at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1283)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1271)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1270)
at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1270)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:697)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:697)
at scala.Option.foreach(Option.scala:236)
at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:697)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1496)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1458)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1447)
at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:567)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:1824)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:1837)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:1914)
at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1.apply$mcV$sp(PairRDDFunctions.scala:1055)
at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1.apply(PairRDDFunctions.scala:998)
at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1.apply(PairRDDFunctions.scala:998)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:147)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:108)
at org.apache.spark.rdd.RDD.withScope(RDD.scala:310)
at org.apache.spark.rdd.PairRDDFunctions.saveAsNewAPIHadoopDataset(PairRDDFunctions.scala:998)
at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopFile$2.apply$mcV$sp(PairRDDFunctions.scala:938)
at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopFile$2.apply(PairRDDFunctions.scala:930)
at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopFile$2.apply(PairRDDFunctions.scala:930)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:147)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:108)
at org.apache.spark.rdd.RDD.withScope(RDD.scala:310)
at org.apache.spark.rdd.PairRDDFunctions.saveAsNewAPIHadoopFile(PairRDDFunctions.scala:930)
at org.apache.spark.api.java.JavaPairRDD.saveAsNewAPIHadoopFile(JavaPairRDD.scala:809)
at org.apache.tinkerpop.gremlin.spark.structure.io.OutputFormatRDD.writeMemoryRDD(OutputFormatRDD.java:65)
at org.apache.tinkerpop.gremlin.spark.process.computer.SparkGraphComputer.lambda$submitWithExecutor$1(SparkGraphComputer.java:271)
at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1590)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: Mkdirs failed to create /usr/src/tinkerpop/spark-gremlin/target/test-case-data/SparkHadoopGraphProvider/graph-provider-data/~reducing/_temporary/0/_temporary/attempt_201611051613_15726_r_000000_0 (exists=false, cwd=file:/usr/src/tinkerpop/spark-gremlin)
at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:450)
at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:435)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:909)
at org.apache.hadoop.io.SequenceFile$Writer.<init>(SequenceFile.java:1135)
at org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:273)
at org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:530)
at org.apache.hadoop.mapreduce.lib.output.SequenceFileOutputFormat.getSequenceWriter(SequenceFileOutputFormat.java:64)
at org.apache.hadoop.mapreduce.lib.output.SequenceFileOutputFormat.getRecordWriter(SequenceFileOutputFormat.java:75)
at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12.apply(PairRDDFunctions.scala:1030)
at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12.apply(PairRDDFunctions.scala:1014)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
at org.apache.spark.scheduler.Task.run(Task.scala:88)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
```
could just be environmental in some way, but i don't seem to get this on the tp31 branch itself. I just had a clean build of that this morning.
@dkuppitz could you give this a shot? Also, recognize that exception i pasted? it seems familiar to me. did we ever diagnose that?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] tinkerpop issue #457: TINKERPOP-1493 Groovy project doesn't build on Windows
Posted by spmallette <gi...@git.apache.org>.
Github user spmallette commented on the issue:
https://github.com/apache/tinkerpop/pull/457
The stack trace is pasted above.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] tinkerpop issue #457: TINKERPOP-1493 Groovy project doesn't build on Windows
Posted by spmallette <gi...@git.apache.org>.
Github user spmallette commented on the issue:
https://github.com/apache/tinkerpop/pull/457
I think we can get this merged once #498 goes in as it solves my problems with the build it seems. @pauljackson i will ping back here when that merges and request an additional rebase from you that I can then test. Thanks for your patience on this one.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] tinkerpop issue #457: TINKERPOP-1493 Groovy project doesn't build on Windows
Posted by spmallette <gi...@git.apache.org>.
Github user spmallette commented on the issue:
https://github.com/apache/tinkerpop/pull/457
@dkuppitz does that make any sense to you? is docker not using 2.7.2 somehow?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] tinkerpop issue #457: TINKERPOP-1493 Groovy project doesn't build on Windows
Posted by pluradj <gi...@git.apache.org>.
Github user pluradj commented on the issue:
https://github.com/apache/tinkerpop/pull/457
I configured a system with the same specs as listed, but I got 3 consecutive successes. Not sure what the issue could be.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] tinkerpop issue #457: TINKERPOP-1493 Groovy project doesn't build on Windows
Posted by pauljackson <gi...@git.apache.org>.
Github user pauljackson commented on the issue:
https://github.com/apache/tinkerpop/pull/457
Sorry, I\u2019m fluent in git (shame). From reading, I gleaned that rebase is a type of merge from one branch to another. Since I created the pull request on tp31 I don\u2019t understand what it means to rebase it on tp31. I\u2019m happy to follow directions if you could give me more explicit steps.
Thanks,
-Paul
From: stephen mallette [mailto:notifications@github.com]
Sent: Saturday, November 19, 2016 7:52 AM
To: apache/tinkerpop
Cc: Paul A. Jackson; Mention
Subject: Re: [apache/tinkerpop] TINKERPOP-1493 Groovy project doesn't build on Windows (#457)
Looks like #498<https://github.com/apache/tinkerpop/pull/498> has merged. @pauljackson<https://github.com/pauljackson> if you could kindly rebase this on tp31 for me I can give it a final test.
\u2014
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub<https://github.com/apache/tinkerpop/pull/457#issuecomment-261712200>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AAUcSOuRaIqzUeUX5cxZ2PqRUqWOhi8Iks5q_vDngaJpZM4KVbcO>.
________________________________
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] tinkerpop issue #457: TINKERPOP-1493 Groovy project doesn't build on Windows
Posted by spmallette <gi...@git.apache.org>.
Github user spmallette commented on the issue:
https://github.com/apache/tinkerpop/pull/457
I'm still in a position where tp31 is fine on docker but this PR is not. I rely on docker pretty heavily, so until I figure out what's wrong with my environment, I'm hesitant to +1 this though I admit that this change really shouldn't affect spark at all (yet somehow it does).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] tinkerpop issue #457: TINKERPOP-1493 Groovy project doesn't build on Windows
Posted by robertdale <gi...@git.apache.org>.
Github user robertdale commented on the issue:
https://github.com/apache/tinkerpop/pull/457
Either one appears to work cleanly. Just update first - `git fetch --all`.
Obviously, make sure you're on your branch `git checkout tp31`.
Then either `git rebase upstream/tp31` or `git merge upstream/tp31`.
Finally, push it back to your repo. `git push` or `git push origin tp31` or however it's named.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] tinkerpop issue #457: TINKERPOP-1493 Groovy project doesn't build on Windows
Posted by pluradj <gi...@git.apache.org>.
Github user pluradj commented on the issue:
https://github.com/apache/tinkerpop/pull/457
I reconfirmed with the latest updates and reaffirm my +1 vote from before.
- On Docker, this builds cleanly with `docker/build.sh -t -n -i`
- On Mac, this builds cleanly with `mvn clean install -DskipIntegrationTests=false`
- On Windows, this builds cleanly with `mvn clean install -DskipTests`
- Following it up with `mvn clean install -DskipIntegrationTests=false` succeeds all the way until `hadoop-gremlin`
```
[INFO] Apache TinkerPop ................................... SUCCESS [ 11.578 s]
[INFO] Apache TinkerPop :: Gremlin Shaded ................. SUCCESS [ 3.609 s]
[INFO] Apache TinkerPop :: Gremlin Core ................... SUCCESS [ 45.703 s]
[INFO] Apache TinkerPop :: Gremlin Test ................... SUCCESS [ 15.797 s]
[INFO] Apache TinkerPop :: Gremlin Groovy ................. SUCCESS [ 59.453 s]
[INFO] Apache TinkerPop :: Gremlin Groovy Test ............ SUCCESS [ 10.000 s]
[INFO] Apache TinkerPop :: TinkerGraph Gremlin ............ SUCCESS [04:31 min]
[INFO] Apache TinkerPop :: Hadoop Gremlin ................. FAILURE [06:13 min]
```
Although this doesn't completely solve TinkerPop builds on Windows, it passes a lot more than it used to.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] tinkerpop issue #457: TINKERPOP-1493 Groovy project doesn't build on Windows
Posted by pluradj <gi...@git.apache.org>.
Github user pluradj commented on the issue:
https://github.com/apache/tinkerpop/pull/457
any update on your Docker build @spmallette? still works ok for me.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] tinkerpop issue #457: TINKERPOP-1493 Groovy project doesn't build on Windows
Posted by pluradj <gi...@git.apache.org>.
Github user pluradj commented on the issue:
https://github.com/apache/tinkerpop/pull/457
I verified multiple platforms to ensure the changes didn't have any adverse affects.
- On Docker, this builds cleanly with `docker/build.sh -t -n -i`
- On Mac, this builds cleanly with `mvn clean install -DskipIntegrationTests=false`
- On Windows, this builds cleanly with `mvn clean install -DskipTests`. Following it up with `mvn clean install -DskipIntegrationTests=true` succeeds all the way through `hadoop-gremlin`, and then previously witnessed build failures on Windows start occurring in `spark-gremlin`. Skipping past `spark-gremlin` and `giraph-gremlin`, this builds cleanly to completion with `mvn clean install -DskipIntegrationTests=true -DincludeNeo4j -rf :neo4j-gremlin`.
Code changes look good to me.
VOTE: +1
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] tinkerpop issue #457: TINKERPOP-1493 Groovy project doesn't build on Windows
Posted by spmallette <gi...@git.apache.org>.
Github user spmallette commented on the issue:
https://github.com/apache/tinkerpop/pull/457
All tests pass with `docker/build.sh -t -n -i`
VOTE +1
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] tinkerpop issue #457: TINKERPOP-1493 Groovy project doesn't build on Windows
Posted by robertdale <gi...@git.apache.org>.
Github user robertdale commented on the issue:
https://github.com/apache/tinkerpop/pull/457
I didn't consider that your remote branch was behind. Duh. What should have been done is a forced push `git push -f origin tp31` to replace your remote branch with your local branch. Instead git ended up doing a merge and the history here looks a little weird because it pulled in all the changes since your changes. But git does the right thing and it merges cleanly on tp31.
Build passes on Linux. Can't test on windows.
VOTE: +0.5
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] tinkerpop issue #457: TINKERPOP-1493 Groovy project doesn't build on Windows
Posted by pluradj <gi...@git.apache.org>.
Github user pluradj commented on the issue:
https://github.com/apache/tinkerpop/pull/457
@spmallette what's your Docker version and host platform? Curious to see if I can recreate the failure.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---