You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@tinkerpop.apache.org by GitBox <gi...@apache.org> on 2019/09/08 07:14:32 UTC
[GitHub] [tinkerpop] okram commented on issue #1188: [TINKERPOP-2274] Fixed the issues revealed while compiling and runnin…
okram commented on issue #1188: [TINKERPOP-2274] Fixed the issues revealed while compiling and runnin…
URL: https://github.com/apache/tinkerpop/pull/1188#issuecomment-529176936
Wow. Rad PR man. Love your README.
Goodness to you,
Marko.
http://rredux.com <http://rredux.com/>
> On Sep 7, 2019, at 7:35 PM, Rusi Popov <no...@github.com> wrote:
>
> [TINKERPOP-2274] Fixed the issues revealed while compiling and running tests under Windows
>
> File system differences
>
> Revealed that the Spark context is not closed between the tests, while the tests remove its storage files, therefore explicitluy store Spark Cintext for every test, making the tests independent and (more) corect on Spark use.
>
> Revealed that Spark does not close the FileInputStreams it iterates upon, so that locks hang up in the file system. Thus, not being able to fix Spark/Hadoop and specifically the use of MultiIterator, call explicitly System.gc() in order to finalize the streams remaining open after closing the context.
>
> In Windows the file locking is diffrent than in LINUX, so rm() tests failed.
>
> See: [https://issues.apache.org/jira/browse/SPARK-12216]
>
> Documentaton
>
> Documented the convention to use only UNIX-formatted file paths as of the Storage API.
> Revealed the use and documented the Storage interface.
> File names and conventions
>
> Delegated file parsing to File class for platform independence
> Migrated to use unform UNIX-formatted paths, even under Windows for platform indepenence.
> Imposed Path / File classes use instead of manually composing file names.
> Avoided strings as Storage file system paths.
> Resolved inconsistency of Storage (file system) API returning null.
> Refactored FileSystemStorage to match the use & specification.
> Fixed possible leaking resources.
> Fixed inconsistencies in the use of / and home directory.
> Changed Storage.rm() to report if it succeeded to remove at least a file (in scope).
> Changed Spark tests close use the non-persistent context, this way closing it when not needed
> Changed tests
>
> Fixed EarlyLimitStrategyTest to allow running it
> Fixed the loger matching tests to ignore platform-specific EOLN char(s)
> Fixed org.apache.tinkerpop.gremlin.hadoop.process.computer.AbstractHadoopGraphComputerTest to delete the temorary files first before attempting to create new ones with the same name. Provided better error diagnostic messages.
> Renamed org.apache.tinkerpop.gremlin.TestHelper to CoreTestHelper to avoid confusing it with other TestHelper class.
> Updated AbstractStorageCheck#AbstractStorageCheck tests to ease their understanding and help further improvement of error reporting. Fixed the tests of rm(), cp() methods.
> Fixed tests
>
> org.apache.tinkerpop.gremlin.spark.SparkGremlinGryoSerializerTest
> org.apache.tinkerpop.gremlin.spark.SparkGremlinTest
> that used to fail with:
> ERROR shouldSupportCopyMethods(org.apache.tinkerpop.gremlin.spark.structure.io.SparkContextStorageCheck)
> java.lang.AssertionError
> at org.apache.tinkerpop.gremlin.spark.structure.io.SparkContextStorageCheck.shouldSupportCopyMethods(SparkContextStorageCheck.java:75)
> (NOTE: this is an example of "cryptic" error message reported on failed tests)
> Fixed reflection-based test to match their parameters at runtime.
> Dates
>
> Used SimpleDateFormat with explicit US locale to parse the test data.
> POM changes
>
> Changed POM to dump the whole stack trace in case of test failure.
>
> Set POM to check if HADOOP_HOME and HADOOP_GREMLIN_LIBS env. vars are set and warn if not, in order not to fail the build with NullPointerException.
>
> In order to avoid MAVEN warnings, provided explicitly the most-recent version of jacoco in and removed .
>
> Made the required HADOOP_HOME and HADOOP_GREMLIN_LIBS set up optional to allow Travis build pass, but this basically negates the use of Maven enforcer at all - in case of not installed Hadoop the tests will fail with no indication why, instead of the script to define & test its environment clearly.
>
> Artifacts
>
> Added generation of the tests jar to publish the TestHelper class, that is used both in core and tests projects, as a preparation to avoid the collistion of TestHelper classes in both projects
> In hadoop project added Readme.md to describe how to install hadoop, spark and OS integration.
> You can view, comment on, or merge this pull request online at:
>
> https://github.com/apache/tinkerpop/pull/1188 <https://github.com/apache/tinkerpop/pull/1188>
> Commit Summary
>
> [TINKERPOP-2274] Fixed the issues revealed while compiling and running tests under Windows
> File Changes
>
> M gremlin-console/src/test/java/org/apache/tinkerpop/gremlin/console/jsr223/DriverRemoteAcceptorIntegrateTest.java <https://github.com/apache/tinkerpop/pull/1188/files#diff-0> (25)
> M gremlin-console/src/test/java/org/apache/tinkerpop/gremlin/console/jsr223/DriverRemoteAcceptorTest.java <https://github.com/apache/tinkerpop/pull/1188/files#diff-1> (4)
> M gremlin-core/pom.xml <https://github.com/apache/tinkerpop/pull/1188/files#diff-2> (17)
> M gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/structure/io/Storage.java <https://github.com/apache/tinkerpop/pull/1188/files#diff-3> (105)
> R gremlin-core/src/test/java/org/apache/tinkerpop/gremlin/CoreTestHelper.java <https://github.com/apache/tinkerpop/pull/1188/files#diff-4> (102)
> M gremlin-core/src/test/java/org/apache/tinkerpop/gremlin/jsr223/DefaultScriptCustomizerTest.java <https://github.com/apache/tinkerpop/pull/1188/files#diff-5> (6)
> M gremlin-core/src/test/java/org/apache/tinkerpop/gremlin/jsr223/ScriptEngineCacheTest.java <https://github.com/apache/tinkerpop/pull/1188/files#diff-6> (4)
> M gremlin-core/src/test/java/org/apache/tinkerpop/gremlin/jsr223/ScriptFileGremlinPluginTest.java <https://github.com/apache/tinkerpop/pull/1188/files#diff-7> (11)
> M gremlin-core/src/test/java/org/apache/tinkerpop/gremlin/process/traversal/OrderTest.java <https://github.com/apache/tinkerpop/pull/1188/files#diff-8> (11)
> M gremlin-core/src/test/java/org/apache/tinkerpop/gremlin/process/traversal/strategy/optimization/EarlyLimitStrategyTest.java <https://github.com/apache/tinkerpop/pull/1188/files#diff-9> (4)
> M gremlin-core/src/test/java/org/apache/tinkerpop/gremlin/process/traversal/strategy/optimization/OrderLimitStrategyTest.java <https://github.com/apache/tinkerpop/pull/1188/files#diff-10> (2)
> M gremlin-core/src/test/java/org/apache/tinkerpop/gremlin/structure/io/GraphMigratorTest.java <https://github.com/apache/tinkerpop/pull/1188/files#diff-11> (4)
> M gremlin-core/src/test/java/org/apache/tinkerpop/gremlin/structure/io/IoCoreTest.java <https://github.com/apache/tinkerpop/pull/1188/files#diff-12> (4)
> M gremlin-core/src/test/java/org/apache/tinkerpop/gremlin/structure/io/graphson/GraphSONTokensTest.java <https://github.com/apache/tinkerpop/pull/1188/files#diff-13> (4)
> M gremlin-core/src/test/java/org/apache/tinkerpop/gremlin/structure/io/graphson/GraphSONUtilTest.java <https://github.com/apache/tinkerpop/pull/1188/files#diff-14> (4)
> M gremlin-core/src/test/java/org/apache/tinkerpop/gremlin/structure/util/ComparatorsTest.java <https://github.com/apache/tinkerpop/pull/1188/files#diff-15> (4)
> M gremlin-core/src/test/java/org/apache/tinkerpop/gremlin/structure/util/ElementHelperTest.java <https://github.com/apache/tinkerpop/pull/1188/files#diff-16> (4)
> M gremlin-core/src/test/java/org/apache/tinkerpop/gremlin/structure/util/GraphFactoryTest.java <https://github.com/apache/tinkerpop/pull/1188/files#diff-17> (31)
> M gremlin-core/src/test/java/org/apache/tinkerpop/gremlin/structure/util/GraphVariableHelperTest.java <https://github.com/apache/tinkerpop/pull/1188/files#diff-18> (4)
> M gremlin-core/src/test/java/org/apache/tinkerpop/gremlin/structure/util/StringFactoryTest.java <https://github.com/apache/tinkerpop/pull/1188/files#diff-19> (4)
> M gremlin-core/src/test/java/org/apache/tinkerpop/gremlin/util/GremlinTest.java <https://github.com/apache/tinkerpop/pull/1188/files#diff-20> (4)
> M gremlin-core/src/test/java/org/apache/tinkerpop/gremlin/util/SerializerTest.java <https://github.com/apache/tinkerpop/pull/1188/files#diff-21> (4)
> M gremlin-core/src/test/java/org/apache/tinkerpop/gremlin/util/TimeUtilTest.java <https://github.com/apache/tinkerpop/pull/1188/files#diff-22> (4)
> M gremlin-core/src/test/java/org/apache/tinkerpop/gremlin/util/function/FunctionUtilsTest.java <https://github.com/apache/tinkerpop/pull/1188/files#diff-23> (4)
> M gremlin-core/src/test/java/org/apache/tinkerpop/gremlin/util/iterator/IteratorUtilsTest.java <https://github.com/apache/tinkerpop/pull/1188/files#diff-24> (4)
> M gremlin-core/src/test/java/org/apache/tinkerpop/gremlin/util/tools/MultiMapTest.java <https://github.com/apache/tinkerpop/pull/1188/files#diff-25> (4)
> M gremlin-groovy/src/test/java/org/apache/tinkerpop/gremlin/groovy/engine/GremlinExecutorTest.java <https://github.com/apache/tinkerpop/pull/1188/files#diff-26> (4)
> M gremlin-groovy/src/test/java/org/apache/tinkerpop/gremlin/groovy/jsr223/GremlinGroovyScriptEngineFileSandboxTest.java <https://github.com/apache/tinkerpop/pull/1188/files#diff-27> (5)
> M gremlin-groovy/src/test/java/org/apache/tinkerpop/gremlin/groovy/util/DependencyGrabberIntegrateTest.java <https://github.com/apache/tinkerpop/pull/1188/files#diff-28> (3)
> M gremlin-groovy/src/test/java/org/apache/tinkerpop/gremlin/util/TinkerGraphProvider.java <https://github.com/apache/tinkerpop/pull/1188/files#diff-29> (4)
> M gremlin-javascript/pom.xml <https://github.com/apache/tinkerpop/pull/1188/files#diff-30> (2)
> M gremlin-server/src/test/java/org/apache/tinkerpop/gremlin/server/GremlinDriverIntegrateTest.java <https://github.com/apache/tinkerpop/pull/1188/files#diff-31> (6)
> M gremlin-server/src/test/java/org/apache/tinkerpop/gremlin/server/ResponseHandlerContextTest.java <https://github.com/apache/tinkerpop/pull/1188/files#diff-32> (6)
> M gremlin-server/src/test/java/org/apache/tinkerpop/gremlin/server/ServerTestHelper.java <https://github.com/apache/tinkerpop/pull/1188/files#diff-33> (60)
> M gremlin-server/src/test/java/org/apache/tinkerpop/gremlin/util/Log4jRecordingAppender.java <https://github.com/apache/tinkerpop/pull/1188/files#diff-34> (22)
> M gremlin-test/pom.xml <https://github.com/apache/tinkerpop/pull/1188/files#diff-35> (6)
> M gremlin-test/src/main/java/org/apache/tinkerpop/gremlin/GraphProvider.java <https://github.com/apache/tinkerpop/pull/1188/files#diff-36> (3)
> M gremlin-test/src/main/java/org/apache/tinkerpop/gremlin/TestHelper.java <https://github.com/apache/tinkerpop/pull/1188/files#diff-37> (147)
> M gremlin-test/src/main/java/org/apache/tinkerpop/gremlin/jsr223/BindingsScriptEngineTest.java <https://github.com/apache/tinkerpop/pull/1188/files#diff-38> (3)
> M gremlin-test/src/main/java/org/apache/tinkerpop/gremlin/process/computer/bulkloading/BulkLoaderVertexProgramTest.java <https://github.com/apache/tinkerpop/pull/1188/files#diff-39> (2)
> M gremlin-test/src/main/java/org/apache/tinkerpop/gremlin/structure/io/IoGraphTest.java <https://github.com/apache/tinkerpop/pull/1188/files#diff-40> (8)
> M gremlin-test/src/main/java/org/apache/tinkerpop/gremlin/structure/io/IoTest.java <https://github.com/apache/tinkerpop/pull/1188/files#diff-41> (31)
> A hadoop-gremlin/README.md <https://github.com/apache/tinkerpop/pull/1188/files#diff-42> (52)
> M hadoop-gremlin/pom.xml <https://github.com/apache/tinkerpop/pull/1188/files#diff-43> (29)
> M hadoop-gremlin/src/main/java/org/apache/tinkerpop/gremlin/hadoop/Constants.java <https://github.com/apache/tinkerpop/pull/1188/files#diff-44> (22)
> M hadoop-gremlin/src/main/java/org/apache/tinkerpop/gremlin/hadoop/process/computer/AbstractHadoopGraphComputer.java <https://github.com/apache/tinkerpop/pull/1188/files#diff-45> (12)
> M hadoop-gremlin/src/main/java/org/apache/tinkerpop/gremlin/hadoop/structure/io/FileSystemStorage.java <https://github.com/apache/tinkerpop/pull/1188/files#diff-46> (101)
> M hadoop-gremlin/src/main/java/org/apache/tinkerpop/gremlin/hadoop/structure/io/InputOutputHelper.java <https://github.com/apache/tinkerpop/pull/1188/files#diff-47> (2)
> M hadoop-gremlin/src/test/java/org/apache/tinkerpop/gremlin/hadoop/HadoopGraphProvider.java <https://github.com/apache/tinkerpop/pull/1188/files#diff-48> (12)
> M hadoop-gremlin/src/test/java/org/apache/tinkerpop/gremlin/hadoop/process/computer/AbstractHadoopGraphComputerTest.java <https://github.com/apache/tinkerpop/pull/1188/files#diff-49> (59)
> M hadoop-gremlin/src/test/java/org/apache/tinkerpop/gremlin/hadoop/structure/io/AbstractIoRegistryCheck.java <https://github.com/apache/tinkerpop/pull/1188/files#diff-50> (7)
> M hadoop-gremlin/src/test/java/org/apache/tinkerpop/gremlin/hadoop/structure/io/AbstractStorageCheck.java <https://github.com/apache/tinkerpop/pull/1188/files#diff-51> (48)
> M hadoop-gremlin/src/test/java/org/apache/tinkerpop/gremlin/hadoop/structure/io/FileSystemStorageCheck.java <https://github.com/apache/tinkerpop/pull/1188/files#diff-52> (4)
> M hadoop-gremlin/src/test/java/org/apache/tinkerpop/gremlin/hadoop/structure/io/RecordReaderWriterTest.java <https://github.com/apache/tinkerpop/pull/1188/files#diff-53> (5)
> M pom.xml <https://github.com/apache/tinkerpop/pull/1188/files#diff-54> (7)
> M spark-gremlin/src/main/java/org/apache/tinkerpop/gremlin/spark/structure/Spark.java <https://github.com/apache/tinkerpop/pull/1188/files#diff-55> (7)
> M spark-gremlin/src/main/java/org/apache/tinkerpop/gremlin/spark/structure/io/SparkContextStorage.java <https://github.com/apache/tinkerpop/pull/1188/files#diff-56> (115)
> M spark-gremlin/src/test/java/org/apache/tinkerpop/gremlin/spark/process/computer/LocalPropertyTest.java <https://github.com/apache/tinkerpop/pull/1188/files#diff-57> (2)
> M spark-gremlin/src/test/java/org/apache/tinkerpop/gremlin/spark/process/computer/SparkHadoopGraphGryoSerializerProvider.java <https://github.com/apache/tinkerpop/pull/1188/files#diff-58> (1)
> M spark-gremlin/src/test/java/org/apache/tinkerpop/gremlin/spark/process/computer/SparkHadoopGraphProvider.java <https://github.com/apache/tinkerpop/pull/1188/files#diff-59> (5)
> M spark-gremlin/src/test/java/org/apache/tinkerpop/gremlin/spark/structure/SparkTest.java <https://github.com/apache/tinkerpop/pull/1188/files#diff-60> (3)
> M spark-gremlin/src/test/java/org/apache/tinkerpop/gremlin/spark/structure/io/PersistedInputOutputRDDIntegrateTest.java <https://github.com/apache/tinkerpop/pull/1188/files#diff-61> (8)
> M spark-gremlin/src/test/java/org/apache/tinkerpop/gremlin/spark/structure/io/SparkContextStorageCheck.java <https://github.com/apache/tinkerpop/pull/1188/files#diff-62> (37)
> M spark-gremlin/src/test/java/org/apache/tinkerpop/gremlin/spark/structure/io/gryo/GryoSerializerIntegrateTest.java <https://github.com/apache/tinkerpop/pull/1188/files#diff-63> (4)
> M tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/TinkerGraphProvider.java <https://github.com/apache/tinkerpop/pull/1188/files#diff-64> (16)
> M tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/TinkerGraphUUIDProvider.java <https://github.com/apache/tinkerpop/pull/1188/files#diff-65> (4)
> M tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/IoDataGenerationTest.java <https://github.com/apache/tinkerpop/pull/1188/files#diff-66> (141)
> M tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/TinkerGraphTest.java <https://github.com/apache/tinkerpop/pull/1188/files#diff-67> (13)
> Patch Links:
>
> https://github.com/apache/tinkerpop/pull/1188.patch <https://github.com/apache/tinkerpop/pull/1188.patch>
> https://github.com/apache/tinkerpop/pull/1188.diff <https://github.com/apache/tinkerpop/pull/1188.diff>
> —
> You are receiving this because you are subscribed to this thread.
> Reply to this email directly, view it on GitHub <https://github.com/apache/tinkerpop/pull/1188?email_source=notifications&email_token=AABELPM6K7SFUR6NK2M5YDDQIRJHFA5CNFSM4IUR3CDKYY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4HJ7O3XQ>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AABELPMTPEEUNTZ3FXYBEBLQIRJHFANCNFSM4IUR3CDA>.
>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services