[jira] [Created] (PARQUET-1923) Upgrading Hadoop to 2.7.4 to fix java.version problem

["Alexander Bayandin (Jira)" <ji...@apache.org>]

Alexander Bayandin created PARQUET-1923:
-------------------------------------------

             Summary: Upgrading Hadoop to 2.7.4 to fix java.version problem
                 Key: PARQUET-1923
                 URL: https://issues.apache.org/jira/browse/PARQUET-1923
             Project: Parquet
          Issue Type: Task
    Affects Versions: 1.11.0
         Environment: {noformat}
$ mvn --version
Apache Maven 3.6.3 (cecedd343002696d0abb50b32b541b8a6ba2883f)
Maven home: /usr/local/Cellar/maven/3.6.3_1/libexec
Java version: 15, vendor: N/A, runtime: /usr/local/Cellar/openjdk/15/libexec/openjdk.jdk/Contents/Home
Default locale: en_GB, platform encoding: UTF-8
OS name: "mac os x", version: "10.15.7", arch: "x86_64", family: "mac"

$ java -version
openjdk version "15" 2020-09-15
OpenJDK Runtime Environment (build 15+36)
OpenJDK 64-Bit Server VM (build 15+36, mixed mode, sharing)

$ sw_vers 
ProductName:	Mac OS X
ProductVersion:	10.15.7
BuildVersion:	19H2
{noformat}
 
            Reporter: Alexander Bayandin


{{mvn clean package -Plocal}} for parquet-tools 1.11.0 fails with a failing test {{testConverter(org.apache.parquet.tools.read.TestSimpleRecordConverter)}}.

{{mvn clean -Dtest=TestSimpleRecordConverter "-Plocal" test}}:
{noformat}
--------------------------------------------------------------------------------------------------------------------------------------------------------------Test set: org.apache.parquet.tools.read.TestSimpleRecordConverter-------------------------------------------------------------------------------Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.338 sec <<< FAILURE!testConverter(org.apache.parquet.tools.read.TestSimpleRecordConverter)  Time elapsed: 0.268 sec  <<< ERROR!java.lang.ExceptionInInitializerError at org.apache.hadoop.util.StringUtils.<clinit>(StringUtils.java:80) at org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:2823) at org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:2818) at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2684) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:373) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:172) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:357) at org.apache.hadoop.fs.Path.getFileSystem(Path.java:295) at org.apache.parquet.hadoop.util.HadoopOutputFile.fromPath(HadoopOutputFile.java:58) at org.apache.parquet.hadoop.ParquetWriter.<init>(ParquetWriter.java:227) at org.apache.parquet.hadoop.ParquetWriter.<init>(ParquetWriter.java:192) at org.apache.parquet.tools.read.TestSimpleRecordConverter.createTestParquetFile(TestSimpleRecordConverter.java:114) at org.apache.parquet.tools.read.TestSimpleRecordConverter.setUp(TestSimpleRecordConverter.java:90) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:64) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:564) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24) at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) at org.junit.rules.RunRules.evaluate(RunRules.java:20) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.runners.ParentRunner.run(ParentRunner.java:363) at org.apache.maven.surefire.junit4.JUnit4TestSet.execute(JUnit4TestSet.java:53) at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:123) at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:104) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:64) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:564) at org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:164) at org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:110) at org.apache.maven.surefire.booter.SurefireStarter.invokeProvider(SurefireStarter.java:175) at org.apache.maven.surefire.booter.SurefireStarter.runSuitesInProcessWhenForked(SurefireStarter.java:107) at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:68)Caused by: java.lang.StringIndexOutOfBoundsException: begin 0, end 3, length 2 at java.base/java.lang.String.checkBoundsBeginEnd(String.java:3734) at java.base/java.lang.String.substring(String.java:1903) at org.apache.hadoop.util.Shell.<clinit>(Shell.java:52) ... 44 more{noformat}

The error looks similar to SPARK-26134, so I suppose a fix should be the same.

Notes:
- I use parquet-tools 1.11.0 with https://github.com/apache/parquet-mr/commit/b6d07ae0744ba47aa9a8868ef2d7cbb232a60b22 patch
- I can't build current master at all: https://issues.apache.org/jira/browse/PARQUET-1896?focusedCommentId=17203146&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17203146



--
This message was sent by Atlassian Jira
(v8.3.4#803005)