You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@beam.apache.org by "Tomo Suzuki (Jira)" <ji...@apache.org> on 2021/01/13 21:00:00 UTC
[jira] [Updated] (BEAM-11626) Upgrade non-vendored Guava to the
latest version
[ https://issues.apache.org/jira/browse/BEAM-11626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tomo Suzuki updated BEAM-11626:
-------------------------------
Description:
Related to https://issues.apache.org/jira/browse/BEAM-5559
As per BEAM-8911, Cassandra-all requires Guava 25.1. But other parts of Beam dependencies (such as gcsio) may want to use newer version of Guava. Excerpt from Kiley's finding using Linkage Checker today:
{noformat}
(com.google.guava:guava:25.1-jre) com.google.common.util.concurrent.Uninterruptibles's method sleepUninterruptibly(java.time.Duration) is not found;
referenced by 3 class files
com.google.cloud.hadoop.gcsio.cooplock.CoopLockOperationDao (com.google.cloud.bigdataoss:gcsio:2.1.6)
com.google.cloud.hadoop.gcsio.cooplock.CoopLockRecordsDao (com.google.cloud.bigdataoss:gcsio:2.1.6)
com.google.cloud.hadoop.gcsio.testing.InMemoryObjectEntry (com.google.cloud.bigdataoss:gcsio:2.1.6)
{noformat}
Note that Guava's [Uninterruptibles.sleepUninterruptibly(Duration)|https://guava.dev/releases/30.1-jre/api/docs/com/google/common/util/concurrent/Uninterruptibles.html#sleepUninterruptibly(java.time.Duration)] has been introduced since 28.0.
This ticket is attempts to declare the newer version of Guava while keeping the Guava version used by Beam's Cassandra tests.
h1. Failures with Guava 30.1-jre
https://gist.github.com/suztomo/f7bbf77b54265eb80fd3c2d230b678dc
Failures in my local tests:
org.apache.beam.sdk.io.hdfs.HadoopFileSystemRegistrarTest and other tests in the package:
{noformat}
java.lang.NoSuchMethodError: com.google.common.util.concurrent.Futures.addCallback(Lcom/google/common/util/concurrent/ListenableFuture;Lcom/google/common/util/concurrent/FutureCallback;)V
at org.apache.hadoop.hdfs.server.datanode.checker.ThrottledAsyncChecker.addResultCachingCallback(ThrottledAsyncChecker.java:167)
at org.apache.hadoop.hdfs.server.datanode.checker.ThrottledAsyncChecker.schedule(ThrottledAsyncChecker.java:156)
at org.apache.hadoop.hdfs.server.datanode.checker.StorageLocationChecker.check(StorageLocationChecker.java:165)
at org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:2703)
at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:2613)
at org.apache.hadoop.hdfs.MiniDFSCluster.startDataNodes(MiniDFSCluster.java:1683)
at org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:905)
at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:517)
at org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:476)
at org.apache.beam.sdk.io.hdfs.HadoopFileSystemRegistrarTest.setUp(HadoopFileSystemRegistrarTest.java:58)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
{noformat}
The deprecated [Futures.addCallback|https://guava.dev/releases/25.1-jre/api/docs/com/google/common/util/concurrent/Futures.html] has been removed since 26.0.
The build (master) fails in my machine:
{noformat}
suztomo@suztomo:~/beam$ ./gradlew :sdks:java:io:cassandra:test
Configuration on demand is an incubating feature.
> Task :sdks:java:io:cassandra:test
#
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGSEGV (0xb) at pc=0x00007fbdbc875a3f, pid=3453267, tid=0x000000000034b177
#
# JRE version: OpenJDK Runtime Environment (8.0_181) (build 1.8.0_181-google-v7-341329124-341329124)
# Java VM: OpenJDK 64-Bit Server VM (25.181-b01 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# C [libc.so.6+0xbaa3f] __gettimeofday+0x7f
#
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
...
{noformat}
{{:sdks:java:io:cassandra:test}} works in my MacBook Pro. Other tests in {{:sdks:java:io}} passed in Linux.
was:
As per BEAM-8911, Cassandra-all requires Guava 25.1. But other parts of Beam dependencies (such as gcsio) may want to use newer version of Guava. Excerpt from Kiley's finding using Linkage Checker today:
{noformat}
(com.google.guava:guava:25.1-jre) com.google.common.util.concurrent.Uninterruptibles's method sleepUninterruptibly(java.time.Duration) is not found;
referenced by 3 class files
com.google.cloud.hadoop.gcsio.cooplock.CoopLockOperationDao (com.google.cloud.bigdataoss:gcsio:2.1.6)
com.google.cloud.hadoop.gcsio.cooplock.CoopLockRecordsDao (com.google.cloud.bigdataoss:gcsio:2.1.6)
com.google.cloud.hadoop.gcsio.testing.InMemoryObjectEntry (com.google.cloud.bigdataoss:gcsio:2.1.6)
{noformat}
Note that Guava's [Uninterruptibles.sleepUninterruptibly(Duration)|https://guava.dev/releases/30.1-jre/api/docs/com/google/common/util/concurrent/Uninterruptibles.html#sleepUninterruptibly(java.time.Duration)] has been introduced since 28.0.
This ticket is attempts to declare the newer version of Guava while keeping the Guava version used by Beam's Cassandra tests.
h1. Failures with Guava 30.1-jre
https://gist.github.com/suztomo/f7bbf77b54265eb80fd3c2d230b678dc
Failures in my local tests:
org.apache.beam.sdk.io.hdfs.HadoopFileSystemRegistrarTest and other tests in the package:
{noformat}
java.lang.NoSuchMethodError: com.google.common.util.concurrent.Futures.addCallback(Lcom/google/common/util/concurrent/ListenableFuture;Lcom/google/common/util/concurrent/FutureCallback;)V
at org.apache.hadoop.hdfs.server.datanode.checker.ThrottledAsyncChecker.addResultCachingCallback(ThrottledAsyncChecker.java:167)
at org.apache.hadoop.hdfs.server.datanode.checker.ThrottledAsyncChecker.schedule(ThrottledAsyncChecker.java:156)
at org.apache.hadoop.hdfs.server.datanode.checker.StorageLocationChecker.check(StorageLocationChecker.java:165)
at org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:2703)
at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:2613)
at org.apache.hadoop.hdfs.MiniDFSCluster.startDataNodes(MiniDFSCluster.java:1683)
at org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:905)
at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:517)
at org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:476)
at org.apache.beam.sdk.io.hdfs.HadoopFileSystemRegistrarTest.setUp(HadoopFileSystemRegistrarTest.java:58)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
{noformat}
The deprecated [Futures.addCallback|https://guava.dev/releases/25.1-jre/api/docs/com/google/common/util/concurrent/Futures.html] has been removed since 26.0.
The build (master) fails in my machine:
{noformat}
suztomo@suztomo:~/beam$ ./gradlew :sdks:java:io:cassandra:test
Configuration on demand is an incubating feature.
> Task :sdks:java:io:cassandra:test
#
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGSEGV (0xb) at pc=0x00007fbdbc875a3f, pid=3453267, tid=0x000000000034b177
#
# JRE version: OpenJDK Runtime Environment (8.0_181) (build 1.8.0_181-google-v7-341329124-341329124)
# Java VM: OpenJDK 64-Bit Server VM (25.181-b01 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# C [libc.so.6+0xbaa3f] __gettimeofday+0x7f
#
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
...
{noformat}
{{:sdks:java:io:cassandra:test}} works in my MacBook Pro. Other tests in {{:sdks:java:io}} passed in Linux.
Summary: Upgrade non-vendored Guava to the latest version (was: Split Guava version to Cassandra and others)
> Upgrade non-vendored Guava to the latest version
> ------------------------------------------------
>
> Key: BEAM-11626
> URL: https://issues.apache.org/jira/browse/BEAM-11626
> Project: Beam
> Issue Type: Improvement
> Components: build-system
> Reporter: Tomo Suzuki
> Assignee: Tomo Suzuki
> Priority: P2
> Time Spent: 3h 50m
> Remaining Estimate: 0h
>
> Related to https://issues.apache.org/jira/browse/BEAM-5559
> As per BEAM-8911, Cassandra-all requires Guava 25.1. But other parts of Beam dependencies (such as gcsio) may want to use newer version of Guava. Excerpt from Kiley's finding using Linkage Checker today:
> {noformat}
> (com.google.guava:guava:25.1-jre) com.google.common.util.concurrent.Uninterruptibles's method sleepUninterruptibly(java.time.Duration) is not found;
> referenced by 3 class files
> com.google.cloud.hadoop.gcsio.cooplock.CoopLockOperationDao (com.google.cloud.bigdataoss:gcsio:2.1.6)
> com.google.cloud.hadoop.gcsio.cooplock.CoopLockRecordsDao (com.google.cloud.bigdataoss:gcsio:2.1.6)
> com.google.cloud.hadoop.gcsio.testing.InMemoryObjectEntry (com.google.cloud.bigdataoss:gcsio:2.1.6)
> {noformat}
> Note that Guava's [Uninterruptibles.sleepUninterruptibly(Duration)|https://guava.dev/releases/30.1-jre/api/docs/com/google/common/util/concurrent/Uninterruptibles.html#sleepUninterruptibly(java.time.Duration)] has been introduced since 28.0.
> This ticket is attempts to declare the newer version of Guava while keeping the Guava version used by Beam's Cassandra tests.
> h1. Failures with Guava 30.1-jre
> https://gist.github.com/suztomo/f7bbf77b54265eb80fd3c2d230b678dc
> Failures in my local tests:
> org.apache.beam.sdk.io.hdfs.HadoopFileSystemRegistrarTest and other tests in the package:
> {noformat}
> java.lang.NoSuchMethodError: com.google.common.util.concurrent.Futures.addCallback(Lcom/google/common/util/concurrent/ListenableFuture;Lcom/google/common/util/concurrent/FutureCallback;)V
> at org.apache.hadoop.hdfs.server.datanode.checker.ThrottledAsyncChecker.addResultCachingCallback(ThrottledAsyncChecker.java:167)
> at org.apache.hadoop.hdfs.server.datanode.checker.ThrottledAsyncChecker.schedule(ThrottledAsyncChecker.java:156)
> at org.apache.hadoop.hdfs.server.datanode.checker.StorageLocationChecker.check(StorageLocationChecker.java:165)
> at org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:2703)
> at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:2613)
> at org.apache.hadoop.hdfs.MiniDFSCluster.startDataNodes(MiniDFSCluster.java:1683)
> at org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:905)
> at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:517)
> at org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:476)
> at org.apache.beam.sdk.io.hdfs.HadoopFileSystemRegistrarTest.setUp(HadoopFileSystemRegistrarTest.java:58)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> {noformat}
> The deprecated [Futures.addCallback|https://guava.dev/releases/25.1-jre/api/docs/com/google/common/util/concurrent/Futures.html] has been removed since 26.0.
> The build (master) fails in my machine:
> {noformat}
> suztomo@suztomo:~/beam$ ./gradlew :sdks:java:io:cassandra:test
> Configuration on demand is an incubating feature.
> > Task :sdks:java:io:cassandra:test
> #
> # A fatal error has been detected by the Java Runtime Environment:
> #
> # SIGSEGV (0xb) at pc=0x00007fbdbc875a3f, pid=3453267, tid=0x000000000034b177
> #
> # JRE version: OpenJDK Runtime Environment (8.0_181) (build 1.8.0_181-google-v7-341329124-341329124)
> # Java VM: OpenJDK 64-Bit Server VM (25.181-b01 mixed mode linux-amd64 compressed oops)
> # Problematic frame:
> # C [libc.so.6+0xbaa3f] __gettimeofday+0x7f
> #
> # Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
> ...
> {noformat}
> {{:sdks:java:io:cassandra:test}} works in my MacBook Pro. Other tests in {{:sdks:java:io}} passed in Linux.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)