You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Sergei Poganshev <s....@slice.com> on 2018/12/06 13:03:01 UTC

flink-s3-fs-presto:1.7.0 is missing shaded com/facebook/presto/hadoop

When I try to configure checkpointing using Presto in 1.7.0 the following
exception occurs:

java.lang.NoClassDefFoundError:
org/apache/flink/fs/s3presto/shaded/com/facebook/presto/hadoop/HadoopFileStatus
at
org.apache.flink.fs.s3presto.shaded.com.facebook.presto.hive.PrestoS3FileSystem.directory(PrestoS3FileSystem.java:446)
at
org.apache.flink.fs.s3presto.shaded.com.facebook.presto.hive.PrestoS3FileSystem.delete(PrestoS3FileSystem.java:423)
at
org.apache.flink.fs.s3.common.hadoop.HadoopFileSystem.delete(HadoopFileSystem.java:147)
at
org.apache.flink.runtime.state.filesystem.FileStateHandle.discardState(FileStateHandle.java:80)
at
org.apache.flink.runtime.checkpoint.CompletedCheckpoint.doDiscard(CompletedCheckpoint.java:250)
at
org.apache.flink.runtime.checkpoint.CompletedCheckpoint.discardOnSubsume(CompletedCheckpoint.java:219)
at
org.apache.flink.runtime.checkpoint.StandaloneCompletedCheckpointStore.addCheckpoint(StandaloneCompletedCheckpointStore.java:72)
at
org.apache.flink.runtime.checkpoint.CheckpointCoordinator.completePendingCheckpoint(CheckpointCoordinator.java:844)
at
org.apache.flink.runtime.checkpoint.CheckpointCoordinator.receiveAcknowledgeMessage(CheckpointCoordinator.java:756)
at
org.apache.flink.runtime.jobmaster.JobMaster.lambda$acknowledgeCheckpoint$8(JobMaster.java:680)
at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:39)
at
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:415)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at
scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

HadoopFileStatus is indeed absent in flink-s3-fs-presto:1.7.0, while it is
present in previous versions (1.6.2 for example)

I've worked it around by just adding this class to my job project:

package org.apache.flink.fs.s3presto.shaded.com.facebook.presto.hadoop;

import org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.FileStatus;

public final class HadoopFileStatus {
    public static boolean isDirectory(FileStatus status) {
        return status.isDirectory();
    }

    public static boolean isFile(FileStatus status) {
        return status.isFile();
    }
}


But it's a dirty hack of course and hopefully the fix will be added in next
patch release.

Re: flink-s3-fs-presto:1.7.0 is missing shaded com/facebook/presto/hadoop

Posted by Chesnay Schepler <ch...@apache.org>.
I've filed a JIRA: https://issues.apache.org/jira/browse/FLINK-11085

On 06.12.2018 14:03, Sergei Poganshev wrote:
> When I try to configure checkpointing using Presto in 1.7.0 the 
> following exception occurs:
>
> java.lang.NoClassDefFoundError: 
> org/apache/flink/fs/s3presto/shaded/com/facebook/presto/hadoop/HadoopFileStatus
> at 
> org.apache.flink.fs.s3presto.shaded.com.facebook.presto.hive.PrestoS3FileSystem.directory(PrestoS3FileSystem.java:446)
> at 
> org.apache.flink.fs.s3presto.shaded.com.facebook.presto.hive.PrestoS3FileSystem.delete(PrestoS3FileSystem.java:423)
> at 
> org.apache.flink.fs.s3.common.hadoop.HadoopFileSystem.delete(HadoopFileSystem.java:147)
> at 
> org.apache.flink.runtime.state.filesystem.FileStateHandle.discardState(FileStateHandle.java:80)
> at 
> org.apache.flink.runtime.checkpoint.CompletedCheckpoint.doDiscard(CompletedCheckpoint.java:250)
> at 
> org.apache.flink.runtime.checkpoint.CompletedCheckpoint.discardOnSubsume(CompletedCheckpoint.java:219)
> at 
> org.apache.flink.runtime.checkpoint.StandaloneCompletedCheckpointStore.addCheckpoint(StandaloneCompletedCheckpointStore.java:72)
> at 
> org.apache.flink.runtime.checkpoint.CheckpointCoordinator.completePendingCheckpoint(CheckpointCoordinator.java:844)
> at 
> org.apache.flink.runtime.checkpoint.CheckpointCoordinator.receiveAcknowledgeMessage(CheckpointCoordinator.java:756)
> at 
> org.apache.flink.runtime.jobmaster.JobMaster.lambda$acknowledgeCheckpoint$8(JobMaster.java:680)
> at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:39)
> at 
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:415)
> at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
> at 
> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
> at 
> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at 
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>
> HadoopFileStatus is indeed absent in flink-s3-fs-presto:1.7.0, while 
> it is present in previous versions (1.6.2 for example)
>
> I've worked it around by just adding this class to my job project:
>
> package org.apache.flink.fs.s3presto.shaded.com.facebook.presto.hadoop; import org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.FileStatus; public final class HadoopFileStatus {
>      public static boolean isDirectory(FileStatus status) {
>          return status.isDirectory(); }
>
>      public static boolean isFile(FileStatus status) {
>          return status.isFile(); }
> }
>
> But it's a dirty hack of course and hopefully the fix will be added in 
> next patch release.