You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@streampark.apache.org by "zfz748 (via GitHub)" <gi...@apache.org> on 2023/12/28 10:16:57 UTC
[D] s3存储，yarn application模式运行任务失败 [incubator-streampark]
GitHub user zfz748 created a discussion: s3存储，yarn application模式运行任务失败
Hadoop3.3.6，flink1.17.1，streampark2.1.2，在yarn application模式下使用s3存储，flink_home的plugins目录下已添加目录s3-fs-presto，运行任务报错，但是在集群上运行不会报错。
运行：CREATE CATALOG pre_catalog WITH (
'type' = 'paimon',
'warehouse' = 's3://paimon/warehouse',
's3.endpoint' = 'http://xxxx:xxxx',
's3.access-key' = 'xxxx',
's3.secret-key' = 'xxxx'
);
报错信息：
Caused by: org.apache.flink.core.fs.UnsupportedFileSystemSchemeException: Could not find a file system implementation for scheme 's3'. The scheme is directly supported by Flink through the following plugin(s): flink-s3-fs-hadoop, flink-s3-fs-presto. Please ensure that each plugin resides within its own subfolder within the plugins directory. See https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/plugins/ for more information. If you want to use a Hadoop file system for that scheme, please add the scheme to the configuration fs.allowed-fallback-filesystems. For a full list of supported file systems, please see https://nightlies.apache.org/flink/flink-docs-stable/ops/filesystems/.
GitHub link: https://github.com/apache/incubator-streampark/discussions/3437
----
This is an automatically sent email for dev@streampark.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@streampark.apache.org
Re: [D] s3存储，yarn application模式运行任务失败 [incubator-streampark]
Posted by "lumaozhiyexuanshou (via GitHub)" <gi...@apache.org>.
GitHub user lumaozhiyexuanshou added a comment to the discussion: s3存储，yarn application模式运行任务失败
可以将对应的hadoop-aws-xxx.jar 和 aws-java-sdk-bundle-x.x.xxx.jar传到streampark的lib目录下
GitHub link: https://github.com/apache/incubator-streampark/discussions/3437#discussioncomment-8702951
----
This is an automatically sent email for dev@streampark.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@streampark.apache.org
Re: [D] s3存储，yarn application模式运行任务失败 [incubator-streampark]
Posted by "zfz748 (via GitHub)" <gi...@apache.org>.
GitHub user zfz748 closed a discussion: s3存储，yarn application模式运行任务失败
Hadoop3.3.6，flink1.17.1，streampark2.1.2，在yarn application模式下使用s3存储，flink_home的plugins目录下已添加目录s3-fs-presto，运行任务报错，但是在集群上运行不会报错。
运行：CREATE CATALOG pre_catalog WITH (
'type' = 'paimon',
'warehouse' = 's3://paimon/warehouse',
's3.endpoint' = 'http://xxxx:xxxx',
's3.access-key' = 'xxxx',
's3.secret-key' = 'xxxx'
);
报错信息：
Caused by: org.apache.flink.core.fs.UnsupportedFileSystemSchemeException: Could not find a file system implementation for scheme 's3'. The scheme is directly supported by Flink through the following plugin(s): flink-s3-fs-hadoop, flink-s3-fs-presto. Please ensure that each plugin resides within its own subfolder within the plugins directory. See https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/plugins/ for more information. If you want to use a Hadoop file system for that scheme, please add the scheme to the configuration fs.allowed-fallback-filesystems. For a full list of supported file systems, please see https://nightlies.apache.org/flink/flink-docs-stable/ops/filesystems/.
GitHub link: https://github.com/apache/incubator-streampark/discussions/3437
----
This is an automatically sent email for dev@streampark.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@streampark.apache.org