You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kylin.apache.org by "peilun.zheng (Jira)" <ji...@apache.org> on 2022/04/12 07:55:00 UTC
[jira] [Created] (KYLIN-5177) Kylin4.0.1使用S3存储无法清理数据
peilun.zheng created KYLIN-5177:
-----------------------------------
Summary: Kylin4.0.1使用S3存储无法清理数据
Key: KYLIN-5177
URL: https://issues.apache.org/jira/browse/KYLIN-5177
Project: Kylin
Issue Type: Bug
Components: Storage - Parquet
Affects Versions: v4.0.1
Environment: aws emr5.33.1
Reporter: peilun.zheng
kylin4.0.1在aws emr5.33.1版本上使用bin/kylin.sh org.apache.kylin.tool.StorageCleanupJob
无法清理S3上的存储
log:
{code:java}
Retrieving hadoop conf dir...
Retrieving Spark dependency...
OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=512M; support was removed in 8.0
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/hadoop/kylin/tool/kylin-tool-4.0.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/hadoop/kylin/ext/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/hadoop/kylin/spark/jars/slf4j-log4j12-1.7.16.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
2022-04-12 07:35:09,765 INFO [main] common.KylinConfig:117 : Loading kylin-defaults.properties from file:/home/hadoop/kylin/tool/kylin-tool-4.0.1.jar!/kylin-defaults.properties
2022-04-12 07:35:09,776 INFO [main] common.KylinConfig:369 : Use KYLIN_HOME=/home/hadoop/kylin
2022-04-12 07:35:09,819 INFO [main] common.KylinConfigBase:263 : Kylin Config was updated with kylin.metadata.url.identifier : kylin_instance
2022-04-12 07:35:09,822 INFO [main] common.KylinConfigBase:263 : Kylin Config was updated with kylin.log.spark-executor-properties-file : /home/hadoop/kylin/conf/spark-executor-log4j.properties
2022-04-12 07:35:09,822 INFO [main] common.KylinConfig:160 : Initialized a new KylinConfig from getInstanceFromEnv : 1845904670
2022-04-12 07:35:10,011 WARN [main] util.NativeCodeLoader:62 : Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Exception in thread "main" java.lang.RuntimeException: org.apache.hadoop.fs.UnsupportedFileSystemException: No FileSystem for scheme "s3"
at org.apache.kylin.common.KylinConfigBase.getHdfsWorkingDirectory(KylinConfigBase.java:313)
at org.apache.kylin.common.util.HadoopUtil.getWorkingFileSystem(HadoopUtil.java:100)
at org.apache.kylin.rest.job.StorageCleanupJob.<init>(StorageCleanupJob.java:121)
at org.apache.kylin.tool.StorageCleanupJob.main(StorageCleanupJob.java:26)
Caused by: org.apache.hadoop.fs.UnsupportedFileSystemException: No FileSystem for scheme "s3"
at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:3273)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3293)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:120)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3344)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3312)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:478)
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:361)
at org.apache.kylin.common.KylinConfigBase.getHdfsWorkingDirectory(KylinConfigBase.java:310)
... 3 more
{code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)