You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@kylin.apache.org by "mukvin (Jira)" <ji...@apache.org> on 2022/04/29 03:10:00 UTC
[jira] [Resolved] (KYLIN-5177) Kylin4.0.1使用S3存储无法清理数据
[ https://issues.apache.org/jira/browse/KYLIN-5177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
mukvin resolved KYLIN-5177.
---------------------------
Fix Version/s: v4.1.0
Resolution: Fixed
> Kylin4.0.1使用S3存储无法清理数据
> ----------------------
>
> Key: KYLIN-5177
> URL: https://issues.apache.org/jira/browse/KYLIN-5177
> Project: Kylin
> Issue Type: Bug
> Components: Storage - Parquet
> Affects Versions: v4.0.1
> Environment: aws emr5.33.1
> Reporter: peilun.zheng
> Assignee: mukvin
> Priority: Major
> Fix For: v4.1.0
>
>
> kylin4.0.1在aws emr5.33.1版本上使用bin/kylin.sh org.apache.kylin.tool.StorageCleanupJob
> 无法清理S3上的存储
> log:
> {code:java}
> Retrieving hadoop conf dir...
> Retrieving Spark dependency...
> OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=512M; support was removed in 8.0
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in [jar:file:/home/hadoop/kylin/tool/kylin-tool-4.0.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in [jar:file:/home/hadoop/kylin/ext/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in [jar:file:/home/hadoop/kylin/spark/jars/slf4j-log4j12-1.7.16.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
> 2022-04-12 07:35:09,765 INFO [main] common.KylinConfig:117 : Loading kylin-defaults.properties from file:/home/hadoop/kylin/tool/kylin-tool-4.0.1.jar!/kylin-defaults.properties
> 2022-04-12 07:35:09,776 INFO [main] common.KylinConfig:369 : Use KYLIN_HOME=/home/hadoop/kylin
> 2022-04-12 07:35:09,819 INFO [main] common.KylinConfigBase:263 : Kylin Config was updated with kylin.metadata.url.identifier : kylin_instance
> 2022-04-12 07:35:09,822 INFO [main] common.KylinConfigBase:263 : Kylin Config was updated with kylin.log.spark-executor-properties-file : /home/hadoop/kylin/conf/spark-executor-log4j.properties
> 2022-04-12 07:35:09,822 INFO [main] common.KylinConfig:160 : Initialized a new KylinConfig from getInstanceFromEnv : 1845904670
> 2022-04-12 07:35:10,011 WARN [main] util.NativeCodeLoader:62 : Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
> Exception in thread "main" java.lang.RuntimeException: org.apache.hadoop.fs.UnsupportedFileSystemException: No FileSystem for scheme "s3"
> at org.apache.kylin.common.KylinConfigBase.getHdfsWorkingDirectory(KylinConfigBase.java:313)
> at org.apache.kylin.common.util.HadoopUtil.getWorkingFileSystem(HadoopUtil.java:100)
> at org.apache.kylin.rest.job.StorageCleanupJob.<init>(StorageCleanupJob.java:121)
> at org.apache.kylin.tool.StorageCleanupJob.main(StorageCleanupJob.java:26)
> Caused by: org.apache.hadoop.fs.UnsupportedFileSystemException: No FileSystem for scheme "s3"
> at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:3273)
> at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3293)
> at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:120)
> at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3344)
> at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3312)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:478)
> at org.apache.hadoop.fs.Path.getFileSystem(Path.java:361)
> at org.apache.kylin.common.KylinConfigBase.getHdfsWorkingDirectory(KylinConfigBase.java:310)
> ... 3 more
> {code}
--
This message was sent by Atlassian Jira
(v8.20.7#820007)