You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@kylin.apache.org by "mukvin (Jira)" <ji...@apache.org> on 2022/04/29 03:07:00 UTC
[jira] [Resolved] (KYLIN-5179) Kylin4无法通过StorageCleanupJob移除HDFS中失效的segment
[ https://issues.apache.org/jira/browse/KYLIN-5179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
mukvin resolved KYLIN-5179.
---------------------------
Resolution: Fixed
> Kylin4无法通过StorageCleanupJob移除HDFS中失效的segment
> --------------------------------------------
>
> Key: KYLIN-5179
> URL: https://issues.apache.org/jira/browse/KYLIN-5179
> Project: Kylin
> Issue Type: Bug
> Components: Storage - Parquet
> Affects Versions: v4.0.1
> Reporter: Sunbuck
> Assignee: mukvin
> Priority: Major
> Fix For: v4.0.1
>
> Attachments: image-2022-04-22-16-20-14-714.png, image-2022-04-22-16-22-09-804.png, image-2022-04-22-16-30-27-879.png, image-2022-04-22-16-31-18-621.png
>
>
> 在测试环境中通过KylinWeb中对cube进行数据构建后，想通过kylin.sh org.apache.kylin.tool.StorageCleanupJob进行失效segment进行清理，目前有效的cube为0RI的parquet storage：
> !image-2022-04-22-16-20-14-714.png!
> 在hdfs中查看该cube的segment情况：
> !image-2022-04-22-16-22-09-804.png!
> 其中MV6和C54的segment是我已经在Web中进行delete segment操作后的遗留垃圾数据，在按照Kylin Wiki（[https://cwiki.apache.org/confluence/display/KYLIN/How+to+clean+up+storage+in+Kylin+4]）中的说明中执行 kylin.sh org.apache.kylin.tool.StorageCleanupJob时，报错如下：
>
> {code:java}
> 2022-04-22 16:26:13,328 INFO [main] persistence.JDBCResourceStore:115 : Table [kylin_metadata] already exists
> 2022-04-22 16:26:13,332 INFO [main] persistence.JDBCResourceStore:115 : Table [kylin_metadata_log] already exists
> 2022-04-22 16:26:13,561 INFO [main] common.KylinConfig:493 : Creating new manager instance of class org.apache.kylin.metadata.cachesync.Broadcaster
> Running org.apache.kylin.rest.job.StorageCleanupJob
> 2022-04-22 16:26:13,571 INFO [main] job.StorageCleanupJob:143 : options: ''
> 2022-04-22 16:26:13,571 INFO [main] job.StorageCleanupJob:160 : ===================================================================
> delete : false; cleanupTableSnapshot : true; cleanupGlobalDict : true; cleanupJobTmp : false; cleanBeforeDate : Fri Apr 15 16:26:13 CST 2022.
> 2022-04-22 16:26:13,574 INFO [main] common.KylinConfig:493 : Creating new manager instance of class org.apache.kylin.metadata.project.ProjectManager
> 2022-04-22 16:26:13,576 INFO [main] project.ProjectManager:81 : Initializing ProjectManager with metadata url kylin_metadata@jdbc,url=jdbc:mysql://10.62.0.245:3306/kylin4,username=cloud,password=tplinkcloud,maxActive=10,maxIdle=10
> 2022-04-22 16:26:13,602 INFO [main] common.KylinConfig:493 : Creating new manager instance of class org.apache.kylin.cube.CubeManager
> 2022-04-22 16:26:13,608 INFO [main] cube.CubeManager:122 : Initializing CubeManager with config kylin_metadata@jdbc,url=jdbc:mysql://10.62.0.245:3306/kylin4,username=cloud,password=tplinkcloud,maxActive=10,maxIdle=10
> 2022-04-22 16:26:13,651 INFO [main] common.KylinConfig:493 : Creating new manager instance of class org.apache.kylin.cube.CubeDescManager
> 2022-04-22 16:26:13,652 INFO [main] cube.CubeDescManager:91 : Initializing CubeDescManager with config kylin_metadata@jdbc,url=jdbc:mysql://10.62.0.245:3306/kylin4,username=cloud,password=tplinkcloud,maxActive=10,maxIdle=10
> 2022-04-22 16:26:13,705 INFO [main] common.KylinConfig:493 : Creating new manager instance of class org.apache.kylin.metadata.model.DataModelManager
> 2022-04-22 16:26:13,709 INFO [main] common.KylinConfig:493 : Creating new manager instance of class org.apache.kylin.metadata.TableMetadataManager
> 2022-04-22 16:26:13,866 INFO [main] job.StorageCleanupJob:174 : Start to clean up unreferenced projects and cubes ...
> 2022-04-22 16:26:13,867 INFO [main] job.StorageCleanupJob:200 : Start to clean up no unreferenced segments ...
> 2022-04-22 16:26:13,867 WARN [main] job.StorageCleanupJob:222 : Cube path doesn't exist! The path is file:/kylin4/kylin_metadata/storage/parquet/mobile_core_analysis_cube
> 2022-04-22 16:26:13,868 WARN [main] job.StorageCleanupJob:222 : Cube path doesn't exist! The path is file:/kylin4/kylin_metadata/storage/parquet/storage_cube{code}
>
>
>
>
--
This message was sent by Atlassian Jira
(v8.20.7#820007)