You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@kylin.apache.org by "nichunen (JIRA)" <ji...@apache.org> on 2019/07/24 07:16:00 UTC
[jira] [Updated] (KYLIN-4107) StorageCleanupJob fails to delete
Hive tables with "Argument list too long" error
[ https://issues.apache.org/jira/browse/KYLIN-4107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
nichunen updated KYLIN-4107:
----------------------------
Fix Version/s: v3.0.0-beta
> StorageCleanupJob fails to delete Hive tables with "Argument list too long" error
> ---------------------------------------------------------------------------------
>
> Key: KYLIN-4107
> URL: https://issues.apache.org/jira/browse/KYLIN-4107
> Project: Kylin
> Issue Type: Bug
> Components: Storage - HBase
> Affects Versions: v2.6.2
> Environment: CentOS 7.6, HDP 2.6.5, Kylin 2.6.3
> Reporter: Vsevolod Ostapenko
> Priority: Major
> Fix For: v3.0.0-beta
>
>
> On a system with multiple Kylin developers that experiment with cube design and (re)build/drop cube segments often intermediate Hive tables and HBase left over tables accumulate very quickly.
> After a certain point storage cleanup cannot be executed using suggested method:
> {{${KYLIN_HOME}/bin/kylin.sh org.apache.kylin.tool.StorageCleanupJob --delete true}}
> Apparently, storage cleanup job creates a single shell command to drop all Hive tables, which fails to execute because command line is just too long. For example:
> {quote}
> 2019-07-23 17:47:31,611 ERROR [main] job.StorageCleanupJob:377 : Error during deleting Hive tables
> java.io.IOException: Cannot run program "/bin/bash": error=7, Argument list too long
> at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048)
> at org.apache.kylin.common.util.CliCommandExecutor.runNativeCommand(CliCommandExecutor.java:133)
> at org.apache.kylin.common.util.CliCommandExecutor.execute(CliCommandExecutor.java:89)
> at org.apache.kylin.common.util.CliCommandExecutor.execute(CliCommandExecutor.java:83)
> at org.apache.kylin.rest.job.StorageCleanupJob.deleteHiveTables(StorageCleanupJob.java:409)
> at org.apache.kylin.rest.job.StorageCleanupJob.cleanUnusedIntermediateHiveTableInternal(StorageCleanupJob.java:375)
> at org.apache.kylin.rest.job.StorageCleanupJob.cleanUnusedIntermediateHiveTable(StorageCleanupJob.java:278)
> at org.apache.kylin.rest.job.StorageCleanupJob.cleanup(StorageCleanupJob.java:151)
> at org.apache.kylin.rest.job.StorageCleanupJob.execute(StorageCleanupJob.java:145)
> at org.apache.kylin.common.util.AbstractApplication.execute(AbstractApplication.java:37)
> at org.apache.kylin.tool.StorageCleanupJob.main(StorageCleanupJob.java:27)
> Caused by: java.io.IOException: error=7, Argument list too long
> at java.lang.UNIXProcess.forkAndExec(Native Method)
> at java.lang.UNIXProcess.<init>(UNIXProcess.java:247)
> at java.lang.ProcessImpl.start(ProcessImpl.java:134)
> at java.lang.ProcessBuilder.start(ProcessBuilder.java:1029)
> ... 10 more
> {quote}
> Instead of composing one long command, storage cleanup need to generate a script and feed that into beeline or hive CLI.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)