[jira] [Updated] (FLINK-22734) ExtractionUtils#getClassReader causes "open too many files" error

["Chesnay Schepler (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/FLINK-22734?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Chesnay Schepler updated FLINK-22734:
-------------------------------------
    Component/s: Table SQL / Runtime

> ExtractionUtils#getClassReader causes "open too many files" error
> -----------------------------------------------------------------
>
>                 Key: FLINK-22734
>                 URL: https://issues.apache.org/jira/browse/FLINK-22734
>             Project: Flink
>          Issue Type: Bug
>          Components: Table SQL / Runtime
>    Affects Versions: 1.12.2
>            Reporter: zhiping lin
>            Priority: Major
>
> when we get classReader in ExtractionUtils 
>  
> {code:java}
>     private static ClassReader getClassReader(Class<?> cls) {        
>         final String className = cls.getName().replaceFirst("^.*\\.", "") + ".class";        
>         try {            
>             return new ClassReader(cls.getResourceAsStream(className));        
>         } catch (IOException e) {
>             throw new IllegalStateException("Could not instantiate ClassReader.", e);
>         }
>     }
> {code}
> we open a inputStream by "cls.getResourceAsStream(className)" and set it as a construct param for  ClassReader without any other steps to close it. Also ClassReader's construct method with only a outside inputStream, classReader won't close the inputStream while it finished read the file. This will leads to fd leak with "open too many files" error when we parse a lot independent sqls with different envs in one jvm(the case is that we want to extract the table/data lineage of our flink jobs).
> {code:java}
> lsof -p 8011 | wc -l 
> 10534
> {code}
> After checked all files and I found they are opened when loaded the udf jars for each loop to build the env. Then I test this method with try-with-source and it works well. So I wonder if it is necessary to change a little bit just like this
>  
> {code:java}
> private static ClassReader getClassReader(Class<?> cls) {
>    final String className = cls.getName().replaceFirst("^.*\\.", "") + ".class";
>    try (InputStream inputStream = cls.getResourceAsStream(className)) {
>       return new ClassReader(inputStream);
>    } catch (IOException e) {
>       throw new IllegalStateException("Could not instantiate ClassReader.", e);
>    }
> }
> {code}
>  
>  
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)