[GitHub] [beam] mosche opened a new issue, #26321: [Bug]: FileSystems have a high risk of leaking resources due to the repeated invocation of FileSystems.setDefaultPipelineOptions

["mosche (via GitHub)" <gi...@apache.org>]

mosche opened a new issue, #26321:
URL: https://github.com/apache/beam/issues/26321

   ### What happened?
   
   The underlying issue here is the repeated initialization of filesystems triggered in `SerializablePipelineOptions` (#18430).
   Every call to `FileSystems.setDefaultPipelineOptions()` will silently replace any existing previous file systems without cleaning up resources.
   
   In the case of `S3FileSystem` this can leak threads if the file system was already in use.
   The simple test below shows the every increasing number of threads in the system.
   
   ```Java
   public class S3FileSystemTest {
     
   
     @Test
     public void testSetDefaultPipelineOptions() throws Exception {
       S3Options s3Options = s3Options();
       s3Options.setAwsCredentialsProvider(StaticCredentialsProvider.create(AwsBasicCredentials.create("key", "secret")));
   
       for (int i = 0; i < 1000; i++) {
         FileSystems.setDefaultPipelineOptions(s3Options);
         FileSystems.match(ImmutableList.of("s3://bucket/key*"));
         System.out.println("Number of threads " + Thread.activeCount());
       }
     }
   }  
   ```
   
   This is particularly bad with some runners, e.g. see #17645:
   > A normal Go Flink ValidatesRunner execution would contain the logging line from this class nearly 80,000 times, making it much more difficult to read the logs.
   
   ### Issue Priority
   
   Priority: 3 (minor)
   
   ### Issue Components
   
   - [ ] Component: Python SDK
   - [X] Component: Java SDK
   - [ ] Component: Go SDK
   - [ ] Component: Typescript SDK
   - [ ] Component: IO connector
   - [ ] Component: Beam examples
   - [ ] Component: Beam playground
   - [ ] Component: Beam katas
   - [ ] Component: Website
   - [ ] Component: Spark Runner
   - [ ] Component: Flink Runner
   - [ ] Component: Samza Runner
   - [ ] Component: Twister2 Runner
   - [ ] Component: Hazelcast Jet Runner
   - [ ] Component: Google Cloud Dataflow Runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] aromanenko-dev closed issue #26321: [Bug]: FileSystems have a high risk of leaking resources due to the repeated invocation of FileSystems.setDefaultPipelineOptions

["aromanenko-dev (via GitHub)" <gi...@apache.org>]

aromanenko-dev closed issue #26321: [Bug]: FileSystems have a high risk of leaking resources due to the repeated invocation of FileSystems.setDefaultPipelineOptions
URL: https://github.com/apache/beam/issues/26321


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org