You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "laurentgo (via GitHub)" <gi...@apache.org> on 2024/05/22 14:52:09 UTC
[I] [Java][Flight] Flight SQL tests are flaky [arrow]
laurentgo opened a new issue, #41782:
URL: https://github.com/apache/arrow/issues/41782
### Describe the bug, including details regarding any error messages, version, and platform.
Several test failures in `flight-sql` module have been observed in multiple job executions:
- https://github.com/apache/arrow/actions/runs/9185953424/job/25260768750
- https://github.com/apache/arrow/actions/runs/9185714602/job/25260156899?pr=41772
The reported issue is
```
Error: Errors:
Error: TestFlightSqlStreams.tearDown:224 ยป IllegalState Memory was leaked by query. Memory leaked: (250384)
Allocator(ROOT) 0/250384/250896/2147483647 (res/actual/peak/limit)
```
Note that there are also multiple messages about unclosed `ManagedChannels` in:
- `org.apache.arrow.flight.auth2.TestBasicAuth2`
- `org.apache.arrow.flight.core/org.apache.arrow.flight.TestFlightGrpcUtils.testMultipleGrpcServices`
- `org.apache.arrow.flight.core/org.apache.arrow.flight.TestDoExchange.setUp`
- `org.apache.arrow.flight.core/org.apache.arrow.flight.TestServerOptions.addHealthCheckService`
```
May 22, 2024 5:50:41 AM io.grpc.internal.ManagedChannelOrphanWrapper$ManagedChannelReference cleanQueue
SEVERE: *~*~*~ Previous channel ManagedChannelImpl{logId=505, target=directaddress:///localhost/127.0.0.1:5555} was garbage collected without being shut down! ~*~*~*
Make sure to call shutdown()/shutdownNow()
java.lang.RuntimeException: ManagedChannel allocation site
at io.grpc.internal@1.63.0/io.grpc.internal.ManagedChannelOrphanWrapper$ManagedChannelReference.<init>(ManagedChannelOrphanWrapper.java:102)
at io.grpc.internal@1.63.0/io.grpc.internal.ManagedChannelOrphanWrapper.<init>(ManagedChannelOrphanWrapper.java:60)
at io.grpc.internal@1.63.0/io.grpc.internal.ManagedChannelOrphanWrapper.<init>(ManagedChannelOrphanWrapper.java:51)
at io.grpc.internal@1.63.0/io.grpc.internal.ManagedChannelImplBuilder.build(ManagedChannelImplBuilder.java:672)
at io.grpc@1.63.0/io.grpc.ForwardingChannelBuilder2.build(ForwardingChannelBuilder2.java:260)
at org.apache.arrow.flight.core/org.apache.arrow.flight.TestServerOptions.addHealthCheckService(TestServerOptions.java:191)
at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
at java.base/java.lang.reflect.Method.invoke(Method.java:580)
```
but those seems to only cause warnings, not errors
### Component(s)
FlightRPC, Java
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Java][Flight] Flight SQL tests are flaky [arrow]
Posted by "laurentgo (via GitHub)" <gi...@apache.org>.
laurentgo commented on issue #41782:
URL: https://github.com/apache/arrow/issues/41782#issuecomment-2125441246
I wonder if the issue is somehow similar to the one hinted at https://github.com/apache/arrow/blob/9ba9253e8527a7f3e2c6e47e631e278b8ca84e53/java/flight/flight-core/src/test/java/org/apache/arrow/flight/TestDoExchange.java#L407 where somehow data is not fully reclaimed? (in my local test for `TestDoExchange` it seems the leak is related to allocations triggered by `Producer`)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Java][Flight] Flight SQL tests are flaky [arrow]
Posted by "vibhatha (via GitHub)" <gi...@apache.org>.
vibhatha commented on issue #41782:
URL: https://github.com/apache/arrow/issues/41782#issuecomment-2127151932
> We optionally track this info and it should be enabled in CI and the error message should be enhanced.
@lidavidm could I get a pointer in how to enable this in CIs?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Java][Flight] Flight SQL tests are flaky [arrow]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on issue #41782:
URL: https://github.com/apache/arrow/issues/41782#issuecomment-2125913169
Can anyone reproduce it locally?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Java][Flight] Flight SQL tests are flaky [arrow]
Posted by "vibhatha (via GitHub)" <gi...@apache.org>.
vibhatha commented on issue #41782:
URL: https://github.com/apache/arrow/issues/41782#issuecomment-2125434301
cc @lidavidm
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Java][Flight] Flight SQL tests are flaky [arrow]
Posted by "laurentgo (via GitHub)" <gi...@apache.org>.
laurentgo commented on issue #41782:
URL: https://github.com/apache/arrow/issues/41782#issuecomment-2127394651
Another occurence of a build failure: https://github.com/apache/arrow/actions/runs/9210096683/job/25336267356?pr=41800
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Java][Flight] Flight SQL tests are flaky [arrow]
Posted by "laurentgo (via GitHub)" <gi...@apache.org>.
laurentgo commented on issue #41782:
URL: https://github.com/apache/arrow/issues/41782#issuecomment-2126222117
I cannot for `TestFlightSqlStreams`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Java][Flight] Flight SQL tests are flaky [arrow]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on issue #41782:
URL: https://github.com/apache/arrow/issues/41782#issuecomment-2126346519
I think we need to do two things:
- tearDown needs to run after each test to narrow down the scope of the leaked memory.
- We need to properly add debug information to print out the leaked buffer's retain/release history and where it was allocated. We optionally track this info and it should be enabled in CI and the error message should be enhanced.
@vibhatha
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Java][Flight] Flight SQL tests are flaky [arrow]
Posted by "vibhatha (via GitHub)" <gi...@apache.org>.
vibhatha commented on issue #41782:
URL: https://github.com/apache/arrow/issues/41782#issuecomment-2126354657
I will take a look. But need sometime, probably earliest next week.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Java][Flight] Flight SQL tests are flaky [arrow]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on issue #41782:
URL: https://github.com/apache/arrow/issues/41782#issuecomment-2127247999
You'll have to look. We neutered it after performance complaints.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Java][Flight] Flight SQL tests are flaky [arrow]
Posted by "vibhatha (via GitHub)" <gi...@apache.org>.
vibhatha commented on issue #41782:
URL: https://github.com/apache/arrow/issues/41782#issuecomment-2128928817
Hard thing is replicating this locally. I tried a few times yesterday, no luck. Need to enable what @lidavidm mentioned in the CIs. I need to look into that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org