You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by GitBox <gi...@apache.org> on 2020/01/23 16:07:15 UTC
[GitHub] [flink] static-max opened a new pull request #10936: [FLINK-13689]
[Connectors/ElasticSearch] Fix thread leak in Elasticsearch connector when
cluster is down
static-max opened a new pull request #10936: [FLINK-13689] [Connectors/ElasticSearch] Fix thread leak in Elasticsearch connector when cluster is down
URL: https://github.com/apache/flink/pull/10936
## What is the purpose of the change
Fixes thread leaks caused by unclosed Elasticsearch clients
## Brief change log
- Created new method *verifyClientConnection()* in *ElasticsearchApiCallBridge*
- Moved test calls in existing Elasticsearch connectors to this new method
- verifyClientConnection() is called in *ElasticsearchSinkBase#open()* after client has been created. If an error occurs, close() in 'ElasticsearchSinkBase' can now clean up the client. This was not possible before as the Exception was thrown before the client variable was assigned.
## Verifying this change
This change is a trivial rework / code cleanup without any test coverage.
I tested the change on our development cluster by applying the change on top of release tag 1.9.1. No problems so far and the threads don't leak anymore. Recovery works fine, I tested different scenarios with the Elasticsearch cluster shut down,.
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): no
- The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no
- The serializers: no
- The runtime per-record code paths (performance sensitive): no
- Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: no
- The S3 file system connector: no
## Documentation
- Does this pull request introduce a new feature? no
- If yes, how is the feature documented? not applicable
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot commented on issue #10936: [FLINK-13689]
[Connectors/ElasticSearch] Fix thread leak in Elasticsearch connector when
cluster is down
Posted by GitBox <gi...@apache.org>.
flinkbot commented on issue #10936: [FLINK-13689] [Connectors/ElasticSearch] Fix thread leak in Elasticsearch connector when cluster is down
URL: https://github.com/apache/flink/pull/10936#issuecomment-577765273
<!--
Meta data
Hash:f12dc4a4d0a8c88a303cc67a283824b01c84b6ec Status:UNKNOWN URL:TBD TriggerType:PUSH TriggerID:f12dc4a4d0a8c88a303cc67a283824b01c84b6ec
-->
## CI report:
* f12dc4a4d0a8c88a303cc67a283824b01c84b6ec UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] static-max commented on issue #10936: [FLINK-13689]
[Connectors/ElasticSearch] Fix thread leak in Elasticsearch connector when
cluster is down
Posted by GitBox <gi...@apache.org>.
static-max commented on issue #10936: [FLINK-13689] [Connectors/ElasticSearch] Fix thread leak in Elasticsearch connector when cluster is down
URL: https://github.com/apache/flink/pull/10936#issuecomment-578054647
Thanks for the review!
I will try to create a test to verfiy that close() gets called, but I'm off for the next days, so it would take at least a week for me to respond, if that's ok.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10936: [FLINK-13689]
[Connectors/ElasticSearch] Fix thread leak in Elasticsearch connector when
cluster is down
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #10936: [FLINK-13689] [Connectors/ElasticSearch] Fix thread leak in Elasticsearch connector when cluster is down
URL: https://github.com/apache/flink/pull/10936#issuecomment-577765273
<!--
Meta data
Hash:f12dc4a4d0a8c88a303cc67a283824b01c84b6ec Status:FAILURE URL:https://travis-ci.com/flink-ci/flink/builds/145795770 TriggerType:PUSH TriggerID:f12dc4a4d0a8c88a303cc67a283824b01c84b6ec
Hash:f12dc4a4d0a8c88a303cc67a283824b01c84b6ec Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4588 TriggerType:PUSH TriggerID:f12dc4a4d0a8c88a303cc67a283824b01c84b6ec
Hash:60254f3a686e9abcc8e76c8204a05edff3c9cd56 Status:UNKNOWN URL:TBD TriggerType:PUSH TriggerID:60254f3a686e9abcc8e76c8204a05edff3c9cd56
-->
## CI report:
* f12dc4a4d0a8c88a303cc67a283824b01c84b6ec Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/145795770) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4588)
* 60254f3a686e9abcc8e76c8204a05edff3c9cd56 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10936: [FLINK-13689]
[Connectors/ElasticSearch] Fix thread leak in Elasticsearch connector when
cluster is down
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #10936: [FLINK-13689] [Connectors/ElasticSearch] Fix thread leak in Elasticsearch connector when cluster is down
URL: https://github.com/apache/flink/pull/10936#issuecomment-577765273
<!--
Meta data
Hash:f12dc4a4d0a8c88a303cc67a283824b01c84b6ec Status:FAILURE URL:https://travis-ci.com/flink-ci/flink/builds/145795770 TriggerType:PUSH TriggerID:f12dc4a4d0a8c88a303cc67a283824b01c84b6ec
Hash:f12dc4a4d0a8c88a303cc67a283824b01c84b6ec Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4588 TriggerType:PUSH TriggerID:f12dc4a4d0a8c88a303cc67a283824b01c84b6ec
Hash:60254f3a686e9abcc8e76c8204a05edff3c9cd56 Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/145839714 TriggerType:PUSH TriggerID:60254f3a686e9abcc8e76c8204a05edff3c9cd56
Hash:60254f3a686e9abcc8e76c8204a05edff3c9cd56 Status:SUCCESS URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4590 TriggerType:PUSH TriggerID:60254f3a686e9abcc8e76c8204a05edff3c9cd56
Hash:ff83bfbcb22e29a08d5da4eb18e2bbc1398614e6 Status:PENDING URL:https://travis-ci.com/flink-ci/flink/builds/146739275 TriggerType:PUSH TriggerID:ff83bfbcb22e29a08d5da4eb18e2bbc1398614e6
Hash:ff83bfbcb22e29a08d5da4eb18e2bbc1398614e6 Status:PENDING URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4675 TriggerType:PUSH TriggerID:ff83bfbcb22e29a08d5da4eb18e2bbc1398614e6
-->
## CI report:
* f12dc4a4d0a8c88a303cc67a283824b01c84b6ec Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/145795770) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4588)
* 60254f3a686e9abcc8e76c8204a05edff3c9cd56 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/145839714) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4590)
* ff83bfbcb22e29a08d5da4eb18e2bbc1398614e6 Travis: [PENDING](https://travis-ci.com/flink-ci/flink/builds/146739275) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4675)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] aljoscha closed pull request #10936: [FLINK-13689]
[Connectors/ElasticSearch] Fix thread leak in Elasticsearch connector when
cluster is down
Posted by GitBox <gi...@apache.org>.
aljoscha closed pull request #10936: [FLINK-13689] [Connectors/ElasticSearch] Fix thread leak in Elasticsearch connector when cluster is down
URL: https://github.com/apache/flink/pull/10936
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10936: [FLINK-13689]
[Connectors/ElasticSearch] Fix thread leak in Elasticsearch connector when
cluster is down
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #10936: [FLINK-13689] [Connectors/ElasticSearch] Fix thread leak in Elasticsearch connector when cluster is down
URL: https://github.com/apache/flink/pull/10936#issuecomment-577765273
<!--
Meta data
Hash:f12dc4a4d0a8c88a303cc67a283824b01c84b6ec Status:FAILURE URL:https://travis-ci.com/flink-ci/flink/builds/145795770 TriggerType:PUSH TriggerID:f12dc4a4d0a8c88a303cc67a283824b01c84b6ec
Hash:f12dc4a4d0a8c88a303cc67a283824b01c84b6ec Status:PENDING URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4588 TriggerType:PUSH TriggerID:f12dc4a4d0a8c88a303cc67a283824b01c84b6ec
-->
## CI report:
* f12dc4a4d0a8c88a303cc67a283824b01c84b6ec Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/145795770) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4588)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10936: [FLINK-13689]
[Connectors/ElasticSearch] Fix thread leak in Elasticsearch connector when
cluster is down
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #10936: [FLINK-13689] [Connectors/ElasticSearch] Fix thread leak in Elasticsearch connector when cluster is down
URL: https://github.com/apache/flink/pull/10936#issuecomment-577765273
<!--
Meta data
Hash:f12dc4a4d0a8c88a303cc67a283824b01c84b6ec Status:FAILURE URL:https://travis-ci.com/flink-ci/flink/builds/145795770 TriggerType:PUSH TriggerID:f12dc4a4d0a8c88a303cc67a283824b01c84b6ec
Hash:f12dc4a4d0a8c88a303cc67a283824b01c84b6ec Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4588 TriggerType:PUSH TriggerID:f12dc4a4d0a8c88a303cc67a283824b01c84b6ec
-->
## CI report:
* f12dc4a4d0a8c88a303cc67a283824b01c84b6ec Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/145795770) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4588)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10936: [FLINK-13689]
[Connectors/ElasticSearch] Fix thread leak in Elasticsearch connector when
cluster is down
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #10936: [FLINK-13689] [Connectors/ElasticSearch] Fix thread leak in Elasticsearch connector when cluster is down
URL: https://github.com/apache/flink/pull/10936#issuecomment-577765273
<!--
Meta data
Hash:f12dc4a4d0a8c88a303cc67a283824b01c84b6ec Status:FAILURE URL:https://travis-ci.com/flink-ci/flink/builds/145795770 TriggerType:PUSH TriggerID:f12dc4a4d0a8c88a303cc67a283824b01c84b6ec
Hash:f12dc4a4d0a8c88a303cc67a283824b01c84b6ec Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4588 TriggerType:PUSH TriggerID:f12dc4a4d0a8c88a303cc67a283824b01c84b6ec
Hash:60254f3a686e9abcc8e76c8204a05edff3c9cd56 Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/145839714 TriggerType:PUSH TriggerID:60254f3a686e9abcc8e76c8204a05edff3c9cd56
Hash:60254f3a686e9abcc8e76c8204a05edff3c9cd56 Status:SUCCESS URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4590 TriggerType:PUSH TriggerID:60254f3a686e9abcc8e76c8204a05edff3c9cd56
-->
## CI report:
* f12dc4a4d0a8c88a303cc67a283824b01c84b6ec Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/145795770) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4588)
* 60254f3a686e9abcc8e76c8204a05edff3c9cd56 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/145839714) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4590)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10936: [FLINK-13689]
[Connectors/ElasticSearch] Fix thread leak in Elasticsearch connector when
cluster is down
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #10936: [FLINK-13689] [Connectors/ElasticSearch] Fix thread leak in Elasticsearch connector when cluster is down
URL: https://github.com/apache/flink/pull/10936#issuecomment-577765273
<!--
Meta data
Hash:f12dc4a4d0a8c88a303cc67a283824b01c84b6ec Status:FAILURE URL:https://travis-ci.com/flink-ci/flink/builds/145795770 TriggerType:PUSH TriggerID:f12dc4a4d0a8c88a303cc67a283824b01c84b6ec
Hash:f12dc4a4d0a8c88a303cc67a283824b01c84b6ec Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4588 TriggerType:PUSH TriggerID:f12dc4a4d0a8c88a303cc67a283824b01c84b6ec
Hash:60254f3a686e9abcc8e76c8204a05edff3c9cd56 Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/145839714 TriggerType:PUSH TriggerID:60254f3a686e9abcc8e76c8204a05edff3c9cd56
Hash:60254f3a686e9abcc8e76c8204a05edff3c9cd56 Status:SUCCESS URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4590 TriggerType:PUSH TriggerID:60254f3a686e9abcc8e76c8204a05edff3c9cd56
Hash:ff83bfbcb22e29a08d5da4eb18e2bbc1398614e6 Status:PENDING URL:https://travis-ci.com/flink-ci/flink/builds/146739275 TriggerType:PUSH TriggerID:ff83bfbcb22e29a08d5da4eb18e2bbc1398614e6
Hash:ff83bfbcb22e29a08d5da4eb18e2bbc1398614e6 Status:SUCCESS URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4675 TriggerType:PUSH TriggerID:ff83bfbcb22e29a08d5da4eb18e2bbc1398614e6
-->
## CI report:
* f12dc4a4d0a8c88a303cc67a283824b01c84b6ec Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/145795770) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4588)
* 60254f3a686e9abcc8e76c8204a05edff3c9cd56 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/145839714) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4590)
* ff83bfbcb22e29a08d5da4eb18e2bbc1398614e6 Travis: [PENDING](https://travis-ci.com/flink-ci/flink/builds/146739275) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4675)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] aljoscha commented on issue #10936: [FLINK-13689]
[Connectors/ElasticSearch] Fix thread leak in Elasticsearch connector when
cluster is down
Posted by GitBox <gi...@apache.org>.
aljoscha commented on issue #10936: [FLINK-13689] [Connectors/ElasticSearch] Fix thread leak in Elasticsearch connector when cluster is down
URL: https://github.com/apache/flink/pull/10936#issuecomment-578065485
Cool, that's fine. Enjoy your days off!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10936: [FLINK-13689]
[Connectors/ElasticSearch] Fix thread leak in Elasticsearch connector when
cluster is down
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #10936: [FLINK-13689] [Connectors/ElasticSearch] Fix thread leak in Elasticsearch connector when cluster is down
URL: https://github.com/apache/flink/pull/10936#issuecomment-577765273
<!--
Meta data
Hash:f12dc4a4d0a8c88a303cc67a283824b01c84b6ec Status:FAILURE URL:https://travis-ci.com/flink-ci/flink/builds/145795770 TriggerType:PUSH TriggerID:f12dc4a4d0a8c88a303cc67a283824b01c84b6ec
Hash:f12dc4a4d0a8c88a303cc67a283824b01c84b6ec Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4588 TriggerType:PUSH TriggerID:f12dc4a4d0a8c88a303cc67a283824b01c84b6ec
Hash:60254f3a686e9abcc8e76c8204a05edff3c9cd56 Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/145839714 TriggerType:PUSH TriggerID:60254f3a686e9abcc8e76c8204a05edff3c9cd56
Hash:60254f3a686e9abcc8e76c8204a05edff3c9cd56 Status:SUCCESS URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4590 TriggerType:PUSH TriggerID:60254f3a686e9abcc8e76c8204a05edff3c9cd56
Hash:ff83bfbcb22e29a08d5da4eb18e2bbc1398614e6 Status:UNKNOWN URL:TBD TriggerType:PUSH TriggerID:ff83bfbcb22e29a08d5da4eb18e2bbc1398614e6
-->
## CI report:
* f12dc4a4d0a8c88a303cc67a283824b01c84b6ec Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/145795770) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4588)
* 60254f3a686e9abcc8e76c8204a05edff3c9cd56 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/145839714) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4590)
* ff83bfbcb22e29a08d5da4eb18e2bbc1398614e6 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] aljoscha commented on issue #10936: [FLINK-13689]
[Connectors/ElasticSearch] Fix thread leak in Elasticsearch connector when
cluster is down
Posted by GitBox <gi...@apache.org>.
aljoscha commented on issue #10936: [FLINK-13689] [Connectors/ElasticSearch] Fix thread leak in Elasticsearch connector when cluster is down
URL: https://github.com/apache/flink/pull/10936#issuecomment-580710192
Thanks a lot! I merged this now.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10936: [FLINK-13689]
[Connectors/ElasticSearch] Fix thread leak in Elasticsearch connector when
cluster is down
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #10936: [FLINK-13689] [Connectors/ElasticSearch] Fix thread leak in Elasticsearch connector when cluster is down
URL: https://github.com/apache/flink/pull/10936#issuecomment-577765273
<!--
Meta data
Hash:f12dc4a4d0a8c88a303cc67a283824b01c84b6ec Status:FAILURE URL:https://travis-ci.com/flink-ci/flink/builds/145795770 TriggerType:PUSH TriggerID:f12dc4a4d0a8c88a303cc67a283824b01c84b6ec
Hash:f12dc4a4d0a8c88a303cc67a283824b01c84b6ec Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4588 TriggerType:PUSH TriggerID:f12dc4a4d0a8c88a303cc67a283824b01c84b6ec
Hash:60254f3a686e9abcc8e76c8204a05edff3c9cd56 Status:PENDING URL:https://travis-ci.com/flink-ci/flink/builds/145839714 TriggerType:PUSH TriggerID:60254f3a686e9abcc8e76c8204a05edff3c9cd56
Hash:60254f3a686e9abcc8e76c8204a05edff3c9cd56 Status:PENDING URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4590 TriggerType:PUSH TriggerID:60254f3a686e9abcc8e76c8204a05edff3c9cd56
-->
## CI report:
* f12dc4a4d0a8c88a303cc67a283824b01c84b6ec Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/145795770) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4588)
* 60254f3a686e9abcc8e76c8204a05edff3c9cd56 Travis: [PENDING](https://travis-ci.com/flink-ci/flink/builds/145839714) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4590)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] aljoscha commented on a change in pull request #10936:
[FLINK-13689] [Connectors/ElasticSearch] Fix thread leak in Elasticsearch
connector when cluster is down
Posted by GitBox <gi...@apache.org>.
aljoscha commented on a change in pull request #10936: [FLINK-13689] [Connectors/ElasticSearch] Fix thread leak in Elasticsearch connector when cluster is down
URL: https://github.com/apache/flink/pull/10936#discussion_r370527319
##########
File path: flink-connectors/flink-connector-elasticsearch6/src/main/java/org/apache/flink/streaming/connectors/elasticsearch6/Elasticsearch6ApiCallBridge.java
##########
@@ -139,4 +127,19 @@ public RequestIndexer createBulkProcessorIndexer(
flushOnCheckpoint,
numPendingRequestsRef);
}
+
+ @Override
+ public void verifyClientConnection(RestHighLevelClient client) throws IOException {
+ if (LOG.isInfoEnabled()) {
+ LOG.info("Pinging Elasticsearch cluster via hosts {} ...", httpHosts);
+ }
+
+ if (!client.ping()) {
+ throw new RuntimeException("There are no reachable Elasticsearch nodes!");
+ }
+
+ if (LOG.isInfoEnabled()) {
+ LOG.info("Created Elasticsearch RestHighLevelClient connected to {}", httpHosts.toString());
Review comment:
```suggestion
LOG.info("Elasticsearch RestHighLevelClient is connected to {}", httpHosts.toString());
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] static-max commented on issue #10936: [FLINK-13689]
[Connectors/ElasticSearch] Fix thread leak in Elasticsearch connector when
cluster is down
Posted by GitBox <gi...@apache.org>.
static-max commented on issue #10936: [FLINK-13689] [Connectors/ElasticSearch] Fix thread leak in Elasticsearch connector when cluster is down
URL: https://github.com/apache/flink/pull/10936#issuecomment-580188405
I did not find a convenient way to check if close() was called. The Elasticsearch client does not expose the underlying CloseableHttpAsyncClient, so we need to use reflection and I don't know if this is wanted by the Flink team.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] aljoscha commented on a change in pull request #10936:
[FLINK-13689] [Connectors/ElasticSearch] Fix thread leak in Elasticsearch
connector when cluster is down
Posted by GitBox <gi...@apache.org>.
aljoscha commented on a change in pull request #10936: [FLINK-13689] [Connectors/ElasticSearch] Fix thread leak in Elasticsearch connector when cluster is down
URL: https://github.com/apache/flink/pull/10936#discussion_r370527156
##########
File path: flink-connectors/flink-connector-elasticsearch5/src/main/java/org/apache/flink/streaming/connectors/elasticsearch5/Elasticsearch5ApiCallBridge.java
##########
@@ -132,4 +119,19 @@ public void configureBulkProcessorBackoff(
builder.setBackoffPolicy(backoffPolicy);
}
+
+ @Override
+ public void verifyClientConnection(TransportClient client) {
+ // verify that we actually are connected to a cluster
+ if (client.connectedNodes().isEmpty()) {
+ // close the transportClient here
+ IOUtils.closeQuietly(client);
+
+ throw new RuntimeException("Elasticsearch client is not connected to any Elasticsearch nodes!");
+ }
+
+ if (LOG.isInfoEnabled()) {
+ LOG.info("Created Elasticsearch TransportClient with connected nodes {}", client.connectedNodes());
Review comment:
```suggestion
LOG.info("Elasticsearch TransportClient is connected to nodes {}", client.connectedNodes());
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] aljoscha commented on a change in pull request #10936:
[FLINK-13689] [Connectors/ElasticSearch] Fix thread leak in Elasticsearch
connector when cluster is down
Posted by GitBox <gi...@apache.org>.
aljoscha commented on a change in pull request #10936: [FLINK-13689] [Connectors/ElasticSearch] Fix thread leak in Elasticsearch connector when cluster is down
URL: https://github.com/apache/flink/pull/10936#discussion_r370527528
##########
File path: flink-connectors/flink-connector-elasticsearch7/src/main/java/org/apache/flink/streaming/connectors/elasticsearch7/Elasticsearch7ApiCallBridge.java
##########
@@ -140,4 +128,19 @@ public RequestIndexer createBulkProcessorIndexer(
flushOnCheckpoint,
numPendingRequestsRef);
}
+
+ @Override
+ public void verifyClientConnection(RestHighLevelClient client) throws IOException {
+ if (LOG.isInfoEnabled()) {
+ LOG.info("Pinging Elasticsearch cluster via hosts {} ...", httpHosts);
+ }
+
+ if (!client.ping(RequestOptions.DEFAULT)) {
+ throw new RuntimeException("There are no reachable Elasticsearch nodes!");
+ }
+
+ if (LOG.isInfoEnabled()) {
+ LOG.info("Created Elasticsearch RestHighLevelClient connected to {}", httpHosts.toString());
Review comment:
```suggestion
LOG.info("Elasticsearch RestHighLevelClient is connected to {}", httpHosts.toString());
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] aljoscha commented on issue #10936: [FLINK-13689]
[Connectors/ElasticSearch] Fix thread leak in Elasticsearch connector when
cluster is down
Posted by GitBox <gi...@apache.org>.
aljoscha commented on issue #10936: [FLINK-13689] [Connectors/ElasticSearch] Fix thread leak in Elasticsearch connector when cluster is down
URL: https://github.com/apache/flink/pull/10936#issuecomment-578046815
Do you think you might be able to add a test that verifies that we close the client even if the `createClient()` methods throws an exception. Not sure that's easily possible so I'm not against merging without such a test.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] static-max edited a comment on issue #10936: [FLINK-13689]
[Connectors/ElasticSearch] Fix thread leak in Elasticsearch connector when
cluster is down
Posted by GitBox <gi...@apache.org>.
static-max edited a comment on issue #10936: [FLINK-13689] [Connectors/ElasticSearch] Fix thread leak in Elasticsearch connector when cluster is down
URL: https://github.com/apache/flink/pull/10936#issuecomment-578054647
Thanks for the review!
I will try to create a test to verify that close() gets called, but I'm off for the next days, so it would take at least a week for me to respond, if that's ok.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10936: [FLINK-13689]
[Connectors/ElasticSearch] Fix thread leak in Elasticsearch connector when
cluster is down
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #10936: [FLINK-13689] [Connectors/ElasticSearch] Fix thread leak in Elasticsearch connector when cluster is down
URL: https://github.com/apache/flink/pull/10936#issuecomment-577765273
<!--
Meta data
Hash:f12dc4a4d0a8c88a303cc67a283824b01c84b6ec Status:FAILURE URL:https://travis-ci.com/flink-ci/flink/builds/145795770 TriggerType:PUSH TriggerID:f12dc4a4d0a8c88a303cc67a283824b01c84b6ec
Hash:f12dc4a4d0a8c88a303cc67a283824b01c84b6ec Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4588 TriggerType:PUSH TriggerID:f12dc4a4d0a8c88a303cc67a283824b01c84b6ec
Hash:60254f3a686e9abcc8e76c8204a05edff3c9cd56 Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/145839714 TriggerType:PUSH TriggerID:60254f3a686e9abcc8e76c8204a05edff3c9cd56
Hash:60254f3a686e9abcc8e76c8204a05edff3c9cd56 Status:PENDING URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4590 TriggerType:PUSH TriggerID:60254f3a686e9abcc8e76c8204a05edff3c9cd56
-->
## CI report:
* f12dc4a4d0a8c88a303cc67a283824b01c84b6ec Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/145795770) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4588)
* 60254f3a686e9abcc8e76c8204a05edff3c9cd56 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/145839714) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4590)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] aljoscha commented on a change in pull request #10936:
[FLINK-13689] [Connectors/ElasticSearch] Fix thread leak in Elasticsearch
connector when cluster is down
Posted by GitBox <gi...@apache.org>.
aljoscha commented on a change in pull request #10936: [FLINK-13689] [Connectors/ElasticSearch] Fix thread leak in Elasticsearch connector when cluster is down
URL: https://github.com/apache/flink/pull/10936#discussion_r370526826
##########
File path: flink-connectors/flink-connector-elasticsearch2/src/main/java/org/apache/flink/streaming/connectors/elasticsearch2/Elasticsearch2ApiCallBridge.java
##########
@@ -126,4 +113,19 @@ public void configureBulkProcessorBackoff(
builder.setBackoffPolicy(backoffPolicy);
}
+
+ @Override
+ public void verifyClientConnection(TransportClient client) {
+ // verify that we actually are connected to a cluster
+ if (client.connectedNodes().isEmpty()) {
+ // close the transportClient here
+ IOUtils.closeQuietly(client);
+
+ throw new RuntimeException("Elasticsearch client is not connected to any Elasticsearch nodes!");
+ }
+
+ if (LOG.isInfoEnabled()) {
+ LOG.info("Created Elasticsearch TransportClient with connected nodes {}", client.connectedNodes());
Review comment:
```suggestion
LOG.info("Elasticsearch TransportClient is connected to nodes {}", client.connectedNodes());
```
This is not `createClient()` anymore.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot commented on issue #10936: [FLINK-13689]
[Connectors/ElasticSearch] Fix thread leak in Elasticsearch connector when
cluster is down
Posted by GitBox <gi...@apache.org>.
flinkbot commented on issue #10936: [FLINK-13689] [Connectors/ElasticSearch] Fix thread leak in Elasticsearch connector when cluster is down
URL: https://github.com/apache/flink/pull/10936#issuecomment-577750433
Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community
to review your pull request. We will use this comment to track the progress of the review.
## Automated Checks
Last check on commit f12dc4a4d0a8c88a303cc67a283824b01c84b6ec (Thu Jan 23 16:09:59 UTC 2020)
**Warnings:**
* No documentation files were touched! Remember to keep the Flink docs up to date!
<sub>Mention the bot in a comment to re-run the automated checks.</sub>
## Review Progress
* ❓ 1. The [description] looks good.
* ❓ 2. There is [consensus] that the contribution should go into to Flink.
* ❓ 3. Needs [attention] from.
* ❓ 4. The change fits into the overall [architecture].
* ❓ 5. Overall code [quality] is good.
Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process.<details>
The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required <summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`)
- `@flinkbot approve all` to approve all aspects
- `@flinkbot approve-until architecture` to approve everything until `architecture`
- `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention
- `@flinkbot disapprove architecture` to remove an approval you gave earlier
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services