You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@kyuubi.apache.org by "Whojohn (via GitHub)" <gi...@apache.org> on 2024/12/26 06:34:32 UTC
[I] [Bug] jdbc or beeline statement cancel is useless in flink [kyuubi]
Whojohn opened a new issue, #6869:
URL: https://github.com/apache/kyuubi/issues/6869
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
### Search before asking
- [X] I have searched in the [issues](https://github.com/apache/kyuubi/issues?q=is%3Aissue) and found no similar issues.
### Describe the bug
# Which case i try
## Problem in beeline sitiation
```
./bin/beeline --incremental=true --verbose=true --incrementalBufferRows=1 --outputformat=table -u 'jdbc:hive2://dev:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=kyuubi;#kyuubi.engine.type=FLINK_SQL'
select sum(a),count(b) from sou
+--------------+---------+------------+
| EXPR$0 | EXPR$1 | op_inside |
+--------------+---------+------------+
| -1861035379 | 1 | +I |
| -1861035379 | 1 | -U |
| -317406356 | 2 | +U |
| -317406356 | 2 | -U |
| -684766290 | 3 | +U |
| -684766290 | 3 | -U |
| -15267534 | 4 | +U |
| -15267534 | 4 | -U |
| 2073227048 | 5 | +U |
| 2073227048 | 5 | -U |
| -734011806 | 6 | +U |
| -734011806 | 6 | -U |
| 786777747 | 7 | +U |
| 786777747 | 7 | -U |
| -707027945 | 8 | +U |
Interrupting... Please be patient this may take some time.
Interrupting... Please be patient this may take some time.
Interrupting... Please be patient this may take some time.
Interrupting... Please be patient this may take some time.
Interrupting... Please be patient this may take some time.
Interrupting... Please be patient this may take some time.
Interrupting... Please be patient this may take some time.
| -707027945 | 8 | -U |
| -681694912 | 9 | +U |
| -681694912 | 9 | -U |
| 950626509 | 10 | +U |
| 950626509 | 10 | -U |
| -1337848862 | 11 | +U |
| -1337848862 | 11 | -U |
| 2077867031 | 12 | +U |
| 2077867031 | 12 | -U |
| 1993739972 | 13 | +U |
| 1993739972 | 13 | -U |
| 1204146208 | 14 | +U |
| 1204146208 | 14 | -U |
------------- no matter how many ctrl-c send , it will stop until kyuubi.session.engine.flink.max.rows is reach.
--
```
## problem in KyuubiHiveDriver jdbc
- my code as follwing using
```
connection = DriverManager.getConnection("jdbc:hive2://localhost:10009/#kyuubi.engine.type=FLINK_SQL", null,
null);
statement = connection.createStatement();
statement.setFetchSize(1);
statement.execute("create table sou(a int,b string) with ('connector' = 'datagen','rows-per-second' = '20')");
resultSet = statement.executeQuery("select sum(a),count(a) from sou");
int a = 0;
while (resultSet.next()) {
a += 1;
if (a > 100) {
// !!! notice statement.cancel() will not stop flink job ;
statement.cancel();
break;
}
}
// do more querying ;
```
- flink sql cluster way(no matter how may cancel, it will still running util another query is send.)
Flink SQL> show jobs;
+----------------------------------+----------+----------+-------------------------+
| job id | job name | status | start time |
+----------------------------------+----------+----------+-------------------------+
| a91f330664aa725067ff641cd3fd6651 | collect | CANCELED | 2024-12-26T06:00:33.033 |
| f7a5538b8e510ccafa47830f982ed724 | collect | CANCELED | 2024-12-26T03:39:15.653 |
| 85dca8c9ab2068f1d48f63add195882c | collect | RUNNING | 2024-12-26T06:01:18.028 |
+----------------------------------+----------+----------+-------------------------+
3 rows in set
## Souce code readed
1. I found any server/engine/flink log i can not see any log about cacnel receive .
2. Debug in beeline or jdbc , client send KyuubiStatement#cancle is ok ,but remote bug in AbstractBackendService#cancelOperation not receive and singal. (debug in engine)
> both beeline and jdbc call KyuubiStatement#cancel to cancel.( guuest from code)
ps : if anybody help me how to debug in thrift cancel calling rpc , sumbit a pr is my pleasure； Forgive my poor English.
### Affects Version(s)
1.9.3
### Kyuubi Server Log Output
_No response_
### Kyuubi Engine Log Output
_No response_
### Kyuubi Server Configurations
```yaml
export FLINK_HOME=/data/flink-1.17.2
export FLINK_HADOOP_CLASSPATH=/data/flink-1.17.2/lib/flink-shaded-hadoop-2-uber-2.10.2-10.0.jar
export JAVA_HOME="/data/jdk-11.0.25+9"
```
### Kyuubi Engine Configurations
```yaml
kyuubi.server.thrift.resultset.default.fetch.size 1
kyuubi.engine.jdbc.fetch.size 1
hive.server2.thrift.resultset.default.fetch.size 1
kyuubi.operation.interrupt.on.cancel true
```
### Additional context
_No response_
### Are you willing to submit PR?
- [ ] Yes. I would be willing to submit a PR with guidance from the Kyuubi community to fix.
- [X] No. I cannot submit a PR at this time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [I] [Bug] jdbc or beeline statement cancel is useless in flink [kyuubi]
Posted by "Whojohn (via GitHub)" <gi...@apache.org>.
Whojohn commented on issue #6869:
URL: https://github.com/apache/kyuubi/issues/6869#issuecomment-2564642563
> @Whojohn would you like to send a PR to fix it?
@pan3793 why not ? My pleasure. I will push a pr and fix this as a option config.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [I] [Bug] jdbc or beeline statement cancel is useless in flink [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on issue #6869:
URL: https://github.com/apache/kyuubi/issues/6869#issuecomment-2563565997
@Whojohn would you like to send PR to fix it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [I] [Bug] Fix Beeline SQL querying cannot be stopped immediately by pressing Ctrl-C. [kyuubi]
Posted by "Whojohn (via GitHub)" <gi...@apache.org>.
Whojohn commented on issue #6869:
URL: https://github.com/apache/kyuubi/issues/6869#issuecomment-2572417653
# Problem Summary:
## There are two situations where a Beeline query cannot be stopped:
1. The current logic of the `BufferedRows` class causes it (data keeps coming in).
2. The current implementation logic of Flink SQL and JDBC causes the query to not stop in streaming mode when there is no new data and the end has been reached (temporarily resolved by #4806).
**This PR only fixes the first situation.**
PS:
> Attempted to fix the issue without using the #4806 timeout, as the current overall process requires non-empty data to be sent. It needs to be modified to allow sending empty data, but the changes are too extensive (involving JDBC statement, `flinksqlengine#IncrementalResultFetchIterator`, etc.). Will the community consider such a disruptive fix for situation 2?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [I] [Bug] jdbc or beeline statement cancel is useless in flink [kyuubi]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #6869:
URL: https://github.com/apache/kyuubi/issues/6869#issuecomment-2562212416
Hello @Whojohn,
Thanks for finding the time to report the issue!
We really appreciate the community's efforts to improve Apache Kyuubi.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [I] [Bug] jdbc or beeline statement cancel is useless in flink [kyuubi]
Posted by "Whojohn (via GitHub)" <gi...@apache.org>.
Whojohn commented on issue #6869:
URL: https://github.com/apache/kyuubi/issues/6869#issuecomment-2562257236
Regardless of the rpc bug, fix this bug by follow way .(!!!danger way !!!)
<img width="1112" alt="image" src="https://github.com/user-attachments/assets/655c6f1e-4b5f-4c66-9d37-66d594c85c86" />
<img width="1607" alt="image" src="https://github.com/user-attachments/assets/af5b5690-cb26-4641-bf3f-aadf34c05929" />
<img width="740" alt="image" src="https://github.com/user-attachments/assets/842eb8c7-2eba-4d4c-b7e8-bd0f3aabf66f" />
<img width="736" alt="image" src="https://github.com/user-attachments/assets/7310dad4-675e-47ee-9266-39837e838917" />
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org