You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@kyuubi.apache.org by "silvermissile (via GitHub)" <gi...@apache.org> on 2023/03/15 11:18:30 UTC
[GitHub] [kyuubi] silvermissile opened a new issue, #4525: [Bug] partition filed can not filter data when querying hive_catalog with hive connector
silvermissile opened a new issue, #4525:
URL: https://github.com/apache/kyuubi/issues/4525
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
### Search before asking
- [X] I have searched in the [issues](https://github.com/apache/kyuubi/issues?q=is%3Aissue) and found no similar issues.
### Describe the bug
partition filed not effective in where contition
``` sql
CREATE TABLE hive_store.tmp.students (name VARCHAR(64), address VARCHAR(64), student_id INT)
USING PARQUET PARTITIONED BY (student_id);
INSERT INTO hive_store.tmp.students VALUES
('Amy Smith', '123 Park Ave, San Jose', 111111),
('1Bush Brown', '456 Taylor St, Cupertino', 111111),
('Bob Brown', '456 Taylor St, Cupertino', 222222),
('2Bob Brown', '456 Taylor St, Cupertino', 222222),
('Bob Brown', '456 Taylor St, Cupertino', 333333),
('3Woods Brown', '456 Taylor St, Cupertino', 333333),
('Cathy Johnson', '789 Race Ave, Palo Alto', 333333);
select * from hive_store.tmp.students ;
+----------------+---------------------------+-------------+
| name | address | student_id |
+----------------+---------------------------+-------------+
| 1Bush Brown | 456 Taylor St, Cupertino | 111111 |
| Amy Smith | 123 Park Ave, San Jose | 111111 |
| 2Bob Brown | 456 Taylor St, Cupertino | 222222 |
| Bob Brown | 456 Taylor St, Cupertino | 222222 |
| Cathy Johnson | 789 Race Ave, Palo Alto | 333333 |
| 3Woods Brown | 456 Taylor St, Cupertino | 333333 |
| Bob Brown | 456 Taylor St, Cupertino | 333333 |
+----------------+---------------------------+-------------+
select student_id, count(1) from hive_store.tmp.students where group by student_id;
+-------------+-----------+
| student_id | count(1) |
+-------------+-----------+
| 111111 | 2 |
| 222222 | 2 |
| 333333 | 3 |
+-------------+-----------+
select * from hive_store.tmp.students where student_id= 333333;
+----------------+---------------------------+-------------+
| name | address | student_id |
+----------------+---------------------------+-------------+
| 1Bush Brown | 456 Taylor St, Cupertino | 111111 |
| Amy Smith | 123 Park Ave, San Jose | 111111 |
| 2Bob Brown | 456 Taylor St, Cupertino | 222222 |
| Bob Brown | 456 Taylor St, Cupertino | 222222 |
| Cathy Johnson | 789 Race Ave, Palo Alto | 333333 |
| 3Woods Brown | 456 Taylor St, Cupertino | 333333 |
| Bob Brown | 456 Taylor St, Cupertino | 333333 |
+----------------+---------------------------+-------------+
-- this is not expected
select * from hive_store.tmp.students where name="Bob Brown" ;
+------------+---------------------------+-------------+
| name | address | student_id |
+------------+---------------------------+-------------+
| Bob Brown | 456 Taylor St, Cupertino | 222222 |
| Bob Brown | 456 Taylor St, Cupertino | 333333 |
+------------+---------------------------+-------------+
```
```
### Affects Version(s)
1.7.0
### Kyuubi Server Log Output
```logtalk
nothing special
```
### Kyuubi Engine Log Output
```logtalk
nothing special
```
### Kyuubi Server Configurations
_No response_
### Kyuubi Engine Configurations
```yaml
spark.sql.catalog.hive_store=org.apache.kyuubi.spark.connector.hive.HiveTableCatalog
spark.sql.catalog.hive_store.spark.sql.hive.metastore.version=2.3.9
spark.sql.catalog.hive_store.hive.metastore.uris=thrift://172.19.79.130:30168
spark.sql.catalog.hive_store.spark.sql.hive.metastore.jars=path
spark.sql.catalog.hive_store.spark.sql.warehouse.dir=hdfs:///data/hive23/warehouse
spark.driver.cores=4;
spark.driver.memory=12g;
spark.executor.instances=2;spark.executor.cores=4;
spark.executor.memory=24g;
kyuubi.engine.share.level=connection;
spark.hadoop.hive.exec.dynamic.partition=true;
spark.hadoop.hive.exec.dynamic.partition.mode=nonstrict
```
### Additional context
nomal when query hive dirctly not from hive_catalog
### Are you willing to submit PR?
- [ ] Yes. I would be willing to submit a PR with guidance from the Kyuubi community to fix.
- [X] No. I cannot submit a PR at this time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [kyuubi] github-actions[bot] commented on issue #4525: [Bug] partition filed can not filter data when querying hive_catalog with hive connector
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #4525:
URL: https://github.com/apache/kyuubi/issues/4525#issuecomment-1469821962
Hello @silvermissile,
Thanks for finding the time to report the issue!
We really appreciate the community's efforts to improve Apache Kyuubi.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [kyuubi] pan3793 commented on issue #4525: [Bug] partition filed can not filter data when querying hive_catalog with hive connector
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on issue #4525:
URL: https://github.com/apache/kyuubi/issues/4525#issuecomment-1475506382
@silvermissile thanks for verification
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [kyuubi] Yikf commented on issue #4525: [Bug] partition filed can not filter data when querying hive_catalog with hive connector
Posted by "Yikf (via GitHub)" <gi...@apache.org>.
Yikf commented on issue #4525:
URL: https://github.com/apache/kyuubi/issues/4525#issuecomment-1469980088
Thanks @silvermissile for reporting this issue to the Apache Kyuubi community.
It looks like that hive-connector has a partition processing issue after Apache Spark pushes the partition predicate, will take a closer look.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [kyuubi] silvermissile commented on issue #4525: [Bug] partition filed can not filter data when querying hive_catalog with hive connector
Posted by "silvermissile (via GitHub)" <gi...@apache.org>.
silvermissile commented on issue #4525:
URL: https://github.com/apache/kyuubi/issues/4525#issuecomment-1475496710
After recompiling kyuubi-spark-connector-hive_2.12-1.7.0.jar, validation has passed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [kyuubi] pan3793 commented on issue #4525: [Bug] partition filed can not filter data when querying hive_catalog with hive connector
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on issue #4525:
URL: https://github.com/apache/kyuubi/issues/4525#issuecomment-1469824228
cc @Yikf
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [kyuubi] pan3793 closed issue #4525: [Bug] partition filed can not filter data when querying hive_catalog with hive connector
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 closed issue #4525: [Bug] partition filed can not filter data when querying hive_catalog with hive connector
URL: https://github.com/apache/kyuubi/issues/4525
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org