You are viewing a plain text version of this content. The canonical link for it is here.
Posted to gitbox@hive.apache.org by "VenuReddy2103 (via GitHub)" <gi...@apache.org> on 2023/02/10 01:52:43 UTC
[GitHub] [hive] VenuReddy2103 opened a new pull request, #4050: HIVE-27065: Exception in partition column statistics update with SQL Server db when histogram statistics is not enabled
VenuReddy2103 opened a new pull request, #4050:
URL: https://github.com/apache/hive/pull/4050
### What changes were proposed in this pull request?
Used `preparedStatement.setBytes()` to set the histogram.
### Why are the changes needed?
`java.sql.BatchUpdateException` thrown from `DirectSqlUpdateStat.insertIntoPartColStatTable()` with SQL Server db when histogram statistics is not enabled.
**Exception Stack:**
`2023-02-10T05:16:42,921 ERROR [HiveServer2-Background-Pool: Thread-112] metastore.DirectSqlUpdateStat: Unable to update Column stats for dynpart
java.sql.BatchUpdateException: Implicit conversion from data type varchar to varbinary(max) is not allowed. Use the CONVERT function to run this query.
at com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement.executeBatch(SQLServerPreparedStatement.java:2303) ~[mssql-jdbc-6.2.1.jre8.jar:?]
at org.apache.hive.com.zaxxer.hikari.pool.ProxyStatement.executeBatch(ProxyStatement.java:127) ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at org.apache.hive.com.zaxxer.hikari.pool.HikariProxyPreparedStatement.executeBatch(HikariProxyPreparedStatement.java) ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at org.apache.hadoop.hive.metastore.DirectSqlUpdateStat.insertIntoPartColStatTable(DirectSqlUpdateStat.java:281) ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at org.apache.hadoop.hive.metastore.DirectSqlUpdateStat.updatePartitionColumnStatistics(DirectSqlUpdateStat.java:612) ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at org.apache.hadoop.hive.metastore.MetaStoreDirectSql.updatePartitionColumnStatisticsBatch(MetaStoreDirectSql.java:3063) ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at org.apache.hadoop.hive.metastore.ObjectStore.updatePartitionColumnStatisticsInBatch(ObjectStore.java:9943) ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]`
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Tested manually with SQL Server
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] sonarcloud[bot] commented on pull request #4050: HIVE-27065: Exception in partition column statistics update with SQL Server db when histogram statistics is not enabled
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #4050:
URL: https://github.com/apache/hive/pull/4050#issuecomment-1425098335
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4050)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4050&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4050&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4050&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4050&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4050&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4050&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4050&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4050&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4050&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4050&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4050&resolved=false&types=CODE_SMELL) [0 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4050&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4050&metric=coverage&view=list) No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4050&metric=duplicated_lines_density&view=list) No Duplication information
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] asolimando closed pull request #4050: HIVE-27065: Exception in partition column statistics update with SQL Server db when histogram statistics is not enabled
Posted by "asolimando (via GitHub)" <gi...@apache.org>.
asolimando closed pull request #4050: HIVE-27065: Exception in partition column statistics update with SQL Server db when histogram statistics is not enabled
URL: https://github.com/apache/hive/pull/4050
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] sonarcloud[bot] commented on pull request #4050: HIVE-27065: Exception in partition column statistics update with SQL Server db when histogram statistics is not enabled
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #4050:
URL: https://github.com/apache/hive/pull/4050#issuecomment-1425318905
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4050)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4050&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4050&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4050&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4050&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4050&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4050&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4050&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4050&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4050&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4050&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4050&resolved=false&types=CODE_SMELL) [0 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4050&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4050&metric=coverage&view=list) No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4050&metric=duplicated_lines_density&view=list) No Duplication information
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] VenuReddy2103 commented on pull request #4050: HIVE-27065: Exception in partition column statistics update with SQL Server db when histogram statistics is not enabled
Posted by "VenuReddy2103 (via GitHub)" <gi...@apache.org>.
VenuReddy2103 commented on PR #4050:
URL: https://github.com/apache/hive/pull/4050#issuecomment-1427522236
> > > @VenuReddy2103 thanks for your PR, I have a couple of questions: did you try with other backend DBs as well? Did you encounter a similar error? I am curious because this part of the code does not seem to be DB specific, so I'd rather confirm quickly if that's an issue only with MSSQL or if it's more general, and that we don't regress on other databases by using `setBytes()` in place of `setObject()`.
> > > Have you tried the same code when histogram is actually computed? Did it fail? I am asking because it might be an artifact of the uninitialized `histogram` field.
> > > Since the `bitvector` field is totally analogous (same data type, `byte array`, and written with `setObject()`), apart from the fact that it is always computed, I am wondering if that's not the real difference, as it's successfully written to MSSQL with the `.setObject()` call.
> > > Can you update here with the actual return value of `mPartitionColumnStatistics.getHistogram()` and of `mPartitionColumnStatistics.getBitVector()` at the time of the call by putting a breakpoint in the `insertIntoPartColStatTable` function? That would help shed some light on what's actually going on there.
> > > Thanks!
> >
> >
> > @asolimando Thanks for reviewing PR. Please find reply for queries.
> >
> > 1. I've tried with Mysql, Postgresql, Oracle and Derby as well. This issue is not observed with them. It is observed only with MSSql. When histogram statistics is disabled, `histogram` field in `MPartitionColumnStatistics` is initialized default with empty byte array. And `mPartitionColumnStatistics.getHistogram()` returns null if `histogram.length` is 0. With `preparedStatement.setObject(18, mPartitionColumnStatistics.getHistogram())` invocation, since the object is null, `SQLServerPreparedStatement.setObject()` inferred the jdbc type as `JDBCType.CHAR`. Hence this issue. Whereas with `preparedStatement.setBytes()`, though null is passed, it don't try to infer type again. Have verified this change with MSSql, MySql, PostgreSql, Oracle and Derby.
> > Have found a similar issue fix here - https://github.com/trinodb/trino/pull/4846/files#:~:text=statement.setBytes(index%2C%20null)%3B
> > 2. When the histogram is computed, it didn't fail because value is not null and driver could infer the type correctly.
> > 3. Yes this issue is not seen with `bitvector` as it is always computed.
> > 4. Debugger screenshots for `MPartitionColumnStatistics.getBitVector()` and `MPartitionColumnStatistics.getHistogram()`
> >
> > <img alt="image" width="974" src="https://user-images.githubusercontent.com/35334869/218059642-40495bcf-a547-48cb-bc5d-0a21697e3c07.png">
> > <img alt="image" width="800" src="https://user-images.githubusercontent.com/35334869/218060043-960bcad7-0578-4b9b-8183-068f9672eaf7.png">
>
> @VenuReddy2103, thanks a lot for checking, it all makes sense to me. One last thing, could you try with the other metastore backend databases if we are still OK with the current fix, both in case of empty (null) and non-empty histogram array?
>
> If you can confirm that, then we are good to go.
@asolimando Have verified the change with MSSql, MySql, PostgreSql, Oracle and Derby dbs with both empty and non-empty histogram cases.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] VenuReddy2103 commented on pull request #4050: HIVE-27065: Exception in partition column statistics update with SQL Server db when histogram statistics is not enabled
Posted by "VenuReddy2103 (via GitHub)" <gi...@apache.org>.
VenuReddy2103 commented on PR #4050:
URL: https://github.com/apache/hive/pull/4050#issuecomment-1425563880
> @VenuReddy2103 thanks for your PR, I have a couple of questions: did you try with other backend DBs as well? Did you encounter a similar error? I am curious because this part of the code does not seem to be DB specific, so I'd rather confirm quickly if that's an issue only with MSSQL or if it's more general, and that we don't regress on other databases by using `setBytes()` in place of `setObject()`.
>
> Have you tried the same code when histogram is actually computed? Did it fail? I am asking because it might be an artifact of the uninitialized `histogram` field.
>
> Since the `bitvector` field is totally analogous (same data type, `byte array`, and written with `setObject()`), apart from the fact that it is always computed, I am wondering if that's not the real difference, as it's successfully written to MSSQL with the `.setObject()` call.
>
> Can you update here with the actual return value of `mPartitionColumnStatistics.getHistogram()` and of `mPartitionColumnStatistics.getBitVector()` at the time of the call by putting a breakpoint in the `insertIntoPartColStatTable` function? That would help shed some light on what's actually going on there.
>
> Thanks!
@asolimando Thanks for reviewing PR. Please find reply for queries.
1. I've tried with Mysql, Postgresql, Oracle and Derby as well. This issue is not observed with them. It is observed only with MSSql. When histogram statistics is disabled, `histogram` field in `MPartitionColumnStatistics` is initialized default with empty byte array. And `mPartitionColumnStatistics.getHistogram()` returns null if `histogram.length` is 0. With `preparedStatement.setObject(18, mPartitionColumnStatistics.getHistogram())` invocation, since the object is null, `SQLServerPreparedStatement.setObject()` inferred the jdbc type as `JDBCType.CHAR`. Hence this issue. Whereas with `preparedStatement.setBytes()`, though null is passed, it don't try to infer type again. Have verified this change with MSSql, MySql, PostgreSql, Oracle and Derby.
Have found a similar issue fix here - https://github.com/trinodb/trino/pull/4846/files#:~:text=statement.setBytes(index%2C%20null)%3B
2. When the histogram is computed, it didn't fail because value is not null and driver could infer the type correctly.
3. Yes this issue is not seen with `bitvector` as it is always computed.
4. Debugger screenshots for MPartitionColumnStatistics.getBitVector() and MPartitionColumnStatistics.getHistogram()
<img width="974" alt="image" src="https://user-images.githubusercontent.com/35334869/218059642-40495bcf-a547-48cb-bc5d-0a21697e3c07.png">
<img width="800" alt="image" src="https://user-images.githubusercontent.com/35334869/218060043-960bcad7-0578-4b9b-8183-068f9672eaf7.png">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] asolimando commented on pull request #4050: HIVE-27065: Exception in partition column statistics update with SQL Server db when histogram statistics is not enabled
Posted by "asolimando (via GitHub)" <gi...@apache.org>.
asolimando commented on PR #4050:
URL: https://github.com/apache/hive/pull/4050#issuecomment-1425333371
@VenuReddy2103 thanks for your PR, I have a couple of questions: did you try with other backend DBs as well? Did you encounter a similar error? I am curious because this part of the code does not seem to be DB specific, so I'd rather confirm quickly if that's an issue only with MSSQL or if it's more general, and that we don't regress on other databases by using `setBytes()` in place of `setObject()`.
Have you tried the same code when histogram is actually computed? Did it fail? I am asking because it might be an artifact of the uninitialized `histogram` field.
Since the `bitvector` field is totally analogous (same data type, `byte array`, and written with `setObject()`), apart from the fact that it is always computed, I am wondering if that's not the real difference, as it's successfully written to MSSQL with the `.setObject()` call.
Can you update here with the actual return value of `mPartitionColumnStatistics.getHistogram()` and of `mPartitionColumnStatistics.getBitVector()` at the time of the call by putting a breakpoint in the `insertIntoPartColStatTable` function? That would help shed some light on what's actually going on there.
Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] asolimando commented on pull request #4050: HIVE-27065: Exception in partition column statistics update with SQL Server db when histogram statistics is not enabled
Posted by "asolimando (via GitHub)" <gi...@apache.org>.
asolimando commented on PR #4050:
URL: https://github.com/apache/hive/pull/4050#issuecomment-1426092206
> > @VenuReddy2103 thanks for your PR, I have a couple of questions: did you try with other backend DBs as well? Did you encounter a similar error? I am curious because this part of the code does not seem to be DB specific, so I'd rather confirm quickly if that's an issue only with MSSQL or if it's more general, and that we don't regress on other databases by using `setBytes()` in place of `setObject()`.
> > Have you tried the same code when histogram is actually computed? Did it fail? I am asking because it might be an artifact of the uninitialized `histogram` field.
> > Since the `bitvector` field is totally analogous (same data type, `byte array`, and written with `setObject()`), apart from the fact that it is always computed, I am wondering if that's not the real difference, as it's successfully written to MSSQL with the `.setObject()` call.
> > Can you update here with the actual return value of `mPartitionColumnStatistics.getHistogram()` and of `mPartitionColumnStatistics.getBitVector()` at the time of the call by putting a breakpoint in the `insertIntoPartColStatTable` function? That would help shed some light on what's actually going on there.
> > Thanks!
>
> @asolimando Thanks for reviewing PR. Please find reply for queries.
>
> 1. I've tried with Mysql, Postgresql, Oracle and Derby as well. This issue is not observed with them. It is observed only with MSSql. When histogram statistics is disabled, `histogram` field in `MPartitionColumnStatistics` is initialized default with empty byte array. And `mPartitionColumnStatistics.getHistogram()` returns null if `histogram.length` is 0. With `preparedStatement.setObject(18, mPartitionColumnStatistics.getHistogram())` invocation, since the object is null, `SQLServerPreparedStatement.setObject()` inferred the jdbc type as `JDBCType.CHAR`. Hence this issue. Whereas with `preparedStatement.setBytes()`, though null is passed, it don't try to infer type again. Have verified this change with MSSql, MySql, PostgreSql, Oracle and Derby.
> Have found a similar issue fix here - https://github.com/trinodb/trino/pull/4846/files#:~:text=statement.setBytes(index%2C%20null)%3B
> 2. When the histogram is computed, it didn't fail because value is not null and driver could infer the type correctly.
> 3. Yes this issue is not seen with `bitvector` as it is always computed.
> 4. Debugger screenshots for `MPartitionColumnStatistics.getBitVector()` and `MPartitionColumnStatistics.getHistogram()`
>
> <img alt="image" width="974" src="https://user-images.githubusercontent.com/35334869/218059642-40495bcf-a547-48cb-bc5d-0a21697e3c07.png">
>
> <img alt="image" width="800" src="https://user-images.githubusercontent.com/35334869/218060043-960bcad7-0578-4b9b-8183-068f9672eaf7.png">
@VenuReddy2103, thanks a lot for checking, it all makes sense to me.
One last thing, could you try with the other metastore backend databases if we are still OK with the current fix, both in case of empty (null) and non-empty histogram array?
If you can confirm that, then we are good to go.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org