You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2022/11/25 12:45:53 UTC
[GitHub] [iceberg] Kirito-Aus opened a new issue, #6273: java.sql.SQLException: Access denied for user 'hive'@'hadoopSlave0' (using password: YES)
Kirito-Aus opened a new issue, #6273:
URL: https://github.com/apache/iceberg/issues/6273
### Query engine
### Apache Iceberg version
1.0.0 (latest release)
### Query engine
Hive + Iceberg + MySQL + HDFS
## Please describe the bug š
### Application:
I want to use Hive+Spark to implement queries.
hostName HDFS YARN
hadoopMaster NameNodećDataNode NodeManagerćResourceManager
hadoopSlave0 DataNodećSecondaryNameNode NodeManager
hadoopSlave1 DataNode NodeManager
### Question:
There is no problem when Hive is used to add, delete, modify, and query data.
However, when using the table created by Iceberg, there is no problem when creating a new table and querying, but an error is reported when inserting new data.
It's ok to creating a new table and querying
hive> create table test_iceberg_tbl1(
id int ,
name string,
age int)
partitioned by (dt string)
stored by 'org.apache.iceberg.mr.hive.HiveIcebergStorageHandler';
hive> add jar /usr/local/hive/lib/iceberg-hive-runtime-1.0.0.jar;
hive> add jar /usr/local/hive/lib/libfb303-0.9.3.jar;
hive> add jar /usr/local/hive/lib/datanucleus-core-4.1.17.jar;
hive> add jar /usr/local/hive/lib/datanucleus-rdbms-4.1.19.jar;
hive> add jar /usr/local/hive/lib/datanucleus-api-jdo-4.2.4.jar;
hive> add jar /usr/local/hive/lib/mysql-connector-java-8.0.11.jar;
hive> add jar /usr/local/hive/lib/mysql-metadata-storage-0.12.0.jar;
hive> select * from test_iceberg_tbl1;
question arise
hive> insert into test_iceberg_tbl1 values (1,"zs",18,"20211212");
2022-11-25 17:02:12,348 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1669349629626_0010_m_000000_0:
2022-11-25 17:02:12,348 INFO [CommitterEvent Processor #1] org.apache.hadoop.hive.metastore.conf.MetastoreConf: Unable to find config file hivemetastore-site.xml
2022-11-25 17:02:12,348 INFO [CommitterEvent Processor #1] org.apache.hadoop.hive.metastore.conf.MetastoreConf: Found configuration file null
2022-11-25 17:02:12,348 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1669349629626_0010_m_000000_0 TaskAttempt Transitioned from SUCCESS_FINISHING_CONTAINER to SUCCEEDED
2022-11-25 17:02:12,348 INFO [CommitterEvent Processor #1] org.apache.hadoop.hive.metastore.conf.MetastoreConf: Unable to find config file metastore-site.xml
2022-11-25 17:02:12,348 INFO [CommitterEvent Processor #1] org.apache.hadoop.hive.metastore.conf.MetastoreConf: Found configuration file null
2022-11-25 17:02:12,348 INFO [ContainerLauncher #1] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_COMPLETED for container container_1669349629626_0010_01_000002 taskAttempt attempt_1669349629626_0010_m_000000_0
2022-11-25 17:02:12,605 INFO [CommitterEvent Processor #1] DataNucleus.Persistence: Property datanucleus.cache.level2 unknown - will be ignored
2022-11-25 17:02:12,792 INFO [CommitterEvent Processor #1] com.zaxxer.hikari.HikariDataSource: HikariPool-1 - Starting...
2022-11-25 17:02:12,802 WARN [CommitterEvent Processor #1] com.zaxxer.hikari.util.DriverDataSource: Registered driver with driverClassName=com.mysql.jdbc.Driver was not found, trying direct instantiation.
2022-11-25 17:02:14,022 ERROR [CommitterEvent Processor #1] com.zaxxer.hikari.pool.HikariPool: HikariPool-1 - Exception during pool initialization.
java.sql.SQLException: Access denied for user 'hive'@'hadoopSlave0' (using password: YES)
at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:127)
at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:95)
at com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:122)
at com.mysql.cj.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:862)
at com.mysql.cj.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:444)
at com.mysql.cj.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:230)
at com.mysql.cj.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:226)
at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:112)
at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:118)
at com.zaxxer.hikari.pool.PoolBase.newConnection(PoolBase.java:341)
at com.zaxxer.hikari.pool.PoolBase.newPoolEntry(PoolBase.java:193)
at com.zaxxer.hikari.pool.HikariPool.createPoolEntry(HikariPool.java:428)
at com.zaxxer.hikari.pool.HikariPool.checkFailFast(HikariPool.java:499)
at com.zaxxer.hikari.pool.HikariPool.<init>(HikariPool.java:112)
at com.zaxxer.hikari.HikariDataSource.<init>(HikariDataSource.java:72)
at org.datanucleus.store.rdbms.connectionpool.HikariCPConnectionPoolFactory.createConnectionPool(HikariCPConnectionPoolFactory.java:176)
at org.datanucleus.store.rdbms.ConnectionFactoryImpl.generateDataSources(ConnectionFactoryImpl.java:213)
at org.datanucleus.store.rdbms.ConnectionFactoryImpl.initialiseDataSources(ConnectionFactoryImpl.java:117)
......
at org.datanucleus.api.jdo.JDOPersistenceManagerFactory.freezeConfiguration(JDOPersistenceManagerFactory.java:817)
at org.datanucleus.api.jdo.JDOPersistenceManagerFactory.createPersistenceManagerFactory(JDOPersistenceManagerFactory.java:334)
.......
2022-11-25 17:02:14,024 ERROR [CommitterEvent Processor #1] DataNucleus.Datastore: Exception thrown creating StoreManager. See the nested exception
Error creating transactional connection factory
org.datanucleus.exceptions.NucleusException: Error creating transactional connection factory
at org.datanucleus.store.AbstractStoreManager.registerConnectionFactory(AbstractStoreManager.java:214)
at org.datanucleus.store.AbstractStoreManager.<init>(AbstractStoreManager.java:162)
at org.datanucleus.store.rdbms.RDBMSStoreManager.<init>(RDBMSStoreManager.java:285)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.datanucleus.plugin.NonManagedPluginRegistry.createExecutableExtension(NonManagedPluginRegistry.java:606)
at org.datanucleus.plugin.PluginManager.createExecutableExtension(PluginManager.java:301)
at org.datanucleus.NucleusContextHelper.createStoreManagerForProperties(NucleusContextHelper.java:133)
at org.datanucleus.PersistenceNucleusContextImpl.initialise(PersistenceNucleusContextImpl.java:422)
at org.datanucleus.api.jdo.JDOPersistenceManagerFactory.freezeConfiguration(JDOPersistenceManagerFactory.java:817)
at
### Debug checks:
I have tried many relevant methods on the Internet:
1. Turn off the firewall
2. Set the accessible hosts of MySQL connected users to%, that is, all can be accessed.
3. Set the ConnectionURL in Hive configuration file to ājdbc: mysql://hadoopMaster:3306/hive?createDatabaseIfNotExist=true ā
etc.
None can be solved
But when Hive is set to the local mode, you can insert and query normally. It seems that there is a problem with the connection with Yarn.
### Environment
We have tried with the following versions:
I want to use Hive+Spark to implement queries.
Build Hive on hadoopMaster and use MySQL to manage metadata.
Hive 3.1.2
Hadoop 3.1.3
MySQL Distrib 5.7.40
Iceberg 1.0.0
### Question
Question
So anybody know what is going on?
I think Yarn's connection was not successfully built.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] Kirito-Aus commented on issue #6273: java.sql.SQLException: Access denied for user 'hive'@'hadoopSlave0' (using password: YES)
Posted by GitBox <gi...@apache.org>.
Kirito-Aus commented on issue #6273:
URL: https://github.com/apache/iceberg/issues/6273#issuecomment-1328892886
@nastra @pvary I use the HiveCli to operate Hive. Hive can run normally independently, and can perform operations such as adding, deleting, modifying, querying, and creating tables to a non-Iceberg table. And once I open the local Hive mode, I can operate the Iceberg table without any problem. I will try the Hive 4.0.0-alpha-2 release and see whether it is applicable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] nastra closed issue #6273: java.sql.SQLException: Access denied for user 'hive'@'hadoopSlave0' (using password: YES)
Posted by GitBox <gi...@apache.org>.
nastra closed issue #6273: java.sql.SQLException: Access denied for user 'hive'@'hadoopSlave0' (using password: YES)
URL: https://github.com/apache/iceberg/issues/6273
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] pvary commented on issue #6273: java.sql.SQLException: Access denied for user 'hive'@'hadoopSlave0' (using password: YES)
Posted by GitBox <gi...@apache.org>.
pvary commented on issue #6273:
URL: https://github.com/apache/iceberg/issues/6273#issuecomment-1328726613
+1 on @nastra's comment. This seems like an issue with the Hive Metastore trying to write data into the backend mysql.
How are you using Hive? Are you using HiveCli? Are you able to write to a non-Iceberg table? Is using the Hive 4.0.0-alpha-2 release an option for you? It has more throughout and tested Iceberg integration OOB
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] nastra commented on issue #6273: java.sql.SQLException: Access denied for user 'hive'@'hadoopSlave0' (using password: YES)
Posted by GitBox <gi...@apache.org>.
nastra commented on issue #6273:
URL: https://github.com/apache/iceberg/issues/6273#issuecomment-1333664922
I'll close this for now. Feel free to re-open if necessary
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] nastra commented on issue #6273: java.sql.SQLException: Access denied for user 'hive'@'hadoopSlave0' (using password: YES)
Posted by GitBox <gi...@apache.org>.
nastra commented on issue #6273:
URL: https://github.com/apache/iceberg/issues/6273#issuecomment-1327520903
@Kirito-Aus I don't think this issue is related to Iceberg itself. Does the `hive` user have the right permissions to do inserts?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org