You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by Sanjay Gupta <sa...@gmail.com> on 2023/12/18 04:18:30 UTC
Fwd: Help with Docker Apache/Hive metastore using mysql remote database
Issue with Docker container using mysql RDBMS ( Failed to load driver)
https://hub.docker.com/r/apache/hive
According to readme
Launch Standalone Metastore With External RDBMS
(Postgres/Oracle/MySql/MsSql)
I want to use MySQL
I tried com.mysql.jdbc.Driver or com.mysql.cj.jdbc.Driver
docker run -it -d -p 9083:9083 --env SERVICE_NAME=metastore
--add-host=host.docker.internal:host-gateway \
--env DB_DRIVER=mysql \
--env SERVICE_OPTS="-Djavax.jdo.option.ConnectionDriverName=com.mysql.jdbc.Driver
-Djavax.jdo.option.ConnectionURL=jdbc:mysql://host.docker.internal:3306/hive?createDatabaseIfNotExist=true
-Djavax.jdo.option.ConnectionUserName=hive
-Djavax.jdo.option.ConnectionPassword=password" \
--mount source=warehouse,target=/opt/hive/data/warehouse \
--name metastore-standalone apache/hive:${HIVE_VERSION}
docker run -it -d -p 9083:9083 --env SERVICE_NAME=metastore
--add-host=host.docker.internal:host-gateway \
--env DB_DRIVER=mysql \
--env SERVICE_OPTS="-Djavax.jdo.option.ConnectionDriverName=com.mysql.cj.jdbc.Driver
-Djavax.jdo.option.ConnectionURL=jdbc:mysql://host.docker.internal:3306/hive?createDatabaseIfNotExist=true
-Djavax.jdo.option.ConnectionUserName=hive
-Djavax.jdo.option.ConnectionPassword=password" \
--mount source=warehouse,target=/opt/hive/data/warehouse \
--name metastore-standalone apache/hive:${HIVE_VERSION}
Docker logs shows this for both drivers ( same error )
docker logs f3
+ : mysql
+ SKIP_SCHEMA_INIT=false
+ export HIVE_CONF_DIR=/opt/hive/conf
+ HIVE_CONF_DIR=/opt/hive/conf
+ '[' -d '' ']'
+ export 'HADOOP_CLIENT_OPTS= -Xmx1G
-Djavax.jdo.option.ConnectionDriverName=com.mysql.cj.jdbc.Driver
-Djavax.jdo.option.ConnectionURL=jdbc:mysql://host.docker.internal:3306/hive?createDatabaseIfNotExist=true
-Djavax.jdo.option.ConnectionUserName=hive
-Djavax.jdo.option.ConnectionPassword=hive'
+ HADOOP_CLIENT_OPTS=' -Xmx1G
-Djavax.jdo.option.ConnectionDriverName=com.mysql.cj.jdbc.Driver
-Djavax.jdo.option.ConnectionURL=jdbc:mysql://host.docker.internal:3306/hive?createDatabaseIfNotExist=true
-Djavax.jdo.option.ConnectionUserName=hive
-Djavax.jdo.option.ConnectionPassword=hive'
+ [[ false == \f\a\l\s\e ]]
+ initialize_hive
+ /opt/hive/bin/schematool -dbType mysql -initSchema
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in
[jar:file:/opt/hive/lib/log4j-slf4j-impl-2.17.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in
[jar:file:/opt/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Metastore connection URL:
jdbc:mysql://host.docker.internal:3306/hive?createDatabaseIfNotExist=true
Metastore Connection Driver : com.mysql.cj.jdbc.Driver
Metastore connection User: hive
org.apache.hadoop.hive.metastore.HiveMetaException: Failed to load driver
Underlying cause: java.lang.ClassNotFoundException : com.mysql.cj.jdbc.Driver
Use --verbose for detailed stacktrace.
*** schemaTool failed ***
+ '[' 1 -eq 0 ']'
+ echo 'Schema initialization failed!'
Schema initialization failed!
+ exit 1
Any idea, why I am getting failed to load driver for MySQL DB.
Isn't docker container comes with MySQL Driver ?
Docker container exits so I can't check whether driver is already installed.
Let me know, what I can do to make it work.
--
Thanks
Sanjay Gupta
--
Thanks
Sanjay Gupta
--
Thanks
Sanjay Gupta
Re: Help with Docker Apache/Hive metastore using mysql remote database
Posted by Akshat m <ak...@gmail.com>.
Hi,
You can download it from here:
https://repo1.maven.org/maven2/mysql/mysql-connector-java/8.0.29/mysql-connector-java-8.0.29.jar
Regards,
Akshat
On Thu, Dec 21, 2023 at 1:40 AM Sanjay Gupta <sa...@gmail.com> wrote:
> There is no mysql driver file in following repo
> https://repo1.maven.org/maven2/org
>
> On Mon, Dec 18, 2023 at 3:10 AM Simhadri G <si...@gmail.com> wrote:
> >
> > We can modify the Dockerfile to wget the necessary driver and copy it to
> /opt/hive/lib/ . This should make it work. The diff is attached below:
> >
> >
> > diff --git a/packaging/src/docker/Dockerfile
> b/packaging/src/docker/Dockerfile
> > --- a/packaging/src/docker/Dockerfile (revision
> dceaf810b32fc266e3e657fdaefcd4507f2191b5)
> > +++ b/packaging/src/docker/Dockerfile (date 1702897518609)
> > @@ -80,6 +80,9 @@
> >
> > ENV PATH=$HIVE_HOME/bin:$HADOOP_HOME/bin:$PATH
> >
> > +RUN wget
> https://repo1.maven.org/maven2/org/postgresql/postgresql/42.5.1/postgresql-42.5.1.jar
> > +RUN cp /postgresql-42.5.1.jar /opt/hive/lib/
> > +
> > COPY entrypoint.sh /
> > COPY conf $HIVE_HOME/conf
> > RUN chmod +x /entrypoint.sh
> >
> > On Mon, Dec 18, 2023, 12:59 PM Ayush Saxena <ay...@gmail.com> wrote:
> >>
> >> I think the similar problem is being chased as part of
> >> https://github.com/apache/hive/pull/4948
> >>
> >> On Mon, 18 Dec 2023 at 09:48, Sanjay Gupta <sa...@gmail.com> wrote:
> >> >
> >> >
> >> >
> >> >
> >> > Issue with Docker container using mysql RDBMS ( Failed to load driver)
> >> >
> >> > https://hub.docker.com/r/apache/hive
> >> >
> >> > According to readme
> >> >
> >> > Launch Standalone Metastore With External RDBMS
> (Postgres/Oracle/MySql/MsSql)
> >> >
> >> > I want to use MySQL
> >> >
> >> > I tried com.mysql.jdbc.Driver or com.mysql.cj.jdbc.Driver
> >> >
> >> > docker run -it -d -p 9083:9083 --env SERVICE_NAME=metastore
> --add-host=host.docker.internal:host-gateway \
> >> > --env DB_DRIVER=mysql \
> >> > --env
> SERVICE_OPTS="-Djavax.jdo.option.ConnectionDriverName=com.mysql.jdbc.Driver
> -Djavax.jdo.option.ConnectionURL=jdbc:mysql://host.docker.internal:3306/hive?createDatabaseIfNotExist=true
> -Djavax.jdo.option.ConnectionUserName=hive
> -Djavax.jdo.option.ConnectionPassword=password" \
> >> > --mount source=warehouse,target=/opt/hive/data/warehouse \
> >> > --name metastore-standalone apache/hive:${HIVE_VERSION}
> >> >
> >> >
> >> > docker run -it -d -p 9083:9083 --env SERVICE_NAME=metastore
> --add-host=host.docker.internal:host-gateway \
> >> > --env DB_DRIVER=mysql \
> >> > --env
> SERVICE_OPTS="-Djavax.jdo.option.ConnectionDriverName=com.mysql.cj.jdbc.Driver
> -Djavax.jdo.option.ConnectionURL=jdbc:mysql://host.docker.internal:3306/hive?createDatabaseIfNotExist=true
> -Djavax.jdo.option.ConnectionUserName=hive
> -Djavax.jdo.option.ConnectionPassword=password" \
> >> > --mount source=warehouse,target=/opt/hive/data/warehouse \
> >> > --name metastore-standalone apache/hive:${HIVE_VERSION}
> >> >
> >> > Docker logs shows this for both drivers ( same error )
> >> >
> >> > docker logs f3
> >> > + : mysql
> >> > + SKIP_SCHEMA_INIT=false
> >> > + export HIVE_CONF_DIR=/opt/hive/conf
> >> > + HIVE_CONF_DIR=/opt/hive/conf
> >> > + '[' -d '' ']'
> >> > + export 'HADOOP_CLIENT_OPTS= -Xmx1G
> -Djavax.jdo.option.ConnectionDriverName=com.mysql.cj.jdbc.Driver
> -Djavax.jdo.option.ConnectionURL=jdbc:mysql://host.docker.internal:3306/hive?createDatabaseIfNotExist=true
> -Djavax.jdo.option.ConnectionUserName=hive
> -Djavax.jdo.option.ConnectionPassword=hive'
> >> > + HADOOP_CLIENT_OPTS=' -Xmx1G
> -Djavax.jdo.option.ConnectionDriverName=com.mysql.cj.jdbc.Driver
> -Djavax.jdo.option.ConnectionURL=jdbc:mysql://host.docker.internal:3306/hive?createDatabaseIfNotExist=true
> -Djavax.jdo.option.ConnectionUserName=hive
> -Djavax.jdo.option.ConnectionPassword=hive'
> >> > + [[ false == \f\a\l\s\e ]]
> >> > + initialize_hive
> >> > + /opt/hive/bin/schematool -dbType mysql -initSchema
> >> > SLF4J: Class path contains multiple SLF4J bindings.
> >> > SLF4J: Found binding in
> [jar:file:/opt/hive/lib/log4j-slf4j-impl-2.17.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> >> > SLF4J: Found binding in
> [jar:file:/opt/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> >> > SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
> >> > SLF4J: Actual binding is of type
> [org.apache.logging.slf4j.Log4jLoggerFactory]
> >> > Metastore connection URL:
> jdbc:mysql://host.docker.internal:3306/hive?createDatabaseIfNotExist=true
> >> > Metastore Connection Driver : com.mysql.cj.jdbc.Driver
> >> > Metastore connection User: hive
> >> > org.apache.hadoop.hive.metastore.HiveMetaException: Failed to load
> driver
> >> > Underlying cause: java.lang.ClassNotFoundException :
> com.mysql.cj.jdbc.Driver
> >> > Use --verbose for detailed stacktrace.
> >> > *** schemaTool failed ***
> >> > + '[' 1 -eq 0 ']'
> >> > + echo 'Schema initialization failed!'
> >> > Schema initialization failed!
> >> > + exit 1
> >> >
> >> > Any idea, why I am getting failed to load driver for MySQL DB.
> >> >
> >> > Isn't docker container comes with MySQL Driver ?
> >> >
> >> > Docker container exits so I can't check whether driver is already
> installed.
> >> >
> >> > Let me know, what I can do to make it work.
> >> >
> >> > --
> >> >
> >> >
> >> > Thanks
> >> > Sanjay Gupta
> >> >
> >> >
> >> >
> >> > --
> >> >
> >> > Thanks
> >> > Sanjay Gupta
> >> >
> >> >
> >> >
> >> > --
> >> >
> >> > Thanks
> >> > Sanjay Gupta
> >> >
>
>
>
> --
>
> Thanks
> Sanjay Gupta
>
Re: Help with Docker Apache/Hive metastore using mysql remote database
Posted by Akshat m <ak...@gmail.com>.
Hi,
You can download it from here:
https://repo1.maven.org/maven2/mysql/mysql-connector-java/8.0.29/mysql-connector-java-8.0.29.jar
Regards,
Akshat
On Thu, Dec 21, 2023 at 1:40 AM Sanjay Gupta <sa...@gmail.com> wrote:
> There is no mysql driver file in following repo
> https://repo1.maven.org/maven2/org
>
> On Mon, Dec 18, 2023 at 3:10 AM Simhadri G <si...@gmail.com> wrote:
> >
> > We can modify the Dockerfile to wget the necessary driver and copy it to
> /opt/hive/lib/ . This should make it work. The diff is attached below:
> >
> >
> > diff --git a/packaging/src/docker/Dockerfile
> b/packaging/src/docker/Dockerfile
> > --- a/packaging/src/docker/Dockerfile (revision
> dceaf810b32fc266e3e657fdaefcd4507f2191b5)
> > +++ b/packaging/src/docker/Dockerfile (date 1702897518609)
> > @@ -80,6 +80,9 @@
> >
> > ENV PATH=$HIVE_HOME/bin:$HADOOP_HOME/bin:$PATH
> >
> > +RUN wget
> https://repo1.maven.org/maven2/org/postgresql/postgresql/42.5.1/postgresql-42.5.1.jar
> > +RUN cp /postgresql-42.5.1.jar /opt/hive/lib/
> > +
> > COPY entrypoint.sh /
> > COPY conf $HIVE_HOME/conf
> > RUN chmod +x /entrypoint.sh
> >
> > On Mon, Dec 18, 2023, 12:59 PM Ayush Saxena <ay...@gmail.com> wrote:
> >>
> >> I think the similar problem is being chased as part of
> >> https://github.com/apache/hive/pull/4948
> >>
> >> On Mon, 18 Dec 2023 at 09:48, Sanjay Gupta <sa...@gmail.com> wrote:
> >> >
> >> >
> >> >
> >> >
> >> > Issue with Docker container using mysql RDBMS ( Failed to load driver)
> >> >
> >> > https://hub.docker.com/r/apache/hive
> >> >
> >> > According to readme
> >> >
> >> > Launch Standalone Metastore With External RDBMS
> (Postgres/Oracle/MySql/MsSql)
> >> >
> >> > I want to use MySQL
> >> >
> >> > I tried com.mysql.jdbc.Driver or com.mysql.cj.jdbc.Driver
> >> >
> >> > docker run -it -d -p 9083:9083 --env SERVICE_NAME=metastore
> --add-host=host.docker.internal:host-gateway \
> >> > --env DB_DRIVER=mysql \
> >> > --env
> SERVICE_OPTS="-Djavax.jdo.option.ConnectionDriverName=com.mysql.jdbc.Driver
> -Djavax.jdo.option.ConnectionURL=jdbc:mysql://host.docker.internal:3306/hive?createDatabaseIfNotExist=true
> -Djavax.jdo.option.ConnectionUserName=hive
> -Djavax.jdo.option.ConnectionPassword=password" \
> >> > --mount source=warehouse,target=/opt/hive/data/warehouse \
> >> > --name metastore-standalone apache/hive:${HIVE_VERSION}
> >> >
> >> >
> >> > docker run -it -d -p 9083:9083 --env SERVICE_NAME=metastore
> --add-host=host.docker.internal:host-gateway \
> >> > --env DB_DRIVER=mysql \
> >> > --env
> SERVICE_OPTS="-Djavax.jdo.option.ConnectionDriverName=com.mysql.cj.jdbc.Driver
> -Djavax.jdo.option.ConnectionURL=jdbc:mysql://host.docker.internal:3306/hive?createDatabaseIfNotExist=true
> -Djavax.jdo.option.ConnectionUserName=hive
> -Djavax.jdo.option.ConnectionPassword=password" \
> >> > --mount source=warehouse,target=/opt/hive/data/warehouse \
> >> > --name metastore-standalone apache/hive:${HIVE_VERSION}
> >> >
> >> > Docker logs shows this for both drivers ( same error )
> >> >
> >> > docker logs f3
> >> > + : mysql
> >> > + SKIP_SCHEMA_INIT=false
> >> > + export HIVE_CONF_DIR=/opt/hive/conf
> >> > + HIVE_CONF_DIR=/opt/hive/conf
> >> > + '[' -d '' ']'
> >> > + export 'HADOOP_CLIENT_OPTS= -Xmx1G
> -Djavax.jdo.option.ConnectionDriverName=com.mysql.cj.jdbc.Driver
> -Djavax.jdo.option.ConnectionURL=jdbc:mysql://host.docker.internal:3306/hive?createDatabaseIfNotExist=true
> -Djavax.jdo.option.ConnectionUserName=hive
> -Djavax.jdo.option.ConnectionPassword=hive'
> >> > + HADOOP_CLIENT_OPTS=' -Xmx1G
> -Djavax.jdo.option.ConnectionDriverName=com.mysql.cj.jdbc.Driver
> -Djavax.jdo.option.ConnectionURL=jdbc:mysql://host.docker.internal:3306/hive?createDatabaseIfNotExist=true
> -Djavax.jdo.option.ConnectionUserName=hive
> -Djavax.jdo.option.ConnectionPassword=hive'
> >> > + [[ false == \f\a\l\s\e ]]
> >> > + initialize_hive
> >> > + /opt/hive/bin/schematool -dbType mysql -initSchema
> >> > SLF4J: Class path contains multiple SLF4J bindings.
> >> > SLF4J: Found binding in
> [jar:file:/opt/hive/lib/log4j-slf4j-impl-2.17.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> >> > SLF4J: Found binding in
> [jar:file:/opt/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> >> > SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
> >> > SLF4J: Actual binding is of type
> [org.apache.logging.slf4j.Log4jLoggerFactory]
> >> > Metastore connection URL:
> jdbc:mysql://host.docker.internal:3306/hive?createDatabaseIfNotExist=true
> >> > Metastore Connection Driver : com.mysql.cj.jdbc.Driver
> >> > Metastore connection User: hive
> >> > org.apache.hadoop.hive.metastore.HiveMetaException: Failed to load
> driver
> >> > Underlying cause: java.lang.ClassNotFoundException :
> com.mysql.cj.jdbc.Driver
> >> > Use --verbose for detailed stacktrace.
> >> > *** schemaTool failed ***
> >> > + '[' 1 -eq 0 ']'
> >> > + echo 'Schema initialization failed!'
> >> > Schema initialization failed!
> >> > + exit 1
> >> >
> >> > Any idea, why I am getting failed to load driver for MySQL DB.
> >> >
> >> > Isn't docker container comes with MySQL Driver ?
> >> >
> >> > Docker container exits so I can't check whether driver is already
> installed.
> >> >
> >> > Let me know, what I can do to make it work.
> >> >
> >> > --
> >> >
> >> >
> >> > Thanks
> >> > Sanjay Gupta
> >> >
> >> >
> >> >
> >> > --
> >> >
> >> > Thanks
> >> > Sanjay Gupta
> >> >
> >> >
> >> >
> >> > --
> >> >
> >> > Thanks
> >> > Sanjay Gupta
> >> >
>
>
>
> --
>
> Thanks
> Sanjay Gupta
>
Re: Help with Docker Apache/Hive metastore using mysql remote database
Posted by Sanjay Gupta <sa...@gmail.com>.
There is no mysql driver file in following repo
https://repo1.maven.org/maven2/org
On Mon, Dec 18, 2023 at 3:10 AM Simhadri G <si...@gmail.com> wrote:
>
> We can modify the Dockerfile to wget the necessary driver and copy it to /opt/hive/lib/ . This should make it work. The diff is attached below:
>
>
> diff --git a/packaging/src/docker/Dockerfile b/packaging/src/docker/Dockerfile
> --- a/packaging/src/docker/Dockerfile (revision dceaf810b32fc266e3e657fdaefcd4507f2191b5)
> +++ b/packaging/src/docker/Dockerfile (date 1702897518609)
> @@ -80,6 +80,9 @@
>
> ENV PATH=$HIVE_HOME/bin:$HADOOP_HOME/bin:$PATH
>
> +RUN wget https://repo1.maven.org/maven2/org/postgresql/postgresql/42.5.1/postgresql-42.5.1.jar
> +RUN cp /postgresql-42.5.1.jar /opt/hive/lib/
> +
> COPY entrypoint.sh /
> COPY conf $HIVE_HOME/conf
> RUN chmod +x /entrypoint.sh
>
> On Mon, Dec 18, 2023, 12:59 PM Ayush Saxena <ay...@gmail.com> wrote:
>>
>> I think the similar problem is being chased as part of
>> https://github.com/apache/hive/pull/4948
>>
>> On Mon, 18 Dec 2023 at 09:48, Sanjay Gupta <sa...@gmail.com> wrote:
>> >
>> >
>> >
>> >
>> > Issue with Docker container using mysql RDBMS ( Failed to load driver)
>> >
>> > https://hub.docker.com/r/apache/hive
>> >
>> > According to readme
>> >
>> > Launch Standalone Metastore With External RDBMS (Postgres/Oracle/MySql/MsSql)
>> >
>> > I want to use MySQL
>> >
>> > I tried com.mysql.jdbc.Driver or com.mysql.cj.jdbc.Driver
>> >
>> > docker run -it -d -p 9083:9083 --env SERVICE_NAME=metastore --add-host=host.docker.internal:host-gateway \
>> > --env DB_DRIVER=mysql \
>> > --env SERVICE_OPTS="-Djavax.jdo.option.ConnectionDriverName=com.mysql.jdbc.Driver -Djavax.jdo.option.ConnectionURL=jdbc:mysql://host.docker.internal:3306/hive?createDatabaseIfNotExist=true -Djavax.jdo.option.ConnectionUserName=hive -Djavax.jdo.option.ConnectionPassword=password" \
>> > --mount source=warehouse,target=/opt/hive/data/warehouse \
>> > --name metastore-standalone apache/hive:${HIVE_VERSION}
>> >
>> >
>> > docker run -it -d -p 9083:9083 --env SERVICE_NAME=metastore --add-host=host.docker.internal:host-gateway \
>> > --env DB_DRIVER=mysql \
>> > --env SERVICE_OPTS="-Djavax.jdo.option.ConnectionDriverName=com.mysql.cj.jdbc.Driver -Djavax.jdo.option.ConnectionURL=jdbc:mysql://host.docker.internal:3306/hive?createDatabaseIfNotExist=true -Djavax.jdo.option.ConnectionUserName=hive -Djavax.jdo.option.ConnectionPassword=password" \
>> > --mount source=warehouse,target=/opt/hive/data/warehouse \
>> > --name metastore-standalone apache/hive:${HIVE_VERSION}
>> >
>> > Docker logs shows this for both drivers ( same error )
>> >
>> > docker logs f3
>> > + : mysql
>> > + SKIP_SCHEMA_INIT=false
>> > + export HIVE_CONF_DIR=/opt/hive/conf
>> > + HIVE_CONF_DIR=/opt/hive/conf
>> > + '[' -d '' ']'
>> > + export 'HADOOP_CLIENT_OPTS= -Xmx1G -Djavax.jdo.option.ConnectionDriverName=com.mysql.cj.jdbc.Driver -Djavax.jdo.option.ConnectionURL=jdbc:mysql://host.docker.internal:3306/hive?createDatabaseIfNotExist=true -Djavax.jdo.option.ConnectionUserName=hive -Djavax.jdo.option.ConnectionPassword=hive'
>> > + HADOOP_CLIENT_OPTS=' -Xmx1G -Djavax.jdo.option.ConnectionDriverName=com.mysql.cj.jdbc.Driver -Djavax.jdo.option.ConnectionURL=jdbc:mysql://host.docker.internal:3306/hive?createDatabaseIfNotExist=true -Djavax.jdo.option.ConnectionUserName=hive -Djavax.jdo.option.ConnectionPassword=hive'
>> > + [[ false == \f\a\l\s\e ]]
>> > + initialize_hive
>> > + /opt/hive/bin/schematool -dbType mysql -initSchema
>> > SLF4J: Class path contains multiple SLF4J bindings.
>> > SLF4J: Found binding in [jar:file:/opt/hive/lib/log4j-slf4j-impl-2.17.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>> > SLF4J: Found binding in [jar:file:/opt/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>> > SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
>> > SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
>> > Metastore connection URL: jdbc:mysql://host.docker.internal:3306/hive?createDatabaseIfNotExist=true
>> > Metastore Connection Driver : com.mysql.cj.jdbc.Driver
>> > Metastore connection User: hive
>> > org.apache.hadoop.hive.metastore.HiveMetaException: Failed to load driver
>> > Underlying cause: java.lang.ClassNotFoundException : com.mysql.cj.jdbc.Driver
>> > Use --verbose for detailed stacktrace.
>> > *** schemaTool failed ***
>> > + '[' 1 -eq 0 ']'
>> > + echo 'Schema initialization failed!'
>> > Schema initialization failed!
>> > + exit 1
>> >
>> > Any idea, why I am getting failed to load driver for MySQL DB.
>> >
>> > Isn't docker container comes with MySQL Driver ?
>> >
>> > Docker container exits so I can't check whether driver is already installed.
>> >
>> > Let me know, what I can do to make it work.
>> >
>> > --
>> >
>> >
>> > Thanks
>> > Sanjay Gupta
>> >
>> >
>> >
>> > --
>> >
>> > Thanks
>> > Sanjay Gupta
>> >
>> >
>> >
>> > --
>> >
>> > Thanks
>> > Sanjay Gupta
>> >
--
Thanks
Sanjay Gupta
Re: Help with Docker Apache/Hive metastore using mysql remote database
Posted by Sanjay Gupta <sa...@gmail.com>.
There is no mysql driver file in following repo
https://repo1.maven.org/maven2/org
On Mon, Dec 18, 2023 at 3:10 AM Simhadri G <si...@gmail.com> wrote:
>
> We can modify the Dockerfile to wget the necessary driver and copy it to /opt/hive/lib/ . This should make it work. The diff is attached below:
>
>
> diff --git a/packaging/src/docker/Dockerfile b/packaging/src/docker/Dockerfile
> --- a/packaging/src/docker/Dockerfile (revision dceaf810b32fc266e3e657fdaefcd4507f2191b5)
> +++ b/packaging/src/docker/Dockerfile (date 1702897518609)
> @@ -80,6 +80,9 @@
>
> ENV PATH=$HIVE_HOME/bin:$HADOOP_HOME/bin:$PATH
>
> +RUN wget https://repo1.maven.org/maven2/org/postgresql/postgresql/42.5.1/postgresql-42.5.1.jar
> +RUN cp /postgresql-42.5.1.jar /opt/hive/lib/
> +
> COPY entrypoint.sh /
> COPY conf $HIVE_HOME/conf
> RUN chmod +x /entrypoint.sh
>
> On Mon, Dec 18, 2023, 12:59 PM Ayush Saxena <ay...@gmail.com> wrote:
>>
>> I think the similar problem is being chased as part of
>> https://github.com/apache/hive/pull/4948
>>
>> On Mon, 18 Dec 2023 at 09:48, Sanjay Gupta <sa...@gmail.com> wrote:
>> >
>> >
>> >
>> >
>> > Issue with Docker container using mysql RDBMS ( Failed to load driver)
>> >
>> > https://hub.docker.com/r/apache/hive
>> >
>> > According to readme
>> >
>> > Launch Standalone Metastore With External RDBMS (Postgres/Oracle/MySql/MsSql)
>> >
>> > I want to use MySQL
>> >
>> > I tried com.mysql.jdbc.Driver or com.mysql.cj.jdbc.Driver
>> >
>> > docker run -it -d -p 9083:9083 --env SERVICE_NAME=metastore --add-host=host.docker.internal:host-gateway \
>> > --env DB_DRIVER=mysql \
>> > --env SERVICE_OPTS="-Djavax.jdo.option.ConnectionDriverName=com.mysql.jdbc.Driver -Djavax.jdo.option.ConnectionURL=jdbc:mysql://host.docker.internal:3306/hive?createDatabaseIfNotExist=true -Djavax.jdo.option.ConnectionUserName=hive -Djavax.jdo.option.ConnectionPassword=password" \
>> > --mount source=warehouse,target=/opt/hive/data/warehouse \
>> > --name metastore-standalone apache/hive:${HIVE_VERSION}
>> >
>> >
>> > docker run -it -d -p 9083:9083 --env SERVICE_NAME=metastore --add-host=host.docker.internal:host-gateway \
>> > --env DB_DRIVER=mysql \
>> > --env SERVICE_OPTS="-Djavax.jdo.option.ConnectionDriverName=com.mysql.cj.jdbc.Driver -Djavax.jdo.option.ConnectionURL=jdbc:mysql://host.docker.internal:3306/hive?createDatabaseIfNotExist=true -Djavax.jdo.option.ConnectionUserName=hive -Djavax.jdo.option.ConnectionPassword=password" \
>> > --mount source=warehouse,target=/opt/hive/data/warehouse \
>> > --name metastore-standalone apache/hive:${HIVE_VERSION}
>> >
>> > Docker logs shows this for both drivers ( same error )
>> >
>> > docker logs f3
>> > + : mysql
>> > + SKIP_SCHEMA_INIT=false
>> > + export HIVE_CONF_DIR=/opt/hive/conf
>> > + HIVE_CONF_DIR=/opt/hive/conf
>> > + '[' -d '' ']'
>> > + export 'HADOOP_CLIENT_OPTS= -Xmx1G -Djavax.jdo.option.ConnectionDriverName=com.mysql.cj.jdbc.Driver -Djavax.jdo.option.ConnectionURL=jdbc:mysql://host.docker.internal:3306/hive?createDatabaseIfNotExist=true -Djavax.jdo.option.ConnectionUserName=hive -Djavax.jdo.option.ConnectionPassword=hive'
>> > + HADOOP_CLIENT_OPTS=' -Xmx1G -Djavax.jdo.option.ConnectionDriverName=com.mysql.cj.jdbc.Driver -Djavax.jdo.option.ConnectionURL=jdbc:mysql://host.docker.internal:3306/hive?createDatabaseIfNotExist=true -Djavax.jdo.option.ConnectionUserName=hive -Djavax.jdo.option.ConnectionPassword=hive'
>> > + [[ false == \f\a\l\s\e ]]
>> > + initialize_hive
>> > + /opt/hive/bin/schematool -dbType mysql -initSchema
>> > SLF4J: Class path contains multiple SLF4J bindings.
>> > SLF4J: Found binding in [jar:file:/opt/hive/lib/log4j-slf4j-impl-2.17.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>> > SLF4J: Found binding in [jar:file:/opt/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>> > SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
>> > SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
>> > Metastore connection URL: jdbc:mysql://host.docker.internal:3306/hive?createDatabaseIfNotExist=true
>> > Metastore Connection Driver : com.mysql.cj.jdbc.Driver
>> > Metastore connection User: hive
>> > org.apache.hadoop.hive.metastore.HiveMetaException: Failed to load driver
>> > Underlying cause: java.lang.ClassNotFoundException : com.mysql.cj.jdbc.Driver
>> > Use --verbose for detailed stacktrace.
>> > *** schemaTool failed ***
>> > + '[' 1 -eq 0 ']'
>> > + echo 'Schema initialization failed!'
>> > Schema initialization failed!
>> > + exit 1
>> >
>> > Any idea, why I am getting failed to load driver for MySQL DB.
>> >
>> > Isn't docker container comes with MySQL Driver ?
>> >
>> > Docker container exits so I can't check whether driver is already installed.
>> >
>> > Let me know, what I can do to make it work.
>> >
>> > --
>> >
>> >
>> > Thanks
>> > Sanjay Gupta
>> >
>> >
>> >
>> > --
>> >
>> > Thanks
>> > Sanjay Gupta
>> >
>> >
>> >
>> > --
>> >
>> > Thanks
>> > Sanjay Gupta
>> >
--
Thanks
Sanjay Gupta
Re: Help with Docker Apache/Hive metastore using mysql remote database
Posted by Simhadri G <si...@gmail.com>.
We can modify the Dockerfile to wget the necessary driver and copy it to
/opt/hive/lib/ . This should make it work. The diff is attached below:
diff --git a/packaging/src/docker/Dockerfile
b/packaging/src/docker/Dockerfile
--- a/packaging/src/docker/Dockerfile (revision
dceaf810b32fc266e3e657fdaefcd4507f2191b5)
+++ b/packaging/src/docker/Dockerfile (date 1702897518609)
@@ -80,6 +80,9 @@
ENV PATH=$HIVE_HOME/bin:$HADOOP_HOME/bin:$PATH
+RUN wget
https://repo1.maven.org/maven2/org/postgresql/postgresql/42.5.1/postgresql-42.5.1.jar
+RUN cp /postgresql-42.5.1.jar /opt/hive/lib/
+
COPY entrypoint.sh /
COPY conf $HIVE_HOME/conf
RUN chmod +x /entrypoint.sh
On Mon, Dec 18, 2023, 12:59 PM Ayush Saxena <ay...@gmail.com> wrote:
> I think the similar problem is being chased as part of
> https://github.com/apache/hive/pull/4948
>
> On Mon, 18 Dec 2023 at 09:48, Sanjay Gupta <sa...@gmail.com> wrote:
> >
> >
> >
> >
> > Issue with Docker container using mysql RDBMS ( Failed to load driver)
> >
> > https://hub.docker.com/r/apache/hive
> >
> > According to readme
> >
> > Launch Standalone Metastore With External RDBMS
> (Postgres/Oracle/MySql/MsSql)
> >
> > I want to use MySQL
> >
> > I tried com.mysql.jdbc.Driver or com.mysql.cj.jdbc.Driver
> >
> > docker run -it -d -p 9083:9083 --env SERVICE_NAME=metastore
> --add-host=host.docker.internal:host-gateway \
> > --env DB_DRIVER=mysql \
> > --env
> SERVICE_OPTS="-Djavax.jdo.option.ConnectionDriverName=com.mysql.jdbc.Driver
> -Djavax.jdo.option.ConnectionURL=jdbc:mysql://host.docker.internal:3306/hive?createDatabaseIfNotExist=true
> -Djavax.jdo.option.ConnectionUserName=hive
> -Djavax.jdo.option.ConnectionPassword=password" \
> > --mount source=warehouse,target=/opt/hive/data/warehouse \
> > --name metastore-standalone apache/hive:${HIVE_VERSION}
> >
> >
> > docker run -it -d -p 9083:9083 --env SERVICE_NAME=metastore
> --add-host=host.docker.internal:host-gateway \
> > --env DB_DRIVER=mysql \
> > --env
> SERVICE_OPTS="-Djavax.jdo.option.ConnectionDriverName=com.mysql.cj.jdbc.Driver
> -Djavax.jdo.option.ConnectionURL=jdbc:mysql://host.docker.internal:3306/hive?createDatabaseIfNotExist=true
> -Djavax.jdo.option.ConnectionUserName=hive
> -Djavax.jdo.option.ConnectionPassword=password" \
> > --mount source=warehouse,target=/opt/hive/data/warehouse \
> > --name metastore-standalone apache/hive:${HIVE_VERSION}
> >
> > Docker logs shows this for both drivers ( same error )
> >
> > docker logs f3
> > + : mysql
> > + SKIP_SCHEMA_INIT=false
> > + export HIVE_CONF_DIR=/opt/hive/conf
> > + HIVE_CONF_DIR=/opt/hive/conf
> > + '[' -d '' ']'
> > + export 'HADOOP_CLIENT_OPTS= -Xmx1G
> -Djavax.jdo.option.ConnectionDriverName=com.mysql.cj.jdbc.Driver
> -Djavax.jdo.option.ConnectionURL=jdbc:mysql://host.docker.internal:3306/hive?createDatabaseIfNotExist=true
> -Djavax.jdo.option.ConnectionUserName=hive
> -Djavax.jdo.option.ConnectionPassword=hive'
> > + HADOOP_CLIENT_OPTS=' -Xmx1G
> -Djavax.jdo.option.ConnectionDriverName=com.mysql.cj.jdbc.Driver
> -Djavax.jdo.option.ConnectionURL=jdbc:mysql://host.docker.internal:3306/hive?createDatabaseIfNotExist=true
> -Djavax.jdo.option.ConnectionUserName=hive
> -Djavax.jdo.option.ConnectionPassword=hive'
> > + [[ false == \f\a\l\s\e ]]
> > + initialize_hive
> > + /opt/hive/bin/schematool -dbType mysql -initSchema
> > SLF4J: Class path contains multiple SLF4J bindings.
> > SLF4J: Found binding in
> [jar:file:/opt/hive/lib/log4j-slf4j-impl-2.17.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> > SLF4J: Found binding in
> [jar:file:/opt/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> > SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
> > SLF4J: Actual binding is of type
> [org.apache.logging.slf4j.Log4jLoggerFactory]
> > Metastore connection URL:
> jdbc:mysql://host.docker.internal:3306/hive?createDatabaseIfNotExist=true
> > Metastore Connection Driver : com.mysql.cj.jdbc.Driver
> > Metastore connection User: hive
> > org.apache.hadoop.hive.metastore.HiveMetaException: Failed to load driver
> > Underlying cause: java.lang.ClassNotFoundException :
> com.mysql.cj.jdbc.Driver
> > Use --verbose for detailed stacktrace.
> > *** schemaTool failed ***
> > + '[' 1 -eq 0 ']'
> > + echo 'Schema initialization failed!'
> > Schema initialization failed!
> > + exit 1
> >
> > Any idea, why I am getting failed to load driver for MySQL DB.
> >
> > Isn't docker container comes with MySQL Driver ?
> >
> > Docker container exits so I can't check whether driver is already
> installed.
> >
> > Let me know, what I can do to make it work.
> >
> > --
> >
> >
> > Thanks
> > Sanjay Gupta
> >
> >
> >
> > --
> >
> > Thanks
> > Sanjay Gupta
> >
> >
> >
> > --
> >
> > Thanks
> > Sanjay Gupta
> >
>
Re: Help with Docker Apache/Hive metastore using mysql remote database
Posted by Simhadri G <si...@gmail.com>.
We can modify the Dockerfile to wget the necessary driver and copy it to
/opt/hive/lib/ . This should make it work. The diff is attached below:
diff --git a/packaging/src/docker/Dockerfile
b/packaging/src/docker/Dockerfile
--- a/packaging/src/docker/Dockerfile (revision
dceaf810b32fc266e3e657fdaefcd4507f2191b5)
+++ b/packaging/src/docker/Dockerfile (date 1702897518609)
@@ -80,6 +80,9 @@
ENV PATH=$HIVE_HOME/bin:$HADOOP_HOME/bin:$PATH
+RUN wget
https://repo1.maven.org/maven2/org/postgresql/postgresql/42.5.1/postgresql-42.5.1.jar
+RUN cp /postgresql-42.5.1.jar /opt/hive/lib/
+
COPY entrypoint.sh /
COPY conf $HIVE_HOME/conf
RUN chmod +x /entrypoint.sh
On Mon, Dec 18, 2023, 12:59 PM Ayush Saxena <ay...@gmail.com> wrote:
> I think the similar problem is being chased as part of
> https://github.com/apache/hive/pull/4948
>
> On Mon, 18 Dec 2023 at 09:48, Sanjay Gupta <sa...@gmail.com> wrote:
> >
> >
> >
> >
> > Issue with Docker container using mysql RDBMS ( Failed to load driver)
> >
> > https://hub.docker.com/r/apache/hive
> >
> > According to readme
> >
> > Launch Standalone Metastore With External RDBMS
> (Postgres/Oracle/MySql/MsSql)
> >
> > I want to use MySQL
> >
> > I tried com.mysql.jdbc.Driver or com.mysql.cj.jdbc.Driver
> >
> > docker run -it -d -p 9083:9083 --env SERVICE_NAME=metastore
> --add-host=host.docker.internal:host-gateway \
> > --env DB_DRIVER=mysql \
> > --env
> SERVICE_OPTS="-Djavax.jdo.option.ConnectionDriverName=com.mysql.jdbc.Driver
> -Djavax.jdo.option.ConnectionURL=jdbc:mysql://host.docker.internal:3306/hive?createDatabaseIfNotExist=true
> -Djavax.jdo.option.ConnectionUserName=hive
> -Djavax.jdo.option.ConnectionPassword=password" \
> > --mount source=warehouse,target=/opt/hive/data/warehouse \
> > --name metastore-standalone apache/hive:${HIVE_VERSION}
> >
> >
> > docker run -it -d -p 9083:9083 --env SERVICE_NAME=metastore
> --add-host=host.docker.internal:host-gateway \
> > --env DB_DRIVER=mysql \
> > --env
> SERVICE_OPTS="-Djavax.jdo.option.ConnectionDriverName=com.mysql.cj.jdbc.Driver
> -Djavax.jdo.option.ConnectionURL=jdbc:mysql://host.docker.internal:3306/hive?createDatabaseIfNotExist=true
> -Djavax.jdo.option.ConnectionUserName=hive
> -Djavax.jdo.option.ConnectionPassword=password" \
> > --mount source=warehouse,target=/opt/hive/data/warehouse \
> > --name metastore-standalone apache/hive:${HIVE_VERSION}
> >
> > Docker logs shows this for both drivers ( same error )
> >
> > docker logs f3
> > + : mysql
> > + SKIP_SCHEMA_INIT=false
> > + export HIVE_CONF_DIR=/opt/hive/conf
> > + HIVE_CONF_DIR=/opt/hive/conf
> > + '[' -d '' ']'
> > + export 'HADOOP_CLIENT_OPTS= -Xmx1G
> -Djavax.jdo.option.ConnectionDriverName=com.mysql.cj.jdbc.Driver
> -Djavax.jdo.option.ConnectionURL=jdbc:mysql://host.docker.internal:3306/hive?createDatabaseIfNotExist=true
> -Djavax.jdo.option.ConnectionUserName=hive
> -Djavax.jdo.option.ConnectionPassword=hive'
> > + HADOOP_CLIENT_OPTS=' -Xmx1G
> -Djavax.jdo.option.ConnectionDriverName=com.mysql.cj.jdbc.Driver
> -Djavax.jdo.option.ConnectionURL=jdbc:mysql://host.docker.internal:3306/hive?createDatabaseIfNotExist=true
> -Djavax.jdo.option.ConnectionUserName=hive
> -Djavax.jdo.option.ConnectionPassword=hive'
> > + [[ false == \f\a\l\s\e ]]
> > + initialize_hive
> > + /opt/hive/bin/schematool -dbType mysql -initSchema
> > SLF4J: Class path contains multiple SLF4J bindings.
> > SLF4J: Found binding in
> [jar:file:/opt/hive/lib/log4j-slf4j-impl-2.17.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> > SLF4J: Found binding in
> [jar:file:/opt/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> > SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
> > SLF4J: Actual binding is of type
> [org.apache.logging.slf4j.Log4jLoggerFactory]
> > Metastore connection URL:
> jdbc:mysql://host.docker.internal:3306/hive?createDatabaseIfNotExist=true
> > Metastore Connection Driver : com.mysql.cj.jdbc.Driver
> > Metastore connection User: hive
> > org.apache.hadoop.hive.metastore.HiveMetaException: Failed to load driver
> > Underlying cause: java.lang.ClassNotFoundException :
> com.mysql.cj.jdbc.Driver
> > Use --verbose for detailed stacktrace.
> > *** schemaTool failed ***
> > + '[' 1 -eq 0 ']'
> > + echo 'Schema initialization failed!'
> > Schema initialization failed!
> > + exit 1
> >
> > Any idea, why I am getting failed to load driver for MySQL DB.
> >
> > Isn't docker container comes with MySQL Driver ?
> >
> > Docker container exits so I can't check whether driver is already
> installed.
> >
> > Let me know, what I can do to make it work.
> >
> > --
> >
> >
> > Thanks
> > Sanjay Gupta
> >
> >
> >
> > --
> >
> > Thanks
> > Sanjay Gupta
> >
> >
> >
> > --
> >
> > Thanks
> > Sanjay Gupta
> >
>
Re: Help with Docker Apache/Hive metastore using mysql remote database
Posted by Ayush Saxena <ay...@gmail.com>.
I think the similar problem is being chased as part of
https://github.com/apache/hive/pull/4948
On Mon, 18 Dec 2023 at 09:48, Sanjay Gupta <sa...@gmail.com> wrote:
>
>
>
>
> Issue with Docker container using mysql RDBMS ( Failed to load driver)
>
> https://hub.docker.com/r/apache/hive
>
> According to readme
>
> Launch Standalone Metastore With External RDBMS (Postgres/Oracle/MySql/MsSql)
>
> I want to use MySQL
>
> I tried com.mysql.jdbc.Driver or com.mysql.cj.jdbc.Driver
>
> docker run -it -d -p 9083:9083 --env SERVICE_NAME=metastore --add-host=host.docker.internal:host-gateway \
> --env DB_DRIVER=mysql \
> --env SERVICE_OPTS="-Djavax.jdo.option.ConnectionDriverName=com.mysql.jdbc.Driver -Djavax.jdo.option.ConnectionURL=jdbc:mysql://host.docker.internal:3306/hive?createDatabaseIfNotExist=true -Djavax.jdo.option.ConnectionUserName=hive -Djavax.jdo.option.ConnectionPassword=password" \
> --mount source=warehouse,target=/opt/hive/data/warehouse \
> --name metastore-standalone apache/hive:${HIVE_VERSION}
>
>
> docker run -it -d -p 9083:9083 --env SERVICE_NAME=metastore --add-host=host.docker.internal:host-gateway \
> --env DB_DRIVER=mysql \
> --env SERVICE_OPTS="-Djavax.jdo.option.ConnectionDriverName=com.mysql.cj.jdbc.Driver -Djavax.jdo.option.ConnectionURL=jdbc:mysql://host.docker.internal:3306/hive?createDatabaseIfNotExist=true -Djavax.jdo.option.ConnectionUserName=hive -Djavax.jdo.option.ConnectionPassword=password" \
> --mount source=warehouse,target=/opt/hive/data/warehouse \
> --name metastore-standalone apache/hive:${HIVE_VERSION}
>
> Docker logs shows this for both drivers ( same error )
>
> docker logs f3
> + : mysql
> + SKIP_SCHEMA_INIT=false
> + export HIVE_CONF_DIR=/opt/hive/conf
> + HIVE_CONF_DIR=/opt/hive/conf
> + '[' -d '' ']'
> + export 'HADOOP_CLIENT_OPTS= -Xmx1G -Djavax.jdo.option.ConnectionDriverName=com.mysql.cj.jdbc.Driver -Djavax.jdo.option.ConnectionURL=jdbc:mysql://host.docker.internal:3306/hive?createDatabaseIfNotExist=true -Djavax.jdo.option.ConnectionUserName=hive -Djavax.jdo.option.ConnectionPassword=hive'
> + HADOOP_CLIENT_OPTS=' -Xmx1G -Djavax.jdo.option.ConnectionDriverName=com.mysql.cj.jdbc.Driver -Djavax.jdo.option.ConnectionURL=jdbc:mysql://host.docker.internal:3306/hive?createDatabaseIfNotExist=true -Djavax.jdo.option.ConnectionUserName=hive -Djavax.jdo.option.ConnectionPassword=hive'
> + [[ false == \f\a\l\s\e ]]
> + initialize_hive
> + /opt/hive/bin/schematool -dbType mysql -initSchema
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in [jar:file:/opt/hive/lib/log4j-slf4j-impl-2.17.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in [jar:file:/opt/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
> SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
> Metastore connection URL: jdbc:mysql://host.docker.internal:3306/hive?createDatabaseIfNotExist=true
> Metastore Connection Driver : com.mysql.cj.jdbc.Driver
> Metastore connection User: hive
> org.apache.hadoop.hive.metastore.HiveMetaException: Failed to load driver
> Underlying cause: java.lang.ClassNotFoundException : com.mysql.cj.jdbc.Driver
> Use --verbose for detailed stacktrace.
> *** schemaTool failed ***
> + '[' 1 -eq 0 ']'
> + echo 'Schema initialization failed!'
> Schema initialization failed!
> + exit 1
>
> Any idea, why I am getting failed to load driver for MySQL DB.
>
> Isn't docker container comes with MySQL Driver ?
>
> Docker container exits so I can't check whether driver is already installed.
>
> Let me know, what I can do to make it work.
>
> --
>
>
> Thanks
> Sanjay Gupta
>
>
>
> --
>
> Thanks
> Sanjay Gupta
>
>
>
> --
>
> Thanks
> Sanjay Gupta
>
Re: Help with Docker Apache/Hive metastore using mysql remote database
Posted by Ayush Saxena <ay...@gmail.com>.
I think the similar problem is being chased as part of
https://github.com/apache/hive/pull/4948
On Mon, 18 Dec 2023 at 09:48, Sanjay Gupta <sa...@gmail.com> wrote:
>
>
>
>
> Issue with Docker container using mysql RDBMS ( Failed to load driver)
>
> https://hub.docker.com/r/apache/hive
>
> According to readme
>
> Launch Standalone Metastore With External RDBMS (Postgres/Oracle/MySql/MsSql)
>
> I want to use MySQL
>
> I tried com.mysql.jdbc.Driver or com.mysql.cj.jdbc.Driver
>
> docker run -it -d -p 9083:9083 --env SERVICE_NAME=metastore --add-host=host.docker.internal:host-gateway \
> --env DB_DRIVER=mysql \
> --env SERVICE_OPTS="-Djavax.jdo.option.ConnectionDriverName=com.mysql.jdbc.Driver -Djavax.jdo.option.ConnectionURL=jdbc:mysql://host.docker.internal:3306/hive?createDatabaseIfNotExist=true -Djavax.jdo.option.ConnectionUserName=hive -Djavax.jdo.option.ConnectionPassword=password" \
> --mount source=warehouse,target=/opt/hive/data/warehouse \
> --name metastore-standalone apache/hive:${HIVE_VERSION}
>
>
> docker run -it -d -p 9083:9083 --env SERVICE_NAME=metastore --add-host=host.docker.internal:host-gateway \
> --env DB_DRIVER=mysql \
> --env SERVICE_OPTS="-Djavax.jdo.option.ConnectionDriverName=com.mysql.cj.jdbc.Driver -Djavax.jdo.option.ConnectionURL=jdbc:mysql://host.docker.internal:3306/hive?createDatabaseIfNotExist=true -Djavax.jdo.option.ConnectionUserName=hive -Djavax.jdo.option.ConnectionPassword=password" \
> --mount source=warehouse,target=/opt/hive/data/warehouse \
> --name metastore-standalone apache/hive:${HIVE_VERSION}
>
> Docker logs shows this for both drivers ( same error )
>
> docker logs f3
> + : mysql
> + SKIP_SCHEMA_INIT=false
> + export HIVE_CONF_DIR=/opt/hive/conf
> + HIVE_CONF_DIR=/opt/hive/conf
> + '[' -d '' ']'
> + export 'HADOOP_CLIENT_OPTS= -Xmx1G -Djavax.jdo.option.ConnectionDriverName=com.mysql.cj.jdbc.Driver -Djavax.jdo.option.ConnectionURL=jdbc:mysql://host.docker.internal:3306/hive?createDatabaseIfNotExist=true -Djavax.jdo.option.ConnectionUserName=hive -Djavax.jdo.option.ConnectionPassword=hive'
> + HADOOP_CLIENT_OPTS=' -Xmx1G -Djavax.jdo.option.ConnectionDriverName=com.mysql.cj.jdbc.Driver -Djavax.jdo.option.ConnectionURL=jdbc:mysql://host.docker.internal:3306/hive?createDatabaseIfNotExist=true -Djavax.jdo.option.ConnectionUserName=hive -Djavax.jdo.option.ConnectionPassword=hive'
> + [[ false == \f\a\l\s\e ]]
> + initialize_hive
> + /opt/hive/bin/schematool -dbType mysql -initSchema
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in [jar:file:/opt/hive/lib/log4j-slf4j-impl-2.17.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in [jar:file:/opt/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
> SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
> Metastore connection URL: jdbc:mysql://host.docker.internal:3306/hive?createDatabaseIfNotExist=true
> Metastore Connection Driver : com.mysql.cj.jdbc.Driver
> Metastore connection User: hive
> org.apache.hadoop.hive.metastore.HiveMetaException: Failed to load driver
> Underlying cause: java.lang.ClassNotFoundException : com.mysql.cj.jdbc.Driver
> Use --verbose for detailed stacktrace.
> *** schemaTool failed ***
> + '[' 1 -eq 0 ']'
> + echo 'Schema initialization failed!'
> Schema initialization failed!
> + exit 1
>
> Any idea, why I am getting failed to load driver for MySQL DB.
>
> Isn't docker container comes with MySQL Driver ?
>
> Docker container exits so I can't check whether driver is already installed.
>
> Let me know, what I can do to make it work.
>
> --
>
>
> Thanks
> Sanjay Gupta
>
>
>
> --
>
> Thanks
> Sanjay Gupta
>
>
>
> --
>
> Thanks
> Sanjay Gupta
>