You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2021/11/10 23:32:07 UTC

[GitHub] [hudi] rubenssoto opened a new issue #3966: [SUPPORT] Hudi Sync Remote Hive - EKS

rubenssoto opened a new issue #3966:
URL: https://github.com/apache/hudi/issues/3966


   Hello Guys,
   
   We are starting to deploy Spark on Kubernetes and we need to Hudi sync with our Hive metastore, but we are facing a lot of errors like you could see on the following logs:
   [log_2.txt](https://github.com/apache/hudi/files/7516536/log_2.txt)
   
   
   Our hive metastore is in another Pod, but in the same cluster. 
   
   We create hive-site.xml, but with spark in kubernetes is not possible to use the $SPARK_HOME/conf, so we use the following path
   
   ENV SPARK_CLASSPATH=$SPARK_HOME/cluster-conf
   
   The Docker File of our Spark Image:
   
   ```
   # Global args
   ARG SPARK_VERSION=3.0.3
   ARG HADOOP_VERSION=3.2.0-cloud
   ARG SCALA_VERSION=2.12
   
   # Build Spark
   FROM openjdk:8-jdk-slim as build
   
   ENV DEBIAN_FRONTEND=noninteractive
   
   ENV M2_HOME=/opt/maven
   ENV MAVEN_HOME=/opt/maven
   ENV PATH=${PATH}:${M2_HOME}/bin
   
   ARG SPARK_VERSION
   ARG HADOOP_VERSION
   ARG SCALA_VERSION
   
   RUN apt-get -qq update && \
       apt-get -qq upgrade -y && \
       apt-get -qq install -y git wget curl && \
       wget -q https://dlcdn.apache.org/maven/maven-3/3.8.1/binaries/apache-maven-3.8.1-bin.tar.gz -P /tmp && \
       tar xf /tmp/apache-maven-3.8.1-bin.tar.gz -C /opt && \
       ln -s /opt/apache-maven-3.8.1 /opt/maven && \
       apt-get -qq install gnupg && \
       echo "deb https://repo.scala-sbt.org/scalasbt/debian all main" | tee -a /etc/apt/sources.list.d/sbt.list && \
       echo "deb https://repo.scala-sbt.org/scalasbt/debian /" | tee -a /etc/apt/sources.list.d/sbt_old.list && \
       curl -sL "https://keyserver.ubuntu.com/pks/lookup?op=get&search=0x2EE0EA64E40A89B84B2DF73499E82A75642AC823" | apt-key add && \
       apt-get -qq update && \
       apt-get -qq install sbt && \
       apt-get -qq install -y r-base && \
       apt install -y python && \
       curl https://bootstrap.pypa.io/pip/2.7/get-pip.py -o get-pip.py && \
       python get-pip.py && \
       apt install -y python3 python3-pip && \
       rm -r /usr/lib/python*/ensurepip && \
       pip install --upgrade pip setuptools
   
   RUN cd / && \
       git clone https://github.com/apache/spark.git --branch v${SPARK_VERSION} --single-branch && \
       cd /spark && \
       dev/make-distribution.sh \
           --name hadoop-${HADOOP_VERSION}-scala-${SCALA_VERSION} --pip --tgz -DskipTests \
           -Phadoop-3.2 \
           -Phadoop-cloud \
           -Pkubernetes \
           -Phive && \
       cp spark-${SPARK_VERSION}-bin-hadoop-${HADOOP_VERSION}-scala-${SCALA_VERSION}.tgz /
   
   # Spark image
   FROM openjdk:8-jre-slim
   
   ENV BASE_IMAGE  openjdk:8-jre-slim
   
   RUN set -ex && \
       sed -i 's/http:/https:/g' /etc/apt/sources.list && \
       apt-get update && \
       ln -s /lib /lib64 && \
       apt install -y bash tini libc6 libpam-modules krb5-user libnss3 wget bzip2 && \
       rm /bin/sh && \
       ln -sv /bin/bash /bin/sh && \
       echo "auth required pam_wheel.so use_uid" >> /etc/pam.d/su && \
       chgrp root /etc/passwd && chmod ug+rw /etc/passwd && \
       rm -rf /var/cache/apt/*
   
   RUN apt-get -qq update && \
       apt-get -qq upgrade -y && \
       apt-get -qq install -y coreutils \
                               cron \
                               initscripts \
                               git \
                               curl \
                               unixodbc-dev \
                               sasl2-bin \
                               libsasl2-2 \
                               libsasl2-modules \
                               libsasl2-dev \
                               g++ \
                               gcc \
                               libspatialindex-dev
   
   ARG SPARK_VERSION
   ARG HADOOP_VERSION
   ARG SCALA_VERSION
   
   ENV SPARK_HOME=/opt/spark
   ENV SPARK_CONF_DIR=$SPARK_HOME/conf
   ENV SPARK_CLASSPATH=$SPARK_HOME/cluster-conf
   
   ENV PYTHONHASHSEED=0
   ENV CONDA_DIR=/opt/conda
   ENV SHELL=/bin/bash
   
   ENV PATH=$PATH:$SPARK_HOME/bin:$CONDA_DIR/bin
   
   ENV M2_HOME=/opt/maven
   ENV MAVEN_HOME=/opt/maven
   ENV PATH=${PATH}:${M2_HOME}/bin 
   
   ARG MINICONDA_VERSION=4.8.3
   ARG MINICONDA_MD5=d63adf39f2c220950a063e0529d4ff74
   ARG CONDA_VERSION=4.8.3
   ARG PYTHON_VERSION=3.7.8
   
   ARG spark_uid=185
   
   # Install Conda (https://github.com/jupyter/docker-stacks/blob/6d42503c684f3de9b17ce92a6b0c952ef2d1ecd8/base-notebook/Dockerfile#L78-L101)
   RUN mkdir -p $CONDA_DIR && \
       cd /tmp && \
       wget --quiet https://repo.continuum.io/miniconda/Miniconda3-py38_${MINICONDA_VERSION}-Linux-x86_64.sh && \
       echo "${MINICONDA_MD5} *Miniconda3-py38_${MINICONDA_VERSION}-Linux-x86_64.sh" | md5sum -c - && \
       /bin/bash Miniconda3-py38_${MINICONDA_VERSION}-Linux-x86_64.sh -f -b -p $CONDA_DIR && \
       rm Miniconda3-py38_${MINICONDA_VERSION}-Linux-x86_64.sh && \
       echo "conda ${CONDA_VERSION}" >> $CONDA_DIR/conda-meta/pinned && \
       conda config --system --prepend channels conda-forge && \
       conda config --system --set auto_update_conda false && \
       conda config --system --set show_channel_urls true && \
       conda config --system --set channel_priority strict && \
       if [ ! $PYTHON_VERSION = 'default' ]; then conda install --yes python=$PYTHON_VERSION; fi && \
       conda list python | grep '^python ' | tr -s ' ' | cut -d '.' -f 1,2 | sed 's/$/.*/' >> $CONDA_DIR/conda-meta/pinned && \
       conda install --quiet --yes conda && \
       conda install --quiet --yes pip && \
       conda install --quiet --yes numpy scipy pandas scikit-learn && \
       conda install --quiet --yes -c conda-forge pyarrow && \
       conda update --all --quiet --yes && \
       conda clean --all -f -y
   
   # Install Spark 
   COPY --from=build /spark-${SPARK_VERSION}-bin-hadoop-${HADOOP_VERSION}-scala-${SCALA_VERSION}.tgz /
   RUN tar -xzf /spark-${SPARK_VERSION}-bin-hadoop-${HADOOP_VERSION}-scala-${SCALA_VERSION}.tgz -C /opt/ && \
       ln -s /opt/spark-${SPARK_VERSION}-bin-hadoop-${HADOOP_VERSION}-scala-${SCALA_VERSION} $SPARK_HOME && \
       rm -f /spark-${SPARK_VERSION}-bin-hadoop-${HADOOP_VERSION}-scala-${SCALA_VERSION}.tgz && \
       mkdir -p $SPARK_HOME/work-dir && \
       mkdir -p $SPARK_HOME/spark-warehouse && \
       mkdir -p $SPARK_HOME/cluster-conf
   
   COPY config/* $SPARK_CONF_DIR/
   COPY config/* $SPARK_HOME/cluster-conf/
   COPY entrypoint.sh /opt/
   RUN chmod +x /opt/entrypoint.sh
   
   WORKDIR $SPARK_HOME/work-dir
   RUN chmod g+w /opt/spark/work-dir
   
   ##Install Hudi
   RUN wget -q https://dlcdn.apache.org/maven/maven-3/3.8.3/binaries/apache-maven-3.8.3-bin.tar.gz -P /tmp && \
       tar xf /tmp/apache-maven-3.8.3-bin.tar.gz -C /opt && \
       ln -s /opt/apache-maven-3.8.3 /opt/maven
   
   RUN mvn dependency:copy -Dartifact=org.apache.hudi:hudi-spark3-bundle_2.12:0.9.0 -DoutputDirectory=$SPARK_HOME/jars/
   RUN mvn dependency:copy -Dartifact=org.apache.spark:spark-avro_2.12:3.1.2 -DoutputDirectory=$SPARK_HOME/jars/
   RUN mvn dependency:copy -Dartifact=org.apache.hudi:hudi-hive-sync:0.9.0 -DoutputDirectory=$SPARK_HOME/jars/
   
   
   ENTRYPOINT [ "/opt/entrypoint.sh" ]
   
   USER ${spark_uid}
   ```
   
   
   Could you help me?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] rubenssoto commented on issue #3966: [SUPPORT] Hudi Sync Remote Hive - EKS

Posted by GitBox <gi...@apache.org>.
rubenssoto commented on issue #3966:
URL: https://github.com/apache/hudi/issues/3966#issuecomment-965906492


   I have no problems to create and query tables in hive using spark
   
   ```
   from os.path import expanduser, join, abspath
   from pyspark.sql import SparkSession
   from pyspark.sql import Row
   
   # warehouse_location points to the default location for managed databases and tables
   warehouse_location = abspath('spark-warehouse')
   
   spark = SparkSession \
       .builder \
       .appName("Python Spark SQL Hive integration example") \
       .enableHiveSupport() \
       .config("hive.metastore.uris", "thrift://hive-metastore.hive.svc.cluster.local:9083") \
       .enableHiveSupport() \
       .getOrCreate()
   
   
   spark.sql("create table test_metastore(a int,b int);")
   
   spark.sql("select count(1) from test_metastore").show()
   ```
   
   <img width="1583" alt="Captura de Tela 2021-11-10 às 22 18 28" src="https://user-images.githubusercontent.com/36298331/141219711-a02a167b-c657-4087-a59b-a18f175d07ba.png">
   <img width="1519" alt="Captura de Tela 2021-11-10 às 22 17 21" src="https://user-images.githubusercontent.com/36298331/141219715-61f484a1-8a7a-46cd-9fe8-678827aa2108.png">
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] rubenssoto commented on issue #3966: [SUPPORT] Hudi Sync Remote Hive - EKS

Posted by GitBox <gi...@apache.org>.
rubenssoto commented on issue #3966:
URL: https://github.com/apache/hudi/issues/3966#issuecomment-967728548


   Hello,
   
   The problem was solved, thank you to Rajesh Mahindra.
   
   I was using Spark Kubernetes Operator, Im trying to configure the URI metastore like a Spark Conf, but the right place is Hadoop Conf.
   
   With these two options:
   ```
   hadoopConf:
       hive.metastore.uris: thrift://metastore123:9083
       hive.metastore.client.socket.timeout: 1500s
   
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] rubenssoto closed issue #3966: [SUPPORT] Hudi Sync Remote Hive - EKS

Posted by GitBox <gi...@apache.org>.
rubenssoto closed issue #3966:
URL: https://github.com/apache/hudi/issues/3966


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] rubenssoto commented on issue #3966: [SUPPORT] Hudi Sync Remote Hive - EKS

Posted by GitBox <gi...@apache.org>.
rubenssoto commented on issue #3966:
URL: https://github.com/apache/hudi/issues/3966#issuecomment-965908176


   My hive-site:
   
   <?xml version="1.0" encoding="UTF-8" standalone="no"?>
   <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
   
   <configuration>
     <property>
       <name>javax.jdo.option.ConnectionURL</name>
       <value>jdbc:postgresql://DATA_BASE_URL:5432/metastore</value>
     </property>
   
     <property>
       <name>javax.jdo.option.ConnectionDriverName</name>
       <value>org.postgresql.Driver</value>
     </property>
   
     <property>
       <name>javax.jdo.option.ConnectionUserName</name>
       <value>USER</value>
     </property>
   
     <property>
       <name>javax.jdo.option.ConnectionPassword</name>
       <value>PASSWORD</value>
     </property>
   
   </configuration>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] rubenssoto edited a comment on issue #3966: [SUPPORT] Hudi Sync Remote Hive - EKS

Posted by GitBox <gi...@apache.org>.
rubenssoto edited a comment on issue #3966:
URL: https://github.com/apache/hudi/issues/3966#issuecomment-965908176


   My hive-site:
   ```
   <?xml version="1.0" encoding="UTF-8" standalone="no"?>
   <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
   
   <configuration>
     <property>
       <name>javax.jdo.option.ConnectionURL</name>
       <value>jdbc:postgresql://DATA_BASE_URL:5432/metastore</value>
     </property>
   
     <property>
       <name>javax.jdo.option.ConnectionDriverName</name>
       <value>org.postgresql.Driver</value>
     </property>
   
     <property>
       <name>javax.jdo.option.ConnectionUserName</name>
       <value>USER</value>
     </property>
   
     <property>
       <name>javax.jdo.option.ConnectionPassword</name>
       <value>PASSWORD</value>
     </property>
   
   </configuration>
   
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org