You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@kyuubi.apache.org by fe...@apache.org on 2021/12/02 09:03:28 UTC
[incubator-kyuubi] branch master updated: [KYUUBI #1484] Add a dependency describe for kyuubi on azure delta-lake doc.
This is an automated email from the ASF dual-hosted git repository.
feiwang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-kyuubi.git
The following commit(s) were added to refs/heads/master by this push:
new 7f3f8af [KYUUBI #1484] Add a dependency describe for kyuubi on azure delta-lake doc.
7f3f8af is described below
commit 7f3f8af902fda3a8d1d422212bed3465a914df96
Author: zhichaoleo <zh...@163.com>
AuthorDate: Thu Dec 2 17:01:31 2021 +0800
[KYUUBI #1484] Add a dependency describe for kyuubi on azure delta-lake doc.
### _Why are the changes needed?_
1. After test again in my local, I found I miss a dependency jar which need to be put into ./spark/jars/
### _How was this patch tested?_
I test thses doc by the steps provided by the doc : https://kyuubi.apache.org/docs/latest/develop_tools/build_document.html
Closes #1484 from zhichaoleo/defect/Fix_Doc_For_Delta.
Closes #1484
a1dff005 [zhichaoleo] Add a dependency describe for kyuubi on azure delta-lake doc.
Authored-by: zhichaoleo <zh...@163.com>
Signed-off-by: Fei Wang <fw...@ebay.com>
---
docs/integrations/delta_lake_with_azure_blob.md | 14 ++++++++------
1 file changed, 8 insertions(+), 6 deletions(-)
diff --git a/docs/integrations/delta_lake_with_azure_blob.md b/docs/integrations/delta_lake_with_azure_blob.md
index 712b433..03a9818 100644
--- a/docs/integrations/delta_lake_with_azure_blob.md
+++ b/docs/integrations/delta_lake_with_azure_blob.md
@@ -57,7 +57,7 @@ Create a new file named core-site.xml under ./spark/conf directory, and add foll
<value>org.apache.hadoop.fs.azure.Wasb</value>
</property>
<property>
- <name>fs.azure.account.key.Your_Azure_Account.blob.core.windows.net</name>
+ <name>fs.azure.account.key.YOUR_AZURE_ACCOUNT.blob.core.windows.net</name>
<value>YOUR_AZURE_ACCOUNT_ACCESS_KEY</value>
</property>
<property>
@@ -77,6 +77,8 @@ Create a new file named core-site.xml under ./spark/conf directory, and add foll
#### Copy Dependencies To Spark
Copy jar packages required by delta lake and microsoft azure to ./spark/jars directory:
```shell
+wget https://repo1.maven.org/maven2/io/delta/delta-core_2.12/1.0.0/delta-core_2.12-1.0.0.jar -O ./spark/jars/delta-core_2.12-1.0.0.jar
+
wget https://repo1.maven.org/maven2/com/microsoft/azure/azure-storage/8.6.6/azure-storage-8.6.6.jar -O ./spark/jars/azure-storage-8.6.6.jar
wget https://repo1.maven.org/maven2/com/azure/azure-storage-blob/12.14.2/azure-storage-blob-12.14.2.jar -O ./spark/jars/azure-storage-blob-12.14.2.jar
@@ -97,8 +99,8 @@ Start spark shell:
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
-Spark context Web UI available at http://host:4040
-Spark context available as 'sc' (master = spark://host:7077, app id = app-20211126172803-0003).
+Spark context Web UI available at http://HOST:4040
+Spark context available as 'sc' (master = spark://HOST:7077, app id = app-20211126172803-0003).
Spark session available as 'spark'.
Welcome to
____ __
@@ -217,7 +219,7 @@ Check kyuubi log, in order to check kyuubi start status and find the jdbc connec
2021-11-26 17:49:50.234 INFO session.KyuubiSessionManager: Service[KyuubiSessionManager] is started.
2021-11-26 17:49:50.234 INFO server.KyuubiBackendService: Service[KyuubiBackendService] is started.
2021-11-26 17:49:50.235 INFO service.ThriftFrontendService: Service[ThriftFrontendService] is started.
-2021-11-26 17:49:50.235 INFO service.ThriftFrontendService: Starting and exposing JDBC connection at: jdbc:hive2://host:10009/
+2021-11-26 17:49:50.235 INFO service.ThriftFrontendService: Starting and exposing JDBC connection at: jdbc:hive2://HOST:10009/
2021-11-26 17:49:50.239 INFO zookeeper.ClientCnxn: Session establishment complete on server host/*.*.*.*:2181, sessionid = 0x100046ec0ca01b5, negotiated timeout = 40000
2021-11-26 17:49:50.245 INFO state.ConnectionStateManager: State change: CONNECTED
2021-11-26 17:49:50.247 INFO client.KyuubiServiceDiscovery: Zookeeper client connection state changed to: CONNECTED
@@ -231,7 +233,7 @@ You can get the jdbc connection url by the log:
```
#### Test The Connectivity Of Kyuubi And Delta Lake
```shell
-/usr/apache/current/spark/bin> ./beeline -u 'jdbc:hive2://HOST:10009/'
+/usr/apache/current/spark/bin> ./beeline -u 'jdbc:hive2://<YOUR_HOST>:10009/'
log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.Shell).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
@@ -240,7 +242,7 @@ Connected to: Spark SQL (version 1.3.1-incubating)
Driver: Hive JDBC (version 2.3.7)
Transaction isolation: TRANSACTION_REPEATABLE_READ
Beeline version 2.3.7 by Apache Hive
-0: jdbc:hive2://host>
+0: jdbc:hive2://YOUR_HOST>
```
At the same time, you can also check whether the engine is running on the spark UI:
