You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@carbondata.apache.org by ajithme <gi...@git.apache.org> on 2018/07/18 11:28:26 UTC
[GitHub] carbondata pull request #2522: [CARBONDATA-2752][CARBONSTORE] Carbon provide...
GitHub user ajithme opened a pull request:
https://github.com/apache/carbondata/pull/2522
[CARBONDATA-2752][CARBONSTORE] Carbon provide Zeppelin support
Apache Zeppelin is a popular open web-based notebook that enables interactive data analytics. This is one of the favored solutions for providing UI frontend as it can support solutions like Spark already. Carbon can leverage this to provide a UI for its operations. After CARBONDATA-2688 which provides a carbon REST server, we can add a UI support from zeppelin to provide a complete solution.
- [ ] Document update required? YES, need to update usage guide for Zeppelin integration
- [ ] Testing done
1. Added UT
2. Done Testing manually by integration with Zeppelin
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/ajithme/carbondata zeppelinsupport
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/carbondata/pull/2522.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2522
----
commit 16c69836c3903150875582f950f20cd1189fc69a
Author: Ajith <aj...@...>
Date: 2018-07-18T11:18:54Z
CARBONDATA-2752 Zeppelin support
commit 13801259e5c46b1a4cc736fe94c4bf3678d75794
Author: Ajith <aj...@...>
Date: 2018-07-18T11:26:29Z
update doc
----
---
[GitHub] carbondata issue #2522: [CARBONDATA-2752][CARBONSTORE] Carbon provide Zeppel...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2522
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7300/
---
[GitHub] carbondata pull request #2522: [CARBONDATA-2752][CARBONSTORE] Carbon provide...
Posted by ajithme <gi...@git.apache.org>.
Github user ajithme commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2522#discussion_r203618808
--- Diff: zeppelin/README.txt ---
@@ -0,0 +1,18 @@
+Please follow below steps to integrate with zeppelin
--- End diff --
Done:
https://github.com/ajithme/carbondata/tree/zeppelinsupport/integration/zeppelin
---
[GitHub] carbondata issue #2522: [CARBONDATA-2752][CARBONSTORE] Carbon provide Zeppel...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2522
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6064/
---
[GitHub] carbondata issue #2522: [CARBONDATA-2752][CARBONSTORE] Carbon provide Zeppel...
Posted by ajithme <gi...@git.apache.org>.
Github user ajithme commented on the issue:
https://github.com/apache/carbondata/pull/2522
Merged
---
[GitHub] carbondata pull request #2522: [CARBONDATA-2752][CARBONSTORE] Carbon provide...
Posted by chenliang613 <gi...@git.apache.org>.
Github user chenliang613 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2522#discussion_r203592254
--- Diff: zeppelin/README.txt ---
@@ -0,0 +1,18 @@
+Please follow below steps to integrate with zeppelin
--- End diff --
1. please use .md format
2. Please add apache license header also.
---
[GitHub] carbondata issue #2522: [CARBONDATA-2752][CARBONSTORE] Carbon provide Zeppel...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2522
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6067/
---
[GitHub] carbondata pull request #2522: [CARBONDATA-2752][CARBONSTORE] Carbon provide...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2522#discussion_r203347146
--- Diff: store/sql/src/main/java/org/apache/carbondata/horizon/rest/controller/SqlHorizonController.java ---
@@ -42,20 +43,24 @@
public ResponseEntity<SqlResponse> sql(@RequestBody SqlRequest request) throws StoreException {
RequestValidator.validateSql(request);
List<Row> rows;
+ Dataset<Row> sqlDataFrame = null;
try {
- rows = SparkSqlWrapper.sql(SqlHorizon.getSession(), request.getSqlStatement())
+ sqlDataFrame = SparkSqlWrapper.sql(SqlHorizon.getSession(),
+ request.getSqlStatement());
+ rows = sqlDataFrame
.collectAsList();
} catch (AnalysisException e) {
throw new StoreException(e.getSimpleMessage());
} catch (Exception e) {
throw new StoreException(e.getMessage());
}
- Object[][] result = new Object[rows.size()][];
+ Object[][] result = new Object[rows.size()+1][];
--- End diff --
add space before and after `+`
---
[GitHub] carbondata pull request #2522: [CARBONDATA-2752][CARBONSTORE] Carbon provide...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2522#discussion_r203347064
--- Diff: store/sql/src/main/java/org/apache/carbondata/horizon/rest/controller/SqlHorizonController.java ---
@@ -42,20 +43,24 @@
public ResponseEntity<SqlResponse> sql(@RequestBody SqlRequest request) throws StoreException {
RequestValidator.validateSql(request);
List<Row> rows;
+ Dataset<Row> sqlDataFrame = null;
try {
- rows = SparkSqlWrapper.sql(SqlHorizon.getSession(), request.getSqlStatement())
+ sqlDataFrame = SparkSqlWrapper.sql(SqlHorizon.getSession(),
+ request.getSqlStatement());
+ rows = sqlDataFrame
.collectAsList();
--- End diff --
move it to previous line
---
[GitHub] carbondata issue #2522: [CARBONDATA-2752][CARBONSTORE] Carbon provide Zeppel...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2522
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7295/
---
[GitHub] carbondata pull request #2522: [CARBONDATA-2752][CARBONSTORE] Carbon provide...
Posted by ajithme <gi...@git.apache.org>.
Github user ajithme closed the pull request at:
https://github.com/apache/carbondata/pull/2522
---
[GitHub] carbondata issue #2522: [CARBONDATA-2752][CARBONSTORE] Carbon provide Zeppel...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2522
Can one of the admins verify this patch?
---
[GitHub] carbondata pull request #2522: [CARBONDATA-2752][CARBONSTORE] Carbon provide...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2522#discussion_r203347726
--- Diff: store/sql/src/main/java/org/apache/carbondata/horizon/rest/controller/SqlHorizonController.java ---
@@ -42,20 +43,24 @@
public ResponseEntity<SqlResponse> sql(@RequestBody SqlRequest request) throws StoreException {
RequestValidator.validateSql(request);
List<Row> rows;
+ Dataset<Row> sqlDataFrame = null;
try {
- rows = SparkSqlWrapper.sql(SqlHorizon.getSession(), request.getSqlStatement())
+ sqlDataFrame = SparkSqlWrapper.sql(SqlHorizon.getSession(),
+ request.getSqlStatement());
+ rows = sqlDataFrame
.collectAsList();
} catch (AnalysisException e) {
throw new StoreException(e.getSimpleMessage());
} catch (Exception e) {
throw new StoreException(e.getMessage());
}
- Object[][] result = new Object[rows.size()][];
+ Object[][] result = new Object[rows.size()+1][];
+ result[0] = sqlDataFrame.schema().fieldNames();
for (int i = 0; i < rows.size(); i++) {
Row row = rows.get(i);
- result[i] = new Object[row.size()];
+ result[i+1] = new Object[row.size()];
for (int j = 0; j < row.size(); j++) {
- result[i][j] = row.get(j);
+ result[i+1][j] = row.get(j);
--- End diff --
Can't use System.arraycopy?
---
[GitHub] carbondata issue #2522: [CARBONDATA-2752][CARBONSTORE] Carbon provide Zeppel...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2522
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7323/
---
[GitHub] carbondata issue #2522: [CARBONDATA-2752][CARBONSTORE] Carbon provide Zeppel...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2522
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6089/
---
[GitHub] carbondata pull request #2522: [CARBONDATA-2752][CARBONSTORE] Carbon provide...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2522#discussion_r203347890
--- Diff: store/sql/src/main/java/org/apache/carbondata/horizon/rest/controller/SqlHorizonController.java ---
@@ -42,20 +43,24 @@
public ResponseEntity<SqlResponse> sql(@RequestBody SqlRequest request) throws StoreException {
RequestValidator.validateSql(request);
List<Row> rows;
+ Dataset<Row> sqlDataFrame = null;
try {
- rows = SparkSqlWrapper.sql(SqlHorizon.getSession(), request.getSqlStatement())
+ sqlDataFrame = SparkSqlWrapper.sql(SqlHorizon.getSession(),
+ request.getSqlStatement());
+ rows = sqlDataFrame
.collectAsList();
} catch (AnalysisException e) {
throw new StoreException(e.getSimpleMessage());
} catch (Exception e) {
throw new StoreException(e.getMessage());
}
- Object[][] result = new Object[rows.size()][];
+ Object[][] result = new Object[rows.size()+1][];
+ result[0] = sqlDataFrame.schema().fieldNames();
for (int i = 0; i < rows.size(); i++) {
--- End diff --
`I ` can start from 1
---
[GitHub] carbondata issue #2522: [CARBONDATA-2752][CARBONSTORE] Carbon provide Zeppel...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2522
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7352/
---
[GitHub] carbondata pull request #2522: [CARBONDATA-2752][CARBONSTORE] Carbon provide...
Posted by ajithme <gi...@git.apache.org>.
Github user ajithme commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2522#discussion_r203618873
--- Diff: store/sql/src/main/java/org/apache/carbondata/horizon/rest/controller/SqlHorizonController.java ---
@@ -42,20 +43,24 @@
public ResponseEntity<SqlResponse> sql(@RequestBody SqlRequest request) throws StoreException {
RequestValidator.validateSql(request);
List<Row> rows;
+ Dataset<Row> sqlDataFrame = null;
try {
- rows = SparkSqlWrapper.sql(SqlHorizon.getSession(), request.getSqlStatement())
+ sqlDataFrame = SparkSqlWrapper.sql(SqlHorizon.getSession(),
+ request.getSqlStatement());
+ rows = sqlDataFrame
.collectAsList();
} catch (AnalysisException e) {
throw new StoreException(e.getSimpleMessage());
} catch (Exception e) {
throw new StoreException(e.getMessage());
}
- Object[][] result = new Object[rows.size()][];
+ Object[][] result = new Object[rows.size()+1][];
+ result[0] = sqlDataFrame.schema().fieldNames();
for (int i = 0; i < rows.size(); i++) {
--- End diff --
replaced with stream construct
---
[GitHub] carbondata pull request #2522: [CARBONDATA-2752][CARBONSTORE] Carbon provide...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2522#discussion_r203954062
--- Diff: zeppelin/README.txt ---
@@ -0,0 +1,18 @@
+Please follow below steps to integrate with zeppelin
--- End diff --
Is this README written for carbon specificly?
---
[GitHub] carbondata issue #2522: [CARBONDATA-2752][CARBONSTORE] Carbon provide Zeppel...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2522
Can one of the admins verify this patch?
---
[GitHub] carbondata pull request #2522: [CARBONDATA-2752][CARBONSTORE] Carbon provide...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2522#discussion_r203954155
--- Diff: zeppelin/assembly/assembly.xml ---
@@ -0,0 +1,37 @@
+<assembly>
--- End diff --
please move zeppelin folder to integration folder
---
[GitHub] carbondata pull request #2522: [CARBONDATA-2752][CARBONSTORE] Carbon provide...
Posted by ajithme <gi...@git.apache.org>.
Github user ajithme commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2522#discussion_r203618896
--- Diff: store/sql/src/main/java/org/apache/carbondata/horizon/rest/controller/SqlHorizonController.java ---
@@ -42,20 +43,24 @@
public ResponseEntity<SqlResponse> sql(@RequestBody SqlRequest request) throws StoreException {
RequestValidator.validateSql(request);
List<Row> rows;
+ Dataset<Row> sqlDataFrame = null;
try {
- rows = SparkSqlWrapper.sql(SqlHorizon.getSession(), request.getSqlStatement())
+ sqlDataFrame = SparkSqlWrapper.sql(SqlHorizon.getSession(),
+ request.getSqlStatement());
+ rows = sqlDataFrame
.collectAsList();
} catch (AnalysisException e) {
throw new StoreException(e.getSimpleMessage());
} catch (Exception e) {
throw new StoreException(e.getMessage());
}
- Object[][] result = new Object[rows.size()][];
+ Object[][] result = new Object[rows.size()+1][];
+ result[0] = sqlDataFrame.schema().fieldNames();
for (int i = 0; i < rows.size(); i++) {
Row row = rows.get(i);
- result[i] = new Object[row.size()];
+ result[i+1] = new Object[row.size()];
for (int j = 0; j < row.size(); j++) {
- result[i][j] = row.get(j);
+ result[i+1][j] = row.get(j);
--- End diff --
Done
---
[GitHub] carbondata pull request #2522: [CARBONDATA-2752][CARBONSTORE] Carbon provide...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2522#discussion_r203954234
--- Diff: zeppelin/src/main/java/org/apache/carbondata/zeppelin/CarbonInterpreter.java ---
@@ -0,0 +1,184 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.carbondata.zeppelin;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.util.Arrays;
+import java.util.Objects;
+import java.util.Optional;
+import java.util.Properties;
+import java.util.function.BiFunction;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+import org.apache.carbondata.zeppelin.response.CarbonResponse;
+
+import org.apache.commons.lang.StringUtils;
+import org.apache.http.HttpResponse;
+import org.apache.http.client.HttpClient;
+import org.apache.http.client.methods.HttpPost;
+import org.apache.http.entity.StringEntity;
+import org.apache.http.impl.client.HttpClientBuilder;
+import org.apache.zeppelin.interpreter.Interpreter;
+import org.apache.zeppelin.interpreter.InterpreterContext;
+import org.apache.zeppelin.interpreter.InterpreterException;
+import org.apache.zeppelin.interpreter.InterpreterResult;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Carbon based interpreter for zeppelin
+ */
+public class CarbonInterpreter extends Interpreter {
+
+ public static final Logger logger = LoggerFactory.getLogger(CarbonInterpreter.class);
--- End diff --
Please use Carbon's LogService
---
[GitHub] carbondata issue #2522: [CARBONDATA-2752][CARBONSTORE] Carbon provide Zeppel...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2522
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6116/
---
[GitHub] carbondata issue #2522: [CARBONDATA-2752][CARBONSTORE] Carbon provide Zeppel...
Posted by chenliang613 <gi...@git.apache.org>.
Github user chenliang613 commented on the issue:
https://github.com/apache/carbondata/pull/2522
add to whitelist
---
[GitHub] carbondata issue #2522: [CARBONDATA-2752][CARBONSTORE] Carbon provide Zeppel...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2522
Can one of the admins verify this patch?
---
[GitHub] carbondata pull request #2522: [CARBONDATA-2752][CARBONSTORE] Carbon provide...
Posted by ajithme <gi...@git.apache.org>.
Github user ajithme commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2522#discussion_r203618953
--- Diff: store/sql/src/main/java/org/apache/carbondata/horizon/rest/controller/SqlHorizonController.java ---
@@ -42,20 +43,24 @@
public ResponseEntity<SqlResponse> sql(@RequestBody SqlRequest request) throws StoreException {
RequestValidator.validateSql(request);
List<Row> rows;
+ Dataset<Row> sqlDataFrame = null;
try {
- rows = SparkSqlWrapper.sql(SqlHorizon.getSession(), request.getSqlStatement())
+ sqlDataFrame = SparkSqlWrapper.sql(SqlHorizon.getSession(),
+ request.getSqlStatement());
+ rows = sqlDataFrame
.collectAsList();
--- End diff --
Done
---
[GitHub] carbondata pull request #2522: [CARBONDATA-2752][CARBONSTORE] Carbon provide...
Posted by ajithme <gi...@git.apache.org>.
Github user ajithme commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2522#discussion_r203618925
--- Diff: store/sql/src/main/java/org/apache/carbondata/horizon/rest/controller/SqlHorizonController.java ---
@@ -42,20 +43,24 @@
public ResponseEntity<SqlResponse> sql(@RequestBody SqlRequest request) throws StoreException {
RequestValidator.validateSql(request);
List<Row> rows;
+ Dataset<Row> sqlDataFrame = null;
try {
- rows = SparkSqlWrapper.sql(SqlHorizon.getSession(), request.getSqlStatement())
+ sqlDataFrame = SparkSqlWrapper.sql(SqlHorizon.getSession(),
+ request.getSqlStatement());
+ rows = sqlDataFrame
.collectAsList();
} catch (AnalysisException e) {
throw new StoreException(e.getSimpleMessage());
} catch (Exception e) {
throw new StoreException(e.getMessage());
}
- Object[][] result = new Object[rows.size()][];
+ Object[][] result = new Object[rows.size()+1][];
--- End diff --
Done
---
[GitHub] carbondata issue #2522: [CARBONDATA-2752][CARBONSTORE] Carbon provide Zeppel...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2522
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7297/
---