You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@carbondata.apache.org by SteNicholas <gi...@git.apache.org> on 2018/12/02 01:40:24 UTC
[GitHub] carbondata pull request #2969: [CARBONDATA-3127]Fix the TestCarbonSerde exce...
GitHub user SteNicholas opened a pull request:
https://github.com/apache/carbondata/pull/2969
[CARBONDATA-3127]Fix the TestCarbonSerde exception
[CARBONDATA-3127]Hive module test case has been commented off,can' t run.
This pull request fix TestCarbonSerde test case exception with maven compile.Method called deserializeAndSerializeLazySimple of TestCarbonSerde class calls the interface called serializeStartKey which is not exsit in CarbonHiveSerDe class.Just modify replace deserializeAndSerializeLazySimple with deserialize and remove comments of TestCarbonSerde.
- [x] Any interfaces changed?
No
- [x] Any backward compatibility impacted?
No
- [x] Document update required?
No
- [x] Testing done
Modify TestCarbonSerde test
- [x] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
No
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/SteNicholas/carbondata CARBONDATA-3127
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/carbondata/pull/2969.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2969
----
commit b53a09c3e76442ebfb868104478a31cca8b23c52
Author: Nicholas Jiang <pr...@...>
Date: 2018-11-27T06:51:56Z
update maven profile
commit 06bce3309ded88a9551371a19b3979c6e9ace66d
Author: Nicholas Jiang <pr...@...>
Date: 2018-11-27T06:57:59Z
update maven profile
commit 45b3f9cab524188751bbf2c6ec87dd3db4457990
Author: Nicholas Jiang <pr...@...>
Date: 2018-12-01T12:07:57Z
merge master branch
commit c13e8e3309fa1a09e26437085cab33ddb65dfc01
Author: Nicholas Jiang <pr...@...>
Date: 2018-12-01T12:11:48Z
merge master branch
commit ceeab809493623f92f0166ae1a32424221ea21e0
Author: Nicholas Jiang <pr...@...>
Date: 2018-12-02T01:06:21Z
update TestCarbonSerde test
----
---
[GitHub] carbondata pull request #2969: [CARBONDATA-3127]Fix the TestCarbonSerde exce...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2969#discussion_r243149331
--- Diff: integration/hive/src/main/scala/org/apache/carbondata/hiveexample/HiveExample.scala ---
@@ -49,17 +49,19 @@ object HiveExample {
sparkSession.sql("""DROP TABLE IF EXISTS HIVE_CARBON_EXAMPLE""".stripMargin)
- sparkSession
- .sql(
- "CREATE TABLE HIVE_CARBON_EXAMPLE (ID int,NAME string,SALARY double) " +
- "STORED BY 'CARBONDATA' ")
+ sparkSession.sql(
+ s"""
+ | CREATE TABLE HIVE_CARBON_EXAMPLE
+ | (ID int,NAME string,SALARY double)
+ | STORED BY 'CARBONDATA'
--- End diff --
```suggestion
| STORED AS CARBONDATA
```
---
[GitHub] carbondata issue #2969: [CARBONDATA-3127]Fix the TestCarbonSerde exception
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2969
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1804/
---
[GitHub] carbondata pull request #2969: [CARBONDATA-3127]Fix the TestCarbonSerde exce...
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2969#discussion_r242391201
--- Diff: integration/hive/src/main/scala/org/apache/carbondata/hiveexample/HiveExample.scala ---
@@ -85,19 +82,25 @@ object HiveExample {
logger.info(s"============HIVE CLI IS STARTED ON PORT $port ==============")
- statement.execute("CREATE TABLE IF NOT EXISTS " + "HIVE_CARBON_EXAMPLE " +
- " (ID int, NAME string,SALARY double)")
+ statement.execute("DROP TABLE IF EXISTS HIVE_CARBON_EXAMPLE")
+
+ statement.execute("CREATE TABLE HIVE_CARBON_EXAMPLE " +
+ " (ID int, NAME string,SALARY double) " +
+ "ROW FORMAT SERDE 'org.apache.carbondata.hive.CarbonHiveSerDe' " +
+ "WITH SERDEPROPERTIES ('mapreduce.input.carboninputformat.databaseName'='default', " +
+ "'mapreduce.input.carboninputformat.tableName'='HIVE_CARBON_EXAMPLE')")
+
statement
.execute(
"ALTER TABLE HIVE_CARBON_EXAMPLE SET FILEFORMAT INPUTFORMAT \"org.apache.carbondata." +
- "hive.MapredCarbonInputFormat\"OUTPUTFORMAT \"org.apache.carbondata.hive." +
- "MapredCarbonOutputFormat\"SERDE \"org.apache.carbondata.hive." +
- "CarbonHiveSerDe\" ")
+ "hive.MapredCarbonInputFormat\"OUTPUTFORMAT \"org.apache.carbondata.hive." +
--- End diff --
Can you add space before OUTPUTFORMAT?
---
[GitHub] carbondata pull request #2969: [CARBONDATA-3127]Fix the HiveExample & TestCa...
Posted by SteNicholas <gi...@git.apache.org>.
Github user SteNicholas closed the pull request at:
https://github.com/apache/carbondata/pull/2969
---
[GitHub] carbondata issue #2969: [CARBONDATA-3127]Fix the TestCarbonSerde exception
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2969
Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/2060/
---
[GitHub] carbondata pull request #2969: [CARBONDATA-3127]Fix the TestCarbonSerde exce...
Posted by SteNicholas <gi...@git.apache.org>.
Github user SteNicholas commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2969#discussion_r243259047
--- Diff: integration/hive/src/main/scala/org/apache/carbondata/hiveexample/HiveExample.scala ---
@@ -85,25 +82,35 @@ object HiveExample {
logger.info(s"============HIVE CLI IS STARTED ON PORT $port ==============")
- statement.execute("CREATE TABLE IF NOT EXISTS " + "HIVE_CARBON_EXAMPLE " +
- " (ID int, NAME string,SALARY double)")
- statement
- .execute(
- "ALTER TABLE HIVE_CARBON_EXAMPLE SET FILEFORMAT INPUTFORMAT \"org.apache.carbondata." +
- "hive.MapredCarbonInputFormat\"OUTPUTFORMAT \"org.apache.carbondata.hive." +
- "MapredCarbonOutputFormat\"SERDE \"org.apache.carbondata.hive." +
- "CarbonHiveSerDe\" ")
+ statement.execute(
+ s"""
+ | CREATE TABLE IF NOT EXISTS HIVE_CARBON_EXAMPLE
+ | (ID int, NAME string,SALARY double)
+ | ROW FORMAT SERDE 'org.apache.carbondata.hive.CarbonHiveSerDe'
+ | WITH SERDEPROPERTIES ('mapreduce.input.carboninputformat.databaseName'='default',
+ | 'mapreduce.input.carboninputformat.tableName'='HIVE_CARBON_EXAMPLE')
+ """.stripMargin)
+
+ statement.execute(
+ s"""
+ | ALTER TABLE HIVE_CARBON_EXAMPLE
+ | SET FILEFORMAT
+ | INPUTFORMAT \"org.apache.carbondata.hive.MapredCarbonInputFormat\"
--- End diff --
Yeah, I really agree with you.But this pull request #3004 doesn't work in HiveExample.
---
[GitHub] carbondata issue #2969: [CARBONDATA-3127]Fix the TestCarbonSerde exception
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on the issue:
https://github.com/apache/carbondata/pull/2969
LGTM
---
[GitHub] carbondata issue #2969: [CARBONDATA-3127]Fix the TestCarbonSerde exception
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2969
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1640/
---
[GitHub] carbondata issue #2969: [CARBONDATA-3127]Fix the TestCarbonSerde exception
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2969
Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/1881/
---
[GitHub] carbondata pull request #2969: [CARBONDATA-3127]Fix the TestCarbonSerde exce...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2969#discussion_r243149546
--- Diff: integration/hive/src/main/scala/org/apache/carbondata/hiveexample/HiveExample.scala ---
@@ -85,25 +82,35 @@ object HiveExample {
logger.info(s"============HIVE CLI IS STARTED ON PORT $port ==============")
- statement.execute("CREATE TABLE IF NOT EXISTS " + "HIVE_CARBON_EXAMPLE " +
- " (ID int, NAME string,SALARY double)")
- statement
- .execute(
- "ALTER TABLE HIVE_CARBON_EXAMPLE SET FILEFORMAT INPUTFORMAT \"org.apache.carbondata." +
- "hive.MapredCarbonInputFormat\"OUTPUTFORMAT \"org.apache.carbondata.hive." +
- "MapredCarbonOutputFormat\"SERDE \"org.apache.carbondata.hive." +
- "CarbonHiveSerDe\" ")
+ statement.execute(
+ s"""
+ | CREATE TABLE IF NOT EXISTS HIVE_CARBON_EXAMPLE
+ | (ID int, NAME string,SALARY double)
+ | ROW FORMAT SERDE 'org.apache.carbondata.hive.CarbonHiveSerDe'
+ | WITH SERDEPROPERTIES ('mapreduce.input.carboninputformat.databaseName'='default',
+ | 'mapreduce.input.carboninputformat.tableName'='HIVE_CARBON_EXAMPLE')
+ """.stripMargin)
+
+ statement.execute(
+ s"""
+ | ALTER TABLE HIVE_CARBON_EXAMPLE
+ | SET FILEFORMAT
+ | INPUTFORMAT \"org.apache.carbondata.hive.MapredCarbonInputFormat\"
--- End diff --
This is fine for this PR, and after merging #3004 this ALTER TABLE will not be required
---
[GitHub] carbondata issue #2969: [CARBONDATA-3127]Fix the TestCarbonSerde exception
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2969
Build Success with Spark 2.3.2, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/9932/
---
[GitHub] carbondata issue #2969: [CARBONDATA-3127]Fix the TestCarbonSerde exception
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2969
Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/2045/
---
[GitHub] carbondata issue #2969: [CARBONDATA-3127]Fix the HiveExample & TestCarbonSer...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2969
Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/2097/
---
[GitHub] carbondata pull request #2969: [CARBONDATA-3127]Fix the TestCarbonSerde exce...
Posted by SteNicholas <gi...@git.apache.org>.
Github user SteNicholas commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2969#discussion_r242815115
--- Diff: integration/hive/src/main/scala/org/apache/carbondata/hiveexample/HiveExample.scala ---
@@ -85,19 +82,25 @@ object HiveExample {
logger.info(s"============HIVE CLI IS STARTED ON PORT $port ==============")
- statement.execute("CREATE TABLE IF NOT EXISTS " + "HIVE_CARBON_EXAMPLE " +
- " (ID int, NAME string,SALARY double)")
+ statement.execute("DROP TABLE IF EXISTS HIVE_CARBON_EXAMPLE")
+
+ statement.execute("CREATE TABLE HIVE_CARBON_EXAMPLE " +
+ " (ID int, NAME string,SALARY double) " +
+ "ROW FORMAT SERDE 'org.apache.carbondata.hive.CarbonHiveSerDe' " +
+ "WITH SERDEPROPERTIES ('mapreduce.input.carboninputformat.databaseName'='default', " +
+ "'mapreduce.input.carboninputformat.tableName'='HIVE_CARBON_EXAMPLE')")
+
statement
.execute(
"ALTER TABLE HIVE_CARBON_EXAMPLE SET FILEFORMAT INPUTFORMAT \"org.apache.carbondata." +
- "hive.MapredCarbonInputFormat\"OUTPUTFORMAT \"org.apache.carbondata.hive." +
- "MapredCarbonOutputFormat\"SERDE \"org.apache.carbondata.hive." +
- "CarbonHiveSerDe\" ")
+ "hive.MapredCarbonInputFormat\"OUTPUTFORMAT \"org.apache.carbondata.hive." +
+ "MapredCarbonOutputFormat\"SERDE \"org.apache.carbondata.hive." +
--- End diff --
I have already added space before SERDE
---
[GitHub] carbondata pull request #2969: [CARBONDATA-3127]Fix the TestCarbonSerde exce...
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2969#discussion_r238085518
--- Diff: integration/hive/src/test/java/org/apache/carbondata/hive/TestCarbonSerde.java ---
@@ -1,133 +1,133 @@
-///*
-// * Licensed to the Apache Software Foundation (ASF) under one or more
-// * contributor license agreements. See the NOTICE file distributed with
-// * this work for additional information regarding copyright ownership.
-// * The ASF licenses this file to You under the Apache License, Version 2.0
-// * (the "License"); you may not use this file except in compliance with
-// * the License. You may obtain a copy of the License at
-// *
-// * http://www.apache.org/licenses/LICENSE-2.0
-// *
-// * Unless required by applicable law or agreed to in writing, software
-// * distributed under the License is distributed on an "AS IS" BASIS,
-// * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-// * See the License for the specific language governing permissions and
-// * limitations under the License.
-// */
-//package org.apache.carbondata.hive;
-//
-//import junit.framework.TestCase;
-//import org.apache.hadoop.conf.Configuration;
-//import org.apache.hadoop.hive.common.type.HiveDecimal;
-//import org.apache.hadoop.hive.serde2.SerDeException;
-//import org.apache.hadoop.hive.serde2.SerDeUtils;
-//import org.apache.hadoop.hive.serde2.io.DoubleWritable;
-//import org.apache.hadoop.hive.serde2.io.HiveDecimalWritable;
-//import org.apache.hadoop.hive.serde2.io.ShortWritable;
-//import org.apache.hadoop.hive.serde2.objectinspector.StructObjectInspector;
-//import org.apache.hadoop.io.*;
-//import org.junit.Test;
-//
-//import java.util.Properties;
-//
-//public class TestCarbonSerde extends TestCase {
-// @Test
-// public void testCarbonHiveSerDe() throws Throwable {
-// try {
-// // Create the SerDe
-// System.out.println("test: testCarbonHiveSerDe");
-//
-// final CarbonHiveSerDe serDe = new CarbonHiveSerDe();
-// final Configuration conf = new Configuration();
-// final Properties tbl = createProperties();
-// SerDeUtils.initializeSerDe(serDe, conf, tbl, null);
-//
-// // Data
-// final Writable[] arr = new Writable[7];
-//
-// //primitive types
-// arr[0] = new ShortWritable((short) 456);
-// arr[1] = new IntWritable(789);
-// arr[2] = new LongWritable(1000l);
-// arr[3] = new DoubleWritable((double) 5.3);
-// arr[4] = new HiveDecimalWritable(HiveDecimal.create(1));
-// arr[5] = new Text("carbonSerde binary".getBytes("UTF-8"));
-//
-// final Writable[] arrayContainer = new Writable[1];
-// final Writable[] array = new Writable[5];
-// for (int i = 0; i < 5; ++i) {
-// array[i] = new IntWritable(i);
-// }
-// arrayContainer[0] = new ArrayWritable(Writable.class, array);
-// arr[6] = new ArrayWritable(Writable.class, arrayContainer);
-//
-// final ArrayWritable arrWritable = new ArrayWritable(Writable.class, arr);
-// // Test
-// deserializeAndSerializeLazySimple(serDe, arrWritable);
-// System.out.println("test: testCarbonHiveSerDe - OK");
-//
-// } catch (final Throwable e) {
-// e.printStackTrace();
-// throw e;
-// }
-// }
-//
-// private void deserializeAndSerializeLazySimple(final CarbonHiveSerDe serDe,
-// final ArrayWritable t) throws SerDeException {
-//
-// // Get the row structure
-// final StructObjectInspector oi = (StructObjectInspector) serDe.getObjectInspector();
-//
-// // Deserialize
-// final Object row = serDe.deserialize(t);
-// assertEquals("deserialization gives the wrong object class", row.getClass(),
-// ArrayWritable.class);
-// assertEquals("size correct after deserialization",
-// serDe.getSerDeStats().getRawDataSize(), t.get().length);
-// assertEquals("deserialization gives the wrong object", t, row);
-//
-// // Serialize
-// final ArrayWritable serializedArr = (ArrayWritable) serDe.serializeStartKey(row, oi);
-// assertEquals("size correct after serialization", serDe.getSerDeStats().getRawDataSize(),
-// serializedArr.get().length);
-// assertTrue("serialized object should be equal to starting object",
-// arrayWritableEquals(t, serializedArr));
-// }
-//
-// private Properties createProperties() {
-// final Properties tbl = new Properties();
-//
-// // Set the configuration parameters
-// tbl.setProperty("columns", "ashort,aint,along,adouble,adecimal,astring,alist");
-// tbl.setProperty("columns.types",
-// "smallint:int:bigint:double:decimal:string:array<int>");
-// tbl.setProperty(org.apache.hadoop.hive.serde.serdeConstants.SERIALIZATION_NULL_FORMAT, "NULL");
-// return tbl;
-// }
-//
-// public static boolean arrayWritableEquals(final ArrayWritable a1, final ArrayWritable a2) {
-// final Writable[] a1Arr = a1.get();
-// final Writable[] a2Arr = a2.get();
-//
-// if (a1Arr.length != a2Arr.length) {
-// return false;
-// }
-//
-// for (int i = 0; i < a1Arr.length; ++i) {
-// if (a1Arr[i] instanceof ArrayWritable) {
-// if (!(a2Arr[i] instanceof ArrayWritable)) {
-// return false;
-// }
-// if (!arrayWritableEquals((ArrayWritable) a1Arr[i], (ArrayWritable) a2Arr[i])) {
-// return false;
-// }
-// } else {
-// if (!a1Arr[i].equals(a2Arr[i])) {
-// return false;
-// }
-// }
-//
-// }
-// return true;
-// }
-//}
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.carbondata.hive;
+
+import junit.framework.TestCase;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.common.type.HiveDecimal;
+import org.apache.hadoop.hive.serde2.SerDeException;
+import org.apache.hadoop.hive.serde2.SerDeUtils;
+import org.apache.hadoop.hive.serde2.io.DoubleWritable;
+import org.apache.hadoop.hive.serde2.io.HiveDecimalWritable;
+import org.apache.hadoop.hive.serde2.io.ShortWritable;
+import org.apache.hadoop.hive.serde2.objectinspector.StructObjectInspector;
+import org.apache.hadoop.io.*;
+import org.junit.Test;
+
+import java.util.Properties;
+
+public class TestCarbonSerde extends TestCase {
+ @Test
+ public void testCarbonHiveSerDe() throws Throwable {
+ try {
+ // Create the SerDe
+ System.out.println("test: testCarbonHiveSerDe");
+
+ final CarbonHiveSerDe serDe = new CarbonHiveSerDe();
+ final Configuration conf = new Configuration();
+ final Properties tbl = createProperties();
+ SerDeUtils.initializeSerDe(serDe, conf, tbl, null);
+
+ // Data
+ final Writable[] arr = new Writable[7];
+
+ //primitive types
+ arr[0] = new ShortWritable((short) 456);
+ arr[1] = new IntWritable(789);
+ arr[2] = new LongWritable(1000l);
+ arr[3] = new DoubleWritable((double) 5.3);
--- End diff --
no need: (double) ,please remove it
---
[GitHub] carbondata issue #2969: [CARBONDATA-3127]Fix the TestCarbonSerde exception
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2969
Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/1820/
---
[GitHub] carbondata issue #2969: [CARBONDATA-3127]Fix the TestCarbonSerde exception
Posted by zzcclp <gi...@git.apache.org>.
Github user zzcclp commented on the issue:
https://github.com/apache/carbondata/pull/2969
LGTM
---
[GitHub] carbondata issue #2969: [CARBONDATA-3127]Fix the TestCarbonSerde exception
Posted by SteNicholas <gi...@git.apache.org>.
Github user SteNicholas commented on the issue:
https://github.com/apache/carbondata/pull/2969
@xubo245 Please review this request again include improvement of the comments referred by @xuchuanyin .
---
[GitHub] carbondata pull request #2969: [CARBONDATA-3127]Fix the TestCarbonSerde exce...
Posted by SteNicholas <gi...@git.apache.org>.
Github user SteNicholas commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2969#discussion_r243259131
--- Diff: integration/hive/src/main/scala/org/apache/carbondata/hiveexample/HiveExample.scala ---
@@ -85,25 +82,35 @@ object HiveExample {
logger.info(s"============HIVE CLI IS STARTED ON PORT $port ==============")
- statement.execute("CREATE TABLE IF NOT EXISTS " + "HIVE_CARBON_EXAMPLE " +
- " (ID int, NAME string,SALARY double)")
- statement
- .execute(
- "ALTER TABLE HIVE_CARBON_EXAMPLE SET FILEFORMAT INPUTFORMAT \"org.apache.carbondata." +
- "hive.MapredCarbonInputFormat\"OUTPUTFORMAT \"org.apache.carbondata.hive." +
- "MapredCarbonOutputFormat\"SERDE \"org.apache.carbondata.hive." +
- "CarbonHiveSerDe\" ")
+ statement.execute(
+ s"""
+ | CREATE TABLE IF NOT EXISTS HIVE_CARBON_EXAMPLE
+ | (ID int, NAME string,SALARY double)
+ | ROW FORMAT SERDE 'org.apache.carbondata.hive.CarbonHiveSerDe'
+ | WITH SERDEPROPERTIES ('mapreduce.input.carboninputformat.databaseName'='default',
+ | 'mapreduce.input.carboninputformat.tableName'='HIVE_CARBON_EXAMPLE')
+ """.stripMargin)
+
+ statement.execute(
+ s"""
+ | ALTER TABLE HIVE_CARBON_EXAMPLE
+ | SET FILEFORMAT
+ | INPUTFORMAT \"org.apache.carbondata.hive.MapredCarbonInputFormat\"
--- End diff --
@jackylk Yeah, I really agree with you.But this pull request #3004 doesn't work in HiveExample.
---
[GitHub] carbondata issue #2969: [CARBONDATA-3127]Fix the TestCarbonSerde exception
Posted by zzcclp <gi...@git.apache.org>.
Github user zzcclp commented on the issue:
https://github.com/apache/carbondata/pull/2969
LGTM
---
[GitHub] carbondata issue #2969: [CARBONDATA-3127]Fix the HiveExample & TestCarbonSer...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2969
Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/2088/
---
[GitHub] carbondata issue #2969: [CARBONDATA-3127]Fix the TestCarbonSerde exception
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2969
Build Success with Spark 2.3.2, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/10094/
---
[GitHub] carbondata issue #2969: [CARBONDATA-3127]Fix the TestCarbonSerde exception
Posted by SteNicholas <gi...@git.apache.org>.
Github user SteNicholas commented on the issue:
https://github.com/apache/carbondata/pull/2969
@xubo245 @zzcclp @xuchuanyin I have already fixed hive integration bug based on HiveExample.Please review these updates.
---
[GitHub] carbondata issue #2969: [CARBONDATA-3127]Fix the TestCarbonSerde exception
Posted by SteNicholas <gi...@git.apache.org>.
Github user SteNicholas commented on the issue:
https://github.com/apache/carbondata/pull/2969
@xuchuanyin @xubo245 Please review this request include improvement of the comments referred by @xuchuanyin .
---
[GitHub] carbondata issue #2969: [CARBONDATA-3127]Fix the TestCarbonSerde exception
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2969
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1837/
---
[GitHub] carbondata issue #2969: [CARBONDATA-3127]Fix the TestCarbonSerde exception
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2969
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1806/
---
[GitHub] carbondata pull request #2969: [CARBONDATA-3127]Fix the TestCarbonSerde exce...
Posted by SteNicholas <gi...@git.apache.org>.
Github user SteNicholas commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2969#discussion_r242815043
--- Diff: integration/hive/src/main/scala/org/apache/carbondata/hiveexample/HiveExample.scala ---
@@ -85,19 +82,25 @@ object HiveExample {
logger.info(s"============HIVE CLI IS STARTED ON PORT $port ==============")
- statement.execute("CREATE TABLE IF NOT EXISTS " + "HIVE_CARBON_EXAMPLE " +
- " (ID int, NAME string,SALARY double)")
+ statement.execute("DROP TABLE IF EXISTS HIVE_CARBON_EXAMPLE")
+
+ statement.execute("CREATE TABLE HIVE_CARBON_EXAMPLE " +
+ " (ID int, NAME string,SALARY double) " +
+ "ROW FORMAT SERDE 'org.apache.carbondata.hive.CarbonHiveSerDe' " +
+ "WITH SERDEPROPERTIES ('mapreduce.input.carboninputformat.databaseName'='default', " +
+ "'mapreduce.input.carboninputformat.tableName'='HIVE_CARBON_EXAMPLE')")
+
statement
.execute(
"ALTER TABLE HIVE_CARBON_EXAMPLE SET FILEFORMAT INPUTFORMAT \"org.apache.carbondata." +
- "hive.MapredCarbonInputFormat\"OUTPUTFORMAT \"org.apache.carbondata.hive." +
- "MapredCarbonOutputFormat\"SERDE \"org.apache.carbondata.hive." +
- "CarbonHiveSerDe\" ")
+ "hive.MapredCarbonInputFormat\"OUTPUTFORMAT \"org.apache.carbondata.hive." +
--- End diff --
I have already added space before OUTPUTFORMAT
---
[GitHub] carbondata pull request #2969: [CARBONDATA-3127]Fix the TestCarbonSerde exce...
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2969#discussion_r242390063
--- Diff: integration/hive/src/main/scala/org/apache/carbondata/hiveexample/HiveExample.scala ---
@@ -43,28 +43,25 @@ object HiveExample {
import org.apache.spark.sql.CarbonSession._
- val carbonSession = SparkSession
- .builder()
- .master("local")
- .appName("HiveExample")
- .config("carbonSession.sql.warehouse.dir", warehouse).enableHiveSupport()
- .getOrCreateCarbonSession(
- store, metaStore_Db)
+ val carbonSession = SparkSession.builder()
+ .master("local").appName("HiveExample")
+ .enableHiveSupport()
+ .config("spark.sql.warehouse.dir", warehouse)
+ .getOrCreateCarbonSession(store, metaStore_Db)
carbonSession.sql("""DROP TABLE IF EXISTS HIVE_CARBON_EXAMPLE""".stripMargin)
carbonSession
.sql(
- """CREATE TABLE HIVE_CARBON_EXAMPLE (ID int,NAME string,SALARY double) STORED BY
- |'CARBONDATA' """
- .stripMargin)
+ "CREATE TABLE HIVE_CARBON_EXAMPLE (ID int,NAME string,SALARY double) " +
+ "STORED BY 'CARBONDATA' ")
carbonSession.sql(
s"""
- LOAD DATA LOCAL INPATH '$rootPath/integration/hive/src/main/resources/data.csv' INTO
- TABLE
- HIVE_CARBON_EXAMPLE
- """)
+ LOAD DATA LOCAL INPATH '$rootPath/integration/hive/src/main/resources/data.csv' INTO
+ TABLE
+ HIVE_CARBON_EXAMPLE
--- End diff --
please optimize it or revert it.
---
[GitHub] carbondata pull request #2969: [CARBONDATA-3127]Fix the TestCarbonSerde exce...
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2969#discussion_r238154723
--- Diff: integration/hive/src/test/java/org/apache/carbondata/hive/TestCarbonSerDe.java ---
@@ -0,0 +1,133 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.carbondata.hive;
+
+import junit.framework.TestCase;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.common.type.HiveDecimal;
+import org.apache.hadoop.hive.serde2.SerDeException;
+import org.apache.hadoop.hive.serde2.SerDeUtils;
+import org.apache.hadoop.hive.serde2.io.DoubleWritable;
+import org.apache.hadoop.hive.serde2.io.HiveDecimalWritable;
+import org.apache.hadoop.hive.serde2.io.ShortWritable;
+import org.apache.hadoop.hive.serde2.objectinspector.StructObjectInspector;
+import org.apache.hadoop.io.*;
+import org.junit.Test;
+
+import java.util.Properties;
+
+public class TestCarbonSerDe extends TestCase {
+ @Test
+ public void testCarbonHiveSerDe() throws Throwable {
+ try {
+ // Create the SerDe
+ System.out.println("test: testCarbonHiveSerDe");
+
+ final CarbonHiveSerDe serDe = new CarbonHiveSerDe();
+ final Configuration conf = new Configuration();
+ final Properties tbl = createProperties();
+ SerDeUtils.initializeSerDe(serDe, conf, tbl, null);
+
+ // Data
+ final Writable[] arr = new Writable[7];
+
+ //primitive types
+ arr[0] = new ShortWritable((short) 456);
+ arr[1] = new IntWritable(789);
+ arr[2] = new LongWritable(1000l);
+ arr[3] = new DoubleWritable(5.3);
+ arr[4] = new HiveDecimalWritable(HiveDecimal.create(1));
+ arr[5] = new Text("carbonSerde binary".getBytes("UTF-8"));
--- End diff --
Please take care the spell format, including the case sensitive，for example：CarbonSerDe
---
[GitHub] carbondata issue #2969: [CARBONDATA-3127]Fix the TestCarbonSerde exception
Posted by xuchuanyin <gi...@git.apache.org>.
Github user xuchuanyin commented on the issue:
https://github.com/apache/carbondata/pull/2969
LGTM
---
[GitHub] carbondata issue #2969: [CARBONDATA-3127]Fix the TestCarbonSerde exception
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2969
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1878/
---
[GitHub] carbondata pull request #2969: [CARBONDATA-3127]Fix the TestCarbonSerde exce...
Posted by SteNicholas <gi...@git.apache.org>.
Github user SteNicholas commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2969#discussion_r242814739
--- Diff: integration/hive/src/main/scala/org/apache/carbondata/hiveexample/HiveExample.scala ---
@@ -43,28 +43,25 @@ object HiveExample {
import org.apache.spark.sql.CarbonSession._
- val carbonSession = SparkSession
- .builder()
- .master("local")
- .appName("HiveExample")
- .config("carbonSession.sql.warehouse.dir", warehouse).enableHiveSupport()
- .getOrCreateCarbonSession(
- store, metaStore_Db)
+ val carbonSession = SparkSession.builder()
+ .master("local").appName("HiveExample")
--- End diff --
Yes,I replace this with ExampleUtil to create carbon session.
---
[GitHub] carbondata issue #2969: [CARBONDATA-3127]Fix the TestCarbonSerde exception
Posted by zzcclp <gi...@git.apache.org>.
Github user zzcclp commented on the issue:
https://github.com/apache/carbondata/pull/2969
LGTM
---
[GitHub] carbondata pull request #2969: [CARBONDATA-3127]Fix the TestCarbonSerde exce...
Posted by SteNicholas <gi...@git.apache.org>.
Github user SteNicholas commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2969#discussion_r240114104
--- Diff: integration/hive/src/test/java/org/apache/carbondata/hive/TestCarbonSerDe.java ---
@@ -0,0 +1,133 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.carbondata.hive;
+
+import junit.framework.TestCase;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.common.type.HiveDecimal;
+import org.apache.hadoop.hive.serde2.SerDeException;
+import org.apache.hadoop.hive.serde2.SerDeUtils;
+import org.apache.hadoop.hive.serde2.io.DoubleWritable;
+import org.apache.hadoop.hive.serde2.io.HiveDecimalWritable;
+import org.apache.hadoop.hive.serde2.io.ShortWritable;
+import org.apache.hadoop.hive.serde2.objectinspector.StructObjectInspector;
+import org.apache.hadoop.io.*;
+import org.junit.Test;
+
+import java.util.Properties;
+
+public class TestCarbonSerDe extends TestCase {
+ @Test
+ public void testCarbonHiveSerDe() throws Throwable {
+ try {
+ // Create the SerDe
+ System.out.println("test: testCarbonHiveSerDe");
+
+ final CarbonHiveSerDe serDe = new CarbonHiveSerDe();
+ final Configuration conf = new Configuration();
+ final Properties tbl = createProperties();
+ SerDeUtils.initializeSerDe(serDe, conf, tbl, null);
+
+ // Data
+ final Writable[] arr = new Writable[7];
+
+ //primitive types
+ arr[0] = new ShortWritable((short) 456);
+ arr[1] = new IntWritable(789);
+ arr[2] = new LongWritable(1000l);
+ arr[3] = new DoubleWritable(5.3);
+ arr[4] = new HiveDecimalWritable(HiveDecimal.create(1));
+ arr[5] = new Text("CarbonSerDe Binary".getBytes("UTF-8"));
+
+ final Writable[] arrayContainer = new Writable[1];
+ final Writable[] array = new Writable[5];
+ for (int i = 0; i < 5; ++i) {
+ array[i] = new IntWritable(i);
+ }
+ arrayContainer[0] = new ArrayWritable(Writable.class, array);
+ arr[6] = new ArrayWritable(Writable.class, arrayContainer);
+
+ final ArrayWritable arrWritable = new ArrayWritable(Writable.class, arr);
+ // Test
+ deserializeAndSerializeLazySimple(serDe, arrWritable);
+ System.out.println("test: testCarbonHiveSerDe - OK");
+
+ } catch (final Throwable e) {
--- End diff --
Yeah,I think so.I have already remove this non-sense catch-rethrow in this method.
---
[GitHub] carbondata issue #2969: [CARBONDATA-3127]Fix the TestCarbonSerde exception
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on the issue:
https://github.com/apache/carbondata/pull/2969
@SteNicholas Please optimize the title
---
[GitHub] carbondata issue #2969: [CARBONDATA-3127]Fix the HiveExample & TestCarbonSer...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2969
Build Success with Spark 2.3.2, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/10140/
---
[GitHub] carbondata pull request #2969: [CARBONDATA-3127]Fix the TestCarbonSerde exce...
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2969#discussion_r242395640
--- Diff: integration/hive/src/main/scala/org/apache/carbondata/hiveexample/HiveExample.scala ---
@@ -193,5 +196,4 @@ object HiveExample {
}
hiveEmbeddedServer2.stop()
--- End diff --
please fix the issue: the hiveExample don't stop after running the code. HiveExample should stop.
---
[GitHub] carbondata issue #2969: [CARBONDATA-3127]Fix the TestCarbonSerde exception
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2969
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1808/
---
[GitHub] carbondata pull request #2969: [CARBONDATA-3127]Fix the TestCarbonSerde exce...
Posted by SteNicholas <gi...@git.apache.org>.
Github user SteNicholas commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2969#discussion_r238089109
--- Diff: integration/hive/src/test/java/org/apache/carbondata/hive/TestCarbonSerde.java ---
@@ -1,133 +1,133 @@
-///*
-// * Licensed to the Apache Software Foundation (ASF) under one or more
-// * contributor license agreements. See the NOTICE file distributed with
-// * this work for additional information regarding copyright ownership.
-// * The ASF licenses this file to You under the Apache License, Version 2.0
-// * (the "License"); you may not use this file except in compliance with
-// * the License. You may obtain a copy of the License at
-// *
-// * http://www.apache.org/licenses/LICENSE-2.0
-// *
-// * Unless required by applicable law or agreed to in writing, software
-// * distributed under the License is distributed on an "AS IS" BASIS,
-// * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-// * See the License for the specific language governing permissions and
-// * limitations under the License.
-// */
-//package org.apache.carbondata.hive;
-//
-//import junit.framework.TestCase;
-//import org.apache.hadoop.conf.Configuration;
-//import org.apache.hadoop.hive.common.type.HiveDecimal;
-//import org.apache.hadoop.hive.serde2.SerDeException;
-//import org.apache.hadoop.hive.serde2.SerDeUtils;
-//import org.apache.hadoop.hive.serde2.io.DoubleWritable;
-//import org.apache.hadoop.hive.serde2.io.HiveDecimalWritable;
-//import org.apache.hadoop.hive.serde2.io.ShortWritable;
-//import org.apache.hadoop.hive.serde2.objectinspector.StructObjectInspector;
-//import org.apache.hadoop.io.*;
-//import org.junit.Test;
-//
-//import java.util.Properties;
-//
-//public class TestCarbonSerde extends TestCase {
-// @Test
-// public void testCarbonHiveSerDe() throws Throwable {
-// try {
-// // Create the SerDe
-// System.out.println("test: testCarbonHiveSerDe");
-//
-// final CarbonHiveSerDe serDe = new CarbonHiveSerDe();
-// final Configuration conf = new Configuration();
-// final Properties tbl = createProperties();
-// SerDeUtils.initializeSerDe(serDe, conf, tbl, null);
-//
-// // Data
-// final Writable[] arr = new Writable[7];
-//
-// //primitive types
-// arr[0] = new ShortWritable((short) 456);
-// arr[1] = new IntWritable(789);
-// arr[2] = new LongWritable(1000l);
-// arr[3] = new DoubleWritable((double) 5.3);
-// arr[4] = new HiveDecimalWritable(HiveDecimal.create(1));
-// arr[5] = new Text("carbonSerde binary".getBytes("UTF-8"));
-//
-// final Writable[] arrayContainer = new Writable[1];
-// final Writable[] array = new Writable[5];
-// for (int i = 0; i < 5; ++i) {
-// array[i] = new IntWritable(i);
-// }
-// arrayContainer[0] = new ArrayWritable(Writable.class, array);
-// arr[6] = new ArrayWritable(Writable.class, arrayContainer);
-//
-// final ArrayWritable arrWritable = new ArrayWritable(Writable.class, arr);
-// // Test
-// deserializeAndSerializeLazySimple(serDe, arrWritable);
-// System.out.println("test: testCarbonHiveSerDe - OK");
-//
-// } catch (final Throwable e) {
-// e.printStackTrace();
-// throw e;
-// }
-// }
-//
-// private void deserializeAndSerializeLazySimple(final CarbonHiveSerDe serDe,
-// final ArrayWritable t) throws SerDeException {
-//
-// // Get the row structure
-// final StructObjectInspector oi = (StructObjectInspector) serDe.getObjectInspector();
-//
-// // Deserialize
-// final Object row = serDe.deserialize(t);
-// assertEquals("deserialization gives the wrong object class", row.getClass(),
-// ArrayWritable.class);
-// assertEquals("size correct after deserialization",
-// serDe.getSerDeStats().getRawDataSize(), t.get().length);
-// assertEquals("deserialization gives the wrong object", t, row);
-//
-// // Serialize
-// final ArrayWritable serializedArr = (ArrayWritable) serDe.serializeStartKey(row, oi);
-// assertEquals("size correct after serialization", serDe.getSerDeStats().getRawDataSize(),
-// serializedArr.get().length);
-// assertTrue("serialized object should be equal to starting object",
-// arrayWritableEquals(t, serializedArr));
-// }
-//
-// private Properties createProperties() {
-// final Properties tbl = new Properties();
-//
-// // Set the configuration parameters
-// tbl.setProperty("columns", "ashort,aint,along,adouble,adecimal,astring,alist");
-// tbl.setProperty("columns.types",
-// "smallint:int:bigint:double:decimal:string:array<int>");
-// tbl.setProperty(org.apache.hadoop.hive.serde.serdeConstants.SERIALIZATION_NULL_FORMAT, "NULL");
-// return tbl;
-// }
-//
-// public static boolean arrayWritableEquals(final ArrayWritable a1, final ArrayWritable a2) {
-// final Writable[] a1Arr = a1.get();
-// final Writable[] a2Arr = a2.get();
-//
-// if (a1Arr.length != a2Arr.length) {
-// return false;
-// }
-//
-// for (int i = 0; i < a1Arr.length; ++i) {
-// if (a1Arr[i] instanceof ArrayWritable) {
-// if (!(a2Arr[i] instanceof ArrayWritable)) {
-// return false;
-// }
-// if (!arrayWritableEquals((ArrayWritable) a1Arr[i], (ArrayWritable) a2Arr[i])) {
-// return false;
-// }
-// } else {
-// if (!a1Arr[i].equals(a2Arr[i])) {
-// return false;
-// }
-// }
-//
-// }
-// return true;
-// }
-//}
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.carbondata.hive;
+
+import junit.framework.TestCase;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.common.type.HiveDecimal;
+import org.apache.hadoop.hive.serde2.SerDeException;
+import org.apache.hadoop.hive.serde2.SerDeUtils;
+import org.apache.hadoop.hive.serde2.io.DoubleWritable;
+import org.apache.hadoop.hive.serde2.io.HiveDecimalWritable;
+import org.apache.hadoop.hive.serde2.io.ShortWritable;
+import org.apache.hadoop.hive.serde2.objectinspector.StructObjectInspector;
+import org.apache.hadoop.io.*;
+import org.junit.Test;
+
+import java.util.Properties;
+
+public class TestCarbonSerde extends TestCase {
+ @Test
+ public void testCarbonHiveSerDe() throws Throwable {
+ try {
+ // Create the SerDe
+ System.out.println("test: testCarbonHiveSerDe");
+
+ final CarbonHiveSerDe serDe = new CarbonHiveSerDe();
+ final Configuration conf = new Configuration();
+ final Properties tbl = createProperties();
+ SerDeUtils.initializeSerDe(serDe, conf, tbl, null);
+
+ // Data
+ final Writable[] arr = new Writable[7];
+
+ //primitive types
+ arr[0] = new ShortWritable((short) 456);
+ arr[1] = new IntWritable(789);
+ arr[2] = new LongWritable(1000l);
+ arr[3] = new DoubleWritable((double) 5.3);
--- End diff --
@xubo245 I have already update this transformation.
---
[GitHub] carbondata pull request #2969: [CARBONDATA-3127]Fix the TestCarbonSerde exce...
Posted by SteNicholas <gi...@git.apache.org>.
Github user SteNicholas commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2969#discussion_r242815245
--- Diff: integration/hive/src/main/scala/org/apache/carbondata/hiveexample/HiveExample.scala ---
@@ -193,5 +196,4 @@ object HiveExample {
}
hiveEmbeddedServer2.stop()
--- End diff --
I have already solved hiveExample stop problem with system.exit()
---
[GitHub] carbondata pull request #2969: [CARBONDATA-3127]Fix the TestCarbonSerde exce...
Posted by SteNicholas <gi...@git.apache.org>.
Github user SteNicholas commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2969#discussion_r242814939
--- Diff: integration/hive/src/main/scala/org/apache/carbondata/hiveexample/HiveExample.scala ---
@@ -43,28 +43,25 @@ object HiveExample {
import org.apache.spark.sql.CarbonSession._
- val carbonSession = SparkSession
- .builder()
- .master("local")
- .appName("HiveExample")
- .config("carbonSession.sql.warehouse.dir", warehouse).enableHiveSupport()
- .getOrCreateCarbonSession(
- store, metaStore_Db)
+ val carbonSession = SparkSession.builder()
+ .master("local").appName("HiveExample")
+ .enableHiveSupport()
+ .config("spark.sql.warehouse.dir", warehouse)
+ .getOrCreateCarbonSession(store, metaStore_Db)
carbonSession.sql("""DROP TABLE IF EXISTS HIVE_CARBON_EXAMPLE""".stripMargin)
carbonSession
.sql(
- """CREATE TABLE HIVE_CARBON_EXAMPLE (ID int,NAME string,SALARY double) STORED BY
- |'CARBONDATA' """
- .stripMargin)
+ "CREATE TABLE HIVE_CARBON_EXAMPLE (ID int,NAME string,SALARY double) " +
+ "STORED BY 'CARBONDATA' ")
carbonSession.sql(
s"""
- LOAD DATA LOCAL INPATH '$rootPath/integration/hive/src/main/resources/data.csv' INTO
- TABLE
- HIVE_CARBON_EXAMPLE
- """)
+ LOAD DATA LOCAL INPATH '$rootPath/integration/hive/src/main/resources/data.csv' INTO
+ TABLE
+ HIVE_CARBON_EXAMPLE
--- End diff --
Yeah, I didn't optimize this SQL format
---
[GitHub] carbondata issue #2969: [CARBONDATA-3127]Fix the TestCarbonSerde exception
Posted by SteNicholas <gi...@git.apache.org>.
Github user SteNicholas commented on the issue:
https://github.com/apache/carbondata/pull/2969
@zzcclp Please review this update.
---
[GitHub] carbondata issue #2969: [CARBONDATA-3127]Fix the HiveExample & TestCarbonSer...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2969
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1885/
---
[GitHub] carbondata pull request #2969: [CARBONDATA-3127]Fix the TestCarbonSerde exce...
Posted by xuchuanyin <gi...@git.apache.org>.
Github user xuchuanyin commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2969#discussion_r239070238
--- Diff: integration/hive/src/test/java/org/apache/carbondata/hive/TestCarbonSerDe.java ---
@@ -0,0 +1,133 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.carbondata.hive;
+
+import junit.framework.TestCase;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.common.type.HiveDecimal;
+import org.apache.hadoop.hive.serde2.SerDeException;
+import org.apache.hadoop.hive.serde2.SerDeUtils;
+import org.apache.hadoop.hive.serde2.io.DoubleWritable;
+import org.apache.hadoop.hive.serde2.io.HiveDecimalWritable;
+import org.apache.hadoop.hive.serde2.io.ShortWritable;
+import org.apache.hadoop.hive.serde2.objectinspector.StructObjectInspector;
+import org.apache.hadoop.io.*;
+import org.junit.Test;
+
+import java.util.Properties;
+
+public class TestCarbonSerDe extends TestCase {
+ @Test
+ public void testCarbonHiveSerDe() throws Throwable {
+ try {
+ // Create the SerDe
+ System.out.println("test: testCarbonHiveSerDe");
--- End diff --
It's not recommended to use stdout in test code. Currently we only use stdout in `example` code not `test` code. If you want to print something, use carbon Logger instead
---
[GitHub] carbondata issue #2969: [CARBONDATA-3127]Fix the TestCarbonSerde exception
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2969
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1670/
---
[GitHub] carbondata pull request #2969: [CARBONDATA-3127]Fix the TestCarbonSerde exce...
Posted by SteNicholas <gi...@git.apache.org>.
Github user SteNicholas commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2969#discussion_r238089149
--- Diff: integration/hive/src/test/java/org/apache/carbondata/hive/TestCarbonSerde.java ---
@@ -1,133 +1,133 @@
-///*
-// * Licensed to the Apache Software Foundation (ASF) under one or more
-// * contributor license agreements. See the NOTICE file distributed with
-// * this work for additional information regarding copyright ownership.
-// * The ASF licenses this file to You under the Apache License, Version 2.0
-// * (the "License"); you may not use this file except in compliance with
-// * the License. You may obtain a copy of the License at
-// *
-// * http://www.apache.org/licenses/LICENSE-2.0
-// *
-// * Unless required by applicable law or agreed to in writing, software
-// * distributed under the License is distributed on an "AS IS" BASIS,
-// * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-// * See the License for the specific language governing permissions and
-// * limitations under the License.
-// */
-//package org.apache.carbondata.hive;
-//
-//import junit.framework.TestCase;
-//import org.apache.hadoop.conf.Configuration;
-//import org.apache.hadoop.hive.common.type.HiveDecimal;
-//import org.apache.hadoop.hive.serde2.SerDeException;
-//import org.apache.hadoop.hive.serde2.SerDeUtils;
-//import org.apache.hadoop.hive.serde2.io.DoubleWritable;
-//import org.apache.hadoop.hive.serde2.io.HiveDecimalWritable;
-//import org.apache.hadoop.hive.serde2.io.ShortWritable;
-//import org.apache.hadoop.hive.serde2.objectinspector.StructObjectInspector;
-//import org.apache.hadoop.io.*;
-//import org.junit.Test;
-//
-//import java.util.Properties;
-//
-//public class TestCarbonSerde extends TestCase {
-// @Test
-// public void testCarbonHiveSerDe() throws Throwable {
-// try {
-// // Create the SerDe
-// System.out.println("test: testCarbonHiveSerDe");
-//
-// final CarbonHiveSerDe serDe = new CarbonHiveSerDe();
-// final Configuration conf = new Configuration();
-// final Properties tbl = createProperties();
-// SerDeUtils.initializeSerDe(serDe, conf, tbl, null);
-//
-// // Data
-// final Writable[] arr = new Writable[7];
-//
-// //primitive types
-// arr[0] = new ShortWritable((short) 456);
-// arr[1] = new IntWritable(789);
-// arr[2] = new LongWritable(1000l);
-// arr[3] = new DoubleWritable((double) 5.3);
-// arr[4] = new HiveDecimalWritable(HiveDecimal.create(1));
-// arr[5] = new Text("carbonSerde binary".getBytes("UTF-8"));
-//
-// final Writable[] arrayContainer = new Writable[1];
-// final Writable[] array = new Writable[5];
-// for (int i = 0; i < 5; ++i) {
-// array[i] = new IntWritable(i);
-// }
-// arrayContainer[0] = new ArrayWritable(Writable.class, array);
-// arr[6] = new ArrayWritable(Writable.class, arrayContainer);
-//
-// final ArrayWritable arrWritable = new ArrayWritable(Writable.class, arr);
-// // Test
-// deserializeAndSerializeLazySimple(serDe, arrWritable);
-// System.out.println("test: testCarbonHiveSerDe - OK");
-//
-// } catch (final Throwable e) {
-// e.printStackTrace();
-// throw e;
-// }
-// }
-//
-// private void deserializeAndSerializeLazySimple(final CarbonHiveSerDe serDe,
-// final ArrayWritable t) throws SerDeException {
-//
-// // Get the row structure
-// final StructObjectInspector oi = (StructObjectInspector) serDe.getObjectInspector();
-//
-// // Deserialize
-// final Object row = serDe.deserialize(t);
-// assertEquals("deserialization gives the wrong object class", row.getClass(),
-// ArrayWritable.class);
-// assertEquals("size correct after deserialization",
-// serDe.getSerDeStats().getRawDataSize(), t.get().length);
-// assertEquals("deserialization gives the wrong object", t, row);
-//
-// // Serialize
-// final ArrayWritable serializedArr = (ArrayWritable) serDe.serializeStartKey(row, oi);
-// assertEquals("size correct after serialization", serDe.getSerDeStats().getRawDataSize(),
-// serializedArr.get().length);
-// assertTrue("serialized object should be equal to starting object",
-// arrayWritableEquals(t, serializedArr));
-// }
-//
-// private Properties createProperties() {
-// final Properties tbl = new Properties();
-//
-// // Set the configuration parameters
-// tbl.setProperty("columns", "ashort,aint,along,adouble,adecimal,astring,alist");
-// tbl.setProperty("columns.types",
-// "smallint:int:bigint:double:decimal:string:array<int>");
-// tbl.setProperty(org.apache.hadoop.hive.serde.serdeConstants.SERIALIZATION_NULL_FORMAT, "NULL");
-// return tbl;
-// }
-//
-// public static boolean arrayWritableEquals(final ArrayWritable a1, final ArrayWritable a2) {
-// final Writable[] a1Arr = a1.get();
-// final Writable[] a2Arr = a2.get();
-//
-// if (a1Arr.length != a2Arr.length) {
-// return false;
-// }
-//
-// for (int i = 0; i < a1Arr.length; ++i) {
-// if (a1Arr[i] instanceof ArrayWritable) {
-// if (!(a2Arr[i] instanceof ArrayWritable)) {
-// return false;
-// }
-// if (!arrayWritableEquals((ArrayWritable) a1Arr[i], (ArrayWritable) a2Arr[i])) {
-// return false;
-// }
-// } else {
-// if (!a1Arr[i].equals(a2Arr[i])) {
-// return false;
-// }
-// }
-//
-// }
-// return true;
-// }
-//}
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.carbondata.hive;
+
+import junit.framework.TestCase;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.common.type.HiveDecimal;
+import org.apache.hadoop.hive.serde2.SerDeException;
+import org.apache.hadoop.hive.serde2.SerDeUtils;
+import org.apache.hadoop.hive.serde2.io.DoubleWritable;
+import org.apache.hadoop.hive.serde2.io.HiveDecimalWritable;
+import org.apache.hadoop.hive.serde2.io.ShortWritable;
+import org.apache.hadoop.hive.serde2.objectinspector.StructObjectInspector;
+import org.apache.hadoop.io.*;
+import org.junit.Test;
+
+import java.util.Properties;
+
+public class TestCarbonSerde extends TestCase {
+ @Test
+ public void testCarbonHiveSerDe() throws Throwable {
+ try {
+ // Create the SerDe
+ System.out.println("test: testCarbonHiveSerDe");
+
+ final CarbonHiveSerDe serDe = new CarbonHiveSerDe();
+ final Configuration conf = new Configuration();
+ final Properties tbl = createProperties();
+ SerDeUtils.initializeSerDe(serDe, conf, tbl, null);
+
+ // Data
+ final Writable[] arr = new Writable[7];
+
+ //primitive types
+ arr[0] = new ShortWritable((short) 456);
+ arr[1] = new IntWritable(789);
+ arr[2] = new LongWritable(1000l);
+ arr[3] = new DoubleWritable((double) 5.3);
+ arr[4] = new HiveDecimalWritable(HiveDecimal.create(1));
+ arr[5] = new Text("carbonSerde binary".getBytes("UTF-8"));
+
+ final Writable[] arrayContainer = new Writable[1];
+ final Writable[] array = new Writable[5];
+ for (int i = 0; i < 5; ++i) {
+ array[i] = new IntWritable(i);
+ }
+ arrayContainer[0] = new ArrayWritable(Writable.class, array);
+ arr[6] = new ArrayWritable(Writable.class, arrayContainer);
+
+ final ArrayWritable arrWritable = new ArrayWritable(Writable.class, arr);
+ // Test
+ deserializeAndSerializeLazySimple(serDe, arrWritable);
+ System.out.println("test: testCarbonHiveSerDe - OK");
+
+ } catch (final Throwable e) {
+ e.printStackTrace();
+ throw e;
+ }
+ }
+
+ private void deserializeAndSerializeLazySimple(final CarbonHiveSerDe serDe,
+ final ArrayWritable t) throws SerDeException {
+
+ // Get the row structure
+ final StructObjectInspector oi = (StructObjectInspector) serDe.getObjectInspector();
+
+ // Deserialize
+ final Object row = serDe.deserialize(t);
+ assertEquals("deserialization gives the wrong object class", row.getClass(),
+ ArrayWritable.class);
+ assertEquals("size correct after deserialization",
+ serDe.getSerDeStats().getRawDataSize(), t.get().length);
+ assertEquals("deserialization gives the wrong object", t, row);
+
+ // Serialize
+ final ArrayWritable serializedArr = (ArrayWritable) serDe.serialize(row, oi);
+ assertEquals("size correct after serialization", serDe.getSerDeStats().getRawDataSize(),
+ serializedArr.get().length);
+ assertTrue("serialized object should be equal to starting object",
+ arrayWritableEquals(t, serializedArr));
+ }
+
+ private Properties createProperties() {
+ final Properties tbl = new Properties();
+
+ // Set the configuration parameters
+ tbl.setProperty("columns", "ashort,aint,along,adouble,adecimal,astring,alist");
--- End diff --
@xubo245 I have already fixed the spell formate you referred.
---
[GitHub] carbondata issue #2969: [CARBONDATA-3127]Fix the TestCarbonSerde exception
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2969
Build Success with Spark 2.3.2, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/9930/
---
[GitHub] carbondata pull request #2969: [CARBONDATA-3127]Fix the TestCarbonSerde exce...
Posted by xuchuanyin <gi...@git.apache.org>.
Github user xuchuanyin commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2969#discussion_r239071026
--- Diff: integration/hive/src/test/java/org/apache/carbondata/hive/TestCarbonSerDe.java ---
@@ -0,0 +1,133 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.carbondata.hive;
+
+import junit.framework.TestCase;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.common.type.HiveDecimal;
+import org.apache.hadoop.hive.serde2.SerDeException;
+import org.apache.hadoop.hive.serde2.SerDeUtils;
+import org.apache.hadoop.hive.serde2.io.DoubleWritable;
+import org.apache.hadoop.hive.serde2.io.HiveDecimalWritable;
+import org.apache.hadoop.hive.serde2.io.ShortWritable;
+import org.apache.hadoop.hive.serde2.objectinspector.StructObjectInspector;
+import org.apache.hadoop.io.*;
+import org.junit.Test;
+
+import java.util.Properties;
+
+public class TestCarbonSerDe extends TestCase {
+ @Test
+ public void testCarbonHiveSerDe() throws Throwable {
+ try {
+ // Create the SerDe
+ System.out.println("test: testCarbonHiveSerDe");
+
+ final CarbonHiveSerDe serDe = new CarbonHiveSerDe();
+ final Configuration conf = new Configuration();
+ final Properties tbl = createProperties();
+ SerDeUtils.initializeSerDe(serDe, conf, tbl, null);
+
+ // Data
+ final Writable[] arr = new Writable[7];
+
+ //primitive types
+ arr[0] = new ShortWritable((short) 456);
+ arr[1] = new IntWritable(789);
+ arr[2] = new LongWritable(1000l);
+ arr[3] = new DoubleWritable(5.3);
+ arr[4] = new HiveDecimalWritable(HiveDecimal.create(1));
+ arr[5] = new Text("CarbonSerDe Binary".getBytes("UTF-8"));
+
+ final Writable[] arrayContainer = new Writable[1];
+ final Writable[] array = new Writable[5];
+ for (int i = 0; i < 5; ++i) {
+ array[i] = new IntWritable(i);
+ }
+ arrayContainer[0] = new ArrayWritable(Writable.class, array);
+ arr[6] = new ArrayWritable(Writable.class, arrayContainer);
+
+ final ArrayWritable arrWritable = new ArrayWritable(Writable.class, arr);
+ // Test
+ deserializeAndSerializeLazySimple(serDe, arrWritable);
+ System.out.println("test: testCarbonHiveSerDe - OK");
+
+ } catch (final Throwable e) {
--- End diff --
after observing the following procedure, I think there is no need to catch-rethrow the exception here. The test framework will handle this
---
[GitHub] carbondata issue #2969: [CARBONDATA-3127]Fix the TestCarbonSerde exception
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2969
Build Failed with Spark 2.3.2, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/10093/
---
[GitHub] carbondata pull request #2969: [CARBONDATA-3127]Fix the TestCarbonSerde exce...
Posted by SteNicholas <gi...@git.apache.org>.
Github user SteNicholas commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2969#discussion_r243259414
--- Diff: integration/hive/src/main/scala/org/apache/carbondata/hiveexample/HiveExample.scala ---
@@ -22,52 +22,49 @@ import java.sql.{DriverManager, ResultSet, Statement}
import org.apache.spark.sql.SparkSession
--- End diff --
@jackylk I have already moved this example to example folder and rename module named carbondata-example-spark2.
---
[GitHub] carbondata pull request #2969: [CARBONDATA-3127]Fix the TestCarbonSerde exce...
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2969#discussion_r242389254
--- Diff: integration/hive/src/main/scala/org/apache/carbondata/hiveexample/HiveExample.scala ---
@@ -43,28 +43,25 @@ object HiveExample {
import org.apache.spark.sql.CarbonSession._
- val carbonSession = SparkSession
- .builder()
- .master("local")
- .appName("HiveExample")
- .config("carbonSession.sql.warehouse.dir", warehouse).enableHiveSupport()
- .getOrCreateCarbonSession(
- store, metaStore_Db)
+ val carbonSession = SparkSession.builder()
+ .master("local").appName("HiveExample")
--- End diff --
Please keep it，no need to change
---
[GitHub] carbondata issue #2969: [CARBONDATA-3127]Fix the TestCarbonSerde exception
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on the issue:
https://github.com/apache/carbondata/pull/2969
LGTM
---
[GitHub] carbondata issue #2969: [CARBONDATA-3127]Fix the HiveExample & TestCarbonSer...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2969
Build Success with Spark 2.3.2, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/10133/
---
[GitHub] carbondata pull request #2969: [CARBONDATA-3127]Fix the TestCarbonSerde exce...
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2969#discussion_r242817509
--- Diff: integration/hive/src/main/scala/org/apache/carbondata/hiveexample/HiveExample.scala ---
@@ -85,19 +82,25 @@ object HiveExample {
logger.info(s"============HIVE CLI IS STARTED ON PORT $port ==============")
- statement.execute("CREATE TABLE IF NOT EXISTS " + "HIVE_CARBON_EXAMPLE " +
- " (ID int, NAME string,SALARY double)")
+ statement.execute("DROP TABLE IF EXISTS HIVE_CARBON_EXAMPLE")
+
+ statement.execute("CREATE TABLE HIVE_CARBON_EXAMPLE " +
+ " (ID int, NAME string,SALARY double) " +
+ "ROW FORMAT SERDE 'org.apache.carbondata.hive.CarbonHiveSerDe' " +
+ "WITH SERDEPROPERTIES ('mapreduce.input.carboninputformat.databaseName'='default', " +
+ "'mapreduce.input.carboninputformat.tableName'='HIVE_CARBON_EXAMPLE')")
+
statement
.execute(
"ALTER TABLE HIVE_CARBON_EXAMPLE SET FILEFORMAT INPUTFORMAT \"org.apache.carbondata." +
- "hive.MapredCarbonInputFormat\"OUTPUTFORMAT \"org.apache.carbondata.hive." +
- "MapredCarbonOutputFormat\"SERDE \"org.apache.carbondata.hive." +
- "CarbonHiveSerDe\" ")
+ "hive.MapredCarbonInputFormat\"OUTPUTFORMAT \"org.apache.carbondata.hive." +
--- End diff --
ok
---
[GitHub] carbondata issue #2969: [CARBONDATA-3127]Fix the HiveExample & TestCarbonSer...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2969
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1888/
---
[GitHub] carbondata issue #2969: [CARBONDATA-3127]Fix the TestCarbonSerde exception
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2969
Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/2019/
---
[GitHub] carbondata issue #2969: [CARBONDATA-3127]Fix the TestCarbonSerde exception
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2969
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1805/
---
[GitHub] carbondata pull request #2969: [CARBONDATA-3127]Fix the TestCarbonSerde exce...
Posted by xuchuanyin <gi...@git.apache.org>.
Github user xuchuanyin commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2969#discussion_r239070652
--- Diff: integration/hive/src/test/java/org/apache/carbondata/hive/TestCarbonSerDe.java ---
@@ -0,0 +1,133 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.carbondata.hive;
+
+import junit.framework.TestCase;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.common.type.HiveDecimal;
+import org.apache.hadoop.hive.serde2.SerDeException;
+import org.apache.hadoop.hive.serde2.SerDeUtils;
+import org.apache.hadoop.hive.serde2.io.DoubleWritable;
+import org.apache.hadoop.hive.serde2.io.HiveDecimalWritable;
+import org.apache.hadoop.hive.serde2.io.ShortWritable;
+import org.apache.hadoop.hive.serde2.objectinspector.StructObjectInspector;
+import org.apache.hadoop.io.*;
+import org.junit.Test;
+
+import java.util.Properties;
+
+public class TestCarbonSerDe extends TestCase {
+ @Test
+ public void testCarbonHiveSerDe() throws Throwable {
+ try {
+ // Create the SerDe
+ System.out.println("test: testCarbonHiveSerDe");
+
+ final CarbonHiveSerDe serDe = new CarbonHiveSerDe();
+ final Configuration conf = new Configuration();
+ final Properties tbl = createProperties();
+ SerDeUtils.initializeSerDe(serDe, conf, tbl, null);
+
+ // Data
+ final Writable[] arr = new Writable[7];
+
+ //primitive types
+ arr[0] = new ShortWritable((short) 456);
+ arr[1] = new IntWritable(789);
+ arr[2] = new LongWritable(1000l);
+ arr[3] = new DoubleWritable(5.3);
+ arr[4] = new HiveDecimalWritable(HiveDecimal.create(1));
+ arr[5] = new Text("CarbonSerDe Binary".getBytes("UTF-8"));
+
+ final Writable[] arrayContainer = new Writable[1];
+ final Writable[] array = new Writable[5];
+ for (int i = 0; i < 5; ++i) {
+ array[i] = new IntWritable(i);
+ }
+ arrayContainer[0] = new ArrayWritable(Writable.class, array);
+ arr[6] = new ArrayWritable(Writable.class, arrayContainer);
+
+ final ArrayWritable arrWritable = new ArrayWritable(Writable.class, arr);
+ // Test
+ deserializeAndSerializeLazySimple(serDe, arrWritable);
+ System.out.println("test: testCarbonHiveSerDe - OK");
+
+ } catch (final Throwable e) {
+ e.printStackTrace();
--- End diff --
use Logger instead of printing stacktarce in test code
---
[GitHub] carbondata pull request #2969: [CARBONDATA-3127]Fix the TestCarbonSerde exce...
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2969#discussion_r242394285
--- Diff: integration/hive/src/main/scala/org/apache/carbondata/hiveexample/HiveExample.scala ---
@@ -31,7 +31,7 @@ object HiveExample {
def main(args: Array[String]) {
--- End diff --
Please optimize this example and RunHiveExampleTest like:org.apache.carbondata.examplesCI.RunExamples.
We should add the example to CI, and add some assert for exception and result. We should avoid that some developer/PR change code and lead to the hiveExample error, the developer need fix it before their PR merged
---
[GitHub] carbondata issue #2969: [CARBONDATA-3127]Fix the TestCarbonSerde exception
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2969
Build Success with Spark 2.3.2, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/10068/
---
[GitHub] carbondata issue #2969: [CARBONDATA-3127]Fix the HiveExample & TestCarbonSer...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2969
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1887/
---
[GitHub] carbondata issue #2969: [CARBONDATA-3127]Fix the TestCarbonSerde exception
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2969
Build Success with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/9870/
---
[GitHub] carbondata issue #2969: [CARBONDATA-3127]Fix the TestCarbonSerde exception
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2969
Build Success with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/9900/
---
[GitHub] carbondata issue #2969: [CARBONDATA-3127]Fix the TestCarbonSerde exception
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2969
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1672/
---
[GitHub] carbondata pull request #2969: [CARBONDATA-3127]Fix the TestCarbonSerde exce...
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2969#discussion_r238085487
--- Diff: integration/hive/src/test/java/org/apache/carbondata/hive/TestCarbonSerde.java ---
@@ -1,133 +1,133 @@
-///*
-// * Licensed to the Apache Software Foundation (ASF) under one or more
-// * contributor license agreements. See the NOTICE file distributed with
-// * this work for additional information regarding copyright ownership.
-// * The ASF licenses this file to You under the Apache License, Version 2.0
-// * (the "License"); you may not use this file except in compliance with
-// * the License. You may obtain a copy of the License at
-// *
-// * http://www.apache.org/licenses/LICENSE-2.0
-// *
-// * Unless required by applicable law or agreed to in writing, software
-// * distributed under the License is distributed on an "AS IS" BASIS,
-// * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-// * See the License for the specific language governing permissions and
-// * limitations under the License.
-// */
-//package org.apache.carbondata.hive;
-//
-//import junit.framework.TestCase;
-//import org.apache.hadoop.conf.Configuration;
-//import org.apache.hadoop.hive.common.type.HiveDecimal;
-//import org.apache.hadoop.hive.serde2.SerDeException;
-//import org.apache.hadoop.hive.serde2.SerDeUtils;
-//import org.apache.hadoop.hive.serde2.io.DoubleWritable;
-//import org.apache.hadoop.hive.serde2.io.HiveDecimalWritable;
-//import org.apache.hadoop.hive.serde2.io.ShortWritable;
-//import org.apache.hadoop.hive.serde2.objectinspector.StructObjectInspector;
-//import org.apache.hadoop.io.*;
-//import org.junit.Test;
-//
-//import java.util.Properties;
-//
-//public class TestCarbonSerde extends TestCase {
-// @Test
-// public void testCarbonHiveSerDe() throws Throwable {
-// try {
-// // Create the SerDe
-// System.out.println("test: testCarbonHiveSerDe");
-//
-// final CarbonHiveSerDe serDe = new CarbonHiveSerDe();
-// final Configuration conf = new Configuration();
-// final Properties tbl = createProperties();
-// SerDeUtils.initializeSerDe(serDe, conf, tbl, null);
-//
-// // Data
-// final Writable[] arr = new Writable[7];
-//
-// //primitive types
-// arr[0] = new ShortWritable((short) 456);
-// arr[1] = new IntWritable(789);
-// arr[2] = new LongWritable(1000l);
-// arr[3] = new DoubleWritable((double) 5.3);
-// arr[4] = new HiveDecimalWritable(HiveDecimal.create(1));
-// arr[5] = new Text("carbonSerde binary".getBytes("UTF-8"));
-//
-// final Writable[] arrayContainer = new Writable[1];
-// final Writable[] array = new Writable[5];
-// for (int i = 0; i < 5; ++i) {
-// array[i] = new IntWritable(i);
-// }
-// arrayContainer[0] = new ArrayWritable(Writable.class, array);
-// arr[6] = new ArrayWritable(Writable.class, arrayContainer);
-//
-// final ArrayWritable arrWritable = new ArrayWritable(Writable.class, arr);
-// // Test
-// deserializeAndSerializeLazySimple(serDe, arrWritable);
-// System.out.println("test: testCarbonHiveSerDe - OK");
-//
-// } catch (final Throwable e) {
-// e.printStackTrace();
-// throw e;
-// }
-// }
-//
-// private void deserializeAndSerializeLazySimple(final CarbonHiveSerDe serDe,
-// final ArrayWritable t) throws SerDeException {
-//
-// // Get the row structure
-// final StructObjectInspector oi = (StructObjectInspector) serDe.getObjectInspector();
-//
-// // Deserialize
-// final Object row = serDe.deserialize(t);
-// assertEquals("deserialization gives the wrong object class", row.getClass(),
-// ArrayWritable.class);
-// assertEquals("size correct after deserialization",
-// serDe.getSerDeStats().getRawDataSize(), t.get().length);
-// assertEquals("deserialization gives the wrong object", t, row);
-//
-// // Serialize
-// final ArrayWritable serializedArr = (ArrayWritable) serDe.serializeStartKey(row, oi);
-// assertEquals("size correct after serialization", serDe.getSerDeStats().getRawDataSize(),
-// serializedArr.get().length);
-// assertTrue("serialized object should be equal to starting object",
-// arrayWritableEquals(t, serializedArr));
-// }
-//
-// private Properties createProperties() {
-// final Properties tbl = new Properties();
-//
-// // Set the configuration parameters
-// tbl.setProperty("columns", "ashort,aint,along,adouble,adecimal,astring,alist");
-// tbl.setProperty("columns.types",
-// "smallint:int:bigint:double:decimal:string:array<int>");
-// tbl.setProperty(org.apache.hadoop.hive.serde.serdeConstants.SERIALIZATION_NULL_FORMAT, "NULL");
-// return tbl;
-// }
-//
-// public static boolean arrayWritableEquals(final ArrayWritable a1, final ArrayWritable a2) {
-// final Writable[] a1Arr = a1.get();
-// final Writable[] a2Arr = a2.get();
-//
-// if (a1Arr.length != a2Arr.length) {
-// return false;
-// }
-//
-// for (int i = 0; i < a1Arr.length; ++i) {
-// if (a1Arr[i] instanceof ArrayWritable) {
-// if (!(a2Arr[i] instanceof ArrayWritable)) {
-// return false;
-// }
-// if (!arrayWritableEquals((ArrayWritable) a1Arr[i], (ArrayWritable) a2Arr[i])) {
-// return false;
-// }
-// } else {
-// if (!a1Arr[i].equals(a2Arr[i])) {
-// return false;
-// }
-// }
-//
-// }
-// return true;
-// }
-//}
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.carbondata.hive;
+
+import junit.framework.TestCase;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.common.type.HiveDecimal;
+import org.apache.hadoop.hive.serde2.SerDeException;
+import org.apache.hadoop.hive.serde2.SerDeUtils;
+import org.apache.hadoop.hive.serde2.io.DoubleWritable;
+import org.apache.hadoop.hive.serde2.io.HiveDecimalWritable;
+import org.apache.hadoop.hive.serde2.io.ShortWritable;
+import org.apache.hadoop.hive.serde2.objectinspector.StructObjectInspector;
+import org.apache.hadoop.io.*;
+import org.junit.Test;
+
+import java.util.Properties;
+
+public class TestCarbonSerde extends TestCase {
+ @Test
+ public void testCarbonHiveSerDe() throws Throwable {
+ try {
+ // Create the SerDe
+ System.out.println("test: testCarbonHiveSerDe");
+
+ final CarbonHiveSerDe serDe = new CarbonHiveSerDe();
+ final Configuration conf = new Configuration();
+ final Properties tbl = createProperties();
+ SerDeUtils.initializeSerDe(serDe, conf, tbl, null);
+
+ // Data
+ final Writable[] arr = new Writable[7];
+
+ //primitive types
+ arr[0] = new ShortWritable((short) 456);
+ arr[1] = new IntWritable(789);
+ arr[2] = new LongWritable(1000l);
+ arr[3] = new DoubleWritable((double) 5.3);
+ arr[4] = new HiveDecimalWritable(HiveDecimal.create(1));
+ arr[5] = new Text("carbonSerde binary".getBytes("UTF-8"));
+
+ final Writable[] arrayContainer = new Writable[1];
+ final Writable[] array = new Writable[5];
+ for (int i = 0; i < 5; ++i) {
+ array[i] = new IntWritable(i);
+ }
+ arrayContainer[0] = new ArrayWritable(Writable.class, array);
+ arr[6] = new ArrayWritable(Writable.class, arrayContainer);
+
+ final ArrayWritable arrWritable = new ArrayWritable(Writable.class, arr);
+ // Test
+ deserializeAndSerializeLazySimple(serDe, arrWritable);
+ System.out.println("test: testCarbonHiveSerDe - OK");
+
+ } catch (final Throwable e) {
+ e.printStackTrace();
+ throw e;
+ }
+ }
+
+ private void deserializeAndSerializeLazySimple(final CarbonHiveSerDe serDe,
+ final ArrayWritable t) throws SerDeException {
+
+ // Get the row structure
+ final StructObjectInspector oi = (StructObjectInspector) serDe.getObjectInspector();
+
+ // Deserialize
+ final Object row = serDe.deserialize(t);
+ assertEquals("deserialization gives the wrong object class", row.getClass(),
+ ArrayWritable.class);
+ assertEquals("size correct after deserialization",
+ serDe.getSerDeStats().getRawDataSize(), t.get().length);
+ assertEquals("deserialization gives the wrong object", t, row);
+
+ // Serialize
+ final ArrayWritable serializedArr = (ArrayWritable) serDe.serialize(row, oi);
+ assertEquals("size correct after serialization", serDe.getSerDeStats().getRawDataSize(),
+ serializedArr.get().length);
+ assertTrue("serialized object should be equal to starting object",
+ arrayWritableEquals(t, serializedArr));
+ }
+
+ private Properties createProperties() {
+ final Properties tbl = new Properties();
+
+ // Set the configuration parameters
+ tbl.setProperty("columns", "ashort,aint,along,adouble,adecimal,astring,alist");
--- End diff --
Please take care the spell format, including the case sensitive， for example: aShort
---
[GitHub] carbondata pull request #2969: [CARBONDATA-3127]Fix the TestCarbonSerde exce...
Posted by SteNicholas <gi...@git.apache.org>.
Github user SteNicholas commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2969#discussion_r242815339
--- Diff: integration/hive/src/main/java/org/apache/carbondata/hive/MapredCarbonInputFormat.java ---
@@ -106,9 +107,17 @@ private static CarbonTable getCarbonTable(Configuration configuration, String pa
CarbonInputSplit split;
for (int i = 0; i < splitList.size(); i++) {
split = (CarbonInputSplit) splitList.get(i);
- splits[i] = new CarbonHiveInputSplit(split.getSegmentId(), split.getPath(), split.getStart(),
- split.getLength(), split.getLocations(), split.getNumberOfBlocklets(), split.getVersion(),
- split.getBlockStorageIdMap());
+ CarbonHiveInputSplit inputSplit = new CarbonHiveInputSplit(split.getSegmentId(),
+ split.getPath(), split.getStart(), split.getLength(),
+ split.getLocations(), split.getNumberOfBlocklets(),
+ split.getVersion(), split.getBlockStorageIdMap());
+ BlockletDetailInfo info = new BlockletDetailInfo();
+ info.setBlockSize(split.getLength());
+ info.setBlockFooterOffset(split.getDetailInfo().getBlockFooterOffset());
+ info.setVersionNumber(split.getVersion().number());
+ info.setUseMinMaxForPruning(false);
--- End diff --
There is no need to use min max for pruning.
---
[GitHub] carbondata issue #2969: [CARBONDATA-3127]Fix the TestCarbonSerde exception
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2969
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1836/
---
[GitHub] carbondata pull request #2969: [CARBONDATA-3127]Fix the TestCarbonSerde exce...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2969#discussion_r243150967
--- Diff: integration/hive/src/main/scala/org/apache/carbondata/hiveexample/HiveExample.scala ---
@@ -22,52 +22,49 @@ import java.sql.{DriverManager, ResultSet, Statement}
import org.apache.spark.sql.SparkSession
--- End diff --
Please move this example to example folder, I suggest to put all example in one module, so instead of adding another example module for hive, I think it is better add it in existing example module and rename it to carbondata-example from carbondata-example-spark2
---
[GitHub] carbondata pull request #2969: [CARBONDATA-3127]Fix the TestCarbonSerde exce...
Posted by SteNicholas <gi...@git.apache.org>.
Github user SteNicholas commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2969#discussion_r238089137
--- Diff: integration/hive/src/test/java/org/apache/carbondata/hive/TestCarbonSerde.java ---
@@ -1,133 +1,133 @@
-///*
-// * Licensed to the Apache Software Foundation (ASF) under one or more
-// * contributor license agreements. See the NOTICE file distributed with
-// * this work for additional information regarding copyright ownership.
-// * The ASF licenses this file to You under the Apache License, Version 2.0
-// * (the "License"); you may not use this file except in compliance with
-// * the License. You may obtain a copy of the License at
-// *
-// * http://www.apache.org/licenses/LICENSE-2.0
-// *
-// * Unless required by applicable law or agreed to in writing, software
-// * distributed under the License is distributed on an "AS IS" BASIS,
-// * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-// * See the License for the specific language governing permissions and
-// * limitations under the License.
-// */
-//package org.apache.carbondata.hive;
-//
-//import junit.framework.TestCase;
-//import org.apache.hadoop.conf.Configuration;
-//import org.apache.hadoop.hive.common.type.HiveDecimal;
-//import org.apache.hadoop.hive.serde2.SerDeException;
-//import org.apache.hadoop.hive.serde2.SerDeUtils;
-//import org.apache.hadoop.hive.serde2.io.DoubleWritable;
-//import org.apache.hadoop.hive.serde2.io.HiveDecimalWritable;
-//import org.apache.hadoop.hive.serde2.io.ShortWritable;
-//import org.apache.hadoop.hive.serde2.objectinspector.StructObjectInspector;
-//import org.apache.hadoop.io.*;
-//import org.junit.Test;
-//
-//import java.util.Properties;
-//
-//public class TestCarbonSerde extends TestCase {
-// @Test
-// public void testCarbonHiveSerDe() throws Throwable {
-// try {
-// // Create the SerDe
-// System.out.println("test: testCarbonHiveSerDe");
-//
-// final CarbonHiveSerDe serDe = new CarbonHiveSerDe();
-// final Configuration conf = new Configuration();
-// final Properties tbl = createProperties();
-// SerDeUtils.initializeSerDe(serDe, conf, tbl, null);
-//
-// // Data
-// final Writable[] arr = new Writable[7];
-//
-// //primitive types
-// arr[0] = new ShortWritable((short) 456);
-// arr[1] = new IntWritable(789);
-// arr[2] = new LongWritable(1000l);
-// arr[3] = new DoubleWritable((double) 5.3);
-// arr[4] = new HiveDecimalWritable(HiveDecimal.create(1));
-// arr[5] = new Text("carbonSerde binary".getBytes("UTF-8"));
-//
-// final Writable[] arrayContainer = new Writable[1];
-// final Writable[] array = new Writable[5];
-// for (int i = 0; i < 5; ++i) {
-// array[i] = new IntWritable(i);
-// }
-// arrayContainer[0] = new ArrayWritable(Writable.class, array);
-// arr[6] = new ArrayWritable(Writable.class, arrayContainer);
-//
-// final ArrayWritable arrWritable = new ArrayWritable(Writable.class, arr);
-// // Test
-// deserializeAndSerializeLazySimple(serDe, arrWritable);
-// System.out.println("test: testCarbonHiveSerDe - OK");
-//
-// } catch (final Throwable e) {
-// e.printStackTrace();
-// throw e;
-// }
-// }
-//
-// private void deserializeAndSerializeLazySimple(final CarbonHiveSerDe serDe,
-// final ArrayWritable t) throws SerDeException {
-//
-// // Get the row structure
-// final StructObjectInspector oi = (StructObjectInspector) serDe.getObjectInspector();
-//
-// // Deserialize
-// final Object row = serDe.deserialize(t);
-// assertEquals("deserialization gives the wrong object class", row.getClass(),
-// ArrayWritable.class);
-// assertEquals("size correct after deserialization",
-// serDe.getSerDeStats().getRawDataSize(), t.get().length);
-// assertEquals("deserialization gives the wrong object", t, row);
-//
-// // Serialize
-// final ArrayWritable serializedArr = (ArrayWritable) serDe.serializeStartKey(row, oi);
-// assertEquals("size correct after serialization", serDe.getSerDeStats().getRawDataSize(),
-// serializedArr.get().length);
-// assertTrue("serialized object should be equal to starting object",
-// arrayWritableEquals(t, serializedArr));
-// }
-//
-// private Properties createProperties() {
-// final Properties tbl = new Properties();
-//
-// // Set the configuration parameters
-// tbl.setProperty("columns", "ashort,aint,along,adouble,adecimal,astring,alist");
-// tbl.setProperty("columns.types",
-// "smallint:int:bigint:double:decimal:string:array<int>");
-// tbl.setProperty(org.apache.hadoop.hive.serde.serdeConstants.SERIALIZATION_NULL_FORMAT, "NULL");
-// return tbl;
-// }
-//
-// public static boolean arrayWritableEquals(final ArrayWritable a1, final ArrayWritable a2) {
-// final Writable[] a1Arr = a1.get();
-// final Writable[] a2Arr = a2.get();
-//
-// if (a1Arr.length != a2Arr.length) {
-// return false;
-// }
-//
-// for (int i = 0; i < a1Arr.length; ++i) {
-// if (a1Arr[i] instanceof ArrayWritable) {
-// if (!(a2Arr[i] instanceof ArrayWritable)) {
-// return false;
-// }
-// if (!arrayWritableEquals((ArrayWritable) a1Arr[i], (ArrayWritable) a2Arr[i])) {
-// return false;
-// }
-// } else {
-// if (!a1Arr[i].equals(a2Arr[i])) {
-// return false;
-// }
-// }
-//
-// }
-// return true;
-// }
-//}
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.carbondata.hive;
+
+import junit.framework.TestCase;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.common.type.HiveDecimal;
+import org.apache.hadoop.hive.serde2.SerDeException;
+import org.apache.hadoop.hive.serde2.SerDeUtils;
+import org.apache.hadoop.hive.serde2.io.DoubleWritable;
+import org.apache.hadoop.hive.serde2.io.HiveDecimalWritable;
+import org.apache.hadoop.hive.serde2.io.ShortWritable;
+import org.apache.hadoop.hive.serde2.objectinspector.StructObjectInspector;
+import org.apache.hadoop.io.*;
+import org.junit.Test;
+
+import java.util.Properties;
+
+public class TestCarbonSerde extends TestCase {
--- End diff --
@xubo245 I have already fixed the spell formate you referred.
---
[GitHub] carbondata pull request #2969: [CARBONDATA-3127]Fix the TestCarbonSerde exce...
Posted by SteNicholas <gi...@git.apache.org>.
Github user SteNicholas commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2969#discussion_r240113918
--- Diff: integration/hive/src/test/java/org/apache/carbondata/hive/TestCarbonSerDe.java ---
@@ -0,0 +1,133 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.carbondata.hive;
+
+import junit.framework.TestCase;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.common.type.HiveDecimal;
+import org.apache.hadoop.hive.serde2.SerDeException;
+import org.apache.hadoop.hive.serde2.SerDeUtils;
+import org.apache.hadoop.hive.serde2.io.DoubleWritable;
+import org.apache.hadoop.hive.serde2.io.HiveDecimalWritable;
+import org.apache.hadoop.hive.serde2.io.ShortWritable;
+import org.apache.hadoop.hive.serde2.objectinspector.StructObjectInspector;
+import org.apache.hadoop.io.*;
+import org.junit.Test;
+
+import java.util.Properties;
+
+public class TestCarbonSerDe extends TestCase {
+ @Test
+ public void testCarbonHiveSerDe() throws Throwable {
+ try {
+ // Create the SerDe
+ System.out.println("test: testCarbonHiveSerDe");
+
+ final CarbonHiveSerDe serDe = new CarbonHiveSerDe();
+ final Configuration conf = new Configuration();
+ final Properties tbl = createProperties();
+ SerDeUtils.initializeSerDe(serDe, conf, tbl, null);
+
+ // Data
+ final Writable[] arr = new Writable[7];
+
+ //primitive types
+ arr[0] = new ShortWritable((short) 456);
+ arr[1] = new IntWritable(789);
+ arr[2] = new LongWritable(1000l);
+ arr[3] = new DoubleWritable(5.3);
+ arr[4] = new HiveDecimalWritable(HiveDecimal.create(1));
+ arr[5] = new Text("CarbonSerDe Binary".getBytes("UTF-8"));
+
+ final Writable[] arrayContainer = new Writable[1];
+ final Writable[] array = new Writable[5];
+ for (int i = 0; i < 5; ++i) {
+ array[i] = new IntWritable(i);
+ }
+ arrayContainer[0] = new ArrayWritable(Writable.class, array);
+ arr[6] = new ArrayWritable(Writable.class, arrayContainer);
+
+ final ArrayWritable arrWritable = new ArrayWritable(Writable.class, arr);
+ // Test
+ deserializeAndSerializeLazySimple(serDe, arrWritable);
+ System.out.println("test: testCarbonHiveSerDe - OK");
+
+ } catch (final Throwable e) {
+ e.printStackTrace();
--- End diff --
I have already remove this catch-rethrow.
---
[GitHub] carbondata issue #2969: [CARBONDATA-3127]Fix the TestCarbonSerde exception
Posted by xuchuanyin <gi...@git.apache.org>.
Github user xuchuanyin commented on the issue:
https://github.com/apache/carbondata/pull/2969
@SteNicholas Nice to see your work on the existed problems. And it seems the previous code has some problem which is extend by your code. So I suggest you to fix them at the same time. Please check the above comments.
---
[GitHub] carbondata pull request #2969: [CARBONDATA-3127]Fix the TestCarbonSerde exce...
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2969#discussion_r238085516
--- Diff: integration/hive/src/test/java/org/apache/carbondata/hive/TestCarbonSerde.java ---
@@ -1,133 +1,133 @@
-///*
-// * Licensed to the Apache Software Foundation (ASF) under one or more
-// * contributor license agreements. See the NOTICE file distributed with
-// * this work for additional information regarding copyright ownership.
-// * The ASF licenses this file to You under the Apache License, Version 2.0
-// * (the "License"); you may not use this file except in compliance with
-// * the License. You may obtain a copy of the License at
-// *
-// * http://www.apache.org/licenses/LICENSE-2.0
-// *
-// * Unless required by applicable law or agreed to in writing, software
-// * distributed under the License is distributed on an "AS IS" BASIS,
-// * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-// * See the License for the specific language governing permissions and
-// * limitations under the License.
-// */
-//package org.apache.carbondata.hive;
-//
-//import junit.framework.TestCase;
-//import org.apache.hadoop.conf.Configuration;
-//import org.apache.hadoop.hive.common.type.HiveDecimal;
-//import org.apache.hadoop.hive.serde2.SerDeException;
-//import org.apache.hadoop.hive.serde2.SerDeUtils;
-//import org.apache.hadoop.hive.serde2.io.DoubleWritable;
-//import org.apache.hadoop.hive.serde2.io.HiveDecimalWritable;
-//import org.apache.hadoop.hive.serde2.io.ShortWritable;
-//import org.apache.hadoop.hive.serde2.objectinspector.StructObjectInspector;
-//import org.apache.hadoop.io.*;
-//import org.junit.Test;
-//
-//import java.util.Properties;
-//
-//public class TestCarbonSerde extends TestCase {
-// @Test
-// public void testCarbonHiveSerDe() throws Throwable {
-// try {
-// // Create the SerDe
-// System.out.println("test: testCarbonHiveSerDe");
-//
-// final CarbonHiveSerDe serDe = new CarbonHiveSerDe();
-// final Configuration conf = new Configuration();
-// final Properties tbl = createProperties();
-// SerDeUtils.initializeSerDe(serDe, conf, tbl, null);
-//
-// // Data
-// final Writable[] arr = new Writable[7];
-//
-// //primitive types
-// arr[0] = new ShortWritable((short) 456);
-// arr[1] = new IntWritable(789);
-// arr[2] = new LongWritable(1000l);
-// arr[3] = new DoubleWritable((double) 5.3);
-// arr[4] = new HiveDecimalWritable(HiveDecimal.create(1));
-// arr[5] = new Text("carbonSerde binary".getBytes("UTF-8"));
-//
-// final Writable[] arrayContainer = new Writable[1];
-// final Writable[] array = new Writable[5];
-// for (int i = 0; i < 5; ++i) {
-// array[i] = new IntWritable(i);
-// }
-// arrayContainer[0] = new ArrayWritable(Writable.class, array);
-// arr[6] = new ArrayWritable(Writable.class, arrayContainer);
-//
-// final ArrayWritable arrWritable = new ArrayWritable(Writable.class, arr);
-// // Test
-// deserializeAndSerializeLazySimple(serDe, arrWritable);
-// System.out.println("test: testCarbonHiveSerDe - OK");
-//
-// } catch (final Throwable e) {
-// e.printStackTrace();
-// throw e;
-// }
-// }
-//
-// private void deserializeAndSerializeLazySimple(final CarbonHiveSerDe serDe,
-// final ArrayWritable t) throws SerDeException {
-//
-// // Get the row structure
-// final StructObjectInspector oi = (StructObjectInspector) serDe.getObjectInspector();
-//
-// // Deserialize
-// final Object row = serDe.deserialize(t);
-// assertEquals("deserialization gives the wrong object class", row.getClass(),
-// ArrayWritable.class);
-// assertEquals("size correct after deserialization",
-// serDe.getSerDeStats().getRawDataSize(), t.get().length);
-// assertEquals("deserialization gives the wrong object", t, row);
-//
-// // Serialize
-// final ArrayWritable serializedArr = (ArrayWritable) serDe.serializeStartKey(row, oi);
-// assertEquals("size correct after serialization", serDe.getSerDeStats().getRawDataSize(),
-// serializedArr.get().length);
-// assertTrue("serialized object should be equal to starting object",
-// arrayWritableEquals(t, serializedArr));
-// }
-//
-// private Properties createProperties() {
-// final Properties tbl = new Properties();
-//
-// // Set the configuration parameters
-// tbl.setProperty("columns", "ashort,aint,along,adouble,adecimal,astring,alist");
-// tbl.setProperty("columns.types",
-// "smallint:int:bigint:double:decimal:string:array<int>");
-// tbl.setProperty(org.apache.hadoop.hive.serde.serdeConstants.SERIALIZATION_NULL_FORMAT, "NULL");
-// return tbl;
-// }
-//
-// public static boolean arrayWritableEquals(final ArrayWritable a1, final ArrayWritable a2) {
-// final Writable[] a1Arr = a1.get();
-// final Writable[] a2Arr = a2.get();
-//
-// if (a1Arr.length != a2Arr.length) {
-// return false;
-// }
-//
-// for (int i = 0; i < a1Arr.length; ++i) {
-// if (a1Arr[i] instanceof ArrayWritable) {
-// if (!(a2Arr[i] instanceof ArrayWritable)) {
-// return false;
-// }
-// if (!arrayWritableEquals((ArrayWritable) a1Arr[i], (ArrayWritable) a2Arr[i])) {
-// return false;
-// }
-// } else {
-// if (!a1Arr[i].equals(a2Arr[i])) {
-// return false;
-// }
-// }
-//
-// }
-// return true;
-// }
-//}
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.carbondata.hive;
+
+import junit.framework.TestCase;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.common.type.HiveDecimal;
+import org.apache.hadoop.hive.serde2.SerDeException;
+import org.apache.hadoop.hive.serde2.SerDeUtils;
+import org.apache.hadoop.hive.serde2.io.DoubleWritable;
+import org.apache.hadoop.hive.serde2.io.HiveDecimalWritable;
+import org.apache.hadoop.hive.serde2.io.ShortWritable;
+import org.apache.hadoop.hive.serde2.objectinspector.StructObjectInspector;
+import org.apache.hadoop.io.*;
+import org.junit.Test;
+
+import java.util.Properties;
+
+public class TestCarbonSerde extends TestCase {
--- End diff --
Please take care the spell format, including the case sensitive，for example：TestCarbonSerDe
Please check other code
---
[GitHub] carbondata issue #2969: [CARBONDATA-3127]Fix the TestCarbonSerde exception
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2969
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1835/
---
[GitHub] carbondata issue #2969: [CARBONDATA-3127]Fix the HiveExample & TestCarbonSer...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on the issue:
https://github.com/apache/carbondata/pull/2969
LGTM, can you squash your commits and rebase to master. I could not squash it
---
[GitHub] carbondata issue #2969: [CARBONDATA-3127]Fix the TestCarbonSerde exception
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2969
Build Success with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/9869/
---
[GitHub] carbondata issue #2969: [CARBONDATA-3127]Fix the TestCarbonSerde exception
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2969
Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/1884/
---
[GitHub] carbondata pull request #2969: [CARBONDATA-3127]Fix the TestCarbonSerde exce...
Posted by SteNicholas <gi...@git.apache.org>.
Github user SteNicholas commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2969#discussion_r242815829
--- Diff: integration/hive/src/main/scala/org/apache/carbondata/hiveexample/HiveExample.scala ---
@@ -31,7 +31,7 @@ object HiveExample {
def main(args: Array[String]) {
--- End diff --
I updated RunHiveExampleTest and RunExamples as same as RunExamples.But this doesn't work in CI.I guess that the problem is caused by CI.
---
[GitHub] carbondata pull request #2969: [CARBONDATA-3127]Fix the TestCarbonSerde exce...
Posted by xuchuanyin <gi...@git.apache.org>.
Github user xuchuanyin commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2969#discussion_r239070613
--- Diff: integration/hive/src/test/java/org/apache/carbondata/hive/TestCarbonSerDe.java ---
@@ -0,0 +1,133 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.carbondata.hive;
+
+import junit.framework.TestCase;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.common.type.HiveDecimal;
+import org.apache.hadoop.hive.serde2.SerDeException;
+import org.apache.hadoop.hive.serde2.SerDeUtils;
+import org.apache.hadoop.hive.serde2.io.DoubleWritable;
+import org.apache.hadoop.hive.serde2.io.HiveDecimalWritable;
+import org.apache.hadoop.hive.serde2.io.ShortWritable;
+import org.apache.hadoop.hive.serde2.objectinspector.StructObjectInspector;
+import org.apache.hadoop.io.*;
+import org.junit.Test;
+
+import java.util.Properties;
+
+public class TestCarbonSerDe extends TestCase {
+ @Test
+ public void testCarbonHiveSerDe() throws Throwable {
+ try {
+ // Create the SerDe
+ System.out.println("test: testCarbonHiveSerDe");
+
+ final CarbonHiveSerDe serDe = new CarbonHiveSerDe();
+ final Configuration conf = new Configuration();
+ final Properties tbl = createProperties();
+ SerDeUtils.initializeSerDe(serDe, conf, tbl, null);
+
+ // Data
+ final Writable[] arr = new Writable[7];
+
+ //primitive types
+ arr[0] = new ShortWritable((short) 456);
+ arr[1] = new IntWritable(789);
+ arr[2] = new LongWritable(1000l);
+ arr[3] = new DoubleWritable(5.3);
+ arr[4] = new HiveDecimalWritable(HiveDecimal.create(1));
+ arr[5] = new Text("CarbonSerDe Binary".getBytes("UTF-8"));
+
+ final Writable[] arrayContainer = new Writable[1];
+ final Writable[] array = new Writable[5];
+ for (int i = 0; i < 5; ++i) {
+ array[i] = new IntWritable(i);
+ }
+ arrayContainer[0] = new ArrayWritable(Writable.class, array);
+ arr[6] = new ArrayWritable(Writable.class, arrayContainer);
+
+ final ArrayWritable arrWritable = new ArrayWritable(Writable.class, arr);
+ // Test
+ deserializeAndSerializeLazySimple(serDe, arrWritable);
+ System.out.println("test: testCarbonHiveSerDe - OK");
+
+ } catch (final Throwable e) {
+ e.printStackTrace();
--- End diff --
use Logger instead of printing stacktarce in test code
---
[GitHub] carbondata issue #2969: [CARBONDATA-3127]Fix the TestCarbonSerde exception
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2969
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1852/
---
[GitHub] carbondata issue #2969: [CARBONDATA-3127]Fix the HiveExample & TestCarbonSer...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2969
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1886/
---
[GitHub] carbondata issue #2969: [CARBONDATA-3127]Fix the TestCarbonSerde exception
Posted by SteNicholas <gi...@git.apache.org>.
Github user SteNicholas commented on the issue:
https://github.com/apache/carbondata/pull/2969
@xubo245 @zzcclp @xuchuanyin I have already fixed hive integration bug based on HiveExample.Please review these updates.
---
[GitHub] carbondata pull request #2969: [CARBONDATA-3127]Fix the TestCarbonSerde exce...
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2969#discussion_r243139107
--- Diff: integration/hive/src/main/scala/org/apache/carbondata/hiveexample/HiveExample.scala ---
@@ -49,17 +49,19 @@ object HiveExample {
sparkSession.sql("""DROP TABLE IF EXISTS HIVE_CARBON_EXAMPLE""".stripMargin)
- sparkSession
- .sql(
- "CREATE TABLE HIVE_CARBON_EXAMPLE (ID int,NAME string,SALARY double) " +
- "STORED BY 'CARBONDATA' ")
+ sparkSession.sql(
+ s"""
+ | CREATE TABLE HIVE_CARBON_EXAMPLE
+ | (ID int,NAME string,SALARY double)
+ | STORED BY 'CARBONDATA'
--- End diff --
can you use lower case carbondata? like: STORED BY 'carbondata'
---
[GitHub] carbondata issue #2969: [CARBONDATA-3127]Fix the HiveExample & TestCarbonSer...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2969
Build Success with Spark 2.3.2, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/10142/
---
[GitHub] carbondata issue #2969: [CARBONDATA-3127]Fix the TestCarbonSerde exception
Posted by SteNicholas <gi...@git.apache.org>.
Github user SteNicholas commented on the issue:
https://github.com/apache/carbondata/pull/2969
@xubo245 @zzcclp @xuchuanyin I have already fixed hive integration bug based on HiveExample.Please review these updates.
---
[GitHub] carbondata issue #2969: [CARBONDATA-3127]Fix the TestCarbonSerde exception
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2969
Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/1821/
---
[GitHub] carbondata issue #2969: [CARBONDATA-3127]Fix the TestCarbonSerde exception
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2969
Build Failed with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/2044/
---
[GitHub] carbondata issue #2969: [CARBONDATA-3127]Fix the TestCarbonSerde exception
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2969
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1809/
---
[GitHub] carbondata issue #2969: [CARBONDATA-3127]Fix the TestCarbonSerde exception
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2969
Build Success with Spark 2.3.2, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/10107/
---
[GitHub] carbondata pull request #2969: [CARBONDATA-3127]Fix the TestCarbonSerde exce...
Posted by xuchuanyin <gi...@git.apache.org>.
Github user xuchuanyin commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2969#discussion_r239069684
--- Diff: integration/hive/src/test/java/org/apache/carbondata/hive/TestCarbonSerDe.java ---
@@ -0,0 +1,133 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.carbondata.hive;
+
+import junit.framework.TestCase;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.common.type.HiveDecimal;
+import org.apache.hadoop.hive.serde2.SerDeException;
+import org.apache.hadoop.hive.serde2.SerDeUtils;
+import org.apache.hadoop.hive.serde2.io.DoubleWritable;
+import org.apache.hadoop.hive.serde2.io.HiveDecimalWritable;
+import org.apache.hadoop.hive.serde2.io.ShortWritable;
+import org.apache.hadoop.hive.serde2.objectinspector.StructObjectInspector;
+import org.apache.hadoop.io.*;
+import org.junit.Test;
+
+import java.util.Properties;
+
+public class TestCarbonSerDe extends TestCase {
--- End diff --
Please refer to the other tests in carbon such as carbon-core module. Actually we do not need to extend TestCase here
---
[GitHub] carbondata issue #2969: [CARBONDATA-3127]Fix the TestCarbonSerde exception
Posted by SteNicholas <gi...@git.apache.org>.
Github user SteNicholas commented on the issue:
https://github.com/apache/carbondata/pull/2969
@xubo245 @zzcclp Please review the small update.
---
[GitHub] carbondata pull request #2969: [CARBONDATA-3127]Fix the TestCarbonSerde exce...
Posted by SteNicholas <gi...@git.apache.org>.
Github user SteNicholas commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2969#discussion_r243258830
--- Diff: integration/hive/src/main/scala/org/apache/carbondata/hiveexample/HiveExample.scala ---
@@ -49,17 +49,19 @@ object HiveExample {
sparkSession.sql("""DROP TABLE IF EXISTS HIVE_CARBON_EXAMPLE""".stripMargin)
- sparkSession
- .sql(
- "CREATE TABLE HIVE_CARBON_EXAMPLE (ID int,NAME string,SALARY double) " +
- "STORED BY 'CARBONDATA' ")
+ sparkSession.sql(
+ s"""
+ | CREATE TABLE HIVE_CARBON_EXAMPLE
+ | (ID int,NAME string,SALARY double)
+ | STORED BY 'CARBONDATA'
--- End diff --
@jackylk This suggestion doesn't work
@xubo245 I have already lower CARBONDATA
---
[GitHub] carbondata issue #2969: [CARBONDATA-3127]Fix the TestCarbonSerde exception
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2969
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1609/
---
[GitHub] carbondata pull request #2969: [CARBONDATA-3127]Fix the TestCarbonSerde exce...
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2969#discussion_r242391224
--- Diff: integration/hive/src/main/scala/org/apache/carbondata/hiveexample/HiveExample.scala ---
@@ -85,19 +82,25 @@ object HiveExample {
logger.info(s"============HIVE CLI IS STARTED ON PORT $port ==============")
- statement.execute("CREATE TABLE IF NOT EXISTS " + "HIVE_CARBON_EXAMPLE " +
- " (ID int, NAME string,SALARY double)")
+ statement.execute("DROP TABLE IF EXISTS HIVE_CARBON_EXAMPLE")
+
+ statement.execute("CREATE TABLE HIVE_CARBON_EXAMPLE " +
+ " (ID int, NAME string,SALARY double) " +
+ "ROW FORMAT SERDE 'org.apache.carbondata.hive.CarbonHiveSerDe' " +
+ "WITH SERDEPROPERTIES ('mapreduce.input.carboninputformat.databaseName'='default', " +
+ "'mapreduce.input.carboninputformat.tableName'='HIVE_CARBON_EXAMPLE')")
+
statement
.execute(
"ALTER TABLE HIVE_CARBON_EXAMPLE SET FILEFORMAT INPUTFORMAT \"org.apache.carbondata." +
- "hive.MapredCarbonInputFormat\"OUTPUTFORMAT \"org.apache.carbondata.hive." +
- "MapredCarbonOutputFormat\"SERDE \"org.apache.carbondata.hive." +
- "CarbonHiveSerDe\" ")
+ "hive.MapredCarbonInputFormat\"OUTPUTFORMAT \"org.apache.carbondata.hive." +
+ "MapredCarbonOutputFormat\"SERDE \"org.apache.carbondata.hive." +
--- End diff --
Can you add space before SERDE?
---
[GitHub] carbondata pull request #2969: [CARBONDATA-3127]Fix the TestCarbonSerde exce...
Posted by SteNicholas <gi...@git.apache.org>.
Github user SteNicholas commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2969#discussion_r240113830
--- Diff: integration/hive/src/test/java/org/apache/carbondata/hive/TestCarbonSerDe.java ---
@@ -0,0 +1,133 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.carbondata.hive;
+
+import junit.framework.TestCase;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.common.type.HiveDecimal;
+import org.apache.hadoop.hive.serde2.SerDeException;
+import org.apache.hadoop.hive.serde2.SerDeUtils;
+import org.apache.hadoop.hive.serde2.io.DoubleWritable;
+import org.apache.hadoop.hive.serde2.io.HiveDecimalWritable;
+import org.apache.hadoop.hive.serde2.io.ShortWritable;
+import org.apache.hadoop.hive.serde2.objectinspector.StructObjectInspector;
+import org.apache.hadoop.io.*;
+import org.junit.Test;
+
+import java.util.Properties;
+
+public class TestCarbonSerDe extends TestCase {
+ @Test
+ public void testCarbonHiveSerDe() throws Throwable {
+ try {
+ // Create the SerDe
+ System.out.println("test: testCarbonHiveSerDe");
--- End diff --
Yeah,I have already replace stdout with Logger as same as the way of other carbon test code.
---
[GitHub] carbondata pull request #2969: [CARBONDATA-3127]Fix the TestCarbonSerde exce...
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2969#discussion_r242817598
--- Diff: integration/hive/src/main/scala/org/apache/carbondata/hiveexample/HiveExample.scala ---
@@ -43,28 +43,25 @@ object HiveExample {
import org.apache.spark.sql.CarbonSession._
- val carbonSession = SparkSession
- .builder()
- .master("local")
- .appName("HiveExample")
- .config("carbonSession.sql.warehouse.dir", warehouse).enableHiveSupport()
- .getOrCreateCarbonSession(
- store, metaStore_Db)
+ val carbonSession = SparkSession.builder()
+ .master("local").appName("HiveExample")
--- End diff --
ok
---
[GitHub] carbondata pull request #2969: [CARBONDATA-3127]Fix the TestCarbonSerde exce...
Posted by SteNicholas <gi...@git.apache.org>.
Github user SteNicholas commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2969#discussion_r238936639
--- Diff: integration/hive/src/test/java/org/apache/carbondata/hive/TestCarbonSerDe.java ---
@@ -0,0 +1,133 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.carbondata.hive;
+
+import junit.framework.TestCase;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.common.type.HiveDecimal;
+import org.apache.hadoop.hive.serde2.SerDeException;
+import org.apache.hadoop.hive.serde2.SerDeUtils;
+import org.apache.hadoop.hive.serde2.io.DoubleWritable;
+import org.apache.hadoop.hive.serde2.io.HiveDecimalWritable;
+import org.apache.hadoop.hive.serde2.io.ShortWritable;
+import org.apache.hadoop.hive.serde2.objectinspector.StructObjectInspector;
+import org.apache.hadoop.io.*;
+import org.junit.Test;
+
+import java.util.Properties;
+
+public class TestCarbonSerDe extends TestCase {
+ @Test
+ public void testCarbonHiveSerDe() throws Throwable {
+ try {
+ // Create the SerDe
+ System.out.println("test: testCarbonHiveSerDe");
+
+ final CarbonHiveSerDe serDe = new CarbonHiveSerDe();
+ final Configuration conf = new Configuration();
+ final Properties tbl = createProperties();
+ SerDeUtils.initializeSerDe(serDe, conf, tbl, null);
+
+ // Data
+ final Writable[] arr = new Writable[7];
+
+ //primitive types
+ arr[0] = new ShortWritable((short) 456);
+ arr[1] = new IntWritable(789);
+ arr[2] = new LongWritable(1000l);
+ arr[3] = new DoubleWritable(5.3);
+ arr[4] = new HiveDecimalWritable(HiveDecimal.create(1));
+ arr[5] = new Text("carbonSerde binary".getBytes("UTF-8"));
--- End diff --
@xubo245 Sorry to forget string spell format,please review it.Sorry for the inconvenience.
---
[GitHub] carbondata issue #2969: [CARBONDATA-3127]Fix the TestCarbonSerde exception
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2969
Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/1851/
---
[GitHub] carbondata issue #2969: [CARBONDATA-3127]Fix the TestCarbonSerde exception
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2969
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1610/
---
[GitHub] carbondata pull request #2969: [CARBONDATA-3127]Fix the TestCarbonSerde exce...
Posted by SteNicholas <gi...@git.apache.org>.
Github user SteNicholas commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2969#discussion_r240113381
--- Diff: integration/hive/src/test/java/org/apache/carbondata/hive/TestCarbonSerDe.java ---
@@ -0,0 +1,133 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.carbondata.hive;
+
+import junit.framework.TestCase;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.common.type.HiveDecimal;
+import org.apache.hadoop.hive.serde2.SerDeException;
+import org.apache.hadoop.hive.serde2.SerDeUtils;
+import org.apache.hadoop.hive.serde2.io.DoubleWritable;
+import org.apache.hadoop.hive.serde2.io.HiveDecimalWritable;
+import org.apache.hadoop.hive.serde2.io.ShortWritable;
+import org.apache.hadoop.hive.serde2.objectinspector.StructObjectInspector;
+import org.apache.hadoop.io.*;
+import org.junit.Test;
+
+import java.util.Properties;
+
+public class TestCarbonSerDe extends TestCase {
--- End diff --
I have already remove "extend TestCase" here, and update asserts of method.
---
[GitHub] carbondata pull request #2969: [CARBONDATA-3127]Fix the TestCarbonSerde exce...
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2969#discussion_r242400169
--- Diff: integration/hive/src/main/java/org/apache/carbondata/hive/MapredCarbonInputFormat.java ---
@@ -106,9 +107,17 @@ private static CarbonTable getCarbonTable(Configuration configuration, String pa
CarbonInputSplit split;
for (int i = 0; i < splitList.size(); i++) {
split = (CarbonInputSplit) splitList.get(i);
- splits[i] = new CarbonHiveInputSplit(split.getSegmentId(), split.getPath(), split.getStart(),
- split.getLength(), split.getLocations(), split.getNumberOfBlocklets(), split.getVersion(),
- split.getBlockStorageIdMap());
+ CarbonHiveInputSplit inputSplit = new CarbonHiveInputSplit(split.getSegmentId(),
+ split.getPath(), split.getStart(), split.getLength(),
+ split.getLocations(), split.getNumberOfBlocklets(),
+ split.getVersion(), split.getBlockStorageIdMap());
+ BlockletDetailInfo info = new BlockletDetailInfo();
+ info.setBlockSize(split.getLength());
+ info.setBlockFooterOffset(split.getDetailInfo().getBlockFooterOffset());
+ info.setVersionNumber(split.getVersion().number());
+ info.setUseMinMaxForPruning(false);
--- End diff --
Why do you set false in here?
---
[GitHub] carbondata issue #2969: [CARBONDATA-3127]Fix the TestCarbonSerde exception
Posted by SteNicholas <gi...@git.apache.org>.
Github user SteNicholas commented on the issue:
https://github.com/apache/carbondata/pull/2969
@zzcclp Please review this update;
---
[GitHub] carbondata issue #2969: [CARBONDATA-3127]Fix the HiveExample & TestCarbonSer...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2969
Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/2094/
---