You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@carbondata.apache.org by jackylk <gi...@git.apache.org> on 2018/07/30 16:25:23 UTC
[GitHub] carbondata pull request #2589: [WIP][CARBONSTORE] add ParallelScanStore inte...
GitHub user jackylk opened a pull request:
https://github.com/apache/carbondata/pull/2589
[WIP][CARBONSTORE] add ParallelScanStore interface
Be sure to do all of the following checklist to help us incorporate
your contribution quickly and easily:
- [ ] Any interfaces changed?
- [ ] Any backward compatibility impacted?
- [ ] Document update required?
- [ ] Testing done
Please provide details on
- Whether new unit test cases have been added or why no new tests are required?
- How it is tested? Please attach test report.
- Is it a performance related change? Please attach the performance test report.
- Any additional information to help reviewers in testing this change.
- [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/jackylk/incubator-carbondata store-parallelscan
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/carbondata/pull/2589.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2589
----
commit 458d3268fddfcb298c30f6bfa34a397e02b6f604
Author: Jacky Li <ja...@...>
Date: 2018-07-30T16:23:19Z
add ParallelScanStore interface
----
---
[GitHub] carbondata issue #2589: [WIP][CARBONSTORE] Refactor CarbonStore API
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2589
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7782/
---
[GitHub] carbondata pull request #2589: [CARBONDATA-2825][CARBONDATA-2828] CarbonStor...
Posted by ajithme <gi...@git.apache.org>.
Github user ajithme commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2589#discussion_r207721786
--- Diff: store/core/src/main/java/org/apache/carbondata/store/impl/master/RegistryServiceImpl.java ---
@@ -20,14 +20,14 @@
import java.io.IOException;
import org.apache.carbondata.common.annotations.InterfaceAudience;
-import org.apache.carbondata.store.impl.rpc.RegistryService;
-import org.apache.carbondata.store.impl.rpc.model.RegisterWorkerRequest;
-import org.apache.carbondata.store.impl.rpc.model.RegisterWorkerResponse;
+import org.apache.carbondata.store.impl.service.RegistryService;
+import org.apache.carbondata.store.impl.service.model.RegisterWorkerRequest;
+import org.apache.carbondata.store.impl.service.model.RegisterWorkerResponse;
import org.apache.hadoop.ipc.ProtocolSignature;
@InterfaceAudience.Internal
-class RegistryServiceImpl implements RegistryService {
+class egistryServiceImpl implements RegistryService {
--- End diff --
typo, pls rename
---
[GitHub] carbondata issue #2589: [WIP][CARBONSTORE] Refactor CarbonStore API
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2589
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7784/
---
[GitHub] carbondata issue #2589: [WIP][CARBONSTORE] Refactor CarbonStore API
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2589
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6165/
---
[GitHub] carbondata pull request #2589: [CARBONDATA-2825][CARBONDATA-2828] CarbonStor...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk closed the pull request at:
https://github.com/apache/carbondata/pull/2589
---
[GitHub] carbondata pull request #2589: [WIP][CARBONSTORE] Refactor CarbonStore API
Posted by ajithme <gi...@git.apache.org>.
Github user ajithme commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2589#discussion_r207704316
--- Diff: hadoop/src/main/java/org/apache/carbondata/hadoop/CarbonInputSplit.java ---
@@ -252,9 +262,24 @@ public Segment getSegment() {
if (dataMapWriterPathExists) {
dataMapWritePath = in.readUTF();
}
+ boolean filePathExists = in.readBoolean();
+ if (filePathExists) {
+ filePath = in.readUTF();
+ } else {
+ filePath = super.getPath().toString();
+ }
}
@Override public void write(DataOutput out) throws IOException {
+ if (super.getPath() != null) {
+ super.write(out);
+ } else {
+ // see HADOOP-13519, after Java deserialization, super.filePath is
+ // null, so write our filePath instead
+ Text.writeString(out, filePath);
+ out.writeLong(getStart());
+ out.writeLong(getLength());
+ }
super.write(out);
--- End diff --
can delete this line else read will fail
---
[GitHub] carbondata issue #2589: [WIP][CARBONSTORE] Refactor CarbonStore API
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2589
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6089/
---
[GitHub] carbondata issue #2589: [WIP][CARBONSTORE] Refactor CarbonStore API
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2589
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6138/
---
[GitHub] carbondata pull request #2589: [WIP][CARBONSTORE] Refactor CarbonStore API
Posted by jackylk <gi...@git.apache.org>.
GitHub user jackylk reopened a pull request:
https://github.com/apache/carbondata/pull/2589
[WIP][CARBONSTORE] Refactor CarbonStore API
Interface of carbon prepared to integrate with Spark DatasourceV2
Be sure to do all of the following checklist to help us incorporate
your contribution quickly and easily:
- [ ] Any interfaces changed?
- [ ] Any backward compatibility impacted?
- [ ] Document update required?
- [ ] Testing done
Please provide details on
- Whether new unit test cases have been added or why no new tests are required?
- How it is tested? Please attach test report.
- Is it a performance related change? Please attach the performance test report.
- Any additional information to help reviewers in testing this change.
- [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/jackylk/incubator-carbondata store-parallelscan
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/carbondata/pull/2589.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2589
----
commit 6d08a5261037c762ea7b5297a22840c00689e890
Author: Jacky Li <ja...@...>
Date: 2018-07-31T18:16:26Z
modify CarbonStore API
commit 63e2c2cba4221f70c788325277b47b93bbc94de3
Author: Jacky Li <ja...@...>
Date: 2018-07-31T18:39:45Z
correct Horizon
----
---
[GitHub] carbondata pull request #2589: [WIP][CARBONSTORE] Refactor CarbonStore API
Posted by ajithme <gi...@git.apache.org>.
Github user ajithme commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2589#discussion_r207501460
--- Diff: store/sdk/src/main/java/org/apache/carbondata/sdk/store/descriptor/ScanDescriptor.java ---
@@ -15,23 +15,33 @@
* limitations under the License.
*/
-package org.apache.carbondata.store.api.descriptor;
+package org.apache.carbondata.sdk.store.descriptor;
+import java.io.DataInput;
+import java.io.DataOutput;
+import java.io.IOException;
import java.util.Objects;
+import org.apache.carbondata.common.annotations.InterfaceAudience;
+import org.apache.carbondata.common.annotations.InterfaceStability;
import org.apache.carbondata.core.scan.expression.Expression;
+import org.apache.carbondata.core.util.ObjectSerializationUtil;
-public class SelectDescriptor {
+import org.apache.hadoop.io.Writable;
+
+@InterfaceAudience.User
+@InterfaceStability.Evolving
+public class ScanDescriptor implements Writable {
private TableIdentifier table;
private String[] projection;
private Expression filter;
private long limit;
--- End diff --
Must be Long.MAX_VALUE
---
[GitHub] carbondata pull request #2589: [WIP][CARBONSTORE] Refactor CarbonStore API
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2589#discussion_r207699730
--- Diff: store/sdk/src/main/java/org/apache/carbondata/sdk/store/ScanUnit.java ---
@@ -15,26 +15,27 @@
* limitations under the License.
*/
-package org.apache.carbondata.store.impl.rpc;
+package org.apache.carbondata.sdk.store;
-import org.apache.carbondata.common.annotations.InterfaceAudience;
-import org.apache.carbondata.store.impl.rpc.model.BaseResponse;
-import org.apache.carbondata.store.impl.rpc.model.LoadDataRequest;
-import org.apache.carbondata.store.impl.rpc.model.QueryResponse;
-import org.apache.carbondata.store.impl.rpc.model.Scan;
-import org.apache.carbondata.store.impl.rpc.model.ShutdownRequest;
-import org.apache.carbondata.store.impl.rpc.model.ShutdownResponse;
-
-import org.apache.hadoop.ipc.VersionedProtocol;
-
-@InterfaceAudience.Internal
-public interface StoreService extends VersionedProtocol {
-
- long versionID = 1L;
+import java.io.Serializable;
- BaseResponse loadData(LoadDataRequest request);
-
- QueryResponse query(Scan scan);
+import org.apache.carbondata.common.annotations.InterfaceAudience;
+import org.apache.carbondata.common.annotations.InterfaceStability;
+import org.apache.carbondata.core.metadata.schema.table.Writable;
- ShutdownResponse shutdown(ShutdownRequest request);
+/**
+ * An unit for the scanner in Carbon Store
+ */
+@InterfaceAudience.User
+@InterfaceStability.Unstable
+public interface ScanUnit<T> extends Serializable, Writable {
--- End diff --
fixed
---
[GitHub] carbondata issue #2589: [WIP][CARBONSTORE] Refactor CarbonStore API
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2589
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6508/
---
[GitHub] carbondata issue #2589: [WIP][CARBONSTORE] add CTable interface in CarbonSto...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2589
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6397/
---
[GitHub] carbondata pull request #2589: [WIP][CARBONSTORE] Refactor CarbonStore API
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk closed the pull request at:
https://github.com/apache/carbondata/pull/2589
---
[GitHub] carbondata pull request #2589: [WIP][CARBONSTORE] Refactor CarbonStore API
Posted by ajithme <gi...@git.apache.org>.
Github user ajithme commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2589#discussion_r207699345
--- Diff: store/sdk/src/main/java/org/apache/carbondata/sdk/store/BlockScanUnit.java ---
@@ -0,0 +1,70 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.carbondata.sdk.store;
+
+import java.io.DataInput;
+import java.io.DataOutput;
+import java.io.IOException;
+
+import org.apache.carbondata.common.annotations.InterfaceAudience;
+import org.apache.carbondata.hadoop.CarbonInputSplit;
+
+/**
+ * It contains a block to scan, and a destination worker who should scan it
+ */
+@InterfaceAudience.Internal
+public class BlockScanUnit implements ScanUnit {
+
+ // the data block to scan
+ private CarbonInputSplit inputSplit;
+
+ // the worker who should scan this unit
+ private Schedulable schedulable;
--- End diff --
Add this in Writable interface else it will be null after deserialization
---
[GitHub] carbondata pull request #2589: [CARBONDATA-2825][CARBONDATA-2828] CarbonStor...
Posted by ajithme <gi...@git.apache.org>.
Github user ajithme commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2589#discussion_r207721977
--- Diff: store/core/src/main/java/org/apache/carbondata/store/impl/MetaOperation.java ---
@@ -145,26 +152,61 @@ private void createTable(TableInfo tableInfo, boolean ifNotExists) throws IOExce
}
}
- public void dropTable(TableIdentifier table) throws IOException {
- String tablePath = store.getTablePath(table.getTableName(), table.getDatabaseName());
+ public void dropTable(TableIdentifier table) throws CarbonException {
+ String tablePath = getTablePath(table.getTableName(), table.getDatabaseName());
cache.remove(tablePath);
- FileFactory.deleteFile(tablePath);
+ try {
+ FileFactory.deleteFile(tablePath);
+ } catch (IOException e) {
+ throw new CarbonException(e);
+ }
+ }
+
+ public TableInfo getTable(TableIdentifier table) throws CarbonException {
+ return getTable(table, storeConf);
}
- public CarbonTable getTable(TableIdentifier table) throws IOException {
- String tablePath = store.getTablePath(table.getTableName(), table.getDatabaseName());
+ public static TableInfo getTable(TableIdentifier table, StoreConf storeConf)
+ throws CarbonException {
+ String tablePath = getTablePath(table.getTableName(), table.getDatabaseName(), storeConf);
if (cache.containsKey(tablePath)) {
return cache.get(tablePath);
} else {
- org.apache.carbondata.format.TableInfo formatTableInfo =
- CarbonUtil.readSchemaFile(CarbonTablePath.getSchemaFilePath(tablePath));
+ org.apache.carbondata.format.TableInfo formatTableInfo = null;
+ try {
+ formatTableInfo = CarbonUtil.readSchemaFile(CarbonTablePath.getSchemaFilePath(tablePath));
+ } catch (IOException e) {
+ throw new CarbonException(e);
+ }
SchemaConverter schemaConverter = new ThriftWrapperSchemaConverterImpl();
TableInfo tableInfo = schemaConverter.fromExternalToWrapperTableInfo(
formatTableInfo, table.getDatabaseName(), table.getTableName(), tablePath);
tableInfo.setTablePath(tablePath);
- CarbonTable carbonTable = CarbonTable.buildFromTableInfo(tableInfo);
- cache.put(tablePath, carbonTable);
- return carbonTable;
+ cache.put(tablePath, tableInfo);
+ return tableInfo;
}
}
-}
\ No newline at end of file
+
+ public List<TableDescriptor> listTable() throws CarbonException {
+ throw new UnsupportedOperationException();
+ }
+
+ public TableDescriptor getDescriptor(TableIdentifier table) throws CarbonException {
+ throw new UnsupportedOperationException();
+ }
+
+ public void alterTable(TableIdentifier table, TableDescriptor newTable) throws CarbonException {
+ throw new UnsupportedOperationException();
+ }
+
+ public String getTablePath(String tableName, String databaseName) {
+ Objects.requireNonNull(tableName);
+ Objects.requireNonNull(databaseName);
+ return String.format("%s/%s", storeConf.storeLocation(), tableName);
+ }
+ public static String getTablePath(String tableName, String databaseName, StoreConf storeConf) {
+ Objects.requireNonNull(tableName);
+ Objects.requireNonNull(databaseName);
+ return String.format("%s/%s", storeConf.storeLocation(), tableName);
--- End diff --
must consider database name in path
---
[GitHub] carbondata issue #2589: [WIP][CARBONSTORE] Refactor CarbonStore API
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2589
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7749/
---
[GitHub] carbondata issue #2589: [WIP][CARBONSTORE] Refactor CarbonStore API
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2589
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6439/
---
[GitHub] carbondata issue #2589: [WIP][CARBONSTORE] Refactor CarbonStore API
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2589
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7781/
---
[GitHub] carbondata issue #2589: [WIP][CARBONSTORE] Refactor CarbonStore API
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2589
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7695/
---
[GitHub] carbondata issue #2589: [CARBONDATA-2825][CARBONDATA-2828] CarbonStore and I...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2589
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6168/
---
[GitHub] carbondata issue #2589: [WIP][CARBONSTORE] Refactor CarbonStore API
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2589
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6421/
---
[GitHub] carbondata issue #2589: [CARBONDATA-2825][CARBONDATA-2828] CarbonStore and I...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2589
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7787/
---
[GitHub] carbondata issue #2589: [WIP][CARBONSTORE] Refactor CarbonStore API
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2589
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6159/
---
[GitHub] carbondata issue #2589: [WIP][CARBONSTORE] Refactor CarbonStore API
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2589
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6142/
---
[GitHub] carbondata issue #2589: [WIP][CARBONSTORE] Refactor CarbonStore API
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2589
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6141/
---
[GitHub] carbondata issue #2589: [WIP][CARBONSTORE] Refactor CarbonStore API
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2589
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6166/
---
[GitHub] carbondata issue #2589: [WIP][CARBONSTORE] add ParallelScanStore interface
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2589
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7629/
---
[GitHub] carbondata issue #2589: [WIP][CARBONSTORE] add CTable interface in CarbonSto...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2589
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7665/
---
[GitHub] carbondata issue #2589: [WIP][CARBONSTORE] Refactor CarbonStore API
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2589
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7777/
---
[GitHub] carbondata pull request #2589: [WIP][CARBONSTORE] Refactor CarbonStore API
Posted by ajithme <gi...@git.apache.org>.
Github user ajithme commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2589#discussion_r207431252
--- Diff: store/sdk/src/main/java/org/apache/carbondata/sdk/store/service/StoreService.java ---
@@ -0,0 +1,53 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.carbondata.sdk.store.service;
+
+import java.util.List;
+
+import org.apache.carbondata.common.annotations.InterfaceAudience;
+import org.apache.carbondata.core.datastore.row.CarbonRow;
+import org.apache.carbondata.core.metadata.schema.table.CarbonTable;
+import org.apache.carbondata.sdk.store.descriptor.LoadDescriptor;
+import org.apache.carbondata.sdk.store.descriptor.ScanDescriptor;
+import org.apache.carbondata.sdk.store.descriptor.TableDescriptor;
+import org.apache.carbondata.sdk.store.descriptor.TableIdentifier;
+import org.apache.carbondata.sdk.store.exception.CarbonException;
+
+import org.apache.hadoop.ipc.VersionedProtocol;
+
+@InterfaceAudience.Internal
+public interface StoreService extends VersionedProtocol {
+ long versionID = 1L;
+
+ void createTable(TableDescriptor descriptor) throws CarbonException;
+
+ void dropTable(TableIdentifier table) throws CarbonException;
+
+ CarbonTable getTable(TableIdentifier table) throws CarbonException;
--- End diff --
hadoop RPC need response object to be a org.apache.hadoop.io.serializer.WritableSerialization
---
[GitHub] carbondata issue #2589: [WIP][CARBONSTORE] Refactor CarbonStore API
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2589
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6164/
---
[GitHub] carbondata pull request #2589: [WIP][CARBONSTORE] add CTable interface in Ca...
Posted by chenliang613 <gi...@git.apache.org>.
Github user chenliang613 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2589#discussion_r206458466
--- Diff: store/core/pom.xml ---
@@ -48,8 +48,8 @@
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<configuration>
- <source>1.7</source>
- <target>1.7</target>
+ <source>8</source>
--- End diff --
1.8 ?
---
[GitHub] carbondata issue #2589: [WIP][CARBONSTORE] Refactor CarbonStore API
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2589
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6088/
---
[GitHub] carbondata issue #2589: [WIP][CARBONSTORE] Refactor CarbonStore API
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2589
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6474/
---
[GitHub] carbondata issue #2589: [WIP][CARBONSTORE] Refactor CarbonStore API
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2589
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7689/
---
[GitHub] carbondata pull request #2589: [WIP][CARBONSTORE] Refactor CarbonStore API
Posted by ajithme <gi...@git.apache.org>.
Github user ajithme commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2589#discussion_r207431095
--- Diff: store/sdk/src/main/java/org/apache/carbondata/sdk/store/ScannerImpl.java ---
@@ -0,0 +1,122 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.carbondata.sdk.store;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Random;
+import java.util.stream.Collectors;
+
+import org.apache.carbondata.common.logging.LogService;
+import org.apache.carbondata.common.logging.LogServiceFactory;
+import org.apache.carbondata.core.datastore.row.CarbonRow;
+import org.apache.carbondata.core.metadata.schema.table.TableInfo;
+import org.apache.carbondata.core.scan.expression.Expression;
+import org.apache.carbondata.hadoop.CarbonInputSplit;
+import org.apache.carbondata.hadoop.CarbonMultiBlockSplit;
+import org.apache.carbondata.hadoop.api.CarbonInputFormat;
+import org.apache.carbondata.sdk.store.conf.StoreConf;
+import org.apache.carbondata.sdk.store.descriptor.ScanDescriptor;
+import org.apache.carbondata.sdk.store.descriptor.TableIdentifier;
+import org.apache.carbondata.sdk.store.exception.CarbonException;
+import org.apache.carbondata.sdk.store.service.DataService;
+import org.apache.carbondata.sdk.store.service.PruneService;
+import org.apache.carbondata.sdk.store.service.ServiceFactory;
+import org.apache.carbondata.sdk.store.service.model.PruneRequest;
+import org.apache.carbondata.sdk.store.service.model.PruneResponse;
+import org.apache.carbondata.sdk.store.service.model.ScanRequest;
+import org.apache.carbondata.sdk.store.service.model.ScanResponse;
+
+import org.apache.hadoop.conf.Configuration;
+
+class ScannerImpl implements Scanner {
+ private static final LogService LOGGER =
+ LogServiceFactory.getLogService(ScannerImpl.class.getCanonicalName());
+
+ private PruneService pruneService;
+ private TableInfo tableInfo;
+
+ ScannerImpl(StoreConf conf, TableInfo tableInfo) throws IOException {
+ this.pruneService = ServiceFactory.createPruneService(
+ conf.masterHost(), conf.registryServicePort());
--- End diff --
must be prune service port
---
[GitHub] carbondata issue #2589: [WIP][CARBONSTORE] Refactor CarbonStore API
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2589
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7783/
---
[GitHub] carbondata issue #2589: [CARBONDATA-2825][CARBONDATA-2828] CarbonStore and I...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2589
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6509/
---
[GitHub] carbondata issue #2589: [WIP][CARBONSTORE] Refactor CarbonStore API
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2589
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6507/
---
[GitHub] carbondata issue #2589: [WIP][CARBONSTORE] Refactor CarbonStore API
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2589
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6481/
---
[GitHub] carbondata issue #2589: [WIP][CARBONSTORE] Refactor CarbonStore API
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2589
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6416/
---
[GitHub] carbondata issue #2589: [CARBONDATA-2825][CARBONDATA-2828] CarbonStore and I...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2589
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6512/
---
[GitHub] carbondata pull request #2589: [WIP][CARBONSTORE] Refactor CarbonStore API
Posted by ajithme <gi...@git.apache.org>.
Github user ajithme commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2589#discussion_r207433215
--- Diff: store/sdk/src/main/java/org/apache/carbondata/sdk/store/ScannerImpl.java ---
@@ -0,0 +1,122 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.carbondata.sdk.store;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Random;
+import java.util.stream.Collectors;
+
+import org.apache.carbondata.common.logging.LogService;
+import org.apache.carbondata.common.logging.LogServiceFactory;
+import org.apache.carbondata.core.datastore.row.CarbonRow;
+import org.apache.carbondata.core.metadata.schema.table.TableInfo;
+import org.apache.carbondata.core.scan.expression.Expression;
+import org.apache.carbondata.hadoop.CarbonInputSplit;
+import org.apache.carbondata.hadoop.CarbonMultiBlockSplit;
+import org.apache.carbondata.hadoop.api.CarbonInputFormat;
+import org.apache.carbondata.sdk.store.conf.StoreConf;
+import org.apache.carbondata.sdk.store.descriptor.ScanDescriptor;
+import org.apache.carbondata.sdk.store.descriptor.TableIdentifier;
+import org.apache.carbondata.sdk.store.exception.CarbonException;
+import org.apache.carbondata.sdk.store.service.DataService;
+import org.apache.carbondata.sdk.store.service.PruneService;
+import org.apache.carbondata.sdk.store.service.ServiceFactory;
+import org.apache.carbondata.sdk.store.service.model.PruneRequest;
+import org.apache.carbondata.sdk.store.service.model.PruneResponse;
+import org.apache.carbondata.sdk.store.service.model.ScanRequest;
+import org.apache.carbondata.sdk.store.service.model.ScanResponse;
+
+import org.apache.hadoop.conf.Configuration;
+
+class ScannerImpl implements Scanner {
+ private static final LogService LOGGER =
+ LogServiceFactory.getLogService(ScannerImpl.class.getCanonicalName());
+
+ private PruneService pruneService;
+ private TableInfo tableInfo;
+
+ ScannerImpl(StoreConf conf, TableInfo tableInfo) throws IOException {
+ this.pruneService = ServiceFactory.createPruneService(
+ conf.masterHost(), conf.registryServicePort());
+ this.tableInfo = tableInfo;
+ }
+
+ /**
+ * Trigger a RPC to Carbon Master to do pruning
+ * @param table table identifier
+ * @param filterExpression expression of filter predicate given by user
+ * @return list of ScanUnit
+ * @throws CarbonException if any error occurs
+ */
+ @Override
+ public List<ScanUnit> prune(TableIdentifier table, Expression filterExpression)
+ throws CarbonException {
+ try {
+ Configuration configuration = new Configuration();
+ CarbonInputFormat.setTableName(configuration, table.getTableName());
--- End diff --
can use CarbonInputFormat.setTableInfo(configuration, tableInfo); else org.apache.carbondata.hadoop.api.CarbonInputFormat#getAbsoluteTableIdentifier will have empty path
---
[GitHub] carbondata issue #2589: [WIP][CARBONSTORE] Refactor CarbonStore API
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2589
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7690/
---
[GitHub] carbondata pull request #2589: [WIP][CARBONSTORE] Refactor CarbonStore API
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2589#discussion_r207699726
--- Diff: store/sdk/src/main/java/org/apache/carbondata/sdk/store/BlockScanUnit.java ---
@@ -0,0 +1,70 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.carbondata.sdk.store;
+
+import java.io.DataInput;
+import java.io.DataOutput;
+import java.io.IOException;
+
+import org.apache.carbondata.common.annotations.InterfaceAudience;
+import org.apache.carbondata.hadoop.CarbonInputSplit;
+
+/**
+ * It contains a block to scan, and a destination worker who should scan it
+ */
+@InterfaceAudience.Internal
+public class BlockScanUnit implements ScanUnit {
+
+ // the data block to scan
+ private CarbonInputSplit inputSplit;
+
+ // the worker who should scan this unit
+ private Schedulable schedulable;
--- End diff --
fixed
---
[GitHub] carbondata pull request #2589: [WIP][CARBONSTORE] Refactor CarbonStore API
Posted by ajithme <gi...@git.apache.org>.
Github user ajithme commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2589#discussion_r207699308
--- Diff: hadoop/src/main/java/org/apache/carbondata/hadoop/CarbonInputSplit.java ---
@@ -444,4 +444,16 @@ public void setFormat(FileFormat fileFormat) {
public Blocklet makeBlocklet() {
return new Blocklet(getPath().getName(), blockletId);
}
+
+ public String[] preferredLocations() {
--- End diff --
The super FileSplit.file is not serializable. Refer HADOOP-13519 so java serialization may return empty
---
[GitHub] carbondata issue #2589: [WIP][CARBONSTORE] Refactor CarbonStore API
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2589
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6506/
---
[GitHub] carbondata issue #2589: [WIP][CARBONSTORE] Refactor CarbonStore API
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2589
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6106/
---
[GitHub] carbondata pull request #2589: [WIP][CARBONSTORE] Refactor CarbonStore API
Posted by ajithme <gi...@git.apache.org>.
Github user ajithme commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2589#discussion_r207699358
--- Diff: store/sdk/src/main/java/org/apache/carbondata/sdk/store/ScanUnit.java ---
@@ -15,26 +15,27 @@
* limitations under the License.
*/
-package org.apache.carbondata.store.impl.rpc;
+package org.apache.carbondata.sdk.store;
-import org.apache.carbondata.common.annotations.InterfaceAudience;
-import org.apache.carbondata.store.impl.rpc.model.BaseResponse;
-import org.apache.carbondata.store.impl.rpc.model.LoadDataRequest;
-import org.apache.carbondata.store.impl.rpc.model.QueryResponse;
-import org.apache.carbondata.store.impl.rpc.model.Scan;
-import org.apache.carbondata.store.impl.rpc.model.ShutdownRequest;
-import org.apache.carbondata.store.impl.rpc.model.ShutdownResponse;
-
-import org.apache.hadoop.ipc.VersionedProtocol;
-
-@InterfaceAudience.Internal
-public interface StoreService extends VersionedProtocol {
-
- long versionID = 1L;
+import java.io.Serializable;
- BaseResponse loadData(LoadDataRequest request);
-
- QueryResponse query(Scan scan);
+import org.apache.carbondata.common.annotations.InterfaceAudience;
+import org.apache.carbondata.common.annotations.InterfaceStability;
+import org.apache.carbondata.core.metadata.schema.table.Writable;
- ShutdownResponse shutdown(ShutdownRequest request);
+/**
+ * An unit for the scanner in Carbon Store
+ */
+@InterfaceAudience.User
+@InterfaceStability.Unstable
+public interface ScanUnit<T> extends Serializable, Writable {
--- End diff --
can remove Generics
---
[GitHub] carbondata pull request #2589: [WIP][CARBONSTORE] Refactor CarbonStore API
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2589#discussion_r207699000
--- Diff: store/sdk/src/main/java/org/apache/carbondata/sdk/store/descriptor/ScanDescriptor.java ---
@@ -15,23 +15,33 @@
* limitations under the License.
*/
-package org.apache.carbondata.store.api.descriptor;
+package org.apache.carbondata.sdk.store.descriptor;
+import java.io.DataInput;
+import java.io.DataOutput;
+import java.io.IOException;
import java.util.Objects;
+import org.apache.carbondata.common.annotations.InterfaceAudience;
+import org.apache.carbondata.common.annotations.InterfaceStability;
import org.apache.carbondata.core.scan.expression.Expression;
+import org.apache.carbondata.core.util.ObjectSerializationUtil;
-public class SelectDescriptor {
+import org.apache.hadoop.io.Writable;
+
+@InterfaceAudience.User
+@InterfaceStability.Evolving
+public class ScanDescriptor implements Writable {
private TableIdentifier table;
private String[] projection;
private Expression filter;
private long limit;
--- End diff --
ok
---
[GitHub] carbondata pull request #2589: [WIP][CARBONSTORE] Refactor CarbonStore API
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2589#discussion_r207698994
--- Diff: store/sdk/src/main/java/org/apache/carbondata/sdk/store/ScannerImpl.java ---
@@ -0,0 +1,122 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.carbondata.sdk.store;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Random;
+import java.util.stream.Collectors;
+
+import org.apache.carbondata.common.logging.LogService;
+import org.apache.carbondata.common.logging.LogServiceFactory;
+import org.apache.carbondata.core.datastore.row.CarbonRow;
+import org.apache.carbondata.core.metadata.schema.table.TableInfo;
+import org.apache.carbondata.core.scan.expression.Expression;
+import org.apache.carbondata.hadoop.CarbonInputSplit;
+import org.apache.carbondata.hadoop.CarbonMultiBlockSplit;
+import org.apache.carbondata.hadoop.api.CarbonInputFormat;
+import org.apache.carbondata.sdk.store.conf.StoreConf;
+import org.apache.carbondata.sdk.store.descriptor.ScanDescriptor;
+import org.apache.carbondata.sdk.store.descriptor.TableIdentifier;
+import org.apache.carbondata.sdk.store.exception.CarbonException;
+import org.apache.carbondata.sdk.store.service.DataService;
+import org.apache.carbondata.sdk.store.service.PruneService;
+import org.apache.carbondata.sdk.store.service.ServiceFactory;
+import org.apache.carbondata.sdk.store.service.model.PruneRequest;
+import org.apache.carbondata.sdk.store.service.model.PruneResponse;
+import org.apache.carbondata.sdk.store.service.model.ScanRequest;
+import org.apache.carbondata.sdk.store.service.model.ScanResponse;
+
+import org.apache.hadoop.conf.Configuration;
+
+class ScannerImpl implements Scanner {
+ private static final LogService LOGGER =
+ LogServiceFactory.getLogService(ScannerImpl.class.getCanonicalName());
+
+ private PruneService pruneService;
+ private TableInfo tableInfo;
+
+ ScannerImpl(StoreConf conf, TableInfo tableInfo) throws IOException {
+ this.pruneService = ServiceFactory.createPruneService(
+ conf.masterHost(), conf.registryServicePort());
+ this.tableInfo = tableInfo;
+ }
+
+ /**
+ * Trigger a RPC to Carbon Master to do pruning
+ * @param table table identifier
+ * @param filterExpression expression of filter predicate given by user
+ * @return list of ScanUnit
+ * @throws CarbonException if any error occurs
+ */
+ @Override
+ public List<ScanUnit> prune(TableIdentifier table, Expression filterExpression)
+ throws CarbonException {
+ try {
+ Configuration configuration = new Configuration();
+ CarbonInputFormat.setTableName(configuration, table.getTableName());
--- End diff --
ok
---
[GitHub] carbondata issue #2589: [WIP][CARBONSTORE] Refactor CarbonStore API
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2589
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6501/
---
[GitHub] carbondata issue #2589: [WIP][CARBONSTORE] Refactor CarbonStore API
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2589
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7756/
---
[GitHub] carbondata issue #2589: [WIP][CARBONSTORE] Refactor CarbonStore API
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2589
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6480/
---
[GitHub] carbondata issue #2589: [WIP][CARBONSTORE] Refactor CarbonStore API
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on the issue:
https://github.com/apache/carbondata/pull/2589
CTable is removed, now I have added new API in CarbonStore interface only
---
[GitHub] carbondata issue #2589: [WIP][CARBONSTORE] Refactor CarbonStore API
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2589
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6479/
---
[GitHub] carbondata issue #2589: [WIP][CARBONSTORE] Refactor CarbonStore API
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2589
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7694/
---
[GitHub] carbondata issue #2589: [WIP][CARBONSTORE] add ParallelScanStore interface
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2589
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6347/
---
[GitHub] carbondata issue #2589: [WIP][CARBONSTORE] add CTable interface in CarbonSto...
Posted by chenliang613 <gi...@git.apache.org>.
Github user chenliang613 commented on the issue:
https://github.com/apache/carbondata/pull/2589
Can you explain "CTable" for what ?
---
[GitHub] carbondata pull request #2589: [WIP][CARBONSTORE] Refactor CarbonStore API
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2589#discussion_r207699719
--- Diff: hadoop/src/main/java/org/apache/carbondata/hadoop/CarbonInputSplit.java ---
@@ -444,4 +444,16 @@ public void setFormat(FileFormat fileFormat) {
public Blocklet makeBlocklet() {
return new Blocklet(getPath().getName(), blockletId);
}
+
+ public String[] preferredLocations() {
--- End diff --
fixed
---
[GitHub] carbondata issue #2589: [WIP][CARBONSTORE] Refactor CarbonStore API
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2589
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6140/
---
[GitHub] carbondata issue #2589: [CARBONDATA-2825][CARBONDATA-2828] CarbonStore and I...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2589
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6167/
---
[GitHub] carbondata issue #2589: [WIP][CARBONSTORE] Refactor CarbonStore API
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2589
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7755/
---
[GitHub] carbondata pull request #2589: [WIP][CARBONSTORE] add CTable interface in Ca...
Posted by ajithme <gi...@git.apache.org>.
Github user ajithme commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2589#discussion_r206492090
--- Diff: store/core/pom.xml ---
@@ -48,8 +48,8 @@
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<configuration>
- <source>1.7</source>
- <target>1.7</target>
+ <source>8</source>
--- End diff --
8 is same as 1.8, as this is passed to javac
https://docs.oracle.com/javase/8/docs/technotes/tools/windows/javac.html
---
[GitHub] carbondata issue #2589: [WIP][CARBONSTORE] Refactor CarbonStore API
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2589
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7713/
---
[GitHub] carbondata issue #2589: [WIP][CARBONSTORE] Refactor CarbonStore API
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2589
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7754/
---