You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@carbondata.apache.org by QiangCai <gi...@git.apache.org> on 2017/12/10 13:29:08 UTC
[GitHub] carbondata pull request #1638: [CARBONDATA-1879] Support alter table to chan...
GitHub user QiangCai opened a pull request:
https://github.com/apache/carbondata/pull/1638
[CARBONDATA-1879] Support alter table to change the status of the streaming segment
Support new SQL command to change the status of the segment from "streaming" to "streaming finish":
Alter table <dbname.tablename> finish streaming
1. If streaming is running, the command will throw an exception.
2. If there isn't streaming segment, do nothing.
- [x] Any interfaces changed?
add new sql command:
Alter table <dbname.tablename> finish streaming
- [x] Any backward compatibility impacted?
no
- [x] Document update required?
required
- [x] Testing done
Please provide details on
- Whether new unit test cases have been added or why no new tests are required?
added
- How it is tested? Please attach test report.
- Is it a performance related change? Please attach the performance test report.
- Any additional information to help reviewers in testing this change.
- [x] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
NA
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/QiangCai/carbondata finish_streaming
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/carbondata/pull/1638.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1638
----
commit fc5318348dea70f57e967a9a38a8cd25c139b7f9
Author: QiangCai <qi...@qq.com>
Date: 2017-12-10T13:01:56Z
alter table finish streaming
----
---
[GitHub] carbondata pull request #1638: [CARBONDATA-1879][Streaming] Support alter ta...
Posted by QiangCai <gi...@git.apache.org>.
Github user QiangCai commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1638#discussion_r156834294
--- Diff: streaming/src/main/java/org/apache/carbondata/streaming/segment/StreamSegment.java ---
@@ -180,6 +182,70 @@ public static String close(CarbonTable table, String segmentId)
}
}
+ /**
+ * change the status of the segment from "streaming" to "streaming finish"
+ */
+ public static void finishStreaming(CarbonTable carbonTable) throws Exception {
+ ICarbonLock lock = CarbonLockFactory.getCarbonLockObj(
+ carbonTable.getTableInfo().getOrCreateAbsoluteTableIdentifier(),
+ LockUsage.TABLE_STATUS_LOCK);
+ try {
+ if (lock.lockWithRetries()) {
+ ICarbonLock streamingLock = CarbonLockFactory.getCarbonLockObj(
+ carbonTable.getTableInfo().getOrCreateAbsoluteTableIdentifier(),
+ LockUsage.STREAMING_LOCK);
+ try {
+ if (streamingLock.lockWithRetries()) {
+ LoadMetadataDetails[] details =
+ SegmentStatusManager.readLoadMetadata(carbonTable.getMetaDataFilepath());
+ boolean updated = false;
+ for (LoadMetadataDetails detail : details) {
+ if (SegmentStatus.STREAMING == detail.getSegmentStatus()) {
+ detail.setLoadEndTime(System.currentTimeMillis());
+ detail.setSegmentStatus(SegmentStatus.STREAMING_FINISH);
+ updated = true;
+ }
+ }
+ if (updated) {
+ CarbonTablePath tablePath =
+ CarbonStorePath.getCarbonTablePath(carbonTable.getAbsoluteTableIdentifier());
+ SegmentStatusManager.writeLoadDetailsIntoFile(
+ tablePath.getTableStatusFilePath(), details);
+ }
+ } else {
+ String msg = "Failed to finish streaming, because streaming is locked for table " +
+ carbonTable.getDatabaseName() + "." + carbonTable.getTableName();
+ LOGGER.error(msg);
+ throw new Exception(msg);
+ }
+ } finally {
+ if (streamingLock.unlock()) {
+ LOGGER.info("Table unlocked successfully after streaming finished" + carbonTable
+ .getDatabaseName() + "." + carbonTable.getTableName());
+ } else {
+ LOGGER.error("Unable to unlock Table lock for table " +
+ carbonTable.getDatabaseName() + "." + carbonTable.getTableName() +
+ " during streaming finished");
+ }
+ }
+ } else {
+ String msg = "Failed to acquire table status lock of " +
+ carbonTable.getDatabaseName() + "." + carbonTable.getTableName();
+ LOGGER.error(msg);
+ throw new Exception(msg);
--- End diff --
fixed
---
[GitHub] carbondata issue #1638: [CARBONDATA-1879][Streaming] Support alter table to ...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on the issue:
https://github.com/apache/carbondata/pull/1638
retest this please
---
[GitHub] carbondata issue #1638: [CARBONDATA-1879][Streaming] Support alter table to ...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1638
Build Success with Spark 2.2.0, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/677/
---
[GitHub] carbondata issue #1638: [CARBONDATA-1879][Streaming] Support alter table to ...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1638
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/1906/
---
[GitHub] carbondata issue #1638: [CARBONDATA-1879][Streaming] Support alter table to ...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1638
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2275/
---
[GitHub] carbondata pull request #1638: [CARBONDATA-1879][Streaming] Support alter ta...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1638#discussion_r156599142
--- Diff: streaming/src/main/java/org/apache/carbondata/streaming/segment/StreamSegment.java ---
@@ -180,6 +182,70 @@ public static String close(CarbonTable table, String segmentId)
}
}
+ /**
+ * change the status of the segment from "streaming" to "streaming finish"
+ */
+ public static void finishStreaming(CarbonTable carbonTable) throws Exception {
+ ICarbonLock lock = CarbonLockFactory.getCarbonLockObj(
+ carbonTable.getTableInfo().getOrCreateAbsoluteTableIdentifier(),
+ LockUsage.TABLE_STATUS_LOCK);
+ try {
+ if (lock.lockWithRetries()) {
+ ICarbonLock streamingLock = CarbonLockFactory.getCarbonLockObj(
+ carbonTable.getTableInfo().getOrCreateAbsoluteTableIdentifier(),
+ LockUsage.STREAMING_LOCK);
--- End diff --
This lock should be acquired first because it is acquired firstly in handoff flow
---
[GitHub] carbondata pull request #1638: [CARBONDATA-1879][Streaming] Support alter ta...
Posted by QiangCai <gi...@git.apache.org>.
Github user QiangCai commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1638#discussion_r156834271
--- Diff: streaming/src/main/java/org/apache/carbondata/streaming/segment/StreamSegment.java ---
@@ -180,6 +182,70 @@ public static String close(CarbonTable table, String segmentId)
}
}
+ /**
+ * change the status of the segment from "streaming" to "streaming finish"
+ */
+ public static void finishStreaming(CarbonTable carbonTable) throws Exception {
+ ICarbonLock lock = CarbonLockFactory.getCarbonLockObj(
+ carbonTable.getTableInfo().getOrCreateAbsoluteTableIdentifier(),
+ LockUsage.TABLE_STATUS_LOCK);
+ try {
+ if (lock.lockWithRetries()) {
+ ICarbonLock streamingLock = CarbonLockFactory.getCarbonLockObj(
+ carbonTable.getTableInfo().getOrCreateAbsoluteTableIdentifier(),
+ LockUsage.STREAMING_LOCK);
--- End diff --
fixed
---
[GitHub] carbondata issue #1638: [CARBONDATA-1879][Streaming] Support alter table to ...
Posted by QiangCai <gi...@git.apache.org>.
Github user QiangCai commented on the issue:
https://github.com/apache/carbondata/pull/1638
retest this please
---
[GitHub] carbondata issue #1638: [CARBONDATA-1879][Streaming] Support alter table to ...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1638
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/1951/
---
[GitHub] carbondata pull request #1638: [CARBONDATA-1879][Streaming] Support alter ta...
Posted by QiangCai <gi...@git.apache.org>.
Github user QiangCai commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1638#discussion_r156832490
--- Diff: integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/management/CarbonAlterTableFinishStreaming.scala ---
@@ -0,0 +1,37 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command.management
+
+import org.apache.spark.sql.{CarbonEnv, Row, SparkSession}
+import org.apache.spark.sql.execution.command.DataCommand
+
+import org.apache.carbondata.streaming.segment.StreamSegment
+
+/**
+ * This command will try to change the status of the segment from "streaming" to "streaming finish"
+ */
+case class CarbonAlterTableFinishStreaming(
+ dbName: Option[String],
+ tableName: String)
+ extends DataCommand {
--- End diff --
fixed
---
[GitHub] carbondata pull request #1638: [CARBONDATA-1879][Streaming] Support alter ta...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/carbondata/pull/1638
---
[GitHub] carbondata issue #1638: [CARBONDATA-1879] Support alter table to change the ...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1638
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/1854/
---
[GitHub] carbondata pull request #1638: [CARBONDATA-1879][Streaming] Support alter ta...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1638#discussion_r156596322
--- Diff: integration/spark2/src/main/scala/org/apache/spark/sql/parser/CarbonSpark2SqlParser.scala ---
@@ -129,6 +129,12 @@ class CarbonSpark2SqlParser extends CarbonDDLSqlParser {
CarbonAlterTableCompactionCommand(altertablemodel)
}
+ protected lazy val alterTableFinishStreaming: Parser[LogicalPlan] =
--- End diff --
add syntax description comment
---
[GitHub] carbondata issue #1638: [CARBONDATA-1879][Streaming] Support alter table to ...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1638
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2232/
---
[GitHub] carbondata pull request #1638: [CARBONDATA-1879][Streaming] Support alter ta...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1638#discussion_r156597691
--- Diff: streaming/src/main/java/org/apache/carbondata/streaming/segment/StreamSegment.java ---
@@ -180,6 +182,70 @@ public static String close(CarbonTable table, String segmentId)
}
}
+ /**
+ * change the status of the segment from "streaming" to "streaming finish"
+ */
+ public static void finishStreaming(CarbonTable carbonTable) throws Exception {
+ ICarbonLock lock = CarbonLockFactory.getCarbonLockObj(
+ carbonTable.getTableInfo().getOrCreateAbsoluteTableIdentifier(),
+ LockUsage.TABLE_STATUS_LOCK);
+ try {
+ if (lock.lockWithRetries()) {
+ ICarbonLock streamingLock = CarbonLockFactory.getCarbonLockObj(
+ carbonTable.getTableInfo().getOrCreateAbsoluteTableIdentifier(),
+ LockUsage.STREAMING_LOCK);
+ try {
+ if (streamingLock.lockWithRetries()) {
+ LoadMetadataDetails[] details =
+ SegmentStatusManager.readLoadMetadata(carbonTable.getMetaDataFilepath());
+ boolean updated = false;
+ for (LoadMetadataDetails detail : details) {
+ if (SegmentStatus.STREAMING == detail.getSegmentStatus()) {
+ detail.setLoadEndTime(System.currentTimeMillis());
+ detail.setSegmentStatus(SegmentStatus.STREAMING_FINISH);
+ updated = true;
+ }
+ }
+ if (updated) {
+ CarbonTablePath tablePath =
+ CarbonStorePath.getCarbonTablePath(carbonTable.getAbsoluteTableIdentifier());
+ SegmentStatusManager.writeLoadDetailsIntoFile(
+ tablePath.getTableStatusFilePath(), details);
+ }
+ } else {
+ String msg = "Failed to finish streaming, because streaming is locked for table " +
+ carbonTable.getDatabaseName() + "." + carbonTable.getTableName();
+ LOGGER.error(msg);
+ throw new Exception(msg);
+ }
+ } finally {
+ if (streamingLock.unlock()) {
+ LOGGER.info("Table unlocked successfully after streaming finished" + carbonTable
+ .getDatabaseName() + "." + carbonTable.getTableName());
+ } else {
+ LOGGER.error("Unable to unlock Table lock for table " +
+ carbonTable.getDatabaseName() + "." + carbonTable.getTableName() +
+ " during streaming finished");
+ }
+ }
+ } else {
+ String msg = "Failed to acquire table status lock of " +
+ carbonTable.getDatabaseName() + "." + carbonTable.getTableName();
+ LOGGER.error(msg);
+ throw new Exception(msg);
--- End diff --
change to IOException
---
[GitHub] carbondata issue #1638: [CARBONDATA-1879][Streaming] Support alter table to ...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1638
Build Success with Spark 2.2.0, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/723/
---
[GitHub] carbondata issue #1638: [CARBONDATA-1879][Streaming] Support alter table to ...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on the issue:
https://github.com/apache/carbondata/pull/1638
LGTM
---
[GitHub] carbondata pull request #1638: [CARBONDATA-1879][Streaming] Support alter ta...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1638#discussion_r156597163
--- Diff: integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/management/CarbonAlterTableFinishStreaming.scala ---
@@ -0,0 +1,37 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command.management
+
+import org.apache.spark.sql.{CarbonEnv, Row, SparkSession}
+import org.apache.spark.sql.execution.command.DataCommand
+
+import org.apache.carbondata.streaming.segment.StreamSegment
+
+/**
+ * This command will try to change the status of the segment from "streaming" to "streaming finish"
+ */
+case class CarbonAlterTableFinishStreaming(
+ dbName: Option[String],
+ tableName: String)
+ extends DataCommand {
--- End diff --
This should be metadata command
---
[GitHub] carbondata pull request #1638: [CARBONDATA-1879][Streaming] Support alter ta...
Posted by QiangCai <gi...@git.apache.org>.
Github user QiangCai commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1638#discussion_r156832564
--- Diff: integration/spark2/src/main/scala/org/apache/spark/sql/parser/CarbonSpark2SqlParser.scala ---
@@ -129,6 +129,12 @@ class CarbonSpark2SqlParser extends CarbonDDLSqlParser {
CarbonAlterTableCompactionCommand(altertablemodel)
}
+ protected lazy val alterTableFinishStreaming: Parser[LogicalPlan] =
--- End diff --
fixed
---
[GitHub] carbondata issue #1638: [CARBONDATA-1879][Streaming] Support alter table to ...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1638
Build Success with Spark 2.2.0, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/671/
---
[GitHub] carbondata issue #1638: [CARBONDATA-1879][Streaming] Support alter table to ...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1638
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2206/
---
[GitHub] carbondata issue #1638: [CARBONDATA-1879] Support alter table to change the ...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1638
Build Success with Spark 2.2.0, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/626/
---
[GitHub] carbondata issue #1638: [CARBONDATA-1879][Streaming] Support alter table to ...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1638
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/1893/
---
[GitHub] carbondata issue #1638: [CARBONDATA-1879][Streaming] Support alter table to ...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1638
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/1889/
---
[GitHub] carbondata issue #1638: [CARBONDATA-1879][Streaming] Support alter table to ...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1638
Build Success with Spark 2.2.0, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/658/
---
[GitHub] carbondata issue #1638: [CARBONDATA-1879][Streaming] Support alter table to ...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1638
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/1900/
---
[GitHub] carbondata issue #1638: [CARBONDATA-1879][Streaming] Support alter table to ...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1638
Build Failed with Spark 2.2.0, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/664/
---
[GitHub] carbondata pull request #1638: [CARBONDATA-1879][Streaming] Support alter ta...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1638#discussion_r156596190
--- Diff: integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/management/CarbonAlterTableFinishStreaming.scala ---
@@ -0,0 +1,37 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command.management
+
+import org.apache.spark.sql.{CarbonEnv, Row, SparkSession}
+import org.apache.spark.sql.execution.command.DataCommand
+
+import org.apache.carbondata.streaming.segment.StreamSegment
+
+/**
+ * This command will try to change the status of the segment from "streaming" to "streaming finish"
+ */
+case class CarbonAlterTableFinishStreaming(
+ dbName: Option[String],
+ tableName: String)
+ extends DataCommand {
--- End diff --
This is not data command
---