You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@carbondata.apache.org by GitBox <gi...@apache.org> on 2021/02/03 15:10:50 UTC
[GitHub] [carbondata] Indhumathi27 opened a new pull request #4087: [WIP] SI compatability issue fix
Indhumathi27 opened a new pull request #4087:
URL: https://github.com/apache/carbondata/pull/4087
### Why is this PR needed?
### What changes were proposed in this PR?
### Does this PR introduce any user interface change?
- No
- Yes. (please explain the change and update document)
### Is any new testcase added?
- No
- Yes
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] akashrn5 commented on a change in pull request #4087: [CARBONDATA-4125] SI compatability issue fix
Posted by GitBox <gi...@apache.org>.
akashrn5 commented on a change in pull request #4087:
URL: https://github.com/apache/carbondata/pull/4087#discussion_r576096839
##########
File path: integration/spark/src/main/scala/org/apache/spark/sql/secondaryindex/command/SICreationCommand.scala
##########
@@ -278,6 +303,31 @@ private[sql] case class CarbonCreateSecondaryIndexCommand(
carbonTable, databaseName,
tableName, indexTableName, absoluteTableIdentifier)
}
+ if (isRegisterIndex && null != tableInfo.getFactTable.getSchemaEvolution &&
+ null != carbonTable.getTableInfo.getFactTable.getSchemaEvolution) {
+ // check if SI table has undergone any alter schema operation before registering it
+ val schemaEvolutionEntryList = tableInfo
+ .getFactTable
+ .getSchemaEvolution
+ .getSchemaEvolutionEntryList
+ val mainTableSchemaEvolutionEntryList = carbonTable
+ .getTableInfo
+ .getFactTable
+ .getSchemaEvolution
+ .getSchemaEvolutionEntryList
+ if (mainTableSchemaEvolutionEntryList.size() != schemaEvolutionEntryList.size()) {
+ val index = mainTableSchemaEvolutionEntryList.size()
+ for (i <- index until schemaEvolutionEntryList.size()) {
Review comment:
use foreach
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] akashrn5 commented on a change in pull request #4087: [CARBONDATA-4125] SI compatability issue fix
Posted by GitBox <gi...@apache.org>.
akashrn5 commented on a change in pull request #4087:
URL: https://github.com/apache/carbondata/pull/4087#discussion_r576141512
##########
File path: integration/spark/src/main/scala/org/apache/spark/sql/secondaryindex/command/SICreationCommand.scala
##########
@@ -278,6 +303,31 @@ private[sql] case class CarbonCreateSecondaryIndexCommand(
carbonTable, databaseName,
tableName, indexTableName, absoluteTableIdentifier)
}
+ if (isRegisterIndex && null != tableInfo.getFactTable.getSchemaEvolution &&
+ null != carbonTable.getTableInfo.getFactTable.getSchemaEvolution) {
+ // check if SI table has undergone any alter schema operation before registering it
+ val schemaEvolutionEntryList = tableInfo
+ .getFactTable
+ .getSchemaEvolution
+ .getSchemaEvolutionEntryList
+ val mainTableSchemaEvolutionEntryList = carbonTable
+ .getTableInfo
+ .getFactTable
+ .getSchemaEvolution
+ .getSchemaEvolutionEntryList
+ if (mainTableSchemaEvolutionEntryList.size() != schemaEvolutionEntryList.size()) {
+ val index = mainTableSchemaEvolutionEntryList.size()
+ for (i <- index until schemaEvolutionEntryList.size()) {
+ val schemaEntry = schemaEvolutionEntryList.get(i)
+ val isRenamed = (schemaEntry.getAdded == null && schemaEntry.getRemoved == null) ||
Review comment:
```suggestion
val isSITableRenamed = (schemaEntry.getAdded == null && schemaEntry.getRemoved == null) ||
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4087: [WIP] SI compatability issue fix
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4087:
URL: https://github.com/apache/carbondata/pull/4087#issuecomment-773286185
Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12444/job/ApacheCarbonPRBuilder2.3/5433/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4087: [WIP] SI compatability issue fix
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4087:
URL: https://github.com/apache/carbondata/pull/4087#issuecomment-773286315
Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12444/job/ApacheCarbon_PR_Builder_2.4.5/3673/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] asfgit closed pull request #4087: [CARBONDATA-4125] SI compatability issue fix
Posted by GitBox <gi...@apache.org>.
asfgit closed pull request #4087:
URL: https://github.com/apache/carbondata/pull/4087
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4087: [CARBONDATA-4125] SI compatability issue fix
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4087:
URL: https://github.com/apache/carbondata/pull/4087#issuecomment-779233932
Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12444/job/ApacheCarbon_PR_Builder_2.4.5/3705/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4087: [CARBONDATA-4125] SI compatability issue fix
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4087:
URL: https://github.com/apache/carbondata/pull/4087#issuecomment-780345475
Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12444/job/ApacheCarbonPRBuilder2.3/5475/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4087: [CARBONDATA-4125] SI compatability issue fix
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4087:
URL: https://github.com/apache/carbondata/pull/4087#issuecomment-778253000
Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12444/job/ApacheCarbon_PR_Builder_2.4.5/3700/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] akashrn5 commented on a change in pull request #4087: [CARBONDATA-4125] SI compatability issue fix
Posted by GitBox <gi...@apache.org>.
akashrn5 commented on a change in pull request #4087:
URL: https://github.com/apache/carbondata/pull/4087#discussion_r576568284
##########
File path: integration/spark/src/main/scala/org/apache/spark/sql/secondaryindex/command/SICreationCommand.scala
##########
@@ -239,6 +249,25 @@ private[sql] case class CarbonCreateSecondaryIndexCommand(
throw new ErrorMessage(
s"Table [$tableName] under database [$databaseName] is already an index table")
}
+ val absoluteTableIdentifier = AbsoluteTableIdentifier.
+ from(tablePath, databaseName, indexTableName)
+ val indexTablePath = CarbonTablePath
+ .getMetadataPath(absoluteTableIdentifier.getTablePath)
+ val mainTblLoadMetadataDetails: Array[LoadMetadataDetails] =
+ SegmentStatusManager.readLoadMetadata(carbonTable.getMetadataPath)
+ var siTblLoadMetadataDetails: Array[LoadMetadataDetails] =
+ SegmentStatusManager.readLoadMetadata(indexTablePath)
+ if (isRegisterIndex) {
+ // check if SI table has undergone any Update or delete operation, which can happen in
+ // case of compatibility scenario. IUD after Refresh SI and before register index
+ val updatedSegmentsCount = siTblLoadMetadataDetails.filter(loadMetaDetail =>
+ !loadMetaDetail.getUpdateStatusFileName.equals(""))
+ if (!updatedSegmentsCount.isEmpty) {
+ throw new ErrorMessage("Cannot Register Secondary index table " + indexTableName +
Review comment:
use the scala string concat as scala file, `s"message ${var}"`. Please check in other places also.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4087: [WIP] SI compatability issue fix
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4087:
URL: https://github.com/apache/carbondata/pull/4087#issuecomment-773224131
Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12444/job/ApacheCarbon_PR_Builder_2.4.5/3671/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4087: [CARBONDATA-4125] SI compatability issue fix
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4087:
URL: https://github.com/apache/carbondata/pull/4087#issuecomment-776648606
Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12444/job/ApacheCarbon_PR_Builder_2.4.5/3691/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4087: [WIP] SI compatability issue fix
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4087:
URL: https://github.com/apache/carbondata/pull/4087#issuecomment-773224131
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] akashrn5 commented on pull request #4087: [CARBONDATA-4125] SI compatability issue fix
Posted by GitBox <gi...@apache.org>.
akashrn5 commented on pull request #4087:
URL: https://github.com/apache/carbondata/pull/4087#issuecomment-780424725
LGTM
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4087: [CARBONDATA-4125] SI compatability issue fix
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4087:
URL: https://github.com/apache/carbondata/pull/4087#issuecomment-777598349
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/5072/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] brijoobopanna commented on pull request #4087: [CARBONDATA-4125] SI compatability issue fix
Posted by GitBox <gi...@apache.org>.
brijoobopanna commented on pull request #4087:
URL: https://github.com/apache/carbondata/pull/4087#issuecomment-778193178
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4087: [CARBONDATA-4125] SI compatability issue fix
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4087:
URL: https://github.com/apache/carbondata/pull/4087#issuecomment-778247716
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12444/job/ApacheCarbonPRBuilder2.3/5463/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] Indhumathi27 commented on a change in pull request #4087: [CARBONDATA-4125] SI compatability issue fix
Posted by GitBox <gi...@apache.org>.
Indhumathi27 commented on a change in pull request #4087:
URL: https://github.com/apache/carbondata/pull/4087#discussion_r576147074
##########
File path: integration/spark/src/main/scala/org/apache/spark/sql/secondaryindex/command/SICreationCommand.scala
##########
@@ -239,6 +248,24 @@ private[sql] case class CarbonCreateSecondaryIndexCommand(
throw new ErrorMessage(
s"Table [$tableName] under database [$databaseName] is already an index table")
}
+ val absoluteTableIdentifier = AbsoluteTableIdentifier.
+ from(tablePath, databaseName, indexTableName)
+ val indexTablePath = CarbonTablePath
+ .getMetadataPath(absoluteTableIdentifier.getTablePath)
+ val mainTblLoadMetadataDetails: Array[LoadMetadataDetails] =
+ SegmentStatusManager.readLoadMetadata(carbonTable.getMetadataPath)
+ var siTblLoadMetadataDetails: Array[LoadMetadataDetails] =
+ SegmentStatusManager.readLoadMetadata(indexTablePath)
+ if (isRegisterIndex) {
+ // check if SI table has undergone any Update or delete operation, which can happen in
+ // case of compatibility scenario. IUD after Refresh SI and before register index
+ val updatedSegmentsCount = siTblLoadMetadataDetails.filter(loadMetaDetail =>
+ !loadMetaDetail.getUpdateStatusFileName.equals(""))
+ if (!updatedSegmentsCount.isEmpty) {
+ throw new ErrorMessage("Cannot Register Secondary index table " + indexTableName +
+ ", as it has undergone update or delete operation.")
Review comment:
done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4087: [WIP] SI compatability issue fix
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4087:
URL: https://github.com/apache/carbondata/pull/4087#issuecomment-773224317
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12444/job/ApacheCarbonPRBuilder2.3/5431/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4087: [CARBONDATA-4125] SI compatability issue fix
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4087:
URL: https://github.com/apache/carbondata/pull/4087#issuecomment-779231349
Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12444/job/ApacheCarbonPRBuilder2.3/5468/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4087: [CARBONDATA-4125] SI compatability issue fix
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4087:
URL: https://github.com/apache/carbondata/pull/4087#issuecomment-777596685
Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/3314/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4087: [CARBONDATA-4125] SI compatability issue fix
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4087:
URL: https://github.com/apache/carbondata/pull/4087#issuecomment-776646101
Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12444/job/ApacheCarbonPRBuilder2.3/5452/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4087: [WIP] SI compatability issue fix
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4087:
URL: https://github.com/apache/carbondata/pull/4087#issuecomment-772650984
Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12444/job/ApacheCarbon_PR_Builder_2.4.5/3661/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] akashrn5 commented on a change in pull request #4087: [CARBONDATA-4125] SI compatability issue fix
Posted by GitBox <gi...@apache.org>.
akashrn5 commented on a change in pull request #4087:
URL: https://github.com/apache/carbondata/pull/4087#discussion_r576568284
##########
File path: integration/spark/src/main/scala/org/apache/spark/sql/secondaryindex/command/SICreationCommand.scala
##########
@@ -239,6 +249,25 @@ private[sql] case class CarbonCreateSecondaryIndexCommand(
throw new ErrorMessage(
s"Table [$tableName] under database [$databaseName] is already an index table")
}
+ val absoluteTableIdentifier = AbsoluteTableIdentifier.
+ from(tablePath, databaseName, indexTableName)
+ val indexTablePath = CarbonTablePath
+ .getMetadataPath(absoluteTableIdentifier.getTablePath)
+ val mainTblLoadMetadataDetails: Array[LoadMetadataDetails] =
+ SegmentStatusManager.readLoadMetadata(carbonTable.getMetadataPath)
+ var siTblLoadMetadataDetails: Array[LoadMetadataDetails] =
+ SegmentStatusManager.readLoadMetadata(indexTablePath)
+ if (isRegisterIndex) {
+ // check if SI table has undergone any Update or delete operation, which can happen in
+ // case of compatibility scenario. IUD after Refresh SI and before register index
+ val updatedSegmentsCount = siTblLoadMetadataDetails.filter(loadMetaDetail =>
+ !loadMetaDetail.getUpdateStatusFileName.equals(""))
+ if (!updatedSegmentsCount.isEmpty) {
+ throw new ErrorMessage("Cannot Register Secondary index table " + indexTableName +
Review comment:
use the scala string concat as scala file, `s"message ${var}"`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4087: [CARBONDATA-4125] SI compatability issue fix
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4087:
URL: https://github.com/apache/carbondata/pull/4087#issuecomment-780348611
Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12444/job/ApacheCarbon_PR_Builder_2.4.5/3711/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4087: [WIP] SI compatability issue fix
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4087:
URL: https://github.com/apache/carbondata/pull/4087#issuecomment-772643405
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12444/job/ApacheCarbonPRBuilder2.3/5422/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] akashrn5 commented on a change in pull request #4087: [CARBONDATA-4125] SI compatability issue fix
Posted by GitBox <gi...@apache.org>.
akashrn5 commented on a change in pull request #4087:
URL: https://github.com/apache/carbondata/pull/4087#discussion_r576096839
##########
File path: integration/spark/src/main/scala/org/apache/spark/sql/secondaryindex/command/SICreationCommand.scala
##########
@@ -278,6 +303,31 @@ private[sql] case class CarbonCreateSecondaryIndexCommand(
carbonTable, databaseName,
tableName, indexTableName, absoluteTableIdentifier)
}
+ if (isRegisterIndex && null != tableInfo.getFactTable.getSchemaEvolution &&
+ null != carbonTable.getTableInfo.getFactTable.getSchemaEvolution) {
+ // check if SI table has undergone any alter schema operation before registering it
+ val schemaEvolutionEntryList = tableInfo
+ .getFactTable
+ .getSchemaEvolution
+ .getSchemaEvolutionEntryList
+ val mainTableSchemaEvolutionEntryList = carbonTable
+ .getTableInfo
+ .getFactTable
+ .getSchemaEvolution
+ .getSchemaEvolutionEntryList
+ if (mainTableSchemaEvolutionEntryList.size() != schemaEvolutionEntryList.size()) {
+ val index = mainTableSchemaEvolutionEntryList.size()
+ for (i <- index until schemaEvolutionEntryList.size()) {
Review comment:
use foreach
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] akashrn5 commented on a change in pull request #4087: [CARBONDATA-4125] SI compatability issue fix
Posted by GitBox <gi...@apache.org>.
akashrn5 commented on a change in pull request #4087:
URL: https://github.com/apache/carbondata/pull/4087#discussion_r576095548
##########
File path: integration/spark/src/main/scala/org/apache/spark/sql/secondaryindex/command/SICreationCommand.scala
##########
@@ -239,6 +248,24 @@ private[sql] case class CarbonCreateSecondaryIndexCommand(
throw new ErrorMessage(
s"Table [$tableName] under database [$databaseName] is already an index table")
}
+ val absoluteTableIdentifier = AbsoluteTableIdentifier.
+ from(tablePath, databaseName, indexTableName)
+ val indexTablePath = CarbonTablePath
+ .getMetadataPath(absoluteTableIdentifier.getTablePath)
+ val mainTblLoadMetadataDetails: Array[LoadMetadataDetails] =
+ SegmentStatusManager.readLoadMetadata(carbonTable.getMetadataPath)
+ var siTblLoadMetadataDetails: Array[LoadMetadataDetails] =
+ SegmentStatusManager.readLoadMetadata(indexTablePath)
+ if (isRegisterIndex) {
+ // check if SI table has undergone any Update or delete operation, which can happen in
+ // case of compatibility scenario. IUD after Refresh SI and before register index
+ val updatedSegmentsCount = siTblLoadMetadataDetails.filter(loadMetaDetail =>
+ !loadMetaDetail.getUpdateStatusFileName.equals(""))
+ if (!updatedSegmentsCount.isEmpty) {
+ throw new ErrorMessage("Cannot Register Secondary index table " + indexTableName +
+ ", as it has undergone update or delete operation.")
Review comment:
in this scenario, if user cant register index then he needs to drop and recreated the SI right, if so, please include that in error message also.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org