You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@carbondata.apache.org by GitBox <gi...@apache.org> on 2021/03/10 13:15:25 UTC
[GitHub] [carbondata] liuhe0702 opened a new pull request #4104: [CARBONDATA-4146]Query fails and the error message "unable to get file status" is displayed. query is normal after the "drop metacache on table" command is executed.
liuhe0702 opened a new pull request #4104:
URL: https://github.com/apache/carbondata/pull/4104
Why is this PR needed?
During compact execution, the status of the new segment is set to success before index files are merged.
After index files are merged, the carbonindex files are deleted.
As a result, the query task cannot find the cached carbonindex files.
What changes were proposed in this PR?
Set the status of the new segment to succeeded after index files are merged.
Does this PR introduce any user interface change?
No
Is any new testcase added?
No
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] liuhe0702 commented on a change in pull request #4104: [CARBONDATA-4146]Query fails and the error message "unable to get file status" is displayed. query is normal after the "drop metacache on table" command is executed.
Posted by GitBox <gi...@apache.org>.
liuhe0702 commented on a change in pull request #4104:
URL: https://github.com/apache/carbondata/pull/4104#discussion_r598454997
##########
File path: processing/src/main/java/org/apache/carbondata/processing/merger/CompactionResultSortProcessor.java
##########
@@ -190,7 +190,12 @@ public boolean execute(List<RawResultIterator> unsortedResultIteratorList,
LOGGER.error(e.getLocalizedMessage(), e);
throw e;
} finally {
- if (partitionSpec != null) {
+ boolean isMergeIndex = Boolean.parseBoolean(CarbonProperties.getInstance().getProperty(
+ CarbonCommonConstants.CARBON_MERGE_INDEX_IN_SEGMENT,
+ CarbonCommonConstants.CARBON_MERGE_INDEX_IN_SEGMENT_DEFAULT));
+ // mergeIndex is true, the segment file not need to be wrotten
+ // and will be wrotten during merging index
+ if (partitionSpec != null && !isMergeIndex) {
Review comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4104: [CARBONDATA-4146]Query fails and the error message "unable to get file status" is displayed. query is normal after the "drop metacache on table" command is executed.
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4104:
URL: https://github.com/apache/carbondata/pull/4104#issuecomment-804039784
Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/3324/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] liuhe0702 commented on pull request #4104: [CARBONDATA-4146]Query fails and the error message "unable to get file status" is displayed. query is normal after the "drop metacache on table" command is executed.
Posted by GitBox <gi...@apache.org>.
liuhe0702 commented on pull request #4104:
URL: https://github.com/apache/carbondata/pull/4104#issuecomment-797346594
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4104: [CARBONDATA-4146]Query fails and the error message "unable to get file status" is displayed. query is normal after the "drop metacache on table" command is executed.
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4104:
URL: https://github.com/apache/carbondata/pull/4104#issuecomment-802047900
Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12444/job/ApacheCarbonPRBuilder2.3/5585/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] ShreelekhyaG commented on a change in pull request #4104: [CARBONDATA-4146]Query fails and the error message "unable to get file status" is displayed. query is normal after the "drop metacache on table" command is executed.
Posted by GitBox <gi...@apache.org>.
ShreelekhyaG commented on a change in pull request #4104:
URL: https://github.com/apache/carbondata/pull/4104#discussion_r596907341
##########
File path: integration/spark/src/main/scala/org/apache/carbondata/spark/rdd/CarbonTableCompactor.scala
##########
@@ -296,26 +299,50 @@ class CarbonTableCompactor(
if (finalMergeStatus) {
val mergedLoadNumber = CarbonDataMergerUtil.getLoadNumberFromLoadName(mergedLoadName)
- var segmentFilesForIUDCompact = new util.ArrayList[Segment]()
var segmentFileName: String = null
+
+ val mergeIndex = CarbonProperties.getInstance().getProperty(
+ CarbonCommonConstants.CARBON_MERGE_INDEX_IN_SEGMENT,
+ CarbonCommonConstants.CARBON_MERGE_INDEX_IN_SEGMENT_DEFAULT).toBoolean
+
+ if (compactionType != CompactionType.IUD_DELETE_DELTA && mergeIndex) {
+ MergeIndexUtil.mergeIndexFilesOnCompaction(compactionCallableModel)
+ }
+
if (carbonTable.isHivePartitionTable) {
- val readPath =
- CarbonTablePath.getSegmentFilesLocation(carbonLoadModel.getTablePath) +
- CarbonCommonConstants.FILE_SEPARATOR + carbonLoadModel.getFactTimeStamp + ".tmp"
- // Merge all partition files into a single file.
- segmentFileName =
- mergedLoadNumber + "_" + carbonLoadModel.getFactTimeStamp
- val segmentFile = SegmentFileStore
- .mergeSegmentFiles(readPath,
- segmentFileName,
- CarbonTablePath.getSegmentFilesLocation(carbonLoadModel.getTablePath))
- if (segmentFile != null) {
- SegmentFileStore
- .moveFromTempFolder(segmentFile,
- carbonLoadModel.getFactTimeStamp + ".tmp",
- carbonLoadModel.getTablePath)
+ if (mergeIndex) {
+ val segmentTmpFileName = carbonLoadModel.getFactTimeStamp + CarbonTablePath.SEGMENT_EXT
+ segmentFileName = mergedLoadNumber + "_" + segmentTmpFileName
+ val segmentTmpFile = FileFactory.getCarbonFile(
+ CarbonTablePath.getSegmentFilePath(carbonTable.getTablePath, segmentTmpFileName))
+ if (!segmentTmpFile.renameForce(
+ CarbonTablePath.getSegmentFilePath(carbonTable.getTablePath, segmentFileName))) {
+ throw new Exception(s"Rename segment file from ${segmentTmpFileName} " +
+ s"to ${segmentFileName} failed.")
+ }
+ val tmpPath =
Review comment:
If we are not using tmp segment files when mergeIndex is true, may be we can avoid writing it. pls check once in `RowResultMergerProcessor `and `CompactionResultSortProcessor`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] akashrn5 commented on a change in pull request #4104: [CARBONDATA-4146]Query fails and the error message "unable to get file status" is displayed. query is normal after the "drop metacache on table" command is executed.
Posted by GitBox <gi...@apache.org>.
akashrn5 commented on a change in pull request #4104:
URL: https://github.com/apache/carbondata/pull/4104#discussion_r597487929
##########
File path: processing/src/main/java/org/apache/carbondata/processing/merger/CompactionResultSortProcessor.java
##########
@@ -190,7 +200,7 @@ public boolean execute(List<RawResultIterator> unsortedResultIteratorList,
LOGGER.error(e.getLocalizedMessage(), e);
throw e;
} finally {
- if (partitionSpec != null) {
Review comment:
yes, @liuhe0702 instead of making parameter changes or adding constructors, check property here only and can avoid changes in MergerRDD
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] ShreelekhyaG commented on pull request #4104: [CARBONDATA-4146]Query fails and the error message "unable to get file status" is displayed. query is normal after the "drop metacache on table" command is executed.
Posted by GitBox <gi...@apache.org>.
ShreelekhyaG commented on pull request #4104:
URL: https://github.com/apache/carbondata/pull/4104#issuecomment-804754945
LGTM
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4104: [CARBONDATA-4146]Query fails and the error message "unable to get file status" is displayed. query is normal after the "drop metacache on table" command is executed.
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4104:
URL: https://github.com/apache/carbondata/pull/4104#issuecomment-802570355
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12444/job/ApacheCarbonPRBuilder2.3/5596/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4104: [CARBONDATA-4146]Query fails and the error message "unable to get file status" is displayed. query is normal after the "drop metacache on table" command is executed.
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4104:
URL: https://github.com/apache/carbondata/pull/4104#issuecomment-796690631
Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12444/job/ApacheCarbon_PR_Builder_2.4.5/3788/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4104: [CARBONDATA-4146]Query fails and the error message "unable to get file status" is displayed. query is normal after the "drop metacache on table" command is executed.
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4104:
URL: https://github.com/apache/carbondata/pull/4104#issuecomment-796688384
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12444/job/ApacheCarbonPRBuilder2.3/5554/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] liuhe0702 commented on pull request #4104: [CARBONDATA-4146]Query fails and the error message "unable to get file status" is displayed. query is normal after the "drop metacache on table" command is executed.
Posted by GitBox <gi...@apache.org>.
liuhe0702 commented on pull request #4104:
URL: https://github.com/apache/carbondata/pull/4104#issuecomment-797432137
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4104: [CARBONDATA-4146]Query fails and the error message "unable to get file status" is displayed. query is normal after the "drop metacache on table" command is executed.
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4104:
URL: https://github.com/apache/carbondata/pull/4104#issuecomment-802737738
Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12444/job/ApacheCarbonPRBuilder2.3/5603/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] liuhe0702 commented on a change in pull request #4104: [CARBONDATA-4146]Query fails and the error message "unable to get file status" is displayed. query is normal after the "drop metacache on table" command is executed.
Posted by GitBox <gi...@apache.org>.
liuhe0702 commented on a change in pull request #4104:
URL: https://github.com/apache/carbondata/pull/4104#discussion_r597028077
##########
File path: integration/spark/src/main/scala/org/apache/carbondata/spark/rdd/CarbonTableCompactor.scala
##########
@@ -296,26 +299,50 @@ class CarbonTableCompactor(
if (finalMergeStatus) {
val mergedLoadNumber = CarbonDataMergerUtil.getLoadNumberFromLoadName(mergedLoadName)
- var segmentFilesForIUDCompact = new util.ArrayList[Segment]()
var segmentFileName: String = null
+
+ val mergeIndex = CarbonProperties.getInstance().getProperty(
+ CarbonCommonConstants.CARBON_MERGE_INDEX_IN_SEGMENT,
+ CarbonCommonConstants.CARBON_MERGE_INDEX_IN_SEGMENT_DEFAULT).toBoolean
+
+ if (compactionType != CompactionType.IUD_DELETE_DELTA && mergeIndex) {
+ MergeIndexUtil.mergeIndexFilesOnCompaction(compactionCallableModel)
+ }
+
if (carbonTable.isHivePartitionTable) {
- val readPath =
- CarbonTablePath.getSegmentFilesLocation(carbonLoadModel.getTablePath) +
- CarbonCommonConstants.FILE_SEPARATOR + carbonLoadModel.getFactTimeStamp + ".tmp"
- // Merge all partition files into a single file.
- segmentFileName =
- mergedLoadNumber + "_" + carbonLoadModel.getFactTimeStamp
- val segmentFile = SegmentFileStore
- .mergeSegmentFiles(readPath,
- segmentFileName,
- CarbonTablePath.getSegmentFilesLocation(carbonLoadModel.getTablePath))
- if (segmentFile != null) {
- SegmentFileStore
- .moveFromTempFolder(segmentFile,
- carbonLoadModel.getFactTimeStamp + ".tmp",
- carbonLoadModel.getTablePath)
+ if (mergeIndex) {
+ val segmentTmpFileName = carbonLoadModel.getFactTimeStamp + CarbonTablePath.SEGMENT_EXT
+ segmentFileName = mergedLoadNumber + "_" + segmentTmpFileName
+ val segmentTmpFile = FileFactory.getCarbonFile(
+ CarbonTablePath.getSegmentFilePath(carbonTable.getTablePath, segmentTmpFileName))
+ if (!segmentTmpFile.renameForce(
+ CarbonTablePath.getSegmentFilePath(carbonTable.getTablePath, segmentFileName))) {
+ throw new Exception(s"Rename segment file from ${segmentTmpFileName} " +
+ s"to ${segmentFileName} failed.")
+ }
+ val tmpPath =
Review comment:
tmpPath is used. During merging index, each task for partition will generated a temporaty segment file in tmpPath and all the temporaty segment files are merged into a final segment file.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] akashrn5 edited a comment on pull request #4104: [CARBONDATA-4146]Query fails and the error message "unable to get file status" is displayed. query is normal after the "drop metacache on table" command is executed.
Posted by GitBox <gi...@apache.org>.
akashrn5 edited a comment on pull request #4104:
URL: https://github.com/apache/carbondata/pull/4104#issuecomment-800028090
@liuhe0702 i think this issue is already being handled in #3988 , so i think no need of this PR, please refer the jira to the jira of #3988 and close as duplicate. @ShreelekhyaG please confirm the same here, where this scenario is being handled or not in your PR.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4104: [CARBONDATA-4146]Query fails and the error message "unable to get file status" is displayed. query is normal after the "drop metacache on table" command is executed.
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4104:
URL: https://github.com/apache/carbondata/pull/4104#issuecomment-795400926
Can one of the admins verify this patch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4104: [CARBONDATA-4146]Query fails and the error message "unable to get file status" is displayed. query is normal after the "drop metacache on table" command is executed.
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4104:
URL: https://github.com/apache/carbondata/pull/4104#issuecomment-797313501
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12444/job/ApacheCarbonPRBuilder2.3/5561/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4104: [CARBONDATA-4146]Query fails and the error message "unable to get file status" is displayed. query is normal after the "drop metacache on table" command is executed.
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4104:
URL: https://github.com/apache/carbondata/pull/4104#issuecomment-803625714
Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/3307/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] liuhe0702 closed pull request #4104: [CARBONDATA-4146]Query fails and the error message "unable to get file status" is displayed. query is normal after the "drop metacache on table" command is executed.
Posted by GitBox <gi...@apache.org>.
liuhe0702 closed pull request #4104:
URL: https://github.com/apache/carbondata/pull/4104
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4104: [CARBONDATA-4146]Query fails and the error message "unable to get file status" is displayed. query is normal after the "drop metacache on table" command is executed.
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4104:
URL: https://github.com/apache/carbondata/pull/4104#issuecomment-802739486
Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12444/job/ApacheCarbon_PR_Builder_2.4.5/3837/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4104: [CARBONDATA-4146]Query fails and the error message "unable to get file status" is displayed. query is normal after the "drop metacache on table" command is executed.
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4104:
URL: https://github.com/apache/carbondata/pull/4104#issuecomment-803933718
Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/5076/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] brijoobopanna commented on pull request #4104: [CARBONDATA-4146]Query fails and the error message "unable to get file status" is displayed. query is normal after the "drop metacache on table" command is executed.
Posted by GitBox <gi...@apache.org>.
brijoobopanna commented on pull request #4104:
URL: https://github.com/apache/carbondata/pull/4104#issuecomment-796452679
add to whitelist
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4104: [CARBONDATA-4146]Query fails and the error message "unable to get file status" is displayed. query is normal after the "drop metacache on table" command is executed.
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4104:
URL: https://github.com/apache/carbondata/pull/4104#issuecomment-797189506
Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12444/job/ApacheCarbon_PR_Builder_2.4.5/3792/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4104: [CARBONDATA-4146]Query fails and the error message "unable to get file status" is displayed. query is normal after the "drop metacache on table" command is executed.
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4104:
URL: https://github.com/apache/carbondata/pull/4104#issuecomment-803763582
Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/5072/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4104: [CARBONDATA-4146]Query fails and the error message "unable to get file status" is displayed. query is normal after the "drop metacache on table" command is executed.
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4104:
URL: https://github.com/apache/carbondata/pull/4104#issuecomment-803722043
Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/3316/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] liuhe0702 commented on pull request #4104: [CARBONDATA-4146]Query fails and the error message "unable to get file status" is displayed. query is normal after the "drop metacache on table" command is executed.
Posted by GitBox <gi...@apache.org>.
liuhe0702 commented on pull request #4104:
URL: https://github.com/apache/carbondata/pull/4104#issuecomment-797202419
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4104: [CARBONDATA-4146]Query fails and the error message "unable to get file status" is displayed. query is normal after the "drop metacache on table" command is executed.
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4104:
URL: https://github.com/apache/carbondata/pull/4104#issuecomment-803860422
Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/3321/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4104: [CARBONDATA-4146]Query fails and the error message "unable to get file status" is displayed. query is normal after the "drop metacache on table" command is executed.
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4104:
URL: https://github.com/apache/carbondata/pull/4104#issuecomment-797430674
Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12444/job/ApacheCarbon_PR_Builder_2.4.5/3797/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] liuhe0702 commented on a change in pull request #4104: [CARBONDATA-4146]Query fails and the error message "unable to get file status" is displayed. query is normal after the "drop metacache on table" command is executed.
Posted by GitBox <gi...@apache.org>.
liuhe0702 commented on a change in pull request #4104:
URL: https://github.com/apache/carbondata/pull/4104#discussion_r597028077
##########
File path: integration/spark/src/main/scala/org/apache/carbondata/spark/rdd/CarbonTableCompactor.scala
##########
@@ -296,26 +299,50 @@ class CarbonTableCompactor(
if (finalMergeStatus) {
val mergedLoadNumber = CarbonDataMergerUtil.getLoadNumberFromLoadName(mergedLoadName)
- var segmentFilesForIUDCompact = new util.ArrayList[Segment]()
var segmentFileName: String = null
+
+ val mergeIndex = CarbonProperties.getInstance().getProperty(
+ CarbonCommonConstants.CARBON_MERGE_INDEX_IN_SEGMENT,
+ CarbonCommonConstants.CARBON_MERGE_INDEX_IN_SEGMENT_DEFAULT).toBoolean
+
+ if (compactionType != CompactionType.IUD_DELETE_DELTA && mergeIndex) {
+ MergeIndexUtil.mergeIndexFilesOnCompaction(compactionCallableModel)
+ }
+
if (carbonTable.isHivePartitionTable) {
- val readPath =
- CarbonTablePath.getSegmentFilesLocation(carbonLoadModel.getTablePath) +
- CarbonCommonConstants.FILE_SEPARATOR + carbonLoadModel.getFactTimeStamp + ".tmp"
- // Merge all partition files into a single file.
- segmentFileName =
- mergedLoadNumber + "_" + carbonLoadModel.getFactTimeStamp
- val segmentFile = SegmentFileStore
- .mergeSegmentFiles(readPath,
- segmentFileName,
- CarbonTablePath.getSegmentFilesLocation(carbonLoadModel.getTablePath))
- if (segmentFile != null) {
- SegmentFileStore
- .moveFromTempFolder(segmentFile,
- carbonLoadModel.getFactTimeStamp + ".tmp",
- carbonLoadModel.getTablePath)
+ if (mergeIndex) {
+ val segmentTmpFileName = carbonLoadModel.getFactTimeStamp + CarbonTablePath.SEGMENT_EXT
+ segmentFileName = mergedLoadNumber + "_" + segmentTmpFileName
+ val segmentTmpFile = FileFactory.getCarbonFile(
+ CarbonTablePath.getSegmentFilePath(carbonTable.getTablePath, segmentTmpFileName))
+ if (!segmentTmpFile.renameForce(
+ CarbonTablePath.getSegmentFilePath(carbonTable.getTablePath, segmentFileName))) {
+ throw new Exception(s"Rename segment file from ${segmentTmpFileName} " +
+ s"to ${segmentFileName} failed.")
+ }
+ val tmpPath =
Review comment:
tmpPath is used. During merging index, each task for partition will generated a temporaty segment file in tmpPath and all the temporaty segment files are merged into a final segment file.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] akashrn5 commented on a change in pull request #4104: [CARBONDATA-4146]Query fails and the error message "unable to get file status" is displayed. query is normal after the "drop metacache on table" command is executed.
Posted by GitBox <gi...@apache.org>.
akashrn5 commented on a change in pull request #4104:
URL: https://github.com/apache/carbondata/pull/4104#discussion_r597407885
##########
File path: integration/spark/src/main/scala/org/apache/carbondata/spark/rdd/CarbonTableCompactor.scala
##########
@@ -296,26 +299,46 @@ class CarbonTableCompactor(
if (finalMergeStatus) {
val mergedLoadNumber = CarbonDataMergerUtil.getLoadNumberFromLoadName(mergedLoadName)
- var segmentFilesForIUDCompact = new util.ArrayList[Segment]()
var segmentFileName: String = null
+
+ val isMergeIndexEnabled = CarbonProperties.getInstance().getProperty(
+ CarbonCommonConstants.CARBON_MERGE_INDEX_IN_SEGMENT,
+ CarbonCommonConstants.CARBON_MERGE_INDEX_IN_SEGMENT_DEFAULT).toBoolean
+
+ if (compactionType != CompactionType.IUD_DELETE_DELTA && isMergeIndexEnabled) {
+ MergeIndexUtil.mergeIndexFilesOnCompaction(compactionCallableModel)
+ }
+
if (carbonTable.isHivePartitionTable) {
- val readPath =
- CarbonTablePath.getSegmentFilesLocation(carbonLoadModel.getTablePath) +
- CarbonCommonConstants.FILE_SEPARATOR + carbonLoadModel.getFactTimeStamp + ".tmp"
- // Merge all partition files into a single file.
- segmentFileName =
- mergedLoadNumber + "_" + carbonLoadModel.getFactTimeStamp
- val segmentFile = SegmentFileStore
- .mergeSegmentFiles(readPath,
- segmentFileName,
- CarbonTablePath.getSegmentFilesLocation(carbonLoadModel.getTablePath))
- if (segmentFile != null) {
- SegmentFileStore
- .moveFromTempFolder(segmentFile,
- carbonLoadModel.getFactTimeStamp + ".tmp",
- carbonLoadModel.getTablePath)
+ if (isMergeIndexEnabled) {
+ val segmentTmpFileName = carbonLoadModel.getFactTimeStamp + CarbonTablePath.SEGMENT_EXT
+ segmentFileName = mergedLoadNumber + "_" + segmentTmpFileName
+ val segmentTmpFile = FileFactory.getCarbonFile(
+ CarbonTablePath.getSegmentFilePath(carbonTable.getTablePath, segmentTmpFileName))
+ if (!segmentTmpFile.renameForce(
+ CarbonTablePath.getSegmentFilePath(carbonTable.getTablePath, segmentFileName))) {
+ throw new Exception(s"Rename segment file from ${segmentTmpFileName} " +
+ s"to ${segmentFileName} failed.")
+ }
+ } else {
+ val readPath =
+ CarbonTablePath.getSegmentFilesLocation(carbonLoadModel.getTablePath) +
+ CarbonCommonConstants.FILE_SEPARATOR + carbonLoadModel.getFactTimeStamp + ".tmp"
+ // Merge all partition files into a single file.
+ segmentFileName =
+ mergedLoadNumber + "_" + carbonLoadModel.getFactTimeStamp
Review comment:
use static constant from CarbonCommonConstants for "_", check and replace in other places also
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] akashrn5 commented on a change in pull request #4104: [CARBONDATA-4146]Query fails and the error message "unable to get file status" is displayed. query is normal after the "drop metacache on table" command is executed.
Posted by GitBox <gi...@apache.org>.
akashrn5 commented on a change in pull request #4104:
URL: https://github.com/apache/carbondata/pull/4104#discussion_r598429980
##########
File path: processing/src/main/java/org/apache/carbondata/processing/merger/CompactionResultSortProcessor.java
##########
@@ -190,7 +190,12 @@ public boolean execute(List<RawResultIterator> unsortedResultIteratorList,
LOGGER.error(e.getLocalizedMessage(), e);
throw e;
} finally {
- if (partitionSpec != null) {
+ boolean isMergeIndex = Boolean.parseBoolean(CarbonProperties.getInstance().getProperty(
+ CarbonCommonConstants.CARBON_MERGE_INDEX_IN_SEGMENT,
+ CarbonCommonConstants.CARBON_MERGE_INDEX_IN_SEGMENT_DEFAULT));
+ // mergeIndex is true, the segment file not need to be wrotten
+ // and will be wrotten during merging index
+ if (partitionSpec != null && !isMergeIndex) {
Review comment:
`wrotten` to `written`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4104: [CARBONDATA-4146]Query fails and the error message "unable to get file status" is displayed. query is normal after the "drop metacache on table" command is executed.
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4104:
URL: https://github.com/apache/carbondata/pull/4104#issuecomment-804036744
Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/5077/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] liuhe0702 commented on pull request #4104: [CARBONDATA-4146]Query fails and the error message "unable to get file status" is displayed. query is normal after the "drop metacache on table" command is executed.
Posted by GitBox <gi...@apache.org>.
liuhe0702 commented on pull request #4104:
URL: https://github.com/apache/carbondata/pull/4104#issuecomment-803696803
retest this please
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] liuhe0702 commented on a change in pull request #4104: [CARBONDATA-4146]Query fails and the error message "unable to get file status" is displayed. query is normal after the "drop metacache on table" command is executed.
Posted by GitBox <gi...@apache.org>.
liuhe0702 commented on a change in pull request #4104:
URL: https://github.com/apache/carbondata/pull/4104#discussion_r597463702
##########
File path: integration/spark/src/main/scala/org/apache/carbondata/spark/rdd/CarbonTableCompactor.scala
##########
@@ -296,26 +299,50 @@ class CarbonTableCompactor(
if (finalMergeStatus) {
val mergedLoadNumber = CarbonDataMergerUtil.getLoadNumberFromLoadName(mergedLoadName)
- var segmentFilesForIUDCompact = new util.ArrayList[Segment]()
var segmentFileName: String = null
+
+ val mergeIndex = CarbonProperties.getInstance().getProperty(
+ CarbonCommonConstants.CARBON_MERGE_INDEX_IN_SEGMENT,
+ CarbonCommonConstants.CARBON_MERGE_INDEX_IN_SEGMENT_DEFAULT).toBoolean
+
+ if (compactionType != CompactionType.IUD_DELETE_DELTA && mergeIndex) {
+ MergeIndexUtil.mergeIndexFilesOnCompaction(compactionCallableModel)
+ }
+
if (carbonTable.isHivePartitionTable) {
- val readPath =
- CarbonTablePath.getSegmentFilesLocation(carbonLoadModel.getTablePath) +
- CarbonCommonConstants.FILE_SEPARATOR + carbonLoadModel.getFactTimeStamp + ".tmp"
- // Merge all partition files into a single file.
- segmentFileName =
- mergedLoadNumber + "_" + carbonLoadModel.getFactTimeStamp
- val segmentFile = SegmentFileStore
- .mergeSegmentFiles(readPath,
- segmentFileName,
- CarbonTablePath.getSegmentFilesLocation(carbonLoadModel.getTablePath))
- if (segmentFile != null) {
- SegmentFileStore
- .moveFromTempFolder(segmentFile,
- carbonLoadModel.getFactTimeStamp + ".tmp",
- carbonLoadModel.getTablePath)
+ if (mergeIndex) {
+ val segmentTmpFileName = carbonLoadModel.getFactTimeStamp + CarbonTablePath.SEGMENT_EXT
+ segmentFileName = mergedLoadNumber + "_" + segmentTmpFileName
+ val segmentTmpFile = FileFactory.getCarbonFile(
+ CarbonTablePath.getSegmentFilePath(carbonTable.getTablePath, segmentTmpFileName))
+ if (!segmentTmpFile.renameForce(
+ CarbonTablePath.getSegmentFilePath(carbonTable.getTablePath, segmentFileName))) {
+ throw new Exception(s"Rename segment file from ${segmentTmpFileName} " +
+ s"to ${segmentFileName} failed.")
+ }
+ val tmpPath =
Review comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] ShreelekhyaG commented on a change in pull request #4104: [CARBONDATA-4146]Query fails and the error message "unable to get file status" is displayed. query is normal after the "drop metacache on table" command is executed.
Posted by GitBox <gi...@apache.org>.
ShreelekhyaG commented on a change in pull request #4104:
URL: https://github.com/apache/carbondata/pull/4104#discussion_r597428016
##########
File path: processing/src/main/java/org/apache/carbondata/processing/merger/CompactionResultSortProcessor.java
##########
@@ -190,7 +200,7 @@ public boolean execute(List<RawResultIterator> unsortedResultIteratorList,
LOGGER.error(e.getLocalizedMessage(), e);
throw e;
} finally {
- if (partitionSpec != null) {
Review comment:
Instead of passing boolean value, cant we directly get `CarbonCommonConstants.CARBON_MERGE_INDEX_IN_SEGMENT `from properties? and the if check should be - if merge index not enabled, then write segment file. As below,
``` boolean isMergeIndexEnable = Boolean.parseBoolean(CarbonProperties.getInstance().getProperty(
CarbonCommonConstants.CARBON_MERGE_INDEX_IN_SEGMENT,
CarbonCommonConstants.CARBON_MERGE_INDEX_IN_SEGMENT_DEFAULT));
if (partitionSpec != null && !isMergeIndexEnable) { ```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4104: [CARBONDATA-4146]Query fails and the error message "unable to get file status" is displayed. query is normal after the "drop metacache on table" command is executed.
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4104:
URL: https://github.com/apache/carbondata/pull/4104#issuecomment-803856068
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/5074/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] akashrn5 commented on a change in pull request #4104: [CARBONDATA-4146]Query fails and the error message "unable to get file status" is displayed. query is normal after the "drop metacache on table" command is executed.
Posted by GitBox <gi...@apache.org>.
akashrn5 commented on a change in pull request #4104:
URL: https://github.com/apache/carbondata/pull/4104#discussion_r597410799
##########
File path: processing/src/main/java/org/apache/carbondata/processing/merger/CompactionResultSortProcessor.java
##########
@@ -149,16 +149,26 @@
private CarbonColumn[] noDicAndComplexColumns;
+ private boolean needSegmentFile;
Review comment:
why this boolean is required? if any specific scenario is failing or being handled, please add a comment
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] akashrn5 commented on a change in pull request #4104: [CARBONDATA-4146]Query fails and the error message "unable to get file status" is displayed. query is normal after the "drop metacache on table" command is executed.
Posted by GitBox <gi...@apache.org>.
akashrn5 commented on a change in pull request #4104:
URL: https://github.com/apache/carbondata/pull/4104#discussion_r597409105
##########
File path: integration/spark/src/main/scala/org/apache/carbondata/spark/rdd/CarbonTableCompactor.scala
##########
@@ -296,26 +299,46 @@ class CarbonTableCompactor(
if (finalMergeStatus) {
val mergedLoadNumber = CarbonDataMergerUtil.getLoadNumberFromLoadName(mergedLoadName)
- var segmentFilesForIUDCompact = new util.ArrayList[Segment]()
var segmentFileName: String = null
+
+ val isMergeIndexEnabled = CarbonProperties.getInstance().getProperty(
+ CarbonCommonConstants.CARBON_MERGE_INDEX_IN_SEGMENT,
+ CarbonCommonConstants.CARBON_MERGE_INDEX_IN_SEGMENT_DEFAULT).toBoolean
+
+ if (compactionType != CompactionType.IUD_DELETE_DELTA && isMergeIndexEnabled) {
+ MergeIndexUtil.mergeIndexFilesOnCompaction(compactionCallableModel)
+ }
+
if (carbonTable.isHivePartitionTable) {
- val readPath =
- CarbonTablePath.getSegmentFilesLocation(carbonLoadModel.getTablePath) +
- CarbonCommonConstants.FILE_SEPARATOR + carbonLoadModel.getFactTimeStamp + ".tmp"
- // Merge all partition files into a single file.
- segmentFileName =
- mergedLoadNumber + "_" + carbonLoadModel.getFactTimeStamp
- val segmentFile = SegmentFileStore
- .mergeSegmentFiles(readPath,
- segmentFileName,
- CarbonTablePath.getSegmentFilesLocation(carbonLoadModel.getTablePath))
- if (segmentFile != null) {
- SegmentFileStore
- .moveFromTempFolder(segmentFile,
- carbonLoadModel.getFactTimeStamp + ".tmp",
- carbonLoadModel.getTablePath)
+ if (isMergeIndexEnabled) {
+ val segmentTmpFileName = carbonLoadModel.getFactTimeStamp + CarbonTablePath.SEGMENT_EXT
+ segmentFileName = mergedLoadNumber + "_" + segmentTmpFileName
+ val segmentTmpFile = FileFactory.getCarbonFile(
+ CarbonTablePath.getSegmentFilePath(carbonTable.getTablePath, segmentTmpFileName))
+ if (!segmentTmpFile.renameForce(
+ CarbonTablePath.getSegmentFilePath(carbonTable.getTablePath, segmentFileName))) {
+ throw new Exception(s"Rename segment file from ${segmentTmpFileName} " +
+ s"to ${segmentFileName} failed.")
+ }
+ } else {
+ val readPath =
+ CarbonTablePath.getSegmentFilesLocation(carbonLoadModel.getTablePath) +
+ CarbonCommonConstants.FILE_SEPARATOR + carbonLoadModel.getFactTimeStamp + ".tmp"
+ // Merge all partition files into a single file.
+ segmentFileName =
+ mergedLoadNumber + "_" + carbonLoadModel.getFactTimeStamp
+ val segmentFile = SegmentFileStore
Review comment:
```suggestion
val mergedSegmentFile = SegmentFileStore
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4104: [CARBONDATA-4146]Query fails and the error message "unable to get file status" is displayed. query is normal after the "drop metacache on table" command is executed.
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4104:
URL: https://github.com/apache/carbondata/pull/4104#issuecomment-802572216
Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12444/job/ApacheCarbon_PR_Builder_2.4.5/3830/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4104: [CARBONDATA-4146]Query fails and the error message "unable to get file status" is displayed. query is normal after the "drop metacache on table" command is executed.
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4104:
URL: https://github.com/apache/carbondata/pull/4104#issuecomment-803768137
Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/3319/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] liuhe0702 commented on pull request #4104: [CARBONDATA-4146]Query fails and the error message "unable to get file status" is displayed. query is normal after the "drop metacache on table" command is executed.
Posted by GitBox <gi...@apache.org>.
liuhe0702 commented on pull request #4104:
URL: https://github.com/apache/carbondata/pull/4104#issuecomment-797257368
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4104: [CARBONDATA-4146]Query fails and the error message "unable to get file status" is displayed. query is normal after the "drop metacache on table" command is executed.
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4104:
URL: https://github.com/apache/carbondata/pull/4104#issuecomment-797496773
Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12444/job/ApacheCarbonPRBuilder2.3/5565/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] liuhe0702 commented on a change in pull request #4104: [CARBONDATA-4146]Query fails and the error message "unable to get file status" is displayed. query is normal after the "drop metacache on table" command is executed.
Posted by GitBox <gi...@apache.org>.
liuhe0702 commented on a change in pull request #4104:
URL: https://github.com/apache/carbondata/pull/4104#discussion_r597382404
##########
File path: integration/spark/src/main/scala/org/apache/carbondata/spark/rdd/CarbonTableCompactor.scala
##########
@@ -296,26 +299,50 @@ class CarbonTableCompactor(
if (finalMergeStatus) {
val mergedLoadNumber = CarbonDataMergerUtil.getLoadNumberFromLoadName(mergedLoadName)
- var segmentFilesForIUDCompact = new util.ArrayList[Segment]()
var segmentFileName: String = null
+
+ val mergeIndex = CarbonProperties.getInstance().getProperty(
+ CarbonCommonConstants.CARBON_MERGE_INDEX_IN_SEGMENT,
+ CarbonCommonConstants.CARBON_MERGE_INDEX_IN_SEGMENT_DEFAULT).toBoolean
+
+ if (compactionType != CompactionType.IUD_DELETE_DELTA && mergeIndex) {
+ MergeIndexUtil.mergeIndexFilesOnCompaction(compactionCallableModel)
+ }
+
if (carbonTable.isHivePartitionTable) {
- val readPath =
- CarbonTablePath.getSegmentFilesLocation(carbonLoadModel.getTablePath) +
- CarbonCommonConstants.FILE_SEPARATOR + carbonLoadModel.getFactTimeStamp + ".tmp"
- // Merge all partition files into a single file.
- segmentFileName =
- mergedLoadNumber + "_" + carbonLoadModel.getFactTimeStamp
- val segmentFile = SegmentFileStore
- .mergeSegmentFiles(readPath,
- segmentFileName,
- CarbonTablePath.getSegmentFilesLocation(carbonLoadModel.getTablePath))
- if (segmentFile != null) {
- SegmentFileStore
- .moveFromTempFolder(segmentFile,
- carbonLoadModel.getFactTimeStamp + ".tmp",
- carbonLoadModel.getTablePath)
+ if (mergeIndex) {
+ val segmentTmpFileName = carbonLoadModel.getFactTimeStamp + CarbonTablePath.SEGMENT_EXT
+ segmentFileName = mergedLoadNumber + "_" + segmentTmpFileName
+ val segmentTmpFile = FileFactory.getCarbonFile(
+ CarbonTablePath.getSegmentFilePath(carbonTable.getTablePath, segmentTmpFileName))
+ if (!segmentTmpFile.renameForce(
+ CarbonTablePath.getSegmentFilePath(carbonTable.getTablePath, segmentFileName))) {
+ throw new Exception(s"Rename segment file from ${segmentTmpFileName} " +
+ s"to ${segmentFileName} failed.")
+ }
+ val tmpPath =
Review comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4104: [CARBONDATA-4146]Query fails and the error message "unable to get file status" is displayed. query is normal after the "drop metacache on table" command is executed.
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4104:
URL: https://github.com/apache/carbondata/pull/4104#issuecomment-803720072
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/5069/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4104: [CARBONDATA-4146]Query fails and the error message "unable to get file status" is displayed. query is normal after the "drop metacache on table" command is executed.
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4104:
URL: https://github.com/apache/carbondata/pull/4104#issuecomment-802052102
Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12444/job/ApacheCarbon_PR_Builder_2.4.5/3820/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4104: [CARBONDATA-4146]Query fails and the error message "unable to get file status" is displayed. query is normal after the "drop metacache on table" command is executed.
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4104:
URL: https://github.com/apache/carbondata/pull/4104#issuecomment-797241039
Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12444/job/ApacheCarbonPRBuilder2.3/5560/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] liuhe0702 removed a comment on pull request #4104: [CARBONDATA-4146]Query fails and the error message "unable to get file status" is displayed. query is normal after the "drop metacache on table" command is executed.
Posted by GitBox <gi...@apache.org>.
liuhe0702 removed a comment on pull request #4104:
URL: https://github.com/apache/carbondata/pull/4104#issuecomment-796741652
add to whitelist
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4104: [CARBONDATA-4146]Query fails and the error message "unable to get file status" is displayed. query is normal after the "drop metacache on table" command is executed.
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4104:
URL: https://github.com/apache/carbondata/pull/4104#issuecomment-796525788
Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12444/job/ApacheCarbon_PR_Builder_2.4.5/3780/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] liuhe0702 commented on pull request #4104: [CARBONDATA-4146]Query fails and the error message "unable to get file status" is displayed. query is normal after the "drop metacache on table" command is executed.
Posted by GitBox <gi...@apache.org>.
liuhe0702 commented on pull request #4104:
URL: https://github.com/apache/carbondata/pull/4104#issuecomment-797146808
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] liuhe0702 commented on a change in pull request #4104: [CARBONDATA-4146]Query fails and the error message "unable to get file status" is displayed. query is normal after the "drop metacache on table" command is executed.
Posted by GitBox <gi...@apache.org>.
liuhe0702 commented on a change in pull request #4104:
URL: https://github.com/apache/carbondata/pull/4104#discussion_r597462247
##########
File path: processing/src/main/java/org/apache/carbondata/processing/merger/CompactionResultSortProcessor.java
##########
@@ -149,16 +149,26 @@
private CarbonColumn[] noDicAndComplexColumns;
+ private boolean needSegmentFile;
Review comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4104: [CARBONDATA-4146]Query fails and the error message "unable to get file status" is displayed. query is normal after the "drop metacache on table" command is executed.
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4104:
URL: https://github.com/apache/carbondata/pull/4104#issuecomment-796526039
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12444/job/ApacheCarbonPRBuilder2.3/5545/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] akashrn5 commented on pull request #4104: [CARBONDATA-4146]Query fails and the error message "unable to get file status" is displayed. query is normal after the "drop metacache on table" command is executed.
Posted by GitBox <gi...@apache.org>.
akashrn5 commented on pull request #4104:
URL: https://github.com/apache/carbondata/pull/4104#issuecomment-800028090
@liuhe0702 i think this issue is already being handled in #3988 , so i think no need of this PR, please refer the jira to the jira of #3988 and close as duplicate.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] liuhe0702 commented on a change in pull request #4104: [CARBONDATA-4146]Query fails and the error message "unable to get file status" is displayed. query is normal after the "drop metacache on table" command is executed.
Posted by GitBox <gi...@apache.org>.
liuhe0702 commented on a change in pull request #4104:
URL: https://github.com/apache/carbondata/pull/4104#discussion_r597462077
##########
File path: integration/spark/src/main/scala/org/apache/carbondata/spark/rdd/CarbonTableCompactor.scala
##########
@@ -296,26 +299,46 @@ class CarbonTableCompactor(
if (finalMergeStatus) {
val mergedLoadNumber = CarbonDataMergerUtil.getLoadNumberFromLoadName(mergedLoadName)
- var segmentFilesForIUDCompact = new util.ArrayList[Segment]()
var segmentFileName: String = null
+
+ val isMergeIndexEnabled = CarbonProperties.getInstance().getProperty(
+ CarbonCommonConstants.CARBON_MERGE_INDEX_IN_SEGMENT,
+ CarbonCommonConstants.CARBON_MERGE_INDEX_IN_SEGMENT_DEFAULT).toBoolean
+
+ if (compactionType != CompactionType.IUD_DELETE_DELTA && isMergeIndexEnabled) {
+ MergeIndexUtil.mergeIndexFilesOnCompaction(compactionCallableModel)
+ }
+
if (carbonTable.isHivePartitionTable) {
- val readPath =
- CarbonTablePath.getSegmentFilesLocation(carbonLoadModel.getTablePath) +
- CarbonCommonConstants.FILE_SEPARATOR + carbonLoadModel.getFactTimeStamp + ".tmp"
- // Merge all partition files into a single file.
- segmentFileName =
- mergedLoadNumber + "_" + carbonLoadModel.getFactTimeStamp
- val segmentFile = SegmentFileStore
- .mergeSegmentFiles(readPath,
- segmentFileName,
- CarbonTablePath.getSegmentFilesLocation(carbonLoadModel.getTablePath))
- if (segmentFile != null) {
- SegmentFileStore
- .moveFromTempFolder(segmentFile,
- carbonLoadModel.getFactTimeStamp + ".tmp",
- carbonLoadModel.getTablePath)
+ if (isMergeIndexEnabled) {
+ val segmentTmpFileName = carbonLoadModel.getFactTimeStamp + CarbonTablePath.SEGMENT_EXT
+ segmentFileName = mergedLoadNumber + "_" + segmentTmpFileName
+ val segmentTmpFile = FileFactory.getCarbonFile(
+ CarbonTablePath.getSegmentFilePath(carbonTable.getTablePath, segmentTmpFileName))
+ if (!segmentTmpFile.renameForce(
+ CarbonTablePath.getSegmentFilePath(carbonTable.getTablePath, segmentFileName))) {
+ throw new Exception(s"Rename segment file from ${segmentTmpFileName} " +
+ s"to ${segmentFileName} failed.")
+ }
+ } else {
+ val readPath =
+ CarbonTablePath.getSegmentFilesLocation(carbonLoadModel.getTablePath) +
+ CarbonCommonConstants.FILE_SEPARATOR + carbonLoadModel.getFactTimeStamp + ".tmp"
+ // Merge all partition files into a single file.
+ segmentFileName =
+ mergedLoadNumber + "_" + carbonLoadModel.getFactTimeStamp
+ val segmentFile = SegmentFileStore
Review comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4104: [CARBONDATA-4146]Query fails and the error message "unable to get file status" is displayed. query is normal after the "drop metacache on table" command is executed.
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4104:
URL: https://github.com/apache/carbondata/pull/4104#issuecomment-797499421
Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12444/job/ApacheCarbon_PR_Builder_2.4.5/3799/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] akashrn5 commented on pull request #4104: [CARBONDATA-4146]Query fails and the error message "unable to get file status" is displayed. query is normal after the "drop metacache on table" command is executed.
Posted by GitBox <gi...@apache.org>.
akashrn5 commented on pull request #4104:
URL: https://github.com/apache/carbondata/pull/4104#issuecomment-804763379
LGTM
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] liuhe0702 commented on pull request #4104: [CARBONDATA-4146]Query fails and the error message "unable to get file status" is displayed. query is normal after the "drop metacache on table" command is executed.
Posted by GitBox <gi...@apache.org>.
liuhe0702 commented on pull request #4104:
URL: https://github.com/apache/carbondata/pull/4104#issuecomment-803984958
retest this please
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4104: [CARBONDATA-4146]Query fails and the error message "unable to get file status" is displayed. query is normal after the "drop metacache on table" command is executed.
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4104:
URL: https://github.com/apache/carbondata/pull/4104#issuecomment-797315710
Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12444/job/ApacheCarbon_PR_Builder_2.4.5/3795/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] liuhe0702 commented on a change in pull request #4104: [CARBONDATA-4146]Query fails and the error message "unable to get file status" is displayed. query is normal after the "drop metacache on table" command is executed.
Posted by GitBox <gi...@apache.org>.
liuhe0702 commented on a change in pull request #4104:
URL: https://github.com/apache/carbondata/pull/4104#discussion_r597462970
##########
File path: integration/spark/src/main/scala/org/apache/carbondata/spark/rdd/CarbonTableCompactor.scala
##########
@@ -296,26 +299,46 @@ class CarbonTableCompactor(
if (finalMergeStatus) {
val mergedLoadNumber = CarbonDataMergerUtil.getLoadNumberFromLoadName(mergedLoadName)
- var segmentFilesForIUDCompact = new util.ArrayList[Segment]()
var segmentFileName: String = null
+
+ val isMergeIndexEnabled = CarbonProperties.getInstance().getProperty(
+ CarbonCommonConstants.CARBON_MERGE_INDEX_IN_SEGMENT,
+ CarbonCommonConstants.CARBON_MERGE_INDEX_IN_SEGMENT_DEFAULT).toBoolean
+
+ if (compactionType != CompactionType.IUD_DELETE_DELTA && isMergeIndexEnabled) {
+ MergeIndexUtil.mergeIndexFilesOnCompaction(compactionCallableModel)
+ }
+
if (carbonTable.isHivePartitionTable) {
- val readPath =
- CarbonTablePath.getSegmentFilesLocation(carbonLoadModel.getTablePath) +
- CarbonCommonConstants.FILE_SEPARATOR + carbonLoadModel.getFactTimeStamp + ".tmp"
- // Merge all partition files into a single file.
- segmentFileName =
- mergedLoadNumber + "_" + carbonLoadModel.getFactTimeStamp
- val segmentFile = SegmentFileStore
- .mergeSegmentFiles(readPath,
- segmentFileName,
- CarbonTablePath.getSegmentFilesLocation(carbonLoadModel.getTablePath))
- if (segmentFile != null) {
- SegmentFileStore
- .moveFromTempFolder(segmentFile,
- carbonLoadModel.getFactTimeStamp + ".tmp",
- carbonLoadModel.getTablePath)
+ if (isMergeIndexEnabled) {
+ val segmentTmpFileName = carbonLoadModel.getFactTimeStamp + CarbonTablePath.SEGMENT_EXT
+ segmentFileName = mergedLoadNumber + "_" + segmentTmpFileName
+ val segmentTmpFile = FileFactory.getCarbonFile(
+ CarbonTablePath.getSegmentFilePath(carbonTable.getTablePath, segmentTmpFileName))
+ if (!segmentTmpFile.renameForce(
+ CarbonTablePath.getSegmentFilePath(carbonTable.getTablePath, segmentFileName))) {
+ throw new Exception(s"Rename segment file from ${segmentTmpFileName} " +
+ s"to ${segmentFileName} failed.")
+ }
+ } else {
+ val readPath =
+ CarbonTablePath.getSegmentFilesLocation(carbonLoadModel.getTablePath) +
+ CarbonCommonConstants.FILE_SEPARATOR + carbonLoadModel.getFactTimeStamp + ".tmp"
+ // Merge all partition files into a single file.
+ segmentFileName =
+ mergedLoadNumber + "_" + carbonLoadModel.getFactTimeStamp
Review comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4104: [CARBONDATA-4146]Query fails and the error message "unable to get file status" is displayed. query is normal after the "drop metacache on table" command is executed.
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4104:
URL: https://github.com/apache/carbondata/pull/4104#issuecomment-803623956
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/5059/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] liuhe0702 commented on a change in pull request #4104: [CARBONDATA-4146]Query fails and the error message "unable to get file status" is displayed. query is normal after the "drop metacache on table" command is executed.
Posted by GitBox <gi...@apache.org>.
liuhe0702 commented on a change in pull request #4104:
URL: https://github.com/apache/carbondata/pull/4104#discussion_r597497963
##########
File path: processing/src/main/java/org/apache/carbondata/processing/merger/CompactionResultSortProcessor.java
##########
@@ -190,7 +200,7 @@ public boolean execute(List<RawResultIterator> unsortedResultIteratorList,
LOGGER.error(e.getLocalizedMessage(), e);
throw e;
} finally {
- if (partitionSpec != null) {
Review comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] liuhe0702 commented on pull request #4104: [CARBONDATA-4146]Query fails and the error message "unable to get file status" is displayed. query is normal after the "drop metacache on table" command is executed.
Posted by GitBox <gi...@apache.org>.
liuhe0702 commented on pull request #4104:
URL: https://github.com/apache/carbondata/pull/4104#issuecomment-796741652
add to whitelist
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4104: [CARBONDATA-4146]Query fails and the error message "unable to get file status" is displayed. query is normal after the "drop metacache on table" command is executed.
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4104:
URL: https://github.com/apache/carbondata/pull/4104#issuecomment-797237859
Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12444/job/ApacheCarbon_PR_Builder_2.4.5/3794/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4104: [CARBONDATA-4146]Query fails and the error message "unable to get file status" is displayed. query is normal after the "drop metacache on table" command is executed.
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4104:
URL: https://github.com/apache/carbondata/pull/4104#issuecomment-803936430
Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/3323/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] akashrn5 commented on pull request #4104: [CARBONDATA-4146]Query fails and the error message "unable to get file status" is displayed. query is normal after the "drop metacache on table" command is executed.
Posted by GitBox <gi...@apache.org>.
akashrn5 commented on pull request #4104:
URL: https://github.com/apache/carbondata/pull/4104#issuecomment-800031297
@liuhe0702 please make the jira as duplicate and comment there as its handled in #3988 and close both jira and PR, thanks
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] asfgit closed pull request #4104: [CARBONDATA-4146]Query fails and the error message "unable to get file status" is displayed. query is normal after the "drop metacache on table" command is executed.
Posted by GitBox <gi...@apache.org>.
asfgit closed pull request #4104:
URL: https://github.com/apache/carbondata/pull/4104
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] liuhe0702 commented on pull request #4104: [CARBONDATA-4146]Query fails and the error message "unable to get file status" is displayed. query is normal after the "drop metacache on table" command is executed.
Posted by GitBox <gi...@apache.org>.
liuhe0702 commented on pull request #4104:
URL: https://github.com/apache/carbondata/pull/4104#issuecomment-803737267
retest this please
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] ShreelekhyaG commented on pull request #4104: [CARBONDATA-4146]Query fails and the error message "unable to get file status" is displayed. query is normal after the "drop metacache on table" command is executed.
Posted by GitBox <gi...@apache.org>.
ShreelekhyaG commented on pull request #4104:
URL: https://github.com/apache/carbondata/pull/4104#issuecomment-800030287
Yes @akashrn5/ @liuhe0702 , the same scenario being handled in PR #3988
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4104: [CARBONDATA-4146]Query fails and the error message "unable to get file status" is displayed. query is normal after the "drop metacache on table" command is executed.
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4104:
URL: https://github.com/apache/carbondata/pull/4104#issuecomment-797190327
Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12444/job/ApacheCarbonPRBuilder2.3/5558/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] liuhe0702 commented on a change in pull request #4104: [CARBONDATA-4146]Query fails and the error message "unable to get file status" is displayed. query is normal after the "drop metacache on table" command is executed.
Posted by GitBox <gi...@apache.org>.
liuhe0702 commented on a change in pull request #4104:
URL: https://github.com/apache/carbondata/pull/4104#discussion_r597382404
##########
File path: integration/spark/src/main/scala/org/apache/carbondata/spark/rdd/CarbonTableCompactor.scala
##########
@@ -296,26 +299,50 @@ class CarbonTableCompactor(
if (finalMergeStatus) {
val mergedLoadNumber = CarbonDataMergerUtil.getLoadNumberFromLoadName(mergedLoadName)
- var segmentFilesForIUDCompact = new util.ArrayList[Segment]()
var segmentFileName: String = null
+
+ val mergeIndex = CarbonProperties.getInstance().getProperty(
+ CarbonCommonConstants.CARBON_MERGE_INDEX_IN_SEGMENT,
+ CarbonCommonConstants.CARBON_MERGE_INDEX_IN_SEGMENT_DEFAULT).toBoolean
+
+ if (compactionType != CompactionType.IUD_DELETE_DELTA && mergeIndex) {
+ MergeIndexUtil.mergeIndexFilesOnCompaction(compactionCallableModel)
+ }
+
if (carbonTable.isHivePartitionTable) {
- val readPath =
- CarbonTablePath.getSegmentFilesLocation(carbonLoadModel.getTablePath) +
- CarbonCommonConstants.FILE_SEPARATOR + carbonLoadModel.getFactTimeStamp + ".tmp"
- // Merge all partition files into a single file.
- segmentFileName =
- mergedLoadNumber + "_" + carbonLoadModel.getFactTimeStamp
- val segmentFile = SegmentFileStore
- .mergeSegmentFiles(readPath,
- segmentFileName,
- CarbonTablePath.getSegmentFilesLocation(carbonLoadModel.getTablePath))
- if (segmentFile != null) {
- SegmentFileStore
- .moveFromTempFolder(segmentFile,
- carbonLoadModel.getFactTimeStamp + ".tmp",
- carbonLoadModel.getTablePath)
+ if (mergeIndex) {
+ val segmentTmpFileName = carbonLoadModel.getFactTimeStamp + CarbonTablePath.SEGMENT_EXT
+ segmentFileName = mergedLoadNumber + "_" + segmentTmpFileName
+ val segmentTmpFile = FileFactory.getCarbonFile(
+ CarbonTablePath.getSegmentFilePath(carbonTable.getTablePath, segmentTmpFileName))
+ if (!segmentTmpFile.renameForce(
+ CarbonTablePath.getSegmentFilePath(carbonTable.getTablePath, segmentFileName))) {
+ throw new Exception(s"Rename segment file from ${segmentTmpFileName} " +
+ s"to ${segmentFileName} failed.")
+ }
+ val tmpPath =
Review comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4104: [CARBONDATA-4146]Query fails and the error message "unable to get file status" is displayed. query is normal after the "drop metacache on table" command is executed.
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4104:
URL: https://github.com/apache/carbondata/pull/4104#issuecomment-797426785
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12444/job/ApacheCarbonPRBuilder2.3/5563/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] liuhe0702 commented on pull request #4104: [CARBONDATA-4146]Query fails and the error message "unable to get file status" is displayed. query is normal after the "drop metacache on table" command is executed.
Posted by GitBox <gi...@apache.org>.
liuhe0702 commented on pull request #4104:
URL: https://github.com/apache/carbondata/pull/4104#issuecomment-803867653
retest this please
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org