You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@carbondata.apache.org by GitBox <gi...@apache.org> on 2021/03/10 07:59:27 UTC
[GitHub] [carbondata] liuhe0702 opened a new pull request #4102: [CARBONDATA-4144] After the alter table xxx compact command is executed, the index size of the segment is 0, and an error is rep…orted while quering
liuhe0702 opened a new pull request #4102:
URL: https://github.com/apache/carbondata/pull/4102
Why is this PR needed?
When compact operation fails, the segment lock is not released. Failed to get segment lock when compaction is executed again.
In this case, the merged segment status is set to success and the segmentfile is xxx_null.segments and the vaule of indexsize is 0.
What changes were proposed in this PR?
If an exception occurs, release the obtained segment locks.
If getting segment locks failed, not update the merged segment status.
Does this PR introduce any user interface change?
No
Is any new testcase added?
No
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] akashrn5 commented on pull request #4102: [CARBONDATA-4144] During compaction, the segment lock of SI table is not released in abnormal scenarios.
Posted by GitBox <gi...@apache.org>.
akashrn5 commented on pull request #4102:
URL: https://github.com/apache/carbondata/pull/4102#issuecomment-802727517
#3988 PR takes time to stabilize, so merging this to avoid issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4102: [CARBONDATA-4144] After the alter table xxx compact command is executed, the index size of the segment is 0, and an error is rep…orted while quering
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4102:
URL: https://github.com/apache/carbondata/pull/4102#issuecomment-796646950
Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12444/job/ApacheCarbon_PR_Builder_2.4.5/3786/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4102: [CARBONDATA-4144] After the alter table xxx compact command is executed, the index size of the segment is 0, and an error is rep…orted while quering
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4102:
URL: https://github.com/apache/carbondata/pull/4102#issuecomment-796719500
Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12444/job/ApacheCarbon_PR_Builder_2.4.5/3789/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4102: [CARBONDATA-4144] After the alter table xxx compact command is executed, the index size of the segment is 0, and an error is rep…orted while quering
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4102:
URL: https://github.com/apache/carbondata/pull/4102#issuecomment-796526090
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12444/job/ApacheCarbonPRBuilder2.3/5547/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4102: [CARBONDATA-4144] After the alter table xxx compact command is executed, the index size of the segment is 0, and an error is rep…orted while quering
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4102:
URL: https://github.com/apache/carbondata/pull/4102#issuecomment-796536532
Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12444/job/ApacheCarbon_PR_Builder_2.4.5/3782/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] akashrn5 commented on pull request #4102: [CARBONDATA-4144] After the alter table xxx compact command is executed, the index size of the segment is 0, and an error is rep…orted while quering
Posted by GitBox <gi...@apache.org>.
akashrn5 commented on pull request #4102:
URL: https://github.com/apache/carbondata/pull/4102#issuecomment-800183773
@liuhe0702 please shorten the PR title with proper summary and in PR description, i can see just compaction explanation, but the changes are in secondary index files during compaction on main table. As In SI there is nothing as compaction, always its a load, so please give a proper description according to the changes made. thanks
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4102: [CARBONDATA-4144] During compaction, the segment lock of SI table is not released in abnormal scenarios.
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4102:
URL: https://github.com/apache/carbondata/pull/4102#issuecomment-801584412
Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12444/job/ApacheCarbon_PR_Builder_2.4.5/3812/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] akashrn5 commented on a change in pull request #4102: [CARBONDATA-4144] During compaction, the segment lock of SI table is not released in abnormal scenarios.
Posted by GitBox <gi...@apache.org>.
akashrn5 commented on a change in pull request #4102:
URL: https://github.com/apache/carbondata/pull/4102#discussion_r596861202
##########
File path: integration/spark/src/main/scala/org/apache/spark/sql/secondaryindex/load/Compactor.scala
##########
@@ -93,6 +93,9 @@ object Compactor {
segmentToSegmentTimestampMap, null,
forceAccessSegment, isCompactionCall = true,
isLoadToFailedSISegments = false)
+ if (segmentLocks.isEmpty) {
Review comment:
please add a log, not comment
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4102: [CARBONDATA-4144] After the alter table xxx compact command is executed, the index size of the segment is 0, and an error is rep…orted while quering
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4102:
URL: https://github.com/apache/carbondata/pull/4102#issuecomment-796714870
Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12444/job/ApacheCarbonPRBuilder2.3/5555/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] akashrn5 commented on pull request #4102: [CARBONDATA-4144] During compaction, the segment lock of SI table is not released in abnormal scenarios.
Posted by GitBox <gi...@apache.org>.
akashrn5 commented on pull request #4102:
URL: https://github.com/apache/carbondata/pull/4102#issuecomment-801950416
LGTM, can be merged once build passes
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4102: [CARBONDATA-4144] During compaction, the segment lock of SI table is not released in abnormal scenarios.
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4102:
URL: https://github.com/apache/carbondata/pull/4102#issuecomment-802099779
Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12444/job/ApacheCarbon_PR_Builder_2.4.5/3821/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] akashrn5 commented on pull request #4102: [CARBONDATA-4144] During compaction, the segment lock of SI table is not released in abnormal scenarios.
Posted by GitBox <gi...@apache.org>.
akashrn5 commented on pull request #4102:
URL: https://github.com/apache/carbondata/pull/4102#issuecomment-801920979
LGTM
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] brijoobopanna commented on pull request #4102: [CARBONDATA-4144] After the alter table xxx compact command is executed, the index size of the segment is 0, and an error is rep…orted while quering
Posted by GitBox <gi...@apache.org>.
brijoobopanna commented on pull request #4102:
URL: https://github.com/apache/carbondata/pull/4102#issuecomment-796452330
add to whitelist
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4102: [CARBONDATA-4144] During compaction, the segment lock of SI table is not released in abnormal scenarios.
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4102:
URL: https://github.com/apache/carbondata/pull/4102#issuecomment-802070582
Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12444/job/ApacheCarbonPRBuilder2.3/5587/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] ydvpankaj99 commented on pull request #4102: [CARBONDATA-4144] After the alter table xxx compact command is executed, the index size of the segment is 0, and an error is rep…orted while quering
Posted by GitBox <gi...@apache.org>.
ydvpankaj99 commented on pull request #4102:
URL: https://github.com/apache/carbondata/pull/4102#issuecomment-795234232
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] liuhe0702 commented on pull request #4102: [CARBONDATA-4144] After the alter table xxx compact command is executed, the index size of the segment is 0, and an error is rep…orted while quering
Posted by GitBox <gi...@apache.org>.
liuhe0702 commented on pull request #4102:
URL: https://github.com/apache/carbondata/pull/4102#issuecomment-796645900
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] liuhe0702 commented on a change in pull request #4102: [CARBONDATA-4144] During compaction, the segment lock of SI table is not released in abnormal scenarios.
Posted by GitBox <gi...@apache.org>.
liuhe0702 commented on a change in pull request #4102:
URL: https://github.com/apache/carbondata/pull/4102#discussion_r596893792
##########
File path: integration/spark/src/main/scala/org/apache/spark/sql/secondaryindex/load/Compactor.scala
##########
@@ -93,6 +93,9 @@ object Compactor {
segmentToSegmentTimestampMap, null,
forceAccessSegment, isCompactionCall = true,
isLoadToFailedSISegments = false)
+ if (segmentLocks.isEmpty) {
Review comment:
done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] akashrn5 commented on a change in pull request #4102: [CARBONDATA-4144] During compaction, the segment lock of SI table is not released in abnormal scenarios.
Posted by GitBox <gi...@apache.org>.
akashrn5 commented on a change in pull request #4102:
URL: https://github.com/apache/carbondata/pull/4102#discussion_r595866555
##########
File path: integration/spark/src/main/scala/org/apache/spark/sql/secondaryindex/load/Compactor.scala
##########
@@ -93,6 +93,9 @@ object Compactor {
segmentToSegmentTimestampMap, null,
forceAccessSegment, isCompactionCall = true,
isLoadToFailedSISegments = false)
+ if (segmentLocks.isEmpty) {
Review comment:
please add a log here saying that the loading of compacted segment into SI table failed, couldn't acquire segment lock on the specific segment
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] asfgit closed pull request #4102: [CARBONDATA-4144] During compaction, the segment lock of SI table is not released in abnormal scenarios.
Posted by GitBox <gi...@apache.org>.
asfgit closed pull request #4102:
URL: https://github.com/apache/carbondata/pull/4102
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4102: [CARBONDATA-4144] During compaction, the segment lock of SI table is not released in abnormal scenarios.
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4102:
URL: https://github.com/apache/carbondata/pull/4102#issuecomment-801582139
Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12444/job/ApacheCarbonPRBuilder2.3/5578/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] liuhe0702 commented on pull request #4102: [CARBONDATA-4144] After the alter table xxx compact command is executed, the index size of the segment is 0, and an error is rep…orted while quering
Posted by GitBox <gi...@apache.org>.
liuhe0702 commented on pull request #4102:
URL: https://github.com/apache/carbondata/pull/4102#issuecomment-796564691
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4102: [CARBONDATA-4144] After the alter table xxx compact command is executed, the index size of the segment is 0, and an error is rep…orted while quering
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4102:
URL: https://github.com/apache/carbondata/pull/4102#issuecomment-796577615
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12444/job/ApacheCarbonPRBuilder2.3/5552/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] akashrn5 removed a comment on pull request #4102: [CARBONDATA-4144] During compaction, the segment lock of SI table is not released in abnormal scenarios.
Posted by GitBox <gi...@apache.org>.
akashrn5 removed a comment on pull request #4102:
URL: https://github.com/apache/carbondata/pull/4102#issuecomment-801920979
LGTM
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] liuhe0702 commented on a change in pull request #4102: [CARBONDATA-4144] During compaction, the segment lock of SI table is not released in abnormal scenarios.
Posted by GitBox <gi...@apache.org>.
liuhe0702 commented on a change in pull request #4102:
URL: https://github.com/apache/carbondata/pull/4102#discussion_r596482836
##########
File path: integration/spark/src/main/scala/org/apache/spark/sql/secondaryindex/load/Compactor.scala
##########
@@ -93,6 +93,9 @@ object Compactor {
segmentToSegmentTimestampMap, null,
forceAccessSegment, isCompactionCall = true,
isLoadToFailedSISegments = false)
+ if (segmentLocks.isEmpty) {
Review comment:
done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4102: [CARBONDATA-4144] After the alter table xxx compact command is executed, the index size of the segment is 0, and an error is rep…orted while quering
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4102:
URL: https://github.com/apache/carbondata/pull/4102#issuecomment-795093613
Can one of the admins verify this patch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org