[jira] [Commented] (CARBONDATA-3471) Spark query carbondata error reporting

["Zhichao Zhang (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/CARBONDATA-3471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16886819#comment-16886819 ] 

Zhichao  Zhang commented on CARBONDATA-3471:
--------------------------------------------

[~tianyouyangying], you found that the content of file 'Metadata/segments/1471_time.segment' is below:

{"locationMap":\{"/Fact/Part0/Segment_1471":{"files":[],"partitions":[],"status":"Success","mergeFileName":"1471_1562963281071.carbonindexmerge","isRelative":true}}}

 

but in the dir Fact/Part0/Segment_1471, there is just a file '1471_1562963281071.carbonindexmerge', no carbondata file, right?

 

> Spark query carbondata error reporting
> --------------------------------------
>
>                 Key: CARBONDATA-3471
>                 URL: https://issues.apache.org/jira/browse/CARBONDATA-3471
>             Project: CarbonData
>          Issue Type: Bug
>          Components: data-query
>    Affects Versions: 1.5.3
>         Environment: cdh5.14.x spark2.3.2 hadoop2.6
>            Reporter: tianyou
>            Priority: Major
>
> Data tables are stored every hour ,delete segment clean file for this table every night.
> It has been running steadily for more than a month.
> But:Now query for error reporting.
> error:
>      caused by: java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
>  at java.util.ArrayList.rangeCheck(ArrayList.java:657)
>  at java.util.ArrayList.get(ArrayList.java:433)
>  at org.apache.carbondata.core.indexstore.blockletindex.BlockletDataMapFactory.getSegmentProperties(BlockletDataMapFactory.java:376)
>  at org.apache.carbondata.core.datamap.TableDataMap.pruneWithFilter(TableDataMap.java:195)
>  at org.apache.carbondata.core.datamap.TableDataMap.prune(TableDataMap.java:171)
>  at org.apache.carbondata.hadoop.api.CarbonInputFormat.getPrunedBlocklets(CarbonInputFormat.java:491)
>  at org.apache.carbondata.hadoop.api.CarbonInputFormat.getDataBlocksOfSegment(CarbonInputFormat.java:414)
>  at org.apache.carbondata.hadoop.api.CarbonTableInputFormat.getSplits(CarbonTableInputFormat.java:494)
>  at org.apache.carbondata.hadoop.api.CarbonTableInputFormat.getSplits(CarbonTableInputFormat.java:218)
>  at org.apache.carbondata.spark.rdd.CarbonScanRDD.internalGetPartitions(CarbonScanRDD.scala:129)
>  at org.apache.carbondata.spark.rdd.CarbonRDD.getPartitions(CarbonRDD.scala:66)
>  at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:253)
>  at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:251)
>  at scala.Option.getOrElse(Option.scala:121)
>  at org.apache.spark.rdd.RDD.partitions(RDD.scala:251)
>  at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:46)
>  at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:253)
>  at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:251)
>  at scala.Option.getOrElse(Option.scala:121)
>  at org.apache.spark.rdd.RDD.partitions(RDD.scala:251)
>  at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:46)
>  at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:253)
>  at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:251)
>  at scala.Option.g



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)