You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@carbondata.apache.org by "weifan (JIRA)" <ji...@apache.org> on 2019/01/29 02:47:00 UTC
[jira] [Updated] (CARBONDATA-3279) where i create datamap using
lucene
[ https://issues.apache.org/jira/browse/CARBONDATA-3279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
weifan updated CARBONDATA-3279:
-------------------------------
Affects Version/s: 1.5.0
Description:
create datamap carbondata_mix6_test on table carbondata_mix6 using 'lucene' DMPROPERTIES('INDEX_COLUMNS'='rk','SPLIT_BLOCKLET'='true');
exception:
Error: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 23.0 failed 4 times, most recent failure: Lost task 0.3 in stage 23.0 (TID 394, node29, executor 1): java.nio.channels.ClosedChannelException
at org.apache.hadoop.hdfs.DFSOutputStream.checkClosed(DFSOutputStream.java:1622)
at org.apache.hadoop.fs.FSOutputSummer.write(FSOutputSummer.java:104)
at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:58)
at java.io.DataOutputStream.write(DataOutputStream.java:107)
at java.util.zip.CheckedOutputStream.write(CheckedOutputStream.java:73)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
at org.apache.lucene.store.OutputStreamIndexOutput.close(OutputStreamIndexOutput.java:68)
at org.apache.lucene.util.IOUtils.close(IOUtils.java:89)
at org.apache.lucene.util.IOUtils.close(IOUtils.java:76)
at org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter.close(BlockTreeTermsWriter.java:1028)
at org.apache.lucene.util.IOUtils.close(IOUtils.java:89)
at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter.close(PerFieldPostingsFormat.java:241)
at org.apache.lucene.util.IOUtils.close(IOUtils.java:89)
at org.apache.lucene.util.IOUtils.close(IOUtils.java:76)
at org.apache.lucene.index.FreqProxTermsWriter.flush(FreqProxTermsWriter.java:111)
at org.apache.lucene.index.DefaultIndexingChain.flush(DefaultIndexingChain.java:134)
at org.apache.lucene.index.DocumentsWriterPerThread.flush(DocumentsWriterPerThread.java:443)
at org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:539)
at org.apache.lucene.index.DocumentsWriter.flushAllThreads(DocumentsWriter.java:653)
at org.apache.lucene.index.IndexWriter.doFlush(IndexWriter.java:3358)
at org.apache.lucene.index.IndexWriter.flush(IndexWriter.java:3333)
at org.apache.lucene.index.IndexWriter.shutdown(IndexWriter.java:1117)
at org.apache.lucene.index.IndexWriter.close(IndexWriter.java:1162)
at org.apache.carbondata.datamap.lucene.LuceneDataMapBuilder.close(LuceneDataMapBuilder.java:171)
at org.apache.carbondata.datamap.lucene.LuceneDataMapBuilder.addRow(LuceneDataMapBuilder.java:136)
at org.apache.carbondata.datamap.IndexDataMapRebuildRDD.internalCompute(IndexDataMapRebuildRDD.scala:394)
at org.apache.carbondata.spark.rdd.CarbonRDD.compute(CarbonRDD.scala:82)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
at org.apache.spark.scheduler.Task.run(Task.scala:99)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:282)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Suppressed: java.nio.channels.ClosedChannelException
at org.apache.hadoop.hdfs.DFSOutputStream.checkClosed(DFSOutputStream.java:1622)
at org.apache.hadoop.fs.FSOutputSummer.write(FSOutputSummer.java:104)
at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:58)
at java.io.DataOutputStream.write(DataOutputStream.java:107)
at java.util.zip.CheckedOutputStream.write(CheckedOutputStream.java:73)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
at java.io.FilterOutputStream.close(FilterOutputStream.java:158)
at org.apache.lucene.store.OutputStreamIndexOutput.close(OutputStreamIndexOutput.java:70)
... 28 more
Suppressed: java.nio.channels.ClosedChannelException
at org.apache.hadoop.hdfs.DFSOutputStream.checkClosed(DFSOutputStream.java:1622)
at org.apache.hadoop.fs.FSOutputSummer.write(FSOutputSummer.java:104)
at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:58)
at java.io.DataOutputStream.write(DataOutputStream.java:107)
at java.util.zip.CheckedOutputStream.write(CheckedOutputStream.java:73)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
at org.apache.lucene.store.OutputStreamIndexOutput.getChecksum(OutputStreamIndexOutput.java:80)
at org.apache.lucene.codecs.CodecUtil.writeCRC(CodecUtil.java:542)
at org.apache.lucene.codecs.CodecUtil.writeFooter(CodecUtil.java:390)
at org.apache.lucene.codecs.lucene50.Lucene50PostingsWriter.close(Lucene50PostingsWriter.java:469)
... 28 more
Component/s: spark-integration
Summary: where i create datamap using lucene (was: where i create datamap using lucene:)
> where i create datamap using lucene
> -----------------------------------
>
> Key: CARBONDATA-3279
> URL: https://issues.apache.org/jira/browse/CARBONDATA-3279
> Project: CarbonData
> Issue Type: Bug
> Components: spark-integration
> Affects Versions: 1.5.0
> Reporter: weifan
> Priority: Major
>
> create datamap carbondata_mix6_test on table carbondata_mix6 using 'lucene' DMPROPERTIES('INDEX_COLUMNS'='rk','SPLIT_BLOCKLET'='true');
>
> exception:
>
> Error: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 23.0 failed 4 times, most recent failure: Lost task 0.3 in stage 23.0 (TID 394, node29, executor 1): java.nio.channels.ClosedChannelException
> at org.apache.hadoop.hdfs.DFSOutputStream.checkClosed(DFSOutputStream.java:1622)
> at org.apache.hadoop.fs.FSOutputSummer.write(FSOutputSummer.java:104)
> at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:58)
> at java.io.DataOutputStream.write(DataOutputStream.java:107)
> at java.util.zip.CheckedOutputStream.write(CheckedOutputStream.java:73)
> at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
> at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
> at org.apache.lucene.store.OutputStreamIndexOutput.close(OutputStreamIndexOutput.java:68)
> at org.apache.lucene.util.IOUtils.close(IOUtils.java:89)
> at org.apache.lucene.util.IOUtils.close(IOUtils.java:76)
> at org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter.close(BlockTreeTermsWriter.java:1028)
> at org.apache.lucene.util.IOUtils.close(IOUtils.java:89)
> at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter.close(PerFieldPostingsFormat.java:241)
> at org.apache.lucene.util.IOUtils.close(IOUtils.java:89)
> at org.apache.lucene.util.IOUtils.close(IOUtils.java:76)
> at org.apache.lucene.index.FreqProxTermsWriter.flush(FreqProxTermsWriter.java:111)
> at org.apache.lucene.index.DefaultIndexingChain.flush(DefaultIndexingChain.java:134)
> at org.apache.lucene.index.DocumentsWriterPerThread.flush(DocumentsWriterPerThread.java:443)
> at org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:539)
> at org.apache.lucene.index.DocumentsWriter.flushAllThreads(DocumentsWriter.java:653)
> at org.apache.lucene.index.IndexWriter.doFlush(IndexWriter.java:3358)
> at org.apache.lucene.index.IndexWriter.flush(IndexWriter.java:3333)
> at org.apache.lucene.index.IndexWriter.shutdown(IndexWriter.java:1117)
> at org.apache.lucene.index.IndexWriter.close(IndexWriter.java:1162)
> at org.apache.carbondata.datamap.lucene.LuceneDataMapBuilder.close(LuceneDataMapBuilder.java:171)
> at org.apache.carbondata.datamap.lucene.LuceneDataMapBuilder.addRow(LuceneDataMapBuilder.java:136)
> at org.apache.carbondata.datamap.IndexDataMapRebuildRDD.internalCompute(IndexDataMapRebuildRDD.scala:394)
> at org.apache.carbondata.spark.rdd.CarbonRDD.compute(CarbonRDD.scala:82)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
> at org.apache.spark.scheduler.Task.run(Task.scala:99)
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:282)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Suppressed: java.nio.channels.ClosedChannelException
> at org.apache.hadoop.hdfs.DFSOutputStream.checkClosed(DFSOutputStream.java:1622)
> at org.apache.hadoop.fs.FSOutputSummer.write(FSOutputSummer.java:104)
> at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:58)
> at java.io.DataOutputStream.write(DataOutputStream.java:107)
> at java.util.zip.CheckedOutputStream.write(CheckedOutputStream.java:73)
> at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
> at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
> at java.io.FilterOutputStream.close(FilterOutputStream.java:158)
> at org.apache.lucene.store.OutputStreamIndexOutput.close(OutputStreamIndexOutput.java:70)
> ... 28 more
> Suppressed: java.nio.channels.ClosedChannelException
> at org.apache.hadoop.hdfs.DFSOutputStream.checkClosed(DFSOutputStream.java:1622)
> at org.apache.hadoop.fs.FSOutputSummer.write(FSOutputSummer.java:104)
> at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:58)
> at java.io.DataOutputStream.write(DataOutputStream.java:107)
> at java.util.zip.CheckedOutputStream.write(CheckedOutputStream.java:73)
> at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
> at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
> at org.apache.lucene.store.OutputStreamIndexOutput.getChecksum(OutputStreamIndexOutput.java:80)
> at org.apache.lucene.codecs.CodecUtil.writeCRC(CodecUtil.java:542)
> at org.apache.lucene.codecs.CodecUtil.writeFooter(CodecUtil.java:390)
> at org.apache.lucene.codecs.lucene50.Lucene50PostingsWriter.close(Lucene50PostingsWriter.java:469)
> ... 28 more
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)