Re: ERROR sortdata.SortTempFileChunkHolder when loading data

[sounak <so...@gmail.com>]

Hi,

Can you please try your test case in spark-2.1 or spark-2.2? In case it is
getting reproduce in these versions then please provide us the test case to
reproduce. Also specify if there is any specific configuration change you
have made.

On Tue, Jan 2, 2018 at 3:16 PM, dylan <dy...@163.com> wrote:

>  hello all:
>   when i loading data on carbondata1.2,spark1.6 on yarn,
>   i get this error:
> 18/01/02 17:27:25 INFO sortdata.SortDataRows: pool-24-thread-1 Time taken
> to
> sort and write sort temp file
> /hadoop7/yarn/nm/usercache/dba/appcache/application_
> 1513058773052_20157/container_e38_1513058773052_20157_01_
> 000002/tmp/7020573636328672_0/test/table1/Fact/Part0/
> Segment_0/0/sortrowtmp/mobeye_o2o_lbs_frequency_daily_20171229_228_
> 1e8bcd325f9343348172b7e6fb10b3f77020814710024274.sorttemp
> is: 347
> 18/01/02 17:27:28 INFO sortdata.SortDataRows: pool-24-thread-1 Time taken
> to
> sort and write sort temp file
> /hadoop7/yarn/nm/usercache/dba/appcache/application_
> 1513058773052_20157/container_e38_1513058773052_20157_01_
> 000002/tmp/7020573636328672_0/test/table1/Fact/Part0/
> Segment_0/0/sortrowtmp/mobeye_o2o_lbs_frequency_daily_20171229_228_
> 1e8bcd325f9343348172b7e6fb10b3f77020818030809514.sorttemp
> is: 334
> 18/01/02 17:27:32 INFO sortdata.SortDataRows: pool-24-thread-1 Time taken
> to
> sort and write sort temp file
> /hadoop7/yarn/nm/usercache/dba/appcache/application_
> 1513058773052_20157/container_e38_1513058773052_20157_01_
> 000002/tmp/7020573636328672_0/test/table1/Fact/Part0/
> Segment_0/0/sortrowtmp/mobeye_o2o_lbs_frequency_daily_20171229_228_
> 1e8bcd325f9343348172b7e6fb10b3f77020821312442170.sorttemp
> is: 348
> 18/01/02 17:27:34 INFO newflow.AbstractDataLoadProcessorStep: Thread-359
> Rows processed in step Sort Processor : 7072000
> 18/01/02 17:27:34 INFO newflow.AbstractDataLoadProcessorStep: Thread-358
> Rows processed in step Data Writer : 0
> 18/01/02 17:27:34 INFO newflow.AbstractDataLoadProcessorStep: Thread-360
> Rows processed in step Data Converter : 7072000
> 18/01/02 17:27:34 INFO newflow.AbstractDataLoadProcessorStep: Thread-361
> Rows processed in step Input Processor : 7073000
> 18/01/02 17:27:35 INFO sortdata.SortDataRows: pool-24-thread-1 Time taken
> to
> sort and write sort temp file
> /hadoop7/yarn/nm/usercache/dba/appcache/application_
> 1513058773052_20157/container_e38_1513058773052_20157_01_
> 000002/tmp/7020573636328672_0/test/table1/Fact/Part0/
> Segment_0/0/sortrowtmp/mobeye_o2o_lbs_frequency_daily_20171229_228_
> 1e8bcd325f9343348172b7e6fb10b3f77020824678303996.sorttemp
> is: 355
> 18/01/02 17:27:41 INFO sortdata.SortDataRows: pool-24-thread-1 Time taken
> to
> sort and write sort temp file
> /hadoop7/yarn/nm/usercache/dba/appcache/application_
> 1513058773052_20157/container_e38_1513058773052_20157_01_
> 000002/tmp/7020573636328672_0/test/table1/Fact/Part0/
> Segment_0/0/sortrowtmp/mobeye_o2o_lbs_frequency_daily_20171229_228_
> 1e8bcd325f9343348172b7e6fb10b3f77020830413379712.sorttemp
> is: 369
> 18/01/02 17:27:42 INFO sortdata.SortDataRows: Executor task launch worker-3
> File based sorting will be used
> 18/01/02 17:27:42 INFO impl.ParallelReadMergeSorterImpl: Executor task
> launch worker-3 Record Processed For table:
> mobeye_o2o_lbs_frequency_daily_20171229_228_1e8bcd325f9343348172b7e6fb10b3
> f7
> 18/01/02 17:27:42 INFO store.SingleThreadFinalSortFilesMerger: Executor
> task
> launch worker-3 Number of temp file: 16
> 18/01/02 17:27:42 INFO store.SingleThreadFinalSortFilesMerger: Executor
> task
> launch worker-3 File Buffer Size: 655360
> 18/01/02 17:27:42 INFO store.SingleThreadFinalSortFilesMerger: Executor
> task
> launch worker-3 Started adding first record from each file
> 18/01/02 17:27:42 ERROR sortdata.SortTempFileChunkHolder: pool-86-thread-2
> java.lang.NegativeArraySizeException
>         at
> org.apache.carbondata.processing.sortandgroupby.sortdata.
> SortTempFileChunkHolder.getRowFromStream(SortTempFileChunkHolder.java:334)
>         at
> org.apache.carbondata.processing.sortandgroupby.sortdata.
> SortTempFileChunkHolder.prefetchRecordsFromFile(
> SortTempFileChunkHolder.java:518)
>         at
> org.apache.carbondata.processing.sortandgroupby.sortdata.
> SortTempFileChunkHolder.access$500(SortTempFileChunkHolder.java:42)
>         at
> org.apache.carbondata.processing.sortandgroupby.sortdata.
> SortTempFileChunkHolder$DataFetcher.call(SortTempFileChunkHolder.java:497)
>         at
> org.apache.carbondata.processing.sortandgroupby.sortdata.
> SortTempFileChunkHolder.initialise(SortTempFileChunkHolder.java:227)
>         at
> org.apache.carbondata.processing.sortandgroupby.sortdata.
> SortTempFileChunkHolder.initialize(SortTempFileChunkHolder.java:215)
>         at
> org.apache.carbondata.processing.store.SingleThreadFinalSortFilesMerg
> er$2.run(SingleThreadFinalSortFilesMerger.java:210)
>         at
> java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1142)
>         at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
>         at java.lang.Thread.run(Thread.java:745)
> 18/01/02 17:27:42 ERROR sortdata.SortTempFileChunkHolder: pool-86-thread-2
> java.lang.NullPointerException
>         at
> org.apache.carbondata.processing.sortandgroupby.sortdata.
> SortTempFileChunkHolder.initialise(SortTempFileChunkHolder.java:228)
>         at
> org.apache.carbondata.processing.sortandgroupby.sortdata.
> SortTempFileChunkHolder.initialize(SortTempFileChunkHolder.java:215)
>         at
> org.apache.carbondata.processing.store.SingleThreadFinalSortFilesMerg
> er$2.run(SingleThreadFinalSortFilesMerger.java:210)
>         at
> java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1142)
>         at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
>         at java.lang.Thread.run(Thread.java:745)
> 18/01/02 17:27:42 ERROR store.SingleThreadFinalSortFilesMerger:
> pool-86-thread-2
> org.apache.carbondata.processing.sortandgroupby.exception.
> CarbonSortKeyAndGroupByException:
> /hadoop7/yarn/nm/usercache/dba/appcache/application_
> 1513058773052_20157/container_e38_1513058773052_20157_01_
> 000002/tmp/7020573636328672_0/test/table1/Fact/Part0/
> Segment_0/0/sortrowtmp/mobeye_o2o_lbs_frequency_daily_20171229_228_
> 1e8bcd325f9343348172b7e6fb10b3f77020721749691375.merge
> Problem while reading
>         at
> org.apache.carbondata.processing.sortandgroupby.sortdata.
> SortTempFileChunkHolder.initialise(SortTempFileChunkHolder.java:246)
>         at
> org.apache.carbondata.processing.sortandgroupby.sortdata.
> SortTempFileChunkHolder.initialize(SortTempFileChunkHolder.java:215)
>         at
> org.apache.carbondata.processing.store.SingleThreadFinalSortFilesMerg
> er$2.run(SingleThreadFinalSortFilesMerger.java:210)
>         at
> java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1142)
>         at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
>         at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.NullPointerException
>         at
> org.apache.carbondata.processing.sortandgroupby.sortdata.
> SortTempFileChunkHolder.initialise(SortTempFileChunkHolder.java:228)
>         ... 5 more
>
>
>
> --
> Sent from: http://apache-carbondata-dev-mailing-list-archive.1130556.
> n5.nabble.com/
>



-- 
Thanks
Sounak