You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by "stack (JIRA)" <ji...@apache.org> on 2009/12/21 17:36:18 UTC
[jira] Created: (HBASE-2063) For hfileoutputformat, on
timeout/failure/kill clean up half-written hfile
For hfileoutputformat, on timeout/failure/kill clean up half-written hfile
--------------------------------------------------------------------------
Key: HBASE-2063
URL: https://issues.apache.org/jira/browse/HBASE-2063
Project: Hadoop HBase
Issue Type: Bug
Reporter: stack
Below is from mailing list. Read from bottom to top:
{code}
I was going to write that perhaps you needed to turn mapred.reduce.tasks.speculative.execution off, but if enabling it and things work, that would seem to indicate that a our reducer first takes longer than the task timeout maximum and secondly, on failure, we should clean up the hfile.
On the first issue, you are using KeyValueSortReducer? Are your values large? We set reducer status every 100 values. Maybe this is not enough? We should set status more frequently? If you call context setstatus more frequently, do things work w/o speculative execution?
On the second, HFileOutputFormat close will set the metadata on the hfile and then close it. On kill, this code is not being called. Let me see if can do something about that (e.g. register a shutdown hook to clean away incomplete files -- ).
Thanks,
St.Ack
On Sun, Dec 20, 2009 at 11:26 PM, ChingShen <ch...@gmail.com> wrote:
I think I found a way.
I set the "mapred.reduce.tasks.speculative.execution" to true and output
hfiles again, then successfully load hfiles into hbase.
Is it best solution? or HFileOutputFormat bug?
Shen
On Mon, Dec 21, 2009 at 8:25 AM, ChingShen <ch...@gmail.com> wrote:
> Thanks, stack.
>
> I checked this file that isn't empty. But I found that as long as the
> "Killed Task Attempts" > 0 in reduce phase, and run the loadtable.rb script
> to load hfiles then failed.
> How to avoid this problem?
>
> Thanks.
>
> Shen
>
>
> On Sat, Dec 19, 2009 at 3:49 AM, stack <st...@duboce.net> wrote:
>
>> Check the
>> file
>> hdfs://domU-12-31-39-09-C5-54.compute-1.internal/osm2_hfile/Level4/197894389945760574.
>> Is it empty? Was there an error during running of your MR job? Perhaps
>> a
>> task failed?
>>
>> St.Ack
>>
>>
>>
>> On Thu, Dec 17, 2009 at 9:46 PM, ChingShen <ch...@gmail.com>
>> wrote:
>>
>> > Hi,
>> > I use the script loadtable.rb to load my hfiles into hbase, but I got
>> an
>> > exception as below.
>> > Does anyone have any suggestions?
>> >
>> > 09/12/17 23:59:33 INFO loadtable: 18 read firstkey of -3.9290_52.5534
>> from
>> >
>> >
>> hdfs://domU-12-31-39-09-C5-54.compute-1.internal/osm2_hfile/Level4/1978943899457605747
>> > org/apache/hadoop/hbase/io/hfile/HFile.java:1335:in `deserialize':
>> > java.io.IOException: Trailer 'header' is wrong; does the trailer size
>> match
>> > content? (NativeException)
>> > from org/apache/hadoop/hbase/io/hfile/HFile.java:813:in `readTrailer'
>> > from org/apache/hadoop/hbase/io/hfile/HFile.java:758:in
>> `loadFileInfo'
>> > from sun.reflect.GeneratedMethodAccessor7:-1:in `invoke'
>> > from sun/reflect/DelegatingMethodAccessorImpl.java:25:in `invoke'
>> > from java/lang/reflect/Method.java:597:in `invoke'
>> > from org/jruby/javasupport/JavaMethod.java:298:in
>> > `invokeWithExceptionHandling'
>> > from org/jruby/javasupport/JavaMethod.java:259:in `invoke'
>> > from org/jruby/java/invokers/InstanceMethodInvoker.java:36:in `call'
>> > ... 18 levels...
>> > from org/jruby/Main.java:94:in `main'
>> > from loadtable.rb:83:in `each'
>> > from loadtable.rb:83
>> > Complete Java stackTrace
>> > java.io.IOException: Trailer 'header' is wrong; does the trailer size
>> match
>> > content?
>> > at
>> >
>> >
>> org.apache.hadoop.hbase.io.hfile.HFile$FixedFileTrailer.deserialize(HFile.java:1335)
>> > at
>> >
>> org.apache.hadoop.hbase.io.hfile.HFile$Reader.readTrailer(HFile.java:813)
>> > at
>> >
>> org.apache.hadoop.hbase.io.hfile.HFile$Reader.loadFileInfo(HFile.java:758)
>> > at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
>> > at
>> >
>> >
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> > at
>> >
>> >
>> org.jruby.javasupport.JavaMethod.invokeWithExceptionHandling(JavaMethod.java:298)
>> > at org.jruby.javasupport.JavaMethod.invoke(JavaMethod.java:259)
>> > at
>> >
>> >
>> org.jruby.java.invokers.InstanceMethodInvoker.call(InstanceMethodInvoker.java:36)
>> > at
>> > org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:70)
>> > at loadtable.ensure_1$RUBY$__ensure___2(loadtable.rb:86)
>> > at loadtable.block_0$RUBY$__for__(loadtable.rb:85)
>> > at loadtableBlockCallback$block_0$RUBY$__for__xx1.call(Unknown
>> Source)
>> > at org.jruby.runtime.CompiledBlock.yield(CompiledBlock.java:102)
>> > at org.jruby.runtime.Block.yield(Block.java:100)
>> > at
>> org.jruby.java.proxies.ArrayJavaProxy.each(ArrayJavaProxy.java:112)
>> > at
>> >
>> >
>> org.jruby.java.proxies.ArrayJavaProxy$i_method_0_0$RUBYINVOKER$each.call(org/jruby/java/proxies/ArrayJavaProxy$i_method_0_0$RUBYINVOKER$each.gen)
>> > at
>> >
>> >
>> org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:263)
>> > at
>> >
>> >
>> org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:81)
>> > at
>> >
>> >
>> org.jruby.runtime.callsite.CachingCallSite.callIter(CachingCallSite.java:96)
>> > at loadtable.__file__(loadtable.rb:83)
>> > at loadtable.load(loadtable.rb)
>> > at org.jruby.Ruby.runScript(Ruby.java:577)
>> > at org.jruby.Ruby.runNormally(Ruby.java:480)
>> > at org.jruby.Ruby.runFromMain(Ruby.java:354)
>> > at org.jruby.Main.run(Main.java:229)
>> > at org.jruby.Main.run(Main.java:110)
>> > at org.jruby.Main.main(Main.java:94)
>> >
>>
>
>
>
>
--
*****************************************************
Ching-Shen Chen
Advanced Technology Center,
Information & Communications Research Lab.
E-mail: chenchingshen@itri.org.tw
Tel:+886-3-5915542
*****************************************************
{code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-2063) For hfileoutputformat, on
timeout/failure/kill clean up half-written hfile
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-2063:
-------------------------
Fix Version/s: 0.21.0
0.20.4
Moving into 0.20.4.
> For hfileoutputformat, on timeout/failure/kill clean up half-written hfile
> --------------------------------------------------------------------------
>
> Key: HBASE-2063
> URL: https://issues.apache.org/jira/browse/HBASE-2063
> Project: Hadoop HBase
> Issue Type: Bug
> Affects Versions: 0.20.3
> Reporter: stack
> Fix For: 0.20.4, 0.21.0
>
> Attachments: HBASE-2063-v1.patch
>
>
> Below is from mailing list. Read from bottom to top:
> {code}
> I was going to write that perhaps you needed to turn mapred.reduce.tasks.speculative.execution off, but if enabling it and things work, that would seem to indicate that a our reducer first takes longer than the task timeout maximum and secondly, on failure, we should clean up the hfile.
> On the first issue, you are using KeyValueSortReducer? Are your values large? We set reducer status every 100 values. Maybe this is not enough? We should set status more frequently? If you call context setstatus more frequently, do things work w/o speculative execution?
>
> On the second, HFileOutputFormat close will set the metadata on the hfile and then close it. On kill, this code is not being called. Let me see if can do something about that (e.g. register a shutdown hook to clean away incomplete files -- ).
> Thanks,
> St.Ack
> On Sun, Dec 20, 2009 at 11:26 PM, ChingShen <ch...@gmail.com> wrote:
> I think I found a way.
> I set the "mapred.reduce.tasks.speculative.execution" to true and output
> hfiles again, then successfully load hfiles into hbase.
> Is it best solution? or HFileOutputFormat bug?
> Shen
> On Mon, Dec 21, 2009 at 8:25 AM, ChingShen <ch...@gmail.com> wrote:
> > Thanks, stack.
> >
> > I checked this file that isn't empty. But I found that as long as the
> > "Killed Task Attempts" > 0 in reduce phase, and run the loadtable.rb script
> > to load hfiles then failed.
> > How to avoid this problem?
> >
> > Thanks.
> >
> > Shen
> >
> >
> > On Sat, Dec 19, 2009 at 3:49 AM, stack <st...@duboce.net> wrote:
> >
> >> Check the
> >> file
> >> hdfs://domU-12-31-39-09-C5-54.compute-1.internal/osm2_hfile/Level4/197894389945760574.
> >> Is it empty? Was there an error during running of your MR job? Perhaps
> >> a
> >> task failed?
> >>
> >> St.Ack
> >>
> >>
> >>
> >> On Thu, Dec 17, 2009 at 9:46 PM, ChingShen <ch...@gmail.com>
> >> wrote:
> >>
> >> > Hi,
> >> > I use the script loadtable.rb to load my hfiles into hbase, but I got
> >> an
> >> > exception as below.
> >> > Does anyone have any suggestions?
> >> >
> >> > 09/12/17 23:59:33 INFO loadtable: 18 read firstkey of -3.9290_52.5534
> >> from
> >> >
> >> >
> >> hdfs://domU-12-31-39-09-C5-54.compute-1.internal/osm2_hfile/Level4/1978943899457605747
> >> > org/apache/hadoop/hbase/io/hfile/HFile.java:1335:in `deserialize':
> >> > java.io.IOException: Trailer 'header' is wrong; does the trailer size
> >> match
> >> > content? (NativeException)
> >> > from org/apache/hadoop/hbase/io/hfile/HFile.java:813:in `readTrailer'
> >> > from org/apache/hadoop/hbase/io/hfile/HFile.java:758:in
> >> `loadFileInfo'
> >> > from sun.reflect.GeneratedMethodAccessor7:-1:in `invoke'
> >> > from sun/reflect/DelegatingMethodAccessorImpl.java:25:in `invoke'
> >> > from java/lang/reflect/Method.java:597:in `invoke'
> >> > from org/jruby/javasupport/JavaMethod.java:298:in
> >> > `invokeWithExceptionHandling'
> >> > from org/jruby/javasupport/JavaMethod.java:259:in `invoke'
> >> > from org/jruby/java/invokers/InstanceMethodInvoker.java:36:in `call'
> >> > ... 18 levels...
> >> > from org/jruby/Main.java:94:in `main'
> >> > from loadtable.rb:83:in `each'
> >> > from loadtable.rb:83
> >> > Complete Java stackTrace
> >> > java.io.IOException: Trailer 'header' is wrong; does the trailer size
> >> match
> >> > content?
> >> > at
> >> >
> >> >
> >> org.apache.hadoop.hbase.io.hfile.HFile$FixedFileTrailer.deserialize(HFile.java:1335)
> >> > at
> >> >
> >> org.apache.hadoop.hbase.io.hfile.HFile$Reader.readTrailer(HFile.java:813)
> >> > at
> >> >
> >> org.apache.hadoop.hbase.io.hfile.HFile$Reader.loadFileInfo(HFile.java:758)
> >> > at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> >> > at
> >> >
> >> >
> >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >> > at java.lang.reflect.Method.invoke(Method.java:597)
> >> > at
> >> >
> >> >
> >> org.jruby.javasupport.JavaMethod.invokeWithExceptionHandling(JavaMethod.java:298)
> >> > at org.jruby.javasupport.JavaMethod.invoke(JavaMethod.java:259)
> >> > at
> >> >
> >> >
> >> org.jruby.java.invokers.InstanceMethodInvoker.call(InstanceMethodInvoker.java:36)
> >> > at
> >> > org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:70)
> >> > at loadtable.ensure_1$RUBY$__ensure___2(loadtable.rb:86)
> >> > at loadtable.block_0$RUBY$__for__(loadtable.rb:85)
> >> > at loadtableBlockCallback$block_0$RUBY$__for__xx1.call(Unknown
> >> Source)
> >> > at org.jruby.runtime.CompiledBlock.yield(CompiledBlock.java:102)
> >> > at org.jruby.runtime.Block.yield(Block.java:100)
> >> > at
> >> org.jruby.java.proxies.ArrayJavaProxy.each(ArrayJavaProxy.java:112)
> >> > at
> >> >
> >> >
> >> org.jruby.java.proxies.ArrayJavaProxy$i_method_0_0$RUBYINVOKER$each.call(org/jruby/java/proxies/ArrayJavaProxy$i_method_0_0$RUBYINVOKER$each.gen)
> >> > at
> >> >
> >> >
> >> org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:263)
> >> > at
> >> >
> >> >
> >> org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:81)
> >> > at
> >> >
> >> >
> >> org.jruby.runtime.callsite.CachingCallSite.callIter(CachingCallSite.java:96)
> >> > at loadtable.__file__(loadtable.rb:83)
> >> > at loadtable.load(loadtable.rb)
> >> > at org.jruby.Ruby.runScript(Ruby.java:577)
> >> > at org.jruby.Ruby.runNormally(Ruby.java:480)
> >> > at org.jruby.Ruby.runFromMain(Ruby.java:354)
> >> > at org.jruby.Main.run(Main.java:229)
> >> > at org.jruby.Main.run(Main.java:110)
> >> > at org.jruby.Main.main(Main.java:94)
> >> >
> >>
> >
> >
> >
> >
> --
> *****************************************************
> Ching-Shen Chen
> Advanced Technology Center,
> Information & Communications Research Lab.
> E-mail: chenchingshen@itri.org.tw
> Tel:+886-3-5915542
> *****************************************************
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-2063) For hfileoutputformat, on
timeout/failure/kill clean up half-written hfile
Posted by "Ruslan Salyakhov (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12840644#action_12840644 ]
Ruslan Salyakhov commented on HBASE-2063:
-----------------------------------------
Ching-Shen: Let's double check, can you see _temporary folder in your output path during MR execution (after MR execution you will have the folder with the same name as a column of your table and _temporary folder will be removed)? if not then probably patch was not applied.
> For hfileoutputformat, on timeout/failure/kill clean up half-written hfile
> --------------------------------------------------------------------------
>
> Key: HBASE-2063
> URL: https://issues.apache.org/jira/browse/HBASE-2063
> Project: Hadoop HBase
> Issue Type: Bug
> Affects Versions: 0.20.3
> Reporter: stack
> Fix For: 0.20.4, 0.21.0
>
> Attachments: HBASE-2063-v1.patch
>
>
> Below is from mailing list. Read from bottom to top:
> {code}
> I was going to write that perhaps you needed to turn mapred.reduce.tasks.speculative.execution off, but if enabling it and things work, that would seem to indicate that a our reducer first takes longer than the task timeout maximum and secondly, on failure, we should clean up the hfile.
> On the first issue, you are using KeyValueSortReducer? Are your values large? We set reducer status every 100 values. Maybe this is not enough? We should set status more frequently? If you call context setstatus more frequently, do things work w/o speculative execution?
>
> On the second, HFileOutputFormat close will set the metadata on the hfile and then close it. On kill, this code is not being called. Let me see if can do something about that (e.g. register a shutdown hook to clean away incomplete files -- ).
> Thanks,
> St.Ack
> On Sun, Dec 20, 2009 at 11:26 PM, ChingShen <ch...@gmail.com> wrote:
> I think I found a way.
> I set the "mapred.reduce.tasks.speculative.execution" to true and output
> hfiles again, then successfully load hfiles into hbase.
> Is it best solution? or HFileOutputFormat bug?
> Shen
> On Mon, Dec 21, 2009 at 8:25 AM, ChingShen <ch...@gmail.com> wrote:
> > Thanks, stack.
> >
> > I checked this file that isn't empty. But I found that as long as the
> > "Killed Task Attempts" > 0 in reduce phase, and run the loadtable.rb script
> > to load hfiles then failed.
> > How to avoid this problem?
> >
> > Thanks.
> >
> > Shen
> >
> >
> > On Sat, Dec 19, 2009 at 3:49 AM, stack <st...@duboce.net> wrote:
> >
> >> Check the
> >> file
> >> hdfs://domU-12-31-39-09-C5-54.compute-1.internal/osm2_hfile/Level4/197894389945760574.
> >> Is it empty? Was there an error during running of your MR job? Perhaps
> >> a
> >> task failed?
> >>
> >> St.Ack
> >>
> >>
> >>
> >> On Thu, Dec 17, 2009 at 9:46 PM, ChingShen <ch...@gmail.com>
> >> wrote:
> >>
> >> > Hi,
> >> > I use the script loadtable.rb to load my hfiles into hbase, but I got
> >> an
> >> > exception as below.
> >> > Does anyone have any suggestions?
> >> >
> >> > 09/12/17 23:59:33 INFO loadtable: 18 read firstkey of -3.9290_52.5534
> >> from
> >> >
> >> >
> >> hdfs://domU-12-31-39-09-C5-54.compute-1.internal/osm2_hfile/Level4/1978943899457605747
> >> > org/apache/hadoop/hbase/io/hfile/HFile.java:1335:in `deserialize':
> >> > java.io.IOException: Trailer 'header' is wrong; does the trailer size
> >> match
> >> > content? (NativeException)
> >> > from org/apache/hadoop/hbase/io/hfile/HFile.java:813:in `readTrailer'
> >> > from org/apache/hadoop/hbase/io/hfile/HFile.java:758:in
> >> `loadFileInfo'
> >> > from sun.reflect.GeneratedMethodAccessor7:-1:in `invoke'
> >> > from sun/reflect/DelegatingMethodAccessorImpl.java:25:in `invoke'
> >> > from java/lang/reflect/Method.java:597:in `invoke'
> >> > from org/jruby/javasupport/JavaMethod.java:298:in
> >> > `invokeWithExceptionHandling'
> >> > from org/jruby/javasupport/JavaMethod.java:259:in `invoke'
> >> > from org/jruby/java/invokers/InstanceMethodInvoker.java:36:in `call'
> >> > ... 18 levels...
> >> > from org/jruby/Main.java:94:in `main'
> >> > from loadtable.rb:83:in `each'
> >> > from loadtable.rb:83
> >> > Complete Java stackTrace
> >> > java.io.IOException: Trailer 'header' is wrong; does the trailer size
> >> match
> >> > content?
> >> > at
> >> >
> >> >
> >> org.apache.hadoop.hbase.io.hfile.HFile$FixedFileTrailer.deserialize(HFile.java:1335)
> >> > at
> >> >
> >> org.apache.hadoop.hbase.io.hfile.HFile$Reader.readTrailer(HFile.java:813)
> >> > at
> >> >
> >> org.apache.hadoop.hbase.io.hfile.HFile$Reader.loadFileInfo(HFile.java:758)
> >> > at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> >> > at
> >> >
> >> >
> >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >> > at java.lang.reflect.Method.invoke(Method.java:597)
> >> > at
> >> >
> >> >
> >> org.jruby.javasupport.JavaMethod.invokeWithExceptionHandling(JavaMethod.java:298)
> >> > at org.jruby.javasupport.JavaMethod.invoke(JavaMethod.java:259)
> >> > at
> >> >
> >> >
> >> org.jruby.java.invokers.InstanceMethodInvoker.call(InstanceMethodInvoker.java:36)
> >> > at
> >> > org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:70)
> >> > at loadtable.ensure_1$RUBY$__ensure___2(loadtable.rb:86)
> >> > at loadtable.block_0$RUBY$__for__(loadtable.rb:85)
> >> > at loadtableBlockCallback$block_0$RUBY$__for__xx1.call(Unknown
> >> Source)
> >> > at org.jruby.runtime.CompiledBlock.yield(CompiledBlock.java:102)
> >> > at org.jruby.runtime.Block.yield(Block.java:100)
> >> > at
> >> org.jruby.java.proxies.ArrayJavaProxy.each(ArrayJavaProxy.java:112)
> >> > at
> >> >
> >> >
> >> org.jruby.java.proxies.ArrayJavaProxy$i_method_0_0$RUBYINVOKER$each.call(org/jruby/java/proxies/ArrayJavaProxy$i_method_0_0$RUBYINVOKER$each.gen)
> >> > at
> >> >
> >> >
> >> org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:263)
> >> > at
> >> >
> >> >
> >> org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:81)
> >> > at
> >> >
> >> >
> >> org.jruby.runtime.callsite.CachingCallSite.callIter(CachingCallSite.java:96)
> >> > at loadtable.__file__(loadtable.rb:83)
> >> > at loadtable.load(loadtable.rb)
> >> > at org.jruby.Ruby.runScript(Ruby.java:577)
> >> > at org.jruby.Ruby.runNormally(Ruby.java:480)
> >> > at org.jruby.Ruby.runFromMain(Ruby.java:354)
> >> > at org.jruby.Main.run(Main.java:229)
> >> > at org.jruby.Main.run(Main.java:110)
> >> > at org.jruby.Main.main(Main.java:94)
> >> >
> >>
> >
> >
> >
> >
> --
> *****************************************************
> Ching-Shen Chen
> Advanced Technology Center,
> Information & Communications Research Lab.
> E-mail: chenchingshen@itri.org.tw
> Tel:+886-3-5915542
> *****************************************************
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-2063) For hfileoutputformat, on
timeout/failure/kill clean up half-written hfile
Posted by "Ruslan Salyakhov (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ruslan Salyakhov updated HBASE-2063:
------------------------------------
Attachment: HBASE-2063-v1.patch
attached is the patch (HBASE-2063-v1.patch) for HFileOutputFormat:
to avoid such kind of problems need to use working path instead of output path in HFileOutputFormat.RecordWriter.
so, temporary data of failed attempts will be removed and corrupted HFiles will not appear in output folder and loadtable.rb script will go without exceptions.
> For hfileoutputformat, on timeout/failure/kill clean up half-written hfile
> --------------------------------------------------------------------------
>
> Key: HBASE-2063
> URL: https://issues.apache.org/jira/browse/HBASE-2063
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: stack
> Attachments: HBASE-2063-v1.patch
>
>
> Below is from mailing list. Read from bottom to top:
> {code}
> I was going to write that perhaps you needed to turn mapred.reduce.tasks.speculative.execution off, but if enabling it and things work, that would seem to indicate that a our reducer first takes longer than the task timeout maximum and secondly, on failure, we should clean up the hfile.
> On the first issue, you are using KeyValueSortReducer? Are your values large? We set reducer status every 100 values. Maybe this is not enough? We should set status more frequently? If you call context setstatus more frequently, do things work w/o speculative execution?
>
> On the second, HFileOutputFormat close will set the metadata on the hfile and then close it. On kill, this code is not being called. Let me see if can do something about that (e.g. register a shutdown hook to clean away incomplete files -- ).
> Thanks,
> St.Ack
> On Sun, Dec 20, 2009 at 11:26 PM, ChingShen <ch...@gmail.com> wrote:
> I think I found a way.
> I set the "mapred.reduce.tasks.speculative.execution" to true and output
> hfiles again, then successfully load hfiles into hbase.
> Is it best solution? or HFileOutputFormat bug?
> Shen
> On Mon, Dec 21, 2009 at 8:25 AM, ChingShen <ch...@gmail.com> wrote:
> > Thanks, stack.
> >
> > I checked this file that isn't empty. But I found that as long as the
> > "Killed Task Attempts" > 0 in reduce phase, and run the loadtable.rb script
> > to load hfiles then failed.
> > How to avoid this problem?
> >
> > Thanks.
> >
> > Shen
> >
> >
> > On Sat, Dec 19, 2009 at 3:49 AM, stack <st...@duboce.net> wrote:
> >
> >> Check the
> >> file
> >> hdfs://domU-12-31-39-09-C5-54.compute-1.internal/osm2_hfile/Level4/197894389945760574.
> >> Is it empty? Was there an error during running of your MR job? Perhaps
> >> a
> >> task failed?
> >>
> >> St.Ack
> >>
> >>
> >>
> >> On Thu, Dec 17, 2009 at 9:46 PM, ChingShen <ch...@gmail.com>
> >> wrote:
> >>
> >> > Hi,
> >> > I use the script loadtable.rb to load my hfiles into hbase, but I got
> >> an
> >> > exception as below.
> >> > Does anyone have any suggestions?
> >> >
> >> > 09/12/17 23:59:33 INFO loadtable: 18 read firstkey of -3.9290_52.5534
> >> from
> >> >
> >> >
> >> hdfs://domU-12-31-39-09-C5-54.compute-1.internal/osm2_hfile/Level4/1978943899457605747
> >> > org/apache/hadoop/hbase/io/hfile/HFile.java:1335:in `deserialize':
> >> > java.io.IOException: Trailer 'header' is wrong; does the trailer size
> >> match
> >> > content? (NativeException)
> >> > from org/apache/hadoop/hbase/io/hfile/HFile.java:813:in `readTrailer'
> >> > from org/apache/hadoop/hbase/io/hfile/HFile.java:758:in
> >> `loadFileInfo'
> >> > from sun.reflect.GeneratedMethodAccessor7:-1:in `invoke'
> >> > from sun/reflect/DelegatingMethodAccessorImpl.java:25:in `invoke'
> >> > from java/lang/reflect/Method.java:597:in `invoke'
> >> > from org/jruby/javasupport/JavaMethod.java:298:in
> >> > `invokeWithExceptionHandling'
> >> > from org/jruby/javasupport/JavaMethod.java:259:in `invoke'
> >> > from org/jruby/java/invokers/InstanceMethodInvoker.java:36:in `call'
> >> > ... 18 levels...
> >> > from org/jruby/Main.java:94:in `main'
> >> > from loadtable.rb:83:in `each'
> >> > from loadtable.rb:83
> >> > Complete Java stackTrace
> >> > java.io.IOException: Trailer 'header' is wrong; does the trailer size
> >> match
> >> > content?
> >> > at
> >> >
> >> >
> >> org.apache.hadoop.hbase.io.hfile.HFile$FixedFileTrailer.deserialize(HFile.java:1335)
> >> > at
> >> >
> >> org.apache.hadoop.hbase.io.hfile.HFile$Reader.readTrailer(HFile.java:813)
> >> > at
> >> >
> >> org.apache.hadoop.hbase.io.hfile.HFile$Reader.loadFileInfo(HFile.java:758)
> >> > at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> >> > at
> >> >
> >> >
> >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >> > at java.lang.reflect.Method.invoke(Method.java:597)
> >> > at
> >> >
> >> >
> >> org.jruby.javasupport.JavaMethod.invokeWithExceptionHandling(JavaMethod.java:298)
> >> > at org.jruby.javasupport.JavaMethod.invoke(JavaMethod.java:259)
> >> > at
> >> >
> >> >
> >> org.jruby.java.invokers.InstanceMethodInvoker.call(InstanceMethodInvoker.java:36)
> >> > at
> >> > org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:70)
> >> > at loadtable.ensure_1$RUBY$__ensure___2(loadtable.rb:86)
> >> > at loadtable.block_0$RUBY$__for__(loadtable.rb:85)
> >> > at loadtableBlockCallback$block_0$RUBY$__for__xx1.call(Unknown
> >> Source)
> >> > at org.jruby.runtime.CompiledBlock.yield(CompiledBlock.java:102)
> >> > at org.jruby.runtime.Block.yield(Block.java:100)
> >> > at
> >> org.jruby.java.proxies.ArrayJavaProxy.each(ArrayJavaProxy.java:112)
> >> > at
> >> >
> >> >
> >> org.jruby.java.proxies.ArrayJavaProxy$i_method_0_0$RUBYINVOKER$each.call(org/jruby/java/proxies/ArrayJavaProxy$i_method_0_0$RUBYINVOKER$each.gen)
> >> > at
> >> >
> >> >
> >> org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:263)
> >> > at
> >> >
> >> >
> >> org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:81)
> >> > at
> >> >
> >> >
> >> org.jruby.runtime.callsite.CachingCallSite.callIter(CachingCallSite.java:96)
> >> > at loadtable.__file__(loadtable.rb:83)
> >> > at loadtable.load(loadtable.rb)
> >> > at org.jruby.Ruby.runScript(Ruby.java:577)
> >> > at org.jruby.Ruby.runNormally(Ruby.java:480)
> >> > at org.jruby.Ruby.runFromMain(Ruby.java:354)
> >> > at org.jruby.Main.run(Main.java:229)
> >> > at org.jruby.Main.run(Main.java:110)
> >> > at org.jruby.Main.main(Main.java:94)
> >> >
> >>
> >
> >
> >
> >
> --
> *****************************************************
> Ching-Shen Chen
> Advanced Technology Center,
> Information & Communications Research Lab.
> E-mail: chenchingshen@itri.org.tw
> Tel:+886-3-5915542
> *****************************************************
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-2063) For hfileoutputformat, on
timeout/failure/kill clean up half-written hfile
Posted by "Ching-Shen Chen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12841097#action_12841097 ]
Ching-Shen Chen commented on HBASE-2063:
----------------------------------------
Thanks Ruslan, It works well. (I think I forgot to distribute the hbase.jar to slaves, It's my fault.)
> For hfileoutputformat, on timeout/failure/kill clean up half-written hfile
> --------------------------------------------------------------------------
>
> Key: HBASE-2063
> URL: https://issues.apache.org/jira/browse/HBASE-2063
> Project: Hadoop HBase
> Issue Type: Bug
> Affects Versions: 0.20.3
> Reporter: stack
> Fix For: 0.20.4, 0.21.0
>
> Attachments: HBASE-2063-v1.patch
>
>
> Below is from mailing list. Read from bottom to top:
> {code}
> I was going to write that perhaps you needed to turn mapred.reduce.tasks.speculative.execution off, but if enabling it and things work, that would seem to indicate that a our reducer first takes longer than the task timeout maximum and secondly, on failure, we should clean up the hfile.
> On the first issue, you are using KeyValueSortReducer? Are your values large? We set reducer status every 100 values. Maybe this is not enough? We should set status more frequently? If you call context setstatus more frequently, do things work w/o speculative execution?
>
> On the second, HFileOutputFormat close will set the metadata on the hfile and then close it. On kill, this code is not being called. Let me see if can do something about that (e.g. register a shutdown hook to clean away incomplete files -- ).
> Thanks,
> St.Ack
> On Sun, Dec 20, 2009 at 11:26 PM, ChingShen <ch...@gmail.com> wrote:
> I think I found a way.
> I set the "mapred.reduce.tasks.speculative.execution" to true and output
> hfiles again, then successfully load hfiles into hbase.
> Is it best solution? or HFileOutputFormat bug?
> Shen
> On Mon, Dec 21, 2009 at 8:25 AM, ChingShen <ch...@gmail.com> wrote:
> > Thanks, stack.
> >
> > I checked this file that isn't empty. But I found that as long as the
> > "Killed Task Attempts" > 0 in reduce phase, and run the loadtable.rb script
> > to load hfiles then failed.
> > How to avoid this problem?
> >
> > Thanks.
> >
> > Shen
> >
> >
> > On Sat, Dec 19, 2009 at 3:49 AM, stack <st...@duboce.net> wrote:
> >
> >> Check the
> >> file
> >> hdfs://domU-12-31-39-09-C5-54.compute-1.internal/osm2_hfile/Level4/197894389945760574.
> >> Is it empty? Was there an error during running of your MR job? Perhaps
> >> a
> >> task failed?
> >>
> >> St.Ack
> >>
> >>
> >>
> >> On Thu, Dec 17, 2009 at 9:46 PM, ChingShen <ch...@gmail.com>
> >> wrote:
> >>
> >> > Hi,
> >> > I use the script loadtable.rb to load my hfiles into hbase, but I got
> >> an
> >> > exception as below.
> >> > Does anyone have any suggestions?
> >> >
> >> > 09/12/17 23:59:33 INFO loadtable: 18 read firstkey of -3.9290_52.5534
> >> from
> >> >
> >> >
> >> hdfs://domU-12-31-39-09-C5-54.compute-1.internal/osm2_hfile/Level4/1978943899457605747
> >> > org/apache/hadoop/hbase/io/hfile/HFile.java:1335:in `deserialize':
> >> > java.io.IOException: Trailer 'header' is wrong; does the trailer size
> >> match
> >> > content? (NativeException)
> >> > from org/apache/hadoop/hbase/io/hfile/HFile.java:813:in `readTrailer'
> >> > from org/apache/hadoop/hbase/io/hfile/HFile.java:758:in
> >> `loadFileInfo'
> >> > from sun.reflect.GeneratedMethodAccessor7:-1:in `invoke'
> >> > from sun/reflect/DelegatingMethodAccessorImpl.java:25:in `invoke'
> >> > from java/lang/reflect/Method.java:597:in `invoke'
> >> > from org/jruby/javasupport/JavaMethod.java:298:in
> >> > `invokeWithExceptionHandling'
> >> > from org/jruby/javasupport/JavaMethod.java:259:in `invoke'
> >> > from org/jruby/java/invokers/InstanceMethodInvoker.java:36:in `call'
> >> > ... 18 levels...
> >> > from org/jruby/Main.java:94:in `main'
> >> > from loadtable.rb:83:in `each'
> >> > from loadtable.rb:83
> >> > Complete Java stackTrace
> >> > java.io.IOException: Trailer 'header' is wrong; does the trailer size
> >> match
> >> > content?
> >> > at
> >> >
> >> >
> >> org.apache.hadoop.hbase.io.hfile.HFile$FixedFileTrailer.deserialize(HFile.java:1335)
> >> > at
> >> >
> >> org.apache.hadoop.hbase.io.hfile.HFile$Reader.readTrailer(HFile.java:813)
> >> > at
> >> >
> >> org.apache.hadoop.hbase.io.hfile.HFile$Reader.loadFileInfo(HFile.java:758)
> >> > at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> >> > at
> >> >
> >> >
> >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >> > at java.lang.reflect.Method.invoke(Method.java:597)
> >> > at
> >> >
> >> >
> >> org.jruby.javasupport.JavaMethod.invokeWithExceptionHandling(JavaMethod.java:298)
> >> > at org.jruby.javasupport.JavaMethod.invoke(JavaMethod.java:259)
> >> > at
> >> >
> >> >
> >> org.jruby.java.invokers.InstanceMethodInvoker.call(InstanceMethodInvoker.java:36)
> >> > at
> >> > org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:70)
> >> > at loadtable.ensure_1$RUBY$__ensure___2(loadtable.rb:86)
> >> > at loadtable.block_0$RUBY$__for__(loadtable.rb:85)
> >> > at loadtableBlockCallback$block_0$RUBY$__for__xx1.call(Unknown
> >> Source)
> >> > at org.jruby.runtime.CompiledBlock.yield(CompiledBlock.java:102)
> >> > at org.jruby.runtime.Block.yield(Block.java:100)
> >> > at
> >> org.jruby.java.proxies.ArrayJavaProxy.each(ArrayJavaProxy.java:112)
> >> > at
> >> >
> >> >
> >> org.jruby.java.proxies.ArrayJavaProxy$i_method_0_0$RUBYINVOKER$each.call(org/jruby/java/proxies/ArrayJavaProxy$i_method_0_0$RUBYINVOKER$each.gen)
> >> > at
> >> >
> >> >
> >> org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:263)
> >> > at
> >> >
> >> >
> >> org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:81)
> >> > at
> >> >
> >> >
> >> org.jruby.runtime.callsite.CachingCallSite.callIter(CachingCallSite.java:96)
> >> > at loadtable.__file__(loadtable.rb:83)
> >> > at loadtable.load(loadtable.rb)
> >> > at org.jruby.Ruby.runScript(Ruby.java:577)
> >> > at org.jruby.Ruby.runNormally(Ruby.java:480)
> >> > at org.jruby.Ruby.runFromMain(Ruby.java:354)
> >> > at org.jruby.Main.run(Main.java:229)
> >> > at org.jruby.Main.run(Main.java:110)
> >> > at org.jruby.Main.main(Main.java:94)
> >> >
> >>
> >
> >
> >
> >
> --
> *****************************************************
> Ching-Shen Chen
> Advanced Technology Center,
> Information & Communications Research Lab.
> E-mail: chenchingshen@itri.org.tw
> Tel:+886-3-5915542
> *****************************************************
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-2063) For hfileoutputformat, on
timeout/failure/kill clean up half-written hfile
Posted by "Ching-Shen Chen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12840411#action_12840411 ]
Ching-Shen Chen commented on HBASE-2063:
----------------------------------------
I just patched this issue, but the problem still exists.
> For hfileoutputformat, on timeout/failure/kill clean up half-written hfile
> --------------------------------------------------------------------------
>
> Key: HBASE-2063
> URL: https://issues.apache.org/jira/browse/HBASE-2063
> Project: Hadoop HBase
> Issue Type: Bug
> Affects Versions: 0.20.3
> Reporter: stack
> Fix For: 0.20.4, 0.21.0
>
> Attachments: HBASE-2063-v1.patch
>
>
> Below is from mailing list. Read from bottom to top:
> {code}
> I was going to write that perhaps you needed to turn mapred.reduce.tasks.speculative.execution off, but if enabling it and things work, that would seem to indicate that a our reducer first takes longer than the task timeout maximum and secondly, on failure, we should clean up the hfile.
> On the first issue, you are using KeyValueSortReducer? Are your values large? We set reducer status every 100 values. Maybe this is not enough? We should set status more frequently? If you call context setstatus more frequently, do things work w/o speculative execution?
>
> On the second, HFileOutputFormat close will set the metadata on the hfile and then close it. On kill, this code is not being called. Let me see if can do something about that (e.g. register a shutdown hook to clean away incomplete files -- ).
> Thanks,
> St.Ack
> On Sun, Dec 20, 2009 at 11:26 PM, ChingShen <ch...@gmail.com> wrote:
> I think I found a way.
> I set the "mapred.reduce.tasks.speculative.execution" to true and output
> hfiles again, then successfully load hfiles into hbase.
> Is it best solution? or HFileOutputFormat bug?
> Shen
> On Mon, Dec 21, 2009 at 8:25 AM, ChingShen <ch...@gmail.com> wrote:
> > Thanks, stack.
> >
> > I checked this file that isn't empty. But I found that as long as the
> > "Killed Task Attempts" > 0 in reduce phase, and run the loadtable.rb script
> > to load hfiles then failed.
> > How to avoid this problem?
> >
> > Thanks.
> >
> > Shen
> >
> >
> > On Sat, Dec 19, 2009 at 3:49 AM, stack <st...@duboce.net> wrote:
> >
> >> Check the
> >> file
> >> hdfs://domU-12-31-39-09-C5-54.compute-1.internal/osm2_hfile/Level4/197894389945760574.
> >> Is it empty? Was there an error during running of your MR job? Perhaps
> >> a
> >> task failed?
> >>
> >> St.Ack
> >>
> >>
> >>
> >> On Thu, Dec 17, 2009 at 9:46 PM, ChingShen <ch...@gmail.com>
> >> wrote:
> >>
> >> > Hi,
> >> > I use the script loadtable.rb to load my hfiles into hbase, but I got
> >> an
> >> > exception as below.
> >> > Does anyone have any suggestions?
> >> >
> >> > 09/12/17 23:59:33 INFO loadtable: 18 read firstkey of -3.9290_52.5534
> >> from
> >> >
> >> >
> >> hdfs://domU-12-31-39-09-C5-54.compute-1.internal/osm2_hfile/Level4/1978943899457605747
> >> > org/apache/hadoop/hbase/io/hfile/HFile.java:1335:in `deserialize':
> >> > java.io.IOException: Trailer 'header' is wrong; does the trailer size
> >> match
> >> > content? (NativeException)
> >> > from org/apache/hadoop/hbase/io/hfile/HFile.java:813:in `readTrailer'
> >> > from org/apache/hadoop/hbase/io/hfile/HFile.java:758:in
> >> `loadFileInfo'
> >> > from sun.reflect.GeneratedMethodAccessor7:-1:in `invoke'
> >> > from sun/reflect/DelegatingMethodAccessorImpl.java:25:in `invoke'
> >> > from java/lang/reflect/Method.java:597:in `invoke'
> >> > from org/jruby/javasupport/JavaMethod.java:298:in
> >> > `invokeWithExceptionHandling'
> >> > from org/jruby/javasupport/JavaMethod.java:259:in `invoke'
> >> > from org/jruby/java/invokers/InstanceMethodInvoker.java:36:in `call'
> >> > ... 18 levels...
> >> > from org/jruby/Main.java:94:in `main'
> >> > from loadtable.rb:83:in `each'
> >> > from loadtable.rb:83
> >> > Complete Java stackTrace
> >> > java.io.IOException: Trailer 'header' is wrong; does the trailer size
> >> match
> >> > content?
> >> > at
> >> >
> >> >
> >> org.apache.hadoop.hbase.io.hfile.HFile$FixedFileTrailer.deserialize(HFile.java:1335)
> >> > at
> >> >
> >> org.apache.hadoop.hbase.io.hfile.HFile$Reader.readTrailer(HFile.java:813)
> >> > at
> >> >
> >> org.apache.hadoop.hbase.io.hfile.HFile$Reader.loadFileInfo(HFile.java:758)
> >> > at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> >> > at
> >> >
> >> >
> >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >> > at java.lang.reflect.Method.invoke(Method.java:597)
> >> > at
> >> >
> >> >
> >> org.jruby.javasupport.JavaMethod.invokeWithExceptionHandling(JavaMethod.java:298)
> >> > at org.jruby.javasupport.JavaMethod.invoke(JavaMethod.java:259)
> >> > at
> >> >
> >> >
> >> org.jruby.java.invokers.InstanceMethodInvoker.call(InstanceMethodInvoker.java:36)
> >> > at
> >> > org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:70)
> >> > at loadtable.ensure_1$RUBY$__ensure___2(loadtable.rb:86)
> >> > at loadtable.block_0$RUBY$__for__(loadtable.rb:85)
> >> > at loadtableBlockCallback$block_0$RUBY$__for__xx1.call(Unknown
> >> Source)
> >> > at org.jruby.runtime.CompiledBlock.yield(CompiledBlock.java:102)
> >> > at org.jruby.runtime.Block.yield(Block.java:100)
> >> > at
> >> org.jruby.java.proxies.ArrayJavaProxy.each(ArrayJavaProxy.java:112)
> >> > at
> >> >
> >> >
> >> org.jruby.java.proxies.ArrayJavaProxy$i_method_0_0$RUBYINVOKER$each.call(org/jruby/java/proxies/ArrayJavaProxy$i_method_0_0$RUBYINVOKER$each.gen)
> >> > at
> >> >
> >> >
> >> org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:263)
> >> > at
> >> >
> >> >
> >> org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:81)
> >> > at
> >> >
> >> >
> >> org.jruby.runtime.callsite.CachingCallSite.callIter(CachingCallSite.java:96)
> >> > at loadtable.__file__(loadtable.rb:83)
> >> > at loadtable.load(loadtable.rb)
> >> > at org.jruby.Ruby.runScript(Ruby.java:577)
> >> > at org.jruby.Ruby.runNormally(Ruby.java:480)
> >> > at org.jruby.Ruby.runFromMain(Ruby.java:354)
> >> > at org.jruby.Main.run(Main.java:229)
> >> > at org.jruby.Main.run(Main.java:110)
> >> > at org.jruby.Main.main(Main.java:94)
> >> >
> >>
> >
> >
> >
> >
> --
> *****************************************************
> Ching-Shen Chen
> Advanced Technology Center,
> Information & Communications Research Lab.
> E-mail: chenchingshen@itri.org.tw
> Tel:+886-3-5915542
> *****************************************************
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-2063) For hfileoutputformat, on
timeout/failure/kill clean up half-written hfile
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-2063:
-------------------------
Resolution: Fixed
Hadoop Flags: [Reviewed]
Status: Resolved (was: Patch Available)
Commmitted branch and trunk. Thanks for the nice patch Ruslan Salyakhov. Thanks for verifying fix Ching-Shen Chen .
> For hfileoutputformat, on timeout/failure/kill clean up half-written hfile
> --------------------------------------------------------------------------
>
> Key: HBASE-2063
> URL: https://issues.apache.org/jira/browse/HBASE-2063
> Project: Hadoop HBase
> Issue Type: Bug
> Affects Versions: 0.20.3
> Reporter: stack
> Fix For: 0.20.4, 0.21.0
>
> Attachments: HBASE-2063-v1.patch
>
>
> Below is from mailing list. Read from bottom to top:
> {code}
> I was going to write that perhaps you needed to turn mapred.reduce.tasks.speculative.execution off, but if enabling it and things work, that would seem to indicate that a our reducer first takes longer than the task timeout maximum and secondly, on failure, we should clean up the hfile.
> On the first issue, you are using KeyValueSortReducer? Are your values large? We set reducer status every 100 values. Maybe this is not enough? We should set status more frequently? If you call context setstatus more frequently, do things work w/o speculative execution?
>
> On the second, HFileOutputFormat close will set the metadata on the hfile and then close it. On kill, this code is not being called. Let me see if can do something about that (e.g. register a shutdown hook to clean away incomplete files -- ).
> Thanks,
> St.Ack
> On Sun, Dec 20, 2009 at 11:26 PM, ChingShen <ch...@gmail.com> wrote:
> I think I found a way.
> I set the "mapred.reduce.tasks.speculative.execution" to true and output
> hfiles again, then successfully load hfiles into hbase.
> Is it best solution? or HFileOutputFormat bug?
> Shen
> On Mon, Dec 21, 2009 at 8:25 AM, ChingShen <ch...@gmail.com> wrote:
> > Thanks, stack.
> >
> > I checked this file that isn't empty. But I found that as long as the
> > "Killed Task Attempts" > 0 in reduce phase, and run the loadtable.rb script
> > to load hfiles then failed.
> > How to avoid this problem?
> >
> > Thanks.
> >
> > Shen
> >
> >
> > On Sat, Dec 19, 2009 at 3:49 AM, stack <st...@duboce.net> wrote:
> >
> >> Check the
> >> file
> >> hdfs://domU-12-31-39-09-C5-54.compute-1.internal/osm2_hfile/Level4/197894389945760574.
> >> Is it empty? Was there an error during running of your MR job? Perhaps
> >> a
> >> task failed?
> >>
> >> St.Ack
> >>
> >>
> >>
> >> On Thu, Dec 17, 2009 at 9:46 PM, ChingShen <ch...@gmail.com>
> >> wrote:
> >>
> >> > Hi,
> >> > I use the script loadtable.rb to load my hfiles into hbase, but I got
> >> an
> >> > exception as below.
> >> > Does anyone have any suggestions?
> >> >
> >> > 09/12/17 23:59:33 INFO loadtable: 18 read firstkey of -3.9290_52.5534
> >> from
> >> >
> >> >
> >> hdfs://domU-12-31-39-09-C5-54.compute-1.internal/osm2_hfile/Level4/1978943899457605747
> >> > org/apache/hadoop/hbase/io/hfile/HFile.java:1335:in `deserialize':
> >> > java.io.IOException: Trailer 'header' is wrong; does the trailer size
> >> match
> >> > content? (NativeException)
> >> > from org/apache/hadoop/hbase/io/hfile/HFile.java:813:in `readTrailer'
> >> > from org/apache/hadoop/hbase/io/hfile/HFile.java:758:in
> >> `loadFileInfo'
> >> > from sun.reflect.GeneratedMethodAccessor7:-1:in `invoke'
> >> > from sun/reflect/DelegatingMethodAccessorImpl.java:25:in `invoke'
> >> > from java/lang/reflect/Method.java:597:in `invoke'
> >> > from org/jruby/javasupport/JavaMethod.java:298:in
> >> > `invokeWithExceptionHandling'
> >> > from org/jruby/javasupport/JavaMethod.java:259:in `invoke'
> >> > from org/jruby/java/invokers/InstanceMethodInvoker.java:36:in `call'
> >> > ... 18 levels...
> >> > from org/jruby/Main.java:94:in `main'
> >> > from loadtable.rb:83:in `each'
> >> > from loadtable.rb:83
> >> > Complete Java stackTrace
> >> > java.io.IOException: Trailer 'header' is wrong; does the trailer size
> >> match
> >> > content?
> >> > at
> >> >
> >> >
> >> org.apache.hadoop.hbase.io.hfile.HFile$FixedFileTrailer.deserialize(HFile.java:1335)
> >> > at
> >> >
> >> org.apache.hadoop.hbase.io.hfile.HFile$Reader.readTrailer(HFile.java:813)
> >> > at
> >> >
> >> org.apache.hadoop.hbase.io.hfile.HFile$Reader.loadFileInfo(HFile.java:758)
> >> > at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> >> > at
> >> >
> >> >
> >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >> > at java.lang.reflect.Method.invoke(Method.java:597)
> >> > at
> >> >
> >> >
> >> org.jruby.javasupport.JavaMethod.invokeWithExceptionHandling(JavaMethod.java:298)
> >> > at org.jruby.javasupport.JavaMethod.invoke(JavaMethod.java:259)
> >> > at
> >> >
> >> >
> >> org.jruby.java.invokers.InstanceMethodInvoker.call(InstanceMethodInvoker.java:36)
> >> > at
> >> > org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:70)
> >> > at loadtable.ensure_1$RUBY$__ensure___2(loadtable.rb:86)
> >> > at loadtable.block_0$RUBY$__for__(loadtable.rb:85)
> >> > at loadtableBlockCallback$block_0$RUBY$__for__xx1.call(Unknown
> >> Source)
> >> > at org.jruby.runtime.CompiledBlock.yield(CompiledBlock.java:102)
> >> > at org.jruby.runtime.Block.yield(Block.java:100)
> >> > at
> >> org.jruby.java.proxies.ArrayJavaProxy.each(ArrayJavaProxy.java:112)
> >> > at
> >> >
> >> >
> >> org.jruby.java.proxies.ArrayJavaProxy$i_method_0_0$RUBYINVOKER$each.call(org/jruby/java/proxies/ArrayJavaProxy$i_method_0_0$RUBYINVOKER$each.gen)
> >> > at
> >> >
> >> >
> >> org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:263)
> >> > at
> >> >
> >> >
> >> org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:81)
> >> > at
> >> >
> >> >
> >> org.jruby.runtime.callsite.CachingCallSite.callIter(CachingCallSite.java:96)
> >> > at loadtable.__file__(loadtable.rb:83)
> >> > at loadtable.load(loadtable.rb)
> >> > at org.jruby.Ruby.runScript(Ruby.java:577)
> >> > at org.jruby.Ruby.runNormally(Ruby.java:480)
> >> > at org.jruby.Ruby.runFromMain(Ruby.java:354)
> >> > at org.jruby.Main.run(Main.java:229)
> >> > at org.jruby.Main.run(Main.java:110)
> >> > at org.jruby.Main.main(Main.java:94)
> >> >
> >>
> >
> >
> >
> >
> --
> *****************************************************
> Ching-Shen Chen
> Advanced Technology Center,
> Information & Communications Research Lab.
> E-mail: chenchingshen@itri.org.tw
> Tel:+886-3-5915542
> *****************************************************
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-2063) For hfileoutputformat, on
timeout/failure/kill clean up half-written hfile
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12840388#action_12840388 ]
stack commented on HBASE-2063:
------------------------------
Ruslan: Patch looks great. Thanks for fixing this. Will apply soon.
> For hfileoutputformat, on timeout/failure/kill clean up half-written hfile
> --------------------------------------------------------------------------
>
> Key: HBASE-2063
> URL: https://issues.apache.org/jira/browse/HBASE-2063
> Project: Hadoop HBase
> Issue Type: Bug
> Affects Versions: 0.20.3
> Reporter: stack
> Fix For: 0.20.4, 0.21.0
>
> Attachments: HBASE-2063-v1.patch
>
>
> Below is from mailing list. Read from bottom to top:
> {code}
> I was going to write that perhaps you needed to turn mapred.reduce.tasks.speculative.execution off, but if enabling it and things work, that would seem to indicate that a our reducer first takes longer than the task timeout maximum and secondly, on failure, we should clean up the hfile.
> On the first issue, you are using KeyValueSortReducer? Are your values large? We set reducer status every 100 values. Maybe this is not enough? We should set status more frequently? If you call context setstatus more frequently, do things work w/o speculative execution?
>
> On the second, HFileOutputFormat close will set the metadata on the hfile and then close it. On kill, this code is not being called. Let me see if can do something about that (e.g. register a shutdown hook to clean away incomplete files -- ).
> Thanks,
> St.Ack
> On Sun, Dec 20, 2009 at 11:26 PM, ChingShen <ch...@gmail.com> wrote:
> I think I found a way.
> I set the "mapred.reduce.tasks.speculative.execution" to true and output
> hfiles again, then successfully load hfiles into hbase.
> Is it best solution? or HFileOutputFormat bug?
> Shen
> On Mon, Dec 21, 2009 at 8:25 AM, ChingShen <ch...@gmail.com> wrote:
> > Thanks, stack.
> >
> > I checked this file that isn't empty. But I found that as long as the
> > "Killed Task Attempts" > 0 in reduce phase, and run the loadtable.rb script
> > to load hfiles then failed.
> > How to avoid this problem?
> >
> > Thanks.
> >
> > Shen
> >
> >
> > On Sat, Dec 19, 2009 at 3:49 AM, stack <st...@duboce.net> wrote:
> >
> >> Check the
> >> file
> >> hdfs://domU-12-31-39-09-C5-54.compute-1.internal/osm2_hfile/Level4/197894389945760574.
> >> Is it empty? Was there an error during running of your MR job? Perhaps
> >> a
> >> task failed?
> >>
> >> St.Ack
> >>
> >>
> >>
> >> On Thu, Dec 17, 2009 at 9:46 PM, ChingShen <ch...@gmail.com>
> >> wrote:
> >>
> >> > Hi,
> >> > I use the script loadtable.rb to load my hfiles into hbase, but I got
> >> an
> >> > exception as below.
> >> > Does anyone have any suggestions?
> >> >
> >> > 09/12/17 23:59:33 INFO loadtable: 18 read firstkey of -3.9290_52.5534
> >> from
> >> >
> >> >
> >> hdfs://domU-12-31-39-09-C5-54.compute-1.internal/osm2_hfile/Level4/1978943899457605747
> >> > org/apache/hadoop/hbase/io/hfile/HFile.java:1335:in `deserialize':
> >> > java.io.IOException: Trailer 'header' is wrong; does the trailer size
> >> match
> >> > content? (NativeException)
> >> > from org/apache/hadoop/hbase/io/hfile/HFile.java:813:in `readTrailer'
> >> > from org/apache/hadoop/hbase/io/hfile/HFile.java:758:in
> >> `loadFileInfo'
> >> > from sun.reflect.GeneratedMethodAccessor7:-1:in `invoke'
> >> > from sun/reflect/DelegatingMethodAccessorImpl.java:25:in `invoke'
> >> > from java/lang/reflect/Method.java:597:in `invoke'
> >> > from org/jruby/javasupport/JavaMethod.java:298:in
> >> > `invokeWithExceptionHandling'
> >> > from org/jruby/javasupport/JavaMethod.java:259:in `invoke'
> >> > from org/jruby/java/invokers/InstanceMethodInvoker.java:36:in `call'
> >> > ... 18 levels...
> >> > from org/jruby/Main.java:94:in `main'
> >> > from loadtable.rb:83:in `each'
> >> > from loadtable.rb:83
> >> > Complete Java stackTrace
> >> > java.io.IOException: Trailer 'header' is wrong; does the trailer size
> >> match
> >> > content?
> >> > at
> >> >
> >> >
> >> org.apache.hadoop.hbase.io.hfile.HFile$FixedFileTrailer.deserialize(HFile.java:1335)
> >> > at
> >> >
> >> org.apache.hadoop.hbase.io.hfile.HFile$Reader.readTrailer(HFile.java:813)
> >> > at
> >> >
> >> org.apache.hadoop.hbase.io.hfile.HFile$Reader.loadFileInfo(HFile.java:758)
> >> > at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> >> > at
> >> >
> >> >
> >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >> > at java.lang.reflect.Method.invoke(Method.java:597)
> >> > at
> >> >
> >> >
> >> org.jruby.javasupport.JavaMethod.invokeWithExceptionHandling(JavaMethod.java:298)
> >> > at org.jruby.javasupport.JavaMethod.invoke(JavaMethod.java:259)
> >> > at
> >> >
> >> >
> >> org.jruby.java.invokers.InstanceMethodInvoker.call(InstanceMethodInvoker.java:36)
> >> > at
> >> > org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:70)
> >> > at loadtable.ensure_1$RUBY$__ensure___2(loadtable.rb:86)
> >> > at loadtable.block_0$RUBY$__for__(loadtable.rb:85)
> >> > at loadtableBlockCallback$block_0$RUBY$__for__xx1.call(Unknown
> >> Source)
> >> > at org.jruby.runtime.CompiledBlock.yield(CompiledBlock.java:102)
> >> > at org.jruby.runtime.Block.yield(Block.java:100)
> >> > at
> >> org.jruby.java.proxies.ArrayJavaProxy.each(ArrayJavaProxy.java:112)
> >> > at
> >> >
> >> >
> >> org.jruby.java.proxies.ArrayJavaProxy$i_method_0_0$RUBYINVOKER$each.call(org/jruby/java/proxies/ArrayJavaProxy$i_method_0_0$RUBYINVOKER$each.gen)
> >> > at
> >> >
> >> >
> >> org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:263)
> >> > at
> >> >
> >> >
> >> org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:81)
> >> > at
> >> >
> >> >
> >> org.jruby.runtime.callsite.CachingCallSite.callIter(CachingCallSite.java:96)
> >> > at loadtable.__file__(loadtable.rb:83)
> >> > at loadtable.load(loadtable.rb)
> >> > at org.jruby.Ruby.runScript(Ruby.java:577)
> >> > at org.jruby.Ruby.runNormally(Ruby.java:480)
> >> > at org.jruby.Ruby.runFromMain(Ruby.java:354)
> >> > at org.jruby.Main.run(Main.java:229)
> >> > at org.jruby.Main.run(Main.java:110)
> >> > at org.jruby.Main.main(Main.java:94)
> >> >
> >>
> >
> >
> >
> >
> --
> *****************************************************
> Ching-Shen Chen
> Advanced Technology Center,
> Information & Communications Research Lab.
> E-mail: chenchingshen@itri.org.tw
> Tel:+886-3-5915542
> *****************************************************
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-2063) For hfileoutputformat, on
timeout/failure/kill clean up half-written hfile
Posted by "Ruslan Salyakhov (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ruslan Salyakhov updated HBASE-2063:
------------------------------------
Affects Version/s: 0.20.3
> For hfileoutputformat, on timeout/failure/kill clean up half-written hfile
> --------------------------------------------------------------------------
>
> Key: HBASE-2063
> URL: https://issues.apache.org/jira/browse/HBASE-2063
> Project: Hadoop HBase
> Issue Type: Bug
> Affects Versions: 0.20.3
> Reporter: stack
> Attachments: HBASE-2063-v1.patch
>
>
> Below is from mailing list. Read from bottom to top:
> {code}
> I was going to write that perhaps you needed to turn mapred.reduce.tasks.speculative.execution off, but if enabling it and things work, that would seem to indicate that a our reducer first takes longer than the task timeout maximum and secondly, on failure, we should clean up the hfile.
> On the first issue, you are using KeyValueSortReducer? Are your values large? We set reducer status every 100 values. Maybe this is not enough? We should set status more frequently? If you call context setstatus more frequently, do things work w/o speculative execution?
>
> On the second, HFileOutputFormat close will set the metadata on the hfile and then close it. On kill, this code is not being called. Let me see if can do something about that (e.g. register a shutdown hook to clean away incomplete files -- ).
> Thanks,
> St.Ack
> On Sun, Dec 20, 2009 at 11:26 PM, ChingShen <ch...@gmail.com> wrote:
> I think I found a way.
> I set the "mapred.reduce.tasks.speculative.execution" to true and output
> hfiles again, then successfully load hfiles into hbase.
> Is it best solution? or HFileOutputFormat bug?
> Shen
> On Mon, Dec 21, 2009 at 8:25 AM, ChingShen <ch...@gmail.com> wrote:
> > Thanks, stack.
> >
> > I checked this file that isn't empty. But I found that as long as the
> > "Killed Task Attempts" > 0 in reduce phase, and run the loadtable.rb script
> > to load hfiles then failed.
> > How to avoid this problem?
> >
> > Thanks.
> >
> > Shen
> >
> >
> > On Sat, Dec 19, 2009 at 3:49 AM, stack <st...@duboce.net> wrote:
> >
> >> Check the
> >> file
> >> hdfs://domU-12-31-39-09-C5-54.compute-1.internal/osm2_hfile/Level4/197894389945760574.
> >> Is it empty? Was there an error during running of your MR job? Perhaps
> >> a
> >> task failed?
> >>
> >> St.Ack
> >>
> >>
> >>
> >> On Thu, Dec 17, 2009 at 9:46 PM, ChingShen <ch...@gmail.com>
> >> wrote:
> >>
> >> > Hi,
> >> > I use the script loadtable.rb to load my hfiles into hbase, but I got
> >> an
> >> > exception as below.
> >> > Does anyone have any suggestions?
> >> >
> >> > 09/12/17 23:59:33 INFO loadtable: 18 read firstkey of -3.9290_52.5534
> >> from
> >> >
> >> >
> >> hdfs://domU-12-31-39-09-C5-54.compute-1.internal/osm2_hfile/Level4/1978943899457605747
> >> > org/apache/hadoop/hbase/io/hfile/HFile.java:1335:in `deserialize':
> >> > java.io.IOException: Trailer 'header' is wrong; does the trailer size
> >> match
> >> > content? (NativeException)
> >> > from org/apache/hadoop/hbase/io/hfile/HFile.java:813:in `readTrailer'
> >> > from org/apache/hadoop/hbase/io/hfile/HFile.java:758:in
> >> `loadFileInfo'
> >> > from sun.reflect.GeneratedMethodAccessor7:-1:in `invoke'
> >> > from sun/reflect/DelegatingMethodAccessorImpl.java:25:in `invoke'
> >> > from java/lang/reflect/Method.java:597:in `invoke'
> >> > from org/jruby/javasupport/JavaMethod.java:298:in
> >> > `invokeWithExceptionHandling'
> >> > from org/jruby/javasupport/JavaMethod.java:259:in `invoke'
> >> > from org/jruby/java/invokers/InstanceMethodInvoker.java:36:in `call'
> >> > ... 18 levels...
> >> > from org/jruby/Main.java:94:in `main'
> >> > from loadtable.rb:83:in `each'
> >> > from loadtable.rb:83
> >> > Complete Java stackTrace
> >> > java.io.IOException: Trailer 'header' is wrong; does the trailer size
> >> match
> >> > content?
> >> > at
> >> >
> >> >
> >> org.apache.hadoop.hbase.io.hfile.HFile$FixedFileTrailer.deserialize(HFile.java:1335)
> >> > at
> >> >
> >> org.apache.hadoop.hbase.io.hfile.HFile$Reader.readTrailer(HFile.java:813)
> >> > at
> >> >
> >> org.apache.hadoop.hbase.io.hfile.HFile$Reader.loadFileInfo(HFile.java:758)
> >> > at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> >> > at
> >> >
> >> >
> >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >> > at java.lang.reflect.Method.invoke(Method.java:597)
> >> > at
> >> >
> >> >
> >> org.jruby.javasupport.JavaMethod.invokeWithExceptionHandling(JavaMethod.java:298)
> >> > at org.jruby.javasupport.JavaMethod.invoke(JavaMethod.java:259)
> >> > at
> >> >
> >> >
> >> org.jruby.java.invokers.InstanceMethodInvoker.call(InstanceMethodInvoker.java:36)
> >> > at
> >> > org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:70)
> >> > at loadtable.ensure_1$RUBY$__ensure___2(loadtable.rb:86)
> >> > at loadtable.block_0$RUBY$__for__(loadtable.rb:85)
> >> > at loadtableBlockCallback$block_0$RUBY$__for__xx1.call(Unknown
> >> Source)
> >> > at org.jruby.runtime.CompiledBlock.yield(CompiledBlock.java:102)
> >> > at org.jruby.runtime.Block.yield(Block.java:100)
> >> > at
> >> org.jruby.java.proxies.ArrayJavaProxy.each(ArrayJavaProxy.java:112)
> >> > at
> >> >
> >> >
> >> org.jruby.java.proxies.ArrayJavaProxy$i_method_0_0$RUBYINVOKER$each.call(org/jruby/java/proxies/ArrayJavaProxy$i_method_0_0$RUBYINVOKER$each.gen)
> >> > at
> >> >
> >> >
> >> org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:263)
> >> > at
> >> >
> >> >
> >> org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:81)
> >> > at
> >> >
> >> >
> >> org.jruby.runtime.callsite.CachingCallSite.callIter(CachingCallSite.java:96)
> >> > at loadtable.__file__(loadtable.rb:83)
> >> > at loadtable.load(loadtable.rb)
> >> > at org.jruby.Ruby.runScript(Ruby.java:577)
> >> > at org.jruby.Ruby.runNormally(Ruby.java:480)
> >> > at org.jruby.Ruby.runFromMain(Ruby.java:354)
> >> > at org.jruby.Main.run(Main.java:229)
> >> > at org.jruby.Main.run(Main.java:110)
> >> > at org.jruby.Main.main(Main.java:94)
> >> >
> >>
> >
> >
> >
> >
> --
> *****************************************************
> Ching-Shen Chen
> Advanced Technology Center,
> Information & Communications Research Lab.
> E-mail: chenchingshen@itri.org.tw
> Tel:+886-3-5915542
> *****************************************************
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-2063) For hfileoutputformat, on
timeout/failure/kill clean up half-written hfile
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-2063:
-------------------------
Status: Patch Available (was: Open)
Marking patch available
> For hfileoutputformat, on timeout/failure/kill clean up half-written hfile
> --------------------------------------------------------------------------
>
> Key: HBASE-2063
> URL: https://issues.apache.org/jira/browse/HBASE-2063
> Project: Hadoop HBase
> Issue Type: Bug
> Affects Versions: 0.20.3
> Reporter: stack
> Fix For: 0.20.4, 0.21.0
>
> Attachments: HBASE-2063-v1.patch
>
>
> Below is from mailing list. Read from bottom to top:
> {code}
> I was going to write that perhaps you needed to turn mapred.reduce.tasks.speculative.execution off, but if enabling it and things work, that would seem to indicate that a our reducer first takes longer than the task timeout maximum and secondly, on failure, we should clean up the hfile.
> On the first issue, you are using KeyValueSortReducer? Are your values large? We set reducer status every 100 values. Maybe this is not enough? We should set status more frequently? If you call context setstatus more frequently, do things work w/o speculative execution?
>
> On the second, HFileOutputFormat close will set the metadata on the hfile and then close it. On kill, this code is not being called. Let me see if can do something about that (e.g. register a shutdown hook to clean away incomplete files -- ).
> Thanks,
> St.Ack
> On Sun, Dec 20, 2009 at 11:26 PM, ChingShen <ch...@gmail.com> wrote:
> I think I found a way.
> I set the "mapred.reduce.tasks.speculative.execution" to true and output
> hfiles again, then successfully load hfiles into hbase.
> Is it best solution? or HFileOutputFormat bug?
> Shen
> On Mon, Dec 21, 2009 at 8:25 AM, ChingShen <ch...@gmail.com> wrote:
> > Thanks, stack.
> >
> > I checked this file that isn't empty. But I found that as long as the
> > "Killed Task Attempts" > 0 in reduce phase, and run the loadtable.rb script
> > to load hfiles then failed.
> > How to avoid this problem?
> >
> > Thanks.
> >
> > Shen
> >
> >
> > On Sat, Dec 19, 2009 at 3:49 AM, stack <st...@duboce.net> wrote:
> >
> >> Check the
> >> file
> >> hdfs://domU-12-31-39-09-C5-54.compute-1.internal/osm2_hfile/Level4/197894389945760574.
> >> Is it empty? Was there an error during running of your MR job? Perhaps
> >> a
> >> task failed?
> >>
> >> St.Ack
> >>
> >>
> >>
> >> On Thu, Dec 17, 2009 at 9:46 PM, ChingShen <ch...@gmail.com>
> >> wrote:
> >>
> >> > Hi,
> >> > I use the script loadtable.rb to load my hfiles into hbase, but I got
> >> an
> >> > exception as below.
> >> > Does anyone have any suggestions?
> >> >
> >> > 09/12/17 23:59:33 INFO loadtable: 18 read firstkey of -3.9290_52.5534
> >> from
> >> >
> >> >
> >> hdfs://domU-12-31-39-09-C5-54.compute-1.internal/osm2_hfile/Level4/1978943899457605747
> >> > org/apache/hadoop/hbase/io/hfile/HFile.java:1335:in `deserialize':
> >> > java.io.IOException: Trailer 'header' is wrong; does the trailer size
> >> match
> >> > content? (NativeException)
> >> > from org/apache/hadoop/hbase/io/hfile/HFile.java:813:in `readTrailer'
> >> > from org/apache/hadoop/hbase/io/hfile/HFile.java:758:in
> >> `loadFileInfo'
> >> > from sun.reflect.GeneratedMethodAccessor7:-1:in `invoke'
> >> > from sun/reflect/DelegatingMethodAccessorImpl.java:25:in `invoke'
> >> > from java/lang/reflect/Method.java:597:in `invoke'
> >> > from org/jruby/javasupport/JavaMethod.java:298:in
> >> > `invokeWithExceptionHandling'
> >> > from org/jruby/javasupport/JavaMethod.java:259:in `invoke'
> >> > from org/jruby/java/invokers/InstanceMethodInvoker.java:36:in `call'
> >> > ... 18 levels...
> >> > from org/jruby/Main.java:94:in `main'
> >> > from loadtable.rb:83:in `each'
> >> > from loadtable.rb:83
> >> > Complete Java stackTrace
> >> > java.io.IOException: Trailer 'header' is wrong; does the trailer size
> >> match
> >> > content?
> >> > at
> >> >
> >> >
> >> org.apache.hadoop.hbase.io.hfile.HFile$FixedFileTrailer.deserialize(HFile.java:1335)
> >> > at
> >> >
> >> org.apache.hadoop.hbase.io.hfile.HFile$Reader.readTrailer(HFile.java:813)
> >> > at
> >> >
> >> org.apache.hadoop.hbase.io.hfile.HFile$Reader.loadFileInfo(HFile.java:758)
> >> > at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> >> > at
> >> >
> >> >
> >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >> > at java.lang.reflect.Method.invoke(Method.java:597)
> >> > at
> >> >
> >> >
> >> org.jruby.javasupport.JavaMethod.invokeWithExceptionHandling(JavaMethod.java:298)
> >> > at org.jruby.javasupport.JavaMethod.invoke(JavaMethod.java:259)
> >> > at
> >> >
> >> >
> >> org.jruby.java.invokers.InstanceMethodInvoker.call(InstanceMethodInvoker.java:36)
> >> > at
> >> > org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:70)
> >> > at loadtable.ensure_1$RUBY$__ensure___2(loadtable.rb:86)
> >> > at loadtable.block_0$RUBY$__for__(loadtable.rb:85)
> >> > at loadtableBlockCallback$block_0$RUBY$__for__xx1.call(Unknown
> >> Source)
> >> > at org.jruby.runtime.CompiledBlock.yield(CompiledBlock.java:102)
> >> > at org.jruby.runtime.Block.yield(Block.java:100)
> >> > at
> >> org.jruby.java.proxies.ArrayJavaProxy.each(ArrayJavaProxy.java:112)
> >> > at
> >> >
> >> >
> >> org.jruby.java.proxies.ArrayJavaProxy$i_method_0_0$RUBYINVOKER$each.call(org/jruby/java/proxies/ArrayJavaProxy$i_method_0_0$RUBYINVOKER$each.gen)
> >> > at
> >> >
> >> >
> >> org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:263)
> >> > at
> >> >
> >> >
> >> org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:81)
> >> > at
> >> >
> >> >
> >> org.jruby.runtime.callsite.CachingCallSite.callIter(CachingCallSite.java:96)
> >> > at loadtable.__file__(loadtable.rb:83)
> >> > at loadtable.load(loadtable.rb)
> >> > at org.jruby.Ruby.runScript(Ruby.java:577)
> >> > at org.jruby.Ruby.runNormally(Ruby.java:480)
> >> > at org.jruby.Ruby.runFromMain(Ruby.java:354)
> >> > at org.jruby.Main.run(Main.java:229)
> >> > at org.jruby.Main.run(Main.java:110)
> >> > at org.jruby.Main.main(Main.java:94)
> >> >
> >>
> >
> >
> >
> >
> --
> *****************************************************
> Ching-Shen Chen
> Advanced Technology Center,
> Information & Communications Research Lab.
> E-mail: chenchingshen@itri.org.tw
> Tel:+886-3-5915542
> *****************************************************
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.