Re: how to figure out the range of a split that failed?

[edward choi <mp...@gmail.com>]

Dear Sharad,

I have come across another problem. I hope you can help me with this too.
I am trying to use SkipBadRecords feature on Hadoop Streaming.
The streaming method I use is: "hadoop jar
$HADOOP_INSTALL/contrib/streaming/hadoop-*-streaming.jar"
But your example uses Java application which I cannot use because I am
trying to use a C++ application connecting it with Hadoop Streaming.

So what I am doing is:
hadoop jar $HADOOP_INSTALL/contrib/streaming/hadoop-*-streaming.jar -D
mapred.skip.mode.enabled=true -D mapred.skip.attempts.to.start.skipping=2 -D
mapred.skip.map.max.skip.records=Long.MAX_VALUE -D mapred.reduce.tasks=0
-file "..." -mapper "..." -input "..." -output "..."

Then I noticed that you have to set
"mapred.skip.map.auto.incr.proc.count=false" and increment
COUNTER_MAP_PROCESSED_RECORDS in your own application. I guess that you can
do this in your example, but I don't know how to do it using my way of
Hadoop Streaming. Could you enlighten me please?

Sincerely, Ed

2010/6/30 Sharad Agarwal <sh...@yahoo-inc.com>

> edward choi wrote:
>
>> Thanks for the quick response.
>> I know the SkipBadRecords feature but unfortunately I cannot use it since
>> I
>> am running my job on Hadoop Streaming.
>> I had asked if there were any way to use SkipBadRecords in Hadoop
>> Streaming
>> but never got an answer. I guess it is not possible at all.
>> Thanks for your concern.
>>
>>
> SkipBadRecords feature can be used for streaming as well. Perhaps the best
> example is the testcase
> ->
> http://svn.apache.org/viewvc/hadoop/mapreduce/trunk/src/contrib/streaming/src/test/org/apache/hadoop/streaming/TestStreamingBadRecords.java?view=markup
>
> Sharad
>

Re: how to figure out the range of a split that failed?

[edward choi <mp...@gmail.com>]

Thanks for the tip.
I actually already have tried your method. The command I wrote is like below

cerr << "reporter:counter:SkippingTaskCounters,MapProcessedRecords,1\n";

This actually produced some skipped records in skip folder. But the problem
is that the skipped records' text was all messed up. So I couldn't recycle
them. The broken text is at the end of this mail.
I don't know the reason. Maybe it's because I wrote some other information
on error stream(such as document ID. The command below is the one)

cerr << "Processing: " << docID << endl;

Anyway if I happen to have any progress, I will update through this post.

Broken Text:
SEQ!org.apache.hadoop.io.LongWritableorg.apache.hadoop.io.Text

*org.apache.hadoop.io.compress.DefaultCodec�9/S庇4z����9/S庇4z��
x��x�`
[0텬�,
wx�c�c]
r8�$x�U]oW}i�H�Hi��Z�b+X�r�y접�"H���\��탈�y��
6�%�b��EMLの]`[}Ø�}�則�霙�綴�愁P�<�q�賽���\��5~庾X�xp�c≠�=��<�嵬p|3:�B�A?-8�勁쫑�2��_�@���F#A�6
G�}殘�蛛甦Q�$潾�,8��0鉛=�츈^曠��Ｇ�└&?日���;=~兪����묽��씔�nl�o]�판��M�d橫罹�|잣��C�噫�0����p`���i�U���z�4t^d芚Mc}�qf��뎬�����_^朽c���V��}S�V콸�Q�%κ쐴*�O_�����s������:�kn��(����b��RX�+oh
S��8���梁=㈍哥�0%DGw當�.
�3
V���y�분�!A�0%�;���瀏��茁�'&?
�N�jP����$��� %�m��밑!c��ls����<``% �����廣�>Y�i���璽��Fb賃$x좃_�W�L
�\F�gi�Ix8���廟別<�毫�<u9�BI(5S��ｕ��後�
#��~��弛欒��鯖�`h�����|�9�c�高�:課Y�Q��������銳�占�쇽��p
-�cK��qY%�OR���55�pJ��潁~��Qt�>HywK�\Dz觴���U��(����/����渾�Re'%�s0k咫키L�%{K$��U+���1�B5�7j�-�f~��~頃�K�`c\�G�+t-��dJc|s��b�vWA�2�荳�f�X�却"��M�����W�.����D=O~��/���$
s��헷1M�e����긺 ^�9��1�揷�]奸��;���0-�굇���
�w욜=l泊�踰=c��u��瓣��S��*�����奈y1룡bM]��X�씬�7h짼Zx�\琅윰'��n?-�s�?��q�弘벧MV�*距�已희�岸侖:N�����刃2�


2010/7/6 Sharad Agarwal <sh...@yahoo-inc.com>

> to be precise you have to write on error stream ->
> for map:
> reporter:counter:SkippingTaskCounters,MapProcessedRecords,<count>
>
> for reduce:
> reporter:counter:SkippingTaskCounters,ReduceProcessedGroups,<count>
>
>
> edward choi wrote:
>
>> Thanks for the response. I went to the web page you told me and several
>> other pages that I found.
>> I am still not sure if I got it right.
>> If I am trying to increment COUNTER_MAP_PROCESS_RECORDS using Hadoop
>> Streaming, is the example below the way to do it? (assuming that I am using
>> c++)
>>
>> example:
>> cerr << "reporter:counter:counters,linecount,1" << endl;
>>
>>
>>
>

Re: how to figure out the range of a split that failed?

[Sharad Agarwal <sh...@yahoo-inc.com>]

to be precise you have to write on error stream ->
for map:
reporter:counter:SkippingTaskCounters,MapProcessedRecords,<count>

for reduce:
reporter:counter:SkippingTaskCounters,ReduceProcessedGroups,<count>

edward choi wrote:
> Thanks for the response. I went to the web page you told me and 
> several other pages that I found.
> I am still not sure if I got it right.
> If I am trying to increment COUNTER_MAP_PROCESS_RECORDS using Hadoop 
> Streaming, is the example below the way to do it? (assuming that I am 
> using c++)
>
> example:
> cerr << "reporter:counter:counters,linecount,1" << endl;
>
>

Re: how to figure out the range of a split that failed?

[edward choi <mp...@gmail.com>]

Thanks for the response. I went to the web page you told me and several
other pages that I found.
I am still not sure if I got it right.
If I am trying to increment COUNTER_MAP_PROCESS_RECORDS using Hadoop
Streaming, is the example below the way to do it? (assuming that I am using
c++)

example:
cerr << "reporter:counter:counters,linecount,1" << endl;

I am not sure how to use "reporter:counter:<group>,<counter>,<amount>".
I wrote the example by referring to "
https://issues.apache.org/jira/browse/HADOOP-1328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
"
Am I doing it right?

2010/7/1 Sharad Agarwal <sh...@yahoo-inc.com>

> edward choi wrote:
>
>>
>>
>> Then I noticed that you have to set
>> "mapred.skip.map.auto.incr.proc.count=false" and increment
>> COUNTER_MAP_PROCESSED_RECORDS in your own application. I guess that you can
>> do this in your example, but I don't know how to do it using my way of
>> Hadoop Streaming. Could you enlighten me please?
>>
>>  You can refer
>
> http://hadoop.apache.org/common/docs/r0.20.0/streaming.html#How+do+I+update+counters+in+streaming+applications%3F
> for knowing how to increment counter in streaming.
>
>
> Sharad
>

Re: how to figure out the range of a split that failed?

[Sharad Agarwal <sh...@yahoo-inc.com>]

edward choi wrote:
>
>
> Then I noticed that you have to set 
> "mapred.skip.map.auto.incr.proc.count=false" and increment 
> COUNTER_MAP_PROCESSED_RECORDS in your own application. I guess that 
> you can do this in your example, but I don't know how to do it using 
> my way of Hadoop Streaming. Could you enlighten me please?
>
You can refer
http://hadoop.apache.org/common/docs/r0.20.0/streaming.html#How+do+I+update+counters+in+streaming+applications%3F 

for knowing how to increment counter in streaming.


Sharad