Issue with partitioning using streaming

[Aleksandr Elbakyan <ra...@yahoo.com>]

Hello All,

I am trying to partition data and sort it in hadoop streaming. 


Most of the time the data is sorted and partitioned correctly but if I run multiple times sometimes data goes to other partition 




The data looks like

asdas 0 ada
asdas 1 asd
12123 1 ccc
12123 0 xxx



  hadoop  jar ${HADOOP_HOME}/contrib/streaming/hadoop-*streaming.jar \
        -D mapred.task.timeout=3600000 \
        -D mapred.map.tasks=${GD_NUM_MAP_TASKS}  \
        -D mapred.reduce.tasks=${GD_NUM_REDUCE_TASKS} \
        -D stream.non.zero.exit.is.failure=true \
        -D stream.num.map.output.key.fields=2 \
        -D mapred.text.key.partitioner.options="-k1,1" \
        -D mapred.output.key.comparator.class=org.apache.hadoop.mapred.lib.KeyFieldBasedComparator \
        -D mapred.text.key.comparator.options=-k1,2n \
        -partitioner org.apache.hadoop.mapred.lib.KeyFieldBasedPartitioner \
        -input input \
        -output output \
        -mapper  "  cat" \
        -reducer " cat" \
        -verbose


in reducer code I have some logic which depend on correct partitioning and sorting.


Regards.