You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@hbase.apache.org by donhoff_h <16...@qq.com> on 2015/04/27 11:30:25 UTC

completebulkload problem

Hi, experts.

I wrote a program to generate a HFile and then used the completebulkload command to load it. The HFile contained both Put and Delete cells. After I loaded it, I found all cells except one were taken effect. The problem cell was a delete cell. Its timestamp was bigger than any other delete cells. Following is part of the content from the HFile. The line in read are the problem cell.
I executed the command many times. The results are the same. It is always that cell could not take effect.  Could anyone help me to solve it? Many thanks!
By the way, the HBase version is 0.98.3 and the Hadoop version is 2.4.0.

K: 2014-10-14#1127158#3/a:F074/1430123822035/Put/vlen=1/mvcc=0 V: 0
K: 2014-10-14#1127158#3/a:F075/1430123822035/Delete/vlen=0/mvcc=0 V:
K: 2014-10-14#1127158#3/a:F076/1430123822035/Delete/vlen=0/mvcc=0 V:
K: 2014-10-14#1127158#3/a:F077/1430123822035/Delete/vlen=0/mvcc=0 V:
K: 2014-10-14#1127295#1/a:F004/1430123822052/Put/vlen=6/mvcc=0 V: 801021
K: 2014-10-14#1127295#1/a:F005/1430123822052/Put/vlen=1/mvcc=0 V: a
K: 2014-10-14#1127295#1/a:F006/1430123822052/Delete/vlen=0/mvcc=0 V:
K: 2014-10-14#1127295#1/a:F007/1430123822052/Put/vlen=4/mvcc=0 V: 0100
K: 2014-10-14#1127295#1/a:F008/1430123822052/Put/vlen=4/mvcc=0 V: 0920
K: 2014-10-14#1127295#1/a:F009/1430123822052/Put/vlen=4/mvcc=0 V: 0901
K: 2014-10-14#1127295#1/a:F010/1430123822051/Put/vlen=4/mvcc=0 V: 1001
K: 2014-10-14#1127295#1/a:F011/1430123822051/Delete/vlen=0/mvcc=0 V:
K: 2014-10-14#1127295#1/a:F012/1430123822051/Delete/vlen=0/mvcc=0 V:

Re: completebulkload problem

Posted by donhoff_h <16...@qq.com>.

Hi, Ted

Yes, I find the color was lost. The delete cell which did not take effect is : K: 2014-10-14#1127295#1/a:F006/1430123822052/Delete/vlen=0/mvcc=0 V:
And I generated the HFile by a mapreduce program.  The correspond code is as following,the variable "qualifierAndValue" is a String array. qualifierAndValue[0] is the name of the quailifer, qualiferAndValue[1] is its value.
while(...){
  if(qualifierAndValue!=null) {
    if(qualiferAndValue.length==2) {
       //Here are the codes generating Put cell. Since Put Cells were all taken effect, I did not show the code to save some space.
      ...
    } else {
      //here are the codes generating Delete cell
      cell = CellUtil.createCell(key.get(),family,Bytes.toBytes(qualiferAndValue[0]),System.currentTimeMIllis(),KeyValue.Type.Delete.getCode(),null);
     treeSet.add(cell)
    }
  }
} //End of While

for(Cell e: treeSet) context.write(key,e)



------------------ Original ------------------
From:  "yuzhihong";<yu...@gmail.com>;
Send time: Monday, Apr 27, 2015 9:44 PM
To: "user@hbase.apache.org"<us...@hbase.apache.org>; 

Subject:  Re: completebulkload problem



There're several Delete cells in the output and color was lost.
Can you let us know which line the Delete cell is on ?

Is it this one ?
K: 2014-10-14#1127295#1/a:F006/1430123822052/Delete/vlen=0/mvcc=0 V:

Can you describe how your program generates HFile ?

Thanks

On Mon, Apr 27, 2015 at 2:30 AM, donhoff_h <16...@qq.com> wrote:

> Hi, experts.
>
> I wrote a program to generate a HFile and then used the completebulkload
> command to load it. The HFile contained both Put and Delete cells. After I
> loaded it, I found all cells except one were taken effect. The problem cell
> was a delete cell. Its timestamp was bigger than any other delete cells.
> Following is part of the content from the HFile. The line in read are the
> problem cell.
> I executed the command many times. The results are the same. It is always
> that cell could not take effect.  Could anyone help me to solve it? Many
> thanks!
> By the way, the HBase version is 0.98.3 and the Hadoop version is 2.4.0.
>
> K: 2014-10-14#1127158#3/a:F074/1430123822035/Put/vlen=1/mvcc=0 V: 0
> K: 2014-10-14#1127158#3/a:F075/1430123822035/Delete/vlen=0/mvcc=0 V:
> K: 2014-10-14#1127158#3/a:F076/1430123822035/Delete/vlen=0/mvcc=0 V:
> K: 2014-10-14#1127158#3/a:F077/1430123822035/Delete/vlen=0/mvcc=0 V:
> K: 2014-10-14#1127295#1/a:F004/1430123822052/Put/vlen=6/mvcc=0 V: 801021
> K: 2014-10-14#1127295#1/a:F005/1430123822052/Put/vlen=1/mvcc=0 V: a
> K: 2014-10-14#1127295#1/a:F006/1430123822052/Delete/vlen=0/mvcc=0 V:
> K: 2014-10-14#1127295#1/a:F007/1430123822052/Put/vlen=4/mvcc=0 V: 0100
> K: 2014-10-14#1127295#1/a:F008/1430123822052/Put/vlen=4/mvcc=0 V: 0920
> K: 2014-10-14#1127295#1/a:F009/1430123822052/Put/vlen=4/mvcc=0 V: 0901
> K: 2014-10-14#1127295#1/a:F010/1430123822051/Put/vlen=4/mvcc=0 V: 1001
> K: 2014-10-14#1127295#1/a:F011/1430123822051/Delete/vlen=0/mvcc=0 V:
> K: 2014-10-14#1127295#1/a:F012/1430123822051/Delete/vlen=0/mvcc=0 V:

Re: completebulkload problem

Posted by Ted Yu <yu...@gmail.com>.

Thanks for the update. 



> On Apr 27, 2015, at 9:35 PM, donhoff_h <16...@qq.com> wrote:
> 
> Hi, Ted
> 
> This problem has been resolved.  I should use KeyValue.Type.DeleteColumn instead of KeyValue.Type.Delete . KeyValue.Type.Delete only deletes the cells with specified timestamp, while KeyValue.Type.DeleteColumn deletes cells which timestamp is smaller than or equal to the specified timestamp. Just share this if any other people meet the same problem as me.
> 
> 
> 
> ------------------ Original ------------------
> From:  "yuzhihong";<yu...@gmail.com>;
> Send time: Monday, Apr 27, 2015 9:44 PM
> To: "user@hbase.apache.org"<us...@hbase.apache.org>; 
> 
> Subject:  Re: completebulkload problem
> 
> 
> 
> There're several Delete cells in the output and color was lost.
> Can you let us know which line the Delete cell is on ?
> 
> Is it this one ?
> K: 2014-10-14#1127295#1/a:F006/1430123822052/Delete/vlen=0/mvcc=0 V:
> 
> Can you describe how your program generates HFile ?
> 
> Thanks
> 
>> On Mon, Apr 27, 2015 at 2:30 AM, donhoff_h <16...@qq.com> wrote:
>> 
>> Hi, experts.
>> 
>> I wrote a program to generate a HFile and then used the completebulkload
>> command to load it. The HFile contained both Put and Delete cells. After I
>> loaded it, I found all cells except one were taken effect. The problem cell
>> was a delete cell. Its timestamp was bigger than any other delete cells.
>> Following is part of the content from the HFile. The line in read are the
>> problem cell.
>> I executed the command many times. The results are the same. It is always
>> that cell could not take effect.  Could anyone help me to solve it? Many
>> thanks!
>> By the way, the HBase version is 0.98.3 and the Hadoop version is 2.4.0.
>> 
>> K: 2014-10-14#1127158#3/a:F074/1430123822035/Put/vlen=1/mvcc=0 V: 0
>> K: 2014-10-14#1127158#3/a:F075/1430123822035/Delete/vlen=0/mvcc=0 V:
>> K: 2014-10-14#1127158#3/a:F076/1430123822035/Delete/vlen=0/mvcc=0 V:
>> K: 2014-10-14#1127158#3/a:F077/1430123822035/Delete/vlen=0/mvcc=0 V:
>> K: 2014-10-14#1127295#1/a:F004/1430123822052/Put/vlen=6/mvcc=0 V: 801021
>> K: 2014-10-14#1127295#1/a:F005/1430123822052/Put/vlen=1/mvcc=0 V: a
>> K: 2014-10-14#1127295#1/a:F006/1430123822052/Delete/vlen=0/mvcc=0 V:
>> K: 2014-10-14#1127295#1/a:F007/1430123822052/Put/vlen=4/mvcc=0 V: 0100
>> K: 2014-10-14#1127295#1/a:F008/1430123822052/Put/vlen=4/mvcc=0 V: 0920
>> K: 2014-10-14#1127295#1/a:F009/1430123822052/Put/vlen=4/mvcc=0 V: 0901
>> K: 2014-10-14#1127295#1/a:F010/1430123822051/Put/vlen=4/mvcc=0 V: 1001
>> K: 2014-10-14#1127295#1/a:F011/1430123822051/Delete/vlen=0/mvcc=0 V:
>> K: 2014-10-14#1127295#1/a:F012/1430123822051/Delete/vlen=0/mvcc=0 V:

Re: completebulkload problem

Posted by donhoff_h <16...@qq.com>.

Hi, Ted

This problem has been resolved.  I should use KeyValue.Type.DeleteColumn instead of KeyValue.Type.Delete . KeyValue.Type.Delete only deletes the cells with specified timestamp, while KeyValue.Type.DeleteColumn deletes cells which timestamp is smaller than or equal to the specified timestamp. Just share this if any other people meet the same problem as me.



------------------ Original ------------------
From:  "yuzhihong";<yu...@gmail.com>;
Send time: Monday, Apr 27, 2015 9:44 PM
To: "user@hbase.apache.org"<us...@hbase.apache.org>; 

Subject:  Re: completebulkload problem



There're several Delete cells in the output and color was lost.
Can you let us know which line the Delete cell is on ?

Is it this one ?
K: 2014-10-14#1127295#1/a:F006/1430123822052/Delete/vlen=0/mvcc=0 V:

Can you describe how your program generates HFile ?

Thanks

On Mon, Apr 27, 2015 at 2:30 AM, donhoff_h <16...@qq.com> wrote:

> Hi, experts.
>
> I wrote a program to generate a HFile and then used the completebulkload
> command to load it. The HFile contained both Put and Delete cells. After I
> loaded it, I found all cells except one were taken effect. The problem cell
> was a delete cell. Its timestamp was bigger than any other delete cells.
> Following is part of the content from the HFile. The line in read are the
> problem cell.
> I executed the command many times. The results are the same. It is always
> that cell could not take effect.  Could anyone help me to solve it? Many
> thanks!
> By the way, the HBase version is 0.98.3 and the Hadoop version is 2.4.0.
>
> K: 2014-10-14#1127158#3/a:F074/1430123822035/Put/vlen=1/mvcc=0 V: 0
> K: 2014-10-14#1127158#3/a:F075/1430123822035/Delete/vlen=0/mvcc=0 V:
> K: 2014-10-14#1127158#3/a:F076/1430123822035/Delete/vlen=0/mvcc=0 V:
> K: 2014-10-14#1127158#3/a:F077/1430123822035/Delete/vlen=0/mvcc=0 V:
> K: 2014-10-14#1127295#1/a:F004/1430123822052/Put/vlen=6/mvcc=0 V: 801021
> K: 2014-10-14#1127295#1/a:F005/1430123822052/Put/vlen=1/mvcc=0 V: a
> K: 2014-10-14#1127295#1/a:F006/1430123822052/Delete/vlen=0/mvcc=0 V:
> K: 2014-10-14#1127295#1/a:F007/1430123822052/Put/vlen=4/mvcc=0 V: 0100
> K: 2014-10-14#1127295#1/a:F008/1430123822052/Put/vlen=4/mvcc=0 V: 0920
> K: 2014-10-14#1127295#1/a:F009/1430123822052/Put/vlen=4/mvcc=0 V: 0901
> K: 2014-10-14#1127295#1/a:F010/1430123822051/Put/vlen=4/mvcc=0 V: 1001
> K: 2014-10-14#1127295#1/a:F011/1430123822051/Delete/vlen=0/mvcc=0 V:
> K: 2014-10-14#1127295#1/a:F012/1430123822051/Delete/vlen=0/mvcc=0 V:

Re: completebulkload problem

Posted by Ted Yu <yu...@gmail.com>.

There're several Delete cells in the output and color was lost.
Can you let us know which line the Delete cell is on ?

Is it this one ?
K: 2014-10-14#1127295#1/a:F006/1430123822052/Delete/vlen=0/mvcc=0 V:

Can you describe how your program generates HFile ?

Thanks

On Mon, Apr 27, 2015 at 2:30 AM, donhoff_h <16...@qq.com> wrote:

> Hi, experts.
>
> I wrote a program to generate a HFile and then used the completebulkload
> command to load it. The HFile contained both Put and Delete cells. After I
> loaded it, I found all cells except one were taken effect. The problem cell
> was a delete cell. Its timestamp was bigger than any other delete cells.
> Following is part of the content from the HFile. The line in read are the
> problem cell.
> I executed the command many times. The results are the same. It is always
> that cell could not take effect.  Could anyone help me to solve it? Many
> thanks!
> By the way, the HBase version is 0.98.3 and the Hadoop version is 2.4.0.
>
> K: 2014-10-14#1127158#3/a:F074/1430123822035/Put/vlen=1/mvcc=0 V: 0
> K: 2014-10-14#1127158#3/a:F075/1430123822035/Delete/vlen=0/mvcc=0 V:
> K: 2014-10-14#1127158#3/a:F076/1430123822035/Delete/vlen=0/mvcc=0 V:
> K: 2014-10-14#1127158#3/a:F077/1430123822035/Delete/vlen=0/mvcc=0 V:
> K: 2014-10-14#1127295#1/a:F004/1430123822052/Put/vlen=6/mvcc=0 V: 801021
> K: 2014-10-14#1127295#1/a:F005/1430123822052/Put/vlen=1/mvcc=0 V: a
> K: 2014-10-14#1127295#1/a:F006/1430123822052/Delete/vlen=0/mvcc=0 V:
> K: 2014-10-14#1127295#1/a:F007/1430123822052/Put/vlen=4/mvcc=0 V: 0100
> K: 2014-10-14#1127295#1/a:F008/1430123822052/Put/vlen=4/mvcc=0 V: 0920
> K: 2014-10-14#1127295#1/a:F009/1430123822052/Put/vlen=4/mvcc=0 V: 0901
> K: 2014-10-14#1127295#1/a:F010/1430123822051/Put/vlen=4/mvcc=0 V: 1001
> K: 2014-10-14#1127295#1/a:F011/1430123822051/Delete/vlen=0/mvcc=0 V:
> K: 2014-10-14#1127295#1/a:F012/1430123822051/Delete/vlen=0/mvcc=0 V: