You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@accumulo.apache.org by "Bell, Philip S CIV SPAWARSYSCEN-PACIFIC, 81320" <ph...@navy.mil> on 2013/04/11 00:02:22 UTC
FW: Deletes not commiting from custom java API but work in shell interface
Using the following code, rows are never deleted even when identified and submitted for deletion to the batchwriter.
After running the code the rows still show up in search.
This has been seen in 1.4.1 and 1.4.2
The same rows can be deleted from the accumulo shell interface
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
for( Entry<Key, Value> e : mainScanner )
{
Text currentUUID = e.getKey().getRow();
Text colFam = e.getKey().getColumnFamily();
Text colQual = e.getKey().getColumnQualifier();
String colVis = e.getKey().getColumnVisibility().toString();
System.out.println( currentUUID + ":" + colFam + ":" + colQual + ":" + colVis + ":" + e.getKey().getTimestamp() );
if( colFam.toString().equalsIgnoreCase( "root" ) || colVis.length() > 0 )
{
Mutation delMutation = new Mutation( currentUUID );
delMutation.putDelete( colFam, colQual, new ColumnVisibility( colVis ), e.getKey().getTimestamp() );
System.out.println( "removing" );
try
{
bw.addMutation( delMutation );
}
catch( MutationsRejectedException e1 )
{
e1.printStackTrace();
}
count++;
if( count % 1000000 == 0 )
{
System.out.println( this.getName() + ": " + count + " completed in " + getTimePassed() );
}
}
}// for each entry found
try
{
bw.flush();
bw.close();
}
catch( Exception e1 )
{
e1.printStackTrace();
}
Re: FW: Deletes not commiting from custom java API but work in shell interface
Posted by John Vines <vi...@apache.org>.
Thanks for the clarification Keith, I wasn't 100% on that.
On Thu, Apr 11, 2013 at 10:01 AM, Keith Turner <ke...@deenlo.com> wrote:
> On Wed, Apr 10, 2013 at 6:06 PM, John Vines <vi...@apache.org> wrote:
> > Deletes will remove all entries which occur before the key. I believe
> there
> > is undefined behavior for when a delete key is identical. It should work
> if
> > you set the delete keys time +1.
>
> The behavior is defined when keys are identical except for delete.
> Delete is the last thing a key is sorted on, and delete keys sort
> before non delete keys. So a delete key with timestamp T will delete
> all keys with the same row and column with timestamps <= T.
>
> >
> > Sent from my phone, please pardon the typos and brevity.
> >
> > On Apr 10, 2013 6:02 PM, "Bell, Philip S CIV SPAWARSYSCEN-PACIFIC, 81320"
> > <ph...@navy.mil> wrote:
> >>
> >> Using the following code, rows are never deleted even when identified
> and
> >> submitted for deletion to the batchwriter.
> >> After running the code the rows still show up in search.
> >>
> >> This has been seen in 1.4.1 and 1.4.2
> >>
> >> The same rows can be deleted from the accumulo shell interface
> >>
> >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>
> >>
> >> for( Entry<Key, Value> e : mainScanner )
> >> {
> >> Text currentUUID = e.getKey().getRow();
> >>
> >> Text colFam = e.getKey().getColumnFamily();
> >> Text colQual = e.getKey().getColumnQualifier();
> >>
> >> String colVis = e.getKey().getColumnVisibility().toString();
> >>
> >> System.out.println( currentUUID + ":" + colFam + ":" + colQual +
> >> ":" + colVis + ":" + e.getKey().getTimestamp() );
> >>
> >> if( colFam.toString().equalsIgnoreCase( "root" ) ||
> >> colVis.length() > 0 )
> >> {
> >> Mutation delMutation = new Mutation( currentUUID );
> >> delMutation.putDelete( colFam, colQual, new
> >> ColumnVisibility( colVis ), e.getKey().getTimestamp() );
> >>
> >> System.out.println( "removing" );
> >>
> >> try
> >> {
> >> bw.addMutation( delMutation );
> >> }
> >> catch( MutationsRejectedException e1 )
> >> {
> >> e1.printStackTrace();
> >> }
> >>
> >> count++;
> >> if( count % 1000000 == 0 )
> >> {
> >> System.out.println( this.getName() + ": " +
> count
> >> + " completed in " + getTimePassed() );
> >> }
> >> }
> >> }// for each entry found
> >>
> >> try
> >> {
> >> bw.flush();
> >> bw.close();
> >> }
> >> catch( Exception e1 )
> >> {
> >> e1.printStackTrace();
> >> }
>
Re: FW: Deletes not commiting from custom java API but work in shell interface
Posted by Keith Turner <ke...@deenlo.com>.
On Wed, Apr 10, 2013 at 6:06 PM, John Vines <vi...@apache.org> wrote:
> Deletes will remove all entries which occur before the key. I believe there
> is undefined behavior for when a delete key is identical. It should work if
> you set the delete keys time +1.
The behavior is defined when keys are identical except for delete.
Delete is the last thing a key is sorted on, and delete keys sort
before non delete keys. So a delete key with timestamp T will delete
all keys with the same row and column with timestamps <= T.
>
> Sent from my phone, please pardon the typos and brevity.
>
> On Apr 10, 2013 6:02 PM, "Bell, Philip S CIV SPAWARSYSCEN-PACIFIC, 81320"
> <ph...@navy.mil> wrote:
>>
>> Using the following code, rows are never deleted even when identified and
>> submitted for deletion to the batchwriter.
>> After running the code the rows still show up in search.
>>
>> This has been seen in 1.4.1 and 1.4.2
>>
>> The same rows can be deleted from the accumulo shell interface
>>
>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>
>>
>> for( Entry<Key, Value> e : mainScanner )
>> {
>> Text currentUUID = e.getKey().getRow();
>>
>> Text colFam = e.getKey().getColumnFamily();
>> Text colQual = e.getKey().getColumnQualifier();
>>
>> String colVis = e.getKey().getColumnVisibility().toString();
>>
>> System.out.println( currentUUID + ":" + colFam + ":" + colQual +
>> ":" + colVis + ":" + e.getKey().getTimestamp() );
>>
>> if( colFam.toString().equalsIgnoreCase( "root" ) ||
>> colVis.length() > 0 )
>> {
>> Mutation delMutation = new Mutation( currentUUID );
>> delMutation.putDelete( colFam, colQual, new
>> ColumnVisibility( colVis ), e.getKey().getTimestamp() );
>>
>> System.out.println( "removing" );
>>
>> try
>> {
>> bw.addMutation( delMutation );
>> }
>> catch( MutationsRejectedException e1 )
>> {
>> e1.printStackTrace();
>> }
>>
>> count++;
>> if( count % 1000000 == 0 )
>> {
>> System.out.println( this.getName() + ": " + count
>> + " completed in " + getTimePassed() );
>> }
>> }
>> }// for each entry found
>>
>> try
>> {
>> bw.flush();
>> bw.close();
>> }
>> catch( Exception e1 )
>> {
>> e1.printStackTrace();
>> }
Re: FW: Deletes not commiting from custom java API but work in shell
interface
Posted by Josh Elser <jo...@gmail.com>.
Another gotcha to be aware of.. make sure your ColumnVisibilities are
equal and not just equivalent.
e.g. a CV of (FOO&BAR) won't delete a Key which has (BAR&FOO)
You can easily get around this by making sure flatten is called on the
ColumnVisibility. I don't think the normal constructors do this for you
(only quickly glanced at the code again).
On 04/10/2013 06:42 PM, Bell, Philip S CIV SPAWARSYSCEN-PACIFIC, 81320
wrote:
> Sanity check complete, running on a VM from the training class (though this is accumulo 1.3.5) deletes work with and without column visibility - I'll be back with more once our development servers are back up
RE: FW: Deletes not commiting from custom java API but work in shell interface
Posted by "Bell, Philip S CIV SPAWARSYSCEN-PACIFIC, 81320" <ph...@navy.mil>.
Sanity check complete, running on a VM from the training class (though this is accumulo 1.3.5) deletes work with and without column visibility - I'll be back with more once our development servers are back up
Re: FW: Deletes not commiting from custom java API but work in shell interface
Posted by Billie Rinaldi <bi...@gmail.com>.
On Wed, Apr 10, 2013 at 3:20 PM, Bell, Philip S CIV SPAWARSYSCEN-PACIFIC,
81320 <ph...@navy.mil> wrote:
> We have not as of yet implemented any column visibility
>
You mentioned you haven't used column visibility, while this line of your
code is only considering entries that have a nonempty column visibility.
Are these conditions true for the entries you're trying to delete?
if (colFam.toString().equalsIgnoreCase("root") || colVis.length() > 0)
Billie
> When I first saw the problem, I created a separate table with some very
> generic test data to try out the deletion, and nothing I do from the java
> API seems to actually cause a delete to go through. I assume if deletes
> work from the shell then the delete iterator is working correctly, hence
> I'm out of ideas.
>
> To add to the fun our test server is under maintenance for now but I can
> hopefully recreate the problem in a VM soon.
>
> I'll work on recreating the dummy data on my VM and see if I still see the
> issue, and also post the dummy data here in case there's some strange thing
> I'm doing.
>
> -----Original Message-----
> From: John Vines [mailto:vines@apache.org]
> Sent: Wednesday, April 10, 2013 3:16 PM
> To: user@accumulo.apache.org
> Subject: RE: FW: Deletes not commiting from custom java API but work in
> shell interface
>
> First, you don't need to flush then close. Close already flushes.
>
> As for your problem, I have only 2 ideas- are you writing to the correct
> table?
>
> And as a sanity check, make sure the authorization you construct equals
> the column visibility you pull out. I wonder if information is getting lost
> in translation.
>
> Sent from my phone, please pardon the typos and brevity.
>
RE: FW: Deletes not commiting from custom java API but work in shell interface
Posted by "Bell, Philip S CIV SPAWARSYSCEN-PACIFIC, 81320" <ph...@navy.mil>.
We have not as of yet implemented any column visibility
When I first saw the problem, I created a separate table with some very generic test data to try out the deletion, and nothing I do from the java API seems to actually cause a delete to go through. I assume if deletes work from the shell then the delete iterator is working correctly, hence I'm out of ideas.
To add to the fun our test server is under maintenance for now but I can hopefully recreate the problem in a VM soon.
I'll work on recreating the dummy data on my VM and see if I still see the issue, and also post the dummy data here in case there's some strange thing I'm doing.
-----Original Message-----
From: John Vines [mailto:vines@apache.org]
Sent: Wednesday, April 10, 2013 3:16 PM
To: user@accumulo.apache.org
Subject: RE: FW: Deletes not commiting from custom java API but work in shell interface
First, you don't need to flush then close. Close already flushes.
As for your problem, I have only 2 ideas- are you writing to the correct table?
And as a sanity check, make sure the authorization you construct equals the column visibility you pull out. I wonder if information is getting lost in translation.
Sent from my phone, please pardon the typos and brevity.
RE: FW: Deletes not commiting from custom java API but work in shell interface
Posted by John Vines <vi...@apache.org>.
First, you don't need to flush then close. Close already flushes.
As for your problem, I have only 2 ideas- are you writing to the correct
table?
And as a sanity check, make sure the authorization you construct equals the
column visibility you pull out. I wonder if information is getting lost in
translation.
Sent from my phone, please pardon the typos and brevity.
On Apr 10, 2013 6:09 PM, "Bell, Philip S CIV SPAWARSYSCEN-PACIFIC, 81320" <
philip.bell@navy.mil> wrote:
> Ah I forgot to clean up my testing code, when I saw deletes weren't
> working I tried setting it to the last version timestamp to see if it
> required an exact match.
>
> So no matter if I submit
> delMutation.putDelete( colFam, colQual, new ColumnVisibility( colVis ),
> e.getKey().getTimestamp() );
>
> or
>
> delMutation.putDelete( colFam, colQual, new ColumnVisibility( colVis ) );
>
> I am still seeing the entries I wanted to delete
>
> -----Original Message-----
> From: John Vines [mailto:vines@apache.org]
> Sent: Wednesday, April 10, 2013 3:07 PM
> To: user@accumulo.apache.org
> Subject: Re: FW: Deletes not commiting from custom java API but work in
> shell interface
>
> Deletes will remove all entries which occur before the key. I believe
> there is undefined behavior for when a delete key is identical. It should
> work if you set the delete keys time +1.
>
> Sent from my phone, please pardon the typos and brevity.
>
> On Apr 10, 2013 6:02 PM, "Bell, Philip S CIV SPAWARSYSCEN-PACIFIC, 81320" <
> philip.bell@navy.mil> wrote:
>
>
> Using the following code, rows are never deleted even when
> identified and submitted for deletion to the batchwriter.
> After running the code the rows still show up in search.
>
> This has been seen in 1.4.1 and 1.4.2
>
> The same rows can be deleted from the accumulo shell interface
>
>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>
>
> for( Entry<Key, Value> e : mainScanner )
> {
> Text currentUUID = e.getKey().getRow();
>
> Text colFam = e.getKey().getColumnFamily();
> Text colQual = e.getKey().getColumnQualifier();
>
> String colVis =
> e.getKey().getColumnVisibility().toString();
>
> System.out.println( currentUUID + ":" + colFam + ":" +
> colQual + ":" + colVis + ":" + e.getKey().getTimestamp() );
>
> if( colFam.toString().equalsIgnoreCase( "root" ) ||
> colVis.length() > 0 )
> {
> Mutation delMutation = new Mutation( currentUUID );
> delMutation.putDelete( colFam, colQual, new
> ColumnVisibility( colVis ), e.getKey().getTimestamp() );
>
> System.out.println( "removing" );
>
> try
> {
> bw.addMutation( delMutation );
> }
> catch( MutationsRejectedException e1 )
> {
> e1.printStackTrace();
> }
>
> count++;
> if( count % 1000000 == 0 )
> {
> System.out.println( this.getName() + ": "
> + count + " completed in " + getTimePassed() );
> }
> }
> }// for each entry found
>
> try
> {
> bw.flush();
> bw.close();
> }
> catch( Exception e1 )
> {
> e1.printStackTrace();
> }
>
>
>
RE: FW: Deletes not commiting from custom java API but work in shell interface
Posted by "Bell, Philip S CIV SPAWARSYSCEN-PACIFIC, 81320" <ph...@navy.mil>.
Ah I forgot to clean up my testing code, when I saw deletes weren't working I tried setting it to the last version timestamp to see if it required an exact match.
So no matter if I submit
delMutation.putDelete( colFam, colQual, new ColumnVisibility( colVis ), e.getKey().getTimestamp() );
or
delMutation.putDelete( colFam, colQual, new ColumnVisibility( colVis ) );
I am still seeing the entries I wanted to delete
-----Original Message-----
From: John Vines [mailto:vines@apache.org]
Sent: Wednesday, April 10, 2013 3:07 PM
To: user@accumulo.apache.org
Subject: Re: FW: Deletes not commiting from custom java API but work in shell interface
Deletes will remove all entries which occur before the key. I believe there is undefined behavior for when a delete key is identical. It should work if you set the delete keys time +1.
Sent from my phone, please pardon the typos and brevity.
On Apr 10, 2013 6:02 PM, "Bell, Philip S CIV SPAWARSYSCEN-PACIFIC, 81320" <ph...@navy.mil> wrote:
Using the following code, rows are never deleted even when identified and submitted for deletion to the batchwriter.
After running the code the rows still show up in search.
This has been seen in 1.4.1 and 1.4.2
The same rows can be deleted from the accumulo shell interface
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
for( Entry<Key, Value> e : mainScanner )
{
Text currentUUID = e.getKey().getRow();
Text colFam = e.getKey().getColumnFamily();
Text colQual = e.getKey().getColumnQualifier();
String colVis = e.getKey().getColumnVisibility().toString();
System.out.println( currentUUID + ":" + colFam + ":" + colQual + ":" + colVis + ":" + e.getKey().getTimestamp() );
if( colFam.toString().equalsIgnoreCase( "root" ) || colVis.length() > 0 )
{
Mutation delMutation = new Mutation( currentUUID );
delMutation.putDelete( colFam, colQual, new ColumnVisibility( colVis ), e.getKey().getTimestamp() );
System.out.println( "removing" );
try
{
bw.addMutation( delMutation );
}
catch( MutationsRejectedException e1 )
{
e1.printStackTrace();
}
count++;
if( count % 1000000 == 0 )
{
System.out.println( this.getName() + ": " + count + " completed in " + getTimePassed() );
}
}
}// for each entry found
try
{
bw.flush();
bw.close();
}
catch( Exception e1 )
{
e1.printStackTrace();
}
Re: FW: Deletes not commiting from custom java API but work in shell interface
Posted by John Vines <vi...@apache.org>.
Deletes will remove all entries which occur before the key. I believe there
is undefined behavior for when a delete key is identical. It should work if
you set the delete keys time +1.
Sent from my phone, please pardon the typos and brevity.
On Apr 10, 2013 6:02 PM, "Bell, Philip S CIV SPAWARSYSCEN-PACIFIC, 81320" <
philip.bell@navy.mil> wrote:
> Using the following code, rows are never deleted even when identified and
> submitted for deletion to the batchwriter.
> After running the code the rows still show up in search.
>
> This has been seen in 1.4.1 and 1.4.2
>
> The same rows can be deleted from the accumulo shell interface
>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>
>
> for( Entry<Key, Value> e : mainScanner )
> {
> Text currentUUID = e.getKey().getRow();
>
> Text colFam = e.getKey().getColumnFamily();
> Text colQual = e.getKey().getColumnQualifier();
>
> String colVis = e.getKey().getColumnVisibility().toString();
>
> System.out.println( currentUUID + ":" + colFam + ":" + colQual +
> ":" + colVis + ":" + e.getKey().getTimestamp() );
>
> if( colFam.toString().equalsIgnoreCase( "root" ) ||
> colVis.length() > 0 )
> {
> Mutation delMutation = new Mutation( currentUUID );
> delMutation.putDelete( colFam, colQual, new
> ColumnVisibility( colVis ), e.getKey().getTimestamp() );
>
> System.out.println( "removing" );
>
> try
> {
> bw.addMutation( delMutation );
> }
> catch( MutationsRejectedException e1 )
> {
> e1.printStackTrace();
> }
>
> count++;
> if( count % 1000000 == 0 )
> {
> System.out.println( this.getName() + ": " + count
> + " completed in " + getTimePassed() );
> }
> }
> }// for each entry found
>
> try
> {
> bw.flush();
> bw.close();
> }
> catch( Exception e1 )
> {
> e1.printStackTrace();
> }
>