javax.crypto.BadPaddingException

[Cyrille Savelief <cs...@gmail.com>]

Hi,

Today, we stopped our Accumulo cluster (1.9.3, with encryption of data at rest) but we are now unable to restart it due to the following error : javax.crypto.BadPaddingException : Given final block not properly padded. Any workaround ?

Best,

Cyrille

Re: RE : javax.crypto.BadPaddingException

[Jeffrey Zeiberg <jz...@gmail.com>]

Can you check to see if your key became corrupted somehow?

On Mon, Oct 21, 2019 at 4:42 PM Cyrille Savelief <cs...@gmail.com>
wrote:

> We are using the built in crypto strategy :
> CachingHDFSSecretKeyEncryptionStrategy with the cypher suite
> AES/CFB/NoPadding
>
>
>
>
>
>
>
> *De : *Mike Miller <mm...@apache.org>
> *Envoyé le :*lundi 21 octobre 2019 22:38
> *À : *user@accumulo.apache.org
> *Objet :*Re: javax.crypto.BadPaddingException
>
>
>
> Hi Cyrille - This could be a number of issues.  The Key could be wrong,
> the block could not finished writing to the WAL when you shutdown, etc...
>
>
>
> What type of crypto do you have setup?  Are you using the built in crypto
> strategies or did you write your own?
>
>
>
> On Mon, Oct 21, 2019 at 3:36 PM Cyrille Savelief <cs...@gmail.com>
> wrote:
>
> Hi,
>
>
>
> Today, we stopped our Accumulo cluster (1.9.3, with encryption of data at
> rest) but we are now unable to restart it due to the following error :
> javax.crypto.BadPaddingException : Given final block not properly padded.
> Any workaround ?
>
>
>
> Best,
>
>
>
> Cyrille
>
>
>
>
>

Re: RE : javax.crypto.BadPaddingException

[Nick Felts <ng...@apache.org>]

I agree with Mike's suggestion that the block was incomplete when you shut
down, given that you had stopped Accumulo the same day this issue occurred.
If this is the case, you should be able to recover some of your data. The
DfsLogger contains the WAL readers/writers that use the CryptoModule.
Specifically, the method readHeaderAndReturnStream provides you with the
file stream you need for recovery. If you read data from this stream one
encrypted block at a time, you should be able to recover the data prior to
the corrupted block.

On Mon, Oct 21, 2019 at 6:41 PM Mike Miller <mm...@apache.org> wrote:

> Also, FWIW the crypto transformation for WALs in 2.0
> uses "AES/CBC/NoPadding".   There was only minimal testing done with that
> mode in 2.0 but it does seem to work with the WALs.
>
> From my understanding, I think you may have issues with CFB mode and the
> block sizes.  The crypto parameters are written out at the beginning of the
> WAL (since the file could be closed prematurely) which doesn't work for all
> cipher modes.
>
> On Mon, Oct 21, 2019 at 6:04 PM Christopher <ct...@apache.org> wrote:
>
>> My understanding is that CFB mode may not be suitable for write-ahead
>> logs. Perhaps CBC is better? I'm not sure. For what it's worth, a lot
>> of work was done in Accumulo 2.0 to help support configuring separate
>> modes for RFiles and WALs
>> (https://accumulo.apache.org/docs/2.x/security/on-disk-encryption),
>> and the updates may better handle the failure conditions that can
>> happen. Of course, this won't help you in 1.9.3.
>>
>> Keep in mind the on-disk encryption features are still experimental.
>> With more feedback, and development time/effort, it'd be cool to see
>> this feature get better over time, but I'm not sure to what extent WAL
>> encryption has been tested in the case of failure scenarios requiring
>> recovery.
>>
>> One relatively easy thing to do to recover from this specific failure
>> (assuming your key is correct, and this is definitely caused by a
>> corrupt block in the WAL file), is to replace the WAL file with an
>> effectively blank one or to surgically delete the WAL entry from the
>> metadata. However, this will likely cause data loss, which is
>> obviously not a great outcome, but it depends on how critical the data
>> is to your application. It could be particularly bad if the WAL
>> contained writes to the root or metadata table.
>>
>> I don't know how much work it would be to try to recover the
>> non-corrupt blocks from the WAL file, but that's an option as well, if
>> the data is critical.
>>
>> On Mon, Oct 21, 2019 at 4:42 PM Cyrille Savelief <cs...@gmail.com>
>> wrote:
>> >
>> > We are using the built in crypto strategy :
>> CachingHDFSSecretKeyEncryptionStrategy with the cypher suite
>> AES/CFB/NoPadding
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> > De : Mike Miller
>> > Envoyé le :lundi 21 octobre 2019 22:38
>> > À : user@accumulo.apache.org
>> > Objet :Re: javax.crypto.BadPaddingException
>> >
>> >
>> >
>> > Hi Cyrille - This could be a number of issues.  The Key could be wrong,
>> the block could not finished writing to the WAL when you shutdown, etc...
>> >
>> >
>> >
>> > What type of crypto do you have setup?  Are you using the built in
>> crypto strategies or did you write your own?
>> >
>> >
>> >
>> > On Mon, Oct 21, 2019 at 3:36 PM Cyrille Savelief <cs...@gmail.com>
>> wrote:
>> >
>> > Hi,
>> >
>> >
>> >
>> > Today, we stopped our Accumulo cluster (1.9.3, with encryption of data
>> at rest) but we are now unable to restart it due to the following error :
>> javax.crypto.BadPaddingException : Given final block not properly padded.
>> Any workaround ?
>> >
>> >
>> >
>> > Best,
>> >
>> >
>> >
>> > Cyrille
>> >
>> >
>> >
>> >
>>
>

Re: RE : javax.crypto.BadPaddingException

[Mike Miller <mm...@apache.org>]

Also, FWIW the crypto transformation for WALs in 2.0
uses "AES/CBC/NoPadding".   There was only minimal testing done with that
mode in 2.0 but it does seem to work with the WALs.

From my understanding, I think you may have issues with CFB mode and the
block sizes.  The crypto parameters are written out at the beginning of the
WAL (since the file could be closed prematurely) which doesn't work for all
cipher modes.

On Mon, Oct 21, 2019 at 6:04 PM Christopher <ct...@apache.org> wrote:

> My understanding is that CFB mode may not be suitable for write-ahead
> logs. Perhaps CBC is better? I'm not sure. For what it's worth, a lot
> of work was done in Accumulo 2.0 to help support configuring separate
> modes for RFiles and WALs
> (https://accumulo.apache.org/docs/2.x/security/on-disk-encryption),
> and the updates may better handle the failure conditions that can
> happen. Of course, this won't help you in 1.9.3.
>
> Keep in mind the on-disk encryption features are still experimental.
> With more feedback, and development time/effort, it'd be cool to see
> this feature get better over time, but I'm not sure to what extent WAL
> encryption has been tested in the case of failure scenarios requiring
> recovery.
>
> One relatively easy thing to do to recover from this specific failure
> (assuming your key is correct, and this is definitely caused by a
> corrupt block in the WAL file), is to replace the WAL file with an
> effectively blank one or to surgically delete the WAL entry from the
> metadata. However, this will likely cause data loss, which is
> obviously not a great outcome, but it depends on how critical the data
> is to your application. It could be particularly bad if the WAL
> contained writes to the root or metadata table.
>
> I don't know how much work it would be to try to recover the
> non-corrupt blocks from the WAL file, but that's an option as well, if
> the data is critical.
>
> On Mon, Oct 21, 2019 at 4:42 PM Cyrille Savelief <cs...@gmail.com>
> wrote:
> >
> > We are using the built in crypto strategy :
> CachingHDFSSecretKeyEncryptionStrategy with the cypher suite
> AES/CFB/NoPadding
> >
> >
> >
> >
> >
> >
> >
> > De : Mike Miller
> > Envoyé le :lundi 21 octobre 2019 22:38
> > À : user@accumulo.apache.org
> > Objet :Re: javax.crypto.BadPaddingException
> >
> >
> >
> > Hi Cyrille - This could be a number of issues.  The Key could be wrong,
> the block could not finished writing to the WAL when you shutdown, etc...
> >
> >
> >
> > What type of crypto do you have setup?  Are you using the built in
> crypto strategies or did you write your own?
> >
> >
> >
> > On Mon, Oct 21, 2019 at 3:36 PM Cyrille Savelief <cs...@gmail.com>
> wrote:
> >
> > Hi,
> >
> >
> >
> > Today, we stopped our Accumulo cluster (1.9.3, with encryption of data
> at rest) but we are now unable to restart it due to the following error :
> javax.crypto.BadPaddingException : Given final block not properly padded.
> Any workaround ?
> >
> >
> >
> > Best,
> >
> >
> >
> > Cyrille
> >
> >
> >
> >
>

Re: RE : javax.crypto.BadPaddingException

[Christopher <ct...@apache.org>]

My understanding is that CFB mode may not be suitable for write-ahead
logs. Perhaps CBC is better? I'm not sure. For what it's worth, a lot
of work was done in Accumulo 2.0 to help support configuring separate
modes for RFiles and WALs
(https://accumulo.apache.org/docs/2.x/security/on-disk-encryption),
and the updates may better handle the failure conditions that can
happen. Of course, this won't help you in 1.9.3.

Keep in mind the on-disk encryption features are still experimental.
With more feedback, and development time/effort, it'd be cool to see
this feature get better over time, but I'm not sure to what extent WAL
encryption has been tested in the case of failure scenarios requiring
recovery.

One relatively easy thing to do to recover from this specific failure
(assuming your key is correct, and this is definitely caused by a
corrupt block in the WAL file), is to replace the WAL file with an
effectively blank one or to surgically delete the WAL entry from the
metadata. However, this will likely cause data loss, which is
obviously not a great outcome, but it depends on how critical the data
is to your application. It could be particularly bad if the WAL
contained writes to the root or metadata table.

I don't know how much work it would be to try to recover the
non-corrupt blocks from the WAL file, but that's an option as well, if
the data is critical.

On Mon, Oct 21, 2019 at 4:42 PM Cyrille Savelief <cs...@gmail.com> wrote:
>
> We are using the built in crypto strategy : CachingHDFSSecretKeyEncryptionStrategy with the cypher suite AES/CFB/NoPadding
>
>
>
>
>
>
>
> De : Mike Miller
> Envoyé le :lundi 21 octobre 2019 22:38
> À : user@accumulo.apache.org
> Objet :Re: javax.crypto.BadPaddingException
>
>
>
> Hi Cyrille - This could be a number of issues.  The Key could be wrong, the block could not finished writing to the WAL when you shutdown, etc...
>
>
>
> What type of crypto do you have setup?  Are you using the built in crypto strategies or did you write your own?
>
>
>
> On Mon, Oct 21, 2019 at 3:36 PM Cyrille Savelief <cs...@gmail.com> wrote:
>
> Hi,
>
>
>
> Today, we stopped our Accumulo cluster (1.9.3, with encryption of data at rest) but we are now unable to restart it due to the following error : javax.crypto.BadPaddingException : Given final block not properly padded. Any workaround ?
>
>
>
> Best,
>
>
>
> Cyrille
>
>
>
>

RE : javax.crypto.BadPaddingException

[Cyrille Savelief <cs...@gmail.com>]

We are using the built in crypto strategy : CachingHDFSSecretKeyEncryptionStrategy with the cypher suite AES/CFB/NoPadding



De : Mike Miller
Envoyé le :lundi 21 octobre 2019 22:38
À : user@accumulo.apache.org
Objet :Re: javax.crypto.BadPaddingException

Hi Cyrille - This could be a number of issues.  The Key could be wrong, the block could not finished writing to the WAL when you shutdown, etc... 

What type of crypto do you have setup?  Are you using the built in crypto strategies or did you write your own?

On Mon, Oct 21, 2019 at 3:36 PM Cyrille Savelief <cs...@gmail.com> wrote:
Hi,
 
Today, we stopped our Accumulo cluster (1.9.3, with encryption of data at rest) but we are now unable to restart it due to the following error : javax.crypto.BadPaddingException : Given final block not properly padded. Any workaround ?
 
Best,
 
Cyrille
 

Re: javax.crypto.BadPaddingException

[Mike Miller <mm...@apache.org>]

Hi Cyrille - This could be a number of issues.  The Key could be wrong, the
block could not finished writing to the WAL when you shutdown, etc...

What type of crypto do you have setup?  Are you using the built in crypto
strategies or did you write your own?

On Mon, Oct 21, 2019 at 3:36 PM Cyrille Savelief <cs...@gmail.com>
wrote:

> Hi,
>
>
>
> Today, we stopped our Accumulo cluster (1.9.3, with encryption of data at
> rest) but we are now unable to restart it due to the following error :
> javax.crypto.BadPaddingException : Given final block not properly padded.
> Any workaround ?
>
>
>
> Best,
>
>
>
> Cyrille
>
>
>