You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by Luciano Afranllie <li...@gmail.com> on 2016/07/22 20:58:45 UTC
Changing hash algorithm to LogCleaner offset map
Hi
We are evaluating to change the hash algorithm used by the SkimpyOffsetMap
used by the LogCleaner from MD5 to SHA-1.
Besides the impact in performance (more memory, more cpu usage) is there
anything that may be impacted?
Regards
Luciano
Re: Changing hash algorithm to LogCleaner offset map
Posted by Luciano Afranllie <li...@gmail.com>.
Thanks Shikhar.
I have created KAFKA-3987 (https://issues.apache.org/jira/browse/KAFKA-3987).
Could anybody please assign that issue to me? I could not do it. I have a
patch ready and will open a pull request tomorrow.
Regards
On Sun, Jul 24, 2016 at 4:45 PM, Shikhar Bhushan <sh...@confluent.io>
wrote:
> Got it, makes sense to make the hash function customizable if there are
> environments in which md5 usage is prevented. The approach you are
> proposing sounds good to me.
> On Sat, Jul 23, 2016 at 14:56 Luciano Afranllie <li...@gmail.com>
> wrote:
>
> > Nothing wrong about using MD5 for that from FIPS point of view, but we
> want
> > to deploy with FIPS 140-2 mode enabled using only RSA security providers.
> > With this settings it is not possible to use MD5.
> >
> > On Fri, Jul 22, 2016 at 8:49 PM, Shikhar Bhushan <sh...@confluent.io>
> > wrote:
> >
> > > Not sure I understand the motivation to use a FIPS-compliant hash
> > function
> > > for log compaction -- what are the security ramifications?
> > >
> > > On Fri, Jul 22, 2016 at 2:56 PM Luciano Afranllie <
> > > listas.luafran@gmail.com>
> > > wrote:
> > >
> > > > A little bit of background first.
> > > >
> > > > We are trying to make a deployment of Kafka that is FIPS 140-2 (
> > > > https://en.wikipedia.org/wiki/FIPS_140-2) complaint and one of the
> > > > requirements is not to use MD5.
> > > >
> > > > As far as we could see, Kafka is using MD5 only to hash message keys
> > in a
> > > > offset map (SkimpyOffsetMap) used by the log cleaner. So, we are
> > planning
> > > > to change the hash algorithm to something allowed by FIPS.
> > > >
> > > > With this in mind we are thinking that it would be great if we can
> add
> > a
> > > > config property LogCleanerHashAlgorithmProp =
> > > "log.cleaner.hash.algorithm"
> > > > with a default value equal to "MD5" and use it in the constructor
> > > > of CleanerConfig. In that case in future versions of Kafka we can
> just
> > > > change the value of this property.
> > > >
> > > > Please let me know if you are Ok with this change.
> > > > It is enough to create a pull request for this? Should I create a
> Jira
> > > > first?
> > > >
> > > > Regards
> > > > Luciano
> > > >
> > > > On Fri, Jul 22, 2016 at 5:58 PM, Luciano Afranllie <
> > > > listas.luafran@gmail.com
> > > > > wrote:
> > > >
> > > > > Hi
> > > > >
> > > > > We are evaluating to change the hash algorithm used by the
> > > > SkimpyOffsetMap
> > > > > used by the LogCleaner from MD5 to SHA-1.
> > > > >
> > > > > Besides the impact in performance (more memory, more cpu usage) is
> > > there
> > > > > anything that may be impacted?
> > > > >
> > > > > Regards
> > > > > Luciano
> > > > >
> > > >
> > >
> >
>
Re: Changing hash algorithm to LogCleaner offset map
Posted by Shikhar Bhushan <sh...@confluent.io>.
Got it, makes sense to make the hash function customizable if there are
environments in which md5 usage is prevented. The approach you are
proposing sounds good to me.
On Sat, Jul 23, 2016 at 14:56 Luciano Afranllie <li...@gmail.com>
wrote:
> Nothing wrong about using MD5 for that from FIPS point of view, but we want
> to deploy with FIPS 140-2 mode enabled using only RSA security providers.
> With this settings it is not possible to use MD5.
>
> On Fri, Jul 22, 2016 at 8:49 PM, Shikhar Bhushan <sh...@confluent.io>
> wrote:
>
> > Not sure I understand the motivation to use a FIPS-compliant hash
> function
> > for log compaction -- what are the security ramifications?
> >
> > On Fri, Jul 22, 2016 at 2:56 PM Luciano Afranllie <
> > listas.luafran@gmail.com>
> > wrote:
> >
> > > A little bit of background first.
> > >
> > > We are trying to make a deployment of Kafka that is FIPS 140-2 (
> > > https://en.wikipedia.org/wiki/FIPS_140-2) complaint and one of the
> > > requirements is not to use MD5.
> > >
> > > As far as we could see, Kafka is using MD5 only to hash message keys
> in a
> > > offset map (SkimpyOffsetMap) used by the log cleaner. So, we are
> planning
> > > to change the hash algorithm to something allowed by FIPS.
> > >
> > > With this in mind we are thinking that it would be great if we can add
> a
> > > config property LogCleanerHashAlgorithmProp =
> > "log.cleaner.hash.algorithm"
> > > with a default value equal to "MD5" and use it in the constructor
> > > of CleanerConfig. In that case in future versions of Kafka we can just
> > > change the value of this property.
> > >
> > > Please let me know if you are Ok with this change.
> > > It is enough to create a pull request for this? Should I create a Jira
> > > first?
> > >
> > > Regards
> > > Luciano
> > >
> > > On Fri, Jul 22, 2016 at 5:58 PM, Luciano Afranllie <
> > > listas.luafran@gmail.com
> > > > wrote:
> > >
> > > > Hi
> > > >
> > > > We are evaluating to change the hash algorithm used by the
> > > SkimpyOffsetMap
> > > > used by the LogCleaner from MD5 to SHA-1.
> > > >
> > > > Besides the impact in performance (more memory, more cpu usage) is
> > there
> > > > anything that may be impacted?
> > > >
> > > > Regards
> > > > Luciano
> > > >
> > >
> >
>
Re: Changing hash algorithm to LogCleaner offset map
Posted by Luciano Afranllie <li...@gmail.com>.
Nothing wrong about using MD5 for that from FIPS point of view, but we want
to deploy with FIPS 140-2 mode enabled using only RSA security providers.
With this settings it is not possible to use MD5.
On Fri, Jul 22, 2016 at 8:49 PM, Shikhar Bhushan <sh...@confluent.io>
wrote:
> Not sure I understand the motivation to use a FIPS-compliant hash function
> for log compaction -- what are the security ramifications?
>
> On Fri, Jul 22, 2016 at 2:56 PM Luciano Afranllie <
> listas.luafran@gmail.com>
> wrote:
>
> > A little bit of background first.
> >
> > We are trying to make a deployment of Kafka that is FIPS 140-2 (
> > https://en.wikipedia.org/wiki/FIPS_140-2) complaint and one of the
> > requirements is not to use MD5.
> >
> > As far as we could see, Kafka is using MD5 only to hash message keys in a
> > offset map (SkimpyOffsetMap) used by the log cleaner. So, we are planning
> > to change the hash algorithm to something allowed by FIPS.
> >
> > With this in mind we are thinking that it would be great if we can add a
> > config property LogCleanerHashAlgorithmProp =
> "log.cleaner.hash.algorithm"
> > with a default value equal to "MD5" and use it in the constructor
> > of CleanerConfig. In that case in future versions of Kafka we can just
> > change the value of this property.
> >
> > Please let me know if you are Ok with this change.
> > It is enough to create a pull request for this? Should I create a Jira
> > first?
> >
> > Regards
> > Luciano
> >
> > On Fri, Jul 22, 2016 at 5:58 PM, Luciano Afranllie <
> > listas.luafran@gmail.com
> > > wrote:
> >
> > > Hi
> > >
> > > We are evaluating to change the hash algorithm used by the
> > SkimpyOffsetMap
> > > used by the LogCleaner from MD5 to SHA-1.
> > >
> > > Besides the impact in performance (more memory, more cpu usage) is
> there
> > > anything that may be impacted?
> > >
> > > Regards
> > > Luciano
> > >
> >
>
Re: Changing hash algorithm to LogCleaner offset map
Posted by Shikhar Bhushan <sh...@confluent.io>.
Not sure I understand the motivation to use a FIPS-compliant hash function
for log compaction -- what are the security ramifications?
On Fri, Jul 22, 2016 at 2:56 PM Luciano Afranllie <li...@gmail.com>
wrote:
> A little bit of background first.
>
> We are trying to make a deployment of Kafka that is FIPS 140-2 (
> https://en.wikipedia.org/wiki/FIPS_140-2) complaint and one of the
> requirements is not to use MD5.
>
> As far as we could see, Kafka is using MD5 only to hash message keys in a
> offset map (SkimpyOffsetMap) used by the log cleaner. So, we are planning
> to change the hash algorithm to something allowed by FIPS.
>
> With this in mind we are thinking that it would be great if we can add a
> config property LogCleanerHashAlgorithmProp = "log.cleaner.hash.algorithm"
> with a default value equal to "MD5" and use it in the constructor
> of CleanerConfig. In that case in future versions of Kafka we can just
> change the value of this property.
>
> Please let me know if you are Ok with this change.
> It is enough to create a pull request for this? Should I create a Jira
> first?
>
> Regards
> Luciano
>
> On Fri, Jul 22, 2016 at 5:58 PM, Luciano Afranllie <
> listas.luafran@gmail.com
> > wrote:
>
> > Hi
> >
> > We are evaluating to change the hash algorithm used by the
> SkimpyOffsetMap
> > used by the LogCleaner from MD5 to SHA-1.
> >
> > Besides the impact in performance (more memory, more cpu usage) is there
> > anything that may be impacted?
> >
> > Regards
> > Luciano
> >
>
Re: Changing hash algorithm to LogCleaner offset map
Posted by Luciano Afranllie <li...@gmail.com>.
A little bit of background first.
We are trying to make a deployment of Kafka that is FIPS 140-2 (
https://en.wikipedia.org/wiki/FIPS_140-2) complaint and one of the
requirements is not to use MD5.
As far as we could see, Kafka is using MD5 only to hash message keys in a
offset map (SkimpyOffsetMap) used by the log cleaner. So, we are planning
to change the hash algorithm to something allowed by FIPS.
With this in mind we are thinking that it would be great if we can add a
config property LogCleanerHashAlgorithmProp = "log.cleaner.hash.algorithm"
with a default value equal to "MD5" and use it in the constructor
of CleanerConfig. In that case in future versions of Kafka we can just
change the value of this property.
Please let me know if you are Ok with this change.
It is enough to create a pull request for this? Should I create a Jira
first?
Regards
Luciano
On Fri, Jul 22, 2016 at 5:58 PM, Luciano Afranllie <listas.luafran@gmail.com
> wrote:
> Hi
>
> We are evaluating to change the hash algorithm used by the SkimpyOffsetMap
> used by the LogCleaner from MD5 to SHA-1.
>
> Besides the impact in performance (more memory, more cpu usage) is there
> anything that may be impacted?
>
> Regards
> Luciano
>