You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by ZHU Hua B <Hu...@alcatel-lucent.com> on 2016/10/14 08:41:10 UTC
A question about kafka
Hi,
I have a question about kafka, could you please help to have a look?
I want to send a message from producer with snappy compression codec. So I run the command "bin/kafka-console-producer.sh --compression-codec snappy --broker-list localhost:9092 --topic test", after that I checked the data log, compresscodec is SnappyCompressionCodec as expectation.
Then I tried another command "bin/kafka-console-producer.sh --producer-property compression.type=snappy --broker-list localhost:9092 --topic test" to send a message from producer with snappy also, but this time I found compresscodec is NoCompressionCodec in data log.
I double checked the option of "kafka-console-producer.sh" as below, seems both "--compression-codec" and "--producer-property" could be used to set up compression codec, why the second way failed? The "key=value" I found in config/producer.properties, so I think it is regular. Thanks!
# bin/kafka-console-producer.sh --help
--compression-codec [compression-codec] The compression codec: either 'none','gzip', 'snappy', or 'lz4'.If specified without value, then it defaults to 'gzip'.
--producer-property <producer_prop> A mechanism to pass user-defined properties in the form key=value to the producer.
--producer.config <config file> Producer config properties file. Note that [producer-property] takes precedence over this config.
Best Regards
Johnny
RE: A question about kafka
Posted by ZHU Hua B <Hu...@alcatel-lucent.com>.
Hi,
Thanks for your reply!
If console producer only allows for compression codec argument, why we could found option —producer-property defined in ConsoleProducer.scala? And we could find the usage also if we running console producer? The version we used is Kafka 0.10.0.0. Thanks!
# ./kafka-console-producer.sh
Read data from standard input and publish it to Kafka.
Option Description
------ -----------
--compression-codec [compression-codec] The compression codec: either 'none',
'gzip', 'snappy', or 'lz4'.If
specified without value, then it
defaults to 'gzip'
--producer-property <producer_prop> A mechanism to pass user-defined
properties in the form key=value to
the producer.
--producer.config <config file> Producer config properties file. Note
that [producer-property] takes
precedence over this config.
--property <prop> A mechanism to pass user-defined
properties in the form key=value to
the message reader. This allows
custom configuration for a user-
defined message reader.
Best Regards
Johnny
From: Radoslaw Gruchalski [mailto:radek@gruchalski.com]
Sent: 2016年10月17日 14:02
To: ZHU Hua B; users@kafka.apache.org
Subject: RE: A question about kafka
Hi,
I believe the answer is in the code. This is where the --compression-codec is processed:
https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/tools/ConsoleProducer.scala#L143
and this is —producer-property:
https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/tools/ConsoleProducer.scala#L234
The usage is here:
https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/tools/ConsoleProducer.scala#L114
The answer is: The console producer allows for compression codec only with —compression-codec argument.
–
Best regards,
Radek Gruchalski
radek@gruchalski.com<ma...@gruchalski.com>
On October 17, 2016 at 7:46:41 AM, ZHU Hua B (hua.b.zhu@alcatel-lucent.com<ma...@alcatel-lucent.com>) wrote:
Hi,
Anybody could help to answer this question? Thanks!
Best Regards
Johnny
-----Original Message-----
From: ZHU Hua B
Sent: 2016年10月14日 16:41
To: users@kafka.apache.org<ma...@kafka.apache.org>
Subject: [COMMERCIAL] A question about kafka
Hi,
I have a question about kafka, could you please help to have a look?
I want to send a message from producer with snappy compression codec. So I run the command "bin/kafka-console-producer.sh --compression-codec snappy --broker-list localhost:9092 --topic test", after that I checked the data log, compresscodec is SnappyCompressionCodec as expectation.
Then I tried another command "bin/kafka-console-producer.sh --producer-property compression.type=snappy --broker-list localhost:9092 --topic test" to send a message from producer with snappy also, but this time I found compresscodec is NoCompressionCodec in data log.
I double checked the option of "kafka-console-producer.sh" as below, seems both "--compression-codec" and "--producer-property" could be used to set up compression codec, why the second way failed? The "key=value" I found in config/producer.properties, so I think it is regular. Thanks!
# bin/kafka-console-producer.sh --help
--compression-codec [compression-codec] The compression codec: either 'none','gzip', 'snappy', or 'lz4'.If specified without value, then it defaults to 'gzip'.
--producer-property <producer_prop> A mechanism to pass user-defined properties in the form key=value to the producer.
--producer.config <config file> Producer config properties file. Note that [producer-property] takes precedence over this config.
Best Regards
Johnny
RE: A question about kafka
Posted by Radoslaw Gruchalski <ra...@gruchalski.com>.
Hi,
I believe the answer is in the code. This is where the --compression-codec
is processed:
https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/tools/ConsoleProducer.scala#L143
and this is —producer-property:
https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/tools/ConsoleProducer.scala#L234
The usage is here:
https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/tools/ConsoleProducer.scala#L114
The answer is: The console producer allows for compression codec only with
—compression-codec argument.
–
Best regards,
Radek Gruchalski
radek@gruchalski.com
On October 17, 2016 at 7:46:41 AM, ZHU Hua B (hua.b.zhu@alcatel-lucent.com)
wrote:
Hi,
Anybody could help to answer this question? Thanks!
Best Regards
Johnny
-----Original Message-----
From: ZHU Hua B
Sent: 2016年10月14日 16:41
To: users@kafka.apache.org
Subject: [COMMERCIAL] A question about kafka
Hi,
I have a question about kafka, could you please help to have a look?
I want to send a message from producer with snappy compression codec. So I
run the command "bin/kafka-console-producer.sh --compression-codec snappy
--broker-list localhost:9092 --topic test", after that I checked the data
log, compresscodec is SnappyCompressionCodec as expectation.
Then I tried another command "bin/kafka-console-producer.sh
--producer-property compression.type=snappy --broker-list localhost:9092
--topic test" to send a message from producer with snappy also, but this
time I found compresscodec is NoCompressionCodec in data log.
I double checked the option of "kafka-console-producer.sh" as below, seems
both "--compression-codec" and "--producer-property" could be used to set
up compression codec, why the second way failed? The "key=value" I found in
config/producer.properties, so I think it is regular. Thanks!
# bin/kafka-console-producer.sh --help
--compression-codec [compression-codec] The compression codec: either
'none','gzip', 'snappy', or 'lz4'.If specified without value, then it
defaults to 'gzip'.
--producer-property <producer_prop> A mechanism to pass user-defined
properties in the form key=value to the producer.
--producer.config <config file> Producer config properties file. Note that
[producer-property] takes precedence over this config.
Best Regards
Johnny
RE: A question about kafka
Posted by ZHU Hua B <Hu...@alcatel-lucent.com>.
Hi,
Thanks for your reply!
I saw the last update for this issue was on 14/Nov/2015, do you know if the has been fixed now? The version we used is 0.10.0.0. Thanks!
Best Regards
Johnny
-----Original Message-----
From: Manikumar [mailto:manikumar.reddy@gmail.com]
Sent: 2016年10月17日 14:03
To: users@kafka.apache.org
Subject: Re: A question about kafka
This is known issue in some of the command line tools.
JIRA is here :
https://issues.apache.org/jira/browse/KAFKA-2619
On Mon, Oct 17, 2016 at 11:16 AM, ZHU Hua B <Hu...@alcatel-lucent.com>
wrote:
> Hi,
>
>
> Anybody could help to answer this question? Thanks!
>
>
>
>
>
>
> Best Regards
>
> Johnny
>
> -----Original Message-----
> From: ZHU Hua B
> Sent: 2016年10月14日 16:41
> To: users@kafka.apache.org
> Subject: [COMMERCIAL] A question about kafka
>
> Hi,
>
>
> I have a question about kafka, could you please help to have a look?
>
> I want to send a message from producer with snappy compression codec.
> So I run the command "bin/kafka-console-producer.sh
> --compression-codec snappy --broker-list localhost:9092 --topic test",
> after that I checked the data log, compresscodec is SnappyCompressionCodec as expectation.
>
> Then I tried another command "bin/kafka-console-producer.sh
> --producer-property compression.type=snappy --broker-list
> localhost:9092 --topic test" to send a message from producer with
> snappy also, but this time I found compresscodec is NoCompressionCodec in data log.
> I double checked the option of "kafka-console-producer.sh" as below,
> seems both "--compression-codec" and "--producer-property" could be
> used to set up compression codec, why the second way failed? The
> "key=value" I found in config/producer.properties, so I think it is regular. Thanks!
>
> # bin/kafka-console-producer.sh --help
>
> --compression-codec [compression-codec] The compression codec: either
> 'none','gzip', 'snappy', or 'lz4'.If specified without value, then it
> defaults to 'gzip'.
>
> --producer-property <producer_prop> A mechanism to pass user-defined
> properties in the form key=value to the producer.
>
> --producer.config <config file> Producer config properties file.
> Note that [producer-property] takes precedence over this config.
>
>
>
>
>
>
> Best Regards
>
> Johnny
>
>
Re: A question about kafka
Posted by Manikumar <ma...@gmail.com>.
This is known issue in some of the command line tools.
JIRA is here :
https://issues.apache.org/jira/browse/KAFKA-2619
On Mon, Oct 17, 2016 at 11:16 AM, ZHU Hua B <Hu...@alcatel-lucent.com>
wrote:
> Hi,
>
>
> Anybody could help to answer this question? Thanks!
>
>
>
>
>
>
> Best Regards
>
> Johnny
>
> -----Original Message-----
> From: ZHU Hua B
> Sent: 2016年10月14日 16:41
> To: users@kafka.apache.org
> Subject: [COMMERCIAL] A question about kafka
>
> Hi,
>
>
> I have a question about kafka, could you please help to have a look?
>
> I want to send a message from producer with snappy compression codec. So I
> run the command "bin/kafka-console-producer.sh --compression-codec snappy
> --broker-list localhost:9092 --topic test", after that I checked the data
> log, compresscodec is SnappyCompressionCodec as expectation.
>
> Then I tried another command "bin/kafka-console-producer.sh
> --producer-property compression.type=snappy --broker-list localhost:9092
> --topic test" to send a message from producer with snappy also, but this
> time I found compresscodec is NoCompressionCodec in data log.
> I double checked the option of "kafka-console-producer.sh" as below, seems
> both "--compression-codec" and "--producer-property" could be used to set
> up compression codec, why the second way failed? The "key=value" I found in
> config/producer.properties, so I think it is regular. Thanks!
>
> # bin/kafka-console-producer.sh --help
>
> --compression-codec [compression-codec] The compression codec: either
> 'none','gzip', 'snappy', or 'lz4'.If specified without value, then it
> defaults to 'gzip'.
>
> --producer-property <producer_prop> A mechanism to pass user-defined
> properties in the form key=value to the producer.
>
> --producer.config <config file> Producer config properties file.
> Note that [producer-property] takes precedence over this config.
>
>
>
>
>
>
> Best Regards
>
> Johnny
>
>
RE: A question about kafka
Posted by ZHU Hua B <Hu...@alcatel-lucent.com>.
Hi,
Anybody could help to answer this question? Thanks!
Best Regards
Johnny
-----Original Message-----
From: ZHU Hua B
Sent: 2016年10月14日 16:41
To: users@kafka.apache.org
Subject: [COMMERCIAL] A question about kafka
Hi,
I have a question about kafka, could you please help to have a look?
I want to send a message from producer with snappy compression codec. So I run the command "bin/kafka-console-producer.sh --compression-codec snappy --broker-list localhost:9092 --topic test", after that I checked the data log, compresscodec is SnappyCompressionCodec as expectation.
Then I tried another command "bin/kafka-console-producer.sh --producer-property compression.type=snappy --broker-list localhost:9092 --topic test" to send a message from producer with snappy also, but this time I found compresscodec is NoCompressionCodec in data log.
I double checked the option of "kafka-console-producer.sh" as below, seems both "--compression-codec" and "--producer-property" could be used to set up compression codec, why the second way failed? The "key=value" I found in config/producer.properties, so I think it is regular. Thanks!
# bin/kafka-console-producer.sh --help
--compression-codec [compression-codec] The compression codec: either 'none','gzip', 'snappy', or 'lz4'.If specified without value, then it defaults to 'gzip'.
--producer-property <producer_prop> A mechanism to pass user-defined properties in the form key=value to the producer.
--producer.config <config file> Producer config properties file. Note that [producer-property] takes precedence over this config.
Best Regards
Johnny