You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@beam.apache.org by Shen Li <cs...@gmail.com> on 2017/10/30 15:14:52 UTC
Is upgrading to Kafka Client 0.10.2.0+ in the roadmap?
Hi,
To use KafkaIO in secure mode, I need to set
-Djava.security.auth.login.config to point to a JAAS configuration file. It
works fine for local execution. But how can I configure the
"java.security.auth.login.config" property in the Beam app when the
pipeline is submitted to a cluster/cloud-service? Even if I use a ParDo to
set the system property, there is no guarantee that the ParDo will run on
the same server with the KafkaIO source.
For this specific problem, it would be helpful to upgrade to Kafka Client
0.10.2.0, which provides a "sasl.jaas.config" property that can be updated
programmatically. Or is there any other work around?
https://cwiki.apache.org/confluence/display/KAFKA/KIP-85%3A+Dynamic+JAAS+configuration+for+Kafka+clients
Thanks,
Shen
Re: Is upgrading to Kafka Client 0.10.2.0+ in the roadmap?
Posted by Raghu Angadi <ra...@google.com.INVALID>.
Shen,
KafkaIO works with all the versions since 0.9. Just include kafka-clients
version you like in your maven dependencies along with Beam dependencies.
Glad to here Kafka 0.10.2 made it simpler to provide this config.
On Mon, Oct 30, 2017 at 8:14 AM, Shen Li <cs...@gmail.com> wrote:
> Hi,
>
> To use KafkaIO in secure mode, I need to set
> -Djava.security.auth.login.config to point to a JAAS configuration file.
> It
> works fine for local execution. But how can I configure the
> "java.security.auth.login.config" property in the Beam app when the
> pipeline is submitted to a cluster/cloud-service? Even if I use a ParDo to
> set the system property, there is no guarantee that the ParDo will run on
> the same server with the KafkaIO source.
>
> For this specific problem, it would be helpful to upgrade to Kafka Client
> 0.10.2.0, which provides a "sasl.jaas.config" property that can be updated
> programmatically. Or is there any other work around?
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-85%3A+Dynamic+JAAS+
> configuration+for+Kafka+clients
>
> Thanks,
> Shen
>
Re: Is upgrading to Kafka Client 0.10.2.0+ in the roadmap?
Posted by Raghu Angadi <ra...@google.com.INVALID>.
>
> Thanks a lot for the information. I am using Beam-2.0.
> https://github.com/apache/beam/blob/release-2.0.0/sdks/
> java/io/kafka/pom.xml#L33
I think we should move kafka-clients dependency in KafkaIO to provided
scope to avoid potential confusion like this.
On Mon, Oct 30, 2017 at 11:10 AM, Shen Li <cs...@gmail.com> wrote:
> Dear All,
>
> Thanks a lot for the information. I am using Beam-2.0.
> https://github.com/apache/beam/blob/release-2.0.0/sdks/
> java/io/kafka/pom.xml#L33
>
> I have just verified that adding Kafka-Client 0.11 in the application
> pom.xml works fine for me. I can now avoid the JAAS configuration file by
> using the "java.security.auth.login.config" property.
>
> Best,
> Shen
>
> On Mon, Oct 30, 2017 at 1:41 PM, Mingmin Xu <mi...@gmail.com> wrote:
>
> > Hi Shen,
> >
> > Can you share which Beam version are you using? Just check master code,
> the
> > default version for Kafka is
> > `<kafka.clients.version>0.11.0.1</kafka.clients.version>`.
> > I cannot recall the usage for old versions, my
> application(2.2.0-SNAPSHOT)
> > works with a customized kafka version based on 0.10.00-SASL. What you
> need
> > to do is
> > 1). exclude the kafka-client in KafkaIO, and add your own Kafka client
> > library in pom.xml;
> > 2). add your configuration like:
> > ```
> > Map<String, Object> consumerPara = new HashMap<String, Object>();
> > //consumerPara.put(ConsumerConfig.GROUP_ID_CONFIG,
> consumerName);
> > //consumerPara.put(ConsumerConfig.CLIENT_ID_CONFIG, clientId);
> > if (secureEnabled) {
> > consumerPara.put("sasl.mechanism", "IAF");
> > consumerPara.put("security.protocol", "SASL_PLAINTEXT");
> > consumerPara.put("sasl.login.class", ".....");
> > consumerPara.put("sasl.callback.handler.class", "...");
> > }
> >
> > KafkaIO.<byte[], byte[]>read()
> > ....
> > .updateConsumerProperties(configUpdates)
> > ....;
> > ```
> >
> > Mingmin
> >
> > On Mon, Oct 30, 2017 at 10:21 AM, Raghu Angadi
> <rangadi@google.com.invalid
> > >
> > wrote:
> >
> > > > https://issues.apache.org/jira/browse/BEAM-307
> > >
> > > This should be closed.
> > >
> > > On Mon, Oct 30, 2017 at 9:00 AM, Lukasz Cwik <lcwik@google.com.invalid
> >
> > > wrote:
> > >
> > > > There has been some discussion about getting Kafka 0.10.x working on
> > > > BEAM-307[1].
> > > >
> > > > As an immediate way to unblock yourself, modify your local copy of
> the
> > > > KafkaIO source to include setting the system property in a static
> block
> > > > before the class is loaded or before the Kafka client is instantiated
> > and
> > > > used.
> > > >
> > > > Also consider contributing to the Kafka connector to getting 0.10.x
> > > > working.
> > > >
> > > > 1: https://issues.apache.org/jira/browse/BEAM-307
> > > >
> > > > On Mon, Oct 30, 2017 at 8:14 AM, Shen Li <cs...@gmail.com>
> wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > To use KafkaIO in secure mode, I need to set
> > > > > -Djava.security.auth.login.config to point to a JAAS configuration
> > > file.
> > > > > It
> > > > > works fine for local execution. But how can I configure the
> > > > > "java.security.auth.login.config" property in the Beam app when
> the
> > > > > pipeline is submitted to a cluster/cloud-service? Even if I use a
> > ParDo
> > > > to
> > > > > set the system property, there is no guarantee that the ParDo will
> > run
> > > on
> > > > > the same server with the KafkaIO source.
> > > > >
> > > > > For this specific problem, it would be helpful to upgrade to Kafka
> > > Client
> > > > > 0.10.2.0, which provides a "sasl.jaas.config" property that can be
> > > > updated
> > > > > programmatically. Or is there any other work around?
> > > > >
> > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > > 85%3A+Dynamic+JAAS+
> > > > > configuration+for+Kafka+clients
> > > > >
> > > > > Thanks,
> > > > > Shen
> > > > >
> > > >
> > >
> >
> >
> >
> > --
> > ----
> > Mingmin
> >
>
Re: Is upgrading to Kafka Client 0.10.2.0+ in the roadmap?
Posted by Mingmin Xu <mi...@gmail.com>.
Thanks for the feedback, glad to know that it works now.
Mingmin
On Mon, Oct 30, 2017 at 11:10 AM, Shen Li <cs...@gmail.com> wrote:
> Dear All,
>
> Thanks a lot for the information. I am using Beam-2.0.
> https://github.com/apache/beam/blob/release-2.0.0/sdks/
> java/io/kafka/pom.xml#L33
>
> I have just verified that adding Kafka-Client 0.11 in the application
> pom.xml works fine for me. I can now avoid the JAAS configuration file by
> using the "java.security.auth.login.config" property.
>
> Best,
> Shen
>
> On Mon, Oct 30, 2017 at 1:41 PM, Mingmin Xu <mi...@gmail.com> wrote:
>
> > Hi Shen,
> >
> > Can you share which Beam version are you using? Just check master code,
> the
> > default version for Kafka is
> > `<kafka.clients.version>0.11.0.1</kafka.clients.version>`.
> > I cannot recall the usage for old versions, my
> application(2.2.0-SNAPSHOT)
> > works with a customized kafka version based on 0.10.00-SASL. What you
> need
> > to do is
> > 1). exclude the kafka-client in KafkaIO, and add your own Kafka client
> > library in pom.xml;
> > 2). add your configuration like:
> > ```
> > Map<String, Object> consumerPara = new HashMap<String, Object>();
> > //consumerPara.put(ConsumerConfig.GROUP_ID_CONFIG,
> consumerName);
> > //consumerPara.put(ConsumerConfig.CLIENT_ID_CONFIG, clientId);
> > if (secureEnabled) {
> > consumerPara.put("sasl.mechanism", "IAF");
> > consumerPara.put("security.protocol", "SASL_PLAINTEXT");
> > consumerPara.put("sasl.login.class", ".....");
> > consumerPara.put("sasl.callback.handler.class", "...");
> > }
> >
> > KafkaIO.<byte[], byte[]>read()
> > ....
> > .updateConsumerProperties(configUpdates)
> > ....;
> > ```
> >
> > Mingmin
> >
> > On Mon, Oct 30, 2017 at 10:21 AM, Raghu Angadi
> <rangadi@google.com.invalid
> > >
> > wrote:
> >
> > > > https://issues.apache.org/jira/browse/BEAM-307
> > >
> > > This should be closed.
> > >
> > > On Mon, Oct 30, 2017 at 9:00 AM, Lukasz Cwik <lcwik@google.com.invalid
> >
> > > wrote:
> > >
> > > > There has been some discussion about getting Kafka 0.10.x working on
> > > > BEAM-307[1].
> > > >
> > > > As an immediate way to unblock yourself, modify your local copy of
> the
> > > > KafkaIO source to include setting the system property in a static
> block
> > > > before the class is loaded or before the Kafka client is instantiated
> > and
> > > > used.
> > > >
> > > > Also consider contributing to the Kafka connector to getting 0.10.x
> > > > working.
> > > >
> > > > 1: https://issues.apache.org/jira/browse/BEAM-307
> > > >
> > > > On Mon, Oct 30, 2017 at 8:14 AM, Shen Li <cs...@gmail.com>
> wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > To use KafkaIO in secure mode, I need to set
> > > > > -Djava.security.auth.login.config to point to a JAAS configuration
> > > file.
> > > > > It
> > > > > works fine for local execution. But how can I configure the
> > > > > "java.security.auth.login.config" property in the Beam app when
> the
> > > > > pipeline is submitted to a cluster/cloud-service? Even if I use a
> > ParDo
> > > > to
> > > > > set the system property, there is no guarantee that the ParDo will
> > run
> > > on
> > > > > the same server with the KafkaIO source.
> > > > >
> > > > > For this specific problem, it would be helpful to upgrade to Kafka
> > > Client
> > > > > 0.10.2.0, which provides a "sasl.jaas.config" property that can be
> > > > updated
> > > > > programmatically. Or is there any other work around?
> > > > >
> > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > > 85%3A+Dynamic+JAAS+
> > > > > configuration+for+Kafka+clients
> > > > >
> > > > > Thanks,
> > > > > Shen
> > > > >
> > > >
> > >
> >
> >
> >
> > --
> > ----
> > Mingmin
> >
>
--
----
Mingmin
Re: Is upgrading to Kafka Client 0.10.2.0+ in the roadmap?
Posted by Shen Li <cs...@gmail.com>.
Dear All,
Thanks a lot for the information. I am using Beam-2.0.
https://github.com/apache/beam/blob/release-2.0.0/sdks/java/io/kafka/pom.xml#L33
I have just verified that adding Kafka-Client 0.11 in the application
pom.xml works fine for me. I can now avoid the JAAS configuration file by
using the "java.security.auth.login.config" property.
Best,
Shen
On Mon, Oct 30, 2017 at 1:41 PM, Mingmin Xu <mi...@gmail.com> wrote:
> Hi Shen,
>
> Can you share which Beam version are you using? Just check master code, the
> default version for Kafka is
> `<kafka.clients.version>0.11.0.1</kafka.clients.version>`.
> I cannot recall the usage for old versions, my application(2.2.0-SNAPSHOT)
> works with a customized kafka version based on 0.10.00-SASL. What you need
> to do is
> 1). exclude the kafka-client in KafkaIO, and add your own Kafka client
> library in pom.xml;
> 2). add your configuration like:
> ```
> Map<String, Object> consumerPara = new HashMap<String, Object>();
> //consumerPara.put(ConsumerConfig.GROUP_ID_CONFIG, consumerName);
> //consumerPara.put(ConsumerConfig.CLIENT_ID_CONFIG, clientId);
> if (secureEnabled) {
> consumerPara.put("sasl.mechanism", "IAF");
> consumerPara.put("security.protocol", "SASL_PLAINTEXT");
> consumerPara.put("sasl.login.class", ".....");
> consumerPara.put("sasl.callback.handler.class", "...");
> }
>
> KafkaIO.<byte[], byte[]>read()
> ....
> .updateConsumerProperties(configUpdates)
> ....;
> ```
>
> Mingmin
>
> On Mon, Oct 30, 2017 at 10:21 AM, Raghu Angadi <rangadi@google.com.invalid
> >
> wrote:
>
> > > https://issues.apache.org/jira/browse/BEAM-307
> >
> > This should be closed.
> >
> > On Mon, Oct 30, 2017 at 9:00 AM, Lukasz Cwik <lc...@google.com.invalid>
> > wrote:
> >
> > > There has been some discussion about getting Kafka 0.10.x working on
> > > BEAM-307[1].
> > >
> > > As an immediate way to unblock yourself, modify your local copy of the
> > > KafkaIO source to include setting the system property in a static block
> > > before the class is loaded or before the Kafka client is instantiated
> and
> > > used.
> > >
> > > Also consider contributing to the Kafka connector to getting 0.10.x
> > > working.
> > >
> > > 1: https://issues.apache.org/jira/browse/BEAM-307
> > >
> > > On Mon, Oct 30, 2017 at 8:14 AM, Shen Li <cs...@gmail.com> wrote:
> > >
> > > > Hi,
> > > >
> > > > To use KafkaIO in secure mode, I need to set
> > > > -Djava.security.auth.login.config to point to a JAAS configuration
> > file.
> > > > It
> > > > works fine for local execution. But how can I configure the
> > > > "java.security.auth.login.config" property in the Beam app when the
> > > > pipeline is submitted to a cluster/cloud-service? Even if I use a
> ParDo
> > > to
> > > > set the system property, there is no guarantee that the ParDo will
> run
> > on
> > > > the same server with the KafkaIO source.
> > > >
> > > > For this specific problem, it would be helpful to upgrade to Kafka
> > Client
> > > > 0.10.2.0, which provides a "sasl.jaas.config" property that can be
> > > updated
> > > > programmatically. Or is there any other work around?
> > > >
> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > 85%3A+Dynamic+JAAS+
> > > > configuration+for+Kafka+clients
> > > >
> > > > Thanks,
> > > > Shen
> > > >
> > >
> >
>
>
>
> --
> ----
> Mingmin
>
Re: Is upgrading to Kafka Client 0.10.2.0+ in the roadmap?
Posted by Mingmin Xu <mi...@gmail.com>.
Hi Shen,
Can you share which Beam version are you using? Just check master code, the
default version for Kafka is
`<kafka.clients.version>0.11.0.1</kafka.clients.version>`.
I cannot recall the usage for old versions, my application(2.2.0-SNAPSHOT)
works with a customized kafka version based on 0.10.00-SASL. What you need
to do is
1). exclude the kafka-client in KafkaIO, and add your own Kafka client
library in pom.xml;
2). add your configuration like:
```
Map<String, Object> consumerPara = new HashMap<String, Object>();
//consumerPara.put(ConsumerConfig.GROUP_ID_CONFIG, consumerName);
//consumerPara.put(ConsumerConfig.CLIENT_ID_CONFIG, clientId);
if (secureEnabled) {
consumerPara.put("sasl.mechanism", "IAF");
consumerPara.put("security.protocol", "SASL_PLAINTEXT");
consumerPara.put("sasl.login.class", ".....");
consumerPara.put("sasl.callback.handler.class", "...");
}
KafkaIO.<byte[], byte[]>read()
....
.updateConsumerProperties(configUpdates)
....;
```
Mingmin
On Mon, Oct 30, 2017 at 10:21 AM, Raghu Angadi <ra...@google.com.invalid>
wrote:
> > https://issues.apache.org/jira/browse/BEAM-307
>
> This should be closed.
>
> On Mon, Oct 30, 2017 at 9:00 AM, Lukasz Cwik <lc...@google.com.invalid>
> wrote:
>
> > There has been some discussion about getting Kafka 0.10.x working on
> > BEAM-307[1].
> >
> > As an immediate way to unblock yourself, modify your local copy of the
> > KafkaIO source to include setting the system property in a static block
> > before the class is loaded or before the Kafka client is instantiated and
> > used.
> >
> > Also consider contributing to the Kafka connector to getting 0.10.x
> > working.
> >
> > 1: https://issues.apache.org/jira/browse/BEAM-307
> >
> > On Mon, Oct 30, 2017 at 8:14 AM, Shen Li <cs...@gmail.com> wrote:
> >
> > > Hi,
> > >
> > > To use KafkaIO in secure mode, I need to set
> > > -Djava.security.auth.login.config to point to a JAAS configuration
> file.
> > > It
> > > works fine for local execution. But how can I configure the
> > > "java.security.auth.login.config" property in the Beam app when the
> > > pipeline is submitted to a cluster/cloud-service? Even if I use a ParDo
> > to
> > > set the system property, there is no guarantee that the ParDo will run
> on
> > > the same server with the KafkaIO source.
> > >
> > > For this specific problem, it would be helpful to upgrade to Kafka
> Client
> > > 0.10.2.0, which provides a "sasl.jaas.config" property that can be
> > updated
> > > programmatically. Or is there any other work around?
> > >
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > 85%3A+Dynamic+JAAS+
> > > configuration+for+Kafka+clients
> > >
> > > Thanks,
> > > Shen
> > >
> >
>
--
----
Mingmin
Re: Is upgrading to Kafka Client 0.10.2.0+ in the roadmap?
Posted by Raghu Angadi <ra...@google.com.INVALID>.
> https://issues.apache.org/jira/browse/BEAM-307
This should be closed.
On Mon, Oct 30, 2017 at 9:00 AM, Lukasz Cwik <lc...@google.com.invalid>
wrote:
> There has been some discussion about getting Kafka 0.10.x working on
> BEAM-307[1].
>
> As an immediate way to unblock yourself, modify your local copy of the
> KafkaIO source to include setting the system property in a static block
> before the class is loaded or before the Kafka client is instantiated and
> used.
>
> Also consider contributing to the Kafka connector to getting 0.10.x
> working.
>
> 1: https://issues.apache.org/jira/browse/BEAM-307
>
> On Mon, Oct 30, 2017 at 8:14 AM, Shen Li <cs...@gmail.com> wrote:
>
> > Hi,
> >
> > To use KafkaIO in secure mode, I need to set
> > -Djava.security.auth.login.config to point to a JAAS configuration file.
> > It
> > works fine for local execution. But how can I configure the
> > "java.security.auth.login.config" property in the Beam app when the
> > pipeline is submitted to a cluster/cloud-service? Even if I use a ParDo
> to
> > set the system property, there is no guarantee that the ParDo will run on
> > the same server with the KafkaIO source.
> >
> > For this specific problem, it would be helpful to upgrade to Kafka Client
> > 0.10.2.0, which provides a "sasl.jaas.config" property that can be
> updated
> > programmatically. Or is there any other work around?
> >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 85%3A+Dynamic+JAAS+
> > configuration+for+Kafka+clients
> >
> > Thanks,
> > Shen
> >
>
Re: Is upgrading to Kafka Client 0.10.2.0+ in the roadmap?
Posted by Lukasz Cwik <lc...@google.com.INVALID>.
There has been some discussion about getting Kafka 0.10.x working on
BEAM-307[1].
As an immediate way to unblock yourself, modify your local copy of the
KafkaIO source to include setting the system property in a static block
before the class is loaded or before the Kafka client is instantiated and
used.
Also consider contributing to the Kafka connector to getting 0.10.x working.
1: https://issues.apache.org/jira/browse/BEAM-307
On Mon, Oct 30, 2017 at 8:14 AM, Shen Li <cs...@gmail.com> wrote:
> Hi,
>
> To use KafkaIO in secure mode, I need to set
> -Djava.security.auth.login.config to point to a JAAS configuration file.
> It
> works fine for local execution. But how can I configure the
> "java.security.auth.login.config" property in the Beam app when the
> pipeline is submitted to a cluster/cloud-service? Even if I use a ParDo to
> set the system property, there is no guarantee that the ParDo will run on
> the same server with the KafkaIO source.
>
> For this specific problem, it would be helpful to upgrade to Kafka Client
> 0.10.2.0, which provides a "sasl.jaas.config" property that can be updated
> programmatically. Or is there any other work around?
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-85%3A+Dynamic+JAAS+
> configuration+for+Kafka+clients
>
> Thanks,
> Shen
>