You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by Clebert Suconic <cl...@gmail.com> on 2017/10/02 13:14:43 UTC

[DISCUSS] URIs on Producer and Consumer

At ActiveMQ and ActiveMQ Artemis, ConnectionFactories have an
interesting feature where you can pass parameters through an URI.

I was looking at Producer and Consumer APIs, and these two classes are
using a method that I considered old for Artemis resembling HornetQ:

Instead of passing a Properties (aka HashMaps), users would be able to
create a Consumer or Producer by simply doing:

new Consumer("tcp::/host:port?properties=values;properties=values...etc");

Example:


Instead of the following:

Map<String, Object> config = new HashMap<>();
config.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9999");
config.put(ConsumerConfig.RECEIVE_BUFFER_CONFIG, -2);
new KafkaConsumer<>(config, new ByteArrayDeserializer(), new
ByteArrayDeserializer());



Someone could do

new KafkaConsumer<>("tcp://localhost:9999?receive.buffer.bytes=-2",
new ByteArrayDeserializer(), new ByteArrayDeserializer());



I don't know if that little API improvement would be welcomed? I would be
able to send a Pull Request but I don't want to do it if that wouldn't
be welcomed in the first place:


Just an idea...  let me know if that is welcomed or not.

If so I can forward the discussion into how I would implement it.

Re: [DISCUSS] URIs on Producer and Consumer

Posted by Christopher Shannon <ch...@gmail.com>.
I think this would be useful as a secondary way to configure.  If others
agree then you can write up a KIP and it can be discussed in more detail.

On Tue, Oct 3, 2017 at 8:56 AM, Clebert Suconic <cl...@gmail.com>
wrote:

> Maybe I didn't make the message clear enough...
>
> Would using an URI to the constructor (in addition to the properties)
> help the API, or anyone see a reason to not do it?
>
> KafkaConsumer consumer = new
> KafkaConsumer<>("tcp://localhost:9999?receive.buffer.bytes=-2", new
> ByteArrayDeserializer(), new ByteArrayDeserializer());
>
> I could send a Pull Request for that. The framework I would write
> would validate if the parameters are valid or not.
>
>
> Thanks in advance
>
>
> On Mon, Oct 2, 2017 at 9:14 AM, Clebert Suconic
> <cl...@gmail.com> wrote:
> > At ActiveMQ and ActiveMQ Artemis, ConnectionFactories have an
> > interesting feature where you can pass parameters through an URI.
> >
> > I was looking at Producer and Consumer APIs, and these two classes are
> > using a method that I considered old for Artemis resembling HornetQ:
> >
> > Instead of passing a Properties (aka HashMaps), users would be able to
> > create a Consumer or Producer by simply doing:
> >
> > new Consumer("tcp::/host:port?properties=values;properties=
> values...etc");
> >
> > Example:
> >
> >
> > Instead of the following:
> >
> > Map<String, Object> config = new HashMap<>();
> > config.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9999");
> > config.put(ConsumerConfig.RECEIVE_BUFFER_CONFIG, -2);
> > new KafkaConsumer<>(config, new ByteArrayDeserializer(), new
> > ByteArrayDeserializer());
> >
> >
> >
> > Someone could do
> >
> > new KafkaConsumer<>("tcp://localhost:9999?receive.buffer.bytes=-2",
> > new ByteArrayDeserializer(), new ByteArrayDeserializer());
> >
> >
> >
> > I don't know if that little API improvement would be welcomed? I would be
> > able to send a Pull Request but I don't want to do it if that wouldn't
> > be welcomed in the first place:
> >
> >
> > Just an idea...  let me know if that is welcomed or not.
> >
> > If so I can forward the discussion into how I would implement it.
>
>
>
> --
> Clebert Suconic
>

Re: [DISCUSS] URIs on Producer and Consumer

Posted by Clebert Suconic <cl...@gmail.com>.
Maybe I didn't make the message clear enough...

Would using an URI to the constructor (in addition to the properties)
help the API, or anyone see a reason to not do it?

KafkaConsumer consumer = new
KafkaConsumer<>("tcp://localhost:9999?receive.buffer.bytes=-2", new
ByteArrayDeserializer(), new ByteArrayDeserializer());

I could send a Pull Request for that. The framework I would write
would validate if the parameters are valid or not.


Thanks in advance


On Mon, Oct 2, 2017 at 9:14 AM, Clebert Suconic
<cl...@gmail.com> wrote:
> At ActiveMQ and ActiveMQ Artemis, ConnectionFactories have an
> interesting feature where you can pass parameters through an URI.
>
> I was looking at Producer and Consumer APIs, and these two classes are
> using a method that I considered old for Artemis resembling HornetQ:
>
> Instead of passing a Properties (aka HashMaps), users would be able to
> create a Consumer or Producer by simply doing:
>
> new Consumer("tcp::/host:port?properties=values;properties=values...etc");
>
> Example:
>
>
> Instead of the following:
>
> Map<String, Object> config = new HashMap<>();
> config.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9999");
> config.put(ConsumerConfig.RECEIVE_BUFFER_CONFIG, -2);
> new KafkaConsumer<>(config, new ByteArrayDeserializer(), new
> ByteArrayDeserializer());
>
>
>
> Someone could do
>
> new KafkaConsumer<>("tcp://localhost:9999?receive.buffer.bytes=-2",
> new ByteArrayDeserializer(), new ByteArrayDeserializer());
>
>
>
> I don't know if that little API improvement would be welcomed? I would be
> able to send a Pull Request but I don't want to do it if that wouldn't
> be welcomed in the first place:
>
>
> Just an idea...  let me know if that is welcomed or not.
>
> If so I can forward the discussion into how I would implement it.



-- 
Clebert Suconic

Re: [DISCUSS] URIs on Producer and Consumer

Posted by Ismael Juma <is...@juma.me.uk>.
You can update your properties file in the same way with the current
clients, right? I don't understand how mapping properties into a single
string makes things easier.

Ismael

On Thu, Oct 5, 2017 at 11:22 PM, Clebert Suconic <cl...@gmail.com>
wrote:

> Another useful feature for this...
>
> Say that I am writing an application... if I write this URI on any of
> my internal properties.. I can tweak my Consumer or Producer without
> changing any code.
>
> Say, you store the URI on your project's configuration xml..read it
> and start your consumer... later on you could just update the XML, and
> your user wouldn't need to rewrite his application.
>
>
> right now if you (as an user) want to do such thing.. you would have
> to write the parser yourself.
>
> On Thu, Oct 5, 2017 at 4:33 PM, Michael Pearce <Mi...@ig.com>
> wrote:
> > To me, this is a lot more in line with many other systems connections,
> to have the ability to have a single connection string / uri, is this
> really that left field suggesting or wanting this?
> >
> > If anything this bring kafka more standardised approach imo, to have a
> unified resource identifier, protocol name and a set schema for that.
> >
> > e.g.
> > Database connection strings like
> >
> > oracle:
> > jdbc:oracle:thin:@(description=(address_list=
> >    (address=(protocol=tcp)(port=1521)(host=prodHost)))
> > (connect_data=(INSTANCE_NAME=ORCL)))
> >
> > or
> >
> > postgres:
> > "jdbc:postgresql://localhost/test?user=fred&password=secret&ssl=true"
> >
> > And then like wise on messaging front, systems like
> >
> > rabbitmq
> > “amqp://myhost?heartbeat=10&connection_timeout=10000”
> >
> >
> > I personally like the suggestion +1.
> >
> >
> > Cheers
> > Mike
> >
> >
> > On 05/10/2017, 20:10, "Clebert Suconic" <cl...@gmail.com>
> wrote:
> >
> >     On Thu, Oct 5, 2017 at 2:20 PM, Colin McCabe <cm...@apache.org>
> wrote:
> >     > We used URIs as file paths in Hadoop.  I think it was a mistake,
> for a
> >     > few different reasons.
> >     >
> >     > URIs are actually very complex.  You probably know about scheme,
> host,
> >     > and port, but did you know about authority, user-info, query,
> fragment,
> >     > scheme-specific-part?  Do you know what they do in Hadoop?  The
> mapping
> >     > isn't obvious (and it wouldn't be obvious in Kafka either).
> >
> >     URIs are just a hashmap of key=string.. just like Properties...
> >
> >     The Consumer and Producer is just having such hashMap.. and these
> >     values are easy to translate to boolean, integer.. etc. We would just
> >     need to add such mapping as part of this task when done. I don't see
> >     anything difficult there.
> >
> >
> >     >
> >     > When you flip back and forth between URIs and strings (and you
> >     > inevitably will do this, when serializing or sending things over
> the
> >     > wire), you run into tons of really hard problems.  Should you
> preserve
> >     > the "fragment" (the thing after the hash mark) for your URI, or
> not?  It
> >     > may not do anything now, but maybe it will do something later.
> URIs
> >     > also have complex string escaping rules.  Parsing URIs is very
> messy,
> >     > especially when you start talking about non-Java programming
> languages.
> >
> >
> >     Why flip back and forth? URIs would generate the same HashMap that's
> >     being generated today.. I don't see any mess here.
> >     Besides... This would be an addition, not replacement...
> >
> >     And I'm talking only about the Java API now.
> >
> >     Again, All the properties on ProducerConfig and ConsumerConfig seems
> >     easy to be mapped as primitive types (String, numbers.. booleans).
> >
> >     Serialization shouldn't be a problem there. it would generate the
> same
> >     properties it's generated now.
> >
> >     >
> >     > URIs are designed for a world where you talk to a single host over
> a
> >     > single port.  That isn't the world distributed systems live in.
> You
> >     > don't want your clients to fail to bootstrap because the single
> server
> >     > you specified is having a bad day, even when the other 8 servers
> are up.
> >
> >     I have seen a few projects using this style of URI: I would make it
> >     doing the same here:
> >
> >     If you have multiple hosts:
> >
> >     KafkaConsumer consumer = new
> >     KafkaConsumer("kafka:(kafka://host1:port,kafka://host2:port)
> ?property1=value");
> >
> >     if you have a single host:
> >     KafkaConsumer consumer = new
> >     KafkaConsumer("kafka://host2:port?property1=value&
> property2=value2");
> >
> >
> >     One example of an apache project using a similar approach is
> qpid-jms:
> >     http://qpid.apache.org/releases/qpid-jms-0.25.0/docs/
> index.html#failover-configuration-options
> >
> >
> >     > The bottom line is that URIs are the wrong abstraction for the job.
> >     > They just don't express what we really want, and they introduce a
> lot of
> >     > complexity and ambiguity.
> >
> >     I have seen the opposite to be honest. this has been simpler for me
> >     and users I know than using a HashMap.. .  users in my experience
> tend
> >     to write this faster.
> >
> >     users can certainly put up with the HashMap.. but this is easier to
> >     remember. I'm just proposing what I think it's a simpler API.
> >
> >
> >
> >
> >     Perhaps we should move into the KIP discussion itself here.. I first
> >     intended to start this thread to see if it would make sense or not...
> >     But I don't have authorization to create the KIP page.. so again..
> >     based on the contributing page.. can someone add me authorizations to
> >     the WIKI space?
> >
> >
> > The information contained in this email is strictly confidential and for
> the use of the addressee only, unless otherwise indicated. If you are not
> the intended recipient, please do not read, copy, use or disclose to others
> this message or any attachment. Please also notify the sender by replying
> to this email or by telephone (+44(020 7896 0011) and then delete the
> email and any copies of it. Opinions, conclusion (etc) that do not relate
> to the official business of this company shall be understood as neither
> given nor endorsed by it. IG is a trading name of IG Markets Limited (a
> company registered in England and Wales, company number 04008957) and IG
> Index Limited (a company registered in England and Wales, company number
> 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
> London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG
> Index Limited (register number 114059) are authorised and regulated by the
> Financial Conduct Authority.
>
>
>
> --
> Clebert Suconic
>

Re: [DISCUSS] URIs on Producer and Consumer

Posted by Clebert Suconic <cl...@gmail.com>.
Another useful feature for this...

Say that I am writing an application... if I write this URI on any of
my internal properties.. I can tweak my Consumer or Producer without
changing any code.

Say, you store the URI on your project's configuration xml..read it
and start your consumer... later on you could just update the XML, and
your user wouldn't need to rewrite his application.


right now if you (as an user) want to do such thing.. you would have
to write the parser yourself.

On Thu, Oct 5, 2017 at 4:33 PM, Michael Pearce <Mi...@ig.com> wrote:
> To me, this is a lot more in line with many other systems connections, to have the ability to have a single connection string / uri, is this really that left field suggesting or wanting this?
>
> If anything this bring kafka more standardised approach imo, to have a unified resource identifier, protocol name and a set schema for that.
>
> e.g.
> Database connection strings like
>
> oracle:
> jdbc:oracle:thin:@(description=(address_list=
>    (address=(protocol=tcp)(port=1521)(host=prodHost)))
> (connect_data=(INSTANCE_NAME=ORCL)))
>
> or
>
> postgres:
> "jdbc:postgresql://localhost/test?user=fred&password=secret&ssl=true"
>
> And then like wise on messaging front, systems like
>
> rabbitmq
> “amqp://myhost?heartbeat=10&connection_timeout=10000”
>
>
> I personally like the suggestion +1.
>
>
> Cheers
> Mike
>
>
> On 05/10/2017, 20:10, "Clebert Suconic" <cl...@gmail.com> wrote:
>
>     On Thu, Oct 5, 2017 at 2:20 PM, Colin McCabe <cm...@apache.org> wrote:
>     > We used URIs as file paths in Hadoop.  I think it was a mistake, for a
>     > few different reasons.
>     >
>     > URIs are actually very complex.  You probably know about scheme, host,
>     > and port, but did you know about authority, user-info, query, fragment,
>     > scheme-specific-part?  Do you know what they do in Hadoop?  The mapping
>     > isn't obvious (and it wouldn't be obvious in Kafka either).
>
>     URIs are just a hashmap of key=string.. just like Properties...
>
>     The Consumer and Producer is just having such hashMap.. and these
>     values are easy to translate to boolean, integer.. etc. We would just
>     need to add such mapping as part of this task when done. I don't see
>     anything difficult there.
>
>
>     >
>     > When you flip back and forth between URIs and strings (and you
>     > inevitably will do this, when serializing or sending things over the
>     > wire), you run into tons of really hard problems.  Should you preserve
>     > the "fragment" (the thing after the hash mark) for your URI, or not?  It
>     > may not do anything now, but maybe it will do something later.  URIs
>     > also have complex string escaping rules.  Parsing URIs is very messy,
>     > especially when you start talking about non-Java programming languages.
>
>
>     Why flip back and forth? URIs would generate the same HashMap that's
>     being generated today.. I don't see any mess here.
>     Besides... This would be an addition, not replacement...
>
>     And I'm talking only about the Java API now.
>
>     Again, All the properties on ProducerConfig and ConsumerConfig seems
>     easy to be mapped as primitive types (String, numbers.. booleans).
>
>     Serialization shouldn't be a problem there. it would generate the same
>     properties it's generated now.
>
>     >
>     > URIs are designed for a world where you talk to a single host over a
>     > single port.  That isn't the world distributed systems live in.  You
>     > don't want your clients to fail to bootstrap because the single server
>     > you specified is having a bad day, even when the other 8 servers are up.
>
>     I have seen a few projects using this style of URI: I would make it
>     doing the same here:
>
>     If you have multiple hosts:
>
>     KafkaConsumer consumer = new
>     KafkaConsumer("kafka:(kafka://host1:port,kafka://host2:port)?property1=value");
>
>     if you have a single host:
>     KafkaConsumer consumer = new
>     KafkaConsumer("kafka://host2:port?property1=value&property2=value2");
>
>
>     One example of an apache project using a similar approach is qpid-jms:
>     http://qpid.apache.org/releases/qpid-jms-0.25.0/docs/index.html#failover-configuration-options
>
>
>     > The bottom line is that URIs are the wrong abstraction for the job.
>     > They just don't express what we really want, and they introduce a lot of
>     > complexity and ambiguity.
>
>     I have seen the opposite to be honest. this has been simpler for me
>     and users I know than using a HashMap.. .  users in my experience tend
>     to write this faster.
>
>     users can certainly put up with the HashMap.. but this is easier to
>     remember. I'm just proposing what I think it's a simpler API.
>
>
>
>
>     Perhaps we should move into the KIP discussion itself here.. I first
>     intended to start this thread to see if it would make sense or not...
>     But I don't have authorization to create the KIP page.. so again..
>     based on the contributing page.. can someone add me authorizations to
>     the WIKI space?
>
>
> The information contained in this email is strictly confidential and for the use of the addressee only, unless otherwise indicated. If you are not the intended recipient, please do not read, copy, use or disclose to others this message or any attachment. Please also notify the sender by replying to this email or by telephone (+44(020 7896 0011) and then delete the email and any copies of it. Opinions, conclusion (etc) that do not relate to the official business of this company shall be understood as neither given nor endorsed by it. IG is a trading name of IG Markets Limited (a company registered in England and Wales, company number 04008957) and IG Index Limited (a company registered in England and Wales, company number 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill, London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG Index Limited (register number 114059) are authorised and regulated by the Financial Conduct Authority.



-- 
Clebert Suconic

Re: [DISCUSS] URIs on Producer and Consumer

Posted by Clebert Suconic <cl...@gmail.com>.
> >
> >
> > Although I can't start a KIP... again.. (lost the count now).. can
> > someone add me authorizations to the WIKI Page so I can create the
> > page and start a proper discussion there?
>
> Hmm, I don't have permissions to add you.  admins?




Perhaps I should email the list with a different subject?

I was just following the contributing page saying I should ask permissions
here to create the page with the KIP template.





>
> regards,
> Colin
>
-- 
Clebert Suconic

Re: [DISCUSS] URIs on Producer and Consumer

Posted by Colin McCabe <cm...@apache.org>.
On Fri, Oct 6, 2017, at 10:12, Clebert Suconic wrote:
> > As I said before, a connection string might be a good idea.  A URI, no.
> 
> 
> Same purpose.. I'm fine with that...

Sounds good to me.  You might want to discuss this on the librdkafka
mailing list as well, to see if there are any ideas that can be shared.

> 
> 
> Although I can't start a KIP... again.. (lost the count now).. can
> someone add me authorizations to the WIKI Page so I can create the
> page and start a proper discussion there?

Hmm, I don't have permissions to add you.  admins?

regards,
Colin

Re: [DISCUSS] URIs on Producer and Consumer

Posted by Clebert Suconic <cl...@gmail.com>.
> As I said before, a connection string might be a good idea.  A URI, no.


Same purpose.. I'm fine with that...


Although I can't start a KIP... again.. (lost the count now).. can
someone add me authorizations to the WIKI Page so I can create the
page and start a proper discussion there?

Re: [DISCUSS] URIs on Producer and Consumer

Posted by Colin McCabe <co...@cmccabe.xyz>.
On Thu, Oct 5, 2017, at 13:33, Michael Pearce wrote:
> To me, this is a lot more in line with many other systems connections, to
> have the ability to have a single connection string / uri, is this really
> that left field suggesting or wanting this?
> 
> If anything this bring kafka more standardised approach imo, to have a
> unified resource identifier, protocol name and a set schema for that.
> 
> e.g.
> Database connection strings like
> 
> oracle:
> jdbc:oracle:thin:@(description=(address_list=
>    (address=(protocol=tcp)(port=1521)(host=prodHost)))
> (connect_data=(INSTANCE_NAME=ORCL)))

Hmm.  That isn't a URI, though, right?  So adopting URIs doesn't help us
integrate with JDBC.  In any case, since Kafka is not a database, it is
a little unclear what better integration with JDBC would look like. 
Perhaps that is worth thinking about at some point, but it seems
unrelated to this URI discussion.

> On 05/10/2017, 20:10, "Clebert Suconic" <cl...@gmail.com>
> wrote:
> 
>     On Thu, Oct 5, 2017 at 2:20 PM, Colin McCabe <cm...@apache.org>
>     wrote:
>     > We used URIs as file paths in Hadoop.  I think it was a mistake, for a
>     > few different reasons.
>     >
>     > URIs are actually very complex.  You probably know about scheme, host,
>     > and port, but did you know about authority, user-info, query, fragment,
>     > scheme-specific-part?  Do you know what they do in Hadoop?  The mapping
>     > isn't obvious (and it wouldn't be obvious in Kafka either).
> 
>     URIs are just a hashmap of key=string.. just like Properties...

You really can't treat a URI as a hashmap.  For one thing, the scheme
and hostname parts are not optional.

You are probably thinking of the "query" part (the part after the
question mark).  This isn't  map either-- it's a sequence of
comma-separated key=value pairs.  The same key can appear multiple
times.  And you have to encode everything with RFC3986 "percent
encoding."

>     The Consumer and Producer is just having such hashMap.. and these
>     values are easy to translate to boolean, integer.. etc. We would just
>     need to add such mapping as part of this task when done. I don't see
>     anything difficult there.

I don't object to having some kind of connection string that rolls up
all the configuration properties.  I just don't think it should be a
URI.

>     >
>     > When you flip back and forth between URIs and strings (and you
>     > inevitably will do this, when serializing or sending things over the
>     > wire), you run into tons of really hard problems.  Should you preserve
>     > the "fragment" (the thing after the hash mark) for your URI, or not?  It
>     > may not do anything now, but maybe it will do something later.  URIs
>     > also have complex string escaping rules.  Parsing URIs is very messy,
>     > especially when you start talking about non-Java programming languages.
> 
> 
>     Why flip back and forth? URIs would generate the same HashMap that's
>     being generated today.. I don't see any mess here.
>     Besides... This would be an addition, not replacement...
> 
>     And I'm talking only about the Java API now.

We have a lot of non-Java clients-- those should be part of the
discussion.

> 
>     Again, All the properties on ProducerConfig and ConsumerConfig seems
>     easy to be mapped as primitive types (String, numbers.. booleans).
> 
>     Serialization shouldn't be a problem there. it would generate the
>     same
>     properties it's generated now.
> 
>     >
>     > URIs are designed for a world where you talk to a single host over a
>     > single port.  That isn't the world distributed systems live in.  You
>     > don't want your clients to fail to bootstrap because the single server
>     > you specified is having a bad day, even when the other 8 servers are up.
> 
>     I have seen a few projects using this style of URI: I would make it
>     doing the same here:
> 
>     If you have multiple hosts:
> 
>     KafkaConsumer consumer = new
>     KafkaConsumer("kafka:(kafka://host1:port,kafka://host2:port)?property1=value");

That's not a valid URI?

> 
>     if you have a single host:
>     KafkaConsumer consumer = new
>     KafkaConsumer("kafka://host2:port?property1=value&property2=value2");
> 
> 
>     One example of an apache project using a similar approach is
>     qpid-jms:
>     http://qpid.apache.org/releases/qpid-jms-0.25.0/docs/index.html#failover-configuration-options
> 
> 
>     > The bottom line is that URIs are the wrong abstraction for the job.
>     > They just don't express what we really want, and they introduce a lot of
>     > complexity and ambiguity.
> 
>     I have seen the opposite to be honest. this has been simpler for me
>     and users I know than using a HashMap.. .  users in my experience
>     tend
>     to write this faster.

Users tend to find make mistakes when writing URIs.  For example, how do
you translate a filename with spaces and commas into a URI?  I had to
debug these issues.  It is why I dislike URIs.

As I said before, a connection string might be a good idea.  A URI, no.

best,
Colin

> 
>     users can certainly put up with the HashMap.. but this is easier to
>     remember. I'm just proposing what I think it's a simpler API.
> 
> 
> 
> 
>     Perhaps we should move into the KIP discussion itself here.. I first
>     intended to start this thread to see if it would make sense or not...
>     But I don't have authorization to create the KIP page.. so again..
>     based on the contributing page.. can someone add me authorizations to
>     the WIKI space?
> 
> 
> The information contained in this email is strictly confidential and for
> the use of the addressee only, unless otherwise indicated. If you are not
> the intended recipient, please do not read, copy, use or disclose to
> others this message or any attachment. Please also notify the sender by
> replying to this email or by telephone (+44(020 7896 0011) and then
> delete the email and any copies of it. Opinions, conclusion (etc) that do
> not relate to the official business of this company shall be understood
> as neither given nor endorsed by it. IG is a trading name of IG Markets
> Limited (a company registered in England and Wales, company number
> 04008957) and IG Index Limited (a company registered in England and
> Wales, company number 01190902). Registered address at Cannon Bridge
> House, 25 Dowgate Hill, London EC4R 2YA. Both IG Markets Limited
> (register number 195355) and IG Index Limited (register number 114059)
> are authorised and regulated by the Financial Conduct Authority.

Re: [DISCUSS] URIs on Producer and Consumer

Posted by Michael Pearce <Mi...@ig.com>.
To me, this is a lot more in line with many other systems connections, to have the ability to have a single connection string / uri, is this really that left field suggesting or wanting this?

If anything this bring kafka more standardised approach imo, to have a unified resource identifier, protocol name and a set schema for that.

e.g.
Database connection strings like

oracle:
jdbc:oracle:thin:@(description=(address_list=
   (address=(protocol=tcp)(port=1521)(host=prodHost)))
(connect_data=(INSTANCE_NAME=ORCL)))

or

postgres:
"jdbc:postgresql://localhost/test?user=fred&password=secret&ssl=true"

And then like wise on messaging front, systems like

rabbitmq
“amqp://myhost?heartbeat=10&connection_timeout=10000”


I personally like the suggestion +1.


Cheers
Mike


On 05/10/2017, 20:10, "Clebert Suconic" <cl...@gmail.com> wrote:

    On Thu, Oct 5, 2017 at 2:20 PM, Colin McCabe <cm...@apache.org> wrote:
    > We used URIs as file paths in Hadoop.  I think it was a mistake, for a
    > few different reasons.
    >
    > URIs are actually very complex.  You probably know about scheme, host,
    > and port, but did you know about authority, user-info, query, fragment,
    > scheme-specific-part?  Do you know what they do in Hadoop?  The mapping
    > isn't obvious (and it wouldn't be obvious in Kafka either).

    URIs are just a hashmap of key=string.. just like Properties...

    The Consumer and Producer is just having such hashMap.. and these
    values are easy to translate to boolean, integer.. etc. We would just
    need to add such mapping as part of this task when done. I don't see
    anything difficult there.


    >
    > When you flip back and forth between URIs and strings (and you
    > inevitably will do this, when serializing or sending things over the
    > wire), you run into tons of really hard problems.  Should you preserve
    > the "fragment" (the thing after the hash mark) for your URI, or not?  It
    > may not do anything now, but maybe it will do something later.  URIs
    > also have complex string escaping rules.  Parsing URIs is very messy,
    > especially when you start talking about non-Java programming languages.


    Why flip back and forth? URIs would generate the same HashMap that's
    being generated today.. I don't see any mess here.
    Besides... This would be an addition, not replacement...

    And I'm talking only about the Java API now.

    Again, All the properties on ProducerConfig and ConsumerConfig seems
    easy to be mapped as primitive types (String, numbers.. booleans).

    Serialization shouldn't be a problem there. it would generate the same
    properties it's generated now.

    >
    > URIs are designed for a world where you talk to a single host over a
    > single port.  That isn't the world distributed systems live in.  You
    > don't want your clients to fail to bootstrap because the single server
    > you specified is having a bad day, even when the other 8 servers are up.

    I have seen a few projects using this style of URI: I would make it
    doing the same here:

    If you have multiple hosts:

    KafkaConsumer consumer = new
    KafkaConsumer("kafka:(kafka://host1:port,kafka://host2:port)?property1=value");

    if you have a single host:
    KafkaConsumer consumer = new
    KafkaConsumer("kafka://host2:port?property1=value&property2=value2");


    One example of an apache project using a similar approach is qpid-jms:
    http://qpid.apache.org/releases/qpid-jms-0.25.0/docs/index.html#failover-configuration-options


    > The bottom line is that URIs are the wrong abstraction for the job.
    > They just don't express what we really want, and they introduce a lot of
    > complexity and ambiguity.

    I have seen the opposite to be honest. this has been simpler for me
    and users I know than using a HashMap.. .  users in my experience tend
    to write this faster.

    users can certainly put up with the HashMap.. but this is easier to
    remember. I'm just proposing what I think it's a simpler API.




    Perhaps we should move into the KIP discussion itself here.. I first
    intended to start this thread to see if it would make sense or not...
    But I don't have authorization to create the KIP page.. so again..
    based on the contributing page.. can someone add me authorizations to
    the WIKI space?


The information contained in this email is strictly confidential and for the use of the addressee only, unless otherwise indicated. If you are not the intended recipient, please do not read, copy, use or disclose to others this message or any attachment. Please also notify the sender by replying to this email or by telephone (+44(020 7896 0011) and then delete the email and any copies of it. Opinions, conclusion (etc) that do not relate to the official business of this company shall be understood as neither given nor endorsed by it. IG is a trading name of IG Markets Limited (a company registered in England and Wales, company number 04008957) and IG Index Limited (a company registered in England and Wales, company number 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill, London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG Index Limited (register number 114059) are authorised and regulated by the Financial Conduct Authority.

Re: [DISCUSS] URIs on Producer and Consumer

Posted by Clebert Suconic <cl...@gmail.com>.
On Thu, Oct 5, 2017 at 2:20 PM, Colin McCabe <cm...@apache.org> wrote:
> We used URIs as file paths in Hadoop.  I think it was a mistake, for a
> few different reasons.
>
> URIs are actually very complex.  You probably know about scheme, host,
> and port, but did you know about authority, user-info, query, fragment,
> scheme-specific-part?  Do you know what they do in Hadoop?  The mapping
> isn't obvious (and it wouldn't be obvious in Kafka either).

URIs are just a hashmap of key=string.. just like Properties...

The Consumer and Producer is just having such hashMap.. and these
values are easy to translate to boolean, integer.. etc. We would just
need to add such mapping as part of this task when done. I don't see
anything difficult there.


>
> When you flip back and forth between URIs and strings (and you
> inevitably will do this, when serializing or sending things over the
> wire), you run into tons of really hard problems.  Should you preserve
> the "fragment" (the thing after the hash mark) for your URI, or not?  It
> may not do anything now, but maybe it will do something later.  URIs
> also have complex string escaping rules.  Parsing URIs is very messy,
> especially when you start talking about non-Java programming languages.


Why flip back and forth? URIs would generate the same HashMap that's
being generated today.. I don't see any mess here.
Besides... This would be an addition, not replacement...

And I'm talking only about the Java API now.

Again, All the properties on ProducerConfig and ConsumerConfig seems
easy to be mapped as primitive types (String, numbers.. booleans).

Serialization shouldn't be a problem there. it would generate the same
properties it's generated now.

>
> URIs are designed for a world where you talk to a single host over a
> single port.  That isn't the world distributed systems live in.  You
> don't want your clients to fail to bootstrap because the single server
> you specified is having a bad day, even when the other 8 servers are up.

I have seen a few projects using this style of URI: I would make it
doing the same here:

If you have multiple hosts:

KafkaConsumer consumer = new
KafkaConsumer("kafka:(kafka://host1:port,kafka://host2:port)?property1=value");

if you have a single host:
KafkaConsumer consumer = new
KafkaConsumer("kafka://host2:port?property1=value&property2=value2");


One example of an apache project using a similar approach is qpid-jms:
http://qpid.apache.org/releases/qpid-jms-0.25.0/docs/index.html#failover-configuration-options


> The bottom line is that URIs are the wrong abstraction for the job.
> They just don't express what we really want, and they introduce a lot of
> complexity and ambiguity.

I have seen the opposite to be honest. this has been simpler for me
and users I know than using a HashMap.. .  users in my experience tend
to write this faster.

users can certainly put up with the HashMap.. but this is easier to
remember. I'm just proposing what I think it's a simpler API.




Perhaps we should move into the KIP discussion itself here.. I first
intended to start this thread to see if it would make sense or not...
But I don't have authorization to create the KIP page.. so again..
based on the contributing page.. can someone add me authorizations to
the WIKI space?

Re: [DISCUSS] URIs on Producer and Consumer

Posted by Colin McCabe <cm...@apache.org>.
We used URIs as file paths in Hadoop.  I think it was a mistake, for a
few different reasons.

URIs are actually very complex.  You probably know about scheme, host,
and port, but did you know about authority, user-info, query, fragment,
scheme-specific-part?  Do you know what they do in Hadoop?  The mapping
isn't obvious (and it wouldn't be obvious in Kafka either).

When you flip back and forth between URIs and strings (and you
inevitably will do this, when serializing or sending things over the
wire), you run into tons of really hard problems.  Should you preserve
the "fragment" (the thing after the hash mark) for your URI, or not?  It
may not do anything now, but maybe it will do something later.  URIs
also have complex string escaping rules.  Parsing URIs is very messy,
especially when you start talking about non-Java programming languages.

URIs are designed for a world where you talk to a single host over a
single port.  That isn't the world distributed systems live in.  You
don't want your clients to fail to bootstrap because the single server
you specified is having a bad day, even when the other 8 servers are up.

What we want is a list of servers.  But URIs don't give us that.  That's
why in HDFS, we introduced another layer of indirection, so that the
"URI hostname" maps to an entry in a configuration file, which then maps
to a list of hostnames.

Later on, we found out that people wanted a unified namespace.  They
wanted to be able to access /foo without caring whether it was on s3,
the first hdfs cluster, or the second hdfs cluster.  But our use of URIs
for paths had made that impossible.  If the path was on s3, it had to be
accessed via s3://mybucketname/foo.  If it was on the first hdfs
cluster, it had to be accessed by hdfs://myhdfs1name/foo.  And so on. 
We had re-invented the equivalent of DOS drive letters: ugly, clunky
drive letter prefixes that to chaperone around every path name.

The bottom line is that URIs are the wrong abstraction for the job. 
They just don't express what we really want, and they introduce a lot of
complexity and ambiguity.

best,
Colin


On Thu, Oct 5, 2017, at 08:08, Clebert Suconic wrote:
> I can start a KIP discussion on this.. or not if you really think this
> is against basic rules...
> 
> 
> I will need authorization to create the page.. if you could assign me
> regardless so I can have it for next time?
> 
> On Thu, Oct 5, 2017 at 10:31 AM, Clebert Suconic
> <cl...@gmail.com> wrote:
> > Just as a facility for users... I think it would be easier to
> > prototype consumers and producer by simply doing new
> > Consumer("tcp://HOST:PORT") or new Producer("tcp://HOST:PORT")...
> >
> > on the other project I work (ActiveMQ Artemis) we used to do a similar
> > way to what Kafka does..we then provided the URI support and I now
> > think the URI was a lot easier.
> >
> > I'm just trying to leverage my experience into here... I'm an apache
> > committer at ActiveMQ Artemis.. I think I could bring some goodies
> > into Kafka.. I see no reason to be a competitor.. instead I'm all up
> > to contribute here as well.  And I was looking for something small and
> > easy to start with.
> >
> >
> >
> >
> >
> >
> > On Thu, Oct 5, 2017 at 10:15 AM, Jay Kreps <ja...@confluent.io> wrote:
> >> Hey Clebert,
> >>
> >> Is there a motivation for adding a second way? We generally try to avoid
> >> having two ways to do something unless it's really needed...I suspect you
> >> have a reason for wanting this, though.
> >>
> >> -Jay
> >>
> >> On Mon, Oct 2, 2017 at 6:15 AM Clebert Suconic <cl...@gmail.com>
> >> wrote:
> >>
> >>> At ActiveMQ and ActiveMQ Artemis, ConnectionFactories have an
> >>> interesting feature where you can pass parameters through an URI.
> >>>
> >>> I was looking at Producer and Consumer APIs, and these two classes are
> >>> using a method that I considered old for Artemis resembling HornetQ:
> >>>
> >>> Instead of passing a Properties (aka HashMaps), users would be able to
> >>> create a Consumer or Producer by simply doing:
> >>>
> >>> new Consumer("tcp::/host:port?properties=values;properties=values...etc");
> >>>
> >>> Example:
> >>>
> >>>
> >>> Instead of the following:
> >>>
> >>> Map<String, Object> config = new HashMap<>();
> >>> config.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9999");
> >>> config.put(ConsumerConfig.RECEIVE_BUFFER_CONFIG, -2);
> >>> new KafkaConsumer<>(config, new ByteArrayDeserializer(), new
> >>> ByteArrayDeserializer());
> >>>
> >>>
> >>>
> >>> Someone could do
> >>>
> >>> new KafkaConsumer<>("tcp://localhost:9999?receive.buffer.bytes=-2",
> >>> new ByteArrayDeserializer(), new ByteArrayDeserializer());
> >>>
> >>>
> >>>
> >>> I don't know if that little API improvement would be welcomed? I would be
> >>> able to send a Pull Request but I don't want to do it if that wouldn't
> >>> be welcomed in the first place:
> >>>
> >>>
> >>> Just an idea...  let me know if that is welcomed or not.
> >>>
> >>> If so I can forward the discussion into how I would implement it.
> >>>
> >
> >
> >
> > --
> > Clebert Suconic
> 
> 
> 
> -- 
> Clebert Suconic

Re: [DISCUSS] URIs on Producer and Consumer

Posted by Clebert Suconic <cl...@gmail.com>.
I can start a KIP discussion on this.. or not if you really think this
is against basic rules...


I will need authorization to create the page.. if you could assign me
regardless so I can have it for next time?

On Thu, Oct 5, 2017 at 10:31 AM, Clebert Suconic
<cl...@gmail.com> wrote:
> Just as a facility for users... I think it would be easier to
> prototype consumers and producer by simply doing new
> Consumer("tcp://HOST:PORT") or new Producer("tcp://HOST:PORT")...
>
> on the other project I work (ActiveMQ Artemis) we used to do a similar
> way to what Kafka does..we then provided the URI support and I now
> think the URI was a lot easier.
>
> I'm just trying to leverage my experience into here... I'm an apache
> committer at ActiveMQ Artemis.. I think I could bring some goodies
> into Kafka.. I see no reason to be a competitor.. instead I'm all up
> to contribute here as well.  And I was looking for something small and
> easy to start with.
>
>
>
>
>
>
> On Thu, Oct 5, 2017 at 10:15 AM, Jay Kreps <ja...@confluent.io> wrote:
>> Hey Clebert,
>>
>> Is there a motivation for adding a second way? We generally try to avoid
>> having two ways to do something unless it's really needed...I suspect you
>> have a reason for wanting this, though.
>>
>> -Jay
>>
>> On Mon, Oct 2, 2017 at 6:15 AM Clebert Suconic <cl...@gmail.com>
>> wrote:
>>
>>> At ActiveMQ and ActiveMQ Artemis, ConnectionFactories have an
>>> interesting feature where you can pass parameters through an URI.
>>>
>>> I was looking at Producer and Consumer APIs, and these two classes are
>>> using a method that I considered old for Artemis resembling HornetQ:
>>>
>>> Instead of passing a Properties (aka HashMaps), users would be able to
>>> create a Consumer or Producer by simply doing:
>>>
>>> new Consumer("tcp::/host:port?properties=values;properties=values...etc");
>>>
>>> Example:
>>>
>>>
>>> Instead of the following:
>>>
>>> Map<String, Object> config = new HashMap<>();
>>> config.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9999");
>>> config.put(ConsumerConfig.RECEIVE_BUFFER_CONFIG, -2);
>>> new KafkaConsumer<>(config, new ByteArrayDeserializer(), new
>>> ByteArrayDeserializer());
>>>
>>>
>>>
>>> Someone could do
>>>
>>> new KafkaConsumer<>("tcp://localhost:9999?receive.buffer.bytes=-2",
>>> new ByteArrayDeserializer(), new ByteArrayDeserializer());
>>>
>>>
>>>
>>> I don't know if that little API improvement would be welcomed? I would be
>>> able to send a Pull Request but I don't want to do it if that wouldn't
>>> be welcomed in the first place:
>>>
>>>
>>> Just an idea...  let me know if that is welcomed or not.
>>>
>>> If so I can forward the discussion into how I would implement it.
>>>
>
>
>
> --
> Clebert Suconic



-- 
Clebert Suconic

Re: [DISCUSS] URIs on Producer and Consumer

Posted by Clebert Suconic <cl...@gmail.com>.
Just as a facility for users... I think it would be easier to
prototype consumers and producer by simply doing new
Consumer("tcp://HOST:PORT") or new Producer("tcp://HOST:PORT")...

on the other project I work (ActiveMQ Artemis) we used to do a similar
way to what Kafka does..we then provided the URI support and I now
think the URI was a lot easier.

I'm just trying to leverage my experience into here... I'm an apache
committer at ActiveMQ Artemis.. I think I could bring some goodies
into Kafka.. I see no reason to be a competitor.. instead I'm all up
to contribute here as well.  And I was looking for something small and
easy to start with.






On Thu, Oct 5, 2017 at 10:15 AM, Jay Kreps <ja...@confluent.io> wrote:
> Hey Clebert,
>
> Is there a motivation for adding a second way? We generally try to avoid
> having two ways to do something unless it's really needed...I suspect you
> have a reason for wanting this, though.
>
> -Jay
>
> On Mon, Oct 2, 2017 at 6:15 AM Clebert Suconic <cl...@gmail.com>
> wrote:
>
>> At ActiveMQ and ActiveMQ Artemis, ConnectionFactories have an
>> interesting feature where you can pass parameters through an URI.
>>
>> I was looking at Producer and Consumer APIs, and these two classes are
>> using a method that I considered old for Artemis resembling HornetQ:
>>
>> Instead of passing a Properties (aka HashMaps), users would be able to
>> create a Consumer or Producer by simply doing:
>>
>> new Consumer("tcp::/host:port?properties=values;properties=values...etc");
>>
>> Example:
>>
>>
>> Instead of the following:
>>
>> Map<String, Object> config = new HashMap<>();
>> config.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9999");
>> config.put(ConsumerConfig.RECEIVE_BUFFER_CONFIG, -2);
>> new KafkaConsumer<>(config, new ByteArrayDeserializer(), new
>> ByteArrayDeserializer());
>>
>>
>>
>> Someone could do
>>
>> new KafkaConsumer<>("tcp://localhost:9999?receive.buffer.bytes=-2",
>> new ByteArrayDeserializer(), new ByteArrayDeserializer());
>>
>>
>>
>> I don't know if that little API improvement would be welcomed? I would be
>> able to send a Pull Request but I don't want to do it if that wouldn't
>> be welcomed in the first place:
>>
>>
>> Just an idea...  let me know if that is welcomed or not.
>>
>> If so I can forward the discussion into how I would implement it.
>>



-- 
Clebert Suconic

Re: [DISCUSS] URIs on Producer and Consumer

Posted by Jay Kreps <ja...@confluent.io>.
Hey Clebert,

Is there a motivation for adding a second way? We generally try to avoid
having two ways to do something unless it's really needed...I suspect you
have a reason for wanting this, though.

-Jay

On Mon, Oct 2, 2017 at 6:15 AM Clebert Suconic <cl...@gmail.com>
wrote:

> At ActiveMQ and ActiveMQ Artemis, ConnectionFactories have an
> interesting feature where you can pass parameters through an URI.
>
> I was looking at Producer and Consumer APIs, and these two classes are
> using a method that I considered old for Artemis resembling HornetQ:
>
> Instead of passing a Properties (aka HashMaps), users would be able to
> create a Consumer or Producer by simply doing:
>
> new Consumer("tcp::/host:port?properties=values;properties=values...etc");
>
> Example:
>
>
> Instead of the following:
>
> Map<String, Object> config = new HashMap<>();
> config.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9999");
> config.put(ConsumerConfig.RECEIVE_BUFFER_CONFIG, -2);
> new KafkaConsumer<>(config, new ByteArrayDeserializer(), new
> ByteArrayDeserializer());
>
>
>
> Someone could do
>
> new KafkaConsumer<>("tcp://localhost:9999?receive.buffer.bytes=-2",
> new ByteArrayDeserializer(), new ByteArrayDeserializer());
>
>
>
> I don't know if that little API improvement would be welcomed? I would be
> able to send a Pull Request but I don't want to do it if that wouldn't
> be welcomed in the first place:
>
>
> Just an idea...  let me know if that is welcomed or not.
>
> If so I can forward the discussion into how I would implement it.
>