generics type for Producer and Consumer do not need to match?

[Yang <te...@gmail.com>]

in the example
https://cwiki.apache.org/confluence/display/KAFKA/0.8.0+Producer+Example

we use a String,String for <K,V>

in the same set of example
https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Group+Example

on the consumer side we use byte[], byte[] for K,V
        ConsumerIterator<byte[], byte[]> it = m_stream.iterator();



we tested in our own code that the above producer-consumer pair does work.


so the K,V types on the 2 sides do NOT need to match? if we argue that
"everything has to come down to byte[], then maybe we don't need the
generics types on the consumer side? in other words, how do we determine
the K,V types on consumer side?

thanks
Yang

Re: generics type for Producer and Consumer do not need to match?

[Yang <te...@gmail.com>]

thanks, I just noticed the consumerconnector.createMessageStream Api, it
has 2 versions

 public <K,V> Map<String, List<KafkaStream<K,V>>>
    createMessageStreams(Map<String, Integer> topicCountMap, Decoder<K>
keyDecoder, Decoder<V> valueDecoder);

  /**
   *  Create a list of message streams of type T for each topic, using the
default decoder.
   */
  public Map<String, List<KafkaStream<byte[], byte[]>>>
createMessageStreams(Map<String, Integer> topicCountMap);


essentially the byte[], byte[] version is a default, kind of a back door ,
in which case the generics declaration of the consumer type chain is
useless indeed.



On Thu, Feb 5, 2015 at 12:18 PM, Joel Koshy <jj...@gmail.com> wrote:

> There has to be an implicit contract between the producer and
> consumer. The K, V pairs don't _need_ to match but generally _should_.
> If producer sends with <PK, PV> the consumer may receive <CK, CV> as
> long as it knows how to convert those raw bytes to <CK, CV>. In the
> example if CK == byte[] and CV == byte[] it is effectively a no-op
> conversion.
>
> On Thu, Feb 05, 2015 at 12:11:03PM -0800, Yang wrote:
> > in the example
> > https://cwiki.apache.org/confluence/display/KAFKA/0.8.0+Producer+Example
> >
> > we use a String,String for <K,V>
> >
> > in the same set of example
> > https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Group+Example
> >
> > on the consumer side we use byte[], byte[] for K,V
> >         ConsumerIterator<byte[], byte[]> it = m_stream.iterator();
> >
> >
> >
> > we tested in our own code that the above producer-consumer pair does
> work.
> >
> >
> > so the K,V types on the 2 sides do NOT need to match? if we argue that
> > "everything has to come down to byte[], then maybe we don't need the
> > generics types on the consumer side? in other words, how do we determine
> > the K,V types on consumer side?
> >
> > thanks
> > Yang
>
>

Re: generics type for Producer and Consumer do not need to match?

[Joel Koshy <jj...@gmail.com>]

There has to be an implicit contract between the producer and
consumer. The K, V pairs don't _need_ to match but generally _should_.
If producer sends with <PK, PV> the consumer may receive <CK, CV> as
long as it knows how to convert those raw bytes to <CK, CV>. In the
example if CK == byte[] and CV == byte[] it is effectively a no-op
conversion.

On Thu, Feb 05, 2015 at 12:11:03PM -0800, Yang wrote:
> in the example
> https://cwiki.apache.org/confluence/display/KAFKA/0.8.0+Producer+Example
> 
> we use a String,String for <K,V>
> 
> in the same set of example
> https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Group+Example
> 
> on the consumer side we use byte[], byte[] for K,V
>         ConsumerIterator<byte[], byte[]> it = m_stream.iterator();
> 
> 
> 
> we tested in our own code that the above producer-consumer pair does work.
> 
> 
> so the K,V types on the 2 sides do NOT need to match? if we argue that
> "everything has to come down to byte[], then maybe we don't need the
> generics types on the consumer side? in other words, how do we determine
> the K,V types on consumer side?
> 
> thanks
> Yang