You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by Subacini Balakrishnan <su...@gmail.com> on 2019/03/19 18:17:19 UTC

LocationStratgies.PreferFixed in Structured Streaming

Hi,

We have a use case where specific kafka partition data needs to be assigned
to specific execuotr node.

In Spark Streaming, this can be achieved using
LocationStrategies.Preferfixed .How do we achieve same in structured
streaming??


*Spark Streaming*

Map<TopicPartition, String> partitionMapToHost = new HashMap<>();

partitionMapToHost.put(new TopicPartition("topic", 0), "node1");

partitionMapToHost.put(new TopicPartition("topic", 1), "node2");

partitionMapToHost.put(new TopicPartition("topic", 2), "node2");

 KafkaUtils.createDirectStream(jssc,LocationStrategies.PreferFixed(partitionMapToHost),
 ConsumerStrategies.Subscribe(topicCollection, kafkaParams));



Regards,
Subacini