Apache Heron Slack Digest

[Apache Heron <he...@joshfischer.io>]

2020-11-29 20:13:37 UTC - Josh Fischer: I think Karthik owns this account.
----
2020-11-30 06:29:27 UTC - Karthik Ramasamy: Pypi repo let me check
----
2020-11-30 06:29:56 UTC - Karthik Ramasamy: I own the account
----
2020-11-30 06:30:10 UTC - Karthik Ramasamy: What do you need?
----
2020-11-30 06:43:50 UTC - W.Wong: we need a good solution to manage how we
should tweet
----
2020-11-30 06:44:07 UTC - W.Wong: even we got the access, we still need to
think what to post on there
----
2020-11-30 07:20:55 UTC - Ulaga Sundar J: Hi Team, I am Ulaga Sundar J,
software engineer in a private security domain company. We are currently
using Apache Heron 0.17.8 version in production in a multi tenant
environment which is working fine. Currently we are focusing for small
clients with their own datacenters and they are expecting with minimum
hardware requirements for 1000EPS and so we are currently performance
tesing the entire product, coming to our design in heron we are using 2
topologies and we run a topology in a separate 4CPU and 16GBRAM AWS machine
and a single topology utilizes all CPU and 6.7GB RAM(PFA.,). When we give
htop command in that box it shows there are many sub process for heron
running in that machine, So currently we are guessing to proceed with 2
solutions to minimize the CPU for upgrade 0.20.1 version and 2.Need to
check whether we can stop any sub process which is not necessary for this
topology(In this topology we are taking data from kafka and there are 3
bolts i)Preprocess bolt ii)Parse bolt iii)Sink bolt again to kafka) Am I
going on correct way? Any suggestions? Please guide us on this. Thanks in
Advance.
----
2020-11-30 11:36:39 UTC - Ulaga Sundar J: Hi team,
----
2020-11-30 11:38:57 UTC - Ulaga Sundar J: In 0.17.8 Builder builder =
Builder.newBuilder(); Streamlet<Map<String, String>> kafkaSource =
builder.newSource(new KafkaSource()); In 0.20.1 it is nt accepting the
source as Kafka
----
2020-11-30 11:46:39 UTC - W.Wong: Are you using Heronpy?
----
2020-11-30 11:49:56 UTC - W.Wong: processes spawned for Heron are those
bolts/spouts
----
2020-11-30 11:50:09 UTC - W.Wong: and also managers and schedulers
----
2020-11-30 11:50:35 UTC - W.Wong: Supporting Kafka or not is based on
whether your codebase support it or not, instead of Heron supporting Kafka
or not.
----
2020-11-30 11:50:49 UTC - W.Wong: If you are using docker, you need to
build your image with librdkafka
----
2020-11-30 12:03:51 UTC - Ulaga Sundar J: no, I am nt using Heronpy
Streamlet<Map<String, String>> kafkaSource = builder.newSource(new
KafkaSource()); kafkaSource
.setNumPartitions(HeronProperties.NUM_CONTAINERS) .transform(new
PreProcessor()) .transform(new Parser()) .toSink(new KafkaSink());
Config.DeliverySemantics deliverySemantics = applyDeliverySemantics();
Config config = Config.newBuilder()
.setNumContainers(HeronProperties.NUM_CONTAINERS)
.setPerContainerCpu(HeronProperties.CPU)
.setDeliverySemantics(deliverySemantics) .build();
config.getHeronConfig().setContainerCpuRequested(HeronProperties.CPU);
----
2020-11-30 12:04:04 UTC - Ulaga Sundar J: using Java Streamlet api
----
2020-11-30 12:10:05 UTC - Josh Fischer: It could be the package change. I
think 0.17.8 is `com.twitter` and releases > 0.17.8 are `org.apache`
----
2020-11-30 12:12:11 UTC - Ulaga Sundar J: yes in addition to that, in
builder also there is some changes
----
2020-11-30 12:12:16 UTC - Ulaga Sundar J: package
org.apache.heron.streamlet; import org.apache.heron.api.spout.IRichSpout;
public interface Builder { public static Builder newBuilder() { //
<editor-fold defaultstate="collapsed" desc="Compiled Code"> /* 0: new #9 //
class org/apache/heron/streamlet/impl/BuilderImpl * 3: dup * 4:
invokespecial #13 // Method
org/apache/heron/streamlet/impl/BuilderImpl."<init>":()V * 7: areturn * */
// </editor-fold> } public <R extends Object> Streamlet<R>
newSource(SerializableSupplier<R> supplier); public <R extends Object>
Streamlet<R> newSource(Source<R> generator); public <R extends Object>
Streamlet<R> newSource(IRichSpout spout);
----
2020-11-30 12:16:34 UTC - Ulaga Sundar J: I cant able to find any
documentation or example to give input as kafka source in builder..any
suggestions?
----
2020-11-30 12:22:37 UTC - Josh Fischer: Can you expand on this? > In 0.20.1
it is nt accepting the source as Kafka
----
2020-11-30 12:47:51 UTC - Ulaga Sundar J:
----
2020-11-30 13:10:55 UTC - Ulaga Sundar J: I found the solution, Need to
implement Source<Map<String, String>> in KafkaSource object... Thanks
+1 : Josh Fischer ----