Enrichment Topology: Split-Join vs Unified

[Stefan Kupstaitis-Dunkler <st...@gmail.com>]

Hi,

what are the key differences of Split-Join and Unified in the enrichment
topology. Which should be used when and why?

Best,
Stefan
-- 
Stefan Kupstaitis-Dunkler
https://datahovel.com/
https://www.meetup.com/Hadoop-User-Group-Vienna/
https://twitter.com/StefanDunkler

Re: Enrichment Topology: Split-Join vs Unified

[Casey Stella <ce...@gmail.com>]

What Nick says is absolutely right, but I want to add just a bit of color
around the architectural differences between split-join vs unified here.

Split-join was the first approach to parallelizing the various enrichment
adapters that we have (e.g. hbase, geo, stellar).  We took a very "stormy"
approach to this (see:
https://groups.google.com/forum/#!topic/storm-user/7Gk34vwUATk).  What we
found, however, was during performance evaluation we had extremely pinched
throughput
with this architecture (see architecture at
https://github.com/apache/metron/tree/master/metron-platform/metron-enrichment#enrichment-architecture).
Specifically, the cost of the network overhead in a split/join topology was
overwhelming us and pinching throughput.

We then moved to the unified topology (architecture:
https://github.com/apache/metron/tree/master/metron-platform/metron-enrichment#unified-enrichment-topology),
which removed the network latency overhead, but did things in a less stormy
way.  Specifically enrichments are done in parallel, but inside of a
threadpool in the enrichment bolt.  This saved us network hops at the
expense of adding a threadpool to storm.  In our tests, we've found this to
be the preferred approach.

Hope this helps add color!

Best,

Casey

On Thu, Jul 26, 2018 at 5:31 AM Stefan Kupstaitis-Dunkler <
stefan.dun@gmail.com> wrote:

> Hi,
>
> what are the key differences of Split-Join and Unified in the enrichment
> topology. Which should be used when and why?
>
> Best,
> Stefan
> --
> Stefan Kupstaitis-Dunkler
> https://datahovel.com/
> https://www.meetup.com/Hadoop-User-Group-Vienna/
> https://twitter.com/StefanDunkler
>

Re: Enrichment Topology: Split-Join vs Unified

[Nick Allen <ni...@nickallen.org>]

The Split-Join topology is the legacy topology.  It exists mainly to
support backwards compatibility.  I would consider it deprecated.

The Unified topology is the new and improved topology.  It is much easier
to tune and boasts much better performance.  Use the Unified topology.

On Thu, Jul 26, 2018 at 5:31 AM, Stefan Kupstaitis-Dunkler <
stefan.dun@gmail.com> wrote:

> Hi,
>
> what are the key differences of Split-Join and Unified in the enrichment
> topology. Which should be used when and why?
>
> Best,
> Stefan
> --
> Stefan Kupstaitis-Dunkler
> https://datahovel.com/
> https://www.meetup.com/Hadoop-User-Group-Vienna/
> https://twitter.com/StefanDunkler
>