You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by Joel D <ga...@gmail.com> on 2018/04/16 15:59:42 UTC
Business Rules Engine for Hive
Hi,
Any suggestions on how to implement Business Rules Engine with Hive ETLs?
For spark based Etl jobs, I was exploring Drools but not sure about Hive.
Thanks.
Re: Business Rules Engine for Hive
Posted by Joel D <ga...@gmail.com>.
Hi Pivonka, we are more inclined towards using open source products and
closer integration with Hive since we've most of our ETL in Hive.
Thanks.
On Mon, Apr 16, 2018 at 12:51 PM Al Pivonka <al...@gmail.com> wrote:
> I am not the product owner an have not implemented it yet.
> I would check out
> http://cask.co/products/rules-engine/
>
> On Mon, Apr 16, 2018 at 11:59 AM, Joel D <ga...@gmail.com> wrote:
>
>> Hi,
>>
>> Any suggestions on how to implement Business Rules Engine with Hive ETLs?
>>
>> For spark based Etl jobs, I was exploring Drools but not sure about Hive.
>>
>> Thanks.
>>
>
>
>
> --
> Those who say it can't be done, are usually interrupted by those doing it.
>
Re: Business Rules Engine for Hive
Posted by Al Pivonka <al...@gmail.com>.
I am not the product owner an have not implemented it yet.
I would check out
http://cask.co/products/rules-engine/
On Mon, Apr 16, 2018 at 11:59 AM, Joel D <ga...@gmail.com> wrote:
> Hi,
>
> Any suggestions on how to implement Business Rules Engine with Hive ETLs?
>
> For spark based Etl jobs, I was exploring Drools but not sure about Hive.
>
> Thanks.
>
--
Those who say it can't be done, are usually interrupted by those doing it.
Re: Business Rules Engine for Hive
Posted by Joel D <ga...@gmail.com>.
The business rules we've here are currently embedded in hive code. They
range from basic standardization using case blocks to complex multi-column
validation.
Thanks.
On Mon, Apr 16, 2018 at 5:03 PM Jörn Franke <jo...@gmail.com> wrote:
> The question is what do your rules do? Do you need to maintain a factbase
> or do they just check data quality within certain tables?
>
> On 16. Apr 2018, at 22:28, Joel D <ga...@gmail.com> wrote:
>
> Ok.
>
> Rough ideas:
> To keep the business logic outside code, I was thinking to give a custom
> UI.
>
> Next read from UI data and build UDFs using the rules defined outside the
> UDF.
>
> 1 UDF per data object.
>
> Not sure these are just thoughts.
>
> On Mon, Apr 16, 2018 at 1:40 PM Jörn Franke <jo...@gmail.com> wrote:
>
>> I would not use Drools with Spark, it does not scale to the distributed
>> setting.
>>
>> You could translate the rules to hive queries but this would not be
>> exactly the same thing.
>>
>> > On 16. Apr 2018, at 17:59, Joel D <ga...@gmail.com> wrote:
>> >
>> > Hi,
>> >
>> > Any suggestions on how to implement Business Rules Engine with Hive
>> ETLs?
>> >
>> > For spark based Etl jobs, I was exploring Drools but not sure about
>> Hive.
>> >
>> > Thanks.
>>
>
Re: Business Rules Engine for Hive
Posted by Jörn Franke <jo...@gmail.com>.
The question is what do your rules do? Do you need to maintain a factbase or do they just check data quality within certain tables?
> On 16. Apr 2018, at 22:28, Joel D <ga...@gmail.com> wrote:
>
> Ok.
>
> Rough ideas:
> To keep the business logic outside code, I was thinking to give a custom UI.
>
> Next read from UI data and build UDFs using the rules defined outside the UDF.
>
> 1 UDF per data object.
>
> Not sure these are just thoughts.
>
>> On Mon, Apr 16, 2018 at 1:40 PM Jörn Franke <jo...@gmail.com> wrote:
>> I would not use Drools with Spark, it does not scale to the distributed setting.
>>
>> You could translate the rules to hive queries but this would not be exactly the same thing.
>>
>> > On 16. Apr 2018, at 17:59, Joel D <ga...@gmail.com> wrote:
>> >
>> > Hi,
>> >
>> > Any suggestions on how to implement Business Rules Engine with Hive ETLs?
>> >
>> > For spark based Etl jobs, I was exploring Drools but not sure about Hive.
>> >
>> > Thanks.
Re: Business Rules Engine for Hive
Posted by Joel D <ga...@gmail.com>.
Ok.
Rough ideas:
To keep the business logic outside code, I was thinking to give a custom UI.
Next read from UI data and build UDFs using the rules defined outside the
UDF.
1 UDF per data object.
Not sure these are just thoughts.
On Mon, Apr 16, 2018 at 1:40 PM Jörn Franke <jo...@gmail.com> wrote:
> I would not use Drools with Spark, it does not scale to the distributed
> setting.
>
> You could translate the rules to hive queries but this would not be
> exactly the same thing.
>
> > On 16. Apr 2018, at 17:59, Joel D <ga...@gmail.com> wrote:
> >
> > Hi,
> >
> > Any suggestions on how to implement Business Rules Engine with Hive ETLs?
> >
> > For spark based Etl jobs, I was exploring Drools but not sure about Hive.
> >
> > Thanks.
>
Re: Business Rules Engine for Hive
Posted by Jörn Franke <jo...@gmail.com>.
I would not use Drools with Spark, it does not scale to the distributed setting.
You could translate the rules to hive queries but this would not be exactly the same thing.
> On 16. Apr 2018, at 17:59, Joel D <ga...@gmail.com> wrote:
>
> Hi,
>
> Any suggestions on how to implement Business Rules Engine with Hive ETLs?
>
> For spark based Etl jobs, I was exploring Drools but not sure about Hive.
>
> Thanks.