You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@beam.apache.org by "Ly, Kiet" <Ki...@finra.org> on 2016/04/12 15:20:51 UTC
Hive Runner
I didn't see Hive runner in Beam. Is there a plan for Hive runner component?
Confidentiality Notice:: This email, including attachments, may include non-public, proprietary, confidential or legally privileged information. If you are not an intended recipient or an authorized agent of an intended recipient, you are hereby notified that any dissemination, distribution or copying of the information contained in or transmitted with this e-mail is unauthorized and strictly prohibited. If you have received this email in error, please notify the sender by replying to this message and permanently delete this e-mail, its attachments, and any copies of it immediately. You should not retain, copy or use this e-mail or any attachment for any purpose, nor disclose all or any part of the contents to any other person. Thank you.
Re: Hive Runner
Posted by Zahoor Mohamed J <za...@zahoor.in>.
Great... Please share the details and I am open to help
./Zahoor@iPhone
> On 12-Apr-2016, at 10:23 PM, Jean-Baptiste Onofré <jb...@nanthrax.net> wrote:
>
> Hi,
>
> yes, I started the MapReduce runner. I can share where I am with you.
>
> Regards
> JB
>
>> On 04/12/2016 06:19 PM, Zahoor Mohamed J wrote:
>> That brings me to MapReduce runner... Any one working on that... Iam interested in helping and learning here.. Any pointers to start looking at .... To design a MapReduce runner?
>>
>> ./Zahoor@iPhone
>>
>>> On 12-Apr-2016, at 7:42 PM, Jean-Baptiste Onofré <jb...@nanthrax.net> wrote:
>>>
>>> We can imagine to translate some Fn (DoPar) as Hive/SQL statements. I don't think it's super interesting, but why not.
>>>
>>> On the other hand, definitely, we will provide an IO for Hive.
>>>
>>> Regards
>>> JB
>>>
>>>> On 04/12/2016 04:04 PM, Aljoscha Krettek wrote:
>>>> Hi,
>>>> what do you mean by Hive Runner? AFAIK Hive provides an SQL like interface
>>>> to data while execution is handled either by a MapReduce backend or the
>>>> newer Tez backend. Therefore I don't think it makes sense to put Beam on
>>>> Hive.
>>>>
>>>> Cheers,
>>>> Aljoscha
>>>>
>>>>> On Tue, 12 Apr 2016 at 15:38 Jean-Baptiste Onofré <jb...@nanthrax.net> wrote:
>>>>>
>>>>> Hi,
>>>>>
>>>>> you are right: for now, we have runners for spark, flink, google cloud
>>>>> platform.
>>>>>
>>>>> Some work are in progress to provide MapReduce, Gearpump runners. And
>>>>> we're also preparing a Runner API to simplify the way of writing runners.
>>>>>
>>>>> On the other hand, we will improve the website to provide a better
>>>>> visibility on the Beam support (current and coming runners, IOs,
>>>>> SDKs/DSLs).
>>>>>
>>>>> If you are interested to work on a Hive runner, please let me know, we
>>>>> love contribution !
>>>>>
>>>>> Thanks,
>>>>> Regards
>>>>> JB
>>>>>
>>>>>> On 04/12/2016 03:20 PM, Ly, Kiet wrote:
>>>>>> I didn't see Hive runner in Beam. Is there a plan for Hive runner
>>>>> component?
>>>>>>
>>>>>> Confidentiality Notice:: This email, including attachments, may include
>>>>> non-public, proprietary, confidential or legally privileged information.
>>>>> If you are not an intended recipient or an authorized agent of an intended
>>>>> recipient, you are hereby notified that any dissemination, distribution or
>>>>> copying of the information contained in or transmitted with this e-mail is
>>>>> unauthorized and strictly prohibited. If you have received this email in
>>>>> error, please notify the sender by replying to this message and permanently
>>>>> delete this e-mail, its attachments, and any copies of it immediately. You
>>>>> should not retain, copy or use this e-mail or any attachment for any
>>>>> purpose, nor disclose all or any part of the contents to any other person.
>>>>> Thank you.
>>>>>
>>>>> --
>>>>> Jean-Baptiste Onofré
>>>>> jbonofre@apache.org
>>>>> http://blog.nanthrax.net
>>>>> Talend - http://www.talend.com
>>>
>>> --
>>> Jean-Baptiste Onofré
>>> jbonofre@apache.org
>>> http://blog.nanthrax.net
>>> Talend - http://www.talend.com
>
> --
> Jean-Baptiste Onofré
> jbonofre@apache.org
> http://blog.nanthrax.net
> Talend - http://www.talend.com
Re: Hive Runner
Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
Hi,
yes, I started the MapReduce runner. I can share where I am with you.
Regards
JB
On 04/12/2016 06:19 PM, Zahoor Mohamed J wrote:
> That brings me to MapReduce runner... Any one working on that... Iam interested in helping and learning here.. Any pointers to start looking at .... To design a MapReduce runner?
>
> ./Zahoor@iPhone
>
>> On 12-Apr-2016, at 7:42 PM, Jean-Baptiste Onofré <jb...@nanthrax.net> wrote:
>>
>> We can imagine to translate some Fn (DoPar) as Hive/SQL statements. I don't think it's super interesting, but why not.
>>
>> On the other hand, definitely, we will provide an IO for Hive.
>>
>> Regards
>> JB
>>
>>> On 04/12/2016 04:04 PM, Aljoscha Krettek wrote:
>>> Hi,
>>> what do you mean by Hive Runner? AFAIK Hive provides an SQL like interface
>>> to data while execution is handled either by a MapReduce backend or the
>>> newer Tez backend. Therefore I don't think it makes sense to put Beam on
>>> Hive.
>>>
>>> Cheers,
>>> Aljoscha
>>>
>>>> On Tue, 12 Apr 2016 at 15:38 Jean-Baptiste Onofré <jb...@nanthrax.net> wrote:
>>>>
>>>> Hi,
>>>>
>>>> you are right: for now, we have runners for spark, flink, google cloud
>>>> platform.
>>>>
>>>> Some work are in progress to provide MapReduce, Gearpump runners. And
>>>> we're also preparing a Runner API to simplify the way of writing runners.
>>>>
>>>> On the other hand, we will improve the website to provide a better
>>>> visibility on the Beam support (current and coming runners, IOs,
>>>> SDKs/DSLs).
>>>>
>>>> If you are interested to work on a Hive runner, please let me know, we
>>>> love contribution !
>>>>
>>>> Thanks,
>>>> Regards
>>>> JB
>>>>
>>>>> On 04/12/2016 03:20 PM, Ly, Kiet wrote:
>>>>> I didn't see Hive runner in Beam. Is there a plan for Hive runner
>>>> component?
>>>>>
>>>>> Confidentiality Notice:: This email, including attachments, may include
>>>> non-public, proprietary, confidential or legally privileged information.
>>>> If you are not an intended recipient or an authorized agent of an intended
>>>> recipient, you are hereby notified that any dissemination, distribution or
>>>> copying of the information contained in or transmitted with this e-mail is
>>>> unauthorized and strictly prohibited. If you have received this email in
>>>> error, please notify the sender by replying to this message and permanently
>>>> delete this e-mail, its attachments, and any copies of it immediately. You
>>>> should not retain, copy or use this e-mail or any attachment for any
>>>> purpose, nor disclose all or any part of the contents to any other person.
>>>> Thank you.
>>>>
>>>> --
>>>> Jean-Baptiste Onofré
>>>> jbonofre@apache.org
>>>> http://blog.nanthrax.net
>>>> Talend - http://www.talend.com
>>
>> --
>> Jean-Baptiste Onofré
>> jbonofre@apache.org
>> http://blog.nanthrax.net
>> Talend - http://www.talend.com
--
Jean-Baptiste Onofré
jbonofre@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com
Re: Hive Runner
Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
Actually, I started to work on it ;)
Regards
JB
On 04/12/2016 06:43 PM, James Malone wrote:
> I do not believe anyone is actively working on a MR runner but we have a
> JIRA open to track interest and possibly track the development should work
> begin.
>
> https://issues.apache.org/jira/browse/BEAM-165
>
> On Tue, Apr 12, 2016 at 9:19 AM, Zahoor Mohamed J <za...@zahoor.in> wrote:
>
>> That brings me to MapReduce runner... Any one working on that... Iam
>> interested in helping and learning here.. Any pointers to start looking at
>> .... To design a MapReduce runner?
>>
>> ./Zahoor@iPhone
>>
>>> On 12-Apr-2016, at 7:42 PM, Jean-Baptiste Onofré <jb...@nanthrax.net>
>> wrote:
>>>
>>> We can imagine to translate some Fn (DoPar) as Hive/SQL statements. I
>> don't think it's super interesting, but why not.
>>>
>>> On the other hand, definitely, we will provide an IO for Hive.
>>>
>>> Regards
>>> JB
>>>
>>>> On 04/12/2016 04:04 PM, Aljoscha Krettek wrote:
>>>> Hi,
>>>> what do you mean by Hive Runner? AFAIK Hive provides an SQL like
>> interface
>>>> to data while execution is handled either by a MapReduce backend or the
>>>> newer Tez backend. Therefore I don't think it makes sense to put Beam on
>>>> Hive.
>>>>
>>>> Cheers,
>>>> Aljoscha
>>>>
>>>>> On Tue, 12 Apr 2016 at 15:38 Jean-Baptiste Onofré <jb...@nanthrax.net>
>> wrote:
>>>>>
>>>>> Hi,
>>>>>
>>>>> you are right: for now, we have runners for spark, flink, google cloud
>>>>> platform.
>>>>>
>>>>> Some work are in progress to provide MapReduce, Gearpump runners. And
>>>>> we're also preparing a Runner API to simplify the way of writing
>> runners.
>>>>>
>>>>> On the other hand, we will improve the website to provide a better
>>>>> visibility on the Beam support (current and coming runners, IOs,
>>>>> SDKs/DSLs).
>>>>>
>>>>> If you are interested to work on a Hive runner, please let me know, we
>>>>> love contribution !
>>>>>
>>>>> Thanks,
>>>>> Regards
>>>>> JB
>>>>>
>>>>>> On 04/12/2016 03:20 PM, Ly, Kiet wrote:
>>>>>> I didn't see Hive runner in Beam. Is there a plan for Hive runner
>>>>> component?
>>>>>>
>>>>>> Confidentiality Notice:: This email, including attachments, may
>> include
>>>>> non-public, proprietary, confidential or legally privileged
>> information.
>>>>> If you are not an intended recipient or an authorized agent of an
>> intended
>>>>> recipient, you are hereby notified that any dissemination,
>> distribution or
>>>>> copying of the information contained in or transmitted with this
>> e-mail is
>>>>> unauthorized and strictly prohibited. If you have received this email
>> in
>>>>> error, please notify the sender by replying to this message and
>> permanently
>>>>> delete this e-mail, its attachments, and any copies of it
>> immediately. You
>>>>> should not retain, copy or use this e-mail or any attachment for any
>>>>> purpose, nor disclose all or any part of the contents to any other
>> person.
>>>>> Thank you.
>>>>>
>>>>> --
>>>>> Jean-Baptiste Onofré
>>>>> jbonofre@apache.org
>>>>> http://blog.nanthrax.net
>>>>> Talend - http://www.talend.com
>>>
>>> --
>>> Jean-Baptiste Onofré
>>> jbonofre@apache.org
>>> http://blog.nanthrax.net
>>> Talend - http://www.talend.com
>>
>
--
Jean-Baptiste Onofré
jbonofre@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com
Re: Hive Runner
Posted by James Malone <ja...@google.com.INVALID>.
I do not believe anyone is actively working on a MR runner but we have a
JIRA open to track interest and possibly track the development should work
begin.
https://issues.apache.org/jira/browse/BEAM-165
On Tue, Apr 12, 2016 at 9:19 AM, Zahoor Mohamed J <za...@zahoor.in> wrote:
> That brings me to MapReduce runner... Any one working on that... Iam
> interested in helping and learning here.. Any pointers to start looking at
> .... To design a MapReduce runner?
>
> ./Zahoor@iPhone
>
> > On 12-Apr-2016, at 7:42 PM, Jean-Baptiste Onofré <jb...@nanthrax.net>
> wrote:
> >
> > We can imagine to translate some Fn (DoPar) as Hive/SQL statements. I
> don't think it's super interesting, but why not.
> >
> > On the other hand, definitely, we will provide an IO for Hive.
> >
> > Regards
> > JB
> >
> >> On 04/12/2016 04:04 PM, Aljoscha Krettek wrote:
> >> Hi,
> >> what do you mean by Hive Runner? AFAIK Hive provides an SQL like
> interface
> >> to data while execution is handled either by a MapReduce backend or the
> >> newer Tez backend. Therefore I don't think it makes sense to put Beam on
> >> Hive.
> >>
> >> Cheers,
> >> Aljoscha
> >>
> >>> On Tue, 12 Apr 2016 at 15:38 Jean-Baptiste Onofré <jb...@nanthrax.net>
> wrote:
> >>>
> >>> Hi,
> >>>
> >>> you are right: for now, we have runners for spark, flink, google cloud
> >>> platform.
> >>>
> >>> Some work are in progress to provide MapReduce, Gearpump runners. And
> >>> we're also preparing a Runner API to simplify the way of writing
> runners.
> >>>
> >>> On the other hand, we will improve the website to provide a better
> >>> visibility on the Beam support (current and coming runners, IOs,
> >>> SDKs/DSLs).
> >>>
> >>> If you are interested to work on a Hive runner, please let me know, we
> >>> love contribution !
> >>>
> >>> Thanks,
> >>> Regards
> >>> JB
> >>>
> >>>> On 04/12/2016 03:20 PM, Ly, Kiet wrote:
> >>>> I didn't see Hive runner in Beam. Is there a plan for Hive runner
> >>> component?
> >>>>
> >>>> Confidentiality Notice:: This email, including attachments, may
> include
> >>> non-public, proprietary, confidential or legally privileged
> information.
> >>> If you are not an intended recipient or an authorized agent of an
> intended
> >>> recipient, you are hereby notified that any dissemination,
> distribution or
> >>> copying of the information contained in or transmitted with this
> e-mail is
> >>> unauthorized and strictly prohibited. If you have received this email
> in
> >>> error, please notify the sender by replying to this message and
> permanently
> >>> delete this e-mail, its attachments, and any copies of it
> immediately. You
> >>> should not retain, copy or use this e-mail or any attachment for any
> >>> purpose, nor disclose all or any part of the contents to any other
> person.
> >>> Thank you.
> >>>
> >>> --
> >>> Jean-Baptiste Onofré
> >>> jbonofre@apache.org
> >>> http://blog.nanthrax.net
> >>> Talend - http://www.talend.com
> >
> > --
> > Jean-Baptiste Onofré
> > jbonofre@apache.org
> > http://blog.nanthrax.net
> > Talend - http://www.talend.com
>
Re: Hive Runner
Posted by Zahoor Mohamed J <za...@zahoor.in>.
That brings me to MapReduce runner... Any one working on that... Iam interested in helping and learning here.. Any pointers to start looking at .... To design a MapReduce runner?
./Zahoor@iPhone
> On 12-Apr-2016, at 7:42 PM, Jean-Baptiste Onofré <jb...@nanthrax.net> wrote:
>
> We can imagine to translate some Fn (DoPar) as Hive/SQL statements. I don't think it's super interesting, but why not.
>
> On the other hand, definitely, we will provide an IO for Hive.
>
> Regards
> JB
>
>> On 04/12/2016 04:04 PM, Aljoscha Krettek wrote:
>> Hi,
>> what do you mean by Hive Runner? AFAIK Hive provides an SQL like interface
>> to data while execution is handled either by a MapReduce backend or the
>> newer Tez backend. Therefore I don't think it makes sense to put Beam on
>> Hive.
>>
>> Cheers,
>> Aljoscha
>>
>>> On Tue, 12 Apr 2016 at 15:38 Jean-Baptiste Onofré <jb...@nanthrax.net> wrote:
>>>
>>> Hi,
>>>
>>> you are right: for now, we have runners for spark, flink, google cloud
>>> platform.
>>>
>>> Some work are in progress to provide MapReduce, Gearpump runners. And
>>> we're also preparing a Runner API to simplify the way of writing runners.
>>>
>>> On the other hand, we will improve the website to provide a better
>>> visibility on the Beam support (current and coming runners, IOs,
>>> SDKs/DSLs).
>>>
>>> If you are interested to work on a Hive runner, please let me know, we
>>> love contribution !
>>>
>>> Thanks,
>>> Regards
>>> JB
>>>
>>>> On 04/12/2016 03:20 PM, Ly, Kiet wrote:
>>>> I didn't see Hive runner in Beam. Is there a plan for Hive runner
>>> component?
>>>>
>>>> Confidentiality Notice:: This email, including attachments, may include
>>> non-public, proprietary, confidential or legally privileged information.
>>> If you are not an intended recipient or an authorized agent of an intended
>>> recipient, you are hereby notified that any dissemination, distribution or
>>> copying of the information contained in or transmitted with this e-mail is
>>> unauthorized and strictly prohibited. If you have received this email in
>>> error, please notify the sender by replying to this message and permanently
>>> delete this e-mail, its attachments, and any copies of it immediately. You
>>> should not retain, copy or use this e-mail or any attachment for any
>>> purpose, nor disclose all or any part of the contents to any other person.
>>> Thank you.
>>>
>>> --
>>> Jean-Baptiste Onofré
>>> jbonofre@apache.org
>>> http://blog.nanthrax.net
>>> Talend - http://www.talend.com
>
> --
> Jean-Baptiste Onofré
> jbonofre@apache.org
> http://blog.nanthrax.net
> Talend - http://www.talend.com
Re: Hive Runner
Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
We can imagine to translate some Fn (DoPar) as Hive/SQL statements. I
don't think it's super interesting, but why not.
On the other hand, definitely, we will provide an IO for Hive.
Regards
JB
On 04/12/2016 04:04 PM, Aljoscha Krettek wrote:
> Hi,
> what do you mean by Hive Runner? AFAIK Hive provides an SQL like interface
> to data while execution is handled either by a MapReduce backend or the
> newer Tez backend. Therefore I don't think it makes sense to put Beam on
> Hive.
>
> Cheers,
> Aljoscha
>
> On Tue, 12 Apr 2016 at 15:38 Jean-Baptiste Onofré <jb...@nanthrax.net> wrote:
>
>> Hi,
>>
>> you are right: for now, we have runners for spark, flink, google cloud
>> platform.
>>
>> Some work are in progress to provide MapReduce, Gearpump runners. And
>> we're also preparing a Runner API to simplify the way of writing runners.
>>
>> On the other hand, we will improve the website to provide a better
>> visibility on the Beam support (current and coming runners, IOs,
>> SDKs/DSLs).
>>
>> If you are interested to work on a Hive runner, please let me know, we
>> love contribution !
>>
>> Thanks,
>> Regards
>> JB
>>
>> On 04/12/2016 03:20 PM, Ly, Kiet wrote:
>>> I didn't see Hive runner in Beam. Is there a plan for Hive runner
>> component?
>>>
>>> Confidentiality Notice:: This email, including attachments, may include
>> non-public, proprietary, confidential or legally privileged information.
>> If you are not an intended recipient or an authorized agent of an intended
>> recipient, you are hereby notified that any dissemination, distribution or
>> copying of the information contained in or transmitted with this e-mail is
>> unauthorized and strictly prohibited. If you have received this email in
>> error, please notify the sender by replying to this message and permanently
>> delete this e-mail, its attachments, and any copies of it immediately. You
>> should not retain, copy or use this e-mail or any attachment for any
>> purpose, nor disclose all or any part of the contents to any other person.
>> Thank you.
>>>
>>
>> --
>> Jean-Baptiste Onofré
>> jbonofre@apache.org
>> http://blog.nanthrax.net
>> Talend - http://www.talend.com
>>
>
--
Jean-Baptiste Onofré
jbonofre@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com
Re: Hive Runner
Posted by Aljoscha Krettek <al...@apache.org>.
Hi,
what do you mean by Hive Runner? AFAIK Hive provides an SQL like interface
to data while execution is handled either by a MapReduce backend or the
newer Tez backend. Therefore I don't think it makes sense to put Beam on
Hive.
Cheers,
Aljoscha
On Tue, 12 Apr 2016 at 15:38 Jean-Baptiste Onofré <jb...@nanthrax.net> wrote:
> Hi,
>
> you are right: for now, we have runners for spark, flink, google cloud
> platform.
>
> Some work are in progress to provide MapReduce, Gearpump runners. And
> we're also preparing a Runner API to simplify the way of writing runners.
>
> On the other hand, we will improve the website to provide a better
> visibility on the Beam support (current and coming runners, IOs,
> SDKs/DSLs).
>
> If you are interested to work on a Hive runner, please let me know, we
> love contribution !
>
> Thanks,
> Regards
> JB
>
> On 04/12/2016 03:20 PM, Ly, Kiet wrote:
> > I didn't see Hive runner in Beam. Is there a plan for Hive runner
> component?
> >
> > Confidentiality Notice:: This email, including attachments, may include
> non-public, proprietary, confidential or legally privileged information.
> If you are not an intended recipient or an authorized agent of an intended
> recipient, you are hereby notified that any dissemination, distribution or
> copying of the information contained in or transmitted with this e-mail is
> unauthorized and strictly prohibited. If you have received this email in
> error, please notify the sender by replying to this message and permanently
> delete this e-mail, its attachments, and any copies of it immediately. You
> should not retain, copy or use this e-mail or any attachment for any
> purpose, nor disclose all or any part of the contents to any other person.
> Thank you.
> >
>
> --
> Jean-Baptiste Onofré
> jbonofre@apache.org
> http://blog.nanthrax.net
> Talend - http://www.talend.com
>
Re: Hive Runner
Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
Hi,
you are right: for now, we have runners for spark, flink, google cloud
platform.
Some work are in progress to provide MapReduce, Gearpump runners. And
we're also preparing a Runner API to simplify the way of writing runners.
On the other hand, we will improve the website to provide a better
visibility on the Beam support (current and coming runners, IOs, SDKs/DSLs).
If you are interested to work on a Hive runner, please let me know, we
love contribution !
Thanks,
Regards
JB
On 04/12/2016 03:20 PM, Ly, Kiet wrote:
> I didn't see Hive runner in Beam. Is there a plan for Hive runner component?
>
> Confidentiality Notice:: This email, including attachments, may include non-public, proprietary, confidential or legally privileged information. If you are not an intended recipient or an authorized agent of an intended recipient, you are hereby notified that any dissemination, distribution or copying of the information contained in or transmitted with this e-mail is unauthorized and strictly prohibited. If you have received this email in error, please notify the sender by replying to this message and permanently delete this e-mail, its attachments, and any copies of it immediately. You should not retain, copy or use this e-mail or any attachment for any purpose, nor disclose all or any part of the contents to any other person. Thank you.
>
--
Jean-Baptiste Onofré
jbonofre@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com