You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@tez.apache.org by Bikas Saha <bi...@apache.org> on 2017/05/01 00:46:42 UTC
Re: How to get user input in the form of a DAG in a text file
IIRC a long time back, Rajesh Balamohan had prototyped something like this to provide a DAG definition via a text file (xml) I think.
This would be nice to get done because then it could provide a simple way for higher level apps to generate Tez DAGs for execution without linking against Tez programatically. And also be quite convenient for visual ETL tools.
Bikas
________________________________
From: Abdul Qadeer <qa...@gmail.com>
Sent: Wednesday, April 26, 2017 4:20 PM
To: user@tez.apache.org
Subject: How to get user input in the form of a DAG in a text file
Hi,
For a pipeline workload, I want to use the
Input-process-output based abstraction for the
end user. End user will need to provide his
pipeline DAG as input. One option is to use
Dot language for user input and then map it
to Tez DAG API
(http://www.graphviz.org/content/dot-language).
My question is:
Is there some facility available in Tez to
get DAG based input in text format from
user? Or
if you could suggest me something better
than my current dot based plan?
Thanks,
-Abdul
Re: How to get user input in the form of a DAG in a text file
Posted by Rajesh Balamohan <rb...@apache.org>.
Yes, this was somewhere around Tez 0.4 timeframe. Patch would have to be
rebased.
~Rajesh.B
On Mon, May 1, 2017 at 6:24 AM, Bikas Saha <bi...@apache.org> wrote:
> Found the JIRA - https://issues.apache.org/jira/browse/TEZ-1100
> ------------------------------
> *From:* Bikas Saha <bi...@apache.org>
> *Sent:* Sunday, April 30, 2017 5:46:42 PM
> *To:* user@tez.apache.org
> *Subject:* Re: How to get user input in the form of a DAG in a text file
>
>
> IIRC a long time back, Rajesh Balamohan had prototyped something like this
> to provide a DAG definition via a text file (xml) I think.
>
>
> This would be nice to get done because then it could provide a simple way
> for higher level apps to generate Tez DAGs for execution without linking
> against Tez programatically. And also be quite convenient for visual ETL
> tools.
>
>
> Bikas
>
>
> ------------------------------
> *From:* Abdul Qadeer <qa...@gmail.com>
> *Sent:* Wednesday, April 26, 2017 4:20 PM
> *To:* user@tez.apache.org
> *Subject:* How to get user input in the form of a DAG in a text file
>
> Hi,
>
> For a pipeline workload, I want to use the
> Input-process-output based abstraction for the
> end user. End user will need to provide his
> pipeline DAG as input. One option is to use
> Dot language for user input and then map it
> to Tez DAG API
> (http://www.graphviz.org/content/dot-language).
>
>
> My question is:
>
> Is there some facility available in Tez to
> get DAG based input in text format from
> user? Or
> if you could suggest me something better
> than my current dot based plan?
>
>
> Thanks,
> -Abdul
>
Re: How to get user input in the form of a DAG in a text file
Posted by Bikas Saha <bi...@apache.org>.
Found the JIRA - https://issues.apache.org/jira/browse/TEZ-1100
________________________________
From: Bikas Saha <bi...@apache.org>
Sent: Sunday, April 30, 2017 5:46:42 PM
To: user@tez.apache.org
Subject: Re: How to get user input in the form of a DAG in a text file
IIRC a long time back, Rajesh Balamohan had prototyped something like this to provide a DAG definition via a text file (xml) I think.
This would be nice to get done because then it could provide a simple way for higher level apps to generate Tez DAGs for execution without linking against Tez programatically. And also be quite convenient for visual ETL tools.
Bikas
________________________________
From: Abdul Qadeer <qa...@gmail.com>
Sent: Wednesday, April 26, 2017 4:20 PM
To: user@tez.apache.org
Subject: How to get user input in the form of a DAG in a text file
Hi,
For a pipeline workload, I want to use the
Input-process-output based abstraction for the
end user. End user will need to provide his
pipeline DAG as input. One option is to use
Dot language for user input and then map it
to Tez DAG API
(http://www.graphviz.org/content/dot-language).
My question is:
Is there some facility available in Tez to
get DAG based input in text format from
user? Or
if you could suggest me something better
than my current dot based plan?
Thanks,
-Abdul