You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by Raajay <ra...@gmail.com> on 2015/08/31 22:06:11 UTC
Serializing dags
Hello,
Currently, I am running Hive on Tez. I wish to make some changes to the
DAGs generated by HIve before running on Tez/Yarn.
Which data structure should i serialize ? DAG or DagPlan ?
- Raajay
Re: Serializing dags
Posted by Raajay <ra...@gmail.com>.
I see from the docs that QueryPlan can be serialized to string using the
toThriftJSONString() function.
How do de-serialize it ? Any pointers would be helpful.
Thanks,
Raajay
On Tue, Sep 1, 2015 at 11:26 AM, Raajay <ra...@gmail.com> wrote:
> Hi Canan,
>
> The changes that I am primarily interested are:
>
> a. Altering the parallelism of the DAG
> b. Change task location hints etc..
>
> In general, I want to make these alterations and run the DAGs on tez,
> without having to go through the hive pipeline.
>
> Raajay
>
> On Mon, Aug 31, 2015 at 11:42 PM, canan chen <cc...@gmail.com> wrote:
>
>> Hi Raajay,
>>
>> Regarding how hive build DAG, you can check
>> https://github.com/apache/hive/blob/72f97fc7760134465333983fc40766e9e864e643/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezTask.java
>>
>> Check method TezTask#build
>>
>> Do you mind share what kind of changes you want to make on DAGs ?
>>
>> On Tue, Sep 1, 2015 at 4:06 AM, Raajay <ra...@gmail.com> wrote:
>>
>>> Hello,
>>>
>>> Currently, I am running Hive on Tez. I wish to make some changes to the
>>> DAGs generated by HIve before running on Tez/Yarn.
>>>
>>> Which data structure should i serialize ? DAG or DagPlan ?
>>>
>>> - Raajay
>>>
>>
>>
>
Re: Serializing dags
Posted by Raajay <ra...@gmail.com>.
Hi Canan,
The changes that I am primarily interested are:
a. Altering the parallelism of the DAG
b. Change task location hints etc..
In general, I want to make these alterations and run the DAGs on tez,
without having to go through the hive pipeline.
Raajay
On Mon, Aug 31, 2015 at 11:42 PM, canan chen <cc...@gmail.com> wrote:
> Hi Raajay,
>
> Regarding how hive build DAG, you can check
> https://github.com/apache/hive/blob/72f97fc7760134465333983fc40766e9e864e643/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezTask.java
>
> Check method TezTask#build
>
> Do you mind share what kind of changes you want to make on DAGs ?
>
> On Tue, Sep 1, 2015 at 4:06 AM, Raajay <ra...@gmail.com> wrote:
>
>> Hello,
>>
>> Currently, I am running Hive on Tez. I wish to make some changes to the
>> DAGs generated by HIve before running on Tez/Yarn.
>>
>> Which data structure should i serialize ? DAG or DagPlan ?
>>
>> - Raajay
>>
>
>
Re: Serializing dags
Posted by canan chen <cc...@gmail.com>.
Hi Raajay,
Regarding how hive build DAG, you can check
https://github.com/apache/hive/blob/72f97fc7760134465333983fc40766e9e864e643/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezTask.java
Check method TezTask#build
Do you mind share what kind of changes you want to make on DAGs ?
On Tue, Sep 1, 2015 at 4:06 AM, Raajay <ra...@gmail.com> wrote:
> Hello,
>
> Currently, I am running Hive on Tez. I wish to make some changes to the
> DAGs generated by HIve before running on Tez/Yarn.
>
> Which data structure should i serialize ? DAG or DagPlan ?
>
> - Raajay
>