[DISCUSS] Exposing user payloads in the Tez UI

[Hitesh Shah <hi...@apache.org>]

Hi folks 

As part of the Tez APIs, every object ( Input / Output / Processor / EdgeManagerPlugin / VertexManagerPlugin ) can be associated with its own user provided payload to set itself up. The format of this payload is not known to Tez as it could be a java serialized object/protobuf/xml, etc. In a lot of cases, this payload is usually a serialized configuration object. To make it easy to expose this configuration aspects in the Tez UI, a setHistoryText API was added to the UserPayload class. 

This history text was meant to be something which could be displayed in the UI. To make this more meaningful, I am proposing that we define a common standard that everyone using this API can conform to. This will allow the UI to make some assumptions about the structure and display the data in a more consumable manner. 

A possible standard could be:
   - history text should be a json blob mapping to a json object containing key-val pairs.
   - the keys that should be used: 
      - description: value should be a simple string
      - config: value should be an object in the format: { “key1”: “value1”, “key2” : “value2”, … }
     
The above is not a final version or the recommended one. Folks like Jonathan and Prakash who are working on TEZ-8 may have stronger recommendations on the json structure. 

@Gunther, @Rohini - any comments from the user point of view for exposing Hive/Pig information in the Tez UI?

Comments/Questions/Suggestions? 

thanks
— HItesh

Re: [DISCUSS] Exposing user payloads in the Tez UI

[Prakash Ramachandran <pr...@hortonworks.com>]

Hi Hitesh,

The json structure looks fine. Does adding the source of the config 
value make it useful? The current userPayloadAsText seems to be adding 
source to the config values.

Thanks
Prakash



On 10/22/14 3:05 AM, Hitesh Shah wrote:
> Hi folks
>
> As part of the Tez APIs, every object ( Input / Output / Processor / EdgeManagerPlugin / VertexManagerPlugin ) can be associated with its own user provided payload to set itself up. The format of this payload is not known to Tez as it could be a java serialized object/protobuf/xml, etc. In a lot of cases, this payload is usually a serialized configuration object. To make it easy to expose this configuration aspects in the Tez UI, a setHistoryText API was added to the UserPayload class.
>
> This history text was meant to be something which could be displayed in the UI. To make this more meaningful, I am proposing that we define a common standard that everyone using this API can conform to. This will allow the UI to make some assumptions about the structure and display the data in a more consumable manner.
>
> A possible standard could be:
>     - history text should be a json blob mapping to a json object containing key-val pairs.
>     - the keys that should be used:
>        - description: value should be a simple string
>        - config: value should be an object in the format: { “key1”: “value1”, “key2” : “value2”, … }
>       
> The above is not a final version or the recommended one. Folks like Jonathan and Prakash who are working on TEZ-8 may have stronger recommendations on the json structure.
>
> @Gunther, @Rohini - any comments from the user point of view for exposing Hive/Pig information in the Tez UI?
>
> Comments/Questions/Suggestions?
>
> thanks
> — HItesh


-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Re: [DISCUSS] Exposing user payloads in the Tez UI

[Prakash Ramachandran <pr...@hortonworks.com>]

Hi Hitesh,

The json structure looks fine. Does adding the source of the config 
value make it useful? The current userPayloadAsText seems to be adding 
source to the config values.

Thanks
Prakash



On 10/22/14 3:05 AM, Hitesh Shah wrote:
> Hi folks
>
> As part of the Tez APIs, every object ( Input / Output / Processor / EdgeManagerPlugin / VertexManagerPlugin ) can be associated with its own user provided payload to set itself up. The format of this payload is not known to Tez as it could be a java serialized object/protobuf/xml, etc. In a lot of cases, this payload is usually a serialized configuration object. To make it easy to expose this configuration aspects in the Tez UI, a setHistoryText API was added to the UserPayload class.
>
> This history text was meant to be something which could be displayed in the UI. To make this more meaningful, I am proposing that we define a common standard that everyone using this API can conform to. This will allow the UI to make some assumptions about the structure and display the data in a more consumable manner.
>
> A possible standard could be:
>     - history text should be a json blob mapping to a json object containing key-val pairs.
>     - the keys that should be used:
>        - description: value should be a simple string
>        - config: value should be an object in the format: { “key1”: “value1”, “key2” : “value2”, … }
>       
> The above is not a final version or the recommended one. Folks like Jonathan and Prakash who are working on TEZ-8 may have stronger recommendations on the json structure.
>
> @Gunther, @Rohini - any comments from the user point of view for exposing Hive/Pig information in the Tez UI?
>
> Comments/Questions/Suggestions?
>
> thanks
> — HItesh


-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.