[jira] [Commented] (PIG-3780) Tez mini cluster tests run for a very long time with TezSession reuse on

["Rohini Palaniswamy (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/PIG-3780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13913341#comment-13913341 ] 

Rohini Palaniswamy commented on PIG-3780:
-----------------------------------------

Some issues with RB and the comments I gave yest wasn't published. Still not able to. So giving it here. 

1) PathFilter getDefaultPathFilter
Just to know, is it the _SUCCESS files that you are trying to filter out? If so can the method be renamed to getSuccessMarkerPathFilter?
Also can we add the method to org.apache.pig.test.Util in test package instead of this class?

2) properties.setProperty(PigConfiguration.TEZ_SESSION_REUSE, "false");
Why don't we set this in MiniCluster m_conf itself. Once we fix the original issue, then it will just be one place that needs to be fixed. 

3) TEZC11.gld
Combiner has gone missing. We need to fix this instead of changing golden file. Can you leave this file as is?

4) Other split related files
These ones be left as is till PIG-3757 is fully done. The filter should not be there and the alias names also have gone missing. We knew we were breaking this. 

> Tez mini cluster tests run for a very long time with TezSession reuse on
> ------------------------------------------------------------------------
>
>                 Key: PIG-3780
>                 URL: https://issues.apache.org/jira/browse/PIG-3780
>             Project: Pig
>          Issue Type: Sub-task
>          Components: tez
>    Affects Versions: tez-branch
>            Reporter: Cheolsoo Park
>            Assignee: Cheolsoo Park
>             Fix For: tez-branch
>
>         Attachments: PIG-3780-1.patch
>
>
> In the current tez branch, mini cluster unit tests are very slow. The reason is as follows:
> * TezSession reuse is by default on.
> * Each test case runs, and it waits for Tez AM to terminate.
> *  After Tez AM times out (usually after several minutes), another test case runs.
> Two questions that I have are:
> # Why doesn't TezSession reuse work in mini cluster?
> # Why is TezSession reuse not disabled in some tests (e.g. TestAccumulator) where we explicitly set "tez.session.reuse" to false?
> As for #2, I realized that "tez.session.reuse" was never set in the properties object that is passed to PigServer. I am going to upload a patch that fixes this problem in this jira.
> As for #1, I don't have an answer yet. But I think we can fix this in a separate jira once we get Tez unit tests working again.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)