[jira] [Commented] (HIVE-17704) Hive QA isn't collecting spark.log anymore

["Sahil Takiar (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HIVE-17704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16195479#comment-16195479 ] 

Sahil Takiar commented on HIVE-17704:
-------------------------------------

Ok had some time to work on this today, here is what I found. The {{spark.log}} file isn't generated locally anymore. I went back to branch-1.1 and couldn't even it generate it locally there either. On branch-1.1, the file will only be generated if Spark picks up {{data/conf/spark/log4j.properties}} as the log4j config file for the Driver and Executors. Hive tries to ensure this happens by copying the config file into {{SPARK_HOME/conf/}}, but as far as I can tell that doesn't always work. I used the {{-Dlog4j.debug}} and found out that Spark will usually pick up the log4j file from some jar on its classpath. Most blog posts online discussing how to configure log4j for Spark require setting {{-Dlog4j.configuration=}} in the {{spark.driver.extraJavaOptions}} config. I tried that locally at it seems to work. I think there are a few other fixes that require this to work for the master branch. For one, Spark is still on log4j1.x, but the properties file we are giving it is for log4j2.x. It would be nice if each executor logged to a different file.

Alternatively, we could just archive everything under {{itests/qtest-spark/target/sparkOnYarn/sparkOnYarn-logDir}} which contains all the container logs. Which is probably the easiest approach

> Hive QA isn't collecting spark.log anymore
> ------------------------------------------
>
>                 Key: HIVE-17704
>                 URL: https://issues.apache.org/jira/browse/HIVE-17704
>             Project: Hive
>          Issue Type: Bug
>          Components: Spark
>            Reporter: Sahil Takiar
>
> Checked a recent Hive QA run, and the {{spark.log}} file isn't being collected by ptest (maybe isn't not even being generated anymore, haven't checked).
> We should re-enable it, its hard to debug Hive QA runs without the {{spark.log}} file.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)