Where are logs for Spark Kafka Yarn on Cloudera

[Rachana Srivastava <Ra...@markmonitor.com>]

Hello all,

I am trying to test JavaKafkaWordCount on Yarn, to make sure Yarn is working fine I am saving the output to hdfs.  The example works fine in local mode but not on yarn mode.  I cannot see any output logged when I changed the mode to yarn-client or yarn-cluster or cannot find any errors logged.  For my application id I was looking for logs under /var/log/hadoop-yarn/containers (e.g /var/log/hadoop-yarn/containers/application_1439517792099_0010/container_1439517792099_0010_01_000003/stderr) but I cannot find anything useful information.   Is it the only location where application logs are logged.

Also tried setting log output to spark.yarn.app.container.log.dir but got access denied error.

Question:  Do we need to have some special setup to run spark streaming on Yarn?  How do we debug?  Where to find more details to test streaming on Yarn.

Thanks,

Rachana

Re: Where are logs for Spark Kafka Yarn on Cloudera

[Marcelo Vanzin <va...@cloudera.com>]

(-dev@)

Try using the "yarn logs" command to read logs for finished
applications. You can also browse the RM UI to find more information
about the applications you ran.

On Mon, Sep 28, 2015 at 11:37 PM, Rachana Srivastava
<Ra...@markmonitor.com> wrote:
> Hello all,
>
>
>
> I am trying to test JavaKafkaWordCount on Yarn, to make sure Yarn is working
> fine I am saving the output to hdfs.  The example works fine in local mode
> but not on yarn mode.  I cannot see any output logged when I changed the
> mode to yarn-client or yarn-cluster or cannot find any errors logged.  For
> my application id I was looking for logs under
> /var/log/hadoop-yarn/containers (e.g
> /var/log/hadoop-yarn/containers/application_1439517792099_0010/container_1439517792099_0010_01_000003/stderr)
> but I cannot find anything useful information.   Is it the only location
> where application logs are logged.
>
>
>
> Also tried setting log output to spark.yarn.app.container.log.dir but got
> access denied error.
>
>
>
> Question:  Do we need to have some special setup to run spark streaming on
> Yarn?  How do we debug?  Where to find more details to test streaming on
> Yarn.
>
>
>
> Thanks,
>
>
>
> Rachana



-- 
Marcelo

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org