You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@apex.apache.org by "Mukkamula, Suryavamshivardhan (CWM-NR)" <su...@rbc.com> on 2017/03/23 13:43:01 UTC
problems viewing the Datatorrent application logs
Hello Team,
One of our data torrent application , we cannot view the logs from web console due to firewall issues(Datatorrent installed by service user 'A' and application is being launched by service user 'B').
Can you please let us know where can I find the path of the application logs (running and history jobs) either on the HDFS or individual nodes ? , we have yarn logs from the Hadoop web console but due to the same fire wall issues we can only view the logs through window server of same domain and it is very painful and time consuming.
While the platform team trying to fix the issue , can you please let us know where can I find the log files on the HDFS or individual nodes.
Regards,
Surya Vamshi
_______________________________________________________________________
If you received this email in error, please advise the sender (by return email or otherwise) immediately. You have consented to receive the attached electronically at the above-noted email address; please retain a copy of this confirmation for future reference.
Si vous recevez ce courriel par erreur, veuillez en aviser l'expéditeur immédiatement, par retour de courriel ou par un autre moyen. Vous avez accepté de recevoir le(s) document(s) ci-joint(s) par voie électronique à l'adresse courriel indiquée ci-dessus; veuillez conserver une copie de cette confirmation pour les fins de reference future.
RE: problems viewing the Datatorrent application logs
Posted by "Mukkamula, Suryavamshivardhan (CWM-NR)" <su...@rbc.com>.
Thank you very much Ram.
Regards,
Surya Vamshi
From: Munagala Ramanath [mailto:ram@datatorrent.com]
Sent: 2017, March, 23 10:31 AM
To: users@apex.apache.org
Subject: Re: problems viewing the Datatorrent application logs
If log file aggregation is enabled, aggregated logs for terminated logs should be available on HDFS and retrievable with:
yarn logs -applicationId <application ID>
Log aggregation is discussed further at: https://hortonworks.com/blog/simplifying-user-logs-management-and-access-in-yarn/
and is controlled by the following configuration parameters:
yarn.log-aggregation-enable
yarn.log-aggregation.retain-check-interval-seconds
yarn.log-aggregation.retain-seconds
yarn.nodemanager.log-aggregation.compression-type
yarn.nodemanager.log-aggregation.roll-monitoring-interval-seconds
For running applications, logs are not on HDFS but rather on the local filesystems of the individual nodes; the location
depends on the following configuration parameters:
yarn.nodemanager.remote-app-log-dir
yarn.log.dir
yarn.nodemanager.log-dirs
Some example locations are discussed in the Beginner's Guide at: http://docs.datatorrent.com/beginner/
in the section entitled: Checking logs
Ram
On Thu, Mar 23, 2017 at 6:43 AM, Mukkamula, Suryavamshivardhan (CWM-NR) <su...@rbc.com>> wrote:
Hello Team,
One of our data torrent application , we cannot view the logs from web console due to firewall issues(Datatorrent installed by service user ‘A’ and application is being launched by service user ‘B’).
Can you please let us know where can I find the path of the application logs (running and history jobs) either on the HDFS or individual nodes ? , we have yarn logs from the Hadoop web console but due to the same fire wall issues we can only view the logs through window server of same domain and it is very painful and time consuming.
While the platform team trying to fix the issue , can you please let us know where can I find the log files on the HDFS or individual nodes.
Regards,
Surya Vamshi
_______________________________________________________________________
If you received this email in error, please advise the sender (by return email or otherwise) immediately. You have consented to receive the attached electronically at the above-noted email address; please retain a copy of this confirmation for future reference.
Si vous recevez ce courriel par erreur, veuillez en aviser l'expéditeur immédiatement, par retour de courriel ou par un autre moyen. Vous avez accepté de recevoir le(s) document(s) ci-joint(s) par voie électronique à l'adresse courriel indiquée ci-dessus; veuillez conserver une copie de cette confirmation pour les fins de reference future.
--
_______________________________________________________
Munagala V. Ramanath
Software Engineer
E: ram@datatorrent.com<ma...@datatorrent.com> | M: (408) 331-5034 | Twitter: @UnknownRam
www.datatorrent.com<http://www.datatorrent.com> | apex.apache.org<http://apex.apache.org>
[https://lh4.googleusercontent.com/gZ-_bA3cXH8UerggUOqWVpDglJsOiKtL3dpoCCWrX0K1YEYQHbJUoR106D3gztJVPaRyfeuQVMLKM3PCDfZo-wzBaliedZdlaL5YSqmZb_Weje_De0qIPsLSMAgzRZiE5lIECrT4]
_______________________________________________________________________
If you received this email in error, please advise the sender (by return email or otherwise) immediately. You have consented to receive the attached electronically at the above-noted email address; please retain a copy of this confirmation for future reference.
Si vous recevez ce courriel par erreur, veuillez en aviser l'expéditeur immédiatement, par retour de courriel ou par un autre moyen. Vous avez accepté de recevoir le(s) document(s) ci-joint(s) par voie électronique à l'adresse courriel indiquée ci-dessus; veuillez conserver une copie de cette confirmation pour les fins de reference future.
Re: problems viewing the Datatorrent application logs
Posted by Munagala Ramanath <ra...@datatorrent.com>.
If log file aggregation is enabled, aggregated logs for terminated logs
should be available on HDFS and retrievable with:
*yarn logs -applicationId **<application ID>*
Log aggregation is discussed further at:
https://hortonworks.com/blog/simplifying-user-logs-management-and-access-in-yarn/
and is controlled by the following configuration parameters:
*yarn.log-aggregation-enable*
*yarn.log-aggregation.retain-check-interval-seconds*
*yarn.log-aggregation.retain-seconds*
*yarn.nodemanager.log-aggregation.compression-type*
*yarn.nodemanager.log-aggregation.roll-monitoring-interval-seconds*
For running applications, logs are not on HDFS but rather on the local
filesystems of the individual nodes; the location
depends on the following configuration parameters:
*yarn.nodemanager.remote-app-log-dir*
*yarn.log.dir*
*yarn.nodemanager.log-dirs*
Some example locations are discussed in the Beginner's Guide at:
http://docs.datatorrent.com/beginner/
in the section entitled: *Checking logs*
Ram
On Thu, Mar 23, 2017 at 6:43 AM, Mukkamula, Suryavamshivardhan (CWM-NR) <
suryavamshivardhan.mukkamula@rbc.com> wrote:
> Hello Team,
>
> One of our data torrent application , we cannot view the logs from web
> console due to firewall issues(Datatorrent installed by service user ‘A’
> and application is being launched by service user ‘B’).
>
> Can you please let us know where can I find the path of the application
> logs (running and history jobs) either on the HDFS or individual nodes ? ,
> we have yarn logs from the Hadoop web console but due to the same fire wall
> issues we can only view the logs through window server of same domain and
> it is very painful and time consuming.
>
> While the platform team trying to fix the issue , can you please let us
> know where can I find the log files on the HDFS or individual nodes.
>
> Regards,
> Surya Vamshi
>
>
>
>
> _______________________________________________________________________
>
> If you received this email in error, please advise the sender (by return
> email or otherwise) immediately. You have consented to receive the attached
> electronically at the above-noted email address; please retain a copy of
> this confirmation for future reference.
>
> Si vous recevez ce courriel par erreur, veuillez en aviser l'expéditeur
> immédiatement, par retour de courriel ou par un autre moyen. Vous avez
> accepté de recevoir le(s) document(s) ci-joint(s) par voie électronique à
> l'adresse courriel indiquée ci-dessus; veuillez conserver une copie de
> cette confirmation pour les fins de reference future.
>
>
--
_______________________________________________________
Munagala V. Ramanath
Software Engineer
E: ram@datatorrent.com | M: (408) 331-5034 | Twitter: @UnknownRam
www.datatorrent.com | apex.apache.org