You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Fawze Abujaber <fa...@gmail.com> on 2018/08/08 04:56:41 UTC
Unable to see completed application in Spark 2 history web UI
Hello Community,
I'm using Spark 2.3 and Spark 1.6.0 in my cluster with Cloudera
distribution 5.13.0.
Both are configured to run on Yarn, but i'm unable to see completed
application in Spark2 history server, while in Spark 1.6.0 i did.
1) I checked the HDFS permissions for both folders and both have the same
permissions.
drwxrwxrwt - cloudera-scm spark 0 2018-08-08 00:46
/user/spark/applicationHistory
drwxrwxrwt - cloudera-scm spark 0 2018-08-08 00:46
/user/spark/spark2ApplicationHistory
The applications file itself running with permissions 770 in both.
-rwxrwx--- 3 fawzea spark 4743751 2018-08-07 23:32
/user/spark/spark2ApplicationHistory/application_1527404701551_672816_1
-rwxrwx--- 3 fawzea spark 134315 2018-08-08 00:41
/user/spark/applicationHistory/application_1527404701551_673359_1
2) No error in the Spark2 history server log.
3) Compared the configurations between Spark 1.6 and Spark 2.3 like system
user, enable log, etc ... all looks the same.
4) Once i changed the permissions for the above Spark2 applications to 777,
i was able to see the application in the spark2 history server UI.
Tried to figure out if the 2 Sparks UIs running with different users but
was unable to find it.
Anyone who ran into this issue and solved it?
Thanks in advance.
--
Take Care
Fawze Abujaber
Re: Unable to see completed application in Spark 2 history web UI
Posted by Fawze Abujaber <fa...@gmail.com>.
Thanks Manu for your response.
I already checked the logs and didn't see anything that can help me
understanding the issue.
The more weird thing, i have a small CI cluster which run on single
NameNode and i see the Spark2 job in the UI, i'm still not sure if it may
related to the NameNode HA, i tried to replace the logdir from NameNode HA
to the activeNameNode like this
http://server:8020/user/spark/spark2historyapplication in the spark2
default conf but the UI still showing the the path with the HA NameNode
event after a restart of Spark2.
The issue become more intersting :)
On Fri, Aug 17, 2018 at 2:01 AM Manu Zhang <ow...@gmail.com> wrote:
> Hi Fawze,
>
> Sorry but I'm not familiar with CM. Maybe you can look into the logs (or
> turn on DEBUG log).
>
> On Thu, Aug 16, 2018 at 3:05 PM Fawze Abujaber <fa...@gmail.com> wrote:
>
>> Hi Manu,
>>
>> I'm using cloudera manager with single user mode and every process is
>> running with cloudera-scm user, the cloudera-scm is a super user and this
>> is why i was confused how it worked in spark 1.6 and not in spark 2.3
>>
>>
>> On Thu, Aug 16, 2018 at 5:34 AM Manu Zhang <ow...@gmail.com>
>> wrote:
>>
>>> If you are able to log onto the node where UI has been launched, then
>>> try `ps -aux | grep HistoryServer` and the first column of output should be
>>> the user.
>>>
>>> On Wed, Aug 15, 2018 at 10:26 PM Fawze Abujaber <fa...@gmail.com>
>>> wrote:
>>>
>>>> Thanks Manu, Do you know how i can see which user the UI is running,
>>>> because i'm using cloudera manager and i created a user for cloudera
>>>> manager and called it spark but this didn't solve me issue and here i'm
>>>> trying to find out the user for the spark hisotry UI.
>>>>
>>>> On Wed, Aug 15, 2018 at 5:11 PM Manu Zhang <ow...@gmail.com>
>>>> wrote:
>>>>
>>>>> Hi Fawze,
>>>>>
>>>>> A) The file permission is currently hard coded to 770 (
>>>>> https://github.com/apache/spark/blob/branch-2.3/core/src/main/scala/org/apache/spark/scheduler/EventLoggingListener.scala#L287
>>>>> ).
>>>>> B) I think add all users (including UI) to the group like Spark will
>>>>> do.
>>>>>
>>>>>
>>>>> On Wed, Aug 15, 2018 at 6:38 PM Fawze Abujaber <fa...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hi Manu,
>>>>>>
>>>>>> Thanks for your response.
>>>>>>
>>>>>> Yes, i see but still interesting to know how i can see these
>>>>>> applications from the spark history UI.
>>>>>>
>>>>>> How i can know with which user i'm logged in when i'm navigating the
>>>>>> spark history UI.
>>>>>>
>>>>>> The Spark process is running with cloudera-scm and the events written
>>>>>> in the spark2history folder at the HDFS written with the user name who is
>>>>>> running the application and group spark (770 permissions).
>>>>>>
>>>>>> I'm interesting to see if i can force these logs to be written with
>>>>>> 774 or 775 permission or finding another solutions that enable Rnd or
>>>>>> anyone to be able to investigate his application logs using the UI.
>>>>>>
>>>>>> for example : can i use such spark conf :
>>>>>> spark.eventLog.permissions=755
>>>>>>
>>>>>> The 2 options i see here:
>>>>>>
>>>>>> A) find a way to enforce these logs to be written with other
>>>>>> permissions.
>>>>>>
>>>>>> B) Find the user that the UI running with as creating LDAP groups and
>>>>>> user that can handle this.
>>>>>>
>>>>>> for example creating group called Spark and create the user that the
>>>>>> UI running with and add this user to the spark group.
>>>>>> not sure if this option will work as i don't know if these steps
>>>>>> authenticate against the LDAP.
>>>>>>
>>>>>
>>>>
>>>> --
>>>> Take Care
>>>> Fawze Abujaber
>>>>
>>>
>>
>> --
>> Take Care
>> Fawze Abujaber
>>
>
--
Take Care
Fawze Abujaber
Re: Unable to see completed application in Spark 2 history web UI
Posted by Manu Zhang <ow...@gmail.com>.
Hi Fawze,
Sorry but I'm not familiar with CM. Maybe you can look into the logs (or
turn on DEBUG log).
On Thu, Aug 16, 2018 at 3:05 PM Fawze Abujaber <fa...@gmail.com> wrote:
> Hi Manu,
>
> I'm using cloudera manager with single user mode and every process is
> running with cloudera-scm user, the cloudera-scm is a super user and this
> is why i was confused how it worked in spark 1.6 and not in spark 2.3
>
>
> On Thu, Aug 16, 2018 at 5:34 AM Manu Zhang <ow...@gmail.com>
> wrote:
>
>> If you are able to log onto the node where UI has been launched, then try
>> `ps -aux | grep HistoryServer` and the first column of output should be the
>> user.
>>
>> On Wed, Aug 15, 2018 at 10:26 PM Fawze Abujaber <fa...@gmail.com>
>> wrote:
>>
>>> Thanks Manu, Do you know how i can see which user the UI is running,
>>> because i'm using cloudera manager and i created a user for cloudera
>>> manager and called it spark but this didn't solve me issue and here i'm
>>> trying to find out the user for the spark hisotry UI.
>>>
>>> On Wed, Aug 15, 2018 at 5:11 PM Manu Zhang <ow...@gmail.com>
>>> wrote:
>>>
>>>> Hi Fawze,
>>>>
>>>> A) The file permission is currently hard coded to 770 (
>>>> https://github.com/apache/spark/blob/branch-2.3/core/src/main/scala/org/apache/spark/scheduler/EventLoggingListener.scala#L287
>>>> ).
>>>> B) I think add all users (including UI) to the group like Spark will do.
>>>>
>>>>
>>>> On Wed, Aug 15, 2018 at 6:38 PM Fawze Abujaber <fa...@gmail.com>
>>>> wrote:
>>>>
>>>>> Hi Manu,
>>>>>
>>>>> Thanks for your response.
>>>>>
>>>>> Yes, i see but still interesting to know how i can see these
>>>>> applications from the spark history UI.
>>>>>
>>>>> How i can know with which user i'm logged in when i'm navigating the
>>>>> spark history UI.
>>>>>
>>>>> The Spark process is running with cloudera-scm and the events written
>>>>> in the spark2history folder at the HDFS written with the user name who is
>>>>> running the application and group spark (770 permissions).
>>>>>
>>>>> I'm interesting to see if i can force these logs to be written with
>>>>> 774 or 775 permission or finding another solutions that enable Rnd or
>>>>> anyone to be able to investigate his application logs using the UI.
>>>>>
>>>>> for example : can i use such spark conf :
>>>>> spark.eventLog.permissions=755
>>>>>
>>>>> The 2 options i see here:
>>>>>
>>>>> A) find a way to enforce these logs to be written with other
>>>>> permissions.
>>>>>
>>>>> B) Find the user that the UI running with as creating LDAP groups and
>>>>> user that can handle this.
>>>>>
>>>>> for example creating group called Spark and create the user that the
>>>>> UI running with and add this user to the spark group.
>>>>> not sure if this option will work as i don't know if these steps
>>>>> authenticate against the LDAP.
>>>>>
>>>>
>>>
>>> --
>>> Take Care
>>> Fawze Abujaber
>>>
>>
>
> --
> Take Care
> Fawze Abujaber
>
Re: Unable to see completed application in Spark 2 history web UI
Posted by Fawze Abujaber <fa...@gmail.com>.
Hi Manu,
I'm using cloudera manager with single user mode and every process is
running with cloudera-scm user, the cloudera-scm is a super user and this
is why i was confused how it worked in spark 1.6 and not in spark 2.3
On Thu, Aug 16, 2018 at 5:34 AM Manu Zhang <ow...@gmail.com> wrote:
> If you are able to log onto the node where UI has been launched, then try
> `ps -aux | grep HistoryServer` and the first column of output should be the
> user.
>
> On Wed, Aug 15, 2018 at 10:26 PM Fawze Abujaber <fa...@gmail.com> wrote:
>
>> Thanks Manu, Do you know how i can see which user the UI is running,
>> because i'm using cloudera manager and i created a user for cloudera
>> manager and called it spark but this didn't solve me issue and here i'm
>> trying to find out the user for the spark hisotry UI.
>>
>> On Wed, Aug 15, 2018 at 5:11 PM Manu Zhang <ow...@gmail.com>
>> wrote:
>>
>>> Hi Fawze,
>>>
>>> A) The file permission is currently hard coded to 770 (
>>> https://github.com/apache/spark/blob/branch-2.3/core/src/main/scala/org/apache/spark/scheduler/EventLoggingListener.scala#L287
>>> ).
>>> B) I think add all users (including UI) to the group like Spark will do.
>>>
>>>
>>> On Wed, Aug 15, 2018 at 6:38 PM Fawze Abujaber <fa...@gmail.com>
>>> wrote:
>>>
>>>> Hi Manu,
>>>>
>>>> Thanks for your response.
>>>>
>>>> Yes, i see but still interesting to know how i can see these
>>>> applications from the spark history UI.
>>>>
>>>> How i can know with which user i'm logged in when i'm navigating the
>>>> spark history UI.
>>>>
>>>> The Spark process is running with cloudera-scm and the events written
>>>> in the spark2history folder at the HDFS written with the user name who is
>>>> running the application and group spark (770 permissions).
>>>>
>>>> I'm interesting to see if i can force these logs to be written with 774
>>>> or 775 permission or finding another solutions that enable Rnd or anyone to
>>>> be able to investigate his application logs using the UI.
>>>>
>>>> for example : can i use such spark conf :
>>>> spark.eventLog.permissions=755
>>>>
>>>> The 2 options i see here:
>>>>
>>>> A) find a way to enforce these logs to be written with other
>>>> permissions.
>>>>
>>>> B) Find the user that the UI running with as creating LDAP groups and
>>>> user that can handle this.
>>>>
>>>> for example creating group called Spark and create the user that the UI
>>>> running with and add this user to the spark group.
>>>> not sure if this option will work as i don't know if these steps
>>>> authenticate against the LDAP.
>>>>
>>>
>>
>> --
>> Take Care
>> Fawze Abujaber
>>
>
--
Take Care
Fawze Abujaber
Re: Unable to see completed application in Spark 2 history web UI
Posted by Manu Zhang <ow...@gmail.com>.
If you are able to log onto the node where UI has been launched, then try
`ps -aux | grep HistoryServer` and the first column of output should be the
user.
On Wed, Aug 15, 2018 at 10:26 PM Fawze Abujaber <fa...@gmail.com> wrote:
> Thanks Manu, Do you know how i can see which user the UI is running,
> because i'm using cloudera manager and i created a user for cloudera
> manager and called it spark but this didn't solve me issue and here i'm
> trying to find out the user for the spark hisotry UI.
>
> On Wed, Aug 15, 2018 at 5:11 PM Manu Zhang <ow...@gmail.com>
> wrote:
>
>> Hi Fawze,
>>
>> A) The file permission is currently hard coded to 770 (
>> https://github.com/apache/spark/blob/branch-2.3/core/src/main/scala/org/apache/spark/scheduler/EventLoggingListener.scala#L287
>> ).
>> B) I think add all users (including UI) to the group like Spark will do.
>>
>>
>> On Wed, Aug 15, 2018 at 6:38 PM Fawze Abujaber <fa...@gmail.com> wrote:
>>
>>> Hi Manu,
>>>
>>> Thanks for your response.
>>>
>>> Yes, i see but still interesting to know how i can see these
>>> applications from the spark history UI.
>>>
>>> How i can know with which user i'm logged in when i'm navigating the
>>> spark history UI.
>>>
>>> The Spark process is running with cloudera-scm and the events written in
>>> the spark2history folder at the HDFS written with the user name who is
>>> running the application and group spark (770 permissions).
>>>
>>> I'm interesting to see if i can force these logs to be written with 774
>>> or 775 permission or finding another solutions that enable Rnd or anyone to
>>> be able to investigate his application logs using the UI.
>>>
>>> for example : can i use such spark conf : spark.eventLog.permissions=755
>>>
>>> The 2 options i see here:
>>>
>>> A) find a way to enforce these logs to be written with other permissions.
>>>
>>> B) Find the user that the UI running with as creating LDAP groups and
>>> user that can handle this.
>>>
>>> for example creating group called Spark and create the user that the UI
>>> running with and add this user to the spark group.
>>> not sure if this option will work as i don't know if these steps
>>> authenticate against the LDAP.
>>>
>>
>
> --
> Take Care
> Fawze Abujaber
>
Re: Unable to see completed application in Spark 2 history web UI
Posted by Fawze Abujaber <fa...@gmail.com>.
Thanks Manu, Do you know how i can see which user the UI is running,
because i'm using cloudera manager and i created a user for cloudera
manager and called it spark but this didn't solve me issue and here i'm
trying to find out the user for the spark hisotry UI.
On Wed, Aug 15, 2018 at 5:11 PM Manu Zhang <ow...@gmail.com> wrote:
> Hi Fawze,
>
> A) The file permission is currently hard coded to 770 (
> https://github.com/apache/spark/blob/branch-2.3/core/src/main/scala/org/apache/spark/scheduler/EventLoggingListener.scala#L287
> ).
> B) I think add all users (including UI) to the group like Spark will do.
>
>
> On Wed, Aug 15, 2018 at 6:38 PM Fawze Abujaber <fa...@gmail.com> wrote:
>
>> Hi Manu,
>>
>> Thanks for your response.
>>
>> Yes, i see but still interesting to know how i can see these applications
>> from the spark history UI.
>>
>> How i can know with which user i'm logged in when i'm navigating the
>> spark history UI.
>>
>> The Spark process is running with cloudera-scm and the events written in
>> the spark2history folder at the HDFS written with the user name who is
>> running the application and group spark (770 permissions).
>>
>> I'm interesting to see if i can force these logs to be written with 774
>> or 775 permission or finding another solutions that enable Rnd or anyone to
>> be able to investigate his application logs using the UI.
>>
>> for example : can i use such spark conf : spark.eventLog.permissions=755
>>
>> The 2 options i see here:
>>
>> A) find a way to enforce these logs to be written with other permissions.
>>
>> B) Find the user that the UI running with as creating LDAP groups and
>> user that can handle this.
>>
>> for example creating group called Spark and create the user that the UI
>> running with and add this user to the spark group.
>> not sure if this option will work as i don't know if these steps
>> authenticate against the LDAP.
>>
>
--
Take Care
Fawze Abujaber
Re: Unable to see completed application in Spark 2 history web UI
Posted by Manu Zhang <ow...@gmail.com>.
Hi Fawze,
A) The file permission is currently hard coded to 770 (
https://github.com/apache/spark/blob/branch-2.3/core/src/main/scala/org/apache/spark/scheduler/EventLoggingListener.scala#L287
).
B) I think add all users (including UI) to the group like Spark will do.
On Wed, Aug 15, 2018 at 6:38 PM Fawze Abujaber <fa...@gmail.com> wrote:
> Hi Manu,
>
> Thanks for your response.
>
> Yes, i see but still interesting to know how i can see these applications
> from the spark history UI.
>
> How i can know with which user i'm logged in when i'm navigating the
> spark history UI.
>
> The Spark process is running with cloudera-scm and the events written in
> the spark2history folder at the HDFS written with the user name who is
> running the application and group spark (770 permissions).
>
> I'm interesting to see if i can force these logs to be written with 774 or
> 775 permission or finding another solutions that enable Rnd or anyone to be
> able to investigate his application logs using the UI.
>
> for example : can i use such spark conf : spark.eventLog.permissions=755
>
> The 2 options i see here:
>
> A) find a way to enforce these logs to be written with other permissions.
>
> B) Find the user that the UI running with as creating LDAP groups and user
> that can handle this.
>
> for example creating group called Spark and create the user that the UI
> running with and add this user to the spark group.
> not sure if this option will work as i don't know if these steps
> authenticate against the LDAP.
>
Re: Unable to see completed application in Spark 2 history web UI
Posted by Fawze Abujaber <fa...@gmail.com>.
Hi Manu,
Thanks for your response.
Yes, i see but still interesting to know how i can see these applications
from the spark history UI.
How i can know with which user i'm logged in when i'm navigating the spark
history UI.
The Spark process is running with cloudera-scm and the events written in
the spark2history folder at the HDFS written with the user name who is
running the application and group spark (770 permissions).
I'm interesting to see if i can force these logs to be written with 774 or
775 permission or finding another solutions that enable Rnd or anyone to be
able to investigate his application logs using the UI.
for example : can i use such spark conf : spark.eventLog.permissions=755
The 2 options i see here:
A) find a way to enforce these logs to be written with other permissions.
B) Find the user that the UI running with as creating LDAP groups and user
that can handle this.
for example creating group called Spark and create the user that the UI
running with and add this user to the spark group.
not sure if this option will work as i don't know if these steps
authenticate against the LDAP.
Re: Unable to see completed application in Spark 2 history web UI
Posted by Manu Zhang <ow...@gmail.com>.
Hi Fawze,
In Spark 2.3, HistoryServer will check for file permissions when reading
event logs written by your applications. (Please check
https://issues.apache.org/jira/browse/SPARK-20172). With file permissions
of 770, HistoryServer is not permitted to read the event log. That's why
you were able to see applications once changing file permissions to 777.
Regards,
Manu Zhang
On Mon, Aug 13, 2018 at 4:53 PM Fawze Abujaber <fa...@gmail.com> wrote:
> Hi Guys,
>
> Any help here?
>
> On Wed, Aug 8, 2018 at 7:56 AM Fawze Abujaber <fa...@gmail.com> wrote:
>
>> Hello Community,
>>
>> I'm using Spark 2.3 and Spark 1.6.0 in my cluster with Cloudera
>> distribution 5.13.0.
>>
>> Both are configured to run on Yarn, but i'm unable to see completed
>> application in Spark2 history server, while in Spark 1.6.0 i did.
>>
>> 1) I checked the HDFS permissions for both folders and both have the same
>> permissions.
>>
>> drwxrwxrwt - cloudera-scm spark 0 2018-08-08 00:46
>> /user/spark/applicationHistory
>> drwxrwxrwt - cloudera-scm spark 0 2018-08-08 00:46
>> /user/spark/spark2ApplicationHistory
>>
>> The applications file itself running with permissions 770 in both.
>>
>> -rwxrwx--- 3 fawzea spark 4743751 2018-08-07 23:32
>> /user/spark/spark2ApplicationHistory/application_1527404701551_672816_1
>> -rwxrwx--- 3 fawzea spark 134315 2018-08-08 00:41
>> /user/spark/applicationHistory/application_1527404701551_673359_1
>>
>> 2) No error in the Spark2 history server log.
>>
>> 3) Compared the configurations between Spark 1.6 and Spark 2.3 like
>> system user, enable log, etc ... all looks the same.
>>
>> 4) Once i changed the permissions for the above Spark2 applications to
>> 777, i was able to see the application in the spark2 history server UI.
>>
>> Tried to figure out if the 2 Sparks UIs running with different users but
>> was unable to find it.
>>
>> Anyone who ran into this issue and solved it?
>>
>> Thanks in advance.
>>
>>
>> --
>> Take Care
>> Fawze Abujaber
>>
>
>
> --
> Take Care
> Fawze Abujaber
>
Re: Unable to see completed application in Spark 2 history web UI
Posted by Fawze Abujaber <fa...@gmail.com>.
Hi Guys,
Any help here?
On Wed, Aug 8, 2018 at 7:56 AM Fawze Abujaber <fa...@gmail.com> wrote:
> Hello Community,
>
> I'm using Spark 2.3 and Spark 1.6.0 in my cluster with Cloudera
> distribution 5.13.0.
>
> Both are configured to run on Yarn, but i'm unable to see completed
> application in Spark2 history server, while in Spark 1.6.0 i did.
>
> 1) I checked the HDFS permissions for both folders and both have the same
> permissions.
>
> drwxrwxrwt - cloudera-scm spark 0 2018-08-08 00:46
> /user/spark/applicationHistory
> drwxrwxrwt - cloudera-scm spark 0 2018-08-08 00:46
> /user/spark/spark2ApplicationHistory
>
> The applications file itself running with permissions 770 in both.
>
> -rwxrwx--- 3 fawzea spark 4743751 2018-08-07 23:32
> /user/spark/spark2ApplicationHistory/application_1527404701551_672816_1
> -rwxrwx--- 3 fawzea spark 134315 2018-08-08 00:41
> /user/spark/applicationHistory/application_1527404701551_673359_1
>
> 2) No error in the Spark2 history server log.
>
> 3) Compared the configurations between Spark 1.6 and Spark 2.3 like system
> user, enable log, etc ... all looks the same.
>
> 4) Once i changed the permissions for the above Spark2 applications to
> 777, i was able to see the application in the spark2 history server UI.
>
> Tried to figure out if the 2 Sparks UIs running with different users but
> was unable to find it.
>
> Anyone who ran into this issue and solved it?
>
> Thanks in advance.
>
>
> --
> Take Care
> Fawze Abujaber
>
--
Take Care
Fawze Abujaber