You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Ashok Kumar <as...@yahoo.com.INVALID> on 2017/06/05 20:45:03 UTC
Edge Node in Spark
Hi,
I am a bit confused between Edge node, Edge server and gateway node in Spark.
Do these mean the same thing?
How does one set up an Edge node to be used in Spark? Is this different from Edge node for Hadoop please?
Thanks
---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org
Re: Edge Node in Spark
Posted by Mich Talebzadeh <mi...@gmail.com>.
Agreed with Ayan.
Essentially an Edge node is a physical host or VM that is used by the
application to run the job. The users or service users start the process
from the Edge node. Edge nodes are added to the cluster for example
DEV/TEST/UAT etc.
Edge node normally has all compatible binaries in this case $KAFKA_HOME
installation to run Spark jobs. These binaries are normally installed
independently (open source) or through parcels say CDH when the cluster
admin adds the node to the cluster.
I have not seen anyone allowed to directly log in to datanodes, namenodes
or kafka nodes (if they are on different Hardware than datanodes etc) to
kijck off sparksubmit. it is all done through edge or network servers. For
example, you may have 8 Spark processes are running on 8 datanodes say
standalone mode one master and multiple worker processes say two on each
datanode. However, no workere process will be running on edge node.
HTH
Dr Mich Talebzadeh
LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
<https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
http://talebzadehmich.wordpress.com
*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.
On 6 June 2017 at 22:42, ayan guha <gu...@gmail.com> wrote:
> They are all same thing. Essentially it means a machine which is not part
> of the cluster but Has all clients.
>
> On Wed, 7 Jun 2017 at 5:48 am, Irving Duran <ir...@gmail.com>
> wrote:
>
>> Where in the documentation did you find "edge node"? Spark would call it
>> worker or executor, but not "edge node". Her is some info about yarn logs
>> -> https://spark.apache.org/docs/latest/running-on-yarn.html.
>>
>>
>> Thank You,
>>
>> Irving Duran
>>
>> On Tue, Jun 6, 2017 at 11:48 AM, Ashok Kumar <as...@yahoo.com>
>> wrote:
>>
>>> Just Straight Spark please.
>>>
>>> Also if I run a spark job using Python or Scala using Yarn where the log
>>> files are kept in the edge node? Are these under logs directory for yarn?
>>>
>>> thanks
>>>
>>>
>>> On Tuesday, 6 June 2017, 14:11, Irving Duran <ir...@gmail.com>
>>> wrote:
>>>
>>>
>>> Ashok,
>>> Are you working with straight spark or referring to GraphX?
>>>
>>>
>>> Thank You,
>>>
>>> Irving Duran
>>>
>>> On Mon, Jun 5, 2017 at 3:45 PM, Ashok Kumar <
>>> ashok34668@yahoo.com.invalid> wrote:
>>>
>>> Hi,
>>>
>>> I am a bit confused between Edge node, Edge server and gateway node in
>>> Spark.
>>>
>>> Do these mean the same thing?
>>>
>>> How does one set up an Edge node to be used in Spark? Is this different
>>> from Edge node for Hadoop please?
>>>
>>> Thanks
>>>
>>> ------------------------------ ------------------------------ ---------
>>> To unsubscribe e-mail: user-unsubscribe@spark.apache. org
>>> <us...@spark.apache.org>
>>>
>>>
>>>
>>>
>>>
>> --
> Best Regards,
> Ayan Guha
>
Re: Edge Node in Spark
Posted by ayan guha <gu...@gmail.com>.
They are all same thing. Essentially it means a machine which is not part
of the cluster but Has all clients.
On Wed, 7 Jun 2017 at 5:48 am, Irving Duran <ir...@gmail.com> wrote:
> Where in the documentation did you find "edge node"? Spark would call it
> worker or executor, but not "edge node". Her is some info about yarn logs
> -> https://spark.apache.org/docs/latest/running-on-yarn.html.
>
>
> Thank You,
>
> Irving Duran
>
> On Tue, Jun 6, 2017 at 11:48 AM, Ashok Kumar <as...@yahoo.com> wrote:
>
>> Just Straight Spark please.
>>
>> Also if I run a spark job using Python or Scala using Yarn where the log
>> files are kept in the edge node? Are these under logs directory for yarn?
>>
>> thanks
>>
>>
>> On Tuesday, 6 June 2017, 14:11, Irving Duran <ir...@gmail.com>
>> wrote:
>>
>>
>> Ashok,
>> Are you working with straight spark or referring to GraphX?
>>
>>
>> Thank You,
>>
>> Irving Duran
>>
>> On Mon, Jun 5, 2017 at 3:45 PM, Ashok Kumar <ashok34668@yahoo.com.invalid
>> > wrote:
>>
>> Hi,
>>
>> I am a bit confused between Edge node, Edge server and gateway node in
>> Spark.
>>
>> Do these mean the same thing?
>>
>> How does one set up an Edge node to be used in Spark? Is this different
>> from Edge node for Hadoop please?
>>
>> Thanks
>>
>> ------------------------------ ------------------------------ ---------
>> To unsubscribe e-mail: user-unsubscribe@spark.apache. org
>> <us...@spark.apache.org>
>>
>>
>>
>>
>>
> --
Best Regards,
Ayan Guha
Re: Edge Node in Spark
Posted by Irving Duran <ir...@gmail.com>.
Where in the documentation did you find "edge node"? Spark would call it
worker or executor, but not "edge node". Her is some info about yarn logs
-> https://spark.apache.org/docs/latest/running-on-yarn.html.
Thank You,
Irving Duran
On Tue, Jun 6, 2017 at 11:48 AM, Ashok Kumar <as...@yahoo.com> wrote:
> Just Straight Spark please.
>
> Also if I run a spark job using Python or Scala using Yarn where the log
> files are kept in the edge node? Are these under logs directory for yarn?
>
> thanks
>
>
> On Tuesday, 6 June 2017, 14:11, Irving Duran <ir...@gmail.com>
> wrote:
>
>
> Ashok,
> Are you working with straight spark or referring to GraphX?
>
>
> Thank You,
>
> Irving Duran
>
> On Mon, Jun 5, 2017 at 3:45 PM, Ashok Kumar <as...@yahoo.com.invalid>
> wrote:
>
> Hi,
>
> I am a bit confused between Edge node, Edge server and gateway node in
> Spark.
>
> Do these mean the same thing?
>
> How does one set up an Edge node to be used in Spark? Is this different
> from Edge node for Hadoop please?
>
> Thanks
>
> ------------------------------ ------------------------------ ---------
> To unsubscribe e-mail: user-unsubscribe@spark.apache. org
> <us...@spark.apache.org>
>
>
>
>
>
Re: Edge Node in Spark
Posted by Ashok Kumar <as...@yahoo.com.INVALID>.
Just Straight Spark please.
Also if I run a spark job using Python or Scala using Yarn where the log files are kept in the edge node? Are these under logs directory for yarn?
thanks
On Tuesday, 6 June 2017, 14:11, Irving Duran <ir...@gmail.com> wrote:
Ashok,Are you working with straight spark or referring to GraphX?
Thank You,
Irving Duran
On Mon, Jun 5, 2017 at 3:45 PM, Ashok Kumar <as...@yahoo.com.invalid> wrote:
Hi,
I am a bit confused between Edge node, Edge server and gateway node in Spark.
Do these mean the same thing?
How does one set up an Edge node to be used in Spark? Is this different from Edge node for Hadoop please?
Thanks
------------------------------ ------------------------------ ---------
To unsubscribe e-mail: user-unsubscribe@spark.apache. org
Re: Edge Node in Spark
Posted by Irving Duran <ir...@gmail.com>.
Ashok,
Are you working with straight spark or referring to GraphX?
Thank You,
Irving Duran
On Mon, Jun 5, 2017 at 3:45 PM, Ashok Kumar <as...@yahoo.com.invalid>
wrote:
> Hi,
>
> I am a bit confused between Edge node, Edge server and gateway node in
> Spark.
>
> Do these mean the same thing?
>
> How does one set up an Edge node to be used in Spark? Is this different
> from Edge node for Hadoop please?
>
> Thanks
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
>
>