You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by "Alaa Zubaidi (PDF)" <al...@pdf.com> on 2017/06/21 23:50:02 UTC
Using YARN w/o HDFS
Hi,
Can we run Spark on YARN with out installing HDFS?
If yes, where would HADOOP_CONF_DIR point to?
Regards,
--
*This message may contain confidential and privileged information. If it
has been sent to you in error, please reply to advise the sender of the
error and then immediately permanently delete it and all attachments to it
from your systems. If you are not the intended recipient, do not read,
copy, disclose or otherwise use this message or any attachments to it. The
sender disclaims any liability for such unauthorized use. PLEASE NOTE that
all incoming e-mails sent to PDF e-mail accounts will be archived and may
be scanned by us and/or by external service providers to detect and prevent
threats to our systems, investigate illegal or inappropriate behavior,
and/or eliminate unsolicited promotional e-mails (“spam”). If you have any
concerns about this process, please contact us at *
*legal.department@pdf.com* <le...@pdf.com>*.*
Re: Using YARN w/o HDFS
Posted by Chen He <ai...@gmail.com>.
chang your fs.defaultFS to point to local file system and have a try
On Wed, Jun 21, 2017 at 4:50 PM, Alaa Zubaidi (PDF) <al...@pdf.com>
wrote:
> Hi,
>
> Can we run Spark on YARN with out installing HDFS?
> If yes, where would HADOOP_CONF_DIR point to?
>
> Regards,
>
> *This message may contain confidential and privileged information. If it
> has been sent to you in error, please reply to advise the sender of the
> error and then immediately permanently delete it and all attachments to it
> from your systems. If you are not the intended recipient, do not read,
> copy, disclose or otherwise use this message or any attachments to it. The
> sender disclaims any liability for such unauthorized use. PLEASE NOTE that
> all incoming e-mails sent to PDF e-mail accounts will be archived and may
> be scanned by us and/or by external service providers to detect and prevent
> threats to our systems, investigate illegal or inappropriate behavior,
> and/or eliminate unsolicited promotional e-mails (“spam”). If you have any
> concerns about this process, please contact us at *
> *legal.department@pdf.com* <le...@pdf.com>*.*
Re: Using YARN w/o HDFS
Posted by Steve Loughran <st...@hortonworks.com>.
you'll need a filesystem with
* consistency
* accessibility everywhere
* supports a binding through one of the hadoop fs connectors
NFS-style distributed filesystems work with file:// ; things like glusterfs need their own connectors.
you can use azure's wasb:// as a drop in replacement for HDFS in Azure.I think google cloud storage is similar, but haven't played with it. Ask google.
You cannot do the same for S3 except on EMR and Amazon's premium emrfs:// offering, which adds the consistency layer.
On 22 Jun 2017, at 00:50, Alaa Zubaidi (PDF) <al...@pdf.com>> wrote:
Hi,
Can we run Spark on YARN with out installing HDFS?
If yes, where would HADOOP_CONF_DIR point to?
Regards,
This message may contain confidential and privileged information. If it has been sent to you in error, please reply to advise the sender of the error and then immediately permanently delete it and all attachments to it from your systems. If you are not the intended recipient, do not read, copy, disclose or otherwise use this message or any attachments to it. The sender disclaims any liability for such unauthorized use. PLEASE NOTE that all incoming e-mails sent to PDF e-mail accounts will be archived and may be scanned by us and/or by external service providers to detect and prevent threats to our systems, investigate illegal or inappropriate behavior, and/or eliminate unsolicited promotional e-mails (“spam”). If you have any concerns about this process, please contact us at legal.department@pdf.com<ma...@pdf.com>.