Re: Predition.io setup with Azure

[Pankil Doshi <fo...@gmail.com>]

Hi Donald,

As per my understanding, Azure doesn't support job submission through YARN.
It doesn't expose all YARN configs (public facing IP). They recommend
"Livy" to submit spark jobs to HDInsight cluster [0]. Any recommendations
around integrating it with Livy ?

Thanks,
Pankil

[0]:
https://blogs.msdn.microsoft.com/azuredatalake/2017/01/06/spark-job-submission-on-hdinsight-101/

On Fri, Apr 28, 2017 at 4:24 PM, Donald Szeto <do...@apache.org> wrote:

> Hi Pankil,
>
> Looks like HDInsight runs Spark on YARN. You would need to point PIO to
> the right Hadoop config files, and specify the correct master URL (yarn)
> when you run "pio train".
>
> Regards,
> Donald
>
> On Fri, Apr 28, 2017 at 12:12 PM Pankil Doshi <fo...@gmail.com> wrote:
>
>> Hi,
>>
>> Has anyone tried running prediction.io with Azure specifically using
>> Azure's HDInsight cluster for spark ?
>>
>> I understand it has capability of pointing to remote spark cluster. But
>> trying to figure out what is involved in making it work on HDInsight
>> cluster? Can we leverage livy server to submit spark job?
>>
>> Any pointers would be really helpful.
>>
>> Thanks,
>> Pankil
>>
>