You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@predictionio.apache.org by Bansari Shah <ba...@gmail.com> on 2016/10/04 12:55:58 UTC
Use of common argument scratch-uri and remote deployment
Hi,
Can you please guide me how to use 'scratch-uri' argument in case of
transferring all necessary files to remote location.
And can you please suggest me any way for deploying model on remote
location which is trained and build on other machine.
Please consider it.
Thank you,
Regards,
Bansari
Re: Use of common argument scratch-uri and remote deployment
Posted by Donald Szeto <do...@apache.org>.
Hi Bansari,
All you need to make sure is that wherever you run build, train, and
deploy, all nodes have the same data storage configuration and can access
those storage.
"scratch-uri" helps you properly run any "pio" commands in YARN cluster
mode, and is otherwise unrelated with the above.
Regards,
Donald
On Tue, Oct 4, 2016 at 9:37 PM, Bansari Shah <ba...@gmail.com>
wrote:
> Thank you for your guidance.
>
> We have to build and train on development machine which can be standalone
> or 3 node cluster and deploy on production environment which is completely
> different cluster. In this case does scratch-uri will work or we have to
> follow another process.
>
> Please suggest me.
>
> Thank you
> Regards,
> Bansari
>
> On Wed, Oct 5, 2016 at 1:30 AM, Donald Szeto <do...@apache.org> wrote:
>
>> Hi Bansari,
>>
>> The --scratch-uri switch is only useful with "pio train/deploy" using
>> YARN cluster mode, which is your case. It tells PredictionIO where to copy
>> PredictionIO JARs and engine.json for YARN cluster mode to work properly.
>>
>> 1. Make sure HADOOP_CONF_DIR is set properly in conf/pio-env.sh.
>> 2. Provide an HDFS URL to --scratch-uri. You need to have write access to
>> this location.
>>
>> Regards,
>> Donald
>>
>> On Tue, Oct 4, 2016 at 11:21 AM, Pat Ferrel <pa...@occamsmachete.com>
>> wrote:
>>
>>> No idea about 'scratch-uri’ but once you build a model if you have
>>> specified (in pio-env.sh) that pio use hdfs for the model storage it will
>>> already be available to any machine that has access to hdfs. It somewhat
>>> depends on the template, the Universal Recommender uses Elasticsearch for
>>> model storage so any machine with access to ES will have the model.
>>>
>>>
>>> On Oct 4, 2016, at 10:09 AM, Bansari Shah <ba...@gmail.com>
>>> wrote:
>>>
>>> Hi Donald,
>>>
>>> I am running my spark cluster of 3 node with YARN and spark driver is
>>> within cluster.
>>>
>>> Thanks
>>> Regards,
>>> Bansari
>>>
>>> On Tue, Oct 4, 2016 at 9:59 PM, Donald Szeto <do...@apache.org> wrote:
>>>
>>>> Hi Bansari,
>>>>
>>>> How are you running your Spark cluster? Standalone, YARN, or Mesos? Are
>>>> you running the Spark driver on the client or within the cluster?
>>>>
>>>> Regards,
>>>> Donald
>>>>
>>>> On Tue, Oct 4, 2016 at 5:55 AM, Bansari Shah <ba...@gmail.com>
>>>> wrote:
>>>>
>>>>> Hi,
>>>>> Can you please guide me how to use 'scratch-uri' argument in case of
>>>>> transferring all necessary files to remote location.
>>>>>
>>>>> And can you please suggest me any way for deploying model on remote
>>>>> location which is trained and build on other machine.
>>>>>
>>>>> Please consider it.
>>>>>
>>>>> Thank you,
>>>>>
>>>>> Regards,
>>>>> Bansari
>>>>>
>>>>
>>>>
>>>
>>>
>>
>
Re: Use of common argument scratch-uri and remote deployment
Posted by Bansari Shah <ba...@gmail.com>.
Thank you for your guidance.
We have to build and train on development machine which can be standalone
or 3 node cluster and deploy on production environment which is completely
different cluster. In this case does scratch-uri will work or we have to
follow another process.
Please suggest me.
Thank you
Regards,
Bansari
On Wed, Oct 5, 2016 at 1:30 AM, Donald Szeto <do...@apache.org> wrote:
> Hi Bansari,
>
> The --scratch-uri switch is only useful with "pio train/deploy" using YARN
> cluster mode, which is your case. It tells PredictionIO where to copy
> PredictionIO JARs and engine.json for YARN cluster mode to work properly.
>
> 1. Make sure HADOOP_CONF_DIR is set properly in conf/pio-env.sh.
> 2. Provide an HDFS URL to --scratch-uri. You need to have write access to
> this location.
>
> Regards,
> Donald
>
> On Tue, Oct 4, 2016 at 11:21 AM, Pat Ferrel <pa...@occamsmachete.com> wrote:
>
>> No idea about 'scratch-uri’ but once you build a model if you have
>> specified (in pio-env.sh) that pio use hdfs for the model storage it will
>> already be available to any machine that has access to hdfs. It somewhat
>> depends on the template, the Universal Recommender uses Elasticsearch for
>> model storage so any machine with access to ES will have the model.
>>
>>
>> On Oct 4, 2016, at 10:09 AM, Bansari Shah <ba...@gmail.com>
>> wrote:
>>
>> Hi Donald,
>>
>> I am running my spark cluster of 3 node with YARN and spark driver is
>> within cluster.
>>
>> Thanks
>> Regards,
>> Bansari
>>
>> On Tue, Oct 4, 2016 at 9:59 PM, Donald Szeto <do...@apache.org> wrote:
>>
>>> Hi Bansari,
>>>
>>> How are you running your Spark cluster? Standalone, YARN, or Mesos? Are
>>> you running the Spark driver on the client or within the cluster?
>>>
>>> Regards,
>>> Donald
>>>
>>> On Tue, Oct 4, 2016 at 5:55 AM, Bansari Shah <ba...@gmail.com>
>>> wrote:
>>>
>>>> Hi,
>>>> Can you please guide me how to use 'scratch-uri' argument in case of
>>>> transferring all necessary files to remote location.
>>>>
>>>> And can you please suggest me any way for deploying model on remote
>>>> location which is trained and build on other machine.
>>>>
>>>> Please consider it.
>>>>
>>>> Thank you,
>>>>
>>>> Regards,
>>>> Bansari
>>>>
>>>
>>>
>>
>>
>
Re: Use of common argument scratch-uri and remote deployment
Posted by Donald Szeto <do...@apache.org>.
Hi Bansari,
The --scratch-uri switch is only useful with "pio train/deploy" using YARN
cluster mode, which is your case. It tells PredictionIO where to copy
PredictionIO JARs and engine.json for YARN cluster mode to work properly.
1. Make sure HADOOP_CONF_DIR is set properly in conf/pio-env.sh.
2. Provide an HDFS URL to --scratch-uri. You need to have write access to
this location.
Regards,
Donald
On Tue, Oct 4, 2016 at 11:21 AM, Pat Ferrel <pa...@occamsmachete.com> wrote:
> No idea about 'scratch-uri’ but once you build a model if you have
> specified (in pio-env.sh) that pio use hdfs for the model storage it will
> already be available to any machine that has access to hdfs. It somewhat
> depends on the template, the Universal Recommender uses Elasticsearch for
> model storage so any machine with access to ES will have the model.
>
>
> On Oct 4, 2016, at 10:09 AM, Bansari Shah <ba...@gmail.com> wrote:
>
> Hi Donald,
>
> I am running my spark cluster of 3 node with YARN and spark driver is
> within cluster.
>
> Thanks
> Regards,
> Bansari
>
> On Tue, Oct 4, 2016 at 9:59 PM, Donald Szeto <do...@apache.org> wrote:
>
>> Hi Bansari,
>>
>> How are you running your Spark cluster? Standalone, YARN, or Mesos? Are
>> you running the Spark driver on the client or within the cluster?
>>
>> Regards,
>> Donald
>>
>> On Tue, Oct 4, 2016 at 5:55 AM, Bansari Shah <ba...@gmail.com>
>> wrote:
>>
>>> Hi,
>>> Can you please guide me how to use 'scratch-uri' argument in case of
>>> transferring all necessary files to remote location.
>>>
>>> And can you please suggest me any way for deploying model on remote
>>> location which is trained and build on other machine.
>>>
>>> Please consider it.
>>>
>>> Thank you,
>>>
>>> Regards,
>>> Bansari
>>>
>>
>>
>
>
Re: Use of common argument scratch-uri and remote deployment
Posted by Pat Ferrel <pa...@occamsmachete.com>.
No idea about 'scratch-uri’ but once you build a model if you have specified (in pio-env.sh) that pio use hdfs for the model storage it will already be available to any machine that has access to hdfs. It somewhat depends on the template, the Universal Recommender uses Elasticsearch for model storage so any machine with access to ES will have the model.
On Oct 4, 2016, at 10:09 AM, Bansari Shah <ba...@gmail.com> wrote:
Hi Donald,
I am running my spark cluster of 3 node with YARN and spark driver is within cluster.
Thanks
Regards,
Bansari
On Tue, Oct 4, 2016 at 9:59 PM, Donald Szeto <donald@apache.org <ma...@apache.org>> wrote:
Hi Bansari,
How are you running your Spark cluster? Standalone, YARN, or Mesos? Are you running the Spark driver on the client or within the cluster?
Regards,
Donald
On Tue, Oct 4, 2016 at 5:55 AM, Bansari Shah <bansari.jan93@gmail.com <ma...@gmail.com>> wrote:
Hi,
Can you please guide me how to use 'scratch-uri' argument in case of transferring all necessary files to remote location.
And can you please suggest me any way for deploying model on remote location which is trained and build on other machine.
Please consider it.
Thank you,
Regards,
Bansari
Re: Use of common argument scratch-uri and remote deployment
Posted by Bansari Shah <ba...@gmail.com>.
Hi Donald,
I am running my spark cluster of 3 node with YARN and spark driver is
within cluster.
Thanks
Regards,
Bansari
On Tue, Oct 4, 2016 at 9:59 PM, Donald Szeto <do...@apache.org> wrote:
> Hi Bansari,
>
> How are you running your Spark cluster? Standalone, YARN, or Mesos? Are
> you running the Spark driver on the client or within the cluster?
>
> Regards,
> Donald
>
> On Tue, Oct 4, 2016 at 5:55 AM, Bansari Shah <ba...@gmail.com>
> wrote:
>
>> Hi,
>> Can you please guide me how to use 'scratch-uri' argument in case of
>> transferring all necessary files to remote location.
>>
>> And can you please suggest me any way for deploying model on remote
>> location which is trained and build on other machine.
>>
>> Please consider it.
>>
>> Thank you,
>>
>> Regards,
>> Bansari
>>
>
>
Re: Use of common argument scratch-uri and remote deployment
Posted by Donald Szeto <do...@apache.org>.
Hi Bansari,
How are you running your Spark cluster? Standalone, YARN, or Mesos? Are you
running the Spark driver on the client or within the cluster?
Regards,
Donald
On Tue, Oct 4, 2016 at 5:55 AM, Bansari Shah <ba...@gmail.com>
wrote:
> Hi,
> Can you please guide me how to use 'scratch-uri' argument in case of
> transferring all necessary files to remote location.
>
> And can you please suggest me any way for deploying model on remote
> location which is trained and build on other machine.
>
> Please consider it.
>
> Thank you,
>
> Regards,
> Bansari
>