You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@predictionio.apache.org by 王斌斌 <he...@163.com> on 2018/05/10 03:12:04 UTC
deploy predictionio with spark standalone cluster
https://stackoverflow.com/questions/50256449/deploy-predictionio-with-spark-standalone-cluster
I use the official Recommendation as a test. I did these steps successfully:
event server installed in a docker container.(successfully)
config eventdata, metadata and all things are stored in mysql.(successfully)
train & deploy server in another docker container.(successfully)
spark standalone cluster.(successfully)
create new app.(successfully)
import enough eventdata.(successfully)
When I train and deploy as follows, it's ok as the docs described :
pio train
pio deploy
But when I use spark cluster, train and deploy as follows, train is ok(new model has been stored in mysql), but deploy isn't success.
pio train -v engine.json -- --master spark://predictionspark:7077 --executor-memory 2G --driver-memory 2G --total-executor-cores 1
pio deploy -v engine.json --feedback --event-server-ip predictionevent --event-server-port 7070 --accesskey Th7k5gE5yEu9ZdTdM6KdAj0InDrLNJQ1U3qEBy7dbMnYgTxWx5ALNAa2hKjqaHSK -- --master spark://predictionspark:7077 --executor-memory 2G --driver-memory 2G --total-executor-cores 1
deploy ERROR log:
...
flb_flb_1 | 2018-05-09T09:56:20.410043835Z [INFO] [Engine] Using persisted model
flb_flb_1 | 2018-05-09T09:56:20.411705255Z [INFO] [Engine] Custom-persisted model detected for algorithm org.example.recommendation.ALSAlgorithm
flb_flb_1 | 2018-05-09T09:56:21.263570490Z [ERROR] [OneForOneStrategy] empty collection
I don't know why.
Re:Re:Re:deploy predictionio with spark standalone cluster
Posted by 王斌斌 <he...@163.com>.
It's interesting. I find the model datas at the spark-worker container.
I am a beginner with spark. I guess the pio uses the spark results on HDFS?（I don't config the HDFS）
If I install the spark standalone cluster in local container, the results will be write on local /tmp, so the deploying can get the datas. If I install the spark on remote container, it wouldn't. It's all a beginner's guess.
At 2018-05-11 11:17:51, "王斌斌" <he...@163.com> wrote:
a standalone cluster installed on another docker container：
a standalone cluster installed local：
At 2018-05-10 12:19:09, "王斌斌" <he...@163.com> wrote:
Instead of using a standalone cluster installed on another docker container, I start a cluster with the spark on the same container with train&deploy server, it's successfully.
I don't understand why it happened. I can't use a local spark, it's very strange.
At 2018-05-10 11:12:04, "王斌斌" <he...@163.com> wrote:
https://stackoverflow.com/questions/50256449/deploy-predictionio-with-spark-standalone-cluster
I use the official Recommendation as a test. I did these steps successfully:
event server installed in a docker container.(successfully)
config eventdata, metadata and all things are stored in mysql.(successfully)
train & deploy server in another docker container.(successfully)
spark standalone cluster.(successfully)
create new app.(successfully)
import enough eventdata.(successfully)
When I train and deploy as follows, it's ok as the docs described :
pio train
pio deploy
But when I use spark cluster, train and deploy as follows, train is ok(new model has been stored in mysql), but deploy isn't success.
pio train -v engine.json -- --master spark://predictionspark:7077 --executor-memory 2G --driver-memory 2G --total-executor-cores 1
pio deploy -v engine.json --feedback --event-server-ip predictionevent --event-server-port 7070 --accesskey Th7k5gE5yEu9ZdTdM6KdAj0InDrLNJQ1U3qEBy7dbMnYgTxWx5ALNAa2hKjqaHSK -- --master spark://predictionspark:7077 --executor-memory 2G --driver-memory 2G --total-executor-cores 1
deploy ERROR log:
...
flb_flb_1 | 2018-05-09T09:56:20.410043835Z [INFO] [Engine] Using persisted model
flb_flb_1 | 2018-05-09T09:56:20.411705255Z [INFO] [Engine] Custom-persisted model detected for algorithm org.example.recommendation.ALSAlgorithm
flb_flb_1 | 2018-05-09T09:56:21.263570490Z [ERROR] [OneForOneStrategy] empty collection
I don't know why.
Re:Re:deploy predictionio with spark standalone cluster
Posted by 王斌斌 <he...@163.com>.
a standalone cluster installed on another docker container：
a standalone cluster installed local：
At 2018-05-10 12:19:09, "王斌斌" <he...@163.com> wrote:
Instead of using a standalone cluster installed on another docker container, I start a cluster with the spark on the same container with train&deploy server, it's successfully.
I don't understand why it happened. I can't use a local spark, it's very strange.
At 2018-05-10 11:12:04, "王斌斌" <he...@163.com> wrote:
https://stackoverflow.com/questions/50256449/deploy-predictionio-with-spark-standalone-cluster
I use the official Recommendation as a test. I did these steps successfully:
event server installed in a docker container.(successfully)
config eventdata, metadata and all things are stored in mysql.(successfully)
train & deploy server in another docker container.(successfully)
spark standalone cluster.(successfully)
create new app.(successfully)
import enough eventdata.(successfully)
When I train and deploy as follows, it's ok as the docs described :
pio train
pio deploy
But when I use spark cluster, train and deploy as follows, train is ok(new model has been stored in mysql), but deploy isn't success.
pio train -v engine.json -- --master spark://predictionspark:7077 --executor-memory 2G --driver-memory 2G --total-executor-cores 1
pio deploy -v engine.json --feedback --event-server-ip predictionevent --event-server-port 7070 --accesskey Th7k5gE5yEu9ZdTdM6KdAj0InDrLNJQ1U3qEBy7dbMnYgTxWx5ALNAa2hKjqaHSK -- --master spark://predictionspark:7077 --executor-memory 2G --driver-memory 2G --total-executor-cores 1
deploy ERROR log:
...
flb_flb_1 | 2018-05-09T09:56:20.410043835Z [INFO] [Engine] Using persisted model
flb_flb_1 | 2018-05-09T09:56:20.411705255Z [INFO] [Engine] Custom-persisted model detected for algorithm org.example.recommendation.ALSAlgorithm
flb_flb_1 | 2018-05-09T09:56:21.263570490Z [ERROR] [OneForOneStrategy] empty collection
I don't know why.
Re:deploy predictionio with spark standalone cluster
Posted by 王斌斌 <he...@163.com>.
Instead of using a standalone cluster installed on another docker container, I start a cluster with the spark on the same container with train&deploy server, it's successfully.
I don't understand why it happened. I can't use a local spark, it's very strange.
At 2018-05-10 11:12:04, "王斌斌" <he...@163.com> wrote:
https://stackoverflow.com/questions/50256449/deploy-predictionio-with-spark-standalone-cluster
I use the official Recommendation as a test. I did these steps successfully:
event server installed in a docker container.(successfully)
config eventdata, metadata and all things are stored in mysql.(successfully)
train & deploy server in another docker container.(successfully)
spark standalone cluster.(successfully)
create new app.(successfully)
import enough eventdata.(successfully)
When I train and deploy as follows, it's ok as the docs described :
pio train
pio deploy
But when I use spark cluster, train and deploy as follows, train is ok(new model has been stored in mysql), but deploy isn't success.
pio train -v engine.json -- --master spark://predictionspark:7077 --executor-memory 2G --driver-memory 2G --total-executor-cores 1
pio deploy -v engine.json --feedback --event-server-ip predictionevent --event-server-port 7070 --accesskey Th7k5gE5yEu9ZdTdM6KdAj0InDrLNJQ1U3qEBy7dbMnYgTxWx5ALNAa2hKjqaHSK -- --master spark://predictionspark:7077 --executor-memory 2G --driver-memory 2G --total-executor-cores 1
deploy ERROR log:
...
flb_flb_1 | 2018-05-09T09:56:20.410043835Z [INFO] [Engine] Using persisted model
flb_flb_1 | 2018-05-09T09:56:20.411705255Z [INFO] [Engine] Custom-persisted model detected for algorithm org.example.recommendation.ALSAlgorithm
flb_flb_1 | 2018-05-09T09:56:21.263570490Z [ERROR] [OneForOneStrategy] empty collection
I don't know why.