You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@predictionio.apache.org by George Yarish <gy...@griddynamics.com> on 2018/04/06 01:43:00 UTC
PIO deploy: old processes still live after redeployment
Hi everyone,
I faced a strange situation recently.
One of our engines has frequently old processes after daily redeployment.
We used to redeploy by crontab rule that looks like:
nohup pio deploy --port 8004 >> ${DEPLOY_LOG_FILE} 2>&1 &
This is output of ps aux command:
[da_mcom_milan@mdc2pr113 ~]$ ps aux | grep 8004
da_mcom+ 7854 0.0 0.0 112660 996 pts/1 S+ 21:34 0:00 grep
--color=auto *8004*
1. da_mcom+ 30932 0.6 0.9 26473632 595932 ? Sl 01:57 7:51 java
-cp /data/PredictionIO-0.11.0-incubating/conf:/data/PredictionIO
-0.11.0-incubating/plugins/*:/data/PredictionIO-0.11.0-incub
ating/lib/spark/*:/data/PredictionIO-0.11.0-incubating/lib/
pio-assembly-0.11.0-incubating.jar:/usr/hdp/2.5.3.
0-37/hadoop/conf:/data/PredictionIO/vendors/hbase/conf
-Dpio.log.dir=/home/da_mcom_milan org.apache.predictionio.tools.console.Console
deploy --port *8004* --pio-home /data/PredictionIO-0.11.0-incubating
2. da_mcom+ 31097 1.2 1.2 17065124 822596 ? Sl 01:57 14:48
/usr/java/default/bin/java -Dhdp.version=2.5.3.0-37 -cp
/data/PredictionIO-0.11.0-incubating/conf/:/usr/hdp/2.5.3.0-
37/hadoop/conf/:/data/PredictionIO/vendors/hbase/conf/:/usr/
hdp/current/spark-client/conf/:/data/PredictionIO/vendors/
spark/lib/spark-assembly-1.6.2.2.5.3.0-37-hadoop2.7.3.2.5.
3.0-37.jar:/data/PredictionIO/vendors/spark/lib/datanucleus-
api-jdo-3.2.6.jar:/data/PredictionIO/vendors/spark/lib/
datanucleus-core-3.2.10.jar:/data/PredictionIO/vendors/
spark/lib/datanucleus-rdbms-3.2.9.jar:/usr/hdp/2.5.3.0-37/
hadoop/conf/:/usr/hdp/current/hadoop-client/lib/aws-java-
sdk-s3-1.10.6.jar:/usr/hdp/current/hadoop-client/lib/aws-
java-sdk-core-1.10.6.jar:/usr/hdp/current/hadoop-client/lib/aws-java-sdk-kms-1.10.6.jar
-Xms1g -Xmx1g -Dpio.log.dir=/home/da_mcom_milan
org.apache.spark.deploy.SparkSubmit --conf spark.driver.extraClassPath=/d
ata/PredictionIO-0.11.0-incubating/conf:/usr/hdp/2.5.3.0-37/
hadoop/conf:/data/PredictionIO/vendors/hbase/conf --conf
spark.driver.extraJavaOptions=-Dpio.log.dir=/home/da_mcom_milan --class
org.apache.predictionio.workflow.CreateServer --jars
file:/data/PredictionIO/some-engine/target/scala-2.10/
universal-recommender_2.10-0.6.0.jar,file:/data/PredictionIO/some-engine
/target/scala-2.10/universal-recommender-assembly-0.6.0-deps.jar,file:/
data/PredictionIO-0.11.0-incubating/lib/spark/pio-data-elast
icsearch1-assembly-0.11.0-incubating.jar,file:/data/Pred
ictionIO-0.11.0-incubating/lib/spark/pio-data-hbase-
assembly-0.11.0-incubating.jar,file:/data/PredictionIO-0.
11.0-incubating/lib/spark/pio-data-hdfs-assembly-0.11.0-incu
bating.jar,file:/data/PredictionIO-0.11.0-incubating/lib/
spark/pio-data-jdbc-assembly-0.11.0-incubating.jar,file:/
data/PredictionIO-0.11.0-incubating/lib/spark/pio-data-local
fs-assembly-0.11.0-incubating.jar --files file:/data/PredictionIO-0.11.0
-incubating/conf/log4j.properties,file:/usr/hdp/2.5.3.0-37/
hadoop/conf/core-site.xml,file:/data/PredictionIO/vendors/hbase/conf/hbase-site.xml
file:/data/PredictionIO-0.11.0-incubating/lib/pio-assembly-0.11.0-incubating.jar
--engineInstanceId AWKSaLQrSe_zVbacBHWI --engine-variant
file:/data/PredictionIO/some-engine/engine.json --ip 0.0.0.0 --port *8004*
--event-server-ip 0.0.0.0 --event-server-port 7070 --json-extractor Both
--env PIO_STORAGE_SOURCES_HBASE_TYPE=hbase,PIO_ENV_LOADED=1,PIO_
STORAGE_REPOSITORIES_METADATA_NAME=pio_meta,PIO_FS_BASEDIR=/
home/da_mcom_milan/.pio_store,PIO_STORAGE_SOURCES_ELASTICSEA
RCH_HOSTS=mdc2vra176,PIO_STORAGE_SOURCES_HBASE_HOME=/
data/PredictionIO/vendors/hbase,PIO_HOME=/data/PredictionIO-0.11.0-
incubating,PIO_FS_ENGINESDIR=/home/da_mcom_milan/.pio_store/
engines,PIO_STORAGE_SOURCES_LOCALFS_PATH=/home/da_mcom_
milan/.pio_store/models,PIO_STORAGE_SOURCES_ELASTICSEARCH_
TYPE=elasticsearch,PIO_STORAGE_REPOSITORIES_METADATA_
SOURCE=ELASTICSEARCH,PIO_STORAGE_REPOSITORIES_MODELDATA_SOURCE=LOCALFS,PIO_
STORAGE_REPOSITORIES_EVENTDATA_NAME=pio_event,PIO_
STORAGE_SOURCES_ELASTICSEARCH_CLUSTERNAME=pros-dev,PIO_FS_
TMPDIR=/home/da_mcom_milan/.pio_store/tmp,PIO_STORAGE_
REPOSITORIES_MODELDATA_NAME=pio_model,PIO_STORAGE_
REPOSITORIES_EVENTDATA_SOURCE=HBASE,PIO_CONF_DIR=/data/PredictionIO-0.11.0-
incubating/conf,PIO_STORAGE_SOURCES_ELASTICSEARCH_PORTS=
9300,PIO_STORAGE_SOURCES_LOCALFS_TYPE=localfs
3. da_mcom+ 32505 0.6 0.8 26473632 581164 ? Sl 02:00 7:54 java
-cp /data/PredictionIO-0.11.0-incubating/conf:/data/PredictionIO
-0.11.0-incubating/plugins/*:/data/PredictionIO-0.11.0-incub
ating/lib/spark/*:/data/PredictionIO-0.11.0-incubating/lib/
pio-assembly-0.11.0-incubating.jar:/usr/hdp/2.5.3.
0-37/hadoop/conf:/data/PredictionIO/vendors/hbase/conf
-Dpio.log.dir=/home/da_mcom_milan org.apache.predictionio.tools.console.Console
deploy --port *8004* --pio-home /data/PredictionIO-0.11.0-incubating
4. da_mcom+ 32671 1.4 1.3 19757488 881448 ? Sl 02:00 16:27
/usr/java/default/bin/java -Dhdp.version=2.5.3.0-37 -cp
/data/PredictionIO-0.11.0-incubating/conf/:/usr/hdp/2.5.3.0-
37/hadoop/conf/:/data/PredictionIO/vendors/hbase/conf/:/usr/
hdp/current/spark-client/conf/:/data/PredictionIO/vendors/
spark/lib/spark-assembly-1.6.2.2.5.3.0-37-hadoop2.7.3.2.5.
3.0-37.jar:/data/PredictionIO/vendors/spark/lib/datanucleus-
api-jdo-3.2.6.jar:/data/PredictionIO/vendors/spark/lib/
datanucleus-core-3.2.10.jar:/data/PredictionIO/vendors/
spark/lib/datanucleus-rdbms-3.2.9.jar:/usr/hdp/2.5.3.0-37/
hadoop/conf/:/usr/hdp/current/hadoop-client/lib/aws-java-
sdk-s3-1.10.6.jar:/usr/hdp/current/hadoop-client/lib/aws-
java-sdk-core-1.10.6.jar:/usr/hdp/current/hadoop-client/lib/aws-java-sdk-kms-1.10.6.jar
-Xms1g -Xmx1g -Dpio.log.dir=/home/da_mcom_milan
org.apache.spark.deploy.SparkSubmit --conf spark.driver.extraClassPath=/d
ata/PredictionIO-0.11.0-incubating/conf:/usr/hdp/2.5.3.0-37/
hadoop/conf:/data/PredictionIO/vendors/hbase/conf --conf
spark.driver.extraJavaOptions=-Dpio.log.dir=/home/da_mcom_milan --class
org.apache.predictionio.workflow.CreateServer --jars
file:/data/PredictionIO/some-engine/target/scala-2.10/
universal-recommender_2.10-0.6.0.jar,file:/data/PredictionIO/some-engine
/target/scala-2.10/universal-recommender-assembly-0.6.0-deps.jar,file:/
data/PredictionIO-0.11.0-incubating/lib/spark/pio-data-elast
icsearch1-assembly-0.11.0-incubating.jar,file:/data/Pred
ictionIO-0.11.0-incubating/lib/spark/pio-data-hbase-
assembly-0.11.0-incubating.jar,file:/data/PredictionIO-0.
11.0-incubating/lib/spark/pio-data-hdfs-assembly-0.11.0-incu
bating.jar,file:/data/PredictionIO-0.11.0-incubating/lib/
spark/pio-data-jdbc-assembly-0.11.0-incubating.jar,file:/
data/PredictionIO-0.11.0-incubating/lib/spark/pio-data-local
fs-assembly-0.11.0-incubating.jar --files file:/data/PredictionIO-0.11.0
-incubating/conf/log4j.properties,file:/usr/hdp/2.5.3.0-37/
hadoop/conf/core-site.xml,file:/data/PredictionIO/vendors/hbase/conf/hbase-site.xml
file:/data/PredictionIO-0.11.0-incubating/lib/pio-assembly-0.11.0-incubating.jar
--engineInstanceId AWKSaLQrSe_zVbacBHWI --engine-variant
file:/data/PredictionIO/some-engine/engine.json --ip 0.0.0.0 --port *8004*
--event-server-ip 0.0.0.0 --event-server-port 7070 --json-extractor Both
--env PIO_STORAGE_SOURCES_HBASE_TYPE=hbase,PIO_ENV_LOADED=1,PIO_
STORAGE_REPOSITORIES_METADATA_NAME=pio_meta,PIO_FS_BASEDIR=/
home/da_mcom_milan/.pio_store,PIO_STORAGE_SOURCES_ELASTICSEA
RCH_HOSTS=mdc2vra176,PIO_STORAGE_SOURCES_HBASE_HOME=/
data/PredictionIO/vendors/hbase,PIO_HOME=/data/PredictionIO-0.11.0-
incubating,PIO_FS_ENGINESDIR=/home/da_mcom_milan/.pio_store/
engines,PIO_STORAGE_SOURCES_LOCALFS_PATH=/home/da_mcom_
milan/.pio_store/models,PIO_STORAGE_SOURCES_ELASTICSEARCH_
TYPE=elasticsearch,PIO_STORAGE_REPOSITORIES_METADATA_
SOURCE=ELASTICSEARCH,PIO_STORAGE_REPOSITORIES_MODELDATA_SOURCE=LOCALFS,PIO_
STORAGE_REPOSITORIES_EVENTDATA_NAME=pio_event,PIO_
STORAGE_SOURCES_ELASTICSEARCH_CLUSTERNAME=pros-dev,PIO_FS_
TMPDIR=/home/da_mcom_milan/.pio_store/tmp,PIO_STORAGE_
REPOSITORIES_MODELDATA_NAME=pio_model,PIO_STORAGE_
REPOSITORIES_EVENTDATA_SOURCE=HBASE,PIO_CONF_DIR=/data/PredictionIO-0.11.0-
incubating/conf,PIO_STORAGE_SOURCES_ELASTICSEARCH_PORTS=
9300,PIO_STORAGE_SOURCES_LOCALFS_TYPE=localfs
For some reasons, processes 30932 and 31097 still alive and process
listened by 32671
Any ideas?
Thank you,
George
Re: PIO deploy: old processes still live after redeployment
Posted by Donald Szeto <do...@apache.org>.
Have you try examining the output of lsof on those processes? It may tell
you which open I/O is holding up the JVM.
On Wed, Apr 11, 2018 at 4:42 PM George Yarish <gy...@griddynamics.com>
wrote:
> That totally makes sense. We use our custom algorithm based on Universal
> Recommender (UR), so we use elasticsearch and hbase in serving code. But we
> use it via PIO api.
> For hbase we use
> org.apache.predictionio.data.storage.hbase.HBLEvents.futureFind
> For elasticsearch we use EsClient from UR code.
>
> On Wed, Apr 11, 2018 at 4:14 PM, Donald Szeto <do...@apache.org> wrote:
>
>> What engine template are you running? I think the server inside the JVM
>> process has already shut down (otherwise the new one won't be able to bind
>> to the same port), but it may have some I/O (e.g. database connections)
>> that is still active which would have hold the JVM process from
>> terminating. Is there a customization like that in your serving code?
>>
>> On Wed, Apr 11, 2018 at 4:09 PM, George Yarish <gy...@griddynamics.com>
>> wrote:
>>
>>> I'm sorry for that, I have a few environments in which that problem
>>> reproduced. That was the one with *8000* port.
>>> It's pretty the same:
>>>
>>> $ ps aux | grep *8000* | wc -l
>>> 13
>>>
>>> and processes looks like:
>>>
>>> da_mcom+ 17028 0.5 0.9 26473704 603220 ? Sl апр06 42:05 java
>>> -cp
>>> /data/PredictionIO-0.11.0-incubating/conf:/data/PredictionIO-0.11.0-incubating/plugins/*:/data/PredictionIO-0.11.0-incubating/lib/spark/*:/data/PredictionIO-0.11.0-incubating/lib/pio-assembly-0.11.0-incubating.jar:/usr/hdp/
>>> 2.6.1.0-129/hbase/conf -Dpio.log.dir=/home/da_mcom_milan
>>> org.apache.predictionio.tools.console.Console deploy --port *8000*
>>> --pio-home /data/PredictionIO-0.11.0-incubating
>>> da_mcom+ 17269 1.1 1.5 19765888 996524 ? Sl апр06 87:45
>>> /usr/java/default/bin/java -Dhdp.version=2.6.1.0-129 -cp
>>> /data/PredictionIO-0.11.0-incubating/conf/:/usr/hdp/2.6.1.0-129/hbase/conf/:/usr/hdp/current/spark-client/conf/:/data/PredictionIO-0.11.0-incubating/vendors/spark/lib/spark-assembly-1.6.3.2.6.1.0-129-hadoop2.7.3.2.6.1.0-129.jar:/data/PredictionIO-0.11.0-incubating/vendors/spark/lib/datanucleus-api-jdo-3.2.6.jar:/data/PredictionIO-0.11.0-incubating/vendors/spark/lib/datanucleus-core-3.2.10.jar:/data/PredictionIO-0.11.0-incubating/vendors/spark/lib/datanucleus-rdbms-3.2.9.jar:/usr/hdp/current/hadoop-client/conf/:/usr/hdp/current/hadoop-client/lib/aws-java-sdk-s3-1.10.6.jar:/usr/hdp/current/hadoop-client/lib/aws-java-sdk-core-1.10.6.jar:/usr/hdp/current/hadoop-client/lib/aws-java-sdk-kms-1.10.6.jar
>>> -Xms1g -Xmx1g -Dpio.log.dir=/home/da_mcom_milan
>>> org.apache.spark.deploy.SparkSubmit --conf
>>> spark.driver.extraClassPath=/data/PredictionIO-0.11.0-incubating/conf:/usr/hdp/2.6.1.0-129/hbase/conf
>>> --conf spark.driver.extraJavaOptions=-Dpio.log.dir=/home/da_mcom_milan
>>> --class org.apache.predictionio.workflow.CreateServer --jars
>>> file:/data/PredictionIO-0.11.0-incubating/engines/some-engine/target/scala-2.10/universal-recommender_2.10-0.6.0.jar,file:/data/PredictionIO-0.11.0-incubating/engines/some-engine/target/scala-2.10/universal-recommender-assembly-0.6.0-deps.jar,file:/data/PredictionIO-0.11.0-incubating/lib/spark/pio-data-elasticsearch1-assembly-0.11.0-incubating.jar,file:/data/PredictionIO-0.11.0-incubating/lib/spark/pio-data-hbase-assembly-0.11.0-incubating.jar,file:/data/PredictionIO-0.11.0-incubating/lib/spark/pio-data-hdfs-assembly-0.11.0-incubating.jar,file:/data/PredictionIO-0.11.0-incubating/lib/spark/pio-data-jdbc-assembly-0.11.0-incubating.jar,file:/data/PredictionIO-0.11.0-incubating/lib/spark/pio-data-localfs-assembly-0.11.0-incubating.jar
>>> --files
>>> file:/data/PredictionIO-0.11.0-incubating/conf/log4j.properties,file:/usr/hdp/2.6.1.0-129/hbase/conf/hbase-site.xml
>>> file:/data/PredictionIO-0.11.0-incubating/lib/pio-assembly-0.11.0-incubating.jar
>>> --engineInstanceId AWKbr8hAdWI0xyRGjxPY --engine-variant
>>> file:/data/PredictionIO-0.11.0-incubating/engines/some-engine/engine.json
>>> --ip 0.0.0.0 --port *8000* --event-server-ip 0.0.0.0
>>> --event-server-port 7070 --json-extractor Both --env
>>> PIO_STORAGE_SOURCES_HBASE_TYPE=hbase,PIO_ENV_LOADED=1,PIO_STORAGE_REPOSITORIES_METADATA_NAME=pio_meta,PIO_FS_BASEDIR=/home/da_mcom_milan/.pio_store,PIO_STORAGE_SOURCES_ELASTICSEARCH_HOSTS=mdc2pr068,mdc2pr069,mdc2pr070,PIO_STORAGE_SOURCES_HBASE_HOME=/usr/hdp/2.6.1.0-129/hbase,PIO_HOME=/data/PredictionIO-0.11.0-incubating,PIO_FS_ENGINESDIR=/home/da_mcom_milan/.pio_store/engines,PIO_STORAGE_SOURCES_HDFS_TYPE=hdfs,PIO_STORAGE_SOURCES_HDFS_PATH=/tmp/models,PIO_STORAGE_SOURCES_ELASTICSEARCH_TYPE=elasticsearch,PIO_STORAGE_REPOSITORIES_METADATA_SOURCE=ELASTICSEARCH,PIO_STORAGE_REPOSITORIES_MODELDATA_SOURCE=HDFS,PIO_STORAGE_REPOSITORIES_EVENTDATA_NAME=pio_event,PIO_STORAGE_SOURCES_ELASTICSEARCH_CLUSTERNAME=pros_es_prod,PIO_FS_TMPDIR=/home/da_mcom_milan/.pio_store/tmp,PIO_STORAGE_REPOSITORIES_MODELDATA_NAME=pio_model,PIO_STORAGE_REPOSITORIES_EVENTDATA_SOURCE=HBASE,PIO_CONF_DIR=/data/PredictionIO-0.11.0-incubating/conf,PIO_STORAGE_SOURCES_ELASTICSEARCH_PORTS=9300,9300,9300
>>> da_mcom+ 17305 11.9 1.0 17214656 662920 pts/1 Sl 19:00 0:46
>>> /usr/java/default/bin/java -Dhdp.version=2.6.1.0-129 -cp
>>> /data/PredictionIO-0.11.0-incubating/conf/:/usr/hdp/2.6.1.0-129/hbase/conf/:/usr/hdp/current/spark-client/conf/:/data/PredictionIO-0.11.0-incubating/vendors/spark/lib/spark-assembly-1.6.3.2.6.1.0-129-hadoop2.7.3.2.6.1.0-129.jar:/data/PredictionIO-0.11.0-incubating/vendors/spark/lib/datanucleus-api-jdo-3.2.6.jar:/data/PredictionIO-0.11.0-incubating/vendors/spark/lib/datanucleus-core-3.2.10.jar:/data/PredictionIO-0.11.0-incubating/vendors/spark/lib/datanucleus-rdbms-3.2.9.jar:/usr/hdp/current/hadoop-client/conf/:/usr/hdp/current/hadoop-client/lib/aws-java-sdk-s3-1.10.6.jar:/usr/hdp/current/hadoop-client/lib/aws-java-sdk-core-1.10.6.jar:/usr/hdp/current/hadoop-client/lib/aws-java-sdk-kms-1.10.6.jar
>>> -Xms1g -Xmx1g -Dpio.log.dir=/home/da_mcom_milan
>>> org.apache.spark.deploy.SparkSubmit --conf
>>> spark.driver.extraClassPath=/data/PredictionIO-0.11.0-incubating/conf:/usr/hdp/2.6.1.0-129/hbase/conf
>>> --conf spark.driver.extraJavaOptions=-Dpio.log.dir=/home/da_mcom_milan
>>> --class org.apache.predictionio.workflow.CreateServer --jars
>>> file:/data/PredictionIO-0.11.0-incubating/engines/some-engine/target/scala-2.10/universal-recommender_2.10-0.6.0.jar,file:/data/PredictionIO-0.11.0-incubating/engines/some-engine/target/scala-2.10/universal-recommender-assembly-0.6.0-deps.jar,file:/data/PredictionIO-0.11.0-incubating/lib/spark/pio-data-elasticsearch1-assembly-0.11.0-incubating.jar,file:/data/PredictionIO-0.11.0-incubating/lib/spark/pio-data-hbase-assembly-0.11.0-incubating.jar,file:/data/PredictionIO-0.11.0-incubating/lib/spark/pio-data-hdfs-assembly-0.11.0-incubating.jar,file:/data/PredictionIO-0.11.0-incubating/lib/spark/pio-data-jdbc-assembly-0.11.0-incubating.jar,file:/data/PredictionIO-0.11.0-incubating/lib/spark/pio-data-localfs-assembly-0.11.0-incubating.jar
>>> --files
>>> file:/data/PredictionIO-0.11.0-incubating/conf/log4j.properties,file:/usr/hdp/2.6.1.0-129/hbase/conf/hbase-site.xml
>>> file:/data/PredictionIO-0.11.0-incubating/lib/pio-assembly-0.11.0-incubating.jar
>>> --engineInstanceId AWK1b5NndWI0xyRGBxWM --engine-variant
>>> file:/data/PredictionIO-0.11.0-incubating/engines/some-engine/engine.json
>>> --ip 0.0.0.0 --port *8000* --event-server-ip 0.0.0.0
>>> --event-server-port 7070 --json-extractor Both --env
>>> PIO_STORAGE_SOURCES_HBASE_TYPE=hbase,PIO_ENV_LOADED=1,PIO_STORAGE_REPOSITORIES_METADATA_NAME=pio_meta,PIO_FS_BASEDIR=/home/da_mcom_milan/.pio_store,PIO_STORAGE_SOURCES_ELASTICSEARCH_HOSTS=mdc2pr068,mdc2pr069,mdc2pr070,PIO_STORAGE_SOURCES_HBASE_HOME=/usr/hdp/2.6.1.0-129/hbase,PIO_HOME=/data/PredictionIO-0.11.0-incubating,PIO_FS_ENGINESDIR=/home/da_mcom_milan/.pio_store/engines,PIO_STORAGE_SOURCES_HDFS_TYPE=hdfs,PIO_STORAGE_SOURCES_HDFS_PATH=/tmp/models,PIO_STORAGE_SOURCES_ELASTICSEARCH_TYPE=elasticsearch,PIO_STORAGE_REPOSITORIES_METADATA_SOURCE=ELASTICSEARCH,PIO_STORAGE_REPOSITORIES_MODELDATA_SOURCE=HDFS,PIO_STORAGE_REPOSITORIES_EVENTDATA_NAME=pio_event,PIO_STORAGE_SOURCES_ELASTICSEARCH_CLUSTERNAME=pros_es_prod,PIO_FS_TMPDIR=/home/da_mcom_milan/.pio_store/tmp,PIO_STORAGE_REPOSITORIES_MODELDATA_NAME=pio_model,PIO_STORAGE_REPOSITORIES_EVENTDATA_SOURCE=HBASE,PIO_CONF_DIR=/data/PredictionIO-0.11.0-incubating/conf,PIO_STORAGE_SOURCES_ELASTICSEARCH_PORTS=9300,9300,9300
>>> da_mcom+ 18988 0.0 0.0 112652 984 pts/1 R+ 19:06 0:00 grep
>>> --color=auto *8000*
>>> da_mcom+ 48849 0.5 0.9 26473704 607720 ? Sl апр05 50:01 java
>>> -cp
>>> /data/PredictionIO-0.11.0-incubating/conf:/data/PredictionIO-0.11.0-incubating/plugins/*:/data/PredictionIO-0.11.0-incubating/lib/spark/*:/data/PredictionIO-0.11.0-incubating/lib/pio-assembly-0.11.0-incubating.jar:/usr/hdp/
>>> 2.6.1.0-129/hbase/conf -Dpio.log.dir=/home/da_mcom_milan
>>> org.apache.predictionio.tools.console.Console deploy --port *8000*
>>> --pio-home /data/PredictionIO-0.11.0-incubating
>>> da_mcom+ 49016 1.1 1.4 19766916 967792 ? Sl апр05 102:52
>>> /usr/java/default/bin/java -Dhdp.version=2.6.1.0-129 -cp
>>> /data/PredictionIO-0.11.0-incubating/conf/:/usr/hdp/2.6.1.0-129/hbase/conf/:/usr/hdp/current/spark-client/conf/:/data/PredictionIO-0.11.0-incubating/vendors/spark/lib/spark-assembly-1.6.3.2.6.1.0-129-hadoop2.7.3.2.6.1.0-129.jar:/data/PredictionIO-0.11.0-incubating/vendors/spark/lib/datanucleus-api-jdo-3.2.6.jar:/data/PredictionIO-0.11.0-incubating/vendors/spark/lib/datanucleus-core-3.2.10.jar:/data/PredictionIO-0.11.0-incubating/vendors/spark/lib/datanucleus-rdbms-3.2.9.jar:/usr/hdp/current/hadoop-client/conf/:/usr/hdp/current/hadoop-client/lib/aws-java-sdk-s3-1.10.6.jar:/usr/hdp/current/hadoop-client/lib/aws-java-sdk-core-1.10.6.jar:/usr/hdp/current/hadoop-client/lib/aws-java-sdk-kms-1.10.6.jar
>>> -Xms1g -Xmx1g -Dpio.log.dir=/home/da_mcom_milan
>>> org.apache.spark.deploy.SparkSubmit --conf
>>> spark.driver.extraClassPath=/data/PredictionIO-0.11.0-incubating/conf:/usr/hdp/2.6.1.0-129/hbase/conf
>>> --conf spark.driver.extraJavaOptions=-Dpio.log.dir=/home/da_mcom_milan
>>> --class org.apache.predictionio.workflow.CreateServer --jars
>>> file:/data/PredictionIO-0.11.0-incubating/engines/some-engine/target/scala-2.10/universal-recommender_2.10-0.6.0.jar,file:/data/PredictionIO-0.11.0-incubating/engines/some-engine/target/scala-2.10/universal-recommender-assembly-0.6.0-deps.jar,file:/data/PredictionIO-0.11.0-incubating/lib/spark/pio-data-elasticsearch1-assembly-0.11.0-incubating.jar,file:/data/PredictionIO-0.11.0-incubating/lib/spark/pio-data-hbase-assembly-0.11.0-incubating.jar,file:/data/PredictionIO-0.11.0-incubating/lib/spark/pio-data-hdfs-assembly-0.11.0-incubating.jar,file:/data/PredictionIO-0.11.0-incubating/lib/spark/pio-data-jdbc-assembly-0.11.0-incubating.jar,file:/data/PredictionIO-0.11.0-incubating/lib/spark/pio-data-localfs-assembly-0.11.0-incubating.jar
>>> --files
>>> file:/data/PredictionIO-0.11.0-incubating/conf/log4j.properties,file:/usr/hdp/2.6.1.0-129/hbase/conf/hbase-site.xml
>>> file:/data/PredictionIO-0.11.0-incubating/lib/pio-assembly-0.11.0-incubating.jar
>>> --engineInstanceId AWKWiWsxdWI0xyRGjxPJ --engine-variant
>>> file:/data/PredictionIO-0.11.0-incubating/engines/some-engine/engine.json
>>> --ip 0.0.0.0 --port *8000* --event-server-ip 0.0.0.0
>>> --event-server-port 7070 --json-extractor Both --env
>>> PIO_STORAGE_SOURCES_HBASE_TYPE=hbase,PIO_ENV_LOADED=1,PIO_STORAGE_REPOSITORIES_METADATA_NAME=pio_meta,PIO_FS_BASEDIR=/home/da_mcom_milan/.pio_store,PIO_STORAGE_SOURCES_ELASTICSEARCH_HOSTS=mdc2pr068,mdc2pr069,mdc2pr070,PIO_STORAGE_SOURCES_HBASE_HOME=/usr/hdp/2.6.1.0-129/hbase,PIO_HOME=/data/PredictionIO-0.11.0-incubating,PIO_FS_ENGINESDIR=/home/da_mcom_milan/.pio_store/engines,PIO_STORAGE_SOURCES_HDFS_TYPE=hdfs,PIO_STORAGE_SOURCES_HDFS_PATH=/tmp/models,PIO_STORAGE_SOURCES_ELASTICSEARCH_TYPE=elasticsearch,PIO_STORAGE_REPOSITORIES_METADATA_SOURCE=ELASTICSEARCH,PIO_STORAGE_REPOSITORIES_MODELDATA_SOURCE=HDFS,PIO_STORAGE_REPOSITORIES_EVENTDATA_NAME=pio_event,PIO_STORAGE_SOURCES_ELASTICSEARCH_CLUSTERNAME=pros_es_prod,PIO_FS_TMPDIR=/home/da_mcom_milan/.pio_store/tmp,PIO_STORAGE_REPOSITORIES_MODELDATA_NAME=pio_model,PIO_STORAGE_REPOSITORIES_EVENTDATA_SOURCE=HBASE,PIO_CONF_DIR=/data/PredictionIO-0.11.0-incubating/conf,PIO_STORAGE_SOURCES_ELASTICSEARCH_PORTS=9300,9300,9300
>>>
>>>
>>>
>>> On Wed, Apr 11, 2018 at 3:39 PM, Donald Szeto <do...@apache.org> wrote:
>>>
>>>> Are you able to find any occurrence of port 8004? The excerpt you
>>>> showed was about port 8000.
>>>>
>>>> On Wed, Apr 11, 2018 at 3:18 PM, George Yarish <
>>>> gyarish@griddynamics.com> wrote:
>>>>
>>>>> Hi Donald
>>>>>
>>>>> Thank you for the answer.
>>>>> it looks fine to me, but processes still alive:
>>>>>
>>>>> 2018-04-11 13:00:25,331 INFO
>>>>> org.apache.predictionio.workflow.MasterActor
>>>>> [pio-server-akka.actor.default-dispatcher-5] - Undeploying any existing
>>>>> engine instance at http://0.0.0.0:8000
>>>>> 2018-04-11 13:00:26,467 INFO
>>>>> org.apache.predictionio.workflow.MasterActor
>>>>> [pio-server-akka.actor.default-dispatcher-9] - Stop server command received.
>>>>> 2018-04-11 13:00:26,468 INFO
>>>>> org.apache.predictionio.workflow.MasterActor
>>>>> [pio-server-akka.actor.default-dispatcher-9] - Server is shutting down.
>>>>> 2018-04-11 13:00:26,617 INFO spray.can.server.HttpListener
>>>>> [pio-server-akka.actor.default-dispatcher-2] - Bound to /0.0.0.0:8000
>>>>> 2018-04-11 13:00:26,620 INFO
>>>>> org.apache.predictionio.workflow.MasterActor
>>>>> [pio-server-akka.actor.default-dispatcher-4] - Engine is deployed and
>>>>> running. Engine API is live at http://0.0.0.0:8000
>>>>>
>>>>> Thank you,
>>>>> George
>>>>>
>>>>>
>>>>> On Wed, Apr 11, 2018 at 11:48 AM, Donald Szeto <do...@apache.org>
>>>>> wrote:
>>>>>
>>>>>> Hi George,
>>>>>>
>>>>>> Mind checking the content of pio.log at where you run `pio deploy`?
>>>>>> It should contain some messages regarding undeploying existing engine
>>>>>> servers on the same port.
>>>>>>
>>>>>> Regards,
>>>>>> Donald
>>>>>>
>>>>>> On Thu, Apr 5, 2018 at 6:43 PM, George Yarish <
>>>>>> gyarish@griddynamics.com> wrote:
>>>>>>
>>>>>>> Hi everyone,
>>>>>>>
>>>>>>> I faced a strange situation recently.
>>>>>>> One of our engines has frequently old processes after daily
>>>>>>> redeployment.
>>>>>>>
>>>>>>> We used to redeploy by crontab rule that looks like:
>>>>>>> nohup pio deploy --port 8004 >> ${DEPLOY_LOG_FILE} 2>&1 &
>>>>>>>
>>>>>>>
>>>>>>> This is output of ps aux command:
>>>>>>>
>>>>>>> [da_mcom_milan@mdc2pr113 ~]$ ps aux | grep 8004
>>>>>>>
>>>>>>> da_mcom+ 7854 0.0 0.0 112660 996 pts/1 S+ 21:34 0:00
>>>>>>> grep --color=auto *8004*
>>>>>>>
>>>>>>> 1. da_mcom+ 30932 0.6 0.9 26473632 595932 ? Sl 01:57 7:51
>>>>>>> java -cp
>>>>>>> /data/PredictionIO-0.11.0-incubating/conf:/data/PredictionIO-0.11.0-incubating/plugins/*:/data/PredictionIO-0.11.0-incubating/lib/spark/*:/data/PredictionIO-0.11.0-incubating/lib/pio-assembly-0.11.0-incubating.jar:/usr/hdp/
>>>>>>> 2.5.3.0-37/hadoop/conf:/data/PredictionIO/vendors/hbase/conf
>>>>>>> -Dpio.log.dir=/home/da_mcom_milan
>>>>>>> org.apache.predictionio.tools.console.Console deploy --port *8004*
>>>>>>> --pio-home /data/PredictionIO-0.11.0-incubating
>>>>>>>
>>>>>>> 2. da_mcom+ 31097 1.2 1.2 17065124 822596 ? Sl 01:57 14:48
>>>>>>> /usr/java/default/bin/java -Dhdp.version=2.5.3.0-37 -cp
>>>>>>> /data/PredictionIO-0.11.0-incubating/conf/:/usr/hdp/2.5.3.0-37/hadoop/conf/:/data/PredictionIO/vendors/hbase/conf/:/usr/hdp/current/spark-client/conf/:/data/PredictionIO/vendors/spark/lib/spark-assembly-1.6.2.2.5.3.0-37-hadoop2.7.3.2.5.3.0-37.jar:/data/PredictionIO/vendors/spark/lib/datanucleus-api-jdo-3.2.6.jar:/data/PredictionIO/vendors/spark/lib/datanucleus-core-3.2.10.jar:/data/PredictionIO/vendors/spark/lib/datanucleus-rdbms-3.2.9.jar:/usr/hdp/2.5.3.0-37/hadoop/conf/:/usr/hdp/current/hadoop-client/lib/aws-java-sdk-s3-1.10.6.jar:/usr/hdp/current/hadoop-client/lib/aws-java-sdk-core-1.10.6.jar:/usr/hdp/current/hadoop-client/lib/aws-java-sdk-kms-1.10.6.jar
>>>>>>> -Xms1g -Xmx1g -Dpio.log.dir=/home/da_mcom_milan
>>>>>>> org.apache.spark.deploy.SparkSubmit --conf
>>>>>>> spark.driver.extraClassPath=/data/PredictionIO-0.11.0-incubating/conf:/usr/hdp/2.5.3.0-37/hadoop/conf:/data/PredictionIO/vendors/hbase/conf
>>>>>>> --conf spark.driver.extraJavaOptions=-Dpio.log.dir=/home/da_mcom_milan
>>>>>>> --class org.apache.predictionio.workflow.CreateServer --jars
>>>>>>> file:/data/PredictionIO/some-engine/target/scala-2.10/universal-recommender_2.10-0.6.0.jar,file:/data/PredictionIO/
>>>>>>> some-engine/target/scala-2.10/universal-recommender-assembly-0.6.0-deps.jar,file:/data/PredictionIO-0.11.0-incubating/lib/spark/pio-data-elasticsearch1-assembly-0.11.0-incubating.jar,file:/data/PredictionIO-0.11.0-incubating/lib/spark/pio-data-hbase-assembly-0.11.0-incubating.jar,file:/data/PredictionIO-0.11.0-incubating/lib/spark/pio-data-hdfs-assembly-0.11.0-incubating.jar,file:/data/PredictionIO-0.11.0-incubating/lib/spark/pio-data-jdbc-assembly-0.11.0-incubating.jar,file:/data/PredictionIO-0.11.0-incubating/lib/spark/pio-data-localfs-assembly-0.11.0-incubating.jar
>>>>>>> --files
>>>>>>> file:/data/PredictionIO-0.11.0-incubating/conf/log4j.properties,file:/usr/hdp/2.5.3.0-37/hadoop/conf/core-site.xml,file:/data/PredictionIO/vendors/hbase/conf/hbase-site.xml
>>>>>>> file:/data/PredictionIO-0.11.0-incubating/lib/pio-assembly-0.11.0-incubating.jar
>>>>>>> --engineInstanceId AWKSaLQrSe_zVbacBHWI --engine-variant
>>>>>>> file:/data/PredictionIO/some-engine/engine.json --ip 0.0.0.0 --port
>>>>>>> *8004* --event-server-ip 0.0.0.0 --event-server-port 7070
>>>>>>> --json-extractor Both --env
>>>>>>> PIO_STORAGE_SOURCES_HBASE_TYPE=hbase,PIO_ENV_LOADED=1,PIO_STORAGE_REPOSITORIES_METADATA_NAME=pio_meta,PIO_FS_BASEDIR=/home/da_mcom_milan/.pio_store,PIO_STORAGE_SOURCES_ELASTICSEARCH_HOSTS=mdc2vra176,PIO_STORAGE_SOURCES_HBASE_HOME=/data/PredictionIO/vendors/hbase,PIO_HOME=/data/PredictionIO-0.11.0-incubating,PIO_FS_ENGINESDIR=/home/da_mcom_milan/.pio_store/engines,PIO_STORAGE_SOURCES_LOCALFS_PATH=/home/da_mcom_milan/.pio_store/models,PIO_STORAGE_SOURCES_ELASTICSEARCH_TYPE=elasticsearch,PIO_STORAGE_REPOSITORIES_METADATA_SOURCE=ELASTICSEARCH,PIO_STORAGE_REPOSITORIES_MODELDATA_SOURCE=LOCALFS,PIO_STORAGE_REPOSITORIES_EVENTDATA_NAME=pio_event,PIO_STORAGE_SOURCES_ELASTICSEARCH_CLUSTERNAME=pros-dev,PIO_FS_TMPDIR=/home/da_mcom_milan/.pio_store/tmp,PIO_STORAGE_REPOSITORIES_MODELDATA_NAME=pio_model,PIO_STORAGE_REPOSITORIES_EVENTDATA_SOURCE=HBASE,PIO_CONF_DIR=/data/PredictionIO-0.11.0-incubating/conf,PIO_STORAGE_SOURCES_ELASTICSEARCH_PORTS=9300,PIO_STORAGE_SOURCES_LOCALFS_TYPE=localfs
>>>>>>>
>>>>>>> 3. da_mcom+ 32505 0.6 0.8 26473632 581164 ? Sl 02:00 7:54
>>>>>>> java -cp
>>>>>>> /data/PredictionIO-0.11.0-incubating/conf:/data/PredictionIO-0.11.0-incubating/plugins/*:/data/PredictionIO-0.11.0-incubating/lib/spark/*:/data/PredictionIO-0.11.0-incubating/lib/pio-assembly-0.11.0-incubating.jar:/usr/hdp/
>>>>>>> 2.5.3.0-37/hadoop/conf:/data/PredictionIO/vendors/hbase/conf
>>>>>>> -Dpio.log.dir=/home/da_mcom_milan
>>>>>>> org.apache.predictionio.tools.console.Console deploy --port *8004*
>>>>>>> --pio-home /data/PredictionIO-0.11.0-incubating
>>>>>>>
>>>>>>> 4. da_mcom+ 32671 1.4 1.3 19757488 881448 ? Sl 02:00 16:27
>>>>>>> /usr/java/default/bin/java -Dhdp.version=2.5.3.0-37 -cp
>>>>>>> /data/PredictionIO-0.11.0-incubating/conf/:/usr/hdp/2.5.3.0-37/hadoop/conf/:/data/PredictionIO/vendors/hbase/conf/:/usr/hdp/current/spark-client/conf/:/data/PredictionIO/vendors/spark/lib/spark-assembly-1.6.2.2.5.3.0-37-hadoop2.7.3.2.5.3.0-37.jar:/data/PredictionIO/vendors/spark/lib/datanucleus-api-jdo-3.2.6.jar:/data/PredictionIO/vendors/spark/lib/datanucleus-core-3.2.10.jar:/data/PredictionIO/vendors/spark/lib/datanucleus-rdbms-3.2.9.jar:/usr/hdp/2.5.3.0-37/hadoop/conf/:/usr/hdp/current/hadoop-client/lib/aws-java-sdk-s3-1.10.6.jar:/usr/hdp/current/hadoop-client/lib/aws-java-sdk-core-1.10.6.jar:/usr/hdp/current/hadoop-client/lib/aws-java-sdk-kms-1.10.6.jar
>>>>>>> -Xms1g -Xmx1g -Dpio.log.dir=/home/da_mcom_milan
>>>>>>> org.apache.spark.deploy.SparkSubmit --conf
>>>>>>> spark.driver.extraClassPath=/data/PredictionIO-0.11.0-incubating/conf:/usr/hdp/2.5.3.0-37/hadoop/conf:/data/PredictionIO/vendors/hbase/conf
>>>>>>> --conf spark.driver.extraJavaOptions=-Dpio.log.dir=/home/da_mcom_milan
>>>>>>> --class org.apache.predictionio.workflow.CreateServer --jars
>>>>>>> file:/data/PredictionIO/some-engine
>>>>>>> /target/scala-2.10/universal-recommender_2.10-0.6.0.jar,file:/data/PredictionIO/
>>>>>>> some-engine/target/scala-2.10/universal-recommender-assembly-0.6.0-deps.jar,file:/data/PredictionIO-0.11.0-incubating/lib/spark/pio-data-elasticsearch1-assembly-0.11.0-incubating.jar,file:/data/PredictionIO-0.11.0-incubating/lib/spark/pio-data-hbase-assembly-0.11.0-incubating.jar,file:/data/PredictionIO-0.11.0-incubating/lib/spark/pio-data-hdfs-assembly-0.11.0-incubating.jar,file:/data/PredictionIO-0.11.0-incubating/lib/spark/pio-data-jdbc-assembly-0.11.0-incubating.jar,file:/data/PredictionIO-0.11.0-incubating/lib/spark/pio-data-localfs-assembly-0.11.0-incubating.jar
>>>>>>> --files
>>>>>>> file:/data/PredictionIO-0.11.0-incubating/conf/log4j.properties,file:/usr/hdp/2.5.3.0-37/hadoop/conf/core-site.xml,file:/data/PredictionIO/vendors/hbase/conf/hbase-site.xml
>>>>>>> file:/data/PredictionIO-0.11.0-incubating/lib/pio-assembly-0.11.0-incubating.jar
>>>>>>> --engineInstanceId AWKSaLQrSe_zVbacBHWI --engine-variant
>>>>>>> file:/data/PredictionIO/some-engine/engine.json --ip 0.0.0.0 --port
>>>>>>> *8004* --event-server-ip 0.0.0.0 --event-server-port 7070
>>>>>>> --json-extractor Both --env
>>>>>>> PIO_STORAGE_SOURCES_HBASE_TYPE=hbase,PIO_ENV_LOADED=1,PIO_STORAGE_REPOSITORIES_METADATA_NAME=pio_meta,PIO_FS_BASEDIR=/home/da_mcom_milan/.pio_store,PIO_STORAGE_SOURCES_ELASTICSEARCH_HOSTS=mdc2vra176,PIO_STORAGE_SOURCES_HBASE_HOME=/data/PredictionIO/vendors/hbase,PIO_HOME=/data/PredictionIO-0.11.0-incubating,PIO_FS_ENGINESDIR=/home/da_mcom_milan/.pio_store/engines,PIO_STORAGE_SOURCES_LOCALFS_PATH=/home/da_mcom_milan/.pio_store/models,PIO_STORAGE_SOURCES_ELASTICSEARCH_TYPE=elasticsearch,PIO_STORAGE_REPOSITORIES_METADATA_SOURCE=ELASTICSEARCH,PIO_STORAGE_REPOSITORIES_MODELDATA_SOURCE=LOCALFS,PIO_STORAGE_REPOSITORIES_EVENTDATA_NAME=pio_event,PIO_STORAGE_SOURCES_ELASTICSEARCH_CLUSTERNAME=pros-dev,PIO_FS_TMPDIR=/home/da_mcom_milan/.pio_store/tmp,PIO_STORAGE_REPOSITORIES_MODELDATA_NAME=pio_model,PIO_STORAGE_REPOSITORIES_EVENTDATA_SOURCE=HBASE,PIO_CONF_DIR=/data/PredictionIO-0.11.0-incubating/conf,PIO_STORAGE_SOURCES_ELASTICSEARCH_PORTS=9300,PIO_STORAGE_SOURCES_LOCALFS_TYPE=localfs
>>>>>>>
>>>>>>> For some reasons, processes 30932 and 31097 still alive and process
>>>>>>> listened by 32671
>>>>>>> Any ideas?
>>>>>>>
>>>>>>> Thank you,
>>>>>>> George
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>>
>>>>> George Yarish, Java Developer
>>>>>
>>>>> Grid Dynamics
>>>>>
>>>>> 197101, Rentgena Str., 5A, St.Petersburg, Russia
>>>>>
>>>>> Cell: +7 950 030-1941
>>>>>
>>>>> Read Grid Dynamics' Tech Blog
>>>>> <http://blog.griddynamics.com/?utm_campaign=Big%20Data%20Blog%20social%20media%20promotion&utm_medium=CTA&utm_source=Email>
>>>>>
>>>>>
>>>>>
>>>>
>>>
>>>
>>> --
>>>
>>> George Yarish, Java Developer
>>>
>>> Grid Dynamics
>>>
>>> 197101, Rentgena Str., 5A, St.Petersburg, Russia
>>>
>>> Cell: +7 950 030-1941
>>>
>>> Read Grid Dynamics' Tech Blog
>>> <http://blog.griddynamics.com/?utm_campaign=Big%20Data%20Blog%20social%20media%20promotion&utm_medium=CTA&utm_source=Email>
>>>
>>>
>>>
>>
>
>
> --
>
> George Yarish, Java Developer
>
> Grid Dynamics
>
> 197101, Rentgena Str., 5A, St.Petersburg, Russia
>
> Cell: +7 950 030-1941
>
> Read Grid Dynamics' Tech Blog
> <http://blog.griddynamics.com/?utm_campaign=Big%20Data%20Blog%20social%20media%20promotion&utm_medium=CTA&utm_source=Email>
>
>
>
Re: PIO deploy: old processes still live after redeployment
Posted by George Yarish <gy...@griddynamics.com>.
That totally makes sense. We use our custom algorithm based on Universal
Recommender (UR), so we use elasticsearch and hbase in serving code. But we
use it via PIO api.
For hbase we use
org.apache.predictionio.data.storage.hbase.HBLEvents.futureFind
For elasticsearch we use EsClient from UR code.
On Wed, Apr 11, 2018 at 4:14 PM, Donald Szeto <do...@apache.org> wrote:
> What engine template are you running? I think the server inside the JVM
> process has already shut down (otherwise the new one won't be able to bind
> to the same port), but it may have some I/O (e.g. database connections)
> that is still active which would have hold the JVM process from
> terminating. Is there a customization like that in your serving code?
>
> On Wed, Apr 11, 2018 at 4:09 PM, George Yarish <gy...@griddynamics.com>
> wrote:
>
>> I'm sorry for that, I have a few environments in which that problem
>> reproduced. That was the one with *8000* port.
>> It's pretty the same:
>>
>> $ ps aux | grep *8000* | wc -l
>> 13
>>
>> and processes looks like:
>>
>> da_mcom+ 17028 0.5 0.9 26473704 603220 ? Sl апр06 42:05 java -cp
>> /data/PredictionIO-0.11.0-incubating/conf:/data/PredictionIO
>> -0.11.0-incubating/plugins/*:/data/PredictionIO-0.11.0-incub
>> ating/lib/spark/*:/data/PredictionIO-0.11.0-incubating/lib/
>> pio-assembly-0.11.0-incubating.jar:/usr/hdp/2.6.1.0-129/hbase/conf
>> -Dpio.log.dir=/home/da_mcom_milan org.apache.predictionio.tools.console.Console
>> deploy --port *8000* --pio-home /data/PredictionIO-0.11.0-incubating
>> da_mcom+ 17269 1.1 1.5 19765888 996524 ? Sl апр06 87:45
>> /usr/java/default/bin/java -Dhdp.version=2.6.1.0-129 -cp
>> /data/PredictionIO-0.11.0-incubating/conf/:/usr/hdp/2.6.1.0-
>> 129/hbase/conf/:/usr/hdp/current/spark-client/conf/:/data/
>> PredictionIO-0.11.0-incubating/vendors/spark/lib/spark-
>> assembly-1.6.3.2.6.1.0-129-hadoop2.7.3.2.6.1.0-129.jar:/
>> data/PredictionIO-0.11.0-incubating/vendors/spark/lib/datanu
>> cleus-api-jdo-3.2.6.jar:/data/PredictionIO-0.11.0-incubating
>> /vendors/spark/lib/datanucleus-core-3.2.10.jar:/data/
>> PredictionIO-0.11.0-incubating/vendors/spark/lib/datanucleus
>> -rdbms-3.2.9.jar:/usr/hdp/current/hadoop-client/conf/:/
>> usr/hdp/current/hadoop-client/lib/aws-java-sdk-s3-1.10.6.
>> jar:/usr/hdp/current/hadoop-client/lib/aws-java-sdk-core-
>> 1.10.6.jar:/usr/hdp/current/hadoop-client/lib/aws-java-sdk-kms-1.10.6.jar
>> -Xms1g -Xmx1g -Dpio.log.dir=/home/da_mcom_milan
>> org.apache.spark.deploy.SparkSubmit --conf spark.driver.extraClassPath=/d
>> ata/PredictionIO-0.11.0-incubating/conf:/usr/hdp/2.6.1.0-129/hbase/conf
>> --conf spark.driver.extraJavaOptions=-Dpio.log.dir=/home/da_mcom_milan
>> --class org.apache.predictionio.workflow.CreateServer --jars
>> file:/data/PredictionIO-0.11.0-incubating/engines/some-engin
>> e/target/scala-2.10/universal-recommender_2.10-0.6.0.jar,
>> file:/data/PredictionIO-0.11.0-incubating/engines/some-engi
>> ne/target/scala-2.10/universal-recommender-assembly-0.6.0-
>> deps.jar,file:/data/PredictionIO-0.11.0-incubating
>> /lib/spark/pio-data-elasticsearch1-assembly-0.11.0-
>> incubating.jar,file:/data/PredictionIO-0.11.0-incubating/
>> lib/spark/pio-data-hbase-assembly-0.11.0-incubating.
>> jar,file:/data/PredictionIO-0.11.0-incubating/lib/spark/pio-
>> data-hdfs-assembly-0.11.0-incubating.jar,file:/data/Predicti
>> onIO-0.11.0-incubating/lib/spark/pio-data-jdbc-assembly-
>> 0.11.0-incubating.jar,file:/data/PredictionIO-0.11.0-incub
>> ating/lib/spark/pio-data-localfs-assembly-0.11.0-incubating.jar --files
>> file:/data/PredictionIO-0.11.0-incubating/conf/log4j.propert
>> ies,file:/usr/hdp/2.6.1.0-129/hbase/conf/hbase-site.xml
>> file:/data/PredictionIO-0.11.0-incubating/lib/pio-assembly-0.11.0-incubating.jar
>> --engineInstanceId AWKbr8hAdWI0xyRGjxPY --engine-variant
>> file:/data/PredictionIO-0.11.0-incubating/engines/some-engine/engine.json
>> --ip 0.0.0.0 --port *8000* --event-server-ip 0.0.0.0 --event-server-port
>> 7070 --json-extractor Both --env PIO_STORAGE_SOURCES_HBASE_TYPE
>> =hbase,PIO_ENV_LOADED=1,PIO_STORAGE_REPOSITORIES_METADATA_
>> NAME=pio_meta,PIO_FS_BASEDIR=/home/da_mcom_milan/.pio_store,
>> PIO_STORAGE_SOURCES_ELASTICSEARCH_HOSTS=mdc2pr068,mdc2pr069,
>> mdc2pr070,PIO_STORAGE_SOURCES_HBASE_HOME=/usr/hdp/2.6.1.0-
>> 129/hbase,PIO_HOME=/data/PredictionIO-0.11.0-
>> incubating,PIO_FS_ENGINESDIR=/home/da_mcom_milan/.pio_store/
>> engines,PIO_STORAGE_SOURCES_HDFS_TYPE=hdfs,PIO_STORAGE_
>> SOURCES_HDFS_PATH=/tmp/models,PIO_STORAGE_SOURCES_
>> ELASTICSEARCH_TYPE=elasticsearch,PIO_STORAGE_REPOSITORIES_
>> METADATA_SOURCE=ELASTICSEARCH,PIO_STORAGE_REPOSITORIES_
>> MODELDATA_SOURCE=HDFS,PIO_STORAGE_REPOSITORIES_EVENTDATA
>> _NAME=pio_event,PIO_STORAGE_SOURCES_ELASTICSEARCH_CLUSTERN
>> AME=pros_es_prod,PIO_FS_TMPDIR=/home/da_mcom_milan/.
>> pio_store/tmp,PIO_STORAGE_REPOSITORIES_MODELDATA_NAME=pio_
>> model,PIO_STORAGE_REPOSITORIES_EVENTDATA_SOURCE=HBASE,PIO_
>> CONF_DIR=/data/PredictionIO-0.11.0-incubating/conf,PIO_
>> STORAGE_SOURCES_ELASTICSEARCH_PORTS=9300,9300,9300
>> da_mcom+ 17305 11.9 1.0 17214656 662920 pts/1 Sl 19:00 0:46
>> /usr/java/default/bin/java -Dhdp.version=2.6.1.0-129 -cp
>> /data/PredictionIO-0.11.0-incubating/conf/:/usr/hdp/2.6.1.0-
>> 129/hbase/conf/:/usr/hdp/current/spark-client/conf/:/data/
>> PredictionIO-0.11.0-incubating/vendors/spark/lib/spark-
>> assembly-1.6.3.2.6.1.0-129-hadoop2.7.3.2.6.1.0-129.jar:/
>> data/PredictionIO-0.11.0-incubating/vendors/spark/lib/datanu
>> cleus-api-jdo-3.2.6.jar:/data/PredictionIO-0.11.0-incubating
>> /vendors/spark/lib/datanucleus-core-3.2.10.jar:/data/
>> PredictionIO-0.11.0-incubating/vendors/spark/lib/datanucleus
>> -rdbms-3.2.9.jar:/usr/hdp/current/hadoop-client/conf/:/
>> usr/hdp/current/hadoop-client/lib/aws-java-sdk-s3-1.10.6.
>> jar:/usr/hdp/current/hadoop-client/lib/aws-java-sdk-core-
>> 1.10.6.jar:/usr/hdp/current/hadoop-client/lib/aws-java-sdk-kms-1.10.6.jar
>> -Xms1g -Xmx1g -Dpio.log.dir=/home/da_mcom_milan
>> org.apache.spark.deploy.SparkSubmit --conf spark.driver.extraClassPath=/d
>> ata/PredictionIO-0.11.0-incubating/conf:/usr/hdp/2.6.1.0-129/hbase/conf
>> --conf spark.driver.extraJavaOptions=-Dpio.log.dir=/home/da_mcom_milan
>> --class org.apache.predictionio.workflow.CreateServer --jars
>> file:/data/PredictionIO-0.11.0-incubating/engines/some-engin
>> e/target/scala-2.10/universal-recommender_2.10-0.6.0.jar,
>> file:/data/PredictionIO-0.11.0-incubating/engines/some-engi
>> ne/target/scala-2.10/universal-recommender-assembly-0.6.0-
>> deps.jar,file:/data/PredictionIO-0.11.0-incubating
>> /lib/spark/pio-data-elasticsearch1-assembly-0.11.0-
>> incubating.jar,file:/data/PredictionIO-0.11.0-incubating/
>> lib/spark/pio-data-hbase-assembly-0.11.0-incubating.
>> jar,file:/data/PredictionIO-0.11.0-incubating/lib/spark/pio-
>> data-hdfs-assembly-0.11.0-incubating.jar,file:/data/Predicti
>> onIO-0.11.0-incubating/lib/spark/pio-data-jdbc-assembly-
>> 0.11.0-incubating.jar,file:/data/PredictionIO-0.11.0-incub
>> ating/lib/spark/pio-data-localfs-assembly-0.11.0-incubating.jar --files
>> file:/data/PredictionIO-0.11.0-incubating/conf/log4j.propert
>> ies,file:/usr/hdp/2.6.1.0-129/hbase/conf/hbase-site.xml
>> file:/data/PredictionIO-0.11.0-incubating/lib/pio-assembly-0.11.0-incubating.jar
>> --engineInstanceId AWK1b5NndWI0xyRGBxWM --engine-variant
>> file:/data/PredictionIO-0.11.0-incubating/engines/some-engine/engine.json
>> --ip 0.0.0.0 --port *8000* --event-server-ip 0.0.0.0 --event-server-port
>> 7070 --json-extractor Both --env PIO_STORAGE_SOURCES_HBASE_TYPE
>> =hbase,PIO_ENV_LOADED=1,PIO_STORAGE_REPOSITORIES_METADATA_
>> NAME=pio_meta,PIO_FS_BASEDIR=/home/da_mcom_milan/.pio_store,
>> PIO_STORAGE_SOURCES_ELASTICSEARCH_HOSTS=mdc2pr068,mdc2pr069,
>> mdc2pr070,PIO_STORAGE_SOURCES_HBASE_HOME=/usr/hdp/2.6.1.0-
>> 129/hbase,PIO_HOME=/data/PredictionIO-0.11.0-
>> incubating,PIO_FS_ENGINESDIR=/home/da_mcom_milan/.pio_store/
>> engines,PIO_STORAGE_SOURCES_HDFS_TYPE=hdfs,PIO_STORAGE_
>> SOURCES_HDFS_PATH=/tmp/models,PIO_STORAGE_SOURCES_
>> ELASTICSEARCH_TYPE=elasticsearch,PIO_STORAGE_REPOSITORIES_
>> METADATA_SOURCE=ELASTICSEARCH,PIO_STORAGE_REPOSITORIES_
>> MODELDATA_SOURCE=HDFS,PIO_STORAGE_REPOSITORIES_EVENTDATA
>> _NAME=pio_event,PIO_STORAGE_SOURCES_ELASTICSEARCH_CLUSTERN
>> AME=pros_es_prod,PIO_FS_TMPDIR=/home/da_mcom_milan/.
>> pio_store/tmp,PIO_STORAGE_REPOSITORIES_MODELDATA_NAME=pio_
>> model,PIO_STORAGE_REPOSITORIES_EVENTDATA_SOURCE=HBASE,PIO_
>> CONF_DIR=/data/PredictionIO-0.11.0-incubating/conf,PIO_
>> STORAGE_SOURCES_ELASTICSEARCH_PORTS=9300,9300,9300
>> da_mcom+ 18988 0.0 0.0 112652 984 pts/1 R+ 19:06 0:00 grep
>> --color=auto *8000*
>> da_mcom+ 48849 0.5 0.9 26473704 607720 ? Sl апр05 50:01 java -cp
>> /data/PredictionIO-0.11.0-incubating/conf:/data/PredictionIO
>> -0.11.0-incubating/plugins/*:/data/PredictionIO-0.11.0-incub
>> ating/lib/spark/*:/data/PredictionIO-0.11.0-incubating/lib/
>> pio-assembly-0.11.0-incubating.jar:/usr/hdp/2.6.1.0-129/hbase/conf
>> -Dpio.log.dir=/home/da_mcom_milan org.apache.predictionio.tools.console.Console
>> deploy --port *8000* --pio-home /data/PredictionIO-0.11.0-incubating
>> da_mcom+ 49016 1.1 1.4 19766916 967792 ? Sl апр05 102:52
>> /usr/java/default/bin/java -Dhdp.version=2.6.1.0-129 -cp
>> /data/PredictionIO-0.11.0-incubating/conf/:/usr/hdp/2.6.1.0-
>> 129/hbase/conf/:/usr/hdp/current/spark-client/conf/:/data/
>> PredictionIO-0.11.0-incubating/vendors/spark/lib/spark-
>> assembly-1.6.3.2.6.1.0-129-hadoop2.7.3.2.6.1.0-129.jar:/
>> data/PredictionIO-0.11.0-incubating/vendors/spark/lib/datanu
>> cleus-api-jdo-3.2.6.jar:/data/PredictionIO-0.11.0-incubating
>> /vendors/spark/lib/datanucleus-core-3.2.10.jar:/data/
>> PredictionIO-0.11.0-incubating/vendors/spark/lib/datanucleus
>> -rdbms-3.2.9.jar:/usr/hdp/current/hadoop-client/conf/:/
>> usr/hdp/current/hadoop-client/lib/aws-java-sdk-s3-1.10.6.
>> jar:/usr/hdp/current/hadoop-client/lib/aws-java-sdk-core-
>> 1.10.6.jar:/usr/hdp/current/hadoop-client/lib/aws-java-sdk-kms-1.10.6.jar
>> -Xms1g -Xmx1g -Dpio.log.dir=/home/da_mcom_milan
>> org.apache.spark.deploy.SparkSubmit --conf spark.driver.extraClassPath=/d
>> ata/PredictionIO-0.11.0-incubating/conf:/usr/hdp/2.6.1.0-129/hbase/conf
>> --conf spark.driver.extraJavaOptions=-Dpio.log.dir=/home/da_mcom_milan
>> --class org.apache.predictionio.workflow.CreateServer --jars
>> file:/data/PredictionIO-0.11.0-incubating/engines/some-engin
>> e/target/scala-2.10/universal-recommender_2.10-0.6.0.jar,
>> file:/data/PredictionIO-0.11.0-incubating/engines/some-engi
>> ne/target/scala-2.10/universal-recommender-assembly-0.6.0-
>> deps.jar,file:/data/PredictionIO-0.11.0-incubating
>> /lib/spark/pio-data-elasticsearch1-assembly-0.11.0-
>> incubating.jar,file:/data/PredictionIO-0.11.0-incubating/
>> lib/spark/pio-data-hbase-assembly-0.11.0-incubating.
>> jar,file:/data/PredictionIO-0.11.0-incubating/lib/spark/pio-
>> data-hdfs-assembly-0.11.0-incubating.jar,file:/data/Predicti
>> onIO-0.11.0-incubating/lib/spark/pio-data-jdbc-assembly-
>> 0.11.0-incubating.jar,file:/data/PredictionIO-0.11.0-incub
>> ating/lib/spark/pio-data-localfs-assembly-0.11.0-incubating.jar --files
>> file:/data/PredictionIO-0.11.0-incubating/conf/log4j.propert
>> ies,file:/usr/hdp/2.6.1.0-129/hbase/conf/hbase-site.xml
>> file:/data/PredictionIO-0.11.0-incubating/lib/pio-assembly-0.11.0-incubating.jar
>> --engineInstanceId AWKWiWsxdWI0xyRGjxPJ --engine-variant
>> file:/data/PredictionIO-0.11.0-incubating/engines/some-engine/engine.json
>> --ip 0.0.0.0 --port *8000* --event-server-ip 0.0.0.0 --event-server-port
>> 7070 --json-extractor Both --env PIO_STORAGE_SOURCES_HBASE_TYPE
>> =hbase,PIO_ENV_LOADED=1,PIO_STORAGE_REPOSITORIES_METADATA_
>> NAME=pio_meta,PIO_FS_BASEDIR=/home/da_mcom_milan/.pio_store,
>> PIO_STORAGE_SOURCES_ELASTICSEARCH_HOSTS=mdc2pr068,mdc2pr069,
>> mdc2pr070,PIO_STORAGE_SOURCES_HBASE_HOME=/usr/hdp/2.6.1.0-
>> 129/hbase,PIO_HOME=/data/PredictionIO-0.11.0-
>> incubating,PIO_FS_ENGINESDIR=/home/da_mcom_milan/.pio_store/
>> engines,PIO_STORAGE_SOURCES_HDFS_TYPE=hdfs,PIO_STORAGE_
>> SOURCES_HDFS_PATH=/tmp/models,PIO_STORAGE_SOURCES_
>> ELASTICSEARCH_TYPE=elasticsearch,PIO_STORAGE_REPOSITORIES_
>> METADATA_SOURCE=ELASTICSEARCH,PIO_STORAGE_REPOSITORIES_
>> MODELDATA_SOURCE=HDFS,PIO_STORAGE_REPOSITORIES_EVENTDATA
>> _NAME=pio_event,PIO_STORAGE_SOURCES_ELASTICSEARCH_CLUSTERN
>> AME=pros_es_prod,PIO_FS_TMPDIR=/home/da_mcom_milan/.
>> pio_store/tmp,PIO_STORAGE_REPOSITORIES_MODELDATA_NAME=pio_
>> model,PIO_STORAGE_REPOSITORIES_EVENTDATA_SOURCE=HBASE,PIO_
>> CONF_DIR=/data/PredictionIO-0.11.0-incubating/conf,PIO_
>> STORAGE_SOURCES_ELASTICSEARCH_PORTS=9300,9300,9300
>>
>>
>>
>> On Wed, Apr 11, 2018 at 3:39 PM, Donald Szeto <do...@apache.org> wrote:
>>
>>> Are you able to find any occurrence of port 8004? The excerpt you showed
>>> was about port 8000.
>>>
>>> On Wed, Apr 11, 2018 at 3:18 PM, George Yarish <gyarish@griddynamics.com
>>> > wrote:
>>>
>>>> Hi Donald
>>>>
>>>> Thank you for the answer.
>>>> it looks fine to me, but processes still alive:
>>>>
>>>> 2018-04-11 13:00:25,331 INFO org.apache.predictionio.workflow.MasterActor
>>>> [pio-server-akka.actor.default-dispatcher-5] - Undeploying any
>>>> existing engine instance at http://0.0.0.0:8000
>>>> 2018-04-11 13:00:26,467 INFO org.apache.predictionio.workflow.MasterActor
>>>> [pio-server-akka.actor.default-dispatcher-9] - Stop server command
>>>> received.
>>>> 2018-04-11 13:00:26,468 INFO org.apache.predictionio.workflow.MasterActor
>>>> [pio-server-akka.actor.default-dispatcher-9] - Server is shutting down.
>>>> 2018-04-11 13:00:26,617 INFO spray.can.server.HttpListener
>>>> [pio-server-akka.actor.default-dispatcher-2] - Bound to /0.0.0.0:8000
>>>> 2018-04-11 13:00:26,620 INFO org.apache.predictionio.workflow.MasterActor
>>>> [pio-server-akka.actor.default-dispatcher-4] - Engine is deployed and
>>>> running. Engine API is live at http://0.0.0.0:8000
>>>>
>>>> Thank you,
>>>> George
>>>>
>>>>
>>>> On Wed, Apr 11, 2018 at 11:48 AM, Donald Szeto <do...@apache.org>
>>>> wrote:
>>>>
>>>>> Hi George,
>>>>>
>>>>> Mind checking the content of pio.log at where you run `pio deploy`? It
>>>>> should contain some messages regarding undeploying existing engine servers
>>>>> on the same port.
>>>>>
>>>>> Regards,
>>>>> Donald
>>>>>
>>>>> On Thu, Apr 5, 2018 at 6:43 PM, George Yarish <
>>>>> gyarish@griddynamics.com> wrote:
>>>>>
>>>>>> Hi everyone,
>>>>>>
>>>>>> I faced a strange situation recently.
>>>>>> One of our engines has frequently old processes after daily
>>>>>> redeployment.
>>>>>>
>>>>>> We used to redeploy by crontab rule that looks like:
>>>>>> nohup pio deploy --port 8004 >> ${DEPLOY_LOG_FILE} 2>&1 &
>>>>>>
>>>>>>
>>>>>> This is output of ps aux command:
>>>>>>
>>>>>> [da_mcom_milan@mdc2pr113 ~]$ ps aux | grep 8004
>>>>>>
>>>>>> da_mcom+ 7854 0.0 0.0 112660 996 pts/1 S+ 21:34 0:00 grep
>>>>>> --color=auto *8004*
>>>>>>
>>>>>> 1. da_mcom+ 30932 0.6 0.9 26473632 595932 ? Sl 01:57 7:51
>>>>>> java -cp /data/PredictionIO-0.11.0-incubating/conf:/data/PredictionIO
>>>>>> -0.11.0-incubating/plugins/*:/data/PredictionIO-0.11.0-incub
>>>>>> ating/lib/spark/*:/data/PredictionIO-0.11.0-incubating/lib/p
>>>>>> io-assembly-0.11.0-incubating.jar:/usr/hdp/2.5.3.0-37/hadoop
>>>>>> /conf:/data/PredictionIO/vendors/hbase/conf
>>>>>> -Dpio.log.dir=/home/da_mcom_milan org.apache.predictionio.tools.console.Console
>>>>>> deploy --port *8004* --pio-home /data/PredictionIO-0.11.0-incubating
>>>>>>
>>>>>> 2. da_mcom+ 31097 1.2 1.2 17065124 822596 ? Sl 01:57 14:48
>>>>>> /usr/java/default/bin/java -Dhdp.version=2.5.3.0-37 -cp
>>>>>> /data/PredictionIO-0.11.0-incubating/conf/:/usr/hdp/2.5.3.0-
>>>>>> 37/hadoop/conf/:/data/PredictionIO/vendors/hbase/conf/:/usr/
>>>>>> hdp/current/spark-client/conf/:/data/PredictionIO/vendors/sp
>>>>>> ark/lib/spark-assembly-1.6.2.2.5.3.0-37-hadoop2.7.3.2.5.3.0-
>>>>>> 37.jar:/data/PredictionIO/vendors/spark/lib/datanucleus-api-
>>>>>> jdo-3.2.6.jar:/data/PredictionIO/vendors/spark/lib/datanucle
>>>>>> us-core-3.2.10.jar:/data/PredictionIO/vendors/spark/lib/data
>>>>>> nucleus-rdbms-3.2.9.jar:/usr/hdp/2.5.3.0-37/hadoop/conf/:/us
>>>>>> r/hdp/current/hadoop-client/lib/aws-java-sdk-s3-1.10.6.jar:/
>>>>>> usr/hdp/current/hadoop-client/lib/aws-java-sdk-core-1.10.6.j
>>>>>> ar:/usr/hdp/current/hadoop-client/lib/aws-java-sdk-kms-1.10.6.jar
>>>>>> -Xms1g -Xmx1g -Dpio.log.dir=/home/da_mcom_milan
>>>>>> org.apache.spark.deploy.SparkSubmit --conf
>>>>>> spark.driver.extraClassPath=/data/PredictionIO-0.11.0-incuba
>>>>>> ting/conf:/usr/hdp/2.5.3.0-37/hadoop/conf:/data/PredictionIO/vendors/hbase/conf
>>>>>> --conf spark.driver.extraJavaOptions=-Dpio.log.dir=/home/da_mcom_milan
>>>>>> --class org.apache.predictionio.workflow.CreateServer --jars
>>>>>> file:/data/PredictionIO/some-engine/target/scala-2.10/univer
>>>>>> sal-recommender_2.10-0.6.0.jar,file:/data/PredictionIO/some-engine
>>>>>> /target/scala-2.10/universal-recommender-assembly-0.6.
>>>>>> 0-deps.jar,file:/data/PredictionIO-0.11.0-incubating/lib/spa
>>>>>> rk/pio-data-elasticsearch1-assembly-0.11.0-incubating.jar,fi
>>>>>> le:/data/PredictionIO-0.11.0-incubating/lib/spark/pio-data-h
>>>>>> base-assembly-0.11.0-incubating.jar,file:/data/PredictionIO-
>>>>>> 0.11.0-incubating/lib/spark/pio-data-hdfs-assembly-0.11.0-in
>>>>>> cubating.jar,file:/data/PredictionIO-0.11.0-incubating/lib/s
>>>>>> park/pio-data-jdbc-assembly-0.11.0-incubating.jar,file:/data
>>>>>> /PredictionIO-0.11.0-incubating/lib/spark/pio-data-localfs-assembly-0.11.0-incubating.jar
>>>>>> --files file:/data/PredictionIO-0.11.0-incubating/conf/log4j.propert
>>>>>> ies,file:/usr/hdp/2.5.3.0-37/hadoop/conf/core-site.xml,file:
>>>>>> /data/PredictionIO/vendors/hbase/conf/hbase-site.xml
>>>>>> file:/data/PredictionIO-0.11.0-incubating/lib/pio-assembly-0.11.0-incubating.jar
>>>>>> --engineInstanceId AWKSaLQrSe_zVbacBHWI --engine-variant
>>>>>> file:/data/PredictionIO/some-engine/engine.json --ip 0.0.0.0 --port
>>>>>> *8004* --event-server-ip 0.0.0.0 --event-server-port 7070
>>>>>> --json-extractor Both --env PIO_STORAGE_SOURCES_HBASE_TYPE
>>>>>> =hbase,PIO_ENV_LOADED=1,PIO_STORAGE_REPOSITORIES_METADATA_NA
>>>>>> ME=pio_meta,PIO_FS_BASEDIR=/home/da_mcom_milan/.pio_store,PI
>>>>>> O_STORAGE_SOURCES_ELASTICSEARCH_HOSTS=mdc2vra176,PIO_STORAGE
>>>>>> _SOURCES_HBASE_HOME=/data/PredictionIO/vendors/hbase,PIO_HOM
>>>>>> E=/data/PredictionIO-0.11.0-incubating,PIO_FS_ENGINESDIR=/ho
>>>>>> me/da_mcom_milan/.pio_store/engines,PIO_STORAGE_SOURCES_LOCA
>>>>>> LFS_PATH=/home/da_mcom_milan/.pio_store/models,PIO_STORAGE_S
>>>>>> OURCES_ELASTICSEARCH_TYPE=elasticsearch,PIO_STORAGE_REPOSITO
>>>>>> RIES_METADATA_SOURCE=ELASTICSEARCH,PIO_STORAGE_REPOSITORIES_
>>>>>> MODELDATA_SOURCE=LOCALFS,PIO_STORAGE_REPOSITORIES_EVENTDATA_
>>>>>> NAME=pio_event,PIO_STORAGE_SOURCES_ELASTICSEARCH_CLUSTERNAME
>>>>>> =pros-dev,PIO_FS_TMPDIR=/home/da_mcom_milan/.pio_store/tmp,
>>>>>> PIO_STORAGE_REPOSITORIES_MODELDATA_NAME=pio_model,PIO_STORAG
>>>>>> E_REPOSITORIES_EVENTDATA_SOURCE=HBASE,PIO_CONF_DIR=/
>>>>>> data/PredictionIO-0.11.0-incubating/conf,PIO_STORAGE_
>>>>>> SOURCES_ELASTICSEARCH_PORTS=9300,PIO_STORAGE_SOURCES_
>>>>>> LOCALFS_TYPE=localfs
>>>>>>
>>>>>> 3. da_mcom+ 32505 0.6 0.8 26473632 581164 ? Sl 02:00 7:54
>>>>>> java -cp /data/PredictionIO-0.11.0-incubating/conf:/data/PredictionIO
>>>>>> -0.11.0-incubating/plugins/*:/data/PredictionIO-0.11.0-incub
>>>>>> ating/lib/spark/*:/data/PredictionIO-0.11.0-incubating/lib/p
>>>>>> io-assembly-0.11.0-incubating.jar:/usr/hdp/2.5.3.0-37/hadoop
>>>>>> /conf:/data/PredictionIO/vendors/hbase/conf
>>>>>> -Dpio.log.dir=/home/da_mcom_milan org.apache.predictionio.tools.console.Console
>>>>>> deploy --port *8004* --pio-home /data/PredictionIO-0.11.0-incubating
>>>>>>
>>>>>> 4. da_mcom+ 32671 1.4 1.3 19757488 881448 ? Sl 02:00 16:27
>>>>>> /usr/java/default/bin/java -Dhdp.version=2.5.3.0-37 -cp
>>>>>> /data/PredictionIO-0.11.0-incubating/conf/:/usr/hdp/2.5.3.0-
>>>>>> 37/hadoop/conf/:/data/PredictionIO/vendors/hbase/conf/:/usr/
>>>>>> hdp/current/spark-client/conf/:/data/PredictionIO/vendors/sp
>>>>>> ark/lib/spark-assembly-1.6.2.2.5.3.0-37-hadoop2.7.3.2.5.3.0-
>>>>>> 37.jar:/data/PredictionIO/vendors/spark/lib/datanucleus-api-
>>>>>> jdo-3.2.6.jar:/data/PredictionIO/vendors/spark/lib/datanucle
>>>>>> us-core-3.2.10.jar:/data/PredictionIO/vendors/spark/lib/data
>>>>>> nucleus-rdbms-3.2.9.jar:/usr/hdp/2.5.3.0-37/hadoop/conf/:/us
>>>>>> r/hdp/current/hadoop-client/lib/aws-java-sdk-s3-1.10.6.jar:/
>>>>>> usr/hdp/current/hadoop-client/lib/aws-java-sdk-core-1.10.6.j
>>>>>> ar:/usr/hdp/current/hadoop-client/lib/aws-java-sdk-kms-1.10.6.jar
>>>>>> -Xms1g -Xmx1g -Dpio.log.dir=/home/da_mcom_milan
>>>>>> org.apache.spark.deploy.SparkSubmit --conf
>>>>>> spark.driver.extraClassPath=/data/PredictionIO-0.11.0-incuba
>>>>>> ting/conf:/usr/hdp/2.5.3.0-37/hadoop/conf:/data/PredictionIO/vendors/hbase/conf
>>>>>> --conf spark.driver.extraJavaOptions=-Dpio.log.dir=/home/da_mcom_milan
>>>>>> --class org.apache.predictionio.workflow.CreateServer --jars
>>>>>> file:/data/PredictionIO/some-engine/target/scala-2.10/univer
>>>>>> sal-recommender_2.10-0.6.0.jar,file:/data/PredictionIO/some-engine
>>>>>> /target/scala-2.10/universal-recommender-assembly-0.6.
>>>>>> 0-deps.jar,file:/data/PredictionIO-0.11.0-incubating/lib/spa
>>>>>> rk/pio-data-elasticsearch1-assembly-0.11.0-incubating.jar,fi
>>>>>> le:/data/PredictionIO-0.11.0-incubating/lib/spark/pio-data-h
>>>>>> base-assembly-0.11.0-incubating.jar,file:/data/PredictionIO-
>>>>>> 0.11.0-incubating/lib/spark/pio-data-hdfs-assembly-0.11.0-in
>>>>>> cubating.jar,file:/data/PredictionIO-0.11.0-incubating/lib/s
>>>>>> park/pio-data-jdbc-assembly-0.11.0-incubating.jar,file:/data
>>>>>> /PredictionIO-0.11.0-incubating/lib/spark/pio-data-localfs-assembly-0.11.0-incubating.jar
>>>>>> --files file:/data/PredictionIO-0.11.0-incubating/conf/log4j.propert
>>>>>> ies,file:/usr/hdp/2.5.3.0-37/hadoop/conf/core-site.xml,file:
>>>>>> /data/PredictionIO/vendors/hbase/conf/hbase-site.xml
>>>>>> file:/data/PredictionIO-0.11.0-incubating/lib/pio-assembly-0.11.0-incubating.jar
>>>>>> --engineInstanceId AWKSaLQrSe_zVbacBHWI --engine-variant
>>>>>> file:/data/PredictionIO/some-engine/engine.json --ip 0.0.0.0 --port
>>>>>> *8004* --event-server-ip 0.0.0.0 --event-server-port 7070
>>>>>> --json-extractor Both --env PIO_STORAGE_SOURCES_HBASE_TYPE
>>>>>> =hbase,PIO_ENV_LOADED=1,PIO_STORAGE_REPOSITORIES_METADATA_NA
>>>>>> ME=pio_meta,PIO_FS_BASEDIR=/home/da_mcom_milan/.pio_store,PI
>>>>>> O_STORAGE_SOURCES_ELASTICSEARCH_HOSTS=mdc2vra176,PIO_STORAGE
>>>>>> _SOURCES_HBASE_HOME=/data/PredictionIO/vendors/hbase,PIO_HOM
>>>>>> E=/data/PredictionIO-0.11.0-incubating,PIO_FS_ENGINESDIR=/ho
>>>>>> me/da_mcom_milan/.pio_store/engines,PIO_STORAGE_SOURCES_LOCA
>>>>>> LFS_PATH=/home/da_mcom_milan/.pio_store/models,PIO_STORAGE_S
>>>>>> OURCES_ELASTICSEARCH_TYPE=elasticsearch,PIO_STORAGE_REPOSITO
>>>>>> RIES_METADATA_SOURCE=ELASTICSEARCH,PIO_STORAGE_REPOSITORIES_
>>>>>> MODELDATA_SOURCE=LOCALFS,PIO_STORAGE_REPOSITORIES_EVENTDATA_
>>>>>> NAME=pio_event,PIO_STORAGE_SOURCES_ELASTICSEARCH_CLUSTERNAME
>>>>>> =pros-dev,PIO_FS_TMPDIR=/home/da_mcom_milan/.pio_store/tmp,
>>>>>> PIO_STORAGE_REPOSITORIES_MODELDATA_NAME=pio_model,PIO_STORAG
>>>>>> E_REPOSITORIES_EVENTDATA_SOURCE=HBASE,PIO_CONF_DIR=/
>>>>>> data/PredictionIO-0.11.0-incubating/conf,PIO_STORAGE_
>>>>>> SOURCES_ELASTICSEARCH_PORTS=9300,PIO_STORAGE_SOURCES_
>>>>>> LOCALFS_TYPE=localfs
>>>>>>
>>>>>> For some reasons, processes 30932 and 31097 still alive and process
>>>>>> listened by 32671
>>>>>> Any ideas?
>>>>>>
>>>>>> Thank you,
>>>>>> George
>>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>>
>>>> George Yarish, Java Developer
>>>>
>>>> Grid Dynamics
>>>>
>>>> 197101, Rentgena Str., 5A, St.Petersburg, Russia
>>>>
>>>> Cell: +7 950 030-1941
>>>>
>>>> Read Grid Dynamics' Tech Blog
>>>> <http://blog.griddynamics.com/?utm_campaign=Big%20Data%20Blog%20social%20media%20promotion&utm_medium=CTA&utm_source=Email>
>>>>
>>>>
>>>>
>>>
>>
>>
>> --
>>
>> George Yarish, Java Developer
>>
>> Grid Dynamics
>>
>> 197101, Rentgena Str., 5A, St.Petersburg, Russia
>>
>> Cell: +7 950 030-1941
>>
>> Read Grid Dynamics' Tech Blog
>> <http://blog.griddynamics.com/?utm_campaign=Big%20Data%20Blog%20social%20media%20promotion&utm_medium=CTA&utm_source=Email>
>>
>>
>>
>
--
George Yarish, Java Developer
Grid Dynamics
197101, Rentgena Str., 5A, St.Petersburg, Russia
Cell: +7 950 030-1941
Read Grid Dynamics' Tech Blog
<http://blog.griddynamics.com/?utm_campaign=Big%20Data%20Blog%20social%20media%20promotion&utm_medium=CTA&utm_source=Email>
Re: PIO deploy: old processes still live after redeployment
Posted by Donald Szeto <do...@apache.org>.
What engine template are you running? I think the server inside the JVM
process has already shut down (otherwise the new one won't be able to bind
to the same port), but it may have some I/O (e.g. database connections)
that is still active which would have hold the JVM process from
terminating. Is there a customization like that in your serving code?
On Wed, Apr 11, 2018 at 4:09 PM, George Yarish <gy...@griddynamics.com>
wrote:
> I'm sorry for that, I have a few environments in which that problem
> reproduced. That was the one with *8000* port.
> It's pretty the same:
>
> $ ps aux | grep *8000* | wc -l
> 13
>
> and processes looks like:
>
> da_mcom+ 17028 0.5 0.9 26473704 603220 ? Sl апр06 42:05 java -cp
> /data/PredictionIO-0.11.0-incubating/conf:/data/PredictionIO-0.11.0-
> incubating/plugins/*:/data/PredictionIO-0.11.0-
> incubating/lib/spark/*:/data/PredictionIO-0.11.0-
> incubating/lib/pio-assembly-0.11.0-incubating.jar:/usr/hdp/2.6.1.0-129/hbase/conf
> -Dpio.log.dir=/home/da_mcom_milan org.apache.predictionio.tools.console.Console
> deploy --port *8000* --pio-home /data/PredictionIO-0.11.0-incubating
> da_mcom+ 17269 1.1 1.5 19765888 996524 ? Sl апр06 87:45
> /usr/java/default/bin/java -Dhdp.version=2.6.1.0-129 -cp
> /data/PredictionIO-0.11.0-incubating/conf/:/usr/hdp/2.6.
> 1.0-129/hbase/conf/:/usr/hdp/current/spark-client/conf/:/
> data/PredictionIO-0.11.0-incubating/vendors/spark/lib/
> spark-assembly-1.6.3.2.6.1.0-129-hadoop2.7.3.2.6.1.0-129.
> jar:/data/PredictionIO-0.11.0-incubating/vendors/spark/lib/
> datanucleus-api-jdo-3.2.6.jar:/data/PredictionIO-0.11.0-
> incubating/vendors/spark/lib/datanucleus-core-3.2.10.jar:/
> data/PredictionIO-0.11.0-incubating/vendors/spark/lib/
> datanucleus-rdbms-3.2.9.jar:/usr/hdp/current/hadoop-client/
> conf/:/usr/hdp/current/hadoop-client/lib/aws-java-sdk-s3-1.
> 10.6.jar:/usr/hdp/current/hadoop-client/lib/aws-java-
> sdk-core-1.10.6.jar:/usr/hdp/current/hadoop-client/lib/aws-java-sdk-kms-1.10.6.jar
> -Xms1g -Xmx1g -Dpio.log.dir=/home/da_mcom_milan org.apache.spark.deploy.SparkSubmit
> --conf spark.driver.extraClassPath=/data/PredictionIO-0.11.0-
> incubating/conf:/usr/hdp/2.6.1.0-129/hbase/conf --conf
> spark.driver.extraJavaOptions=-Dpio.log.dir=/home/da_mcom_milan --class
> org.apache.predictionio.workflow.CreateServer --jars
> file:/data/PredictionIO-0.11.0-incubating/engines/some-
> engine/target/scala-2.10/universal-recommender_2.10-0.6.0.jar,file:/data/
> PredictionIO-0.11.0-incubating/engines/some-engine/target/scala-2.10/
> universal-recommender-assembly-0.6.0-deps.jar,file:/
> data/PredictionIO-0.11.0-incubating/lib/spark/pio-data-
> elasticsearch1-assembly-0.11.0-incubating.jar,file:/data/
> PredictionIO-0.11.0-incubating/lib/spark/pio-data-hbase-assembly-0.11.0-
> incubating.jar,file:/data/PredictionIO-0.11.0-
> incubating/lib/spark/pio-data-hdfs-assembly-0.11.0-
> incubating.jar,file:/data/PredictionIO-0.11.0-
> incubating/lib/spark/pio-data-jdbc-assembly-0.11.0-
> incubating.jar,file:/data/PredictionIO-0.11.0-
> incubating/lib/spark/pio-data-localfs-assembly-0.11.0-incubating.jar
> --files file:/data/PredictionIO-0.11.0-incubating/conf/log4j.
> properties,file:/usr/hdp/2.6.1.0-129/hbase/conf/hbase-site.xml
> file:/data/PredictionIO-0.11.0-incubating/lib/pio-assembly-0.11.0-incubating.jar
> --engineInstanceId AWKbr8hAdWI0xyRGjxPY --engine-variant
> file:/data/PredictionIO-0.11.0-incubating/engines/some-engine/engine.json
> --ip 0.0.0.0 --port *8000* --event-server-ip 0.0.0.0 --event-server-port
> 7070 --json-extractor Both --env PIO_STORAGE_SOURCES_HBASE_
> TYPE=hbase,PIO_ENV_LOADED=1,PIO_STORAGE_REPOSITORIES_
> METADATA_NAME=pio_meta,PIO_FS_BASEDIR=/home/da_mcom_milan/.
> pio_store,PIO_STORAGE_SOURCES_ELASTICSEARCH_HOSTS=mdc2pr068,
> mdc2pr069,mdc2pr070,PIO_STORAGE_SOURCES_HBASE_HOME=/
> usr/hdp/2.6.1.0-129/hbase,PIO_HOME=/data/PredictionIO-0.11.
> 0-incubating,PIO_FS_ENGINESDIR=/home/da_mcom_milan/.pio_store/engines,PIO_
> STORAGE_SOURCES_HDFS_TYPE=hdfs,PIO_STORAGE_SOURCES_HDFS_
> PATH=/tmp/models,PIO_STORAGE_SOURCES_ELASTICSEARCH_TYPE=
> elasticsearch,PIO_STORAGE_REPOSITORIES_METADATA_SOURCE=
> ELASTICSEARCH,PIO_STORAGE_REPOSITORIES_MODELDATA_SOURCE=
> HDFS,PIO_STORAGE_REPOSITORIES_EVENTDATA_NAME=pio_event,PIO_
> STORAGE_SOURCES_ELASTICSEARCH_CLUSTERNAME=pros_es_prod,PIO_
> FS_TMPDIR=/home/da_mcom_milan/.pio_store/tmp,PIO_STORAGE_
> REPOSITORIES_MODELDATA_NAME=pio_model,PIO_STORAGE_
> REPOSITORIES_EVENTDATA_SOURCE=HBASE,PIO_CONF_DIR=/data/
> PredictionIO-0.11.0-incubating/conf,PIO_STORAGE_
> SOURCES_ELASTICSEARCH_PORTS=9300,9300,9300
> da_mcom+ 17305 11.9 1.0 17214656 662920 pts/1 Sl 19:00 0:46
> /usr/java/default/bin/java -Dhdp.version=2.6.1.0-129 -cp
> /data/PredictionIO-0.11.0-incubating/conf/:/usr/hdp/2.6.
> 1.0-129/hbase/conf/:/usr/hdp/current/spark-client/conf/:/
> data/PredictionIO-0.11.0-incubating/vendors/spark/lib/
> spark-assembly-1.6.3.2.6.1.0-129-hadoop2.7.3.2.6.1.0-129.
> jar:/data/PredictionIO-0.11.0-incubating/vendors/spark/lib/
> datanucleus-api-jdo-3.2.6.jar:/data/PredictionIO-0.11.0-
> incubating/vendors/spark/lib/datanucleus-core-3.2.10.jar:/
> data/PredictionIO-0.11.0-incubating/vendors/spark/lib/
> datanucleus-rdbms-3.2.9.jar:/usr/hdp/current/hadoop-client/
> conf/:/usr/hdp/current/hadoop-client/lib/aws-java-sdk-s3-1.
> 10.6.jar:/usr/hdp/current/hadoop-client/lib/aws-java-
> sdk-core-1.10.6.jar:/usr/hdp/current/hadoop-client/lib/aws-java-sdk-kms-1.10.6.jar
> -Xms1g -Xmx1g -Dpio.log.dir=/home/da_mcom_milan org.apache.spark.deploy.SparkSubmit
> --conf spark.driver.extraClassPath=/data/PredictionIO-0.11.0-
> incubating/conf:/usr/hdp/2.6.1.0-129/hbase/conf --conf
> spark.driver.extraJavaOptions=-Dpio.log.dir=/home/da_mcom_milan --class
> org.apache.predictionio.workflow.CreateServer --jars
> file:/data/PredictionIO-0.11.0-incubating/engines/some-
> engine/target/scala-2.10/universal-recommender_2.10-0.6.0.jar,file:/data/
> PredictionIO-0.11.0-incubating/engines/some-engine/target/scala-2.10/
> universal-recommender-assembly-0.6.0-deps.jar,file:/
> data/PredictionIO-0.11.0-incubating/lib/spark/pio-data-
> elasticsearch1-assembly-0.11.0-incubating.jar,file:/data/
> PredictionIO-0.11.0-incubating/lib/spark/pio-data-hbase-assembly-0.11.0-
> incubating.jar,file:/data/PredictionIO-0.11.0-
> incubating/lib/spark/pio-data-hdfs-assembly-0.11.0-
> incubating.jar,file:/data/PredictionIO-0.11.0-
> incubating/lib/spark/pio-data-jdbc-assembly-0.11.0-
> incubating.jar,file:/data/PredictionIO-0.11.0-
> incubating/lib/spark/pio-data-localfs-assembly-0.11.0-incubating.jar
> --files file:/data/PredictionIO-0.11.0-incubating/conf/log4j.
> properties,file:/usr/hdp/2.6.1.0-129/hbase/conf/hbase-site.xml
> file:/data/PredictionIO-0.11.0-incubating/lib/pio-assembly-0.11.0-incubating.jar
> --engineInstanceId AWK1b5NndWI0xyRGBxWM --engine-variant
> file:/data/PredictionIO-0.11.0-incubating/engines/some-engine/engine.json
> --ip 0.0.0.0 --port *8000* --event-server-ip 0.0.0.0 --event-server-port
> 7070 --json-extractor Both --env PIO_STORAGE_SOURCES_HBASE_
> TYPE=hbase,PIO_ENV_LOADED=1,PIO_STORAGE_REPOSITORIES_
> METADATA_NAME=pio_meta,PIO_FS_BASEDIR=/home/da_mcom_milan/.
> pio_store,PIO_STORAGE_SOURCES_ELASTICSEARCH_HOSTS=mdc2pr068,
> mdc2pr069,mdc2pr070,PIO_STORAGE_SOURCES_HBASE_HOME=/
> usr/hdp/2.6.1.0-129/hbase,PIO_HOME=/data/PredictionIO-0.11.
> 0-incubating,PIO_FS_ENGINESDIR=/home/da_mcom_milan/.pio_store/engines,PIO_
> STORAGE_SOURCES_HDFS_TYPE=hdfs,PIO_STORAGE_SOURCES_HDFS_
> PATH=/tmp/models,PIO_STORAGE_SOURCES_ELASTICSEARCH_TYPE=
> elasticsearch,PIO_STORAGE_REPOSITORIES_METADATA_SOURCE=
> ELASTICSEARCH,PIO_STORAGE_REPOSITORIES_MODELDATA_SOURCE=
> HDFS,PIO_STORAGE_REPOSITORIES_EVENTDATA_NAME=pio_event,PIO_
> STORAGE_SOURCES_ELASTICSEARCH_CLUSTERNAME=pros_es_prod,PIO_
> FS_TMPDIR=/home/da_mcom_milan/.pio_store/tmp,PIO_STORAGE_
> REPOSITORIES_MODELDATA_NAME=pio_model,PIO_STORAGE_
> REPOSITORIES_EVENTDATA_SOURCE=HBASE,PIO_CONF_DIR=/data/
> PredictionIO-0.11.0-incubating/conf,PIO_STORAGE_
> SOURCES_ELASTICSEARCH_PORTS=9300,9300,9300
> da_mcom+ 18988 0.0 0.0 112652 984 pts/1 R+ 19:06 0:00 grep
> --color=auto *8000*
> da_mcom+ 48849 0.5 0.9 26473704 607720 ? Sl апр05 50:01 java -cp
> /data/PredictionIO-0.11.0-incubating/conf:/data/PredictionIO-0.11.0-
> incubating/plugins/*:/data/PredictionIO-0.11.0-
> incubating/lib/spark/*:/data/PredictionIO-0.11.0-
> incubating/lib/pio-assembly-0.11.0-incubating.jar:/usr/hdp/2.6.1.0-129/hbase/conf
> -Dpio.log.dir=/home/da_mcom_milan org.apache.predictionio.tools.console.Console
> deploy --port *8000* --pio-home /data/PredictionIO-0.11.0-incubating
> da_mcom+ 49016 1.1 1.4 19766916 967792 ? Sl апр05 102:52
> /usr/java/default/bin/java -Dhdp.version=2.6.1.0-129 -cp
> /data/PredictionIO-0.11.0-incubating/conf/:/usr/hdp/2.6.
> 1.0-129/hbase/conf/:/usr/hdp/current/spark-client/conf/:/
> data/PredictionIO-0.11.0-incubating/vendors/spark/lib/
> spark-assembly-1.6.3.2.6.1.0-129-hadoop2.7.3.2.6.1.0-129.
> jar:/data/PredictionIO-0.11.0-incubating/vendors/spark/lib/
> datanucleus-api-jdo-3.2.6.jar:/data/PredictionIO-0.11.0-
> incubating/vendors/spark/lib/datanucleus-core-3.2.10.jar:/
> data/PredictionIO-0.11.0-incubating/vendors/spark/lib/
> datanucleus-rdbms-3.2.9.jar:/usr/hdp/current/hadoop-client/
> conf/:/usr/hdp/current/hadoop-client/lib/aws-java-sdk-s3-1.
> 10.6.jar:/usr/hdp/current/hadoop-client/lib/aws-java-
> sdk-core-1.10.6.jar:/usr/hdp/current/hadoop-client/lib/aws-java-sdk-kms-1.10.6.jar
> -Xms1g -Xmx1g -Dpio.log.dir=/home/da_mcom_milan org.apache.spark.deploy.SparkSubmit
> --conf spark.driver.extraClassPath=/data/PredictionIO-0.11.0-
> incubating/conf:/usr/hdp/2.6.1.0-129/hbase/conf --conf
> spark.driver.extraJavaOptions=-Dpio.log.dir=/home/da_mcom_milan --class
> org.apache.predictionio.workflow.CreateServer --jars
> file:/data/PredictionIO-0.11.0-incubating/engines/some-
> engine/target/scala-2.10/universal-recommender_2.10-0.6.0.jar,file:/data/
> PredictionIO-0.11.0-incubating/engines/some-engine/target/scala-2.10/
> universal-recommender-assembly-0.6.0-deps.jar,file:/
> data/PredictionIO-0.11.0-incubating/lib/spark/pio-data-
> elasticsearch1-assembly-0.11.0-incubating.jar,file:/data/
> PredictionIO-0.11.0-incubating/lib/spark/pio-data-hbase-assembly-0.11.0-
> incubating.jar,file:/data/PredictionIO-0.11.0-
> incubating/lib/spark/pio-data-hdfs-assembly-0.11.0-
> incubating.jar,file:/data/PredictionIO-0.11.0-
> incubating/lib/spark/pio-data-jdbc-assembly-0.11.0-
> incubating.jar,file:/data/PredictionIO-0.11.0-
> incubating/lib/spark/pio-data-localfs-assembly-0.11.0-incubating.jar
> --files file:/data/PredictionIO-0.11.0-incubating/conf/log4j.
> properties,file:/usr/hdp/2.6.1.0-129/hbase/conf/hbase-site.xml
> file:/data/PredictionIO-0.11.0-incubating/lib/pio-assembly-0.11.0-incubating.jar
> --engineInstanceId AWKWiWsxdWI0xyRGjxPJ --engine-variant
> file:/data/PredictionIO-0.11.0-incubating/engines/some-engine/engine.json
> --ip 0.0.0.0 --port *8000* --event-server-ip 0.0.0.0 --event-server-port
> 7070 --json-extractor Both --env PIO_STORAGE_SOURCES_HBASE_
> TYPE=hbase,PIO_ENV_LOADED=1,PIO_STORAGE_REPOSITORIES_
> METADATA_NAME=pio_meta,PIO_FS_BASEDIR=/home/da_mcom_milan/.
> pio_store,PIO_STORAGE_SOURCES_ELASTICSEARCH_HOSTS=mdc2pr068,
> mdc2pr069,mdc2pr070,PIO_STORAGE_SOURCES_HBASE_HOME=/
> usr/hdp/2.6.1.0-129/hbase,PIO_HOME=/data/PredictionIO-0.11.
> 0-incubating,PIO_FS_ENGINESDIR=/home/da_mcom_milan/.pio_store/engines,PIO_
> STORAGE_SOURCES_HDFS_TYPE=hdfs,PIO_STORAGE_SOURCES_HDFS_
> PATH=/tmp/models,PIO_STORAGE_SOURCES_ELASTICSEARCH_TYPE=
> elasticsearch,PIO_STORAGE_REPOSITORIES_METADATA_SOURCE=
> ELASTICSEARCH,PIO_STORAGE_REPOSITORIES_MODELDATA_SOURCE=
> HDFS,PIO_STORAGE_REPOSITORIES_EVENTDATA_NAME=pio_event,PIO_
> STORAGE_SOURCES_ELASTICSEARCH_CLUSTERNAME=pros_es_prod,PIO_
> FS_TMPDIR=/home/da_mcom_milan/.pio_store/tmp,PIO_STORAGE_
> REPOSITORIES_MODELDATA_NAME=pio_model,PIO_STORAGE_
> REPOSITORIES_EVENTDATA_SOURCE=HBASE,PIO_CONF_DIR=/data/
> PredictionIO-0.11.0-incubating/conf,PIO_STORAGE_
> SOURCES_ELASTICSEARCH_PORTS=9300,9300,9300
>
>
>
> On Wed, Apr 11, 2018 at 3:39 PM, Donald Szeto <do...@apache.org> wrote:
>
>> Are you able to find any occurrence of port 8004? The excerpt you showed
>> was about port 8000.
>>
>> On Wed, Apr 11, 2018 at 3:18 PM, George Yarish <gy...@griddynamics.com>
>> wrote:
>>
>>> Hi Donald
>>>
>>> Thank you for the answer.
>>> it looks fine to me, but processes still alive:
>>>
>>> 2018-04-11 13:00:25,331 INFO org.apache.predictionio.workflow.MasterActor
>>> [pio-server-akka.actor.default-dispatcher-5] - Undeploying any existing
>>> engine instance at http://0.0.0.0:8000
>>> 2018-04-11 13:00:26,467 INFO org.apache.predictionio.workflow.MasterActor
>>> [pio-server-akka.actor.default-dispatcher-9] - Stop server command
>>> received.
>>> 2018-04-11 13:00:26,468 INFO org.apache.predictionio.workflow.MasterActor
>>> [pio-server-akka.actor.default-dispatcher-9] - Server is shutting down.
>>> 2018-04-11 13:00:26,617 INFO spray.can.server.HttpListener
>>> [pio-server-akka.actor.default-dispatcher-2] - Bound to /0.0.0.0:8000
>>> 2018-04-11 13:00:26,620 INFO org.apache.predictionio.workflow.MasterActor
>>> [pio-server-akka.actor.default-dispatcher-4] - Engine is deployed and
>>> running. Engine API is live at http://0.0.0.0:8000
>>>
>>> Thank you,
>>> George
>>>
>>>
>>> On Wed, Apr 11, 2018 at 11:48 AM, Donald Szeto <do...@apache.org>
>>> wrote:
>>>
>>>> Hi George,
>>>>
>>>> Mind checking the content of pio.log at where you run `pio deploy`? It
>>>> should contain some messages regarding undeploying existing engine servers
>>>> on the same port.
>>>>
>>>> Regards,
>>>> Donald
>>>>
>>>> On Thu, Apr 5, 2018 at 6:43 PM, George Yarish <gyarish@griddynamics.com
>>>> > wrote:
>>>>
>>>>> Hi everyone,
>>>>>
>>>>> I faced a strange situation recently.
>>>>> One of our engines has frequently old processes after daily
>>>>> redeployment.
>>>>>
>>>>> We used to redeploy by crontab rule that looks like:
>>>>> nohup pio deploy --port 8004 >> ${DEPLOY_LOG_FILE} 2>&1 &
>>>>>
>>>>>
>>>>> This is output of ps aux command:
>>>>>
>>>>> [da_mcom_milan@mdc2pr113 ~]$ ps aux | grep 8004
>>>>>
>>>>> da_mcom+ 7854 0.0 0.0 112660 996 pts/1 S+ 21:34 0:00 grep
>>>>> --color=auto *8004*
>>>>>
>>>>> 1. da_mcom+ 30932 0.6 0.9 26473632 595932 ? Sl 01:57 7:51
>>>>> java -cp /data/PredictionIO-0.11.0-incubating/conf:/data/PredictionIO
>>>>> -0.11.0-incubating/plugins/*:/data/PredictionIO-0.11.0-incub
>>>>> ating/lib/spark/*:/data/PredictionIO-0.11.0-incubating/lib/p
>>>>> io-assembly-0.11.0-incubating.jar:/usr/hdp/2.5.3.0-37/hadoop
>>>>> /conf:/data/PredictionIO/vendors/hbase/conf
>>>>> -Dpio.log.dir=/home/da_mcom_milan org.apache.predictionio.tools.console.Console
>>>>> deploy --port *8004* --pio-home /data/PredictionIO-0.11.0-incubating
>>>>>
>>>>> 2. da_mcom+ 31097 1.2 1.2 17065124 822596 ? Sl 01:57 14:48
>>>>> /usr/java/default/bin/java -Dhdp.version=2.5.3.0-37 -cp
>>>>> /data/PredictionIO-0.11.0-incubating/conf/:/usr/hdp/2.5.3.0-
>>>>> 37/hadoop/conf/:/data/PredictionIO/vendors/hbase/conf/:/usr/
>>>>> hdp/current/spark-client/conf/:/data/PredictionIO/vendors/sp
>>>>> ark/lib/spark-assembly-1.6.2.2.5.3.0-37-hadoop2.7.3.2.5.3.0-
>>>>> 37.jar:/data/PredictionIO/vendors/spark/lib/datanucleus-api-
>>>>> jdo-3.2.6.jar:/data/PredictionIO/vendors/spark/lib/datanucle
>>>>> us-core-3.2.10.jar:/data/PredictionIO/vendors/spark/lib/data
>>>>> nucleus-rdbms-3.2.9.jar:/usr/hdp/2.5.3.0-37/hadoop/conf/:/us
>>>>> r/hdp/current/hadoop-client/lib/aws-java-sdk-s3-1.10.6.jar:/
>>>>> usr/hdp/current/hadoop-client/lib/aws-java-sdk-core-1.10.6.j
>>>>> ar:/usr/hdp/current/hadoop-client/lib/aws-java-sdk-kms-1.10.6.jar
>>>>> -Xms1g -Xmx1g -Dpio.log.dir=/home/da_mcom_milan
>>>>> org.apache.spark.deploy.SparkSubmit --conf
>>>>> spark.driver.extraClassPath=/data/PredictionIO-0.11.0-incuba
>>>>> ting/conf:/usr/hdp/2.5.3.0-37/hadoop/conf:/data/PredictionIO/vendors/hbase/conf
>>>>> --conf spark.driver.extraJavaOptions=-Dpio.log.dir=/home/da_mcom_milan
>>>>> --class org.apache.predictionio.workflow.CreateServer --jars
>>>>> file:/data/PredictionIO/some-engine/target/scala-2.10/univer
>>>>> sal-recommender_2.10-0.6.0.jar,file:/data/PredictionIO/some-engine
>>>>> /target/scala-2.10/universal-recommender-assembly-0.6.
>>>>> 0-deps.jar,file:/data/PredictionIO-0.11.0-incubating/lib/spa
>>>>> rk/pio-data-elasticsearch1-assembly-0.11.0-incubating.jar,fi
>>>>> le:/data/PredictionIO-0.11.0-incubating/lib/spark/pio-data-h
>>>>> base-assembly-0.11.0-incubating.jar,file:/data/PredictionIO-
>>>>> 0.11.0-incubating/lib/spark/pio-data-hdfs-assembly-0.11.0-in
>>>>> cubating.jar,file:/data/PredictionIO-0.11.0-incubating/lib/
>>>>> spark/pio-data-jdbc-assembly-0.11.0-incubating.jar,file:/
>>>>> data/PredictionIO-0.11.0-incubating/lib/spark/pio-data-
>>>>> localfs-assembly-0.11.0-incubating.jar --files
>>>>> file:/data/PredictionIO-0.11.0-incubating/conf/log4j.propert
>>>>> ies,file:/usr/hdp/2.5.3.0-37/hadoop/conf/core-site.xml,file:
>>>>> /data/PredictionIO/vendors/hbase/conf/hbase-site.xml
>>>>> file:/data/PredictionIO-0.11.0-incubating/lib/pio-assembly-0.11.0-incubating.jar
>>>>> --engineInstanceId AWKSaLQrSe_zVbacBHWI --engine-variant
>>>>> file:/data/PredictionIO/some-engine/engine.json --ip 0.0.0.0 --port
>>>>> *8004* --event-server-ip 0.0.0.0 --event-server-port 7070
>>>>> --json-extractor Both --env PIO_STORAGE_SOURCES_HBASE_TYPE
>>>>> =hbase,PIO_ENV_LOADED=1,PIO_STORAGE_REPOSITORIES_METADATA_NA
>>>>> ME=pio_meta,PIO_FS_BASEDIR=/home/da_mcom_milan/.pio_store,PI
>>>>> O_STORAGE_SOURCES_ELASTICSEARCH_HOSTS=mdc2vra176,PIO_STORAGE
>>>>> _SOURCES_HBASE_HOME=/data/PredictionIO/vendors/hbase,PIO_HOM
>>>>> E=/data/PredictionIO-0.11.0-incubating,PIO_FS_ENGINESDIR=/ho
>>>>> me/da_mcom_milan/.pio_store/engines,PIO_STORAGE_SOURCES_LOCA
>>>>> LFS_PATH=/home/da_mcom_milan/.pio_store/models,PIO_STORAGE_S
>>>>> OURCES_ELASTICSEARCH_TYPE=elasticsearch,PIO_STORAGE_REPOSITO
>>>>> RIES_METADATA_SOURCE=ELASTICSEARCH,PIO_STORAGE_REPOSITORIES_
>>>>> MODELDATA_SOURCE=LOCALFS,PIO_STORAGE_REPOSITORIES_EVENTDATA
>>>>> _NAME=pio_event,PIO_STORAGE_SOURCES_ELASTICSEARCH_CLUSTERN
>>>>> AME=pros-dev,PIO_FS_TMPDIR=/home/da_mcom_milan/.pio_store/
>>>>> tmp,PIO_STORAGE_REPOSITORIES_MODELDATA_NAME=pio_model,PIO_
>>>>> STORAGE_REPOSITORIES_EVENTDATA_SOURCE=HBASE,PIO_
>>>>> CONF_DIR=/data/PredictionIO-0.11.0-incubating/conf,PIO_
>>>>> STORAGE_SOURCES_ELASTICSEARCH_PORTS=9300,PIO_STORAGE_
>>>>> SOURCES_LOCALFS_TYPE=localfs
>>>>>
>>>>> 3. da_mcom+ 32505 0.6 0.8 26473632 581164 ? Sl 02:00 7:54
>>>>> java -cp /data/PredictionIO-0.11.0-incubating/conf:/data/PredictionIO
>>>>> -0.11.0-incubating/plugins/*:/data/PredictionIO-0.11.0-incub
>>>>> ating/lib/spark/*:/data/PredictionIO-0.11.0-incubating/lib/p
>>>>> io-assembly-0.11.0-incubating.jar:/usr/hdp/2.5.3.0-37/hadoop
>>>>> /conf:/data/PredictionIO/vendors/hbase/conf
>>>>> -Dpio.log.dir=/home/da_mcom_milan org.apache.predictionio.tools.console.Console
>>>>> deploy --port *8004* --pio-home /data/PredictionIO-0.11.0-incubating
>>>>>
>>>>> 4. da_mcom+ 32671 1.4 1.3 19757488 881448 ? Sl 02:00 16:27
>>>>> /usr/java/default/bin/java -Dhdp.version=2.5.3.0-37 -cp
>>>>> /data/PredictionIO-0.11.0-incubating/conf/:/usr/hdp/2.5.3.0-
>>>>> 37/hadoop/conf/:/data/PredictionIO/vendors/hbase/conf/:/usr/
>>>>> hdp/current/spark-client/conf/:/data/PredictionIO/vendors/sp
>>>>> ark/lib/spark-assembly-1.6.2.2.5.3.0-37-hadoop2.7.3.2.5.3.0-
>>>>> 37.jar:/data/PredictionIO/vendors/spark/lib/datanucleus-api-
>>>>> jdo-3.2.6.jar:/data/PredictionIO/vendors/spark/lib/datanucle
>>>>> us-core-3.2.10.jar:/data/PredictionIO/vendors/spark/lib/data
>>>>> nucleus-rdbms-3.2.9.jar:/usr/hdp/2.5.3.0-37/hadoop/conf/:/us
>>>>> r/hdp/current/hadoop-client/lib/aws-java-sdk-s3-1.10.6.jar:/
>>>>> usr/hdp/current/hadoop-client/lib/aws-java-sdk-core-1.10.6.j
>>>>> ar:/usr/hdp/current/hadoop-client/lib/aws-java-sdk-kms-1.10.6.jar
>>>>> -Xms1g -Xmx1g -Dpio.log.dir=/home/da_mcom_milan
>>>>> org.apache.spark.deploy.SparkSubmit --conf
>>>>> spark.driver.extraClassPath=/data/PredictionIO-0.11.0-incuba
>>>>> ting/conf:/usr/hdp/2.5.3.0-37/hadoop/conf:/data/PredictionIO/vendors/hbase/conf
>>>>> --conf spark.driver.extraJavaOptions=-Dpio.log.dir=/home/da_mcom_milan
>>>>> --class org.apache.predictionio.workflow.CreateServer --jars
>>>>> file:/data/PredictionIO/some-engine/target/scala-2.10/univer
>>>>> sal-recommender_2.10-0.6.0.jar,file:/data/PredictionIO/some-engine
>>>>> /target/scala-2.10/universal-recommender-assembly-0.6.
>>>>> 0-deps.jar,file:/data/PredictionIO-0.11.0-incubating/lib/spa
>>>>> rk/pio-data-elasticsearch1-assembly-0.11.0-incubating.jar,fi
>>>>> le:/data/PredictionIO-0.11.0-incubating/lib/spark/pio-data-h
>>>>> base-assembly-0.11.0-incubating.jar,file:/data/PredictionIO-
>>>>> 0.11.0-incubating/lib/spark/pio-data-hdfs-assembly-0.11.0-in
>>>>> cubating.jar,file:/data/PredictionIO-0.11.0-incubating/lib/
>>>>> spark/pio-data-jdbc-assembly-0.11.0-incubating.jar,file:/
>>>>> data/PredictionIO-0.11.0-incubating/lib/spark/pio-data-
>>>>> localfs-assembly-0.11.0-incubating.jar --files
>>>>> file:/data/PredictionIO-0.11.0-incubating/conf/log4j.propert
>>>>> ies,file:/usr/hdp/2.5.3.0-37/hadoop/conf/core-site.xml,file:
>>>>> /data/PredictionIO/vendors/hbase/conf/hbase-site.xml
>>>>> file:/data/PredictionIO-0.11.0-incubating/lib/pio-assembly-0.11.0-incubating.jar
>>>>> --engineInstanceId AWKSaLQrSe_zVbacBHWI --engine-variant
>>>>> file:/data/PredictionIO/some-engine/engine.json --ip 0.0.0.0 --port
>>>>> *8004* --event-server-ip 0.0.0.0 --event-server-port 7070
>>>>> --json-extractor Both --env PIO_STORAGE_SOURCES_HBASE_TYPE
>>>>> =hbase,PIO_ENV_LOADED=1,PIO_STORAGE_REPOSITORIES_METADATA_NA
>>>>> ME=pio_meta,PIO_FS_BASEDIR=/home/da_mcom_milan/.pio_store,PI
>>>>> O_STORAGE_SOURCES_ELASTICSEARCH_HOSTS=mdc2vra176,PIO_STORAGE
>>>>> _SOURCES_HBASE_HOME=/data/PredictionIO/vendors/hbase,PIO_HOM
>>>>> E=/data/PredictionIO-0.11.0-incubating,PIO_FS_ENGINESDIR=/ho
>>>>> me/da_mcom_milan/.pio_store/engines,PIO_STORAGE_SOURCES_LOCA
>>>>> LFS_PATH=/home/da_mcom_milan/.pio_store/models,PIO_STORAGE_S
>>>>> OURCES_ELASTICSEARCH_TYPE=elasticsearch,PIO_STORAGE_REPOSITO
>>>>> RIES_METADATA_SOURCE=ELASTICSEARCH,PIO_STORAGE_REPOSITORIES_
>>>>> MODELDATA_SOURCE=LOCALFS,PIO_STORAGE_REPOSITORIES_EVENTDATA
>>>>> _NAME=pio_event,PIO_STORAGE_SOURCES_ELASTICSEARCH_CLUSTERN
>>>>> AME=pros-dev,PIO_FS_TMPDIR=/home/da_mcom_milan/.pio_store/
>>>>> tmp,PIO_STORAGE_REPOSITORIES_MODELDATA_NAME=pio_model,PIO_
>>>>> STORAGE_REPOSITORIES_EVENTDATA_SOURCE=HBASE,PIO_
>>>>> CONF_DIR=/data/PredictionIO-0.11.0-incubating/conf,PIO_
>>>>> STORAGE_SOURCES_ELASTICSEARCH_PORTS=9300,PIO_STORAGE_
>>>>> SOURCES_LOCALFS_TYPE=localfs
>>>>>
>>>>> For some reasons, processes 30932 and 31097 still alive and process
>>>>> listened by 32671
>>>>> Any ideas?
>>>>>
>>>>> Thank you,
>>>>> George
>>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>>
>>> George Yarish, Java Developer
>>>
>>> Grid Dynamics
>>>
>>> 197101, Rentgena Str., 5A, St.Petersburg, Russia
>>>
>>> Cell: +7 950 030-1941
>>>
>>> Read Grid Dynamics' Tech Blog
>>> <http://blog.griddynamics.com/?utm_campaign=Big%20Data%20Blog%20social%20media%20promotion&utm_medium=CTA&utm_source=Email>
>>>
>>>
>>>
>>
>
>
> --
>
> George Yarish, Java Developer
>
> Grid Dynamics
>
> 197101, Rentgena Str., 5A, St.Petersburg, Russia
>
> Cell: +7 950 030-1941
>
> Read Grid Dynamics' Tech Blog
> <http://blog.griddynamics.com/?utm_campaign=Big%20Data%20Blog%20social%20media%20promotion&utm_medium=CTA&utm_source=Email>
>
>
>
Re: PIO deploy: old processes still live after redeployment
Posted by George Yarish <gy...@griddynamics.com>.
I'm sorry for that, I have a few environments in which that problem
reproduced. That was the one with *8000* port.
It's pretty the same:
$ ps aux | grep *8000* | wc -l
13
and processes looks like:
da_mcom+ 17028 0.5 0.9 26473704 603220 ? Sl апр06 42:05 java -cp
/data/PredictionIO-0.11.0-incubating/conf:/data/PredictionIO-0.11.0-incubating/plugins/*:/data/PredictionIO-0.11.0-incubating/lib/spark/*:/data/PredictionIO-0.11.0-incubating/lib/pio-assembly-0.11.0-incubating.jar:/usr/hdp/2.6.1.0-129/hbase/conf
-Dpio.log.dir=/home/da_mcom_milan
org.apache.predictionio.tools.console.Console deploy --port *8000*
--pio-home /data/PredictionIO-0.11.0-incubating
da_mcom+ 17269 1.1 1.5 19765888 996524 ? Sl апр06 87:45
/usr/java/default/bin/java -Dhdp.version=2.6.1.0-129 -cp
/data/PredictionIO-0.11.0-incubating/conf/:/usr/hdp/2.6.1.0-129/hbase/conf/:/usr/hdp/current/spark-client/conf/:/data/PredictionIO-0.11.0-incubating/vendors/spark/lib/spark-assembly-1.6.3.2.6.1.0-129-hadoop2.7.3.2.6.1.0-129.jar:/data/PredictionIO-0.11.0-incubating/vendors/spark/lib/datanucleus-api-jdo-3.2.6.jar:/data/PredictionIO-0.11.0-incubating/vendors/spark/lib/datanucleus-core-3.2.10.jar:/data/PredictionIO-0.11.0-incubating/vendors/spark/lib/datanucleus-rdbms-3.2.9.jar:/usr/hdp/current/hadoop-client/conf/:/usr/hdp/current/hadoop-client/lib/aws-java-sdk-s3-1.10.6.jar:/usr/hdp/current/hadoop-client/lib/aws-java-sdk-core-1.10.6.jar:/usr/hdp/current/hadoop-client/lib/aws-java-sdk-kms-1.10.6.jar
-Xms1g -Xmx1g -Dpio.log.dir=/home/da_mcom_milan
org.apache.spark.deploy.SparkSubmit --conf
spark.driver.extraClassPath=/data/PredictionIO-0.11.0-incubating/conf:/usr/hdp/2.6.1.0-129/hbase/conf
--conf spark.driver.extraJavaOptions=-Dpio.log.dir=/home/da_mcom_milan
--class org.apache.predictionio.workflow.CreateServer --jars
file:/data/PredictionIO-0.11.0-incubating/engines/some-engine/target/scala-2.10/universal-recommender_2.10-0.6.0.jar,file:/data/PredictionIO-0.11.0-incubating/engines/some-engine/target/scala-2.10/universal-recommender-assembly-0.6.0-deps.jar,file:/data/PredictionIO-0.11.0-incubating/lib/spark/pio-data-elasticsearch1-assembly-0.11.0-incubating.jar,file:/data/PredictionIO-0.11.0-incubating/lib/spark/pio-data-hbase-assembly-0.11.0-incubating.jar,file:/data/PredictionIO-0.11.0-incubating/lib/spark/pio-data-hdfs-assembly-0.11.0-incubating.jar,file:/data/PredictionIO-0.11.0-incubating/lib/spark/pio-data-jdbc-assembly-0.11.0-incubating.jar,file:/data/PredictionIO-0.11.0-incubating/lib/spark/pio-data-localfs-assembly-0.11.0-incubating.jar
--files
file:/data/PredictionIO-0.11.0-incubating/conf/log4j.properties,file:/usr/hdp/2.6.1.0-129/hbase/conf/hbase-site.xml
file:/data/PredictionIO-0.11.0-incubating/lib/pio-assembly-0.11.0-incubating.jar
--engineInstanceId AWKbr8hAdWI0xyRGjxPY --engine-variant
file:/data/PredictionIO-0.11.0-incubating/engines/some-engine/engine.json
--ip 0.0.0.0 --port *8000* --event-server-ip 0.0.0.0 --event-server-port
7070 --json-extractor Both --env
PIO_STORAGE_SOURCES_HBASE_TYPE=hbase,PIO_ENV_LOADED=1,PIO_STORAGE_REPOSITORIES_METADATA_NAME=pio_meta,PIO_FS_BASEDIR=/home/da_mcom_milan/.pio_store,PIO_STORAGE_SOURCES_ELASTICSEARCH_HOSTS=mdc2pr068,mdc2pr069,mdc2pr070,PIO_STORAGE_SOURCES_HBASE_HOME=/usr/hdp/2.6.1.0-129/hbase,PIO_HOME=/data/PredictionIO-0.11.0-incubating,PIO_FS_ENGINESDIR=/home/da_mcom_milan/.pio_store/engines,PIO_STORAGE_SOURCES_HDFS_TYPE=hdfs,PIO_STORAGE_SOURCES_HDFS_PATH=/tmp/models,PIO_STORAGE_SOURCES_ELASTICSEARCH_TYPE=elasticsearch,PIO_STORAGE_REPOSITORIES_METADATA_SOURCE=ELASTICSEARCH,PIO_STORAGE_REPOSITORIES_MODELDATA_SOURCE=HDFS,PIO_STORAGE_REPOSITORIES_EVENTDATA_NAME=pio_event,PIO_STORAGE_SOURCES_ELASTICSEARCH_CLUSTERNAME=pros_es_prod,PIO_FS_TMPDIR=/home/da_mcom_milan/.pio_store/tmp,PIO_STORAGE_REPOSITORIES_MODELDATA_NAME=pio_model,PIO_STORAGE_REPOSITORIES_EVENTDATA_SOURCE=HBASE,PIO_CONF_DIR=/data/PredictionIO-0.11.0-incubating/conf,PIO_STORAGE_SOURCES_ELASTICSEARCH_PORTS=9300,9300,9300
da_mcom+ 17305 11.9 1.0 17214656 662920 pts/1 Sl 19:00 0:46
/usr/java/default/bin/java -Dhdp.version=2.6.1.0-129 -cp
/data/PredictionIO-0.11.0-incubating/conf/:/usr/hdp/2.6.1.0-129/hbase/conf/:/usr/hdp/current/spark-client/conf/:/data/PredictionIO-0.11.0-incubating/vendors/spark/lib/spark-assembly-1.6.3.2.6.1.0-129-hadoop2.7.3.2.6.1.0-129.jar:/data/PredictionIO-0.11.0-incubating/vendors/spark/lib/datanucleus-api-jdo-3.2.6.jar:/data/PredictionIO-0.11.0-incubating/vendors/spark/lib/datanucleus-core-3.2.10.jar:/data/PredictionIO-0.11.0-incubating/vendors/spark/lib/datanucleus-rdbms-3.2.9.jar:/usr/hdp/current/hadoop-client/conf/:/usr/hdp/current/hadoop-client/lib/aws-java-sdk-s3-1.10.6.jar:/usr/hdp/current/hadoop-client/lib/aws-java-sdk-core-1.10.6.jar:/usr/hdp/current/hadoop-client/lib/aws-java-sdk-kms-1.10.6.jar
-Xms1g -Xmx1g -Dpio.log.dir=/home/da_mcom_milan
org.apache.spark.deploy.SparkSubmit --conf
spark.driver.extraClassPath=/data/PredictionIO-0.11.0-incubating/conf:/usr/hdp/2.6.1.0-129/hbase/conf
--conf spark.driver.extraJavaOptions=-Dpio.log.dir=/home/da_mcom_milan
--class org.apache.predictionio.workflow.CreateServer --jars
file:/data/PredictionIO-0.11.0-incubating/engines/some-engine/target/scala-2.10/universal-recommender_2.10-0.6.0.jar,file:/data/PredictionIO-0.11.0-incubating/engines/some-engine/target/scala-2.10/universal-recommender-assembly-0.6.0-deps.jar,file:/data/PredictionIO-0.11.0-incubating/lib/spark/pio-data-elasticsearch1-assembly-0.11.0-incubating.jar,file:/data/PredictionIO-0.11.0-incubating/lib/spark/pio-data-hbase-assembly-0.11.0-incubating.jar,file:/data/PredictionIO-0.11.0-incubating/lib/spark/pio-data-hdfs-assembly-0.11.0-incubating.jar,file:/data/PredictionIO-0.11.0-incubating/lib/spark/pio-data-jdbc-assembly-0.11.0-incubating.jar,file:/data/PredictionIO-0.11.0-incubating/lib/spark/pio-data-localfs-assembly-0.11.0-incubating.jar
--files
file:/data/PredictionIO-0.11.0-incubating/conf/log4j.properties,file:/usr/hdp/2.6.1.0-129/hbase/conf/hbase-site.xml
file:/data/PredictionIO-0.11.0-incubating/lib/pio-assembly-0.11.0-incubating.jar
--engineInstanceId AWK1b5NndWI0xyRGBxWM --engine-variant
file:/data/PredictionIO-0.11.0-incubating/engines/some-engine/engine.json
--ip 0.0.0.0 --port *8000* --event-server-ip 0.0.0.0 --event-server-port
7070 --json-extractor Both --env
PIO_STORAGE_SOURCES_HBASE_TYPE=hbase,PIO_ENV_LOADED=1,PIO_STORAGE_REPOSITORIES_METADATA_NAME=pio_meta,PIO_FS_BASEDIR=/home/da_mcom_milan/.pio_store,PIO_STORAGE_SOURCES_ELASTICSEARCH_HOSTS=mdc2pr068,mdc2pr069,mdc2pr070,PIO_STORAGE_SOURCES_HBASE_HOME=/usr/hdp/2.6.1.0-129/hbase,PIO_HOME=/data/PredictionIO-0.11.0-incubating,PIO_FS_ENGINESDIR=/home/da_mcom_milan/.pio_store/engines,PIO_STORAGE_SOURCES_HDFS_TYPE=hdfs,PIO_STORAGE_SOURCES_HDFS_PATH=/tmp/models,PIO_STORAGE_SOURCES_ELASTICSEARCH_TYPE=elasticsearch,PIO_STORAGE_REPOSITORIES_METADATA_SOURCE=ELASTICSEARCH,PIO_STORAGE_REPOSITORIES_MODELDATA_SOURCE=HDFS,PIO_STORAGE_REPOSITORIES_EVENTDATA_NAME=pio_event,PIO_STORAGE_SOURCES_ELASTICSEARCH_CLUSTERNAME=pros_es_prod,PIO_FS_TMPDIR=/home/da_mcom_milan/.pio_store/tmp,PIO_STORAGE_REPOSITORIES_MODELDATA_NAME=pio_model,PIO_STORAGE_REPOSITORIES_EVENTDATA_SOURCE=HBASE,PIO_CONF_DIR=/data/PredictionIO-0.11.0-incubating/conf,PIO_STORAGE_SOURCES_ELASTICSEARCH_PORTS=9300,9300,9300
da_mcom+ 18988 0.0 0.0 112652 984 pts/1 R+ 19:06 0:00 grep
--color=auto *8000*
da_mcom+ 48849 0.5 0.9 26473704 607720 ? Sl апр05 50:01 java -cp
/data/PredictionIO-0.11.0-incubating/conf:/data/PredictionIO-0.11.0-incubating/plugins/*:/data/PredictionIO-0.11.0-incubating/lib/spark/*:/data/PredictionIO-0.11.0-incubating/lib/pio-assembly-0.11.0-incubating.jar:/usr/hdp/2.6.1.0-129/hbase/conf
-Dpio.log.dir=/home/da_mcom_milan
org.apache.predictionio.tools.console.Console deploy --port *8000*
--pio-home /data/PredictionIO-0.11.0-incubating
da_mcom+ 49016 1.1 1.4 19766916 967792 ? Sl апр05 102:52
/usr/java/default/bin/java -Dhdp.version=2.6.1.0-129 -cp
/data/PredictionIO-0.11.0-incubating/conf/:/usr/hdp/2.6.1.0-129/hbase/conf/:/usr/hdp/current/spark-client/conf/:/data/PredictionIO-0.11.0-incubating/vendors/spark/lib/spark-assembly-1.6.3.2.6.1.0-129-hadoop2.7.3.2.6.1.0-129.jar:/data/PredictionIO-0.11.0-incubating/vendors/spark/lib/datanucleus-api-jdo-3.2.6.jar:/data/PredictionIO-0.11.0-incubating/vendors/spark/lib/datanucleus-core-3.2.10.jar:/data/PredictionIO-0.11.0-incubating/vendors/spark/lib/datanucleus-rdbms-3.2.9.jar:/usr/hdp/current/hadoop-client/conf/:/usr/hdp/current/hadoop-client/lib/aws-java-sdk-s3-1.10.6.jar:/usr/hdp/current/hadoop-client/lib/aws-java-sdk-core-1.10.6.jar:/usr/hdp/current/hadoop-client/lib/aws-java-sdk-kms-1.10.6.jar
-Xms1g -Xmx1g -Dpio.log.dir=/home/da_mcom_milan
org.apache.spark.deploy.SparkSubmit --conf
spark.driver.extraClassPath=/data/PredictionIO-0.11.0-incubating/conf:/usr/hdp/2.6.1.0-129/hbase/conf
--conf spark.driver.extraJavaOptions=-Dpio.log.dir=/home/da_mcom_milan
--class org.apache.predictionio.workflow.CreateServer --jars
file:/data/PredictionIO-0.11.0-incubating/engines/some-engine/target/scala-2.10/universal-recommender_2.10-0.6.0.jar,file:/data/PredictionIO-0.11.0-incubating/engines/some-engine/target/scala-2.10/universal-recommender-assembly-0.6.0-deps.jar,file:/data/PredictionIO-0.11.0-incubating/lib/spark/pio-data-elasticsearch1-assembly-0.11.0-incubating.jar,file:/data/PredictionIO-0.11.0-incubating/lib/spark/pio-data-hbase-assembly-0.11.0-incubating.jar,file:/data/PredictionIO-0.11.0-incubating/lib/spark/pio-data-hdfs-assembly-0.11.0-incubating.jar,file:/data/PredictionIO-0.11.0-incubating/lib/spark/pio-data-jdbc-assembly-0.11.0-incubating.jar,file:/data/PredictionIO-0.11.0-incubating/lib/spark/pio-data-localfs-assembly-0.11.0-incubating.jar
--files
file:/data/PredictionIO-0.11.0-incubating/conf/log4j.properties,file:/usr/hdp/2.6.1.0-129/hbase/conf/hbase-site.xml
file:/data/PredictionIO-0.11.0-incubating/lib/pio-assembly-0.11.0-incubating.jar
--engineInstanceId AWKWiWsxdWI0xyRGjxPJ --engine-variant
file:/data/PredictionIO-0.11.0-incubating/engines/some-engine/engine.json
--ip 0.0.0.0 --port *8000* --event-server-ip 0.0.0.0 --event-server-port
7070 --json-extractor Both --env
PIO_STORAGE_SOURCES_HBASE_TYPE=hbase,PIO_ENV_LOADED=1,PIO_STORAGE_REPOSITORIES_METADATA_NAME=pio_meta,PIO_FS_BASEDIR=/home/da_mcom_milan/.pio_store,PIO_STORAGE_SOURCES_ELASTICSEARCH_HOSTS=mdc2pr068,mdc2pr069,mdc2pr070,PIO_STORAGE_SOURCES_HBASE_HOME=/usr/hdp/2.6.1.0-129/hbase,PIO_HOME=/data/PredictionIO-0.11.0-incubating,PIO_FS_ENGINESDIR=/home/da_mcom_milan/.pio_store/engines,PIO_STORAGE_SOURCES_HDFS_TYPE=hdfs,PIO_STORAGE_SOURCES_HDFS_PATH=/tmp/models,PIO_STORAGE_SOURCES_ELASTICSEARCH_TYPE=elasticsearch,PIO_STORAGE_REPOSITORIES_METADATA_SOURCE=ELASTICSEARCH,PIO_STORAGE_REPOSITORIES_MODELDATA_SOURCE=HDFS,PIO_STORAGE_REPOSITORIES_EVENTDATA_NAME=pio_event,PIO_STORAGE_SOURCES_ELASTICSEARCH_CLUSTERNAME=pros_es_prod,PIO_FS_TMPDIR=/home/da_mcom_milan/.pio_store/tmp,PIO_STORAGE_REPOSITORIES_MODELDATA_NAME=pio_model,PIO_STORAGE_REPOSITORIES_EVENTDATA_SOURCE=HBASE,PIO_CONF_DIR=/data/PredictionIO-0.11.0-incubating/conf,PIO_STORAGE_SOURCES_ELASTICSEARCH_PORTS=9300,9300,9300
On Wed, Apr 11, 2018 at 3:39 PM, Donald Szeto <do...@apache.org> wrote:
> Are you able to find any occurrence of port 8004? The excerpt you showed
> was about port 8000.
>
> On Wed, Apr 11, 2018 at 3:18 PM, George Yarish <gy...@griddynamics.com>
> wrote:
>
>> Hi Donald
>>
>> Thank you for the answer.
>> it looks fine to me, but processes still alive:
>>
>> 2018-04-11 13:00:25,331 INFO org.apache.predictionio.workflow.MasterActor
>> [pio-server-akka.actor.default-dispatcher-5] - Undeploying any existing
>> engine instance at http://0.0.0.0:8000
>> 2018-04-11 13:00:26,467 INFO org.apache.predictionio.workflow.MasterActor
>> [pio-server-akka.actor.default-dispatcher-9] - Stop server command
>> received.
>> 2018-04-11 13:00:26,468 INFO org.apache.predictionio.workflow.MasterActor
>> [pio-server-akka.actor.default-dispatcher-9] - Server is shutting down.
>> 2018-04-11 13:00:26,617 INFO spray.can.server.HttpListener
>> [pio-server-akka.actor.default-dispatcher-2] - Bound to /0.0.0.0:8000
>> 2018-04-11 13:00:26,620 INFO org.apache.predictionio.workflow.MasterActor
>> [pio-server-akka.actor.default-dispatcher-4] - Engine is deployed and
>> running. Engine API is live at http://0.0.0.0:8000
>>
>> Thank you,
>> George
>>
>>
>> On Wed, Apr 11, 2018 at 11:48 AM, Donald Szeto <do...@apache.org> wrote:
>>
>>> Hi George,
>>>
>>> Mind checking the content of pio.log at where you run `pio deploy`? It
>>> should contain some messages regarding undeploying existing engine servers
>>> on the same port.
>>>
>>> Regards,
>>> Donald
>>>
>>> On Thu, Apr 5, 2018 at 6:43 PM, George Yarish <gy...@griddynamics.com>
>>> wrote:
>>>
>>>> Hi everyone,
>>>>
>>>> I faced a strange situation recently.
>>>> One of our engines has frequently old processes after daily
>>>> redeployment.
>>>>
>>>> We used to redeploy by crontab rule that looks like:
>>>> nohup pio deploy --port 8004 >> ${DEPLOY_LOG_FILE} 2>&1 &
>>>>
>>>>
>>>> This is output of ps aux command:
>>>>
>>>> [da_mcom_milan@mdc2pr113 ~]$ ps aux | grep 8004
>>>>
>>>> da_mcom+ 7854 0.0 0.0 112660 996 pts/1 S+ 21:34 0:00 grep
>>>> --color=auto *8004*
>>>>
>>>> 1. da_mcom+ 30932 0.6 0.9 26473632 595932 ? Sl 01:57 7:51
>>>> java -cp /data/PredictionIO-0.11.0-incubating/conf:/data/PredictionIO
>>>> -0.11.0-incubating/plugins/*:/data/PredictionIO-0.11.0-incub
>>>> ating/lib/spark/*:/data/PredictionIO-0.11.0-incubating/lib/p
>>>> io-assembly-0.11.0-incubating.jar:/usr/hdp/2.5.3.0-37/hadoop
>>>> /conf:/data/PredictionIO/vendors/hbase/conf
>>>> -Dpio.log.dir=/home/da_mcom_milan org.apache.predictionio.tools.console.Console
>>>> deploy --port *8004* --pio-home /data/PredictionIO-0.11.0-incubating
>>>>
>>>> 2. da_mcom+ 31097 1.2 1.2 17065124 822596 ? Sl 01:57 14:48
>>>> /usr/java/default/bin/java -Dhdp.version=2.5.3.0-37 -cp
>>>> /data/PredictionIO-0.11.0-incubating/conf/:/usr/hdp/2.5.3.0-
>>>> 37/hadoop/conf/:/data/PredictionIO/vendors/hbase/conf/:/usr/
>>>> hdp/current/spark-client/conf/:/data/PredictionIO/vendors/sp
>>>> ark/lib/spark-assembly-1.6.2.2.5.3.0-37-hadoop2.7.3.2.5.3.0-
>>>> 37.jar:/data/PredictionIO/vendors/spark/lib/datanucleus-api-
>>>> jdo-3.2.6.jar:/data/PredictionIO/vendors/spark/lib/datanucle
>>>> us-core-3.2.10.jar:/data/PredictionIO/vendors/spark/lib/data
>>>> nucleus-rdbms-3.2.9.jar:/usr/hdp/2.5.3.0-37/hadoop/conf/:/us
>>>> r/hdp/current/hadoop-client/lib/aws-java-sdk-s3-1.10.6.jar:/
>>>> usr/hdp/current/hadoop-client/lib/aws-java-sdk-core-1.10.6.
>>>> jar:/usr/hdp/current/hadoop-client/lib/aws-java-sdk-kms-1.10.6.jar
>>>> -Xms1g -Xmx1g -Dpio.log.dir=/home/da_mcom_milan
>>>> org.apache.spark.deploy.SparkSubmit --conf
>>>> spark.driver.extraClassPath=/data/PredictionIO-0.11.0-incuba
>>>> ting/conf:/usr/hdp/2.5.3.0-37/hadoop/conf:/data/PredictionIO/vendors/hbase/conf
>>>> --conf spark.driver.extraJavaOptions=-Dpio.log.dir=/home/da_mcom_milan
>>>> --class org.apache.predictionio.workflow.CreateServer --jars
>>>> file:/data/PredictionIO/some-engine/target/scala-2.10/univer
>>>> sal-recommender_2.10-0.6.0.jar,file:/data/PredictionIO/some-engine
>>>> /target/scala-2.10/universal-recommender-assembly-0.6.
>>>> 0-deps.jar,file:/data/PredictionIO-0.11.0-incubating/lib/spa
>>>> rk/pio-data-elasticsearch1-assembly-0.11.0-incubating.jar,fi
>>>> le:/data/PredictionIO-0.11.0-incubating/lib/spark/pio-data-
>>>> hbase-assembly-0.11.0-incubating.jar,file:/data/PredictionIO
>>>> -0.11.0-incubating/lib/spark/pio-data-hdfs-assembly-0.11.0-
>>>> incubating.jar,file:/data/PredictionIO-0.11.0-incubating
>>>> /lib/spark/pio-data-jdbc-assembly-0.11.0-incubating.
>>>> jar,file:/data/PredictionIO-0.11.0-incubating/lib/spark/pio-
>>>> data-localfs-assembly-0.11.0-incubating.jar --files
>>>> file:/data/PredictionIO-0.11.0-incubating/conf/log4j.propert
>>>> ies,file:/usr/hdp/2.5.3.0-37/hadoop/conf/core-site.xml,file:
>>>> /data/PredictionIO/vendors/hbase/conf/hbase-site.xml
>>>> file:/data/PredictionIO-0.11.0-incubating/lib/pio-assembly-0.11.0-incubating.jar
>>>> --engineInstanceId AWKSaLQrSe_zVbacBHWI --engine-variant
>>>> file:/data/PredictionIO/some-engine/engine.json --ip 0.0.0.0 --port
>>>> *8004* --event-server-ip 0.0.0.0 --event-server-port 7070
>>>> --json-extractor Both --env PIO_STORAGE_SOURCES_HBASE_TYPE
>>>> =hbase,PIO_ENV_LOADED=1,PIO_STORAGE_REPOSITORIES_METADATA_NA
>>>> ME=pio_meta,PIO_FS_BASEDIR=/home/da_mcom_milan/.pio_store,PI
>>>> O_STORAGE_SOURCES_ELASTICSEARCH_HOSTS=mdc2vra176,PIO_STORAGE
>>>> _SOURCES_HBASE_HOME=/data/PredictionIO/vendors/hbase,PIO_HOM
>>>> E=/data/PredictionIO-0.11.0-incubating,PIO_FS_ENGINESDIR=/ho
>>>> me/da_mcom_milan/.pio_store/engines,PIO_STORAGE_SOURCES_LOCA
>>>> LFS_PATH=/home/da_mcom_milan/.pio_store/models,PIO_STORAGE_
>>>> SOURCES_ELASTICSEARCH_TYPE=elasticsearch,PIO_STORAGE_
>>>> REPOSITORIES_METADATA_SOURCE=ELASTICSEARCH,PIO_STORAGE_REPOS
>>>> ITORIES_MODELDATA_SOURCE=LOCALFS,PIO_STORAGE_REPOSITORIES_
>>>> EVENTDATA_NAME=pio_event,PIO_STORAGE_SOURCES_ELASTICSEARCH_
>>>> CLUSTERNAME=pros-dev,PIO_FS_TMPDIR=/home/da_mcom_milan/.
>>>> pio_store/tmp,PIO_STORAGE_REPOSITORIES_MODELDATA_NAME=
>>>> pio_model,PIO_STORAGE_REPOSITORIES_EVENTDATA_SOURCE=
>>>> HBASE,PIO_CONF_DIR=/data/PredictionIO-0.11.0-
>>>> incubating/conf,PIO_STORAGE_SOURCES_ELASTICSEARCH_PORTS=
>>>> 9300,PIO_STORAGE_SOURCES_LOCALFS_TYPE=localfs
>>>>
>>>> 3. da_mcom+ 32505 0.6 0.8 26473632 581164 ? Sl 02:00 7:54
>>>> java -cp /data/PredictionIO-0.11.0-incubating/conf:/data/PredictionIO
>>>> -0.11.0-incubating/plugins/*:/data/PredictionIO-0.11.0-incub
>>>> ating/lib/spark/*:/data/PredictionIO-0.11.0-incubating/lib/p
>>>> io-assembly-0.11.0-incubating.jar:/usr/hdp/2.5.3.0-37/hadoop
>>>> /conf:/data/PredictionIO/vendors/hbase/conf
>>>> -Dpio.log.dir=/home/da_mcom_milan org.apache.predictionio.tools.console.Console
>>>> deploy --port *8004* --pio-home /data/PredictionIO-0.11.0-incubating
>>>>
>>>> 4. da_mcom+ 32671 1.4 1.3 19757488 881448 ? Sl 02:00 16:27
>>>> /usr/java/default/bin/java -Dhdp.version=2.5.3.0-37 -cp
>>>> /data/PredictionIO-0.11.0-incubating/conf/:/usr/hdp/2.5.3.0-
>>>> 37/hadoop/conf/:/data/PredictionIO/vendors/hbase/conf/:/usr/
>>>> hdp/current/spark-client/conf/:/data/PredictionIO/vendors/sp
>>>> ark/lib/spark-assembly-1.6.2.2.5.3.0-37-hadoop2.7.3.2.5.3.0-
>>>> 37.jar:/data/PredictionIO/vendors/spark/lib/datanucleus-api-
>>>> jdo-3.2.6.jar:/data/PredictionIO/vendors/spark/lib/datanucle
>>>> us-core-3.2.10.jar:/data/PredictionIO/vendors/spark/lib/data
>>>> nucleus-rdbms-3.2.9.jar:/usr/hdp/2.5.3.0-37/hadoop/conf/:/us
>>>> r/hdp/current/hadoop-client/lib/aws-java-sdk-s3-1.10.6.jar:/
>>>> usr/hdp/current/hadoop-client/lib/aws-java-sdk-core-1.10.6.
>>>> jar:/usr/hdp/current/hadoop-client/lib/aws-java-sdk-kms-1.10.6.jar
>>>> -Xms1g -Xmx1g -Dpio.log.dir=/home/da_mcom_milan
>>>> org.apache.spark.deploy.SparkSubmit --conf
>>>> spark.driver.extraClassPath=/data/PredictionIO-0.11.0-incuba
>>>> ting/conf:/usr/hdp/2.5.3.0-37/hadoop/conf:/data/PredictionIO/vendors/hbase/conf
>>>> --conf spark.driver.extraJavaOptions=-Dpio.log.dir=/home/da_mcom_milan
>>>> --class org.apache.predictionio.workflow.CreateServer --jars
>>>> file:/data/PredictionIO/some-engine/target/scala-2.10/univer
>>>> sal-recommender_2.10-0.6.0.jar,file:/data/PredictionIO/some-engine
>>>> /target/scala-2.10/universal-recommender-assembly-0.6.
>>>> 0-deps.jar,file:/data/PredictionIO-0.11.0-incubating/lib/spa
>>>> rk/pio-data-elasticsearch1-assembly-0.11.0-incubating.jar,fi
>>>> le:/data/PredictionIO-0.11.0-incubating/lib/spark/pio-data-
>>>> hbase-assembly-0.11.0-incubating.jar,file:/data/PredictionIO
>>>> -0.11.0-incubating/lib/spark/pio-data-hdfs-assembly-0.11.0-
>>>> incubating.jar,file:/data/PredictionIO-0.11.0-incubating
>>>> /lib/spark/pio-data-jdbc-assembly-0.11.0-incubating.
>>>> jar,file:/data/PredictionIO-0.11.0-incubating/lib/spark/pio-
>>>> data-localfs-assembly-0.11.0-incubating.jar --files
>>>> file:/data/PredictionIO-0.11.0-incubating/conf/log4j.propert
>>>> ies,file:/usr/hdp/2.5.3.0-37/hadoop/conf/core-site.xml,file:
>>>> /data/PredictionIO/vendors/hbase/conf/hbase-site.xml
>>>> file:/data/PredictionIO-0.11.0-incubating/lib/pio-assembly-0.11.0-incubating.jar
>>>> --engineInstanceId AWKSaLQrSe_zVbacBHWI --engine-variant
>>>> file:/data/PredictionIO/some-engine/engine.json --ip 0.0.0.0 --port
>>>> *8004* --event-server-ip 0.0.0.0 --event-server-port 7070
>>>> --json-extractor Both --env PIO_STORAGE_SOURCES_HBASE_TYPE
>>>> =hbase,PIO_ENV_LOADED=1,PIO_STORAGE_REPOSITORIES_METADATA_NA
>>>> ME=pio_meta,PIO_FS_BASEDIR=/home/da_mcom_milan/.pio_store,PI
>>>> O_STORAGE_SOURCES_ELASTICSEARCH_HOSTS=mdc2vra176,PIO_STORAGE
>>>> _SOURCES_HBASE_HOME=/data/PredictionIO/vendors/hbase,PIO_HOM
>>>> E=/data/PredictionIO-0.11.0-incubating,PIO_FS_ENGINESDIR=/ho
>>>> me/da_mcom_milan/.pio_store/engines,PIO_STORAGE_SOURCES_LOCA
>>>> LFS_PATH=/home/da_mcom_milan/.pio_store/models,PIO_STORAGE_
>>>> SOURCES_ELASTICSEARCH_TYPE=elasticsearch,PIO_STORAGE_
>>>> REPOSITORIES_METADATA_SOURCE=ELASTICSEARCH,PIO_STORAGE_REPOS
>>>> ITORIES_MODELDATA_SOURCE=LOCALFS,PIO_STORAGE_REPOSITORIES_
>>>> EVENTDATA_NAME=pio_event,PIO_STORAGE_SOURCES_ELASTICSEARCH_
>>>> CLUSTERNAME=pros-dev,PIO_FS_TMPDIR=/home/da_mcom_milan/.
>>>> pio_store/tmp,PIO_STORAGE_REPOSITORIES_MODELDATA_NAME=
>>>> pio_model,PIO_STORAGE_REPOSITORIES_EVENTDATA_SOURCE=
>>>> HBASE,PIO_CONF_DIR=/data/PredictionIO-0.11.0-
>>>> incubating/conf,PIO_STORAGE_SOURCES_ELASTICSEARCH_PORTS=
>>>> 9300,PIO_STORAGE_SOURCES_LOCALFS_TYPE=localfs
>>>>
>>>> For some reasons, processes 30932 and 31097 still alive and process
>>>> listened by 32671
>>>> Any ideas?
>>>>
>>>> Thank you,
>>>> George
>>>>
>>>
>>>
>>
>>
>> --
>>
>> George Yarish, Java Developer
>>
>> Grid Dynamics
>>
>> 197101, Rentgena Str., 5A, St.Petersburg, Russia
>>
>> Cell: +7 950 030-1941
>>
>> Read Grid Dynamics' Tech Blog
>> <http://blog.griddynamics.com/?utm_campaign=Big%20Data%20Blog%20social%20media%20promotion&utm_medium=CTA&utm_source=Email>
>>
>>
>>
>
--
George Yarish, Java Developer
Grid Dynamics
197101, Rentgena Str., 5A, St.Petersburg, Russia
Cell: +7 950 030-1941
Read Grid Dynamics' Tech Blog
<http://blog.griddynamics.com/?utm_campaign=Big%20Data%20Blog%20social%20media%20promotion&utm_medium=CTA&utm_source=Email>
Re: PIO deploy: old processes still live after redeployment
Posted by Donald Szeto <do...@apache.org>.
Are you able to find any occurrence of port 8004? The excerpt you showed
was about port 8000.
On Wed, Apr 11, 2018 at 3:18 PM, George Yarish <gy...@griddynamics.com>
wrote:
> Hi Donald
>
> Thank you for the answer.
> it looks fine to me, but processes still alive:
>
> 2018-04-11 13:00:25,331 INFO org.apache.predictionio.workflow.MasterActor
> [pio-server-akka.actor.default-dispatcher-5] - Undeploying any existing
> engine instance at http://0.0.0.0:8000
> 2018-04-11 13:00:26,467 INFO org.apache.predictionio.workflow.MasterActor
> [pio-server-akka.actor.default-dispatcher-9] - Stop server command
> received.
> 2018-04-11 13:00:26,468 INFO org.apache.predictionio.workflow.MasterActor
> [pio-server-akka.actor.default-dispatcher-9] - Server is shutting down.
> 2018-04-11 13:00:26,617 INFO spray.can.server.HttpListener
> [pio-server-akka.actor.default-dispatcher-2] - Bound to /0.0.0.0:8000
> 2018-04-11 13:00:26,620 INFO org.apache.predictionio.workflow.MasterActor
> [pio-server-akka.actor.default-dispatcher-4] - Engine is deployed and
> running. Engine API is live at http://0.0.0.0:8000
>
> Thank you,
> George
>
>
> On Wed, Apr 11, 2018 at 11:48 AM, Donald Szeto <do...@apache.org> wrote:
>
>> Hi George,
>>
>> Mind checking the content of pio.log at where you run `pio deploy`? It
>> should contain some messages regarding undeploying existing engine servers
>> on the same port.
>>
>> Regards,
>> Donald
>>
>> On Thu, Apr 5, 2018 at 6:43 PM, George Yarish <gy...@griddynamics.com>
>> wrote:
>>
>>> Hi everyone,
>>>
>>> I faced a strange situation recently.
>>> One of our engines has frequently old processes after daily redeployment.
>>>
>>> We used to redeploy by crontab rule that looks like:
>>> nohup pio deploy --port 8004 >> ${DEPLOY_LOG_FILE} 2>&1 &
>>>
>>>
>>> This is output of ps aux command:
>>>
>>> [da_mcom_milan@mdc2pr113 ~]$ ps aux | grep 8004
>>>
>>> da_mcom+ 7854 0.0 0.0 112660 996 pts/1 S+ 21:34 0:00 grep
>>> --color=auto *8004*
>>>
>>> 1. da_mcom+ 30932 0.6 0.9 26473632 595932 ? Sl 01:57 7:51 java
>>> -cp /data/PredictionIO-0.11.0-incubating/conf:/data/PredictionIO
>>> -0.11.0-incubating/plugins/*:/data/PredictionIO-0.11.0-incub
>>> ating/lib/spark/*:/data/PredictionIO-0.11.0-incubating/lib/p
>>> io-assembly-0.11.0-incubating.jar:/usr/hdp/2.5.3.0-37/hadoop
>>> /conf:/data/PredictionIO/vendors/hbase/conf
>>> -Dpio.log.dir=/home/da_mcom_milan org.apache.predictionio.tools.console.Console
>>> deploy --port *8004* --pio-home /data/PredictionIO-0.11.0-incubating
>>>
>>> 2. da_mcom+ 31097 1.2 1.2 17065124 822596 ? Sl 01:57 14:48
>>> /usr/java/default/bin/java -Dhdp.version=2.5.3.0-37 -cp
>>> /data/PredictionIO-0.11.0-incubating/conf/:/usr/hdp/2.5.3.0-
>>> 37/hadoop/conf/:/data/PredictionIO/vendors/hbase/conf/:/usr/
>>> hdp/current/spark-client/conf/:/data/PredictionIO/vendors/sp
>>> ark/lib/spark-assembly-1.6.2.2.5.3.0-37-hadoop2.7.3.2.5.3.0-
>>> 37.jar:/data/PredictionIO/vendors/spark/lib/datanucleus-api-
>>> jdo-3.2.6.jar:/data/PredictionIO/vendors/spark/lib/datanucle
>>> us-core-3.2.10.jar:/data/PredictionIO/vendors/spark/lib/data
>>> nucleus-rdbms-3.2.9.jar:/usr/hdp/2.5.3.0-37/hadoop/conf/:/
>>> usr/hdp/current/hadoop-client/lib/aws-java-sdk-s3-1.10.6.
>>> jar:/usr/hdp/current/hadoop-client/lib/aws-java-sdk-core-
>>> 1.10.6.jar:/usr/hdp/current/hadoop-client/lib/aws-java-sdk-kms-1.10.6.jar
>>> -Xms1g -Xmx1g -Dpio.log.dir=/home/da_mcom_milan
>>> org.apache.spark.deploy.SparkSubmit --conf
>>> spark.driver.extraClassPath=/data/PredictionIO-0.11.0-incuba
>>> ting/conf:/usr/hdp/2.5.3.0-37/hadoop/conf:/data/PredictionIO/vendors/hbase/conf
>>> --conf spark.driver.extraJavaOptions=-Dpio.log.dir=/home/da_mcom_milan
>>> --class org.apache.predictionio.workflow.CreateServer --jars
>>> file:/data/PredictionIO/some-engine/target/scala-2.10/univer
>>> sal-recommender_2.10-0.6.0.jar,file:/data/PredictionIO/some-engine
>>> /target/scala-2.10/universal-recommender-assembly-0.6.
>>> 0-deps.jar,file:/data/PredictionIO-0.11.0-incubating/lib/spa
>>> rk/pio-data-elasticsearch1-assembly-0.11.0-incubating.jar,
>>> file:/data/PredictionIO-0.11.0-incubating/lib/spark/pio-dat
>>> a-hbase-assembly-0.11.0-incubating.jar,file:/data/Prediction
>>> IO-0.11.0-incubating/lib/spark/pio-data-hdfs-assembly-
>>> 0.11.0-incubating.jar,file:/data/PredictionIO-0.11.0-
>>> incubating/lib/spark/pio-data-jdbc-assembly-0.11.0-incubatin
>>> g.jar,file:/data/PredictionIO-0.11.0-incubating/lib/spark/
>>> pio-data-localfs-assembly-0.11.0-incubating.jar --files
>>> file:/data/PredictionIO-0.11.0-incubating/conf/log4j.propert
>>> ies,file:/usr/hdp/2.5.3.0-37/hadoop/conf/core-site.xml,file:
>>> /data/PredictionIO/vendors/hbase/conf/hbase-site.xml
>>> file:/data/PredictionIO-0.11.0-incubating/lib/pio-assembly-0.11.0-incubating.jar
>>> --engineInstanceId AWKSaLQrSe_zVbacBHWI --engine-variant
>>> file:/data/PredictionIO/some-engine/engine.json --ip 0.0.0.0 --port
>>> *8004* --event-server-ip 0.0.0.0 --event-server-port 7070
>>> --json-extractor Both --env PIO_STORAGE_SOURCES_HBASE_TYPE
>>> =hbase,PIO_ENV_LOADED=1,PIO_STORAGE_REPOSITORIES_METADATA_NA
>>> ME=pio_meta,PIO_FS_BASEDIR=/home/da_mcom_milan/.pio_store,PI
>>> O_STORAGE_SOURCES_ELASTICSEARCH_HOSTS=mdc2vra176,PIO_STORAGE
>>> _SOURCES_HBASE_HOME=/data/PredictionIO/vendors/hbase,PIO_HOM
>>> E=/data/PredictionIO-0.11.0-incubating,PIO_FS_ENGINESDIR=/
>>> home/da_mcom_milan/.pio_store/engines,PIO_STORAGE_SOURCES_
>>> LOCALFS_PATH=/home/da_mcom_milan/.pio_store/models,PIO_
>>> STORAGE_SOURCES_ELASTICSEARCH_TYPE=elasticsearch,PIO_STORAGE
>>> _REPOSITORIES_METADATA_SOURCE=ELASTICSEARCH,PIO_STORAGE_
>>> REPOSITORIES_MODELDATA_SOURCE=LOCALFS,PIO_STORAGE_
>>> REPOSITORIES_EVENTDATA_NAME=pio_event,PIO_STORAGE_SOURCES_
>>> ELASTICSEARCH_CLUSTERNAME=pros-dev,PIO_FS_TMPDIR=/home/
>>> da_mcom_milan/.pio_store/tmp,PIO_STORAGE_REPOSITORIES_
>>> MODELDATA_NAME=pio_model,PIO_STORAGE_REPOSITORIES_
>>> EVENTDATA_SOURCE=HBASE,PIO_CONF_DIR=/data/PredictionIO-0.
>>> 11.0-incubating/conf,PIO_STORAGE_SOURCES_ELASTICSEARCH_
>>> PORTS=9300,PIO_STORAGE_SOURCES_LOCALFS_TYPE=localfs
>>>
>>> 3. da_mcom+ 32505 0.6 0.8 26473632 581164 ? Sl 02:00 7:54 java
>>> -cp /data/PredictionIO-0.11.0-incubating/conf:/data/PredictionIO
>>> -0.11.0-incubating/plugins/*:/data/PredictionIO-0.11.0-incub
>>> ating/lib/spark/*:/data/PredictionIO-0.11.0-incubating/lib/p
>>> io-assembly-0.11.0-incubating.jar:/usr/hdp/2.5.3.0-37/hadoop
>>> /conf:/data/PredictionIO/vendors/hbase/conf
>>> -Dpio.log.dir=/home/da_mcom_milan org.apache.predictionio.tools.console.Console
>>> deploy --port *8004* --pio-home /data/PredictionIO-0.11.0-incubating
>>>
>>> 4. da_mcom+ 32671 1.4 1.3 19757488 881448 ? Sl 02:00 16:27
>>> /usr/java/default/bin/java -Dhdp.version=2.5.3.0-37 -cp
>>> /data/PredictionIO-0.11.0-incubating/conf/:/usr/hdp/2.5.3.0-
>>> 37/hadoop/conf/:/data/PredictionIO/vendors/hbase/conf/:/usr/
>>> hdp/current/spark-client/conf/:/data/PredictionIO/vendors/sp
>>> ark/lib/spark-assembly-1.6.2.2.5.3.0-37-hadoop2.7.3.2.5.3.0-
>>> 37.jar:/data/PredictionIO/vendors/spark/lib/datanucleus-api-
>>> jdo-3.2.6.jar:/data/PredictionIO/vendors/spark/lib/datanucle
>>> us-core-3.2.10.jar:/data/PredictionIO/vendors/spark/lib/data
>>> nucleus-rdbms-3.2.9.jar:/usr/hdp/2.5.3.0-37/hadoop/conf/:/
>>> usr/hdp/current/hadoop-client/lib/aws-java-sdk-s3-1.10.6.
>>> jar:/usr/hdp/current/hadoop-client/lib/aws-java-sdk-core-
>>> 1.10.6.jar:/usr/hdp/current/hadoop-client/lib/aws-java-sdk-kms-1.10.6.jar
>>> -Xms1g -Xmx1g -Dpio.log.dir=/home/da_mcom_milan
>>> org.apache.spark.deploy.SparkSubmit --conf
>>> spark.driver.extraClassPath=/data/PredictionIO-0.11.0-incuba
>>> ting/conf:/usr/hdp/2.5.3.0-37/hadoop/conf:/data/PredictionIO/vendors/hbase/conf
>>> --conf spark.driver.extraJavaOptions=-Dpio.log.dir=/home/da_mcom_milan
>>> --class org.apache.predictionio.workflow.CreateServer --jars
>>> file:/data/PredictionIO/some-engine/target/scala-2.10/univer
>>> sal-recommender_2.10-0.6.0.jar,file:/data/PredictionIO/some-engine
>>> /target/scala-2.10/universal-recommender-assembly-0.6.
>>> 0-deps.jar,file:/data/PredictionIO-0.11.0-incubating/lib/spa
>>> rk/pio-data-elasticsearch1-assembly-0.11.0-incubating.jar,
>>> file:/data/PredictionIO-0.11.0-incubating/lib/spark/pio-dat
>>> a-hbase-assembly-0.11.0-incubating.jar,file:/data/Prediction
>>> IO-0.11.0-incubating/lib/spark/pio-data-hdfs-assembly-
>>> 0.11.0-incubating.jar,file:/data/PredictionIO-0.11.0-
>>> incubating/lib/spark/pio-data-jdbc-assembly-0.11.0-incubatin
>>> g.jar,file:/data/PredictionIO-0.11.0-incubating/lib/spark/
>>> pio-data-localfs-assembly-0.11.0-incubating.jar --files
>>> file:/data/PredictionIO-0.11.0-incubating/conf/log4j.propert
>>> ies,file:/usr/hdp/2.5.3.0-37/hadoop/conf/core-site.xml,file:
>>> /data/PredictionIO/vendors/hbase/conf/hbase-site.xml
>>> file:/data/PredictionIO-0.11.0-incubating/lib/pio-assembly-0.11.0-incubating.jar
>>> --engineInstanceId AWKSaLQrSe_zVbacBHWI --engine-variant
>>> file:/data/PredictionIO/some-engine/engine.json --ip 0.0.0.0 --port
>>> *8004* --event-server-ip 0.0.0.0 --event-server-port 7070
>>> --json-extractor Both --env PIO_STORAGE_SOURCES_HBASE_TYPE
>>> =hbase,PIO_ENV_LOADED=1,PIO_STORAGE_REPOSITORIES_METADATA_NA
>>> ME=pio_meta,PIO_FS_BASEDIR=/home/da_mcom_milan/.pio_store,PI
>>> O_STORAGE_SOURCES_ELASTICSEARCH_HOSTS=mdc2vra176,PIO_STORAGE
>>> _SOURCES_HBASE_HOME=/data/PredictionIO/vendors/hbase,PIO_HOM
>>> E=/data/PredictionIO-0.11.0-incubating,PIO_FS_ENGINESDIR=/
>>> home/da_mcom_milan/.pio_store/engines,PIO_STORAGE_SOURCES_
>>> LOCALFS_PATH=/home/da_mcom_milan/.pio_store/models,PIO_
>>> STORAGE_SOURCES_ELASTICSEARCH_TYPE=elasticsearch,PIO_STORAGE
>>> _REPOSITORIES_METADATA_SOURCE=ELASTICSEARCH,PIO_STORAGE_
>>> REPOSITORIES_MODELDATA_SOURCE=LOCALFS,PIO_STORAGE_
>>> REPOSITORIES_EVENTDATA_NAME=pio_event,PIO_STORAGE_SOURCES_
>>> ELASTICSEARCH_CLUSTERNAME=pros-dev,PIO_FS_TMPDIR=/home/
>>> da_mcom_milan/.pio_store/tmp,PIO_STORAGE_REPOSITORIES_
>>> MODELDATA_NAME=pio_model,PIO_STORAGE_REPOSITORIES_
>>> EVENTDATA_SOURCE=HBASE,PIO_CONF_DIR=/data/PredictionIO-0.
>>> 11.0-incubating/conf,PIO_STORAGE_SOURCES_ELASTICSEARCH_
>>> PORTS=9300,PIO_STORAGE_SOURCES_LOCALFS_TYPE=localfs
>>>
>>> For some reasons, processes 30932 and 31097 still alive and process
>>> listened by 32671
>>> Any ideas?
>>>
>>> Thank you,
>>> George
>>>
>>
>>
>
>
> --
>
> George Yarish, Java Developer
>
> Grid Dynamics
>
> 197101, Rentgena Str., 5A, St.Petersburg, Russia
>
> Cell: +7 950 030-1941
>
> Read Grid Dynamics' Tech Blog
> <http://blog.griddynamics.com/?utm_campaign=Big%20Data%20Blog%20social%20media%20promotion&utm_medium=CTA&utm_source=Email>
>
>
>
Re: PIO deploy: old processes still live after redeployment
Posted by George Yarish <gy...@griddynamics.com>.
Hi Donald
Thank you for the answer.
it looks fine to me, but processes still alive:
2018-04-11 13:00:25,331 INFO org.apache.predictionio.workflow.MasterActor
[pio-server-akka.actor.default-dispatcher-5] - Undeploying any existing
engine instance at http://0.0.0.0:8000
2018-04-11 13:00:26,467 INFO org.apache.predictionio.workflow.MasterActor
[pio-server-akka.actor.default-dispatcher-9] - Stop server command received.
2018-04-11 13:00:26,468 INFO org.apache.predictionio.workflow.MasterActor
[pio-server-akka.actor.default-dispatcher-9] - Server is shutting down.
2018-04-11 13:00:26,617 INFO spray.can.server.HttpListener
[pio-server-akka.actor.default-dispatcher-2] - Bound to /0.0.0.0:8000
2018-04-11 13:00:26,620 INFO org.apache.predictionio.workflow.MasterActor
[pio-server-akka.actor.default-dispatcher-4] - Engine is deployed and
running. Engine API is live at http://0.0.0.0:8000
Thank you,
George
On Wed, Apr 11, 2018 at 11:48 AM, Donald Szeto <do...@apache.org> wrote:
> Hi George,
>
> Mind checking the content of pio.log at where you run `pio deploy`? It
> should contain some messages regarding undeploying existing engine servers
> on the same port.
>
> Regards,
> Donald
>
> On Thu, Apr 5, 2018 at 6:43 PM, George Yarish <gy...@griddynamics.com>
> wrote:
>
>> Hi everyone,
>>
>> I faced a strange situation recently.
>> One of our engines has frequently old processes after daily redeployment.
>>
>> We used to redeploy by crontab rule that looks like:
>> nohup pio deploy --port 8004 >> ${DEPLOY_LOG_FILE} 2>&1 &
>>
>>
>> This is output of ps aux command:
>>
>> [da_mcom_milan@mdc2pr113 ~]$ ps aux | grep 8004
>>
>> da_mcom+ 7854 0.0 0.0 112660 996 pts/1 S+ 21:34 0:00 grep
>> --color=auto *8004*
>>
>> 1. da_mcom+ 30932 0.6 0.9 26473632 595932 ? Sl 01:57 7:51 java
>> -cp /data/PredictionIO-0.11.0-incubating/conf:/data/PredictionIO
>> -0.11.0-incubating/plugins/*:/data/PredictionIO-0.11.0-incub
>> ating/lib/spark/*:/data/PredictionIO-0.11.0-incubating/lib/p
>> io-assembly-0.11.0-incubating.jar:/usr/hdp/2.5.3.0-37/hadoop
>> /conf:/data/PredictionIO/vendors/hbase/conf
>> -Dpio.log.dir=/home/da_mcom_milan org.apache.predictionio.tools.console.Console
>> deploy --port *8004* --pio-home /data/PredictionIO-0.11.0-incubating
>>
>> 2. da_mcom+ 31097 1.2 1.2 17065124 822596 ? Sl 01:57 14:48
>> /usr/java/default/bin/java -Dhdp.version=2.5.3.0-37 -cp
>> /data/PredictionIO-0.11.0-incubating/conf/:/usr/hdp/2.5.3.0-
>> 37/hadoop/conf/:/data/PredictionIO/vendors/hbase/conf/:/usr/
>> hdp/current/spark-client/conf/:/data/PredictionIO/vendors/sp
>> ark/lib/spark-assembly-1.6.2.2.5.3.0-37-hadoop2.7.3.2.5.3.0-
>> 37.jar:/data/PredictionIO/vendors/spark/lib/datanucleus-api-
>> jdo-3.2.6.jar:/data/PredictionIO/vendors/spark/lib/datanucle
>> us-core-3.2.10.jar:/data/PredictionIO/vendors/spark/lib/
>> datanucleus-rdbms-3.2.9.jar:/usr/hdp/2.5.3.0-37/hadoop/
>> conf/:/usr/hdp/current/hadoop-client/lib/aws-java-sdk-s3-1.1
>> 0.6.jar:/usr/hdp/current/hadoop-client/lib/aws-java-sdk-
>> core-1.10.6.jar:/usr/hdp/current/hadoop-client/lib/aws-java-sdk-kms-1.10.6.jar
>> -Xms1g -Xmx1g -Dpio.log.dir=/home/da_mcom_milan
>> org.apache.spark.deploy.SparkSubmit --conf spark.driver.extraClassPath=/d
>> ata/PredictionIO-0.11.0-incubating/conf:/usr/hdp/2.5.3.0-37/
>> hadoop/conf:/data/PredictionIO/vendors/hbase/conf --conf
>> spark.driver.extraJavaOptions=-Dpio.log.dir=/home/da_mcom_milan --class
>> org.apache.predictionio.workflow.CreateServer --jars
>> file:/data/PredictionIO/some-engine/target/scala-2.10/univer
>> sal-recommender_2.10-0.6.0.jar,file:/data/PredictionIO/some-engine
>> /target/scala-2.10/universal-recommender-assembly-0.6.
>> 0-deps.jar,file:/data/PredictionIO-0.11.0-incubating/lib/
>> spark/pio-data-elasticsearch1-assembly-0.11.0-incubating.
>> jar,file:/data/PredictionIO-0.11.0-incubating/lib/spark/pio-
>> data-hbase-assembly-0.11.0-incubating.jar,file:/data/Pred
>> ictionIO-0.11.0-incubating/lib/spark/pio-data-hdfs-
>> assembly-0.11.0-incubating.jar,file:/data/PredictionIO-0.
>> 11.0-incubating/lib/spark/pio-data-jdbc-assembly-0.11.0-incu
>> bating.jar,file:/data/PredictionIO-0.11.0-incubating/lib/
>> spark/pio-data-localfs-assembly-0.11.0-incubating.jar --files
>> file:/data/PredictionIO-0.11.0-incubating/conf/log4j.propert
>> ies,file:/usr/hdp/2.5.3.0-37/hadoop/conf/core-site.xml,file:
>> /data/PredictionIO/vendors/hbase/conf/hbase-site.xml
>> file:/data/PredictionIO-0.11.0-incubating/lib/pio-assembly-0.11.0-incubating.jar
>> --engineInstanceId AWKSaLQrSe_zVbacBHWI --engine-variant
>> file:/data/PredictionIO/some-engine/engine.json --ip 0.0.0.0 --port
>> *8004* --event-server-ip 0.0.0.0 --event-server-port 7070
>> --json-extractor Both --env PIO_STORAGE_SOURCES_HBASE_TYPE
>> =hbase,PIO_ENV_LOADED=1,PIO_STORAGE_REPOSITORIES_METADATA_NA
>> ME=pio_meta,PIO_FS_BASEDIR=/home/da_mcom_milan/.pio_store,PI
>> O_STORAGE_SOURCES_ELASTICSEARCH_HOSTS=mdc2vra176,PIO_STORAGE
>> _SOURCES_HBASE_HOME=/data/PredictionIO/vendors/hbase,PIO_
>> HOME=/data/PredictionIO-0.11.0-incubating,PIO_FS_ENGINESDIR
>> =/home/da_mcom_milan/.pio_store/engines,PIO_STORAGE_
>> SOURCES_LOCALFS_PATH=/home/da_mcom_milan/.pio_store/models,
>> PIO_STORAGE_SOURCES_ELASTICSEARCH_TYPE=elasticsearch,PIO_
>> STORAGE_REPOSITORIES_METADATA_SOURCE=ELASTICSEARCH,PIO_
>> STORAGE_REPOSITORIES_MODELDATA_SOURCE=LOCALFS,PIO_STORAGE_REPOSITORIES_
>> EVENTDATA_NAME=pio_event,PIO_STORAGE_SOURCES_ELASTICSEARCH_
>> CLUSTERNAME=pros-dev,PIO_FS_TMPDIR=/home/da_mcom_milan/.
>> pio_store/tmp,PIO_STORAGE_REPOSITORIES_MODELDATA_NAME=
>> pio_model,PIO_STORAGE_REPOSITORIES_EVENTDATA_SOURCE=
>> HBASE,PIO_CONF_DIR=/data/PredictionIO-0.11.0-incubating/conf,PIO_STORAGE_
>> SOURCES_ELASTICSEARCH_PORTS=9300,PIO_STORAGE_SOURCES_LOCALFS_TYPE=localfs
>>
>> 3. da_mcom+ 32505 0.6 0.8 26473632 581164 ? Sl 02:00 7:54 java
>> -cp /data/PredictionIO-0.11.0-incubating/conf:/data/PredictionIO
>> -0.11.0-incubating/plugins/*:/data/PredictionIO-0.11.0-incub
>> ating/lib/spark/*:/data/PredictionIO-0.11.0-incubating/lib/p
>> io-assembly-0.11.0-incubating.jar:/usr/hdp/2.5.3.0-37/hadoop
>> /conf:/data/PredictionIO/vendors/hbase/conf
>> -Dpio.log.dir=/home/da_mcom_milan org.apache.predictionio.tools.console.Console
>> deploy --port *8004* --pio-home /data/PredictionIO-0.11.0-incubating
>>
>> 4. da_mcom+ 32671 1.4 1.3 19757488 881448 ? Sl 02:00 16:27
>> /usr/java/default/bin/java -Dhdp.version=2.5.3.0-37 -cp
>> /data/PredictionIO-0.11.0-incubating/conf/:/usr/hdp/2.5.3.0-
>> 37/hadoop/conf/:/data/PredictionIO/vendors/hbase/conf/:/usr/
>> hdp/current/spark-client/conf/:/data/PredictionIO/vendors/sp
>> ark/lib/spark-assembly-1.6.2.2.5.3.0-37-hadoop2.7.3.2.5.3.0-
>> 37.jar:/data/PredictionIO/vendors/spark/lib/datanucleus-api-
>> jdo-3.2.6.jar:/data/PredictionIO/vendors/spark/lib/datanucle
>> us-core-3.2.10.jar:/data/PredictionIO/vendors/spark/lib/
>> datanucleus-rdbms-3.2.9.jar:/usr/hdp/2.5.3.0-37/hadoop/
>> conf/:/usr/hdp/current/hadoop-client/lib/aws-java-sdk-s3-1.1
>> 0.6.jar:/usr/hdp/current/hadoop-client/lib/aws-java-sdk-
>> core-1.10.6.jar:/usr/hdp/current/hadoop-client/lib/aws-java-sdk-kms-1.10.6.jar
>> -Xms1g -Xmx1g -Dpio.log.dir=/home/da_mcom_milan
>> org.apache.spark.deploy.SparkSubmit --conf spark.driver.extraClassPath=/d
>> ata/PredictionIO-0.11.0-incubating/conf:/usr/hdp/2.5.3.0-37/
>> hadoop/conf:/data/PredictionIO/vendors/hbase/conf --conf
>> spark.driver.extraJavaOptions=-Dpio.log.dir=/home/da_mcom_milan --class
>> org.apache.predictionio.workflow.CreateServer --jars
>> file:/data/PredictionIO/some-engine/target/scala-2.10/univer
>> sal-recommender_2.10-0.6.0.jar,file:/data/PredictionIO/some-engine
>> /target/scala-2.10/universal-recommender-assembly-0.6.
>> 0-deps.jar,file:/data/PredictionIO-0.11.0-incubating/lib/
>> spark/pio-data-elasticsearch1-assembly-0.11.0-incubating.
>> jar,file:/data/PredictionIO-0.11.0-incubating/lib/spark/pio-
>> data-hbase-assembly-0.11.0-incubating.jar,file:/data/Pred
>> ictionIO-0.11.0-incubating/lib/spark/pio-data-hdfs-
>> assembly-0.11.0-incubating.jar,file:/data/PredictionIO-0.
>> 11.0-incubating/lib/spark/pio-data-jdbc-assembly-0.11.0-incu
>> bating.jar,file:/data/PredictionIO-0.11.0-incubating/lib/
>> spark/pio-data-localfs-assembly-0.11.0-incubating.jar --files
>> file:/data/PredictionIO-0.11.0-incubating/conf/log4j.propert
>> ies,file:/usr/hdp/2.5.3.0-37/hadoop/conf/core-site.xml,file:
>> /data/PredictionIO/vendors/hbase/conf/hbase-site.xml
>> file:/data/PredictionIO-0.11.0-incubating/lib/pio-assembly-0.11.0-incubating.jar
>> --engineInstanceId AWKSaLQrSe_zVbacBHWI --engine-variant
>> file:/data/PredictionIO/some-engine/engine.json --ip 0.0.0.0 --port
>> *8004* --event-server-ip 0.0.0.0 --event-server-port 7070
>> --json-extractor Both --env PIO_STORAGE_SOURCES_HBASE_TYPE
>> =hbase,PIO_ENV_LOADED=1,PIO_STORAGE_REPOSITORIES_METADATA_NA
>> ME=pio_meta,PIO_FS_BASEDIR=/home/da_mcom_milan/.pio_store,PI
>> O_STORAGE_SOURCES_ELASTICSEARCH_HOSTS=mdc2vra176,PIO_STORAGE
>> _SOURCES_HBASE_HOME=/data/PredictionIO/vendors/hbase,PIO_
>> HOME=/data/PredictionIO-0.11.0-incubating,PIO_FS_ENGINESDIR
>> =/home/da_mcom_milan/.pio_store/engines,PIO_STORAGE_
>> SOURCES_LOCALFS_PATH=/home/da_mcom_milan/.pio_store/models,
>> PIO_STORAGE_SOURCES_ELASTICSEARCH_TYPE=elasticsearch,PIO_
>> STORAGE_REPOSITORIES_METADATA_SOURCE=ELASTICSEARCH,PIO_
>> STORAGE_REPOSITORIES_MODELDATA_SOURCE=LOCALFS,PIO_STORAGE_REPOSITORIES_
>> EVENTDATA_NAME=pio_event,PIO_STORAGE_SOURCES_ELASTICSEARCH_
>> CLUSTERNAME=pros-dev,PIO_FS_TMPDIR=/home/da_mcom_milan/.
>> pio_store/tmp,PIO_STORAGE_REPOSITORIES_MODELDATA_NAME=
>> pio_model,PIO_STORAGE_REPOSITORIES_EVENTDATA_SOURCE=
>> HBASE,PIO_CONF_DIR=/data/PredictionIO-0.11.0-incubating/conf,PIO_STORAGE_
>> SOURCES_ELASTICSEARCH_PORTS=9300,PIO_STORAGE_SOURCES_LOCALFS_TYPE=localfs
>>
>> For some reasons, processes 30932 and 31097 still alive and process
>> listened by 32671
>> Any ideas?
>>
>> Thank you,
>> George
>>
>
>
--
George Yarish, Java Developer
Grid Dynamics
197101, Rentgena Str., 5A, St.Petersburg, Russia
Cell: +7 950 030-1941
Read Grid Dynamics' Tech Blog
<http://blog.griddynamics.com/?utm_campaign=Big%20Data%20Blog%20social%20media%20promotion&utm_medium=CTA&utm_source=Email>
Re: PIO deploy: old processes still live after redeployment
Posted by Donald Szeto <do...@apache.org>.
Hi George,
Mind checking the content of pio.log at where you run `pio deploy`? It
should contain some messages regarding undeploying existing engine servers
on the same port.
Regards,
Donald
On Thu, Apr 5, 2018 at 6:43 PM, George Yarish <gy...@griddynamics.com>
wrote:
> Hi everyone,
>
> I faced a strange situation recently.
> One of our engines has frequently old processes after daily redeployment.
>
> We used to redeploy by crontab rule that looks like:
> nohup pio deploy --port 8004 >> ${DEPLOY_LOG_FILE} 2>&1 &
>
>
> This is output of ps aux command:
>
> [da_mcom_milan@mdc2pr113 ~]$ ps aux | grep 8004
>
> da_mcom+ 7854 0.0 0.0 112660 996 pts/1 S+ 21:34 0:00 grep
> --color=auto *8004*
>
> 1. da_mcom+ 30932 0.6 0.9 26473632 595932 ? Sl 01:57 7:51 java
> -cp /data/PredictionIO-0.11.0-incubating/conf:/data/PredictionIO
> -0.11.0-incubating/plugins/*:/data/PredictionIO-0.11.0-incub
> ating/lib/spark/*:/data/PredictionIO-0.11.0-incubating/lib/p
> io-assembly-0.11.0-incubating.jar:/usr/hdp/2.5.3.0-37/
> hadoop/conf:/data/PredictionIO/vendors/hbase/conf
> -Dpio.log.dir=/home/da_mcom_milan org.apache.predictionio.tools.console.Console
> deploy --port *8004* --pio-home /data/PredictionIO-0.11.0-incubating
>
> 2. da_mcom+ 31097 1.2 1.2 17065124 822596 ? Sl 01:57 14:48
> /usr/java/default/bin/java -Dhdp.version=2.5.3.0-37 -cp
> /data/PredictionIO-0.11.0-incubating/conf/:/usr/hdp/2.5.3.0-
> 37/hadoop/conf/:/data/PredictionIO/vendors/hbase/conf/:/usr/
> hdp/current/spark-client/conf/:/data/PredictionIO/vendors/sp
> ark/lib/spark-assembly-1.6.2.2.5.3.0-37-hadoop2.7.3.2.5.3.
> 0-37.jar:/data/PredictionIO/vendors/spark/lib/datanucleus-ap
> i-jdo-3.2.6.jar:/data/PredictionIO/vendors/spark/lib/datanuc
> leus-core-3.2.10.jar:/data/PredictionIO/vendors/spark/
> lib/datanucleus-rdbms-3.2.9.jar:/usr/hdp/2.5.3.0-37/hadoop
> /conf/:/usr/hdp/current/hadoop-client/lib/aws-java-sdk-s3-1.
> 10.6.jar:/usr/hdp/current/hadoop-client/lib/aws-java-
> sdk-core-1.10.6.jar:/usr/hdp/current/hadoop-client/lib/aws-java-sdk-kms-1.10.6.jar
> -Xms1g -Xmx1g -Dpio.log.dir=/home/da_mcom_milan
> org.apache.spark.deploy.SparkSubmit --conf spark.driver.extraClassPath=/d
> ata/PredictionIO-0.11.0-incubating/conf:/usr/hdp/2.5.3.0-37/
> hadoop/conf:/data/PredictionIO/vendors/hbase/conf --conf
> spark.driver.extraJavaOptions=-Dpio.log.dir=/home/da_mcom_milan --class
> org.apache.predictionio.workflow.CreateServer --jars
> file:/data/PredictionIO/some-engine/target/scala-2.10/univer
> sal-recommender_2.10-0.6.0.jar,file:/data/PredictionIO/some-engine
> /target/scala-2.10/universal-recommender-assembly
> -0.6.0-deps.jar,file:/data/PredictionIO-0.11.0-incubating
> /lib/spark/pio-data-elasticsearch1-assembly-0.11.0-incubatin
> g.jar,file:/data/PredictionIO-0.11.0-incubating/lib/spark/
> pio-data-hbase-assembly-0.11.0-incubating.jar,file:/data/
> PredictionIO-0.11.0-incubating/lib/spark/pio-data-
> hdfs-assembly-0.11.0-incubating.jar,file:/data/PredictionIO-
> 0.11.0-incubating/lib/spark/pio-data-jdbc-assembly-0.11.0-
> incubating.jar,file:/data/PredictionIO-0.11.0-incubating
> /lib/spark/pio-data-localfs-assembly-0.11.0-incubating.jar --files
> file:/data/PredictionIO-0.11.0-incubating/conf/log4j.propert
> ies,file:/usr/hdp/2.5.3.0-37/hadoop/conf/core-site.xml,file:
> /data/PredictionIO/vendors/hbase/conf/hbase-site.xml
> file:/data/PredictionIO-0.11.0-incubating/lib/pio-assembly-0.11.0-incubating.jar
> --engineInstanceId AWKSaLQrSe_zVbacBHWI --engine-variant
> file:/data/PredictionIO/some-engine/engine.json --ip 0.0.0.0 --port *8004*
> --event-server-ip 0.0.0.0 --event-server-port 7070 --json-extractor Both
> --env PIO_STORAGE_SOURCES_HBASE_TYPE=hbase,PIO_ENV_LOADED=1,PIO_ST
> ORAGE_REPOSITORIES_METADATA_NAME=pio_meta,PIO_FS_BASEDIR=/ho
> me/da_mcom_milan/.pio_store,PIO_STORAGE_SOURCES_ELASTICSEARC
> H_HOSTS=mdc2vra176,PIO_STORAGE_SOURCES_HBASE_HOME=/data/
> PredictionIO/vendors/hbase,PIO_HOME=/data/PredictionIO-0.
> 11.0-incubating,PIO_FS_ENGINESDIR=/home/da_mcom_
> milan/.pio_store/engines,PIO_STORAGE_SOURCES_LOCALFS_PATH=/
> home/da_mcom_milan/.pio_store/models,PIO_STORAGE_SOURCES_
> ELASTICSEARCH_TYPE=elasticsearch,PIO_STORAGE_REPOSITORIES_METADATA_SOURCE=
> ELASTICSEARCH,PIO_STORAGE_REPOSITORIES_MODELDATA_SOURCE=
> LOCALFS,PIO_STORAGE_REPOSITORIES_EVENTDATA_NAME=
> pio_event,PIO_STORAGE_SOURCES_ELASTICSEARCH_CLUSTERNAME=
> pros-dev,PIO_FS_TMPDIR=/home/da_mcom_milan/.pio_store/tmp,
> PIO_STORAGE_REPOSITORIES_MODELDATA_NAME=pio_model,PIO_
> STORAGE_REPOSITORIES_EVENTDATA_SOURCE=HBASE,PIO_
> CONF_DIR=/data/PredictionIO-0.11.0-incubating/conf,PIO_
> STORAGE_SOURCES_ELASTICSEARCH_PORTS=9300,PIO_STORAGE_
> SOURCES_LOCALFS_TYPE=localfs
>
> 3. da_mcom+ 32505 0.6 0.8 26473632 581164 ? Sl 02:00 7:54 java
> -cp /data/PredictionIO-0.11.0-incubating/conf:/data/PredictionIO
> -0.11.0-incubating/plugins/*:/data/PredictionIO-0.11.0-incub
> ating/lib/spark/*:/data/PredictionIO-0.11.0-incubating/lib/p
> io-assembly-0.11.0-incubating.jar:/usr/hdp/2.5.3.0-37/
> hadoop/conf:/data/PredictionIO/vendors/hbase/conf
> -Dpio.log.dir=/home/da_mcom_milan org.apache.predictionio.tools.console.Console
> deploy --port *8004* --pio-home /data/PredictionIO-0.11.0-incubating
>
> 4. da_mcom+ 32671 1.4 1.3 19757488 881448 ? Sl 02:00 16:27
> /usr/java/default/bin/java -Dhdp.version=2.5.3.0-37 -cp
> /data/PredictionIO-0.11.0-incubating/conf/:/usr/hdp/2.5.3.0-
> 37/hadoop/conf/:/data/PredictionIO/vendors/hbase/conf/:/usr/
> hdp/current/spark-client/conf/:/data/PredictionIO/vendors/sp
> ark/lib/spark-assembly-1.6.2.2.5.3.0-37-hadoop2.7.3.2.5.3.
> 0-37.jar:/data/PredictionIO/vendors/spark/lib/datanucleus-ap
> i-jdo-3.2.6.jar:/data/PredictionIO/vendors/spark/lib/datanuc
> leus-core-3.2.10.jar:/data/PredictionIO/vendors/spark/
> lib/datanucleus-rdbms-3.2.9.jar:/usr/hdp/2.5.3.0-37/hadoop
> /conf/:/usr/hdp/current/hadoop-client/lib/aws-java-sdk-s3-1.
> 10.6.jar:/usr/hdp/current/hadoop-client/lib/aws-java-
> sdk-core-1.10.6.jar:/usr/hdp/current/hadoop-client/lib/aws-java-sdk-kms-1.10.6.jar
> -Xms1g -Xmx1g -Dpio.log.dir=/home/da_mcom_milan
> org.apache.spark.deploy.SparkSubmit --conf spark.driver.extraClassPath=/d
> ata/PredictionIO-0.11.0-incubating/conf:/usr/hdp/2.5.3.0-37/
> hadoop/conf:/data/PredictionIO/vendors/hbase/conf --conf
> spark.driver.extraJavaOptions=-Dpio.log.dir=/home/da_mcom_milan --class
> org.apache.predictionio.workflow.CreateServer --jars
> file:/data/PredictionIO/some-engine/target/scala-2.10/univer
> sal-recommender_2.10-0.6.0.jar,file:/data/PredictionIO/some-engine
> /target/scala-2.10/universal-recommender-assembly
> -0.6.0-deps.jar,file:/data/PredictionIO-0.11.0-incubating
> /lib/spark/pio-data-elasticsearch1-assembly-0.11.0-incubatin
> g.jar,file:/data/PredictionIO-0.11.0-incubating/lib/spark/
> pio-data-hbase-assembly-0.11.0-incubating.jar,file:/data/
> PredictionIO-0.11.0-incubating/lib/spark/pio-data-
> hdfs-assembly-0.11.0-incubating.jar,file:/data/PredictionIO-
> 0.11.0-incubating/lib/spark/pio-data-jdbc-assembly-0.11.0-
> incubating.jar,file:/data/PredictionIO-0.11.0-incubating
> /lib/spark/pio-data-localfs-assembly-0.11.0-incubating.jar --files
> file:/data/PredictionIO-0.11.0-incubating/conf/log4j.propert
> ies,file:/usr/hdp/2.5.3.0-37/hadoop/conf/core-site.xml,file:
> /data/PredictionIO/vendors/hbase/conf/hbase-site.xml
> file:/data/PredictionIO-0.11.0-incubating/lib/pio-assembly-0.11.0-incubating.jar
> --engineInstanceId AWKSaLQrSe_zVbacBHWI --engine-variant
> file:/data/PredictionIO/some-engine/engine.json --ip 0.0.0.0 --port *8004*
> --event-server-ip 0.0.0.0 --event-server-port 7070 --json-extractor Both
> --env PIO_STORAGE_SOURCES_HBASE_TYPE=hbase,PIO_ENV_LOADED=1,PIO_ST
> ORAGE_REPOSITORIES_METADATA_NAME=pio_meta,PIO_FS_BASEDIR=/ho
> me/da_mcom_milan/.pio_store,PIO_STORAGE_SOURCES_ELASTICSEARC
> H_HOSTS=mdc2vra176,PIO_STORAGE_SOURCES_HBASE_HOME=/data/
> PredictionIO/vendors/hbase,PIO_HOME=/data/PredictionIO-0.
> 11.0-incubating,PIO_FS_ENGINESDIR=/home/da_mcom_
> milan/.pio_store/engines,PIO_STORAGE_SOURCES_LOCALFS_PATH=/
> home/da_mcom_milan/.pio_store/models,PIO_STORAGE_SOURCES_
> ELASTICSEARCH_TYPE=elasticsearch,PIO_STORAGE_REPOSITORIES_METADATA_SOURCE=
> ELASTICSEARCH,PIO_STORAGE_REPOSITORIES_MODELDATA_SOURCE=
> LOCALFS,PIO_STORAGE_REPOSITORIES_EVENTDATA_NAME=
> pio_event,PIO_STORAGE_SOURCES_ELASTICSEARCH_CLUSTERNAME=
> pros-dev,PIO_FS_TMPDIR=/home/da_mcom_milan/.pio_store/tmp,
> PIO_STORAGE_REPOSITORIES_MODELDATA_NAME=pio_model,PIO_
> STORAGE_REPOSITORIES_EVENTDATA_SOURCE=HBASE,PIO_
> CONF_DIR=/data/PredictionIO-0.11.0-incubating/conf,PIO_
> STORAGE_SOURCES_ELASTICSEARCH_PORTS=9300,PIO_STORAGE_
> SOURCES_LOCALFS_TYPE=localfs
>
> For some reasons, processes 30932 and 31097 still alive and process
> listened by 32671
> Any ideas?
>
> Thank you,
> George
>