You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@dolphinscheduler.apache.org by wenhe li <li...@gmail.com> on 2020/02/19 07:42:40 UTC
[SUGGESTION] how to build a docker image
hi,
how to build a docker image?
For now, I created a new docker image for my application, and its size is
about 600MB.
The docker image startup as follow:
You can start a dolphinscheduler instance
1. $ docker run -d --name dolphinscheduler \
2. -e POSTGRESQL_USERNAME=test -e POSTGRESQL_PASSWORD=test \
3. -p 8888:8888 \
4. dolphinscheduler all
The default postgres user root, postgres password root and database
dolphinscheduler are created in the startup.sh.
The default zookeeper is created in the startup.sh.
Or via Environment Variables POSTGRESQL_HOST POSTGRESQL_PORT
ZOOKEEPER_QUORUM
You can specify existing postgres service. Example:
1. $ docker run -d --name dolphinscheduler \
2. -e POSTGRESQL_HOST="192.168.x.x" -e POSTGRESQL_PORT="5432" \
3. -e POSTGRESQL_USERNAME="test" -e POSTGRESQL_PASSWORD="test" \
4. -p 8888:8888 \
5. dolphinscheduler all
You can specify existing zookeeper service. Example:
1. $ docker run -d --name dolphinscheduler \
2. -e ZOOKEEPER_QUORUM="l92.168.x.x:2181"
3. -e POSTGRESQL_USERNAME="test" -e POSTGRESQL_PASSWORD="test" \
4. -p 8888:8888 \
5. dolphinscheduler all
Or start a standalone dolphinscheduler server
You can start a standalone dolphinscheduler server.
- Start a master server, For example:
1. $ docker run -d --name dolphinscheduler \
2. -e ZOOKEEPER_QUORUM="l92.168.x.x:2181"
3. -e POSTGRESQL_HOST="192.168.x.x" -e POSTGRESQL_PORT="5432" \
4. -e POSTGRESQL_USERNAME="test" -e POSTGRESQL_PASSWORD="test" \
5. dolphinscheduler master-server
- Start a worker server, For example:
1. $ docker run -d --name dolphinscheduler \
2. -e ZOOKEEPER_QUORUM="l92.168.x.x:2181"
3. -e POSTGRESQL_HOST="192.168.x.x" -e POSTGRESQL_PORT="5432" \
4. -e POSTGRESQL_USERNAME="test" -e POSTGRESQL_PASSWORD="test" \
5. dolphinscheduler worker-server
- Start a api server, For example:
1. $ docker run -d --name dolphinscheduler \
2. -e POSTGRESQL_HOST="192.168.x.x" -e POSTGRESQL_PORT="5432" \
3. -e POSTGRESQL_USERNAME="test" -e POSTGRESQL_PASSWORD="test" \
4. -p 12345:12345 \
5. dolphinscheduler api-server
- Start a alert server, For example:
1. $ docker run -d --name dolphinscheduler \
2. -e POSTGRESQL_HOST="192.168.x.x" -e POSTGRESQL_PORT="5432" \
3. -e POSTGRESQL_USERNAME="test" -e POSTGRESQL_PASSWORD="test" \
4. dolphinscheduler alert-server
- Start a frontend, For example:
1. $ docker run -d --name dolphinscheduler \
2. -e FRONTEND_API_SERVER_HOST="192.168.x.x" -e
FRONTEND_API_SERVER_PORT="12345" \
3. -p 8888:8888 \
4. dolphinscheduler frontend
Note: You must be specify POSTGRESQL_HOST POSTGRESQL_PORT ZOOKEEPER_QUORUM when
start a standalone dolphinscheduler server.
Environment Variables
The Dolphin Scheduler image uses several environment variables which are
easy to miss. While none of the variables are required, they may
significantly aid you in using the image.
POSTGRESQL_HOST
This environment variable sets the host for PostgreSQL. The default value
is 127.0.0.1.
Note: You must be specify it when start a standalone dolphinscheduler
server. Like master-server, worker-server, api-server, alert-server.
POSTGRESQL_PORT
This environment variable sets the port for PostgreSQL. The default value
is 5432.
Note: You must be specify it when start a standalone dolphinscheduler
server. Like master-server, worker-server, api-server, alert-server.
POSTGRESQL_USERNAME
This environment variable sets the username for PostgreSQL. The default
value is root.
POSTGRESQL_PASSWORD
This environment variable sets the password for PostgreSQL. The default
value is root.
DOLPHINSCHEDULER_ENV_PATH
This environment variable sets the runtime environment for task. The
default value is /opt/dolphinscheduler/conf/env/dolphinscheduler_env.sh.
TASK_QUEUE
This environment variable sets the task queue for master-server and
worker-serverr. The default value is zookeeper.
ZOOKEEPER_QUORUM
This environment variable sets zookeeper quorum for master-server and
worker-serverr. The default value is 127.0.0.1:2181.
Note: You must be specify it when start a standalone dolphinscheduler
server. Like master-server, worker-server.
MASTER_EXEC_THREADS
This environment variable sets exec thread num for master-server. The
default value is 100.
MASTER_EXEC_TASK_NUM
This environment variable sets exec task num for master-server. The default
value is 20.
MASTER_HEARTBEAT_INTERVAL
This environment variable sets heartbeat interval for master-server. The
default value is 10.
MASTER_TASK_COMMIT_RETRYTIMES
This environment variable sets task commit retry times for master-server.
The default value is 5.
MASTER_TASK_COMMIT_INTERVAL
This environment variable sets task commit interval for master-server. The
default value is 1000.
MASTER_MAX_CPULOAD_AVG
This environment variable sets max cpu load avg for master-server. The
default value is 100.
MASTER_RESERVED_MEMORY
This environment variable sets reserved memory for master-server. The
default value is 0.1.
WORKER_EXEC_THREADS
This environment variable sets exec thread num for worker-server. The
default value is 100.
WORKER_HEARTBEAT_INTERVAL
This environment variable sets heartbeat interval for worker-server. The
default value is 10.
WORKER_FETCH_TASK_NUM
This environment variable sets fetch task num for worker-server. The
default value is 3.
WORKER_MAX_CPULOAD_AVG
This environment variable sets max cpu load avg for worker-server. The
default value is 100.
WORKER_RESERVED_MEMORY
This environment variable sets reserved memory for worker-server. The
default value is 0.1.
XLS_FILE_PATH
This environment variable sets xls file path for alert-server. The default
value is /tmp/xls.
MAIL_SERVER_HOST
This environment variable sets mail server host for alert-server. The
default value is empty.
MAIL_SERVER_PORT
This environment variable sets mail server port for alert-server. The
default value is empty.
MAIL_SENDER
This environment variable sets mail sender for alert-server. The default
value is empty.
MAIL_USER=
This environment variable sets mail user for alert-server. The default
value is empty.
MAIL_PASSWD
This environment variable sets mail password for alert-server. The default
value is empty.
MAIL_SMTP_STARTTLS_ENABLE
This environment variable sets SMTP tls for alert-server. The default value
is true.
MAIL_SMTP_SSL_ENABLE
This environment variable sets SMTP ssl for alert-server. The default value
is false.
MAIL_SMTP_SSL_TRUST
This environment variable sets SMTP ssl truest for alert-server. The
default value is empty.
ENTERPRISE_WECHAT_ENABLE
This environment variable sets enterprise wechat enable for alert-server.
The default value is false.
ENTERPRISE_WECHAT_CORP_ID
This environment variable sets enterprise wechat corp id for alert-server.
The default value is empty.
ENTERPRISE_WECHAT_SECRET
This environment variable sets enterprise wechat secret for alert-server.
The default value is empty.
ENTERPRISE_WECHAT_AGENT_ID
This environment variable sets enterprise wechat agent id for alert-server.
The default value is empty.
ENTERPRISE_WECHAT_USERS
This environment variable sets enterprise wechat users for alert-server.
The default value is empty.
FRONTEND_API_SERVER_HOST
This environment variable sets api server host for frontend. The default
value is 127.0.0.1.
Note: You must be specify it when start a standalone dolphinscheduler
server. Like api-server.
FRONTEND_API_SERVER_PORT
This environment variable sets api server port for frontend. The default
value is 123451.
Note: You must be specify it when start a standalone dolphinscheduler
server. Like api-server.
Initialization scripts
If you would like to do additional initialization in an image derived from
this one, add one or more environment variable under
/root/start-init-conf.sh, and modify template files in
/opt/dolphinscheduler/conf/*.tpl.
For example, to add an environment variable API_SERVER_PORT in
/root/start-init-conf.sh:
1. export API_SERVER_PORT=5555
and to modify /opt/dolphinscheduler/conf/application-api.properties.tpl
template
file, add server port:
1. server.port=${API_SERVER_PORT}
/root/start-init-conf.sh will dynamically generate config file:
1. echo "generate app config"
2. ls ${DOLPHINSCHEDULER_HOME}/conf/ | grep ".tpl" | while read line; do
3. eval "cat << EOF
4. $(cat ${DOLPHINSCHEDULER_HOME}/conf/${line})
5. EOF
6. " > ${DOLPHINSCHEDULER_HOME}/conf/${line%.*}
7. done
8.
9. echo "generate nginx config"
10. sed -i "s/FRONTEND_API_SERVER_HOST/${FRONTEND_API_SERVER_HOST}/g"
/etc/nginx/conf.d/dolphinscheduler.conf
11. sed -i "s/FRONTEND_API_SERVER_PORT/${FRONTEND_API_SERVER_PORT}/g"
/etc/nginx/conf.d/dolphinscheduler.conf
Re: [SUGGESTION] how to build a docker image
Posted by Xiaochun Liu <li...@apache.org>.
I want to add a suggestion:
Can we keep the single responsibility of the docker image and only support one service,
then each service needs to be started separately, and we can add a startdocker.sh script to start all,
especially zk and postgresql external services, it can also reduce the size of the image
Best Regards
---------------
DolphinScheduler(Incubator) Committer
Xiaochun Liu 刘小春
liuxiaochun@apache.org
---------------
> 在 2020年2月19日,下午4:25,Xiaochun Liu <li...@apache.org> 写道:
>
> It’s a great suggestion.
> But I have a problem, do you have zk and pgsql built in DolphinScheduler?
>
> Best Regards
> ---------------
> DolphinScheduler(Incubator) Committer
> Xiaochun Liu 刘小春
> liuxiaochun@apache.org <ma...@apache.org>
> ---------------
>
>
>
>> 在 2020年2月19日,下午3:42,wenhe li <liwenhe93@gmail.com <ma...@gmail.com>> 写道:
>>
>> 1. echo "generate app config"
>> 2. ls ${DOLPHINSCHEDULER_HOME}/conf/ | grep ".tpl" | while read line; do
>> 3. eval "cat << EOF
>> 4. $(cat ${DOLPHINSCHEDULER_HOME}/conf/${line})
>> 5. EOF
>> 6. " > ${DOLPHINSCHEDULER_HOME}/conf/${line%.*}
>> 7. done
>> 8.
>> 9. echo "generate nginx config"
>> 10. sed -i "s/FRONTEND_API_SERVER_HOST/${FRONTEND_API_SERVER_HOST}/g"
>> /etc/nginx/conf.d/dolphinscheduler.conf
>> 11. sed -i "s/FRONTEND_API_SERVER_PORT/${FRONTEND_API_SERVER_PORT}/g"
>> /etc/nginx/conf.d/dolphinscheduler.conf
>
Re: [SUGGESTION] how to build a docker image
Posted by Xiaochun Liu <li...@apache.org>.
It’s a great suggestion.
But I have a problem, do you have zk and pgsql built in DolphinScheduler?
Best Regards
---------------
DolphinScheduler(Incubator) Committer
Xiaochun Liu 刘小春
liuxiaochun@apache.org
---------------
> 在 2020年2月19日,下午3:42,wenhe li <li...@gmail.com> 写道:
>
> 1. echo "generate app config"
> 2. ls ${DOLPHINSCHEDULER_HOME}/conf/ | grep ".tpl" | while read line; do
> 3. eval "cat << EOF
> 4. $(cat ${DOLPHINSCHEDULER_HOME}/conf/${line})
> 5. EOF
> 6. " > ${DOLPHINSCHEDULER_HOME}/conf/${line%.*}
> 7. done
> 8.
> 9. echo "generate nginx config"
> 10. sed -i "s/FRONTEND_API_SERVER_HOST/${FRONTEND_API_SERVER_HOST}/g"
> /etc/nginx/conf.d/dolphinscheduler.conf
> 11. sed -i "s/FRONTEND_API_SERVER_PORT/${FRONTEND_API_SERVER_PORT}/g"
> /etc/nginx/conf.d/dolphinscheduler.conf