[jira] [Created] (NIFI-11482) Make repository paths configurable via Docker env vars

["Tamas Neumer (Jira)" <ji...@apache.org>]

Tamas Neumer created NIFI-11482:
-----------------------------------

             Summary: Make repository paths configurable via Docker env vars
                 Key: NIFI-11482
                 URL: https://issues.apache.org/jira/browse/NIFI-11482
             Project: Apache NiFi
          Issue Type: New Feature
          Components: Configuration, Docker
    Affects Versions: 1.21.0
            Reporter: Tamas Neumer


Hi,

I'm running Apache NiFi in Docker. To make it stateful I had to apply the following changes:

 *nifi.properties*
{code:java}
nifi.database.directory=/opt/nifi/repositories/database_repository
nifi.flowfile.repository.directory=/opt/nifi/repositories/flowfile_repository
nifi.content.repository.directory.default=/opt/nifi/repositories/content_repository
nifi.provenance.repository.directory.default=/opt/nifi/repositories/provenance_repository
nifi.status.repository.questdb.persist.location=/opt/nifi/repositories/status_repository
nifi.flow.configuration.file=/opt/nifi/repositories/configuration_repository/flow.xml.gz
nifi.flow.configuration.json.file=/opt/nifi/repositories/configuration_repository/flow.json.gz
nifi.flow.configuration.archive.dir=/opt/nifi/repositories/configuration_repository/archive/ {code}
*state-management.xml*
{code:java}
<property name="Directory">/opt/nifi/repositories/state_repository/local</property>{code}

It would be really convenient to set these values using environment variables, however the current NiFi Docker image has no support for this.

As a temporary solution I have overwritten the image's default entrypoint and used a self-written script:
{code:java}
#!/bin/bash  
  
scripts_dir='/opt/nifi/scripts'  
  
[ -f "$${scripts_dir}/common.sh" ] && . "$${scripts_dir}/common.sh"  
  
prop_replace 'nifi.database.directory'                          "/opt/nifi/repositories/database_repository"  
prop_replace 'nifi.flowfile.repository.directory'               "/opt/nifi/repositories/flowfile_repository"  
prop_replace 'nifi.content.repository.directory.default'        "/opt/nifi/repositories/content_repository"  
prop_replace 'nifi.provenance.repository.directory.default'     "/opt/nifi/repositories/provenance_repository"  
prop_replace 'nifi.status.repository.questdb.persist.location'  "/opt/nifi/repositories/status_repository"  
prop_replace 'nifi.flow.configuration.file'                     "/opt/nifi/repositories/configuration_repository/flow.xml.gz"  
prop_replace 'nifi.flow.configuration.json.file'                "/opt/nifi/repositories/configuration_repository/flow.json.gz"  
prop_replace 'nifi.flow.configuration.archive.dir'              "/opt/nifi/repositories/configuration_repository/archive/"  
  
echo "Hacking nifi.properties for stateful operation completed"  
  
sed -i -e 's|<property name="Directory">./state/local</property>|<property name="Directory">/opt/nifi/repositories/state_repository/local</property>|'  $${NIFI_HOME}/conf/state-management.xml  
  
echo "Hacking state-management.xml for stateful operation completed"  
echo "Calling start.sh"  
  
. /opt/nifi/scripts/start.sh{code}
Proposed solution

I would like to craete a new shell-script ({*}update_stateful_paths.sh{*}) for setting the required variables:
{code:java}
#!/bin/sh -e
#    Licensed to the Apache Software Foundation (ASF) under one or more
#    contributor license agreements.  See the NOTICE file distributed with
#    this work for additional information regarding copyright ownership.
#    The ASF licenses this file to You under the Apache License, Version 2.0
#    (the "License"); you may not use this file except in compliance with
#    the License.  You may obtain a copy of the License at
#
#       http://www.apache.org/licenses/LICENSE-2.0
#
#    Unless required by applicable law or agreed to in writing, software
#    distributed under the License is distributed on an "AS IS" BASIS,
#    WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
#    See the License for the specific language governing permissions and
#    limitations under the License.
prop_replace 'nifi.database.directory'                          "${NIFI_DATABASE_DIRECTORY:-./database_repository}"
prop_replace 'nifi.flowfile.repository.directory'               "${NIFI_FLOWFILE_REPOSITORY_DIRECTORY:-./flowfile_repository}"
prop_replace 'nifi.content.repository.directory.default'        "${NIFI_CONTENT_REPOSITORY_DIRECTORY_DEFAULT:-./content_repository}"
prop_replace 'nifi.provenance.repository.directory.default'     "${NIFI_PROVENANCE_REPOSITORY_DIRECTORY_DEFAULT:-./provenance_repository}"
prop_replace 'nifi.status.repository.questdb.persist.location'  "${NIFI_STATUS_REPOSITORY_QUESTDB_PERSIST_LOCATION:-./status_repository}"
prop_replace 'nifi.flow.configuration.file'                     "${NIFI_FLOW_CONFIGURATION_FILE:-./conf/flow.xml.gz}"
prop_replace 'nifi.flow.configuration.json.file'                "${NIFI_FLOW_CONFIGURATION_JSON_FILE:-./conf/flow.json.gz}"
prop_replace 'nifi.flow.configuration.archive.dir'              "${NIFI_FLOW_CONFIGURATION_ARCHIVE_DIR:-./conf/archive/}"
sed -i -e "s|<property name=\"Directory\">./state/local</property>|<property name=\"Directory\">${NIFI_STATE_REPOSITORY_DIRECTORY:-./state/local}</property>|"  ${NIFI_HOME}/conf/state-management.xml
{code}
The *start.sh* would call this new script:

 
{code:java}
        . "${scripts_dir}/secure.sh"
        . "${scripts_dir}/update_oidc_properties.sh"
        ;;
esac
# ...
# NEW LINE:
. "${scripts_dir}/update_stateful_paths.sh"
# ...
"${NIFI_HOME}/bin/nifi.sh" run &
nifi_pid="$!"
tail -F --pid=${nifi_pid} "${NIFI_HOME}/logs/nifi-app.log" &
{code}
I have tested it with the following env file:
{code:java}
NIFI_DATABASE_DIRECTORY=/opt/nifi/repositories/db
NIFI_FLOWFILE_REPOSITORY_DIRECTORY=/opt/nifi/repositories/flow
NIFI_CONTENT_REPOSITORY_DIRECTORY_DEFAULT=/opt/nifi/repositories/content
NIFI_PROVENANCE_REPOSITORY_DIRECTORY_DEFAULT=/opt/nifi/repositories/prov
NIFI_STATUS_REPOSITORY_QUESTDB_PERSIST_LOCATION=/opt/nifi/repositories/status
NIFI_FLOW_CONFIGURATION_FILE=/opt/nifi/repositories/conf/flow.xml.gz
NIFI_FLOW_CONFIGURATION_JSON_FILE=/opt/nifi/repositories/conf/flow.json.gz
NIFI_FLOW_CONFIGURATION_ARCHIVE_DIR=/opt/nifi/repositories/conf/archive
NIFI_STATE_REPOSITORY_DIRECTORY=/opt/nifi/repositories/state{code}
And then running Docker:
{code:java}
docker run -it \
--env-file /Users/tneumer/Downloads/nifi_env_file \
-v /Users/tneumer/Downloads/nifi_temp:/opt/nifi/repositories \
-v /Users/tneumer/temp/nifi/nifi-docker/dockerhub/sh/start.sh:/opt/nifi/scripts/start.sh \
-v /Users/tneumer/temp/nifi/nifi-docker/dockerhub/sh/update_stateful_paths.sh:/opt/nifi/scripts/update_stateful_paths.sh \
apache/nifi:1.21.0 bash{code}
If you approve, I would create a pull-request with the proposed solution!

Kind regards,
Tamas



--
This message was sent by Atlassian Jira
(v8.20.10#820010)