You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by GitBox <gi...@apache.org> on 2020/12/14 17:11:04 UTC

[GitHub] [druid] harinirajendran opened a new pull request #10679: Adding example config files for indexer process. Also added status/selfDiscovered endpoint to indexer for self discovery.

harinirajendran opened a new pull request #10679:
URL: https://github.com/apache/druid/pull/10679


   In this PR I have added example config files that can be used to spin up the indexer process. And have also added the `status/selfDiscovered` endpoint to indexer which we use as readiness probe for indexer in our indexer process's docker config.
   
   Testing
   Built my own docker image, added the following in `docker-compose.yml` file
   ```
   +  indexer:
   +    image: apache/druid:harini
   +    container_name: indexer
   +    volumes:
   +      - ./storage:/opt/data
   +      - middle_var:/opt/druid/var
   +    depends_on:
   +      - zookeeper
   +      - postgres
   +      - coordinator
   +    ports:
   +      - "8092:8092"
   +    command:
   +      - indexer
   +    env_file:
   +      - environment
   +
   ```
   
   Brought all the services up
   ```
   hrajendran@Harini-Rajendran's-MBP16 docker % docker-compose up -d
   Creating network "docker_default" with the default driver
   Creating postgres  ... done
   Creating zookeeper ... done
   Creating coordinator ... done
   Creating indexer       ... done
   Creating router        ... done
   Creating broker        ... done
   Creating historical    ... done
   Creating middlemanager ... done
   hrajendran@Harini-Rajendran's-MBP16 docker % docker ps
   CONTAINER ID        IMAGE                 COMMAND                  CREATED             STATUS              PORTS                                    NAMES
   d4adafa2d20b        apache/druid:harini   "/druid.sh broker"       21 minutes ago      Up 21 minutes       0.0.0.0:8082->8082/tcp                   broker
   23d2d9dab52a        apache/druid:harini   "/druid.sh middleMan…"   21 minutes ago      Up 21 minutes       0.0.0.0:8091->8091/tcp                   middlemanager
   4af88a4f819b        apache/druid:harini   "/druid.sh historical"   21 minutes ago      Up 21 minutes       0.0.0.0:8083->8083/tcp                   historical
   8247c53aff87        apache/druid:harini   "/druid.sh router"       21 minutes ago      Up 21 minutes       0.0.0.0:8888->8888/tcp                   router
   664d268118dd        apache/druid:harini   "/druid.sh indexer"      21 minutes ago      Up 21 minutes       0.0.0.0:8092->8092/tcp                   indexer
   07ffca0bc593        apache/druid:harini   "/druid.sh coordinat…"   21 minutes ago      Up 21 minutes       0.0.0.0:8081->8081/tcp                   coordinator
   b16704e3ee3d        postgres:latest       "docker-entrypoint.s…"   22 minutes ago      Up 21 minutes       5432/tcp                                 postgres
   9a61879c560e        zookeeper:3.5         "/docker-entrypoint.…"   22 minutes ago      Up 21 minutes       2181/tcp, 2888/tcp, 3888/tcp, 8080/tcp   zookeeper
   ```
   
   Made `status/selfDiscovered` curl call to indexer running on port 8092 and got back 200 OK.
   ```
   hrajendran@Harini-Rajendran's-MBP16 docker % curl -v http://localhost:8092/status/selfDiscovered
   *   Trying ::1...
   * TCP_NODELAY set
   * Connected to localhost (::1) port 8092 (#0)
   > GET /status/selfDiscovered HTTP/1.1
   > Host: localhost:8092
   > User-Agent: curl/7.64.1
   > Accept: */*
   > 
   < HTTP/1.1 200 OK
   < Date: Mon, 14 Dec 2020 17:09:49 GMT
   < Content-Type: application/json
   < Vary: Accept-Encoding, User-Agent
   < Content-Length: 0
   < 
   * Connection #0 to host localhost left intact
   * Closing connection 0
   ```


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org

[GitHub] [druid] xvrl commented on a change in pull request #10679: Adding example config files for indexer process. Also added status/selfDiscovered endpoint to indexer for self discovery.

Posted by GitBox <gi...@apache.org>.
xvrl commented on a change in pull request #10679:
URL: https://github.com/apache/druid/pull/10679#discussion_r542563847



##########
File path: examples/conf/druid/cluster/data/indexer/runtime.properties
##########
@@ -0,0 +1,38 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+druid.service=druid/indexer
+druid.plaintextPort=8092

Review comment:
       any reason this is different than the default indexer port?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org

[GitHub] [druid] xvrl commented on a change in pull request #10679: Adding example config files for indexer process. Also added status/selfDiscovered endpoint to indexer for self discovery.

Posted by GitBox <gi...@apache.org>.
xvrl commented on a change in pull request #10679:
URL: https://github.com/apache/druid/pull/10679#discussion_r542784759



##########
File path: examples/conf/druid/cluster/data/indexer/runtime.properties
##########
@@ -0,0 +1,38 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+druid.service=druid/indexer
+druid.plaintextPort=8091
+
+# Number of tasks per indexer
+druid.worker.capacity=2

Review comment:
       the corresponding middleManager config has 4 tasks, should we match that for consistency?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org

[GitHub] [druid] xvrl commented on pull request #10679: Adding example config files for indexer process. Also added status/selfDiscovered endpoint to indexer for self discovery

Posted by GitBox <gi...@apache.org>.
xvrl commented on pull request #10679:
URL: https://github.com/apache/druid/pull/10679#issuecomment-745021418


   merging since test failures appear unrelated.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org

[GitHub] [druid] xvrl merged pull request #10679: Adding example config files for indexer process. Also added status/selfDiscovered endpoint to indexer for self discovery

Posted by GitBox <gi...@apache.org>.
xvrl merged pull request #10679:
URL: https://github.com/apache/druid/pull/10679


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org

[GitHub] [druid] xvrl commented on a change in pull request #10679: Adding example config files for indexer process. Also added status/selfDiscovered endpoint to indexer

Posted by GitBox <gi...@apache.org>.
xvrl commented on a change in pull request #10679:
URL: https://github.com/apache/druid/pull/10679#discussion_r542786845



##########
File path: examples/conf/druid/cluster/data/indexer/jvm.config
##########
@@ -0,0 +1,9 @@
+-server
+-Xms128m
+-Xmx128m
+-XX:MaxDirectMemorySize=1g

Review comment:
       we probably need more heap to match our middle-manager config, which had 1GB per task.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org

[GitHub] [druid] xvrl commented on a change in pull request #10679: Adding example config files for indexer process. Also added status/selfDiscovered endpoint to indexer

Posted by GitBox <gi...@apache.org>.
xvrl commented on a change in pull request #10679:
URL: https://github.com/apache/druid/pull/10679#discussion_r542786016



##########
File path: examples/conf/druid/cluster/data/indexer/runtime.properties
##########
@@ -0,0 +1,38 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+druid.service=druid/indexer
+druid.plaintextPort=8091
+
+# Number of tasks per indexer
+druid.worker.capacity=2
+
+# Task launch parameters
+druid.indexer.task.baseTaskDir=var/druid/task
+
+# HTTP server threads
+druid.server.http.numThreads=60
+
+# Processing threads and buffers on Indexer
+druid.processing.numMergeBuffers=2
+druid.processing.buffer.sizeBytes=100MiB
+druid.processing.numThreads=1

Review comment:
       given we have 1 processing thread per task in the example middle-manager config, I would set this to equal the worker capacity to have a comparable config.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org

[GitHub] [druid] harinirajendran commented on a change in pull request #10679: Adding example config files for indexer process. Also added status/selfDiscovered endpoint to indexer for self discovery

Posted by GitBox <gi...@apache.org>.
harinirajendran commented on a change in pull request #10679:
URL: https://github.com/apache/druid/pull/10679#discussion_r542796281



##########
File path: examples/conf/druid/cluster/data/indexer/runtime.properties
##########
@@ -0,0 +1,38 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+druid.service=druid/indexer
+druid.plaintextPort=8091
+
+# Number of tasks per indexer
+druid.worker.capacity=2

Review comment:
       Sure! I can update it to match with MM config.

##########
File path: examples/conf/druid/cluster/data/indexer/jvm.config
##########
@@ -0,0 +1,9 @@
+-server
+-Xms128m
+-Xmx128m
+-XX:MaxDirectMemorySize=1g

Review comment:
       Made it 4g and also increased `druid.processing.buffer.sizeBytes` to 500MiB 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org

[GitHub] [druid] harinirajendran commented on a change in pull request #10679: Adding example config files for indexer process. Also added status/selfDiscovered endpoint to indexer for self discovery

Posted by GitBox <gi...@apache.org>.
harinirajendran commented on a change in pull request #10679:
URL: https://github.com/apache/druid/pull/10679#discussion_r542803426



##########
File path: examples/conf/druid/cluster/data/indexer/jvm.config
##########
@@ -0,0 +1,9 @@
+-server
+-Xms128m
+-Xmx128m
+-XX:MaxDirectMemorySize=1g

Review comment:
       Made it 4g and also increased `druid.processing.buffer.sizeBytes` to 500MiB . Also increased -Xms and -Xmx setting to be 4x.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org

[GitHub] [druid] harinirajendran commented on a change in pull request #10679: Adding example config files for indexer process. Also added status/selfDiscovered endpoint to indexer for self discovery.

Posted by GitBox <gi...@apache.org>.
harinirajendran commented on a change in pull request #10679:
URL: https://github.com/apache/druid/pull/10679#discussion_r542741175



##########
File path: examples/conf/druid/cluster/data/indexer/runtime.properties
##########
@@ -0,0 +1,38 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+druid.service=druid/indexer
+druid.plaintextPort=8092

Review comment:
       Actually, I set it back to 8091 and tested the docker image. To avoid localhost port conflict I was using the following port mappings: 8091:8091 for MM and 8092:8092 for indexer. Now updated the indexer port mapping to 8092:8091.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org

[GitHub] [druid] xvrl commented on a change in pull request #10679: Adding example config files for indexer process. Also added status/selfDiscovered endpoint to indexer for self discovery

Posted by GitBox <gi...@apache.org>.
xvrl commented on a change in pull request #10679:
URL: https://github.com/apache/druid/pull/10679#discussion_r542820582



##########
File path: examples/conf/druid/cluster/data/indexer/runtime.properties
##########
@@ -0,0 +1,38 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+druid.service=druid/indexer
+druid.plaintextPort=8091
+
+# Number of tasks per indexer
+druid.worker.capacity=4
+
+# Task launch parameters
+druid.indexer.task.baseTaskDir=var/druid/task
+
+# HTTP server threads
+druid.server.http.numThreads=60
+
+# Processing threads and buffers on Indexer
+druid.processing.numMergeBuffers=2
+druid.processing.buffer.sizeBytes=500MiB

Review comment:
       processing buffers are per thread, so we you can leave that to 100MB to match the middle manager




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org