You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@streampipes.apache.org by "dominikriemer (via GitHub)" <gi...@apache.org> on 2023/02/21 16:28:34 UTC
[PR] Support backend-only development mode (#1327) (streampipes)
dominikriemer opened a new pull request, #1339:
URL: https://github.com/apache/streampipes/pull/1339
<!--
~ Licensed to the Apache Software Foundation (ASF) under one or more
~ contributor license agreements. See the NOTICE file distributed with
~ this work for additional information regarding copyright ownership.
~ The ASF licenses this file to You under the Apache License, Version 2.0
~ (the "License"); you may not use this file except in compliance with
~ the License. You may obtain a copy of the License at
~
~ http://www.apache.org/licenses/LICENSE-2.0
~
~ Unless required by applicable law or agreed to in writing, software
~ distributed under the License is distributed on an "AS IS" BASIS,
~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
~ See the License for the specific language governing permissions and
~ limitations under the License.
~
-->
<!--
Thanks for contributing! Here are some tips you can follow to help us incorporate your contribution quickly and easily:
1. If this is your first time, please read our contributor guidelines:
- https://streampipes.apache.org/getinvolved.html
- https://cwiki.apache.org/confluence/display/STREAMPIPES/Getting+Started
2. Make sure the PR title is formatted like: `[#<GitHub issue id>] PR title ...`
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., `[WIP][#<GitHub issue id>] PR title ...`.
4. Please write your PR title to summarize what this PR proposes/fixes.
5. Link the PR to the corresponding GitHub issue (if present) in the `Development` section in the right menu bar.
6. Be sure to keep the PR description updated to reflect all changes.
7. If possible, provide a concise example to reproduce the issue for a faster review.
8. Make sure tests pass via `mvn clean install`.
9. (Optional) If the contribution is large, please file an Apache ICLA
- http://apache.org/licenses/icla.pdf
-->
### Purpose
<!--
Please clarify what changes you are proposing and describe how those changes will address the issue.
Furthermore, describe potential consequences the changes might have.
-->
Modify the `backend` environment which allows to run StreamPipes for backend-only development purposes.
Added a new service definition `ui-custom-conf` which adds a volume mapping from a custom Nginx `default.conf` file where the upstream of the reverse proxy can be set to a hostname that links to the locally running backend.
### Remarks
<!--
Is there anything left we need to pay attention on?
Are there some references that might be important? E.g. links to Confluence, or discussions
on the mailing list or GitHub.
-->
PR introduces (a) breaking change(s): no
PR introduces (a) deprecation(s): no
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@streampipes.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support backend-only development mode (#1327) (streampipes)
Posted by "dominikriemer (via GitHub)" <gi...@apache.org>.
dominikriemer merged PR #1339:
URL: https://github.com/apache/streampipes/pull/1339
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@streampipes.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support backend-only development mode (#1327) (streampipes)
Posted by "dominikriemer (via GitHub)" <gi...@apache.org>.
dominikriemer commented on PR #1339:
URL: https://github.com/apache/streampipes/pull/1339#issuecomment-1440269657
Great!
Can you check which IP or hostname is assigned to the backend upon start (there should be some logging info right at the beginning). The `extensions-all-jvm` service should receive the hostname of the backend from Consul and use this host to connect to the backend.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@streampipes.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support backend-only development mode (#1327) (streampipes)
Posted by "RobertIndie (via GitHub)" <gi...@apache.org>.
RobertIndie commented on code in PR #1339:
URL: https://github.com/apache/streampipes/pull/1339#discussion_r1115731662
##########
installer/cli/deploy/standalone/extensions-all-jvm/docker-compose.yml:
##########
@@ -19,10 +19,12 @@ services:
image: ${SP_DOCKER_REGISTRY}/extensions-all-jvm:${SP_VERSION}
depends_on:
- consul
- - backend
- couchdb
volumes:
- files:/spImages
+# Set SP_HOST to an IP or hostname that can be resolved from the Docker container and local-running backend
Review Comment:
Should we cleanup this comment?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@streampipes.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support backend-only development mode (#1327) (streampipes)
Posted by "RobertIndie (via GitHub)" <gi...@apache.org>.
RobertIndie commented on PR #1339:
URL: https://github.com/apache/streampipes/pull/1339#issuecomment-1441670768
Hi @dominikriemer
I restart all my services and it all works now! Thank you. This PR is awesome.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@streampipes.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support backend-only development mode (#1327) (streampipes)
Posted by "dominikriemer (via GitHub)" <gi...@apache.org>.
dominikriemer commented on PR #1339:
URL: https://github.com/apache/streampipes/pull/1339#issuecomment-1438809738
@RobertIndie I had a look at the backend-only configuration and think this setup has never worked as the UI could not properly connect to the locally running backend.
The `backend` environment now includes a new service `ui-custom-conf` where a volume mapping is included that links to a default.conf file that can be customized in the `deploy/standalone/ui-custom-conf` folder. I set this to `host.docker.internal` which should work on Windows and Mac systems but not under Linux (there the local IP needs to be set that can be resolved from the Docker container where the UI is running).
I also had some problems when trying to create adapters and pipelines with the extensions service running in Docker as more environment variables need to be set in order to make sure the extensions get the right broker connection settings from the backend. So I removed the `extensions-all-jvm` service and this would need to be started locally as well in this mode.
Sorry for the long explanation - can you please give it a try? I hope this workaround works ;-)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@streampipes.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support backend-only development mode (#1327) (streampipes)
Posted by "dominikriemer (via GitHub)" <gi...@apache.org>.
dominikriemer commented on code in PR #1339:
URL: https://github.com/apache/streampipes/pull/1339#discussion_r1115944057
##########
installer/cli/deploy/standalone/extensions-all-jvm/docker-compose.yml:
##########
@@ -19,10 +19,12 @@ services:
image: ${SP_DOCKER_REGISTRY}/extensions-all-jvm:${SP_VERSION}
depends_on:
- consul
- - backend
- couchdb
volumes:
- files:/spImages
+# Set SP_HOST to an IP or hostname that can be resolved from the Docker container and local-running backend
Review Comment:
I removed the comment and will add the info to the documentation, thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@streampipes.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support backend-only development mode (#1327) (streampipes)
Posted by "RobertIndie (via GitHub)" <gi...@apache.org>.
RobertIndie commented on PR #1339:
URL: https://github.com/apache/streampipes/pull/1339#issuecomment-1440145632
@dominikriemer Hi, Thanks for your PR. The UI works now!
But when I start `all-extensions-jvm` locally, it raises these errors:
```
2023-02-22T22:34:56.060+08:00 ERROR 70202 --- [ main] o.a.s.s.e.connect.ConnectRestClient : Could not register adapter at url - is a 'StreamPipes Core' service running?
org.apache.streampipes.commons.exceptions.SpRuntimeException: Could not connect to the StreamPipes API - please check that StreamPipes is available
at org.apache.streampipes.client.http.HttpRequest.executeRequest(HttpRequest.java:111)
at org.apache.streampipes.client.api.AbstractClientApi.post(AbstractClientApi.java:46)
at org.apache.streampipes.client.api.AdminApi.registerAdapters(AdminApi.java:36)
at org.apache.streampipes.service.extensions.connect.ConnectRestClient.register(ConnectRestClient.java:38)
at org.apache.streampipes.service.extensions.connect.ConnectWorkerRegistrationService.registerWorker(ConnectWorkerRegistrationService.java:35)
at org.apache.streampipes.service.extensions.ExtensionsModelSubmitter.afterServiceRegistered(ExtensionsModelSubmitter.java:53)
at org.apache.streampipes.service.extensions.StreamPipesExtensionsServiceBase.startExtensionsService(StreamPipesExtensionsServiceBase.java:77)
at org.apache.streampipes.service.extensions.StreamPipesExtensionsServiceBase.init(StreamPipesExtensionsServiceBase.java:54)
at org.apache.streampipes.service.extensions.StreamPipesExtensionsServiceBase.init(StreamPipesExtensionsServiceBase.java:44)
at org.apache.streampipes.extensions.all.jvm.AllExtensionsInit.main(AllExtensionsInit.java:38)
Caused by: org.apache.http.client.ClientProtocolException: null
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:187)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
at org.apache.http.client.fluent.Request.internalExecute(Request.java:173)
at org.apache.http.client.fluent.Request.execute(Request.java:177)
at org.apache.streampipes.client.http.HttpRequest.executeRequest(HttpRequest.java:98)
... 9 common frames omitted
Caused by: org.apache.http.ProtocolException: Target host is not specified
at org.apache.http.impl.conn.DefaultRoutePlanner.determineRoute(DefaultRoutePlanner.java:71)
at org.apache.http.impl.client.InternalHttpClient.determineRoute(InternalHttpClient.java:125)
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
... 14 common frames omitted
```
Do I miss something? Semes the target host is not specified. How can I set the host?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@streampipes.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org