You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@hop.apache.org by "ravamo (via GitHub)" <gi...@apache.org> on 2024/11/28 11:10:04 UTC
[D] Question About Database Connections in Apache Hop Server Mode (hop)
GitHub user ravamo created a discussion: Question About Database Connections in Apache Hop Server Mode
Hi,
I have a question. I'm using Apache Hop in server mode and triggering the ETLs through the API. Do I need to manually create the database connection file on the server? Is there a way for it to be created automatically, similar to Pentaho? What would be the best approach?
Thanks
GitHub link: https://github.com/apache/hop/discussions/4634
----
This is an automatically sent email for users@hop.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@hop.apache.org
Re: [D] Question About Database Connections in Apache Hop Server Mode (hop)
Posted by "hansva (via GitHub)" <gi...@apache.org>.
GitHub user hansva added a comment to the discussion: Question About Database Connections in Apache Hop Server Mode
The only thing I am not sure of is changes will work without restarting the server
GitHub link: https://github.com/apache/hop/discussions/4634#discussioncomment-11415462
----
This is an automatically sent email for users@hop.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@hop.apache.org
Re: [D] Question About Database Connections in Apache Hop Server Mode (hop)
Posted by "ravamo (via GitHub)" <gi...@apache.org>.
GitHub user ravamo added a comment to the discussion: Question About Database Connections in Apache Hop Server Mode
let me check !! thanks
GitHub link: https://github.com/apache/hop/discussions/4634#discussioncomment-11405726
----
This is an automatically sent email for users@hop.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@hop.apache.org
Re: [D] Question About Database Connections in Apache Hop Server Mode (hop)
Posted by "ravamo (via GitHub)" <gi...@apache.org>.
GitHub user ravamo added a comment to the discussion: Question About Database Connections in Apache Hop Server Mode
Thanks, @hansva , for such a detailed response, especially over the weekend—it wasn’t necessary. As I mentioned before, I didn’t mean to sound rude or bother you. I just wanted to fully understand everything to provide my client with a solid solution, and if I need to create a tool, make sure it actually makes sense.
GitHub link: https://github.com/apache/hop/discussions/4634#discussioncomment-11424396
----
This is an automatically sent email for users@hop.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@hop.apache.org
Re: [D] Question About Database Connections in Apache Hop Server Mode (hop)
Posted by "pr-desai (via GitHub)" <gi...@apache.org>.
GitHub user pr-desai deleted a comment on the discussion: Question About Database Connections in Apache Hop Server Mode
we are getting the response as like this
<HTML>
<HEAD>
<TITLE>
!RegisterPipelineServlet.Title!
</TITLE>
<META http-equiv="Content-Type" content="text/html; charset=UTF-8">
<link rel="icon" type="image/svg+xml" href="/static/images/favicon.svg">
</HEAD>
<BODY>
<p>
<pre>
org.apache.hop.core.exception.HopXmlException:
Error reading information from XML string :
Premature end of file.
at org.apache.hop.core.xml.XmlHandler.loadXmlString(XmlHandler.java:719)
at org.apache.hop.core.xml.XmlHandler.loadXmlString(XmlHandler.java:699)
at org.apache.hop.core.xml.XmlHandler.loadXmlString(XmlHandler.java:673)
at org.apache.hop.pipeline.PipelineConfiguration.fromXml(PipelineConfiguration.java:79)
at org.apache.hop.www.RegisterPipelineServlet.generateBody(RegisterPipelineServlet.java:54)
at org.apache.hop.www.BodyHttpServlet.doGet(BodyHttpServlet.java:65)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:687)
at org.apache.hop.www.BaseHttpServlet.service(BaseHttpServlet.java:102)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:550)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1435)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1350)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:234)
at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:59)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:567)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.eclipse.jetty.server.Server.handle(Server.java:516)
at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:388)
at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:633)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:380)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:383)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:882)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1036)
at java.base/java.lang.Thread.run(Thread.java:842)
Caused by: org.xml.sax.SAXParseException; Premature end of file.
at org.apache.xerces.parsers.DOMParser.parse(Unknown Source)
at org.apache.xerces.jaxp.DocumentBuilderImpl.parse(Unknown Source)
at org.apache.hop.core.xml.XmlHandler.loadXmlString(XmlHandler.java:710)
... 40 more
</pre>
<p>
</BODY>
</HTML>
request: http://localhost:8080/hop/registerPipeline
Payload details for the registerPipeline
<pipeline_configuration>
<pipeline>
<filename>C:/Users/shiva/Downloads/apache-hop-client-2.14.0/hop/config/projects/default/Exampl.hpl</filename>
</pipeline>
<pipeline_execution_configuration>
<variables></variables>
<parameters></parameters>
<pass_export>N</pass_export>
<log_level>Basic</log_level>
<log_file>N</log_file>
<log_filename></log_filename>
<log_file_append>N</log_file_append>
<create_parent_folder>N</create_parent_folder>
<clear_log>Y</clear_log>
<show_subcomponents>Y</show_subcomponents>
<run_configuration>local</run_configuration>
</pipeline_execution_configuration>
<metastore_json></metastore_json>
</pipeline_configuration>
GitHub link: https://github.com/apache/hop/discussions/4634#discussioncomment-14857997
----
This is an automatically sent email for users@hop.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@hop.apache.org
Re: [D] Question About Database Connections in Apache Hop Server Mode (hop)
Posted by "UEMravamo (via GitHub)" <gi...@apache.org>.
GitHub user UEMravamo deleted a comment on the discussion: Question About Database Connections in Apache Hop Server Mode
yes we've this tag complete `<metadata_folder>/servers/hop/hop/config/projects/default/metadata</metadata_folder>`
GitHub link: https://github.com/apache/hop/discussions/4634#discussioncomment-11471299
----
This is an automatically sent email for users@hop.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@hop.apache.org
Re: [D] Question About Database Connections in Apache Hop Server Mode (hop)
Posted by "bamaer (via GitHub)" <gi...@apache.org>.
GitHub user bamaer added a comment to the discussion: Question About Database Connections in Apache Hop Server Mode
what's a pentaho? do you export the connections with the metadata when you register your workflows/pipelines on the server?
GitHub link: https://github.com/apache/hop/discussions/4634#discussioncomment-11405172
----
This is an automatically sent email for users@hop.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@hop.apache.org
Re: [D] Question About Database Connections in Apache Hop Server Mode (hop)
Posted by "hansva (via GitHub)" <gi...@apache.org>.
GitHub user hansva added a comment to the discussion: Question About Database Connections in Apache Hop Server Mode
to get a full list of everything that is configured inside your hop client is to start the client, click on the logo in the top left button and press the "export metadata.json" button.
GitHub link: https://github.com/apache/hop/discussions/4634#discussioncomment-11965436
----
This is an automatically sent email for users@hop.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@hop.apache.org
Re: [D] Question About Database Connections in Apache Hop Server Mode (hop)
Posted by "RaffaelloC94 (via GitHub)" <gi...@apache.org>.
GitHub user RaffaelloC94 added a comment to the discussion: Question About Database Connections in Apache Hop Server Mode
I have a running server, so i can send my workflow+pipeline using my local hop gui? How can i do that?
GitHub link: https://github.com/apache/hop/discussions/4634#discussioncomment-13295073
----
This is an automatically sent email for users@hop.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@hop.apache.org
Re: [D] Question About Database Connections in Apache Hop Server Mode (hop)
Posted by "ggrohmann (via GitHub)" <gi...@apache.org>.
GitHub user ggrohmann edited a comment on the discussion: Question About Database Connections in Apache Hop Server Mode
Hey folks,
I have a question, which is related to the described matter, in particular running a project on a remote server. I managed to run single pipelines and workflows on a hop server in a docker container (long-lived). However, I struggle with setting up the run configuration for a workflow with linked pipelines and sub workflows. I followed the documentation as far as I could (https://hop.apache.org//manual/latest/workflow/workflow-run-configurations/native-remote-workflow-engine.html).
However, I receive the error, that the pipelines couldn't be resolved, when I run the start workflow in my local hop gui with the remote environment:
ERROR: Error starting workflow
ERROR: org.apache.hop.core.exception.HopException:
Error serializing export interface [C:\\.../start.hwf : start]
start - Unable to load pipeline [${PROJECT_HOME}/ingestion/get-csv-file-metadata.hpl] : can't find directory
I tried different options, but none worked out, the last one being displayed in the screenshot (tried it first with both folder settings empty).
It's a rather simple workflow
According to the documentation, the project home (variable) should be replaced on the server with a local folder. That being said, I have not created the container with any particular environment variables or volume mounts etc. as this would counter the purpose of the use case.
Any suggestions what the correct configuration would look like? The workflow runs fine with local runtime.
Thanks
GitHub link: https://github.com/apache/hop/discussions/4634#discussioncomment-12821552
----
This is an automatically sent email for users@hop.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@hop.apache.org
Re: [D] Question About Database Connections in Apache Hop Server Mode (hop)
Posted by "hansva (via GitHub)" <gi...@apache.org>.
GitHub user hansva added a comment to the discussion: Question About Database Connections in Apache Hop Server Mode
That request is missing crucial parts. Such as your pipeline this has to be the acutal content not just a path to a file.
You can see a valid request [here](https://hop.apache.org//manual/latest/hop-server/rest-api.html#_register_pipeline)
GitHub link: https://github.com/apache/hop/discussions/4634#discussioncomment-14858067
----
This is an automatically sent email for users@hop.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@hop.apache.org
Re: [D] Question About Database Connections in Apache Hop Server Mode (hop)
Posted by "bamaer (via GitHub)" <gi...@apache.org>.
GitHub user bamaer added a comment to the discussion: Question About Database Connections in Apache Hop Server Mode
you didn't come across as rude, no worries.
the metadata needs to exist on the server. The solution Hans mentioned will work for the entire server, otherwise I guess you'll need to add all of the required information for all of your workflows and pipelines that use that connection afaik
GitHub link: https://github.com/apache/hop/discussions/4634#discussioncomment-11421716
----
This is an automatically sent email for users@hop.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@hop.apache.org
Re: [D] Question About Database Connections in Apache Hop Server Mode (hop)
Posted by "ggrohmann (via GitHub)" <gi...@apache.org>.
GitHub user ggrohmann edited a comment on the discussion: Question About Database Connections in Apache Hop Server Mode
Hey folks,
I have a question, which is related to the described matter, in particular running a project on a remote server. I managed to run single pipelines and workflows on a hop server in a docker container (long-lived). However, I struggle with setting up the run configuration for a workflow with linked pipelines and sub workflows. I followed the documentation as far as I could (https://hop.apache.org//manual/latest/workflow/workflow-run-configurations/native-remote-workflow-engine.html).
However, I receive the error, that the pipelines couldn't be resolved, when I run the start workflow in my local hop gui with the remote environment:
ERROR: Error starting workflow
ERROR: org.apache.hop.core.exception.HopException:
Error serializing export interface [C:\.../start.hwf : start]
start - Unable to load pipeline [${PROJECT_HOME}/ingestion/get-csv-file-metadata.hpl] : can't find directory
I tried different options, but none worked out, the last one being displayed in the screenshot (tried it first with both folder settings empty).
According to the documentation, the project home (variable) should be replaced on the server with a local folder. That being said, I have not created the container with any particular environment variables or volume mounts etc. as this would counter the purpose of the use case.
Any suggestions what the correct configuration would look like? The workflow runs fine with local runtime.
Thanks
GitHub link: https://github.com/apache/hop/discussions/4634#discussioncomment-12821552
----
This is an automatically sent email for users@hop.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@hop.apache.org
Re: [D] Question About Database Connections in Apache Hop Server Mode (hop)
Posted by "ggrohmann (via GitHub)" <gi...@apache.org>.
GitHub user ggrohmann edited a comment on the discussion: Question About Database Connections in Apache Hop Server Mode
Hey folks,
I have a question, which is related to the described matter, in particular running a project on a remote server. I managed to run single pipelines and workflows on a hop server in a docker container (long-lived). However, I struggle with setting up the run configuration for a workflow with linked pipelines and sub workflows. I followed the documentation as far as I could (https://hop.apache.org//manual/latest/workflow/workflow-run-configurations/native-remote-workflow-engine.html).
However, I receive the error, that the pipelines couldn't be resolved, when I run the start workflow in my local hop gui with the remote environment:
ERROR: Error starting workflow
ERROR: org.apache.hop.core.exception.HopException:
Error serializing export interface [C:\.../start.hwf : start]
start - Unable to load pipeline [${PROJECT_HOME}/ingestion/get-csv-file-metadata.hpl] : can't find directory
I tried different options, but none worked out, the last one being displayed in the screenshot (tried it first with both folder settings empty).
According to the documentation, the project home (variable) should be replaced on the server with a local folder. That being said, I have not created the container with any particular environment variables or volume mounts etc. as this would counter the purpose of the use case.
Any suggestions what the correct configuration would look like?
Thanks
GitHub link: https://github.com/apache/hop/discussions/4634#discussioncomment-12821552
----
This is an automatically sent email for users@hop.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@hop.apache.org
Re: [D] Question About Database Connections in Apache Hop Server Mode (hop)
Posted by "pr-desai (via GitHub)" <gi...@apache.org>.
GitHub user pr-desai added a comment to the discussion: Question About Database Connections in Apache Hop Server Mode
we are getting the response as like this
<HTML>
<HEAD>
<TITLE>
!RegisterPipelineServlet.Title!
</TITLE>
<META http-equiv="Content-Type" content="text/html; charset=UTF-8">
<link rel="icon" type="image/svg+xml" href="/static/images/favicon.svg">
</HEAD>
<BODY>
<p>
<pre>
org.apache.hop.core.exception.HopXmlException:
Error reading information from XML string :
Premature end of file.
at org.apache.hop.core.xml.XmlHandler.loadXmlString(XmlHandler.java:719)
at org.apache.hop.core.xml.XmlHandler.loadXmlString(XmlHandler.java:699)
at org.apache.hop.core.xml.XmlHandler.loadXmlString(XmlHandler.java:673)
at org.apache.hop.pipeline.PipelineConfiguration.fromXml(PipelineConfiguration.java:79)
at org.apache.hop.www.RegisterPipelineServlet.generateBody(RegisterPipelineServlet.java:54)
at org.apache.hop.www.BodyHttpServlet.doGet(BodyHttpServlet.java:65)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:687)
at org.apache.hop.www.BaseHttpServlet.service(BaseHttpServlet.java:102)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:550)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1435)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1350)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:234)
at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:59)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:567)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.eclipse.jetty.server.Server.handle(Server.java:516)
at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:388)
at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:633)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:380)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:383)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:882)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1036)
at java.base/java.lang.Thread.run(Thread.java:842)
Caused by: org.xml.sax.SAXParseException; Premature end of file.
at org.apache.xerces.parsers.DOMParser.parse(Unknown Source)
at org.apache.xerces.jaxp.DocumentBuilderImpl.parse(Unknown Source)
at org.apache.hop.core.xml.XmlHandler.loadXmlString(XmlHandler.java:710)
... 40 more
</pre>
<p>
</BODY>
</HTML>
request: http://localhost:8080/hop/registerPipeline
Payload details for the registerPipeline
<pipeline_configuration>
<pipeline>
<filename>C:/Users/shiva/Downloads/apache-hop-client-2.14.0/hop/config/projects/default/Exampl.hpl</filename>
</pipeline>
<pipeline_execution_configuration>
<variables></variables>
<parameters></parameters>
<pass_export>N</pass_export>
<log_level>Basic</log_level>
<log_file>N</log_file>
<log_filename></log_filename>
<log_file_append>N</log_file_append>
<create_parent_folder>N</create_parent_folder>
<clear_log>Y</clear_log>
<show_subcomponents>Y</show_subcomponents>
<run_configuration>local</run_configuration>
</pipeline_execution_configuration>
<metastore_json></metastore_json>
</pipeline_configuration>
GitHub link: https://github.com/apache/hop/discussions/4634#discussioncomment-14857997
----
This is an automatically sent email for users@hop.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@hop.apache.org
Re: [D] Question About Database Connections in Apache Hop Server Mode (hop)
Posted by "ggrohmann (via GitHub)" <gi...@apache.org>.
GitHub user ggrohmann edited a comment on the discussion: Question About Database Connections in Apache Hop Server Mode
Hey folks,
I have a question, which is related to the described matter, in particular running a project on a remote server. I managed to run single pipelines and workflows on a hop server in a docker container (long-lived). However, I struggle with setting up the run configuration for a workflow with linked pipelines and sub workflows. I followed the documentation as far as I could (https://hop.apache.org//manual/latest/workflow/workflow-run-configurations/native-remote-workflow-engine.html).
However, I receive the error, that the pipelines couldn't be resolved, when I run the start workflow in my local hop gui with the remote environment:
ERROR: Error starting workflow
ERROR: org.apache.hop.core.exception.HopException:
Error serializing export interface [C:\\.../start.hwf : start]
start - Unable to load pipeline [${PROJECT_HOME}/ingestion/get-csv-file-metadata.hpl] : can't find directory
(this is a pipeline within the pipeline get-csv-files-direct.hpl - see screenshot below, being executed by a pipeline transformator)
I tried different options, but none worked out, the last one being displayed in the screenshot (tried it first with both folder settings empty).
It's a rather simple workflow
According to the documentation, the project home (variable) should be replaced on the server with a local folder. That being said, I have not created the container with any particular environment variables or volume mounts etc. as this would counter the purpose of the use case.
Any suggestions what the correct configuration would look like? The workflow runs fine with local runtime.
Thanks
GitHub link: https://github.com/apache/hop/discussions/4634#discussioncomment-12821552
----
This is an automatically sent email for users@hop.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@hop.apache.org
Re: [D] Question About Database Connections in Apache Hop Server Mode (hop)
Posted by "ggrohmann (via GitHub)" <gi...@apache.org>.
GitHub user ggrohmann edited a comment on the discussion: Question About Database Connections in Apache Hop Server Mode
Hey folks,
I have a question, which is related to the described matter, in particular running a project on a remote server. I managed to run single pipelines and workflows on a hop server in a docker container (long-lived). However, I struggle with setting up the run configuration for a workflow with linked pipelines and sub workflows. I followed the documentation as far as I could (https://hop.apache.org//manual/latest/workflow/workflow-run-configurations/native-remote-workflow-engine.html).
However, I receive the error, that the pipelines couldn't be resolved, when I run the start workflow in my local hop gui with the remote environment:
ERROR: Error starting workflow
ERROR: org.apache.hop.core.exception.HopException:
Error serializing export interface [C:\.../start.hwf : start]
start - Unable to load pipeline [${PROJECT_HOME}/ingestion/get-csv-file-metadata.hpl] : can't find directory
I tried different options, but none worked out, the last one being displayed in the screenshot (tried it first with both folder settings empty).
It's a rather simple workflow
According to the documentation, the project home (variable) should be replaced on the server with a local folder. That being said, I have not created the container with any particular environment variables or volume mounts etc. as this would counter the purpose of the use case.
Any suggestions what the correct configuration would look like? The workflow runs fine with local runtime.
Thanks
GitHub link: https://github.com/apache/hop/discussions/4634#discussioncomment-12821552
----
This is an automatically sent email for users@hop.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@hop.apache.org
Re: [D] Question About Database Connections in Apache Hop Server Mode (hop)
Posted by "hansva (via GitHub)" <gi...@apache.org>.
GitHub user hansva added a comment to the discussion: Question About Database Connections in Apache Hop Server Mode
You can also start the server with a configuration in the XML `metadata_folder` then you could leave that part empty given that all information is then stored on the server side
GitHub link: https://github.com/apache/hop/discussions/4634#discussioncomment-11415350
----
This is an automatically sent email for users@hop.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@hop.apache.org
Re: [D] Question About Database Connections in Apache Hop Server Mode (hop)
Posted by "ravamo (via GitHub)" <gi...@apache.org>.
GitHub user ravamo added a comment to the discussion: Question About Database Connections in Apache Hop Server Mode
Thanks, @hansva , for such a detailed response, especially over the weekend—it wasn’t necessary. As I mentioned before, I didn’t mean to sound rude or bother you. I just wanted to fully understand everything to provide my client with a solid solution, and if I need to create a tool, make sure it actually makes sense.
GitHub link: https://github.com/apache/hop/discussions/4634#discussioncomment-11424390
----
This is an automatically sent email for users@hop.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@hop.apache.org
Re: [D] Question About Database Connections in Apache Hop Server Mode (hop)
Posted by "hansva (via GitHub)" <gi...@apache.org>.
GitHub user hansva added a comment to the discussion: Question About Database Connections in Apache Hop Server Mode
Code for the creation of the export can be found [here](https://github.com/apache/hop/blob/a6deda30ecea0d254b67c676583bdd4df5a3ba3a/engine/src/main/java/org/apache/hop/www/AddExportServlet.java#L60). When running via the UI and sending it to the server the zipfiles are also stored in a temp location so you could take a look at how those are structured.
GitHub link: https://github.com/apache/hop/discussions/4634#discussioncomment-13294919
----
This is an automatically sent email for users@hop.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@hop.apache.org
Re: [D] Question About Database Connections in Apache Hop Server Mode (hop)
Posted by "ravamo (via GitHub)" <gi...@apache.org>.
GitHub user ravamo added a comment to the discussion: Question About Database Connections in Apache Hop Server Mode
I tried implementing a workaround to see if it would solve the issue more efficiently. I added the connection file `./projects/default/metadata/rdbms/Test.json` with the definition to that path and restarted the service, but I’m getting this error: `remove rows - ERROR: No database connection is defined`. Is there anything else I should do?
GitHub link: https://github.com/apache/hop/discussions/4634#discussioncomment-11470739
----
This is an automatically sent email for users@hop.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@hop.apache.org
Re: [D] Question About Database Connections in Apache Hop Server Mode (hop)
Posted by "ravamo (via GitHub)" <gi...@apache.org>.
GitHub user ravamo added a comment to the discussion: Question About Database Connections in Apache Hop Server Mode
Fixed it! The problem was the space and the missing trailing / in the path.
GitHub link: https://github.com/apache/hop/discussions/4634#discussioncomment-11472376
----
This is an automatically sent email for users@hop.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@hop.apache.org
Re: [D] Question About Database Connections in Apache Hop Server Mode (hop)
Posted by "hansva (via GitHub)" <gi...@apache.org>.
GitHub user hansva added a comment to the discussion: Question About Database Connections in Apache Hop Server Mode
This could have been a separate discussion as it is not really linked to this one.
I use the default value of `${PROJECT_HOME}` in Named resources reference source/target and everything seems to be working fine
GitHub link: https://github.com/apache/hop/discussions/4634#discussioncomment-12838262
----
This is an automatically sent email for users@hop.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@hop.apache.org
Re: [D] Question About Database Connections in Apache Hop Server Mode (hop)
Posted by "ggrohmann (via GitHub)" <gi...@apache.org>.
GitHub user ggrohmann added a comment to the discussion: Question About Database Connections in Apache Hop Server Mode
Hey folks,
I have a question, which is related to the described matter, in particular running a project on a remote server. I managed to run single pipelines and workflows on a hop server in a docker container (long-lived). However, I struggle with setting up the run configuration for a workflow with linked pipelines and sub workflows. I followed the documentation as far as I could (https://hop.apache.org//manual/latest/workflow/workflow-run-configurations/native-remote-workflow-engine.html).
However, I receive the error, that the pipelines couldn't be resolved, when I run the start workflow in my local hop gui with the remote environment:
ERROR: Error starting workflow
ERROR: org.apache.hop.core.exception.HopException:
Error serializing export interface [C:\.../start.hwf : start]
start - Unable to load pipeline [${PROJECT_HOME}/ingestion/get-csv-file-metadata.hpl] : can't find directory
I tried different options, but none worked out, the last one being displayed in the screenshot.
According to the documentation, the project home (variable) should be replaced on the server with a local folder. That being said, I have not created the container with any particular environment variables or volume mounts etc. as this would counter the purpose of the use case.
Any suggestions what the correct configuration would look like?
Thanks
GitHub link: https://github.com/apache/hop/discussions/4634#discussioncomment-12821552
----
This is an automatically sent email for users@hop.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@hop.apache.org
Re: [D] Question About Database Connections in Apache Hop Server Mode (hop)
Posted by "shivanandullagaddi (via GitHub)" <gi...@apache.org>.
GitHub user shivanandullagaddi added a comment to the discussion: Question About Database Connections in Apache Hop Server Mode
for this request can you check the where configuration is missing
<?xml version="1.0" encoding="UTF-8"?>
<pipeline_configuration>
<pipeline>
<info>
<name>pipeline_example_1</name>
<name_sync_with_filename>Y</name_sync_with_filename>
<description></description>
<extended_description></extended_description>
<pipeline_version></pipeline_version>
<pipeline_type>Normal</pipeline_type>
<parameters></parameters>
<capture_transform_performance>N</capture_transform_performance>
<transform_performance_capturing_delay>1000</transform_performance_capturing_delay>
<transform_performance_capturing_size_limit>100</transform_performance_capturing_size_limit>
<created_user>-</created_user>
<created_date>2025/10/21 12:51:22.638</created_date>
<modified_user>-</modified_user>
<modified_date>2025/10/21 12:51:22.638</modified_date>
</info>
<notepads></notepads>
<order>
<hop>
<from>Salesforce input</from>
<to>Salesforce insert</to>
<enabled>Y</enabled>
</hop>
</order>
<transform>
<name>Salesforce input</name>
<type>SalesforceInput</type>
<description></description>
<distribute>Y</distribute>
<custom_distribution></custom_distribution>
<copies>1</copies>
<partitioning>
<method>none</method>
<schema_name></schema_name>
</partitioning>
<targeturl>https://login.salesforce.com/services/Soap/u/50.0</targeturl>
<username>shivanand@xyzu.com</username>
<password>Encrypted 53686976754032337232595156746166cdf3fca602c3c6a5801ab856d0a780dc</password>
<timeout>60000</timeout>
<useCompression>N</useCompression>
<module>Account</module>
<condition></condition>
<specifyQuery>Y</specifyQuery>
<query>Select id, name, phone, description from account</query>
<include_targeturl>N</include_targeturl>
<targeturl_field></targeturl_field>
<include_module>N</include_module>
<module_field></module_field>
<include_rownum>N</include_rownum>
<include_deletion_date>N</include_deletion_date>
<deletion_date_field></deletion_date_field>
<rownum_field></rownum_field>
<include_sql>N</include_sql>
<sql_field></sql_field>
<include_Timestamp>N</include_Timestamp>
<timestamp_field></timestamp_field>
<read_from></read_from>
<read_to></read_to>
<records_filter>all</records_filter>
<queryAll>N</queryAll>
<fields>
<field>
<name>Id</name>
<field>Id</field>
<idlookup>N</idlookup>
<type>String</type>
<length>-1</length>
<precision>-1</precision>
<trim_type>none</trim_type>
<repeat>N</repeat>
</field>
<field>
<name>Name</name>
<field>Name</field>
<idlookup>N</idlookup>
<type>String</type>
<length>-1</length>
<precision>-1</precision>
<trim_type>none</trim_type>
<repeat>N</repeat>
</field>
<field>
<name>Phone</name>
<field>Phone</field>
<idlookup>N</idlookup>
<type>String</type>
<length>-1</length>
<precision>-1</precision>
<trim_type>none</trim_type>
<repeat>N</repeat>
</field>
<field>
<name>Description</name>
<field>Description</field>
<idlookup>N</idlookup>
<type>String</type>
<length>-1</length>
<precision>-1</precision>
<trim_type>none</trim_type>
<repeat>N</repeat>
</field>
</fields>
<limit>0</limit>
<attributes></attributes>
<GUI>
<xloc>384</xloc>
<yloc>224</yloc>
</GUI>
</transform>
<transform>
<name>Salesforce insert</name>
<type>SalesforceInsert</type>
<description></description>
<distribute>Y</distribute>
<custom_distribution></custom_distribution>
<copies>1</copies>
<partitioning>
<method>none</method>
<schema_name></schema_name>
</partitioning>
<targeturl>https://login.salesforce.com/services/Soap/u/50.0</targeturl>
<username>shivanand@xyz.com</username>
<password>Encrypted 536869767540313349587957746f354557fcf9e1bc06df8abdb10aa824cba3a189</password>
<timeout>60000</timeout>
<useCompression>N</useCompression>
<module>Account</module>
<batchSize>10</batchSize>
<salesforceIDFieldName>Id</salesforceIDFieldName>
<fields>
<field>
<name>Name</name>
<field>Name</field>
<useExternalId>N</useExternalId>
</field>
<field>
<name>Phone</name>
<field>Phone</field>
<useExternalId>N</useExternalId>
</field>
<field>
<name>Description</name>
<field>Description</field>
<useExternalId>N</useExternalId>
</field>
</fields>
<rollbackAllChangesOnError>Y</rollbackAllChangesOnError>
<attributes></attributes>
<GUI>
<xloc>528</xloc>
<yloc>224</yloc>
</GUI>
</transform>
<transform_error_handling></transform_error_handling>
<attributes></attributes>
</pipeline>
<pipeline_execution_configuration>
<run_configuration>local</run_configuration>
<log_level>Basic</log_level>
<clear_log>Y</clear_log>
<show_subcomponents>Y</show_subcomponents>
</pipeline_execution_configuration>
<metastore_json></metastore_json>
</pipeline_configuration>
Response:
<HTML>
<HEAD>
<TITLE>
!RegisterPipelineServlet.Title!
</TITLE>
<META http-equiv="Content-Type" content="text/html; charset=UTF-8">
<link rel="icon" type="image/svg+xml" href="/static/images/favicon.svg">
</HEAD>
<BODY>
<p>
<pre>
org.apache.hop.core.exception.HopXmlException:
Error reading information from XML string :
Premature end of file.
at org.apache.hop.core.xml.XmlHandler.loadXmlString(XmlHandler.java:719)
at org.apache.hop.core.xml.XmlHandler.loadXmlString(XmlHandler.java:699)
at org.apache.hop.core.xml.XmlHandler.loadXmlString(XmlHandler.java:673)
at org.apache.hop.pipeline.PipelineConfiguration.fromXml(PipelineConfiguration.java:79)
at org.apache.hop.www.RegisterPipelineServlet.generateBody(RegisterPipelineServlet.java:54)
at org.apache.hop.www.BodyHttpServlet.doGet(BodyHttpServlet.java:65)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:687)
at org.apache.hop.www.BaseHttpServlet.service(BaseHttpServlet.java:102)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:550)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1435)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1350)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:234)
at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:59)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:567)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.eclipse.jetty.server.Server.handle(Server.java:516)
at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:388)
at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:633)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:380)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:383)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:882)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1036)
at java.base/java.lang.Thread.run(Thread.java:842)
Caused by: org.xml.sax.SAXParseException; Premature end of file.
at org.apache.xerces.parsers.DOMParser.parse(Unknown Source)
at org.apache.xerces.jaxp.DocumentBuilderImpl.parse(Unknown Source)
at org.apache.hop.core.xml.XmlHandler.loadXmlString(XmlHandler.java:710)
... 40 more
</pre>
<p>
</BODY>
</HTML>
GitHub link: https://github.com/apache/hop/discussions/4634#discussioncomment-14865499
----
This is an automatically sent email for users@hop.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@hop.apache.org
Re: [D] Question About Database Connections in Apache Hop Server Mode (hop)
Posted by "RaffaelloC94 (via GitHub)" <gi...@apache.org>.
GitHub user RaffaelloC94 added a comment to the discussion: Question About Database Connections in Apache Hop Server Mode
> When you want to run a workflow with multiple pipelines via the API you need to use the addExport endpoint and this requires a zip file with a specific structure.
Sorry for jumping in — I’d like to know what the structure of the zip file should be when sending it to the addExport API. Thanks!
GitHub link: https://github.com/apache/hop/discussions/4634#discussioncomment-13294315
----
This is an automatically sent email for users@hop.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@hop.apache.org
Re: [D] Question About Database Connections in Apache Hop Server Mode (hop)
Posted by "ravamo (via GitHub)" <gi...@apache.org>.
GitHub user ravamo added a comment to the discussion: Question About Database Connections in Apache Hop Server Mode
Thank you for your response. I don’t have a process that I clone from a Git repository... Could that be the issue? Could you recommend a specific part of the documentation to look into this?
GitHub link: https://github.com/apache/hop/discussions/4634#discussioncomment-11405230
----
This is an automatically sent email for users@hop.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@hop.apache.org
Re: [D] Question About Database Connections in Apache Hop Server Mode (hop)
Posted by "ravamo (via GitHub)" <gi...@apache.org>.
GitHub user ravamo added a comment to the discussion: Question About Database Connections in Apache Hop Server Mode
Thanks for your reaply @bamaer .
OK, that's a problem, but we'll have to deal with it. So, I would need to create a program or something similar to inject that part into the ETLs with the required information, right?
This brings me to my next question: what exactly should <metastore_json> contain? Is it a checksum of the previous information or something like that..
So, the workflow to run the ETLs via API would be:
1. Create the ETL.
2. Inject the XML part into each ETL.
3. Register the Pipeline or Workflow.
Would this be the correct flow to focus on the issues?
GitHub link: https://github.com/apache/hop/discussions/4634#discussioncomment-11415142
----
This is an automatically sent email for users@hop.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@hop.apache.org
Re: [D] Question About Database Connections in Apache Hop Server Mode (hop)
Posted by "ravamo (via GitHub)" <gi...@apache.org>.
GitHub user ravamo deleted a comment on the discussion: Question About Database Connections in Apache Hop Server Mode
Thanks, @hansva , for such a detailed response, especially over the weekend—it wasn’t necessary. As I mentioned before, I didn’t mean to sound rude or bother you. I just wanted to fully understand everything to provide my client with a solid solution, and if I need to create a tool, make sure it actually makes sense.
GitHub link: https://github.com/apache/hop/discussions/4634#discussioncomment-11424390
----
This is an automatically sent email for users@hop.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@hop.apache.org
Re: [D] Question About Database Connections in Apache Hop Server Mode (hop)
Posted by "hansva (via GitHub)" <gi...@apache.org>.
GitHub user hansva added a comment to the discussion: Question About Database Connections in Apache Hop Server Mode
I'll try to write a full and cohesive answer.
Let's start by rephrasing the original question:
> I want to run a pipeline/workflow on Hop Server using the Rest API directly, without using Hop GUI or hop-run. How do I do this?
## Intro
### Hop Server
**What it is**
Hop server is a stateless server, its main purpose is to be used as an extension to Hop GUI to run a pipeline or workflow in a remote environment
**What it isn't**
Hop Server isn't the typical server that you would use for scheduling/monitoring it does not retain state or history, it does not store this information (except in memory for a short period). After a restart all previous information is lost.
By circling back to what it is, we can also discuss why it is poorly documented. We do **not** want it to be used in a stand-alone way; it wasn't made for this. We did add the endpoints to our [documentation](https://hop.apache.org//manual/latest/hop-server/rest-api.html) because people were asking for them, but honestly, it was never designed to be used without using the GUI or Hop Run. There are better ways, eg. using short lived containers which provide more flexibility and in combination with airflow ([tutorial](https://hop.apache.org//manual/latest/how-to-guides/run-hop-in-apache-airflow.html)) you can also use webhooks to start things.
Shameless plug: We (know.bi) are working on something better which we hope to showcase soon.
This covers our disclaimer, let's get back to the subject.
## Running a single pipeline
The process to start something on Hop Server is split up in 3 different categories:
- A single pipeline
- A single workflow
- A workflow with other pipelines/workflows
let's discuss starting a single pipeline, there are 3 steps that need to be taken to start a pipeline on Hop Server.
### registerPipeline
The first step is to send the pipeline and all needed environment information to the server. As stated before the server is stateless so it knows nothing it needs all information to create a successful execution.
The XML format of the request:
```
<pipeline_configuration>
<pipeline>
</pipeline>
<pipeline_execution_configuration>
<variables></variables>
<parameters></parameters>
<pass_export>N</pass_export>
<log_level>Basic</log_level>
<log_file>N</log_file>
<log_filename/>
<log_file_append>N</log_file_append>
<create_parent_folder>N</create_parent_folder>
<clear_log>Y</clear_log>
<show_subcomponents>Y</show_subcomponents>
<run_configuration>local</run_configuration>
</pipeline_execution_configuration>
<metastore_json>
</metastore_json>
</pipeline_configuration>
```
3 blocks of information need to be included in this request:
**pipeline:**
this one is simple it's the hpl file that you wish to execute on the server.
**pipeline_execution_configuration:**
This block contains an export of all Hop variables in the <variables> section and parameters/variables you have defined in the Run Options dialog
The variables section will also contain all variables you have defined in your environment, if you have defined database username/password and so on to an environment file they get added there.
Each variable looks like
`<variable><name>VARIABLE_NAME</name><value>VALUE</value></variable>`
**metastore_json:**
This is the part where it gets hard. The metastore_json is a Base64 encoded gzip stream.
To get a fast/simple preview of what's in this you could take the example from our docs and throw it in [this](https://www.bugdays.com/gzip-base64) website.
It boils down to a json containing all objects you have defined in the metadata perspective/metadata folder.
example if you only have a PostgreSQL connection, but it also needs to contain your run targets and all other objects that are available in your metadata folder. Another note: each database type can have different fields (just like in the UI) most of them are shared, but eg MSSQL Server has more fields.
```
{
"rdbms": [
{
"rdbms": {
"POSTGRESQL": {
"databaseName": "postgres",
"pluginId": "POSTGRESQL",
"indexTablespace": null,
"dataTablespace": null,
"accessType": 0,
"hostname": "localhost",
"password": "",
"pluginName": "PostgreSQL",
"port": "5432",
"servername": null,
"attributes": {
"SUPPORTS_TIMESTAMP_DATA_TYPE": "N",
"QUOTE_ALL_FIELDS": "N",
"SUPPORTS_BOOLEAN_DATA_TYPE": "Y",
"FORCE_IDENTIFIERS_TO_LOWERCASE": "N",
"PRESERVE_RESERVED_WORD_CASE": "Y",
"SQL_CONNECT": "",
"FORCE_IDENTIFIERS_TO_UPPERCASE": "N",
"PREFERRED_SCHEMA_NAME": ""
},
"manualUrl": "",
"username": "postgres"
}
},
"name": "pg"
}
]
}
```
After building and sending this request to the server (POST) you will get a response:
```
<webresult>
<result>OK</result>
<message>Pipeline 'variables' was added to HopServer with id 08bdff17-0d75-43a3-b890-05783376cbb2</message>
<id>08bdff17-0d75-43a3-b890-05783376cbb2</id>
</webresult>
```
### prepareExec
After you get back the Id you have to hit the prepareExec with a GET request
`GET /hop/prepareExec/?name=variables&xml=Y&id=08bdff17-0d75-43a3-b890-05783376cbb2`
response:
```
<webresult>
<result>OK</result>
<message/>
<id/>
</webresult>
```
This will prepare the pipeline for execution and it will enter a "waiting state"
### startExec
The final step is a GET to startExec to start the actual execution
`GET /hop/startExec/?name=variables&xml=Y&id=08bdff17-0d75-43a3-b890-05783376cbb2`
response
```
<webresult>
<result>OK</result>
<message/>
<id/>
</webresult>
```
You can follow up how everything is going with the pipelineStatus endpoint.
## Closing note
These steps should help you use the REST API directly to start a pipeline, running a single workflow is a similar process.
Running a combination of workflows and pipelines requires more work as this is a specially crafted zip file that is sent to the server.
Happy coding,
Hans
GitHub link: https://github.com/apache/hop/discussions/4634#discussioncomment-11422350
----
This is an automatically sent email for users@hop.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@hop.apache.org
Re: [D] Question About Database Connections in Apache Hop Server Mode (hop)
Posted by "ravamo (via GitHub)" <gi...@apache.org>.
GitHub user ravamo added a comment to the discussion: Question About Database Connections in Apache Hop Server Mode
yep we've this tag complete :`<metadata_folder>/servers/hop/hop/config/projects/default/metadata </metadata_folder>`
GitHub link: https://github.com/apache/hop/discussions/4634#discussioncomment-11471340
----
This is an automatically sent email for users@hop.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@hop.apache.org
Re: [D] Question About Database Connections in Apache Hop Server Mode (hop)
Posted by "anilaladag (via GitHub)" <gi...@apache.org>.
GitHub user anilaladag added a comment to the discussion: Question About Database Connections in Apache Hop Server Mode
I could not implement a workflow that contains two pipelines. I got the error: "The server has no path of pipeline." How can I properly structure my workflow and pipelines to avoid this error?
Some thoughts on possible solutions:
1) Using the registerPipeline API, but I'm unsure how to define the correct path for the pipeline. It seems like the pipeline is not being stored anywhere when I send it this way.
2) Building a Docker image that already includes the pipelines in the correct path and structuring the setup accordingly.
I can successfully send metadata, but I don't understand how to send the pipelines that are used inside the workflow.
Can you help me with this?
GitHub link: https://github.com/apache/hop/discussions/4634#discussioncomment-12427072
----
This is an automatically sent email for users@hop.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@hop.apache.org
Re: [D] Question About Database Connections in Apache Hop Server Mode (hop)
Posted by "UEMravamo (via GitHub)" <gi...@apache.org>.
GitHub user UEMravamo added a comment to the discussion: Question About Database Connections in Apache Hop Server Mode
yes we've this tag complete `<metadata_folder>/servers/hop/hop/config/projects/default/metadata</metadata_folder>`
GitHub link: https://github.com/apache/hop/discussions/4634#discussioncomment-11471299
----
This is an automatically sent email for users@hop.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@hop.apache.org
Re: [D] Question About Database Connections in Apache Hop Server Mode (hop)
Posted by "ravamo (via GitHub)" <gi...@apache.org>.
GitHub user ravamo added a comment to the discussion: Question About Database Connections in Apache Hop Server Mode
> you have to add the `pipeline_execution_configuration` part yourself if you want to send your pipelines to the server. using short-lived containers would make your life easier, if that's an option.
Could you provide me with more information about this and why it might be a solution? The thing is, manually entering that information feels like over-engineering. I thought there might be a way to export the ETLs with that information already included
GitHub link: https://github.com/apache/hop/discussions/4634#discussioncomment-11416274
----
This is an automatically sent email for users@hop.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@hop.apache.org
Re: [D] Question About Database Connections in Apache Hop Server Mode (hop)
Posted by "ravamo (via GitHub)" <gi...@apache.org>.
GitHub user ravamo added a comment to the discussion: Question About Database Connections in Apache Hop Server Mode
Of course, @hansva , it makes perfect sense for it to be dynamic. However, to focus my question a bit: if you have a database connection, its definition doesn't change at runtime; instead, what changes are the values associated with that connection, such as the host, user, password, etc. My question revolves around this: let's say I develop a complex (or simple) ETL and have a database connection. I save it and upload it to my repository. If I then clone that project on another host, would I need to manually create the database connection? (Whether or not that makes sense is a separate discussion.)
But if that’s the case, how can I create the connection on a host without a graphical interface? Isn't there a way to inject that information into the XML? I mention this because if I need to create a tool, I'd like to contribute it to the community.
I missed these kinds of chats :D
GitHub link: https://github.com/apache/hop/discussions/4634#discussioncomment-11418322
----
This is an automatically sent email for users@hop.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@hop.apache.org
Re: [D] Question About Database Connections in Apache Hop Server Mode (hop)
Posted by "ravamo (via GitHub)" <gi...@apache.org>.
GitHub user ravamo added a comment to the discussion: Question About Database Connections in Apache Hop Server Mode
OK roger that @bamaer , first of all, I didn’t mean to sound rude in my comments—you’ve done an amazing job. About your point, I totally get it; we all make decisions when starting on something, and later realize it might not have been the best, but you’ve got to start somewhere.
Regarding my question, I have two follow-ups:
- Would I need to add that information to all the pipelines and workflows, or would it be enough to do it in the main one?
- In the case of the database, what tag should I add?
- Host,User,Pass,Databasename, Schema , etc..
GitHub link: https://github.com/apache/hop/discussions/4634#discussioncomment-11421619
----
This is an automatically sent email for users@hop.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@hop.apache.org
Re: [D] Question About Database Connections in Apache Hop Server Mode (hop)
Posted by "hansva (via GitHub)" <gi...@apache.org>.
GitHub user hansva added a comment to the discussion: Question About Database Connections in Apache Hop Server Mode
No rudeness spotted at all. You are not the first to ask, so I took the time to write a full response. When I have some spare time I can convert this into our documentation.
GitHub link: https://github.com/apache/hop/discussions/4634#discussioncomment-11425341
----
This is an automatically sent email for users@hop.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@hop.apache.org
Re: [D] Question About Database Connections in Apache Hop Server Mode (hop)
Posted by "shivanandullagaddi (via GitHub)" <gi...@apache.org>.
GitHub user shivanandullagaddi added a comment to the discussion: Question About Database Connections in Apache Hop Server Mode
we are getting the response as like this for registerPipeline from postman
<TITLE> !RegisterPipelineServlet.Title! </TITLE>
org.apache.hop.core.exception.HopXmlException:
Error reading information from XML string :
Premature end of file.
at org.apache.hop.core.xml.XmlHandler.loadXmlString(XmlHandler.java:719)
at org.apache.hop.core.xml.XmlHandler.loadXmlString(XmlHandler.java:699)
at org.apache.hop.core.xml.XmlHandler.loadXmlString(XmlHandler.java:673)
at org.apache.hop.pipeline.PipelineConfiguration.fromXml(PipelineConfiguration.java:79)
at org.apache.hop.www.RegisterPipelineServlet.generateBody(RegisterPipelineServlet.java:54)
at org.apache.hop.www.BodyHttpServlet.doGet(BodyHttpServlet.java:65)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:687)
at org.apache.hop.www.BaseHttpServlet.service(BaseHttpServlet.java:102)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:550)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1435)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1350)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:234)
at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:59)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:567)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.eclipse.jetty.server.Server.handle(Server.java:516)
at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:388)
at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:633)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:380)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:383)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:882)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1036)
at java.base/java.lang.Thread.run(Thread.java:842)
Caused by: org.xml.sax.SAXParseException; Premature end of file.
at org.apache.xerces.parsers.DOMParser.parse(Unknown Source)
at org.apache.xerces.jaxp.DocumentBuilderImpl.parse(Unknown Source)
at org.apache.hop.core.xml.XmlHandler.loadXmlString(XmlHandler.java:710)
... 40 more
request: http://localhost:8080/hop/registerPipeline
Payload details for the registerPipeline:
<pipeline_configuration>
C:/Users/Downloads/apache-hop-client-2.14.0/hop/config/projects/default/Exampl.hpl
<pipeline_execution_configuration>
<pass_export>N</pass_export>
<log_level>Basic</log_level>
<log_file>N</log_file>
<log_filename></log_filename>
<log_file_append>N</log_file_append>
<create_parent_folder>N</create_parent_folder>
<clear_log>Y</clear_log>
<show_subcomponents>Y</show_subcomponents>
<run_configuration>local</run_configuration>
</pipeline_execution_configuration>
<metastore_json></metastore_json>
</pipeline_configuration>
GitHub link: https://github.com/apache/hop/discussions/4634#discussioncomment-14858037
----
This is an automatically sent email for users@hop.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@hop.apache.org
Re: [D] Question About Database Connections in Apache Hop Server Mode (hop)
Posted by "bamaer (via GitHub)" <gi...@apache.org>.
GitHub user bamaer added a comment to the discussion: Question About Database Connections in Apache Hop Server Mode
have you looked at [registerPipeline](https://hop.apache.org//manual/latest/hop-server/rest-api.html#_register_pipeline)? (similar for workflow)
GitHub link: https://github.com/apache/hop/discussions/4634#discussioncomment-11405619
----
This is an automatically sent email for users@hop.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@hop.apache.org
Re: [D] Question About Database Connections in Apache Hop Server Mode (hop)
Posted by "hansva (via GitHub)" <gi...@apache.org>.
GitHub user hansva added a comment to the discussion: Question About Database Connections in Apache Hop Server Mode
are you using the server-xml with a path to the metadata folder? That is needed to load these objects in Hop Server.
https://hop.apache.org//manual/latest/hop-server/index.html#_start_hop_server_with_a_configuration_file
GitHub link: https://github.com/apache/hop/discussions/4634#discussioncomment-11470930
----
This is an automatically sent email for users@hop.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@hop.apache.org
Re: [D] Question About Database Connections in Apache Hop Server Mode (hop)
Posted by "hansva (via GitHub)" <gi...@apache.org>.
GitHub user hansva added a comment to the discussion: Question About Database Connections in Apache Hop Server Mode
The metastore_json is base64 encoded GZip content, it contains all metadata information that is know to the project (database connections, run configurations....) everything that can be found in the metadata folder of your project
GitHub link: https://github.com/apache/hop/discussions/4634#discussioncomment-11415338
----
This is an automatically sent email for users@hop.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@hop.apache.org
Re: [D] Question About Database Connections in Apache Hop Server Mode (hop)
Posted by "ravamo (via GitHub)" <gi...@apache.org>.
GitHub user ravamo added a comment to the discussion: Question About Database Connections in Apache Hop Server Mode
roger that @hansva, make sense . So does the workflow make sense, or is there an error in my steps?
I mention this because we create the ETLs locally, then push them to a Git repository, and later clone the repository on the server. I just want to know if there's anything else I should keep in mind.
Thanks!
GitHub link: https://github.com/apache/hop/discussions/4634#discussioncomment-11415454
----
This is an automatically sent email for users@hop.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@hop.apache.org
Re: [D] Question About Database Connections in Apache Hop Server Mode (hop)
Posted by "hansva (via GitHub)" <gi...@apache.org>.
GitHub user hansva added a comment to the discussion: Question About Database Connections in Apache Hop Server Mode
Because this is runtime information and not static content, it could be different on every run.
It does get exported and added when using a remote-run configuration. If you are not using the Hop API to build/send your pipeline to the server you will have to add it yourself. Note: you can also use hop-run to start pipelines on the server
GitHub link: https://github.com/apache/hop/discussions/4634#discussioncomment-11416597
----
This is an automatically sent email for users@hop.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@hop.apache.org
Re: [D] Question About Database Connections in Apache Hop Server Mode (hop)
Posted by "bamaer (via GitHub)" <gi...@apache.org>.
GitHub user bamaer added a comment to the discussion: Question About Database Connections in Apache Hop Server Mode
creating _anything_ on a host without a graphical interface sort of defeats the purpose of using a platform that has "visually designed" in its tagline. that said, the current server implementation can be flexible, but that requires quite a bit of work. we probably wouldn't repeat most of the design choices that were made for the current hop server, but here we are...
GitHub link: https://github.com/apache/hop/discussions/4634#discussioncomment-11418367
----
This is an automatically sent email for users@hop.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@hop.apache.org
Re: [D] Question About Database Connections in Apache Hop Server Mode (hop)
Posted by "arunkumarArunachalam99 (via GitHub)" <gi...@apache.org>.
GitHub user arunkumarArunachalam99 added a comment to the discussion: Question About Database Connections in Apache Hop Server Mode
If I do not have any database components in my pipeline, what should I put in the metastore.json, and where can I get the json data to convert into metestore json?
GitHub link: https://github.com/apache/hop/discussions/4634#discussioncomment-11964933
----
This is an automatically sent email for users@hop.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@hop.apache.org
Re: [D] Question About Database Connections in Apache Hop Server Mode (hop)
Posted by "ravamo (via GitHub)" <gi...@apache.org>.
GitHub user ravamo added a comment to the discussion: Question About Database Connections in Apache Hop Server Mode
Is there any documentation about how the process would work if I have to go file by file, or how the overall process should be? The documentation is a bit confusing, and I’m having trouble understanding exactly what I need to send.
Additionally, I noticed that when I look at the source code of my ETLs, it differs slightly from the example. For instance, I don’t have any pipeline_execution_configuration information in any of my project
GitHub link: https://github.com/apache/hop/discussions/4634#discussioncomment-11414965
----
This is an automatically sent email for users@hop.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@hop.apache.org
Re: [D] Question About Database Connections in Apache Hop Server Mode (hop)
Posted by "bamaer (via GitHub)" <gi...@apache.org>.
GitHub user bamaer added a comment to the discussion: Question About Database Connections in Apache Hop Server Mode
you have to add the `pipeline_execution_configuration` part yourself if you want to send your pipelines to the server. using short-lived containers would make your life easier, if that's an option.
GitHub link: https://github.com/apache/hop/discussions/4634#discussioncomment-11415057
----
This is an automatically sent email for users@hop.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@hop.apache.org
Re: [D] Question About Database Connections in Apache Hop Server Mode (hop)
Posted by "hansva (via GitHub)" <gi...@apache.org>.
GitHub user hansva added a comment to the discussion: Question About Database Connections in Apache Hop Server Mode
When you want to run a workflow with multiple pipelines via the API you need to use the addExport endpoint and this requires a zip file with a specific structure.
GitHub link: https://github.com/apache/hop/discussions/4634#discussioncomment-12427877
----
This is an automatically sent email for users@hop.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@hop.apache.org
Re: [D] Question About Database Connections in Apache Hop Server Mode (hop)
Posted by "hansva (via GitHub)" <gi...@apache.org>.
GitHub user hansva added a comment to the discussion: Question About Database Connections in Apache Hop Server Mode
You create a [remote run configuration](https://hop.apache.org//manual/latest/pipeline/pipeline-run-configurations/native-remote-pipeline-engine.html) and then you can start your pipeline there. The same exists for workflows.
GitHub link: https://github.com/apache/hop/discussions/4634#discussioncomment-13295281
----
This is an automatically sent email for users@hop.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@hop.apache.org
Re: [D] Question About Database Connections in Apache Hop Server Mode (hop)
Posted by "RaffaelloC94 (via GitHub)" <gi...@apache.org>.
GitHub user RaffaelloC94 added a comment to the discussion: Question About Database Connections in Apache Hop Server Mode
Ok thank you! It works from gui and run on the server. Can i do the same just using the api addexport and passing the same zip? How can i run the workflow after the import? If i call startWorkflow after addexport it tell the workflow cannot be found
GitHub link: https://github.com/apache/hop/discussions/4634#discussioncomment-13297451
----
This is an automatically sent email for users@hop.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@hop.apache.org
Re: [D] Question About Database Connections in Apache Hop Server Mode (hop)
Posted by "anilaladag (via GitHub)" <gi...@apache.org>.
GitHub user anilaladag added a comment to the discussion: Question About Database Connections in Apache Hop Server Mode
If I upload the entire set of pipelines, workflows, etc., via the addExport API, do I still need to unzip it? I want to run my workflow, which includes multiple pipelines, on a remote server via APIs. Am I misunderstanding something? Could you provide an example with more complex logic?
Additionally, when I upload a pipeline or workflow to a remote server, it seems to store it in a MAP. Is there a way to save it to a specific directory instead? Could you provide an example with more complex logic?
GitHub link: https://github.com/apache/hop/discussions/4634#discussioncomment-12441215
----
This is an automatically sent email for users@hop.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@hop.apache.org
Re: [D] Question About Database Connections in Apache Hop Server Mode (hop)
Posted by "shivanandullagaddi (via GitHub)" <gi...@apache.org>.
GitHub user shivanandullagaddi edited a comment on the discussion: Question About Database Connections in Apache Hop Server Mode
we are getting the response as like this for registerPipeline
<TITLE> !RegisterPipelineServlet.Title! </TITLE>
org.apache.hop.core.exception.HopXmlException:
Error reading information from XML string :
Premature end of file.
at org.apache.hop.core.xml.XmlHandler.loadXmlString(XmlHandler.java:719)
at org.apache.hop.core.xml.XmlHandler.loadXmlString(XmlHandler.java:699)
at org.apache.hop.core.xml.XmlHandler.loadXmlString(XmlHandler.java:673)
at org.apache.hop.pipeline.PipelineConfiguration.fromXml(PipelineConfiguration.java:79)
at org.apache.hop.www.RegisterPipelineServlet.generateBody(RegisterPipelineServlet.java:54)
at org.apache.hop.www.BodyHttpServlet.doGet(BodyHttpServlet.java:65)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:687)
at org.apache.hop.www.BaseHttpServlet.service(BaseHttpServlet.java:102)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:550)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1435)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1350)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:234)
at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:59)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:567)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.eclipse.jetty.server.Server.handle(Server.java:516)
at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:388)
at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:633)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:380)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:383)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:882)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1036)
at java.base/java.lang.Thread.run(Thread.java:842)
Caused by: org.xml.sax.SAXParseException; Premature end of file.
at org.apache.xerces.parsers.DOMParser.parse(Unknown Source)
at org.apache.xerces.jaxp.DocumentBuilderImpl.parse(Unknown Source)
at org.apache.hop.core.xml.XmlHandler.loadXmlString(XmlHandler.java:710)
... 40 more
request: http://localhost:8080/hop/registerPipeline
Payload details for the registerPipeline
<pipeline_configuration>
C:/Users/Downloads/apache-hop-client-2.14.0/hop/config/projects/default/Exampl.hpl
<pipeline_execution_configuration>
<pass_export>N</pass_export>
<log_level>Basic</log_level>
<log_file>N</log_file>
<log_filename></log_filename>
<log_file_append>N</log_file_append>
<create_parent_folder>N</create_parent_folder>
<clear_log>Y</clear_log>
<show_subcomponents>Y</show_subcomponents>
<run_configuration>local</run_configuration>
</pipeline_execution_configuration>
<metastore_json></metastore_json>
</pipeline_configuration>
GitHub link: https://github.com/apache/hop/discussions/4634#discussioncomment-14858044
----
This is an automatically sent email for users@hop.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@hop.apache.org
Re: [D] Question About Database Connections in Apache Hop Server Mode (hop)
Posted by "shivanandullagaddi (via GitHub)" <gi...@apache.org>.
GitHub user shivanandullagaddi added a comment to the discussion: Question About Database Connections in Apache Hop Server Mode
we are getting the response as like this
<TITLE> !RegisterPipelineServlet.Title! </TITLE>
org.apache.hop.core.exception.HopXmlException:
Error reading information from XML string :
Premature end of file.
at org.apache.hop.core.xml.XmlHandler.loadXmlString(XmlHandler.java:719)
at org.apache.hop.core.xml.XmlHandler.loadXmlString(XmlHandler.java:699)
at org.apache.hop.core.xml.XmlHandler.loadXmlString(XmlHandler.java:673)
at org.apache.hop.pipeline.PipelineConfiguration.fromXml(PipelineConfiguration.java:79)
at org.apache.hop.www.RegisterPipelineServlet.generateBody(RegisterPipelineServlet.java:54)
at org.apache.hop.www.BodyHttpServlet.doGet(BodyHttpServlet.java:65)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:687)
at org.apache.hop.www.BaseHttpServlet.service(BaseHttpServlet.java:102)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:550)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1435)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1350)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:234)
at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:59)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:567)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.eclipse.jetty.server.Server.handle(Server.java:516)
at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:388)
at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:633)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:380)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:383)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:882)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1036)
at java.base/java.lang.Thread.run(Thread.java:842)
Caused by: org.xml.sax.SAXParseException; Premature end of file.
at org.apache.xerces.parsers.DOMParser.parse(Unknown Source)
at org.apache.xerces.jaxp.DocumentBuilderImpl.parse(Unknown Source)
at org.apache.hop.core.xml.XmlHandler.loadXmlString(XmlHandler.java:710)
... 40 more
request: http://localhost:8080/hop/registerPipeline
Payload details for the registerPipeline
<pipeline_configuration>
C:/Users/Downloads/apache-hop-client-2.14.0/hop/config/projects/default/Exampl.hpl
<pipeline_execution_configuration>
<pass_export>N</pass_export>
<log_level>Basic</log_level>
<log_file>N</log_file>
<log_filename></log_filename>
<log_file_append>N</log_file_append>
<create_parent_folder>N</create_parent_folder>
<clear_log>Y</clear_log>
<show_subcomponents>Y</show_subcomponents>
<run_configuration>local</run_configuration>
</pipeline_execution_configuration>
<metastore_json></metastore_json>
</pipeline_configuration>
GitHub link: https://github.com/apache/hop/discussions/4634#discussioncomment-14858044
----
This is an automatically sent email for users@hop.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@hop.apache.org