You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@hop.apache.org by Phillip Brown <ph...@gmail.com> on 2023/03/09 05:16:35 UTC
HOP Server REST API
Hi
Trying to understand the Hop Server REST API, and not finding the user
manual all that helpful (Note, the Pentaho documentation is not any
better in this regard)
First, there seems to be duplication without any explanation of when to
use some calls rather than other calls. For example, when would I use
addPipeline vs registerPipeline? Why would I use prepareExec and
startExec instead of startPipeline?
Second, "Request body should contain xml containing
pipeline_configuration (pipeline and pipeline_execution_configuration
wrapped in pipeline_configuration tag)" is there without any real
explanation of how you go about creating that request body, or where the
pieces come from. There appears to be the implicit assumption that
people should just know what it means. And the example in
registerPipeline doesn't really help, and has an additional
"metastore_json is base64 encoded GZip content" which also isn't explained.
Finally, how would I go about doing something like running a pipeline
from another application like Oracle APEX (see, for example,
https://pretius.com/blog/pentaho-data-integration-oracle-apex/) ?
Regards
Phil Brown
Re: HOP Server REST API
Posted by Matt Casters <ma...@leanwithdata.com>.
Documentation is available over here:
https://hop.apache.org/manual/latest/hop-server/async-web-service.html
It contains the POST sample for the async web service (executing a
workflow). The same idea would work for a web service (pipeline).
HTH,
Matt
On Tue, 21 Mar 2023 at 13:56, <po...@gmx.com> wrote:
>
> So Matt, I have no idea how can I sent JSON to Carte server. Maybe you
> have some example?
> Best
>
> *Sent:* Tuesday, March 21, 2023 at 1:43 PM
> *From:* "Matt Casters" <ma...@leanwithdata.com>
> *To:* users@hop.apache.org
> *Subject:* Re: HOP Server REST API
> Again, you can already do this using the existing web service. The only
> difference is that your pipeline is wrapped in a bit of metadata, something
> most people would want anyway for obvious reasons.
>
>
> On Tue, 21 Mar 2023 at 13:40, <po...@gmx.com> wrote:
>
>> I meant *Hop Server* could handle POST payload. Now it is only possible
>> to call it with parameters, for instance:
>>
>> http://127.0.0.1:8181/kettle/executeTrans/?trans=sometransformation.ktr&
>> patameter1=m42
>>
>> But sometimes I need to post XML, JSON or else
>> *Sent:* Tuesday, March 21, 2023 at 1:06 PM
>> *From:* "Matt Casters" <ma...@leanwithdata.com>
>> *To:* users@hop.apache.org
>> *Subject:* Re: HOP Server REST API
>> Both the Web Service (pipelines) and Asynchronous Web Service (workflows)
>> calls support both a content body as well as parameters.
>>
>> Cheers,
>> Matt
>>
>> https://hop.apache.org/manual/latest/hop-server/web-service.html
>> https://hop.apache.org/manual/latest/metadata-types/asyncwebservice.html
>>
>>
>> On Tue, 21 Mar 2023 at 12:51, <po...@gmx.com> wrote:
>>
>>>
>>> Would be great if we could sent some palyload to Carte (I do not know
>>> what's the name in Hop) - not only parameters. That's what I've always
>>> missed in Kettle.
>>>
>>> *Sent:* Thursday, March 09, 2023 at 9:36 AM
>>> *From:* "Matt Casters" <ma...@leanwithdata.com>
>>> *To:* users@hop.apache.org
>>> *Subject:* Re: HOP Server REST API
>>> Hi Phil,
>>>
>>> The easiest way to start a pipeline (or workflow) from another
>>> application is probably to use a named synchronous or asynchronous web
>>> service.
>>> See here for more information:
>>> https://hop.apache.org//manual/latest/hop-server/web-service.html
>>>
>>> Your point about the lacking REST API is taken though. We're
>>> actually in the process of making a better set of services.
>>> Please let us know what you need in terms of the interface so we can
>>> build this out over time. We'll have a docker container to go along with
>>> that as well.
>>> https://hop.apache.org//manual/next/hop-rest/index.html
>>>
>>> All the best,
>>> Matt
>>>
>>> On Thu, 9 Mar 2023 at 06:16, Phillip Brown <ph...@gmail.com>
>>> wrote:
>>>
>>>> Hi
>>>>
>>>> Trying to understand the Hop Server REST API, and not finding the user
>>>> manual all that helpful (Note, the Pentaho documentation is not any
>>>> better in this regard)
>>>>
>>>> First, there seems to be duplication without any explanation of when to
>>>> use some calls rather than other calls. For example, when would I use
>>>> addPipeline vs registerPipeline? Why would I use prepareExec and
>>>> startExec instead of startPipeline?
>>>>
>>>> Second, "Request body should contain xml containing
>>>> pipeline_configuration (pipeline and pipeline_execution_configuration
>>>> wrapped in pipeline_configuration tag)" is there without any real
>>>> explanation of how you go about creating that request body, or where the
>>>> pieces come from. There appears to be the implicit assumption that
>>>> people should just know what it means. And the example in
>>>> registerPipeline doesn't really help, and has an additional
>>>> "metastore_json is base64 encoded GZip content" which also isn't
>>>> explained.
>>>>
>>>> Finally, how would I go about doing something like running a pipeline
>>>> from another application like Oracle APEX (see, for example,
>>>> https://pretius.com/blog/pentaho-data-integration-oracle-apex/) ?
>>>>
>>>> Regards
>>>>
>>>> Phil Brown
>>>>
>>>
>>>
>>>
>>>
>>
>>
>>
>
>
>
Re: HOP Server REST API
Posted by po...@gmx.com.
So Matt, I have no idea how can I sent JSON to Carte server. Maybe you have
some example?
Best
**Sent:** Tuesday, March 21, 2023 at 1:43 PM
**From:** "Matt Casters" <ma...@leanwithdata.com>
**To:** users@hop.apache.org
**Subject:** Re: HOP Server REST API
Again, you can already do this using the existing web service. The only
difference is that your pipeline is wrapped in a bit of metadata, something
most people would want anyway for obvious reasons.
On Tue, 21 Mar 2023 at 13:40, <[podunk@gmx.com](mailto:podunk@gmx.com)> wrote:
> I meant **Hop Server** could handle POST payload. Now it is only possible to
> call it with parameters, for instance:
>
>
>
>
> <http://127.0.0.1:8181/kettle/executeTrans/?trans=>sometransformation.ktr&patameter1=m42
>
>
>
> But sometimes I need to post XML, JSON or else
>
> **Sent:** Tuesday, March 21, 2023 at 1:06 PM
> **From:** "Matt Casters"
> <[matt.casters@leanwithdata.com](mailto:matt.casters@leanwithdata.com)>
> **To:** [users@hop.apache.org](mailto:users@hop.apache.org)
> **Subject:** Re: HOP Server REST API
>
> Both the Web Service (pipelines) and Asynchronous Web Service (workflows)
> calls support both a content body as well as parameters.
>
>
>
> Cheers,
> Matt
>
>
>
> <https://hop.apache.org/manual/latest/hop-server/web-service.html>
>
> <https://hop.apache.org/manual/latest/metadata-types/asyncwebservice.html>
>
>
>
>
>
> On Tue, 21 Mar 2023 at 12:51, <[podunk@gmx.com](mailto:podunk@gmx.com)>
> wrote:
>
>>
>>
>> Would be great if we could sent some palyload to Carte (I do not know
what's the name in Hop) - not only parameters. That's what I've always missed
in Kettle.
>>
>>
>>
>> **Sent:** Thursday, March 09, 2023 at 9:36 AM
> **From:** "Matt Casters"
> <[matt.casters@leanwithdata.com](mailto:matt.casters@leanwithdata.com)>
> **To:** [users@hop.apache.org](mailto:users@hop.apache.org)
> **Subject:** Re: HOP Server REST API
>>
>> Hi Phil,
>>
>>
>>
>> The easiest way to start a pipeline (or workflow) from another application
is probably to use a named synchronous or asynchronous web service.
>>
>> See here for more information: <https://hop.apache.org//manual/latest/hop-
server/web-service.html>
>>
>>
>>
>> Your point about the lacking REST API is taken though. We're actually in
the process of making a better set of services.
>>
>> Please let us know what you need in terms of the interface so we can build
this out over time. We'll have a docker container to go along with that as
well.
>>
>> <https://hop.apache.org//manual/next/hop-rest/index.html>
>>
>>
>>
>> All the best,
> Matt
>>
>>
>>
>> On Thu, 9 Mar 2023 at 06:16, Phillip Brown
<[phillip.brown.j@gmail.com](mailto:phillip.brown.j@gmail.com)> wrote:
>>
>>> Hi
>
> Trying to understand the Hop Server REST API, and not finding the user
> manual all that helpful (Note, the Pentaho documentation is not any
> better in this regard)
>
> First, there seems to be duplication without any explanation of when to
> use some calls rather than other calls. For example, when would I use
> addPipeline vs registerPipeline? Why would I use prepareExec and
> startExec instead of startPipeline?
>
> Second, "Request body should contain xml containing
> pipeline_configuration (pipeline and pipeline_execution_configuration
> wrapped in pipeline_configuration tag)" is there without any real
> explanation of how you go about creating that request body, or where the
> pieces come from. There appears to be the implicit assumption that
> people should just know what it means. And the example in
> registerPipeline doesn't really help, and has an additional
> "metastore_json is base64 encoded GZip content" which also isn't explained.
>
> Finally, how would I go about doing something like running a pipeline
> from another application like Oracle APEX (see, for example,
> <https://pretius.com/blog/pentaho-data-integration-oracle-apex/>) ?
>
> Regards
>
> Phil Brown
>
>>
>>
>>
>>
>
>
>
>
Re: HOP Server REST API
Posted by Matt Casters <ma...@leanwithdata.com>.
Again, you can already do this using the existing web service. The only
difference is that your pipeline is wrapped in a bit of metadata, something
most people would want anyway for obvious reasons.
On Tue, 21 Mar 2023 at 13:40, <po...@gmx.com> wrote:
> I meant *Hop Server* could handle POST payload. Now it is only possible
> to call it with parameters, for instance:
>
> http://127.0.0.1:8181/kettle/executeTrans/?trans=sometransformation.ktr&
> patameter1=m42
>
> But sometimes I need to post XML, JSON or else
> *Sent:* Tuesday, March 21, 2023 at 1:06 PM
> *From:* "Matt Casters" <ma...@leanwithdata.com>
> *To:* users@hop.apache.org
> *Subject:* Re: HOP Server REST API
> Both the Web Service (pipelines) and Asynchronous Web Service (workflows)
> calls support both a content body as well as parameters.
>
> Cheers,
> Matt
>
> https://hop.apache.org/manual/latest/hop-server/web-service.html
> https://hop.apache.org/manual/latest/metadata-types/asyncwebservice.html
>
>
> On Tue, 21 Mar 2023 at 12:51, <po...@gmx.com> wrote:
>
>>
>> Would be great if we could sent some palyload to Carte (I do not know
>> what's the name in Hop) - not only parameters. That's what I've always
>> missed in Kettle.
>>
>> *Sent:* Thursday, March 09, 2023 at 9:36 AM
>> *From:* "Matt Casters" <ma...@leanwithdata.com>
>> *To:* users@hop.apache.org
>> *Subject:* Re: HOP Server REST API
>> Hi Phil,
>>
>> The easiest way to start a pipeline (or workflow) from another
>> application is probably to use a named synchronous or asynchronous web
>> service.
>> See here for more information:
>> https://hop.apache.org//manual/latest/hop-server/web-service.html
>>
>> Your point about the lacking REST API is taken though. We're actually in
>> the process of making a better set of services.
>> Please let us know what you need in terms of the interface so we can
>> build this out over time. We'll have a docker container to go along with
>> that as well.
>> https://hop.apache.org//manual/next/hop-rest/index.html
>>
>> All the best,
>> Matt
>>
>> On Thu, 9 Mar 2023 at 06:16, Phillip Brown <ph...@gmail.com>
>> wrote:
>>
>>> Hi
>>>
>>> Trying to understand the Hop Server REST API, and not finding the user
>>> manual all that helpful (Note, the Pentaho documentation is not any
>>> better in this regard)
>>>
>>> First, there seems to be duplication without any explanation of when to
>>> use some calls rather than other calls. For example, when would I use
>>> addPipeline vs registerPipeline? Why would I use prepareExec and
>>> startExec instead of startPipeline?
>>>
>>> Second, "Request body should contain xml containing
>>> pipeline_configuration (pipeline and pipeline_execution_configuration
>>> wrapped in pipeline_configuration tag)" is there without any real
>>> explanation of how you go about creating that request body, or where the
>>> pieces come from. There appears to be the implicit assumption that
>>> people should just know what it means. And the example in
>>> registerPipeline doesn't really help, and has an additional
>>> "metastore_json is base64 encoded GZip content" which also isn't
>>> explained.
>>>
>>> Finally, how would I go about doing something like running a pipeline
>>> from another application like Oracle APEX (see, for example,
>>> https://pretius.com/blog/pentaho-data-integration-oracle-apex/) ?
>>>
>>> Regards
>>>
>>> Phil Brown
>>>
>>
>>
>>
>>
>
>
>
Re: HOP Server REST API
Posted by po...@gmx.com.
I meant **Hop Server** could handle POST payload. Now it is only possible to
call it with parameters, for instance:
http://127.0.0.1:8181/kettle/executeTrans/?trans=sometransformation.ktr&patameter1=m42
But sometimes I need to post XML, JSON or else
**Sent:** Tuesday, March 21, 2023 at 1:06 PM
**From:** "Matt Casters" <ma...@leanwithdata.com>
**To:** users@hop.apache.org
**Subject:** Re: HOP Server REST API
Both the Web Service (pipelines) and Asynchronous Web Service (workflows)
calls support both a content body as well as parameters.
Cheers,
Matt
<https://hop.apache.org/manual/latest/hop-server/web-service.html>
<https://hop.apache.org/manual/latest/metadata-types/asyncwebservice.html>
On Tue, 21 Mar 2023 at 12:51, <[podunk@gmx.com](mailto:podunk@gmx.com)> wrote:
>
>
> Would be great if we could sent some palyload to Carte (I do not know what's
> the name in Hop) - not only parameters. That's what I've always missed in
> Kettle.
>
>
>
> **Sent:** Thursday, March 09, 2023 at 9:36 AM
> **From:** "Matt Casters"
> <[matt.casters@leanwithdata.com](mailto:matt.casters@leanwithdata.com)>
> **To:** [users@hop.apache.org](mailto:users@hop.apache.org)
> **Subject:** Re: HOP Server REST API
>
> Hi Phil,
>
>
>
> The easiest way to start a pipeline (or workflow) from another application
> is probably to use a named synchronous or asynchronous web service.
>
> See here for more information: <https://hop.apache.org//manual/latest/hop-
> server/web-service.html>
>
>
>
> Your point about the lacking REST API is taken though. We're actually in
> the process of making a better set of services.
>
> Please let us know what you need in terms of the interface so we can build
> this out over time. We'll have a docker container to go along with that as
> well.
>
> <https://hop.apache.org//manual/next/hop-rest/index.html>
>
>
>
> All the best,
> Matt
>
>
>
> On Thu, 9 Mar 2023 at 06:16, Phillip Brown
> <[phillip.brown.j@gmail.com](mailto:phillip.brown.j@gmail.com)> wrote:
>
>> Hi
>
> Trying to understand the Hop Server REST API, and not finding the user
> manual all that helpful (Note, the Pentaho documentation is not any
> better in this regard)
>
> First, there seems to be duplication without any explanation of when to
> use some calls rather than other calls. For example, when would I use
> addPipeline vs registerPipeline? Why would I use prepareExec and
> startExec instead of startPipeline?
>
> Second, "Request body should contain xml containing
> pipeline_configuration (pipeline and pipeline_execution_configuration
> wrapped in pipeline_configuration tag)" is there without any real
> explanation of how you go about creating that request body, or where the
> pieces come from. There appears to be the implicit assumption that
> people should just know what it means. And the example in
> registerPipeline doesn't really help, and has an additional
> "metastore_json is base64 encoded GZip content" which also isn't explained.
>
> Finally, how would I go about doing something like running a pipeline
> from another application like Oracle APEX (see, for example,
> <https://pretius.com/blog/pentaho-data-integration-oracle-apex/>) ?
>
> Regards
>
> Phil Brown
>
>
>
>
>
Re: HOP Server REST API
Posted by Matt Casters <ma...@leanwithdata.com>.
Both the Web Service (pipelines) and Asynchronous Web Service (workflows)
calls support both a content body as well as parameters.
Cheers,
Matt
https://hop.apache.org/manual/latest/hop-server/web-service.html
https://hop.apache.org/manual/latest/metadata-types/asyncwebservice.html
On Tue, 21 Mar 2023 at 12:51, <po...@gmx.com> wrote:
>
> Would be great if we could sent some palyload to Carte (I do not know
> what's the name in Hop) - not only parameters. That's what I've always
> missed in Kettle.
>
> *Sent:* Thursday, March 09, 2023 at 9:36 AM
> *From:* "Matt Casters" <ma...@leanwithdata.com>
> *To:* users@hop.apache.org
> *Subject:* Re: HOP Server REST API
> Hi Phil,
>
> The easiest way to start a pipeline (or workflow) from another application
> is probably to use a named synchronous or asynchronous web service.
> See here for more information:
> https://hop.apache.org//manual/latest/hop-server/web-service.html
>
> Your point about the lacking REST API is taken though. We're actually in
> the process of making a better set of services.
> Please let us know what you need in terms of the interface so we can build
> this out over time. We'll have a docker container to go along with that as
> well.
> https://hop.apache.org//manual/next/hop-rest/index.html
>
> All the best,
> Matt
>
> On Thu, 9 Mar 2023 at 06:16, Phillip Brown <ph...@gmail.com>
> wrote:
>
>> Hi
>>
>> Trying to understand the Hop Server REST API, and not finding the user
>> manual all that helpful (Note, the Pentaho documentation is not any
>> better in this regard)
>>
>> First, there seems to be duplication without any explanation of when to
>> use some calls rather than other calls. For example, when would I use
>> addPipeline vs registerPipeline? Why would I use prepareExec and
>> startExec instead of startPipeline?
>>
>> Second, "Request body should contain xml containing
>> pipeline_configuration (pipeline and pipeline_execution_configuration
>> wrapped in pipeline_configuration tag)" is there without any real
>> explanation of how you go about creating that request body, or where the
>> pieces come from. There appears to be the implicit assumption that
>> people should just know what it means. And the example in
>> registerPipeline doesn't really help, and has an additional
>> "metastore_json is base64 encoded GZip content" which also isn't
>> explained.
>>
>> Finally, how would I go about doing something like running a pipeline
>> from another application like Oracle APEX (see, for example,
>> https://pretius.com/blog/pentaho-data-integration-oracle-apex/) ?
>>
>> Regards
>>
>> Phil Brown
>>
>
>
>
>
Re: HOP Server REST API
Posted by po...@gmx.com.
Would be great if we could sent some palyload to Carte (I do not know what's
the name in Hop) - not only parameters. That's what I've always missed in
Kettle.
**Sent:** Thursday, March 09, 2023 at 9:36 AM
**From:** "Matt Casters" <ma...@leanwithdata.com>
**To:** users@hop.apache.org
**Subject:** Re: HOP Server REST API
Hi Phil,
The easiest way to start a pipeline (or workflow) from another application is
probably to use a named synchronous or asynchronous web service.
See here for more information: <https://hop.apache.org//manual/latest/hop-
server/web-service.html>
Your point about the lacking REST API is taken though. We're actually in the
process of making a better set of services.
Please let us know what you need in terms of the interface so we can build
this out over time. We'll have a docker container to go along with that as
well.
<https://hop.apache.org//manual/next/hop-rest/index.html>
All the best,
Matt
On Thu, 9 Mar 2023 at 06:16, Phillip Brown
<[phillip.brown.j@gmail.com](mailto:phillip.brown.j@gmail.com)> wrote:
> Hi
>
> Trying to understand the Hop Server REST API, and not finding the user
> manual all that helpful (Note, the Pentaho documentation is not any
> better in this regard)
>
> First, there seems to be duplication without any explanation of when to
> use some calls rather than other calls. For example, when would I use
> addPipeline vs registerPipeline? Why would I use prepareExec and
> startExec instead of startPipeline?
>
> Second, "Request body should contain xml containing
> pipeline_configuration (pipeline and pipeline_execution_configuration
> wrapped in pipeline_configuration tag)" is there without any real
> explanation of how you go about creating that request body, or where the
> pieces come from. There appears to be the implicit assumption that
> people should just know what it means. And the example in
> registerPipeline doesn't really help, and has an additional
> "metastore_json is base64 encoded GZip content" which also isn't explained.
>
> Finally, how would I go about doing something like running a pipeline
> from another application like Oracle APEX (see, for example,
> <https://pretius.com/blog/pentaho-data-integration-oracle-apex/>) ?
>
> Regards
>
> Phil Brown
>
Re: HOP Server REST API
Posted by Matt Casters <ma...@leanwithdata.com>.
Hi Phil,
The easiest way to start a pipeline (or workflow) from another application
is probably to use a named synchronous or asynchronous web service.
See here for more information:
https://hop.apache.org//manual/latest/hop-server/web-service.html
Your point about the lacking REST API is taken though. We're actually in
the process of making a better set of services.
Please let us know what you need in terms of the interface so we can build
this out over time. We'll have a docker container to go along with that as
well.
https://hop.apache.org//manual/next/hop-rest/index.html
All the best,
Matt
On Thu, 9 Mar 2023 at 06:16, Phillip Brown <ph...@gmail.com>
wrote:
> Hi
>
> Trying to understand the Hop Server REST API, and not finding the user
> manual all that helpful (Note, the Pentaho documentation is not any
> better in this regard)
>
> First, there seems to be duplication without any explanation of when to
> use some calls rather than other calls. For example, when would I use
> addPipeline vs registerPipeline? Why would I use prepareExec and
> startExec instead of startPipeline?
>
> Second, "Request body should contain xml containing
> pipeline_configuration (pipeline and pipeline_execution_configuration
> wrapped in pipeline_configuration tag)" is there without any real
> explanation of how you go about creating that request body, or where the
> pieces come from. There appears to be the implicit assumption that
> people should just know what it means. And the example in
> registerPipeline doesn't really help, and has an additional
> "metastore_json is base64 encoded GZip content" which also isn't explained.
>
> Finally, how would I go about doing something like running a pipeline
> from another application like Oracle APEX (see, for example,
> https://pretius.com/blog/pentaho-data-integration-oracle-apex/) ?
>
> Regards
>
> Phil Brown
>
>