Transfer components to streamPipes Sinks

[Robert Edward Herding <ro...@yandex.com>]

Hello folks,

After a few weeks of work I am proud to say; now I have some working adapters,
processors and Sinks.

Have to tell you that took quite effort to make it work. But now is really
working like a charm.

And I can also say that take me minutes now to build a new one.

Now I am gonna bother you guys with probably the last question (regarding
basics, of course).

How can I "upload" the component to a Streampipes container in kubernets ?

I am thinking on creating a new container and running the java program. Then I
think StreamPipes will connect to that server (setting the ip in the manage
endpoints)?  
Can you please once more give me a north on this question?

I will really appreciate any help or experience sharing. Because I am pretty
sure that there is not only one way of doing it.

Thanks again.

Patrick I still have a few more notes to care, hope this helps.

Best regards

Robert Edward Hearding

Re: Transfer components to streamPipes Sinks

[Dominik Riemer <ri...@apache.org>]

Hi Robert,

great and thanks for your feedback! It would be interesting to hear where you had problems or where you missed documentation, so that we can further improve the process.

There are two steps involved in deploying a custom pipeline element:

Regarding deployment, in case you used the Maven archetype for development, there should be a Dockerfile in the root directory. It defines an env variable "CONSUL_LOCATION" which defaults to the hostname "consul". You can use this Dockerfile to create an image. 

Second, the archetype also create a "Config" class in the config package. In this class, you can find a config variable "HOST" which registers the hostname. You can change the default hostname as needed, you just need to make sure that it matches the service name in your docker-compose file or your k8s config.

When running the pipeline element in a containerized network, make sure that the pipeline element is in the same network as the other StreamPipes module (e.g., when using Docker Compose, there is a network spnet and a similar concept is also in the helm charts). Once you start the container and Consul is reachable, the pipeline element will automatically connect to Consul and will appear in the "Pipeline Element Installation" section of StreamPipes so that you don't need to manually set the ip under managed endpoints.

One side note, in the next release, this process will be further simplified and StreamPipes will detect the correct host/ip automatically and the config class will be removed.

Hope this helps! Please ask if anything is unclear!

Dominik

On 2021/09/22 10:08:24, Robert Edward Herding <ro...@yandex.com> wrote: 
> Hello folks,
> 
> After a few weeks of work I am proud to say; now I have some working adapters,
> processors and Sinks.
> 
> Have to tell you that took quite effort to make it work. But now is really
> working like a charm.
> 
> And I can also say that take me minutes now to build a new one.
> 
> Now I am gonna bother you guys with probably the last question (regarding
> basics, of course).
> 
> How can I "upload" the component to a Streampipes container in kubernets ?
> 
> I am thinking on creating a new container and running the java program. Then I
> think StreamPipes will connect to that server (setting the ip in the manage
> endpoints)?  
> Can you please once more give me a north on this question?
> 
> I will really appreciate any help or experience sharing. Because I am pretty
> sure that there is not only one way of doing it.
> 
> Thanks again.
> 
> Patrick I still have a few more notes to care, hope this helps.
> 
> Best regards
> 
> Robert Edward Hearding
> 
>