You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@streampipes.apache.org by "Canteri, Michele" <Mi...@gft.com> on 2021/07/21 10:05:12 UTC
Rest publisher
Hi all,
I am trying to expose data via the Rest publisher sink but I don't get how it works, especially if and how it is possible to expose the data on a public address.
My configuration of the Rest publisher is currently:
http://X.X.X.X:5001/My-Test
where X.X.X.X is the public IP and I can access the data exposed only on the same server. I've correctly configured the firewalls to open the port 5001 for both TCP and UDP connections.
I tried using netcat in this way:
nc -l -k X.X.X.X 5001
and I have the POST data but I think only because it is routed on the 0.0.0.0 address and I verified it having the same result using
nc -l -k 0.0.0.0 5001
In both cases I have this result:
POST /My-Test HTTP/1.1
Content-Length: 211
Content-Type: application/json; charset=UTF-8
Host: X.X.X.X:5001
Connection: Keep-Alive
User-Agent: Apache-HttpClient/4.5.13 (Java/1.8.0_292)
Accept-Encoding: gzip,deflate
{"mass_flow":9.2311,"underflow":false,"overflow":false,"level":70.0957,"sensorId_0":"pressure01","temperature":43.8133,"sensorId_1":"level01","pressure":63.8947,"timestamp":1626860022495,"sensorId":"flowrate01"}
That is the correct data from the data simulators.
Any help in understanding what I am doing wrong is appreciated and let me know if some logs can be of help.
Thank you in advance,
Michele
Michele Canteri
_________________________________________________________________
GFT Italia S.r.l.
VIA SILE 18 / floor 4
20139 Milano, Italy
T +39 02 58158503
F +39 02 58158621
Michele.Canteri@gft.com
www.gft.com/it<http://www.gft.com/it>
https://blog.gft.com
https://www.linkedin.com/company/gft-group/
www.twitter.com/gft_it <http://www.twitter.com/gft_it>
[cid:ActoForTheFuture_GFT@1,2x_68f73108-d76e-4497-aa5c-ff8a5693ce57.png]<https://www.gft.com/it/it/index/societa/stampa/press-releases/2021/sostenibilita-e-innovazione-sociale--gft-lancia-il-greencoding-e-in-italia-act-for-the-future/>
Registered office: GFT Italia S.r.l., Via Sile 18, 20139 Milan (Italy)
VAT Code and number of registration with the Companies' Register of Milan: 00819200478
Corporate capital: Euro 1,000,000 fully paid-in
Company subject to the direction and coordination activity of GFT Technologies SE
Ai sensi del Regolamento UE 2016/679 (GDPR) e delle Linee Guida per posta elettronica ed Internet del Garante Privacy si rende noto che le informazioni contenute nella presente comunicazione e i relativi allegati possono essere riservate e sono, comunque, destinate esclusivamente alle persone o alla Societ? sopraindicati. Le stesse non sono da considerarsi comunicazioni personali, quindi eventuali risposte potranno essere conosciute da persone appartenenti all'azienda.
In accordance with Regulation EU 2016/679 (General Data Protection Regulation) and Guidelines Applying to the Use of E-Mails and the Internet in the Employment Context of Italian Data Protection Authority it is announced that the information in this communication (which includes any documents with it) is confidential and meant only for above-mentioned persons and/or Companies; the information is not personal therefore, It is intended for the addressee only and any replies will be known by people belonging to the company.
_________________________________________________________________
RE: Rest publisher
Posted by "Canteri, Michele" <Mi...@gft.com>.
Hi Patrick,
thank you for your answer.
I don't have a specialized Restful endpoint running, we will set it up asap and then will come back if something is still missing.
I am on a linux (Ubuntu 20.04 LTS) server.
Best regards,
Michele
Michele Canteri
_________________________________________________________________
GFT Italia S.r.l.
VIA SILE 18 / floor 4
20139 Milano, Italy
T +39 02 58158503
F +39 02 58158621
Michele.Canteri@gft.com
www.gft.com/it<http://www.gft.com/it>
https://blog.gft.com
https://www.linkedin.com/company/gft-group/
www.twitter.com/gft_it <http://www.twitter.com/gft_it>
[cid:ActoForTheFuture_GFT@1,2x_68f73108-d76e-4497-aa5c-ff8a5693ce57.png]<https://www.gft.com/it/it/index/societa/stampa/press-releases/2021/sostenibilita-e-innovazione-sociale--gft-lancia-il-greencoding-e-in-italia-act-for-the-future/>
Registered office: GFT Italia S.r.l., Via Sile 18, 20139 Milan (Italy)
VAT Code and number of registration with the Companies' Register of Milan: 00819200478
Corporate capital: Euro 1,000,000 fully paid-in
Company subject to the direction and coordination activity of GFT Technologies SE
Ai sensi del Regolamento UE 2016/679 (GDPR) e delle Linee Guida per posta elettronica ed Internet del Garante Privacy si rende noto che le informazioni contenute nella presente comunicazione e i relativi allegati possono essere riservate e sono, comunque, destinate esclusivamente alle persone o alla Società sopraindicati. Le stesse non sono da considerarsi comunicazioni personali, quindi eventuali risposte potranno essere conosciute da persone appartenenti all'azienda.
In accordance with Regulation EU 2016/679 (General Data Protection Regulation) and Guidelines Applying to the Use of E-Mails and the Internet in the Employment Context of Italian Data Protection Authority it is announced that the information in this communication (which includes any documents with it) is confidential and meant only for above-mentioned persons and/or Companies; the information is not personal therefore, It is intended for the addressee only and any replies will be known by people belonging to the company.
_________________________________________________________________
From: Patrick Wiener <wi...@apache.org>
Sent: mercoledì 21 luglio 2021 14:47
To: users@streampipes.apache.org
Subject: Re: Rest publisher
[EXTERNAL EMAIL]
This email originated from outside of the organization or was sent unauthenticated. Do not click links or open attachments unless you confirm the sender and know the content is safe.
Hi Michele,
I'm not quite sure if I completely understood what you're trying to achieve.
Let's say you have a pipeline: flowrate -> REST publisher (with the endpoint http://<https://eur02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fx.x.x.x%3A5001%2FMy-Test&data=04%7C01%7CMichele.Canteri%40gft.com%7Cbe72ba566408495f3a7108d94c45989a%7C55a2bc67aec14ad29a9c5b2457b91dcd%7C0%7C0%7C637624684062881904%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=O%2FN8gBGWyBHirSU%2Fq1mVOm57xPsVN4ehaXoAQmk%2BjYg%3D&reserved=0>X.X.X.X:5001/My-Test<https://eur02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fx.x.x.x%3A5001%2FMy-Test&data=04%7C01%7CMichele.Canteri%40gft.com%7Cbe72ba566408495f3a7108d94c45989a%7C55a2bc67aec14ad29a9c5b2457b91dcd%7C0%7C0%7C637624684062881904%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=O%2FN8gBGWyBHirSU%2Fq1mVOm57xPsVN4ehaXoAQmk%2BjYg%3D&reserved=0>)
The REST publisher sends a HTTP POST with the event serialized as JSON to a resolvable RESTful endpoint external to StreamPipes.
Calling StreamPipes using HTTP GET won't work.
Thus, I assume you have such a RESTful service running on your machine on X.X.X.X:5001?
If the host X.X.X.X is the same machine as StreamPipes is running the REST publisher running inside Docker should not have a problem connecting to it in order to POST data to that endpoint.
If it's another host you need to make that the network is setup right and that the other host is reachable and the port 5001 is accessible.
Whats your system? Linux/Windows/Mac?
Cheers
Patrick
Am 21.07.2021 um 12:05 schrieb Canteri, Michele <Mi...@gft.com>>:
Hi all,
I am trying to expose data via the Rest publisher sink but I don't get how it works, especially if and how it is possible to expose the data on a public address.
My configuration of the Rest publisher is currently:
http://X.X.X.X:5001/My-Test<https://eur02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fx.x.x.x%3A5001%2FMy-Test&data=04%7C01%7CMichele.Canteri%40gft.com%7Cbe72ba566408495f3a7108d94c45989a%7C55a2bc67aec14ad29a9c5b2457b91dcd%7C0%7C0%7C637624684062891859%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=hqfHbVVFvpnZowy6huAHV1imIzYHcRu%2BVlVt72bfeTU%3D&reserved=0>
where X.X.X.X is the public IP and I can access the data exposed only on the same server. I've correctly configured the firewalls to open the port 5001 for both TCP and UDP connections.
I tried using netcat in this way:
nc -l -k X.X.X.X 5001
and I have the POST data but I think only because it is routed on the 0.0.0.0 address and I verified it having the same result using
nc -l -k 0.0.0.0 5001
In both cases I have this result:
POST /My-Test HTTP/1.1
Content-Length: 211
Content-Type: application/json; charset=UTF-8
Host: X.X.X.X:5001
Connection: Keep-Alive
User-Agent: Apache-HttpClient/4.5.13 (Java/1.8.0_292)
Accept-Encoding: gzip,deflate
{"mass_flow":9.2311,"underflow":false,"overflow":false,"level":70.0957,"sensorId_0":"pressure01","temperature":43.8133,"sensorId_1":"level01","pressure":63.8947,"timestamp":1626860022495,"sensorId":"flowrate01"}
That is the correct data from the data simulators.
Any help in understanding what I am doing wrong is appreciated and let me know if some logs can be of help.
Thank you in advance,
Michele
Michele Canteri
_________________________________________________________________
GFT Italia S.r.l.
VIA SILE 18 / floor 4
20139 Milano, Italy
T +39 02 58158503
F +39 02 58158621
Michele.Canteri@gft.com<ma...@gft.com>
www.gft.com/it<https://eur02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.gft.com%2Fit&data=04%7C01%7CMichele.Canteri%40gft.com%7Cbe72ba566408495f3a7108d94c45989a%7C55a2bc67aec14ad29a9c5b2457b91dcd%7C0%7C0%7C637624684062901817%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=dhMR0NL4PBuii4DiNQTPvlUhIq%2FLPi%2Fi37GDBwHfzlI%3D&reserved=0>
https://blog.gft.com<https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fblog.gft.com%2F&data=04%7C01%7CMichele.Canteri%40gft.com%7Cbe72ba566408495f3a7108d94c45989a%7C55a2bc67aec14ad29a9c5b2457b91dcd%7C0%7C0%7C637624684062911775%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=QwQoifDHD9E0xOJ1DHzkHVASBgqh%2F3bWTwZa1Iljv6w%3D&reserved=0>
https://www.linkedin.com/company/gft-group/<https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.linkedin.com%2Fcompany%2Fgft-group%2F&data=04%7C01%7CMichele.Canteri%40gft.com%7Cbe72ba566408495f3a7108d94c45989a%7C55a2bc67aec14ad29a9c5b2457b91dcd%7C0%7C0%7C637624684062911775%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=FOPKn%2Fb7U61GPaI%2FNy3SxJB0xXlQuJ9ynXwGbR2LWx4%3D&reserved=0>
www.twitter.com/gft_it <https://eur02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.twitter.com%2Fgft_it&data=04%7C01%7CMichele.Canteri%40gft.com%7Cbe72ba566408495f3a7108d94c45989a%7C55a2bc67aec14ad29a9c5b2457b91dcd%7C0%7C0%7C637624684062921728%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=g%2FJEzwNXk%2BK7z%2BEpj0Q1E7PqUAjqdS0i9lpQDVdHi%2Fs%3D&reserved=0>
<ActoForTheFuture_GFT@1,2x_68f73108-d76e-4497-aa5c-ff8a5693ce57.png><https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.gft.com%2Fit%2Fit%2Findex%2Fsocieta%2Fstampa%2Fpress-releases%2F2021%2Fsostenibilita-e-innovazione-sociale--gft-lancia-il-greencoding-e-in-italia-act-for-the-future%2F&data=04%7C01%7CMichele.Canteri%40gft.com%7Cbe72ba566408495f3a7108d94c45989a%7C55a2bc67aec14ad29a9c5b2457b91dcd%7C0%7C0%7C637624684062931682%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=g9bzyxx1huUJXDdnqUzW8filHMHzNTDyM4NQetGIBVM%3D&reserved=0>
Registered office: GFT Italia S.r.l., Via Sile 18, 20139 Milan (Italy)
VAT Code and number of registration with the Companies' Register of Milan: 00819200478
Corporate capital: Euro 1,000,000 fully paid-in
Company subject to the direction and coordination activity of GFT Technologies SE
Ai sensi del Regolamento UE 2016/679 (GDPR) e delle Linee Guida per posta elettronica ed Internet del Garante Privacy si rende noto che le informazioni contenute nella presente comunicazione e i relativi allegati possono essere riservate e sono, comunque, destinate esclusivamente alle persone o alla Società sopraindicati. Le stesse non sono da considerarsi comunicazioni personali, quindi eventuali risposte potranno essere conosciute da persone appartenenti all'azienda.
In accordance with Regulation EU 2016/679 (General Data Protection Regulation) and Guidelines Applying to the Use of E-Mails and the Internet in the Employment Context of Italian Data Protection Authority it is announced that the information in this communication (which includes any documents with it) is confidential and meant only for above-mentioned persons and/or Companies; the information is not personal therefore, It is intended for the addressee only and any replies will be known by people belonging to the company.
_________________________________________________________________
Re: Rest publisher
Posted by Patrick Wiener <wi...@apache.org>.
Hi Michele,
I’m not quite sure if I completely understood what you’re trying to achieve.
Let’s say you have a pipeline: flowrate -> REST publisher (with the endpoint http:// <http://x.x.x.x:5001/My-Test>X.X.X.X:5001/My-Test <http://x.x.x.x:5001/My-Test>)
The REST publisher sends a HTTP POST with the event serialized as JSON to a resolvable RESTful endpoint external to StreamPipes.
Calling StreamPipes using HTTP GET won’t work.
Thus, I assume you have such a RESTful service running on your machine on X.X.X.X:5001?
If the host X.X.X.X is the same machine as StreamPipes is running the REST publisher running inside Docker should not have a problem connecting to it in order to POST data to that endpoint.
If it’s another host you need to make that the network is setup right and that the other host is reachable and the port 5001 is accessible.
Whats your system? Linux/Windows/Mac?
Cheers
Patrick
> Am 21.07.2021 um 12:05 schrieb Canteri, Michele <Mi...@gft.com>:
>
> Hi all,
>
> I am trying to expose data via the Rest publisher sink but I don’t get how it works, especially if and how it is possible to expose the data on a public address.
> My configuration of the Rest publisher is currently:
>
> http://X.X.X.X:5001/My-Test <http://x.x.x.x:5001/My-Test>
>
> where X.X.X.X is the public IP and I can access the data exposed only on the same server. I’ve correctly configured the firewalls to open the port 5001 for both TCP and UDP connections.
> I tried using netcat in this way:
>
> nc -l -k X.X.X.X 5001
>
> and I have the POST data but I think only because it is routed on the 0.0.0.0 address and I verified it having the same result using
>
> nc -l -k 0.0.0.0 5001
>
> In both cases I have this result:
>
> POST /My-Test HTTP/1.1
> Content-Length: 211
> Content-Type: application/json; charset=UTF-8
> Host: X.X.X.X:5001
> Connection: Keep-Alive
> User-Agent: Apache-HttpClient/4.5.13 (Java/1.8.0_292)
> Accept-Encoding: gzip,deflate
>
> {"mass_flow":9.2311,"underflow":false,"overflow":false,"level":70.0957,"sensorId_0":"pressure01","temperature":43.8133,"sensorId_1":"level01","pressure":63.8947,"timestamp":1626860022495,"sensorId":"flowrate01"}
>
> That is the correct data from the data simulators.
>
> Any help in understanding what I am doing wrong is appreciated and let me know if some logs can be of help.
>
> Thank you in advance,
> Michele
> Michele Canteri
> _________________________________________________________________
>
> GFT Italia S.r.l.
> VIA SILE 18 / floor 4
> 20139 Milano, Italy
>
> T +39 02 58158503
> F +39 02 58158621
> Michele.Canteri@gft.com <ma...@gft.com>
> www.gft.com/it <http://www.gft.com/it>
> https://blog.gft.com <https://blog.gft.com/>
> https://www.linkedin.com/company/gft-group/ <https://www.linkedin.com/company/gft-group/>
> www.twitter.com/gft_it <http://www.twitter.com/gft_it>
> <ActoForTheFuture_GFT@1,2x_68f73108-d76e-4497-aa5c-ff8a5693ce57.png> <https://www.gft.com/it/it/index/societa/stampa/press-releases/2021/sostenibilita-e-innovazione-sociale--gft-lancia-il-greencoding-e-in-italia-act-for-the-future/>
>
> Registered office: GFT Italia S.r.l., Via Sile 18, 20139 Milan (Italy)
> VAT Code and number of registration with the Companies' Register of Milan: 00819200478
> Corporate capital: Euro 1,000,000 fully paid-in
>
> Company subject to the direction and coordination activity of GFT Technologies SE
>
> Ai sensi del Regolamento UE 2016/679 (GDPR) e delle Linee Guida per posta elettronica ed Internet del Garante Privacy si rende noto che le informazioni contenute nella presente comunicazione e i relativi allegati possono essere riservate e sono, comunque, destinate esclusivamente alle persone o alla Società sopraindicati. Le stesse non sono da considerarsi comunicazioni personali, quindi eventuali risposte potranno essere conosciute da persone appartenenti all’azienda.
>
> In accordance with Regulation EU 2016/679 (General Data Protection Regulation) and Guidelines Applying to the Use of E-Mails and the Internet in the Employment Context of Italian Data Protection Authority it is announced that the information in this communication (which includes any documents with it) is confidential and meant only for above-mentioned persons and/or Companies; the information is not personal therefore, It is intended for the addressee only and any replies will be known by people belonging to the company.
> _________________________________________________________________
>