You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@streampipes.apache.org by "Rullo, Federico" <Fe...@gft.com> on 2022/11/21 16:44:41 UTC
Cannot create new datastream from JSON file
Hello,
The critical issue we are having is that when we create a new datastream from a json file, the data cannot either be previewed or they do, depending on the value of the replay speed, for example, if we put a default 1 we don't see any data previewing when the adapter is created but if we put 50 we do get the preview.
In both cases, no output is produced from the created adapters, and no errors is displayed in the logs.
The steps we are taking to create the adapters are:
1. Upload the JSON file
2. Input 1 as replay speed
3. Select JSON and JSONArray as file format
4. Selecting the data under the "data" filed
5. We already have a date_time_unix timestamp field, so we select that as the timestamp with no conversion
6. Name the adapter and start it
After this we either get no preview of the data or, if we put 50 as replay speed, get a preview of the data.
Finally, if we create a new pipeline using the newly created adapter, no data is passed.
Hence, we see no data in the data explorer and inside the influxdb itself.
We had a similar issue with the preview's version, do you know if there is anything we are doing wrong or if there is some problem handling jsons?
The version we are currently using is the 0.70.0 set on the pipeline-element environment.
Thanks in advance,
Best Regards,
Federico Rullo
Federico Rullo
_________________________________________________________________
GFT Italia S.r.l.
Via A. Rimassa, 51/2
16129 Genova, Italy
T +39 025 8158169
Federico.Rullo@gft.com
www.gft.com/it<http://www.gft.com/it>
https://blog.gft.com
https://www.linkedin.com/company/gft-group/
www.twitter.com/gft_it <http://www.twitter.com/gft_it>
Registered office: GFT Italia S.r.l., Via Sile 18, 20139 Milan (Italy)
VAT Code and number of registration with the Companies' Register of Milan: 00819200478
Corporate capital: Euro 1,000,000 fully paid-in
Company subject to the direction and coordination activity of GFT Technologies SE
Ai sensi del Regolamento UE 2016/679 (GDPR) e delle Linee Guida per posta elettronica ed Internet del Garante Privacy si rende noto che le informazioni contenute nella presente comunicazione e i relativi allegati possono essere riservate e sono, comunque, destinate esclusivamente alle persone o alla Societ? sopraindicati. Le stesse non sono da considerarsi comunicazioni personali, quindi eventuali risposte potranno essere conosciute da persone appartenenti all'azienda.
In accordance with Regulation EU 2016/679 (General Data Protection Regulation) and Guidelines Applying to the Use of E-Mails and the Internet in the Employment Context of Italian Data Protection Authority it is announced that the information in this communication (which includes any documents with it) is confidential and meant only for above-mentioned persons and/or Companies; the information is not personal therefore, It is intended for the addressee only and any replies will be known by people belonging to the company.
_________________________________________________________________
[cid:firmaGPTWdefinitivo_da82df40-8928-4ee9-afd7-5c490fe3d6f0.png]<https://www.gft.com/it/it/solutions/SOLUTIONS/it/IT-Metaverso>
AW: Cannot create new datastream from JSON file
Posted by Philipp Zehnder <ph...@bytefabrik.ai>.
Hi Federico,
in general, StreamPipes expects that the event schema is fix.
So, the data type should not change. However, sometime this can happen. I would say in such a case either the field or the event should be ignored by the adapter.
But the adapter should not fail if there is a problem in the data. If that is the case, please open an issue with a description and ideally a small minimal sample that reproduces the problem.
We now use GitHub Issues [1] for issue tracking and GitHub Discussions [2] to discuss new functionalities or problems.
You are welcome to enter your issues there.
Cheers,
Philipp
[1] https://github.com/apache/streampipes/issues
[2] https://github.com/apache/streampipes/discussions
Von: Rullo, Federico <Fe...@gft.com>
Datum: Montag, 28. November 2022 um 10:39
An: users@streampipes.apache.org <us...@streampipes.apache.org>
Betreff: RE: Cannot create new datastream from JSON file
Hi Philipp,
Thanks for the reply.
Yes, what you have done is correct.
We are using the DataStream component, not the dataset since we need to simulate real-time data.
And when we are creating the data stream, the data is not being written onto the data lake, even if there are no errors in the logs.
We have seen that this might be an issue with some fields that contain strings instead of numbers, and when they are missing data, the entire stream won’t work.
This should not be an issue since it should just output an empty string if I am not mistaken.
Let me know what you think,
Best,
Federico.
Federico Rullo
_________________________________________________________________
GFT Italia S.r.l.
Via A. Rimassa, 51/2
16129 Genova, Italy
T +39 025 8158169
Federico.Rullo@gft.com
www.gft.com/it<http://www.gft.com/it>
https://blog.gft.com
https://www.linkedin.com/company/gft-group/
www.twitter.com/gft_it <http://www.twitter.com/gft_it%20>
Registered office: GFT Italia S.r.l., Via Sile 18, 20139 Milan (Italy)
VAT Code and number of registration with the Companies' Register of Milan: 00819200478
Corporate capital: Euro 1,000,000 fully paid-in
Company subject to the direction and coordination activity of GFT Technologies SE
Ai sensi del Regolamento UE 2016/679 (GDPR) e delle Linee Guida per posta elettronica ed Internet del Garante Privacy si rende noto che le informazioni contenute nella presente comunicazione e i relativi allegati possono essere riservate e sono, comunque, destinate esclusivamente alle persone o alla Società sopraindicati. Le stesse non sono da considerarsi comunicazioni personali, quindi eventuali risposte potranno essere conosciute da persone appartenenti all’azienda.
In accordance with Regulation EU 2016/679 (General Data Protection Regulation) and Guidelines Applying to the Use of E-Mails and the Internet in the Employment Context of Italian Data Protection Authority it is announced that the information in this communication (which includes any documents with it) is confidential and meant only for above-mentioned persons and/or Companies; the information is not personal therefore, It is intended for the addressee only and any replies will be known by people belonging to the company.
_________________________________________________________________
[cid:image001.png@01D90315.A93E01B0]<https://www.gft.com/it/it/solutions/SOLUTIONS/it/IT-Metaverso>
From: Philipp Zehnder <ph...@bytefabrik.ai>
Sent: giovedì 24 novembre 2022 16:57
To: users@streampipes.apache.org
Subject: AW: Cannot create new datastream from JSON file
[EXTERNAL EMAIL]
This email originated from outside of the organization or was sent unauthenticated. Do not click links or open attachments unless you confirm the sender and know the content is safe.
Hi Federico,
thanks a lot for providing the sample file.
I loaded it into a local StreamPipes instance and used the file adapters to read it.
As the format I had to choose “Array Field” with the “Key: data”, because the data is in the array with the key “data”, correct?
I selected the field “date_unix_ts” for the timestamp and I was able to write the data into the data lake and visualize it in a data view.
For that I used the “File Set” adapter (this adapter replays the file only once). This is a good way to load data into the data lake.
If you want to work with the data in a pipeline, I would recommend to use the “File Stream” adapter. This will play back the original data in an infinite loop, but you must remember that the original timestamps will also be used as the original replay speed.
I hope this helps. Please let us know if you have any other problems.
Cheers,
Philipp
Von: Rullo, Federico <Fe...@gft.com>>
Datum: Donnerstag, 24. November 2022 um 10:09
An: users@streampipes.apache.org<ma...@streampipes.apache.org> <us...@streampipes.apache.org>>
Betreff: RE: Cannot create new datastream from JSON file
Hi Philipp,
Here is a file sample.
F.
Federico Rullo
_________________________________________________________________
GFT Italia S.r.l.
Via A. Rimassa, 51/2
16129 Genova, Italy
T +39 025 8158169
Federico.Rullo@gft.com<ma...@gft.com>
www.gft.com/it<https://eur02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.gft.com%2Fit&data=05%7C01%7CFederico.Rullo%40gft.com%7Cfcf2fe127c44413a112608dace349f21%7C55a2bc67aec14ad29a9c5b2457b91dcd%7C0%7C0%7C638049022659960608%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=oI9A0vh92%2BwiOEhFce%2FwSGbt4Zgp2vuiQqELcydTHQo%3D&reserved=0>
https://blog.gft.com<https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fblog.gft.com%2F&data=05%7C01%7CFederico.Rullo%40gft.com%7Cfcf2fe127c44413a112608dace349f21%7C55a2bc67aec14ad29a9c5b2457b91dcd%7C0%7C0%7C638049022659960608%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=zUt5DZimM7n%2BDnj0mgXHzgxiFGV2KHYqURnqHQDYWpk%3D&reserved=0>
https://www.linkedin.com/company/gft-group/<https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.linkedin.com%2Fcompany%2Fgft-group%2F&data=05%7C01%7CFederico.Rullo%40gft.com%7Cfcf2fe127c44413a112608dace349f21%7C55a2bc67aec14ad29a9c5b2457b91dcd%7C0%7C0%7C638049022659960608%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=chF9FUlklL%2Fegh42XMimJ19hDUJloSO4r8ZGqvAaYxw%3D&reserved=0>
www.twitter.com/gft_it <https://eur02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.twitter.com%2Fgft_it%2520&data=05%7C01%7CFederico.Rullo%40gft.com%7Cfcf2fe127c44413a112608dace349f21%7C55a2bc67aec14ad29a9c5b2457b91dcd%7C0%7C0%7C638049022659960608%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=c9cTdvlxq0r9Nff%2F4iGeBAfaK6DSL%2FoO5d4B9SPiKvc%3D&reserved=0>
Registered office: GFT Italia S.r.l., Via Sile 18, 20139 Milan (Italy)
VAT Code and number of registration with the Companies' Register of Milan: 00819200478
Corporate capital: Euro 1,000,000 fully paid-in
Company subject to the direction and coordination activity of GFT Technologies SE
Ai sensi del Regolamento UE 2016/679 (GDPR) e delle Linee Guida per posta elettronica ed Internet del Garante Privacy si rende noto che le informazioni contenute nella presente comunicazione e i relativi allegati possono essere riservate e sono, comunque, destinate esclusivamente alle persone o alla Società sopraindicati. Le stesse non sono da considerarsi comunicazioni personali, quindi eventuali risposte potranno essere conosciute da persone appartenenti all’azienda.
In accordance with Regulation EU 2016/679 (General Data Protection Regulation) and Guidelines Applying to the Use of E-Mails and the Internet in the Employment Context of Italian Data Protection Authority it is announced that the information in this communication (which includes any documents with it) is confidential and meant only for above-mentioned persons and/or Companies; the information is not personal therefore, It is intended for the addressee only and any replies will be known by people belonging to the company.
_________________________________________________________________
[cid:image001.png@01D90315.A93E01B0]<https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.gft.com%2Fit%2Fit%2Fsolutions%2FSOLUTIONS%2Fit%2FIT-Metaverso&data=05%7C01%7CFederico.Rullo%40gft.com%7Cfcf2fe127c44413a112608dace349f21%7C55a2bc67aec14ad29a9c5b2457b91dcd%7C0%7C0%7C638049022659960608%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=dPuu3G14B0dgack%2FkCCXV3gXB7fj1RCBpU2AeSscZMw%3D&reserved=0>
From: Philipp Zehnder <ph...@bytefabrik.ai>>
Sent: mercoledì 23 novembre 2022 17:58
To: users@streampipes.apache.org<ma...@streampipes.apache.org>
Subject: AW: Cannot create new datastream from JSON file
[EXTERNAL EMAIL]
This email originated from outside of the organization or was sent unauthenticated. Do not click links or open attachments unless you confirm the sender and know the content is safe.
Hi Federico,
can you provide a minimal sample json that is not working?
Then I will test it locally to see what is going wrong.
It is strange that the preview feature works with a different replay speed.
The replay speed configuration should only be used once the adapter is started to reduce the time between the events.
Cheers,
Philipp
Von: Rullo, Federico <Fe...@gft.com>>
Datum: Montag, 21. November 2022 um 17:54
An: users@streampipes.apache.org<ma...@streampipes.apache.org> <us...@streampipes.apache.org>>
Betreff: Cannot create new datastream from JSON file
Hello,
The critical issue we are having is that when we create a new datastream from a json file, the data cannot either be previewed or they do, depending on the value of the replay speed, for example, if we put a default 1 we don’t see any data previewing when the adapter is created but if we put 50 we do get the preview.
In both cases, no output is produced from the created adapters, and no errors is displayed in the logs.
The steps we are taking to create the adapters are:
1. Upload the JSON file
2. Input 1 as replay speed
3. Select JSON and JSONArray as file format
4. Selecting the data under the “data” filed
5. We already have a date_time_unix timestamp field, so we select that as the timestamp with no conversion
6. Name the adapter and start it
After this we either get no preview of the data or, if we put 50 as replay speed, get a preview of the data.
Finally, if we create a new pipeline using the newly created adapter, no data is passed.
Hence, we see no data in the data explorer and inside the influxdb itself.
We had a similar issue with the preview’s version, do you know if there is anything we are doing wrong or if there is some problem handling jsons?
The version we are currently using is the 0.70.0 set on the pipeline-element environment.
Thanks in advance,
Best Regards,
Federico Rullo
Federico Rullo
_________________________________________________________________
GFT Italia S.r.l.
Via A. Rimassa, 51/2
16129 Genova, Italy
T +39 025 8158169
Federico.Rullo@gft.com<ma...@gft.com>
www.gft.com/it<https://eur02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.gft.com%2Fit&data=05%7C01%7CFederico.Rullo%40gft.com%7Cfcf2fe127c44413a112608dace349f21%7C55a2bc67aec14ad29a9c5b2457b91dcd%7C0%7C0%7C638049022659960608%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=oI9A0vh92%2BwiOEhFce%2FwSGbt4Zgp2vuiQqELcydTHQo%3D&reserved=0>
https://blog.gft.com<https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fblog.gft.com%2F&data=05%7C01%7CFederico.Rullo%40gft.com%7Cfcf2fe127c44413a112608dace349f21%7C55a2bc67aec14ad29a9c5b2457b91dcd%7C0%7C0%7C638049022659960608%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=zUt5DZimM7n%2BDnj0mgXHzgxiFGV2KHYqURnqHQDYWpk%3D&reserved=0>
https://www.linkedin.com/company/gft-group/<https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.linkedin.com%2Fcompany%2Fgft-group%2F&data=05%7C01%7CFederico.Rullo%40gft.com%7Cfcf2fe127c44413a112608dace349f21%7C55a2bc67aec14ad29a9c5b2457b91dcd%7C0%7C0%7C638049022659960608%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=chF9FUlklL%2Fegh42XMimJ19hDUJloSO4r8ZGqvAaYxw%3D&reserved=0>
www.twitter.com/gft_it <https://eur02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.twitter.com%2Fgft_it%2520&data=05%7C01%7CFederico.Rullo%40gft.com%7Cfcf2fe127c44413a112608dace349f21%7C55a2bc67aec14ad29a9c5b2457b91dcd%7C0%7C0%7C638049022660116810%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=g3YzYsKYFB1uAlGcNjXyUtoJ%2B2JEm5Z%2F3vEofNlk3P0%3D&reserved=0>
Registered office: GFT Italia S.r.l., Via Sile 18, 20139 Milan (Italy)
VAT Code and number of registration with the Companies' Register of Milan: 00819200478
Corporate capital: Euro 1,000,000 fully paid-in
Company subject to the direction and coordination activity of GFT Technologies SE
Ai sensi del Regolamento UE 2016/679 (GDPR) e delle Linee Guida per posta elettronica ed Internet del Garante Privacy si rende noto che le informazioni contenute nella presente comunicazione e i relativi allegati possono essere riservate e sono, comunque, destinate esclusivamente alle persone o alla Società sopraindicati. Le stesse non sono da considerarsi comunicazioni personali, quindi eventuali risposte potranno essere conosciute da persone appartenenti all’azienda.
In accordance with Regulation EU 2016/679 (General Data Protection Regulation) and Guidelines Applying to the Use of E-Mails and the Internet in the Employment Context of Italian Data Protection Authority it is announced that the information in this communication (which includes any documents with it) is confidential and meant only for above-mentioned persons and/or Companies; the information is not personal therefore, It is intended for the addressee only and any replies will be known by people belonging to the company.
_________________________________________________________________
[cid:image001.png@01D90315.A93E01B0]<https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.gft.com%2Fit%2Fit%2Fsolutions%2FSOLUTIONS%2Fit%2FIT-Metaverso&data=05%7C01%7CFederico.Rullo%40gft.com%7Cfcf2fe127c44413a112608dace349f21%7C55a2bc67aec14ad29a9c5b2457b91dcd%7C0%7C0%7C638049022660116810%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=wKPTAwEI77zpNFcrAp8eRXa6FqCwN8YKs5chGYIlkzs%3D&reserved=0>
RE: Cannot create new datastream from JSON file
Posted by "Rullo, Federico" <Fe...@gft.com>.
Hi Philipp,
Thanks for the reply.
Yes, what you have done is correct.
We are using the DataStream component, not the dataset since we need to simulate real-time data.
And when we are creating the data stream, the data is not being written onto the data lake, even if there are no errors in the logs.
We have seen that this might be an issue with some fields that contain strings instead of numbers, and when they are missing data, the entire stream won't work.
This should not be an issue since it should just output an empty string if I am not mistaken.
Let me know what you think,
Best,
Federico.
Federico Rullo
_________________________________________________________________
GFT Italia S.r.l.
Via A. Rimassa, 51/2
16129 Genova, Italy
T +39 025 8158169
Federico.Rullo@gft.com
www.gft.com/it<http://www.gft.com/it>
https://blog.gft.com
https://www.linkedin.com/company/gft-group/
www.twitter.com/gft_it <http://www.twitter.com/gft_it>
Registered office: GFT Italia S.r.l., Via Sile 18, 20139 Milan (Italy)
VAT Code and number of registration with the Companies' Register of Milan: 00819200478
Corporate capital: Euro 1,000,000 fully paid-in
Company subject to the direction and coordination activity of GFT Technologies SE
Ai sensi del Regolamento UE 2016/679 (GDPR) e delle Linee Guida per posta elettronica ed Internet del Garante Privacy si rende noto che le informazioni contenute nella presente comunicazione e i relativi allegati possono essere riservate e sono, comunque, destinate esclusivamente alle persone o alla Società sopraindicati. Le stesse non sono da considerarsi comunicazioni personali, quindi eventuali risposte potranno essere conosciute da persone appartenenti all'azienda.
In accordance with Regulation EU 2016/679 (General Data Protection Regulation) and Guidelines Applying to the Use of E-Mails and the Internet in the Employment Context of Italian Data Protection Authority it is announced that the information in this communication (which includes any documents with it) is confidential and meant only for above-mentioned persons and/or Companies; the information is not personal therefore, It is intended for the addressee only and any replies will be known by people belonging to the company.
_________________________________________________________________
[cid:firmaGPTWdefinitivo_da82df40-8928-4ee9-afd7-5c490fe3d6f0.png]<https://www.gft.com/it/it/solutions/SOLUTIONS/it/IT-Metaverso>
From: Philipp Zehnder <ph...@bytefabrik.ai>
Sent: giovedì 24 novembre 2022 16:57
To: users@streampipes.apache.org
Subject: AW: Cannot create new datastream from JSON file
[EXTERNAL EMAIL]
This email originated from outside of the organization or was sent unauthenticated. Do not click links or open attachments unless you confirm the sender and know the content is safe.
Hi Federico,
thanks a lot for providing the sample file.
I loaded it into a local StreamPipes instance and used the file adapters to read it.
As the format I had to choose "Array Field" with the "Key: data", because the data is in the array with the key "data", correct?
I selected the field "date_unix_ts" for the timestamp and I was able to write the data into the data lake and visualize it in a data view.
For that I used the "File Set" adapter (this adapter replays the file only once). This is a good way to load data into the data lake.
If you want to work with the data in a pipeline, I would recommend to use the "File Stream" adapter. This will play back the original data in an infinite loop, but you must remember that the original timestamps will also be used as the original replay speed.
I hope this helps. Please let us know if you have any other problems.
Cheers,
Philipp
Von: Rullo, Federico <Fe...@gft.com>>
Datum: Donnerstag, 24. November 2022 um 10:09
An: users@streampipes.apache.org<ma...@streampipes.apache.org> <us...@streampipes.apache.org>>
Betreff: RE: Cannot create new datastream from JSON file
Hi Philipp,
Here is a file sample.
F.
Federico Rullo
_________________________________________________________________
GFT Italia S.r.l.
Via A. Rimassa, 51/2
16129 Genova, Italy
T +39 025 8158169
Federico.Rullo@gft.com<ma...@gft.com>
www.gft.com/it<https://eur02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.gft.com%2Fit&data=05%7C01%7CFederico.Rullo%40gft.com%7Cfcf2fe127c44413a112608dace349f21%7C55a2bc67aec14ad29a9c5b2457b91dcd%7C0%7C0%7C638049022659960608%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=oI9A0vh92%2BwiOEhFce%2FwSGbt4Zgp2vuiQqELcydTHQo%3D&reserved=0>
https://blog.gft.com<https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fblog.gft.com%2F&data=05%7C01%7CFederico.Rullo%40gft.com%7Cfcf2fe127c44413a112608dace349f21%7C55a2bc67aec14ad29a9c5b2457b91dcd%7C0%7C0%7C638049022659960608%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=zUt5DZimM7n%2BDnj0mgXHzgxiFGV2KHYqURnqHQDYWpk%3D&reserved=0>
https://www.linkedin.com/company/gft-group/<https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.linkedin.com%2Fcompany%2Fgft-group%2F&data=05%7C01%7CFederico.Rullo%40gft.com%7Cfcf2fe127c44413a112608dace349f21%7C55a2bc67aec14ad29a9c5b2457b91dcd%7C0%7C0%7C638049022659960608%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=chF9FUlklL%2Fegh42XMimJ19hDUJloSO4r8ZGqvAaYxw%3D&reserved=0>
www.twitter.com/gft_it <https://eur02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.twitter.com%2Fgft_it%2520&data=05%7C01%7CFederico.Rullo%40gft.com%7Cfcf2fe127c44413a112608dace349f21%7C55a2bc67aec14ad29a9c5b2457b91dcd%7C0%7C0%7C638049022659960608%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=c9cTdvlxq0r9Nff%2F4iGeBAfaK6DSL%2FoO5d4B9SPiKvc%3D&reserved=0>
Registered office: GFT Italia S.r.l., Via Sile 18, 20139 Milan (Italy)
VAT Code and number of registration with the Companies' Register of Milan: 00819200478
Corporate capital: Euro 1,000,000 fully paid-in
Company subject to the direction and coordination activity of GFT Technologies SE
Ai sensi del Regolamento UE 2016/679 (GDPR) e delle Linee Guida per posta elettronica ed Internet del Garante Privacy si rende noto che le informazioni contenute nella presente comunicazione e i relativi allegati possono essere riservate e sono, comunque, destinate esclusivamente alle persone o alla Società sopraindicati. Le stesse non sono da considerarsi comunicazioni personali, quindi eventuali risposte potranno essere conosciute da persone appartenenti all'azienda.
In accordance with Regulation EU 2016/679 (General Data Protection Regulation) and Guidelines Applying to the Use of E-Mails and the Internet in the Employment Context of Italian Data Protection Authority it is announced that the information in this communication (which includes any documents with it) is confidential and meant only for above-mentioned persons and/or Companies; the information is not personal therefore, It is intended for the addressee only and any replies will be known by people belonging to the company.
_________________________________________________________________
[cid:image001.png@01D90315.A93E01B0]<https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.gft.com%2Fit%2Fit%2Fsolutions%2FSOLUTIONS%2Fit%2FIT-Metaverso&data=05%7C01%7CFederico.Rullo%40gft.com%7Cfcf2fe127c44413a112608dace349f21%7C55a2bc67aec14ad29a9c5b2457b91dcd%7C0%7C0%7C638049022659960608%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=dPuu3G14B0dgack%2FkCCXV3gXB7fj1RCBpU2AeSscZMw%3D&reserved=0>
From: Philipp Zehnder <ph...@bytefabrik.ai>>
Sent: mercoledì 23 novembre 2022 17:58
To: users@streampipes.apache.org<ma...@streampipes.apache.org>
Subject: AW: Cannot create new datastream from JSON file
[EXTERNAL EMAIL]
This email originated from outside of the organization or was sent unauthenticated. Do not click links or open attachments unless you confirm the sender and know the content is safe.
Hi Federico,
can you provide a minimal sample json that is not working?
Then I will test it locally to see what is going wrong.
It is strange that the preview feature works with a different replay speed.
The replay speed configuration should only be used once the adapter is started to reduce the time between the events.
Cheers,
Philipp
Von: Rullo, Federico <Fe...@gft.com>>
Datum: Montag, 21. November 2022 um 17:54
An: users@streampipes.apache.org<ma...@streampipes.apache.org> <us...@streampipes.apache.org>>
Betreff: Cannot create new datastream from JSON file
Hello,
The critical issue we are having is that when we create a new datastream from a json file, the data cannot either be previewed or they do, depending on the value of the replay speed, for example, if we put a default 1 we don't see any data previewing when the adapter is created but if we put 50 we do get the preview.
In both cases, no output is produced from the created adapters, and no errors is displayed in the logs.
The steps we are taking to create the adapters are:
1. Upload the JSON file
2. Input 1 as replay speed
3. Select JSON and JSONArray as file format
4. Selecting the data under the "data" filed
5. We already have a date_time_unix timestamp field, so we select that as the timestamp with no conversion
6. Name the adapter and start it
After this we either get no preview of the data or, if we put 50 as replay speed, get a preview of the data.
Finally, if we create a new pipeline using the newly created adapter, no data is passed.
Hence, we see no data in the data explorer and inside the influxdb itself.
We had a similar issue with the preview's version, do you know if there is anything we are doing wrong or if there is some problem handling jsons?
The version we are currently using is the 0.70.0 set on the pipeline-element environment.
Thanks in advance,
Best Regards,
Federico Rullo
Federico Rullo
_________________________________________________________________
GFT Italia S.r.l.
Via A. Rimassa, 51/2
16129 Genova, Italy
T +39 025 8158169
Federico.Rullo@gft.com<ma...@gft.com>
www.gft.com/it<https://eur02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.gft.com%2Fit&data=05%7C01%7CFederico.Rullo%40gft.com%7Cfcf2fe127c44413a112608dace349f21%7C55a2bc67aec14ad29a9c5b2457b91dcd%7C0%7C0%7C638049022659960608%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=oI9A0vh92%2BwiOEhFce%2FwSGbt4Zgp2vuiQqELcydTHQo%3D&reserved=0>
https://blog.gft.com<https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fblog.gft.com%2F&data=05%7C01%7CFederico.Rullo%40gft.com%7Cfcf2fe127c44413a112608dace349f21%7C55a2bc67aec14ad29a9c5b2457b91dcd%7C0%7C0%7C638049022659960608%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=zUt5DZimM7n%2BDnj0mgXHzgxiFGV2KHYqURnqHQDYWpk%3D&reserved=0>
https://www.linkedin.com/company/gft-group/<https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.linkedin.com%2Fcompany%2Fgft-group%2F&data=05%7C01%7CFederico.Rullo%40gft.com%7Cfcf2fe127c44413a112608dace349f21%7C55a2bc67aec14ad29a9c5b2457b91dcd%7C0%7C0%7C638049022659960608%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=chF9FUlklL%2Fegh42XMimJ19hDUJloSO4r8ZGqvAaYxw%3D&reserved=0>
www.twitter.com/gft_it <https://eur02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.twitter.com%2Fgft_it%2520&data=05%7C01%7CFederico.Rullo%40gft.com%7Cfcf2fe127c44413a112608dace349f21%7C55a2bc67aec14ad29a9c5b2457b91dcd%7C0%7C0%7C638049022660116810%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=g3YzYsKYFB1uAlGcNjXyUtoJ%2B2JEm5Z%2F3vEofNlk3P0%3D&reserved=0>
Registered office: GFT Italia S.r.l., Via Sile 18, 20139 Milan (Italy)
VAT Code and number of registration with the Companies' Register of Milan: 00819200478
Corporate capital: Euro 1,000,000 fully paid-in
Company subject to the direction and coordination activity of GFT Technologies SE
Ai sensi del Regolamento UE 2016/679 (GDPR) e delle Linee Guida per posta elettronica ed Internet del Garante Privacy si rende noto che le informazioni contenute nella presente comunicazione e i relativi allegati possono essere riservate e sono, comunque, destinate esclusivamente alle persone o alla Società sopraindicati. Le stesse non sono da considerarsi comunicazioni personali, quindi eventuali risposte potranno essere conosciute da persone appartenenti all'azienda.
In accordance with Regulation EU 2016/679 (General Data Protection Regulation) and Guidelines Applying to the Use of E-Mails and the Internet in the Employment Context of Italian Data Protection Authority it is announced that the information in this communication (which includes any documents with it) is confidential and meant only for above-mentioned persons and/or Companies; the information is not personal therefore, It is intended for the addressee only and any replies will be known by people belonging to the company.
_________________________________________________________________
[cid:image001.png@01D90315.A93E01B0]<https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.gft.com%2Fit%2Fit%2Fsolutions%2FSOLUTIONS%2Fit%2FIT-Metaverso&data=05%7C01%7CFederico.Rullo%40gft.com%7Cfcf2fe127c44413a112608dace349f21%7C55a2bc67aec14ad29a9c5b2457b91dcd%7C0%7C0%7C638049022660116810%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=wKPTAwEI77zpNFcrAp8eRXa6FqCwN8YKs5chGYIlkzs%3D&reserved=0>
AW: Cannot create new datastream from JSON file
Posted by Philipp Zehnder <ph...@bytefabrik.ai>.
Hi Federico,
thanks a lot for providing the sample file.
I loaded it into a local StreamPipes instance and used the file adapters to read it.
As the format I had to choose “Array Field” with the “Key: data”, because the data is in the array with the key “data”, correct?
I selected the field “date_unix_ts” for the timestamp and I was able to write the data into the data lake and visualize it in a data view.
For that I used the “File Set” adapter (this adapter replays the file only once). This is a good way to load data into the data lake.
If you want to work with the data in a pipeline, I would recommend to use the “File Stream” adapter. This will play back the original data in an infinite loop, but you must remember that the original timestamps will also be used as the original replay speed.
I hope this helps. Please let us know if you have any other problems.
Cheers,
Philipp
Von: Rullo, Federico <Fe...@gft.com>
Datum: Donnerstag, 24. November 2022 um 10:09
An: users@streampipes.apache.org <us...@streampipes.apache.org>
Betreff: RE: Cannot create new datastream from JSON file
Hi Philipp,
Here is a file sample.
F.
Federico Rullo
_________________________________________________________________
GFT Italia S.r.l.
Via A. Rimassa, 51/2
16129 Genova, Italy
T +39 025 8158169
Federico.Rullo@gft.com
www.gft.com/it<http://www.gft.com/it>
https://blog.gft.com
https://www.linkedin.com/company/gft-group/
www.twitter.com/gft_it <http://www.twitter.com/gft_it%20>
Registered office: GFT Italia S.r.l., Via Sile 18, 20139 Milan (Italy)
VAT Code and number of registration with the Companies' Register of Milan: 00819200478
Corporate capital: Euro 1,000,000 fully paid-in
Company subject to the direction and coordination activity of GFT Technologies SE
Ai sensi del Regolamento UE 2016/679 (GDPR) e delle Linee Guida per posta elettronica ed Internet del Garante Privacy si rende noto che le informazioni contenute nella presente comunicazione e i relativi allegati possono essere riservate e sono, comunque, destinate esclusivamente alle persone o alla Società sopraindicati. Le stesse non sono da considerarsi comunicazioni personali, quindi eventuali risposte potranno essere conosciute da persone appartenenti all’azienda.
In accordance with Regulation EU 2016/679 (General Data Protection Regulation) and Guidelines Applying to the Use of E-Mails and the Internet in the Employment Context of Italian Data Protection Authority it is announced that the information in this communication (which includes any documents with it) is confidential and meant only for above-mentioned persons and/or Companies; the information is not personal therefore, It is intended for the addressee only and any replies will be known by people belonging to the company.
_________________________________________________________________
[cid:image001.png@01D8FFEC.90150110]<https://www.gft.com/it/it/solutions/SOLUTIONS/it/IT-Metaverso>
From: Philipp Zehnder <ph...@bytefabrik.ai>
Sent: mercoledì 23 novembre 2022 17:58
To: users@streampipes.apache.org
Subject: AW: Cannot create new datastream from JSON file
[EXTERNAL EMAIL]
This email originated from outside of the organization or was sent unauthenticated. Do not click links or open attachments unless you confirm the sender and know the content is safe.
Hi Federico,
can you provide a minimal sample json that is not working?
Then I will test it locally to see what is going wrong.
It is strange that the preview feature works with a different replay speed.
The replay speed configuration should only be used once the adapter is started to reduce the time between the events.
Cheers,
Philipp
Von: Rullo, Federico <Fe...@gft.com>>
Datum: Montag, 21. November 2022 um 17:54
An: users@streampipes.apache.org<ma...@streampipes.apache.org> <us...@streampipes.apache.org>>
Betreff: Cannot create new datastream from JSON file
Hello,
The critical issue we are having is that when we create a new datastream from a json file, the data cannot either be previewed or they do, depending on the value of the replay speed, for example, if we put a default 1 we don’t see any data previewing when the adapter is created but if we put 50 we do get the preview.
In both cases, no output is produced from the created adapters, and no errors is displayed in the logs.
The steps we are taking to create the adapters are:
1. Upload the JSON file
2. Input 1 as replay speed
3. Select JSON and JSONArray as file format
4. Selecting the data under the “data” filed
5. We already have a date_time_unix timestamp field, so we select that as the timestamp with no conversion
6. Name the adapter and start it
After this we either get no preview of the data or, if we put 50 as replay speed, get a preview of the data.
Finally, if we create a new pipeline using the newly created adapter, no data is passed.
Hence, we see no data in the data explorer and inside the influxdb itself.
We had a similar issue with the preview’s version, do you know if there is anything we are doing wrong or if there is some problem handling jsons?
The version we are currently using is the 0.70.0 set on the pipeline-element environment.
Thanks in advance,
Best Regards,
Federico Rullo
Federico Rullo
_________________________________________________________________
GFT Italia S.r.l.
Via A. Rimassa, 51/2
16129 Genova, Italy
T +39 025 8158169
Federico.Rullo@gft.com<ma...@gft.com>
www.gft.com/it<https://eur02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.gft.com%2Fit&data=05%7C01%7CFederico.Rullo%40gft.com%7C9d630320320c4df6e9e108dacd73f4db%7C55a2bc67aec14ad29a9c5b2457b91dcd%7C0%7C0%7C638048195183279684%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=wTGCGWNuNVE7pto%2BS6jcmBCZHeHVpppFl0TW2mo3%2Bqs%3D&reserved=0>
https://blog.gft.com<https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fblog.gft.com%2F&data=05%7C01%7CFederico.Rullo%40gft.com%7C9d630320320c4df6e9e108dacd73f4db%7C55a2bc67aec14ad29a9c5b2457b91dcd%7C0%7C0%7C638048195183435892%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=mt6m0TtoeT5bSmK4QzEDHgYSbyAm6s%2F1UtFVCSEXnC4%3D&reserved=0>
https://www.linkedin.com/company/gft-group/<https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.linkedin.com%2Fcompany%2Fgft-group%2F&data=05%7C01%7CFederico.Rullo%40gft.com%7C9d630320320c4df6e9e108dacd73f4db%7C55a2bc67aec14ad29a9c5b2457b91dcd%7C0%7C0%7C638048195183435892%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=U61BOSD8mVYtOq%2BzkOhgicfHtS2b9nnQxaQmecT7c%2Fs%3D&reserved=0>
www.twitter.com/gft_it <https://eur02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.twitter.com%2Fgft_it%2520&data=05%7C01%7CFederico.Rullo%40gft.com%7C9d630320320c4df6e9e108dacd73f4db%7C55a2bc67aec14ad29a9c5b2457b91dcd%7C0%7C0%7C638048195183435892%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=CPiRwQNwbTLZ4kvrEtZ0KA4iI7onqVr0sgTfQlH7is4%3D&reserved=0>
Registered office: GFT Italia S.r.l., Via Sile 18, 20139 Milan (Italy)
VAT Code and number of registration with the Companies' Register of Milan: 00819200478
Corporate capital: Euro 1,000,000 fully paid-in
Company subject to the direction and coordination activity of GFT Technologies SE
Ai sensi del Regolamento UE 2016/679 (GDPR) e delle Linee Guida per posta elettronica ed Internet del Garante Privacy si rende noto che le informazioni contenute nella presente comunicazione e i relativi allegati possono essere riservate e sono, comunque, destinate esclusivamente alle persone o alla Società sopraindicati. Le stesse non sono da considerarsi comunicazioni personali, quindi eventuali risposte potranno essere conosciute da persone appartenenti all’azienda.
In accordance with Regulation EU 2016/679 (General Data Protection Regulation) and Guidelines Applying to the Use of E-Mails and the Internet in the Employment Context of Italian Data Protection Authority it is announced that the information in this communication (which includes any documents with it) is confidential and meant only for above-mentioned persons and/or Companies; the information is not personal therefore, It is intended for the addressee only and any replies will be known by people belonging to the company.
_________________________________________________________________
[cid:image001.png@01D8FFEC.90150110]<https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.gft.com%2Fit%2Fit%2Fsolutions%2FSOLUTIONS%2Fit%2FIT-Metaverso&data=05%7C01%7CFederico.Rullo%40gft.com%7C9d630320320c4df6e9e108dacd73f4db%7C55a2bc67aec14ad29a9c5b2457b91dcd%7C0%7C0%7C638048195183435892%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=KZcQTHhxXJsNXEcQdATMHHp1UPwtwQ%2BMFif%2BeI7R3vo%3D&reserved=0>
RE: Cannot create new datastream from JSON file
Posted by "Rullo, Federico" <Fe...@gft.com>.
Hi Philipp,
Here is a file sample.
F.
Federico Rullo
_________________________________________________________________
GFT Italia S.r.l.
Via A. Rimassa, 51/2
16129 Genova, Italy
T +39 025 8158169
Federico.Rullo@gft.com
www.gft.com/it<http://www.gft.com/it>
https://blog.gft.com
https://www.linkedin.com/company/gft-group/
www.twitter.com/gft_it <http://www.twitter.com/gft_it>
Registered office: GFT Italia S.r.l., Via Sile 18, 20139 Milan (Italy)
VAT Code and number of registration with the Companies' Register of Milan: 00819200478
Corporate capital: Euro 1,000,000 fully paid-in
Company subject to the direction and coordination activity of GFT Technologies SE
Ai sensi del Regolamento UE 2016/679 (GDPR) e delle Linee Guida per posta elettronica ed Internet del Garante Privacy si rende noto che le informazioni contenute nella presente comunicazione e i relativi allegati possono essere riservate e sono, comunque, destinate esclusivamente alle persone o alla Società sopraindicati. Le stesse non sono da considerarsi comunicazioni personali, quindi eventuali risposte potranno essere conosciute da persone appartenenti all'azienda.
In accordance with Regulation EU 2016/679 (General Data Protection Regulation) and Guidelines Applying to the Use of E-Mails and the Internet in the Employment Context of Italian Data Protection Authority it is announced that the information in this communication (which includes any documents with it) is confidential and meant only for above-mentioned persons and/or Companies; the information is not personal therefore, It is intended for the addressee only and any replies will be known by people belonging to the company.
_________________________________________________________________
[cid:firmaGPTWdefinitivo_da82df40-8928-4ee9-afd7-5c490fe3d6f0.png]<https://www.gft.com/it/it/solutions/SOLUTIONS/it/IT-Metaverso>
From: Philipp Zehnder <ph...@bytefabrik.ai>
Sent: mercoledì 23 novembre 2022 17:58
To: users@streampipes.apache.org
Subject: AW: Cannot create new datastream from JSON file
[EXTERNAL EMAIL]
This email originated from outside of the organization or was sent unauthenticated. Do not click links or open attachments unless you confirm the sender and know the content is safe.
Hi Federico,
can you provide a minimal sample json that is not working?
Then I will test it locally to see what is going wrong.
It is strange that the preview feature works with a different replay speed.
The replay speed configuration should only be used once the adapter is started to reduce the time between the events.
Cheers,
Philipp
Von: Rullo, Federico <Fe...@gft.com>>
Datum: Montag, 21. November 2022 um 17:54
An: users@streampipes.apache.org<ma...@streampipes.apache.org> <us...@streampipes.apache.org>>
Betreff: Cannot create new datastream from JSON file
Hello,
The critical issue we are having is that when we create a new datastream from a json file, the data cannot either be previewed or they do, depending on the value of the replay speed, for example, if we put a default 1 we don't see any data previewing when the adapter is created but if we put 50 we do get the preview.
In both cases, no output is produced from the created adapters, and no errors is displayed in the logs.
The steps we are taking to create the adapters are:
1. Upload the JSON file
2. Input 1 as replay speed
3. Select JSON and JSONArray as file format
4. Selecting the data under the "data" filed
5. We already have a date_time_unix timestamp field, so we select that as the timestamp with no conversion
6. Name the adapter and start it
After this we either get no preview of the data or, if we put 50 as replay speed, get a preview of the data.
Finally, if we create a new pipeline using the newly created adapter, no data is passed.
Hence, we see no data in the data explorer and inside the influxdb itself.
We had a similar issue with the preview's version, do you know if there is anything we are doing wrong or if there is some problem handling jsons?
The version we are currently using is the 0.70.0 set on the pipeline-element environment.
Thanks in advance,
Best Regards,
Federico Rullo
Federico Rullo
_________________________________________________________________
GFT Italia S.r.l.
Via A. Rimassa, 51/2
16129 Genova, Italy
T +39 025 8158169
Federico.Rullo@gft.com<ma...@gft.com>
www.gft.com/it<https://eur02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.gft.com%2Fit&data=05%7C01%7CFederico.Rullo%40gft.com%7C9d630320320c4df6e9e108dacd73f4db%7C55a2bc67aec14ad29a9c5b2457b91dcd%7C0%7C0%7C638048195183279684%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=wTGCGWNuNVE7pto%2BS6jcmBCZHeHVpppFl0TW2mo3%2Bqs%3D&reserved=0>
https://blog.gft.com<https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fblog.gft.com%2F&data=05%7C01%7CFederico.Rullo%40gft.com%7C9d630320320c4df6e9e108dacd73f4db%7C55a2bc67aec14ad29a9c5b2457b91dcd%7C0%7C0%7C638048195183435892%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=mt6m0TtoeT5bSmK4QzEDHgYSbyAm6s%2F1UtFVCSEXnC4%3D&reserved=0>
https://www.linkedin.com/company/gft-group/<https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.linkedin.com%2Fcompany%2Fgft-group%2F&data=05%7C01%7CFederico.Rullo%40gft.com%7C9d630320320c4df6e9e108dacd73f4db%7C55a2bc67aec14ad29a9c5b2457b91dcd%7C0%7C0%7C638048195183435892%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=U61BOSD8mVYtOq%2BzkOhgicfHtS2b9nnQxaQmecT7c%2Fs%3D&reserved=0>
www.twitter.com/gft_it <https://eur02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.twitter.com%2Fgft_it%2520&data=05%7C01%7CFederico.Rullo%40gft.com%7C9d630320320c4df6e9e108dacd73f4db%7C55a2bc67aec14ad29a9c5b2457b91dcd%7C0%7C0%7C638048195183435892%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=CPiRwQNwbTLZ4kvrEtZ0KA4iI7onqVr0sgTfQlH7is4%3D&reserved=0>
Registered office: GFT Italia S.r.l., Via Sile 18, 20139 Milan (Italy)
VAT Code and number of registration with the Companies' Register of Milan: 00819200478
Corporate capital: Euro 1,000,000 fully paid-in
Company subject to the direction and coordination activity of GFT Technologies SE
Ai sensi del Regolamento UE 2016/679 (GDPR) e delle Linee Guida per posta elettronica ed Internet del Garante Privacy si rende noto che le informazioni contenute nella presente comunicazione e i relativi allegati possono essere riservate e sono, comunque, destinate esclusivamente alle persone o alla Società sopraindicati. Le stesse non sono da considerarsi comunicazioni personali, quindi eventuali risposte potranno essere conosciute da persone appartenenti all'azienda.
In accordance with Regulation EU 2016/679 (General Data Protection Regulation) and Guidelines Applying to the Use of E-Mails and the Internet in the Employment Context of Italian Data Protection Authority it is announced that the information in this communication (which includes any documents with it) is confidential and meant only for above-mentioned persons and/or Companies; the information is not personal therefore, It is intended for the addressee only and any replies will be known by people belonging to the company.
_________________________________________________________________
[cid:image001.png@01D8FFEC.90150110]<https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.gft.com%2Fit%2Fit%2Fsolutions%2FSOLUTIONS%2Fit%2FIT-Metaverso&data=05%7C01%7CFederico.Rullo%40gft.com%7C9d630320320c4df6e9e108dacd73f4db%7C55a2bc67aec14ad29a9c5b2457b91dcd%7C0%7C0%7C638048195183435892%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=KZcQTHhxXJsNXEcQdATMHHp1UPwtwQ%2BMFif%2BeI7R3vo%3D&reserved=0>
AW: Cannot create new datastream from JSON file
Posted by Philipp Zehnder <ph...@bytefabrik.ai>.
Hi Federico,
can you provide a minimal sample json that is not working?
Then I will test it locally to see what is going wrong.
It is strange that the preview feature works with a different replay speed.
The replay speed configuration should only be used once the adapter is started to reduce the time between the events.
Cheers,
Philipp
Von: Rullo, Federico <Fe...@gft.com>
Datum: Montag, 21. November 2022 um 17:54
An: users@streampipes.apache.org <us...@streampipes.apache.org>
Betreff: Cannot create new datastream from JSON file
Hello,
The critical issue we are having is that when we create a new datastream from a json file, the data cannot either be previewed or they do, depending on the value of the replay speed, for example, if we put a default 1 we don’t see any data previewing when the adapter is created but if we put 50 we do get the preview.
In both cases, no output is produced from the created adapters, and no errors is displayed in the logs.
The steps we are taking to create the adapters are:
1. Upload the JSON file
2. Input 1 as replay speed
3. Select JSON and JSONArray as file format
4. Selecting the data under the “data” filed
5. We already have a date_time_unix timestamp field, so we select that as the timestamp with no conversion
6. Name the adapter and start it
After this we either get no preview of the data or, if we put 50 as replay speed, get a preview of the data.
Finally, if we create a new pipeline using the newly created adapter, no data is passed.
Hence, we see no data in the data explorer and inside the influxdb itself.
We had a similar issue with the preview’s version, do you know if there is anything we are doing wrong or if there is some problem handling jsons?
The version we are currently using is the 0.70.0 set on the pipeline-element environment.
Thanks in advance,
Best Regards,
Federico Rullo
Federico Rullo
_________________________________________________________________
GFT Italia S.r.l.
Via A. Rimassa, 51/2
16129 Genova, Italy
T +39 025 8158169
Federico.Rullo@gft.com
www.gft.com/it<http://www.gft.com/it>
https://blog.gft.com
https://www.linkedin.com/company/gft-group/
www.twitter.com/gft_it <http://www.twitter.com/gft_it%20>
Registered office: GFT Italia S.r.l., Via Sile 18, 20139 Milan (Italy)
VAT Code and number of registration with the Companies' Register of Milan: 00819200478
Corporate capital: Euro 1,000,000 fully paid-in
Company subject to the direction and coordination activity of GFT Technologies SE
Ai sensi del Regolamento UE 2016/679 (GDPR) e delle Linee Guida per posta elettronica ed Internet del Garante Privacy si rende noto che le informazioni contenute nella presente comunicazione e i relativi allegati possono essere riservate e sono, comunque, destinate esclusivamente alle persone o alla Società sopraindicati. Le stesse non sono da considerarsi comunicazioni personali, quindi eventuali risposte potranno essere conosciute da persone appartenenti all’azienda.
In accordance with Regulation EU 2016/679 (General Data Protection Regulation) and Guidelines Applying to the Use of E-Mails and the Internet in the Employment Context of Italian Data Protection Authority it is announced that the information in this communication (which includes any documents with it) is confidential and meant only for above-mentioned persons and/or Companies; the information is not personal therefore, It is intended for the addressee only and any replies will be known by people belonging to the company.
_________________________________________________________________
[cid:firmaGPTWdefinitivo_da82df40-8928-4ee9-afd7-5c490fe3d6f0.png]<https://www.gft.com/it/it/solutions/SOLUTIONS/it/IT-Metaverso>