You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@streampipes.apache.org by "Iorio, Michele" <mi...@gft.com> on 2022/05/02 17:23:00 UTC

RE: Error on the Data Lake component

Hi Philipp,
initially I had an error of another type, which I have now managed to solve, but currently in the data explorer I am unable to visualize the array field I added to the event.
I also tried to add other types of fields (double) to the event instead of the array, but in data explorer they are not displayed.
Here is the code I used to add the output fields to the event:

event.addField("appendedText_xi", xi);
event.addField("appendedText_yi", yi);

out.collect(event);


The two fields added to the event (appendedText_xi, appendedText_yi), are not present in the data explorer as in the image below:

[cid:image001.png@01D85E59.5BDD7510]


Can you give me some indication to solve the problem?
Thanks
Michele



Michele Iorio
_________________________________________________________________

GFT Italia S.r.l.
Via A. Rimassa, 51/2
16129 Genova, Italy

T +39 010 583158
michele.iorio@gft.com
www.gft.com/it<http://www.gft.com/it>
https://blog.gft.com
https://www.linkedin.com/company/gft-group/
www.twitter.com/gft_it <http://www.twitter.com/gft_it>

[cid:ActoForTheFuture_GFT@1,2x_68f73108-d76e-4497-aa5c-ff8a5693ce57.png]<https://www.gft.com/it/it/about-us/Sustainability>

Registered office: GFT Italia S.r.l., Via Sile 18, 20139 Milan (Italy)
VAT Code and number of registration with the Companies' Register of Milan: 00819200478
Corporate capital: Euro 1,000,000 fully paid-in

Company subject to the direction and coordination activity of GFT Technologies SE

Ai sensi del Regolamento UE 2016/679 (GDPR) e delle Linee Guida per posta elettronica ed Internet del Garante Privacy si rende noto che le informazioni contenute nella presente comunicazione e i relativi allegati possono essere riservate e sono, comunque, destinate esclusivamente alle persone o alla Società sopraindicati. Le stesse non sono da considerarsi comunicazioni personali, quindi eventuali risposte potranno essere conosciute da persone appartenenti all'azienda.

In accordance with Regulation EU 2016/679 (General Data Protection Regulation) and Guidelines Applying to the Use of E-Mails and the Internet in the Employment Context of Italian Data Protection Authority it is announced that the information in this communication (which includes any documents with it) is confidential and meant only for above-mentioned persons and/or Companies; the information is not personal therefore, It is intended for the addressee only and any replies will be known by people belonging to the company.
_________________________________________________________________
From: Philipp Zehnder zehnder@apache.org<ma...@apache.org>
Sent: sabato 30 aprile 2022 07:28
To: users@streampipes.apache.org
Subject: Re: Error on the Data Lake component


Non si ricevono spesso messaggi di posta elettronica da zehnder@apache.org<ma...@apache.org>. Informazioni sul perché è importante<https://aka.ms/LearnAboutSenderIdentification>


[EXTERNAL EMAIL]
This email originated from outside of the organization or was sent unauthenticated. Do not click links or open attachments unless you confirm the sender and know the content is safe.
Hi Michele,

thanks for reporting the issue.
I tried it out with an example json containing an array field. For me I could store the events in the data lake, but unfortunately get an error in the data explorer when I select the array field for visualization (e.g. in the table).
When I deselect the array field the data is presented as expected, so I guess the UI has a problem visualizing array values.

Do you have the same behavior, or do you have a different error?

Cheers,
Philipp


Von: "Iorio, Michele" <mi...@gft.com>>
Antworten an: "users@streampipes.apache.org<ma...@streampipes.apache.org>" <us...@streampipes.apache.org>>
Datum: Donnerstag, 28. April 2022 um 19:21
An: "users@streampipes.apache.org<ma...@streampipes.apache.org>" <us...@streampipes.apache.org>>
Betreff: Error on the Data Lake component

Hi,
I implemented a component that outputs an array of doubles and connected it to a Data Lake, but unfortunately I get an error.
The error appears to be caused by the Data Lake.
Which component should I use instead?

Thanks
Michele Iorio


Michele Iorio
_________________________________________________________________

GFT Italia S.r.l.
Via A. Rimassa, 51/2
16129 Genova, Italy

T +39 010 583158
michele.iorio@gft.com<ma...@gft.com>
www.gft.com/it<https://eur02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.gft.com%2Fit&data=05%7C01%7Cmichele.iorio%40gft.com%7C3ef91c26672c4c85a86108da2a6a3a80%7C55a2bc67aec14ad29a9c5b2457b91dcd%7C0%7C0%7C637868932981666361%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=jTeCb%2B28Ph0Jd30RXahHL712Z1eYq7rBdKeUU61CGx4%3D&reserved=0>
https://blog.gft.com<https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fblog.gft.com%2F&data=05%7C01%7Cmichele.iorio%40gft.com%7C3ef91c26672c4c85a86108da2a6a3a80%7C55a2bc67aec14ad29a9c5b2457b91dcd%7C0%7C0%7C637868932981666361%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=n6btOpWjLKNs0VO4IZuJXjZzsDLm74xxlVZkkcBG0KE%3D&reserved=0>
https://www.linkedin.com/company/gft-group/<https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.linkedin.com%2Fcompany%2Fgft-group%2F&data=05%7C01%7Cmichele.iorio%40gft.com%7C3ef91c26672c4c85a86108da2a6a3a80%7C55a2bc67aec14ad29a9c5b2457b91dcd%7C0%7C0%7C637868932981666361%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=dNUYuC94rJ4KcS%2BfHsfnsCoJOAVfidxQVlOWSzCzARc%3D&reserved=0>
www.twitter.com/gft_it <https://eur02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.twitter.com%2Fgft_it%2520&data=05%7C01%7Cmichele.iorio%40gft.com%7C3ef91c26672c4c85a86108da2a6a3a80%7C55a2bc67aec14ad29a9c5b2457b91dcd%7C0%7C0%7C637868932981666361%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=5WtMGV74xjbdsm%2F0VUbpHFzrFE0jxWTcUj6cxRu5yUA%3D&reserved=0>

[cid:image003.png@01D85E5A.0A951680]<https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.gft.com%2Fit%2Fit%2Fabout-us%2FSustainability&data=05%7C01%7Cmichele.iorio%40gft.com%7C3ef91c26672c4c85a86108da2a6a3a80%7C55a2bc67aec14ad29a9c5b2457b91dcd%7C0%7C0%7C637868932981666361%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=J5kfDYz15QyV%2F%2FlGYvsm874yUd%2FkiyMEcTkY0aaLFUQ%3D&reserved=0>

Registered office: GFT Italia S.r.l., Via Sile 18, 20139 Milan (Italy)
VAT Code and number of registration with the Companies' Register of Milan: 00819200478
Corporate capital: Euro 1,000,000 fully paid-in

Company subject to the direction and coordination activity of GFT Technologies SE

Ai sensi del Regolamento UE 2016/679 (GDPR) e delle Linee Guida per posta elettronica ed Internet del Garante Privacy si rende noto che le informazioni contenute nella presente comunicazione e i relativi allegati possono essere riservate e sono, comunque, destinate esclusivamente alle persone o alla Società sopraindicati. Le stesse non sono da considerarsi comunicazioni personali, quindi eventuali risposte potranno essere conosciute da persone appartenenti all'azienda.

In accordance with Regulation EU 2016/679 (General Data Protection Regulation) and Guidelines Applying to the Use of E-Mails and the Internet in the Employment Context of Italian Data Protection Authority it is announced that the information in this communication (which includes any documents with it) is confidential and meant only for above-mentioned persons and/or Companies; the information is not personal therefore, It is intended for the addressee only and any replies will be known by people belonging to the company.
_________________________________________________________________

Re: Error on the Data Lake component

Posted by Philipp Zehnder <ze...@apache.org>.
Hi Michele,

 

I'm glad to hear it's working now.

 

It seems that the description of the DataProcessor was not updated correctly.

You can also trigger this under “Install Pipeline Elements”, then select the configurations of your processor (“…”) and click update.

This will update the description of the processing element without reinstalling StreamPipes.

 

Please let me know if there are any other issues.

 

Cheers,

Philipp

 

 

 

 

Von: "Iorio, Michele" <mi...@gft.com>
Antworten an: "users@streampipes.apache.org" <us...@streampipes.apache.org>
Datum: Mittwoch, 4. Mai 2022 um 11:09
An: "users@streampipes.apache.org" <us...@streampipes.apache.org>
Betreff: RE: Error on the Data Lake component

 

 

Hi Philipp,

thanks for the reply.

Below you can find the declareModel method that I implemented where the two custom parameters are inserted in the output:

 

 

  @Override

  public DataProcessorDescription declareModel() {

    return ProcessingElementBuilder.create("org.example.pe.example.processor")

            .withAssets(Assets.DOCUMENTATION, Assets.ICON)

            .withLocales(Locales.EN)

            .category(DataProcessorType.AGGREGATE)

            .requiredStream(StreamRequirementsBuilder.create()

                    .requiredPropertyWithUnaryMapping(EpRequirements.numberReq(),

                            Labels.withId(STARTVALUE), PropertyScope.NONE)

                    .build())

            .requiredFloatParameter(Labels.withId(ENDVALUE))

            .requiredFloatParameter(Labels.withId(COUNT))

            .outputStrategy(OutputStrategies.append(PrimitivePropertyBuilder.create(Datatypes.Double, "appendedText_yi").build(),

                    PrimitivePropertyBuilder.create(Datatypes.Double, "appendedText_xi").build()))

 

            .build();

  }

 

 

I tried cleaning SP and pipelines and regenerating the process.

Now the parameters I entered are displayed in the output.

 

 

Thanks for the support.

Cheers,

Michele

 

 

 

Michele Iorio
_________________________________________________________________

GFT Italia S.r.l.
Via A. Rimassa, 51/2
16129 Genova, Italy

T +39 010 583158
michele.iorio@gft.com
www.gft.com/it
https://blog.gft.com
https://www.linkedin.com/company/gft-group/
www.twitter.com/gft_it 



Registered office: GFT Italia S.r.l., Via Sile 18, 20139 Milan (Italy)
VAT Code and number of registration with the Companies' Register of Milan: 00819200478
Corporate capital: Euro 1,000,000 fully paid-in

Company subject to the direction and coordination activity of GFT Technologies SE

Ai sensi del Regolamento UE 2016/679 (GDPR) e delle Linee Guida per posta elettronica ed Internet del Garante Privacy si rende noto che le informazioni contenute nella presente comunicazione e i relativi allegati possono essere riservate e sono, comunque, destinate esclusivamente alle persone o alla Società sopraindicati. Le stesse non sono da considerarsi comunicazioni personali, quindi eventuali risposte potranno essere conosciute da persone appartenenti all’azienda. 

In accordance with Regulation EU 2016/679 (General Data Protection Regulation) and Guidelines Applying to the Use of E-Mails and the Internet in the Employment Context of Italian Data Protection Authority it is announced that the information in this communication (which includes any documents with it) is confidential and meant only for above-mentioned persons and/or Companies; the information is not personal therefore, It is intended for the addressee only and any replies will be known by people belonging to the company.
_________________________________________________________________

From: Philipp Zehnder <ze...@apache.org> 
Sent: martedì 3 maggio 2022 18:22
To: users@streampipes.apache.org
Subject: Re: Error on the Data Lake component

 

Non si ricevono spesso messaggi di posta elettronica da zehnder@apache.org. Informazioni sull'importanza di questo fatto
[EXTERNAL EMAIL]
This email originated from outside of the organization or was sent unauthenticated. Do not click links or open attachments unless you confirm the sender and know the content is safe.

Hi Michele,

 

can you show me how the declareModel method looks like?

Which output strategy did you select and did you add these two fields to the event schema?

 

Cheers,

Philipp⁄

 

 

 

Von: "Iorio, Michele" <mi...@gft.com>
Antworten an: "users@streampipes.apache.org" <us...@streampipes.apache.org>
Datum: Montag, 2. Mai 2022 um 19:23
An: "users@streampipes.apache.org" <us...@streampipes.apache.org>
Betreff: RE: Error on the Data Lake component

 

Hi Philipp,

initially I had an error of another type, which I have now managed to solve, but currently in the data explorer I am unable to visualize the array field I added to the event.

I also tried to add other types of fields (double) to the event instead of the array, but in data explorer they are not displayed.

Here is the code I used to add the output fields to the event:

 

event.addField("appendedText_xi", xi);
event.addField("appendedText_yi", yi);

out.collect(event);

 

 

The two fields added to the event (appendedText_xi, appendedText_yi), are not present in the data explorer as in the image below:

 

 

 

Can you give me some indication to solve the problem?

Thanks

Michele

 

 

Michele Iorio
_________________________________________________________________

GFT Italia S.r.l.
Via A. Rimassa, 51/2
16129 Genova, Italy

T +39 010 583158
michele.iorio@gft.com
www.gft.com/it
https://blog.gft.com
https://www.linkedin.com/company/gft-group/
www.twitter.com/gft_it 



Registered office: GFT Italia S.r.l., Via Sile 18, 20139 Milan (Italy)
VAT Code and number of registration with the Companies' Register of Milan: 00819200478
Corporate capital: Euro 1,000,000 fully paid-in

Company subject to the direction and coordination activity of GFT Technologies SE

Ai sensi del Regolamento UE 2016/679 (GDPR) e delle Linee Guida per posta elettronica ed Internet del Garante Privacy si rende noto che le informazioni contenute nella presente comunicazione e i relativi allegati possono essere riservate e sono, comunque, destinate esclusivamente alle persone o alla Società sopraindicati. Le stesse non sono da considerarsi comunicazioni personali, quindi eventuali risposte potranno essere conosciute da persone appartenenti all’azienda. 

In accordance with Regulation EU 2016/679 (General Data Protection Regulation) and Guidelines Applying to the Use of E-Mails and the Internet in the Employment Context of Italian Data Protection Authority it is announced that the information in this communication (which includes any documents with it) is confidential and meant only for above-mentioned persons and/or Companies; the information is not personal therefore, It is intended for the addressee only and any replies will be known by people belonging to the company.
_________________________________________________________________

From: Philipp Zehnder zehnder@apache.org 
Sent: sabato 30 aprile 2022 07:28
To: users@streampipes.apache.org
Subject: Re: Error on the Data Lake component

 

Non si ricevono spesso messaggi di posta elettronica da zehnder@apache.org. Informazioni sul perché è importante
[EXTERNAL EMAIL]
This email originated from outside of the organization or was sent unauthenticated. Do not click links or open attachments unless you confirm the sender and know the content is safe.

Hi Michele,

 

thanks for reporting the issue. 

I tried it out with an example json containing an array field. For me I could store the events in the data lake, but unfortunately get an error in the data explorer when I select the array field for visualization (e.g. in the table).

When I deselect the array field the data is presented as expected, so I guess the UI has a problem visualizing array values.

 

Do you have the same behavior, or do you have a different error?

 

Cheers,

Philipp

 

 

Von: "Iorio, Michele" <mi...@gft.com>
Antworten an: "users@streampipes.apache.org" <us...@streampipes.apache.org>
Datum: Donnerstag, 28. April 2022 um 19:21
An: "users@streampipes.apache.org" <us...@streampipes.apache.org>
Betreff: Error on the Data Lake component

 

Hi,

I implemented a component that outputs an array of doubles and connected it to a Data Lake, but unfortunately I get an error.

The error appears to be caused by the Data Lake.

Which component should I use instead?

 

Thanks

Michele Iorio

 

Michele Iorio
_________________________________________________________________

GFT Italia S.r.l.
Via A. Rimassa, 51/2
16129 Genova, Italy

T +39 010 583158
michele.iorio@gft.com
www.gft.com/it
https://blog.gft.com
https://www.linkedin.com/company/gft-group/
www.twitter.com/gft_it 



Registered office: GFT Italia S.r.l., Via Sile 18, 20139 Milan (Italy)
VAT Code and number of registration with the Companies' Register of Milan: 00819200478
Corporate capital: Euro 1,000,000 fully paid-in

Company subject to the direction and coordination activity of GFT Technologies SE

Ai sensi del Regolamento UE 2016/679 (GDPR) e delle Linee Guida per posta elettronica ed Internet del Garante Privacy si rende noto che le informazioni contenute nella presente comunicazione e i relativi allegati possono essere riservate e sono, comunque, destinate esclusivamente alle persone o alla Società sopraindicati. Le stesse non sono da considerarsi comunicazioni personali, quindi eventuali risposte potranno essere conosciute da persone appartenenti all’azienda. 

In accordance with Regulation EU 2016/679 (General Data Protection Regulation) and Guidelines Applying to the Use of E-Mails and the Internet in the Employment Context of Italian Data Protection Authority it is announced that the information in this communication (which includes any documents with it) is confidential and meant only for above-mentioned persons and/or Companies; the information is not personal therefore, It is intended for the addressee only and any replies will be known by people belonging to the company.
_________________________________________________________________

RE: Error on the Data Lake component

Posted by "Iorio, Michele" <mi...@gft.com>.
Hi Philipp,
thanks for the reply.
Below you can find the declareModel method that I implemented where the two custom parameters are inserted in the output:


  @Override
  public DataProcessorDescription declareModel() {
    return ProcessingElementBuilder.create("org.example.pe.example.processor")
            .withAssets(Assets.DOCUMENTATION, Assets.ICON)
            .withLocales(Locales.EN)
            .category(DataProcessorType.AGGREGATE)
            .requiredStream(StreamRequirementsBuilder.create()
                    .requiredPropertyWithUnaryMapping(EpRequirements.numberReq(),
                            Labels.withId(STARTVALUE), PropertyScope.NONE)
                    .build())
            .requiredFloatParameter(Labels.withId(ENDVALUE))
            .requiredFloatParameter(Labels.withId(COUNT))
            .outputStrategy(OutputStrategies.append(PrimitivePropertyBuilder.create(Datatypes.Double, "appendedText_yi").build(),
                    PrimitivePropertyBuilder.create(Datatypes.Double, "appendedText_xi").build()))

            .build();
  }


I tried cleaning SP and pipelines and regenerating the process.
Now the parameters I entered are displayed in the output.

[cid:image004.png@01D85FA7.4E227D90]

Thanks for the support.
Cheers,
Michele




Michele Iorio
_________________________________________________________________

GFT Italia S.r.l.
Via A. Rimassa, 51/2
16129 Genova, Italy

T +39 010 583158
michele.iorio@gft.com
www.gft.com/it<http://www.gft.com/it>
https://blog.gft.com
https://www.linkedin.com/company/gft-group/
www.twitter.com/gft_it <http://www.twitter.com/gft_it>

[cid:ActoForTheFuture_GFT@1,2x_68f73108-d76e-4497-aa5c-ff8a5693ce57.png]<https://www.gft.com/it/it/about-us/Sustainability>

Registered office: GFT Italia S.r.l., Via Sile 18, 20139 Milan (Italy)
VAT Code and number of registration with the Companies' Register of Milan: 00819200478
Corporate capital: Euro 1,000,000 fully paid-in

Company subject to the direction and coordination activity of GFT Technologies SE

Ai sensi del Regolamento UE 2016/679 (GDPR) e delle Linee Guida per posta elettronica ed Internet del Garante Privacy si rende noto che le informazioni contenute nella presente comunicazione e i relativi allegati possono essere riservate e sono, comunque, destinate esclusivamente alle persone o alla Società sopraindicati. Le stesse non sono da considerarsi comunicazioni personali, quindi eventuali risposte potranno essere conosciute da persone appartenenti all’azienda.

In accordance with Regulation EU 2016/679 (General Data Protection Regulation) and Guidelines Applying to the Use of E-Mails and the Internet in the Employment Context of Italian Data Protection Authority it is announced that the information in this communication (which includes any documents with it) is confidential and meant only for above-mentioned persons and/or Companies; the information is not personal therefore, It is intended for the addressee only and any replies will be known by people belonging to the company.
_________________________________________________________________
From: Philipp Zehnder <ze...@apache.org>
Sent: martedì 3 maggio 2022 18:22
To: users@streampipes.apache.org
Subject: Re: Error on the Data Lake component


Non si ricevono spesso messaggi di posta elettronica da zehnder@apache.org<ma...@apache.org>. Informazioni sull'importanza di questo fatto<http://aka.ms/LearnAboutSenderIdentification>


[EXTERNAL EMAIL]
This email originated from outside of the organization or was sent unauthenticated. Do not click links or open attachments unless you confirm the sender and know the content is safe.
Hi Michele,

can you show me how the declareModel method looks like?
Which output strategy did you select and did you add these two fields to the event schema?

Cheers,
Philipp⁄



Von: "Iorio, Michele" <mi...@gft.com>>
Antworten an: "users@streampipes.apache.org<ma...@streampipes.apache.org>" <us...@streampipes.apache.org>>
Datum: Montag, 2. Mai 2022 um 19:23
An: "users@streampipes.apache.org<ma...@streampipes.apache.org>" <us...@streampipes.apache.org>>
Betreff: RE: Error on the Data Lake component

Hi Philipp,
initially I had an error of another type, which I have now managed to solve, but currently in the data explorer I am unable to visualize the array field I added to the event.
I also tried to add other types of fields (double) to the event instead of the array, but in data explorer they are not displayed.
Here is the code I used to add the output fields to the event:

event.addField("appendedText_xi", xi);
event.addField("appendedText_yi", yi);

out.collect(event);


The two fields added to the event (appendedText_xi, appendedText_yi), are not present in the data explorer as in the image below:

[cid:image001.png@01D85FA7.4E3171B0]


Can you give me some indication to solve the problem?
Thanks
Michele



Michele Iorio
_________________________________________________________________

GFT Italia S.r.l.
Via A. Rimassa, 51/2
16129 Genova, Italy

T +39 010 583158
michele.iorio@gft.com<ma...@gft.com>
www.gft.com/it<https://eur02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.gft.com%2Fit&data=05%7C01%7Cmichele.iorio%40gft.com%7C8b06f767bc7f461beb3308da2d211bcd%7C55a2bc67aec14ad29a9c5b2457b91dcd%7C0%7C0%7C637871917485349650%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=1Q8pb6%2BcVrDqq7oK70m%2BVFEYYFrRaOord5vZJtDOkVg%3D&reserved=0>
https://blog.gft.com<https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fblog.gft.com%2F&data=05%7C01%7Cmichele.iorio%40gft.com%7C8b06f767bc7f461beb3308da2d211bcd%7C55a2bc67aec14ad29a9c5b2457b91dcd%7C0%7C0%7C637871917485349650%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=hFV5fG%2FQOVYW4v5SlCzXw53BcUvF%2F0vFM2CX7Cr2AK8%3D&reserved=0>
https://www.linkedin.com/company/gft-group/<https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.linkedin.com%2Fcompany%2Fgft-group%2F&data=05%7C01%7Cmichele.iorio%40gft.com%7C8b06f767bc7f461beb3308da2d211bcd%7C55a2bc67aec14ad29a9c5b2457b91dcd%7C0%7C0%7C637871917485349650%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=KRjUHBP7WbO0iZkJwVNxPdyyi5xwE%2Fo9Cnt4Xqb2W%2BA%3D&reserved=0>
www.twitter.com/gft_it <https://eur02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.twitter.com%2Fgft_it%2520&data=05%7C01%7Cmichele.iorio%40gft.com%7C8b06f767bc7f461beb3308da2d211bcd%7C55a2bc67aec14ad29a9c5b2457b91dcd%7C0%7C0%7C637871917485349650%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=y2rxhqyLDNymtvt%2B4Fxp8hwjxkZeJnT1zKNLtumRIZ4%3D&reserved=0>

[cid:image002.png@01D85FA7.4E3171B0]<https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.gft.com%2Fit%2Fit%2Fabout-us%2FSustainability&data=05%7C01%7Cmichele.iorio%40gft.com%7C8b06f767bc7f461beb3308da2d211bcd%7C55a2bc67aec14ad29a9c5b2457b91dcd%7C0%7C0%7C637871917485349650%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=xDENm2odCD7FqqP7eSdzMDsWJ9H%2FLxgZJmhBEPMMYhY%3D&reserved=0>

Registered office: GFT Italia S.r.l., Via Sile 18, 20139 Milan (Italy)
VAT Code and number of registration with the Companies' Register of Milan: 00819200478
Corporate capital: Euro 1,000,000 fully paid-in

Company subject to the direction and coordination activity of GFT Technologies SE

Ai sensi del Regolamento UE 2016/679 (GDPR) e delle Linee Guida per posta elettronica ed Internet del Garante Privacy si rende noto che le informazioni contenute nella presente comunicazione e i relativi allegati possono essere riservate e sono, comunque, destinate esclusivamente alle persone o alla Società sopraindicati. Le stesse non sono da considerarsi comunicazioni personali, quindi eventuali risposte potranno essere conosciute da persone appartenenti all’azienda.

In accordance with Regulation EU 2016/679 (General Data Protection Regulation) and Guidelines Applying to the Use of E-Mails and the Internet in the Employment Context of Italian Data Protection Authority it is announced that the information in this communication (which includes any documents with it) is confidential and meant only for above-mentioned persons and/or Companies; the information is not personal therefore, It is intended for the addressee only and any replies will be known by people belonging to the company.
_________________________________________________________________
From: Philipp Zehnder zehnder@apache.org<ma...@apache.org>
Sent: sabato 30 aprile 2022 07:28
To: users@streampipes.apache.org<ma...@streampipes.apache.org>
Subject: Re: Error on the Data Lake component


Non si ricevono spesso messaggi di posta elettronica da zehnder@apache.org<ma...@apache.org>. Informazioni sul perché è importante<https://aka.ms/LearnAboutSenderIdentification>


[EXTERNAL EMAIL]
This email originated from outside of the organization or was sent unauthenticated. Do not click links or open attachments unless you confirm the sender and know the content is safe.
Hi Michele,

thanks for reporting the issue.
I tried it out with an example json containing an array field. For me I could store the events in the data lake, but unfortunately get an error in the data explorer when I select the array field for visualization (e.g. in the table).
When I deselect the array field the data is presented as expected, so I guess the UI has a problem visualizing array values.

Do you have the same behavior, or do you have a different error?

Cheers,
Philipp


Von: "Iorio, Michele" <mi...@gft.com>>
Antworten an: "users@streampipes.apache.org<ma...@streampipes.apache.org>" <us...@streampipes.apache.org>>
Datum: Donnerstag, 28. April 2022 um 19:21
An: "users@streampipes.apache.org<ma...@streampipes.apache.org>" <us...@streampipes.apache.org>>
Betreff: Error on the Data Lake component

Hi,
I implemented a component that outputs an array of doubles and connected it to a Data Lake, but unfortunately I get an error.
The error appears to be caused by the Data Lake.
Which component should I use instead?

Thanks
Michele Iorio


Michele Iorio
_________________________________________________________________

GFT Italia S.r.l.
Via A. Rimassa, 51/2
16129 Genova, Italy

T +39 010 583158
michele.iorio@gft.com<ma...@gft.com>
www.gft.com/it<https://eur02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.gft.com%2Fit&data=05%7C01%7Cmichele.iorio%40gft.com%7C8b06f767bc7f461beb3308da2d211bcd%7C55a2bc67aec14ad29a9c5b2457b91dcd%7C0%7C0%7C637871917485349650%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=1Q8pb6%2BcVrDqq7oK70m%2BVFEYYFrRaOord5vZJtDOkVg%3D&reserved=0>
https://blog.gft.com<https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fblog.gft.com%2F&data=05%7C01%7Cmichele.iorio%40gft.com%7C8b06f767bc7f461beb3308da2d211bcd%7C55a2bc67aec14ad29a9c5b2457b91dcd%7C0%7C0%7C637871917485349650%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=hFV5fG%2FQOVYW4v5SlCzXw53BcUvF%2F0vFM2CX7Cr2AK8%3D&reserved=0>
https://www.linkedin.com/company/gft-group/<https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.linkedin.com%2Fcompany%2Fgft-group%2F&data=05%7C01%7Cmichele.iorio%40gft.com%7C8b06f767bc7f461beb3308da2d211bcd%7C55a2bc67aec14ad29a9c5b2457b91dcd%7C0%7C0%7C637871917485349650%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=KRjUHBP7WbO0iZkJwVNxPdyyi5xwE%2Fo9Cnt4Xqb2W%2BA%3D&reserved=0>
www.twitter.com/gft_it <https://eur02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.twitter.com%2Fgft_it%2520&data=05%7C01%7Cmichele.iorio%40gft.com%7C8b06f767bc7f461beb3308da2d211bcd%7C55a2bc67aec14ad29a9c5b2457b91dcd%7C0%7C0%7C637871917485349650%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=y2rxhqyLDNymtvt%2B4Fxp8hwjxkZeJnT1zKNLtumRIZ4%3D&reserved=0>

[cid:image003.png@01D85F9B.5EC8FEF0]<https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.gft.com%2Fit%2Fit%2Fabout-us%2FSustainability&data=05%7C01%7Cmichele.iorio%40gft.com%7C8b06f767bc7f461beb3308da2d211bcd%7C55a2bc67aec14ad29a9c5b2457b91dcd%7C0%7C0%7C637871917485349650%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=xDENm2odCD7FqqP7eSdzMDsWJ9H%2FLxgZJmhBEPMMYhY%3D&reserved=0>

Registered office: GFT Italia S.r.l., Via Sile 18, 20139 Milan (Italy)
VAT Code and number of registration with the Companies' Register of Milan: 00819200478
Corporate capital: Euro 1,000,000 fully paid-in

Company subject to the direction and coordination activity of GFT Technologies SE

Ai sensi del Regolamento UE 2016/679 (GDPR) e delle Linee Guida per posta elettronica ed Internet del Garante Privacy si rende noto che le informazioni contenute nella presente comunicazione e i relativi allegati possono essere riservate e sono, comunque, destinate esclusivamente alle persone o alla Società sopraindicati. Le stesse non sono da considerarsi comunicazioni personali, quindi eventuali risposte potranno essere conosciute da persone appartenenti all’azienda.

In accordance with Regulation EU 2016/679 (General Data Protection Regulation) and Guidelines Applying to the Use of E-Mails and the Internet in the Employment Context of Italian Data Protection Authority it is announced that the information in this communication (which includes any documents with it) is confidential and meant only for above-mentioned persons and/or Companies; the information is not personal therefore, It is intended for the addressee only and any replies will be known by people belonging to the company.
_________________________________________________________________

Re: Error on the Data Lake component

Posted by Philipp Zehnder <ze...@apache.org>.
Hi Michele,

 

can you show me how the declareModel method looks like?

Which output strategy did you select and did you add these two fields to the event schema?

 

Cheers,

Philipp⁄

 

 

 

Von: "Iorio, Michele" <mi...@gft.com>
Antworten an: "users@streampipes.apache.org" <us...@streampipes.apache.org>
Datum: Montag, 2. Mai 2022 um 19:23
An: "users@streampipes.apache.org" <us...@streampipes.apache.org>
Betreff: RE: Error on the Data Lake component

 

Hi Philipp,

initially I had an error of another type, which I have now managed to solve, but currently in the data explorer I am unable to visualize the array field I added to the event.

I also tried to add other types of fields (double) to the event instead of the array, but in data explorer they are not displayed.

Here is the code I used to add the output fields to the event:

 

event.addField("appendedText_xi", xi);
event.addField("appendedText_yi", yi);

out.collect(event);

 

 

The two fields added to the event (appendedText_xi, appendedText_yi), are not present in the data explorer as in the image below:

 

 

 

Can you give me some indication to solve the problem?

Thanks

Michele

 

 

Michele Iorio
_________________________________________________________________

GFT Italia S.r.l.
Via A. Rimassa, 51/2
16129 Genova, Italy

T +39 010 583158
michele.iorio@gft.com
www.gft.com/it
https://blog.gft.com
https://www.linkedin.com/company/gft-group/
www.twitter.com/gft_it 



Registered office: GFT Italia S.r.l., Via Sile 18, 20139 Milan (Italy)
VAT Code and number of registration with the Companies' Register of Milan: 00819200478
Corporate capital: Euro 1,000,000 fully paid-in

Company subject to the direction and coordination activity of GFT Technologies SE

Ai sensi del Regolamento UE 2016/679 (GDPR) e delle Linee Guida per posta elettronica ed Internet del Garante Privacy si rende noto che le informazioni contenute nella presente comunicazione e i relativi allegati possono essere riservate e sono, comunque, destinate esclusivamente alle persone o alla Società sopraindicati. Le stesse non sono da considerarsi comunicazioni personali, quindi eventuali risposte potranno essere conosciute da persone appartenenti all’azienda. 

In accordance with Regulation EU 2016/679 (General Data Protection Regulation) and Guidelines Applying to the Use of E-Mails and the Internet in the Employment Context of Italian Data Protection Authority it is announced that the information in this communication (which includes any documents with it) is confidential and meant only for above-mentioned persons and/or Companies; the information is not personal therefore, It is intended for the addressee only and any replies will be known by people belonging to the company.
_________________________________________________________________

From: Philipp Zehnder zehnder@apache.org 
Sent: sabato 30 aprile 2022 07:28
To: users@streampipes.apache.org
Subject: Re: Error on the Data Lake component

 

Non si ricevono spesso messaggi di posta elettronica da zehnder@apache.org. Informazioni sul perché è importante
[EXTERNAL EMAIL]
This email originated from outside of the organization or was sent unauthenticated. Do not click links or open attachments unless you confirm the sender and know the content is safe.

Hi Michele,

 

thanks for reporting the issue. 

I tried it out with an example json containing an array field. For me I could store the events in the data lake, but unfortunately get an error in the data explorer when I select the array field for visualization (e.g. in the table).

When I deselect the array field the data is presented as expected, so I guess the UI has a problem visualizing array values.

 

Do you have the same behavior, or do you have a different error?

 

Cheers,

Philipp

 

 

Von: "Iorio, Michele" <mi...@gft.com>
Antworten an: "users@streampipes.apache.org" <us...@streampipes.apache.org>
Datum: Donnerstag, 28. April 2022 um 19:21
An: "users@streampipes.apache.org" <us...@streampipes.apache.org>
Betreff: Error on the Data Lake component

 

Hi,

I implemented a component that outputs an array of doubles and connected it to a Data Lake, but unfortunately I get an error.

The error appears to be caused by the Data Lake.

Which component should I use instead?

 

Thanks

Michele Iorio

 

Michele Iorio
_________________________________________________________________

GFT Italia S.r.l.
Via A. Rimassa, 51/2
16129 Genova, Italy

T +39 010 583158
michele.iorio@gft.com
www.gft.com/it
https://blog.gft.com
https://www.linkedin.com/company/gft-group/
www.twitter.com/gft_it 



Registered office: GFT Italia S.r.l., Via Sile 18, 20139 Milan (Italy)
VAT Code and number of registration with the Companies' Register of Milan: 00819200478
Corporate capital: Euro 1,000,000 fully paid-in

Company subject to the direction and coordination activity of GFT Technologies SE

Ai sensi del Regolamento UE 2016/679 (GDPR) e delle Linee Guida per posta elettronica ed Internet del Garante Privacy si rende noto che le informazioni contenute nella presente comunicazione e i relativi allegati possono essere riservate e sono, comunque, destinate esclusivamente alle persone o alla Società sopraindicati. Le stesse non sono da considerarsi comunicazioni personali, quindi eventuali risposte potranno essere conosciute da persone appartenenti all’azienda. 

In accordance with Regulation EU 2016/679 (General Data Protection Regulation) and Guidelines Applying to the Use of E-Mails and the Internet in the Employment Context of Italian Data Protection Authority it is announced that the information in this communication (which includes any documents with it) is confidential and meant only for above-mentioned persons and/or Companies; the information is not personal therefore, It is intended for the addressee only and any replies will be known by people belonging to the company.
_________________________________________________________________