You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@streampipes.apache.org by Grainier Perera <gr...@gmail.com> on 2020/05/13 12:41:27 UTC

Data processor to evaluate JavaScript

Hi all,

I'm planning to implement a JavaScript eval data processor. As the name
suggests, users will be able to write some JavaScript code which takes in
an event (as a map), do some processing on the event, and return a new map
which then gets converted to an sp-event.

example js:

function process(event) {
    // do processing here.
    // return processed event.
    return {id: event.id, tempInCelsius: (event.tempInKelvin - 273.15)};
};


Will this be a good addition to pipeline-elements? What do you think?

Regards,
Grainier.

Re: Data processor to evaluate JavaScript

Posted by Patrick Wiener <wi...@apache.org>.

Cool.

As for the base image discussion: I guess this is worth discussing in a different thread. So feel free to open.

Patrick

> Am 17.06.2020 um 11:21 schrieb Philipp Zehnder <ze...@apache.org>:
> 
> Hi Dominik,
> 
> unfortunately this was not the problem, but this could become a problem in the future ;)
> The problem was the java version. So far we used Java 8 in the docker images, after updating it to Java 9 it worked.
> 
> So far I just changed the docker versions of the images using the JS processor and it should work now.
> I saw that we use java 11 for the arm images. Which version should we use for x86? I think it makes sense to harmonize the base image for all containers.
> So the question is, which base image should we use? Any suggestions?
> 
> Philipp
> 
>> On 15. Jun 2020, at 23:11, Dominik Riemer <ri...@apache.org> wrote:
>> 
>> Hi Philipp,
>> 
>> coming back to this (and with later I really didn't mean ten days, sorry for that) ;-)
>> I could also reproduce this now - it seems to be only an issue when running in Docker.
>> 
>> When looking at the container logs when running the element in Docker, I get a NullPointerException in line 45 of JsEval (which could indicate that either engine is null or code is null).
>> 
>> So could this be an issue with the Java version we're using in the Docker images? The currently used version is AdoptOpenJDK 11.
>> 
>> I found a similar issue on so:
>> https://stackoverflow.com/questions/60916424/getenginebynamejavascript-returning-null-on-java-11
>> 
>> Could this be the cause? 
>> 
>> Dominik
>> 
>> On 2020/06/05 09:02:10, "Dominik Riemer" <ri...@apache.org> wrote: 
>>> Hi Philipp,
>>> 
>>> I can try to reproduce it later.
>>> 
>>> Dominik
>>> 
>>> -----Original Message-----
>>> From: Philipp Zehnder <ze...@apache.org> 
>>> Sent: Friday, June 5, 2020 11:01 AM
>>> To: dev@streampipes.apache.org
>>> Subject: Re: Data processor to evaluate JavaScript
>>> 
>>> Hi, 
>>> 
>>> I just tried it again and removed all docker images from the VM did a fresh pull.
>>> Unfortunately, with the same result. I tried the processor in both containers (pipeline-elements-all-jvm / processors-enricher-jvm).
>>> 
>>> Does anyone else have an idea what the problem could be?
>>> 
>>> Philipp
>>> 
>>>> On 4. Jun 2020, at 12:00, Grainier Perera <gr...@apache.org> wrote:
>>>> 
>>>> Hmm, That's odd. It doesn't use any 3rd party libraries for the PE (we 
>>>> use javax.script.*). However, I came across a similar situation with 
>>>> this RDFMappingException[1]. The reason for that is JSEval use newly 
>>>> introduced .requiredCodeblock() controller prop, and for some reason, 
>>>> there's a cached Docker image without it. But when I cleaned that and 
>>>> re-build, it worked. Could this be the reason?
>>>> 
>>>> [1] io.fogsy.empire.pinto.RDFMappingException: Could not create an 
>>>> instance of class org.apache.streampipes.model.output.OutputStrategy, 
>>>> it does not have a default constructor
>>>> 
>>>> Grainier Perera.
>>>> 
>>>> 
>>>> On Thu, 4 Jun 2020 at 15:17, Philipp Zehnder <ze...@apache.org> wrote:
>>>> 
>>>>> Hi all,
>>>>> 
>>>>> I tried to use the Processor JavaScript Eval in a deployment of 
>>>>> StreamPipes (Lite), running in Docker.
>>>>> Unfortunately, I get an error in the UI when starting the pipeline. 
>>>>> There is no specific error message, just an error that the JavaScript 
>>>>> Eval did not start.
>>>>> I also did not see anything in the logs. Did anyone else have a 
>>>>> similar issue?
>>>>> 
>>>>> The processor runs in the container pipeline-elements-all-jvm using 
>>>>> the CLI.
>>>>> 
>>>>> Running the same pipeline locally in my IDE works well, so I guess 
>>>>> there is a problem when building the project.
>>>>> Is it possible that some libraries are not bundled correctly?
>>>>> 
>>>>> Philipp
>>>>> 
>>>>>> On 22. May 2020, at 16:57, Grainier Perera 
>>>>>> <gr...@gmail.com>
>>>>> wrote:
>>>>>> 
>>>>>> Hi Dominik,
>>>>>> 
>>>>>> My bad. There were some cached images & that was the issue. Also 
>>>>>> fixed
>>>>> the
>>>>>> PR as per your comments. Please review and merge [1].
>>>>>> 
>>>>>> [1] 
>>>>>> https://github.com/apache/incubator-streampipes-extensions/pull/15
>>>>>> 
>>>>>> Thanks & Regards,
>>>>>> Grainier
>>>>>> 
>>>>>> On Fri, 22 May 2020 at 16:25, Dominik Riemer <ri...@apache.org> wrote:
>>>>>> 
>>>>>>> Hi Grainier,
>>>>>>> 
>>>>>>> we have a Github Actions workflow [1] for core+extensions that 
>>>>>>> usually builds and pushes the latest development containers for dev 
>>>>>>> on each
>>>>> push.
>>>>>>> You are using the latest CLI from dev and the tmpl_env or .env 
>>>>>>> version
>>>>> is
>>>>>>> set to 0.67.0-SNAPSHOT, right?
>>>>>>> 
>>>>>>> I just tested the JsEval processor (with CLI 0.67.0-SNAPSHOT, local 
>>>>>>> backend with the latest changes from dev and your PR) and it works
>>>>> great 😊
>>>>>>> The only thing I had to change was a typo in line 59 (use outEvent
>>>>> instead
>>>>>>> of event).
>>>>>>> 
>>>>>>> In the controller, the supportedProtocols and supportedFormats are 
>>>>>>> no longer needed, as they can now be declared for a whole module in 
>>>>>>> the
>>>>> Init
>>>>>>> class (registerFormats/Protocols). I need to update this in the 
>>>>>>> documentation.
>>>>>>> 
>>>>>>> Dominik
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>> https://github.com/apache/incubator-streampipes/blob/dev/.github/work
>>>>> flows/build.yml
>>>>>>> 
>>>>>>> -----Original Message-----
>>>>>>> From: Grainier Perera <gr...@gmail.com>
>>>>>>> Sent: Friday, May 22, 2020 11:58 AM
>>>>>>> To: dev@streampipes.apache.org
>>>>>>> Subject: Re: Data processor to evaluate JavaScript
>>>>>>> 
>>>>>>> Hi Dominik,
>>>>>>> 
>>>>>>> I've implemented the following[1]. However, when I try to install 
>>>>>>> the processor it gives me the following error. Any clue? I've used 
>>>>>>> CLI to
>>>>> start
>>>>>>> sp. So, maybe the snapshot docker image doesn't have the recent
>>>>> changes? If
>>>>>>> so what are the steps to build the images with changes in-place?
>>>>>>> 
>>>>>>> [1] 
>>>>>>> https://github.com/apache/incubator-streampipes-extensions/pull/15
>>>>>>> 
>>>>>>> ERROR: 
>>>>>>> https://streampipes.org/vocabulary/v1/UserDefinedOutputStrategy
>>>>>>> io.fogsy.empire.pinto.RDFMappingException: Could not create an 
>>>>>>> instance
>>>>> of
>>>>>>> class org.apache.streampipes.model.output.OutputStrategy, it does 
>>>>>>> not
>>>>> have
>>>>>>> a default constructor at
>>>>>>> io.fogsy.empire.pinto.RDFMapper.newInstance(RDFMapper.java:178)
>>>>>>> at io.fogsy.empire.pinto.RDFMapper.readValue(RDFMapper.java:261)
>>>>>>> at 
>>>>>>> io.fogsy.empire.pinto.RDFMapper.valueToObject(RDFMapper.java:695)
>>>>>>> at 
>>>>>>> io.fogsy.empire.pinto.RDFMapper.lambda$toObject$3(RDFMapper.java:38
>>>>>>> 8)
>>>>>>> at
>>>>>>> 
>>>>> io.fogsy.empire.pinto.RDFMapper$$Lambda$802/0000000000096040.apply(Un
>>>>> known
>>>>>>> Source)
>>>>>>> at
>>>>>>> 
>>>>> io.fogsy.empire.pinto.RDFMapper$$Lambda$803/0000000000096580.apply(Un
>>>>> known
>>>>>>> Source)
>>>>>>> at java.util.stream.ReferencePipeline$3$1.accept(Unknown Source) at 
>>>>>>> java.util.ArrayList$ArrayListSpliterator.forEachRemaining(Unknown
>>>>> Source)
>>>>>>> at java.util.stream.AbstractPipeline.copyInto(Unknown Source) at 
>>>>>>> java.util.stream.AbstractPipeline.wrapAndCopyInto(Unknown Source) 
>>>>>>> at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(Unknown
>>>>> Source) at
>>>>>>> java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(Unkn
>>>>>>> own
>>>>>>> Source)
>>>>>>> at java.util.stream.AbstractPipeline.evaluate(Unknown Source) at 
>>>>>>> java.util.stream.ReferencePipeline.forEach(Unknown Source) at
>>>>>>> io.fogsy.empire.pinto.RDFMapper.readValue(RDFMapper.java:299)
>>>>>>> at
>>>>>>> 
>>>>>>> 
>>>>> org.apache.streampipes.serializers.jsonld.JsonLdTransformer.fromJsonL
>>>>> d(JsonLdTransformer.java:100)
>>>>>>> at
>>>>>>> 
>>>>>>> 
>>>>> org.apache.streampipes.manager.verification.ElementVerifier.transform
>>>>> (ElementVerifier.java:169)
>>>>>>> at
>>>>>>> 
>>>>>>> 
>>>>> org.apache.streampipes.manager.verification.ElementVerifier.verifyAnd
>>>>> Add(ElementVerifier.java:84)
>>>>>>> at
>>>>>>> 
>>>>>>> 
>>>>> org.apache.streampipes.manager.operations.Operations.verifyAndAddElem
>>>>> ent(Operations.java:96)
>>>>>>> at
>>>>>>> 
>>>>>>> 
>>>>> org.apache.streampipes.manager.endpoint.EndpointItemParser.parseAndAd
>>>>> dEndpointItem(EndpointItemParser.java:37)
>>>>>>> at
>>>>>>> 
>>>>>>> 
>>>>> org.apache.streampipes.rest.impl.PipelineElementImport.verifyAndAddEl
>>>>> ement(PipelineElementImport.java:93)
>>>>>>> at
>>>>>>> 
>>>>>>> 
>>>>> org.apache.streampipes.rest.impl.PipelineElementImport.addElement(Pip
>>>>> elineElementImport.java:89)
>>>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at 
>>>>>>> sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
>>>>>>> 
>>>>>>> Thanks,
>>>>>>> Grainier.
>>>>>>> 
>>>>>>> On Thu, 21 May 2020 at 03:39, Dominik Riemer <ri...@apache.org> wrote:
>>>>>>> 
>>>>>>>> Hi Grainier,
>>>>>>>> 
>>>>>>>> sounds good!
>>>>>>>> I've just pushed a commit that provides an initial version of the 
>>>>>>>> user-defined output strategy. It lets you select the input schema 
>>>>>>>> from the incoming stream and add/remove event properties. There is 
>>>>>>>> one limitation, the component currently only works for flat event 
>>>>>>>> structures, but it shouldn't be a problem to have a more advanced 
>>>>>>>> schema editor for this output strategy soon (we already have this 
>>>>>>>> in the StreamPipes connect code, but unfortunately, the pipeline 
>>>>>>>> editor has yet to be migrated from AngularJS to Angular 2+ to 
>>>>>>>> reuse these components). This is something we plan to do within the next weeks.
>>>>>>>> 
>>>>>>>> I added an example how to use the strategy to the examples repo:
>>>>>>>> 
>>>>>>>> https://github.com/apache/incubator-streampipes-examples/blob/dev/
>>>>>>>> stre 
>>>>>>>> ampipes-pipeline-elements-examples-processors-jvm/src/main/java/or
>>>>>>>> g/ap 
>>>>>>>> ache/streampipes/pe/examples/jvm/staticproperty/CodeInputExampleCo
>>>>>>>> ntro
>>>>>>>> ller.java
>>>>>>>> 
>>>>>>>> Let me know what you think and if you need anything more/else!
>>>>>>>> 
>>>>>>>> Dominik
>>>>>>>> 
>>>>>>>> On 2020/05/19 17:14:21, Grainier Perera 
>>>>>>>> <gr...@gmail.com>
>>>>>>>> wrote:
>>>>>>>>> Hi Dominik,
>>>>>>>>> 
>>>>>>>>> Awesome. This is really cool & exactly what I was looking for. 
>>>>>>>>> I'll use this with my implementation, and keep this thread posted 
>>>>>>>>> with the
>>>>>>>> updates.
>>>>>>>>> ps: let us know when you add output strategy as well.
>>>>>>>>> 
>>>>>>>>> Thanks,
>>>>>>>>> Grainier.
>>>>>>>>> 
>>>>>>>>> On Tue, 19 May 2020 at 03:05, Dominik Riemer <ri...@apache.org>
>>>>>>> wrote:
>>>>>>>>> 
>>>>>>>>>> Hi Grainier,
>>>>>>>>>> 
>>>>>>>>>> I just pushed an initial version of the static property to add
>>>>>>>> javascript
>>>>>>>>>> code.
>>>>>>>>>> It's a simple codeblock with linting enabled and a (currently 
>>>>>>>>>> very
>>>>>>>> simple)
>>>>>>>>>> autocomplete feature that suggests existing properties from the 
>>>>>>>>>> input
>>>>>>>> event
>>>>>>>>>> (currently, only if you press ctrl-space directly after typing
>>>>>>> "event"
>>>>>>>>>> followed by a dot, I'll fix that soon ;-)
>>>>>>>>>> 
>>>>>>>>>> You can find an example how to use it here:
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>> https://github.com/apache/incubator-streampipes-examples/blob/dev/
>>>>>>>> stre 
>>>>>>>> ampipes-pipeline-elements-examples-processors-jvm/src/main/java/or
>>>>>>>> g/ap 
>>>>>>>> ache/streampipes/pe/examples/jvm/staticproperty/CodeInputExampleCo
>>>>>>>> ntro
>>>>>>>> ller.java
>>>>>>>>>> 
>>>>>>>>>> Feedback and ideas for improvement are welcome!
>>>>>>>>>> 
>>>>>>>>>> Dominik
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> On 2020/05/17 12:36:05, Grainier Perera 
>>>>>>>>>> <gr...@gmail.com>
>>>>>>>>>> wrote:
>>>>>>>>>>> Hi Dominik,
>>>>>>>>>>> 
>>>>>>>>>>> Thank you. I'll work on `Implement JS evaluator` [1].
>>>>>>>>>>> 
>>>>>>>>>>> [1] https://issues.apache.org/jira/browse/STREAMPIPES-135
>>>>>>>>>>> 
>>>>>>>>>>> Grainier.
>>>>>>>>>>> 
>>>>>>>>>>> On Sun, 17 May 2020 at 18:02, Dominik Riemer 
>>>>>>>>>>> <ri...@apache.org>
>>>>>>>> wrote:
>>>>>>>>>>> 
>>>>>>>>>>>> Hi Grainier,
>>>>>>>>>>>> 
>>>>>>>>>>>> great! I created an umbrella issue for that:
>>>>>>>>>>>> https://issues.apache.org/jira/browse/STREAMPIPES-132
>>>>>>>>>>>> 
>>>>>>>>>>>> I'll now start to implement the new static property for 
>>>>>>>>>>>> entering
>>>>>>>> code.
>>>>>>>>>>>> 
>>>>>>>>>>>> Dominik
>>>>>>>>>>>> 
>>>>>>>>>>>> On 2020/05/15 01:29:44, Grainier Perera 
>>>>>>>>>>>> <grainier.perera@gmail.com
>>>>>>>>> 
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>> 
>>>>>>>>>>>>> I'm +1 to the idea of `UserDefinedOutputStrategy`. As you 
>>>>>>>>>>>>> said,
>>>>>>>>>> that'll
>>>>>>>>>>>> be
>>>>>>>>>>>>> re-usable in the future processors as well. So let's proceed 
>>>>>>>>>>>>> with
>>>>>>>>>> that.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>> Grainier
>>>>>>>>>>>>> 
>>>>>>>>>>>>> On Fri, 15 May 2020 at 01:37, Dominik Riemer 
>>>>>>>>>>>>> <ri...@apache.org>
>>>>>>>>>> wrote:
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>> cool, that sounds very good!
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Concerning the output, I think there are two ways to achieve
>>>>>>>> this:
>>>>>>>>>>>>>> - A collection static property in conjunction with a runtime
>>>>>>>>>> resolvable
>>>>>>>>>>>>>> output strategy that creates the output schema based on the
>>>>>>>> input
>>>>>>>>>> from
>>>>>>>>>>>> the
>>>>>>>>>>>>>> collection
>>>>>>>>>>>>>> - Or we create a new output strategy, e.g., let's call it 
>>>>>>>>>>>>>> UserDefinedOutputStrategy, which would render a UI component
>>>>>>>> to let
>>>>>>>>>>>> users
>>>>>>>>>>>>>> add/remove/change event properties from the input schema.
>>>>>>>>>>>>>> I
>>>>>>>> also
>>>>>>>>>> like
>>>>>>>>>>>> the
>>>>>>>>>>>>>> idea of preselecting all input properties.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> I personally tend towards the second option as such a 
>>>>>>>>>>>>>> strategy
>>>>>>>>>> might be
>>>>>>>>>>>>>> useful for other pipeline elements as well and it would ease
>>>>>>>> the
>>>>>>>>>> model
>>>>>>>>>>>>>> definition. We could also extend this component in the 
>>>>>>>>>>>>>> future,
>>>>>>>>>> e.g., by
>>>>>>>>>>>>>> letting users enrich metadata such as measurement units
>>>>>>>> directly
>>>>>>>>>> in a
>>>>>>>>>>>>>> processor.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> But I'm also open to the first option and we can also just
>>>>>>>> start
>>>>>>>>>>>>>> implementing to see what works best.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> I'd also be happy to help with the implementation, e.g., I
>>>>>>>> could
>>>>>>>>>> work
>>>>>>>>>>>> on
>>>>>>>>>>>>>> the new static property for code input or the output
>>>>>>> strategy!
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Dominik
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> On 2020/05/14 11:59:36, Grainier Perera <
>>>>>>>> grainier.perera@gmail.com
>>>>>>>>>>> 
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> +1 to the idea of having a collection property containing 
>>>>>>>>>>>>>>> +the
>>>>>>>>>> fields
>>>>>>>>>>>> of
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>> output properties. Also, If we can show the user with
>>>>>>>> available
>>>>>>>>>>>> fields
>>>>>>>>>>>>>> with
>>>>>>>>>>>>>>> metadata (of input event) populated in that same collection
>>>>>>>>>> property,
>>>>>>>>>>>>>> then
>>>>>>>>>>>>>>> the user can add/remove/updates fields easily. Let's say 
>>>>>>>>>>>>>>> they
>>>>>>>>>> just
>>>>>>>>>>>> have
>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>> return a map containing all the fields they defined in 
>>>>>>>>>>>>>>> output
>>>>>>>>>>>> properties
>>>>>>>>>>>>>>> (i.e return {"existingId": 1, "existingTempInKelvin":
>>>>>>>>>>>>>>> 301,
>>>>>>>>>>>>>>> "newTempInCelsius": 28 } ).
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> [image: sample.png]
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>> Grainier.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> On Thu, 14 May 2020 at 12:33, Patrick Wiener <
>>>>>>>> wiener@apache.org>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> sounds reasonable.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> I share the thoughts of Philipp regarding the output
>>>>>>>> schema.
>>>>>>>>>> But
>>>>>>>>>>>> as you
>>>>>>>>>>>>>>>> said Grainier, we can also improve :)
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> btw: I really like the idea to also allow the more
>>>>>>>> technical
>>>>>>>>>> user
>>>>>>>>>>>> to
>>>>>>>>>>>>>>>> easily integrate their function snippets without having to 
>>>>>>>>>>>>>>>> care about the actual execution. This gets a more FaaS
>>>>>>>> like
>>>>>>>>>>>> feeling
>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>> streaming.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Patrick
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Am 14.05.2020 um 08:26 schrieb Philipp Zehnder <
>>>>>>>>>>>> zehnder@apache.org>:
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> it would be very useful to have such a processor.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> I like the ideas for 1 & 2 and I think we can do it this
>>>>>>>> way.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Actually I think that the definition of the output scheme
>>>>>>>>>> could
>>>>>>>>>>>> be
>>>>>>>>>>>>>> quite
>>>>>>>>>>>>>>>> tricky.
>>>>>>>>>>>>>>>>> Because users can do anything with the event.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> How about we provide an option property?
>>>>>>>>>>>>>>>>> Users can select whether they want to define the whole
>>>>>>>> event
>>>>>>>>>> or
>>>>>>>>>>>> just
>>>>>>>>>>>>>>>> append properties.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> For the configurations of the properties we could do
>>>>>>>>>> something
>>>>>>>>>>>>>> similar
>>>>>>>>>>>>>>>> then we have in the PLC4xS7Adapter.
>>>>>>>>>>>>>>>>> A collection property containing the fields of the output
>>>>>>>>>>>> properties.
>>>>>>>>>>>>>>>>> Therefore, we have to decide what information we have to
>>>>>>>>>>>> provide. I
>>>>>>>>>>>>>>>> think if we would support all options it becomes very
>>>>>>>> cluttered
>>>>>>>>>>>> for the
>>>>>>>>>>>>>>>> user.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> What do you think?
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Philipp
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> On 14. May 2020, at 06:55, Grainier Perera <
>>>>>>>>>>>>>> grainier.perera@gmail.com>
>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Hi Dominik,
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> I agree. We have to do some enhancements make this
>>>>>>>> processor
>>>>>>>>>>>> user
>>>>>>>>>>>>>>>> friendly.
>>>>>>>>>>>>>>>>>> Like you said it's nice to have; 1. A new static 
>>>>>>>>>>>>>>>>>> property that supports code
>>>>>>>> highlighting &
>>>>>>>>>>>> syntax
>>>>>>>>>>>>>>>>>> checking. We can simply do such client-side validations
>>>>>>>>>> using
>>>>>>>>>>>>>>>> highlight.js
>>>>>>>>>>>>>>>>>> (BSD) [1], jshint (MIT) [2], etc...
>>>>>>>>>>>>>>>>>> 2. A mechanism to show the user with available fields
>>>>>>>>>> they can
>>>>>>>>>>>>>> use to
>>>>>>>>>>>>>>>>>> write code (similar to what you've done with usable
>>>>>>>>>> templates in
>>>>>>>>>>>>>>>>>> EmailPublisher using HtmlInputParameter).
>>>>>>>>>>>>>>>>>> 3. A mechanism to map output schema.
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> I think 1 & 2 go together and for the 3rd requirement
>>>>>>>> given
>>>>>>>>>>>> that we
>>>>>>>>>>>>>> are
>>>>>>>>>>>>>>>>>> targetting a more technical user group for this
>>>>>>>> processor,
>>>>>>>>>> we
>>>>>>>>>>>> can
>>>>>>>>>>>>>> let
>>>>>>>>>>>>>>>> them
>>>>>>>>>>>>>>>>>> define the output. Otherwise, it would be quite
>>>>>>>> difficult to
>>>>>>>>>>>> derive
>>>>>>>>>>>>>>>> return
>>>>>>>>>>>>>>>>>> type from the JS. Moreover, I'm planning to use Java's
>>>>>>>>>> builtin
>>>>>>>>>>>>>>>> ScriptEngine
>>>>>>>>>>>>>>>>>> to eval. Of course, there'll be limitations. But, we can
>>>>>>>>>> always
>>>>>>>>>>>>>> improve
>>>>>>>>>>>>>>>>>> them :)
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> [1] https://highlightjs.org/ [2] https://jshint.com/
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>>>> Grainier.
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> On Thu, 14 May 2020 at 00:53, Dominik Riemer <
>>>>>>>>>> riemer@apache.org
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Hi Grainier,
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> in my opinion, this would definitely be a very
>>>>>>>> interesting
>>>>>>>>>>>>>> addition! We
>>>>>>>>>>>>>>>>>>> already briefly discussed such a feature in the past,
>>>>>>>>>> before we
>>>>>>>>>>>>>> went
>>>>>>>>>>>>>>>> to the
>>>>>>>>>>>>>>>>>>> ASF. I’d say that a JS evaluator would be
>>>>>>>>>>>>>>>>>>> definitely
>>>>>>>>>> useful to
>>>>>>>>>>>> a
>>>>>>>>>>>>>>>> slightly
>>>>>>>>>>>>>>>>>>> more technical target user group, but would
>>>>>>>>>>>>>>>>>>> provide
>>>>>>>> great
>>>>>>>>>>>>>> flexibility
>>>>>>>>>>>>>>>> for,
>>>>>>>>>>>>>>>>>>> e.g., data harmonization tasks. So let’s discuss
>>>>>>>>>>>>>>>>>>> what
>>>>>>>> is
>>>>>>>>>>>> needed to
>>>>>>>>>>>>>>>>>>> implement this! I’d guess that we (maybe) need to
>>>>>>>>>>>>>>>>>>> add
>>>>>>>> some
>>>>>>>>>>>>>>>> enhancements to
>>>>>>>>>>>>>>>>>>> the core to make this processor good from a
>>>>>>>>>>>>>>>>>>> usability
>>>>>>>>>>>> perspective.
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> From a conceptual point of view, I guess the JS
>>>>>>>> processor
>>>>>>>>>> would
>>>>>>>>>>>>>> have no
>>>>>>>>>>>>>>>>>>> specific input requirements and a single static
>>>>>>>> property
>>>>>>>>>> that
>>>>>>>>>>>>>> allows
>>>>>>>>>>>>>>>> users
>>>>>>>>>>>>>>>>>>> to enter the code (maybe it would be cool to add a
>>>>>>>>>>>>>>>>>>> new
>>>>>>>>>> static
>>>>>>>>>>>>>> property
>>>>>>>>>>>>>>>> here
>>>>>>>>>>>>>>>>>>> that supports code highlighting or even syntax
>>>>>>>> checking).
>>>>>>>>>> The
>>>>>>>>>>>> only
>>>>>>>>>>>>>> open
>>>>>>>>>>>>>>>>>>> issue I see is the output schema – we would
>>>>>>>>>>>>>>>>>>> somehow
>>>>>>>> need to
>>>>>>>>>>>>>> extract the
>>>>>>>>>>>>>>>>>>> output from the JS function. This could probably
>>>>>>>> partially
>>>>>>>>>> be
>>>>>>>>>>>> done
>>>>>>>>>>>>>>>> using
>>>>>>>>>>>>>>>>>>> the CustomTransformOutput (see
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>> http://streampipes.apache.org/docs/docs/dev-guide-output-strategie
>>>>>>>>>> s/),
>>>>>>>>>>>>>>>>>>> but we would somehow need to derive type
>>>>>>>>>>>>>>>>>>> information
>>>>>>>> from
>>>>>>>>>> the
>>>>>>>>>>>> JS
>>>>>>>>>>>>>>>> function
>>>>>>>>>>>>>>>>>>> or introduce a feature to let users define/refine
>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>> output
>>>>>>>>>>>>>> manually
>>>>>>>>>>>>>>>>>>> (e.g., to add semantic metadata).
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> What do you think would be best? And would you
>>>>>>>>>>>>>>>>>>> evaluate
>>>>>>>>>> the JS
>>>>>>>>>>>>>> code in
>>>>>>>>>>>>>>>>>>> Java or somehow else?
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Also, as I’ve just started to improve the SDK
>>>>>>>>>>>>>>>>>>> (e.g.,
>>>>>>>>>> supporting
>>>>>>>>>>>>>>>> optional
>>>>>>>>>>>>>>>>>>> static properties and an easier way to define and
>>>>>>>> extract
>>>>>>>>>> model
>>>>>>>>>>>>>>>>>>> parameters), we can collect your requirements for
>>>>>>>>>>>>>>>>>>> the
>>>>>>>> JS
>>>>>>>>>>>> processor
>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>> extend the SDK if needed, just say what you would
>>>>>>>>>>>>>>>>>>> like
>>>>>>>> to
>>>>>>>>>> have
>>>>>>>>>>>> 😉
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Dominik
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> From: Grainier Perera <gr...@gmail.com>
>>>>>>>>>>>>>>>>>>> Sent: Wednesday, May 13, 2020 2:41 PM
>>>>>>>>>>>>>>>>>>> To: dev@streampipes.apache.org
>>>>>>>>>>>>>>>>>>> Subject: Data processor to evaluate JavaScript
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Hi all,
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> I'm planning to implement a JavaScript eval data
>>>>>>>>>> processor. As
>>>>>>>>>>>> the
>>>>>>>>>>>>>> name
>>>>>>>>>>>>>>>>>>> suggests, users will be able to write some
>>>>>>>>>>>>>>>>>>> JavaScript
>>>>>>>> code
>>>>>>>>>>>> which
>>>>>>>>>>>>>> takes
>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>> an event (as a map), do some processing on the
>>>>>>>>>>>>>>>>>>> event,
>>>>>>>> and
>>>>>>>>>>>> return a
>>>>>>>>>>>>>> new
>>>>>>>>>>>>>>>> map
>>>>>>>>>>>>>>>>>>> which then gets converted to an sp-event.
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> example js:
>>>>>>>>>>>>>>>>>>> function process(event) {
>>>>>>>>>>>>>>>>>>> // do processing here.
>>>>>>>>>>>>>>>>>>> // return processed event.
>>>>>>>>>>>>>>>>>>> return {id: http://event.id, tempInCelsius:
>>>>>>>>>>>> (event.tempInKelvin
>>>>>>>>>>>>>> -
>>>>>>>>>>>>>>>>>>> 273.15)};
>>>>>>>>>>>>>>>>>>> };
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Will this be a good addition to pipeline-elements?
>>>>>>>> What do
>>>>>>>>>> you
>>>>>>>>>>>>>> think?
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>>>>> Grainier.
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>> 
>>> 
>>> 
>>> 
> 
>

Re: Data processor to evaluate JavaScript

Posted by Philipp Zehnder <ze...@apache.org>.

Hi Dominik,

unfortunately this was not the problem, but this could become a problem in the future ;)
The problem was the java version. So far we used Java 8 in the docker images, after updating it to Java 9 it worked.

So far I just changed the docker versions of the images using the JS processor and it should work now.
I saw that we use java 11 for the arm images. Which version should we use for x86? I think it makes sense to harmonize the base image for all containers.
So the question is, which base image should we use? Any suggestions?

Philipp

> On 15. Jun 2020, at 23:11, Dominik Riemer <ri...@apache.org> wrote:
> 
> Hi Philipp,
> 
> coming back to this (and with later I really didn't mean ten days, sorry for that) ;-)
> I could also reproduce this now - it seems to be only an issue when running in Docker.
> 
> When looking at the container logs when running the element in Docker, I get a NullPointerException in line 45 of JsEval (which could indicate that either engine is null or code is null).
> 
> So could this be an issue with the Java version we're using in the Docker images? The currently used version is AdoptOpenJDK 11.
> 
> I found a similar issue on so:
> https://stackoverflow.com/questions/60916424/getenginebynamejavascript-returning-null-on-java-11
> 
> Could this be the cause? 
> 
> Dominik
> 
> On 2020/06/05 09:02:10, "Dominik Riemer" <ri...@apache.org> wrote: 
>> Hi Philipp,
>> 
>> I can try to reproduce it later.
>> 
>> Dominik
>> 
>> -----Original Message-----
>> From: Philipp Zehnder <ze...@apache.org> 
>> Sent: Friday, June 5, 2020 11:01 AM
>> To: dev@streampipes.apache.org
>> Subject: Re: Data processor to evaluate JavaScript
>> 
>> Hi, 
>> 
>> I just tried it again and removed all docker images from the VM did a fresh pull.
>> Unfortunately, with the same result. I tried the processor in both containers (pipeline-elements-all-jvm / processors-enricher-jvm).
>> 
>> Does anyone else have an idea what the problem could be?
>> 
>> Philipp
>> 
>>> On 4. Jun 2020, at 12:00, Grainier Perera <gr...@apache.org> wrote:
>>> 
>>> Hmm, That's odd. It doesn't use any 3rd party libraries for the PE (we 
>>> use javax.script.*). However, I came across a similar situation with 
>>> this RDFMappingException[1]. The reason for that is JSEval use newly 
>>> introduced .requiredCodeblock() controller prop, and for some reason, 
>>> there's a cached Docker image without it. But when I cleaned that and 
>>> re-build, it worked. Could this be the reason?
>>> 
>>> [1] io.fogsy.empire.pinto.RDFMappingException: Could not create an 
>>> instance of class org.apache.streampipes.model.output.OutputStrategy, 
>>> it does not have a default constructor
>>> 
>>> Grainier Perera.
>>> 
>>> 
>>> On Thu, 4 Jun 2020 at 15:17, Philipp Zehnder <ze...@apache.org> wrote:
>>> 
>>>> Hi all,
>>>> 
>>>> I tried to use the Processor JavaScript Eval in a deployment of 
>>>> StreamPipes (Lite), running in Docker.
>>>> Unfortunately, I get an error in the UI when starting the pipeline. 
>>>> There is no specific error message, just an error that the JavaScript 
>>>> Eval did not start.
>>>> I also did not see anything in the logs. Did anyone else have a 
>>>> similar issue?
>>>> 
>>>> The processor runs in the container pipeline-elements-all-jvm using 
>>>> the CLI.
>>>> 
>>>> Running the same pipeline locally in my IDE works well, so I guess 
>>>> there is a problem when building the project.
>>>> Is it possible that some libraries are not bundled correctly?
>>>> 
>>>> Philipp
>>>> 
>>>>> On 22. May 2020, at 16:57, Grainier Perera 
>>>>> <gr...@gmail.com>
>>>> wrote:
>>>>> 
>>>>> Hi Dominik,
>>>>> 
>>>>> My bad. There were some cached images & that was the issue. Also 
>>>>> fixed
>>>> the
>>>>> PR as per your comments. Please review and merge [1].
>>>>> 
>>>>> [1] 
>>>>> https://github.com/apache/incubator-streampipes-extensions/pull/15
>>>>> 
>>>>> Thanks & Regards,
>>>>> Grainier
>>>>> 
>>>>> On Fri, 22 May 2020 at 16:25, Dominik Riemer <ri...@apache.org> wrote:
>>>>> 
>>>>>> Hi Grainier,
>>>>>> 
>>>>>> we have a Github Actions workflow [1] for core+extensions that 
>>>>>> usually builds and pushes the latest development containers for dev 
>>>>>> on each
>>>> push.
>>>>>> You are using the latest CLI from dev and the tmpl_env or .env 
>>>>>> version
>>>> is
>>>>>> set to 0.67.0-SNAPSHOT, right?
>>>>>> 
>>>>>> I just tested the JsEval processor (with CLI 0.67.0-SNAPSHOT, local 
>>>>>> backend with the latest changes from dev and your PR) and it works
>>>> great 😊
>>>>>> The only thing I had to change was a typo in line 59 (use outEvent
>>>> instead
>>>>>> of event).
>>>>>> 
>>>>>> In the controller, the supportedProtocols and supportedFormats are 
>>>>>> no longer needed, as they can now be declared for a whole module in 
>>>>>> the
>>>> Init
>>>>>> class (registerFormats/Protocols). I need to update this in the 
>>>>>> documentation.
>>>>>> 
>>>>>> Dominik
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>> https://github.com/apache/incubator-streampipes/blob/dev/.github/work
>>>> flows/build.yml
>>>>>> 
>>>>>> -----Original Message-----
>>>>>> From: Grainier Perera <gr...@gmail.com>
>>>>>> Sent: Friday, May 22, 2020 11:58 AM
>>>>>> To: dev@streampipes.apache.org
>>>>>> Subject: Re: Data processor to evaluate JavaScript
>>>>>> 
>>>>>> Hi Dominik,
>>>>>> 
>>>>>> I've implemented the following[1]. However, when I try to install 
>>>>>> the processor it gives me the following error. Any clue? I've used 
>>>>>> CLI to
>>>> start
>>>>>> sp. So, maybe the snapshot docker image doesn't have the recent
>>>> changes? If
>>>>>> so what are the steps to build the images with changes in-place?
>>>>>> 
>>>>>> [1] 
>>>>>> https://github.com/apache/incubator-streampipes-extensions/pull/15
>>>>>> 
>>>>>> ERROR: 
>>>>>> https://streampipes.org/vocabulary/v1/UserDefinedOutputStrategy
>>>>>> io.fogsy.empire.pinto.RDFMappingException: Could not create an 
>>>>>> instance
>>>> of
>>>>>> class org.apache.streampipes.model.output.OutputStrategy, it does 
>>>>>> not
>>>> have
>>>>>> a default constructor at
>>>>>> io.fogsy.empire.pinto.RDFMapper.newInstance(RDFMapper.java:178)
>>>>>> at io.fogsy.empire.pinto.RDFMapper.readValue(RDFMapper.java:261)
>>>>>> at 
>>>>>> io.fogsy.empire.pinto.RDFMapper.valueToObject(RDFMapper.java:695)
>>>>>> at 
>>>>>> io.fogsy.empire.pinto.RDFMapper.lambda$toObject$3(RDFMapper.java:38
>>>>>> 8)
>>>>>> at
>>>>>> 
>>>> io.fogsy.empire.pinto.RDFMapper$$Lambda$802/0000000000096040.apply(Un
>>>> known
>>>>>> Source)
>>>>>> at
>>>>>> 
>>>> io.fogsy.empire.pinto.RDFMapper$$Lambda$803/0000000000096580.apply(Un
>>>> known
>>>>>> Source)
>>>>>> at java.util.stream.ReferencePipeline$3$1.accept(Unknown Source) at 
>>>>>> java.util.ArrayList$ArrayListSpliterator.forEachRemaining(Unknown
>>>> Source)
>>>>>> at java.util.stream.AbstractPipeline.copyInto(Unknown Source) at 
>>>>>> java.util.stream.AbstractPipeline.wrapAndCopyInto(Unknown Source) 
>>>>>> at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(Unknown
>>>> Source) at
>>>>>> java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(Unkn
>>>>>> own
>>>>>> Source)
>>>>>> at java.util.stream.AbstractPipeline.evaluate(Unknown Source) at 
>>>>>> java.util.stream.ReferencePipeline.forEach(Unknown Source) at
>>>>>> io.fogsy.empire.pinto.RDFMapper.readValue(RDFMapper.java:299)
>>>>>> at
>>>>>> 
>>>>>> 
>>>> org.apache.streampipes.serializers.jsonld.JsonLdTransformer.fromJsonL
>>>> d(JsonLdTransformer.java:100)
>>>>>> at
>>>>>> 
>>>>>> 
>>>> org.apache.streampipes.manager.verification.ElementVerifier.transform
>>>> (ElementVerifier.java:169)
>>>>>> at
>>>>>> 
>>>>>> 
>>>> org.apache.streampipes.manager.verification.ElementVerifier.verifyAnd
>>>> Add(ElementVerifier.java:84)
>>>>>> at
>>>>>> 
>>>>>> 
>>>> org.apache.streampipes.manager.operations.Operations.verifyAndAddElem
>>>> ent(Operations.java:96)
>>>>>> at
>>>>>> 
>>>>>> 
>>>> org.apache.streampipes.manager.endpoint.EndpointItemParser.parseAndAd
>>>> dEndpointItem(EndpointItemParser.java:37)
>>>>>> at
>>>>>> 
>>>>>> 
>>>> org.apache.streampipes.rest.impl.PipelineElementImport.verifyAndAddEl
>>>> ement(PipelineElementImport.java:93)
>>>>>> at
>>>>>> 
>>>>>> 
>>>> org.apache.streampipes.rest.impl.PipelineElementImport.addElement(Pip
>>>> elineElementImport.java:89)
>>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at 
>>>>>> sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
>>>>>> 
>>>>>> Thanks,
>>>>>> Grainier.
>>>>>> 
>>>>>> On Thu, 21 May 2020 at 03:39, Dominik Riemer <ri...@apache.org> wrote:
>>>>>> 
>>>>>>> Hi Grainier,
>>>>>>> 
>>>>>>> sounds good!
>>>>>>> I've just pushed a commit that provides an initial version of the 
>>>>>>> user-defined output strategy. It lets you select the input schema 
>>>>>>> from the incoming stream and add/remove event properties. There is 
>>>>>>> one limitation, the component currently only works for flat event 
>>>>>>> structures, but it shouldn't be a problem to have a more advanced 
>>>>>>> schema editor for this output strategy soon (we already have this 
>>>>>>> in the StreamPipes connect code, but unfortunately, the pipeline 
>>>>>>> editor has yet to be migrated from AngularJS to Angular 2+ to 
>>>>>>> reuse these components). This is something we plan to do within the next weeks.
>>>>>>> 
>>>>>>> I added an example how to use the strategy to the examples repo:
>>>>>>> 
>>>>>>> https://github.com/apache/incubator-streampipes-examples/blob/dev/
>>>>>>> stre 
>>>>>>> ampipes-pipeline-elements-examples-processors-jvm/src/main/java/or
>>>>>>> g/ap 
>>>>>>> ache/streampipes/pe/examples/jvm/staticproperty/CodeInputExampleCo
>>>>>>> ntro
>>>>>>> ller.java
>>>>>>> 
>>>>>>> Let me know what you think and if you need anything more/else!
>>>>>>> 
>>>>>>> Dominik
>>>>>>> 
>>>>>>> On 2020/05/19 17:14:21, Grainier Perera 
>>>>>>> <gr...@gmail.com>
>>>>>>> wrote:
>>>>>>>> Hi Dominik,
>>>>>>>> 
>>>>>>>> Awesome. This is really cool & exactly what I was looking for. 
>>>>>>>> I'll use this with my implementation, and keep this thread posted 
>>>>>>>> with the
>>>>>>> updates.
>>>>>>>> ps: let us know when you add output strategy as well.
>>>>>>>> 
>>>>>>>> Thanks,
>>>>>>>> Grainier.
>>>>>>>> 
>>>>>>>> On Tue, 19 May 2020 at 03:05, Dominik Riemer <ri...@apache.org>
>>>>>> wrote:
>>>>>>>> 
>>>>>>>>> Hi Grainier,
>>>>>>>>> 
>>>>>>>>> I just pushed an initial version of the static property to add
>>>>>>> javascript
>>>>>>>>> code.
>>>>>>>>> It's a simple codeblock with linting enabled and a (currently 
>>>>>>>>> very
>>>>>>> simple)
>>>>>>>>> autocomplete feature that suggests existing properties from the 
>>>>>>>>> input
>>>>>>> event
>>>>>>>>> (currently, only if you press ctrl-space directly after typing
>>>>>> "event"
>>>>>>>>> followed by a dot, I'll fix that soon ;-)
>>>>>>>>> 
>>>>>>>>> You can find an example how to use it here:
>>>>>>>>> 
>>>>>>>>> 
>>>>>>> https://github.com/apache/incubator-streampipes-examples/blob/dev/
>>>>>>> stre 
>>>>>>> ampipes-pipeline-elements-examples-processors-jvm/src/main/java/or
>>>>>>> g/ap 
>>>>>>> ache/streampipes/pe/examples/jvm/staticproperty/CodeInputExampleCo
>>>>>>> ntro
>>>>>>> ller.java
>>>>>>>>> 
>>>>>>>>> Feedback and ideas for improvement are welcome!
>>>>>>>>> 
>>>>>>>>> Dominik
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> On 2020/05/17 12:36:05, Grainier Perera 
>>>>>>>>> <gr...@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>> Hi Dominik,
>>>>>>>>>> 
>>>>>>>>>> Thank you. I'll work on `Implement JS evaluator` [1].
>>>>>>>>>> 
>>>>>>>>>> [1] https://issues.apache.org/jira/browse/STREAMPIPES-135
>>>>>>>>>> 
>>>>>>>>>> Grainier.
>>>>>>>>>> 
>>>>>>>>>> On Sun, 17 May 2020 at 18:02, Dominik Riemer 
>>>>>>>>>> <ri...@apache.org>
>>>>>>> wrote:
>>>>>>>>>> 
>>>>>>>>>>> Hi Grainier,
>>>>>>>>>>> 
>>>>>>>>>>> great! I created an umbrella issue for that:
>>>>>>>>>>> https://issues.apache.org/jira/browse/STREAMPIPES-132
>>>>>>>>>>> 
>>>>>>>>>>> I'll now start to implement the new static property for 
>>>>>>>>>>> entering
>>>>>>> code.
>>>>>>>>>>> 
>>>>>>>>>>> Dominik
>>>>>>>>>>> 
>>>>>>>>>>> On 2020/05/15 01:29:44, Grainier Perera 
>>>>>>>>>>> <grainier.perera@gmail.com
>>>>>>>> 
>>>>>>>>>>> wrote:
>>>>>>>>>>>> Hi,
>>>>>>>>>>>> 
>>>>>>>>>>>> I'm +1 to the idea of `UserDefinedOutputStrategy`. As you 
>>>>>>>>>>>> said,
>>>>>>>>> that'll
>>>>>>>>>>> be
>>>>>>>>>>>> re-usable in the future processors as well. So let's proceed 
>>>>>>>>>>>> with
>>>>>>>>> that.
>>>>>>>>>>>> 
>>>>>>>>>>>> Regards,
>>>>>>>>>>>> Grainier
>>>>>>>>>>>> 
>>>>>>>>>>>> On Fri, 15 May 2020 at 01:37, Dominik Riemer 
>>>>>>>>>>>> <ri...@apache.org>
>>>>>>>>> wrote:
>>>>>>>>>>>> 
>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>> cool, that sounds very good!
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Concerning the output, I think there are two ways to achieve
>>>>>>> this:
>>>>>>>>>>>>> - A collection static property in conjunction with a runtime
>>>>>>>>> resolvable
>>>>>>>>>>>>> output strategy that creates the output schema based on the
>>>>>>> input
>>>>>>>>> from
>>>>>>>>>>> the
>>>>>>>>>>>>> collection
>>>>>>>>>>>>> - Or we create a new output strategy, e.g., let's call it 
>>>>>>>>>>>>> UserDefinedOutputStrategy, which would render a UI component
>>>>>>> to let
>>>>>>>>>>> users
>>>>>>>>>>>>> add/remove/change event properties from the input schema.
>>>>>>>>>>>>> I
>>>>>>> also
>>>>>>>>> like
>>>>>>>>>>> the
>>>>>>>>>>>>> idea of preselecting all input properties.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> I personally tend towards the second option as such a 
>>>>>>>>>>>>> strategy
>>>>>>>>> might be
>>>>>>>>>>>>> useful for other pipeline elements as well and it would ease
>>>>>>> the
>>>>>>>>> model
>>>>>>>>>>>>> definition. We could also extend this component in the 
>>>>>>>>>>>>> future,
>>>>>>>>> e.g., by
>>>>>>>>>>>>> letting users enrich metadata such as measurement units
>>>>>>> directly
>>>>>>>>> in a
>>>>>>>>>>>>> processor.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> But I'm also open to the first option and we can also just
>>>>>>> start
>>>>>>>>>>>>> implementing to see what works best.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> I'd also be happy to help with the implementation, e.g., I
>>>>>>> could
>>>>>>>>> work
>>>>>>>>>>> on
>>>>>>>>>>>>> the new static property for code input or the output
>>>>>> strategy!
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Dominik
>>>>>>>>>>>>> 
>>>>>>>>>>>>> On 2020/05/14 11:59:36, Grainier Perera <
>>>>>>> grainier.perera@gmail.com
>>>>>>>>>> 
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> +1 to the idea of having a collection property containing 
>>>>>>>>>>>>>> +the
>>>>>>>>> fields
>>>>>>>>>>> of
>>>>>>>>>>>>> the
>>>>>>>>>>>>>> output properties. Also, If we can show the user with
>>>>>>> available
>>>>>>>>>>> fields
>>>>>>>>>>>>> with
>>>>>>>>>>>>>> metadata (of input event) populated in that same collection
>>>>>>>>> property,
>>>>>>>>>>>>> then
>>>>>>>>>>>>>> the user can add/remove/updates fields easily. Let's say 
>>>>>>>>>>>>>> they
>>>>>>>>> just
>>>>>>>>>>> have
>>>>>>>>>>>>> to
>>>>>>>>>>>>>> return a map containing all the fields they defined in 
>>>>>>>>>>>>>> output
>>>>>>>>>>> properties
>>>>>>>>>>>>>> (i.e return {"existingId": 1, "existingTempInKelvin":
>>>>>>>>>>>>>> 301,
>>>>>>>>>>>>>> "newTempInCelsius": 28 } ).
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> [image: sample.png]
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>> Grainier.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> On Thu, 14 May 2020 at 12:33, Patrick Wiener <
>>>>>>> wiener@apache.org>
>>>>>>>>>>> wrote:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> sounds reasonable.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> I share the thoughts of Philipp regarding the output
>>>>>>> schema.
>>>>>>>>> But
>>>>>>>>>>> as you
>>>>>>>>>>>>>>> said Grainier, we can also improve :)
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> btw: I really like the idea to also allow the more
>>>>>>> technical
>>>>>>>>> user
>>>>>>>>>>> to
>>>>>>>>>>>>>>> easily integrate their function snippets without having to 
>>>>>>>>>>>>>>> care about the actual execution. This gets a more FaaS
>>>>>>> like
>>>>>>>>>>> feeling
>>>>>>>>>>>>> for
>>>>>>>>>>>>>>> streaming.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Patrick
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Am 14.05.2020 um 08:26 schrieb Philipp Zehnder <
>>>>>>>>>>> zehnder@apache.org>:
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> it would be very useful to have such a processor.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> I like the ideas for 1 & 2 and I think we can do it this
>>>>>>> way.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Actually I think that the definition of the output scheme
>>>>>>>>> could
>>>>>>>>>>> be
>>>>>>>>>>>>> quite
>>>>>>>>>>>>>>> tricky.
>>>>>>>>>>>>>>>> Because users can do anything with the event.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> How about we provide an option property?
>>>>>>>>>>>>>>>> Users can select whether they want to define the whole
>>>>>>> event
>>>>>>>>> or
>>>>>>>>>>> just
>>>>>>>>>>>>>>> append properties.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> For the configurations of the properties we could do
>>>>>>>>> something
>>>>>>>>>>>>> similar
>>>>>>>>>>>>>>> then we have in the PLC4xS7Adapter.
>>>>>>>>>>>>>>>> A collection property containing the fields of the output
>>>>>>>>>>> properties.
>>>>>>>>>>>>>>>> Therefore, we have to decide what information we have to
>>>>>>>>>>> provide. I
>>>>>>>>>>>>>>> think if we would support all options it becomes very
>>>>>>> cluttered
>>>>>>>>>>> for the
>>>>>>>>>>>>>>> user.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> What do you think?
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Philipp
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> On 14. May 2020, at 06:55, Grainier Perera <
>>>>>>>>>>>>> grainier.perera@gmail.com>
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Hi Dominik,
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> I agree. We have to do some enhancements make this
>>>>>>> processor
>>>>>>>>>>> user
>>>>>>>>>>>>>>> friendly.
>>>>>>>>>>>>>>>>> Like you said it's nice to have; 1. A new static 
>>>>>>>>>>>>>>>>> property that supports code
>>>>>>> highlighting &
>>>>>>>>>>> syntax
>>>>>>>>>>>>>>>>> checking. We can simply do such client-side validations
>>>>>>>>> using
>>>>>>>>>>>>>>> highlight.js
>>>>>>>>>>>>>>>>> (BSD) [1], jshint (MIT) [2], etc...
>>>>>>>>>>>>>>>>> 2. A mechanism to show the user with available fields
>>>>>>>>> they can
>>>>>>>>>>>>> use to
>>>>>>>>>>>>>>>>> write code (similar to what you've done with usable
>>>>>>>>> templates in
>>>>>>>>>>>>>>>>> EmailPublisher using HtmlInputParameter).
>>>>>>>>>>>>>>>>> 3. A mechanism to map output schema.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> I think 1 & 2 go together and for the 3rd requirement
>>>>>>> given
>>>>>>>>>>> that we
>>>>>>>>>>>>> are
>>>>>>>>>>>>>>>>> targetting a more technical user group for this
>>>>>>> processor,
>>>>>>>>> we
>>>>>>>>>>> can
>>>>>>>>>>>>> let
>>>>>>>>>>>>>>> them
>>>>>>>>>>>>>>>>> define the output. Otherwise, it would be quite
>>>>>>> difficult to
>>>>>>>>>>> derive
>>>>>>>>>>>>>>> return
>>>>>>>>>>>>>>>>> type from the JS. Moreover, I'm planning to use Java's
>>>>>>>>> builtin
>>>>>>>>>>>>>>> ScriptEngine
>>>>>>>>>>>>>>>>> to eval. Of course, there'll be limitations. But, we can
>>>>>>>>> always
>>>>>>>>>>>>> improve
>>>>>>>>>>>>>>>>> them :)
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> [1] https://highlightjs.org/ [2] https://jshint.com/
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>>> Grainier.
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>> On Thu, 14 May 2020 at 00:53, Dominik Riemer <
>>>>>>>>> riemer@apache.org
>>>>>>>>>>>> 
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Hi Grainier,
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> in my opinion, this would definitely be a very
>>>>>>> interesting
>>>>>>>>>>>>> addition! We
>>>>>>>>>>>>>>>>>> already briefly discussed such a feature in the past,
>>>>>>>>> before we
>>>>>>>>>>>>> went
>>>>>>>>>>>>>>> to the
>>>>>>>>>>>>>>>>>> ASF. I’d say that a JS evaluator would be
>>>>>>>>>>>>>>>>>> definitely
>>>>>>>>> useful to
>>>>>>>>>>> a
>>>>>>>>>>>>>>> slightly
>>>>>>>>>>>>>>>>>> more technical target user group, but would
>>>>>>>>>>>>>>>>>> provide
>>>>>>> great
>>>>>>>>>>>>> flexibility
>>>>>>>>>>>>>>> for,
>>>>>>>>>>>>>>>>>> e.g., data harmonization tasks. So let’s discuss
>>>>>>>>>>>>>>>>>> what
>>>>>>> is
>>>>>>>>>>> needed to
>>>>>>>>>>>>>>>>>> implement this! I’d guess that we (maybe) need to
>>>>>>>>>>>>>>>>>> add
>>>>>>> some
>>>>>>>>>>>>>>> enhancements to
>>>>>>>>>>>>>>>>>> the core to make this processor good from a
>>>>>>>>>>>>>>>>>> usability
>>>>>>>>>>> perspective.
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> From a conceptual point of view, I guess the JS
>>>>>>> processor
>>>>>>>>> would
>>>>>>>>>>>>> have no
>>>>>>>>>>>>>>>>>> specific input requirements and a single static
>>>>>>> property
>>>>>>>>> that
>>>>>>>>>>>>> allows
>>>>>>>>>>>>>>> users
>>>>>>>>>>>>>>>>>> to enter the code (maybe it would be cool to add a
>>>>>>>>>>>>>>>>>> new
>>>>>>>>> static
>>>>>>>>>>>>> property
>>>>>>>>>>>>>>> here
>>>>>>>>>>>>>>>>>> that supports code highlighting or even syntax
>>>>>>> checking).
>>>>>>>>> The
>>>>>>>>>>> only
>>>>>>>>>>>>> open
>>>>>>>>>>>>>>>>>> issue I see is the output schema – we would
>>>>>>>>>>>>>>>>>> somehow
>>>>>>> need to
>>>>>>>>>>>>> extract the
>>>>>>>>>>>>>>>>>> output from the JS function. This could probably
>>>>>>> partially
>>>>>>>>> be
>>>>>>>>>>> done
>>>>>>>>>>>>>>> using
>>>>>>>>>>>>>>>>>> the CustomTransformOutput (see
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>> http://streampipes.apache.org/docs/docs/dev-guide-output-strategie
>>>>>>>>> s/),
>>>>>>>>>>>>>>>>>> but we would somehow need to derive type
>>>>>>>>>>>>>>>>>> information
>>>>>>> from
>>>>>>>>> the
>>>>>>>>>>> JS
>>>>>>>>>>>>>>> function
>>>>>>>>>>>>>>>>>> or introduce a feature to let users define/refine
>>>>>>>>>>>>>>>>>> the
>>>>>>>>> output
>>>>>>>>>>>>> manually
>>>>>>>>>>>>>>>>>> (e.g., to add semantic metadata).
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> What do you think would be best? And would you
>>>>>>>>>>>>>>>>>> evaluate
>>>>>>>>> the JS
>>>>>>>>>>>>> code in
>>>>>>>>>>>>>>>>>> Java or somehow else?
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Also, as I’ve just started to improve the SDK
>>>>>>>>>>>>>>>>>> (e.g.,
>>>>>>>>> supporting
>>>>>>>>>>>>>>> optional
>>>>>>>>>>>>>>>>>> static properties and an easier way to define and
>>>>>>> extract
>>>>>>>>> model
>>>>>>>>>>>>>>>>>> parameters), we can collect your requirements for
>>>>>>>>>>>>>>>>>> the
>>>>>>> JS
>>>>>>>>>>> processor
>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>> extend the SDK if needed, just say what you would
>>>>>>>>>>>>>>>>>> like
>>>>>>> to
>>>>>>>>> have
>>>>>>>>>>> 😉
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Dominik
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> From: Grainier Perera <gr...@gmail.com>
>>>>>>>>>>>>>>>>>> Sent: Wednesday, May 13, 2020 2:41 PM
>>>>>>>>>>>>>>>>>> To: dev@streampipes.apache.org
>>>>>>>>>>>>>>>>>> Subject: Data processor to evaluate JavaScript
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Hi all,
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> I'm planning to implement a JavaScript eval data
>>>>>>>>> processor. As
>>>>>>>>>>> the
>>>>>>>>>>>>> name
>>>>>>>>>>>>>>>>>> suggests, users will be able to write some
>>>>>>>>>>>>>>>>>> JavaScript
>>>>>>> code
>>>>>>>>>>> which
>>>>>>>>>>>>> takes
>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>> an event (as a map), do some processing on the
>>>>>>>>>>>>>>>>>> event,
>>>>>>> and
>>>>>>>>>>> return a
>>>>>>>>>>>>> new
>>>>>>>>>>>>>>> map
>>>>>>>>>>>>>>>>>> which then gets converted to an sp-event.
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> example js:
>>>>>>>>>>>>>>>>>> function process(event) {
>>>>>>>>>>>>>>>>>> // do processing here.
>>>>>>>>>>>>>>>>>> // return processed event.
>>>>>>>>>>>>>>>>>> return {id: http://event.id, tempInCelsius:
>>>>>>>>>>> (event.tempInKelvin
>>>>>>>>>>>>> -
>>>>>>>>>>>>>>>>>> 273.15)};
>>>>>>>>>>>>>>>>>> };
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Will this be a good addition to pipeline-elements?
>>>>>>> What do
>>>>>>>>> you
>>>>>>>>>>>>> think?
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>>>> Grainier.
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>>> 
>>>> 
>>>> 
>>>> 
>> 
>> 
>> 
>>

Re: RE: Data processor to evaluate JavaScript

Posted by Dominik Riemer <ri...@apache.org>.

Hi Philipp,

coming back to this (and with later I really didn't mean ten days, sorry for that) ;-)
I could also reproduce this now - it seems to be only an issue when running in Docker.

When looking at the container logs when running the element in Docker, I get a NullPointerException in line 45 of JsEval (which could indicate that either engine is null or code is null).

So could this be an issue with the Java version we're using in the Docker images? The currently used version is AdoptOpenJDK 11.
 
I found a similar issue on so:
https://stackoverflow.com/questions/60916424/getenginebynamejavascript-returning-null-on-java-11

Could this be the cause? 

Dominik

On 2020/06/05 09:02:10, "Dominik Riemer" <ri...@apache.org> wrote: 
> Hi Philipp,
> 
> I can try to reproduce it later.
> 
> Dominik
> 
> -----Original Message-----
> From: Philipp Zehnder <ze...@apache.org> 
> Sent: Friday, June 5, 2020 11:01 AM
> To: dev@streampipes.apache.org
> Subject: Re: Data processor to evaluate JavaScript
> 
> Hi, 
> 
> I just tried it again and removed all docker images from the VM did a fresh pull.
> Unfortunately, with the same result. I tried the processor in both containers (pipeline-elements-all-jvm / processors-enricher-jvm).
> 
> Does anyone else have an idea what the problem could be?
> 
> Philipp
> 
> > On 4. Jun 2020, at 12:00, Grainier Perera <gr...@apache.org> wrote:
> > 
> > Hmm, That's odd. It doesn't use any 3rd party libraries for the PE (we 
> > use javax.script.*). However, I came across a similar situation with 
> > this RDFMappingException[1]. The reason for that is JSEval use newly 
> > introduced .requiredCodeblock() controller prop, and for some reason, 
> > there's a cached Docker image without it. But when I cleaned that and 
> > re-build, it worked. Could this be the reason?
> > 
> > [1] io.fogsy.empire.pinto.RDFMappingException: Could not create an 
> > instance of class org.apache.streampipes.model.output.OutputStrategy, 
> > it does not have a default constructor
> > 
> > Grainier Perera.
> > 
> > 
> > On Thu, 4 Jun 2020 at 15:17, Philipp Zehnder <ze...@apache.org> wrote:
> > 
> >> Hi all,
> >> 
> >> I tried to use the Processor JavaScript Eval in a deployment of 
> >> StreamPipes (Lite), running in Docker.
> >> Unfortunately, I get an error in the UI when starting the pipeline. 
> >> There is no specific error message, just an error that the JavaScript 
> >> Eval did not start.
> >> I also did not see anything in the logs. Did anyone else have a 
> >> similar issue?
> >> 
> >> The processor runs in the container pipeline-elements-all-jvm using 
> >> the CLI.
> >> 
> >> Running the same pipeline locally in my IDE works well, so I guess 
> >> there is a problem when building the project.
> >> Is it possible that some libraries are not bundled correctly?
> >> 
> >> Philipp
> >> 
> >>> On 22. May 2020, at 16:57, Grainier Perera 
> >>> <gr...@gmail.com>
> >> wrote:
> >>> 
> >>> Hi Dominik,
> >>> 
> >>> My bad. There were some cached images & that was the issue. Also 
> >>> fixed
> >> the
> >>> PR as per your comments. Please review and merge [1].
> >>> 
> >>> [1] 
> >>> https://github.com/apache/incubator-streampipes-extensions/pull/15
> >>> 
> >>> Thanks & Regards,
> >>> Grainier
> >>> 
> >>> On Fri, 22 May 2020 at 16:25, Dominik Riemer <ri...@apache.org> wrote:
> >>> 
> >>>> Hi Grainier,
> >>>> 
> >>>> we have a Github Actions workflow [1] for core+extensions that 
> >>>> usually builds and pushes the latest development containers for dev 
> >>>> on each
> >> push.
> >>>> You are using the latest CLI from dev and the tmpl_env or .env 
> >>>> version
> >> is
> >>>> set to 0.67.0-SNAPSHOT, right?
> >>>> 
> >>>> I just tested the JsEval processor (with CLI 0.67.0-SNAPSHOT, local 
> >>>> backend with the latest changes from dev and your PR) and it works
> >> great 😊
> >>>> The only thing I had to change was a typo in line 59 (use outEvent
> >> instead
> >>>> of event).
> >>>> 
> >>>> In the controller, the supportedProtocols and supportedFormats are 
> >>>> no longer needed, as they can now be declared for a whole module in 
> >>>> the
> >> Init
> >>>> class (registerFormats/Protocols). I need to update this in the 
> >>>> documentation.
> >>>> 
> >>>> Dominik
> >>>> 
> >>>> 
> >>>> 
> >>>> 
> >>>> 
> >>>> 
> >> https://github.com/apache/incubator-streampipes/blob/dev/.github/work
> >> flows/build.yml
> >>>> 
> >>>> -----Original Message-----
> >>>> From: Grainier Perera <gr...@gmail.com>
> >>>> Sent: Friday, May 22, 2020 11:58 AM
> >>>> To: dev@streampipes.apache.org
> >>>> Subject: Re: Data processor to evaluate JavaScript
> >>>> 
> >>>> Hi Dominik,
> >>>> 
> >>>> I've implemented the following[1]. However, when I try to install 
> >>>> the processor it gives me the following error. Any clue? I've used 
> >>>> CLI to
> >> start
> >>>> sp. So, maybe the snapshot docker image doesn't have the recent
> >> changes? If
> >>>> so what are the steps to build the images with changes in-place?
> >>>> 
> >>>> [1] 
> >>>> https://github.com/apache/incubator-streampipes-extensions/pull/15
> >>>> 
> >>>> ERROR: 
> >>>> https://streampipes.org/vocabulary/v1/UserDefinedOutputStrategy
> >>>> io.fogsy.empire.pinto.RDFMappingException: Could not create an 
> >>>> instance
> >> of
> >>>> class org.apache.streampipes.model.output.OutputStrategy, it does 
> >>>> not
> >> have
> >>>> a default constructor at
> >>>> io.fogsy.empire.pinto.RDFMapper.newInstance(RDFMapper.java:178)
> >>>> at io.fogsy.empire.pinto.RDFMapper.readValue(RDFMapper.java:261)
> >>>> at 
> >>>> io.fogsy.empire.pinto.RDFMapper.valueToObject(RDFMapper.java:695)
> >>>> at 
> >>>> io.fogsy.empire.pinto.RDFMapper.lambda$toObject$3(RDFMapper.java:38
> >>>> 8)
> >>>> at
> >>>> 
> >> io.fogsy.empire.pinto.RDFMapper$$Lambda$802/0000000000096040.apply(Un
> >> known
> >>>> Source)
> >>>> at
> >>>> 
> >> io.fogsy.empire.pinto.RDFMapper$$Lambda$803/0000000000096580.apply(Un
> >> known
> >>>> Source)
> >>>> at java.util.stream.ReferencePipeline$3$1.accept(Unknown Source) at 
> >>>> java.util.ArrayList$ArrayListSpliterator.forEachRemaining(Unknown
> >> Source)
> >>>> at java.util.stream.AbstractPipeline.copyInto(Unknown Source) at 
> >>>> java.util.stream.AbstractPipeline.wrapAndCopyInto(Unknown Source) 
> >>>> at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(Unknown
> >> Source) at
> >>>> java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(Unkn
> >>>> own
> >>>> Source)
> >>>> at java.util.stream.AbstractPipeline.evaluate(Unknown Source) at 
> >>>> java.util.stream.ReferencePipeline.forEach(Unknown Source) at
> >>>> io.fogsy.empire.pinto.RDFMapper.readValue(RDFMapper.java:299)
> >>>> at
> >>>> 
> >>>> 
> >> org.apache.streampipes.serializers.jsonld.JsonLdTransformer.fromJsonL
> >> d(JsonLdTransformer.java:100)
> >>>> at
> >>>> 
> >>>> 
> >> org.apache.streampipes.manager.verification.ElementVerifier.transform
> >> (ElementVerifier.java:169)
> >>>> at
> >>>> 
> >>>> 
> >> org.apache.streampipes.manager.verification.ElementVerifier.verifyAnd
> >> Add(ElementVerifier.java:84)
> >>>> at
> >>>> 
> >>>> 
> >> org.apache.streampipes.manager.operations.Operations.verifyAndAddElem
> >> ent(Operations.java:96)
> >>>> at
> >>>> 
> >>>> 
> >> org.apache.streampipes.manager.endpoint.EndpointItemParser.parseAndAd
> >> dEndpointItem(EndpointItemParser.java:37)
> >>>> at
> >>>> 
> >>>> 
> >> org.apache.streampipes.rest.impl.PipelineElementImport.verifyAndAddEl
> >> ement(PipelineElementImport.java:93)
> >>>> at
> >>>> 
> >>>> 
> >> org.apache.streampipes.rest.impl.PipelineElementImport.addElement(Pip
> >> elineElementImport.java:89)
> >>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at 
> >>>> sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
> >>>> 
> >>>> Thanks,
> >>>> Grainier.
> >>>> 
> >>>> On Thu, 21 May 2020 at 03:39, Dominik Riemer <ri...@apache.org> wrote:
> >>>> 
> >>>>> Hi Grainier,
> >>>>> 
> >>>>> sounds good!
> >>>>> I've just pushed a commit that provides an initial version of the 
> >>>>> user-defined output strategy. It lets you select the input schema 
> >>>>> from the incoming stream and add/remove event properties. There is 
> >>>>> one limitation, the component currently only works for flat event 
> >>>>> structures, but it shouldn't be a problem to have a more advanced 
> >>>>> schema editor for this output strategy soon (we already have this 
> >>>>> in the StreamPipes connect code, but unfortunately, the pipeline 
> >>>>> editor has yet to be migrated from AngularJS to Angular 2+ to 
> >>>>> reuse these components). This is something we plan to do within the next weeks.
> >>>>> 
> >>>>> I added an example how to use the strategy to the examples repo:
> >>>>> 
> >>>>> https://github.com/apache/incubator-streampipes-examples/blob/dev/
> >>>>> stre 
> >>>>> ampipes-pipeline-elements-examples-processors-jvm/src/main/java/or
> >>>>> g/ap 
> >>>>> ache/streampipes/pe/examples/jvm/staticproperty/CodeInputExampleCo
> >>>>> ntro
> >>>>> ller.java
> >>>>> 
> >>>>> Let me know what you think and if you need anything more/else!
> >>>>> 
> >>>>> Dominik
> >>>>> 
> >>>>> On 2020/05/19 17:14:21, Grainier Perera 
> >>>>> <gr...@gmail.com>
> >>>>> wrote:
> >>>>>> Hi Dominik,
> >>>>>> 
> >>>>>> Awesome. This is really cool & exactly what I was looking for. 
> >>>>>> I'll use this with my implementation, and keep this thread posted 
> >>>>>> with the
> >>>>> updates.
> >>>>>> ps: let us know when you add output strategy as well.
> >>>>>> 
> >>>>>> Thanks,
> >>>>>> Grainier.
> >>>>>> 
> >>>>>> On Tue, 19 May 2020 at 03:05, Dominik Riemer <ri...@apache.org>
> >>>> wrote:
> >>>>>> 
> >>>>>>> Hi Grainier,
> >>>>>>> 
> >>>>>>> I just pushed an initial version of the static property to add
> >>>>> javascript
> >>>>>>> code.
> >>>>>>> It's a simple codeblock with linting enabled and a (currently 
> >>>>>>> very
> >>>>> simple)
> >>>>>>> autocomplete feature that suggests existing properties from the 
> >>>>>>> input
> >>>>> event
> >>>>>>> (currently, only if you press ctrl-space directly after typing
> >>>> "event"
> >>>>>>> followed by a dot, I'll fix that soon ;-)
> >>>>>>> 
> >>>>>>> You can find an example how to use it here:
> >>>>>>> 
> >>>>>>> 
> >>>>> https://github.com/apache/incubator-streampipes-examples/blob/dev/
> >>>>> stre 
> >>>>> ampipes-pipeline-elements-examples-processors-jvm/src/main/java/or
> >>>>> g/ap 
> >>>>> ache/streampipes/pe/examples/jvm/staticproperty/CodeInputExampleCo
> >>>>> ntro
> >>>>> ller.java
> >>>>>>> 
> >>>>>>> Feedback and ideas for improvement are welcome!
> >>>>>>> 
> >>>>>>> Dominik
> >>>>>>> 
> >>>>>>> 
> >>>>>>> On 2020/05/17 12:36:05, Grainier Perera 
> >>>>>>> <gr...@gmail.com>
> >>>>>>> wrote:
> >>>>>>>> Hi Dominik,
> >>>>>>>> 
> >>>>>>>> Thank you. I'll work on `Implement JS evaluator` [1].
> >>>>>>>> 
> >>>>>>>> [1] https://issues.apache.org/jira/browse/STREAMPIPES-135
> >>>>>>>> 
> >>>>>>>> Grainier.
> >>>>>>>> 
> >>>>>>>> On Sun, 17 May 2020 at 18:02, Dominik Riemer 
> >>>>>>>> <ri...@apache.org>
> >>>>> wrote:
> >>>>>>>> 
> >>>>>>>>> Hi Grainier,
> >>>>>>>>> 
> >>>>>>>>> great! I created an umbrella issue for that:
> >>>>>>>>> https://issues.apache.org/jira/browse/STREAMPIPES-132
> >>>>>>>>> 
> >>>>>>>>> I'll now start to implement the new static property for 
> >>>>>>>>> entering
> >>>>> code.
> >>>>>>>>> 
> >>>>>>>>> Dominik
> >>>>>>>>> 
> >>>>>>>>> On 2020/05/15 01:29:44, Grainier Perera 
> >>>>>>>>> <grainier.perera@gmail.com
> >>>>>> 
> >>>>>>>>> wrote:
> >>>>>>>>>> Hi,
> >>>>>>>>>> 
> >>>>>>>>>> I'm +1 to the idea of `UserDefinedOutputStrategy`. As you 
> >>>>>>>>>> said,
> >>>>>>> that'll
> >>>>>>>>> be
> >>>>>>>>>> re-usable in the future processors as well. So let's proceed 
> >>>>>>>>>> with
> >>>>>>> that.
> >>>>>>>>>> 
> >>>>>>>>>> Regards,
> >>>>>>>>>> Grainier
> >>>>>>>>>> 
> >>>>>>>>>> On Fri, 15 May 2020 at 01:37, Dominik Riemer 
> >>>>>>>>>> <ri...@apache.org>
> >>>>>>> wrote:
> >>>>>>>>>> 
> >>>>>>>>>>> Hi,
> >>>>>>>>>>> cool, that sounds very good!
> >>>>>>>>>>> 
> >>>>>>>>>>> Concerning the output, I think there are two ways to achieve
> >>>>> this:
> >>>>>>>>>>> - A collection static property in conjunction with a runtime
> >>>>>>> resolvable
> >>>>>>>>>>> output strategy that creates the output schema based on the
> >>>>> input
> >>>>>>> from
> >>>>>>>>> the
> >>>>>>>>>>> collection
> >>>>>>>>>>> - Or we create a new output strategy, e.g., let's call it 
> >>>>>>>>>>> UserDefinedOutputStrategy, which would render a UI component
> >>>>> to let
> >>>>>>>>> users
> >>>>>>>>>>> add/remove/change event properties from the input schema.
> >>>>>>>>>>> I
> >>>>> also
> >>>>>>> like
> >>>>>>>>> the
> >>>>>>>>>>> idea of preselecting all input properties.
> >>>>>>>>>>> 
> >>>>>>>>>>> I personally tend towards the second option as such a 
> >>>>>>>>>>> strategy
> >>>>>>> might be
> >>>>>>>>>>> useful for other pipeline elements as well and it would ease
> >>>>> the
> >>>>>>> model
> >>>>>>>>>>> definition. We could also extend this component in the 
> >>>>>>>>>>> future,
> >>>>>>> e.g., by
> >>>>>>>>>>> letting users enrich metadata such as measurement units
> >>>>> directly
> >>>>>>> in a
> >>>>>>>>>>> processor.
> >>>>>>>>>>> 
> >>>>>>>>>>> But I'm also open to the first option and we can also just
> >>>>> start
> >>>>>>>>>>> implementing to see what works best.
> >>>>>>>>>>> 
> >>>>>>>>>>> I'd also be happy to help with the implementation, e.g., I
> >>>>> could
> >>>>>>> work
> >>>>>>>>> on
> >>>>>>>>>>> the new static property for code input or the output
> >>>> strategy!
> >>>>>>>>>>> 
> >>>>>>>>>>> Dominik
> >>>>>>>>>>> 
> >>>>>>>>>>> On 2020/05/14 11:59:36, Grainier Perera <
> >>>>> grainier.perera@gmail.com
> >>>>>>>> 
> >>>>>>>>>>> wrote:
> >>>>>>>>>>>> Hi,
> >>>>>>>>>>>> 
> >>>>>>>>>>>> +1 to the idea of having a collection property containing 
> >>>>>>>>>>>> +the
> >>>>>>> fields
> >>>>>>>>> of
> >>>>>>>>>>> the
> >>>>>>>>>>>> output properties. Also, If we can show the user with
> >>>>> available
> >>>>>>>>> fields
> >>>>>>>>>>> with
> >>>>>>>>>>>> metadata (of input event) populated in that same collection
> >>>>>>> property,
> >>>>>>>>>>> then
> >>>>>>>>>>>> the user can add/remove/updates fields easily. Let's say 
> >>>>>>>>>>>> they
> >>>>>>> just
> >>>>>>>>> have
> >>>>>>>>>>> to
> >>>>>>>>>>>> return a map containing all the fields they defined in 
> >>>>>>>>>>>> output
> >>>>>>>>> properties
> >>>>>>>>>>>> (i.e return {"existingId": 1, "existingTempInKelvin":
> >>>>>>>>>>>> 301,
> >>>>>>>>>>>> "newTempInCelsius": 28 } ).
> >>>>>>>>>>>> 
> >>>>>>>>>>>> [image: sample.png]
> >>>>>>>>>>>> 
> >>>>>>>>>>>> Regards,
> >>>>>>>>>>>> Grainier.
> >>>>>>>>>>>> 
> >>>>>>>>>>>> On Thu, 14 May 2020 at 12:33, Patrick Wiener <
> >>>>> wiener@apache.org>
> >>>>>>>>> wrote:
> >>>>>>>>>>>> 
> >>>>>>>>>>>>> Hi,
> >>>>>>>>>>>>> 
> >>>>>>>>>>>>> sounds reasonable.
> >>>>>>>>>>>>> 
> >>>>>>>>>>>>> I share the thoughts of Philipp regarding the output
> >>>>> schema.
> >>>>>>> But
> >>>>>>>>> as you
> >>>>>>>>>>>>> said Grainier, we can also improve :)
> >>>>>>>>>>>>> 
> >>>>>>>>>>>>> btw: I really like the idea to also allow the more
> >>>>> technical
> >>>>>>> user
> >>>>>>>>> to
> >>>>>>>>>>>>> easily integrate their function snippets without having to 
> >>>>>>>>>>>>> care about the actual execution. This gets a more FaaS
> >>>>> like
> >>>>>>>>> feeling
> >>>>>>>>>>> for
> >>>>>>>>>>>>> streaming.
> >>>>>>>>>>>>> 
> >>>>>>>>>>>>> Patrick
> >>>>>>>>>>>>> 
> >>>>>>>>>>>>>> Am 14.05.2020 um 08:26 schrieb Philipp Zehnder <
> >>>>>>>>> zehnder@apache.org>:
> >>>>>>>>>>>>>> 
> >>>>>>>>>>>>>> Hi,
> >>>>>>>>>>>>>> 
> >>>>>>>>>>>>>> it would be very useful to have such a processor.
> >>>>>>>>>>>>>> 
> >>>>>>>>>>>>>> I like the ideas for 1 & 2 and I think we can do it this
> >>>>> way.
> >>>>>>>>>>>>>> 
> >>>>>>>>>>>>>> Actually I think that the definition of the output scheme
> >>>>>>> could
> >>>>>>>>> be
> >>>>>>>>>>> quite
> >>>>>>>>>>>>> tricky.
> >>>>>>>>>>>>>> Because users can do anything with the event.
> >>>>>>>>>>>>>> 
> >>>>>>>>>>>>>> How about we provide an option property?
> >>>>>>>>>>>>>> Users can select whether they want to define the whole
> >>>>> event
> >>>>>>> or
> >>>>>>>>> just
> >>>>>>>>>>>>> append properties.
> >>>>>>>>>>>>>> 
> >>>>>>>>>>>>>> For the configurations of the properties we could do
> >>>>>>> something
> >>>>>>>>>>> similar
> >>>>>>>>>>>>> then we have in the PLC4xS7Adapter.
> >>>>>>>>>>>>>> A collection property containing the fields of the output
> >>>>>>>>> properties.
> >>>>>>>>>>>>>> Therefore, we have to decide what information we have to
> >>>>>>>>> provide. I
> >>>>>>>>>>>>> think if we would support all options it becomes very
> >>>>> cluttered
> >>>>>>>>> for the
> >>>>>>>>>>>>> user.
> >>>>>>>>>>>>>> 
> >>>>>>>>>>>>>> What do you think?
> >>>>>>>>>>>>>> 
> >>>>>>>>>>>>>> Philipp
> >>>>>>>>>>>>>> 
> >>>>>>>>>>>>>> 
> >>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>> On 14. May 2020, at 06:55, Grainier Perera <
> >>>>>>>>>>> grainier.perera@gmail.com>
> >>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>> Hi Dominik,
> >>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>> I agree. We have to do some enhancements make this
> >>>>> processor
> >>>>>>>>> user
> >>>>>>>>>>>>> friendly.
> >>>>>>>>>>>>>>> Like you said it's nice to have; 1. A new static 
> >>>>>>>>>>>>>>> property that supports code
> >>>>> highlighting &
> >>>>>>>>> syntax
> >>>>>>>>>>>>>>> checking. We can simply do such client-side validations
> >>>>>>> using
> >>>>>>>>>>>>> highlight.js
> >>>>>>>>>>>>>>> (BSD) [1], jshint (MIT) [2], etc...
> >>>>>>>>>>>>>>> 2. A mechanism to show the user with available fields
> >>>>>>> they can
> >>>>>>>>>>> use to
> >>>>>>>>>>>>>>> write code (similar to what you've done with usable
> >>>>>>> templates in
> >>>>>>>>>>>>>>> EmailPublisher using HtmlInputParameter).
> >>>>>>>>>>>>>>> 3. A mechanism to map output schema.
> >>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>> I think 1 & 2 go together and for the 3rd requirement
> >>>>> given
> >>>>>>>>> that we
> >>>>>>>>>>> are
> >>>>>>>>>>>>>>> targetting a more technical user group for this
> >>>>> processor,
> >>>>>>> we
> >>>>>>>>> can
> >>>>>>>>>>> let
> >>>>>>>>>>>>> them
> >>>>>>>>>>>>>>> define the output. Otherwise, it would be quite
> >>>>> difficult to
> >>>>>>>>> derive
> >>>>>>>>>>>>> return
> >>>>>>>>>>>>>>> type from the JS. Moreover, I'm planning to use Java's
> >>>>>>> builtin
> >>>>>>>>>>>>> ScriptEngine
> >>>>>>>>>>>>>>> to eval. Of course, there'll be limitations. But, we can
> >>>>>>> always
> >>>>>>>>>>> improve
> >>>>>>>>>>>>>>> them :)
> >>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>> [1] https://highlightjs.org/ [2] https://jshint.com/
> >>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>> Regards,
> >>>>>>>>>>>>>>> Grainier.
> >>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>> On Thu, 14 May 2020 at 00:53, Dominik Riemer <
> >>>>>>> riemer@apache.org
> >>>>>>>>>> 
> >>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>> Hi Grainier,
> >>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>> in my opinion, this would definitely be a very
> >>>>> interesting
> >>>>>>>>>>> addition! We
> >>>>>>>>>>>>>>>> already briefly discussed such a feature in the past,
> >>>>>>> before we
> >>>>>>>>>>> went
> >>>>>>>>>>>>> to the
> >>>>>>>>>>>>>>>> ASF. I’d say that a JS evaluator would be
> >>>>>>>>>>>>>>>> definitely
> >>>>>>> useful to
> >>>>>>>>> a
> >>>>>>>>>>>>> slightly
> >>>>>>>>>>>>>>>> more technical target user group, but would
> >>>>>>>>>>>>>>>> provide
> >>>>> great
> >>>>>>>>>>> flexibility
> >>>>>>>>>>>>> for,
> >>>>>>>>>>>>>>>> e.g., data harmonization tasks. So let’s discuss
> >>>>>>>>>>>>>>>> what
> >>>>> is
> >>>>>>>>> needed to
> >>>>>>>>>>>>>>>> implement this! I’d guess that we (maybe) need to
> >>>>>>>>>>>>>>>> add
> >>>>> some
> >>>>>>>>>>>>> enhancements to
> >>>>>>>>>>>>>>>> the core to make this processor good from a
> >>>>>>>>>>>>>>>> usability
> >>>>>>>>> perspective.
> >>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>> From a conceptual point of view, I guess the JS
> >>>>> processor
> >>>>>>> would
> >>>>>>>>>>> have no
> >>>>>>>>>>>>>>>> specific input requirements and a single static
> >>>>> property
> >>>>>>> that
> >>>>>>>>>>> allows
> >>>>>>>>>>>>> users
> >>>>>>>>>>>>>>>> to enter the code (maybe it would be cool to add a
> >>>>>>>>>>>>>>>> new
> >>>>>>> static
> >>>>>>>>>>> property
> >>>>>>>>>>>>> here
> >>>>>>>>>>>>>>>> that supports code highlighting or even syntax
> >>>>> checking).
> >>>>>>> The
> >>>>>>>>> only
> >>>>>>>>>>> open
> >>>>>>>>>>>>>>>> issue I see is the output schema – we would
> >>>>>>>>>>>>>>>> somehow
> >>>>> need to
> >>>>>>>>>>> extract the
> >>>>>>>>>>>>>>>> output from the JS function. This could probably
> >>>>> partially
> >>>>>>> be
> >>>>>>>>> done
> >>>>>>>>>>>>> using
> >>>>>>>>>>>>>>>> the CustomTransformOutput (see
> >>>>>>>>>>>>>>>> 
> >>>>>>>>>>> 
> >>>>>>> http://streampipes.apache.org/docs/docs/dev-guide-output-strategie
> >>>>>>> s/),
> >>>>>>>>>>>>>>>> but we would somehow need to derive type
> >>>>>>>>>>>>>>>> information
> >>>>> from
> >>>>>>> the
> >>>>>>>>> JS
> >>>>>>>>>>>>> function
> >>>>>>>>>>>>>>>> or introduce a feature to let users define/refine
> >>>>>>>>>>>>>>>> the
> >>>>>>> output
> >>>>>>>>>>> manually
> >>>>>>>>>>>>>>>> (e.g., to add semantic metadata).
> >>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>> What do you think would be best? And would you
> >>>>>>>>>>>>>>>> evaluate
> >>>>>>> the JS
> >>>>>>>>>>> code in
> >>>>>>>>>>>>>>>> Java or somehow else?
> >>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>> Also, as I’ve just started to improve the SDK
> >>>>>>>>>>>>>>>> (e.g.,
> >>>>>>> supporting
> >>>>>>>>>>>>> optional
> >>>>>>>>>>>>>>>> static properties and an easier way to define and
> >>>>> extract
> >>>>>>> model
> >>>>>>>>>>>>>>>> parameters), we can collect your requirements for
> >>>>>>>>>>>>>>>> the
> >>>>> JS
> >>>>>>>>> processor
> >>>>>>>>>>> and
> >>>>>>>>>>>>>>>> extend the SDK if needed, just say what you would
> >>>>>>>>>>>>>>>> like
> >>>>> to
> >>>>>>> have
> >>>>>>>>> 😉
> >>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>> Dominik
> >>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>> From: Grainier Perera <gr...@gmail.com>
> >>>>>>>>>>>>>>>> Sent: Wednesday, May 13, 2020 2:41 PM
> >>>>>>>>>>>>>>>> To: dev@streampipes.apache.org
> >>>>>>>>>>>>>>>> Subject: Data processor to evaluate JavaScript
> >>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>> Hi all,
> >>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>> I'm planning to implement a JavaScript eval data
> >>>>>>> processor. As
> >>>>>>>>> the
> >>>>>>>>>>> name
> >>>>>>>>>>>>>>>> suggests, users will be able to write some
> >>>>>>>>>>>>>>>> JavaScript
> >>>>> code
> >>>>>>>>> which
> >>>>>>>>>>> takes
> >>>>>>>>>>>>> in
> >>>>>>>>>>>>>>>> an event (as a map), do some processing on the
> >>>>>>>>>>>>>>>> event,
> >>>>> and
> >>>>>>>>> return a
> >>>>>>>>>>> new
> >>>>>>>>>>>>> map
> >>>>>>>>>>>>>>>> which then gets converted to an sp-event.
> >>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>> example js:
> >>>>>>>>>>>>>>>> function process(event) {
> >>>>>>>>>>>>>>>> // do processing here.
> >>>>>>>>>>>>>>>> // return processed event.
> >>>>>>>>>>>>>>>> return {id: http://event.id, tempInCelsius:
> >>>>>>>>> (event.tempInKelvin
> >>>>>>>>>>> -
> >>>>>>>>>>>>>>>> 273.15)};
> >>>>>>>>>>>>>>>> };
> >>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>> Will this be a good addition to pipeline-elements?
> >>>>> What do
> >>>>>>> you
> >>>>>>>>>>> think?
> >>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>> Regards,
> >>>>>>>>>>>>>>>> Grainier.
> >>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>> 
> >>>>>>>>>>>>>> 
> >>>>>>>>>>>>> 
> >>>>>>>>>>>>> 
> >>>>>>>>>>>> 
> >>>>>>>>>>> 
> >>>>>>>>>> 
> >>>>>>>>> 
> >>>>>>>> 
> >>>>>>> 
> >>>>>> 
> >>>>> 
> >>>> 
> >>>> 
> >> 
> >> 
> >> 
> 
> 
> 
>

RE: Data processor to evaluate JavaScript

Posted by Dominik Riemer <ri...@apache.org>.

Hi Philipp,

I can try to reproduce it later.

Dominik

-----Original Message-----
From: Philipp Zehnder <ze...@apache.org> 
Sent: Friday, June 5, 2020 11:01 AM
To: dev@streampipes.apache.org
Subject: Re: Data processor to evaluate JavaScript

Hi, 

I just tried it again and removed all docker images from the VM did a fresh pull.
Unfortunately, with the same result. I tried the processor in both containers (pipeline-elements-all-jvm / processors-enricher-jvm).

Does anyone else have an idea what the problem could be?

Philipp

> On 4. Jun 2020, at 12:00, Grainier Perera <gr...@apache.org> wrote:
> 
> Hmm, That's odd. It doesn't use any 3rd party libraries for the PE (we 
> use javax.script.*). However, I came across a similar situation with 
> this RDFMappingException[1]. The reason for that is JSEval use newly 
> introduced .requiredCodeblock() controller prop, and for some reason, 
> there's a cached Docker image without it. But when I cleaned that and 
> re-build, it worked. Could this be the reason?
> 
> [1] io.fogsy.empire.pinto.RDFMappingException: Could not create an 
> instance of class org.apache.streampipes.model.output.OutputStrategy, 
> it does not have a default constructor
> 
> Grainier Perera.
> 
> 
> On Thu, 4 Jun 2020 at 15:17, Philipp Zehnder <ze...@apache.org> wrote:
> 
>> Hi all,
>> 
>> I tried to use the Processor JavaScript Eval in a deployment of 
>> StreamPipes (Lite), running in Docker.
>> Unfortunately, I get an error in the UI when starting the pipeline. 
>> There is no specific error message, just an error that the JavaScript 
>> Eval did not start.
>> I also did not see anything in the logs. Did anyone else have a 
>> similar issue?
>> 
>> The processor runs in the container pipeline-elements-all-jvm using 
>> the CLI.
>> 
>> Running the same pipeline locally in my IDE works well, so I guess 
>> there is a problem when building the project.
>> Is it possible that some libraries are not bundled correctly?
>> 
>> Philipp
>> 
>>> On 22. May 2020, at 16:57, Grainier Perera 
>>> <gr...@gmail.com>
>> wrote:
>>> 
>>> Hi Dominik,
>>> 
>>> My bad. There were some cached images & that was the issue. Also 
>>> fixed
>> the
>>> PR as per your comments. Please review and merge [1].
>>> 
>>> [1] 
>>> https://github.com/apache/incubator-streampipes-extensions/pull/15
>>> 
>>> Thanks & Regards,
>>> Grainier
>>> 
>>> On Fri, 22 May 2020 at 16:25, Dominik Riemer <ri...@apache.org> wrote:
>>> 
>>>> Hi Grainier,
>>>> 
>>>> we have a Github Actions workflow [1] for core+extensions that 
>>>> usually builds and pushes the latest development containers for dev 
>>>> on each
>> push.
>>>> You are using the latest CLI from dev and the tmpl_env or .env 
>>>> version
>> is
>>>> set to 0.67.0-SNAPSHOT, right?
>>>> 
>>>> I just tested the JsEval processor (with CLI 0.67.0-SNAPSHOT, local 
>>>> backend with the latest changes from dev and your PR) and it works
>> great 😊
>>>> The only thing I had to change was a typo in line 59 (use outEvent
>> instead
>>>> of event).
>>>> 
>>>> In the controller, the supportedProtocols and supportedFormats are 
>>>> no longer needed, as they can now be declared for a whole module in 
>>>> the
>> Init
>>>> class (registerFormats/Protocols). I need to update this in the 
>>>> documentation.
>>>> 
>>>> Dominik
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>> https://github.com/apache/incubator-streampipes/blob/dev/.github/work
>> flows/build.yml
>>>> 
>>>> -----Original Message-----
>>>> From: Grainier Perera <gr...@gmail.com>
>>>> Sent: Friday, May 22, 2020 11:58 AM
>>>> To: dev@streampipes.apache.org
>>>> Subject: Re: Data processor to evaluate JavaScript
>>>> 
>>>> Hi Dominik,
>>>> 
>>>> I've implemented the following[1]. However, when I try to install 
>>>> the processor it gives me the following error. Any clue? I've used 
>>>> CLI to
>> start
>>>> sp. So, maybe the snapshot docker image doesn't have the recent
>> changes? If
>>>> so what are the steps to build the images with changes in-place?
>>>> 
>>>> [1] 
>>>> https://github.com/apache/incubator-streampipes-extensions/pull/15
>>>> 
>>>> ERROR: 
>>>> https://streampipes.org/vocabulary/v1/UserDefinedOutputStrategy
>>>> io.fogsy.empire.pinto.RDFMappingException: Could not create an 
>>>> instance
>> of
>>>> class org.apache.streampipes.model.output.OutputStrategy, it does 
>>>> not
>> have
>>>> a default constructor at
>>>> io.fogsy.empire.pinto.RDFMapper.newInstance(RDFMapper.java:178)
>>>> at io.fogsy.empire.pinto.RDFMapper.readValue(RDFMapper.java:261)
>>>> at 
>>>> io.fogsy.empire.pinto.RDFMapper.valueToObject(RDFMapper.java:695)
>>>> at 
>>>> io.fogsy.empire.pinto.RDFMapper.lambda$toObject$3(RDFMapper.java:38
>>>> 8)
>>>> at
>>>> 
>> io.fogsy.empire.pinto.RDFMapper$$Lambda$802/0000000000096040.apply(Un
>> known
>>>> Source)
>>>> at
>>>> 
>> io.fogsy.empire.pinto.RDFMapper$$Lambda$803/0000000000096580.apply(Un
>> known
>>>> Source)
>>>> at java.util.stream.ReferencePipeline$3$1.accept(Unknown Source) at 
>>>> java.util.ArrayList$ArrayListSpliterator.forEachRemaining(Unknown
>> Source)
>>>> at java.util.stream.AbstractPipeline.copyInto(Unknown Source) at 
>>>> java.util.stream.AbstractPipeline.wrapAndCopyInto(Unknown Source) 
>>>> at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(Unknown
>> Source) at
>>>> java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(Unkn
>>>> own
>>>> Source)
>>>> at java.util.stream.AbstractPipeline.evaluate(Unknown Source) at 
>>>> java.util.stream.ReferencePipeline.forEach(Unknown Source) at
>>>> io.fogsy.empire.pinto.RDFMapper.readValue(RDFMapper.java:299)
>>>> at
>>>> 
>>>> 
>> org.apache.streampipes.serializers.jsonld.JsonLdTransformer.fromJsonL
>> d(JsonLdTransformer.java:100)
>>>> at
>>>> 
>>>> 
>> org.apache.streampipes.manager.verification.ElementVerifier.transform
>> (ElementVerifier.java:169)
>>>> at
>>>> 
>>>> 
>> org.apache.streampipes.manager.verification.ElementVerifier.verifyAnd
>> Add(ElementVerifier.java:84)
>>>> at
>>>> 
>>>> 
>> org.apache.streampipes.manager.operations.Operations.verifyAndAddElem
>> ent(Operations.java:96)
>>>> at
>>>> 
>>>> 
>> org.apache.streampipes.manager.endpoint.EndpointItemParser.parseAndAd
>> dEndpointItem(EndpointItemParser.java:37)
>>>> at
>>>> 
>>>> 
>> org.apache.streampipes.rest.impl.PipelineElementImport.verifyAndAddEl
>> ement(PipelineElementImport.java:93)
>>>> at
>>>> 
>>>> 
>> org.apache.streampipes.rest.impl.PipelineElementImport.addElement(Pip
>> elineElementImport.java:89)
>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at 
>>>> sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
>>>> 
>>>> Thanks,
>>>> Grainier.
>>>> 
>>>> On Thu, 21 May 2020 at 03:39, Dominik Riemer <ri...@apache.org> wrote:
>>>> 
>>>>> Hi Grainier,
>>>>> 
>>>>> sounds good!
>>>>> I've just pushed a commit that provides an initial version of the 
>>>>> user-defined output strategy. It lets you select the input schema 
>>>>> from the incoming stream and add/remove event properties. There is 
>>>>> one limitation, the component currently only works for flat event 
>>>>> structures, but it shouldn't be a problem to have a more advanced 
>>>>> schema editor for this output strategy soon (we already have this 
>>>>> in the StreamPipes connect code, but unfortunately, the pipeline 
>>>>> editor has yet to be migrated from AngularJS to Angular 2+ to 
>>>>> reuse these components). This is something we plan to do within the next weeks.
>>>>> 
>>>>> I added an example how to use the strategy to the examples repo:
>>>>> 
>>>>> https://github.com/apache/incubator-streampipes-examples/blob/dev/
>>>>> stre 
>>>>> ampipes-pipeline-elements-examples-processors-jvm/src/main/java/or
>>>>> g/ap 
>>>>> ache/streampipes/pe/examples/jvm/staticproperty/CodeInputExampleCo
>>>>> ntro
>>>>> ller.java
>>>>> 
>>>>> Let me know what you think and if you need anything more/else!
>>>>> 
>>>>> Dominik
>>>>> 
>>>>> On 2020/05/19 17:14:21, Grainier Perera 
>>>>> <gr...@gmail.com>
>>>>> wrote:
>>>>>> Hi Dominik,
>>>>>> 
>>>>>> Awesome. This is really cool & exactly what I was looking for. 
>>>>>> I'll use this with my implementation, and keep this thread posted 
>>>>>> with the
>>>>> updates.
>>>>>> ps: let us know when you add output strategy as well.
>>>>>> 
>>>>>> Thanks,
>>>>>> Grainier.
>>>>>> 
>>>>>> On Tue, 19 May 2020 at 03:05, Dominik Riemer <ri...@apache.org>
>>>> wrote:
>>>>>> 
>>>>>>> Hi Grainier,
>>>>>>> 
>>>>>>> I just pushed an initial version of the static property to add
>>>>> javascript
>>>>>>> code.
>>>>>>> It's a simple codeblock with linting enabled and a (currently 
>>>>>>> very
>>>>> simple)
>>>>>>> autocomplete feature that suggests existing properties from the 
>>>>>>> input
>>>>> event
>>>>>>> (currently, only if you press ctrl-space directly after typing
>>>> "event"
>>>>>>> followed by a dot, I'll fix that soon ;-)
>>>>>>> 
>>>>>>> You can find an example how to use it here:
>>>>>>> 
>>>>>>> 
>>>>> https://github.com/apache/incubator-streampipes-examples/blob/dev/
>>>>> stre 
>>>>> ampipes-pipeline-elements-examples-processors-jvm/src/main/java/or
>>>>> g/ap 
>>>>> ache/streampipes/pe/examples/jvm/staticproperty/CodeInputExampleCo
>>>>> ntro
>>>>> ller.java
>>>>>>> 
>>>>>>> Feedback and ideas for improvement are welcome!
>>>>>>> 
>>>>>>> Dominik
>>>>>>> 
>>>>>>> 
>>>>>>> On 2020/05/17 12:36:05, Grainier Perera 
>>>>>>> <gr...@gmail.com>
>>>>>>> wrote:
>>>>>>>> Hi Dominik,
>>>>>>>> 
>>>>>>>> Thank you. I'll work on `Implement JS evaluator` [1].
>>>>>>>> 
>>>>>>>> [1] https://issues.apache.org/jira/browse/STREAMPIPES-135
>>>>>>>> 
>>>>>>>> Grainier.
>>>>>>>> 
>>>>>>>> On Sun, 17 May 2020 at 18:02, Dominik Riemer 
>>>>>>>> <ri...@apache.org>
>>>>> wrote:
>>>>>>>> 
>>>>>>>>> Hi Grainier,
>>>>>>>>> 
>>>>>>>>> great! I created an umbrella issue for that:
>>>>>>>>> https://issues.apache.org/jira/browse/STREAMPIPES-132
>>>>>>>>> 
>>>>>>>>> I'll now start to implement the new static property for 
>>>>>>>>> entering
>>>>> code.
>>>>>>>>> 
>>>>>>>>> Dominik
>>>>>>>>> 
>>>>>>>>> On 2020/05/15 01:29:44, Grainier Perera 
>>>>>>>>> <grainier.perera@gmail.com
>>>>>> 
>>>>>>>>> wrote:
>>>>>>>>>> Hi,
>>>>>>>>>> 
>>>>>>>>>> I'm +1 to the idea of `UserDefinedOutputStrategy`. As you 
>>>>>>>>>> said,
>>>>>>> that'll
>>>>>>>>> be
>>>>>>>>>> re-usable in the future processors as well. So let's proceed 
>>>>>>>>>> with
>>>>>>> that.
>>>>>>>>>> 
>>>>>>>>>> Regards,
>>>>>>>>>> Grainier
>>>>>>>>>> 
>>>>>>>>>> On Fri, 15 May 2020 at 01:37, Dominik Riemer 
>>>>>>>>>> <ri...@apache.org>
>>>>>>> wrote:
>>>>>>>>>> 
>>>>>>>>>>> Hi,
>>>>>>>>>>> cool, that sounds very good!
>>>>>>>>>>> 
>>>>>>>>>>> Concerning the output, I think there are two ways to achieve
>>>>> this:
>>>>>>>>>>> - A collection static property in conjunction with a runtime
>>>>>>> resolvable
>>>>>>>>>>> output strategy that creates the output schema based on the
>>>>> input
>>>>>>> from
>>>>>>>>> the
>>>>>>>>>>> collection
>>>>>>>>>>> - Or we create a new output strategy, e.g., let's call it 
>>>>>>>>>>> UserDefinedOutputStrategy, which would render a UI component
>>>>> to let
>>>>>>>>> users
>>>>>>>>>>> add/remove/change event properties from the input schema.
>>>>>>>>>>> I
>>>>> also
>>>>>>> like
>>>>>>>>> the
>>>>>>>>>>> idea of preselecting all input properties.
>>>>>>>>>>> 
>>>>>>>>>>> I personally tend towards the second option as such a 
>>>>>>>>>>> strategy
>>>>>>> might be
>>>>>>>>>>> useful for other pipeline elements as well and it would ease
>>>>> the
>>>>>>> model
>>>>>>>>>>> definition. We could also extend this component in the 
>>>>>>>>>>> future,
>>>>>>> e.g., by
>>>>>>>>>>> letting users enrich metadata such as measurement units
>>>>> directly
>>>>>>> in a
>>>>>>>>>>> processor.
>>>>>>>>>>> 
>>>>>>>>>>> But I'm also open to the first option and we can also just
>>>>> start
>>>>>>>>>>> implementing to see what works best.
>>>>>>>>>>> 
>>>>>>>>>>> I'd also be happy to help with the implementation, e.g., I
>>>>> could
>>>>>>> work
>>>>>>>>> on
>>>>>>>>>>> the new static property for code input or the output
>>>> strategy!
>>>>>>>>>>> 
>>>>>>>>>>> Dominik
>>>>>>>>>>> 
>>>>>>>>>>> On 2020/05/14 11:59:36, Grainier Perera <
>>>>> grainier.perera@gmail.com
>>>>>>>> 
>>>>>>>>>>> wrote:
>>>>>>>>>>>> Hi,
>>>>>>>>>>>> 
>>>>>>>>>>>> +1 to the idea of having a collection property containing 
>>>>>>>>>>>> +the
>>>>>>> fields
>>>>>>>>> of
>>>>>>>>>>> the
>>>>>>>>>>>> output properties. Also, If we can show the user with
>>>>> available
>>>>>>>>> fields
>>>>>>>>>>> with
>>>>>>>>>>>> metadata (of input event) populated in that same collection
>>>>>>> property,
>>>>>>>>>>> then
>>>>>>>>>>>> the user can add/remove/updates fields easily. Let's say 
>>>>>>>>>>>> they
>>>>>>> just
>>>>>>>>> have
>>>>>>>>>>> to
>>>>>>>>>>>> return a map containing all the fields they defined in 
>>>>>>>>>>>> output
>>>>>>>>> properties
>>>>>>>>>>>> (i.e return {"existingId": 1, "existingTempInKelvin":
>>>>>>>>>>>> 301,
>>>>>>>>>>>> "newTempInCelsius": 28 } ).
>>>>>>>>>>>> 
>>>>>>>>>>>> [image: sample.png]
>>>>>>>>>>>> 
>>>>>>>>>>>> Regards,
>>>>>>>>>>>> Grainier.
>>>>>>>>>>>> 
>>>>>>>>>>>> On Thu, 14 May 2020 at 12:33, Patrick Wiener <
>>>>> wiener@apache.org>
>>>>>>>>> wrote:
>>>>>>>>>>>> 
>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>> 
>>>>>>>>>>>>> sounds reasonable.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> I share the thoughts of Philipp regarding the output
>>>>> schema.
>>>>>>> But
>>>>>>>>> as you
>>>>>>>>>>>>> said Grainier, we can also improve :)
>>>>>>>>>>>>> 
>>>>>>>>>>>>> btw: I really like the idea to also allow the more
>>>>> technical
>>>>>>> user
>>>>>>>>> to
>>>>>>>>>>>>> easily integrate their function snippets without having to 
>>>>>>>>>>>>> care about the actual execution. This gets a more FaaS
>>>>> like
>>>>>>>>> feeling
>>>>>>>>>>> for
>>>>>>>>>>>>> streaming.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Patrick
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Am 14.05.2020 um 08:26 schrieb Philipp Zehnder <
>>>>>>>>> zehnder@apache.org>:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> it would be very useful to have such a processor.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> I like the ideas for 1 & 2 and I think we can do it this
>>>>> way.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Actually I think that the definition of the output scheme
>>>>>>> could
>>>>>>>>> be
>>>>>>>>>>> quite
>>>>>>>>>>>>> tricky.
>>>>>>>>>>>>>> Because users can do anything with the event.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> How about we provide an option property?
>>>>>>>>>>>>>> Users can select whether they want to define the whole
>>>>> event
>>>>>>> or
>>>>>>>>> just
>>>>>>>>>>>>> append properties.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> For the configurations of the properties we could do
>>>>>>> something
>>>>>>>>>>> similar
>>>>>>>>>>>>> then we have in the PLC4xS7Adapter.
>>>>>>>>>>>>>> A collection property containing the fields of the output
>>>>>>>>> properties.
>>>>>>>>>>>>>> Therefore, we have to decide what information we have to
>>>>>>>>> provide. I
>>>>>>>>>>>>> think if we would support all options it becomes very
>>>>> cluttered
>>>>>>>>> for the
>>>>>>>>>>>>> user.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> What do you think?
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Philipp
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> On 14. May 2020, at 06:55, Grainier Perera <
>>>>>>>>>>> grainier.perera@gmail.com>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Hi Dominik,
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> I agree. We have to do some enhancements make this
>>>>> processor
>>>>>>>>> user
>>>>>>>>>>>>> friendly.
>>>>>>>>>>>>>>> Like you said it's nice to have; 1. A new static 
>>>>>>>>>>>>>>> property that supports code
>>>>> highlighting &
>>>>>>>>> syntax
>>>>>>>>>>>>>>> checking. We can simply do such client-side validations
>>>>>>> using
>>>>>>>>>>>>> highlight.js
>>>>>>>>>>>>>>> (BSD) [1], jshint (MIT) [2], etc...
>>>>>>>>>>>>>>> 2. A mechanism to show the user with available fields
>>>>>>> they can
>>>>>>>>>>> use to
>>>>>>>>>>>>>>> write code (similar to what you've done with usable
>>>>>>> templates in
>>>>>>>>>>>>>>> EmailPublisher using HtmlInputParameter).
>>>>>>>>>>>>>>> 3. A mechanism to map output schema.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> I think 1 & 2 go together and for the 3rd requirement
>>>>> given
>>>>>>>>> that we
>>>>>>>>>>> are
>>>>>>>>>>>>>>> targetting a more technical user group for this
>>>>> processor,
>>>>>>> we
>>>>>>>>> can
>>>>>>>>>>> let
>>>>>>>>>>>>> them
>>>>>>>>>>>>>>> define the output. Otherwise, it would be quite
>>>>> difficult to
>>>>>>>>> derive
>>>>>>>>>>>>> return
>>>>>>>>>>>>>>> type from the JS. Moreover, I'm planning to use Java's
>>>>>>> builtin
>>>>>>>>>>>>> ScriptEngine
>>>>>>>>>>>>>>> to eval. Of course, there'll be limitations. But, we can
>>>>>>> always
>>>>>>>>>>> improve
>>>>>>>>>>>>>>> them :)
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> [1] https://highlightjs.org/ [2] https://jshint.com/
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>> Grainier.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> On Thu, 14 May 2020 at 00:53, Dominik Riemer <
>>>>>>> riemer@apache.org
>>>>>>>>>> 
>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Hi Grainier,
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> in my opinion, this would definitely be a very
>>>>> interesting
>>>>>>>>>>> addition! We
>>>>>>>>>>>>>>>> already briefly discussed such a feature in the past,
>>>>>>> before we
>>>>>>>>>>> went
>>>>>>>>>>>>> to the
>>>>>>>>>>>>>>>> ASF. I’d say that a JS evaluator would be
>>>>>>>>>>>>>>>> definitely
>>>>>>> useful to
>>>>>>>>> a
>>>>>>>>>>>>> slightly
>>>>>>>>>>>>>>>> more technical target user group, but would
>>>>>>>>>>>>>>>> provide
>>>>> great
>>>>>>>>>>> flexibility
>>>>>>>>>>>>> for,
>>>>>>>>>>>>>>>> e.g., data harmonization tasks. So let’s discuss
>>>>>>>>>>>>>>>> what
>>>>> is
>>>>>>>>> needed to
>>>>>>>>>>>>>>>> implement this! I’d guess that we (maybe) need to
>>>>>>>>>>>>>>>> add
>>>>> some
>>>>>>>>>>>>> enhancements to
>>>>>>>>>>>>>>>> the core to make this processor good from a
>>>>>>>>>>>>>>>> usability
>>>>>>>>> perspective.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> From a conceptual point of view, I guess the JS
>>>>> processor
>>>>>>> would
>>>>>>>>>>> have no
>>>>>>>>>>>>>>>> specific input requirements and a single static
>>>>> property
>>>>>>> that
>>>>>>>>>>> allows
>>>>>>>>>>>>> users
>>>>>>>>>>>>>>>> to enter the code (maybe it would be cool to add a
>>>>>>>>>>>>>>>> new
>>>>>>> static
>>>>>>>>>>> property
>>>>>>>>>>>>> here
>>>>>>>>>>>>>>>> that supports code highlighting or even syntax
>>>>> checking).
>>>>>>> The
>>>>>>>>> only
>>>>>>>>>>> open
>>>>>>>>>>>>>>>> issue I see is the output schema – we would
>>>>>>>>>>>>>>>> somehow
>>>>> need to
>>>>>>>>>>> extract the
>>>>>>>>>>>>>>>> output from the JS function. This could probably
>>>>> partially
>>>>>>> be
>>>>>>>>> done
>>>>>>>>>>>>> using
>>>>>>>>>>>>>>>> the CustomTransformOutput (see
>>>>>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>> http://streampipes.apache.org/docs/docs/dev-guide-output-strategie
>>>>>>> s/),
>>>>>>>>>>>>>>>> but we would somehow need to derive type
>>>>>>>>>>>>>>>> information
>>>>> from
>>>>>>> the
>>>>>>>>> JS
>>>>>>>>>>>>> function
>>>>>>>>>>>>>>>> or introduce a feature to let users define/refine
>>>>>>>>>>>>>>>> the
>>>>>>> output
>>>>>>>>>>> manually
>>>>>>>>>>>>>>>> (e.g., to add semantic metadata).
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> What do you think would be best? And would you
>>>>>>>>>>>>>>>> evaluate
>>>>>>> the JS
>>>>>>>>>>> code in
>>>>>>>>>>>>>>>> Java or somehow else?
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Also, as I’ve just started to improve the SDK
>>>>>>>>>>>>>>>> (e.g.,
>>>>>>> supporting
>>>>>>>>>>>>> optional
>>>>>>>>>>>>>>>> static properties and an easier way to define and
>>>>> extract
>>>>>>> model
>>>>>>>>>>>>>>>> parameters), we can collect your requirements for
>>>>>>>>>>>>>>>> the
>>>>> JS
>>>>>>>>> processor
>>>>>>>>>>> and
>>>>>>>>>>>>>>>> extend the SDK if needed, just say what you would
>>>>>>>>>>>>>>>> like
>>>>> to
>>>>>>> have
>>>>>>>>> 😉
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Dominik
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> From: Grainier Perera <gr...@gmail.com>
>>>>>>>>>>>>>>>> Sent: Wednesday, May 13, 2020 2:41 PM
>>>>>>>>>>>>>>>> To: dev@streampipes.apache.org
>>>>>>>>>>>>>>>> Subject: Data processor to evaluate JavaScript
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Hi all,
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> I'm planning to implement a JavaScript eval data
>>>>>>> processor. As
>>>>>>>>> the
>>>>>>>>>>> name
>>>>>>>>>>>>>>>> suggests, users will be able to write some
>>>>>>>>>>>>>>>> JavaScript
>>>>> code
>>>>>>>>> which
>>>>>>>>>>> takes
>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>> an event (as a map), do some processing on the
>>>>>>>>>>>>>>>> event,
>>>>> and
>>>>>>>>> return a
>>>>>>>>>>> new
>>>>>>>>>>>>> map
>>>>>>>>>>>>>>>> which then gets converted to an sp-event.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> example js:
>>>>>>>>>>>>>>>> function process(event) {
>>>>>>>>>>>>>>>> // do processing here.
>>>>>>>>>>>>>>>> // return processed event.
>>>>>>>>>>>>>>>> return {id: http://event.id, tempInCelsius:
>>>>>>>>> (event.tempInKelvin
>>>>>>>>>>> -
>>>>>>>>>>>>>>>> 273.15)};
>>>>>>>>>>>>>>>> };
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Will this be a good addition to pipeline-elements?
>>>>> What do
>>>>>>> you
>>>>>>>>>>> think?
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>> Grainier.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>>>> 
>> 
>> 
>>

Re: Data processor to evaluate JavaScript

Posted by Philipp Zehnder <ze...@apache.org>.

Hi, 

I just tried it again and removed all docker images from the VM did a fresh pull.
Unfortunately, with the same result. I tried the processor in both containers (pipeline-elements-all-jvm / processors-enricher-jvm).

Does anyone else have an idea what the problem could be?

Philipp

> On 4. Jun 2020, at 12:00, Grainier Perera <gr...@apache.org> wrote:
> 
> Hmm, That's odd. It doesn't use any 3rd party libraries for the PE (we use
> javax.script.*). However, I came across a similar situation with
> this RDFMappingException[1]. The reason for that is JSEval use newly
> introduced .requiredCodeblock() controller prop, and for some reason,
> there's a cached Docker image without it. But when I cleaned that and
> re-build, it worked. Could this be the reason?
> 
> [1] io.fogsy.empire.pinto.RDFMappingException: Could not create an instance
> of class org.apache.streampipes.model.output.OutputStrategy, it does not
> have a default constructor
> 
> Grainier Perera.
> 
> 
> On Thu, 4 Jun 2020 at 15:17, Philipp Zehnder <ze...@apache.org> wrote:
> 
>> Hi all,
>> 
>> I tried to use the Processor JavaScript Eval in a deployment of
>> StreamPipes (Lite), running in Docker.
>> Unfortunately, I get an error in the UI when starting the pipeline. There
>> is no specific error message, just an error that the JavaScript Eval did
>> not start.
>> I also did not see anything in the logs. Did anyone else have a similar
>> issue?
>> 
>> The processor runs in the container pipeline-elements-all-jvm using the
>> CLI.
>> 
>> Running the same pipeline locally in my IDE works well, so I guess there
>> is a problem when building the project.
>> Is it possible that some libraries are not bundled correctly?
>> 
>> Philipp
>> 
>>> On 22. May 2020, at 16:57, Grainier Perera <gr...@gmail.com>
>> wrote:
>>> 
>>> Hi Dominik,
>>> 
>>> My bad. There were some cached images & that was the issue. Also fixed
>> the
>>> PR as per your comments. Please review and merge [1].
>>> 
>>> [1] https://github.com/apache/incubator-streampipes-extensions/pull/15
>>> 
>>> Thanks & Regards,
>>> Grainier
>>> 
>>> On Fri, 22 May 2020 at 16:25, Dominik Riemer <ri...@apache.org> wrote:
>>> 
>>>> Hi Grainier,
>>>> 
>>>> we have a Github Actions workflow [1] for core+extensions that usually
>>>> builds and pushes the latest development containers for dev on each
>> push.
>>>> You are using the latest CLI from dev and the tmpl_env or .env version
>> is
>>>> set to 0.67.0-SNAPSHOT, right?
>>>> 
>>>> I just tested the JsEval processor (with CLI 0.67.0-SNAPSHOT, local
>>>> backend with the latest changes from dev and your PR) and it works
>> great 😊
>>>> The only thing I had to change was a typo in line 59 (use outEvent
>> instead
>>>> of event).
>>>> 
>>>> In the controller, the supportedProtocols and supportedFormats are no
>>>> longer needed, as they can now be declared for a whole module in the
>> Init
>>>> class (registerFormats/Protocols). I need to update this in the
>>>> documentation.
>>>> 
>>>> Dominik
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>> https://github.com/apache/incubator-streampipes/blob/dev/.github/workflows/build.yml
>>>> 
>>>> -----Original Message-----
>>>> From: Grainier Perera <gr...@gmail.com>
>>>> Sent: Friday, May 22, 2020 11:58 AM
>>>> To: dev@streampipes.apache.org
>>>> Subject: Re: Data processor to evaluate JavaScript
>>>> 
>>>> Hi Dominik,
>>>> 
>>>> I've implemented the following[1]. However, when I try to install the
>>>> processor it gives me the following error. Any clue? I've used CLI to
>> start
>>>> sp. So, maybe the snapshot docker image doesn't have the recent
>> changes? If
>>>> so what are the steps to build the images with changes in-place?
>>>> 
>>>> [1] https://github.com/apache/incubator-streampipes-extensions/pull/15
>>>> 
>>>> ERROR: https://streampipes.org/vocabulary/v1/UserDefinedOutputStrategy
>>>> io.fogsy.empire.pinto.RDFMappingException: Could not create an instance
>> of
>>>> class org.apache.streampipes.model.output.OutputStrategy, it does not
>> have
>>>> a default constructor at
>>>> io.fogsy.empire.pinto.RDFMapper.newInstance(RDFMapper.java:178)
>>>> at io.fogsy.empire.pinto.RDFMapper.readValue(RDFMapper.java:261)
>>>> at io.fogsy.empire.pinto.RDFMapper.valueToObject(RDFMapper.java:695)
>>>> at io.fogsy.empire.pinto.RDFMapper.lambda$toObject$3(RDFMapper.java:388)
>>>> at
>>>> 
>> io.fogsy.empire.pinto.RDFMapper$$Lambda$802/0000000000096040.apply(Unknown
>>>> Source)
>>>> at
>>>> 
>> io.fogsy.empire.pinto.RDFMapper$$Lambda$803/0000000000096580.apply(Unknown
>>>> Source)
>>>> at java.util.stream.ReferencePipeline$3$1.accept(Unknown Source) at
>>>> java.util.ArrayList$ArrayListSpliterator.forEachRemaining(Unknown
>> Source)
>>>> at java.util.stream.AbstractPipeline.copyInto(Unknown Source) at
>>>> java.util.stream.AbstractPipeline.wrapAndCopyInto(Unknown Source) at
>>>> java.util.stream.ForEachOps$ForEachOp.evaluateSequential(Unknown
>> Source) at
>>>> java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(Unknown
>>>> Source)
>>>> at java.util.stream.AbstractPipeline.evaluate(Unknown Source) at
>>>> java.util.stream.ReferencePipeline.forEach(Unknown Source) at
>>>> io.fogsy.empire.pinto.RDFMapper.readValue(RDFMapper.java:299)
>>>> at
>>>> 
>>>> 
>> org.apache.streampipes.serializers.jsonld.JsonLdTransformer.fromJsonLd(JsonLdTransformer.java:100)
>>>> at
>>>> 
>>>> 
>> org.apache.streampipes.manager.verification.ElementVerifier.transform(ElementVerifier.java:169)
>>>> at
>>>> 
>>>> 
>> org.apache.streampipes.manager.verification.ElementVerifier.verifyAndAdd(ElementVerifier.java:84)
>>>> at
>>>> 
>>>> 
>> org.apache.streampipes.manager.operations.Operations.verifyAndAddElement(Operations.java:96)
>>>> at
>>>> 
>>>> 
>> org.apache.streampipes.manager.endpoint.EndpointItemParser.parseAndAddEndpointItem(EndpointItemParser.java:37)
>>>> at
>>>> 
>>>> 
>> org.apache.streampipes.rest.impl.PipelineElementImport.verifyAndAddElement(PipelineElementImport.java:93)
>>>> at
>>>> 
>>>> 
>> org.apache.streampipes.rest.impl.PipelineElementImport.addElement(PipelineElementImport.java:89)
>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
>>>> sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
>>>> 
>>>> Thanks,
>>>> Grainier.
>>>> 
>>>> On Thu, 21 May 2020 at 03:39, Dominik Riemer <ri...@apache.org> wrote:
>>>> 
>>>>> Hi Grainier,
>>>>> 
>>>>> sounds good!
>>>>> I've just pushed a commit that provides an initial version of the
>>>>> user-defined output strategy. It lets you select the input schema from
>>>>> the incoming stream and add/remove event properties. There is one
>>>>> limitation, the component currently only works for flat event
>>>>> structures, but it shouldn't be a problem to have a more advanced
>>>>> schema editor for this output strategy soon (we already have this in
>>>>> the StreamPipes connect code, but unfortunately, the pipeline editor
>>>>> has yet to be migrated from AngularJS to Angular 2+ to reuse these
>>>>> components). This is something we plan to do within the next weeks.
>>>>> 
>>>>> I added an example how to use the strategy to the examples repo:
>>>>> 
>>>>> https://github.com/apache/incubator-streampipes-examples/blob/dev/stre
>>>>> ampipes-pipeline-elements-examples-processors-jvm/src/main/java/org/ap
>>>>> ache/streampipes/pe/examples/jvm/staticproperty/CodeInputExampleContro
>>>>> ller.java
>>>>> 
>>>>> Let me know what you think and if you need anything more/else!
>>>>> 
>>>>> Dominik
>>>>> 
>>>>> On 2020/05/19 17:14:21, Grainier Perera <gr...@gmail.com>
>>>>> wrote:
>>>>>> Hi Dominik,
>>>>>> 
>>>>>> Awesome. This is really cool & exactly what I was looking for. I'll
>>>>>> use this with my implementation, and keep this thread posted with
>>>>>> the
>>>>> updates.
>>>>>> ps: let us know when you add output strategy as well.
>>>>>> 
>>>>>> Thanks,
>>>>>> Grainier.
>>>>>> 
>>>>>> On Tue, 19 May 2020 at 03:05, Dominik Riemer <ri...@apache.org>
>>>> wrote:
>>>>>> 
>>>>>>> Hi Grainier,
>>>>>>> 
>>>>>>> I just pushed an initial version of the static property to add
>>>>> javascript
>>>>>>> code.
>>>>>>> It's a simple codeblock with linting enabled and a (currently very
>>>>> simple)
>>>>>>> autocomplete feature that suggests existing properties from the
>>>>>>> input
>>>>> event
>>>>>>> (currently, only if you press ctrl-space directly after typing
>>>> "event"
>>>>>>> followed by a dot, I'll fix that soon ;-)
>>>>>>> 
>>>>>>> You can find an example how to use it here:
>>>>>>> 
>>>>>>> 
>>>>> https://github.com/apache/incubator-streampipes-examples/blob/dev/stre
>>>>> ampipes-pipeline-elements-examples-processors-jvm/src/main/java/org/ap
>>>>> ache/streampipes/pe/examples/jvm/staticproperty/CodeInputExampleContro
>>>>> ller.java
>>>>>>> 
>>>>>>> Feedback and ideas for improvement are welcome!
>>>>>>> 
>>>>>>> Dominik
>>>>>>> 
>>>>>>> 
>>>>>>> On 2020/05/17 12:36:05, Grainier Perera
>>>>>>> <gr...@gmail.com>
>>>>>>> wrote:
>>>>>>>> Hi Dominik,
>>>>>>>> 
>>>>>>>> Thank you. I'll work on `Implement JS evaluator` [1].
>>>>>>>> 
>>>>>>>> [1] https://issues.apache.org/jira/browse/STREAMPIPES-135
>>>>>>>> 
>>>>>>>> Grainier.
>>>>>>>> 
>>>>>>>> On Sun, 17 May 2020 at 18:02, Dominik Riemer <ri...@apache.org>
>>>>> wrote:
>>>>>>>> 
>>>>>>>>> Hi Grainier,
>>>>>>>>> 
>>>>>>>>> great! I created an umbrella issue for that:
>>>>>>>>> https://issues.apache.org/jira/browse/STREAMPIPES-132
>>>>>>>>> 
>>>>>>>>> I'll now start to implement the new static property for
>>>>>>>>> entering
>>>>> code.
>>>>>>>>> 
>>>>>>>>> Dominik
>>>>>>>>> 
>>>>>>>>> On 2020/05/15 01:29:44, Grainier Perera
>>>>>>>>> <grainier.perera@gmail.com
>>>>>> 
>>>>>>>>> wrote:
>>>>>>>>>> Hi,
>>>>>>>>>> 
>>>>>>>>>> I'm +1 to the idea of `UserDefinedOutputStrategy`. As you
>>>>>>>>>> said,
>>>>>>> that'll
>>>>>>>>> be
>>>>>>>>>> re-usable in the future processors as well. So let's proceed
>>>>>>>>>> with
>>>>>>> that.
>>>>>>>>>> 
>>>>>>>>>> Regards,
>>>>>>>>>> Grainier
>>>>>>>>>> 
>>>>>>>>>> On Fri, 15 May 2020 at 01:37, Dominik Riemer
>>>>>>>>>> <ri...@apache.org>
>>>>>>> wrote:
>>>>>>>>>> 
>>>>>>>>>>> Hi,
>>>>>>>>>>> cool, that sounds very good!
>>>>>>>>>>> 
>>>>>>>>>>> Concerning the output, I think there are two ways to
>>>>>>>>>>> achieve
>>>>> this:
>>>>>>>>>>> - A collection static property in conjunction with a
>>>>>>>>>>> runtime
>>>>>>> resolvable
>>>>>>>>>>> output strategy that creates the output schema based on
>>>>>>>>>>> the
>>>>> input
>>>>>>> from
>>>>>>>>> the
>>>>>>>>>>> collection
>>>>>>>>>>> - Or we create a new output strategy, e.g., let's call it
>>>>>>>>>>> UserDefinedOutputStrategy, which would render a UI
>>>>>>>>>>> component
>>>>> to let
>>>>>>>>> users
>>>>>>>>>>> add/remove/change event properties from the input schema.
>>>>>>>>>>> I
>>>>> also
>>>>>>> like
>>>>>>>>> the
>>>>>>>>>>> idea of preselecting all input properties.
>>>>>>>>>>> 
>>>>>>>>>>> I personally tend towards the second option as such a
>>>>>>>>>>> strategy
>>>>>>> might be
>>>>>>>>>>> useful for other pipeline elements as well and it would
>>>>>>>>>>> ease
>>>>> the
>>>>>>> model
>>>>>>>>>>> definition. We could also extend this component in the
>>>>>>>>>>> future,
>>>>>>> e.g., by
>>>>>>>>>>> letting users enrich metadata such as measurement units
>>>>> directly
>>>>>>> in a
>>>>>>>>>>> processor.
>>>>>>>>>>> 
>>>>>>>>>>> But I'm also open to the first option and we can also just
>>>>> start
>>>>>>>>>>> implementing to see what works best.
>>>>>>>>>>> 
>>>>>>>>>>> I'd also be happy to help with the implementation, e.g., I
>>>>> could
>>>>>>> work
>>>>>>>>> on
>>>>>>>>>>> the new static property for code input or the output
>>>> strategy!
>>>>>>>>>>> 
>>>>>>>>>>> Dominik
>>>>>>>>>>> 
>>>>>>>>>>> On 2020/05/14 11:59:36, Grainier Perera <
>>>>> grainier.perera@gmail.com
>>>>>>>> 
>>>>>>>>>>> wrote:
>>>>>>>>>>>> Hi,
>>>>>>>>>>>> 
>>>>>>>>>>>> +1 to the idea of having a collection property
>>>>>>>>>>>> +containing the
>>>>>>> fields
>>>>>>>>> of
>>>>>>>>>>> the
>>>>>>>>>>>> output properties. Also, If we can show the user with
>>>>> available
>>>>>>>>> fields
>>>>>>>>>>> with
>>>>>>>>>>>> metadata (of input event) populated in that same
>>>>>>>>>>>> collection
>>>>>>> property,
>>>>>>>>>>> then
>>>>>>>>>>>> the user can add/remove/updates fields easily. Let's say
>>>>>>>>>>>> they
>>>>>>> just
>>>>>>>>> have
>>>>>>>>>>> to
>>>>>>>>>>>> return a map containing all the fields they defined in
>>>>>>>>>>>> output
>>>>>>>>> properties
>>>>>>>>>>>> (i.e return {"existingId": 1, "existingTempInKelvin":
>>>>>>>>>>>> 301,
>>>>>>>>>>>> "newTempInCelsius": 28 } ).
>>>>>>>>>>>> 
>>>>>>>>>>>> [image: sample.png]
>>>>>>>>>>>> 
>>>>>>>>>>>> Regards,
>>>>>>>>>>>> Grainier.
>>>>>>>>>>>> 
>>>>>>>>>>>> On Thu, 14 May 2020 at 12:33, Patrick Wiener <
>>>>> wiener@apache.org>
>>>>>>>>> wrote:
>>>>>>>>>>>> 
>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>> 
>>>>>>>>>>>>> sounds reasonable.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> I share the thoughts of Philipp regarding the output
>>>>> schema.
>>>>>>> But
>>>>>>>>> as you
>>>>>>>>>>>>> said Grainier, we can also improve :)
>>>>>>>>>>>>> 
>>>>>>>>>>>>> btw: I really like the idea to also allow the more
>>>>> technical
>>>>>>> user
>>>>>>>>> to
>>>>>>>>>>>>> easily integrate their function snippets without
>>>>>>>>>>>>> having to care about the actual execution. This gets a
>>>>>>>>>>>>> more FaaS
>>>>> like
>>>>>>>>> feeling
>>>>>>>>>>> for
>>>>>>>>>>>>> streaming.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Patrick
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Am 14.05.2020 um 08:26 schrieb Philipp Zehnder <
>>>>>>>>> zehnder@apache.org>:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> it would be very useful to have such a processor.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> I like the ideas for 1 & 2 and I think we can do it
>>>>>>>>>>>>>> this
>>>>> way.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Actually I think that the definition of the output
>>>>>>>>>>>>>> scheme
>>>>>>> could
>>>>>>>>> be
>>>>>>>>>>> quite
>>>>>>>>>>>>> tricky.
>>>>>>>>>>>>>> Because users can do anything with the event.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> How about we provide an option property?
>>>>>>>>>>>>>> Users can select whether they want to define the
>>>>>>>>>>>>>> whole
>>>>> event
>>>>>>> or
>>>>>>>>> just
>>>>>>>>>>>>> append properties.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> For the configurations of the properties we could do
>>>>>>> something
>>>>>>>>>>> similar
>>>>>>>>>>>>> then we have in the PLC4xS7Adapter.
>>>>>>>>>>>>>> A collection property containing the fields of the
>>>>>>>>>>>>>> output
>>>>>>>>> properties.
>>>>>>>>>>>>>> Therefore, we have to decide what information we
>>>>>>>>>>>>>> have to
>>>>>>>>> provide. I
>>>>>>>>>>>>> think if we would support all options it becomes very
>>>>> cluttered
>>>>>>>>> for the
>>>>>>>>>>>>> user.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> What do you think?
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Philipp
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> On 14. May 2020, at 06:55, Grainier Perera <
>>>>>>>>>>> grainier.perera@gmail.com>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Hi Dominik,
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> I agree. We have to do some enhancements make this
>>>>> processor
>>>>>>>>> user
>>>>>>>>>>>>> friendly.
>>>>>>>>>>>>>>> Like you said it's nice to have;
>>>>>>>>>>>>>>> 1. A new static property that supports code
>>>>> highlighting &
>>>>>>>>> syntax
>>>>>>>>>>>>>>> checking. We can simply do such client-side
>>>>>>>>>>>>>>> validations
>>>>>>> using
>>>>>>>>>>>>> highlight.js
>>>>>>>>>>>>>>> (BSD) [1], jshint (MIT) [2], etc...
>>>>>>>>>>>>>>> 2. A mechanism to show the user with available
>>>>>>>>>>>>>>> fields
>>>>>>> they can
>>>>>>>>>>> use to
>>>>>>>>>>>>>>> write code (similar to what you've done with usable
>>>>>>> templates in
>>>>>>>>>>>>>>> EmailPublisher using HtmlInputParameter).
>>>>>>>>>>>>>>> 3. A mechanism to map output schema.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> I think 1 & 2 go together and for the 3rd
>>>>>>>>>>>>>>> requirement
>>>>> given
>>>>>>>>> that we
>>>>>>>>>>> are
>>>>>>>>>>>>>>> targetting a more technical user group for this
>>>>> processor,
>>>>>>> we
>>>>>>>>> can
>>>>>>>>>>> let
>>>>>>>>>>>>> them
>>>>>>>>>>>>>>> define the output. Otherwise, it would be quite
>>>>> difficult to
>>>>>>>>> derive
>>>>>>>>>>>>> return
>>>>>>>>>>>>>>> type from the JS. Moreover, I'm planning to use
>>>>>>>>>>>>>>> Java's
>>>>>>> builtin
>>>>>>>>>>>>> ScriptEngine
>>>>>>>>>>>>>>> to eval. Of course, there'll be limitations. But,
>>>>>>>>>>>>>>> we can
>>>>>>> always
>>>>>>>>>>> improve
>>>>>>>>>>>>>>> them :)
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> [1] https://highlightjs.org/ [2]
>>>>>>>>>>>>>>> https://jshint.com/
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>> Grainier.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> On Thu, 14 May 2020 at 00:53, Dominik Riemer <
>>>>>>> riemer@apache.org
>>>>>>>>>> 
>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Hi Grainier,
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> in my opinion, this would definitely be a very
>>>>> interesting
>>>>>>>>>>> addition! We
>>>>>>>>>>>>>>>> already briefly discussed such a feature in the
>>>>>>>>>>>>>>>> past,
>>>>>>> before we
>>>>>>>>>>> went
>>>>>>>>>>>>> to the
>>>>>>>>>>>>>>>> ASF. I’d say that a JS evaluator would be
>>>>>>>>>>>>>>>> definitely
>>>>>>> useful to
>>>>>>>>> a
>>>>>>>>>>>>> slightly
>>>>>>>>>>>>>>>> more technical target user group, but would
>>>>>>>>>>>>>>>> provide
>>>>> great
>>>>>>>>>>> flexibility
>>>>>>>>>>>>> for,
>>>>>>>>>>>>>>>> e.g., data harmonization tasks. So let’s discuss
>>>>>>>>>>>>>>>> what
>>>>> is
>>>>>>>>> needed to
>>>>>>>>>>>>>>>> implement this! I’d guess that we (maybe) need to
>>>>>>>>>>>>>>>> add
>>>>> some
>>>>>>>>>>>>> enhancements to
>>>>>>>>>>>>>>>> the core to make this processor good from a
>>>>>>>>>>>>>>>> usability
>>>>>>>>> perspective.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> From a conceptual point of view, I guess the JS
>>>>> processor
>>>>>>> would
>>>>>>>>>>> have no
>>>>>>>>>>>>>>>> specific input requirements and a single static
>>>>> property
>>>>>>> that
>>>>>>>>>>> allows
>>>>>>>>>>>>> users
>>>>>>>>>>>>>>>> to enter the code (maybe it would be cool to add a
>>>>>>>>>>>>>>>> new
>>>>>>> static
>>>>>>>>>>> property
>>>>>>>>>>>>> here
>>>>>>>>>>>>>>>> that supports code highlighting or even syntax
>>>>> checking).
>>>>>>> The
>>>>>>>>> only
>>>>>>>>>>> open
>>>>>>>>>>>>>>>> issue I see is the output schema – we would
>>>>>>>>>>>>>>>> somehow
>>>>> need to
>>>>>>>>>>> extract the
>>>>>>>>>>>>>>>> output from the JS function. This could probably
>>>>> partially
>>>>>>> be
>>>>>>>>> done
>>>>>>>>>>>>> using
>>>>>>>>>>>>>>>> the CustomTransformOutput (see
>>>>>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>> http://streampipes.apache.org/docs/docs/dev-guide-output-strategie
>>>>>>> s/),
>>>>>>>>>>>>>>>> but we would somehow need to derive type
>>>>>>>>>>>>>>>> information
>>>>> from
>>>>>>> the
>>>>>>>>> JS
>>>>>>>>>>>>> function
>>>>>>>>>>>>>>>> or introduce a feature to let users define/refine
>>>>>>>>>>>>>>>> the
>>>>>>> output
>>>>>>>>>>> manually
>>>>>>>>>>>>>>>> (e.g., to add semantic metadata).
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> What do you think would be best? And would you
>>>>>>>>>>>>>>>> evaluate
>>>>>>> the JS
>>>>>>>>>>> code in
>>>>>>>>>>>>>>>> Java or somehow else?
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Also, as I’ve just started to improve the SDK
>>>>>>>>>>>>>>>> (e.g.,
>>>>>>> supporting
>>>>>>>>>>>>> optional
>>>>>>>>>>>>>>>> static properties and an easier way to define and
>>>>> extract
>>>>>>> model
>>>>>>>>>>>>>>>> parameters), we can collect your requirements for
>>>>>>>>>>>>>>>> the
>>>>> JS
>>>>>>>>> processor
>>>>>>>>>>> and
>>>>>>>>>>>>>>>> extend the SDK if needed, just say what you would
>>>>>>>>>>>>>>>> like
>>>>> to
>>>>>>> have
>>>>>>>>> 😉
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Dominik
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> From: Grainier Perera <gr...@gmail.com>
>>>>>>>>>>>>>>>> Sent: Wednesday, May 13, 2020 2:41 PM
>>>>>>>>>>>>>>>> To: dev@streampipes.apache.org
>>>>>>>>>>>>>>>> Subject: Data processor to evaluate JavaScript
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Hi all,
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> I'm planning to implement a JavaScript eval data
>>>>>>> processor. As
>>>>>>>>> the
>>>>>>>>>>> name
>>>>>>>>>>>>>>>> suggests, users will be able to write some
>>>>>>>>>>>>>>>> JavaScript
>>>>> code
>>>>>>>>> which
>>>>>>>>>>> takes
>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>> an event (as a map), do some processing on the
>>>>>>>>>>>>>>>> event,
>>>>> and
>>>>>>>>> return a
>>>>>>>>>>> new
>>>>>>>>>>>>> map
>>>>>>>>>>>>>>>> which then gets converted to an sp-event.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> example js:
>>>>>>>>>>>>>>>> function process(event) {
>>>>>>>>>>>>>>>> // do processing here.
>>>>>>>>>>>>>>>> // return processed event.
>>>>>>>>>>>>>>>> return {id: http://event.id, tempInCelsius:
>>>>>>>>> (event.tempInKelvin
>>>>>>>>>>> -
>>>>>>>>>>>>>>>> 273.15)};
>>>>>>>>>>>>>>>> };
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Will this be a good addition to pipeline-elements?
>>>>> What do
>>>>>>> you
>>>>>>>>>>> think?
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>> Grainier.
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>>>> 
>> 
>> 
>>

Re: Data processor to evaluate JavaScript

Posted by Grainier Perera <gr...@apache.org>.

Hmm, That's odd. It doesn't use any 3rd party libraries for the PE (we use
javax.script.*). However, I came across a similar situation with
this RDFMappingException[1]. The reason for that is JSEval use newly
introduced .requiredCodeblock() controller prop, and for some reason,
there's a cached Docker image without it. But when I cleaned that and
re-build, it worked. Could this be the reason?

[1] io.fogsy.empire.pinto.RDFMappingException: Could not create an instance
of class org.apache.streampipes.model.output.OutputStrategy, it does not
have a default constructor

Grainier Perera.


On Thu, 4 Jun 2020 at 15:17, Philipp Zehnder <ze...@apache.org> wrote:

> Hi all,
>
> I tried to use the Processor JavaScript Eval in a deployment of
> StreamPipes (Lite), running in Docker.
> Unfortunately, I get an error in the UI when starting the pipeline. There
> is no specific error message, just an error that the JavaScript Eval did
> not start.
> I also did not see anything in the logs. Did anyone else have a similar
> issue?
>
> The processor runs in the container pipeline-elements-all-jvm using the
> CLI.
>
> Running the same pipeline locally in my IDE works well, so I guess there
> is a problem when building the project.
> Is it possible that some libraries are not bundled correctly?
>
> Philipp
>
> > On 22. May 2020, at 16:57, Grainier Perera <gr...@gmail.com>
> wrote:
> >
> > Hi Dominik,
> >
> > My bad. There were some cached images & that was the issue. Also fixed
> the
> > PR as per your comments. Please review and merge [1].
> >
> > [1] https://github.com/apache/incubator-streampipes-extensions/pull/15
> >
> > Thanks & Regards,
> > Grainier
> >
> > On Fri, 22 May 2020 at 16:25, Dominik Riemer <ri...@apache.org> wrote:
> >
> >> Hi Grainier,
> >>
> >> we have a Github Actions workflow [1] for core+extensions that usually
> >> builds and pushes the latest development containers for dev on each
> push.
> >> You are using the latest CLI from dev and the tmpl_env or .env version
> is
> >> set to 0.67.0-SNAPSHOT, right?
> >>
> >> I just tested the JsEval processor (with CLI 0.67.0-SNAPSHOT, local
> >> backend with the latest changes from dev and your PR) and it works
> great 😊
> >> The only thing I had to change was a typo in line 59 (use outEvent
> instead
> >> of event).
> >>
> >> In the controller, the supportedProtocols and supportedFormats are no
> >> longer needed, as they can now be declared for a whole module in the
> Init
> >> class (registerFormats/Protocols). I need to update this in the
> >> documentation.
> >>
> >> Dominik
> >>
> >>
> >>
> >>
> >>
> >>
> https://github.com/apache/incubator-streampipes/blob/dev/.github/workflows/build.yml
> >>
> >> -----Original Message-----
> >> From: Grainier Perera <gr...@gmail.com>
> >> Sent: Friday, May 22, 2020 11:58 AM
> >> To: dev@streampipes.apache.org
> >> Subject: Re: Data processor to evaluate JavaScript
> >>
> >> Hi Dominik,
> >>
> >> I've implemented the following[1]. However, when I try to install the
> >> processor it gives me the following error. Any clue? I've used CLI to
> start
> >> sp. So, maybe the snapshot docker image doesn't have the recent
> changes? If
> >> so what are the steps to build the images with changes in-place?
> >>
> >> [1] https://github.com/apache/incubator-streampipes-extensions/pull/15
> >>
> >> ERROR: https://streampipes.org/vocabulary/v1/UserDefinedOutputStrategy
> >> io.fogsy.empire.pinto.RDFMappingException: Could not create an instance
> of
> >> class org.apache.streampipes.model.output.OutputStrategy, it does not
> have
> >> a default constructor at
> >> io.fogsy.empire.pinto.RDFMapper.newInstance(RDFMapper.java:178)
> >> at io.fogsy.empire.pinto.RDFMapper.readValue(RDFMapper.java:261)
> >> at io.fogsy.empire.pinto.RDFMapper.valueToObject(RDFMapper.java:695)
> >> at io.fogsy.empire.pinto.RDFMapper.lambda$toObject$3(RDFMapper.java:388)
> >> at
> >>
> io.fogsy.empire.pinto.RDFMapper$$Lambda$802/0000000000096040.apply(Unknown
> >> Source)
> >> at
> >>
> io.fogsy.empire.pinto.RDFMapper$$Lambda$803/0000000000096580.apply(Unknown
> >> Source)
> >> at java.util.stream.ReferencePipeline$3$1.accept(Unknown Source) at
> >> java.util.ArrayList$ArrayListSpliterator.forEachRemaining(Unknown
> Source)
> >> at java.util.stream.AbstractPipeline.copyInto(Unknown Source) at
> >> java.util.stream.AbstractPipeline.wrapAndCopyInto(Unknown Source) at
> >> java.util.stream.ForEachOps$ForEachOp.evaluateSequential(Unknown
> Source) at
> >> java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(Unknown
> >> Source)
> >> at java.util.stream.AbstractPipeline.evaluate(Unknown Source) at
> >> java.util.stream.ReferencePipeline.forEach(Unknown Source) at
> >> io.fogsy.empire.pinto.RDFMapper.readValue(RDFMapper.java:299)
> >> at
> >>
> >>
> org.apache.streampipes.serializers.jsonld.JsonLdTransformer.fromJsonLd(JsonLdTransformer.java:100)
> >> at
> >>
> >>
> org.apache.streampipes.manager.verification.ElementVerifier.transform(ElementVerifier.java:169)
> >> at
> >>
> >>
> org.apache.streampipes.manager.verification.ElementVerifier.verifyAndAdd(ElementVerifier.java:84)
> >> at
> >>
> >>
> org.apache.streampipes.manager.operations.Operations.verifyAndAddElement(Operations.java:96)
> >> at
> >>
> >>
> org.apache.streampipes.manager.endpoint.EndpointItemParser.parseAndAddEndpointItem(EndpointItemParser.java:37)
> >> at
> >>
> >>
> org.apache.streampipes.rest.impl.PipelineElementImport.verifyAndAddElement(PipelineElementImport.java:93)
> >> at
> >>
> >>
> org.apache.streampipes.rest.impl.PipelineElementImport.addElement(PipelineElementImport.java:89)
> >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
> >> sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
> >>
> >> Thanks,
> >> Grainier.
> >>
> >> On Thu, 21 May 2020 at 03:39, Dominik Riemer <ri...@apache.org> wrote:
> >>
> >>> Hi Grainier,
> >>>
> >>> sounds good!
> >>> I've just pushed a commit that provides an initial version of the
> >>> user-defined output strategy. It lets you select the input schema from
> >>> the incoming stream and add/remove event properties. There is one
> >>> limitation, the component currently only works for flat event
> >>> structures, but it shouldn't be a problem to have a more advanced
> >>> schema editor for this output strategy soon (we already have this in
> >>> the StreamPipes connect code, but unfortunately, the pipeline editor
> >>> has yet to be migrated from AngularJS to Angular 2+ to reuse these
> >>> components). This is something we plan to do within the next weeks.
> >>>
> >>> I added an example how to use the strategy to the examples repo:
> >>>
> >>> https://github.com/apache/incubator-streampipes-examples/blob/dev/stre
> >>> ampipes-pipeline-elements-examples-processors-jvm/src/main/java/org/ap
> >>> ache/streampipes/pe/examples/jvm/staticproperty/CodeInputExampleContro
> >>> ller.java
> >>>
> >>> Let me know what you think and if you need anything more/else!
> >>>
> >>> Dominik
> >>>
> >>> On 2020/05/19 17:14:21, Grainier Perera <gr...@gmail.com>
> >>> wrote:
> >>>> Hi Dominik,
> >>>>
> >>>> Awesome. This is really cool & exactly what I was looking for. I'll
> >>>> use this with my implementation, and keep this thread posted with
> >>>> the
> >>> updates.
> >>>> ps: let us know when you add output strategy as well.
> >>>>
> >>>> Thanks,
> >>>> Grainier.
> >>>>
> >>>> On Tue, 19 May 2020 at 03:05, Dominik Riemer <ri...@apache.org>
> >> wrote:
> >>>>
> >>>>> Hi Grainier,
> >>>>>
> >>>>> I just pushed an initial version of the static property to add
> >>> javascript
> >>>>> code.
> >>>>> It's a simple codeblock with linting enabled and a (currently very
> >>> simple)
> >>>>> autocomplete feature that suggests existing properties from the
> >>>>> input
> >>> event
> >>>>> (currently, only if you press ctrl-space directly after typing
> >> "event"
> >>>>> followed by a dot, I'll fix that soon ;-)
> >>>>>
> >>>>> You can find an example how to use it here:
> >>>>>
> >>>>>
> >>> https://github.com/apache/incubator-streampipes-examples/blob/dev/stre
> >>> ampipes-pipeline-elements-examples-processors-jvm/src/main/java/org/ap
> >>> ache/streampipes/pe/examples/jvm/staticproperty/CodeInputExampleContro
> >>> ller.java
> >>>>>
> >>>>> Feedback and ideas for improvement are welcome!
> >>>>>
> >>>>> Dominik
> >>>>>
> >>>>>
> >>>>> On 2020/05/17 12:36:05, Grainier Perera
> >>>>> <gr...@gmail.com>
> >>>>> wrote:
> >>>>>> Hi Dominik,
> >>>>>>
> >>>>>> Thank you. I'll work on `Implement JS evaluator` [1].
> >>>>>>
> >>>>>> [1] https://issues.apache.org/jira/browse/STREAMPIPES-135
> >>>>>>
> >>>>>> Grainier.
> >>>>>>
> >>>>>> On Sun, 17 May 2020 at 18:02, Dominik Riemer <ri...@apache.org>
> >>> wrote:
> >>>>>>
> >>>>>>> Hi Grainier,
> >>>>>>>
> >>>>>>> great! I created an umbrella issue for that:
> >>>>>>> https://issues.apache.org/jira/browse/STREAMPIPES-132
> >>>>>>>
> >>>>>>> I'll now start to implement the new static property for
> >>>>>>> entering
> >>> code.
> >>>>>>>
> >>>>>>> Dominik
> >>>>>>>
> >>>>>>> On 2020/05/15 01:29:44, Grainier Perera
> >>>>>>> <grainier.perera@gmail.com
> >>>>
> >>>>>>> wrote:
> >>>>>>>> Hi,
> >>>>>>>>
> >>>>>>>> I'm +1 to the idea of `UserDefinedOutputStrategy`. As you
> >>>>>>>> said,
> >>>>> that'll
> >>>>>>> be
> >>>>>>>> re-usable in the future processors as well. So let's proceed
> >>>>>>>> with
> >>>>> that.
> >>>>>>>>
> >>>>>>>> Regards,
> >>>>>>>> Grainier
> >>>>>>>>
> >>>>>>>> On Fri, 15 May 2020 at 01:37, Dominik Riemer
> >>>>>>>> <ri...@apache.org>
> >>>>> wrote:
> >>>>>>>>
> >>>>>>>>> Hi,
> >>>>>>>>> cool, that sounds very good!
> >>>>>>>>>
> >>>>>>>>> Concerning the output, I think there are two ways to
> >>>>>>>>> achieve
> >>> this:
> >>>>>>>>> - A collection static property in conjunction with a
> >>>>>>>>> runtime
> >>>>> resolvable
> >>>>>>>>> output strategy that creates the output schema based on
> >>>>>>>>> the
> >>> input
> >>>>> from
> >>>>>>> the
> >>>>>>>>> collection
> >>>>>>>>> - Or we create a new output strategy, e.g., let's call it
> >>>>>>>>> UserDefinedOutputStrategy, which would render a UI
> >>>>>>>>> component
> >>> to let
> >>>>>>> users
> >>>>>>>>> add/remove/change event properties from the input schema.
> >>>>>>>>> I
> >>> also
> >>>>> like
> >>>>>>> the
> >>>>>>>>> idea of preselecting all input properties.
> >>>>>>>>>
> >>>>>>>>> I personally tend towards the second option as such a
> >>>>>>>>> strategy
> >>>>> might be
> >>>>>>>>> useful for other pipeline elements as well and it would
> >>>>>>>>> ease
> >>> the
> >>>>> model
> >>>>>>>>> definition. We could also extend this component in the
> >>>>>>>>> future,
> >>>>> e.g., by
> >>>>>>>>> letting users enrich metadata such as measurement units
> >>> directly
> >>>>> in a
> >>>>>>>>> processor.
> >>>>>>>>>
> >>>>>>>>> But I'm also open to the first option and we can also just
> >>> start
> >>>>>>>>> implementing to see what works best.
> >>>>>>>>>
> >>>>>>>>> I'd also be happy to help with the implementation, e.g., I
> >>> could
> >>>>> work
> >>>>>>> on
> >>>>>>>>> the new static property for code input or the output
> >> strategy!
> >>>>>>>>>
> >>>>>>>>> Dominik
> >>>>>>>>>
> >>>>>>>>> On 2020/05/14 11:59:36, Grainier Perera <
> >>> grainier.perera@gmail.com
> >>>>>>
> >>>>>>>>> wrote:
> >>>>>>>>>> Hi,
> >>>>>>>>>>
> >>>>>>>>>> +1 to the idea of having a collection property
> >>>>>>>>>> +containing the
> >>>>> fields
> >>>>>>> of
> >>>>>>>>> the
> >>>>>>>>>> output properties. Also, If we can show the user with
> >>> available
> >>>>>>> fields
> >>>>>>>>> with
> >>>>>>>>>> metadata (of input event) populated in that same
> >>>>>>>>>> collection
> >>>>> property,
> >>>>>>>>> then
> >>>>>>>>>> the user can add/remove/updates fields easily. Let's say
> >>>>>>>>>> they
> >>>>> just
> >>>>>>> have
> >>>>>>>>> to
> >>>>>>>>>> return a map containing all the fields they defined in
> >>>>>>>>>> output
> >>>>>>> properties
> >>>>>>>>>> (i.e return {"existingId": 1, "existingTempInKelvin":
> >>>>>>>>>> 301,
> >>>>>>>>>> "newTempInCelsius": 28 } ).
> >>>>>>>>>>
> >>>>>>>>>> [image: sample.png]
> >>>>>>>>>>
> >>>>>>>>>> Regards,
> >>>>>>>>>> Grainier.
> >>>>>>>>>>
> >>>>>>>>>> On Thu, 14 May 2020 at 12:33, Patrick Wiener <
> >>> wiener@apache.org>
> >>>>>>> wrote:
> >>>>>>>>>>
> >>>>>>>>>>> Hi,
> >>>>>>>>>>>
> >>>>>>>>>>> sounds reasonable.
> >>>>>>>>>>>
> >>>>>>>>>>> I share the thoughts of Philipp regarding the output
> >>> schema.
> >>>>> But
> >>>>>>> as you
> >>>>>>>>>>> said Grainier, we can also improve :)
> >>>>>>>>>>>
> >>>>>>>>>>> btw: I really like the idea to also allow the more
> >>> technical
> >>>>> user
> >>>>>>> to
> >>>>>>>>>>> easily integrate their function snippets without
> >>>>>>>>>>> having to care about the actual execution. This gets a
> >>>>>>>>>>> more FaaS
> >>> like
> >>>>>>> feeling
> >>>>>>>>> for
> >>>>>>>>>>> streaming.
> >>>>>>>>>>>
> >>>>>>>>>>> Patrick
> >>>>>>>>>>>
> >>>>>>>>>>>> Am 14.05.2020 um 08:26 schrieb Philipp Zehnder <
> >>>>>>> zehnder@apache.org>:
> >>>>>>>>>>>>
> >>>>>>>>>>>> Hi,
> >>>>>>>>>>>>
> >>>>>>>>>>>> it would be very useful to have such a processor.
> >>>>>>>>>>>>
> >>>>>>>>>>>> I like the ideas for 1 & 2 and I think we can do it
> >>>>>>>>>>>> this
> >>> way.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Actually I think that the definition of the output
> >>>>>>>>>>>> scheme
> >>>>> could
> >>>>>>> be
> >>>>>>>>> quite
> >>>>>>>>>>> tricky.
> >>>>>>>>>>>> Because users can do anything with the event.
> >>>>>>>>>>>>
> >>>>>>>>>>>> How about we provide an option property?
> >>>>>>>>>>>> Users can select whether they want to define the
> >>>>>>>>>>>> whole
> >>> event
> >>>>> or
> >>>>>>> just
> >>>>>>>>>>> append properties.
> >>>>>>>>>>>>
> >>>>>>>>>>>> For the configurations of the properties we could do
> >>>>> something
> >>>>>>>>> similar
> >>>>>>>>>>> then we have in the PLC4xS7Adapter.
> >>>>>>>>>>>> A collection property containing the fields of the
> >>>>>>>>>>>> output
> >>>>>>> properties.
> >>>>>>>>>>>> Therefore, we have to decide what information we
> >>>>>>>>>>>> have to
> >>>>>>> provide. I
> >>>>>>>>>>> think if we would support all options it becomes very
> >>> cluttered
> >>>>>>> for the
> >>>>>>>>>>> user.
> >>>>>>>>>>>>
> >>>>>>>>>>>> What do you think?
> >>>>>>>>>>>>
> >>>>>>>>>>>> Philipp
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>> On 14. May 2020, at 06:55, Grainier Perera <
> >>>>>>>>> grainier.perera@gmail.com>
> >>>>>>>>>>> wrote:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Hi Dominik,
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> I agree. We have to do some enhancements make this
> >>> processor
> >>>>>>> user
> >>>>>>>>>>> friendly.
> >>>>>>>>>>>>> Like you said it's nice to have;
> >>>>>>>>>>>>>  1. A new static property that supports code
> >>> highlighting &
> >>>>>>> syntax
> >>>>>>>>>>>>> checking. We can simply do such client-side
> >>>>>>>>>>>>> validations
> >>>>> using
> >>>>>>>>>>> highlight.js
> >>>>>>>>>>>>> (BSD) [1], jshint (MIT) [2], etc...
> >>>>>>>>>>>>>  2. A mechanism to show the user with available
> >>>>>>>>>>>>> fields
> >>>>> they can
> >>>>>>>>> use to
> >>>>>>>>>>>>> write code (similar to what you've done with usable
> >>>>> templates in
> >>>>>>>>>>>>> EmailPublisher using HtmlInputParameter).
> >>>>>>>>>>>>>  3. A mechanism to map output schema.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> I think 1 & 2 go together and for the 3rd
> >>>>>>>>>>>>> requirement
> >>> given
> >>>>>>> that we
> >>>>>>>>> are
> >>>>>>>>>>>>> targetting a more technical user group for this
> >>> processor,
> >>>>> we
> >>>>>>> can
> >>>>>>>>> let
> >>>>>>>>>>> them
> >>>>>>>>>>>>> define the output. Otherwise, it would be quite
> >>> difficult to
> >>>>>>> derive
> >>>>>>>>>>> return
> >>>>>>>>>>>>> type from the JS. Moreover, I'm planning to use
> >>>>>>>>>>>>> Java's
> >>>>> builtin
> >>>>>>>>>>> ScriptEngine
> >>>>>>>>>>>>> to eval. Of course, there'll be limitations. But,
> >>>>>>>>>>>>> we can
> >>>>> always
> >>>>>>>>> improve
> >>>>>>>>>>>>> them :)
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> [1] https://highlightjs.org/ [2]
> >>>>>>>>>>>>> https://jshint.com/
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Regards,
> >>>>>>>>>>>>> Grainier.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> On Thu, 14 May 2020 at 00:53, Dominik Riemer <
> >>>>> riemer@apache.org
> >>>>>>>>
> >>>>>>>>> wrote:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> Hi Grainier,
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> in my opinion, this would definitely be a very
> >>> interesting
> >>>>>>>>> addition! We
> >>>>>>>>>>>>>> already briefly discussed such a feature in the
> >>>>>>>>>>>>>> past,
> >>>>> before we
> >>>>>>>>> went
> >>>>>>>>>>> to the
> >>>>>>>>>>>>>> ASF. I’d say that a JS evaluator would be
> >>>>>>>>>>>>>> definitely
> >>>>> useful to
> >>>>>>> a
> >>>>>>>>>>> slightly
> >>>>>>>>>>>>>> more technical target user group, but would
> >>>>>>>>>>>>>> provide
> >>> great
> >>>>>>>>> flexibility
> >>>>>>>>>>> for,
> >>>>>>>>>>>>>> e.g., data harmonization tasks. So let’s discuss
> >>>>>>>>>>>>>> what
> >>> is
> >>>>>>> needed to
> >>>>>>>>>>>>>> implement this! I’d guess that we (maybe) need to
> >>>>>>>>>>>>>> add
> >>> some
> >>>>>>>>>>> enhancements to
> >>>>>>>>>>>>>> the core to make this processor good from a
> >>>>>>>>>>>>>> usability
> >>>>>>> perspective.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> From a conceptual point of view, I guess the JS
> >>> processor
> >>>>> would
> >>>>>>>>> have no
> >>>>>>>>>>>>>> specific input requirements and a single static
> >>> property
> >>>>> that
> >>>>>>>>> allows
> >>>>>>>>>>> users
> >>>>>>>>>>>>>> to enter the code (maybe it would be cool to add a
> >>>>>>>>>>>>>> new
> >>>>> static
> >>>>>>>>> property
> >>>>>>>>>>> here
> >>>>>>>>>>>>>> that supports code highlighting or even syntax
> >>> checking).
> >>>>> The
> >>>>>>> only
> >>>>>>>>> open
> >>>>>>>>>>>>>> issue I see is the output schema – we would
> >>>>>>>>>>>>>> somehow
> >>> need to
> >>>>>>>>> extract the
> >>>>>>>>>>>>>> output from the JS function. This could probably
> >>> partially
> >>>>> be
> >>>>>>> done
> >>>>>>>>>>> using
> >>>>>>>>>>>>>> the CustomTransformOutput (see
> >>>>>>>>>>>>>>
> >>>>>>>>>
> >>>>> http://streampipes.apache.org/docs/docs/dev-guide-output-strategie
> >>>>> s/),
> >>>>>>>>>>>>>> but we would somehow need to derive type
> >>>>>>>>>>>>>> information
> >>> from
> >>>>> the
> >>>>>>> JS
> >>>>>>>>>>> function
> >>>>>>>>>>>>>> or introduce a feature to let users define/refine
> >>>>>>>>>>>>>> the
> >>>>> output
> >>>>>>>>> manually
> >>>>>>>>>>>>>> (e.g., to add semantic metadata).
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> What do you think would be best? And would you
> >>>>>>>>>>>>>> evaluate
> >>>>> the JS
> >>>>>>>>> code in
> >>>>>>>>>>>>>> Java or somehow else?
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Also, as I’ve just started to improve the SDK
> >>>>>>>>>>>>>> (e.g.,
> >>>>> supporting
> >>>>>>>>>>> optional
> >>>>>>>>>>>>>> static properties and an easier way to define and
> >>> extract
> >>>>> model
> >>>>>>>>>>>>>> parameters), we can collect your requirements for
> >>>>>>>>>>>>>> the
> >>> JS
> >>>>>>> processor
> >>>>>>>>> and
> >>>>>>>>>>>>>> extend the SDK if needed, just say what you would
> >>>>>>>>>>>>>> like
> >>> to
> >>>>> have
> >>>>>>> 😉
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Dominik
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> From: Grainier Perera <gr...@gmail.com>
> >>>>>>>>>>>>>> Sent: Wednesday, May 13, 2020 2:41 PM
> >>>>>>>>>>>>>> To: dev@streampipes.apache.org
> >>>>>>>>>>>>>> Subject: Data processor to evaluate JavaScript
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Hi all,
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> I'm planning to implement a JavaScript eval data
> >>>>> processor. As
> >>>>>>> the
> >>>>>>>>> name
> >>>>>>>>>>>>>> suggests, users will be able to write some
> >>>>>>>>>>>>>> JavaScript
> >>> code
> >>>>>>> which
> >>>>>>>>> takes
> >>>>>>>>>>> in
> >>>>>>>>>>>>>> an event (as a map), do some processing on the
> >>>>>>>>>>>>>> event,
> >>> and
> >>>>>>> return a
> >>>>>>>>> new
> >>>>>>>>>>> map
> >>>>>>>>>>>>>> which then gets converted to an sp-event.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> example js:
> >>>>>>>>>>>>>> function process(event) {
> >>>>>>>>>>>>>>  // do processing here.
> >>>>>>>>>>>>>>  // return processed event.
> >>>>>>>>>>>>>>  return {id: http://event.id, tempInCelsius:
> >>>>>>> (event.tempInKelvin
> >>>>>>>>> -
> >>>>>>>>>>>>>> 273.15)};
> >>>>>>>>>>>>>> };
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Will this be a good addition to pipeline-elements?
> >>> What do
> >>>>> you
> >>>>>>>>> think?
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Regards,
> >>>>>>>>>>>>>> Grainier.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>
> >>
>
>
>

Re: Data processor to evaluate JavaScript

Posted by Philipp Zehnder <ze...@apache.org>.

Hi all,

I tried to use the Processor JavaScript Eval in a deployment of StreamPipes (Lite), running in Docker.
Unfortunately, I get an error in the UI when starting the pipeline. There is no specific error message, just an error that the JavaScript Eval did not start.
I also did not see anything in the logs. Did anyone else have a similar issue?

The processor runs in the container pipeline-elements-all-jvm using the CLI.

Running the same pipeline locally in my IDE works well, so I guess there is a problem when building the project.
Is it possible that some libraries are not bundled correctly?

Philipp

> On 22. May 2020, at 16:57, Grainier Perera <gr...@gmail.com> wrote:
> 
> Hi Dominik,
> 
> My bad. There were some cached images & that was the issue. Also fixed the
> PR as per your comments. Please review and merge [1].
> 
> [1] https://github.com/apache/incubator-streampipes-extensions/pull/15
> 
> Thanks & Regards,
> Grainier
> 
> On Fri, 22 May 2020 at 16:25, Dominik Riemer <ri...@apache.org> wrote:
> 
>> Hi Grainier,
>> 
>> we have a Github Actions workflow [1] for core+extensions that usually
>> builds and pushes the latest development containers for dev on each push.
>> You are using the latest CLI from dev and the tmpl_env or .env version is
>> set to 0.67.0-SNAPSHOT, right?
>> 
>> I just tested the JsEval processor (with CLI 0.67.0-SNAPSHOT, local
>> backend with the latest changes from dev and your PR) and it works great 😊
>> The only thing I had to change was a typo in line 59 (use outEvent instead
>> of event).
>> 
>> In the controller, the supportedProtocols and supportedFormats are no
>> longer needed, as they can now be declared for a whole module in the Init
>> class (registerFormats/Protocols). I need to update this in the
>> documentation.
>> 
>> Dominik
>> 
>> 
>> 
>> 
>> 
>> https://github.com/apache/incubator-streampipes/blob/dev/.github/workflows/build.yml
>> 
>> -----Original Message-----
>> From: Grainier Perera <gr...@gmail.com>
>> Sent: Friday, May 22, 2020 11:58 AM
>> To: dev@streampipes.apache.org
>> Subject: Re: Data processor to evaluate JavaScript
>> 
>> Hi Dominik,
>> 
>> I've implemented the following[1]. However, when I try to install the
>> processor it gives me the following error. Any clue? I've used CLI to start
>> sp. So, maybe the snapshot docker image doesn't have the recent changes? If
>> so what are the steps to build the images with changes in-place?
>> 
>> [1] https://github.com/apache/incubator-streampipes-extensions/pull/15
>> 
>> ERROR: https://streampipes.org/vocabulary/v1/UserDefinedOutputStrategy
>> io.fogsy.empire.pinto.RDFMappingException: Could not create an instance of
>> class org.apache.streampipes.model.output.OutputStrategy, it does not have
>> a default constructor at
>> io.fogsy.empire.pinto.RDFMapper.newInstance(RDFMapper.java:178)
>> at io.fogsy.empire.pinto.RDFMapper.readValue(RDFMapper.java:261)
>> at io.fogsy.empire.pinto.RDFMapper.valueToObject(RDFMapper.java:695)
>> at io.fogsy.empire.pinto.RDFMapper.lambda$toObject$3(RDFMapper.java:388)
>> at
>> io.fogsy.empire.pinto.RDFMapper$$Lambda$802/0000000000096040.apply(Unknown
>> Source)
>> at
>> io.fogsy.empire.pinto.RDFMapper$$Lambda$803/0000000000096580.apply(Unknown
>> Source)
>> at java.util.stream.ReferencePipeline$3$1.accept(Unknown Source) at
>> java.util.ArrayList$ArrayListSpliterator.forEachRemaining(Unknown Source)
>> at java.util.stream.AbstractPipeline.copyInto(Unknown Source) at
>> java.util.stream.AbstractPipeline.wrapAndCopyInto(Unknown Source) at
>> java.util.stream.ForEachOps$ForEachOp.evaluateSequential(Unknown Source) at
>> java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(Unknown
>> Source)
>> at java.util.stream.AbstractPipeline.evaluate(Unknown Source) at
>> java.util.stream.ReferencePipeline.forEach(Unknown Source) at
>> io.fogsy.empire.pinto.RDFMapper.readValue(RDFMapper.java:299)
>> at
>> 
>> org.apache.streampipes.serializers.jsonld.JsonLdTransformer.fromJsonLd(JsonLdTransformer.java:100)
>> at
>> 
>> org.apache.streampipes.manager.verification.ElementVerifier.transform(ElementVerifier.java:169)
>> at
>> 
>> org.apache.streampipes.manager.verification.ElementVerifier.verifyAndAdd(ElementVerifier.java:84)
>> at
>> 
>> org.apache.streampipes.manager.operations.Operations.verifyAndAddElement(Operations.java:96)
>> at
>> 
>> org.apache.streampipes.manager.endpoint.EndpointItemParser.parseAndAddEndpointItem(EndpointItemParser.java:37)
>> at
>> 
>> org.apache.streampipes.rest.impl.PipelineElementImport.verifyAndAddElement(PipelineElementImport.java:93)
>> at
>> 
>> org.apache.streampipes.rest.impl.PipelineElementImport.addElement(PipelineElementImport.java:89)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
>> sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
>> 
>> Thanks,
>> Grainier.
>> 
>> On Thu, 21 May 2020 at 03:39, Dominik Riemer <ri...@apache.org> wrote:
>> 
>>> Hi Grainier,
>>> 
>>> sounds good!
>>> I've just pushed a commit that provides an initial version of the
>>> user-defined output strategy. It lets you select the input schema from
>>> the incoming stream and add/remove event properties. There is one
>>> limitation, the component currently only works for flat event
>>> structures, but it shouldn't be a problem to have a more advanced
>>> schema editor for this output strategy soon (we already have this in
>>> the StreamPipes connect code, but unfortunately, the pipeline editor
>>> has yet to be migrated from AngularJS to Angular 2+ to reuse these
>>> components). This is something we plan to do within the next weeks.
>>> 
>>> I added an example how to use the strategy to the examples repo:
>>> 
>>> https://github.com/apache/incubator-streampipes-examples/blob/dev/stre
>>> ampipes-pipeline-elements-examples-processors-jvm/src/main/java/org/ap
>>> ache/streampipes/pe/examples/jvm/staticproperty/CodeInputExampleContro
>>> ller.java
>>> 
>>> Let me know what you think and if you need anything more/else!
>>> 
>>> Dominik
>>> 
>>> On 2020/05/19 17:14:21, Grainier Perera <gr...@gmail.com>
>>> wrote:
>>>> Hi Dominik,
>>>> 
>>>> Awesome. This is really cool & exactly what I was looking for. I'll
>>>> use this with my implementation, and keep this thread posted with
>>>> the
>>> updates.
>>>> ps: let us know when you add output strategy as well.
>>>> 
>>>> Thanks,
>>>> Grainier.
>>>> 
>>>> On Tue, 19 May 2020 at 03:05, Dominik Riemer <ri...@apache.org>
>> wrote:
>>>> 
>>>>> Hi Grainier,
>>>>> 
>>>>> I just pushed an initial version of the static property to add
>>> javascript
>>>>> code.
>>>>> It's a simple codeblock with linting enabled and a (currently very
>>> simple)
>>>>> autocomplete feature that suggests existing properties from the
>>>>> input
>>> event
>>>>> (currently, only if you press ctrl-space directly after typing
>> "event"
>>>>> followed by a dot, I'll fix that soon ;-)
>>>>> 
>>>>> You can find an example how to use it here:
>>>>> 
>>>>> 
>>> https://github.com/apache/incubator-streampipes-examples/blob/dev/stre
>>> ampipes-pipeline-elements-examples-processors-jvm/src/main/java/org/ap
>>> ache/streampipes/pe/examples/jvm/staticproperty/CodeInputExampleContro
>>> ller.java
>>>>> 
>>>>> Feedback and ideas for improvement are welcome!
>>>>> 
>>>>> Dominik
>>>>> 
>>>>> 
>>>>> On 2020/05/17 12:36:05, Grainier Perera
>>>>> <gr...@gmail.com>
>>>>> wrote:
>>>>>> Hi Dominik,
>>>>>> 
>>>>>> Thank you. I'll work on `Implement JS evaluator` [1].
>>>>>> 
>>>>>> [1] https://issues.apache.org/jira/browse/STREAMPIPES-135
>>>>>> 
>>>>>> Grainier.
>>>>>> 
>>>>>> On Sun, 17 May 2020 at 18:02, Dominik Riemer <ri...@apache.org>
>>> wrote:
>>>>>> 
>>>>>>> Hi Grainier,
>>>>>>> 
>>>>>>> great! I created an umbrella issue for that:
>>>>>>> https://issues.apache.org/jira/browse/STREAMPIPES-132
>>>>>>> 
>>>>>>> I'll now start to implement the new static property for
>>>>>>> entering
>>> code.
>>>>>>> 
>>>>>>> Dominik
>>>>>>> 
>>>>>>> On 2020/05/15 01:29:44, Grainier Perera
>>>>>>> <grainier.perera@gmail.com
>>>> 
>>>>>>> wrote:
>>>>>>>> Hi,
>>>>>>>> 
>>>>>>>> I'm +1 to the idea of `UserDefinedOutputStrategy`. As you
>>>>>>>> said,
>>>>> that'll
>>>>>>> be
>>>>>>>> re-usable in the future processors as well. So let's proceed
>>>>>>>> with
>>>>> that.
>>>>>>>> 
>>>>>>>> Regards,
>>>>>>>> Grainier
>>>>>>>> 
>>>>>>>> On Fri, 15 May 2020 at 01:37, Dominik Riemer
>>>>>>>> <ri...@apache.org>
>>>>> wrote:
>>>>>>>> 
>>>>>>>>> Hi,
>>>>>>>>> cool, that sounds very good!
>>>>>>>>> 
>>>>>>>>> Concerning the output, I think there are two ways to
>>>>>>>>> achieve
>>> this:
>>>>>>>>> - A collection static property in conjunction with a
>>>>>>>>> runtime
>>>>> resolvable
>>>>>>>>> output strategy that creates the output schema based on
>>>>>>>>> the
>>> input
>>>>> from
>>>>>>> the
>>>>>>>>> collection
>>>>>>>>> - Or we create a new output strategy, e.g., let's call it
>>>>>>>>> UserDefinedOutputStrategy, which would render a UI
>>>>>>>>> component
>>> to let
>>>>>>> users
>>>>>>>>> add/remove/change event properties from the input schema.
>>>>>>>>> I
>>> also
>>>>> like
>>>>>>> the
>>>>>>>>> idea of preselecting all input properties.
>>>>>>>>> 
>>>>>>>>> I personally tend towards the second option as such a
>>>>>>>>> strategy
>>>>> might be
>>>>>>>>> useful for other pipeline elements as well and it would
>>>>>>>>> ease
>>> the
>>>>> model
>>>>>>>>> definition. We could also extend this component in the
>>>>>>>>> future,
>>>>> e.g., by
>>>>>>>>> letting users enrich metadata such as measurement units
>>> directly
>>>>> in a
>>>>>>>>> processor.
>>>>>>>>> 
>>>>>>>>> But I'm also open to the first option and we can also just
>>> start
>>>>>>>>> implementing to see what works best.
>>>>>>>>> 
>>>>>>>>> I'd also be happy to help with the implementation, e.g., I
>>> could
>>>>> work
>>>>>>> on
>>>>>>>>> the new static property for code input or the output
>> strategy!
>>>>>>>>> 
>>>>>>>>> Dominik
>>>>>>>>> 
>>>>>>>>> On 2020/05/14 11:59:36, Grainier Perera <
>>> grainier.perera@gmail.com
>>>>>> 
>>>>>>>>> wrote:
>>>>>>>>>> Hi,
>>>>>>>>>> 
>>>>>>>>>> +1 to the idea of having a collection property
>>>>>>>>>> +containing the
>>>>> fields
>>>>>>> of
>>>>>>>>> the
>>>>>>>>>> output properties. Also, If we can show the user with
>>> available
>>>>>>> fields
>>>>>>>>> with
>>>>>>>>>> metadata (of input event) populated in that same
>>>>>>>>>> collection
>>>>> property,
>>>>>>>>> then
>>>>>>>>>> the user can add/remove/updates fields easily. Let's say
>>>>>>>>>> they
>>>>> just
>>>>>>> have
>>>>>>>>> to
>>>>>>>>>> return a map containing all the fields they defined in
>>>>>>>>>> output
>>>>>>> properties
>>>>>>>>>> (i.e return {"existingId": 1, "existingTempInKelvin":
>>>>>>>>>> 301,
>>>>>>>>>> "newTempInCelsius": 28 } ).
>>>>>>>>>> 
>>>>>>>>>> [image: sample.png]
>>>>>>>>>> 
>>>>>>>>>> Regards,
>>>>>>>>>> Grainier.
>>>>>>>>>> 
>>>>>>>>>> On Thu, 14 May 2020 at 12:33, Patrick Wiener <
>>> wiener@apache.org>
>>>>>>> wrote:
>>>>>>>>>> 
>>>>>>>>>>> Hi,
>>>>>>>>>>> 
>>>>>>>>>>> sounds reasonable.
>>>>>>>>>>> 
>>>>>>>>>>> I share the thoughts of Philipp regarding the output
>>> schema.
>>>>> But
>>>>>>> as you
>>>>>>>>>>> said Grainier, we can also improve :)
>>>>>>>>>>> 
>>>>>>>>>>> btw: I really like the idea to also allow the more
>>> technical
>>>>> user
>>>>>>> to
>>>>>>>>>>> easily integrate their function snippets without
>>>>>>>>>>> having to care about the actual execution. This gets a
>>>>>>>>>>> more FaaS
>>> like
>>>>>>> feeling
>>>>>>>>> for
>>>>>>>>>>> streaming.
>>>>>>>>>>> 
>>>>>>>>>>> Patrick
>>>>>>>>>>> 
>>>>>>>>>>>> Am 14.05.2020 um 08:26 schrieb Philipp Zehnder <
>>>>>>> zehnder@apache.org>:
>>>>>>>>>>>> 
>>>>>>>>>>>> Hi,
>>>>>>>>>>>> 
>>>>>>>>>>>> it would be very useful to have such a processor.
>>>>>>>>>>>> 
>>>>>>>>>>>> I like the ideas for 1 & 2 and I think we can do it
>>>>>>>>>>>> this
>>> way.
>>>>>>>>>>>> 
>>>>>>>>>>>> Actually I think that the definition of the output
>>>>>>>>>>>> scheme
>>>>> could
>>>>>>> be
>>>>>>>>> quite
>>>>>>>>>>> tricky.
>>>>>>>>>>>> Because users can do anything with the event.
>>>>>>>>>>>> 
>>>>>>>>>>>> How about we provide an option property?
>>>>>>>>>>>> Users can select whether they want to define the
>>>>>>>>>>>> whole
>>> event
>>>>> or
>>>>>>> just
>>>>>>>>>>> append properties.
>>>>>>>>>>>> 
>>>>>>>>>>>> For the configurations of the properties we could do
>>>>> something
>>>>>>>>> similar
>>>>>>>>>>> then we have in the PLC4xS7Adapter.
>>>>>>>>>>>> A collection property containing the fields of the
>>>>>>>>>>>> output
>>>>>>> properties.
>>>>>>>>>>>> Therefore, we have to decide what information we
>>>>>>>>>>>> have to
>>>>>>> provide. I
>>>>>>>>>>> think if we would support all options it becomes very
>>> cluttered
>>>>>>> for the
>>>>>>>>>>> user.
>>>>>>>>>>>> 
>>>>>>>>>>>> What do you think?
>>>>>>>>>>>> 
>>>>>>>>>>>> Philipp
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>>> On 14. May 2020, at 06:55, Grainier Perera <
>>>>>>>>> grainier.perera@gmail.com>
>>>>>>>>>>> wrote:
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Hi Dominik,
>>>>>>>>>>>>> 
>>>>>>>>>>>>> I agree. We have to do some enhancements make this
>>> processor
>>>>>>> user
>>>>>>>>>>> friendly.
>>>>>>>>>>>>> Like you said it's nice to have;
>>>>>>>>>>>>>  1. A new static property that supports code
>>> highlighting &
>>>>>>> syntax
>>>>>>>>>>>>> checking. We can simply do such client-side
>>>>>>>>>>>>> validations
>>>>> using
>>>>>>>>>>> highlight.js
>>>>>>>>>>>>> (BSD) [1], jshint (MIT) [2], etc...
>>>>>>>>>>>>>  2. A mechanism to show the user with available
>>>>>>>>>>>>> fields
>>>>> they can
>>>>>>>>> use to
>>>>>>>>>>>>> write code (similar to what you've done with usable
>>>>> templates in
>>>>>>>>>>>>> EmailPublisher using HtmlInputParameter).
>>>>>>>>>>>>>  3. A mechanism to map output schema.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> I think 1 & 2 go together and for the 3rd
>>>>>>>>>>>>> requirement
>>> given
>>>>>>> that we
>>>>>>>>> are
>>>>>>>>>>>>> targetting a more technical user group for this
>>> processor,
>>>>> we
>>>>>>> can
>>>>>>>>> let
>>>>>>>>>>> them
>>>>>>>>>>>>> define the output. Otherwise, it would be quite
>>> difficult to
>>>>>>> derive
>>>>>>>>>>> return
>>>>>>>>>>>>> type from the JS. Moreover, I'm planning to use
>>>>>>>>>>>>> Java's
>>>>> builtin
>>>>>>>>>>> ScriptEngine
>>>>>>>>>>>>> to eval. Of course, there'll be limitations. But,
>>>>>>>>>>>>> we can
>>>>> always
>>>>>>>>> improve
>>>>>>>>>>>>> them :)
>>>>>>>>>>>>> 
>>>>>>>>>>>>> [1] https://highlightjs.org/ [2]
>>>>>>>>>>>>> https://jshint.com/
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>> Grainier.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> On Thu, 14 May 2020 at 00:53, Dominik Riemer <
>>>>> riemer@apache.org
>>>>>>>> 
>>>>>>>>> wrote:
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Hi Grainier,
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> in my opinion, this would definitely be a very
>>> interesting
>>>>>>>>> addition! We
>>>>>>>>>>>>>> already briefly discussed such a feature in the
>>>>>>>>>>>>>> past,
>>>>> before we
>>>>>>>>> went
>>>>>>>>>>> to the
>>>>>>>>>>>>>> ASF. I’d say that a JS evaluator would be
>>>>>>>>>>>>>> definitely
>>>>> useful to
>>>>>>> a
>>>>>>>>>>> slightly
>>>>>>>>>>>>>> more technical target user group, but would
>>>>>>>>>>>>>> provide
>>> great
>>>>>>>>> flexibility
>>>>>>>>>>> for,
>>>>>>>>>>>>>> e.g., data harmonization tasks. So let’s discuss
>>>>>>>>>>>>>> what
>>> is
>>>>>>> needed to
>>>>>>>>>>>>>> implement this! I’d guess that we (maybe) need to
>>>>>>>>>>>>>> add
>>> some
>>>>>>>>>>> enhancements to
>>>>>>>>>>>>>> the core to make this processor good from a
>>>>>>>>>>>>>> usability
>>>>>>> perspective.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> From a conceptual point of view, I guess the JS
>>> processor
>>>>> would
>>>>>>>>> have no
>>>>>>>>>>>>>> specific input requirements and a single static
>>> property
>>>>> that
>>>>>>>>> allows
>>>>>>>>>>> users
>>>>>>>>>>>>>> to enter the code (maybe it would be cool to add a
>>>>>>>>>>>>>> new
>>>>> static
>>>>>>>>> property
>>>>>>>>>>> here
>>>>>>>>>>>>>> that supports code highlighting or even syntax
>>> checking).
>>>>> The
>>>>>>> only
>>>>>>>>> open
>>>>>>>>>>>>>> issue I see is the output schema – we would
>>>>>>>>>>>>>> somehow
>>> need to
>>>>>>>>> extract the
>>>>>>>>>>>>>> output from the JS function. This could probably
>>> partially
>>>>> be
>>>>>>> done
>>>>>>>>>>> using
>>>>>>>>>>>>>> the CustomTransformOutput (see
>>>>>>>>>>>>>> 
>>>>>>>>> 
>>>>> http://streampipes.apache.org/docs/docs/dev-guide-output-strategie
>>>>> s/),
>>>>>>>>>>>>>> but we would somehow need to derive type
>>>>>>>>>>>>>> information
>>> from
>>>>> the
>>>>>>> JS
>>>>>>>>>>> function
>>>>>>>>>>>>>> or introduce a feature to let users define/refine
>>>>>>>>>>>>>> the
>>>>> output
>>>>>>>>> manually
>>>>>>>>>>>>>> (e.g., to add semantic metadata).
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> What do you think would be best? And would you
>>>>>>>>>>>>>> evaluate
>>>>> the JS
>>>>>>>>> code in
>>>>>>>>>>>>>> Java or somehow else?
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Also, as I’ve just started to improve the SDK
>>>>>>>>>>>>>> (e.g.,
>>>>> supporting
>>>>>>>>>>> optional
>>>>>>>>>>>>>> static properties and an easier way to define and
>>> extract
>>>>> model
>>>>>>>>>>>>>> parameters), we can collect your requirements for
>>>>>>>>>>>>>> the
>>> JS
>>>>>>> processor
>>>>>>>>> and
>>>>>>>>>>>>>> extend the SDK if needed, just say what you would
>>>>>>>>>>>>>> like
>>> to
>>>>> have
>>>>>>> 😉
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Dominik
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> From: Grainier Perera <gr...@gmail.com>
>>>>>>>>>>>>>> Sent: Wednesday, May 13, 2020 2:41 PM
>>>>>>>>>>>>>> To: dev@streampipes.apache.org
>>>>>>>>>>>>>> Subject: Data processor to evaluate JavaScript
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Hi all,
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> I'm planning to implement a JavaScript eval data
>>>>> processor. As
>>>>>>> the
>>>>>>>>> name
>>>>>>>>>>>>>> suggests, users will be able to write some
>>>>>>>>>>>>>> JavaScript
>>> code
>>>>>>> which
>>>>>>>>> takes
>>>>>>>>>>> in
>>>>>>>>>>>>>> an event (as a map), do some processing on the
>>>>>>>>>>>>>> event,
>>> and
>>>>>>> return a
>>>>>>>>> new
>>>>>>>>>>> map
>>>>>>>>>>>>>> which then gets converted to an sp-event.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> example js:
>>>>>>>>>>>>>> function process(event) {
>>>>>>>>>>>>>>  // do processing here.
>>>>>>>>>>>>>>  // return processed event.
>>>>>>>>>>>>>>  return {id: http://event.id, tempInCelsius:
>>>>>>> (event.tempInKelvin
>>>>>>>>> -
>>>>>>>>>>>>>> 273.15)};
>>>>>>>>>>>>>> };
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Will this be a good addition to pipeline-elements?
>>> What do
>>>>> you
>>>>>>>>> think?
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>> Grainier.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>>> 
>> 
>>

Re: Data processor to evaluate JavaScript

Posted by Grainier Perera <gr...@gmail.com>.

Hi Dominik,

My bad. There were some cached images & that was the issue. Also fixed the
PR as per your comments. Please review and merge [1].

[1] https://github.com/apache/incubator-streampipes-extensions/pull/15

Thanks & Regards,
Grainier

On Fri, 22 May 2020 at 16:25, Dominik Riemer <ri...@apache.org> wrote:

> Hi Grainier,
>
> we have a Github Actions workflow [1] for core+extensions that usually
> builds and pushes the latest development containers for dev on each push.
> You are using the latest CLI from dev and the tmpl_env or .env version is
> set to 0.67.0-SNAPSHOT, right?
>
> I just tested the JsEval processor (with CLI 0.67.0-SNAPSHOT, local
> backend with the latest changes from dev and your PR) and it works great 😊
> The only thing I had to change was a typo in line 59 (use outEvent instead
> of event).
>
> In the controller, the supportedProtocols and supportedFormats are no
> longer needed, as they can now be declared for a whole module in the Init
> class (registerFormats/Protocols). I need to update this in the
> documentation.
>
> Dominik
>
>
>
>
>
> https://github.com/apache/incubator-streampipes/blob/dev/.github/workflows/build.yml
>
> -----Original Message-----
> From: Grainier Perera <gr...@gmail.com>
> Sent: Friday, May 22, 2020 11:58 AM
> To: dev@streampipes.apache.org
> Subject: Re: Data processor to evaluate JavaScript
>
> Hi Dominik,
>
> I've implemented the following[1]. However, when I try to install the
> processor it gives me the following error. Any clue? I've used CLI to start
> sp. So, maybe the snapshot docker image doesn't have the recent changes? If
> so what are the steps to build the images with changes in-place?
>
> [1] https://github.com/apache/incubator-streampipes-extensions/pull/15
>
> ERROR: https://streampipes.org/vocabulary/v1/UserDefinedOutputStrategy
> io.fogsy.empire.pinto.RDFMappingException: Could not create an instance of
> class org.apache.streampipes.model.output.OutputStrategy, it does not have
> a default constructor at
> io.fogsy.empire.pinto.RDFMapper.newInstance(RDFMapper.java:178)
> at io.fogsy.empire.pinto.RDFMapper.readValue(RDFMapper.java:261)
> at io.fogsy.empire.pinto.RDFMapper.valueToObject(RDFMapper.java:695)
> at io.fogsy.empire.pinto.RDFMapper.lambda$toObject$3(RDFMapper.java:388)
> at
> io.fogsy.empire.pinto.RDFMapper$$Lambda$802/0000000000096040.apply(Unknown
> Source)
> at
> io.fogsy.empire.pinto.RDFMapper$$Lambda$803/0000000000096580.apply(Unknown
> Source)
> at java.util.stream.ReferencePipeline$3$1.accept(Unknown Source) at
> java.util.ArrayList$ArrayListSpliterator.forEachRemaining(Unknown Source)
> at java.util.stream.AbstractPipeline.copyInto(Unknown Source) at
> java.util.stream.AbstractPipeline.wrapAndCopyInto(Unknown Source) at
> java.util.stream.ForEachOps$ForEachOp.evaluateSequential(Unknown Source) at
> java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(Unknown
> Source)
> at java.util.stream.AbstractPipeline.evaluate(Unknown Source) at
> java.util.stream.ReferencePipeline.forEach(Unknown Source) at
> io.fogsy.empire.pinto.RDFMapper.readValue(RDFMapper.java:299)
> at
>
> org.apache.streampipes.serializers.jsonld.JsonLdTransformer.fromJsonLd(JsonLdTransformer.java:100)
> at
>
> org.apache.streampipes.manager.verification.ElementVerifier.transform(ElementVerifier.java:169)
> at
>
> org.apache.streampipes.manager.verification.ElementVerifier.verifyAndAdd(ElementVerifier.java:84)
> at
>
> org.apache.streampipes.manager.operations.Operations.verifyAndAddElement(Operations.java:96)
> at
>
> org.apache.streampipes.manager.endpoint.EndpointItemParser.parseAndAddEndpointItem(EndpointItemParser.java:37)
> at
>
> org.apache.streampipes.rest.impl.PipelineElementImport.verifyAndAddElement(PipelineElementImport.java:93)
> at
>
> org.apache.streampipes.rest.impl.PipelineElementImport.addElement(PipelineElementImport.java:89)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
> sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
>
> Thanks,
> Grainier.
>
> On Thu, 21 May 2020 at 03:39, Dominik Riemer <ri...@apache.org> wrote:
>
> > Hi Grainier,
> >
> > sounds good!
> > I've just pushed a commit that provides an initial version of the
> > user-defined output strategy. It lets you select the input schema from
> > the incoming stream and add/remove event properties. There is one
> > limitation, the component currently only works for flat event
> > structures, but it shouldn't be a problem to have a more advanced
> > schema editor for this output strategy soon (we already have this in
> > the StreamPipes connect code, but unfortunately, the pipeline editor
> > has yet to be migrated from AngularJS to Angular 2+ to reuse these
> > components). This is something we plan to do within the next weeks.
> >
> > I added an example how to use the strategy to the examples repo:
> >
> > https://github.com/apache/incubator-streampipes-examples/blob/dev/stre
> > ampipes-pipeline-elements-examples-processors-jvm/src/main/java/org/ap
> > ache/streampipes/pe/examples/jvm/staticproperty/CodeInputExampleContro
> > ller.java
> >
> > Let me know what you think and if you need anything more/else!
> >
> > Dominik
> >
> > On 2020/05/19 17:14:21, Grainier Perera <gr...@gmail.com>
> > wrote:
> > > Hi Dominik,
> > >
> > > Awesome. This is really cool & exactly what I was looking for. I'll
> > > use this with my implementation, and keep this thread posted with
> > > the
> > updates.
> > > ps: let us know when you add output strategy as well.
> > >
> > > Thanks,
> > > Grainier.
> > >
> > > On Tue, 19 May 2020 at 03:05, Dominik Riemer <ri...@apache.org>
> wrote:
> > >
> > > > Hi Grainier,
> > > >
> > > > I just pushed an initial version of the static property to add
> > javascript
> > > > code.
> > > > It's a simple codeblock with linting enabled and a (currently very
> > simple)
> > > > autocomplete feature that suggests existing properties from the
> > > > input
> > event
> > > > (currently, only if you press ctrl-space directly after typing
> "event"
> > > > followed by a dot, I'll fix that soon ;-)
> > > >
> > > > You can find an example how to use it here:
> > > >
> > > >
> > https://github.com/apache/incubator-streampipes-examples/blob/dev/stre
> > ampipes-pipeline-elements-examples-processors-jvm/src/main/java/org/ap
> > ache/streampipes/pe/examples/jvm/staticproperty/CodeInputExampleContro
> > ller.java
> > > >
> > > > Feedback and ideas for improvement are welcome!
> > > >
> > > > Dominik
> > > >
> > > >
> > > > On 2020/05/17 12:36:05, Grainier Perera
> > > > <gr...@gmail.com>
> > > > wrote:
> > > > > Hi Dominik,
> > > > >
> > > > > Thank you. I'll work on `Implement JS evaluator` [1].
> > > > >
> > > > > [1] https://issues.apache.org/jira/browse/STREAMPIPES-135
> > > > >
> > > > > Grainier.
> > > > >
> > > > > On Sun, 17 May 2020 at 18:02, Dominik Riemer <ri...@apache.org>
> > wrote:
> > > > >
> > > > > > Hi Grainier,
> > > > > >
> > > > > > great! I created an umbrella issue for that:
> > > > > > https://issues.apache.org/jira/browse/STREAMPIPES-132
> > > > > >
> > > > > > I'll now start to implement the new static property for
> > > > > > entering
> > code.
> > > > > >
> > > > > > Dominik
> > > > > >
> > > > > > On 2020/05/15 01:29:44, Grainier Perera
> > > > > > <grainier.perera@gmail.com
> > >
> > > > > > wrote:
> > > > > > > Hi,
> > > > > > >
> > > > > > > I'm +1 to the idea of `UserDefinedOutputStrategy`. As you
> > > > > > > said,
> > > > that'll
> > > > > > be
> > > > > > > re-usable in the future processors as well. So let's proceed
> > > > > > > with
> > > > that.
> > > > > > >
> > > > > > > Regards,
> > > > > > > Grainier
> > > > > > >
> > > > > > > On Fri, 15 May 2020 at 01:37, Dominik Riemer
> > > > > > > <ri...@apache.org>
> > > > wrote:
> > > > > > >
> > > > > > > > Hi,
> > > > > > > > cool, that sounds very good!
> > > > > > > >
> > > > > > > > Concerning the output, I think there are two ways to
> > > > > > > > achieve
> > this:
> > > > > > > > - A collection static property in conjunction with a
> > > > > > > > runtime
> > > > resolvable
> > > > > > > > output strategy that creates the output schema based on
> > > > > > > > the
> > input
> > > > from
> > > > > > the
> > > > > > > > collection
> > > > > > > > - Or we create a new output strategy, e.g., let's call it
> > > > > > > > UserDefinedOutputStrategy, which would render a UI
> > > > > > > > component
> > to let
> > > > > > users
> > > > > > > > add/remove/change event properties from the input schema.
> > > > > > > > I
> > also
> > > > like
> > > > > > the
> > > > > > > > idea of preselecting all input properties.
> > > > > > > >
> > > > > > > > I personally tend towards the second option as such a
> > > > > > > > strategy
> > > > might be
> > > > > > > > useful for other pipeline elements as well and it would
> > > > > > > > ease
> > the
> > > > model
> > > > > > > > definition. We could also extend this component in the
> > > > > > > > future,
> > > > e.g., by
> > > > > > > > letting users enrich metadata such as measurement units
> > directly
> > > > in a
> > > > > > > > processor.
> > > > > > > >
> > > > > > > > But I'm also open to the first option and we can also just
> > start
> > > > > > > > implementing to see what works best.
> > > > > > > >
> > > > > > > > I'd also be happy to help with the implementation, e.g., I
> > could
> > > > work
> > > > > > on
> > > > > > > > the new static property for code input or the output
> strategy!
> > > > > > > >
> > > > > > > > Dominik
> > > > > > > >
> > > > > > > > On 2020/05/14 11:59:36, Grainier Perera <
> > grainier.perera@gmail.com
> > > > >
> > > > > > > > wrote:
> > > > > > > > > Hi,
> > > > > > > > >
> > > > > > > > > +1 to the idea of having a collection property
> > > > > > > > > +containing the
> > > > fields
> > > > > > of
> > > > > > > > the
> > > > > > > > > output properties. Also, If we can show the user with
> > available
> > > > > > fields
> > > > > > > > with
> > > > > > > > > metadata (of input event) populated in that same
> > > > > > > > > collection
> > > > property,
> > > > > > > > then
> > > > > > > > > the user can add/remove/updates fields easily. Let's say
> > > > > > > > > they
> > > > just
> > > > > > have
> > > > > > > > to
> > > > > > > > > return a map containing all the fields they defined in
> > > > > > > > > output
> > > > > > properties
> > > > > > > > > (i.e return {"existingId": 1, "existingTempInKelvin":
> > > > > > > > > 301,
> > > > > > > > > "newTempInCelsius": 28 } ).
> > > > > > > > >
> > > > > > > > > [image: sample.png]
> > > > > > > > >
> > > > > > > > > Regards,
> > > > > > > > > Grainier.
> > > > > > > > >
> > > > > > > > > On Thu, 14 May 2020 at 12:33, Patrick Wiener <
> > wiener@apache.org>
> > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi,
> > > > > > > > > >
> > > > > > > > > > sounds reasonable.
> > > > > > > > > >
> > > > > > > > > > I share the thoughts of Philipp regarding the output
> > schema.
> > > > But
> > > > > > as you
> > > > > > > > > > said Grainier, we can also improve :)
> > > > > > > > > >
> > > > > > > > > > btw: I really like the idea to also allow the more
> > technical
> > > > user
> > > > > > to
> > > > > > > > > > easily integrate their function snippets without
> > > > > > > > > > having to care about the actual execution. This gets a
> > > > > > > > > > more FaaS
> > like
> > > > > > feeling
> > > > > > > > for
> > > > > > > > > > streaming.
> > > > > > > > > >
> > > > > > > > > > Patrick
> > > > > > > > > >
> > > > > > > > > > > Am 14.05.2020 um 08:26 schrieb Philipp Zehnder <
> > > > > > zehnder@apache.org>:
> > > > > > > > > > >
> > > > > > > > > > > Hi,
> > > > > > > > > > >
> > > > > > > > > > > it would be very useful to have such a processor.
> > > > > > > > > > >
> > > > > > > > > > > I like the ideas for 1 & 2 and I think we can do it
> > > > > > > > > > > this
> > way.
> > > > > > > > > > >
> > > > > > > > > > > Actually I think that the definition of the output
> > > > > > > > > > > scheme
> > > > could
> > > > > > be
> > > > > > > > quite
> > > > > > > > > > tricky.
> > > > > > > > > > > Because users can do anything with the event.
> > > > > > > > > > >
> > > > > > > > > > > How about we provide an option property?
> > > > > > > > > > > Users can select whether they want to define the
> > > > > > > > > > > whole
> > event
> > > > or
> > > > > > just
> > > > > > > > > > append properties.
> > > > > > > > > > >
> > > > > > > > > > > For the configurations of the properties we could do
> > > > something
> > > > > > > > similar
> > > > > > > > > > then we have in the PLC4xS7Adapter.
> > > > > > > > > > > A collection property containing the fields of the
> > > > > > > > > > > output
> > > > > > properties.
> > > > > > > > > > > Therefore, we have to decide what information we
> > > > > > > > > > > have to
> > > > > > provide. I
> > > > > > > > > > think if we would support all options it becomes very
> > cluttered
> > > > > > for the
> > > > > > > > > > user.
> > > > > > > > > > >
> > > > > > > > > > > What do you think?
> > > > > > > > > > >
> > > > > > > > > > > Philipp
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >> On 14. May 2020, at 06:55, Grainier Perera <
> > > > > > > > grainier.perera@gmail.com>
> > > > > > > > > > wrote:
> > > > > > > > > > >>
> > > > > > > > > > >> Hi Dominik,
> > > > > > > > > > >>
> > > > > > > > > > >> I agree. We have to do some enhancements make this
> > processor
> > > > > > user
> > > > > > > > > > friendly.
> > > > > > > > > > >> Like you said it's nice to have;
> > > > > > > > > > >>   1. A new static property that supports code
> > highlighting &
> > > > > > syntax
> > > > > > > > > > >> checking. We can simply do such client-side
> > > > > > > > > > >> validations
> > > > using
> > > > > > > > > > highlight.js
> > > > > > > > > > >> (BSD) [1], jshint (MIT) [2], etc...
> > > > > > > > > > >>   2. A mechanism to show the user with available
> > > > > > > > > > >> fields
> > > > they can
> > > > > > > > use to
> > > > > > > > > > >> write code (similar to what you've done with usable
> > > > templates in
> > > > > > > > > > >> EmailPublisher using HtmlInputParameter).
> > > > > > > > > > >>   3. A mechanism to map output schema.
> > > > > > > > > > >>
> > > > > > > > > > >> I think 1 & 2 go together and for the 3rd
> > > > > > > > > > >> requirement
> > given
> > > > > > that we
> > > > > > > > are
> > > > > > > > > > >> targetting a more technical user group for this
> > processor,
> > > > we
> > > > > > can
> > > > > > > > let
> > > > > > > > > > them
> > > > > > > > > > >> define the output. Otherwise, it would be quite
> > difficult to
> > > > > > derive
> > > > > > > > > > return
> > > > > > > > > > >> type from the JS. Moreover, I'm planning to use
> > > > > > > > > > >> Java's
> > > > builtin
> > > > > > > > > > ScriptEngine
> > > > > > > > > > >> to eval. Of course, there'll be limitations. But,
> > > > > > > > > > >> we can
> > > > always
> > > > > > > > improve
> > > > > > > > > > >> them :)
> > > > > > > > > > >>
> > > > > > > > > > >> [1] https://highlightjs.org/ [2]
> > > > > > > > > > >> https://jshint.com/
> > > > > > > > > > >>
> > > > > > > > > > >> Regards,
> > > > > > > > > > >> Grainier.
> > > > > > > > > > >>
> > > > > > > > > > >> On Thu, 14 May 2020 at 00:53, Dominik Riemer <
> > > > riemer@apache.org
> > > > > > >
> > > > > > > > wrote:
> > > > > > > > > > >>
> > > > > > > > > > >>> Hi Grainier,
> > > > > > > > > > >>>
> > > > > > > > > > >>> in my opinion, this would definitely be a very
> > interesting
> > > > > > > > addition! We
> > > > > > > > > > >>> already briefly discussed such a feature in the
> > > > > > > > > > >>> past,
> > > > before we
> > > > > > > > went
> > > > > > > > > > to the
> > > > > > > > > > >>> ASF. I’d say that a JS evaluator would be
> > > > > > > > > > >>> definitely
> > > > useful to
> > > > > > a
> > > > > > > > > > slightly
> > > > > > > > > > >>> more technical target user group, but would
> > > > > > > > > > >>> provide
> > great
> > > > > > > > flexibility
> > > > > > > > > > for,
> > > > > > > > > > >>> e.g., data harmonization tasks. So let’s discuss
> > > > > > > > > > >>> what
> > is
> > > > > > needed to
> > > > > > > > > > >>> implement this! I’d guess that we (maybe) need to
> > > > > > > > > > >>> add
> > some
> > > > > > > > > > enhancements to
> > > > > > > > > > >>> the core to make this processor good from a
> > > > > > > > > > >>> usability
> > > > > > perspective.
> > > > > > > > > > >>>
> > > > > > > > > > >>> From a conceptual point of view, I guess the JS
> > processor
> > > > would
> > > > > > > > have no
> > > > > > > > > > >>> specific input requirements and a single static
> > property
> > > > that
> > > > > > > > allows
> > > > > > > > > > users
> > > > > > > > > > >>> to enter the code (maybe it would be cool to add a
> > > > > > > > > > >>> new
> > > > static
> > > > > > > > property
> > > > > > > > > > here
> > > > > > > > > > >>> that supports code highlighting or even syntax
> > checking).
> > > > The
> > > > > > only
> > > > > > > > open
> > > > > > > > > > >>> issue I see is the output schema – we would
> > > > > > > > > > >>> somehow
> > need to
> > > > > > > > extract the
> > > > > > > > > > >>> output from the JS function. This could probably
> > partially
> > > > be
> > > > > > done
> > > > > > > > > > using
> > > > > > > > > > >>> the CustomTransformOutput (see
> > > > > > > > > > >>>
> > > > > > > >
> > > > http://streampipes.apache.org/docs/docs/dev-guide-output-strategie
> > > > s/),
> > > > > > > > > > >>> but we would somehow need to derive type
> > > > > > > > > > >>> information
> > from
> > > > the
> > > > > > JS
> > > > > > > > > > function
> > > > > > > > > > >>> or introduce a feature to let users define/refine
> > > > > > > > > > >>> the
> > > > output
> > > > > > > > manually
> > > > > > > > > > >>> (e.g., to add semantic metadata).
> > > > > > > > > > >>>
> > > > > > > > > > >>> What do you think would be best? And would you
> > > > > > > > > > >>> evaluate
> > > > the JS
> > > > > > > > code in
> > > > > > > > > > >>> Java or somehow else?
> > > > > > > > > > >>>
> > > > > > > > > > >>> Also, as I’ve just started to improve the SDK
> > > > > > > > > > >>> (e.g.,
> > > > supporting
> > > > > > > > > > optional
> > > > > > > > > > >>> static properties and an easier way to define and
> > extract
> > > > model
> > > > > > > > > > >>> parameters), we can collect your requirements for
> > > > > > > > > > >>> the
> > JS
> > > > > > processor
> > > > > > > > and
> > > > > > > > > > >>> extend the SDK if needed, just say what you would
> > > > > > > > > > >>> like
> > to
> > > > have
> > > > > > 😉
> > > > > > > > > > >>>
> > > > > > > > > > >>> Dominik
> > > > > > > > > > >>>
> > > > > > > > > > >>>
> > > > > > > > > > >>> From: Grainier Perera <gr...@gmail.com>
> > > > > > > > > > >>> Sent: Wednesday, May 13, 2020 2:41 PM
> > > > > > > > > > >>> To: dev@streampipes.apache.org
> > > > > > > > > > >>> Subject: Data processor to evaluate JavaScript
> > > > > > > > > > >>>
> > > > > > > > > > >>> Hi all,
> > > > > > > > > > >>>
> > > > > > > > > > >>> I'm planning to implement a JavaScript eval data
> > > > processor. As
> > > > > > the
> > > > > > > > name
> > > > > > > > > > >>> suggests, users will be able to write some
> > > > > > > > > > >>> JavaScript
> > code
> > > > > > which
> > > > > > > > takes
> > > > > > > > > > in
> > > > > > > > > > >>> an event (as a map), do some processing on the
> > > > > > > > > > >>> event,
> > and
> > > > > > return a
> > > > > > > > new
> > > > > > > > > > map
> > > > > > > > > > >>> which then gets converted to an sp-event.
> > > > > > > > > > >>>
> > > > > > > > > > >>> example js:
> > > > > > > > > > >>> function process(event) {
> > > > > > > > > > >>>   // do processing here.
> > > > > > > > > > >>>   // return processed event.
> > > > > > > > > > >>>   return {id: http://event.id, tempInCelsius:
> > > > > > (event.tempInKelvin
> > > > > > > > -
> > > > > > > > > > >>> 273.15)};
> > > > > > > > > > >>> };
> > > > > > > > > > >>>
> > > > > > > > > > >>> Will this be a good addition to pipeline-elements?
> > What do
> > > > you
> > > > > > > > think?
> > > > > > > > > > >>>
> > > > > > > > > > >>> Regards,
> > > > > > > > > > >>> Grainier.
> > > > > > > > > > >>>
> > > > > > > > > > >>>
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
>

RE: Data processor to evaluate JavaScript

Posted by Dominik Riemer <ri...@apache.org>.

Hi Grainier,

we have a Github Actions workflow [1] for core+extensions that usually builds and pushes the latest development containers for dev on each push. You are using the latest CLI from dev and the tmpl_env or .env version is set to 0.67.0-SNAPSHOT, right?

I just tested the JsEval processor (with CLI 0.67.0-SNAPSHOT, local backend with the latest changes from dev and your PR) and it works great 😊
The only thing I had to change was a typo in line 59 (use outEvent instead of event).

In the controller, the supportedProtocols and supportedFormats are no longer needed, as they can now be declared for a whole module in the Init class (registerFormats/Protocols). I need to update this in the documentation.

Dominik




https://github.com/apache/incubator-streampipes/blob/dev/.github/workflows/build.yml

-----Original Message-----
From: Grainier Perera <gr...@gmail.com> 
Sent: Friday, May 22, 2020 11:58 AM
To: dev@streampipes.apache.org
Subject: Re: Data processor to evaluate JavaScript

Hi Dominik,

I've implemented the following[1]. However, when I try to install the processor it gives me the following error. Any clue? I've used CLI to start sp. So, maybe the snapshot docker image doesn't have the recent changes? If so what are the steps to build the images with changes in-place?

[1] https://github.com/apache/incubator-streampipes-extensions/pull/15

ERROR: https://streampipes.org/vocabulary/v1/UserDefinedOutputStrategy
io.fogsy.empire.pinto.RDFMappingException: Could not create an instance of class org.apache.streampipes.model.output.OutputStrategy, it does not have a default constructor at io.fogsy.empire.pinto.RDFMapper.newInstance(RDFMapper.java:178)
at io.fogsy.empire.pinto.RDFMapper.readValue(RDFMapper.java:261)
at io.fogsy.empire.pinto.RDFMapper.valueToObject(RDFMapper.java:695)
at io.fogsy.empire.pinto.RDFMapper.lambda$toObject$3(RDFMapper.java:388)
at
io.fogsy.empire.pinto.RDFMapper$$Lambda$802/0000000000096040.apply(Unknown
Source)
at
io.fogsy.empire.pinto.RDFMapper$$Lambda$803/0000000000096580.apply(Unknown
Source)
at java.util.stream.ReferencePipeline$3$1.accept(Unknown Source) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(Unknown Source) at java.util.stream.AbstractPipeline.copyInto(Unknown Source) at java.util.stream.AbstractPipeline.wrapAndCopyInto(Unknown Source) at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(Unknown Source) at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(Unknown
Source)
at java.util.stream.AbstractPipeline.evaluate(Unknown Source) at java.util.stream.ReferencePipeline.forEach(Unknown Source) at io.fogsy.empire.pinto.RDFMapper.readValue(RDFMapper.java:299)
at
org.apache.streampipes.serializers.jsonld.JsonLdTransformer.fromJsonLd(JsonLdTransformer.java:100)
at
org.apache.streampipes.manager.verification.ElementVerifier.transform(ElementVerifier.java:169)
at
org.apache.streampipes.manager.verification.ElementVerifier.verifyAndAdd(ElementVerifier.java:84)
at
org.apache.streampipes.manager.operations.Operations.verifyAndAddElement(Operations.java:96)
at
org.apache.streampipes.manager.endpoint.EndpointItemParser.parseAndAddEndpointItem(EndpointItemParser.java:37)
at
org.apache.streampipes.rest.impl.PipelineElementImport.verifyAndAddElement(PipelineElementImport.java:93)
at
org.apache.streampipes.rest.impl.PipelineElementImport.addElement(PipelineElementImport.java:89)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)

Thanks,
Grainier.

On Thu, 21 May 2020 at 03:39, Dominik Riemer <ri...@apache.org> wrote:

> Hi Grainier,
>
> sounds good!
> I've just pushed a commit that provides an initial version of the 
> user-defined output strategy. It lets you select the input schema from 
> the incoming stream and add/remove event properties. There is one 
> limitation, the component currently only works for flat event 
> structures, but it shouldn't be a problem to have a more advanced 
> schema editor for this output strategy soon (we already have this in 
> the StreamPipes connect code, but unfortunately, the pipeline editor 
> has yet to be migrated from AngularJS to Angular 2+ to reuse these 
> components). This is something we plan to do within the next weeks.
>
> I added an example how to use the strategy to the examples repo:
>
> https://github.com/apache/incubator-streampipes-examples/blob/dev/stre
> ampipes-pipeline-elements-examples-processors-jvm/src/main/java/org/ap
> ache/streampipes/pe/examples/jvm/staticproperty/CodeInputExampleContro
> ller.java
>
> Let me know what you think and if you need anything more/else!
>
> Dominik
>
> On 2020/05/19 17:14:21, Grainier Perera <gr...@gmail.com>
> wrote:
> > Hi Dominik,
> >
> > Awesome. This is really cool & exactly what I was looking for. I'll 
> > use this with my implementation, and keep this thread posted with 
> > the
> updates.
> > ps: let us know when you add output strategy as well.
> >
> > Thanks,
> > Grainier.
> >
> > On Tue, 19 May 2020 at 03:05, Dominik Riemer <ri...@apache.org> wrote:
> >
> > > Hi Grainier,
> > >
> > > I just pushed an initial version of the static property to add
> javascript
> > > code.
> > > It's a simple codeblock with linting enabled and a (currently very
> simple)
> > > autocomplete feature that suggests existing properties from the 
> > > input
> event
> > > (currently, only if you press ctrl-space directly after typing "event"
> > > followed by a dot, I'll fix that soon ;-)
> > >
> > > You can find an example how to use it here:
> > >
> > >
> https://github.com/apache/incubator-streampipes-examples/blob/dev/stre
> ampipes-pipeline-elements-examples-processors-jvm/src/main/java/org/ap
> ache/streampipes/pe/examples/jvm/staticproperty/CodeInputExampleContro
> ller.java
> > >
> > > Feedback and ideas for improvement are welcome!
> > >
> > > Dominik
> > >
> > >
> > > On 2020/05/17 12:36:05, Grainier Perera 
> > > <gr...@gmail.com>
> > > wrote:
> > > > Hi Dominik,
> > > >
> > > > Thank you. I'll work on `Implement JS evaluator` [1].
> > > >
> > > > [1] https://issues.apache.org/jira/browse/STREAMPIPES-135
> > > >
> > > > Grainier.
> > > >
> > > > On Sun, 17 May 2020 at 18:02, Dominik Riemer <ri...@apache.org>
> wrote:
> > > >
> > > > > Hi Grainier,
> > > > >
> > > > > great! I created an umbrella issue for that:
> > > > > https://issues.apache.org/jira/browse/STREAMPIPES-132
> > > > >
> > > > > I'll now start to implement the new static property for 
> > > > > entering
> code.
> > > > >
> > > > > Dominik
> > > > >
> > > > > On 2020/05/15 01:29:44, Grainier Perera 
> > > > > <grainier.perera@gmail.com
> >
> > > > > wrote:
> > > > > > Hi,
> > > > > >
> > > > > > I'm +1 to the idea of `UserDefinedOutputStrategy`. As you 
> > > > > > said,
> > > that'll
> > > > > be
> > > > > > re-usable in the future processors as well. So let's proceed 
> > > > > > with
> > > that.
> > > > > >
> > > > > > Regards,
> > > > > > Grainier
> > > > > >
> > > > > > On Fri, 15 May 2020 at 01:37, Dominik Riemer 
> > > > > > <ri...@apache.org>
> > > wrote:
> > > > > >
> > > > > > > Hi,
> > > > > > > cool, that sounds very good!
> > > > > > >
> > > > > > > Concerning the output, I think there are two ways to 
> > > > > > > achieve
> this:
> > > > > > > - A collection static property in conjunction with a 
> > > > > > > runtime
> > > resolvable
> > > > > > > output strategy that creates the output schema based on 
> > > > > > > the
> input
> > > from
> > > > > the
> > > > > > > collection
> > > > > > > - Or we create a new output strategy, e.g., let's call it 
> > > > > > > UserDefinedOutputStrategy, which would render a UI 
> > > > > > > component
> to let
> > > > > users
> > > > > > > add/remove/change event properties from the input schema. 
> > > > > > > I
> also
> > > like
> > > > > the
> > > > > > > idea of preselecting all input properties.
> > > > > > >
> > > > > > > I personally tend towards the second option as such a 
> > > > > > > strategy
> > > might be
> > > > > > > useful for other pipeline elements as well and it would 
> > > > > > > ease
> the
> > > model
> > > > > > > definition. We could also extend this component in the 
> > > > > > > future,
> > > e.g., by
> > > > > > > letting users enrich metadata such as measurement units
> directly
> > > in a
> > > > > > > processor.
> > > > > > >
> > > > > > > But I'm also open to the first option and we can also just
> start
> > > > > > > implementing to see what works best.
> > > > > > >
> > > > > > > I'd also be happy to help with the implementation, e.g., I
> could
> > > work
> > > > > on
> > > > > > > the new static property for code input or the output strategy!
> > > > > > >
> > > > > > > Dominik
> > > > > > >
> > > > > > > On 2020/05/14 11:59:36, Grainier Perera <
> grainier.perera@gmail.com
> > > >
> > > > > > > wrote:
> > > > > > > > Hi,
> > > > > > > >
> > > > > > > > +1 to the idea of having a collection property 
> > > > > > > > +containing the
> > > fields
> > > > > of
> > > > > > > the
> > > > > > > > output properties. Also, If we can show the user with
> available
> > > > > fields
> > > > > > > with
> > > > > > > > metadata (of input event) populated in that same 
> > > > > > > > collection
> > > property,
> > > > > > > then
> > > > > > > > the user can add/remove/updates fields easily. Let's say 
> > > > > > > > they
> > > just
> > > > > have
> > > > > > > to
> > > > > > > > return a map containing all the fields they defined in 
> > > > > > > > output
> > > > > properties
> > > > > > > > (i.e return {"existingId": 1, "existingTempInKelvin": 
> > > > > > > > 301,
> > > > > > > > "newTempInCelsius": 28 } ).
> > > > > > > >
> > > > > > > > [image: sample.png]
> > > > > > > >
> > > > > > > > Regards,
> > > > > > > > Grainier.
> > > > > > > >
> > > > > > > > On Thu, 14 May 2020 at 12:33, Patrick Wiener <
> wiener@apache.org>
> > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi,
> > > > > > > > >
> > > > > > > > > sounds reasonable.
> > > > > > > > >
> > > > > > > > > I share the thoughts of Philipp regarding the output
> schema.
> > > But
> > > > > as you
> > > > > > > > > said Grainier, we can also improve :)
> > > > > > > > >
> > > > > > > > > btw: I really like the idea to also allow the more
> technical
> > > user
> > > > > to
> > > > > > > > > easily integrate their function snippets without 
> > > > > > > > > having to care about the actual execution. This gets a 
> > > > > > > > > more FaaS
> like
> > > > > feeling
> > > > > > > for
> > > > > > > > > streaming.
> > > > > > > > >
> > > > > > > > > Patrick
> > > > > > > > >
> > > > > > > > > > Am 14.05.2020 um 08:26 schrieb Philipp Zehnder <
> > > > > zehnder@apache.org>:
> > > > > > > > > >
> > > > > > > > > > Hi,
> > > > > > > > > >
> > > > > > > > > > it would be very useful to have such a processor.
> > > > > > > > > >
> > > > > > > > > > I like the ideas for 1 & 2 and I think we can do it 
> > > > > > > > > > this
> way.
> > > > > > > > > >
> > > > > > > > > > Actually I think that the definition of the output 
> > > > > > > > > > scheme
> > > could
> > > > > be
> > > > > > > quite
> > > > > > > > > tricky.
> > > > > > > > > > Because users can do anything with the event.
> > > > > > > > > >
> > > > > > > > > > How about we provide an option property?
> > > > > > > > > > Users can select whether they want to define the 
> > > > > > > > > > whole
> event
> > > or
> > > > > just
> > > > > > > > > append properties.
> > > > > > > > > >
> > > > > > > > > > For the configurations of the properties we could do
> > > something
> > > > > > > similar
> > > > > > > > > then we have in the PLC4xS7Adapter.
> > > > > > > > > > A collection property containing the fields of the 
> > > > > > > > > > output
> > > > > properties.
> > > > > > > > > > Therefore, we have to decide what information we 
> > > > > > > > > > have to
> > > > > provide. I
> > > > > > > > > think if we would support all options it becomes very
> cluttered
> > > > > for the
> > > > > > > > > user.
> > > > > > > > > >
> > > > > > > > > > What do you think?
> > > > > > > > > >
> > > > > > > > > > Philipp
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >> On 14. May 2020, at 06:55, Grainier Perera <
> > > > > > > grainier.perera@gmail.com>
> > > > > > > > > wrote:
> > > > > > > > > >>
> > > > > > > > > >> Hi Dominik,
> > > > > > > > > >>
> > > > > > > > > >> I agree. We have to do some enhancements make this
> processor
> > > > > user
> > > > > > > > > friendly.
> > > > > > > > > >> Like you said it's nice to have;
> > > > > > > > > >>   1. A new static property that supports code
> highlighting &
> > > > > syntax
> > > > > > > > > >> checking. We can simply do such client-side 
> > > > > > > > > >> validations
> > > using
> > > > > > > > > highlight.js
> > > > > > > > > >> (BSD) [1], jshint (MIT) [2], etc...
> > > > > > > > > >>   2. A mechanism to show the user with available 
> > > > > > > > > >> fields
> > > they can
> > > > > > > use to
> > > > > > > > > >> write code (similar to what you've done with usable
> > > templates in
> > > > > > > > > >> EmailPublisher using HtmlInputParameter).
> > > > > > > > > >>   3. A mechanism to map output schema.
> > > > > > > > > >>
> > > > > > > > > >> I think 1 & 2 go together and for the 3rd 
> > > > > > > > > >> requirement
> given
> > > > > that we
> > > > > > > are
> > > > > > > > > >> targetting a more technical user group for this
> processor,
> > > we
> > > > > can
> > > > > > > let
> > > > > > > > > them
> > > > > > > > > >> define the output. Otherwise, it would be quite
> difficult to
> > > > > derive
> > > > > > > > > return
> > > > > > > > > >> type from the JS. Moreover, I'm planning to use 
> > > > > > > > > >> Java's
> > > builtin
> > > > > > > > > ScriptEngine
> > > > > > > > > >> to eval. Of course, there'll be limitations. But, 
> > > > > > > > > >> we can
> > > always
> > > > > > > improve
> > > > > > > > > >> them :)
> > > > > > > > > >>
> > > > > > > > > >> [1] https://highlightjs.org/ [2] 
> > > > > > > > > >> https://jshint.com/
> > > > > > > > > >>
> > > > > > > > > >> Regards,
> > > > > > > > > >> Grainier.
> > > > > > > > > >>
> > > > > > > > > >> On Thu, 14 May 2020 at 00:53, Dominik Riemer <
> > > riemer@apache.org
> > > > > >
> > > > > > > wrote:
> > > > > > > > > >>
> > > > > > > > > >>> Hi Grainier,
> > > > > > > > > >>>
> > > > > > > > > >>> in my opinion, this would definitely be a very
> interesting
> > > > > > > addition! We
> > > > > > > > > >>> already briefly discussed such a feature in the 
> > > > > > > > > >>> past,
> > > before we
> > > > > > > went
> > > > > > > > > to the
> > > > > > > > > >>> ASF. I’d say that a JS evaluator would be 
> > > > > > > > > >>> definitely
> > > useful to
> > > > > a
> > > > > > > > > slightly
> > > > > > > > > >>> more technical target user group, but would 
> > > > > > > > > >>> provide
> great
> > > > > > > flexibility
> > > > > > > > > for,
> > > > > > > > > >>> e.g., data harmonization tasks. So let’s discuss 
> > > > > > > > > >>> what
> is
> > > > > needed to
> > > > > > > > > >>> implement this! I’d guess that we (maybe) need to 
> > > > > > > > > >>> add
> some
> > > > > > > > > enhancements to
> > > > > > > > > >>> the core to make this processor good from a 
> > > > > > > > > >>> usability
> > > > > perspective.
> > > > > > > > > >>>
> > > > > > > > > >>> From a conceptual point of view, I guess the JS
> processor
> > > would
> > > > > > > have no
> > > > > > > > > >>> specific input requirements and a single static
> property
> > > that
> > > > > > > allows
> > > > > > > > > users
> > > > > > > > > >>> to enter the code (maybe it would be cool to add a 
> > > > > > > > > >>> new
> > > static
> > > > > > > property
> > > > > > > > > here
> > > > > > > > > >>> that supports code highlighting or even syntax
> checking).
> > > The
> > > > > only
> > > > > > > open
> > > > > > > > > >>> issue I see is the output schema – we would 
> > > > > > > > > >>> somehow
> need to
> > > > > > > extract the
> > > > > > > > > >>> output from the JS function. This could probably
> partially
> > > be
> > > > > done
> > > > > > > > > using
> > > > > > > > > >>> the CustomTransformOutput (see
> > > > > > > > > >>>
> > > > > > >
> > > http://streampipes.apache.org/docs/docs/dev-guide-output-strategie
> > > s/),
> > > > > > > > > >>> but we would somehow need to derive type 
> > > > > > > > > >>> information
> from
> > > the
> > > > > JS
> > > > > > > > > function
> > > > > > > > > >>> or introduce a feature to let users define/refine 
> > > > > > > > > >>> the
> > > output
> > > > > > > manually
> > > > > > > > > >>> (e.g., to add semantic metadata).
> > > > > > > > > >>>
> > > > > > > > > >>> What do you think would be best? And would you 
> > > > > > > > > >>> evaluate
> > > the JS
> > > > > > > code in
> > > > > > > > > >>> Java or somehow else?
> > > > > > > > > >>>
> > > > > > > > > >>> Also, as I’ve just started to improve the SDK 
> > > > > > > > > >>> (e.g.,
> > > supporting
> > > > > > > > > optional
> > > > > > > > > >>> static properties and an easier way to define and
> extract
> > > model
> > > > > > > > > >>> parameters), we can collect your requirements for 
> > > > > > > > > >>> the
> JS
> > > > > processor
> > > > > > > and
> > > > > > > > > >>> extend the SDK if needed, just say what you would 
> > > > > > > > > >>> like
> to
> > > have
> > > > > 😉
> > > > > > > > > >>>
> > > > > > > > > >>> Dominik
> > > > > > > > > >>>
> > > > > > > > > >>>
> > > > > > > > > >>> From: Grainier Perera <gr...@gmail.com>
> > > > > > > > > >>> Sent: Wednesday, May 13, 2020 2:41 PM
> > > > > > > > > >>> To: dev@streampipes.apache.org
> > > > > > > > > >>> Subject: Data processor to evaluate JavaScript
> > > > > > > > > >>>
> > > > > > > > > >>> Hi all,
> > > > > > > > > >>>
> > > > > > > > > >>> I'm planning to implement a JavaScript eval data
> > > processor. As
> > > > > the
> > > > > > > name
> > > > > > > > > >>> suggests, users will be able to write some 
> > > > > > > > > >>> JavaScript
> code
> > > > > which
> > > > > > > takes
> > > > > > > > > in
> > > > > > > > > >>> an event (as a map), do some processing on the 
> > > > > > > > > >>> event,
> and
> > > > > return a
> > > > > > > new
> > > > > > > > > map
> > > > > > > > > >>> which then gets converted to an sp-event.
> > > > > > > > > >>>
> > > > > > > > > >>> example js:
> > > > > > > > > >>> function process(event) {
> > > > > > > > > >>>   // do processing here.
> > > > > > > > > >>>   // return processed event.
> > > > > > > > > >>>   return {id: http://event.id, tempInCelsius:
> > > > > (event.tempInKelvin
> > > > > > > -
> > > > > > > > > >>> 273.15)};
> > > > > > > > > >>> };
> > > > > > > > > >>>
> > > > > > > > > >>> Will this be a good addition to pipeline-elements?
> What do
> > > you
> > > > > > > think?
> > > > > > > > > >>>
> > > > > > > > > >>> Regards,
> > > > > > > > > >>> Grainier.
> > > > > > > > > >>>
> > > > > > > > > >>>
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Data processor to evaluate JavaScript

Posted by Grainier Perera <gr...@gmail.com>.

Hi Dominik,

I've implemented the following[1]. However, when I try to install the
processor it gives me the following error. Any clue? I've used CLI to start
sp. So, maybe the snapshot docker image doesn't have the recent changes? If
so what are the steps to build the images with changes in-place?

[1] https://github.com/apache/incubator-streampipes-extensions/pull/15

ERROR: https://streampipes.org/vocabulary/v1/UserDefinedOutputStrategy
io.fogsy.empire.pinto.RDFMappingException: Could not create an instance of
class org.apache.streampipes.model.output.OutputStrategy, it does not have
a default constructor
at io.fogsy.empire.pinto.RDFMapper.newInstance(RDFMapper.java:178)
at io.fogsy.empire.pinto.RDFMapper.readValue(RDFMapper.java:261)
at io.fogsy.empire.pinto.RDFMapper.valueToObject(RDFMapper.java:695)
at io.fogsy.empire.pinto.RDFMapper.lambda$toObject$3(RDFMapper.java:388)
at
io.fogsy.empire.pinto.RDFMapper$$Lambda$802/0000000000096040.apply(Unknown
Source)
at
io.fogsy.empire.pinto.RDFMapper$$Lambda$803/0000000000096580.apply(Unknown
Source)
at java.util.stream.ReferencePipeline$3$1.accept(Unknown Source)
at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(Unknown Source)
at java.util.stream.AbstractPipeline.copyInto(Unknown Source)
at java.util.stream.AbstractPipeline.wrapAndCopyInto(Unknown Source)
at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(Unknown Source)
at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(Unknown
Source)
at java.util.stream.AbstractPipeline.evaluate(Unknown Source)
at java.util.stream.ReferencePipeline.forEach(Unknown Source)
at io.fogsy.empire.pinto.RDFMapper.readValue(RDFMapper.java:299)
at
org.apache.streampipes.serializers.jsonld.JsonLdTransformer.fromJsonLd(JsonLdTransformer.java:100)
at
org.apache.streampipes.manager.verification.ElementVerifier.transform(ElementVerifier.java:169)
at
org.apache.streampipes.manager.verification.ElementVerifier.verifyAndAdd(ElementVerifier.java:84)
at
org.apache.streampipes.manager.operations.Operations.verifyAndAddElement(Operations.java:96)
at
org.apache.streampipes.manager.endpoint.EndpointItemParser.parseAndAddEndpointItem(EndpointItemParser.java:37)
at
org.apache.streampipes.rest.impl.PipelineElementImport.verifyAndAddElement(PipelineElementImport.java:93)
at
org.apache.streampipes.rest.impl.PipelineElementImport.addElement(PipelineElementImport.java:89)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)

Thanks,
Grainier.

On Thu, 21 May 2020 at 03:39, Dominik Riemer <ri...@apache.org> wrote:

> Hi Grainier,
>
> sounds good!
> I've just pushed a commit that provides an initial version of the
> user-defined output strategy. It lets you select the input schema from the
> incoming stream and add/remove event properties. There is one limitation,
> the component currently only works for flat event structures, but it
> shouldn't be a problem to have a more advanced schema editor for this
> output strategy soon (we already have this in the StreamPipes connect code,
> but unfortunately, the pipeline editor has yet to be migrated from
> AngularJS to Angular 2+ to reuse these components). This is something we
> plan to do within the next weeks.
>
> I added an example how to use the strategy to the examples repo:
>
> https://github.com/apache/incubator-streampipes-examples/blob/dev/streampipes-pipeline-elements-examples-processors-jvm/src/main/java/org/apache/streampipes/pe/examples/jvm/staticproperty/CodeInputExampleController.java
>
> Let me know what you think and if you need anything more/else!
>
> Dominik
>
> On 2020/05/19 17:14:21, Grainier Perera <gr...@gmail.com>
> wrote:
> > Hi Dominik,
> >
> > Awesome. This is really cool & exactly what I was looking for. I'll use
> > this with my implementation, and keep this thread posted with the
> updates.
> > ps: let us know when you add output strategy as well.
> >
> > Thanks,
> > Grainier.
> >
> > On Tue, 19 May 2020 at 03:05, Dominik Riemer <ri...@apache.org> wrote:
> >
> > > Hi Grainier,
> > >
> > > I just pushed an initial version of the static property to add
> javascript
> > > code.
> > > It's a simple codeblock with linting enabled and a (currently very
> simple)
> > > autocomplete feature that suggests existing properties from the input
> event
> > > (currently, only if you press ctrl-space directly after typing "event"
> > > followed by a dot, I'll fix that soon ;-)
> > >
> > > You can find an example how to use it here:
> > >
> > >
> https://github.com/apache/incubator-streampipes-examples/blob/dev/streampipes-pipeline-elements-examples-processors-jvm/src/main/java/org/apache/streampipes/pe/examples/jvm/staticproperty/CodeInputExampleController.java
> > >
> > > Feedback and ideas for improvement are welcome!
> > >
> > > Dominik
> > >
> > >
> > > On 2020/05/17 12:36:05, Grainier Perera <gr...@gmail.com>
> > > wrote:
> > > > Hi Dominik,
> > > >
> > > > Thank you. I'll work on `Implement JS evaluator` [1].
> > > >
> > > > [1] https://issues.apache.org/jira/browse/STREAMPIPES-135
> > > >
> > > > Grainier.
> > > >
> > > > On Sun, 17 May 2020 at 18:02, Dominik Riemer <ri...@apache.org>
> wrote:
> > > >
> > > > > Hi Grainier,
> > > > >
> > > > > great! I created an umbrella issue for that:
> > > > > https://issues.apache.org/jira/browse/STREAMPIPES-132
> > > > >
> > > > > I'll now start to implement the new static property for entering
> code.
> > > > >
> > > > > Dominik
> > > > >
> > > > > On 2020/05/15 01:29:44, Grainier Perera <grainier.perera@gmail.com
> >
> > > > > wrote:
> > > > > > Hi,
> > > > > >
> > > > > > I'm +1 to the idea of `UserDefinedOutputStrategy`. As you said,
> > > that'll
> > > > > be
> > > > > > re-usable in the future processors as well. So let's proceed with
> > > that.
> > > > > >
> > > > > > Regards,
> > > > > > Grainier
> > > > > >
> > > > > > On Fri, 15 May 2020 at 01:37, Dominik Riemer <ri...@apache.org>
> > > wrote:
> > > > > >
> > > > > > > Hi,
> > > > > > > cool, that sounds very good!
> > > > > > >
> > > > > > > Concerning the output, I think there are two ways to achieve
> this:
> > > > > > > - A collection static property in conjunction with a runtime
> > > resolvable
> > > > > > > output strategy that creates the output schema based on the
> input
> > > from
> > > > > the
> > > > > > > collection
> > > > > > > - Or we create a new output strategy, e.g., let's call it
> > > > > > > UserDefinedOutputStrategy, which would render a UI component
> to let
> > > > > users
> > > > > > > add/remove/change event properties from the input schema. I
> also
> > > like
> > > > > the
> > > > > > > idea of preselecting all input properties.
> > > > > > >
> > > > > > > I personally tend towards the second option as such a strategy
> > > might be
> > > > > > > useful for other pipeline elements as well and it would ease
> the
> > > model
> > > > > > > definition. We could also extend this component in the future,
> > > e.g., by
> > > > > > > letting users enrich metadata such as measurement units
> directly
> > > in a
> > > > > > > processor.
> > > > > > >
> > > > > > > But I'm also open to the first option and we can also just
> start
> > > > > > > implementing to see what works best.
> > > > > > >
> > > > > > > I'd also be happy to help with the implementation, e.g., I
> could
> > > work
> > > > > on
> > > > > > > the new static property for code input or the output strategy!
> > > > > > >
> > > > > > > Dominik
> > > > > > >
> > > > > > > On 2020/05/14 11:59:36, Grainier Perera <
> grainier.perera@gmail.com
> > > >
> > > > > > > wrote:
> > > > > > > > Hi,
> > > > > > > >
> > > > > > > > +1 to the idea of having a collection property containing the
> > > fields
> > > > > of
> > > > > > > the
> > > > > > > > output properties. Also, If we can show the user with
> available
> > > > > fields
> > > > > > > with
> > > > > > > > metadata (of input event) populated in that same collection
> > > property,
> > > > > > > then
> > > > > > > > the user can add/remove/updates fields easily. Let's say they
> > > just
> > > > > have
> > > > > > > to
> > > > > > > > return a map containing all the fields they defined in output
> > > > > properties
> > > > > > > > (i.e return {"existingId": 1, "existingTempInKelvin": 301,
> > > > > > > > "newTempInCelsius": 28 } ).
> > > > > > > >
> > > > > > > > [image: sample.png]
> > > > > > > >
> > > > > > > > Regards,
> > > > > > > > Grainier.
> > > > > > > >
> > > > > > > > On Thu, 14 May 2020 at 12:33, Patrick Wiener <
> wiener@apache.org>
> > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi,
> > > > > > > > >
> > > > > > > > > sounds reasonable.
> > > > > > > > >
> > > > > > > > > I share the thoughts of Philipp regarding the output
> schema.
> > > But
> > > > > as you
> > > > > > > > > said Grainier, we can also improve :)
> > > > > > > > >
> > > > > > > > > btw: I really like the idea to also allow the more
> technical
> > > user
> > > > > to
> > > > > > > > > easily integrate their function snippets without having
> > > > > > > > > to care about the actual execution. This gets a more FaaS
> like
> > > > > feeling
> > > > > > > for
> > > > > > > > > streaming.
> > > > > > > > >
> > > > > > > > > Patrick
> > > > > > > > >
> > > > > > > > > > Am 14.05.2020 um 08:26 schrieb Philipp Zehnder <
> > > > > zehnder@apache.org>:
> > > > > > > > > >
> > > > > > > > > > Hi,
> > > > > > > > > >
> > > > > > > > > > it would be very useful to have such a processor.
> > > > > > > > > >
> > > > > > > > > > I like the ideas for 1 & 2 and I think we can do it this
> way.
> > > > > > > > > >
> > > > > > > > > > Actually I think that the definition of the output scheme
> > > could
> > > > > be
> > > > > > > quite
> > > > > > > > > tricky.
> > > > > > > > > > Because users can do anything with the event.
> > > > > > > > > >
> > > > > > > > > > How about we provide an option property?
> > > > > > > > > > Users can select whether they want to define the whole
> event
> > > or
> > > > > just
> > > > > > > > > append properties.
> > > > > > > > > >
> > > > > > > > > > For the configurations of the properties we could do
> > > something
> > > > > > > similar
> > > > > > > > > then we have in the PLC4xS7Adapter.
> > > > > > > > > > A collection property containing the fields of the output
> > > > > properties.
> > > > > > > > > > Therefore, we have to decide what information we have to
> > > > > provide. I
> > > > > > > > > think if we would support all options it becomes very
> cluttered
> > > > > for the
> > > > > > > > > user.
> > > > > > > > > >
> > > > > > > > > > What do you think?
> > > > > > > > > >
> > > > > > > > > > Philipp
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >> On 14. May 2020, at 06:55, Grainier Perera <
> > > > > > > grainier.perera@gmail.com>
> > > > > > > > > wrote:
> > > > > > > > > >>
> > > > > > > > > >> Hi Dominik,
> > > > > > > > > >>
> > > > > > > > > >> I agree. We have to do some enhancements make this
> processor
> > > > > user
> > > > > > > > > friendly.
> > > > > > > > > >> Like you said it's nice to have;
> > > > > > > > > >>   1. A new static property that supports code
> highlighting &
> > > > > syntax
> > > > > > > > > >> checking. We can simply do such client-side validations
> > > using
> > > > > > > > > highlight.js
> > > > > > > > > >> (BSD) [1], jshint (MIT) [2], etc...
> > > > > > > > > >>   2. A mechanism to show the user with available fields
> > > they can
> > > > > > > use to
> > > > > > > > > >> write code (similar to what you've done with usable
> > > templates in
> > > > > > > > > >> EmailPublisher using HtmlInputParameter).
> > > > > > > > > >>   3. A mechanism to map output schema.
> > > > > > > > > >>
> > > > > > > > > >> I think 1 & 2 go together and for the 3rd requirement
> given
> > > > > that we
> > > > > > > are
> > > > > > > > > >> targetting a more technical user group for this
> processor,
> > > we
> > > > > can
> > > > > > > let
> > > > > > > > > them
> > > > > > > > > >> define the output. Otherwise, it would be quite
> difficult to
> > > > > derive
> > > > > > > > > return
> > > > > > > > > >> type from the JS. Moreover, I'm planning to use Java's
> > > builtin
> > > > > > > > > ScriptEngine
> > > > > > > > > >> to eval. Of course, there'll be limitations. But, we can
> > > always
> > > > > > > improve
> > > > > > > > > >> them :)
> > > > > > > > > >>
> > > > > > > > > >> [1] https://highlightjs.org/
> > > > > > > > > >> [2] https://jshint.com/
> > > > > > > > > >>
> > > > > > > > > >> Regards,
> > > > > > > > > >> Grainier.
> > > > > > > > > >>
> > > > > > > > > >> On Thu, 14 May 2020 at 00:53, Dominik Riemer <
> > > riemer@apache.org
> > > > > >
> > > > > > > wrote:
> > > > > > > > > >>
> > > > > > > > > >>> Hi Grainier,
> > > > > > > > > >>>
> > > > > > > > > >>> in my opinion, this would definitely be a very
> interesting
> > > > > > > addition! We
> > > > > > > > > >>> already briefly discussed such a feature in the past,
> > > before we
> > > > > > > went
> > > > > > > > > to the
> > > > > > > > > >>> ASF. I’d say that a JS evaluator would be definitely
> > > useful to
> > > > > a
> > > > > > > > > slightly
> > > > > > > > > >>> more technical target user group, but would provide
> great
> > > > > > > flexibility
> > > > > > > > > for,
> > > > > > > > > >>> e.g., data harmonization tasks. So let’s discuss what
> is
> > > > > needed to
> > > > > > > > > >>> implement this! I’d guess that we (maybe) need to add
> some
> > > > > > > > > enhancements to
> > > > > > > > > >>> the core to make this processor good from a usability
> > > > > perspective.
> > > > > > > > > >>>
> > > > > > > > > >>> From a conceptual point of view, I guess the JS
> processor
> > > would
> > > > > > > have no
> > > > > > > > > >>> specific input requirements and a single static
> property
> > > that
> > > > > > > allows
> > > > > > > > > users
> > > > > > > > > >>> to enter the code (maybe it would be cool to add a new
> > > static
> > > > > > > property
> > > > > > > > > here
> > > > > > > > > >>> that supports code highlighting or even syntax
> checking).
> > > The
> > > > > only
> > > > > > > open
> > > > > > > > > >>> issue I see is the output schema – we would somehow
> need to
> > > > > > > extract the
> > > > > > > > > >>> output from the JS function. This could probably
> partially
> > > be
> > > > > done
> > > > > > > > > using
> > > > > > > > > >>> the CustomTransformOutput (see
> > > > > > > > > >>>
> > > > > > >
> > > http://streampipes.apache.org/docs/docs/dev-guide-output-strategies/),
> > > > > > > > > >>> but we would somehow need to derive type information
> from
> > > the
> > > > > JS
> > > > > > > > > function
> > > > > > > > > >>> or introduce a feature to let users define/refine the
> > > output
> > > > > > > manually
> > > > > > > > > >>> (e.g., to add semantic metadata).
> > > > > > > > > >>>
> > > > > > > > > >>> What do you think would be best? And would you evaluate
> > > the JS
> > > > > > > code in
> > > > > > > > > >>> Java or somehow else?
> > > > > > > > > >>>
> > > > > > > > > >>> Also, as I’ve just started to improve the SDK (e.g.,
> > > supporting
> > > > > > > > > optional
> > > > > > > > > >>> static properties and an easier way to define and
> extract
> > > model
> > > > > > > > > >>> parameters), we can collect your requirements for the
> JS
> > > > > processor
> > > > > > > and
> > > > > > > > > >>> extend the SDK if needed, just say what you would like
> to
> > > have
> > > > > 😉
> > > > > > > > > >>>
> > > > > > > > > >>> Dominik
> > > > > > > > > >>>
> > > > > > > > > >>>
> > > > > > > > > >>> From: Grainier Perera <gr...@gmail.com>
> > > > > > > > > >>> Sent: Wednesday, May 13, 2020 2:41 PM
> > > > > > > > > >>> To: dev@streampipes.apache.org
> > > > > > > > > >>> Subject: Data processor to evaluate JavaScript
> > > > > > > > > >>>
> > > > > > > > > >>> Hi all,
> > > > > > > > > >>>
> > > > > > > > > >>> I'm planning to implement a JavaScript eval data
> > > processor. As
> > > > > the
> > > > > > > name
> > > > > > > > > >>> suggests, users will be able to write some JavaScript
> code
> > > > > which
> > > > > > > takes
> > > > > > > > > in
> > > > > > > > > >>> an event (as a map), do some processing on the event,
> and
> > > > > return a
> > > > > > > new
> > > > > > > > > map
> > > > > > > > > >>> which then gets converted to an sp-event.
> > > > > > > > > >>>
> > > > > > > > > >>> example js:
> > > > > > > > > >>> function process(event) {
> > > > > > > > > >>>   // do processing here.
> > > > > > > > > >>>   // return processed event.
> > > > > > > > > >>>   return {id: http://event.id, tempInCelsius:
> > > > > (event.tempInKelvin
> > > > > > > -
> > > > > > > > > >>> 273.15)};
> > > > > > > > > >>> };
> > > > > > > > > >>>
> > > > > > > > > >>> Will this be a good addition to pipeline-elements?
> What do
> > > you
> > > > > > > think?
> > > > > > > > > >>>
> > > > > > > > > >>> Regards,
> > > > > > > > > >>> Grainier.
> > > > > > > > > >>>
> > > > > > > > > >>>
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Data processor to evaluate JavaScript

Posted by Dominik Riemer <ri...@apache.org>.

Hi Grainier,

sounds good!
I've just pushed a commit that provides an initial version of the user-defined output strategy. It lets you select the input schema from the incoming stream and add/remove event properties. There is one limitation, the component currently only works for flat event structures, but it shouldn't be a problem to have a more advanced schema editor for this output strategy soon (we already have this in the StreamPipes connect code, but unfortunately, the pipeline editor has yet to be migrated from AngularJS to Angular 2+ to reuse these components). This is something we plan to do within the next weeks.

I added an example how to use the strategy to the examples repo:
https://github.com/apache/incubator-streampipes-examples/blob/dev/streampipes-pipeline-elements-examples-processors-jvm/src/main/java/org/apache/streampipes/pe/examples/jvm/staticproperty/CodeInputExampleController.java

Let me know what you think and if you need anything more/else!

Dominik

On 2020/05/19 17:14:21, Grainier Perera <gr...@gmail.com> wrote: 
> Hi Dominik,
> 
> Awesome. This is really cool & exactly what I was looking for. I'll use
> this with my implementation, and keep this thread posted with the updates.
> ps: let us know when you add output strategy as well.
> 
> Thanks,
> Grainier.
> 
> On Tue, 19 May 2020 at 03:05, Dominik Riemer <ri...@apache.org> wrote:
> 
> > Hi Grainier,
> >
> > I just pushed an initial version of the static property to add javascript
> > code.
> > It's a simple codeblock with linting enabled and a (currently very simple)
> > autocomplete feature that suggests existing properties from the input event
> > (currently, only if you press ctrl-space directly after typing "event"
> > followed by a dot, I'll fix that soon ;-)
> >
> > You can find an example how to use it here:
> >
> > https://github.com/apache/incubator-streampipes-examples/blob/dev/streampipes-pipeline-elements-examples-processors-jvm/src/main/java/org/apache/streampipes/pe/examples/jvm/staticproperty/CodeInputExampleController.java
> >
> > Feedback and ideas for improvement are welcome!
> >
> > Dominik
> >
> >
> > On 2020/05/17 12:36:05, Grainier Perera <gr...@gmail.com>
> > wrote:
> > > Hi Dominik,
> > >
> > > Thank you. I'll work on `Implement JS evaluator` [1].
> > >
> > > [1] https://issues.apache.org/jira/browse/STREAMPIPES-135
> > >
> > > Grainier.
> > >
> > > On Sun, 17 May 2020 at 18:02, Dominik Riemer <ri...@apache.org> wrote:
> > >
> > > > Hi Grainier,
> > > >
> > > > great! I created an umbrella issue for that:
> > > > https://issues.apache.org/jira/browse/STREAMPIPES-132
> > > >
> > > > I'll now start to implement the new static property for entering code.
> > > >
> > > > Dominik
> > > >
> > > > On 2020/05/15 01:29:44, Grainier Perera <gr...@gmail.com>
> > > > wrote:
> > > > > Hi,
> > > > >
> > > > > I'm +1 to the idea of `UserDefinedOutputStrategy`. As you said,
> > that'll
> > > > be
> > > > > re-usable in the future processors as well. So let's proceed with
> > that.
> > > > >
> > > > > Regards,
> > > > > Grainier
> > > > >
> > > > > On Fri, 15 May 2020 at 01:37, Dominik Riemer <ri...@apache.org>
> > wrote:
> > > > >
> > > > > > Hi,
> > > > > > cool, that sounds very good!
> > > > > >
> > > > > > Concerning the output, I think there are two ways to achieve this:
> > > > > > - A collection static property in conjunction with a runtime
> > resolvable
> > > > > > output strategy that creates the output schema based on the input
> > from
> > > > the
> > > > > > collection
> > > > > > - Or we create a new output strategy, e.g., let's call it
> > > > > > UserDefinedOutputStrategy, which would render a UI component to let
> > > > users
> > > > > > add/remove/change event properties from the input schema. I also
> > like
> > > > the
> > > > > > idea of preselecting all input properties.
> > > > > >
> > > > > > I personally tend towards the second option as such a strategy
> > might be
> > > > > > useful for other pipeline elements as well and it would ease the
> > model
> > > > > > definition. We could also extend this component in the future,
> > e.g., by
> > > > > > letting users enrich metadata such as measurement units directly
> > in a
> > > > > > processor.
> > > > > >
> > > > > > But I'm also open to the first option and we can also just start
> > > > > > implementing to see what works best.
> > > > > >
> > > > > > I'd also be happy to help with the implementation, e.g., I could
> > work
> > > > on
> > > > > > the new static property for code input or the output strategy!
> > > > > >
> > > > > > Dominik
> > > > > >
> > > > > > On 2020/05/14 11:59:36, Grainier Perera <grainier.perera@gmail.com
> > >
> > > > > > wrote:
> > > > > > > Hi,
> > > > > > >
> > > > > > > +1 to the idea of having a collection property containing the
> > fields
> > > > of
> > > > > > the
> > > > > > > output properties. Also, If we can show the user with available
> > > > fields
> > > > > > with
> > > > > > > metadata (of input event) populated in that same collection
> > property,
> > > > > > then
> > > > > > > the user can add/remove/updates fields easily. Let's say they
> > just
> > > > have
> > > > > > to
> > > > > > > return a map containing all the fields they defined in output
> > > > properties
> > > > > > > (i.e return {"existingId": 1, "existingTempInKelvin": 301,
> > > > > > > "newTempInCelsius": 28 } ).
> > > > > > >
> > > > > > > [image: sample.png]
> > > > > > >
> > > > > > > Regards,
> > > > > > > Grainier.
> > > > > > >
> > > > > > > On Thu, 14 May 2020 at 12:33, Patrick Wiener <wi...@apache.org>
> > > > wrote:
> > > > > > >
> > > > > > > > Hi,
> > > > > > > >
> > > > > > > > sounds reasonable.
> > > > > > > >
> > > > > > > > I share the thoughts of Philipp regarding the output schema.
> > But
> > > > as you
> > > > > > > > said Grainier, we can also improve :)
> > > > > > > >
> > > > > > > > btw: I really like the idea to also allow the more technical
> > user
> > > > to
> > > > > > > > easily integrate their function snippets without having
> > > > > > > > to care about the actual execution. This gets a more FaaS like
> > > > feeling
> > > > > > for
> > > > > > > > streaming.
> > > > > > > >
> > > > > > > > Patrick
> > > > > > > >
> > > > > > > > > Am 14.05.2020 um 08:26 schrieb Philipp Zehnder <
> > > > zehnder@apache.org>:
> > > > > > > > >
> > > > > > > > > Hi,
> > > > > > > > >
> > > > > > > > > it would be very useful to have such a processor.
> > > > > > > > >
> > > > > > > > > I like the ideas for 1 & 2 and I think we can do it this way.
> > > > > > > > >
> > > > > > > > > Actually I think that the definition of the output scheme
> > could
> > > > be
> > > > > > quite
> > > > > > > > tricky.
> > > > > > > > > Because users can do anything with the event.
> > > > > > > > >
> > > > > > > > > How about we provide an option property?
> > > > > > > > > Users can select whether they want to define the whole event
> > or
> > > > just
> > > > > > > > append properties.
> > > > > > > > >
> > > > > > > > > For the configurations of the properties we could do
> > something
> > > > > > similar
> > > > > > > > then we have in the PLC4xS7Adapter.
> > > > > > > > > A collection property containing the fields of the output
> > > > properties.
> > > > > > > > > Therefore, we have to decide what information we have to
> > > > provide. I
> > > > > > > > think if we would support all options it becomes very cluttered
> > > > for the
> > > > > > > > user.
> > > > > > > > >
> > > > > > > > > What do you think?
> > > > > > > > >
> > > > > > > > > Philipp
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >> On 14. May 2020, at 06:55, Grainier Perera <
> > > > > > grainier.perera@gmail.com>
> > > > > > > > wrote:
> > > > > > > > >>
> > > > > > > > >> Hi Dominik,
> > > > > > > > >>
> > > > > > > > >> I agree. We have to do some enhancements make this processor
> > > > user
> > > > > > > > friendly.
> > > > > > > > >> Like you said it's nice to have;
> > > > > > > > >>   1. A new static property that supports code highlighting &
> > > > syntax
> > > > > > > > >> checking. We can simply do such client-side validations
> > using
> > > > > > > > highlight.js
> > > > > > > > >> (BSD) [1], jshint (MIT) [2], etc...
> > > > > > > > >>   2. A mechanism to show the user with available fields
> > they can
> > > > > > use to
> > > > > > > > >> write code (similar to what you've done with usable
> > templates in
> > > > > > > > >> EmailPublisher using HtmlInputParameter).
> > > > > > > > >>   3. A mechanism to map output schema.
> > > > > > > > >>
> > > > > > > > >> I think 1 & 2 go together and for the 3rd requirement given
> > > > that we
> > > > > > are
> > > > > > > > >> targetting a more technical user group for this processor,
> > we
> > > > can
> > > > > > let
> > > > > > > > them
> > > > > > > > >> define the output. Otherwise, it would be quite difficult to
> > > > derive
> > > > > > > > return
> > > > > > > > >> type from the JS. Moreover, I'm planning to use Java's
> > builtin
> > > > > > > > ScriptEngine
> > > > > > > > >> to eval. Of course, there'll be limitations. But, we can
> > always
> > > > > > improve
> > > > > > > > >> them :)
> > > > > > > > >>
> > > > > > > > >> [1] https://highlightjs.org/
> > > > > > > > >> [2] https://jshint.com/
> > > > > > > > >>
> > > > > > > > >> Regards,
> > > > > > > > >> Grainier.
> > > > > > > > >>
> > > > > > > > >> On Thu, 14 May 2020 at 00:53, Dominik Riemer <
> > riemer@apache.org
> > > > >
> > > > > > wrote:
> > > > > > > > >>
> > > > > > > > >>> Hi Grainier,
> > > > > > > > >>>
> > > > > > > > >>> in my opinion, this would definitely be a very interesting
> > > > > > addition! We
> > > > > > > > >>> already briefly discussed such a feature in the past,
> > before we
> > > > > > went
> > > > > > > > to the
> > > > > > > > >>> ASF. I’d say that a JS evaluator would be definitely
> > useful to
> > > > a
> > > > > > > > slightly
> > > > > > > > >>> more technical target user group, but would provide great
> > > > > > flexibility
> > > > > > > > for,
> > > > > > > > >>> e.g., data harmonization tasks. So let’s discuss what is
> > > > needed to
> > > > > > > > >>> implement this! I’d guess that we (maybe) need to add some
> > > > > > > > enhancements to
> > > > > > > > >>> the core to make this processor good from a usability
> > > > perspective.
> > > > > > > > >>>
> > > > > > > > >>> From a conceptual point of view, I guess the JS processor
> > would
> > > > > > have no
> > > > > > > > >>> specific input requirements and a single static property
> > that
> > > > > > allows
> > > > > > > > users
> > > > > > > > >>> to enter the code (maybe it would be cool to add a new
> > static
> > > > > > property
> > > > > > > > here
> > > > > > > > >>> that supports code highlighting or even syntax checking).
> > The
> > > > only
> > > > > > open
> > > > > > > > >>> issue I see is the output schema – we would somehow need to
> > > > > > extract the
> > > > > > > > >>> output from the JS function. This could probably partially
> > be
> > > > done
> > > > > > > > using
> > > > > > > > >>> the CustomTransformOutput (see
> > > > > > > > >>>
> > > > > >
> > http://streampipes.apache.org/docs/docs/dev-guide-output-strategies/),
> > > > > > > > >>> but we would somehow need to derive type information from
> > the
> > > > JS
> > > > > > > > function
> > > > > > > > >>> or introduce a feature to let users define/refine the
> > output
> > > > > > manually
> > > > > > > > >>> (e.g., to add semantic metadata).
> > > > > > > > >>>
> > > > > > > > >>> What do you think would be best? And would you evaluate
> > the JS
> > > > > > code in
> > > > > > > > >>> Java or somehow else?
> > > > > > > > >>>
> > > > > > > > >>> Also, as I’ve just started to improve the SDK (e.g.,
> > supporting
> > > > > > > > optional
> > > > > > > > >>> static properties and an easier way to define and extract
> > model
> > > > > > > > >>> parameters), we can collect your requirements for the JS
> > > > processor
> > > > > > and
> > > > > > > > >>> extend the SDK if needed, just say what you would like to
> > have
> > > > 😉
> > > > > > > > >>>
> > > > > > > > >>> Dominik
> > > > > > > > >>>
> > > > > > > > >>>
> > > > > > > > >>> From: Grainier Perera <gr...@gmail.com>
> > > > > > > > >>> Sent: Wednesday, May 13, 2020 2:41 PM
> > > > > > > > >>> To: dev@streampipes.apache.org
> > > > > > > > >>> Subject: Data processor to evaluate JavaScript
> > > > > > > > >>>
> > > > > > > > >>> Hi all,
> > > > > > > > >>>
> > > > > > > > >>> I'm planning to implement a JavaScript eval data
> > processor. As
> > > > the
> > > > > > name
> > > > > > > > >>> suggests, users will be able to write some JavaScript code
> > > > which
> > > > > > takes
> > > > > > > > in
> > > > > > > > >>> an event (as a map), do some processing on the event, and
> > > > return a
> > > > > > new
> > > > > > > > map
> > > > > > > > >>> which then gets converted to an sp-event.
> > > > > > > > >>>
> > > > > > > > >>> example js:
> > > > > > > > >>> function process(event) {
> > > > > > > > >>>   // do processing here.
> > > > > > > > >>>   // return processed event.
> > > > > > > > >>>   return {id: http://event.id, tempInCelsius:
> > > > (event.tempInKelvin
> > > > > > -
> > > > > > > > >>> 273.15)};
> > > > > > > > >>> };
> > > > > > > > >>>
> > > > > > > > >>> Will this be a good addition to pipeline-elements? What do
> > you
> > > > > > think?
> > > > > > > > >>>
> > > > > > > > >>> Regards,
> > > > > > > > >>> Grainier.
> > > > > > > > >>>
> > > > > > > > >>>
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Data processor to evaluate JavaScript

Posted by Grainier Perera <gr...@gmail.com>.

Hi Dominik,

Awesome. This is really cool & exactly what I was looking for. I'll use
this with my implementation, and keep this thread posted with the updates.
ps: let us know when you add output strategy as well.

Thanks,
Grainier.

On Tue, 19 May 2020 at 03:05, Dominik Riemer <ri...@apache.org> wrote:

> Hi Grainier,
>
> I just pushed an initial version of the static property to add javascript
> code.
> It's a simple codeblock with linting enabled and a (currently very simple)
> autocomplete feature that suggests existing properties from the input event
> (currently, only if you press ctrl-space directly after typing "event"
> followed by a dot, I'll fix that soon ;-)
>
> You can find an example how to use it here:
>
> https://github.com/apache/incubator-streampipes-examples/blob/dev/streampipes-pipeline-elements-examples-processors-jvm/src/main/java/org/apache/streampipes/pe/examples/jvm/staticproperty/CodeInputExampleController.java
>
> Feedback and ideas for improvement are welcome!
>
> Dominik
>
>
> On 2020/05/17 12:36:05, Grainier Perera <gr...@gmail.com>
> wrote:
> > Hi Dominik,
> >
> > Thank you. I'll work on `Implement JS evaluator` [1].
> >
> > [1] https://issues.apache.org/jira/browse/STREAMPIPES-135
> >
> > Grainier.
> >
> > On Sun, 17 May 2020 at 18:02, Dominik Riemer <ri...@apache.org> wrote:
> >
> > > Hi Grainier,
> > >
> > > great! I created an umbrella issue for that:
> > > https://issues.apache.org/jira/browse/STREAMPIPES-132
> > >
> > > I'll now start to implement the new static property for entering code.
> > >
> > > Dominik
> > >
> > > On 2020/05/15 01:29:44, Grainier Perera <gr...@gmail.com>
> > > wrote:
> > > > Hi,
> > > >
> > > > I'm +1 to the idea of `UserDefinedOutputStrategy`. As you said,
> that'll
> > > be
> > > > re-usable in the future processors as well. So let's proceed with
> that.
> > > >
> > > > Regards,
> > > > Grainier
> > > >
> > > > On Fri, 15 May 2020 at 01:37, Dominik Riemer <ri...@apache.org>
> wrote:
> > > >
> > > > > Hi,
> > > > > cool, that sounds very good!
> > > > >
> > > > > Concerning the output, I think there are two ways to achieve this:
> > > > > - A collection static property in conjunction with a runtime
> resolvable
> > > > > output strategy that creates the output schema based on the input
> from
> > > the
> > > > > collection
> > > > > - Or we create a new output strategy, e.g., let's call it
> > > > > UserDefinedOutputStrategy, which would render a UI component to let
> > > users
> > > > > add/remove/change event properties from the input schema. I also
> like
> > > the
> > > > > idea of preselecting all input properties.
> > > > >
> > > > > I personally tend towards the second option as such a strategy
> might be
> > > > > useful for other pipeline elements as well and it would ease the
> model
> > > > > definition. We could also extend this component in the future,
> e.g., by
> > > > > letting users enrich metadata such as measurement units directly
> in a
> > > > > processor.
> > > > >
> > > > > But I'm also open to the first option and we can also just start
> > > > > implementing to see what works best.
> > > > >
> > > > > I'd also be happy to help with the implementation, e.g., I could
> work
> > > on
> > > > > the new static property for code input or the output strategy!
> > > > >
> > > > > Dominik
> > > > >
> > > > > On 2020/05/14 11:59:36, Grainier Perera <grainier.perera@gmail.com
> >
> > > > > wrote:
> > > > > > Hi,
> > > > > >
> > > > > > +1 to the idea of having a collection property containing the
> fields
> > > of
> > > > > the
> > > > > > output properties. Also, If we can show the user with available
> > > fields
> > > > > with
> > > > > > metadata (of input event) populated in that same collection
> property,
> > > > > then
> > > > > > the user can add/remove/updates fields easily. Let's say they
> just
> > > have
> > > > > to
> > > > > > return a map containing all the fields they defined in output
> > > properties
> > > > > > (i.e return {"existingId": 1, "existingTempInKelvin": 301,
> > > > > > "newTempInCelsius": 28 } ).
> > > > > >
> > > > > > [image: sample.png]
> > > > > >
> > > > > > Regards,
> > > > > > Grainier.
> > > > > >
> > > > > > On Thu, 14 May 2020 at 12:33, Patrick Wiener <wi...@apache.org>
> > > wrote:
> > > > > >
> > > > > > > Hi,
> > > > > > >
> > > > > > > sounds reasonable.
> > > > > > >
> > > > > > > I share the thoughts of Philipp regarding the output schema.
> But
> > > as you
> > > > > > > said Grainier, we can also improve :)
> > > > > > >
> > > > > > > btw: I really like the idea to also allow the more technical
> user
> > > to
> > > > > > > easily integrate their function snippets without having
> > > > > > > to care about the actual execution. This gets a more FaaS like
> > > feeling
> > > > > for
> > > > > > > streaming.
> > > > > > >
> > > > > > > Patrick
> > > > > > >
> > > > > > > > Am 14.05.2020 um 08:26 schrieb Philipp Zehnder <
> > > zehnder@apache.org>:
> > > > > > > >
> > > > > > > > Hi,
> > > > > > > >
> > > > > > > > it would be very useful to have such a processor.
> > > > > > > >
> > > > > > > > I like the ideas for 1 & 2 and I think we can do it this way.
> > > > > > > >
> > > > > > > > Actually I think that the definition of the output scheme
> could
> > > be
> > > > > quite
> > > > > > > tricky.
> > > > > > > > Because users can do anything with the event.
> > > > > > > >
> > > > > > > > How about we provide an option property?
> > > > > > > > Users can select whether they want to define the whole event
> or
> > > just
> > > > > > > append properties.
> > > > > > > >
> > > > > > > > For the configurations of the properties we could do
> something
> > > > > similar
> > > > > > > then we have in the PLC4xS7Adapter.
> > > > > > > > A collection property containing the fields of the output
> > > properties.
> > > > > > > > Therefore, we have to decide what information we have to
> > > provide. I
> > > > > > > think if we would support all options it becomes very cluttered
> > > for the
> > > > > > > user.
> > > > > > > >
> > > > > > > > What do you think?
> > > > > > > >
> > > > > > > > Philipp
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >> On 14. May 2020, at 06:55, Grainier Perera <
> > > > > grainier.perera@gmail.com>
> > > > > > > wrote:
> > > > > > > >>
> > > > > > > >> Hi Dominik,
> > > > > > > >>
> > > > > > > >> I agree. We have to do some enhancements make this processor
> > > user
> > > > > > > friendly.
> > > > > > > >> Like you said it's nice to have;
> > > > > > > >>   1. A new static property that supports code highlighting &
> > > syntax
> > > > > > > >> checking. We can simply do such client-side validations
> using
> > > > > > > highlight.js
> > > > > > > >> (BSD) [1], jshint (MIT) [2], etc...
> > > > > > > >>   2. A mechanism to show the user with available fields
> they can
> > > > > use to
> > > > > > > >> write code (similar to what you've done with usable
> templates in
> > > > > > > >> EmailPublisher using HtmlInputParameter).
> > > > > > > >>   3. A mechanism to map output schema.
> > > > > > > >>
> > > > > > > >> I think 1 & 2 go together and for the 3rd requirement given
> > > that we
> > > > > are
> > > > > > > >> targetting a more technical user group for this processor,
> we
> > > can
> > > > > let
> > > > > > > them
> > > > > > > >> define the output. Otherwise, it would be quite difficult to
> > > derive
> > > > > > > return
> > > > > > > >> type from the JS. Moreover, I'm planning to use Java's
> builtin
> > > > > > > ScriptEngine
> > > > > > > >> to eval. Of course, there'll be limitations. But, we can
> always
> > > > > improve
> > > > > > > >> them :)
> > > > > > > >>
> > > > > > > >> [1] https://highlightjs.org/
> > > > > > > >> [2] https://jshint.com/
> > > > > > > >>
> > > > > > > >> Regards,
> > > > > > > >> Grainier.
> > > > > > > >>
> > > > > > > >> On Thu, 14 May 2020 at 00:53, Dominik Riemer <
> riemer@apache.org
> > > >
> > > > > wrote:
> > > > > > > >>
> > > > > > > >>> Hi Grainier,
> > > > > > > >>>
> > > > > > > >>> in my opinion, this would definitely be a very interesting
> > > > > addition! We
> > > > > > > >>> already briefly discussed such a feature in the past,
> before we
> > > > > went
> > > > > > > to the
> > > > > > > >>> ASF. I’d say that a JS evaluator would be definitely
> useful to
> > > a
> > > > > > > slightly
> > > > > > > >>> more technical target user group, but would provide great
> > > > > flexibility
> > > > > > > for,
> > > > > > > >>> e.g., data harmonization tasks. So let’s discuss what is
> > > needed to
> > > > > > > >>> implement this! I’d guess that we (maybe) need to add some
> > > > > > > enhancements to
> > > > > > > >>> the core to make this processor good from a usability
> > > perspective.
> > > > > > > >>>
> > > > > > > >>> From a conceptual point of view, I guess the JS processor
> would
> > > > > have no
> > > > > > > >>> specific input requirements and a single static property
> that
> > > > > allows
> > > > > > > users
> > > > > > > >>> to enter the code (maybe it would be cool to add a new
> static
> > > > > property
> > > > > > > here
> > > > > > > >>> that supports code highlighting or even syntax checking).
> The
> > > only
> > > > > open
> > > > > > > >>> issue I see is the output schema – we would somehow need to
> > > > > extract the
> > > > > > > >>> output from the JS function. This could probably partially
> be
> > > done
> > > > > > > using
> > > > > > > >>> the CustomTransformOutput (see
> > > > > > > >>>
> > > > >
> http://streampipes.apache.org/docs/docs/dev-guide-output-strategies/),
> > > > > > > >>> but we would somehow need to derive type information from
> the
> > > JS
> > > > > > > function
> > > > > > > >>> or introduce a feature to let users define/refine the
> output
> > > > > manually
> > > > > > > >>> (e.g., to add semantic metadata).
> > > > > > > >>>
> > > > > > > >>> What do you think would be best? And would you evaluate
> the JS
> > > > > code in
> > > > > > > >>> Java or somehow else?
> > > > > > > >>>
> > > > > > > >>> Also, as I’ve just started to improve the SDK (e.g.,
> supporting
> > > > > > > optional
> > > > > > > >>> static properties and an easier way to define and extract
> model
> > > > > > > >>> parameters), we can collect your requirements for the JS
> > > processor
> > > > > and
> > > > > > > >>> extend the SDK if needed, just say what you would like to
> have
> > > 😉
> > > > > > > >>>
> > > > > > > >>> Dominik
> > > > > > > >>>
> > > > > > > >>>
> > > > > > > >>> From: Grainier Perera <gr...@gmail.com>
> > > > > > > >>> Sent: Wednesday, May 13, 2020 2:41 PM
> > > > > > > >>> To: dev@streampipes.apache.org
> > > > > > > >>> Subject: Data processor to evaluate JavaScript
> > > > > > > >>>
> > > > > > > >>> Hi all,
> > > > > > > >>>
> > > > > > > >>> I'm planning to implement a JavaScript eval data
> processor. As
> > > the
> > > > > name
> > > > > > > >>> suggests, users will be able to write some JavaScript code
> > > which
> > > > > takes
> > > > > > > in
> > > > > > > >>> an event (as a map), do some processing on the event, and
> > > return a
> > > > > new
> > > > > > > map
> > > > > > > >>> which then gets converted to an sp-event.
> > > > > > > >>>
> > > > > > > >>> example js:
> > > > > > > >>> function process(event) {
> > > > > > > >>>   // do processing here.
> > > > > > > >>>   // return processed event.
> > > > > > > >>>   return {id: http://event.id, tempInCelsius:
> > > (event.tempInKelvin
> > > > > -
> > > > > > > >>> 273.15)};
> > > > > > > >>> };
> > > > > > > >>>
> > > > > > > >>> Will this be a good addition to pipeline-elements? What do
> you
> > > > > think?
> > > > > > > >>>
> > > > > > > >>> Regards,
> > > > > > > >>> Grainier.
> > > > > > > >>>
> > > > > > > >>>
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Data processor to evaluate JavaScript

Posted by Dominik Riemer <ri...@apache.org>.

Hi Grainier,

I just pushed an initial version of the static property to add javascript code.
It's a simple codeblock with linting enabled and a (currently very simple) autocomplete feature that suggests existing properties from the input event (currently, only if you press ctrl-space directly after typing "event" followed by a dot, I'll fix that soon ;-)

You can find an example how to use it here:
https://github.com/apache/incubator-streampipes-examples/blob/dev/streampipes-pipeline-elements-examples-processors-jvm/src/main/java/org/apache/streampipes/pe/examples/jvm/staticproperty/CodeInputExampleController.java

Feedback and ideas for improvement are welcome!

Dominik


On 2020/05/17 12:36:05, Grainier Perera <gr...@gmail.com> wrote: 
> Hi Dominik,
> 
> Thank you. I'll work on `Implement JS evaluator` [1].
> 
> [1] https://issues.apache.org/jira/browse/STREAMPIPES-135
> 
> Grainier.
> 
> On Sun, 17 May 2020 at 18:02, Dominik Riemer <ri...@apache.org> wrote:
> 
> > Hi Grainier,
> >
> > great! I created an umbrella issue for that:
> > https://issues.apache.org/jira/browse/STREAMPIPES-132
> >
> > I'll now start to implement the new static property for entering code.
> >
> > Dominik
> >
> > On 2020/05/15 01:29:44, Grainier Perera <gr...@gmail.com>
> > wrote:
> > > Hi,
> > >
> > > I'm +1 to the idea of `UserDefinedOutputStrategy`. As you said, that'll
> > be
> > > re-usable in the future processors as well. So let's proceed with that.
> > >
> > > Regards,
> > > Grainier
> > >
> > > On Fri, 15 May 2020 at 01:37, Dominik Riemer <ri...@apache.org> wrote:
> > >
> > > > Hi,
> > > > cool, that sounds very good!
> > > >
> > > > Concerning the output, I think there are two ways to achieve this:
> > > > - A collection static property in conjunction with a runtime resolvable
> > > > output strategy that creates the output schema based on the input from
> > the
> > > > collection
> > > > - Or we create a new output strategy, e.g., let's call it
> > > > UserDefinedOutputStrategy, which would render a UI component to let
> > users
> > > > add/remove/change event properties from the input schema. I also like
> > the
> > > > idea of preselecting all input properties.
> > > >
> > > > I personally tend towards the second option as such a strategy might be
> > > > useful for other pipeline elements as well and it would ease the model
> > > > definition. We could also extend this component in the future, e.g., by
> > > > letting users enrich metadata such as measurement units directly in a
> > > > processor.
> > > >
> > > > But I'm also open to the first option and we can also just start
> > > > implementing to see what works best.
> > > >
> > > > I'd also be happy to help with the implementation, e.g., I could work
> > on
> > > > the new static property for code input or the output strategy!
> > > >
> > > > Dominik
> > > >
> > > > On 2020/05/14 11:59:36, Grainier Perera <gr...@gmail.com>
> > > > wrote:
> > > > > Hi,
> > > > >
> > > > > +1 to the idea of having a collection property containing the fields
> > of
> > > > the
> > > > > output properties. Also, If we can show the user with available
> > fields
> > > > with
> > > > > metadata (of input event) populated in that same collection property,
> > > > then
> > > > > the user can add/remove/updates fields easily. Let's say they just
> > have
> > > > to
> > > > > return a map containing all the fields they defined in output
> > properties
> > > > > (i.e return {"existingId": 1, "existingTempInKelvin": 301,
> > > > > "newTempInCelsius": 28 } ).
> > > > >
> > > > > [image: sample.png]
> > > > >
> > > > > Regards,
> > > > > Grainier.
> > > > >
> > > > > On Thu, 14 May 2020 at 12:33, Patrick Wiener <wi...@apache.org>
> > wrote:
> > > > >
> > > > > > Hi,
> > > > > >
> > > > > > sounds reasonable.
> > > > > >
> > > > > > I share the thoughts of Philipp regarding the output schema. But
> > as you
> > > > > > said Grainier, we can also improve :)
> > > > > >
> > > > > > btw: I really like the idea to also allow the more technical user
> > to
> > > > > > easily integrate their function snippets without having
> > > > > > to care about the actual execution. This gets a more FaaS like
> > feeling
> > > > for
> > > > > > streaming.
> > > > > >
> > > > > > Patrick
> > > > > >
> > > > > > > Am 14.05.2020 um 08:26 schrieb Philipp Zehnder <
> > zehnder@apache.org>:
> > > > > > >
> > > > > > > Hi,
> > > > > > >
> > > > > > > it would be very useful to have such a processor.
> > > > > > >
> > > > > > > I like the ideas for 1 & 2 and I think we can do it this way.
> > > > > > >
> > > > > > > Actually I think that the definition of the output scheme could
> > be
> > > > quite
> > > > > > tricky.
> > > > > > > Because users can do anything with the event.
> > > > > > >
> > > > > > > How about we provide an option property?
> > > > > > > Users can select whether they want to define the whole event or
> > just
> > > > > > append properties.
> > > > > > >
> > > > > > > For the configurations of the properties we could do something
> > > > similar
> > > > > > then we have in the PLC4xS7Adapter.
> > > > > > > A collection property containing the fields of the output
> > properties.
> > > > > > > Therefore, we have to decide what information we have to
> > provide. I
> > > > > > think if we would support all options it becomes very cluttered
> > for the
> > > > > > user.
> > > > > > >
> > > > > > > What do you think?
> > > > > > >
> > > > > > > Philipp
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >> On 14. May 2020, at 06:55, Grainier Perera <
> > > > grainier.perera@gmail.com>
> > > > > > wrote:
> > > > > > >>
> > > > > > >> Hi Dominik,
> > > > > > >>
> > > > > > >> I agree. We have to do some enhancements make this processor
> > user
> > > > > > friendly.
> > > > > > >> Like you said it's nice to have;
> > > > > > >>   1. A new static property that supports code highlighting &
> > syntax
> > > > > > >> checking. We can simply do such client-side validations using
> > > > > > highlight.js
> > > > > > >> (BSD) [1], jshint (MIT) [2], etc...
> > > > > > >>   2. A mechanism to show the user with available fields they can
> > > > use to
> > > > > > >> write code (similar to what you've done with usable templates in
> > > > > > >> EmailPublisher using HtmlInputParameter).
> > > > > > >>   3. A mechanism to map output schema.
> > > > > > >>
> > > > > > >> I think 1 & 2 go together and for the 3rd requirement given
> > that we
> > > > are
> > > > > > >> targetting a more technical user group for this processor, we
> > can
> > > > let
> > > > > > them
> > > > > > >> define the output. Otherwise, it would be quite difficult to
> > derive
> > > > > > return
> > > > > > >> type from the JS. Moreover, I'm planning to use Java's builtin
> > > > > > ScriptEngine
> > > > > > >> to eval. Of course, there'll be limitations. But, we can always
> > > > improve
> > > > > > >> them :)
> > > > > > >>
> > > > > > >> [1] https://highlightjs.org/
> > > > > > >> [2] https://jshint.com/
> > > > > > >>
> > > > > > >> Regards,
> > > > > > >> Grainier.
> > > > > > >>
> > > > > > >> On Thu, 14 May 2020 at 00:53, Dominik Riemer <riemer@apache.org
> > >
> > > > wrote:
> > > > > > >>
> > > > > > >>> Hi Grainier,
> > > > > > >>>
> > > > > > >>> in my opinion, this would definitely be a very interesting
> > > > addition! We
> > > > > > >>> already briefly discussed such a feature in the past, before we
> > > > went
> > > > > > to the
> > > > > > >>> ASF. I’d say that a JS evaluator would be definitely useful to
> > a
> > > > > > slightly
> > > > > > >>> more technical target user group, but would provide great
> > > > flexibility
> > > > > > for,
> > > > > > >>> e.g., data harmonization tasks. So let’s discuss what is
> > needed to
> > > > > > >>> implement this! I’d guess that we (maybe) need to add some
> > > > > > enhancements to
> > > > > > >>> the core to make this processor good from a usability
> > perspective.
> > > > > > >>>
> > > > > > >>> From a conceptual point of view, I guess the JS processor would
> > > > have no
> > > > > > >>> specific input requirements and a single static property that
> > > > allows
> > > > > > users
> > > > > > >>> to enter the code (maybe it would be cool to add a new static
> > > > property
> > > > > > here
> > > > > > >>> that supports code highlighting or even syntax checking). The
> > only
> > > > open
> > > > > > >>> issue I see is the output schema – we would somehow need to
> > > > extract the
> > > > > > >>> output from the JS function. This could probably partially be
> > done
> > > > > > using
> > > > > > >>> the CustomTransformOutput (see
> > > > > > >>>
> > > > http://streampipes.apache.org/docs/docs/dev-guide-output-strategies/),
> > > > > > >>> but we would somehow need to derive type information from the
> > JS
> > > > > > function
> > > > > > >>> or introduce a feature to let users define/refine the output
> > > > manually
> > > > > > >>> (e.g., to add semantic metadata).
> > > > > > >>>
> > > > > > >>> What do you think would be best? And would you evaluate the JS
> > > > code in
> > > > > > >>> Java or somehow else?
> > > > > > >>>
> > > > > > >>> Also, as I’ve just started to improve the SDK (e.g., supporting
> > > > > > optional
> > > > > > >>> static properties and an easier way to define and extract model
> > > > > > >>> parameters), we can collect your requirements for the JS
> > processor
> > > > and
> > > > > > >>> extend the SDK if needed, just say what you would like to have
> > 😉
> > > > > > >>>
> > > > > > >>> Dominik
> > > > > > >>>
> > > > > > >>>
> > > > > > >>> From: Grainier Perera <gr...@gmail.com>
> > > > > > >>> Sent: Wednesday, May 13, 2020 2:41 PM
> > > > > > >>> To: dev@streampipes.apache.org
> > > > > > >>> Subject: Data processor to evaluate JavaScript
> > > > > > >>>
> > > > > > >>> Hi all,
> > > > > > >>>
> > > > > > >>> I'm planning to implement a JavaScript eval data processor. As
> > the
> > > > name
> > > > > > >>> suggests, users will be able to write some JavaScript code
> > which
> > > > takes
> > > > > > in
> > > > > > >>> an event (as a map), do some processing on the event, and
> > return a
> > > > new
> > > > > > map
> > > > > > >>> which then gets converted to an sp-event.
> > > > > > >>>
> > > > > > >>> example js:
> > > > > > >>> function process(event) {
> > > > > > >>>   // do processing here.
> > > > > > >>>   // return processed event.
> > > > > > >>>   return {id: http://event.id, tempInCelsius:
> > (event.tempInKelvin
> > > > -
> > > > > > >>> 273.15)};
> > > > > > >>> };
> > > > > > >>>
> > > > > > >>> Will this be a good addition to pipeline-elements? What do you
> > > > think?
> > > > > > >>>
> > > > > > >>> Regards,
> > > > > > >>> Grainier.
> > > > > > >>>
> > > > > > >>>
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Data processor to evaluate JavaScript

Posted by Grainier Perera <gr...@gmail.com>.

Hi Dominik,

Thank you. I'll work on `Implement JS evaluator` [1].

[1] https://issues.apache.org/jira/browse/STREAMPIPES-135

Grainier.

On Sun, 17 May 2020 at 18:02, Dominik Riemer <ri...@apache.org> wrote:

> Hi Grainier,
>
> great! I created an umbrella issue for that:
> https://issues.apache.org/jira/browse/STREAMPIPES-132
>
> I'll now start to implement the new static property for entering code.
>
> Dominik
>
> On 2020/05/15 01:29:44, Grainier Perera <gr...@gmail.com>
> wrote:
> > Hi,
> >
> > I'm +1 to the idea of `UserDefinedOutputStrategy`. As you said, that'll
> be
> > re-usable in the future processors as well. So let's proceed with that.
> >
> > Regards,
> > Grainier
> >
> > On Fri, 15 May 2020 at 01:37, Dominik Riemer <ri...@apache.org> wrote:
> >
> > > Hi,
> > > cool, that sounds very good!
> > >
> > > Concerning the output, I think there are two ways to achieve this:
> > > - A collection static property in conjunction with a runtime resolvable
> > > output strategy that creates the output schema based on the input from
> the
> > > collection
> > > - Or we create a new output strategy, e.g., let's call it
> > > UserDefinedOutputStrategy, which would render a UI component to let
> users
> > > add/remove/change event properties from the input schema. I also like
> the
> > > idea of preselecting all input properties.
> > >
> > > I personally tend towards the second option as such a strategy might be
> > > useful for other pipeline elements as well and it would ease the model
> > > definition. We could also extend this component in the future, e.g., by
> > > letting users enrich metadata such as measurement units directly in a
> > > processor.
> > >
> > > But I'm also open to the first option and we can also just start
> > > implementing to see what works best.
> > >
> > > I'd also be happy to help with the implementation, e.g., I could work
> on
> > > the new static property for code input or the output strategy!
> > >
> > > Dominik
> > >
> > > On 2020/05/14 11:59:36, Grainier Perera <gr...@gmail.com>
> > > wrote:
> > > > Hi,
> > > >
> > > > +1 to the idea of having a collection property containing the fields
> of
> > > the
> > > > output properties. Also, If we can show the user with available
> fields
> > > with
> > > > metadata (of input event) populated in that same collection property,
> > > then
> > > > the user can add/remove/updates fields easily. Let's say they just
> have
> > > to
> > > > return a map containing all the fields they defined in output
> properties
> > > > (i.e return {"existingId": 1, "existingTempInKelvin": 301,
> > > > "newTempInCelsius": 28 } ).
> > > >
> > > > [image: sample.png]
> > > >
> > > > Regards,
> > > > Grainier.
> > > >
> > > > On Thu, 14 May 2020 at 12:33, Patrick Wiener <wi...@apache.org>
> wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > sounds reasonable.
> > > > >
> > > > > I share the thoughts of Philipp regarding the output schema. But
> as you
> > > > > said Grainier, we can also improve :)
> > > > >
> > > > > btw: I really like the idea to also allow the more technical user
> to
> > > > > easily integrate their function snippets without having
> > > > > to care about the actual execution. This gets a more FaaS like
> feeling
> > > for
> > > > > streaming.
> > > > >
> > > > > Patrick
> > > > >
> > > > > > Am 14.05.2020 um 08:26 schrieb Philipp Zehnder <
> zehnder@apache.org>:
> > > > > >
> > > > > > Hi,
> > > > > >
> > > > > > it would be very useful to have such a processor.
> > > > > >
> > > > > > I like the ideas for 1 & 2 and I think we can do it this way.
> > > > > >
> > > > > > Actually I think that the definition of the output scheme could
> be
> > > quite
> > > > > tricky.
> > > > > > Because users can do anything with the event.
> > > > > >
> > > > > > How about we provide an option property?
> > > > > > Users can select whether they want to define the whole event or
> just
> > > > > append properties.
> > > > > >
> > > > > > For the configurations of the properties we could do something
> > > similar
> > > > > then we have in the PLC4xS7Adapter.
> > > > > > A collection property containing the fields of the output
> properties.
> > > > > > Therefore, we have to decide what information we have to
> provide. I
> > > > > think if we would support all options it becomes very cluttered
> for the
> > > > > user.
> > > > > >
> > > > > > What do you think?
> > > > > >
> > > > > > Philipp
> > > > > >
> > > > > >
> > > > > >
> > > > > >> On 14. May 2020, at 06:55, Grainier Perera <
> > > grainier.perera@gmail.com>
> > > > > wrote:
> > > > > >>
> > > > > >> Hi Dominik,
> > > > > >>
> > > > > >> I agree. We have to do some enhancements make this processor
> user
> > > > > friendly.
> > > > > >> Like you said it's nice to have;
> > > > > >>   1. A new static property that supports code highlighting &
> syntax
> > > > > >> checking. We can simply do such client-side validations using
> > > > > highlight.js
> > > > > >> (BSD) [1], jshint (MIT) [2], etc...
> > > > > >>   2. A mechanism to show the user with available fields they can
> > > use to
> > > > > >> write code (similar to what you've done with usable templates in
> > > > > >> EmailPublisher using HtmlInputParameter).
> > > > > >>   3. A mechanism to map output schema.
> > > > > >>
> > > > > >> I think 1 & 2 go together and for the 3rd requirement given
> that we
> > > are
> > > > > >> targetting a more technical user group for this processor, we
> can
> > > let
> > > > > them
> > > > > >> define the output. Otherwise, it would be quite difficult to
> derive
> > > > > return
> > > > > >> type from the JS. Moreover, I'm planning to use Java's builtin
> > > > > ScriptEngine
> > > > > >> to eval. Of course, there'll be limitations. But, we can always
> > > improve
> > > > > >> them :)
> > > > > >>
> > > > > >> [1] https://highlightjs.org/
> > > > > >> [2] https://jshint.com/
> > > > > >>
> > > > > >> Regards,
> > > > > >> Grainier.
> > > > > >>
> > > > > >> On Thu, 14 May 2020 at 00:53, Dominik Riemer <riemer@apache.org
> >
> > > wrote:
> > > > > >>
> > > > > >>> Hi Grainier,
> > > > > >>>
> > > > > >>> in my opinion, this would definitely be a very interesting
> > > addition! We
> > > > > >>> already briefly discussed such a feature in the past, before we
> > > went
> > > > > to the
> > > > > >>> ASF. I’d say that a JS evaluator would be definitely useful to
> a
> > > > > slightly
> > > > > >>> more technical target user group, but would provide great
> > > flexibility
> > > > > for,
> > > > > >>> e.g., data harmonization tasks. So let’s discuss what is
> needed to
> > > > > >>> implement this! I’d guess that we (maybe) need to add some
> > > > > enhancements to
> > > > > >>> the core to make this processor good from a usability
> perspective.
> > > > > >>>
> > > > > >>> From a conceptual point of view, I guess the JS processor would
> > > have no
> > > > > >>> specific input requirements and a single static property that
> > > allows
> > > > > users
> > > > > >>> to enter the code (maybe it would be cool to add a new static
> > > property
> > > > > here
> > > > > >>> that supports code highlighting or even syntax checking). The
> only
> > > open
> > > > > >>> issue I see is the output schema – we would somehow need to
> > > extract the
> > > > > >>> output from the JS function. This could probably partially be
> done
> > > > > using
> > > > > >>> the CustomTransformOutput (see
> > > > > >>>
> > > http://streampipes.apache.org/docs/docs/dev-guide-output-strategies/),
> > > > > >>> but we would somehow need to derive type information from the
> JS
> > > > > function
> > > > > >>> or introduce a feature to let users define/refine the output
> > > manually
> > > > > >>> (e.g., to add semantic metadata).
> > > > > >>>
> > > > > >>> What do you think would be best? And would you evaluate the JS
> > > code in
> > > > > >>> Java or somehow else?
> > > > > >>>
> > > > > >>> Also, as I’ve just started to improve the SDK (e.g., supporting
> > > > > optional
> > > > > >>> static properties and an easier way to define and extract model
> > > > > >>> parameters), we can collect your requirements for the JS
> processor
> > > and
> > > > > >>> extend the SDK if needed, just say what you would like to have
> 😉
> > > > > >>>
> > > > > >>> Dominik
> > > > > >>>
> > > > > >>>
> > > > > >>> From: Grainier Perera <gr...@gmail.com>
> > > > > >>> Sent: Wednesday, May 13, 2020 2:41 PM
> > > > > >>> To: dev@streampipes.apache.org
> > > > > >>> Subject: Data processor to evaluate JavaScript
> > > > > >>>
> > > > > >>> Hi all,
> > > > > >>>
> > > > > >>> I'm planning to implement a JavaScript eval data processor. As
> the
> > > name
> > > > > >>> suggests, users will be able to write some JavaScript code
> which
> > > takes
> > > > > in
> > > > > >>> an event (as a map), do some processing on the event, and
> return a
> > > new
> > > > > map
> > > > > >>> which then gets converted to an sp-event.
> > > > > >>>
> > > > > >>> example js:
> > > > > >>> function process(event) {
> > > > > >>>   // do processing here.
> > > > > >>>   // return processed event.
> > > > > >>>   return {id: http://event.id, tempInCelsius:
> (event.tempInKelvin
> > > -
> > > > > >>> 273.15)};
> > > > > >>> };
> > > > > >>>
> > > > > >>> Will this be a good addition to pipeline-elements? What do you
> > > think?
> > > > > >>>
> > > > > >>> Regards,
> > > > > >>> Grainier.
> > > > > >>>
> > > > > >>>
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > >
> > >
> >
>

Re: Data processor to evaluate JavaScript

Posted by Dominik Riemer <ri...@apache.org>.

Hi Grainier,

great! I created an umbrella issue for that: https://issues.apache.org/jira/browse/STREAMPIPES-132

I'll now start to implement the new static property for entering code.

Dominik

On 2020/05/15 01:29:44, Grainier Perera <gr...@gmail.com> wrote: 
> Hi,
> 
> I'm +1 to the idea of `UserDefinedOutputStrategy`. As you said, that'll be
> re-usable in the future processors as well. So let's proceed with that.
> 
> Regards,
> Grainier
> 
> On Fri, 15 May 2020 at 01:37, Dominik Riemer <ri...@apache.org> wrote:
> 
> > Hi,
> > cool, that sounds very good!
> >
> > Concerning the output, I think there are two ways to achieve this:
> > - A collection static property in conjunction with a runtime resolvable
> > output strategy that creates the output schema based on the input from the
> > collection
> > - Or we create a new output strategy, e.g., let's call it
> > UserDefinedOutputStrategy, which would render a UI component to let users
> > add/remove/change event properties from the input schema. I also like the
> > idea of preselecting all input properties.
> >
> > I personally tend towards the second option as such a strategy might be
> > useful for other pipeline elements as well and it would ease the model
> > definition. We could also extend this component in the future, e.g., by
> > letting users enrich metadata such as measurement units directly in a
> > processor.
> >
> > But I'm also open to the first option and we can also just start
> > implementing to see what works best.
> >
> > I'd also be happy to help with the implementation, e.g., I could work on
> > the new static property for code input or the output strategy!
> >
> > Dominik
> >
> > On 2020/05/14 11:59:36, Grainier Perera <gr...@gmail.com>
> > wrote:
> > > Hi,
> > >
> > > +1 to the idea of having a collection property containing the fields of
> > the
> > > output properties. Also, If we can show the user with available fields
> > with
> > > metadata (of input event) populated in that same collection property,
> > then
> > > the user can add/remove/updates fields easily. Let's say they just have
> > to
> > > return a map containing all the fields they defined in output properties
> > > (i.e return {"existingId": 1, "existingTempInKelvin": 301,
> > > "newTempInCelsius": 28 } ).
> > >
> > > [image: sample.png]
> > >
> > > Regards,
> > > Grainier.
> > >
> > > On Thu, 14 May 2020 at 12:33, Patrick Wiener <wi...@apache.org> wrote:
> > >
> > > > Hi,
> > > >
> > > > sounds reasonable.
> > > >
> > > > I share the thoughts of Philipp regarding the output schema. But as you
> > > > said Grainier, we can also improve :)
> > > >
> > > > btw: I really like the idea to also allow the more technical user to
> > > > easily integrate their function snippets without having
> > > > to care about the actual execution. This gets a more FaaS like feeling
> > for
> > > > streaming.
> > > >
> > > > Patrick
> > > >
> > > > > Am 14.05.2020 um 08:26 schrieb Philipp Zehnder <ze...@apache.org>:
> > > > >
> > > > > Hi,
> > > > >
> > > > > it would be very useful to have such a processor.
> > > > >
> > > > > I like the ideas for 1 & 2 and I think we can do it this way.
> > > > >
> > > > > Actually I think that the definition of the output scheme could be
> > quite
> > > > tricky.
> > > > > Because users can do anything with the event.
> > > > >
> > > > > How about we provide an option property?
> > > > > Users can select whether they want to define the whole event or just
> > > > append properties.
> > > > >
> > > > > For the configurations of the properties we could do something
> > similar
> > > > then we have in the PLC4xS7Adapter.
> > > > > A collection property containing the fields of the output properties.
> > > > > Therefore, we have to decide what information we have to provide. I
> > > > think if we would support all options it becomes very cluttered for the
> > > > user.
> > > > >
> > > > > What do you think?
> > > > >
> > > > > Philipp
> > > > >
> > > > >
> > > > >
> > > > >> On 14. May 2020, at 06:55, Grainier Perera <
> > grainier.perera@gmail.com>
> > > > wrote:
> > > > >>
> > > > >> Hi Dominik,
> > > > >>
> > > > >> I agree. We have to do some enhancements make this processor user
> > > > friendly.
> > > > >> Like you said it's nice to have;
> > > > >>   1. A new static property that supports code highlighting & syntax
> > > > >> checking. We can simply do such client-side validations using
> > > > highlight.js
> > > > >> (BSD) [1], jshint (MIT) [2], etc...
> > > > >>   2. A mechanism to show the user with available fields they can
> > use to
> > > > >> write code (similar to what you've done with usable templates in
> > > > >> EmailPublisher using HtmlInputParameter).
> > > > >>   3. A mechanism to map output schema.
> > > > >>
> > > > >> I think 1 & 2 go together and for the 3rd requirement given that we
> > are
> > > > >> targetting a more technical user group for this processor, we can
> > let
> > > > them
> > > > >> define the output. Otherwise, it would be quite difficult to derive
> > > > return
> > > > >> type from the JS. Moreover, I'm planning to use Java's builtin
> > > > ScriptEngine
> > > > >> to eval. Of course, there'll be limitations. But, we can always
> > improve
> > > > >> them :)
> > > > >>
> > > > >> [1] https://highlightjs.org/
> > > > >> [2] https://jshint.com/
> > > > >>
> > > > >> Regards,
> > > > >> Grainier.
> > > > >>
> > > > >> On Thu, 14 May 2020 at 00:53, Dominik Riemer <ri...@apache.org>
> > wrote:
> > > > >>
> > > > >>> Hi Grainier,
> > > > >>>
> > > > >>> in my opinion, this would definitely be a very interesting
> > addition! We
> > > > >>> already briefly discussed such a feature in the past, before we
> > went
> > > > to the
> > > > >>> ASF. I’d say that a JS evaluator would be definitely useful to a
> > > > slightly
> > > > >>> more technical target user group, but would provide great
> > flexibility
> > > > for,
> > > > >>> e.g., data harmonization tasks. So let’s discuss what is needed to
> > > > >>> implement this! I’d guess that we (maybe) need to add some
> > > > enhancements to
> > > > >>> the core to make this processor good from a usability perspective.
> > > > >>>
> > > > >>> From a conceptual point of view, I guess the JS processor would
> > have no
> > > > >>> specific input requirements and a single static property that
> > allows
> > > > users
> > > > >>> to enter the code (maybe it would be cool to add a new static
> > property
> > > > here
> > > > >>> that supports code highlighting or even syntax checking). The only
> > open
> > > > >>> issue I see is the output schema – we would somehow need to
> > extract the
> > > > >>> output from the JS function. This could probably partially be done
> > > > using
> > > > >>> the CustomTransformOutput (see
> > > > >>>
> > http://streampipes.apache.org/docs/docs/dev-guide-output-strategies/),
> > > > >>> but we would somehow need to derive type information from the JS
> > > > function
> > > > >>> or introduce a feature to let users define/refine the output
> > manually
> > > > >>> (e.g., to add semantic metadata).
> > > > >>>
> > > > >>> What do you think would be best? And would you evaluate the JS
> > code in
> > > > >>> Java or somehow else?
> > > > >>>
> > > > >>> Also, as I’ve just started to improve the SDK (e.g., supporting
> > > > optional
> > > > >>> static properties and an easier way to define and extract model
> > > > >>> parameters), we can collect your requirements for the JS processor
> > and
> > > > >>> extend the SDK if needed, just say what you would like to have 😉
> > > > >>>
> > > > >>> Dominik
> > > > >>>
> > > > >>>
> > > > >>> From: Grainier Perera <gr...@gmail.com>
> > > > >>> Sent: Wednesday, May 13, 2020 2:41 PM
> > > > >>> To: dev@streampipes.apache.org
> > > > >>> Subject: Data processor to evaluate JavaScript
> > > > >>>
> > > > >>> Hi all,
> > > > >>>
> > > > >>> I'm planning to implement a JavaScript eval data processor. As the
> > name
> > > > >>> suggests, users will be able to write some JavaScript code which
> > takes
> > > > in
> > > > >>> an event (as a map), do some processing on the event, and return a
> > new
> > > > map
> > > > >>> which then gets converted to an sp-event.
> > > > >>>
> > > > >>> example js:
> > > > >>> function process(event) {
> > > > >>>   // do processing here.
> > > > >>>   // return processed event.
> > > > >>>   return {id: http://event.id, tempInCelsius: (event.tempInKelvin
> > -
> > > > >>> 273.15)};
> > > > >>> };
> > > > >>>
> > > > >>> Will this be a good addition to pipeline-elements? What do you
> > think?
> > > > >>>
> > > > >>> Regards,
> > > > >>> Grainier.
> > > > >>>
> > > > >>>
> > > > >
> > > > >
> > > >
> > > >
> > >
> >
>

Re: Data processor to evaluate JavaScript

Posted by Grainier Perera <gr...@gmail.com>.

Hi,

I'm +1 to the idea of `UserDefinedOutputStrategy`. As you said, that'll be
re-usable in the future processors as well. So let's proceed with that.

Regards,
Grainier

On Fri, 15 May 2020 at 01:37, Dominik Riemer <ri...@apache.org> wrote:

> Hi,
> cool, that sounds very good!
>
> Concerning the output, I think there are two ways to achieve this:
> - A collection static property in conjunction with a runtime resolvable
> output strategy that creates the output schema based on the input from the
> collection
> - Or we create a new output strategy, e.g., let's call it
> UserDefinedOutputStrategy, which would render a UI component to let users
> add/remove/change event properties from the input schema. I also like the
> idea of preselecting all input properties.
>
> I personally tend towards the second option as such a strategy might be
> useful for other pipeline elements as well and it would ease the model
> definition. We could also extend this component in the future, e.g., by
> letting users enrich metadata such as measurement units directly in a
> processor.
>
> But I'm also open to the first option and we can also just start
> implementing to see what works best.
>
> I'd also be happy to help with the implementation, e.g., I could work on
> the new static property for code input or the output strategy!
>
> Dominik
>
> On 2020/05/14 11:59:36, Grainier Perera <gr...@gmail.com>
> wrote:
> > Hi,
> >
> > +1 to the idea of having a collection property containing the fields of
> the
> > output properties. Also, If we can show the user with available fields
> with
> > metadata (of input event) populated in that same collection property,
> then
> > the user can add/remove/updates fields easily. Let's say they just have
> to
> > return a map containing all the fields they defined in output properties
> > (i.e return {"existingId": 1, "existingTempInKelvin": 301,
> > "newTempInCelsius": 28 } ).
> >
> > [image: sample.png]
> >
> > Regards,
> > Grainier.
> >
> > On Thu, 14 May 2020 at 12:33, Patrick Wiener <wi...@apache.org> wrote:
> >
> > > Hi,
> > >
> > > sounds reasonable.
> > >
> > > I share the thoughts of Philipp regarding the output schema. But as you
> > > said Grainier, we can also improve :)
> > >
> > > btw: I really like the idea to also allow the more technical user to
> > > easily integrate their function snippets without having
> > > to care about the actual execution. This gets a more FaaS like feeling
> for
> > > streaming.
> > >
> > > Patrick
> > >
> > > > Am 14.05.2020 um 08:26 schrieb Philipp Zehnder <ze...@apache.org>:
> > > >
> > > > Hi,
> > > >
> > > > it would be very useful to have such a processor.
> > > >
> > > > I like the ideas for 1 & 2 and I think we can do it this way.
> > > >
> > > > Actually I think that the definition of the output scheme could be
> quite
> > > tricky.
> > > > Because users can do anything with the event.
> > > >
> > > > How about we provide an option property?
> > > > Users can select whether they want to define the whole event or just
> > > append properties.
> > > >
> > > > For the configurations of the properties we could do something
> similar
> > > then we have in the PLC4xS7Adapter.
> > > > A collection property containing the fields of the output properties.
> > > > Therefore, we have to decide what information we have to provide. I
> > > think if we would support all options it becomes very cluttered for the
> > > user.
> > > >
> > > > What do you think?
> > > >
> > > > Philipp
> > > >
> > > >
> > > >
> > > >> On 14. May 2020, at 06:55, Grainier Perera <
> grainier.perera@gmail.com>
> > > wrote:
> > > >>
> > > >> Hi Dominik,
> > > >>
> > > >> I agree. We have to do some enhancements make this processor user
> > > friendly.
> > > >> Like you said it's nice to have;
> > > >>   1. A new static property that supports code highlighting & syntax
> > > >> checking. We can simply do such client-side validations using
> > > highlight.js
> > > >> (BSD) [1], jshint (MIT) [2], etc...
> > > >>   2. A mechanism to show the user with available fields they can
> use to
> > > >> write code (similar to what you've done with usable templates in
> > > >> EmailPublisher using HtmlInputParameter).
> > > >>   3. A mechanism to map output schema.
> > > >>
> > > >> I think 1 & 2 go together and for the 3rd requirement given that we
> are
> > > >> targetting a more technical user group for this processor, we can
> let
> > > them
> > > >> define the output. Otherwise, it would be quite difficult to derive
> > > return
> > > >> type from the JS. Moreover, I'm planning to use Java's builtin
> > > ScriptEngine
> > > >> to eval. Of course, there'll be limitations. But, we can always
> improve
> > > >> them :)
> > > >>
> > > >> [1] https://highlightjs.org/
> > > >> [2] https://jshint.com/
> > > >>
> > > >> Regards,
> > > >> Grainier.
> > > >>
> > > >> On Thu, 14 May 2020 at 00:53, Dominik Riemer <ri...@apache.org>
> wrote:
> > > >>
> > > >>> Hi Grainier,
> > > >>>
> > > >>> in my opinion, this would definitely be a very interesting
> addition! We
> > > >>> already briefly discussed such a feature in the past, before we
> went
> > > to the
> > > >>> ASF. I’d say that a JS evaluator would be definitely useful to a
> > > slightly
> > > >>> more technical target user group, but would provide great
> flexibility
> > > for,
> > > >>> e.g., data harmonization tasks. So let’s discuss what is needed to
> > > >>> implement this! I’d guess that we (maybe) need to add some
> > > enhancements to
> > > >>> the core to make this processor good from a usability perspective.
> > > >>>
> > > >>> From a conceptual point of view, I guess the JS processor would
> have no
> > > >>> specific input requirements and a single static property that
> allows
> > > users
> > > >>> to enter the code (maybe it would be cool to add a new static
> property
> > > here
> > > >>> that supports code highlighting or even syntax checking). The only
> open
> > > >>> issue I see is the output schema – we would somehow need to
> extract the
> > > >>> output from the JS function. This could probably partially be done
> > > using
> > > >>> the CustomTransformOutput (see
> > > >>>
> http://streampipes.apache.org/docs/docs/dev-guide-output-strategies/),
> > > >>> but we would somehow need to derive type information from the JS
> > > function
> > > >>> or introduce a feature to let users define/refine the output
> manually
> > > >>> (e.g., to add semantic metadata).
> > > >>>
> > > >>> What do you think would be best? And would you evaluate the JS
> code in
> > > >>> Java or somehow else?
> > > >>>
> > > >>> Also, as I’ve just started to improve the SDK (e.g., supporting
> > > optional
> > > >>> static properties and an easier way to define and extract model
> > > >>> parameters), we can collect your requirements for the JS processor
> and
> > > >>> extend the SDK if needed, just say what you would like to have 😉
> > > >>>
> > > >>> Dominik
> > > >>>
> > > >>>
> > > >>> From: Grainier Perera <gr...@gmail.com>
> > > >>> Sent: Wednesday, May 13, 2020 2:41 PM
> > > >>> To: dev@streampipes.apache.org
> > > >>> Subject: Data processor to evaluate JavaScript
> > > >>>
> > > >>> Hi all,
> > > >>>
> > > >>> I'm planning to implement a JavaScript eval data processor. As the
> name
> > > >>> suggests, users will be able to write some JavaScript code which
> takes
> > > in
> > > >>> an event (as a map), do some processing on the event, and return a
> new
> > > map
> > > >>> which then gets converted to an sp-event.
> > > >>>
> > > >>> example js:
> > > >>> function process(event) {
> > > >>>   // do processing here.
> > > >>>   // return processed event.
> > > >>>   return {id: http://event.id, tempInCelsius: (event.tempInKelvin
> -
> > > >>> 273.15)};
> > > >>> };
> > > >>>
> > > >>> Will this be a good addition to pipeline-elements? What do you
> think?
> > > >>>
> > > >>> Regards,
> > > >>> Grainier.
> > > >>>
> > > >>>
> > > >
> > > >
> > >
> > >
> >
>

Re: Data processor to evaluate JavaScript

Posted by Dominik Riemer <ri...@apache.org>.

Hi,
cool, that sounds very good!

Concerning the output, I think there are two ways to achieve this:
- A collection static property in conjunction with a runtime resolvable output strategy that creates the output schema based on the input from the collection 
- Or we create a new output strategy, e.g., let's call it UserDefinedOutputStrategy, which would render a UI component to let users add/remove/change event properties from the input schema. I also like the idea of preselecting all input properties.

I personally tend towards the second option as such a strategy might be useful for other pipeline elements as well and it would ease the model definition. We could also extend this component in the future, e.g., by letting users enrich metadata such as measurement units directly in a processor.

But I'm also open to the first option and we can also just start implementing to see what works best.

I'd also be happy to help with the implementation, e.g., I could work on the new static property for code input or the output strategy!

Dominik

On 2020/05/14 11:59:36, Grainier Perera <gr...@gmail.com> wrote: 
> Hi,
> 
> +1 to the idea of having a collection property containing the fields of the
> output properties. Also, If we can show the user with available fields with
> metadata (of input event) populated in that same collection property, then
> the user can add/remove/updates fields easily. Let's say they just have to
> return a map containing all the fields they defined in output properties
> (i.e return {"existingId": 1, "existingTempInKelvin": 301,
> "newTempInCelsius": 28 } ).
> 
> [image: sample.png]
> 
> Regards,
> Grainier.
> 
> On Thu, 14 May 2020 at 12:33, Patrick Wiener <wi...@apache.org> wrote:
> 
> > Hi,
> >
> > sounds reasonable.
> >
> > I share the thoughts of Philipp regarding the output schema. But as you
> > said Grainier, we can also improve :)
> >
> > btw: I really like the idea to also allow the more technical user to
> > easily integrate their function snippets without having
> > to care about the actual execution. This gets a more FaaS like feeling for
> > streaming.
> >
> > Patrick
> >
> > > Am 14.05.2020 um 08:26 schrieb Philipp Zehnder <ze...@apache.org>:
> > >
> > > Hi,
> > >
> > > it would be very useful to have such a processor.
> > >
> > > I like the ideas for 1 & 2 and I think we can do it this way.
> > >
> > > Actually I think that the definition of the output scheme could be quite
> > tricky.
> > > Because users can do anything with the event.
> > >
> > > How about we provide an option property?
> > > Users can select whether they want to define the whole event or just
> > append properties.
> > >
> > > For the configurations of the properties we could do something similar
> > then we have in the PLC4xS7Adapter.
> > > A collection property containing the fields of the output properties.
> > > Therefore, we have to decide what information we have to provide. I
> > think if we would support all options it becomes very cluttered for the
> > user.
> > >
> > > What do you think?
> > >
> > > Philipp
> > >
> > >
> > >
> > >> On 14. May 2020, at 06:55, Grainier Perera <gr...@gmail.com>
> > wrote:
> > >>
> > >> Hi Dominik,
> > >>
> > >> I agree. We have to do some enhancements make this processor user
> > friendly.
> > >> Like you said it's nice to have;
> > >>   1. A new static property that supports code highlighting & syntax
> > >> checking. We can simply do such client-side validations using
> > highlight.js
> > >> (BSD) [1], jshint (MIT) [2], etc...
> > >>   2. A mechanism to show the user with available fields they can use to
> > >> write code (similar to what you've done with usable templates in
> > >> EmailPublisher using HtmlInputParameter).
> > >>   3. A mechanism to map output schema.
> > >>
> > >> I think 1 & 2 go together and for the 3rd requirement given that we are
> > >> targetting a more technical user group for this processor, we can let
> > them
> > >> define the output. Otherwise, it would be quite difficult to derive
> > return
> > >> type from the JS. Moreover, I'm planning to use Java's builtin
> > ScriptEngine
> > >> to eval. Of course, there'll be limitations. But, we can always improve
> > >> them :)
> > >>
> > >> [1] https://highlightjs.org/
> > >> [2] https://jshint.com/
> > >>
> > >> Regards,
> > >> Grainier.
> > >>
> > >> On Thu, 14 May 2020 at 00:53, Dominik Riemer <ri...@apache.org> wrote:
> > >>
> > >>> Hi Grainier,
> > >>>
> > >>> in my opinion, this would definitely be a very interesting addition! We
> > >>> already briefly discussed such a feature in the past, before we went
> > to the
> > >>> ASF. I’d say that a JS evaluator would be definitely useful to a
> > slightly
> > >>> more technical target user group, but would provide great flexibility
> > for,
> > >>> e.g., data harmonization tasks. So let’s discuss what is needed to
> > >>> implement this! I’d guess that we (maybe) need to add some
> > enhancements to
> > >>> the core to make this processor good from a usability perspective.
> > >>>
> > >>> From a conceptual point of view, I guess the JS processor would have no
> > >>> specific input requirements and a single static property that allows
> > users
> > >>> to enter the code (maybe it would be cool to add a new static property
> > here
> > >>> that supports code highlighting or even syntax checking). The only open
> > >>> issue I see is the output schema – we would somehow need to extract the
> > >>> output from the JS function. This could probably partially be done
> > using
> > >>> the CustomTransformOutput (see
> > >>> http://streampipes.apache.org/docs/docs/dev-guide-output-strategies/),
> > >>> but we would somehow need to derive type information from the JS
> > function
> > >>> or introduce a feature to let users define/refine the output manually
> > >>> (e.g., to add semantic metadata).
> > >>>
> > >>> What do you think would be best? And would you evaluate the JS code in
> > >>> Java or somehow else?
> > >>>
> > >>> Also, as I’ve just started to improve the SDK (e.g., supporting
> > optional
> > >>> static properties and an easier way to define and extract model
> > >>> parameters), we can collect your requirements for the JS processor and
> > >>> extend the SDK if needed, just say what you would like to have 😉
> > >>>
> > >>> Dominik
> > >>>
> > >>>
> > >>> From: Grainier Perera <gr...@gmail.com>
> > >>> Sent: Wednesday, May 13, 2020 2:41 PM
> > >>> To: dev@streampipes.apache.org
> > >>> Subject: Data processor to evaluate JavaScript
> > >>>
> > >>> Hi all,
> > >>>
> > >>> I'm planning to implement a JavaScript eval data processor. As the name
> > >>> suggests, users will be able to write some JavaScript code which takes
> > in
> > >>> an event (as a map), do some processing on the event, and return a new
> > map
> > >>> which then gets converted to an sp-event.
> > >>>
> > >>> example js:
> > >>> function process(event) {
> > >>>   // do processing here.
> > >>>   // return processed event.
> > >>>   return {id: http://event.id, tempInCelsius: (event.tempInKelvin -
> > >>> 273.15)};
> > >>> };
> > >>>
> > >>> Will this be a good addition to pipeline-elements? What do you think?
> > >>>
> > >>> Regards,
> > >>> Grainier.
> > >>>
> > >>>
> > >
> > >
> >
> >
>

Re: Data processor to evaluate JavaScript

Posted by Grainier Perera <gr...@gmail.com>.

Hi,

+1 to the idea of having a collection property containing the fields of the
output properties. Also, If we can show the user with available fields with
metadata (of input event) populated in that same collection property, then
the user can add/remove/updates fields easily. Let's say they just have to
return a map containing all the fields they defined in output properties
(i.e return {"existingId": 1, "existingTempInKelvin": 301,
"newTempInCelsius": 28 } ).

[image: sample.png]

Regards,
Grainier.

On Thu, 14 May 2020 at 12:33, Patrick Wiener <wi...@apache.org> wrote:

> Hi,
>
> sounds reasonable.
>
> I share the thoughts of Philipp regarding the output schema. But as you
> said Grainier, we can also improve :)
>
> btw: I really like the idea to also allow the more technical user to
> easily integrate their function snippets without having
> to care about the actual execution. This gets a more FaaS like feeling for
> streaming.
>
> Patrick
>
> > Am 14.05.2020 um 08:26 schrieb Philipp Zehnder <ze...@apache.org>:
> >
> > Hi,
> >
> > it would be very useful to have such a processor.
> >
> > I like the ideas for 1 & 2 and I think we can do it this way.
> >
> > Actually I think that the definition of the output scheme could be quite
> tricky.
> > Because users can do anything with the event.
> >
> > How about we provide an option property?
> > Users can select whether they want to define the whole event or just
> append properties.
> >
> > For the configurations of the properties we could do something similar
> then we have in the PLC4xS7Adapter.
> > A collection property containing the fields of the output properties.
> > Therefore, we have to decide what information we have to provide. I
> think if we would support all options it becomes very cluttered for the
> user.
> >
> > What do you think?
> >
> > Philipp
> >
> >
> >
> >> On 14. May 2020, at 06:55, Grainier Perera <gr...@gmail.com>
> wrote:
> >>
> >> Hi Dominik,
> >>
> >> I agree. We have to do some enhancements make this processor user
> friendly.
> >> Like you said it's nice to have;
> >>   1. A new static property that supports code highlighting & syntax
> >> checking. We can simply do such client-side validations using
> highlight.js
> >> (BSD) [1], jshint (MIT) [2], etc...
> >>   2. A mechanism to show the user with available fields they can use to
> >> write code (similar to what you've done with usable templates in
> >> EmailPublisher using HtmlInputParameter).
> >>   3. A mechanism to map output schema.
> >>
> >> I think 1 & 2 go together and for the 3rd requirement given that we are
> >> targetting a more technical user group for this processor, we can let
> them
> >> define the output. Otherwise, it would be quite difficult to derive
> return
> >> type from the JS. Moreover, I'm planning to use Java's builtin
> ScriptEngine
> >> to eval. Of course, there'll be limitations. But, we can always improve
> >> them :)
> >>
> >> [1] https://highlightjs.org/
> >> [2] https://jshint.com/
> >>
> >> Regards,
> >> Grainier.
> >>
> >> On Thu, 14 May 2020 at 00:53, Dominik Riemer <ri...@apache.org> wrote:
> >>
> >>> Hi Grainier,
> >>>
> >>> in my opinion, this would definitely be a very interesting addition! We
> >>> already briefly discussed such a feature in the past, before we went
> to the
> >>> ASF. I’d say that a JS evaluator would be definitely useful to a
> slightly
> >>> more technical target user group, but would provide great flexibility
> for,
> >>> e.g., data harmonization tasks. So let’s discuss what is needed to
> >>> implement this! I’d guess that we (maybe) need to add some
> enhancements to
> >>> the core to make this processor good from a usability perspective.
> >>>
> >>> From a conceptual point of view, I guess the JS processor would have no
> >>> specific input requirements and a single static property that allows
> users
> >>> to enter the code (maybe it would be cool to add a new static property
> here
> >>> that supports code highlighting or even syntax checking). The only open
> >>> issue I see is the output schema – we would somehow need to extract the
> >>> output from the JS function. This could probably partially be done
> using
> >>> the CustomTransformOutput (see
> >>> http://streampipes.apache.org/docs/docs/dev-guide-output-strategies/),
> >>> but we would somehow need to derive type information from the JS
> function
> >>> or introduce a feature to let users define/refine the output manually
> >>> (e.g., to add semantic metadata).
> >>>
> >>> What do you think would be best? And would you evaluate the JS code in
> >>> Java or somehow else?
> >>>
> >>> Also, as I’ve just started to improve the SDK (e.g., supporting
> optional
> >>> static properties and an easier way to define and extract model
> >>> parameters), we can collect your requirements for the JS processor and
> >>> extend the SDK if needed, just say what you would like to have 😉
> >>>
> >>> Dominik
> >>>
> >>>
> >>> From: Grainier Perera <gr...@gmail.com>
> >>> Sent: Wednesday, May 13, 2020 2:41 PM
> >>> To: dev@streampipes.apache.org
> >>> Subject: Data processor to evaluate JavaScript
> >>>
> >>> Hi all,
> >>>
> >>> I'm planning to implement a JavaScript eval data processor. As the name
> >>> suggests, users will be able to write some JavaScript code which takes
> in
> >>> an event (as a map), do some processing on the event, and return a new
> map
> >>> which then gets converted to an sp-event.
> >>>
> >>> example js:
> >>> function process(event) {
> >>>   // do processing here.
> >>>   // return processed event.
> >>>   return {id: http://event.id, tempInCelsius: (event.tempInKelvin -
> >>> 273.15)};
> >>> };
> >>>
> >>> Will this be a good addition to pipeline-elements? What do you think?
> >>>
> >>> Regards,
> >>> Grainier.
> >>>
> >>>
> >
> >
>
>

Re: Data processor to evaluate JavaScript

Posted by Patrick Wiener <wi...@apache.org>.

Hi,

sounds reasonable.

I share the thoughts of Philipp regarding the output schema. But as you said Grainier, we can also improve :)

btw: I really like the idea to also allow the more technical user to easily integrate their function snippets without having
to care about the actual execution. This gets a more FaaS like feeling for streaming.

Patrick

> Am 14.05.2020 um 08:26 schrieb Philipp Zehnder <ze...@apache.org>:
> 
> Hi,
> 
> it would be very useful to have such a processor.
> 
> I like the ideas for 1 & 2 and I think we can do it this way.
> 
> Actually I think that the definition of the output scheme could be quite tricky.
> Because users can do anything with the event.
> 
> How about we provide an option property?
> Users can select whether they want to define the whole event or just append properties.
> 
> For the configurations of the properties we could do something similar then we have in the PLC4xS7Adapter.
> A collection property containing the fields of the output properties.
> Therefore, we have to decide what information we have to provide. I think if we would support all options it becomes very cluttered for the user.
> 
> What do you think?
> 
> Philipp
> 
> 
> 
>> On 14. May 2020, at 06:55, Grainier Perera <gr...@gmail.com> wrote:
>> 
>> Hi Dominik,
>> 
>> I agree. We have to do some enhancements make this processor user friendly.
>> Like you said it's nice to have;
>>   1. A new static property that supports code highlighting & syntax
>> checking. We can simply do such client-side validations using highlight.js
>> (BSD) [1], jshint (MIT) [2], etc...
>>   2. A mechanism to show the user with available fields they can use to
>> write code (similar to what you've done with usable templates in
>> EmailPublisher using HtmlInputParameter).
>>   3. A mechanism to map output schema.
>> 
>> I think 1 & 2 go together and for the 3rd requirement given that we are
>> targetting a more technical user group for this processor, we can let them
>> define the output. Otherwise, it would be quite difficult to derive return
>> type from the JS. Moreover, I'm planning to use Java's builtin ScriptEngine
>> to eval. Of course, there'll be limitations. But, we can always improve
>> them :)
>> 
>> [1] https://highlightjs.org/
>> [2] https://jshint.com/
>> 
>> Regards,
>> Grainier.
>> 
>> On Thu, 14 May 2020 at 00:53, Dominik Riemer <ri...@apache.org> wrote:
>> 
>>> Hi Grainier,
>>> 
>>> in my opinion, this would definitely be a very interesting addition! We
>>> already briefly discussed such a feature in the past, before we went to the
>>> ASF. I’d say that a JS evaluator would be definitely useful to a slightly
>>> more technical target user group, but would provide great flexibility for,
>>> e.g., data harmonization tasks. So let’s discuss what is needed to
>>> implement this! I’d guess that we (maybe) need to add some enhancements to
>>> the core to make this processor good from a usability perspective.
>>> 
>>> From a conceptual point of view, I guess the JS processor would have no
>>> specific input requirements and a single static property that allows users
>>> to enter the code (maybe it would be cool to add a new static property here
>>> that supports code highlighting or even syntax checking). The only open
>>> issue I see is the output schema – we would somehow need to extract the
>>> output from the JS function. This could probably partially be done using
>>> the CustomTransformOutput (see
>>> http://streampipes.apache.org/docs/docs/dev-guide-output-strategies/),
>>> but we would somehow need to derive type information from the JS function
>>> or introduce a feature to let users define/refine the output manually
>>> (e.g., to add semantic metadata).
>>> 
>>> What do you think would be best? And would you evaluate the JS code in
>>> Java or somehow else?
>>> 
>>> Also, as I’ve just started to improve the SDK (e.g., supporting optional
>>> static properties and an easier way to define and extract model
>>> parameters), we can collect your requirements for the JS processor and
>>> extend the SDK if needed, just say what you would like to have 😉
>>> 
>>> Dominik
>>> 
>>> 
>>> From: Grainier Perera <gr...@gmail.com>
>>> Sent: Wednesday, May 13, 2020 2:41 PM
>>> To: dev@streampipes.apache.org
>>> Subject: Data processor to evaluate JavaScript
>>> 
>>> Hi all,
>>> 
>>> I'm planning to implement a JavaScript eval data processor. As the name
>>> suggests, users will be able to write some JavaScript code which takes in
>>> an event (as a map), do some processing on the event, and return a new map
>>> which then gets converted to an sp-event.
>>> 
>>> example js:
>>> function process(event) {
>>>   // do processing here.
>>>   // return processed event.
>>>   return {id: http://event.id, tempInCelsius: (event.tempInKelvin -
>>> 273.15)};
>>> };
>>> 
>>> Will this be a good addition to pipeline-elements? What do you think?
>>> 
>>> Regards,
>>> Grainier.
>>> 
>>> 
> 
>

Re: Data processor to evaluate JavaScript

Posted by Philipp Zehnder <ze...@apache.org>.

Hi,

it would be very useful to have such a processor.

I like the ideas for 1 & 2 and I think we can do it this way.

Actually I think that the definition of the output scheme could be quite tricky.
Because users can do anything with the event.

How about we provide an option property?
Users can select whether they want to define the whole event or just append properties.

For the configurations of the properties we could do something similar then we have in the PLC4xS7Adapter.
A collection property containing the fields of the output properties.
Therefore, we have to decide what information we have to provide. I think if we would support all options it becomes very cluttered for the user.

What do you think?

Philipp



> On 14. May 2020, at 06:55, Grainier Perera <gr...@gmail.com> wrote:
> 
> Hi Dominik,
> 
> I agree. We have to do some enhancements make this processor user friendly.
> Like you said it's nice to have;
>    1. A new static property that supports code highlighting & syntax
> checking. We can simply do such client-side validations using highlight.js
> (BSD) [1], jshint (MIT) [2], etc...
>    2. A mechanism to show the user with available fields they can use to
> write code (similar to what you've done with usable templates in
> EmailPublisher using HtmlInputParameter).
>    3. A mechanism to map output schema.
> 
> I think 1 & 2 go together and for the 3rd requirement given that we are
> targetting a more technical user group for this processor, we can let them
> define the output. Otherwise, it would be quite difficult to derive return
> type from the JS. Moreover, I'm planning to use Java's builtin ScriptEngine
> to eval. Of course, there'll be limitations. But, we can always improve
> them :)
> 
> [1] https://highlightjs.org/
> [2] https://jshint.com/
> 
> Regards,
> Grainier.
> 
> On Thu, 14 May 2020 at 00:53, Dominik Riemer <ri...@apache.org> wrote:
> 
>> Hi Grainier,
>> 
>> in my opinion, this would definitely be a very interesting addition! We
>> already briefly discussed such a feature in the past, before we went to the
>> ASF. I’d say that a JS evaluator would be definitely useful to a slightly
>> more technical target user group, but would provide great flexibility for,
>> e.g., data harmonization tasks. So let’s discuss what is needed to
>> implement this! I’d guess that we (maybe) need to add some enhancements to
>> the core to make this processor good from a usability perspective.
>> 
>> From a conceptual point of view, I guess the JS processor would have no
>> specific input requirements and a single static property that allows users
>> to enter the code (maybe it would be cool to add a new static property here
>> that supports code highlighting or even syntax checking). The only open
>> issue I see is the output schema – we would somehow need to extract the
>> output from the JS function. This could probably partially be done using
>> the CustomTransformOutput (see
>> http://streampipes.apache.org/docs/docs/dev-guide-output-strategies/),
>> but we would somehow need to derive type information from the JS function
>> or introduce a feature to let users define/refine the output manually
>> (e.g., to add semantic metadata).
>> 
>> What do you think would be best? And would you evaluate the JS code in
>> Java or somehow else?
>> 
>> Also, as I’ve just started to improve the SDK (e.g., supporting optional
>> static properties and an easier way to define and extract model
>> parameters), we can collect your requirements for the JS processor and
>> extend the SDK if needed, just say what you would like to have 😉
>> 
>> Dominik
>> 
>> 
>> From: Grainier Perera <gr...@gmail.com>
>> Sent: Wednesday, May 13, 2020 2:41 PM
>> To: dev@streampipes.apache.org
>> Subject: Data processor to evaluate JavaScript
>> 
>> Hi all,
>> 
>> I'm planning to implement a JavaScript eval data processor. As the name
>> suggests, users will be able to write some JavaScript code which takes in
>> an event (as a map), do some processing on the event, and return a new map
>> which then gets converted to an sp-event.
>> 
>> example js:
>> function process(event) {
>>    // do processing here.
>>    // return processed event.
>>    return {id: http://event.id, tempInCelsius: (event.tempInKelvin -
>> 273.15)};
>> };
>> 
>> Will this be a good addition to pipeline-elements? What do you think?
>> 
>> Regards,
>> Grainier.
>> 
>>

Re: Data processor to evaluate JavaScript

Posted by Grainier Perera <gr...@gmail.com>.

Hi Dominik,

I agree. We have to do some enhancements make this processor user friendly.
Like you said it's nice to have;
    1. A new static property that supports code highlighting & syntax
checking. We can simply do such client-side validations using highlight.js
(BSD) [1], jshint (MIT) [2], etc...
    2. A mechanism to show the user with available fields they can use to
write code (similar to what you've done with usable templates in
EmailPublisher using HtmlInputParameter).
    3. A mechanism to map output schema.

I think 1 & 2 go together and for the 3rd requirement given that we are
targetting a more technical user group for this processor, we can let them
define the output. Otherwise, it would be quite difficult to derive return
type from the JS. Moreover, I'm planning to use Java's builtin ScriptEngine
to eval. Of course, there'll be limitations. But, we can always improve
them :)

[1] https://highlightjs.org/
[2] https://jshint.com/

Regards,
Grainier.

On Thu, 14 May 2020 at 00:53, Dominik Riemer <ri...@apache.org> wrote:

> Hi Grainier,
>
> in my opinion, this would definitely be a very interesting addition! We
> already briefly discussed such a feature in the past, before we went to the
> ASF. I’d say that a JS evaluator would be definitely useful to a slightly
> more technical target user group, but would provide great flexibility for,
> e.g., data harmonization tasks. So let’s discuss what is needed to
> implement this! I’d guess that we (maybe) need to add some enhancements to
> the core to make this processor good from a usability perspective.
>
> From a conceptual point of view, I guess the JS processor would have no
> specific input requirements and a single static property that allows users
> to enter the code (maybe it would be cool to add a new static property here
> that supports code highlighting or even syntax checking). The only open
> issue I see is the output schema – we would somehow need to extract the
> output from the JS function. This could probably partially be done using
> the CustomTransformOutput (see
> http://streampipes.apache.org/docs/docs/dev-guide-output-strategies/),
> but we would somehow need to derive type information from the JS function
> or introduce a feature to let users define/refine the output manually
> (e.g., to add semantic metadata).
>
> What do you think would be best? And would you evaluate the JS code in
> Java or somehow else?
>
> Also, as I’ve just started to improve the SDK (e.g., supporting optional
> static properties and an easier way to define and extract model
> parameters), we can collect your requirements for the JS processor and
> extend the SDK if needed, just say what you would like to have 😉
>
> Dominik
>
>
> From: Grainier Perera <gr...@gmail.com>
> Sent: Wednesday, May 13, 2020 2:41 PM
> To: dev@streampipes.apache.org
> Subject: Data processor to evaluate JavaScript
>
> Hi all,
>
> I'm planning to implement a JavaScript eval data processor. As the name
> suggests, users will be able to write some JavaScript code which takes in
> an event (as a map), do some processing on the event, and return a new map
> which then gets converted to an sp-event.
>
> example js:
> function process(event) {
>     // do processing here.
>     // return processed event.
>     return {id: http://event.id, tempInCelsius: (event.tempInKelvin -
> 273.15)};
> };
>
> Will this be a good addition to pipeline-elements? What do you think?
>
> Regards,
> Grainier.
>
>

RE: Data processor to evaluate JavaScript

Posted by Dominik Riemer <ri...@apache.org>.

Hi Grainier,

in my opinion, this would definitely be a very interesting addition! We already briefly discussed such a feature in the past, before we went to the ASF. I’d say that a JS evaluator would be definitely useful to a slightly more technical target user group, but would provide great flexibility for, e.g., data harmonization tasks. So let’s discuss what is needed to implement this! I’d guess that we (maybe) need to add some enhancements to the core to make this processor good from a usability perspective.

From a conceptual point of view, I guess the JS processor would have no specific input requirements and a single static property that allows users to enter the code (maybe it would be cool to add a new static property here that supports code highlighting or even syntax checking). The only open issue I see is the output schema – we would somehow need to extract the output from the JS function. This could probably partially be done using the CustomTransformOutput (see http://streampipes.apache.org/docs/docs/dev-guide-output-strategies/), but we would somehow need to derive type information from the JS function or introduce a feature to let users define/refine the output manually (e.g., to add semantic metadata).

What do you think would be best? And would you evaluate the JS code in Java or somehow else?

Also, as I’ve just started to improve the SDK (e.g., supporting optional static properties and an easier way to define and extract model parameters), we can collect your requirements for the JS processor and extend the SDK if needed, just say what you would like to have 😉

Dominik


From: Grainier Perera <gr...@gmail.com> 
Sent: Wednesday, May 13, 2020 2:41 PM
To: dev@streampipes.apache.org
Subject: Data processor to evaluate JavaScript

Hi all,

I'm planning to implement a JavaScript eval data processor. As the name suggests, users will be able to write some JavaScript code which takes in an event (as a map), do some processing on the event, and return a new map which then gets converted to an sp-event. 

example js:
function process(event) {
    // do processing here.
    // return processed event.
    return {id: http://event.id, tempInCelsius: (event.tempInKelvin - 273.15)};
};

Will this be a good addition to pipeline-elements? What do you think?

Regards,
Grainier.