You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@wayang.apache.org by MatthewJ Sanyoto <ma...@gmail.com> on 2022/07/13 16:12:22 UTC
WAYANG-28
Hi,
If I remember correctly there was an abstract HackIT Experiment Class where
you could test / check if the wayang debugger worked as intended (tagging
and sniffing) using HackItSparkRdd, if I am not mistaken. Was this class
removed from this branch? Since Apache Wayang is not an execution engine, a
suggestion for how to test HackIT Plugin would be very helpful and
appreciated.
Best regards,
Matthew
Re: WAYANG-28
Posted by MatthewJ Sanyoto <ma...@gmail.com>.
Hi Bertty,
Yes, I have fixed the problem and as you may have guessed the problem was
with the IDE and Scala.
Could I ask you some questions regarding the idea of HackIt?
The Idea is to have a DataQuanta wrapper or HackItDataQuanta that contains
the functionality / operator like map, filter and so on which I have done
and users can easily tag using the API.
So HackIt is mainly for the execution layer, correct me if I am wrong and
not in wayang itself I mean in the meta processing framework. To rephrase
that we don't have need to create an HackIt Abstraction in the Wayang level
and then using Java Plugin and Spark Plugin the HackIt will automatically
translate to the execution platform but directly in Java/Spark level. Is my
understanding correct?
Best regards,
Matthew
On Thu, Aug 4, 2022, 12:01 bertty contreras <be...@gmail.com>
wrote:
> I think the problem is because you are using the scala version, exist one
> Dataquanta Java, because the conversion of the types between the language
> are different.
>
> You may try that, if that does not work, let me know.
>
> Best regards,
> Bertty
>
> El lun, 25 jul 2022 a las 22:20, MatthewJ Sanyoto (<
> mattjsanyoto719@gmail.com>) escribió:
>
> > Hi,
> >
> > Here is the link of my code.
> >
> >
> >
> https://github.com/regaleo605/incubator-wayang/blob/HackItDataQuanta/wayang-plugins/wayang-hackit/wayang-hackit-api/src/main/java/HackItDataQuanta.java
> >
> > I thank you in advance for your time to check my code.
> >
> > Best regards,
> > Matthew
> >
> >
> >
> > On Mon, 25 Jul 2022 at 16:17, bertty contreras <
> berttycontreras@gmail.com>
> > wrote:
> >
> > > could you send the location of your code and the full logs of the issue
> > to
> > > take a look?
> > >
> > > Best regards,
> > > Bertty
> > >
> > > El dom, 24 jul 2022 a las 9:47, MatthewJ Sanyoto (<
> > > mattjsanyoto719@gmail.com>)
> > > escribió:
> > >
> > > > Hi Bertty,
> > > >
> > > > I am currently having a problem with the map Operator using Java for
> > the
> > > > DataQuanta. I checked your Scala Code and tried to implement it in
> Java
> > > to
> > > > experiment.
> > > > I tried to map the DataQuanta<T> to DataQuanta<HackItTuple<Long,T>
> and
> > > vice
> > > > versa, however I keep having problems with the map. I tried both
> using
> > > map
> > > > and mapJava and I am pretty sure I input the parameters correctly.
> > > > However, I keep getting the same error.
> > > >
> > > >
> > >
> >
> 'mapJava(org.apache.wayang.core.function.FunctionDescriptor.SerializableFunction
> > > > ,T>, org.apache.wayang.core.optimizer.costs.LoadProfileEstimator,
> > > > scala.reflect.ClassTag )' in
> > > 'org.apache.wayang.api.dataquanta.DataQuanta'
> > > > cannot be applied to
> > > >
> > '(org.apache.wayang.core.function.FunctionDescriptor.SerializableFunction
> > > > ,T>)' or '()'
> > > >
> > > > I assume the problem is because I directly call the map function
> using
> > > the
> > > > DataQuanta Class and not using the DataQuantaBuilder or in a sense, I
> > can
> > > > not just map directly unlinke in Scala or something else.
> > > > I would be grateful if you could maybe clear some confusions for me.
> > > >
> > > > Best regards,
> > > > Matthew
> > > >
> > > > On Fri, 22 Jul 2022 at 07:09, MatthewJ Sanyoto <
> > > mattjsanyoto719@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi Bertty,
> > > > >
> > > > > Sorry for the late reply.
> > > > >
> > > > > I understand now. Thank you for your advice.
> > > > >
> > > > > Best regards,
> > > > > Matthew
> > > > >
> > > > > On Sat, 16 Jul 2022 at 19:46, bertty contreras <
> > > > berttycontreras@gmail.com>
> > > > > wrote:
> > > > >
> > > > >> Hi Matthew,
> > > > >>
> > > > >> Sorry for the late replay.
> > > > >>
> > > > >> But it just spark; the idea is to use the DataQuanta for have the
> > > > >> functionality of wayang, because that is the API of Wayang
> > > > >>
> > > > >> Best regards,
> > > > >> Bertty
> > > > >>
> > > > >> El jue, 14 jul 2022 a las 12:23, MatthewJ Sanyoto (<
> > > > >> mattjsanyoto719@gmail.com>) escribió:
> > > > >>
> > > > >> > Hi Bertty,
> > > > >> >
> > > > >> > Thanks for the helpful advice and links. For curiosity's sake,
> is
> > > the
> > > > >> > HackItExperiment (the HackitSparkRDD) is for the spark side of
> > > HackIT
> > > > >> > Plugin? Or is it from the wayang side but executed using Spark?
> > > > >> >
> > > > >> > Best regards,
> > > > >> > Matthew
> > > > >> >
> > > > >> > On Thu, 14 Jul 2022 at 10:40, Bertty Contreras <
> > bertty@databloom.ai
> > > >
> > > > >> > wrote:
> > > > >> >
> > > > >> > > Hi Matthew,
> > > > >> > >
> > > > >> > > In the case of wayang, you need to create a class like Hackit
> > [2]
> > > (
> > > > >> this
> > > > >> > is
> > > > >> > > similar to hackitSparkRDD), and you can test your changes
> using
> > > the
> > > > >> test
> > > > >> > > [1].
> > > > >> > >
> > > > >> > > Does that respond to your question?
> > > > >> > >
> > > > >> > > Best regards,
> > > > >> > > Bertty
> > > > >> > >
> > > > >> > > [1]
> > > > >> > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> https://github.com/apache/incubator-wayang/blob/WAYANG-28/wayang-plugins/wayang-hackit/wayang-hackit-api/src/test/scala/org/apache/wayang/plugin/hackit/api/ApiExtensionTest.scala
> > > > >> > > [2]
> > > > >> > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> https://github.com/apache/incubator-wayang/blob/WAYANG-28/wayang-plugins/wayang-hackit/wayang-hackit-api/src/main/scala/org/apache/wayang/plugin/hackit/api/Hackit.scala
> > > > >> > >
> > > > >> > > On Wed, Jul 13, 2022 at 6:12 PM MatthewJ Sanyoto <
> > > > >> > > mattjsanyoto719@gmail.com>
> > > > >> > > wrote:
> > > > >> > >
> > > > >> > > > Hi,
> > > > >> > > >
> > > > >> > > > If I remember correctly there was an abstract HackIT
> > Experiment
> > > > >> Class
> > > > >> > > where
> > > > >> > > > you could test / check if the wayang debugger worked as
> > intended
> > > > >> > (tagging
> > > > >> > > > and sniffing) using HackItSparkRdd, if I am not mistaken.
> Was
> > > this
> > > > >> > class
> > > > >> > > > removed from this branch? Since Apache Wayang is not an
> > > execution
> > > > >> > > engine, a
> > > > >> > > > suggestion for how to test HackIT Plugin would be very
> helpful
> > > and
> > > > >> > > > appreciated.
> > > > >> > > >
> > > > >> > > > Best regards,
> > > > >> > > > Matthew
> > > > >> > > >
> > > > >> > >
> > > > >> > >
> > > > >> > > --
> > > > >> > > Best,
> > > > >> > > Bertty
> > > > >> > >
> > > > >> >
> > > > >>
> > > > >
> > > >
> > >
> >
>
Re: WAYANG-28
Posted by bertty contreras <be...@gmail.com>.
I think the problem is because you are using the scala version, exist one
Dataquanta Java, because the conversion of the types between the language
are different.
You may try that, if that does not work, let me know.
Best regards,
Bertty
El lun, 25 jul 2022 a las 22:20, MatthewJ Sanyoto (<
mattjsanyoto719@gmail.com>) escribió:
> Hi,
>
> Here is the link of my code.
>
>
> https://github.com/regaleo605/incubator-wayang/blob/HackItDataQuanta/wayang-plugins/wayang-hackit/wayang-hackit-api/src/main/java/HackItDataQuanta.java
>
> I thank you in advance for your time to check my code.
>
> Best regards,
> Matthew
>
>
>
> On Mon, 25 Jul 2022 at 16:17, bertty contreras <be...@gmail.com>
> wrote:
>
> > could you send the location of your code and the full logs of the issue
> to
> > take a look?
> >
> > Best regards,
> > Bertty
> >
> > El dom, 24 jul 2022 a las 9:47, MatthewJ Sanyoto (<
> > mattjsanyoto719@gmail.com>)
> > escribió:
> >
> > > Hi Bertty,
> > >
> > > I am currently having a problem with the map Operator using Java for
> the
> > > DataQuanta. I checked your Scala Code and tried to implement it in Java
> > to
> > > experiment.
> > > I tried to map the DataQuanta<T> to DataQuanta<HackItTuple<Long,T> and
> > vice
> > > versa, however I keep having problems with the map. I tried both using
> > map
> > > and mapJava and I am pretty sure I input the parameters correctly.
> > > However, I keep getting the same error.
> > >
> > >
> >
> 'mapJava(org.apache.wayang.core.function.FunctionDescriptor.SerializableFunction
> > > ,T>, org.apache.wayang.core.optimizer.costs.LoadProfileEstimator,
> > > scala.reflect.ClassTag )' in
> > 'org.apache.wayang.api.dataquanta.DataQuanta'
> > > cannot be applied to
> > >
> '(org.apache.wayang.core.function.FunctionDescriptor.SerializableFunction
> > > ,T>)' or '()'
> > >
> > > I assume the problem is because I directly call the map function using
> > the
> > > DataQuanta Class and not using the DataQuantaBuilder or in a sense, I
> can
> > > not just map directly unlinke in Scala or something else.
> > > I would be grateful if you could maybe clear some confusions for me.
> > >
> > > Best regards,
> > > Matthew
> > >
> > > On Fri, 22 Jul 2022 at 07:09, MatthewJ Sanyoto <
> > mattjsanyoto719@gmail.com>
> > > wrote:
> > >
> > > > Hi Bertty,
> > > >
> > > > Sorry for the late reply.
> > > >
> > > > I understand now. Thank you for your advice.
> > > >
> > > > Best regards,
> > > > Matthew
> > > >
> > > > On Sat, 16 Jul 2022 at 19:46, bertty contreras <
> > > berttycontreras@gmail.com>
> > > > wrote:
> > > >
> > > >> Hi Matthew,
> > > >>
> > > >> Sorry for the late replay.
> > > >>
> > > >> But it just spark; the idea is to use the DataQuanta for have the
> > > >> functionality of wayang, because that is the API of Wayang
> > > >>
> > > >> Best regards,
> > > >> Bertty
> > > >>
> > > >> El jue, 14 jul 2022 a las 12:23, MatthewJ Sanyoto (<
> > > >> mattjsanyoto719@gmail.com>) escribió:
> > > >>
> > > >> > Hi Bertty,
> > > >> >
> > > >> > Thanks for the helpful advice and links. For curiosity's sake, is
> > the
> > > >> > HackItExperiment (the HackitSparkRDD) is for the spark side of
> > HackIT
> > > >> > Plugin? Or is it from the wayang side but executed using Spark?
> > > >> >
> > > >> > Best regards,
> > > >> > Matthew
> > > >> >
> > > >> > On Thu, 14 Jul 2022 at 10:40, Bertty Contreras <
> bertty@databloom.ai
> > >
> > > >> > wrote:
> > > >> >
> > > >> > > Hi Matthew,
> > > >> > >
> > > >> > > In the case of wayang, you need to create a class like Hackit
> [2]
> > (
> > > >> this
> > > >> > is
> > > >> > > similar to hackitSparkRDD), and you can test your changes using
> > the
> > > >> test
> > > >> > > [1].
> > > >> > >
> > > >> > > Does that respond to your question?
> > > >> > >
> > > >> > > Best regards,
> > > >> > > Bertty
> > > >> > >
> > > >> > > [1]
> > > >> > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> https://github.com/apache/incubator-wayang/blob/WAYANG-28/wayang-plugins/wayang-hackit/wayang-hackit-api/src/test/scala/org/apache/wayang/plugin/hackit/api/ApiExtensionTest.scala
> > > >> > > [2]
> > > >> > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> https://github.com/apache/incubator-wayang/blob/WAYANG-28/wayang-plugins/wayang-hackit/wayang-hackit-api/src/main/scala/org/apache/wayang/plugin/hackit/api/Hackit.scala
> > > >> > >
> > > >> > > On Wed, Jul 13, 2022 at 6:12 PM MatthewJ Sanyoto <
> > > >> > > mattjsanyoto719@gmail.com>
> > > >> > > wrote:
> > > >> > >
> > > >> > > > Hi,
> > > >> > > >
> > > >> > > > If I remember correctly there was an abstract HackIT
> Experiment
> > > >> Class
> > > >> > > where
> > > >> > > > you could test / check if the wayang debugger worked as
> intended
> > > >> > (tagging
> > > >> > > > and sniffing) using HackItSparkRdd, if I am not mistaken. Was
> > this
> > > >> > class
> > > >> > > > removed from this branch? Since Apache Wayang is not an
> > execution
> > > >> > > engine, a
> > > >> > > > suggestion for how to test HackIT Plugin would be very helpful
> > and
> > > >> > > > appreciated.
> > > >> > > >
> > > >> > > > Best regards,
> > > >> > > > Matthew
> > > >> > > >
> > > >> > >
> > > >> > >
> > > >> > > --
> > > >> > > Best,
> > > >> > > Bertty
> > > >> > >
> > > >> >
> > > >>
> > > >
> > >
> >
>
Re: WAYANG-28
Posted by MatthewJ Sanyoto <ma...@gmail.com>.
Hi,
Here is the link of my code.
https://github.com/regaleo605/incubator-wayang/blob/HackItDataQuanta/wayang-plugins/wayang-hackit/wayang-hackit-api/src/main/java/HackItDataQuanta.java
I thank you in advance for your time to check my code.
Best regards,
Matthew
On Mon, 25 Jul 2022 at 16:17, bertty contreras <be...@gmail.com>
wrote:
> could you send the location of your code and the full logs of the issue to
> take a look?
>
> Best regards,
> Bertty
>
> El dom, 24 jul 2022 a las 9:47, MatthewJ Sanyoto (<
> mattjsanyoto719@gmail.com>)
> escribió:
>
> > Hi Bertty,
> >
> > I am currently having a problem with the map Operator using Java for the
> > DataQuanta. I checked your Scala Code and tried to implement it in Java
> to
> > experiment.
> > I tried to map the DataQuanta<T> to DataQuanta<HackItTuple<Long,T> and
> vice
> > versa, however I keep having problems with the map. I tried both using
> map
> > and mapJava and I am pretty sure I input the parameters correctly.
> > However, I keep getting the same error.
> >
> >
> 'mapJava(org.apache.wayang.core.function.FunctionDescriptor.SerializableFunction
> > ,T>, org.apache.wayang.core.optimizer.costs.LoadProfileEstimator,
> > scala.reflect.ClassTag )' in
> 'org.apache.wayang.api.dataquanta.DataQuanta'
> > cannot be applied to
> > '(org.apache.wayang.core.function.FunctionDescriptor.SerializableFunction
> > ,T>)' or '()'
> >
> > I assume the problem is because I directly call the map function using
> the
> > DataQuanta Class and not using the DataQuantaBuilder or in a sense, I can
> > not just map directly unlinke in Scala or something else.
> > I would be grateful if you could maybe clear some confusions for me.
> >
> > Best regards,
> > Matthew
> >
> > On Fri, 22 Jul 2022 at 07:09, MatthewJ Sanyoto <
> mattjsanyoto719@gmail.com>
> > wrote:
> >
> > > Hi Bertty,
> > >
> > > Sorry for the late reply.
> > >
> > > I understand now. Thank you for your advice.
> > >
> > > Best regards,
> > > Matthew
> > >
> > > On Sat, 16 Jul 2022 at 19:46, bertty contreras <
> > berttycontreras@gmail.com>
> > > wrote:
> > >
> > >> Hi Matthew,
> > >>
> > >> Sorry for the late replay.
> > >>
> > >> But it just spark; the idea is to use the DataQuanta for have the
> > >> functionality of wayang, because that is the API of Wayang
> > >>
> > >> Best regards,
> > >> Bertty
> > >>
> > >> El jue, 14 jul 2022 a las 12:23, MatthewJ Sanyoto (<
> > >> mattjsanyoto719@gmail.com>) escribió:
> > >>
> > >> > Hi Bertty,
> > >> >
> > >> > Thanks for the helpful advice and links. For curiosity's sake, is
> the
> > >> > HackItExperiment (the HackitSparkRDD) is for the spark side of
> HackIT
> > >> > Plugin? Or is it from the wayang side but executed using Spark?
> > >> >
> > >> > Best regards,
> > >> > Matthew
> > >> >
> > >> > On Thu, 14 Jul 2022 at 10:40, Bertty Contreras <bertty@databloom.ai
> >
> > >> > wrote:
> > >> >
> > >> > > Hi Matthew,
> > >> > >
> > >> > > In the case of wayang, you need to create a class like Hackit [2]
> (
> > >> this
> > >> > is
> > >> > > similar to hackitSparkRDD), and you can test your changes using
> the
> > >> test
> > >> > > [1].
> > >> > >
> > >> > > Does that respond to your question?
> > >> > >
> > >> > > Best regards,
> > >> > > Bertty
> > >> > >
> > >> > > [1]
> > >> > >
> > >> > >
> > >> >
> > >>
> >
> https://github.com/apache/incubator-wayang/blob/WAYANG-28/wayang-plugins/wayang-hackit/wayang-hackit-api/src/test/scala/org/apache/wayang/plugin/hackit/api/ApiExtensionTest.scala
> > >> > > [2]
> > >> > >
> > >> > >
> > >> >
> > >>
> >
> https://github.com/apache/incubator-wayang/blob/WAYANG-28/wayang-plugins/wayang-hackit/wayang-hackit-api/src/main/scala/org/apache/wayang/plugin/hackit/api/Hackit.scala
> > >> > >
> > >> > > On Wed, Jul 13, 2022 at 6:12 PM MatthewJ Sanyoto <
> > >> > > mattjsanyoto719@gmail.com>
> > >> > > wrote:
> > >> > >
> > >> > > > Hi,
> > >> > > >
> > >> > > > If I remember correctly there was an abstract HackIT Experiment
> > >> Class
> > >> > > where
> > >> > > > you could test / check if the wayang debugger worked as intended
> > >> > (tagging
> > >> > > > and sniffing) using HackItSparkRdd, if I am not mistaken. Was
> this
> > >> > class
> > >> > > > removed from this branch? Since Apache Wayang is not an
> execution
> > >> > > engine, a
> > >> > > > suggestion for how to test HackIT Plugin would be very helpful
> and
> > >> > > > appreciated.
> > >> > > >
> > >> > > > Best regards,
> > >> > > > Matthew
> > >> > > >
> > >> > >
> > >> > >
> > >> > > --
> > >> > > Best,
> > >> > > Bertty
> > >> > >
> > >> >
> > >>
> > >
> >
>
Re: WAYANG-28
Posted by bertty contreras <be...@gmail.com>.
could you send the location of your code and the full logs of the issue to
take a look?
Best regards,
Bertty
El dom, 24 jul 2022 a las 9:47, MatthewJ Sanyoto (<ma...@gmail.com>)
escribió:
> Hi Bertty,
>
> I am currently having a problem with the map Operator using Java for the
> DataQuanta. I checked your Scala Code and tried to implement it in Java to
> experiment.
> I tried to map the DataQuanta<T> to DataQuanta<HackItTuple<Long,T> and vice
> versa, however I keep having problems with the map. I tried both using map
> and mapJava and I am pretty sure I input the parameters correctly.
> However, I keep getting the same error.
>
> 'mapJava(org.apache.wayang.core.function.FunctionDescriptor.SerializableFunction
> ,T>, org.apache.wayang.core.optimizer.costs.LoadProfileEstimator,
> scala.reflect.ClassTag )' in 'org.apache.wayang.api.dataquanta.DataQuanta'
> cannot be applied to
> '(org.apache.wayang.core.function.FunctionDescriptor.SerializableFunction
> ,T>)' or '()'
>
> I assume the problem is because I directly call the map function using the
> DataQuanta Class and not using the DataQuantaBuilder or in a sense, I can
> not just map directly unlinke in Scala or something else.
> I would be grateful if you could maybe clear some confusions for me.
>
> Best regards,
> Matthew
>
> On Fri, 22 Jul 2022 at 07:09, MatthewJ Sanyoto <ma...@gmail.com>
> wrote:
>
> > Hi Bertty,
> >
> > Sorry for the late reply.
> >
> > I understand now. Thank you for your advice.
> >
> > Best regards,
> > Matthew
> >
> > On Sat, 16 Jul 2022 at 19:46, bertty contreras <
> berttycontreras@gmail.com>
> > wrote:
> >
> >> Hi Matthew,
> >>
> >> Sorry for the late replay.
> >>
> >> But it just spark; the idea is to use the DataQuanta for have the
> >> functionality of wayang, because that is the API of Wayang
> >>
> >> Best regards,
> >> Bertty
> >>
> >> El jue, 14 jul 2022 a las 12:23, MatthewJ Sanyoto (<
> >> mattjsanyoto719@gmail.com>) escribió:
> >>
> >> > Hi Bertty,
> >> >
> >> > Thanks for the helpful advice and links. For curiosity's sake, is the
> >> > HackItExperiment (the HackitSparkRDD) is for the spark side of HackIT
> >> > Plugin? Or is it from the wayang side but executed using Spark?
> >> >
> >> > Best regards,
> >> > Matthew
> >> >
> >> > On Thu, 14 Jul 2022 at 10:40, Bertty Contreras <be...@databloom.ai>
> >> > wrote:
> >> >
> >> > > Hi Matthew,
> >> > >
> >> > > In the case of wayang, you need to create a class like Hackit [2] (
> >> this
> >> > is
> >> > > similar to hackitSparkRDD), and you can test your changes using the
> >> test
> >> > > [1].
> >> > >
> >> > > Does that respond to your question?
> >> > >
> >> > > Best regards,
> >> > > Bertty
> >> > >
> >> > > [1]
> >> > >
> >> > >
> >> >
> >>
> https://github.com/apache/incubator-wayang/blob/WAYANG-28/wayang-plugins/wayang-hackit/wayang-hackit-api/src/test/scala/org/apache/wayang/plugin/hackit/api/ApiExtensionTest.scala
> >> > > [2]
> >> > >
> >> > >
> >> >
> >>
> https://github.com/apache/incubator-wayang/blob/WAYANG-28/wayang-plugins/wayang-hackit/wayang-hackit-api/src/main/scala/org/apache/wayang/plugin/hackit/api/Hackit.scala
> >> > >
> >> > > On Wed, Jul 13, 2022 at 6:12 PM MatthewJ Sanyoto <
> >> > > mattjsanyoto719@gmail.com>
> >> > > wrote:
> >> > >
> >> > > > Hi,
> >> > > >
> >> > > > If I remember correctly there was an abstract HackIT Experiment
> >> Class
> >> > > where
> >> > > > you could test / check if the wayang debugger worked as intended
> >> > (tagging
> >> > > > and sniffing) using HackItSparkRdd, if I am not mistaken. Was this
> >> > class
> >> > > > removed from this branch? Since Apache Wayang is not an execution
> >> > > engine, a
> >> > > > suggestion for how to test HackIT Plugin would be very helpful and
> >> > > > appreciated.
> >> > > >
> >> > > > Best regards,
> >> > > > Matthew
> >> > > >
> >> > >
> >> > >
> >> > > --
> >> > > Best,
> >> > > Bertty
> >> > >
> >> >
> >>
> >
>
Re: WAYANG-28
Posted by MatthewJ Sanyoto <ma...@gmail.com>.
Hi Bertty,
I am currently having a problem with the map Operator using Java for the
DataQuanta. I checked your Scala Code and tried to implement it in Java to
experiment.
I tried to map the DataQuanta<T> to DataQuanta<HackItTuple<Long,T> and vice
versa, however I keep having problems with the map. I tried both using map
and mapJava and I am pretty sure I input the parameters correctly.
However, I keep getting the same error.
'mapJava(org.apache.wayang.core.function.FunctionDescriptor.SerializableFunction
,T>, org.apache.wayang.core.optimizer.costs.LoadProfileEstimator,
scala.reflect.ClassTag )' in 'org.apache.wayang.api.dataquanta.DataQuanta'
cannot be applied to
'(org.apache.wayang.core.function.FunctionDescriptor.SerializableFunction
,T>)' or '()'
I assume the problem is because I directly call the map function using the
DataQuanta Class and not using the DataQuantaBuilder or in a sense, I can
not just map directly unlinke in Scala or something else.
I would be grateful if you could maybe clear some confusions for me.
Best regards,
Matthew
On Fri, 22 Jul 2022 at 07:09, MatthewJ Sanyoto <ma...@gmail.com>
wrote:
> Hi Bertty,
>
> Sorry for the late reply.
>
> I understand now. Thank you for your advice.
>
> Best regards,
> Matthew
>
> On Sat, 16 Jul 2022 at 19:46, bertty contreras <be...@gmail.com>
> wrote:
>
>> Hi Matthew,
>>
>> Sorry for the late replay.
>>
>> But it just spark; the idea is to use the DataQuanta for have the
>> functionality of wayang, because that is the API of Wayang
>>
>> Best regards,
>> Bertty
>>
>> El jue, 14 jul 2022 a las 12:23, MatthewJ Sanyoto (<
>> mattjsanyoto719@gmail.com>) escribió:
>>
>> > Hi Bertty,
>> >
>> > Thanks for the helpful advice and links. For curiosity's sake, is the
>> > HackItExperiment (the HackitSparkRDD) is for the spark side of HackIT
>> > Plugin? Or is it from the wayang side but executed using Spark?
>> >
>> > Best regards,
>> > Matthew
>> >
>> > On Thu, 14 Jul 2022 at 10:40, Bertty Contreras <be...@databloom.ai>
>> > wrote:
>> >
>> > > Hi Matthew,
>> > >
>> > > In the case of wayang, you need to create a class like Hackit [2] (
>> this
>> > is
>> > > similar to hackitSparkRDD), and you can test your changes using the
>> test
>> > > [1].
>> > >
>> > > Does that respond to your question?
>> > >
>> > > Best regards,
>> > > Bertty
>> > >
>> > > [1]
>> > >
>> > >
>> >
>> https://github.com/apache/incubator-wayang/blob/WAYANG-28/wayang-plugins/wayang-hackit/wayang-hackit-api/src/test/scala/org/apache/wayang/plugin/hackit/api/ApiExtensionTest.scala
>> > > [2]
>> > >
>> > >
>> >
>> https://github.com/apache/incubator-wayang/blob/WAYANG-28/wayang-plugins/wayang-hackit/wayang-hackit-api/src/main/scala/org/apache/wayang/plugin/hackit/api/Hackit.scala
>> > >
>> > > On Wed, Jul 13, 2022 at 6:12 PM MatthewJ Sanyoto <
>> > > mattjsanyoto719@gmail.com>
>> > > wrote:
>> > >
>> > > > Hi,
>> > > >
>> > > > If I remember correctly there was an abstract HackIT Experiment
>> Class
>> > > where
>> > > > you could test / check if the wayang debugger worked as intended
>> > (tagging
>> > > > and sniffing) using HackItSparkRdd, if I am not mistaken. Was this
>> > class
>> > > > removed from this branch? Since Apache Wayang is not an execution
>> > > engine, a
>> > > > suggestion for how to test HackIT Plugin would be very helpful and
>> > > > appreciated.
>> > > >
>> > > > Best regards,
>> > > > Matthew
>> > > >
>> > >
>> > >
>> > > --
>> > > Best,
>> > > Bertty
>> > >
>> >
>>
>
Re: WAYANG-28
Posted by MatthewJ Sanyoto <ma...@gmail.com>.
Hi Bertty,
Sorry for the late reply.
I understand now. Thank you for your advice.
Best regards,
Matthew
On Sat, 16 Jul 2022 at 19:46, bertty contreras <be...@gmail.com>
wrote:
> Hi Matthew,
>
> Sorry for the late replay.
>
> But it just spark; the idea is to use the DataQuanta for have the
> functionality of wayang, because that is the API of Wayang
>
> Best regards,
> Bertty
>
> El jue, 14 jul 2022 a las 12:23, MatthewJ Sanyoto (<
> mattjsanyoto719@gmail.com>) escribió:
>
> > Hi Bertty,
> >
> > Thanks for the helpful advice and links. For curiosity's sake, is the
> > HackItExperiment (the HackitSparkRDD) is for the spark side of HackIT
> > Plugin? Or is it from the wayang side but executed using Spark?
> >
> > Best regards,
> > Matthew
> >
> > On Thu, 14 Jul 2022 at 10:40, Bertty Contreras <be...@databloom.ai>
> > wrote:
> >
> > > Hi Matthew,
> > >
> > > In the case of wayang, you need to create a class like Hackit [2] (
> this
> > is
> > > similar to hackitSparkRDD), and you can test your changes using the
> test
> > > [1].
> > >
> > > Does that respond to your question?
> > >
> > > Best regards,
> > > Bertty
> > >
> > > [1]
> > >
> > >
> >
> https://github.com/apache/incubator-wayang/blob/WAYANG-28/wayang-plugins/wayang-hackit/wayang-hackit-api/src/test/scala/org/apache/wayang/plugin/hackit/api/ApiExtensionTest.scala
> > > [2]
> > >
> > >
> >
> https://github.com/apache/incubator-wayang/blob/WAYANG-28/wayang-plugins/wayang-hackit/wayang-hackit-api/src/main/scala/org/apache/wayang/plugin/hackit/api/Hackit.scala
> > >
> > > On Wed, Jul 13, 2022 at 6:12 PM MatthewJ Sanyoto <
> > > mattjsanyoto719@gmail.com>
> > > wrote:
> > >
> > > > Hi,
> > > >
> > > > If I remember correctly there was an abstract HackIT Experiment Class
> > > where
> > > > you could test / check if the wayang debugger worked as intended
> > (tagging
> > > > and sniffing) using HackItSparkRdd, if I am not mistaken. Was this
> > class
> > > > removed from this branch? Since Apache Wayang is not an execution
> > > engine, a
> > > > suggestion for how to test HackIT Plugin would be very helpful and
> > > > appreciated.
> > > >
> > > > Best regards,
> > > > Matthew
> > > >
> > >
> > >
> > > --
> > > Best,
> > > Bertty
> > >
> >
>
Re: WAYANG-28
Posted by bertty contreras <be...@gmail.com>.
Hi Matthew,
Sorry for the late replay.
But it just spark; the idea is to use the DataQuanta for have the
functionality of wayang, because that is the API of Wayang
Best regards,
Bertty
El jue, 14 jul 2022 a las 12:23, MatthewJ Sanyoto (<
mattjsanyoto719@gmail.com>) escribió:
> Hi Bertty,
>
> Thanks for the helpful advice and links. For curiosity's sake, is the
> HackItExperiment (the HackitSparkRDD) is for the spark side of HackIT
> Plugin? Or is it from the wayang side but executed using Spark?
>
> Best regards,
> Matthew
>
> On Thu, 14 Jul 2022 at 10:40, Bertty Contreras <be...@databloom.ai>
> wrote:
>
> > Hi Matthew,
> >
> > In the case of wayang, you need to create a class like Hackit [2] ( this
> is
> > similar to hackitSparkRDD), and you can test your changes using the test
> > [1].
> >
> > Does that respond to your question?
> >
> > Best regards,
> > Bertty
> >
> > [1]
> >
> >
> https://github.com/apache/incubator-wayang/blob/WAYANG-28/wayang-plugins/wayang-hackit/wayang-hackit-api/src/test/scala/org/apache/wayang/plugin/hackit/api/ApiExtensionTest.scala
> > [2]
> >
> >
> https://github.com/apache/incubator-wayang/blob/WAYANG-28/wayang-plugins/wayang-hackit/wayang-hackit-api/src/main/scala/org/apache/wayang/plugin/hackit/api/Hackit.scala
> >
> > On Wed, Jul 13, 2022 at 6:12 PM MatthewJ Sanyoto <
> > mattjsanyoto719@gmail.com>
> > wrote:
> >
> > > Hi,
> > >
> > > If I remember correctly there was an abstract HackIT Experiment Class
> > where
> > > you could test / check if the wayang debugger worked as intended
> (tagging
> > > and sniffing) using HackItSparkRdd, if I am not mistaken. Was this
> class
> > > removed from this branch? Since Apache Wayang is not an execution
> > engine, a
> > > suggestion for how to test HackIT Plugin would be very helpful and
> > > appreciated.
> > >
> > > Best regards,
> > > Matthew
> > >
> >
> >
> > --
> > Best,
> > Bertty
> >
>
Re: WAYANG-28
Posted by MatthewJ Sanyoto <ma...@gmail.com>.
Hi Bertty,
Thanks for the helpful advice and links. For curiosity's sake, is the
HackItExperiment (the HackitSparkRDD) is for the spark side of HackIT
Plugin? Or is it from the wayang side but executed using Spark?
Best regards,
Matthew
On Thu, 14 Jul 2022 at 10:40, Bertty Contreras <be...@databloom.ai> wrote:
> Hi Matthew,
>
> In the case of wayang, you need to create a class like Hackit [2] ( this is
> similar to hackitSparkRDD), and you can test your changes using the test
> [1].
>
> Does that respond to your question?
>
> Best regards,
> Bertty
>
> [1]
>
> https://github.com/apache/incubator-wayang/blob/WAYANG-28/wayang-plugins/wayang-hackit/wayang-hackit-api/src/test/scala/org/apache/wayang/plugin/hackit/api/ApiExtensionTest.scala
> [2]
>
> https://github.com/apache/incubator-wayang/blob/WAYANG-28/wayang-plugins/wayang-hackit/wayang-hackit-api/src/main/scala/org/apache/wayang/plugin/hackit/api/Hackit.scala
>
> On Wed, Jul 13, 2022 at 6:12 PM MatthewJ Sanyoto <
> mattjsanyoto719@gmail.com>
> wrote:
>
> > Hi,
> >
> > If I remember correctly there was an abstract HackIT Experiment Class
> where
> > you could test / check if the wayang debugger worked as intended (tagging
> > and sniffing) using HackItSparkRdd, if I am not mistaken. Was this class
> > removed from this branch? Since Apache Wayang is not an execution
> engine, a
> > suggestion for how to test HackIT Plugin would be very helpful and
> > appreciated.
> >
> > Best regards,
> > Matthew
> >
>
>
> --
> Best,
> Bertty
>
Re: WAYANG-28
Posted by Bertty Contreras <be...@databloom.ai>.
Hi Matthew,
In the case of wayang, you need to create a class like Hackit [2] ( this is
similar to hackitSparkRDD), and you can test your changes using the test
[1].
Does that respond to your question?
Best regards,
Bertty
[1]
https://github.com/apache/incubator-wayang/blob/WAYANG-28/wayang-plugins/wayang-hackit/wayang-hackit-api/src/test/scala/org/apache/wayang/plugin/hackit/api/ApiExtensionTest.scala
[2]
https://github.com/apache/incubator-wayang/blob/WAYANG-28/wayang-plugins/wayang-hackit/wayang-hackit-api/src/main/scala/org/apache/wayang/plugin/hackit/api/Hackit.scala
On Wed, Jul 13, 2022 at 6:12 PM MatthewJ Sanyoto <ma...@gmail.com>
wrote:
> Hi,
>
> If I remember correctly there was an abstract HackIT Experiment Class where
> you could test / check if the wayang debugger worked as intended (tagging
> and sniffing) using HackItSparkRdd, if I am not mistaken. Was this class
> removed from this branch? Since Apache Wayang is not an execution engine, a
> suggestion for how to test HackIT Plugin would be very helpful and
> appreciated.
>
> Best regards,
> Matthew
>
--
Best,
Bertty