You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@wayang.apache.org by Lefteris Lymperopoulos <le...@gmail.com> on 2022/01/25 15:07:29 UTC
Help with implementing Wayang to access multiple systems
Hello Dev Team,
Excuse me if this mail isn't appropriate for this mail address. Since I
could not find any documentation for Wayang besides that in the official
website I would like to ask you if you could help me with this issue. I
have 3 VMs that connect to each other. In the first VM I want to run Spark,
in the second I want to run Postgres and in the third GraphChi. I also have
Wayang installed in the first VM and I intend to develop my app in Java. Is
it possible to connect Wayang to Spark, Postgres and GraphChi in order to
run my queries? If yes, could you please show me how to do it? Or do these
platforms have to be in the same VM? Any help would be greatly appreciated.
Best regards,
Lefteris Lymperopoulos
Re: Help with implementing Wayang to access multiple systems
Posted by Bertty Contreras <be...@databloom.ai>.
Hi Lefteris,
Sorry for the late reply, In the case of spark you need to provide a
configuration like configuration on the test [1], also you can add the
configuration as the one in the spark configuration [2]
In the case of postgres you can take as an example the configuration of
postgres platform [3] the same is true for graphchi [4].
If they are in different machines you just need to give in the path with
the ip of the VM and the only thing that you need to be sure is that all
the machines in the network are reachable(I mean not firewall rule that can
denied the access), and if you need more help just let me know.
Best regards,
Bertty
[1]
https://github.com/apache/incubator-wayang/blob/main/wayang-benchmark/code/test/resources/rheem.properties
[2]
https://github.com/apache/incubator-wayang/blob/main/wayang-platforms/wayang-spark/code/main/resources/wayang-spark-defaults.properties
[3]
https://github.com/apache/incubator-wayang/blob/main/wayang-platforms/wayang-postgres/src/main/resources/wayang-postgres-defaults.properties
[4]
https://github.com/apache/incubator-wayang/blob/main/wayang-platforms/wayang-graphchi/src/main/resources/wayang-graphchi-defaults.properties
On Thu, Jan 27, 2022 at 4:59 PM Lefteris Lymperopoulos <
lefterislib@gmail.com> wrote:
> Hello and thank you for your reply,
> I want to run benchmarks from TPC such like this
> http://tpc.org/tpcds/default5.asp which consist of simple SQL queries such
> as UPDATE, DELETE and JOIN in order to measure Wayang's performance when
> connected to Spark, Postgres and GraphChi. Could you please provide me with
> more details about how to connect those three systems to Wayang as well as
> Spark, Postgres and GraphChi plugins, especially when they run on different
> VMs? Excuse me if I misunderstood something, I am a student with no prior
> experience in these systems. Thank you for your time.
> Best regards,
> Lefteris
>
> On Tue, Jan 25, 2022 at 10:35 PM Bertty Contreras <be...@databloom.ai>
> wrote:
>
> > Hi Lefteris,
> >
> > It is possible to run the query that you describe using Apache Wayang;
> one
> > example that could help you understand how to connect different platforms
> > in one task is the Query3Hybrid [1]. Nevertheless, if you could explain
> > more about the shape of the query will look, we can give you more hints
> on
> > how you could do it.
> >
> > I mean with a shape something like the following example:
> >
> > The need to run an SQL query in top Postgres, then join the result with
> > data stored in HDFS or S3 and transform it to execute a page rank using
> > Graphchi; after the results are processed, the information needs to be
> > joined with another dataset.
> >
> > Let me know the shape, and we will be able provide more information.
> >
> > Thank you so much for your question; this will help us a lot to know what
> > to write in the documentation in the example section :D
> >
> > Best regards,
> > Bertty
> >
> > [1]
> >
> >
> https://github.com/apache/incubator-wayang/blob/main/wayang-benchmark/code/main/scala/org/apache/wayang/apps/tpch/queries/Query3Hybrid.scala
> >
> > On Tue, Jan 25, 2022 at 4:08 PM Lefteris Lymperopoulos <
> > lefterislib@gmail.com> wrote:
> >
> > > Hello Dev Team,
> > > Excuse me if this mail isn't appropriate for this mail address. Since I
> > > could not find any documentation for Wayang besides that in the
> official
> > > website I would like to ask you if you could help me with this issue. I
> > > have 3 VMs that connect to each other. In the first VM I want to run
> > Spark,
> > > in the second I want to run Postgres and in the third GraphChi. I also
> > have
> > > Wayang installed in the first VM and I intend to develop my app in
> Java.
> > Is
> > > it possible to connect Wayang to Spark, Postgres and GraphChi in order
> to
> > > run my queries? If yes, could you please show me how to do it? Or do
> > these
> > > platforms have to be in the same VM? Any help would be greatly
> > appreciated.
> > > Best regards,
> > > Lefteris Lymperopoulos
> > >
> >
>
Re: Help with implementing Wayang to access multiple systems
Posted by Lefteris Lymperopoulos <le...@gmail.com>.
Hello and thank you for your reply,
I want to run benchmarks from TPC such like this
http://tpc.org/tpcds/default5.asp which consist of simple SQL queries such
as UPDATE, DELETE and JOIN in order to measure Wayang's performance when
connected to Spark, Postgres and GraphChi. Could you please provide me with
more details about how to connect those three systems to Wayang as well as
Spark, Postgres and GraphChi plugins, especially when they run on different
VMs? Excuse me if I misunderstood something, I am a student with no prior
experience in these systems. Thank you for your time.
Best regards,
Lefteris
On Tue, Jan 25, 2022 at 10:35 PM Bertty Contreras <be...@databloom.ai>
wrote:
> Hi Lefteris,
>
> It is possible to run the query that you describe using Apache Wayang; one
> example that could help you understand how to connect different platforms
> in one task is the Query3Hybrid [1]. Nevertheless, if you could explain
> more about the shape of the query will look, we can give you more hints on
> how you could do it.
>
> I mean with a shape something like the following example:
>
> The need to run an SQL query in top Postgres, then join the result with
> data stored in HDFS or S3 and transform it to execute a page rank using
> Graphchi; after the results are processed, the information needs to be
> joined with another dataset.
>
> Let me know the shape, and we will be able provide more information.
>
> Thank you so much for your question; this will help us a lot to know what
> to write in the documentation in the example section :D
>
> Best regards,
> Bertty
>
> [1]
>
> https://github.com/apache/incubator-wayang/blob/main/wayang-benchmark/code/main/scala/org/apache/wayang/apps/tpch/queries/Query3Hybrid.scala
>
> On Tue, Jan 25, 2022 at 4:08 PM Lefteris Lymperopoulos <
> lefterislib@gmail.com> wrote:
>
> > Hello Dev Team,
> > Excuse me if this mail isn't appropriate for this mail address. Since I
> > could not find any documentation for Wayang besides that in the official
> > website I would like to ask you if you could help me with this issue. I
> > have 3 VMs that connect to each other. In the first VM I want to run
> Spark,
> > in the second I want to run Postgres and in the third GraphChi. I also
> have
> > Wayang installed in the first VM and I intend to develop my app in Java.
> Is
> > it possible to connect Wayang to Spark, Postgres and GraphChi in order to
> > run my queries? If yes, could you please show me how to do it? Or do
> these
> > platforms have to be in the same VM? Any help would be greatly
> appreciated.
> > Best regards,
> > Lefteris Lymperopoulos
> >
>
Re: Help with implementing Wayang to access multiple systems
Posted by Bertty Contreras <be...@databloom.ai>.
Hi Lefteris,
It is possible to run the query that you describe using Apache Wayang; one
example that could help you understand how to connect different platforms
in one task is the Query3Hybrid [1]. Nevertheless, if you could explain
more about the shape of the query will look, we can give you more hints on
how you could do it.
I mean with a shape something like the following example:
The need to run an SQL query in top Postgres, then join the result with
data stored in HDFS or S3 and transform it to execute a page rank using
Graphchi; after the results are processed, the information needs to be
joined with another dataset.
Let me know the shape, and we will be able provide more information.
Thank you so much for your question; this will help us a lot to know what
to write in the documentation in the example section :D
Best regards,
Bertty
[1]
https://github.com/apache/incubator-wayang/blob/main/wayang-benchmark/code/main/scala/org/apache/wayang/apps/tpch/queries/Query3Hybrid.scala
On Tue, Jan 25, 2022 at 4:08 PM Lefteris Lymperopoulos <
lefterislib@gmail.com> wrote:
> Hello Dev Team,
> Excuse me if this mail isn't appropriate for this mail address. Since I
> could not find any documentation for Wayang besides that in the official
> website I would like to ask you if you could help me with this issue. I
> have 3 VMs that connect to each other. In the first VM I want to run Spark,
> in the second I want to run Postgres and in the third GraphChi. I also have
> Wayang installed in the first VM and I intend to develop my app in Java. Is
> it possible to connect Wayang to Spark, Postgres and GraphChi in order to
> run my queries? If yes, could you please show me how to do it? Or do these
> platforms have to be in the same VM? Any help would be greatly appreciated.
> Best regards,
> Lefteris Lymperopoulos
>