You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by PieterJanVanAeken <gi...@git.apache.org> on 2015/08/18 17:01:35 UTC

[GitHub] flink pull request: Add Scala version of GraphAlgorithm

GitHub user PieterJanVanAeken opened a pull request:

    https://github.com/apache/flink/pull/1033

    Add Scala version of GraphAlgorithm

    The Scala Gelly API already has a wrapper for running java-based GraphAlgorithms. It is needed for running pre-built algorithms such as the PageRankAlgorithm. This small addition maintains compatibility with these java-based GraphAlgorithms while giving the user an additional possibility of developing their own GraphAlgorithm that makes use of the syntactic sugar methods that the new Scala Gelly API offers.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/PieterJanVanAeken/flink scala-gelly-algorithm

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/flink/pull/1033.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #1033
    
----
commit 0a5c46314ba2cf1e4bb8d7703b17c4cde1fbca51
Author: Pieter-Jan Van Aeken <pi...@euranova.eu>
Date:   2015-08-18T14:48:44Z

    Add Scala version of Graph Algorithm that allows for syntactic sugar inside the algorithm.

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: Add Scala version of GraphAlgorithm

Posted by fhueske <gi...@git.apache.org>.
Github user fhueske commented on the pull request:

    https://github.com/apache/flink/pull/1033#issuecomment-139245906
  
    Hi @PieterJanVanAeken, 
    I think it is fine to just update the test file that your PR touches and port this one to the `collect()` approach. Then we don't need to wait for the other tests to be fixed and get this PR in. Have a look at the `ReduceITCase` in `flink-tests` which uses the `collect()` pattern.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: Add Scala version of GraphAlgorithm

Posted by PieterJanVanAeken <gi...@git.apache.org>.
Github user PieterJanVanAeken commented on the pull request:

    https://github.com/apache/flink/pull/1033#issuecomment-133328052
  
    @tillrohrmann I rebased on the latest master, but the tests are based on older Java Gelly tests. They all need to be reworked then, since they all use the same file-storing approach. But indeed I have noticed that the Travis build fails seemingly at random.
    
    I personally only use the Scala API's so I am not really bothered with Java compatibility, but I can tell the community is. I can have the Scala graph algorithm extend the Java one if it helps. But that doesn't solve the type issues and I would hate having to build Scala algorithms with Java primitives just so that they will work on the Java API. It just seems unnatural. And I'm guessing that more problems will arise. Breeze vectors for instance aren't well-known for being Java friendly either.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: Add Scala version of GraphAlgorithm

Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:

    https://github.com/apache/flink/pull/1033#discussion_r37513671
  
    --- Diff: flink-staging/flink-gelly-scala/src/main/scala/org/apache/flink/graph/scala/GraphAlgorithm.scala ---
    @@ -0,0 +1,30 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one
    + * or more contributor license agreements.  See the NOTICE file
    + * distributed with this work for additional information
    + * regarding copyright ownership.  The ASF licenses this file
    + * to you under the Apache License, Version 2.0 (the
    + * "License"); you may not use this file except in compliance
    + * with the License.  You may obtain a copy of the License at
    + *
    + *     http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.flink.graph.scala
    +
    +/**
    --- End diff --
    
    ScalaDocs is missing here.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: Add Scala version of GraphAlgorithm

Posted by chiwanpark <gi...@git.apache.org>.
Github user chiwanpark commented on a diff in the pull request:

    https://github.com/apache/flink/pull/1033#discussion_r37376484
  
    --- Diff: flink-staging/flink-gelly-scala/src/main/scala/org/apache/flink/graph/scala/GraphAlgorithm.scala ---
    @@ -0,0 +1,30 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one
    + * or more contributor license agreements.  See the NOTICE file
    + * distributed with this work for additional information
    + * regarding copyright ownership.  The ASF licenses this file
    + * to you under the Apache License, Version 2.0 (the
    + * "License"); you may not use this file except in compliance
    + * with the License.  You may obtain a copy of the License at
    + *
    + *     http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.flink.graph.scala
    +
    +/**
    + * @tparam K key type
    + * @tparam VV vertex value type
    + * @tparam EV edge value type
    + */
    +abstract class GraphAlgorithm[K, VV, EV] {
    +
    +  def run(graph: Graph[K, VV, EV]): Graph[K,VV,EV]
    --- End diff --
    
    Please add space after comma :)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: Add Scala version of GraphAlgorithm

Posted by chiwanpark <gi...@git.apache.org>.
Github user chiwanpark commented on the pull request:

    https://github.com/apache/flink/pull/1033#issuecomment-132429945
  
    Looks good. There are some code style issues. Please modify them.
    
    BTW, Is it possible to run graph algorithm implemented with Scala in Java? If yes, it would be better add a test case about that.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: Add Scala version of GraphAlgorithm

Posted by chiwanpark <gi...@git.apache.org>.
Github user chiwanpark commented on the pull request:

    https://github.com/apache/flink/pull/1033#issuecomment-135360987
  
    I'm inclined to the @tillrohrmann 's comment. I think that it is okay to call GraphAlgorithm based on Scala API in Java API and the user has the responsibility to verify type.
    
    Since cf88407, we have to add return type into generic parameter.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: Add Scala version of GraphAlgorithm

Posted by PieterJanVanAeken <gi...@git.apache.org>.
Github user PieterJanVanAeken commented on the pull request:

    https://github.com/apache/flink/pull/1033#issuecomment-132249704
  
    By the way, let me know if I need to create a JIRA for this first but it seemed overkill for such a trivial change.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: Add Scala version of GraphAlgorithm

Posted by chiwanpark <gi...@git.apache.org>.
Github user chiwanpark commented on a diff in the pull request:

    https://github.com/apache/flink/pull/1033#discussion_r37376469
  
    --- Diff: flink-staging/flink-gelly-scala/src/main/scala/org/apache/flink/graph/scala/Graph.scala ---
    @@ -649,10 +650,13 @@ TypeInformation : ClassTag](jgraph: jg.Graph[K, VV, EV]) {
           jtuple.f1))
       }
     
    -  def run(algorithm: GraphAlgorithm[K, VV, EV]) = {
    +  def run(algorithm: JGraphAlgorithm[K, VV, EV]): Graph[K,VV,EV] = {
    --- End diff --
    
    Please add space after comma :)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #1033: Add Scala version of GraphAlgorithm

Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:

    https://github.com/apache/flink/pull/1033


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: Add Scala version of GraphAlgorithm

Posted by PieterJanVanAeken <gi...@git.apache.org>.
Github user PieterJanVanAeken commented on the pull request:

    https://github.com/apache/flink/pull/1033#issuecomment-132241973
  
    Please wait a sec with this one. Something is off with my local changes vs my repo changes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: Add Scala version of GraphAlgorithm

Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:

    https://github.com/apache/flink/pull/1033#discussion_r37514284
  
    --- Diff: flink-staging/flink-gelly-scala/src/test/scala/org/apache/flink/graph/scala/test/operations/MapEdgesITCase.scala ---
    @@ -92,6 +92,30 @@ MultipleProgramsTestBase(mode) {
           "5,1,52\n"
       }
     
    +  @Test
    +  @throws(classOf[Exception])
    +  def testWithSameValueAlgorithm {
    +    val env: ExecutionEnvironment = ExecutionEnvironment.getExecutionEnvironment
    +    val graph: Graph[Long, Long, Long] = Graph.fromDataSet(TestGraphUtils
    +      .getLongLongVertexData(env), TestGraphUtils.getLongLongEdgeData(env), env)
    +
    +   class SameValueAlgorithm extends GraphAlgorithm[Long, Long, Long] {
    +     override def run(graph: Graph[Long, Long, Long]): Graph[Long, Long, Long] = {
    +       graph.mapEdges(edge => edge.getValue + 1)
    +     }
    +   }
    +
    +    graph.run(new SameValueAlgorithm).getEdgesAsTuple3().writeAsCsv(resultPath)
    --- End diff --
    
    If I'm not mistaken, then writing to files while running on Travis can cause sometimes test failures. Therefore, we've started using the `collect` method and then to compare the results in your case with `compareResultAsTuples`. That way, we only work on data kept in memory. In the current master version of the `MapEdgesITCase` this is already the case. Maybe you haven't rebased on the latest master?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: Add Scala version of GraphAlgorithm

Posted by chiwanpark <gi...@git.apache.org>.
Github user chiwanpark commented on a diff in the pull request:

    https://github.com/apache/flink/pull/1033#discussion_r37376477
  
    --- Diff: flink-staging/flink-gelly-scala/src/main/scala/org/apache/flink/graph/scala/Graph.scala ---
    @@ -649,10 +650,13 @@ TypeInformation : ClassTag](jgraph: jg.Graph[K, VV, EV]) {
           jtuple.f1))
       }
     
    -  def run(algorithm: GraphAlgorithm[K, VV, EV]) = {
    +  def run(algorithm: JGraphAlgorithm[K, VV, EV]): Graph[K,VV,EV] = {
         wrapGraph(jgraph.run(algorithm))
       }
     
    +  def run(algorithm: GraphAlgorithm[K,VV,EV]): Graph[K,VV,EV] = {
    --- End diff --
    
    Please add space after comma.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: Add Scala version of GraphAlgorithm

Posted by PieterJanVanAeken <gi...@git.apache.org>.
Github user PieterJanVanAeken commented on the pull request:

    https://github.com/apache/flink/pull/1033#issuecomment-132495162
  
    Fixed the style issues. 
    
    At this point in time, it is not possible to run Scala-based graph algorithms from inside the Java API. Do you think this would be desirable behaviour? Using "Scala stuff" in the Java API seems wrong. But we could probably get it to work by having scala.GraphAlgorithm extend java.GraphAlgorithm where the java run method simply wraps the scala layer around the graph before applying the algorithm and then unwraps it afterwards.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink issue #1033: Add Scala version of GraphAlgorithm

Posted by greghogan <gi...@git.apache.org>.
Github user greghogan commented on the issue:

    https://github.com/apache/flink/pull/1033
  
    Is this PR's functionality still requested or can it be closed?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: Add Scala version of GraphAlgorithm

Posted by PieterJanVanAeken <gi...@git.apache.org>.
Github user PieterJanVanAeken commented on the pull request:

    https://github.com/apache/flink/pull/1033#issuecomment-132502725
  
    **EDIT:** I tried the approach above, and technically it works. But when I create a GraphAlgorithm[Long,Long,Long] in Scala, it does not work on a Java Graph[Long,Long,Long] because a Scala Long is not the same as a Java Long so it is not at all user friendly. Personally, I feel ease of use dictates we keep the java and scala graph algorithms separately.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: Add Scala version of GraphAlgorithm

Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on the pull request:

    https://github.com/apache/flink/pull/1033#issuecomment-133332575
  
    I guess that all the tests should then be reworked as you've stated before merging this PR.
    
    I think it's fair enough to say that the developer of the algorithm can decide on the input types of the algorithm and the user has the responsibility to convert the input data in the right format.
    
    Well as far as I know, you can use Breeze from Java. The only problem is that you have to assemble the implicit operations yourself, because Java is lacking this feature. But once you've done this, it should actually work.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: Add Scala version of GraphAlgorithm

Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on the pull request:

    https://github.com/apache/flink/pull/1033#issuecomment-132966338
  
    I'm not a Gelly expert, but conceptually I don't think that it is wrong to call an algorithm which was developed with the Scala API from within the Java API. Otherwise this would mean that every algorithm which was implemented using the Scala API would have to be reimplemented for the Java API as well. And then it is the question why developing the algorithm with the Scala API in the first place, if you can run algorithms written with the Java API from within the Scala API? But maybe I'm mistaken here.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---