You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@rya.apache.org by Ejgood21 <gi...@git.apache.org> on 2016/08/18 15:22:59 UTC

[GitHub] incubator-rya pull request #70: Feature/spark

GitHub user Ejgood21 opened a pull request:

    https://github.com/apache/incubator-rya/pull/70

    Feature/spark

    Spark GraphX Support - WIP

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/Ejgood21/incubator-rya feature/spark

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/incubator-rya/pull/70.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #70
    
----
commit 18552132e8bfecec33b57aae302e25d93c28e7e3
Author: Jesse Hatfield <je...@parsons.com>
Date:   2016-06-30T19:50:08Z

    Consolidated MapReduce API and applications into toplevel project.
    
    Changes include:
    - Made RdfFileInputFormat threaded, allowing it to handle all RDF input
    - Added entity-centric indexing to RyaOutputFormat
    - Added methods for using Rya input and output formats to AbstractAccumuloMRTool (renamed setupInputFormat to setupAccumuloInput since there are now multiple input options)
    - Removed *NullIndexer classes, which were only used in RyaOutputFormat
    - Removed StatementWritable, changed to RyaStatementWritable where applicable (for consistency)
    - Fixed hashCode/compareTo/equals methods in RyaStatementWritable, RyaType, and RyaURI
    - Minor renaming and repackaging (standalone tools go in "tools", "fileinput" and "utils" removed)
    - Minor tweaks to AccumuloHDFSFileInputFormat, which seemed to be broken with Accumulo 1.6
    - Documented code and added a MapReduce page to the manual
    - Added "examples" package with one example
    - Removed RyaStatementMapper and RyaStatementReducer: these simply insert records into Rya.
      This functionality can be achieved with RyaOutputFormat and the default mapper/reducer,
      so these two classes seem redundant.
    - Removed BulkNtripsInputTool, BulkNtripsInputToolIndexing, RdfFileInputByLineTool,
      and RyaBatchWriterInputTool: Fixes to RdfFileInputFormat now allow RdfFileInputTool
      to now handle all the file input use cases (configurable secondary indexers, handles
      any format, scales), rendering the other file import tools redundant. (Previously, all
      five tools had largely overlapping but subtly different behavior.)

commit dd727af8d6d97f273ca168637a15f02f5eb1a605
Author: Evan Good <ev...@parsons.com>
Date:   2016-08-18T15:15:45Z

    initial spark support

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-rya issue #70: [WIP] RYA-109 Add Spark Support

Posted by amihalik <gi...@git.apache.org>.
Github user amihalik commented on the issue:

    https://github.com/apache/incubator-rya/pull/70
  
    Something is screwy with Jesse's commit that you pulled in.  The commit you have has hash 	1855213 and the one in the apache/develop branch has 01489ef.  This needs to be fixed before we can pull in your PR, and it's probably easier to fix now rather than later.  Let me know if you need any assistance with this and I can walk you through it.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-rya pull request #70: [WIP] RYA-109 Add Spark Support

Posted by Ejgood21 <gi...@git.apache.org>.
Github user Ejgood21 closed the pull request at:

    https://github.com/apache/incubator-rya/pull/70


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---