You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@rya.apache.org by kchilton2 <gi...@git.apache.org> on 2017/03/07 21:25:53 UTC
[GitHub] incubator-rya pull request #144: RYA-255 Provide an example class that demon...
GitHub user kchilton2 opened a pull request:
https://github.com/apache/incubator-rya/pull/144
RYA-255 Provide an example class that demonstrates the Prospector.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/kchilton2/incubator-rya RYA-255
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-rya/pull/144.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #144
----
commit 5b14ce37999a24c177242f59ac9c72744488378a
Author: Kevin Chilton <ke...@parsons.com>
Date: 2017-03-01T22:42:08Z
RYA-253 Converted the rya.prospector class from Groovy to Java. Also added documentation to the project.
commit d86be8fa0e7f389fc17e2a80c2c6adec60c9fd02
Author: Kevin Chilton <ke...@parsons.com>
Date: 2017-03-06T22:11:30Z
RYA-254 Added a factory method to the ProspectorServiceEvalStatsDAO to make it more convenient to construct. This required moving some of the Accumulo specific configuration parameters into the AccumuloRdfConfiguration object from ConfigUtils.
commit 96ee13f495a3263dd05f513e09eebef9b3ce5af9
Author: Kevin Chilton <ke...@parsons.com>
Date: 2017-03-07T21:02:16Z
RYA-255 Provide an example class that demonstrates the Prospector.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-rya pull request #144: RYA-255 Provide an example class that demon...
Posted by ejwhite922 <gi...@git.apache.org>.
Github user ejwhite922 commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/144#discussion_r105437184
--- Diff: extras/indexingExample/src/main/java/ProspectorExample.java ---
@@ -0,0 +1,193 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import java.util.List;
+
+import org.apache.hadoop.util.ToolRunner;
+import org.apache.log4j.ConsoleAppender;
+import org.apache.log4j.Level;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+import org.apache.log4j.PatternLayout;
+import org.apache.rya.accumulo.AccumuloRdfConfiguration;
+import org.apache.rya.accumulo.utils.ConnectorFactory;
+import org.apache.rya.api.persist.RdfEvalStatsDAO.CARDINALITY_OF;
+import org.apache.rya.prospector.mr.Prospector;
+import org.apache.rya.prospector.service.ProspectorServiceEvalStatsDAO;
+import org.apache.rya.sail.config.RyaSailFactory;
+import org.openrdf.model.Statement;
+import org.openrdf.model.URI;
+import org.openrdf.model.ValueFactory;
+import org.openrdf.model.impl.ValueFactoryImpl;
+import org.openrdf.sail.Sail;
+import org.openrdf.sail.SailConnection;
+
+import com.beust.jcommander.internal.Lists;
+
+/**
+ * Demonstrates how you can use the {@link Prospector} to count values that appear within an instance of Rya and
+ * then use the {@link ProspectorServiceEvalStatsDAO} to fetch those counts.
+ */
+public class ProspectorExample {
+ private static final Logger log = Logger.getLogger(RyaClientExample.class);
+
+ private static final ValueFactory VALUE_FACTORY = new ValueFactoryImpl();
+
+ private static final URI ALICE = VALUE_FACTORY.createURI("urn:alice");
+ private static final URI BOB = VALUE_FACTORY.createURI("urn:bob");
+ private static final URI CHARLIE = VALUE_FACTORY.createURI("urn:charlie");
+
+ private static final URI WORKS_AT = VALUE_FACTORY.createURI("urn:worksAt");
+ private static final URI ADMIRES = VALUE_FACTORY.createURI("urn:admires");
+ private static final URI LIVES_WITH = VALUE_FACTORY.createURI("urn:livesWith");
+
+ private static final URI BURGER_JOINT = VALUE_FACTORY.createURI("urn:burgerJoint");
+ private static final URI DONUT_SHOP= VALUE_FACTORY.createURI("urn:donutShop");
+
+ public static void main(final String[] args) throws Exception {
+ setupLogging();
+
+ // Configure Rya to use a mock instance.
+ final AccumuloRdfConfiguration config = new AccumuloRdfConfiguration();
+ config.useMockInstance(true);
+ config.setTablePrefix("rya_");
+ config.setUsername("user");
+ config.setPassword("pass");
+ config.setInstanceName("accumulo");
+
+ // Load some data into Rya.
+ final List<Statement> statements = Lists.newArrayList(
+ VALUE_FACTORY.createStatement(ALICE, WORKS_AT, BURGER_JOINT),
+ VALUE_FACTORY.createStatement(ALICE, ADMIRES, BOB),
+ VALUE_FACTORY.createStatement(BOB, WORKS_AT, DONUT_SHOP),
+ VALUE_FACTORY.createStatement(CHARLIE, WORKS_AT, DONUT_SHOP),
+ VALUE_FACTORY.createStatement(CHARLIE, LIVES_WITH, BOB),
+ VALUE_FACTORY.createStatement(BOB, LIVES_WITH, CHARLIE),
+ VALUE_FACTORY.createStatement(BOB, LIVES_WITH, ALICE));
+
+ final Sail sail = RyaSailFactory.getInstance(config);
+ final SailConnection conn = sail.getConnection();
+ log.info("Loading the following statements into a Mock instance of Accumulo Rya:");
+ conn.begin();
+ for(final Statement statement : statements) {
+ log.info(" " + statement.toString());
+ conn.addStatement(statement.getSubject(), statement.getPredicate(), statement.getObject());
+ }
+ conn.commit();
+ conn.close();
+
+ // Create the table that the Prospector's results will be written to.
+ ConnectorFactory.connect(config)
+ .tableOperations()
+ .create("rya_prospects");
+
+ // Run the Prospector using the configuration file that is in the resources directory.
+ log.info("");
+ log.info("Running the Map Reduce job that computes the Prospector results.");
+ ToolRunner.run(new Prospector(), new String[]{ "src/main/resources/stats_cluster_config.xml" });
+
+ // Print the table that was created by the Prospector.
+ log.info("");
+ log.info("The following cardinalities were written to the Prospector table:");
+ final ProspectorServiceEvalStatsDAO dao = ProspectorServiceEvalStatsDAO.make(config);
+
+ // Do each of the Subjects.
+ double cardinality = dao.getCardinality(config, CARDINALITY_OF.SUBJECT, Lists.newArrayList(ALICE));
--- End diff --
A lot of code duplication here. Could be changed to
```
private static void displayCardinality(final ProspectorServiceEvalStatsDAO dao, final AccumuloRdfConfiguration config, final CARDINALITY_OF cardinalityOf, final List<URI> uris) {
final double cardinality = dao.getCardinality(config, cardinalityOf, uris);
log.info(" " + cardinalityOf.toString() + ": " + Joiner.on("/").join(uris) + ", cardinality: " + cardinality);
}
...
...
final ImmutableMap<CARDINALITY_OF, List<List<URI>>> map = ImmutableMap.<CARDINALITY_OF, List<List<URI>>>builder()
.put(CARDINALITY_OF.SUBJECT,
Lists.newArrayList(
Lists.newArrayList(ALICE),
Lists.newArrayList(BOB),
Lists.newArrayList(CHARLIE)
)
)
.put(CARDINALITY_OF.OBJECT,
Lists.newArrayList(
Lists.newArrayList(WORKS_WITH),
Lists.newArrayList(ADMIRES),
Lists.newArrayList(LIVES_WITH)
)
)
...
.build();
for (final Entry<CARDINALITY_OF, List<List<URI>>> entry : map.entrySet()) {
final CARDINALITY_OF cardinalityOf = entry.getKey();
for (final List<URI> uris : entry.getValue()) {
displayCardinality(dao, config, cardinalityOf, uris);
}
}
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-rya pull request #144: RYA-255 Provide an example class that demon...
Posted by kchilton2 <gi...@git.apache.org>.
Github user kchilton2 commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/144#discussion_r110795434
--- Diff: extras/indexingExample/src/main/java/ProspectorExample.java ---
@@ -0,0 +1,193 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import java.util.List;
+
+import org.apache.hadoop.util.ToolRunner;
+import org.apache.log4j.ConsoleAppender;
+import org.apache.log4j.Level;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+import org.apache.log4j.PatternLayout;
+import org.apache.rya.accumulo.AccumuloRdfConfiguration;
+import org.apache.rya.accumulo.utils.ConnectorFactory;
+import org.apache.rya.api.persist.RdfEvalStatsDAO.CARDINALITY_OF;
+import org.apache.rya.prospector.mr.Prospector;
+import org.apache.rya.prospector.service.ProspectorServiceEvalStatsDAO;
+import org.apache.rya.sail.config.RyaSailFactory;
+import org.openrdf.model.Statement;
+import org.openrdf.model.URI;
+import org.openrdf.model.ValueFactory;
+import org.openrdf.model.impl.ValueFactoryImpl;
+import org.openrdf.sail.Sail;
+import org.openrdf.sail.SailConnection;
+
+import com.beust.jcommander.internal.Lists;
+
+/**
+ * Demonstrates how you can use the {@link Prospector} to count values that appear within an instance of Rya and
+ * then use the {@link ProspectorServiceEvalStatsDAO} to fetch those counts.
+ */
+public class ProspectorExample {
+ private static final Logger log = Logger.getLogger(RyaClientExample.class);
+
+ private static final ValueFactory VALUE_FACTORY = new ValueFactoryImpl();
+
+ private static final URI ALICE = VALUE_FACTORY.createURI("urn:alice");
+ private static final URI BOB = VALUE_FACTORY.createURI("urn:bob");
+ private static final URI CHARLIE = VALUE_FACTORY.createURI("urn:charlie");
+
+ private static final URI WORKS_AT = VALUE_FACTORY.createURI("urn:worksAt");
+ private static final URI ADMIRES = VALUE_FACTORY.createURI("urn:admires");
+ private static final URI LIVES_WITH = VALUE_FACTORY.createURI("urn:livesWith");
+
+ private static final URI BURGER_JOINT = VALUE_FACTORY.createURI("urn:burgerJoint");
+ private static final URI DONUT_SHOP= VALUE_FACTORY.createURI("urn:donutShop");
+
+ public static void main(final String[] args) throws Exception {
+ setupLogging();
+
+ // Configure Rya to use a mock instance.
+ final AccumuloRdfConfiguration config = new AccumuloRdfConfiguration();
+ config.useMockInstance(true);
+ config.setTablePrefix("rya_");
+ config.setUsername("user");
+ config.setPassword("pass");
+ config.setInstanceName("accumulo");
+
+ // Load some data into Rya.
+ final List<Statement> statements = Lists.newArrayList(
+ VALUE_FACTORY.createStatement(ALICE, WORKS_AT, BURGER_JOINT),
+ VALUE_FACTORY.createStatement(ALICE, ADMIRES, BOB),
+ VALUE_FACTORY.createStatement(BOB, WORKS_AT, DONUT_SHOP),
+ VALUE_FACTORY.createStatement(CHARLIE, WORKS_AT, DONUT_SHOP),
+ VALUE_FACTORY.createStatement(CHARLIE, LIVES_WITH, BOB),
+ VALUE_FACTORY.createStatement(BOB, LIVES_WITH, CHARLIE),
+ VALUE_FACTORY.createStatement(BOB, LIVES_WITH, ALICE));
+
+ final Sail sail = RyaSailFactory.getInstance(config);
+ final SailConnection conn = sail.getConnection();
+ log.info("Loading the following statements into a Mock instance of Accumulo Rya:");
+ conn.begin();
+ for(final Statement statement : statements) {
+ log.info(" " + statement.toString());
+ conn.addStatement(statement.getSubject(), statement.getPredicate(), statement.getObject());
+ }
+ conn.commit();
+ conn.close();
+
+ // Create the table that the Prospector's results will be written to.
+ ConnectorFactory.connect(config)
+ .tableOperations()
+ .create("rya_prospects");
+
+ // Run the Prospector using the configuration file that is in the resources directory.
+ log.info("");
+ log.info("Running the Map Reduce job that computes the Prospector results.");
+ ToolRunner.run(new Prospector(), new String[]{ "src/main/resources/stats_cluster_config.xml" });
+
+ // Print the table that was created by the Prospector.
+ log.info("");
+ log.info("The following cardinalities were written to the Prospector table:");
+ final ProspectorServiceEvalStatsDAO dao = ProspectorServiceEvalStatsDAO.make(config);
+
+ // Do each of the Subjects.
+ double cardinality = dao.getCardinality(config, CARDINALITY_OF.SUBJECT, Lists.newArrayList(ALICE));
--- End diff --
I think reducing the lines of the code, in this case, actually makes the example a lot harder to understand.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-rya issue #144: RYA-255 Provide an example class that demonstrates...
Posted by kchilton2 <gi...@git.apache.org>.
Github user kchilton2 commented on the issue:
https://github.com/apache/incubator-rya/pull/144
Once RYA-254 is pulled in, I'll update this pull request.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-rya pull request #144: RYA-255 Provide an example class that demon...
Posted by meiercaleb <gi...@git.apache.org>.
Github user meiercaleb commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/144#discussion_r110792504
--- Diff: extras/indexingExample/src/main/java/ProspectorExample.java ---
@@ -0,0 +1,193 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import java.util.List;
+
+import org.apache.hadoop.util.ToolRunner;
+import org.apache.log4j.ConsoleAppender;
+import org.apache.log4j.Level;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+import org.apache.log4j.PatternLayout;
+import org.apache.rya.accumulo.AccumuloRdfConfiguration;
+import org.apache.rya.accumulo.utils.ConnectorFactory;
+import org.apache.rya.api.persist.RdfEvalStatsDAO.CARDINALITY_OF;
+import org.apache.rya.prospector.mr.Prospector;
+import org.apache.rya.prospector.service.ProspectorServiceEvalStatsDAO;
+import org.apache.rya.sail.config.RyaSailFactory;
+import org.openrdf.model.Statement;
+import org.openrdf.model.URI;
+import org.openrdf.model.ValueFactory;
+import org.openrdf.model.impl.ValueFactoryImpl;
+import org.openrdf.sail.Sail;
+import org.openrdf.sail.SailConnection;
+
+import com.beust.jcommander.internal.Lists;
+
+/**
+ * Demonstrates how you can use the {@link Prospector} to count values that appear within an instance of Rya and
+ * then use the {@link ProspectorServiceEvalStatsDAO} to fetch those counts.
+ */
+public class ProspectorExample {
+ private static final Logger log = Logger.getLogger(RyaClientExample.class);
+
+ private static final ValueFactory VALUE_FACTORY = new ValueFactoryImpl();
+
+ private static final URI ALICE = VALUE_FACTORY.createURI("urn:alice");
+ private static final URI BOB = VALUE_FACTORY.createURI("urn:bob");
+ private static final URI CHARLIE = VALUE_FACTORY.createURI("urn:charlie");
+
+ private static final URI WORKS_AT = VALUE_FACTORY.createURI("urn:worksAt");
+ private static final URI ADMIRES = VALUE_FACTORY.createURI("urn:admires");
+ private static final URI LIVES_WITH = VALUE_FACTORY.createURI("urn:livesWith");
+
+ private static final URI BURGER_JOINT = VALUE_FACTORY.createURI("urn:burgerJoint");
+ private static final URI DONUT_SHOP= VALUE_FACTORY.createURI("urn:donutShop");
+
+ public static void main(final String[] args) throws Exception {
+ setupLogging();
+
+ // Configure Rya to use a mock instance.
+ final AccumuloRdfConfiguration config = new AccumuloRdfConfiguration();
+ config.useMockInstance(true);
+ config.setTablePrefix("rya_");
+ config.setUsername("user");
+ config.setPassword("pass");
+ config.setInstanceName("accumulo");
+
+ // Load some data into Rya.
+ final List<Statement> statements = Lists.newArrayList(
+ VALUE_FACTORY.createStatement(ALICE, WORKS_AT, BURGER_JOINT),
+ VALUE_FACTORY.createStatement(ALICE, ADMIRES, BOB),
+ VALUE_FACTORY.createStatement(BOB, WORKS_AT, DONUT_SHOP),
+ VALUE_FACTORY.createStatement(CHARLIE, WORKS_AT, DONUT_SHOP),
+ VALUE_FACTORY.createStatement(CHARLIE, LIVES_WITH, BOB),
+ VALUE_FACTORY.createStatement(BOB, LIVES_WITH, CHARLIE),
+ VALUE_FACTORY.createStatement(BOB, LIVES_WITH, ALICE));
+
+ final Sail sail = RyaSailFactory.getInstance(config);
+ final SailConnection conn = sail.getConnection();
+ log.info("Loading the following statements into a Mock instance of Accumulo Rya:");
+ conn.begin();
+ for(final Statement statement : statements) {
+ log.info(" " + statement.toString());
+ conn.addStatement(statement.getSubject(), statement.getPredicate(), statement.getObject());
+ }
+ conn.commit();
+ conn.close();
+
+ // Create the table that the Prospector's results will be written to.
+ ConnectorFactory.connect(config)
+ .tableOperations()
+ .create("rya_prospects");
+
+ // Run the Prospector using the configuration file that is in the resources directory.
+ log.info("");
+ log.info("Running the Map Reduce job that computes the Prospector results.");
+ ToolRunner.run(new Prospector(), new String[]{ "src/main/resources/stats_cluster_config.xml" });
+
+ // Print the table that was created by the Prospector.
+ log.info("");
+ log.info("The following cardinalities were written to the Prospector table:");
+ final ProspectorServiceEvalStatsDAO dao = ProspectorServiceEvalStatsDAO.make(config);
+
+ // Do each of the Subjects.
+ double cardinality = dao.getCardinality(config, CARDINALITY_OF.SUBJECT, Lists.newArrayList(ALICE));
--- End diff --
Eric's comment seems very valid to me.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-rya pull request #144: RYA-255 Provide an example class that demon...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/incubator-rya/pull/144
---
[GitHub] incubator-rya issue #144: RYA-255 Provide an example class that demonstrates...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit commented on the issue:
https://github.com/apache/incubator-rya/pull/144
Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/incubator-rya-master-with-optionals-pull-requests/499/
---
[GitHub] incubator-rya issue #144: RYA-255 Provide an example class that demonstrates...
Posted by pujav65 <gi...@git.apache.org>.
Github user pujav65 commented on the issue:
https://github.com/apache/incubator-rya/pull/144
hey @kchilton2 can you fix the conflict? then i will merge.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---