You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@rya.apache.org by jdasch <gi...@git.apache.org> on 2017/07/27 17:19:38 UTC
[GitHub] incubator-rya pull request #188: RYA-318 Rya Shell improvements for packagin...
GitHub user jdasch opened a pull request:
https://github.com/apache/incubator-rya/pull/188
RYA-318 Rya Shell improvements for packaging, loading and querying.
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
## Description
>What Changed?
- Linked in the ability to load RDF statement data from file. Modified default behavior to not flush on every statement.
- Added the ability to perform sparql queries from the shell, either by reading in a query file, or by typing in a new multi-line query (similar to what you'd have in mysql/psql).
- Improved packaging so this is a distributable application (*-bin.tar.gz) that can be installed on a system and made available on a users $PATH, such that they only need to type the command "rya" to launch a shell.
- added the ability to install a rya instance from a script.
- Added user configurable logging to simplify difficult to debug/diagnose nested exceptions that may not get reported to the user as the top level exception message.
- added an example script for creating a rya instance and running a lubm benchmark. Omitted rdf and queries due to licensing concerns.
### Tests
Added RyaCommandsTest.java
Updated RyaAdminCommandsTest.java to support code changes and improved coverage of added features.
Updated RyaConnectionCommandsIT.java, AccumuloLoadStatementsFileIT.java, to support method signature changes.
### Links
[Jira](https://issues.apache.org/jira/browse/RYA-318)
### Checklist
- [ ] Code Review
- [ ] Squash Commits
#### People To Reivew
@amihalik
@meiercaleb
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/jdasch/incubator-rya RYA-318
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-rya/pull/188.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #188
----
commit 6b669be91fad503c3a5977425959e04daab33737
Author: jdasch <jd...@localhost.localdomain>
Date: 2017-07-27T15:00:52Z
RYA-318 Rya Shell improvements for packaging, loading and querying.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-rya issue #188: RYA-318 Rya Shell improvements for packaging, load...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit commented on the issue:
https://github.com/apache/incubator-rya/pull/188
Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/incubator-rya-master-with-optionals-pull-requests/292/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-rya issue #188: RYA-318 Rya Shell improvements for packaging, load...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit commented on the issue:
https://github.com/apache/incubator-rya/pull/188
Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/incubator-rya-master-with-optionals-pull-requests/331/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-rya pull request #188: RYA-318 Rya Shell improvements for packagin...
Posted by jdasch <gi...@git.apache.org>.
Github user jdasch commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/188#discussion_r130160135
--- Diff: common/rya.api/src/main/java/org/apache/rya/api/client/ExecuteSparqlQuery.java ---
@@ -0,0 +1,39 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.rya.api.client;
+
+import edu.umd.cs.findbugs.annotations.DefaultAnnotation;
+import edu.umd.cs.findbugs.annotations.NonNull;
+
+/**
+ * Loads a SPARQL Query and executes the query against an instance of Rya.
+ */
+@DefaultAnnotation(NonNull.class)
+public interface ExecuteSparqlQuery {
+ /**
+ * Loads a SPARQL Query and executes the query against an instance of Rya.
+ *
+ * @param ryaInstanceName - The name of the Rya instance the query will be executed against. (not null)
+ * @param sparqlQuery - A single SPARQL Query. (not null)
+ * @return A user-friendly String representation of the query results.
--- End diff --
Yup. I thought about that. Current implementation is limited to a string of length Integer.MAX_INT. That is a more advanced feature that I think is beyond the scope of this PR. Besides, you can always use LIMIT and OFFSET in your sparql query to get around this limitation. I've been adding out of scope features as I go to this https://issues.apache.org/jira/browse/RYA-314 as subtasks.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-rya issue #188: RYA-318 Rya Shell improvements for packaging, load...
Posted by amihalik <gi...@git.apache.org>.
Github user amihalik commented on the issue:
https://github.com/apache/incubator-rya/pull/188
@jdasch `ERROR: Maven JVM terminated unexpectedly with exit code 137`. The build servers are dead to me.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-rya pull request #188: RYA-318 Rya Shell improvements for packagin...
Posted by kchilton2 <gi...@git.apache.org>.
Github user kchilton2 commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/188#discussion_r130146633
--- Diff: extras/rya.console/src/main/java/org/apache/rya/shell/RyaAdminCommands.java ---
@@ -200,8 +201,65 @@ public String install() {
}
}
- @CliCommand(value = GET_INSTANCE_DETAILS_CMD, help = "Print information about how the Rya instance is configured.")
- public String getInstanceDetails() {
+ @CliCommand(value = INSTALL_PARAMETERS_CMD, help = "Create a new instance of Rya with command line parameters.")
+ public String installWithParameters(
+ @CliOption(key = {"instanceName"}, mandatory = true, help = "The name of the Rya instance to create.")
+ final String instanceName,
+
+ @CliOption(key = {"enableTableHashPrefix"}, mandatory = false, help = "Use Shard Balancing (improves streamed input write speeds).", unspecifiedDefaultValue = "false", specifiedDefaultValue = "true")
+ final boolean enableTableHashPrefix,
+
+ @CliOption(key = {"enableEntityCentricIndex"}, mandatory = false, help = "Use Entity Centric Indexing.", unspecifiedDefaultValue = "false", specifiedDefaultValue = "true")
+ final boolean enableEntityCentricIndex,
+
+ @CliOption(key = {"enableFreeTextIndex"}, mandatory = false, help = "Use Free Text Indexing.", unspecifiedDefaultValue = "false", specifiedDefaultValue = "true")
+ final boolean enableFreeTextIndex,
+
+ @CliOption(key = {"enableGeospatialIndex"}, mandatory = false, help = "Use Geospatial Indexing.", unspecifiedDefaultValue = "false", specifiedDefaultValue = "true")
+ final boolean enableGeospatialIndex,
+
+ @CliOption(key = {"enableTemporalIndex"}, mandatory = false, help = "Use Temporal Indexing.", unspecifiedDefaultValue = "false", specifiedDefaultValue = "true")
+ final boolean enableTemporalIndex,
+
+ @CliOption(key = {"enablePcjIndex"}, mandatory = false, help = "Use Precomputed Join (PCJ) Indexing.", unspecifiedDefaultValue = "false", specifiedDefaultValue = "true")
+ final boolean enablePcjIndex,
+
+ @CliOption(key = {"fluoPcjAppName"}, mandatory = false, help = "Fluo Application Name for PCJ Index Updater (fluo app must be initialized and enablePcjIndex=true).")
--- End diff --
You should have a different prompt depending on if you're installing to mongo and accumulo if they support different features.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-rya issue #188: RYA-318 Rya Shell improvements for packaging, load...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit commented on the issue:
https://github.com/apache/incubator-rya/pull/188
Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/incubator-rya-master-with-optionals-pull-requests/303/<h2>Build result: ABORTED</span></h2>[...truncated 5.05 MB...][WARNING] Failed to notify spy hudson.maven.Maven3Builder$JenkinsEventSpy: java.util.concurrent.ExecutionException: Invalid object ID 27 iota=55[INFO] [INFO] --- maven-enforcer-plugin:1.3.1:enforce (enforce-mvn) @ rya.console ---[WARNING] Failed to notify spy hudson.maven.Maven3Builder$JenkinsEventSpy: java.util.concurrent.ExecutionException: Invalid object ID 27 iota=55[INFO] [INFO] --- jacoco-maven-plugin:0.7.9:prepare-agent (default-prepare-agent) @ rya.console ---[INFO] argLine set to -javaagent:/home/jenkins/jenkins-slave/maven-repositories/1/org/jacoco/org.jacoco.agent/0.7.9/org.jacoco.agent-0.7.9-runtime.jar=destfile=/home/jenkins/jenkins-slave/workspace/incubator-rya-master-with-optionals-pull-requests/extras/rya.console/target/jacoco.exec[WARNING] Failed to notify spy hudson.maven.Maven3Builder$JenkinsEventSpy: java.util.concur
rent.ExecutionException: Invalid object ID 27 iota=55[INFO] [INFO] --- license-maven-plugin:2.6:format (default) @ rya.console ---[INFO] Updating license headers...[WARNING] Unknown file extension: /home/jenkins/jenkins-slave/workspace/incubator-rya-master-with-optionals-pull-requests/extras/rya.console/src/main/examples/Query1.sparql[WARNING] Unknown file extension: /home/jenkins/jenkins-slave/workspace/incubator-rya-master-with-optionals-pull-requests/extras/rya.console/src/main/examples/example.script[WARNING] Unknown file extension: /home/jenkins/jenkins-slave/workspace/incubator-rya-master-with-optionals-pull-requests/extras/rya.console/src/main/examples/triples.nt[WARNING] Unknown file extension: /home/jenkins/jenkins-slave/workspace/incubator-rya-master-with-optionals-pull-requests/extras/rya.console/src/main/scripts/rya[WARNING] Unknown file extension: /home/jenkins/jenkins-slave/workspace/incubator-rya-master-with-optionals-pull-requests/extras/rya.console/src/test/resource
s/Query1.sparqlBuild was aborted[WARNING] Failed to notify spy hudson.maven.Maven3Builder$JenkinsEventSpy: java.util.concurrent.ExecutionException: Invalid object ID 27 iota=55[INFO] [INFO] --- maven-remote-resources-plugin:1.5:process (default) @ rya.console ---channel stoppedSetting status of f907398e79a0c09e0f2f64d2894c59ef7821f942 to FAILURE with url https://builds.apache.org/job/incubator-rya-master-with-optionals-pull-requests/303/ and message: 'FAILURE 'Using context: Jenkins: clean package -Pgeoindexing
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-rya issue #188: RYA-318 Rya Shell improvements for packaging, load...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit commented on the issue:
https://github.com/apache/incubator-rya/pull/188
Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/incubator-rya-master-with-optionals-pull-requests/287/<h2>Build result: ABORTED</span></h2>[...truncated 6.29 MB...][INFO] ------------------------------------------------------------------------[INFO] Building Apache Rya PCJ Fluo Demo 3.2.11-incubating-SNAPSHOT[INFO] ------------------------------------------------------------------------[INFO] [INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ rya.pcj.fluo.demo ---[WARNING] Failed to notify spy hudson.maven.Maven3Builder$JenkinsEventSpy: java.util.concurrent.ExecutionException: Invalid object ID 12 iota=55[INFO] [INFO] --- apache-rat-plugin:0.11:check (check-licenses) @ rya.pcj.fluo.demo ---[INFO] 51 implicit excludes (use -debug for more details).[INFO] Exclude: **/resources/META-INF/services/**[INFO] 4 resources included (use -debug for more details)[INFO] Rat check: Summary of files. Unapproved: 0 unknown: 0 generated: 0 approved: 4 licence.[WARNING] Failed to notify spy hudson.maven.Mave
n3Builder$JenkinsEventSpy: java.util.concurrent.ExecutionException: Invalid object ID 12 iota=55[INFO] [INFO] --- maven-enforcer-plugin:1.3.1:enforce (enforce-mvn) @ rya.pcj.fluo.demo ---[WARNING] Failed to notify spy hudson.maven.Maven3Builder$JenkinsEventSpy: java.util.concurrent.ExecutionException: Invalid object ID 12 iota=55[INFO] [INFO] --- maven-remote-resources-plugin:1.5:process (default) @ rya.pcj.fluo.demo ---Build was abortedAborted by mihalikchannel stoppedSetting status of 6b669be91fad503c3a5977425959e04daab33737 to FAILURE with url https://builds.apache.org/job/incubator-rya-master-with-optionals-pull-requests/287/ and message: 'FAILURE 'Using context: Jenkins: clean package -Pgeoindexing
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-rya pull request #188: RYA-318 Rya Shell improvements for packagin...
Posted by kchilton2 <gi...@git.apache.org>.
Github user kchilton2 commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/188#discussion_r130145866
--- Diff: common/rya.api/src/main/java/org/apache/rya/api/client/ExecuteSparqlQuery.java ---
@@ -0,0 +1,39 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.rya.api.client;
+
+import edu.umd.cs.findbugs.annotations.DefaultAnnotation;
+import edu.umd.cs.findbugs.annotations.NonNull;
+
+/**
+ * Loads a SPARQL Query and executes the query against an instance of Rya.
+ */
+@DefaultAnnotation(NonNull.class)
+public interface ExecuteSparqlQuery {
+ /**
+ * Loads a SPARQL Query and executes the query against an instance of Rya.
+ *
+ * @param ryaInstanceName - The name of the Rya instance the query will be executed against. (not null)
+ * @param sparqlQuery - A single SPARQL Query. (not null)
+ * @return A user-friendly String representation of the query results.
--- End diff --
I see further down that you actually have an idea about what should be in a query result. Like, timing information and stuff. You might want to actually return an object with all of that in it here (and maybe add paging), then format what it will look like in the console code.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-rya issue #188: RYA-318 Rya Shell improvements for packaging, load...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit commented on the issue:
https://github.com/apache/incubator-rya/pull/188
Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/incubator-rya-master-with-optionals-pull-requests/302/<h2>Build result: FAILURE</span></h2>[...truncated 4.70 MB...] Compare (>) Var (name=t) ValueConstant (value="8"^^<http://www.w3.org/2001/XMLSchema#integer>) Join Join StatementPattern Var (name=s) Var (name=-const-http://www.w3.org/1999/02/22-rdf-syntax-ns#type, value=http://www.w3.org/1999/02/22-rdf-syntax-ns#type, anonymous) Var (name=t) StatementPattern Var (name=t) Var (name=-const-http://www.w3.org/2000/01/rdf-schema#label, value=http://www.w3.org/2000/01/rdf-schema#label, anonymous) Var (name=u) StatementPattern Var (name=u) Var (name=-const-uri:talksTo, value=uri:talksTo, anonymous) Var (name=s)Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.013 sec - in org.apache.rya.indexing.IndexPl
anValidator.VarConstantIndexListPrunerTestRunning org.apache.rya.indexing.IndexPlanValidator.ThreshholdPlanSelectorTestERROR: Maven JVM terminated unexpectedly with exit code 137Setting status of 6d894f384730523eb2b0b38b7a6ebab846ca71fa to FAILURE with url https://builds.apache.org/job/incubator-rya-master-with-optionals-pull-requests/302/ and message: 'FAILURE 'Using context: Jenkins: clean package -Pgeoindexing
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-rya pull request #188: RYA-318 Rya Shell improvements for packagin...
Posted by pujav65 <gi...@git.apache.org>.
Github user pujav65 commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/188#discussion_r129999450
--- Diff: extras/indexing/src/main/java/org/apache/rya/api/client/accumulo/AccumuloExecuteSparqlQuery.java ---
@@ -0,0 +1,150 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.rya.api.client.accumulo;
+
+import static java.util.Objects.requireNonNull;
+
+import java.text.DecimalFormat;
+
+import org.apache.accumulo.core.client.AccumuloException;
+import org.apache.accumulo.core.client.AccumuloSecurityException;
+import org.apache.accumulo.core.client.Connector;
+import org.apache.log4j.Logger;
+import org.apache.rya.accumulo.AccumuloRdfConfiguration;
+import org.apache.rya.api.client.ExecuteSparqlQuery;
+import org.apache.rya.api.client.InstanceDoesNotExistException;
+import org.apache.rya.api.client.InstanceExists;
+import org.apache.rya.api.client.LoadStatementsFile;
+import org.apache.rya.api.client.RyaClientException;
+import org.apache.rya.api.persist.RyaDAOException;
+import org.apache.rya.rdftriplestore.inference.InferenceEngineException;
+import org.apache.rya.sail.config.RyaSailFactory;
+import org.openrdf.query.MalformedQueryException;
+import org.openrdf.query.QueryEvaluationException;
+import org.openrdf.query.QueryLanguage;
+import org.openrdf.query.TupleQuery;
+import org.openrdf.query.TupleQueryResult;
+import org.openrdf.repository.RepositoryException;
+import org.openrdf.repository.sail.SailRepository;
+import org.openrdf.repository.sail.SailRepositoryConnection;
+import org.openrdf.sail.Sail;
+import org.openrdf.sail.SailException;
+
+import edu.umd.cs.findbugs.annotations.DefaultAnnotation;
+import edu.umd.cs.findbugs.annotations.NonNull;
+
+/**
+ * An Accumulo implementation of the {@link LoadStatementsFile} command.
+ */
+@DefaultAnnotation(NonNull.class)
+public class AccumuloExecuteSparqlQuery extends AccumuloCommand implements ExecuteSparqlQuery {
+ private static final Logger log = Logger.getLogger(AccumuloExecuteSparqlQuery.class);
+
+ private final InstanceExists instanceExists;
+
+ /**
+ * Constructs an instance of {@link AccumuloExecuteSparqlQuery}.
+ *
+ * @param connectionDetails - Details about the values that were used to create
+ * the connector to the cluster. (not null)
+ * @param connector - Provides programmatic access to the instance of Accumulo
+ * that hosts Rya instance. (not null)
+ */
+ public AccumuloExecuteSparqlQuery(final AccumuloConnectionDetails connectionDetails, final Connector connector) {
+ super(connectionDetails, connector);
+ instanceExists = new AccumuloInstanceExists(connectionDetails, connector);
+ }
+
+
+ @Override
+ public String executeSparqlQuery(final String ryaInstanceName, final String sparqlQuery)
+ throws InstanceDoesNotExistException, RyaClientException {
+ requireNonNull(ryaInstanceName);
+ requireNonNull(sparqlQuery);
+
+ // Ensure the Rya Instance exists.
+ if(!instanceExists.exists(ryaInstanceName)) {
+ throw new InstanceDoesNotExistException(String.format("There is no Rya instance named '%s'.", ryaInstanceName));
+ }
+
+
+ Sail sail = null;
+ SailRepository sailRepo = null;
+ SailRepositoryConnection sailRepoConn = null;
+
+ try {
+ // Get a Sail object that is connected to the Rya instance.
+ final AccumuloRdfConfiguration ryaConf = getAccumuloConnectionDetails().buildAccumuloRdfConfiguration(ryaInstanceName);
+ sail = RyaSailFactory.getInstance(ryaConf);
+
+ // Load the file.
+ sailRepo = new SailRepository(sail);
+ sailRepoConn = sailRepo.getConnection();
+ final long start = System.currentTimeMillis();
+ final TupleQuery tupleQuery = sailRepoConn.prepareTupleQuery(QueryLanguage.SPARQL, sparqlQuery);
+ final TupleQueryResult result = tupleQuery.evaluate();
+ final StringBuilder sb = new StringBuilder();
+ final String newline = "\n";
+ sb.append("Query Result:").append(newline);
+ sb.append("Binding Names: ").append(result.getBindingNames().toString()).append(newline);
+
+ int count = 0;
+ while(result.hasNext()) {
+ sb.append(result.next().toString()).append(newline);
+ count++;
+ }
+ final String seconds = new DecimalFormat("0.0##").format((System.currentTimeMillis() - start) / 1000.0);
+ sb.append("Retrieved ").append(count).append(" results in ").append(seconds).append(" seconds.");
--- End diff --
why are we appending the timing? we do this in examples, but i'm not sure if its appropriate here.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-rya pull request #188: RYA-318 Rya Shell improvements for packagin...
Posted by jdasch <gi...@git.apache.org>.
Github user jdasch commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/188#discussion_r130091543
--- Diff: common/rya.api/src/main/java/org/apache/rya/api/client/LoadStatementsFile.java ---
@@ -31,14 +31,16 @@
@DefaultAnnotation(NonNull.class)
public interface LoadStatementsFile {
- /**
- * Loads a local file of RDF statements into an instance of Rya.
- *
- * @param ryaInstanceName - The name of the Rya instance the statements will be loaded into. (not null)
- * @param statementsFile - A file that holds RDF statements that will be loaded. (not null)
- * @param format - The format of the statements file. (not null)
- * @throws InstanceDoesNotExistException No instance of Rya exists for the provided name.
- * @throws RyaClientException Something caused the command to fail.
- */
- public void loadStatements(String ryaInstanceName, Path statementsFile, RDFFormat format) throws InstanceDoesNotExistException, RyaClientException;
+ /**
+ * Loads a local file of RDF statements into an instance of Rya.
+ *
+ * @param ryaInstanceName - The name of the Rya instance the statements will be loaded into. (not null)
+ * @param statementsFile - A file that holds RDF statements that will be loaded. (not null)
+ * @param format - The format of the statements file. (not null)
+ * @param flushEachUpdate - If true, each statement will be flushed individually. If false, statements will be
+ * grouped in an unspecified manner. A value of false will typically yield higher ingest rates.
+ * @throws InstanceDoesNotExistException No instance of Rya exists for the provided name.
+ * @throws RyaClientException Something caused the command to fail.
+ */
+ public void loadStatements(String ryaInstanceName, Path statementsFile, RDFFormat format, boolean flushEachUpdate) throws InstanceDoesNotExistException, RyaClientException;
--- End diff --
Good catch. I think this (and other future config settings) will need to be handled by the RyaConnectionCommands.connectToAccumulo() function. Currently it is nice to be able to change it while being connected, but it keeps the API cleaner if we move it. I'll make the adjustment.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-rya pull request #188: RYA-318 Rya Shell improvements for packagin...
Posted by kchilton2 <gi...@git.apache.org>.
Github user kchilton2 commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/188#discussion_r130143520
--- Diff: common/rya.api/src/main/java/org/apache/rya/api/client/ExecuteSparqlQuery.java ---
@@ -0,0 +1,39 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.rya.api.client;
+
+import edu.umd.cs.findbugs.annotations.DefaultAnnotation;
+import edu.umd.cs.findbugs.annotations.NonNull;
+
+/**
+ * Loads a SPARQL Query and executes the query against an instance of Rya.
+ */
+@DefaultAnnotation(NonNull.class)
+public interface ExecuteSparqlQuery {
+ /**
+ * Loads a SPARQL Query and executes the query against an instance of Rya.
+ *
+ * @param ryaInstanceName - The name of the Rya instance the query will be executed against. (not null)
+ * @param sparqlQuery - A single SPARQL Query. (not null)
+ * @return A user-friendly String representation of the query results.
--- End diff --
What if thousands of results are returned by the query? Tens of thousands? More? This interface doesn't suggest any sort of pagination over the search results, so I'd worry that it'll load everything into memory and format it as a single String for output. It also must force some sort of display format within the interface. Seems like it would be a little more flexible if the interface returned an iterator over BindingSets that are the results of the query.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-rya pull request #188: RYA-318 Rya Shell improvements for packagin...
Posted by pujav65 <gi...@git.apache.org>.
Github user pujav65 commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/188#discussion_r129999856
--- Diff: extras/rya.console/src/main/java/org/apache/rya/shell/RyaAdminCommands.java ---
@@ -200,8 +201,65 @@ public String install() {
}
}
- @CliCommand(value = GET_INSTANCE_DETAILS_CMD, help = "Print information about how the Rya instance is configured.")
- public String getInstanceDetails() {
+ @CliCommand(value = INSTALL_PARAMETERS_CMD, help = "Create a new instance of Rya with command line parameters.")
+ public String installWithParameters(
+ @CliOption(key = {"instanceName"}, mandatory = true, help = "The name of the Rya instance to create.")
+ final String instanceName,
+
+ @CliOption(key = {"enableTableHashPrefix"}, mandatory = false, help = "Use Shard Balancing (improves streamed input write speeds).", unspecifiedDefaultValue = "false", specifiedDefaultValue = "true")
+ final boolean enableTableHashPrefix,
+
+ @CliOption(key = {"enableEntityCentricIndex"}, mandatory = false, help = "Use Entity Centric Indexing.", unspecifiedDefaultValue = "false", specifiedDefaultValue = "true")
+ final boolean enableEntityCentricIndex,
+
+ @CliOption(key = {"enableFreeTextIndex"}, mandatory = false, help = "Use Free Text Indexing.", unspecifiedDefaultValue = "false", specifiedDefaultValue = "true")
+ final boolean enableFreeTextIndex,
+
+ @CliOption(key = {"enableGeospatialIndex"}, mandatory = false, help = "Use Geospatial Indexing.", unspecifiedDefaultValue = "false", specifiedDefaultValue = "true")
+ final boolean enableGeospatialIndex,
+
+ @CliOption(key = {"enableTemporalIndex"}, mandatory = false, help = "Use Temporal Indexing.", unspecifiedDefaultValue = "false", specifiedDefaultValue = "true")
+ final boolean enableTemporalIndex,
+
+ @CliOption(key = {"enablePcjIndex"}, mandatory = false, help = "Use Precomputed Join (PCJ) Indexing.", unspecifiedDefaultValue = "false", specifiedDefaultValue = "true")
+ final boolean enablePcjIndex,
+
+ @CliOption(key = {"fluoPcjAppName"}, mandatory = false, help = "Fluo Application Name for PCJ Index Updater (fluo app must be initialized and enablePcjIndex=true).")
--- End diff --
some of this stuff is very accumulo specific -- i'm not sure how to address that for when we implement mongo support.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-rya pull request #188: RYA-318 Rya Shell improvements for packagin...
Posted by kchilton2 <gi...@git.apache.org>.
Github user kchilton2 commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/188#discussion_r130144295
--- Diff: common/rya.api/src/main/java/org/apache/rya/api/client/LoadStatementsFile.java ---
@@ -31,14 +31,16 @@
@DefaultAnnotation(NonNull.class)
public interface LoadStatementsFile {
- /**
- * Loads a local file of RDF statements into an instance of Rya.
- *
- * @param ryaInstanceName - The name of the Rya instance the statements will be loaded into. (not null)
- * @param statementsFile - A file that holds RDF statements that will be loaded. (not null)
- * @param format - The format of the statements file. (not null)
- * @throws InstanceDoesNotExistException No instance of Rya exists for the provided name.
- * @throws RyaClientException Something caused the command to fail.
- */
- public void loadStatements(String ryaInstanceName, Path statementsFile, RDFFormat format) throws InstanceDoesNotExistException, RyaClientException;
+ /**
+ * Loads a local file of RDF statements into an instance of Rya.
+ *
+ * @param ryaInstanceName - The name of the Rya instance the statements will be loaded into. (not null)
+ * @param statementsFile - A file that holds RDF statements that will be loaded. (not null)
+ * @param format - The format of the statements file. (not null)
+ * @param flushEachUpdate - If true, each statement will be flushed individually. If false, statements will be
+ * grouped in an unspecified manner. A value of false will typically yield higher ingest rates.
+ * @throws InstanceDoesNotExistException No instance of Rya exists for the provided name.
+ * @throws RyaClientException Something caused the command to fail.
+ */
+ public void loadStatements(String ryaInstanceName, Path statementsFile, RDFFormat format, boolean flushEachUpdate) throws InstanceDoesNotExistException, RyaClientException;
--- End diff --
I think it makes more sense to expose the batching as a "batch size" and "timeout" pair if you're going for flexibility. Or provide something that is responsible for managing the batching.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-rya pull request #188: RYA-318 Rya Shell improvements for packagin...
Posted by amihalik <gi...@git.apache.org>.
Github user amihalik commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/188#discussion_r129999333
--- Diff: extras/indexing/src/main/java/org/apache/rya/api/client/accumulo/AccumuloExecuteSparqlQuery.java ---
@@ -0,0 +1,150 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.rya.api.client.accumulo;
+
+import static java.util.Objects.requireNonNull;
+
+import java.text.DecimalFormat;
+
+import org.apache.accumulo.core.client.AccumuloException;
+import org.apache.accumulo.core.client.AccumuloSecurityException;
+import org.apache.accumulo.core.client.Connector;
+import org.apache.log4j.Logger;
+import org.apache.rya.accumulo.AccumuloRdfConfiguration;
+import org.apache.rya.api.client.ExecuteSparqlQuery;
+import org.apache.rya.api.client.InstanceDoesNotExistException;
+import org.apache.rya.api.client.InstanceExists;
+import org.apache.rya.api.client.LoadStatementsFile;
+import org.apache.rya.api.client.RyaClientException;
+import org.apache.rya.api.persist.RyaDAOException;
+import org.apache.rya.rdftriplestore.inference.InferenceEngineException;
+import org.apache.rya.sail.config.RyaSailFactory;
+import org.openrdf.query.MalformedQueryException;
+import org.openrdf.query.QueryEvaluationException;
+import org.openrdf.query.QueryLanguage;
+import org.openrdf.query.TupleQuery;
+import org.openrdf.query.TupleQueryResult;
+import org.openrdf.repository.RepositoryException;
+import org.openrdf.repository.sail.SailRepository;
+import org.openrdf.repository.sail.SailRepositoryConnection;
+import org.openrdf.sail.Sail;
+import org.openrdf.sail.SailException;
+
+import edu.umd.cs.findbugs.annotations.DefaultAnnotation;
+import edu.umd.cs.findbugs.annotations.NonNull;
+
+/**
+ * An Accumulo implementation of the {@link LoadStatementsFile} command.
+ */
+@DefaultAnnotation(NonNull.class)
+public class AccumuloExecuteSparqlQuery extends AccumuloCommand implements ExecuteSparqlQuery {
+ private static final Logger log = Logger.getLogger(AccumuloExecuteSparqlQuery.class);
+
+ private final InstanceExists instanceExists;
+
+ /**
+ * Constructs an instance of {@link AccumuloExecuteSparqlQuery}.
+ *
+ * @param connectionDetails - Details about the values that were used to create
+ * the connector to the cluster. (not null)
+ * @param connector - Provides programmatic access to the instance of Accumulo
+ * that hosts Rya instance. (not null)
+ */
+ public AccumuloExecuteSparqlQuery(final AccumuloConnectionDetails connectionDetails, final Connector connector) {
+ super(connectionDetails, connector);
+ instanceExists = new AccumuloInstanceExists(connectionDetails, connector);
+ }
+
+
+ @Override
+ public String executeSparqlQuery(final String ryaInstanceName, final String sparqlQuery)
+ throws InstanceDoesNotExistException, RyaClientException {
+ requireNonNull(ryaInstanceName);
+ requireNonNull(sparqlQuery);
+
+ // Ensure the Rya Instance exists.
+ if(!instanceExists.exists(ryaInstanceName)) {
+ throw new InstanceDoesNotExistException(String.format("There is no Rya instance named '%s'.", ryaInstanceName));
+ }
+
+
+ Sail sail = null;
+ SailRepository sailRepo = null;
+ SailRepositoryConnection sailRepoConn = null;
+
+ try {
+ // Get a Sail object that is connected to the Rya instance.
+ final AccumuloRdfConfiguration ryaConf = getAccumuloConnectionDetails().buildAccumuloRdfConfiguration(ryaInstanceName);
+ sail = RyaSailFactory.getInstance(ryaConf);
+
+ // Load the file.
+ sailRepo = new SailRepository(sail);
+ sailRepoConn = sailRepo.getConnection();
+ final long start = System.currentTimeMillis();
+ final TupleQuery tupleQuery = sailRepoConn.prepareTupleQuery(QueryLanguage.SPARQL, sparqlQuery);
+ final TupleQueryResult result = tupleQuery.evaluate();
+ final StringBuilder sb = new StringBuilder();
+ final String newline = "\n";
+ sb.append("Query Result:").append(newline);
+ sb.append("Binding Names: ").append(result.getBindingNames().toString()).append(newline);
+
+ int count = 0;
+ while(result.hasNext()) {
--- End diff --
There are some nice tools to print this (like the SPARQLResultsCSVWriter)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-rya pull request #188: RYA-318 Rya Shell improvements for packagin...
Posted by jdasch <gi...@git.apache.org>.
Github user jdasch commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/188#discussion_r130091740
--- Diff: extras/indexing/src/main/java/org/apache/rya/api/client/accumulo/AccumuloExecuteSparqlQuery.java ---
@@ -0,0 +1,150 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.rya.api.client.accumulo;
+
+import static java.util.Objects.requireNonNull;
+
+import java.text.DecimalFormat;
+
+import org.apache.accumulo.core.client.AccumuloException;
+import org.apache.accumulo.core.client.AccumuloSecurityException;
+import org.apache.accumulo.core.client.Connector;
+import org.apache.log4j.Logger;
+import org.apache.rya.accumulo.AccumuloRdfConfiguration;
+import org.apache.rya.api.client.ExecuteSparqlQuery;
+import org.apache.rya.api.client.InstanceDoesNotExistException;
+import org.apache.rya.api.client.InstanceExists;
+import org.apache.rya.api.client.LoadStatementsFile;
+import org.apache.rya.api.client.RyaClientException;
+import org.apache.rya.api.persist.RyaDAOException;
+import org.apache.rya.rdftriplestore.inference.InferenceEngineException;
+import org.apache.rya.sail.config.RyaSailFactory;
+import org.openrdf.query.MalformedQueryException;
+import org.openrdf.query.QueryEvaluationException;
+import org.openrdf.query.QueryLanguage;
+import org.openrdf.query.TupleQuery;
+import org.openrdf.query.TupleQueryResult;
+import org.openrdf.repository.RepositoryException;
+import org.openrdf.repository.sail.SailRepository;
+import org.openrdf.repository.sail.SailRepositoryConnection;
+import org.openrdf.sail.Sail;
+import org.openrdf.sail.SailException;
+
+import edu.umd.cs.findbugs.annotations.DefaultAnnotation;
+import edu.umd.cs.findbugs.annotations.NonNull;
+
+/**
+ * An Accumulo implementation of the {@link LoadStatementsFile} command.
+ */
+@DefaultAnnotation(NonNull.class)
+public class AccumuloExecuteSparqlQuery extends AccumuloCommand implements ExecuteSparqlQuery {
+ private static final Logger log = Logger.getLogger(AccumuloExecuteSparqlQuery.class);
+
+ private final InstanceExists instanceExists;
+
+ /**
+ * Constructs an instance of {@link AccumuloExecuteSparqlQuery}.
+ *
+ * @param connectionDetails - Details about the values that were used to create
+ * the connector to the cluster. (not null)
+ * @param connector - Provides programmatic access to the instance of Accumulo
+ * that hosts Rya instance. (not null)
+ */
+ public AccumuloExecuteSparqlQuery(final AccumuloConnectionDetails connectionDetails, final Connector connector) {
+ super(connectionDetails, connector);
+ instanceExists = new AccumuloInstanceExists(connectionDetails, connector);
+ }
+
+
+ @Override
+ public String executeSparqlQuery(final String ryaInstanceName, final String sparqlQuery)
+ throws InstanceDoesNotExistException, RyaClientException {
+ requireNonNull(ryaInstanceName);
+ requireNonNull(sparqlQuery);
+
+ // Ensure the Rya Instance exists.
+ if(!instanceExists.exists(ryaInstanceName)) {
+ throw new InstanceDoesNotExistException(String.format("There is no Rya instance named '%s'.", ryaInstanceName));
+ }
+
+
+ Sail sail = null;
+ SailRepository sailRepo = null;
+ SailRepositoryConnection sailRepoConn = null;
+
+ try {
+ // Get a Sail object that is connected to the Rya instance.
+ final AccumuloRdfConfiguration ryaConf = getAccumuloConnectionDetails().buildAccumuloRdfConfiguration(ryaInstanceName);
+ sail = RyaSailFactory.getInstance(ryaConf);
+
+ // Load the file.
--- End diff --
Sorry. CPD...
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-rya pull request #188: RYA-318 Rya Shell improvements for packagin...
Posted by kchilton2 <gi...@git.apache.org>.
Github user kchilton2 commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/188#discussion_r130147972
--- Diff: extras/rya.console/src/main/java/org/apache/rya/shell/util/ConsolePrinter.java ---
@@ -0,0 +1,85 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.rya.shell.util;
+
+import java.io.IOException;
+
+import edu.umd.cs.findbugs.annotations.DefaultAnnotation;
+import edu.umd.cs.findbugs.annotations.NonNull;
+import jline.console.ConsoleReader;
+
+/**
+ * A mechanism for printing content to the console.
+ */
+@DefaultAnnotation(NonNull.class)
+public interface ConsolePrinter {
+
+ /**
+ * Prints the provided content to the console.
--- End diff --
nit: Some tab characters in this classs
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-rya issue #188: RYA-318 Rya Shell improvements for packaging, load...
Posted by pujav65 <gi...@git.apache.org>.
Github user pujav65 commented on the issue:
https://github.com/apache/incubator-rya/pull/188
My main concern here is that we exposed a number of accumulo specific operations that are not really optional which will make implementing a mongo console implementation very difficult. I think maybe adding some checks to see if you are creating an accumulo instance might alleviate that concern.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-rya pull request #188: RYA-318 Rya Shell improvements for packagin...
Posted by jdasch <gi...@git.apache.org>.
Github user jdasch commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/188#discussion_r130092162
--- Diff: extras/rya.console/src/main/examples/lubm.script ---
@@ -0,0 +1,39 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+connect-accumulo --username accumulo_user --instanceName accumulo_instance --zookeepers zoo1,zoo2,zoo3,zoo4,zoo5
--- End diff --
Yeah, I had the full lubm in here and then I removed it due to licensing concerns. I thought this was generic enough, but I'll rename it to example.script and put an example query in there.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-rya pull request #188: RYA-318 Rya Shell improvements for packagin...
Posted by amihalik <gi...@git.apache.org>.
Github user amihalik commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/188#discussion_r129999058
--- Diff: extras/indexing/src/main/java/org/apache/rya/api/client/accumulo/AccumuloExecuteSparqlQuery.java ---
@@ -0,0 +1,150 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.rya.api.client.accumulo;
+
+import static java.util.Objects.requireNonNull;
+
+import java.text.DecimalFormat;
+
+import org.apache.accumulo.core.client.AccumuloException;
+import org.apache.accumulo.core.client.AccumuloSecurityException;
+import org.apache.accumulo.core.client.Connector;
+import org.apache.log4j.Logger;
+import org.apache.rya.accumulo.AccumuloRdfConfiguration;
+import org.apache.rya.api.client.ExecuteSparqlQuery;
+import org.apache.rya.api.client.InstanceDoesNotExistException;
+import org.apache.rya.api.client.InstanceExists;
+import org.apache.rya.api.client.LoadStatementsFile;
+import org.apache.rya.api.client.RyaClientException;
+import org.apache.rya.api.persist.RyaDAOException;
+import org.apache.rya.rdftriplestore.inference.InferenceEngineException;
+import org.apache.rya.sail.config.RyaSailFactory;
+import org.openrdf.query.MalformedQueryException;
+import org.openrdf.query.QueryEvaluationException;
+import org.openrdf.query.QueryLanguage;
+import org.openrdf.query.TupleQuery;
+import org.openrdf.query.TupleQueryResult;
+import org.openrdf.repository.RepositoryException;
+import org.openrdf.repository.sail.SailRepository;
+import org.openrdf.repository.sail.SailRepositoryConnection;
+import org.openrdf.sail.Sail;
+import org.openrdf.sail.SailException;
+
+import edu.umd.cs.findbugs.annotations.DefaultAnnotation;
+import edu.umd.cs.findbugs.annotations.NonNull;
+
+/**
+ * An Accumulo implementation of the {@link LoadStatementsFile} command.
+ */
+@DefaultAnnotation(NonNull.class)
+public class AccumuloExecuteSparqlQuery extends AccumuloCommand implements ExecuteSparqlQuery {
+ private static final Logger log = Logger.getLogger(AccumuloExecuteSparqlQuery.class);
+
+ private final InstanceExists instanceExists;
+
+ /**
+ * Constructs an instance of {@link AccumuloExecuteSparqlQuery}.
+ *
+ * @param connectionDetails - Details about the values that were used to create
+ * the connector to the cluster. (not null)
+ * @param connector - Provides programmatic access to the instance of Accumulo
+ * that hosts Rya instance. (not null)
+ */
+ public AccumuloExecuteSparqlQuery(final AccumuloConnectionDetails connectionDetails, final Connector connector) {
+ super(connectionDetails, connector);
+ instanceExists = new AccumuloInstanceExists(connectionDetails, connector);
+ }
+
+
+ @Override
+ public String executeSparqlQuery(final String ryaInstanceName, final String sparqlQuery)
+ throws InstanceDoesNotExistException, RyaClientException {
+ requireNonNull(ryaInstanceName);
+ requireNonNull(sparqlQuery);
+
+ // Ensure the Rya Instance exists.
+ if(!instanceExists.exists(ryaInstanceName)) {
+ throw new InstanceDoesNotExistException(String.format("There is no Rya instance named '%s'.", ryaInstanceName));
+ }
+
+
+ Sail sail = null;
+ SailRepository sailRepo = null;
+ SailRepositoryConnection sailRepoConn = null;
+
+ try {
+ // Get a Sail object that is connected to the Rya instance.
+ final AccumuloRdfConfiguration ryaConf = getAccumuloConnectionDetails().buildAccumuloRdfConfiguration(ryaInstanceName);
+ sail = RyaSailFactory.getInstance(ryaConf);
+
+ // Load the file.
--- End diff --
?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-rya pull request #188: RYA-318 Rya Shell improvements for packagin...
Posted by jdasch <gi...@git.apache.org>.
Github user jdasch closed the pull request at:
https://github.com/apache/incubator-rya/pull/188
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-rya pull request #188: RYA-318 Rya Shell improvements for packagin...
Posted by jdasch <gi...@git.apache.org>.
Github user jdasch commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/188#discussion_r130160343
--- Diff: extras/rya.console/src/main/java/org/apache/rya/shell/util/ConsolePrinter.java ---
@@ -0,0 +1,85 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.rya.shell.util;
+
+import java.io.IOException;
+
+import edu.umd.cs.findbugs.annotations.DefaultAnnotation;
+import edu.umd.cs.findbugs.annotations.NonNull;
+import jline.console.ConsoleReader;
+
+/**
+ * A mechanism for printing content to the console.
+ */
+@DefaultAnnotation(NonNull.class)
+public interface ConsolePrinter {
+
+ /**
+ * Prints the provided content to the console.
--- End diff --
good catch
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-rya issue #188: RYA-318 Rya Shell improvements for packaging, load...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit commented on the issue:
https://github.com/apache/incubator-rya/pull/188
Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/incubator-rya-master-with-optionals-pull-requests/304/<h2>Build result: ABORTED</span></h2>[...truncated 5.43 MB...][INFO] [INFO] ------------------------------------------------------------------------[INFO] Building Apache Rya PCJ Fluo Demo 3.2.11-incubating-SNAPSHOT[INFO] ------------------------------------------------------------------------[INFO] [INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ rya.pcj.fluo.demo ---[WARNING] Failed to notify spy hudson.maven.Maven3Builder$JenkinsEventSpy: java.util.concurrent.ExecutionException: Invalid object ID 12 iota=55[INFO] [INFO] --- apache-rat-plugin:0.11:check (check-licenses) @ rya.pcj.fluo.demo ---[INFO] 51 implicit excludes (use -debug for more details).[INFO] Exclude: **/resources/META-INF/services/**[INFO] 4 resources included (use -debug for more details)[INFO] Rat check: Summary of files. Unapproved: 0 unknown: 0 ge
nerated: 0 approved: 4 licence.Build was aborted[WARNING] Failed to notify spy hudson.maven.Maven3Builder$JenkinsEventSpy: java.util.concurrent.ExecutionException: Invalid object ID 12 iota=55[INFO] [INFO] --- maven-enforcer-plugin:1.3.1:enforce (enforce-mvn) @ rya.pcj.fluo.demo ---[WARNING] Failed to notify spy hudson.maven.Maven3Builder$JenkinsEventSpy: hudson.remoting.ChannelClosedException: channel is already closed[INFO] [INFO] --- maven-remote-resources-plugin:1.5:process (default) @ rya.pcj.fluo.demo ---channel stoppedSetting status of f907398e79a0c09e0f2f64d2894c59ef7821f942 to FAILURE with url https://builds.apache.org/job/incubator-rya-master-with-optionals-pull-requests/304/ and message: 'FAILURE 'Using context: Jenkins: clean package -Pgeoindexing
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-rya pull request #188: RYA-318 Rya Shell improvements for packagin...
Posted by jdasch <gi...@git.apache.org>.
Github user jdasch commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/188#discussion_r130091994
--- Diff: extras/indexing/src/main/java/org/apache/rya/api/client/accumulo/AccumuloExecuteSparqlQuery.java ---
@@ -0,0 +1,150 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.rya.api.client.accumulo;
+
+import static java.util.Objects.requireNonNull;
+
+import java.text.DecimalFormat;
+
+import org.apache.accumulo.core.client.AccumuloException;
+import org.apache.accumulo.core.client.AccumuloSecurityException;
+import org.apache.accumulo.core.client.Connector;
+import org.apache.log4j.Logger;
+import org.apache.rya.accumulo.AccumuloRdfConfiguration;
+import org.apache.rya.api.client.ExecuteSparqlQuery;
+import org.apache.rya.api.client.InstanceDoesNotExistException;
+import org.apache.rya.api.client.InstanceExists;
+import org.apache.rya.api.client.LoadStatementsFile;
+import org.apache.rya.api.client.RyaClientException;
+import org.apache.rya.api.persist.RyaDAOException;
+import org.apache.rya.rdftriplestore.inference.InferenceEngineException;
+import org.apache.rya.sail.config.RyaSailFactory;
+import org.openrdf.query.MalformedQueryException;
+import org.openrdf.query.QueryEvaluationException;
+import org.openrdf.query.QueryLanguage;
+import org.openrdf.query.TupleQuery;
+import org.openrdf.query.TupleQueryResult;
+import org.openrdf.repository.RepositoryException;
+import org.openrdf.repository.sail.SailRepository;
+import org.openrdf.repository.sail.SailRepositoryConnection;
+import org.openrdf.sail.Sail;
+import org.openrdf.sail.SailException;
+
+import edu.umd.cs.findbugs.annotations.DefaultAnnotation;
+import edu.umd.cs.findbugs.annotations.NonNull;
+
+/**
+ * An Accumulo implementation of the {@link LoadStatementsFile} command.
+ */
+@DefaultAnnotation(NonNull.class)
+public class AccumuloExecuteSparqlQuery extends AccumuloCommand implements ExecuteSparqlQuery {
+ private static final Logger log = Logger.getLogger(AccumuloExecuteSparqlQuery.class);
+
+ private final InstanceExists instanceExists;
+
+ /**
+ * Constructs an instance of {@link AccumuloExecuteSparqlQuery}.
+ *
+ * @param connectionDetails - Details about the values that were used to create
+ * the connector to the cluster. (not null)
+ * @param connector - Provides programmatic access to the instance of Accumulo
+ * that hosts Rya instance. (not null)
+ */
+ public AccumuloExecuteSparqlQuery(final AccumuloConnectionDetails connectionDetails, final Connector connector) {
+ super(connectionDetails, connector);
+ instanceExists = new AccumuloInstanceExists(connectionDetails, connector);
+ }
+
+
+ @Override
+ public String executeSparqlQuery(final String ryaInstanceName, final String sparqlQuery)
+ throws InstanceDoesNotExistException, RyaClientException {
+ requireNonNull(ryaInstanceName);
+ requireNonNull(sparqlQuery);
+
+ // Ensure the Rya Instance exists.
+ if(!instanceExists.exists(ryaInstanceName)) {
+ throw new InstanceDoesNotExistException(String.format("There is no Rya instance named '%s'.", ryaInstanceName));
+ }
+
+
+ Sail sail = null;
+ SailRepository sailRepo = null;
+ SailRepositoryConnection sailRepoConn = null;
+
+ try {
+ // Get a Sail object that is connected to the Rya instance.
+ final AccumuloRdfConfiguration ryaConf = getAccumuloConnectionDetails().buildAccumuloRdfConfiguration(ryaInstanceName);
+ sail = RyaSailFactory.getInstance(ryaConf);
+
+ // Load the file.
+ sailRepo = new SailRepository(sail);
+ sailRepoConn = sailRepo.getConnection();
+ final long start = System.currentTimeMillis();
+ final TupleQuery tupleQuery = sailRepoConn.prepareTupleQuery(QueryLanguage.SPARQL, sparqlQuery);
+ final TupleQueryResult result = tupleQuery.evaluate();
+ final StringBuilder sb = new StringBuilder();
+ final String newline = "\n";
+ sb.append("Query Result:").append(newline);
+ sb.append("Binding Names: ").append(result.getBindingNames().toString()).append(newline);
+
+ int count = 0;
+ while(result.hasNext()) {
+ sb.append(result.next().toString()).append(newline);
+ count++;
+ }
+ final String seconds = new DecimalFormat("0.0##").format((System.currentTimeMillis() - start) / 1000.0);
+ sb.append("Retrieved ").append(count).append(" results in ").append(seconds).append(" seconds.");
--- End diff --
I think it is appropriate. The contract specifies that this method will return "A user-friendly String representation of the query results." If you use other query shells, like impala, they also provide query execution timing. It provides basic feedback to the user for how performant their query is. If you prefer, we could move the timing section to rya.console and append it to the content that is returned from this method. Let me know which you prefer, but I really like seeing timing when I issue a query.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-rya issue #188: RYA-318 Rya Shell improvements for packaging, load...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit commented on the issue:
https://github.com/apache/incubator-rya/pull/188
Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/incubator-rya-master-with-optionals-pull-requests/301/<h2>Build result: FAILURE</span></h2>[...truncated 4.70 MB...] Compare (>) Var (name=t) ValueConstant (value="8"^^<http://www.w3.org/2001/XMLSchema#integer>) Join Join StatementPattern Var (name=s) Var (name=-const-http://www.w3.org/1999/02/22-rdf-syntax-ns#type, value=http://www.w3.org/1999/02/22-rdf-syntax-ns#type, anonymous) Var (name=t) StatementPattern Var (name=t) Var (name=-const-http://www.w3.org/2000/01/rdf-schema#label, value=http://www.w3.org/2000/01/rdf-schema#label, anonymous) Var (name=u) StatementPattern Var (name=u) Var (name=-const-uri:talksTo, value=uri:talksTo, anonymous) Var (name=s)Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.009 sec - in org.apache.rya.indexing.IndexPl
anValidator.VarConstantIndexListPrunerTestRunning org.apache.rya.indexing.IndexPlanValidator.ThreshholdPlanSelectorTestERROR: Maven JVM terminated unexpectedly with exit code 137Setting status of 09968c0e596986e762ad0310f00047e6a2f638ec to FAILURE with url https://builds.apache.org/job/incubator-rya-master-with-optionals-pull-requests/301/ and message: 'FAILURE 'Using context: Jenkins: clean package -Pgeoindexing
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-rya pull request #188: RYA-318 Rya Shell improvements for packagin...
Posted by kchilton2 <gi...@git.apache.org>.
Github user kchilton2 commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/188#discussion_r130147195
--- Diff: extras/rya.console/src/main/java/org/apache/rya/shell/RyaAdminCommands.java ---
@@ -200,8 +201,65 @@ public String install() {
}
}
- @CliCommand(value = GET_INSTANCE_DETAILS_CMD, help = "Print information about how the Rya instance is configured.")
- public String getInstanceDetails() {
+ @CliCommand(value = INSTALL_PARAMETERS_CMD, help = "Create a new instance of Rya with command line parameters.")
+ public String installWithParameters(
+ @CliOption(key = {"instanceName"}, mandatory = true, help = "The name of the Rya instance to create.")
+ final String instanceName,
+
+ @CliOption(key = {"enableTableHashPrefix"}, mandatory = false, help = "Use Shard Balancing (improves streamed input write speeds).", unspecifiedDefaultValue = "false", specifiedDefaultValue = "true")
+ final boolean enableTableHashPrefix,
+
+ @CliOption(key = {"enableEntityCentricIndex"}, mandatory = false, help = "Use Entity Centric Indexing.", unspecifiedDefaultValue = "false", specifiedDefaultValue = "true")
+ final boolean enableEntityCentricIndex,
+
+ @CliOption(key = {"enableFreeTextIndex"}, mandatory = false, help = "Use Free Text Indexing.", unspecifiedDefaultValue = "false", specifiedDefaultValue = "true")
+ final boolean enableFreeTextIndex,
+
+ @CliOption(key = {"enableGeospatialIndex"}, mandatory = false, help = "Use Geospatial Indexing.", unspecifiedDefaultValue = "false", specifiedDefaultValue = "true")
+ final boolean enableGeospatialIndex,
+
+ @CliOption(key = {"enableTemporalIndex"}, mandatory = false, help = "Use Temporal Indexing.", unspecifiedDefaultValue = "false", specifiedDefaultValue = "true")
+ final boolean enableTemporalIndex,
+
+ @CliOption(key = {"enablePcjIndex"}, mandatory = false, help = "Use Precomputed Join (PCJ) Indexing.", unspecifiedDefaultValue = "false", specifiedDefaultValue = "true")
+ final boolean enablePcjIndex,
+
+ @CliOption(key = {"fluoPcjAppName"}, mandatory = false, help = "Fluo Application Name for PCJ Index Updater (fluo app must be initialized and enablePcjIndex=true).")
--- End diff --
Likewise, different command line options should be available depending on if you're installing to Mongo or Accumulo. Yea I basically just said what you said.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-rya pull request #188: RYA-318 Rya Shell improvements for packagin...
Posted by jdasch <gi...@git.apache.org>.
Github user jdasch commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/188#discussion_r130092402
--- Diff: extras/rya.console/src/main/java/org/apache/rya/shell/RyaAdminCommands.java ---
@@ -200,8 +201,65 @@ public String install() {
}
}
- @CliCommand(value = GET_INSTANCE_DETAILS_CMD, help = "Print information about how the Rya instance is configured.")
- public String getInstanceDetails() {
+ @CliCommand(value = INSTALL_PARAMETERS_CMD, help = "Create a new instance of Rya with command line parameters.")
+ public String installWithParameters(
+ @CliOption(key = {"instanceName"}, mandatory = true, help = "The name of the Rya instance to create.")
+ final String instanceName,
+
+ @CliOption(key = {"enableTableHashPrefix"}, mandatory = false, help = "Use Shard Balancing (improves streamed input write speeds).", unspecifiedDefaultValue = "false", specifiedDefaultValue = "true")
+ final boolean enableTableHashPrefix,
+
+ @CliOption(key = {"enableEntityCentricIndex"}, mandatory = false, help = "Use Entity Centric Indexing.", unspecifiedDefaultValue = "false", specifiedDefaultValue = "true")
+ final boolean enableEntityCentricIndex,
+
+ @CliOption(key = {"enableFreeTextIndex"}, mandatory = false, help = "Use Free Text Indexing.", unspecifiedDefaultValue = "false", specifiedDefaultValue = "true")
+ final boolean enableFreeTextIndex,
+
+ @CliOption(key = {"enableGeospatialIndex"}, mandatory = false, help = "Use Geospatial Indexing.", unspecifiedDefaultValue = "false", specifiedDefaultValue = "true")
+ final boolean enableGeospatialIndex,
+
+ @CliOption(key = {"enableTemporalIndex"}, mandatory = false, help = "Use Temporal Indexing.", unspecifiedDefaultValue = "false", specifiedDefaultValue = "true")
+ final boolean enableTemporalIndex,
+
+ @CliOption(key = {"enablePcjIndex"}, mandatory = false, help = "Use Precomputed Join (PCJ) Indexing.", unspecifiedDefaultValue = "false", specifiedDefaultValue = "true")
+ final boolean enablePcjIndex,
+
+ @CliOption(key = {"fluoPcjAppName"}, mandatory = false, help = "Fluo Application Name for PCJ Index Updater (fluo app must be initialized and enablePcjIndex=true).")
--- End diff --
I agree that it is Accumulo specific, but what I'm doing here is no different than the interactive code that was already in the InstallPrompt. When Mongo support is added, we can label these features as Accumulo only in the command help, maybe even prefix the names to make it more obvious -- and they'll fail immediately if the user tries to use them. The interactive command can just hide invalid options from the user.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-rya pull request #188: RYA-318 Rya Shell improvements for packagin...
Posted by pujav65 <gi...@git.apache.org>.
Github user pujav65 commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/188#discussion_r129999701
--- Diff: extras/rya.console/src/main/examples/lubm.script ---
@@ -0,0 +1,39 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+connect-accumulo --username accumulo_user --instanceName accumulo_instance --zookeepers zoo1,zoo2,zoo3,zoo4,zoo5
--- End diff --
this might be more appropriate as an example or something rather than checked in here. also i remember there being some concern with creating distribution artifacts -- i think at one point we were told to try to avoid due to licensing concerns. I don't remember the details though.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-rya pull request #188: RYA-318 Rya Shell improvements for packagin...
Posted by jdasch <gi...@git.apache.org>.
Github user jdasch commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/188#discussion_r130113701
--- Diff: common/rya.api/src/main/java/org/apache/rya/api/client/LoadStatementsFile.java ---
@@ -31,14 +31,16 @@
@DefaultAnnotation(NonNull.class)
public interface LoadStatementsFile {
- /**
- * Loads a local file of RDF statements into an instance of Rya.
- *
- * @param ryaInstanceName - The name of the Rya instance the statements will be loaded into. (not null)
- * @param statementsFile - A file that holds RDF statements that will be loaded. (not null)
- * @param format - The format of the statements file. (not null)
- * @throws InstanceDoesNotExistException No instance of Rya exists for the provided name.
- * @throws RyaClientException Something caused the command to fail.
- */
- public void loadStatements(String ryaInstanceName, Path statementsFile, RDFFormat format) throws InstanceDoesNotExistException, RyaClientException;
+ /**
+ * Loads a local file of RDF statements into an instance of Rya.
+ *
+ * @param ryaInstanceName - The name of the Rya instance the statements will be loaded into. (not null)
+ * @param statementsFile - A file that holds RDF statements that will be loaded. (not null)
+ * @param format - The format of the statements file. (not null)
+ * @param flushEachUpdate - If true, each statement will be flushed individually. If false, statements will be
+ * grouped in an unspecified manner. A value of false will typically yield higher ingest rates.
+ * @throws InstanceDoesNotExistException No instance of Rya exists for the provided name.
+ * @throws RyaClientException Something caused the command to fail.
+ */
+ public void loadStatements(String ryaInstanceName, Path statementsFile, RDFFormat format, boolean flushEachUpdate) throws InstanceDoesNotExistException, RyaClientException;
--- End diff --
Agree. It would be nice to have the option to override default connection configs without disconnecting and reconnecting - especially for the case where the user wants to quickly visualize the difference in results when loading/querying data with different visibilities/authorizations. Would require an API change which is out of scope for this.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-rya pull request #188: RYA-318 Rya Shell improvements for packagin...
Posted by jdasch <gi...@git.apache.org>.
Github user jdasch commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/188#discussion_r130091818
--- Diff: extras/indexing/src/main/java/org/apache/rya/api/client/accumulo/AccumuloExecuteSparqlQuery.java ---
@@ -0,0 +1,150 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.rya.api.client.accumulo;
+
+import static java.util.Objects.requireNonNull;
+
+import java.text.DecimalFormat;
+
+import org.apache.accumulo.core.client.AccumuloException;
+import org.apache.accumulo.core.client.AccumuloSecurityException;
+import org.apache.accumulo.core.client.Connector;
+import org.apache.log4j.Logger;
+import org.apache.rya.accumulo.AccumuloRdfConfiguration;
+import org.apache.rya.api.client.ExecuteSparqlQuery;
+import org.apache.rya.api.client.InstanceDoesNotExistException;
+import org.apache.rya.api.client.InstanceExists;
+import org.apache.rya.api.client.LoadStatementsFile;
+import org.apache.rya.api.client.RyaClientException;
+import org.apache.rya.api.persist.RyaDAOException;
+import org.apache.rya.rdftriplestore.inference.InferenceEngineException;
+import org.apache.rya.sail.config.RyaSailFactory;
+import org.openrdf.query.MalformedQueryException;
+import org.openrdf.query.QueryEvaluationException;
+import org.openrdf.query.QueryLanguage;
+import org.openrdf.query.TupleQuery;
+import org.openrdf.query.TupleQueryResult;
+import org.openrdf.repository.RepositoryException;
+import org.openrdf.repository.sail.SailRepository;
+import org.openrdf.repository.sail.SailRepositoryConnection;
+import org.openrdf.sail.Sail;
+import org.openrdf.sail.SailException;
+
+import edu.umd.cs.findbugs.annotations.DefaultAnnotation;
+import edu.umd.cs.findbugs.annotations.NonNull;
+
+/**
+ * An Accumulo implementation of the {@link LoadStatementsFile} command.
+ */
+@DefaultAnnotation(NonNull.class)
+public class AccumuloExecuteSparqlQuery extends AccumuloCommand implements ExecuteSparqlQuery {
+ private static final Logger log = Logger.getLogger(AccumuloExecuteSparqlQuery.class);
+
+ private final InstanceExists instanceExists;
+
+ /**
+ * Constructs an instance of {@link AccumuloExecuteSparqlQuery}.
+ *
+ * @param connectionDetails - Details about the values that were used to create
+ * the connector to the cluster. (not null)
+ * @param connector - Provides programmatic access to the instance of Accumulo
+ * that hosts Rya instance. (not null)
+ */
+ public AccumuloExecuteSparqlQuery(final AccumuloConnectionDetails connectionDetails, final Connector connector) {
+ super(connectionDetails, connector);
+ instanceExists = new AccumuloInstanceExists(connectionDetails, connector);
+ }
+
+
+ @Override
+ public String executeSparqlQuery(final String ryaInstanceName, final String sparqlQuery)
+ throws InstanceDoesNotExistException, RyaClientException {
+ requireNonNull(ryaInstanceName);
+ requireNonNull(sparqlQuery);
+
+ // Ensure the Rya Instance exists.
+ if(!instanceExists.exists(ryaInstanceName)) {
+ throw new InstanceDoesNotExistException(String.format("There is no Rya instance named '%s'.", ryaInstanceName));
+ }
+
+
+ Sail sail = null;
+ SailRepository sailRepo = null;
+ SailRepositoryConnection sailRepoConn = null;
+
+ try {
+ // Get a Sail object that is connected to the Rya instance.
+ final AccumuloRdfConfiguration ryaConf = getAccumuloConnectionDetails().buildAccumuloRdfConfiguration(ryaInstanceName);
+ sail = RyaSailFactory.getInstance(ryaConf);
+
+ // Load the file.
+ sailRepo = new SailRepository(sail);
+ sailRepoConn = sailRepo.getConnection();
+ final long start = System.currentTimeMillis();
+ final TupleQuery tupleQuery = sailRepoConn.prepareTupleQuery(QueryLanguage.SPARQL, sparqlQuery);
+ final TupleQueryResult result = tupleQuery.evaluate();
+ final StringBuilder sb = new StringBuilder();
+ final String newline = "\n";
+ sb.append("Query Result:").append(newline);
+ sb.append("Binding Names: ").append(result.getBindingNames().toString()).append(newline);
+
+ int count = 0;
+ while(result.hasNext()) {
--- End diff --
Thanks, I'll make a tweak here.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-rya pull request #188: RYA-318 Rya Shell improvements for packagin...
Posted by amihalik <gi...@git.apache.org>.
Github user amihalik commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/188#discussion_r130093053
--- Diff: extras/indexing/src/main/java/org/apache/rya/api/client/accumulo/AccumuloExecuteSparqlQuery.java ---
@@ -0,0 +1,150 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.rya.api.client.accumulo;
+
+import static java.util.Objects.requireNonNull;
+
+import java.text.DecimalFormat;
+
+import org.apache.accumulo.core.client.AccumuloException;
+import org.apache.accumulo.core.client.AccumuloSecurityException;
+import org.apache.accumulo.core.client.Connector;
+import org.apache.log4j.Logger;
+import org.apache.rya.accumulo.AccumuloRdfConfiguration;
+import org.apache.rya.api.client.ExecuteSparqlQuery;
+import org.apache.rya.api.client.InstanceDoesNotExistException;
+import org.apache.rya.api.client.InstanceExists;
+import org.apache.rya.api.client.LoadStatementsFile;
+import org.apache.rya.api.client.RyaClientException;
+import org.apache.rya.api.persist.RyaDAOException;
+import org.apache.rya.rdftriplestore.inference.InferenceEngineException;
+import org.apache.rya.sail.config.RyaSailFactory;
+import org.openrdf.query.MalformedQueryException;
+import org.openrdf.query.QueryEvaluationException;
+import org.openrdf.query.QueryLanguage;
+import org.openrdf.query.TupleQuery;
+import org.openrdf.query.TupleQueryResult;
+import org.openrdf.repository.RepositoryException;
+import org.openrdf.repository.sail.SailRepository;
+import org.openrdf.repository.sail.SailRepositoryConnection;
+import org.openrdf.sail.Sail;
+import org.openrdf.sail.SailException;
+
+import edu.umd.cs.findbugs.annotations.DefaultAnnotation;
+import edu.umd.cs.findbugs.annotations.NonNull;
+
+/**
+ * An Accumulo implementation of the {@link LoadStatementsFile} command.
+ */
+@DefaultAnnotation(NonNull.class)
+public class AccumuloExecuteSparqlQuery extends AccumuloCommand implements ExecuteSparqlQuery {
+ private static final Logger log = Logger.getLogger(AccumuloExecuteSparqlQuery.class);
+
+ private final InstanceExists instanceExists;
+
+ /**
+ * Constructs an instance of {@link AccumuloExecuteSparqlQuery}.
+ *
+ * @param connectionDetails - Details about the values that were used to create
+ * the connector to the cluster. (not null)
+ * @param connector - Provides programmatic access to the instance of Accumulo
+ * that hosts Rya instance. (not null)
+ */
+ public AccumuloExecuteSparqlQuery(final AccumuloConnectionDetails connectionDetails, final Connector connector) {
+ super(connectionDetails, connector);
+ instanceExists = new AccumuloInstanceExists(connectionDetails, connector);
+ }
+
+
+ @Override
+ public String executeSparqlQuery(final String ryaInstanceName, final String sparqlQuery)
+ throws InstanceDoesNotExistException, RyaClientException {
+ requireNonNull(ryaInstanceName);
+ requireNonNull(sparqlQuery);
+
+ // Ensure the Rya Instance exists.
+ if(!instanceExists.exists(ryaInstanceName)) {
+ throw new InstanceDoesNotExistException(String.format("There is no Rya instance named '%s'.", ryaInstanceName));
+ }
+
+
+ Sail sail = null;
+ SailRepository sailRepo = null;
+ SailRepositoryConnection sailRepoConn = null;
+
+ try {
+ // Get a Sail object that is connected to the Rya instance.
+ final AccumuloRdfConfiguration ryaConf = getAccumuloConnectionDetails().buildAccumuloRdfConfiguration(ryaInstanceName);
+ sail = RyaSailFactory.getInstance(ryaConf);
+
+ // Load the file.
+ sailRepo = new SailRepository(sail);
+ sailRepoConn = sailRepo.getConnection();
+ final long start = System.currentTimeMillis();
+ final TupleQuery tupleQuery = sailRepoConn.prepareTupleQuery(QueryLanguage.SPARQL, sparqlQuery);
+ final TupleQueryResult result = tupleQuery.evaluate();
+ final StringBuilder sb = new StringBuilder();
+ final String newline = "\n";
+ sb.append("Query Result:").append(newline);
+ sb.append("Binding Names: ").append(result.getBindingNames().toString()).append(newline);
+
+ int count = 0;
+ while(result.hasNext()) {
+ sb.append(result.next().toString()).append(newline);
+ count++;
+ }
+ final String seconds = new DecimalFormat("0.0##").format((System.currentTimeMillis() - start) / 1000.0);
+ sb.append("Retrieved ").append(count).append(" results in ").append(seconds).append(" seconds.");
--- End diff --
I'd prefer you leave it. I use it all the time when I'm tailing the rya web logs. I thought it was a good addition.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-rya issue #188: RYA-318 Rya Shell improvements for packaging, load...
Posted by jdasch <gi...@git.apache.org>.
Github user jdasch commented on the issue:
https://github.com/apache/incubator-rya/pull/188
asfbot build
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-rya issue #188: RYA-318 Rya Shell improvements for packaging, load...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit commented on the issue:
https://github.com/apache/incubator-rya/pull/188
Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/incubator-rya-master-with-optionals-pull-requests/305/<h2>Build result: ABORTED</span></h2>[...truncated 5.04 MB...][INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ rya.console ---[WARNING] Failed to notify spy hudson.maven.Maven3Builder$JenkinsEventSpy: java.util.concurrent.ExecutionException: Invalid object ID 27 iota=55[INFO] [INFO] --- apache-rat-plugin:0.11:check (check-licenses) @ rya.console ---[INFO] 51 implicit excludes (use -debug for more details).[INFO] Exclude: spring-shell.log[INFO] 37 resources included (use -debug for more details)[INFO] Rat check: Summary of files. Unapproved: 0 unknown: 0 generated: 0 approved: 36 licence.[WARNING] Failed to notify spy hudson.maven.Maven3Builder$JenkinsEventSpy: java.util.concurrent.ExecutionException: Invalid object ID 27 iota=55[INFO] [INFO] --- maven-enforcer-plugin:1.3.1:enforce (enforce-mvn) @ rya.console ---[WARNING] Failed to notify spy hudson.maven.Maven3Builder$JenkinsEventSpy: jav
a.util.concurrent.ExecutionException: Invalid object ID 27 iota=55[INFO] [INFO] --- jacoco-maven-plugin:0.7.9:prepare-agent (default-prepare-agent) @ rya.console ---[INFO] argLine set to -javaagent:/home/jenkins/jenkins-slave/maven-repositories/1/org/jacoco/org.jacoco.agent/0.7.9/org.jacoco.agent-0.7.9-runtime.jar=destfile=/home/jenkins/jenkins-slave/workspace/incubator-rya-master-with-optionals-pull-requests/extras/rya.console/target/jacoco.exec[WARNING] Failed to notify spy hudson.maven.Maven3Builder$JenkinsEventSpy: java.util.concurrent.ExecutionException: Invalid object ID 27 iota=55[INFO] [INFO] --- license-maven-plugin:2.6:format (default) @ rya.console ---Build was aborted[INFO] Updating license headers...channel stoppedSetting status of 52b5bb0b2844a249b3bf2dd5f8831b7f2d898621 to FAILURE with url https://builds.apache.org/job/incubator-rya-master-with-optionals-pull-requests/305/ and message: 'FAILURE 'Using context: Jenkins: clean package -Pgeoindexing
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-rya pull request #188: RYA-318 Rya Shell improvements for packagin...
Posted by jdasch <gi...@git.apache.org>.
Github user jdasch commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/188#discussion_r130091678
--- Diff: extras/indexing/src/main/java/org/apache/rya/api/client/accumulo/AccumuloExecuteSparqlQuery.java ---
@@ -0,0 +1,150 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.rya.api.client.accumulo;
+
+import static java.util.Objects.requireNonNull;
+
+import java.text.DecimalFormat;
+
+import org.apache.accumulo.core.client.AccumuloException;
+import org.apache.accumulo.core.client.AccumuloSecurityException;
+import org.apache.accumulo.core.client.Connector;
+import org.apache.log4j.Logger;
+import org.apache.rya.accumulo.AccumuloRdfConfiguration;
+import org.apache.rya.api.client.ExecuteSparqlQuery;
+import org.apache.rya.api.client.InstanceDoesNotExistException;
+import org.apache.rya.api.client.InstanceExists;
+import org.apache.rya.api.client.LoadStatementsFile;
+import org.apache.rya.api.client.RyaClientException;
+import org.apache.rya.api.persist.RyaDAOException;
+import org.apache.rya.rdftriplestore.inference.InferenceEngineException;
+import org.apache.rya.sail.config.RyaSailFactory;
+import org.openrdf.query.MalformedQueryException;
+import org.openrdf.query.QueryEvaluationException;
+import org.openrdf.query.QueryLanguage;
+import org.openrdf.query.TupleQuery;
+import org.openrdf.query.TupleQueryResult;
+import org.openrdf.repository.RepositoryException;
+import org.openrdf.repository.sail.SailRepository;
+import org.openrdf.repository.sail.SailRepositoryConnection;
+import org.openrdf.sail.Sail;
+import org.openrdf.sail.SailException;
+
+import edu.umd.cs.findbugs.annotations.DefaultAnnotation;
+import edu.umd.cs.findbugs.annotations.NonNull;
+
+/**
+ * An Accumulo implementation of the {@link LoadStatementsFile} command.
--- End diff --
Good catch.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-rya issue #188: RYA-318 Rya Shell improvements for packaging, load...
Posted by jdasch <gi...@git.apache.org>.
Github user jdasch commented on the issue:
https://github.com/apache/incubator-rya/pull/188
Just squashed everything and rebased on master.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-rya issue #188: RYA-318 Rya Shell improvements for packaging, load...
Posted by jdasch <gi...@git.apache.org>.
Github user jdasch commented on the issue:
https://github.com/apache/incubator-rya/pull/188
asfbot build
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-rya pull request #188: RYA-318 Rya Shell improvements for packagin...
Posted by meiercaleb <gi...@git.apache.org>.
Github user meiercaleb commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/188#discussion_r130108687
--- Diff: common/rya.api/src/main/java/org/apache/rya/api/client/LoadStatementsFile.java ---
@@ -31,14 +31,16 @@
@DefaultAnnotation(NonNull.class)
public interface LoadStatementsFile {
- /**
- * Loads a local file of RDF statements into an instance of Rya.
- *
- * @param ryaInstanceName - The name of the Rya instance the statements will be loaded into. (not null)
- * @param statementsFile - A file that holds RDF statements that will be loaded. (not null)
- * @param format - The format of the statements file. (not null)
- * @throws InstanceDoesNotExistException No instance of Rya exists for the provided name.
- * @throws RyaClientException Something caused the command to fail.
- */
- public void loadStatements(String ryaInstanceName, Path statementsFile, RDFFormat format) throws InstanceDoesNotExistException, RyaClientException;
+ /**
+ * Loads a local file of RDF statements into an instance of Rya.
+ *
+ * @param ryaInstanceName - The name of the Rya instance the statements will be loaded into. (not null)
+ * @param statementsFile - A file that holds RDF statements that will be loaded. (not null)
+ * @param format - The format of the statements file. (not null)
+ * @param flushEachUpdate - If true, each statement will be flushed individually. If false, statements will be
+ * grouped in an unspecified manner. A value of false will typically yield higher ingest rates.
+ * @throws InstanceDoesNotExistException No instance of Rya exists for the provided name.
+ * @throws RyaClientException Something caused the command to fail.
+ */
+ public void loadStatements(String ryaInstanceName, Path statementsFile, RDFFormat format, boolean flushEachUpdate) throws InstanceDoesNotExistException, RyaClientException;
--- End diff --
I can see value in exposing the flush parameter and other parameters to the user that are specific to the execution instance (as opposed to the particular connection instance). In the case of the flush parameter, a user might be more concerned with higher ingest and less concerned with having the table accurately reflect the data during ingest. On the other hand, when ingesting another file, they might want the table to immediately represent their data. In general, I think that there are a number of execution instance parameters that a user should be able to specify for some of the shell commands. While issuing a query through the shell, the user should be able to indicate which indexers they want to use and whether or not they want to use PCJs. These are all execution specific parameters. Unfortunately, the current API does not support this, and a user has to reconnect each time they want to change their query configuration or ingest configuration. Might be best to write
up a Jira ticket for this and make it a wish for a new feature.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-rya pull request #188: RYA-318 Rya Shell improvements for packagin...
Posted by jdasch <gi...@git.apache.org>.
Github user jdasch commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/188#discussion_r130163790
--- Diff: common/rya.api/src/main/java/org/apache/rya/api/client/ExecuteSparqlQuery.java ---
@@ -0,0 +1,39 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.rya.api.client;
+
+import edu.umd.cs.findbugs.annotations.DefaultAnnotation;
+import edu.umd.cs.findbugs.annotations.NonNull;
+
+/**
+ * Loads a SPARQL Query and executes the query against an instance of Rya.
+ */
+@DefaultAnnotation(NonNull.class)
+public interface ExecuteSparqlQuery {
+ /**
+ * Loads a SPARQL Query and executes the query against an instance of Rya.
+ *
+ * @param ryaInstanceName - The name of the Rya instance the query will be executed against. (not null)
+ * @param sparqlQuery - A single SPARQL Query. (not null)
+ * @return A user-friendly String representation of the query results.
--- End diff --
Captured in https://issues.apache.org/jira/browse/RYA-328
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-rya pull request #188: RYA-318 Rya Shell improvements for packagin...
Posted by amihalik <gi...@git.apache.org>.
Github user amihalik commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/188#discussion_r129998856
--- Diff: extras/indexing/src/main/java/org/apache/rya/api/client/accumulo/AccumuloExecuteSparqlQuery.java ---
@@ -0,0 +1,150 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.rya.api.client.accumulo;
+
+import static java.util.Objects.requireNonNull;
+
+import java.text.DecimalFormat;
+
+import org.apache.accumulo.core.client.AccumuloException;
+import org.apache.accumulo.core.client.AccumuloSecurityException;
+import org.apache.accumulo.core.client.Connector;
+import org.apache.log4j.Logger;
+import org.apache.rya.accumulo.AccumuloRdfConfiguration;
+import org.apache.rya.api.client.ExecuteSparqlQuery;
+import org.apache.rya.api.client.InstanceDoesNotExistException;
+import org.apache.rya.api.client.InstanceExists;
+import org.apache.rya.api.client.LoadStatementsFile;
+import org.apache.rya.api.client.RyaClientException;
+import org.apache.rya.api.persist.RyaDAOException;
+import org.apache.rya.rdftriplestore.inference.InferenceEngineException;
+import org.apache.rya.sail.config.RyaSailFactory;
+import org.openrdf.query.MalformedQueryException;
+import org.openrdf.query.QueryEvaluationException;
+import org.openrdf.query.QueryLanguage;
+import org.openrdf.query.TupleQuery;
+import org.openrdf.query.TupleQueryResult;
+import org.openrdf.repository.RepositoryException;
+import org.openrdf.repository.sail.SailRepository;
+import org.openrdf.repository.sail.SailRepositoryConnection;
+import org.openrdf.sail.Sail;
+import org.openrdf.sail.SailException;
+
+import edu.umd.cs.findbugs.annotations.DefaultAnnotation;
+import edu.umd.cs.findbugs.annotations.NonNull;
+
+/**
+ * An Accumulo implementation of the {@link LoadStatementsFile} command.
--- End diff --
ExecuteSparqlQuery?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-rya pull request #188: RYA-318 Rya Shell improvements for packagin...
Posted by pujav65 <gi...@git.apache.org>.
Github user pujav65 commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/188#discussion_r129999335
--- Diff: common/rya.api/src/main/java/org/apache/rya/api/client/LoadStatementsFile.java ---
@@ -31,14 +31,16 @@
@DefaultAnnotation(NonNull.class)
public interface LoadStatementsFile {
- /**
- * Loads a local file of RDF statements into an instance of Rya.
- *
- * @param ryaInstanceName - The name of the Rya instance the statements will be loaded into. (not null)
- * @param statementsFile - A file that holds RDF statements that will be loaded. (not null)
- * @param format - The format of the statements file. (not null)
- * @throws InstanceDoesNotExistException No instance of Rya exists for the provided name.
- * @throws RyaClientException Something caused the command to fail.
- */
- public void loadStatements(String ryaInstanceName, Path statementsFile, RDFFormat format) throws InstanceDoesNotExistException, RyaClientException;
+ /**
+ * Loads a local file of RDF statements into an instance of Rya.
+ *
+ * @param ryaInstanceName - The name of the Rya instance the statements will be loaded into. (not null)
+ * @param statementsFile - A file that holds RDF statements that will be loaded. (not null)
+ * @param format - The format of the statements file. (not null)
+ * @param flushEachUpdate - If true, each statement will be flushed individually. If false, statements will be
+ * grouped in an unspecified manner. A value of false will typically yield higher ingest rates.
+ * @throws InstanceDoesNotExistException No instance of Rya exists for the provided name.
+ * @throws RyaClientException Something caused the command to fail.
+ */
+ public void loadStatements(String ryaInstanceName, Path statementsFile, RDFFormat format, boolean flushEachUpdate) throws InstanceDoesNotExistException, RyaClientException;
--- End diff --
why was the flush parameter exposed?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---