You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@rya.apache.org by kchilton2 <gi...@git.apache.org> on 2016/08/01 23:43:35 UTC
[GitHub] incubator-rya pull request #60: Rya 81
GitHub user kchilton2 opened a pull request:
https://github.com/apache/incubator-rya/pull/60
Rya 81
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/kchilton2/incubator-rya RYA-81
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-rya/pull/60.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #60
----
commit 78a087d091f953106f6b4ee702097cbb3d957c3f
Author: Kevin Chilton <ke...@parsons.com>
Date: 2016-06-04T21:46:43Z
RYA-81 Added a structure that provides metadata about an instance of Rya as well as a repository that may be used to store and update it. This information could be used to configure clients when querying the instance.
commit e18df668356dbf3b707b8a5b15b6156a30a7c977
Author: isper3at <sm...@gmail.com>
Date: 2016-06-08T06:37:20Z
RYA-84 Add Mongo Support to Admin Table
Implemented admin table interfaces with MongoDB.
Added Test coverage for the Mongo object to RyaDetails
adapter and integration tests for the repo.
commit 4999e4b82ba6bc0b289fd8bfc2038a475dcc45b2
Author: Kevin Chilton <ke...@parsons.com>
Date: 2016-06-28T20:22:54Z
RYA-81 Converting the PCJ Details to be a map for easier lookup.
commit 3a48dcd01adca8fa8d6eac4ea712126872bdb380
Author: isper3at <sm...@gmail.com>
Date: 2016-06-10T20:51:43Z
RYA-85 Configuration glue code
Added a configuration utility for updating the
configuration used in Rya based on what is
in RyaDetails, the admin table.
Added integration examples, these will need to
be updated to use JUnit
commit 26ad30830d7118bf89c319e3d00277051462de0c
Author: Kevin Chilton <ke...@parsons.com>
Date: 2016-07-20T01:33:29Z
RYA-81 Fixed a bug in the RyaSailFactory where the Rya DAO wasn't been placed within the store object and where the Accumulo table names were not matching the table prefix.
commit d9be0fb2283c65a6d86dda9ac0115c4b1e55d303
Author: Caleb Meier <me...@gmail.com>
Date: 2016-07-21T20:05:28Z
RYA-99 Use PrecomputedJoinStorage to get Pcj tables instead of TableOperations
commit d08a8ee3197c19f2dae625bcec269675704e5fbe
Author: David W. Lotts <da...@parsons.com>
Date: 2016-07-28T04:19:17Z
RYA-112 Integrate RyaDetails with secondary indexers
commit 967ee5fbc18a5eaa97be4664ce41738acc6481ba
Author: Kevin Chilton <ke...@parsons.com>
Date: 2016-08-01T22:16:15Z
RYA-81 Making tests pass.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-rya pull request #60: Rya 81
Posted by isper3at <gi...@git.apache.org>.
Github user isper3at commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/60#discussion_r73414516
--- Diff: common/rya.api/src/main/java/mvm/rya/api/instance/RyaDetails.java ---
@@ -0,0 +1,908 @@
+package mvm.rya.api.instance;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import static java.util.Objects.requireNonNull;
+
+import java.io.Serializable;
+import java.util.Date;
+import java.util.Objects;
+
+import javax.annotation.Nullable;
+import javax.annotation.ParametersAreNonnullByDefault;
+import javax.annotation.concurrent.Immutable;
+
+import com.google.common.base.Optional;
+import com.google.common.collect.ImmutableList;
+
+/**
+ * Details about how a Rya instance's state.
--- End diff --
wording is funny
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-rya pull request #60: Rya 81
Posted by isper3at <gi...@git.apache.org>.
Github user isper3at commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/60#discussion_r73414878
--- Diff: common/rya.api/src/main/java/mvm/rya/api/instance/RyaDetails.java ---
@@ -0,0 +1,908 @@
+package mvm.rya.api.instance;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import static java.util.Objects.requireNonNull;
+
+import java.io.Serializable;
+import java.util.Date;
+import java.util.Objects;
+
+import javax.annotation.Nullable;
+import javax.annotation.ParametersAreNonnullByDefault;
+import javax.annotation.concurrent.Immutable;
+
+import com.google.common.base.Optional;
+import com.google.common.collect.ImmutableList;
+
+/**
+ * Details about how a Rya instance's state.
+ */
+@Immutable
+@ParametersAreNonnullByDefault
+public class RyaDetails implements Serializable {
+ private static final long serialVersionUID = 1L;
+
+ // General metadata about the instance.
+ private final String instanceName;
+ private final String version;
+
+ // Secondary Index Details.
+ private final EntityCentricIndexDetails entityCentricDetails;
+ private final GeoIndexDetails geoDetails;
+ private final PCJIndexDetails pcjDetails;
+ private final TemporalIndexDetails temporalDetails;
+ private final FreeTextIndexDetails freeTextDetails;
+
+ // Statistics Details.
+ private final ProspectorDetails prospectorDetails;
+ private final JoinSelectivityDetails joinSelectivityDetails;
+
+ /**
+ * Private to prevent initialization through the constructor. To build
+ * instances of this class, use the {@link Builder}.
+ */
+ private RyaDetails(
+ final String instanceName,
+ final String version,
+ final EntityCentricIndexDetails entityCentricDetails,
+ final GeoIndexDetails geoDetails,
+ final PCJIndexDetails pcjDetails,
+ final TemporalIndexDetails temporalDetails,
+ final FreeTextIndexDetails freeTextDetails,
+ final ProspectorDetails prospectorDetails,
+ final JoinSelectivityDetails joinSelectivityDetails) {
+ this.instanceName = requireNonNull(instanceName);
+ this.version = requireNonNull(version);
+ this.entityCentricDetails = requireNonNull(entityCentricDetails);
+ this.geoDetails = requireNonNull(geoDetails);
+ this.pcjDetails = requireNonNull(pcjDetails);
+ this.temporalDetails = requireNonNull(temporalDetails);
+ this.freeTextDetails = requireNonNull(freeTextDetails);
+ this.prospectorDetails = requireNonNull(prospectorDetails);
+ this.joinSelectivityDetails = requireNonNull(joinSelectivityDetails);
+ }
+
+ /**
+ * @return The name that uniquely identifies the instance of Rya within
+ * the system that hosts it.
+ */
+ public String getRyaInstanceName() {
+ return instanceName;
+ }
+
+ /**
+ * @return The version of Rya this instance uses.
+ */
+ public String getRyaVersion() {
+ return version;
+ }
+
+ /**
+ * @return Information about the instance's Entity Centric Index.
+ */
+ public EntityCentricIndexDetails getEntityCentricIndexDetails() {
+ return entityCentricDetails;
+ }
+
+ /**
+ * @return Information about the instance's Geospatial Index.
+ */
+ public GeoIndexDetails getGeoIndexDetails() {
+ return geoDetails;
+ }
+
+ /**
+ * @return Information about the instance's Precomputed Join Index.
+ */
+ public PCJIndexDetails getPCJIndexDetails() {
+ return pcjDetails;
+ }
+
+ /**
+ * @return Information about the instance's Temporal Index.
+ */
+ public TemporalIndexDetails getTemporalIndexDetails() {
+ return temporalDetails;
+ }
+
+ /**
+ * @return Information about the instance's Free Text Index.
+ */
+ public FreeTextIndexDetails getFreeTextIndexDetails() {
+ return freeTextDetails;
+ }
+
+ /**
+ * @return Information about the instance's Prospector Statistics.
+ */
+ public ProspectorDetails getProspectorDetails() {
+ return prospectorDetails;
+ }
+
+ /**
+ * @return Information about the instance's Join Selectivity Statistics.
+ */
+ public JoinSelectivityDetails getJoinSelectivityDetails() {
+ return joinSelectivityDetails;
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(
+ instanceName,
+ version,
+ entityCentricDetails,
+ geoDetails,
+ pcjDetails,
+ temporalDetails,
+ freeTextDetails,
+ prospectorDetails,
+ joinSelectivityDetails);
+ }
+
+ @Override
+ public boolean equals(final Object obj) {
+ if(this == obj) {
+ return true;
+ }
+ if(obj instanceof RyaDetails) {
+ final RyaDetails details = (RyaDetails) obj;
+ return Objects.equals(instanceName, details.instanceName) &&
+ Objects.equals(version, details.version) &&
+ Objects.equals(entityCentricDetails, details.entityCentricDetails) &&
+ Objects.equals(geoDetails, details.geoDetails) &&
+ Objects.equals(pcjDetails, details.pcjDetails) &&
+ Objects.equals(temporalDetails, details.temporalDetails) &&
+ Objects.equals(freeTextDetails, details.freeTextDetails) &&
+ Objects.equals(prospectorDetails, details.prospectorDetails) &&
+ Objects.equals(joinSelectivityDetails, details.joinSelectivityDetails);
+ }
+ return false;
+ }
+
+ /**
+ * @return An empty instance of {@link Builder}.
+ */
+ public static Builder builder() {
+ return new Builder();
+ }
+
+ /**
+ * Builds instances of {@link RyaDetails}.
+ */
+ @ParametersAreNonnullByDefault
+ public static class Builder {
+
+ // General metadata about the instance.
+ private String instanceName;
+ private String version;
+
+ // Secondary Index Details.
+ private EntityCentricIndexDetails entityCentricDetails;
+ private GeoIndexDetails geoDetails;
+ private PCJIndexDetails pcjDetails;
+ private TemporalIndexDetails temporalDetails;
+ private FreeTextIndexDetails freeTextDetails;
+
+ // Statistics Details.
+ private ProspectorDetails prospectorDetails;
+ private JoinSelectivityDetails joinSelectivityDetails;
+
+ /**
+ * Construcst an empty instance of {@link Builder}.
+ */
+ public Builder() { }
+
+ /**
+ * Constructs an instance of {@link Builder} that is initialized with
+ * a {@link RyaDetails}'s values.
+ *
+ * @param detials - The builder will be initialized with this object's values. (not null)
+ */
+ public Builder(final RyaDetails details) {
+ requireNonNull(details);
+
+ this.instanceName = details.instanceName;
+ this.version = details.version;
+ this.entityCentricDetails = details.entityCentricDetails;
+ this.geoDetails = details.geoDetails;
+ this.pcjDetails = details.pcjDetails;
+ this.temporalDetails = details.temporalDetails;
+ this.freeTextDetails = details.freeTextDetails;
+ this.prospectorDetails = details.prospectorDetails;
+ this.joinSelectivityDetails = details.joinSelectivityDetails;
+ }
+
+ /**
+ * @param instanceName - The name that uniquely identifies the instance of Rya within
+ * the system that hosts it.
+ * @return This {@link Builder} so that method invocations may be chained.
+ */
+ public Builder setRyaInstanceName(@Nullable final String instanceName) {
+ this.instanceName = instanceName;
+ return this;
+ }
+
+ /**
+ * @param version - The version of Rya this instance uses.
+ * @return This {@link Builder} so that method invocations may be chained.
+ */
+ public Builder setRyaVersion(@Nullable final String version) {
+ this.version = version;
+ return this;
+ }
+
+ /**
+ * @param entityCentricDetails - Information about the instance's Entity Centric Index.
+ * @return This {@link Builder} so that method invocations may be chained.
+ */
+ public Builder setEntityCentricIndexDetails(@Nullable final EntityCentricIndexDetails entityCentricDetails) {
+ this.entityCentricDetails = entityCentricDetails;
+ return this;
+ }
+
+ /**
+ *
+ * @param geoDetails - Information about the instance's Geospatial Index.
+ * @return This {@link Builder} so that method invocations may be chained.
+ */
+ public Builder setGeoIndexDetails(@Nullable final GeoIndexDetails geoDetails) {
+ this.geoDetails = geoDetails;
+ return this;
+ }
+
+ /**
+ * @param temporalDetails - Information about the instance's Temporal Index.
+ * @return This {@link Builder} so that method invocations may be chained.
+ */
+ public Builder setTemporalIndexDetails(@Nullable final TemporalIndexDetails temporalDetails) {
+ this.temporalDetails = temporalDetails;
+ return this;
+ }
+
+ /**
+ * @param freeTextDetails - Information about the instance's Free Text Index.
+ * @return This {@link Builder} so that method invocations may be chained.
+ */
+ public Builder setFreeTextDetails(@Nullable final FreeTextIndexDetails freeTextDetails) {
+ this.freeTextDetails = freeTextDetails;
+ return this;
+ }
+
+ /**
+ * @param pcjDetails - Information about the instance's Precomputed Join Index.
+ * @return This {@link Builder} so that method invocations may be chained.
+ */
+ public Builder setPCJIndexDetails(@Nullable final PCJIndexDetails pcjDetails) {
+ this.pcjDetails = pcjDetails;
+ return this;
+ }
+
+ /**
+ * @param prospectorDetails - Information about the instance's Prospector Statistics.
+ * @return This {@link Builder} so that method invocations may be chained.
+ */
+ public Builder setProspectorDetails(@Nullable final ProspectorDetails prospectorDetails) {
+ this.prospectorDetails = prospectorDetails;
+ return this;
+ }
+
+ /**
+ * @param joinSelectivityDetails - Information about the instance's Join Selectivity Statistics.
+ * @return This {@link Builder} so that method invocations may be chained.
+ */
+ public Builder setJoinSelectivityDetails(@Nullable final JoinSelectivityDetails joinSelectivityDetails) {
+ this.joinSelectivityDetails = joinSelectivityDetails;
+ return this;
+ }
+
+ /**
+ * @return An instance of {@link RyaDetails} built using this
+ * builder's values.
+ */
+ public RyaDetails build() {
+ return new RyaDetails(
+ instanceName,
+ version,
+ entityCentricDetails,
+ geoDetails,
+ pcjDetails,
+ temporalDetails,
+ freeTextDetails,
+ prospectorDetails,
+ joinSelectivityDetails);
+ }
+ }
+
+ /**
+ * Details about a Rya instance's Geospatial Index.
+ */
+ @Immutable
+ @ParametersAreNonnullByDefault
+ public static class GeoIndexDetails implements Serializable {
+ private static final long serialVersionUID = 1L;
+
+ private final boolean enabled;
+
+ /**
+ * Constructs an instance of {@link GeoIndexDetails}.
+ *
+ * @param enabled - Whether or not a Geospatial Index will be maintained by the Rya instance.
+ */
+ public GeoIndexDetails(final boolean enabled) {
+ this.enabled = enabled;
+ }
+
+ /**
+ * @return Whether or not a Geospatial Index will be maintained by the Rya instance.
+ */
+ public boolean isEnabled() {
+ return enabled;
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash( enabled );
+ }
+
+ @Override
+ public boolean equals(final Object obj) {
+ if(this == obj) {
+ return true;
+ }
+ if(obj instanceof GeoIndexDetails) {
+ final GeoIndexDetails details = (GeoIndexDetails) obj;
+ return enabled == details.enabled;
+ }
+ return false;
+ }
+ }
+
+ /**
+ * Details about a Rya instance's Temporal Index.
+ */
+ @Immutable
+ @ParametersAreNonnullByDefault
+ public static class TemporalIndexDetails implements Serializable {
+ private static final long serialVersionUID = 1L;
+
+ private final boolean enabled;
+
+ /**
+ * Constructs an instance of {@link TemporalIndexDetails}.
+ *
+ * @param enabled - Whether or not a Temporal Index will be maintained by the Rya instance.
+ */
+ public TemporalIndexDetails(final boolean enabled) {
+ this.enabled = enabled;
+ }
+
+ /**
+ * @return Whether or not a Temporal Index will be maintained by the Rya instance.
+ */
+ public boolean isEnabled() {
+ return enabled;
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash( enabled );
+ }
+
+ @Override
+ public boolean equals(final Object obj) {
+ if(this == obj) {
+ return true;
+ }
+ if(obj instanceof TemporalIndexDetails) {
+ final TemporalIndexDetails details = (TemporalIndexDetails) obj;
+ return enabled == details.enabled;
+ }
+ return false;
+ }
+ }
+
+ /**
+ * Details about a Rya instance's Entity Centric Index.
+ */
+ @Immutable
+ @ParametersAreNonnullByDefault
+ public static class EntityCentricIndexDetails implements Serializable {
+ private static final long serialVersionUID = 1L;
+
+ private final boolean enabled;
+
+ /**
+ * Constructs an instance of {@link EntityCentricIndexDetails}.
+ *
+ * @param enabled - Whether or not a Entity Centric Index will be maintained by the Rya instance.
+ */
+ public EntityCentricIndexDetails(final boolean enabled) {
+ this.enabled = enabled;
+ }
+
+ /**
+ * @return Whether or not a Entity Centric Index will be maintained by the Rya instance.
+ */
+ public boolean isEnabled() {
+ return enabled;
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash( enabled );
+ }
+
+ @Override
+ public boolean equals(final Object obj) {
+ if(this == obj) {
+ return true;
+ }
+ if(obj instanceof EntityCentricIndexDetails) {
+ final EntityCentricIndexDetails details = (EntityCentricIndexDetails) obj;
+ return enabled == details.enabled;
+ }
+ return false;
+ }
+ }
+
+ /**
+ * Details about a Rya instance's Free Text Index.
+ */
+ @Immutable
+ @ParametersAreNonnullByDefault
+ public static class FreeTextIndexDetails implements Serializable {
+ private static final long serialVersionUID = 1L;
+
+ private final boolean enabled;
+
+ /**
+ * Constructs an instance of {@link FreeTextIndexDetails}.
+ *
+ * @param enabled - Whether or not a Free Text Index will be maintained by the Rya instance.
+ */
+ public FreeTextIndexDetails(final boolean enabled) {
+ this.enabled = enabled;
+ }
+
+ /**
+ * @return Whether or not a Free Text Index will be maintained by the Rya instance.
+ */
+ public boolean isEnabled() {
+ return enabled;
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash( enabled );
+ }
+
+ @Override
+ public boolean equals(final Object obj) {
+ if(this == obj) {
+ return true;
+ }
+ if(obj instanceof FreeTextIndexDetails) {
+ final FreeTextIndexDetails details = (FreeTextIndexDetails) obj;
+ return enabled == details.enabled;
+ }
+ return false;
+ }
+ }
+
+ /**
+ * Details about a Rya instance's PCJ Index.
+ */
+ @Immutable
+ @ParametersAreNonnullByDefault
+ public static class PCJIndexDetails implements Serializable {
+ private static final long serialVersionUID = 1L;
+
+ public final boolean enabled;
+ private final Optional<FluoDetails> fluoDetails;
+ private final ImmutableList<PCJDetails> pcjDetails;
+
+ /**
+ * Private to prevent initialization through the constructor. To build
+ * instances of this class, use the {@link Builder}.
+ *
+ * @param enabled - Whether or not a Precomputed Join Index will be maintained by the Rya instance.
+ * @param fluoDetails - Details about a Fluo application that is used to
+ * incrementally update PCJs if one has been installed for this RYA
+ * instance. (not null)
+ * @param pcjDetails - Details about the PCJs that have been created
+ * for this Rya instance. (not null)
+ */
+ private PCJIndexDetails(
+ final boolean enabled,
+ final Optional<FluoDetails> fluoDetails,
+ final ImmutableList<PCJDetails> pcjDetails) {
+ this.enabled = enabled;
+ this.fluoDetails = requireNonNull(fluoDetails);
+ this.pcjDetails = requireNonNull(pcjDetails);
+ }
+
+ /**
+ * @return Whether or not a Precomputed Join Index will be maintained by the Rya instance.
+ */
+ public boolean isEnabled() {
+ return enabled;
+ }
+
+ /**
+ * @return Details about a Fluo application that is used to incrementally
+ * update PCJs if one has been installed for this RYA instance.
+ */
+ public Optional<FluoDetails> getFluoDetails() {
+ return fluoDetails;
+ }
+
+ /**
+ * @return Details about the PCJs that have been created for this Rya instance.
+ */
+ public ImmutableList<PCJDetails> getPCJDetails() {
+ return pcjDetails;
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(enabled, fluoDetails, pcjDetails);
+ }
+
+ @Override
+ public boolean equals(final Object obj) {
+ if(this == obj) {
+ return true;
+ }
+ if(obj instanceof PCJIndexDetails) {
+ final PCJIndexDetails details = (PCJIndexDetails) obj;
+ return Objects.equals(enabled, details.enabled) &&
+ Objects.equals(fluoDetails, details.fluoDetails) &&
+ Objects.equals(pcjDetails, details.pcjDetails);
+ }
+ return false;
+ }
+
+ /**
+ * @return A new instance of {@link Builder}.
+ */
+ public static Builder builder() {
+ return new Builder();
+ }
+
+ /**
+ * Builds instance of {@link PCJIndexDetails).
+ */
+ @ParametersAreNonnullByDefault
+ public static class Builder {
+
+ private Boolean enabled = null;
+ private FluoDetails fluoDetails = null;
+ private final ImmutableList.Builder<PCJDetails> pcjDetails = ImmutableList.builder();
+
+ /**
+ * @param enabled - Whether or not a Precomputed Join Index will be maintained by the Rya instance.
+ * @return This {@link Builder} so that method invocations may be chained.
+ */
+ public Builder setEnabled(final Boolean enabled) {
+ this.enabled = enabled;
+ return this;
+ }
+
+ /**
+ * @param fluoDetails - Details about a Fluo application that is used
+ * to incrementally update PCJs if one has been installed for this RYA instance.
+ * @return This {@link Builder} so that method invocations may be chained.
+ */
+ public Builder setFluoDetails(@Nullable final FluoDetails fluoDetails) {
+ this.fluoDetails = fluoDetails;
+ return this;
+ }
+
+ /**
+ * @param pcjDetails - Details about the PCJs that have been created for this Rya instance.
+ * @return This {@link Builder} so that method invocations may be chained.
+ */
+ public Builder addPCJDetails(@Nullable final PCJDetails pcjDetails) {
+ this.pcjDetails.add( pcjDetails );
+ return this;
+ }
+
+ /**
+ * @return Builds an instance of {@link PCJIndexDetails} using this builder's values.
+ */
+ public PCJIndexDetails build() {
+ return new PCJIndexDetails(
+ enabled,
+ Optional.fromNullable( fluoDetails ),
+ pcjDetails.build());
+ }
+ }
+
+ /**
+ * Details about a Fluo Incremental PCJ application that has been installed
+ * as part of this Rya instance.
+ */
+ @Immutable
+ @ParametersAreNonnullByDefault
+ public static class FluoDetails implements Serializable {
+ private static final long serialVersionUID = 1L;
+
+ private final String updateAppName;
+
+ /**
+ * Constructs an instance of {@link FluoDetails}.
+ *
+ * @param updateAppName - The name of the Fluo application that is
+ * updating this Rya instance's incremental PCJs. (not null)
+ */
+ public FluoDetails(final String updateAppName) {
+ this.updateAppName = requireNonNull(updateAppName);
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(updateAppName);
+ }
+
+ @Override
+ public boolean equals(final Object obj) {
+ if(this == obj) {
+ return true;
+ }
+ if(obj instanceof FluoDetails) {
+ final FluoDetails details = (FluoDetails) obj;
+ return Objects.equals(updateAppName, details.updateAppName);
+ }
+ return false;
+ }
+ }
+
+ /**
+ * Details about a specific PCJ that is being maintained within the Rya instance.
+ */
+ @Immutable
+ @ParametersAreNonnullByDefault
+ public static class PCJDetails implements Serializable {
+ private static final long serialVersionUID = 1L;
+
+ private final String id;
+ private final PCJUpdateStrategy updateStrategy;
+ private final Optional<Date> lastUpdateTime;
+
+ /**
+ * Private to prevent initialization through the constructor. To build
+ * instances of this class, use the {@link Builder}.
+ *
+ * @param id - Uniquely identifies the PCJ within this instance of Rya. (not null)
+ * @param updateStrategy - Describes how the PCJ is being updated. (not null)
+ * @param lastUpdateTime - The last time the PCJ was updated. This information
+ * may not be provided. (not null)
+ */
+ private PCJDetails(
+ final String id,
+ final PCJUpdateStrategy updateStrategy,
+ final Optional<Date> lastUpdateTime) {
+ this.id = requireNonNull(id);
+ this.updateStrategy = requireNonNull(updateStrategy);
+ this.lastUpdateTime = requireNonNull(lastUpdateTime);
+ }
+
+ /**
+ * @return Uniquely identifies the PCJ within this instance of Rya.
+ */
+ public String getId() {
+ return id;
+ }
+
+ /**
+ * @return Describes how the PCJ is being updated.
+ */
+ public PCJUpdateStrategy getUpdateStrategy() {
+ return updateStrategy;
+ }
+
+ /**
+ * @return The last time the PCJ was updated. This information
+ * may not be provided.
+ */
+ public Optional<Date> getLastUpdateTime() {
+ return lastUpdateTime;
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(id, updateStrategy, lastUpdateTime);
+ }
+
+ @Override
+ public boolean equals(final Object obj) {
+ if(this == obj) {
+ return true;
+ }
+ if(obj instanceof PCJDetails) {
+ final PCJDetails details = (PCJDetails) obj;
+ return Objects.equals(id, details.id) &&
+ Objects.equals(updateStrategy, details.updateStrategy) &&
+ Objects.equals(lastUpdateTime, details.lastUpdateTime);
+ }
+ return false;
+ }
+
+ /**
+ * @return A new instance of {@link Builder}.
+ */
+ public static Builder builder() {
+ return new Builder();
+ }
+
+ /**
+ * Builds instance of {@link PCJDetails}.
+ */
+ @ParametersAreNonnullByDefault
+ public static class Builder {
+
+ private String id;
+ private PCJUpdateStrategy updateStrategy;
+ private Date lastUpdateTime;
+
+ /**
+ * @param id - Uniquely identifies the PCJ within this instance of Rya.
+ * @return This {@link Builder} so that method invocations may be chained.
+ */
+ public Builder setId(@Nullable final String id) {
+ this.id = id;
+ return this;
+ }
+
+ /**
+ * @param updateStrategy - Describes how the PCJ is being updated.
+ * @return This {@link Builder} so that method invocations may be chained.
+ */
+ public Builder setUpdateStrategy(@Nullable final PCJUpdateStrategy updateStrategy) {
+ this.updateStrategy = updateStrategy;
+ return this;
+ }
+
+ /**
+ * @param lastUpdateTime - The last time the PCJ was updated. This information
+ * may not be provided.
+ * @return This {@link Builder} so that method invocations may be chained.
+ */
+ public Builder setLastUpdateTime(@Nullable final Date lastUpdateTime) {
+ this.lastUpdateTime = lastUpdateTime;
+ return this;
+ }
+
+ /**
+ * @return An instance of {@link PCJDetails} built using this builder's values.
+ */
+ public PCJDetails build() {
+ return new PCJDetails(id, updateStrategy, Optional.fromNullable(lastUpdateTime));
+ }
+ }
+
+ /**
+ * Describes the different strategies that may be used to update a PCJ index.
+ */
+ public static enum PCJUpdateStrategy {
+ /**
+ * The PCJ is being updated by periodically rebuilding all of the results.
+ */
+ BATCH,
+
+ /**
+ * The PCJ is being updated frequently and incrementally as new
+ * Statements are inserted into the Tya instance.
+ */
+ INCREMENTAL;
+ }
+ }
+ }
+
+ /**
+ * Details about a Rya instance's Prospector statistics.
+ */
+ @Immutable
+ @ParametersAreNonnullByDefault
+ public static class ProspectorDetails implements Serializable {
+ private static final long serialVersionUID = 1L;
+
+ private final Optional<Date> lastUpdated;
+
+ /**
+ * Constructs an instance of {@link ProspectorDetails}.
+ *
+ * @param lastUpdated - The last time the Prospector statistics were updated for the Rya instance. (not null)
+ */
+ public ProspectorDetails(final Optional<Date> lastUpdated) {
+ this.lastUpdated = requireNonNull(lastUpdated);
+ }
+
+ /**
+ * @return The last time the Prospector statistics were updated for the Rya instance.
+ */
+ public Optional<Date> getLastUpdated() {
+ return lastUpdated;
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash( lastUpdated );
+ }
+
+ @Override
+ public boolean equals(final Object obj) {
+ if(this == obj) {
+ return true;
+ }
+ if(obj instanceof ProspectorDetails) {
+ final ProspectorDetails details = (ProspectorDetails) obj;
+ return Objects.equals(lastUpdated, details.lastUpdated);
+ }
+ return false;
+ }
+ }
+
+ /**
+ * Details about a Rya instance's Join Selectivity statistics.
+ */
+ @Immutable
+ @ParametersAreNonnullByDefault
+ public static class JoinSelectivityDetails implements Serializable {
--- End diff --
any reason these aren't in their own files?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-rya pull request #60: Rya 81
Posted by isper3at <gi...@git.apache.org>.
Github user isper3at commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/60#discussion_r73416910
--- Diff: extras/indexing/src/main/java/mvm/rya/indexing/pcj/matching/PCJOptimizer.java ---
@@ -334,9 +334,11 @@ public void meet(Filter node) {
} else {
//if no tables are provided by user, get ids for rya instance id, create table name,
//and associate table name with sparql
+ // TODO: storage.listPcjs() returns tablenames, not PCJ-IDs.
+ // See mvm.rya.indexing.external.accumulo.AccumuloPcjStorage.dropPcj(String)
--- End diff --
in caleb's commit there is a getPcjId()
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-rya pull request #60: Rya 81
Posted by isper3at <gi...@git.apache.org>.
Github user isper3at commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/60#discussion_r73417405
--- Diff: extras/indexing/src/test/java/mvm/rya/indexing/external/tupleSet/AccumuloIndexSetColumnVisibilityTest.java ---
@@ -60,6 +36,34 @@
import com.google.common.collect.Sets;
import com.google.common.io.Files;
+/*
--- End diff --
thought we were putting these above package
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-rya issue #60: Rya 81
Posted by amihalik <gi...@git.apache.org>.
Github user amihalik commented on the issue:
https://github.com/apache/incubator-rya/pull/60
Merged.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-rya pull request #60: Rya 81
Posted by isper3at <gi...@git.apache.org>.
Github user isper3at commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/60#discussion_r73417159
--- Diff: extras/indexing/src/test/java/mvm/rya/indexing/accumulo/temporal/AccumuloTemporalIndexerTest.java ---
@@ -237,17 +219,15 @@ public static void tearDownAfterClass() throws Exception {
@Before
public void setUp() throws Exception {
conf = new Configuration();
+ conf.set(RdfCloudTripleStoreConfiguration.CONF_TBL_PREFIX, "triplestore_");
conf.setBoolean(ConfigUtils.USE_MOCK_INSTANCE, true);
- conf.set(ConfigUtils.TEMPORAL_TABLENAME, uniquePerTestTemporalIndexTableName);
- // This is from http://linkedevents.org/ontology
+ // The temporal predicates are from http://linkedevents.org/ontology
--- End diff --
already linked in the class doc
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-rya pull request #60: Rya 81
Posted by isper3at <gi...@git.apache.org>.
Github user isper3at commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/60#discussion_r73416279
--- Diff: extras/indexing/src/main/java/mvm/rya/indexing/pcj/matching/PCJOptimizer.java ---
@@ -76,273 +80,277 @@
*/
public class PCJOptimizer implements QueryOptimizer, Configurable {
- private static final Logger log = Logger.getLogger(PCJOptimizer.class);
- private List<ExternalTupleSet> indexSet;
- private Configuration conf;
- private boolean init = false;
-
- public PCJOptimizer() {
- }
-
- public PCJOptimizer(Configuration conf) {
- this.conf = conf;
- try {
- indexSet = PCJOptimizerUtilities.getValidPCJs(getAccIndices(conf)); // TODO
- // validate
- // PCJs
- // during
- // table
- // creation
- } catch (MalformedQueryException | SailException
- | QueryEvaluationException | TableNotFoundException
- | AccumuloException | AccumuloSecurityException | PcjException e) {
- e.printStackTrace();
- }
- init = true;
- }
-
- public PCJOptimizer(List<ExternalTupleSet> indices, boolean useOptimalPcj) {
- this.indexSet = PCJOptimizerUtilities.getValidPCJs(indices);
- conf = new Configuration();
- conf.setBoolean(ConfigUtils.USE_OPTIMAL_PCJ, useOptimalPcj);
- }
-
- @Override
- public void setConf(Configuration conf) {
- this.conf = conf;
- if (!init) {
- try {
- indexSet = PCJOptimizerUtilities
- .getValidPCJs(getAccIndices(conf));
- } catch (MalformedQueryException | SailException
- | QueryEvaluationException | TableNotFoundException
- | AccumuloException | AccumuloSecurityException
- | PcjException e) {
- e.printStackTrace();
- }
- init = true;
- }
- }
-
- @Override
- public Configuration getConf() {
- return conf;
- }
-
- /**
- * This method optimizes a specified query by matching subsets of it with
- * PCJ queries.
- *
- * @param tupleExpr
- * - the query to be optimized
- */
- @Override
- public void optimize(TupleExpr tupleExpr, Dataset dataset,
- BindingSet bindings) {
-
- Projection projection = PCJOptimizerUtilities.getProjection(tupleExpr);
- if (projection == null) {
- log.debug("TupleExpr has no Projection. Invalid TupleExpr.");
- return;
- }
- IndexedExecutionPlanGenerator iep = new IndexedExecutionPlanGenerator(
- tupleExpr, indexSet);
- List<ExternalTupleSet> pcjs = iep.getNormalizedIndices();
- // first standardize query by pulling all filters to top of query if
- // they exist
- // using TopOfQueryFilterRelocator
- tupleExpr = TopOfQueryFilterRelocator.moveFiltersToTop(tupleExpr);
-
- if (ConfigUtils.getUseOptimalPCJ(conf) && pcjs.size() > 0) {
-
- // get potential relevant index combinations
- ValidIndexCombinationGenerator vic = new ValidIndexCombinationGenerator(
- tupleExpr);
- Iterator<List<ExternalTupleSet>> iter = vic
- .getValidIndexCombos(pcjs);
- TupleExpr bestTup = null;
- TupleExpr tempTup = null;
- double tempCost = 0;
- double minCost = Double.MAX_VALUE;
-
- while (iter.hasNext()) {
- // apply join visitor to place external index nodes in query
- TupleExpr clone = tupleExpr.clone();
- QuerySegmentPCJMatchVisitor.matchPCJs(clone, iter.next());
-
- // get all valid execution plans for given external index
- // combination by considering all
- // permutations of nodes in TupleExpr
- IndexPlanValidator ipv = new IndexPlanValidator(false);
- Iterator<TupleExpr> validTups = ipv
- .getValidTuples(TupleReArranger.getTupleReOrderings(
- clone).iterator());
-
- // set valid plan according to a specified cost threshold, where
- // cost depends on specified weights
- // for number of external index nodes, common variables among
- // joins in execution plan, and number of
- // external products in execution plan
- ThreshholdPlanSelector tps = new ThreshholdPlanSelector(
- tupleExpr);
- tempTup = tps.getThreshholdQueryPlan(validTups, .4, .5, .2, .3);
-
- // choose best threshhold TupleExpr among all index node
- // combinations
- tempCost = tps.getCost(tempTup, .5, .2, .3);
- if (tempCost < minCost) {
- minCost = tempCost;
- bestTup = tempTup;
- }
- }
- if (bestTup != null) {
- Projection bestTupProject = PCJOptimizerUtilities
- .getProjection(bestTup);
- projection.setArg(bestTupProject.getArg());
- }
- return;
- } else if (pcjs.size() > 0) {
- QuerySegmentPCJMatchVisitor.matchPCJs(tupleExpr, pcjs);
- } else {
- return;
- }
- }
-
- /**
- * This visitor navigates query until it reaches either a Join, Filter, or
- * LeftJoin. Once it reaches this node, it gets the appropriate PCJMatcher
- * from the {@link QuerySegmentPCJMatchVisitor} and uses this to match each
- * of the PCJs to the {@link QuerySegment} starting with the Join, Filter,
- * or LeftJoin. Once each PCJ has been compared for matching, the portion of
- * the query starting with the Join, Filter, or LeftJoin is replaced by the
- * {@link TupleExpr} returned by {@link PCJMatcher#getQuery()}. This visitor
- * then visits each of the nodes returned by {@link PCJMatcher#getUnmatchedArgs()}.
- *
- */
- static class QuerySegmentPCJMatchVisitor extends
- QueryModelVisitorBase<RuntimeException> {
-
- private static List<ExternalTupleSet> pcjs;
- private static final QuerySegmentPCJMatchVisitor INSTANCE = new QuerySegmentPCJMatchVisitor();
-
- private QuerySegmentPCJMatchVisitor() {
- };
-
- public static void matchPCJs(TupleExpr te,
- List<ExternalTupleSet> indexSet) {
- pcjs = indexSet;
- te.visit(INSTANCE);
- }
-
- @Override
- public void meet(Join node) {
- PCJMatcher matcher = PCJMatcherFactory.getPCJMatcher(node);
- for (ExternalTupleSet pcj : pcjs) {
- matcher.matchPCJ(pcj);
- }
-
- node.replaceWith(matcher.getQuery());
- Set<TupleExpr> unmatched = matcher.getUnmatchedArgs();
- PCJOptimizerUtilities.relocateFilters(matcher.getFilters());
-
- for (TupleExpr tupleExpr : unmatched) {
- tupleExpr.visit(this);
- }
- }
-
- @Override
- public void meet(LeftJoin node) {
- PCJMatcher matcher = PCJMatcherFactory.getPCJMatcher(node);
- for (ExternalTupleSet pcj : pcjs) {
- matcher.matchPCJ(pcj);
- }
-
- node.replaceWith(matcher.getQuery());
- Set<TupleExpr> unmatched = matcher.getUnmatchedArgs();
- PCJOptimizerUtilities.relocateFilters(matcher.getFilters());
-
- for (TupleExpr tupleExpr : unmatched) {
- tupleExpr.visit(this);
- }
- }
-
- @Override
- public void meet(Filter node) {
- PCJMatcher matcher = PCJMatcherFactory.getPCJMatcher(node);
- for (ExternalTupleSet pcj : pcjs) {
- matcher.matchPCJ(pcj);
- }
-
- node.replaceWith(matcher.getQuery());
- Set<TupleExpr> unmatched = matcher.getUnmatchedArgs();
- PCJOptimizerUtilities.relocateFilters(matcher.getFilters());
-
- for (TupleExpr tupleExpr : unmatched) {
- tupleExpr.visit(this);
- }
- }
-
- }
-
- /**
- *
- *
- * @param conf
- * - client configuration
- *
- * @return - list of {@link ExternalTupleSet}s or PCJs that are either
- * specified by user in Configuration or exist in system.
- *
- * @throws MalformedQueryException
- * @throws SailException
- * @throws QueryEvaluationException
- * @throws TableNotFoundException
- * @throws AccumuloException
- * @throws AccumuloSecurityException
- * @throws PcjException
- */
- private static List<ExternalTupleSet> getAccIndices(Configuration conf)
- throws MalformedQueryException, SailException,
- QueryEvaluationException, TableNotFoundException,
- AccumuloException, AccumuloSecurityException, PcjException {
-
- List<String> tables = null;
-
- if (conf instanceof RdfCloudTripleStoreConfiguration) {
- tables = ((RdfCloudTripleStoreConfiguration) conf).getPcjTables();
- }
-
- String tablePrefix = conf
- .get(RdfCloudTripleStoreConfiguration.CONF_TBL_PREFIX);
- Connector c = ConfigUtils.getConnector(conf);
- Map<String, String> indexTables = Maps.newLinkedHashMap();
- PcjTables pcj = new PcjTables();
-
- if (tables != null && !tables.isEmpty()) {
- for (final String table : tables) {
- indexTables
- .put(table, pcj.getPcjMetadata(c, table).getSparql());
- }
- } else {
- for (final String table : c.tableOperations().list()) {
- if (table.startsWith(tablePrefix + "INDEX")) {
- indexTables.put(table, pcj.getPcjMetadata(c, table)
- .getSparql());
- }
- }
-
- }
- final List<ExternalTupleSet> index = Lists.newArrayList();
-
- if (indexTables.isEmpty()) {
- System.out.println("No Index found");
- } else {
- for (final String table : indexTables.keySet()) {
- final String indexSparqlString = indexTables.get(table);
- index.add(new AccumuloIndexSet(indexSparqlString, conf, table));
- }
- }
- return index;
- }
+ private static final Logger log = Logger.getLogger(PCJOptimizer.class);
+ private List<ExternalTupleSet> indexSet;
+ private Configuration conf;
+ private boolean init = false;
+
+ public PCJOptimizer() {
+ }
+
+ public PCJOptimizer(Configuration conf) {
+ this.conf = conf;
+ try {
+ indexSet = PCJOptimizerUtilities.getValidPCJs(getAccIndices(conf)); // TODO
+ // validate
+ // PCJs
+ // during
+ // table
+ // creation
+ } catch (MalformedQueryException | SailException
+ | QueryEvaluationException | TableNotFoundException
+ | AccumuloException | AccumuloSecurityException | PcjException e) {
+ e.printStackTrace();
+ }
+ init = true;
+ }
+
+ public PCJOptimizer(List<ExternalTupleSet> indices, boolean useOptimalPcj) {
+ this.indexSet = PCJOptimizerUtilities.getValidPCJs(indices);
+ conf = new Configuration();
+ conf.setBoolean(ConfigUtils.USE_OPTIMAL_PCJ, useOptimalPcj);
+ }
+
+ @Override
+ public void setConf(Configuration conf) {
+ this.conf = conf;
+ if (!init) {
+ try {
+ indexSet = PCJOptimizerUtilities
+ .getValidPCJs(getAccIndices(conf));
+ } catch (MalformedQueryException | SailException
+ | QueryEvaluationException | TableNotFoundException
+ | AccumuloException | AccumuloSecurityException
+ | PcjException e) {
+ e.printStackTrace();
+ }
+ init = true;
+ }
+ }
+
+ @Override
+ public Configuration getConf() {
+ return conf;
+ }
+
+ /**
+ * This method optimizes a specified query by matching subsets of it with
+ * PCJ queries.
+ *
+ * @param tupleExpr
+ * - the query to be optimized
+ */
+ @Override
+ public void optimize(TupleExpr tupleExpr, Dataset dataset,
+ BindingSet bindings) {
+
+ Projection projection = PCJOptimizerUtilities.getProjection(tupleExpr);
+ if (projection == null) {
+ log.debug("TupleExpr has no Projection. Invalid TupleExpr.");
+ return;
+ }
+ IndexedExecutionPlanGenerator iep = new IndexedExecutionPlanGenerator(
+ tupleExpr, indexSet);
+ List<ExternalTupleSet> pcjs = iep.getNormalizedIndices();
+ // first standardize query by pulling all filters to top of query if
+ // they exist
+ // using TopOfQueryFilterRelocator
+ tupleExpr = TopOfQueryFilterRelocator.moveFiltersToTop(tupleExpr);
+
+ if (ConfigUtils.getUseOptimalPCJ(conf) && pcjs.size() > 0) {
+
+ // get potential relevant index combinations
+ ValidIndexCombinationGenerator vic = new ValidIndexCombinationGenerator(
+ tupleExpr);
+ Iterator<List<ExternalTupleSet>> iter = vic
+ .getValidIndexCombos(pcjs);
+ TupleExpr bestTup = null;
+ TupleExpr tempTup = null;
+ double tempCost = 0;
+ double minCost = Double.MAX_VALUE;
+
+ while (iter.hasNext()) {
+ // apply join visitor to place external index nodes in query
+ TupleExpr clone = tupleExpr.clone();
+ QuerySegmentPCJMatchVisitor.matchPCJs(clone, iter.next());
+
+ // get all valid execution plans for given external index
+ // combination by considering all
+ // permutations of nodes in TupleExpr
+ IndexPlanValidator ipv = new IndexPlanValidator(false);
+ Iterator<TupleExpr> validTups = ipv
+ .getValidTuples(TupleReArranger.getTupleReOrderings(
+ clone).iterator());
+
+ // set valid plan according to a specified cost threshold, where
+ // cost depends on specified weights
+ // for number of external index nodes, common variables among
+ // joins in execution plan, and number of
+ // external products in execution plan
+ ThreshholdPlanSelector tps = new ThreshholdPlanSelector(
+ tupleExpr);
+ tempTup = tps.getThreshholdQueryPlan(validTups, .4, .5, .2, .3);
+
+ // choose best threshhold TupleExpr among all index node
+ // combinations
+ tempCost = tps.getCost(tempTup, .5, .2, .3);
+ if (tempCost < minCost) {
+ minCost = tempCost;
+ bestTup = tempTup;
+ }
+ }
+ if (bestTup != null) {
+ Projection bestTupProject = PCJOptimizerUtilities
+ .getProjection(bestTup);
+ projection.setArg(bestTupProject.getArg());
+ }
+ return;
+ } else if (pcjs.size() > 0) {
+ QuerySegmentPCJMatchVisitor.matchPCJs(tupleExpr, pcjs);
+ } else {
+ return;
+ }
+ }
+
+ /**
+ * This visitor navigates query until it reaches either a Join, Filter, or
+ * LeftJoin. Once it reaches this node, it gets the appropriate PCJMatcher
+ * from the {@link QuerySegmentPCJMatchVisitor} and uses this to match each
+ * of the PCJs to the {@link QuerySegment} starting with the Join, Filter,
+ * or LeftJoin. Once each PCJ has been compared for matching, the portion of
+ * the query starting with the Join, Filter, or LeftJoin is replaced by the
+ * {@link TupleExpr} returned by {@link PCJMatcher#getQuery()}. This visitor
+ * then visits each of the nodes returned by {@link PCJMatcher#getUnmatchedArgs()}.
+ *
+ */
+ static class QuerySegmentPCJMatchVisitor extends
--- End diff --
word wrap is a little short
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-rya pull request #60: Rya 81
Posted by isper3at <gi...@git.apache.org>.
Github user isper3at commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/60#discussion_r73417056
--- Diff: extras/indexing/src/test/java/mvm/rya/indexing/accumulo/temporal/AccumuloTemporalIndexerTest.java ---
@@ -21,9 +21,8 @@
import static mvm.rya.api.resolver.RdfToRyaConversions.convertStatement;
-import static org.junit.Assert.assertNotNull;
-import static org.junit.Assert.assertTrue;
-
+import static org.junit.Assert.*;
--- End diff --
dont use *
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-rya pull request #60: Rya 81
Posted by isper3at <gi...@git.apache.org>.
Github user isper3at commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/60#discussion_r73416389
--- Diff: extras/indexing/src/main/java/mvm/rya/indexing/pcj/matching/PCJOptimizer.java ---
@@ -76,273 +80,277 @@
*/
public class PCJOptimizer implements QueryOptimizer, Configurable {
- private static final Logger log = Logger.getLogger(PCJOptimizer.class);
- private List<ExternalTupleSet> indexSet;
- private Configuration conf;
- private boolean init = false;
-
- public PCJOptimizer() {
- }
-
- public PCJOptimizer(Configuration conf) {
- this.conf = conf;
- try {
- indexSet = PCJOptimizerUtilities.getValidPCJs(getAccIndices(conf)); // TODO
- // validate
- // PCJs
- // during
- // table
- // creation
- } catch (MalformedQueryException | SailException
- | QueryEvaluationException | TableNotFoundException
- | AccumuloException | AccumuloSecurityException | PcjException e) {
- e.printStackTrace();
- }
- init = true;
- }
-
- public PCJOptimizer(List<ExternalTupleSet> indices, boolean useOptimalPcj) {
- this.indexSet = PCJOptimizerUtilities.getValidPCJs(indices);
- conf = new Configuration();
- conf.setBoolean(ConfigUtils.USE_OPTIMAL_PCJ, useOptimalPcj);
- }
-
- @Override
- public void setConf(Configuration conf) {
- this.conf = conf;
- if (!init) {
- try {
- indexSet = PCJOptimizerUtilities
- .getValidPCJs(getAccIndices(conf));
- } catch (MalformedQueryException | SailException
- | QueryEvaluationException | TableNotFoundException
- | AccumuloException | AccumuloSecurityException
- | PcjException e) {
- e.printStackTrace();
- }
- init = true;
- }
- }
-
- @Override
- public Configuration getConf() {
- return conf;
- }
-
- /**
- * This method optimizes a specified query by matching subsets of it with
- * PCJ queries.
- *
- * @param tupleExpr
- * - the query to be optimized
- */
- @Override
- public void optimize(TupleExpr tupleExpr, Dataset dataset,
- BindingSet bindings) {
-
- Projection projection = PCJOptimizerUtilities.getProjection(tupleExpr);
- if (projection == null) {
- log.debug("TupleExpr has no Projection. Invalid TupleExpr.");
- return;
- }
- IndexedExecutionPlanGenerator iep = new IndexedExecutionPlanGenerator(
- tupleExpr, indexSet);
- List<ExternalTupleSet> pcjs = iep.getNormalizedIndices();
- // first standardize query by pulling all filters to top of query if
- // they exist
- // using TopOfQueryFilterRelocator
- tupleExpr = TopOfQueryFilterRelocator.moveFiltersToTop(tupleExpr);
-
- if (ConfigUtils.getUseOptimalPCJ(conf) && pcjs.size() > 0) {
-
- // get potential relevant index combinations
- ValidIndexCombinationGenerator vic = new ValidIndexCombinationGenerator(
- tupleExpr);
- Iterator<List<ExternalTupleSet>> iter = vic
- .getValidIndexCombos(pcjs);
- TupleExpr bestTup = null;
- TupleExpr tempTup = null;
- double tempCost = 0;
- double minCost = Double.MAX_VALUE;
-
- while (iter.hasNext()) {
- // apply join visitor to place external index nodes in query
- TupleExpr clone = tupleExpr.clone();
- QuerySegmentPCJMatchVisitor.matchPCJs(clone, iter.next());
-
- // get all valid execution plans for given external index
- // combination by considering all
- // permutations of nodes in TupleExpr
- IndexPlanValidator ipv = new IndexPlanValidator(false);
- Iterator<TupleExpr> validTups = ipv
- .getValidTuples(TupleReArranger.getTupleReOrderings(
- clone).iterator());
-
- // set valid plan according to a specified cost threshold, where
- // cost depends on specified weights
- // for number of external index nodes, common variables among
- // joins in execution plan, and number of
- // external products in execution plan
- ThreshholdPlanSelector tps = new ThreshholdPlanSelector(
- tupleExpr);
- tempTup = tps.getThreshholdQueryPlan(validTups, .4, .5, .2, .3);
-
- // choose best threshhold TupleExpr among all index node
- // combinations
- tempCost = tps.getCost(tempTup, .5, .2, .3);
- if (tempCost < minCost) {
- minCost = tempCost;
- bestTup = tempTup;
- }
- }
- if (bestTup != null) {
- Projection bestTupProject = PCJOptimizerUtilities
- .getProjection(bestTup);
- projection.setArg(bestTupProject.getArg());
- }
- return;
- } else if (pcjs.size() > 0) {
- QuerySegmentPCJMatchVisitor.matchPCJs(tupleExpr, pcjs);
- } else {
- return;
- }
- }
-
- /**
- * This visitor navigates query until it reaches either a Join, Filter, or
- * LeftJoin. Once it reaches this node, it gets the appropriate PCJMatcher
- * from the {@link QuerySegmentPCJMatchVisitor} and uses this to match each
- * of the PCJs to the {@link QuerySegment} starting with the Join, Filter,
- * or LeftJoin. Once each PCJ has been compared for matching, the portion of
- * the query starting with the Join, Filter, or LeftJoin is replaced by the
- * {@link TupleExpr} returned by {@link PCJMatcher#getQuery()}. This visitor
- * then visits each of the nodes returned by {@link PCJMatcher#getUnmatchedArgs()}.
- *
- */
- static class QuerySegmentPCJMatchVisitor extends
- QueryModelVisitorBase<RuntimeException> {
-
- private static List<ExternalTupleSet> pcjs;
- private static final QuerySegmentPCJMatchVisitor INSTANCE = new QuerySegmentPCJMatchVisitor();
-
- private QuerySegmentPCJMatchVisitor() {
- };
-
- public static void matchPCJs(TupleExpr te,
- List<ExternalTupleSet> indexSet) {
- pcjs = indexSet;
- te.visit(INSTANCE);
- }
-
- @Override
- public void meet(Join node) {
- PCJMatcher matcher = PCJMatcherFactory.getPCJMatcher(node);
- for (ExternalTupleSet pcj : pcjs) {
- matcher.matchPCJ(pcj);
- }
-
- node.replaceWith(matcher.getQuery());
- Set<TupleExpr> unmatched = matcher.getUnmatchedArgs();
- PCJOptimizerUtilities.relocateFilters(matcher.getFilters());
-
- for (TupleExpr tupleExpr : unmatched) {
- tupleExpr.visit(this);
- }
- }
-
- @Override
- public void meet(LeftJoin node) {
- PCJMatcher matcher = PCJMatcherFactory.getPCJMatcher(node);
- for (ExternalTupleSet pcj : pcjs) {
- matcher.matchPCJ(pcj);
- }
-
- node.replaceWith(matcher.getQuery());
- Set<TupleExpr> unmatched = matcher.getUnmatchedArgs();
- PCJOptimizerUtilities.relocateFilters(matcher.getFilters());
-
- for (TupleExpr tupleExpr : unmatched) {
- tupleExpr.visit(this);
- }
- }
-
- @Override
- public void meet(Filter node) {
- PCJMatcher matcher = PCJMatcherFactory.getPCJMatcher(node);
- for (ExternalTupleSet pcj : pcjs) {
- matcher.matchPCJ(pcj);
- }
-
- node.replaceWith(matcher.getQuery());
- Set<TupleExpr> unmatched = matcher.getUnmatchedArgs();
- PCJOptimizerUtilities.relocateFilters(matcher.getFilters());
-
- for (TupleExpr tupleExpr : unmatched) {
- tupleExpr.visit(this);
- }
- }
-
- }
-
- /**
- *
- *
- * @param conf
- * - client configuration
- *
- * @return - list of {@link ExternalTupleSet}s or PCJs that are either
- * specified by user in Configuration or exist in system.
- *
- * @throws MalformedQueryException
- * @throws SailException
- * @throws QueryEvaluationException
- * @throws TableNotFoundException
- * @throws AccumuloException
- * @throws AccumuloSecurityException
- * @throws PcjException
- */
- private static List<ExternalTupleSet> getAccIndices(Configuration conf)
- throws MalformedQueryException, SailException,
- QueryEvaluationException, TableNotFoundException,
- AccumuloException, AccumuloSecurityException, PcjException {
-
- List<String> tables = null;
-
- if (conf instanceof RdfCloudTripleStoreConfiguration) {
- tables = ((RdfCloudTripleStoreConfiguration) conf).getPcjTables();
- }
-
- String tablePrefix = conf
- .get(RdfCloudTripleStoreConfiguration.CONF_TBL_PREFIX);
- Connector c = ConfigUtils.getConnector(conf);
- Map<String, String> indexTables = Maps.newLinkedHashMap();
- PcjTables pcj = new PcjTables();
-
- if (tables != null && !tables.isEmpty()) {
- for (final String table : tables) {
- indexTables
- .put(table, pcj.getPcjMetadata(c, table).getSparql());
- }
- } else {
- for (final String table : c.tableOperations().list()) {
- if (table.startsWith(tablePrefix + "INDEX")) {
- indexTables.put(table, pcj.getPcjMetadata(c, table)
- .getSparql());
- }
- }
-
- }
- final List<ExternalTupleSet> index = Lists.newArrayList();
-
- if (indexTables.isEmpty()) {
- System.out.println("No Index found");
- } else {
- for (final String table : indexTables.keySet()) {
- final String indexSparqlString = indexTables.get(table);
- index.add(new AccumuloIndexSet(indexSparqlString, conf, table));
- }
- }
- return index;
- }
+ private static final Logger log = Logger.getLogger(PCJOptimizer.class);
+ private List<ExternalTupleSet> indexSet;
+ private Configuration conf;
+ private boolean init = false;
+
+ public PCJOptimizer() {
+ }
+
+ public PCJOptimizer(Configuration conf) {
+ this.conf = conf;
+ try {
+ indexSet = PCJOptimizerUtilities.getValidPCJs(getAccIndices(conf)); // TODO
+ // validate
+ // PCJs
+ // during
+ // table
+ // creation
+ } catch (MalformedQueryException | SailException
+ | QueryEvaluationException | TableNotFoundException
+ | AccumuloException | AccumuloSecurityException | PcjException e) {
+ e.printStackTrace();
+ }
+ init = true;
+ }
+
+ public PCJOptimizer(List<ExternalTupleSet> indices, boolean useOptimalPcj) {
+ this.indexSet = PCJOptimizerUtilities.getValidPCJs(indices);
+ conf = new Configuration();
+ conf.setBoolean(ConfigUtils.USE_OPTIMAL_PCJ, useOptimalPcj);
+ }
+
+ @Override
+ public void setConf(Configuration conf) {
+ this.conf = conf;
+ if (!init) {
+ try {
+ indexSet = PCJOptimizerUtilities
+ .getValidPCJs(getAccIndices(conf));
+ } catch (MalformedQueryException | SailException
+ | QueryEvaluationException | TableNotFoundException
+ | AccumuloException | AccumuloSecurityException
+ | PcjException e) {
+ e.printStackTrace();
+ }
+ init = true;
+ }
+ }
+
+ @Override
+ public Configuration getConf() {
+ return conf;
+ }
+
+ /**
+ * This method optimizes a specified query by matching subsets of it with
+ * PCJ queries.
+ *
+ * @param tupleExpr
+ * - the query to be optimized
+ */
+ @Override
+ public void optimize(TupleExpr tupleExpr, Dataset dataset,
+ BindingSet bindings) {
+
+ Projection projection = PCJOptimizerUtilities.getProjection(tupleExpr);
+ if (projection == null) {
+ log.debug("TupleExpr has no Projection. Invalid TupleExpr.");
+ return;
+ }
+ IndexedExecutionPlanGenerator iep = new IndexedExecutionPlanGenerator(
+ tupleExpr, indexSet);
+ List<ExternalTupleSet> pcjs = iep.getNormalizedIndices();
+ // first standardize query by pulling all filters to top of query if
+ // they exist
+ // using TopOfQueryFilterRelocator
+ tupleExpr = TopOfQueryFilterRelocator.moveFiltersToTop(tupleExpr);
+
+ if (ConfigUtils.getUseOptimalPCJ(conf) && pcjs.size() > 0) {
+
+ // get potential relevant index combinations
+ ValidIndexCombinationGenerator vic = new ValidIndexCombinationGenerator(
+ tupleExpr);
+ Iterator<List<ExternalTupleSet>> iter = vic
+ .getValidIndexCombos(pcjs);
+ TupleExpr bestTup = null;
+ TupleExpr tempTup = null;
+ double tempCost = 0;
+ double minCost = Double.MAX_VALUE;
+
+ while (iter.hasNext()) {
+ // apply join visitor to place external index nodes in query
+ TupleExpr clone = tupleExpr.clone();
+ QuerySegmentPCJMatchVisitor.matchPCJs(clone, iter.next());
+
+ // get all valid execution plans for given external index
+ // combination by considering all
+ // permutations of nodes in TupleExpr
+ IndexPlanValidator ipv = new IndexPlanValidator(false);
+ Iterator<TupleExpr> validTups = ipv
+ .getValidTuples(TupleReArranger.getTupleReOrderings(
+ clone).iterator());
+
+ // set valid plan according to a specified cost threshold, where
+ // cost depends on specified weights
+ // for number of external index nodes, common variables among
+ // joins in execution plan, and number of
+ // external products in execution plan
+ ThreshholdPlanSelector tps = new ThreshholdPlanSelector(
+ tupleExpr);
+ tempTup = tps.getThreshholdQueryPlan(validTups, .4, .5, .2, .3);
+
+ // choose best threshhold TupleExpr among all index node
+ // combinations
+ tempCost = tps.getCost(tempTup, .5, .2, .3);
+ if (tempCost < minCost) {
+ minCost = tempCost;
+ bestTup = tempTup;
+ }
+ }
+ if (bestTup != null) {
+ Projection bestTupProject = PCJOptimizerUtilities
+ .getProjection(bestTup);
+ projection.setArg(bestTupProject.getArg());
+ }
+ return;
+ } else if (pcjs.size() > 0) {
+ QuerySegmentPCJMatchVisitor.matchPCJs(tupleExpr, pcjs);
+ } else {
+ return;
+ }
+ }
+
+ /**
+ * This visitor navigates query until it reaches either a Join, Filter, or
+ * LeftJoin. Once it reaches this node, it gets the appropriate PCJMatcher
+ * from the {@link QuerySegmentPCJMatchVisitor} and uses this to match each
+ * of the PCJs to the {@link QuerySegment} starting with the Join, Filter,
+ * or LeftJoin. Once each PCJ has been compared for matching, the portion of
+ * the query starting with the Join, Filter, or LeftJoin is replaced by the
+ * {@link TupleExpr} returned by {@link PCJMatcher#getQuery()}. This visitor
+ * then visits each of the nodes returned by {@link PCJMatcher#getUnmatchedArgs()}.
+ *
+ */
+ static class QuerySegmentPCJMatchVisitor extends
+ QueryModelVisitorBase<RuntimeException> {
+
+ private static List<ExternalTupleSet> pcjs;
+ private static final QuerySegmentPCJMatchVisitor INSTANCE = new QuerySegmentPCJMatchVisitor();
+
+ private QuerySegmentPCJMatchVisitor() {
+ };
+
+ public static void matchPCJs(TupleExpr te,
+ List<ExternalTupleSet> indexSet) {
+ pcjs = indexSet;
+ te.visit(INSTANCE);
+ }
+
+ @Override
+ public void meet(Join node) {
+ PCJMatcher matcher = PCJMatcherFactory.getPCJMatcher(node);
+ for (ExternalTupleSet pcj : pcjs) {
+ matcher.matchPCJ(pcj);
+ }
+
+ node.replaceWith(matcher.getQuery());
+ Set<TupleExpr> unmatched = matcher.getUnmatchedArgs();
+ PCJOptimizerUtilities.relocateFilters(matcher.getFilters());
+
+ for (TupleExpr tupleExpr : unmatched) {
+ tupleExpr.visit(this);
+ }
+ }
+
+ @Override
+ public void meet(LeftJoin node) {
+ PCJMatcher matcher = PCJMatcherFactory.getPCJMatcher(node);
+ for (ExternalTupleSet pcj : pcjs) {
+ matcher.matchPCJ(pcj);
+ }
+
+ node.replaceWith(matcher.getQuery());
+ Set<TupleExpr> unmatched = matcher.getUnmatchedArgs();
+ PCJOptimizerUtilities.relocateFilters(matcher.getFilters());
+
+ for (TupleExpr tupleExpr : unmatched) {
+ tupleExpr.visit(this);
+ }
+ }
+
+ @Override
+ public void meet(Filter node) {
+ PCJMatcher matcher = PCJMatcherFactory.getPCJMatcher(node);
+ for (ExternalTupleSet pcj : pcjs) {
+ matcher.matchPCJ(pcj);
+ }
+
+ node.replaceWith(matcher.getQuery());
+ Set<TupleExpr> unmatched = matcher.getUnmatchedArgs();
+ PCJOptimizerUtilities.relocateFilters(matcher.getFilters());
+
+ for (TupleExpr tupleExpr : unmatched) {
+ tupleExpr.visit(this);
+ }
+ }
+
+ }
+
+ /**
+ *
--- End diff --
docs
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-rya pull request #60: Rya 81
Posted by kchilton2 <gi...@git.apache.org>.
Github user kchilton2 closed the pull request at:
https://github.com/apache/incubator-rya/pull/60
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---