You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@rya.apache.org by thn-w <gi...@git.apache.org> on 2017/06/08 18:27:25 UTC
[GitHub] incubator-rya pull request #169: RYA-284: updated to support bloom filter
GitHub user thn-w opened a pull request:
https://github.com/apache/incubator-rya/pull/169
RYA-284: updated to support bloom filter
## Description
Updated dao/accumulo.rya to support the use of bloom filter
### Tests
no new tests
### Links
[JIRA -> RYA-284](https://issues.apache.org/jira/browse/RYA-284)
### People To Review
@amihalik
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/thn-w/incubator-rya RYA-284
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-rya/pull/169.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #169
----
commit cb19ee589b64edade97c2d2254a1888218ed1d42
Author: Tri H. Nguyen <th...@gmail.com>
Date: 2017-06-08T18:18:55Z
RYA-284: updated to support bloom filter
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-rya pull request #169: RYA-284: updated to support bloom filter
Posted by thn-w <gi...@git.apache.org>.
Github user thn-w commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/169#discussion_r121130647
--- Diff: dao/accumulo.rya/src/main/java/org/apache/rya/accumulo/query/AccumuloRyaQueryEngine.java ---
@@ -374,56 +387,64 @@ protected String getData(RyaType ryaType) {
final TripleRowRegex tripleRowRegex = strategy.buildRegex(regexSubject, regexPredicate, regexObject, null, null);
final String table = layoutToTable(layout, tableLayoutStrategy);
- boolean useBatchScanner = ranges.size() > maxRanges;
+ final boolean useBatchScanner = ranges.size() > maxRanges;
FluentCloseableIterable<RyaStatement> results = null;
if (useBatchScanner) {
- BatchScanner scanner = connector.createBatchScanner(table, authorizations, numQueryThreads);
+ final BatchScanner scanner = connector.createBatchScanner(table, authorizations, numQueryThreads);
scanner.setRanges(ranges);
fillScanner(scanner, context, null, ttl, null, tripleRowRegex, ryaQuery.getConf());
- results = FluentCloseableIterable.from(new ScannerBaseCloseableIterable(scanner))
- .transform(keyValueToRyaStatementFunctionMap.get(layout));
- } else {
+ results = FluentCloseableIterable.from(new ScannerBaseCloseableIterable(scanner)).transform(keyValueToRyaStatementFunctionMap.get(layout));
+ }
+ else {
final RyaURI fcontext = context;
final RdfCloudTripleStoreConfiguration fconf = ryaQuery.getConf();
- FluentIterable<RyaStatement> fluent = FluentIterable.from(ranges)
- .transformAndConcat(new Function<Range, Iterable<Map.Entry<Key, Value>>>() {
- @Override
- public Iterable<Map.Entry<Key, Value>> apply(Range range) {
- try {
- Scanner scanner = connector.createScanner(table, authorizations);
- scanner.setRange(range);
- fillScanner(scanner, fcontext, null, ttl, null, tripleRowRegex, fconf);
- return scanner;
- } catch (Exception e) {
- throw new RuntimeException(e);
- }
- }
- }).transform(keyValueToRyaStatementFunctionMap.get(layout));
+ final FluentIterable<RyaStatement> fluent = FluentIterable.from(ranges).transformAndConcat(range -> {
+ try {
+ final Scanner scanner = connector.createScanner(table, authorizations);
+ scanner.setRange(range);
+ fillScanner(scanner, fcontext, null, ttl, null, tripleRowRegex, fconf);
+ return scanner;
+ }
+ catch (final Exception e) {
+ throw new RuntimeException(e);
+ }
+ }).transform(keyValueToRyaStatementFunctionMap.get(layout));
results = FluentCloseableIterable.from(CloseableIterables.wrap(fluent));
}
if (maxResults != null) {
results = results.limit(maxResults.intValue());
}
return results;
- } catch (Exception e) {
+ }
+ catch (final Exception e) {
throw new RyaDAOException(e);
}
}
- protected void fillScanner(ScannerBase scanner, RyaURI context, String qualifier, Long ttl, Long currentTime,
- TripleRowRegex tripleRowRegex, RdfCloudTripleStoreConfiguration conf) throws IOException {
- if (context != null && qualifier != null) {
+ private static Text createText(final byte[] data, final RyaType ryaType) throws RyaTypeResolverException {
--- End diff --
Okay. Will put up a new PR with the new method name.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-rya issue #169: RYA-284: updated to support bloom filter
Posted by amihalik <gi...@git.apache.org>.
Github user amihalik commented on the issue:
https://github.com/apache/incubator-rya/pull/169
Can you submit another PR with only the required changes? There are a lot of whitespace and final changes.
Normally I would power through this, but the github "ignore whitespace" hack doesn't seem to be working.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-rya issue #169: RYA-284: updated to support bloom filter
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit commented on the issue:
https://github.com/apache/incubator-rya/pull/169
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-rya pull request #169: RYA-284: updated to support bloom filter
Posted by thn-w <gi...@git.apache.org>.
Github user thn-w closed the pull request at:
https://github.com/apache/incubator-rya/pull/169
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-rya pull request #169: RYA-284: updated to support bloom filter
Posted by amihalik <gi...@git.apache.org>.
Github user amihalik commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/169#discussion_r120970729
--- Diff: dao/accumulo.rya/src/main/java/org/apache/rya/accumulo/query/AccumuloRyaQueryEngine.java ---
@@ -304,67 +311,73 @@ protected String getData(RyaType ryaType) {
// use range to set scanner
// populate scanner based on authorizations, ttl
- String table = layoutToTable(layout, tableLayoutStrategy);
- Scanner scanner = connector.createScanner(table, authorizations);
+ final String table = layoutToTable(layout, tableLayoutStrategy);
+ final Scanner scanner = connector.createScanner(table, authorizations);
scanner.setRange(range);
if (batchSize != null) {
scanner.setBatchSize(batchSize);
}
fillScanner(scanner, context, qualifier, ttl, currentTime, tripleRowRegex, ryaQuery.getConf());
- FluentCloseableIterable<RyaStatement> results = FluentCloseableIterable.from(new ScannerBaseCloseableIterable(scanner))
- .transform(keyValueToRyaStatementFunctionMap.get(layout));
+ FluentCloseableIterable<RyaStatement> results = FluentCloseableIterable.from(new ScannerBaseCloseableIterable(scanner)).transform(keyValueToRyaStatementFunctionMap.get(layout));
if (maxResults != null) {
results = results.limit(maxResults.intValue());
}
return results;
- } catch (Exception e) {
+ }
+ catch (final Exception e) {
throw new RyaDAOException(e);
}
}
@Override
public CloseableIterable<RyaStatement> query(BatchRyaQuery ryaQuery) throws RyaDAOException {
Preconditions.checkNotNull(ryaQuery);
- Iterable<RyaStatement> stmts = ryaQuery.getQueries();
+ final Iterable<RyaStatement> stmts = ryaQuery.getQueries();
Preconditions.checkNotNull(stmts);
// query configuration
- String[] auths = ryaQuery.getAuths();
+ final String[] auths = ryaQuery.getAuths();
final Authorizations authorizations = auths != null ? new Authorizations(auths) : configuration.getAuthorizations();
final Long ttl = ryaQuery.getTtl();
- Long currentTime = ryaQuery.getCurrentTime();
- Long maxResults = ryaQuery.getMaxResults();
- Integer batchSize = ryaQuery.getBatchSize();
- Integer numQueryThreads = ryaQuery.getNumQueryThreads();
- String regexSubject = ryaQuery.getRegexSubject();
- String regexPredicate = ryaQuery.getRegexPredicate();
- String regexObject = ryaQuery.getRegexObject();
- TableLayoutStrategy tableLayoutStrategy = configuration.getTableLayoutStrategy();
- int maxRanges = ryaQuery.getMaxRanges();
+ final Long currentTime = ryaQuery.getCurrentTime();
+ final Long maxResults = ryaQuery.getMaxResults();
+ final Integer batchSize = ryaQuery.getBatchSize();
+ final Integer numQueryThreads = ryaQuery.getNumQueryThreads();
+ final String regexSubject = ryaQuery.getRegexSubject();
+ final String regexPredicate = ryaQuery.getRegexPredicate();
+ final String regexObject = ryaQuery.getRegexObject();
+ final TableLayoutStrategy tableLayoutStrategy = configuration.getTableLayoutStrategy();
+ final int maxRanges = ryaQuery.getMaxRanges();
// TODO: cannot span multiple tables here
try {
- Collection<Range> ranges = new HashSet<Range>();
+ final Collection<Range> ranges = new HashSet<Range>();
+
TABLE_LAYOUT layout = null;
RyaURI context = null;
+
TriplePatternStrategy strategy = null;
- for (RyaStatement stmt : stmts) {
+ for (final RyaStatement stmt : stmts) {
context = stmt.getContext(); // TODO: This will be overwritten
strategy = ryaContext.retrieveStrategy(stmt);
if (strategy == null) {
throw new IllegalArgumentException("TriplePattern[" + stmt + "] not supported");
}
- Map.Entry<RdfCloudTripleStoreConstants.TABLE_LAYOUT, ByteRange> entry = strategy.defineRange(stmt.getSubject(),
- stmt.getPredicate(), stmt.getObject(), stmt.getContext(), null);
+ final Map.Entry<RdfCloudTripleStoreConstants.TABLE_LAYOUT, ByteRange> entry = strategy.defineRange(stmt.getSubject(), stmt.getPredicate(), stmt.getObject(), stmt.getContext(), null);
// use range to set scanner
// populate scanner based on authorizations, ttl
layout = entry.getKey();
- ByteRange byteRange = entry.getValue();
+ final ByteRange byteRange = entry.getValue();
+
Range range = new Range(new Text(byteRange.getStart()), new Text(byteRange.getEnd()));
+ if ((stmt.getSubject() != null) && (stmt.getPredicate() != null) && (stmt.getObject() != null)) {
+ range = Range.exact(createText(byteRange.getStart(), stmt.getObject()));
--- End diff --
does `byteRange.getStart()` ever have the type appended to it at this point? (if so, then this won't work)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-rya issue #169: RYA-284: updated to support bloom filter
Posted by thn-w <gi...@git.apache.org>.
Github user thn-w commented on the issue:
https://github.com/apache/incubator-rya/pull/169
Closed and will submit a new PR as requested.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-rya pull request #169: RYA-284: updated to support bloom filter
Posted by amihalik <gi...@git.apache.org>.
Github user amihalik commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/169#discussion_r120970204
--- Diff: dao/accumulo.rya/src/main/java/org/apache/rya/accumulo/query/AccumuloRyaQueryEngine.java ---
@@ -374,56 +387,64 @@ protected String getData(RyaType ryaType) {
final TripleRowRegex tripleRowRegex = strategy.buildRegex(regexSubject, regexPredicate, regexObject, null, null);
final String table = layoutToTable(layout, tableLayoutStrategy);
- boolean useBatchScanner = ranges.size() > maxRanges;
+ final boolean useBatchScanner = ranges.size() > maxRanges;
FluentCloseableIterable<RyaStatement> results = null;
if (useBatchScanner) {
- BatchScanner scanner = connector.createBatchScanner(table, authorizations, numQueryThreads);
+ final BatchScanner scanner = connector.createBatchScanner(table, authorizations, numQueryThreads);
scanner.setRanges(ranges);
fillScanner(scanner, context, null, ttl, null, tripleRowRegex, ryaQuery.getConf());
- results = FluentCloseableIterable.from(new ScannerBaseCloseableIterable(scanner))
- .transform(keyValueToRyaStatementFunctionMap.get(layout));
- } else {
+ results = FluentCloseableIterable.from(new ScannerBaseCloseableIterable(scanner)).transform(keyValueToRyaStatementFunctionMap.get(layout));
+ }
+ else {
final RyaURI fcontext = context;
final RdfCloudTripleStoreConfiguration fconf = ryaQuery.getConf();
- FluentIterable<RyaStatement> fluent = FluentIterable.from(ranges)
- .transformAndConcat(new Function<Range, Iterable<Map.Entry<Key, Value>>>() {
- @Override
- public Iterable<Map.Entry<Key, Value>> apply(Range range) {
- try {
- Scanner scanner = connector.createScanner(table, authorizations);
- scanner.setRange(range);
- fillScanner(scanner, fcontext, null, ttl, null, tripleRowRegex, fconf);
- return scanner;
- } catch (Exception e) {
- throw new RuntimeException(e);
- }
- }
- }).transform(keyValueToRyaStatementFunctionMap.get(layout));
+ final FluentIterable<RyaStatement> fluent = FluentIterable.from(ranges).transformAndConcat(range -> {
+ try {
+ final Scanner scanner = connector.createScanner(table, authorizations);
+ scanner.setRange(range);
+ fillScanner(scanner, fcontext, null, ttl, null, tripleRowRegex, fconf);
+ return scanner;
+ }
+ catch (final Exception e) {
+ throw new RuntimeException(e);
+ }
+ }).transform(keyValueToRyaStatementFunctionMap.get(layout));
results = FluentCloseableIterable.from(CloseableIterables.wrap(fluent));
}
if (maxResults != null) {
results = results.limit(maxResults.intValue());
}
return results;
- } catch (Exception e) {
+ }
+ catch (final Exception e) {
throw new RyaDAOException(e);
}
}
- protected void fillScanner(ScannerBase scanner, RyaURI context, String qualifier, Long ttl, Long currentTime,
- TripleRowRegex tripleRowRegex, RdfCloudTripleStoreConfiguration conf) throws IOException {
- if (context != null && qualifier != null) {
+ private static Text createText(final byte[] data, final RyaType ryaType) throws RyaTypeResolverException {
--- End diff --
"appendType" would be a better name for this method.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-rya pull request #169: RYA-284: updated to support bloom filter
Posted by thn-w <gi...@git.apache.org>.
Github user thn-w commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/169#discussion_r121138068
--- Diff: dao/accumulo.rya/src/main/java/org/apache/rya/accumulo/query/AccumuloRyaQueryEngine.java ---
@@ -304,67 +311,73 @@ protected String getData(RyaType ryaType) {
// use range to set scanner
// populate scanner based on authorizations, ttl
- String table = layoutToTable(layout, tableLayoutStrategy);
- Scanner scanner = connector.createScanner(table, authorizations);
+ final String table = layoutToTable(layout, tableLayoutStrategy);
+ final Scanner scanner = connector.createScanner(table, authorizations);
scanner.setRange(range);
if (batchSize != null) {
scanner.setBatchSize(batchSize);
}
fillScanner(scanner, context, qualifier, ttl, currentTime, tripleRowRegex, ryaQuery.getConf());
- FluentCloseableIterable<RyaStatement> results = FluentCloseableIterable.from(new ScannerBaseCloseableIterable(scanner))
- .transform(keyValueToRyaStatementFunctionMap.get(layout));
+ FluentCloseableIterable<RyaStatement> results = FluentCloseableIterable.from(new ScannerBaseCloseableIterable(scanner)).transform(keyValueToRyaStatementFunctionMap.get(layout));
if (maxResults != null) {
results = results.limit(maxResults.intValue());
}
return results;
- } catch (Exception e) {
+ }
+ catch (final Exception e) {
throw new RyaDAOException(e);
}
}
@Override
public CloseableIterable<RyaStatement> query(BatchRyaQuery ryaQuery) throws RyaDAOException {
Preconditions.checkNotNull(ryaQuery);
- Iterable<RyaStatement> stmts = ryaQuery.getQueries();
+ final Iterable<RyaStatement> stmts = ryaQuery.getQueries();
Preconditions.checkNotNull(stmts);
// query configuration
- String[] auths = ryaQuery.getAuths();
+ final String[] auths = ryaQuery.getAuths();
final Authorizations authorizations = auths != null ? new Authorizations(auths) : configuration.getAuthorizations();
final Long ttl = ryaQuery.getTtl();
- Long currentTime = ryaQuery.getCurrentTime();
- Long maxResults = ryaQuery.getMaxResults();
- Integer batchSize = ryaQuery.getBatchSize();
- Integer numQueryThreads = ryaQuery.getNumQueryThreads();
- String regexSubject = ryaQuery.getRegexSubject();
- String regexPredicate = ryaQuery.getRegexPredicate();
- String regexObject = ryaQuery.getRegexObject();
- TableLayoutStrategy tableLayoutStrategy = configuration.getTableLayoutStrategy();
- int maxRanges = ryaQuery.getMaxRanges();
+ final Long currentTime = ryaQuery.getCurrentTime();
+ final Long maxResults = ryaQuery.getMaxResults();
+ final Integer batchSize = ryaQuery.getBatchSize();
+ final Integer numQueryThreads = ryaQuery.getNumQueryThreads();
+ final String regexSubject = ryaQuery.getRegexSubject();
+ final String regexPredicate = ryaQuery.getRegexPredicate();
+ final String regexObject = ryaQuery.getRegexObject();
+ final TableLayoutStrategy tableLayoutStrategy = configuration.getTableLayoutStrategy();
+ final int maxRanges = ryaQuery.getMaxRanges();
// TODO: cannot span multiple tables here
try {
- Collection<Range> ranges = new HashSet<Range>();
+ final Collection<Range> ranges = new HashSet<Range>();
+
TABLE_LAYOUT layout = null;
RyaURI context = null;
+
TriplePatternStrategy strategy = null;
- for (RyaStatement stmt : stmts) {
+ for (final RyaStatement stmt : stmts) {
context = stmt.getContext(); // TODO: This will be overwritten
strategy = ryaContext.retrieveStrategy(stmt);
if (strategy == null) {
throw new IllegalArgumentException("TriplePattern[" + stmt + "] not supported");
}
- Map.Entry<RdfCloudTripleStoreConstants.TABLE_LAYOUT, ByteRange> entry = strategy.defineRange(stmt.getSubject(),
- stmt.getPredicate(), stmt.getObject(), stmt.getContext(), null);
+ final Map.Entry<RdfCloudTripleStoreConstants.TABLE_LAYOUT, ByteRange> entry = strategy.defineRange(stmt.getSubject(), stmt.getPredicate(), stmt.getObject(), stmt.getContext(), null);
// use range to set scanner
// populate scanner based on authorizations, ttl
layout = entry.getKey();
- ByteRange byteRange = entry.getValue();
+ final ByteRange byteRange = entry.getValue();
+
Range range = new Range(new Text(byteRange.getStart()), new Text(byteRange.getEnd()));
+ if ((stmt.getSubject() != null) && (stmt.getPredicate() != null) && (stmt.getObject() != null)) {
+ range = Range.exact(createText(byteRange.getStart(), stmt.getObject()));
--- End diff --
that what createText (current method name) is supposed to do.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---