You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@rya.apache.org by DLotts <gi...@git.apache.org> on 2017/03/31 18:13:52 UTC
[GitHub] incubator-rya pull request #149: RYA-266 Added calls to init() where-ever th...
GitHub user DLotts opened a pull request:
https://github.com/apache/incubator-rya/pull/149
RYA-266 Added calls to init() where-ever the Accumulo temporal indexer is used for storing
## Description
The dao manages multitable batch writer, but that was not being used, so writers might never be flushed.
### Tests
Repaired tests. No new tests.
### Links
[Jira](https://issues.apache.org/jira/browse/RYA-266)
### Checklist
- [ ] Code Review
#### People To Reivew
People that know indexing, or know RyaOutputFormatTest
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/DLotts/incubator-rya RYA-266_tempo_flush
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-rya/pull/149.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #149
----
commit 72925fe26719c7c006aced94ba975d959655d008
Author: David W. Lotts <da...@parsons.com>
Date: 2017-03-31T18:05:07Z
RYA-266 Added init() and other calls where ever the Accumulo temporal indexer is created and used for storing.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-rya issue #149: RYA-266 Added calls to init() where-ever the Accum...
Posted by DLotts <gi...@git.apache.org>.
Github user DLotts commented on the issue:
https://github.com/apache/incubator-rya/pull/149
I think this fix is important and should be in the next release: 3.2.11
Let me know if you disagree. works great on my pc. I am typing this before jenkins finishes.
---
[GitHub] incubator-rya pull request #149: RYA-266 Added calls to init() where-ever th...
Posted by meiercaleb <gi...@git.apache.org>.
Github user meiercaleb commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/149#discussion_r134584063
--- Diff: extras/indexing/src/main/java/org/apache/rya/indexing/accumulo/temporal/AccumuloTemporalIndexer.java ---
@@ -102,42 +103,50 @@
private boolean isInit = false;
+ /**
+ * intilize the temporal index.
+ * This is dependent on a few set method calls before init:
+ * > Connector = ConfigUtils.getConnector(conf);
+ * > MultiTableBatchWriter mtbw = connector.createMultiTableBatchWriter(new BatchWriterConfig());
+ * > // optional: temporal.setConnector(connector);
+ * > temporal.setMultiTableBatchWriter(mtbw);
+ * > temporal.init();
+ */
+ @Override
+ public void init() {
+ if (!isInit) {
+ try {
+ initInternal();
+ isInit = true;
+ } catch (final AccumuloException | AccumuloSecurityException | TableNotFoundException | TableExistsException | RyaClientException e) {
+ logger.warn("Unable to initialize index. Throwing Runtime Exception. ", e);
+ throw new RuntimeException(e);
+ }
+ }
+ }
private void initInternal() throws AccumuloException, AccumuloSecurityException, TableNotFoundException,
- TableExistsException {
+ TableExistsException, RyaClientException {
+ if (mtbw == null)
+ throw new RyaClientException("Failed to initialize temporal index, setMultiTableBatchWriter() was not set.");
+ if (conf == null)
+ throw new RyaClientException("Failed to initialize temporal index, setConf() was not set.");
+
temporalIndexTableName = getTableName();
// Create one index table on first run.
- ConfigUtils.createTableIfNotExists(conf, temporalIndexTableName);
-
- mtbw = ConfigUtils.createMultitableBatchWriter(conf);
-
+ Boolean isCreated = ConfigUtils.createTableIfNotExists(conf, temporalIndexTableName);
+ if (isCreated) {
+ logger.info("First run, created temporal index table: " + temporalIndexTableName);
+ }
temporalIndexBatchWriter = mtbw.getBatchWriter(temporalIndexTableName);
-
validPredicates = ConfigUtils.getTemporalPredicates(conf);
}
//initialization occurs in setConf because index is created using reflection
@Override
public void setConf(final Configuration conf) {
this.conf = conf;
- if (!isInit) {
--- End diff --
Unfortunately, you cannot delete this method. This is how we initialize our Indices when they are created via reflection in the AccumuloRyaDAO.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-rya pull request #149: RYA-266 Added calls to init() where-ever th...
Posted by meiercaleb <gi...@git.apache.org>.
Github user meiercaleb commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/149#discussion_r136105065
--- Diff: extras/indexing/src/main/java/org/apache/rya/indexing/accumulo/temporal/AccumuloTemporalIndexer.java ---
@@ -158,16 +167,20 @@ public Configuration getConf() {
private void storeStatement(final Statement statement) throws IOException, IllegalArgumentException {
// if the predicate list is empty, accept all predicates.
// Otherwise, make sure the predicate is on the "valid" list
- final boolean isValidPredicate = validPredicates.isEmpty() || validPredicates.contains(statement.getPredicate());
+ final boolean isValidPredicate = validPredicates == null || validPredicates.isEmpty() || validPredicates.contains(statement.getPredicate());
if (!isValidPredicate || !(statement.getObject() instanceof Literal)) {
return;
}
+
final DateTime[] indexDateTimes = new DateTime[2]; // 0 begin, 1 end of interval
extractDateTime(statement, indexDateTimes);
if (indexDateTimes[0]==null) {
return;
}
+ if (!this.isInit)
--- End diff --
Okay.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-rya issue #149: RYA-266 Added calls to init() where-ever the Accum...
Posted by DLotts <gi...@git.apache.org>.
Github user DLotts commented on the issue:
https://github.com/apache/incubator-rya/pull/149
asfbot build
---
[GitHub] incubator-rya pull request #149: RYA-266 Added calls to init() where-ever th...
Posted by meiercaleb <gi...@git.apache.org>.
Github user meiercaleb commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/149#discussion_r134583719
--- Diff: extras/indexing/src/main/java/org/apache/rya/indexing/accumulo/temporal/AccumuloTemporalIndexer.java ---
@@ -158,16 +167,20 @@ public Configuration getConf() {
private void storeStatement(final Statement statement) throws IOException, IllegalArgumentException {
// if the predicate list is empty, accept all predicates.
// Otherwise, make sure the predicate is on the "valid" list
- final boolean isValidPredicate = validPredicates.isEmpty() || validPredicates.contains(statement.getPredicate());
+ final boolean isValidPredicate = validPredicates == null || validPredicates.isEmpty() || validPredicates.contains(statement.getPredicate());
if (!isValidPredicate || !(statement.getObject() instanceof Literal)) {
return;
}
+
final DateTime[] indexDateTimes = new DateTime[2]; // 0 begin, 1 end of interval
extractDateTime(statement, indexDateTimes);
if (indexDateTimes[0]==null) {
return;
}
+ if (!this.isInit)
--- End diff --
i.e., check if it has been initialized, and if it hasn't, then call init()
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-rya issue #149: RYA-266 Added calls to init() where-ever the Accum...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit commented on the issue:
https://github.com/apache/incubator-rya/pull/149
Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/incubator-rya-master-with-optionals-pull-requests/289/<h2>Failed Tests: <span class='status-failure'>2</span></h2><h3><a name='incubator-rya-master-with-optionals-pull-requests/org.apache.rya:rya.indexing.example' /><a href='https://builds.apache.org/job/incubator-rya-master-with-optionals-pull-requests/289/org.apache.rya$rya.indexing.example/testReport'>incubator-rya-master-with-optionals-pull-requests/org.apache.rya:rya.indexing.example</a>: <span class='status-failure'>1</span></h3><ul><li><a href='https://builds.apache.org/job/incubator-rya-master-with-optionals-pull-requests/289/org.apache.rya$rya.indexing.example/testReport/(root)/ExamplesTest/RyaDirectExampleTest/'><strong>ExamplesTest.RyaDirectExampleTest</strong></a></li></ul><h3><a name='incubator-rya-master-with-optionals-pull-requests/org.apache.rya:rya.pcj.fluo.integration' /><a href='https://builds.apache.org/job/incubator-rya-master-with-optionals-pull-requests/289/org.apache.rya$rya.p
cj.fluo.integration/testReport'>incubator-rya-master-with-optionals-pull-requests/org.apache.rya:rya.pcj.fluo.integration</a>: <span class='status-failure'>1</span></h3><ul><li><a href='https://builds.apache.org/job/incubator-rya-master-with-optionals-pull-requests/289/org.apache.rya$rya.pcj.fluo.integration/testReport/org.apache.rya.indexing.pcj.fluo.integration/KafkaExportIT/embeddedKafkaTest/'><strong>org.apache.rya.indexing.pcj.fluo.integration.KafkaExportIT.embeddedKafkaTest</strong></a></li></ul>
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-rya issue #149: RYA-266 Added calls to init() where-ever the Accum...
Posted by DLotts <gi...@git.apache.org>.
Github user DLotts commented on the issue:
https://github.com/apache/incubator-rya/pull/149
asfbot build
---
[GitHub] incubator-rya issue #149: RYA-266 Added calls to init() where-ever the Accum...
Posted by pujav65 <gi...@git.apache.org>.
Github user pujav65 commented on the issue:
https://github.com/apache/incubator-rya/pull/149
asfbot build
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-rya pull request #149: RYA-266 Added calls to init() where-ever th...
Posted by DLotts <gi...@git.apache.org>.
Github user DLotts commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/149#discussion_r135073929
--- Diff: extras/indexing/src/main/java/org/apache/rya/indexing/accumulo/temporal/AccumuloTemporalIndexer.java ---
@@ -158,16 +167,20 @@ public Configuration getConf() {
private void storeStatement(final Statement statement) throws IOException, IllegalArgumentException {
// if the predicate list is empty, accept all predicates.
// Otherwise, make sure the predicate is on the "valid" list
- final boolean isValidPredicate = validPredicates.isEmpty() || validPredicates.contains(statement.getPredicate());
+ final boolean isValidPredicate = validPredicates == null || validPredicates.isEmpty() || validPredicates.contains(statement.getPredicate());
if (!isValidPredicate || !(statement.getObject() instanceof Literal)) {
return;
}
+
final DateTime[] indexDateTimes = new DateTime[2]; // 0 begin, 1 end of interval
extractDateTime(statement, indexDateTimes);
if (indexDateTimes[0]==null) {
return;
}
+ if (!this.isInit)
--- End diff --
Indexes should not create their own batchwriter because it does not get flushed properly.
If we call init() here, then the batchwriter may not have been set. If it did not initialize at this point, then this is a bug and should fail.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-rya issue #149: RYA-266 Added calls to init() where-ever the Accum...
Posted by pujav65 <gi...@git.apache.org>.
Github user pujav65 commented on the issue:
https://github.com/apache/incubator-rya/pull/149
asfbot build
---
[GitHub] incubator-rya pull request #149: RYA-266 Added calls to init() where-ever th...
Posted by meiercaleb <gi...@git.apache.org>.
Github user meiercaleb commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/149#discussion_r136105878
--- Diff: mapreduce/src/main/java/org/apache/rya/accumulo/mr/RyaOutputFormat.java ---
@@ -216,12 +215,22 @@ private static FreeTextIndexer getFreeTextIndexer(Configuration conf) {
return freeText;
}
- private static TemporalIndexer getTemporalIndexer(Configuration conf) {
+ private static TemporalIndexer getTemporalIndexer(Configuration conf) throws IOException {
if (!conf.getBoolean(ENABLE_TEMPORAL, true)) {
return null;
}
AccumuloTemporalIndexer temporal = new AccumuloTemporalIndexer();
temporal.setConf(conf);
+ Connector connector;
+ try {
+ connector = ConfigUtils.getConnector(conf);
+ } catch (AccumuloException | AccumuloSecurityException e) {
+ throw new IOException("Error when attempting to create a connection for writing the temporal index.", e);
+ }
+ MultiTableBatchWriter mtbw = connector.createMultiTableBatchWriter(new BatchWriterConfig());
--- End diff --
Why are you creating the multitable batch writer here? Can't you use the same batchwriter that the other indexes are using? Have your getTemporalIndexer(...) method take in a MultiTableBatchWriter as well.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-rya pull request #149: RYA-266 Added calls to init() where-ever th...
Posted by meiercaleb <gi...@git.apache.org>.
Github user meiercaleb commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/149#discussion_r136104402
--- Diff: extras/indexing/src/main/java/org/apache/rya/indexing/accumulo/temporal/AccumuloTemporalIndexer.java ---
@@ -102,42 +103,50 @@
private boolean isInit = false;
+ /**
+ * intilize the temporal index.
+ * This is dependent on a few set method calls before init:
+ * > Connector = ConfigUtils.getConnector(conf);
+ * > MultiTableBatchWriter mtbw = connector.createMultiTableBatchWriter(new BatchWriterConfig());
+ * > // optional: temporal.setConnector(connector);
+ * > temporal.setMultiTableBatchWriter(mtbw);
+ * > temporal.init();
+ */
+ @Override
+ public void init() {
+ if (!isInit) {
+ try {
+ initInternal();
+ isInit = true;
+ } catch (final AccumuloException | AccumuloSecurityException | TableNotFoundException | TableExistsException | RyaClientException e) {
+ logger.warn("Unable to initialize index. Throwing Runtime Exception. ", e);
+ throw new RuntimeException(e);
+ }
+ }
+ }
private void initInternal() throws AccumuloException, AccumuloSecurityException, TableNotFoundException,
- TableExistsException {
+ TableExistsException, RyaClientException {
+ if (mtbw == null)
+ throw new RyaClientException("Failed to initialize temporal index, setMultiTableBatchWriter() was not set.");
+ if (conf == null)
+ throw new RyaClientException("Failed to initialize temporal index, setConf() was not set.");
+
temporalIndexTableName = getTableName();
// Create one index table on first run.
- ConfigUtils.createTableIfNotExists(conf, temporalIndexTableName);
-
- mtbw = ConfigUtils.createMultitableBatchWriter(conf);
-
+ Boolean isCreated = ConfigUtils.createTableIfNotExists(conf, temporalIndexTableName);
+ if (isCreated) {
+ logger.info("First run, created temporal index table: " + temporalIndexTableName);
+ }
temporalIndexBatchWriter = mtbw.getBatchWriter(temporalIndexTableName);
-
validPredicates = ConfigUtils.getTemporalPredicates(conf);
}
//initialization occurs in setConf because index is created using reflection
@Override
public void setConf(final Configuration conf) {
this.conf = conf;
- if (!isInit) {
--- End diff --
After looking at this a bit more, you are right. I think this is the way to go. I was confusing how we construct our Indexes with how we construct our optimizers. I think this is fine.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-rya pull request #149: RYA-266 Added calls to init() where-ever th...
Posted by meiercaleb <gi...@git.apache.org>.
Github user meiercaleb commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/149#discussion_r134583614
--- Diff: extras/indexing/src/main/java/org/apache/rya/indexing/accumulo/temporal/AccumuloTemporalIndexer.java ---
@@ -158,16 +167,20 @@ public Configuration getConf() {
private void storeStatement(final Statement statement) throws IOException, IllegalArgumentException {
// if the predicate list is empty, accept all predicates.
// Otherwise, make sure the predicate is on the "valid" list
- final boolean isValidPredicate = validPredicates.isEmpty() || validPredicates.contains(statement.getPredicate());
+ final boolean isValidPredicate = validPredicates == null || validPredicates.isEmpty() || validPredicates.contains(statement.getPredicate());
if (!isValidPredicate || !(statement.getObject() instanceof Literal)) {
return;
}
+
final DateTime[] indexDateTimes = new DateTime[2]; // 0 begin, 1 end of interval
extractDateTime(statement, indexDateTimes);
if (indexDateTimes[0]==null) {
return;
}
+ if (!this.isInit)
--- End diff --
Why not just call init at the beginning of this method to ensure that everything has been initialized?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-rya issue #149: RYA-266 Added calls to init() where-ever the Accum...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit commented on the issue:
https://github.com/apache/incubator-rya/pull/149
Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/incubator-rya-master-with-optionals-pull-requests/495/<h2>Failed Tests: <span class='status-failure'>1</span></h2><h3><a name='incubator-rya-master-with-optionals-pull-requests/org.apache.rya:rya.indexing.example' /><a href='https://builds.apache.org/job/incubator-rya-master-with-optionals-pull-requests/495/org.apache.rya$rya.indexing.example/testReport'>incubator-rya-master-with-optionals-pull-requests/org.apache.rya:rya.indexing.example</a>: <span class='status-failure'>1</span></h3><ul><li><a href='https://builds.apache.org/job/incubator-rya-master-with-optionals-pull-requests/495/org.apache.rya$rya.indexing.example/testReport/(root)/ExamplesTest/RyaDirectExampleTest/'><strong>ExamplesTest.RyaDirectExampleTest</strong></a></li></ul>
---
[GitHub] incubator-rya pull request #149: RYA-266 Added calls to init() where-ever th...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/incubator-rya/pull/149
---
[GitHub] incubator-rya issue #149: RYA-266 Added calls to init() where-ever the Accum...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit commented on the issue:
https://github.com/apache/incubator-rya/pull/149
Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/incubator-rya-master-with-optionals-pull-requests/498/<h2>Failed Tests: <span class='status-failure'>1</span></h2><h3><a name='incubator-rya-master-with-optionals-pull-requests/org.apache.rya:rya.mapreduce' /><a href='https://builds.apache.org/job/incubator-rya-master-with-optionals-pull-requests/498/org.apache.rya$rya.mapreduce/testReport'>incubator-rya-master-with-optionals-pull-requests/org.apache.rya:rya.mapreduce</a>: <span class='status-failure'>1</span></h3><ul><li><a href='https://builds.apache.org/job/incubator-rya-master-with-optionals-pull-requests/498/org.apache.rya$rya.mapreduce/testReport/org.apache.rya.accumulo.mr/RyaOutputFormatTest/testFreeTextIndexing/'><strong>org.apache.rya.accumulo.mr.RyaOutputFormatTest.testFreeTextIndexing</strong></a></li></ul>
---
[GitHub] incubator-rya issue #149: RYA-266 Added calls to init() where-ever the Accum...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit commented on the issue:
https://github.com/apache/incubator-rya/pull/149
Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/incubator-rya-master-with-optionals-pull-requests/500/<h2>Failed Tests: <span class='status-failure'>3</span></h2><h3><a name='incubator-rya-master-with-optionals-pull-requests/org.apache.rya:rya.prospector' /><a href='https://builds.apache.org/job/incubator-rya-master-with-optionals-pull-requests/500/org.apache.rya$rya.prospector/testReport'>incubator-rya-master-with-optionals-pull-requests/org.apache.rya:rya.prospector</a>: <span class='status-failure'>3</span></h3><ul><li><a href='https://builds.apache.org/job/incubator-rya-master-with-optionals-pull-requests/500/org.apache.rya$rya.prospector/testReport/org.apache.rya.prospector.mr/ProspectorTest/testCount/'><strong>org.apache.rya.prospector.mr.ProspectorTest.testCount</strong></a></li><li><a href='https://builds.apache.org/job/incubator-rya-master-with-optionals-pull-requests/500/org.apache.rya$rya.prospector/testReport/org.apache.rya.prospector.service/ProspectorServiceEvalStatsDAOTest/testCount
/'><strong>org.apache.rya.prospector.service.ProspectorServiceEvalStatsDAOTest.testCount</strong></a></li><li><a href='https://builds.apache.org/job/incubator-rya-master-with-optionals-pull-requests/500/org.apache.rya$rya.prospector/testReport/org.apache.rya.prospector.service/ProspectorServiceEvalStatsDAOTest/testNoAuthsCount/'><strong>org.apache.rya.prospector.service.ProspectorServiceEvalStatsDAOTest.testNoAuthsCount</strong></a></li></ul>
---
[GitHub] incubator-rya pull request #149: RYA-266 Added calls to init() where-ever th...
Posted by DLotts <gi...@git.apache.org>.
Github user DLotts commented on a diff in the pull request:
https://github.com/apache/incubator-rya/pull/149#discussion_r135072243
--- Diff: extras/indexing/src/main/java/org/apache/rya/indexing/accumulo/temporal/AccumuloTemporalIndexer.java ---
@@ -102,42 +103,50 @@
private boolean isInit = false;
+ /**
+ * intilize the temporal index.
+ * This is dependent on a few set method calls before init:
+ * > Connector = ConfigUtils.getConnector(conf);
+ * > MultiTableBatchWriter mtbw = connector.createMultiTableBatchWriter(new BatchWriterConfig());
+ * > // optional: temporal.setConnector(connector);
+ * > temporal.setMultiTableBatchWriter(mtbw);
+ * > temporal.init();
+ */
+ @Override
+ public void init() {
+ if (!isInit) {
+ try {
+ initInternal();
+ isInit = true;
+ } catch (final AccumuloException | AccumuloSecurityException | TableNotFoundException | TableExistsException | RyaClientException e) {
+ logger.warn("Unable to initialize index. Throwing Runtime Exception. ", e);
+ throw new RuntimeException(e);
+ }
+ }
+ }
private void initInternal() throws AccumuloException, AccumuloSecurityException, TableNotFoundException,
- TableExistsException {
+ TableExistsException, RyaClientException {
+ if (mtbw == null)
+ throw new RyaClientException("Failed to initialize temporal index, setMultiTableBatchWriter() was not set.");
+ if (conf == null)
+ throw new RyaClientException("Failed to initialize temporal index, setConf() was not set.");
+
temporalIndexTableName = getTableName();
// Create one index table on first run.
- ConfigUtils.createTableIfNotExists(conf, temporalIndexTableName);
-
- mtbw = ConfigUtils.createMultitableBatchWriter(conf);
-
+ Boolean isCreated = ConfigUtils.createTableIfNotExists(conf, temporalIndexTableName);
+ if (isCreated) {
+ logger.info("First run, created temporal index table: " + temporalIndexTableName);
+ }
temporalIndexBatchWriter = mtbw.getBatchWriter(temporalIndexTableName);
-
validPredicates = ConfigUtils.getTemporalPredicates(conf);
}
//initialization occurs in setConf because index is created using reflection
@Override
public void setConf(final Configuration conf) {
this.conf = conf;
- if (!isInit) {
--- End diff --
In AccumuloRyaDAO, for each secondary index, it calls setConf, then creates a batchwriter, then sets the batchwriter, then calls init() last. Indexes should not create there own batchwriter because it does not get flushed properly.
If we call init() from setConf(), then the batchwriter does not exist yet. I think several other indexes have this issue.
This was discovered by someone outside our team because it does not fail for unit tests. I discussed it with Aaron. He said that Indexes should accept a batchwriter , and then implement init() to be called by AccumuloRyaDAO.
Am I missing something? Here is the code from AccumuloRyaDAO.
```
for (AccumuloIndexer index : secondaryIndexers) {
index.setConf(conf);
}
... Create a batch writer used by all indexes here ...
...
for (AccumuloIndexer index : secondaryIndexers) {
index.setConnector(connector);
index.setMultiTableBatchWriter(mt_bw);
index.init();
}
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-rya issue #149: RYA-266 Added calls to init() where-ever the Accum...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit commented on the issue:
https://github.com/apache/incubator-rya/pull/149
Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/incubator-rya-master-with-optionals-pull-requests/497/<h2>Failed Tests: <span class='status-failure'>1</span></h2><h3><a name='incubator-rya-master-with-optionals-pull-requests/org.apache.rya:rya.indexing.example' /><a href='https://builds.apache.org/job/incubator-rya-master-with-optionals-pull-requests/497/org.apache.rya$rya.indexing.example/testReport'>incubator-rya-master-with-optionals-pull-requests/org.apache.rya:rya.indexing.example</a>: <span class='status-failure'>1</span></h3><ul><li><a href='https://builds.apache.org/job/incubator-rya-master-with-optionals-pull-requests/497/org.apache.rya$rya.indexing.example/testReport/(root)/ExamplesTest/RyaDirectExampleTest/'><strong>ExamplesTest.RyaDirectExampleTest</strong></a></li></ul>
---
[GitHub] incubator-rya issue #149: RYA-266 Added calls to init() where-ever the Accum...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit commented on the issue:
https://github.com/apache/incubator-rya/pull/149
Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/incubator-rya-master-with-optionals-pull-requests/501/
---