You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@metron.apache.org by nickwallen <gi...@git.apache.org> on 2016/09/14 18:10:33 UTC
[GitHub] incubator-metron pull request #251: METRON-418 Set TTL on HBase Puts
GitHub user nickwallen opened a pull request:
https://github.com/apache/incubator-metron/pull/251
METRON-418 Set TTL on HBase Puts
For METRON-417, the goal is to use HBase's TTL mechanism to expire or age-out Profile data. This PR provides the mechanism to set TTL values on HBase Puts.
The HBaseClient is used to write ProfileMeasurement values to HBase. The client was updated so that a TTL can be attached to each Put. This will allow us to attach a TTL to each ProfileMeasurement written by the Profiler.
Tests were added to validate successful reads/write to HBase with a TTL, along with successful expiration of data based on a TTL.
Depends on #246 .
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/nickwallen/incubator-metron METRON-418
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-metron/pull/251.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #251
----
commit c2ffd4a94ed59da718ebc17d5d093bb2085e508e
Author: Nick Allen <ni...@nickallen.org>
Date: 2016-09-10T02:10:03Z
METRON-411 Support Greater Range of Profile Periods
commit f259ea09545fb7a334ab2d763e73ba051b3d44a3
Author: Nick Allen <ni...@nickallen.org>
Date: 2016-09-14T18:04:23Z
METRON-418 Set TTL on HBase Puts
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #251: METRON-418 Set TTL on HBase Puts
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on the issue:
https://github.com/apache/incubator-metron/pull/251
I'm betting travis is overloaded, try closing and reopening.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #251: METRON-418 Set TTL on HBase Puts
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/251#discussion_r78999765
--- Diff: metron-analytics/metron-profiler-client/src/test/java/org/apache/metron/profiler/client/ProfileWriter.java ---
@@ -92,8 +92,8 @@ private void write(ProfileMeasurement m, List<Object> groups) {
byte[] rowKey = rowKeyBuilder.rowKey(m, groups);
ColumnList cols = columnBuilder.columns(m);
- List<Mutation> mutations = hbaseClient.constructMutationReq(rowKey, cols, Durability.SKIP_WAL);
- hbaseClient.batchMutate(mutations);
+ hbaseClient.addMutation(rowKey, cols, Durability.SKIP_WAL);
--- End diff --
I don't think you want to `SKIP_WAL` That will result in data loss on region server crash.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #251: METRON-418 Set TTL on HBase Puts
Posted by nickwallen <gi...@git.apache.org>.
GitHub user nickwallen reopened a pull request:
https://github.com/apache/incubator-metron/pull/251
METRON-418 Set TTL on HBase Puts
For METRON-417, the goal is to use HBase's TTL mechanism to expire or age-out Profile data. This PR provides the mechanism to set TTL values on HBase Puts.
The HBaseClient is used to write ProfileMeasurement values to HBase. The client was updated so that a TTL can be attached to each Put. This will allow us to attach a TTL to each ProfileMeasurement written by the Profiler.
Tests were added to validate successful reads/write to HBase with a TTL, along with successful expiration of data based on a TTL.
Depends on the following existing PRs...
- [x] #246
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/nickwallen/incubator-metron METRON-418
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-metron/pull/251.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #251
----
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #251: METRON-418 Set TTL on HBase Puts
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen closed the pull request at:
https://github.com/apache/incubator-metron/pull/251
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #251: METRON-418 Set TTL on HBase Puts
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on the issue:
https://github.com/apache/incubator-metron/pull/251
This looks good; I'm +1
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #251: METRON-418 Set TTL on HBase Puts
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/251#discussion_r79002242
--- Diff: metron-analytics/metron-profiler-client/src/test/java/org/apache/metron/profiler/client/ProfileWriter.java ---
@@ -92,8 +92,8 @@ private void write(ProfileMeasurement m, List<Object> groups) {
byte[] rowKey = rowKeyBuilder.rowKey(m, groups);
ColumnList cols = columnBuilder.columns(m);
- List<Mutation> mutations = hbaseClient.constructMutationReq(rowKey, cols, Durability.SKIP_WAL);
- hbaseClient.batchMutate(mutations);
+ hbaseClient.addMutation(rowKey, cols, Durability.SKIP_WAL);
--- End diff --
The `ProfileWriter` is only used for testing. It writes Profile data that I can then read with the client API. I have it in under `src/test/...`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #251: METRON-418 Set TTL on HBase Puts
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on the issue:
https://github.com/apache/incubator-metron/pull/251
CI failure saying that ElasticSearchIndexIntegrationTest took too long. Seems unrelated. Hmm.
```
test(org.apache.metron.elasticsearch.integration.ElasticsearchIndexingIntegrationTest) Time elapsed: 145.656 sec <<< ERROR!
java.lang.RuntimeException: Took too long to complete: 120038 > 120000
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #251: METRON-418 Set TTL on HBase Puts
Posted by nickwallen <gi...@git.apache.org>.
GitHub user nickwallen reopened a pull request:
https://github.com/apache/incubator-metron/pull/251
METRON-418 Set TTL on HBase Puts
For METRON-417, the goal is to use HBase's TTL mechanism to expire or age-out Profile data. This PR provides the mechanism to set TTL values on HBase Puts.
The HBaseClient is used to write ProfileMeasurement values to HBase. The client was updated so that a TTL can be attached to each Put. This will allow us to attach a TTL to each ProfileMeasurement written by the Profiler.
Tests were added to validate successful reads/write to HBase with a TTL, along with successful expiration of data based on a TTL.
Depends on the following existing PRs...
- [x] #246
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/nickwallen/incubator-metron METRON-418
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-metron/pull/251.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #251
----
commit c2ffd4a94ed59da718ebc17d5d093bb2085e508e
Author: Nick Allen <ni...@nickallen.org>
Date: 2016-09-10T02:10:03Z
METRON-411 Support Greater Range of Profile Periods
commit f259ea09545fb7a334ab2d763e73ba051b3d44a3
Author: Nick Allen <ni...@nickallen.org>
Date: 2016-09-14T18:04:23Z
METRON-418 Set TTL on HBase Puts
commit 97bad6e90866093bd15b7402adadfc4a7d35f679
Author: Nick Allen <ni...@nickallen.org>
Date: 2016-09-14T18:28:59Z
METRON-418 Missed a file in the commit
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #251: METRON-418 Set TTL on HBase Puts
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen closed the pull request at:
https://github.com/apache/incubator-metron/pull/251
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #251: METRON-418 Set TTL on HBase Puts
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/incubator-metron/pull/251
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron issue #251: METRON-418 Set TTL on HBase Puts
Posted by nickwallen <gi...@git.apache.org>.
Github user nickwallen commented on the issue:
https://github.com/apache/incubator-metron/pull/251
Reconciled with master.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-metron pull request #251: METRON-418 Set TTL on HBase Puts
Posted by cestella <gi...@git.apache.org>.
Github user cestella commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/251#discussion_r79014810
--- Diff: metron-analytics/metron-profiler-client/src/test/java/org/apache/metron/profiler/client/ProfileWriter.java ---
@@ -92,8 +92,8 @@ private void write(ProfileMeasurement m, List<Object> groups) {
byte[] rowKey = rowKeyBuilder.rowKey(m, groups);
ColumnList cols = columnBuilder.columns(m);
- List<Mutation> mutations = hbaseClient.constructMutationReq(rowKey, cols, Durability.SKIP_WAL);
- hbaseClient.batchMutate(mutations);
+ hbaseClient.addMutation(rowKey, cols, Durability.SKIP_WAL);
--- End diff --
ahh, crap, yes, I see. nvm :)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---