You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@accumulo.apache.org by keith-turner <gi...@git.apache.org> on 2017/04/05 17:43:27 UTC
[GitHub] accumulo pull request #243: ACCUMULO-4619 fix split hanging on Error
GitHub user keith-turner opened a pull request:
https://github.com/apache/accumulo/pull/243
ACCUMULO-4619 fix split hanging on Error
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/keith-turner/accumulo ACCUMULO-4619
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/accumulo/pull/243.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #243
----
commit a245caa9a6030f72704265e023fafd2a3734239f
Author: Keith Turner <kt...@apache.org>
Date: 2017-04-05T17:42:20Z
ACCUMULO-4619 fix split hanging on Error
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] accumulo pull request #243: ACCUMULO-4619 fix split hanging on Error
Posted by keith-turner <gi...@git.apache.org>.
Github user keith-turner commented on a diff in the pull request:
https://github.com/apache/accumulo/pull/243#discussion_r110007369
--- Diff: core/src/main/java/org/apache/accumulo/core/client/impl/TableOperationsImpl.java ---
@@ -389,14 +389,17 @@ public void addSplits(String tableName, SortedSet<Text> partitionKeys) throws Ta
if (exception.get() != null) {
executor.shutdownNow();
Throwable excep = exception.get();
- if (excep instanceof TableNotFoundException)
- throw (TableNotFoundException) excep;
- else if (excep instanceof AccumuloException)
- throw (AccumuloException) excep;
- else if (excep instanceof AccumuloSecurityException)
- throw (AccumuloSecurityException) excep;
+ if (excep instanceof TableNotFoundException) {
+ TableNotFoundException tnfe = (TableNotFoundException) excep;
+ throw new TableNotFoundException(tableId, tableName, "Table not found by background thread", tnfe);
+ } else if (excep instanceof AccumuloSecurityException) {
+ // base == background accumulo security exception
+ AccumuloSecurityException base = (AccumuloSecurityException) excep;
+ throw new AccumuloSecurityException(base.getUser(), base.asThriftException().getCode(), base.getTableInfo(), excep);
+ } else if (excep instanceof Error)
+ throw new Error(excep);
--- End diff --
I modified the code to throw an Error and then ran the SplitIT.
Below is what the stack trace looks like when I just cast the error.
```
java.lang.NoSuchMethodError
at org.apache.accumulo.core.client.impl.TableOperationsImpl.addSplits(TableOperationsImpl.java:415)
at org.apache.accumulo.core.client.impl.TableOperationsImpl.access$0(TableOperationsImpl.java:412)
at org.apache.accumulo.core.client.impl.TableOperationsImpl$SplitTask.run(TableOperationsImpl.java:349)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
at java.lang.Thread.run(Thread.java:745)
```
below is what the stack trace looks like when I wrap the Error... .this has the important info that AddSplitIT.java line 62 called addSplits()
```
java.lang.Error: java.lang.NoSuchMethodError
at org.apache.accumulo.core.client.impl.TableOperationsImpl.addSplits(TableOperationsImpl.java:400)
at org.apache.accumulo.test.functional.AddSplitIT.addSplitTest(AddSplitIT.java:62)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.NoSuchMethodError
at org.apache.accumulo.core.client.impl.TableOperationsImpl.addSplits(TableOperationsImpl.java:415)
at org.apache.accumulo.core.client.impl.TableOperationsImpl.access$0(TableOperationsImpl.java:412)
at org.apache.accumulo.core.client.impl.TableOperationsImpl$SplitTask.run(TableOperationsImpl.java:349)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
... 1 more
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] accumulo issue #243: ACCUMULO-4619 fix split hanging on Error
Posted by mjwall <gi...@git.apache.org>.
Github user mjwall commented on the issue:
https://github.com/apache/accumulo/pull/243
Sweet, thank you sir.
On Tue, Apr 18, 2017 at 2:45 PM Keith Turner <no...@github.com>
wrote:
> @mjwall <https://github.com/mjwall> I merged this in 8c0f03a
> <https://github.com/apache/accumulo/commit/8c0f03ac2d8e06b220e2882d54914f2034625e15>
> and forgot to close the PR
>
> \u2014
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/apache/accumulo/pull/243#issuecomment-294942724>, or mute
> the thread
> <https://github.com/notifications/unsubscribe-auth/AAAJJpfzZVu7E5hspSRFUAa9uGHjT-55ks5rxQSsgaJpZM4M0lRZ>
> .
>
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] accumulo pull request #243: ACCUMULO-4619 fix split hanging on Error
Posted by keith-turner <gi...@git.apache.org>.
Github user keith-turner commented on a diff in the pull request:
https://github.com/apache/accumulo/pull/243#discussion_r110004090
--- Diff: core/src/main/java/org/apache/accumulo/core/client/impl/TableOperationsImpl.java ---
@@ -389,14 +389,17 @@ public void addSplits(String tableName, SortedSet<Text> partitionKeys) throws Ta
if (exception.get() != null) {
executor.shutdownNow();
Throwable excep = exception.get();
- if (excep instanceof TableNotFoundException)
- throw (TableNotFoundException) excep;
- else if (excep instanceof AccumuloException)
- throw (AccumuloException) excep;
- else if (excep instanceof AccumuloSecurityException)
- throw (AccumuloSecurityException) excep;
+ if (excep instanceof TableNotFoundException) {
+ TableNotFoundException tnfe = (TableNotFoundException) excep;
+ throw new TableNotFoundException(tableId, tableName, "Table not found by background thread", tnfe);
+ } else if (excep instanceof AccumuloSecurityException) {
+ // base == background accumulo security exception
+ AccumuloSecurityException base = (AccumuloSecurityException) excep;
+ throw new AccumuloSecurityException(base.getUser(), base.asThriftException().getCode(), base.getTableInfo(), excep);
+ } else if (excep instanceof Error)
+ throw new Error(excep);
--- End diff --
Thats the case I was rambling about earlier. Because if the user prints this stack trace, they will not see the code path that called TablerOps.addSplits(). They will just see the stack trace for some background thread that TablerOps.addSplits() created. So if your code has multiple places that call TablerOps.addSplits(), then you may not know which one triggered the exception.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] accumulo pull request #243: ACCUMULO-4619 fix split hanging on Error
Posted by keith-turner <gi...@git.apache.org>.
Github user keith-turner closed the pull request at:
https://github.com/apache/accumulo/pull/243
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] accumulo pull request #243: ACCUMULO-4619 fix split hanging on Error
Posted by joshelser <gi...@git.apache.org>.
Github user joshelser commented on a diff in the pull request:
https://github.com/apache/accumulo/pull/243#discussion_r110003407
--- Diff: core/src/main/java/org/apache/accumulo/core/client/impl/TableOperationsImpl.java ---
@@ -389,14 +389,17 @@ public void addSplits(String tableName, SortedSet<Text> partitionKeys) throws Ta
if (exception.get() != null) {
executor.shutdownNow();
Throwable excep = exception.get();
- if (excep instanceof TableNotFoundException)
- throw (TableNotFoundException) excep;
- else if (excep instanceof AccumuloException)
- throw (AccumuloException) excep;
- else if (excep instanceof AccumuloSecurityException)
- throw (AccumuloSecurityException) excep;
+ if (excep instanceof TableNotFoundException) {
+ TableNotFoundException tnfe = (TableNotFoundException) excep;
+ throw new TableNotFoundException(tableId, tableName, "Table not found by background thread", tnfe);
+ } else if (excep instanceof AccumuloSecurityException) {
+ // base == background accumulo security exception
+ AccumuloSecurityException base = (AccumuloSecurityException) excep;
+ throw new AccumuloSecurityException(base.getUser(), base.asThriftException().getCode(), base.getTableInfo(), excep);
+ } else if (excep instanceof Error)
+ throw new Error(excep);
--- End diff --
Why not just `throw (Error) excep`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] accumulo issue #243: ACCUMULO-4619 fix split hanging on Error
Posted by keith-turner <gi...@git.apache.org>.
Github user keith-turner commented on the issue:
https://github.com/apache/accumulo/pull/243
> If we do get an Error, do we want to keep running?
What I am doing is similar to what a Future would do, except it would wrap the Throwable with an ExecutionException. This is why I think this code would be nicer if it used Future. However, since it constructs a tree of parallel operations that's not easy to represent without CompletableFuture.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] accumulo issue #243: ACCUMULO-4619 fix split hanging on Error
Posted by keith-turner <gi...@git.apache.org>.
Github user keith-turner commented on the issue:
https://github.com/apache/accumulo/pull/243
@joshelser I agree with your comment about `throw (Error) excep`. However I have a general issue with all of the addsplit code that rethrows, it only shows the stack trace for the background thread. The stack trace for what called addsplits is not present. That could be critical debugging info. So I think all of these should do something like `throw new TableNotFoundException(...., backgroundTableNFE)`. Then a user can actually see what code of theirs called addsplits. I'll open another issue about this.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] accumulo issue #243: ACCUMULO-4619 fix split hanging on Error
Posted by keith-turner <gi...@git.apache.org>.
Github user keith-turner commented on the issue:
https://github.com/apache/accumulo/pull/243
In 9523996 I wrapped the exceptions instead of throwing an exception from a background thread. I also rethrew error as a wrapped error.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] accumulo pull request #243: ACCUMULO-4619 fix split hanging on Error
Posted by keith-turner <gi...@git.apache.org>.
Github user keith-turner commented on a diff in the pull request:
https://github.com/apache/accumulo/pull/243#discussion_r110004523
--- Diff: core/src/main/java/org/apache/accumulo/core/client/impl/TableOperationsImpl.java ---
@@ -389,14 +389,17 @@ public void addSplits(String tableName, SortedSet<Text> partitionKeys) throws Ta
if (exception.get() != null) {
executor.shutdownNow();
Throwable excep = exception.get();
- if (excep instanceof TableNotFoundException)
- throw (TableNotFoundException) excep;
- else if (excep instanceof AccumuloException)
- throw (AccumuloException) excep;
- else if (excep instanceof AccumuloSecurityException)
- throw (AccumuloSecurityException) excep;
+ if (excep instanceof TableNotFoundException) {
+ TableNotFoundException tnfe = (TableNotFoundException) excep;
+ throw new TableNotFoundException(tableId, tableName, "Table not found by background thread", tnfe);
+ } else if (excep instanceof AccumuloSecurityException) {
+ // base == background accumulo security exception
+ AccumuloSecurityException base = (AccumuloSecurityException) excep;
+ throw new AccumuloSecurityException(base.getUser(), base.asThriftException().getCode(), base.getTableInfo(), excep);
+ } else if (excep instanceof Error)
+ throw new Error(excep);
--- End diff --
I am only aware of this issue because I ran into it. I don't have the stack traces lying around that show the issue more clearly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] accumulo pull request #243: ACCUMULO-4619 fix split hanging on Error
Posted by joshelser <gi...@git.apache.org>.
Github user joshelser commented on a diff in the pull request:
https://github.com/apache/accumulo/pull/243#discussion_r110008157
--- Diff: core/src/main/java/org/apache/accumulo/core/client/impl/TableOperationsImpl.java ---
@@ -389,14 +389,17 @@ public void addSplits(String tableName, SortedSet<Text> partitionKeys) throws Ta
if (exception.get() != null) {
executor.shutdownNow();
Throwable excep = exception.get();
- if (excep instanceof TableNotFoundException)
- throw (TableNotFoundException) excep;
- else if (excep instanceof AccumuloException)
- throw (AccumuloException) excep;
- else if (excep instanceof AccumuloSecurityException)
- throw (AccumuloSecurityException) excep;
+ if (excep instanceof TableNotFoundException) {
+ TableNotFoundException tnfe = (TableNotFoundException) excep;
+ throw new TableNotFoundException(tableId, tableName, "Table not found by background thread", tnfe);
+ } else if (excep instanceof AccumuloSecurityException) {
+ // base == background accumulo security exception
+ AccumuloSecurityException base = (AccumuloSecurityException) excep;
+ throw new AccumuloSecurityException(base.getUser(), base.asThriftException().getCode(), base.getTableInfo(), excep);
+ } else if (excep instanceof Error)
+ throw new Error(excep);
--- End diff --
Aha, I see now. That was helpful. Thanks.
+1
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] accumulo issue #243: ACCUMULO-4619 fix split hanging on Error
Posted by joshelser <gi...@git.apache.org>.
Github user joshelser commented on the issue:
https://github.com/apache/accumulo/pull/243
> Do you think I should check the type and rethrow an Error?
The more I think about it, I still think there should be:
```java
if (excep instanceof Error) {
throw (Error) excep;
}
throw new RuntimeException(excep);
```
But, I haven't found "official" guidance in the case of Futures/Executors.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] accumulo issue #243: ACCUMULO-4619 fix split hanging on Error
Posted by keith-turner <gi...@git.apache.org>.
Github user keith-turner commented on the issue:
https://github.com/apache/accumulo/pull/243
> Seems reasonable. Any advantage to using an explicit uncaught exception handler longer term?
I was thinking about this and did not come to an firm conclusions.
I made some rambling comments on the Jira issue about what I learned while working on this. If I had only change the ExecutorService calls from `submit()` to `execute()` then this would have enabled the LoggingRunnable (created by the NamingThreadFactory) to log the error. However, the users code calling addsplits would still hang forever. That's why I also changed it to catch Throwable.
I think in master, this code could be changed to use completable future. Then the addsplits could wait on that CompletableFuture and it would automatically handle propagating throwables to the user.
I am not sure, but I suspect we could migrate other code to Future and CompletableFuture and leverage their abilities to handle exceptions/errors in other threads. I will try to use CompletableFuture when I merge forward and if that goes well open a general issue for using it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] accumulo issue #243: ACCUMULO-4619 fix split hanging on Error
Posted by joshelser <gi...@git.apache.org>.
Github user joshelser commented on the issue:
https://github.com/apache/accumulo/pull/243
If we do get an `Error`, do we want to keep running? I know there was at least one place where we tried to bail out if we got an Error. I'm worried about masking Errors that are unrecoverable by re-throwing them as RTEs.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] accumulo issue #243: ACCUMULO-4619 fix split hanging on Error
Posted by mjwall <gi...@git.apache.org>.
Github user mjwall commented on the issue:
https://github.com/apache/accumulo/pull/243
+1 Nice cleanup Keith.
I went back to https://issues.apache.org/jira/browse/ACCUMULO-4332 to see if we could do something similar but didn't see anything.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] accumulo issue #243: ACCUMULO-4619 fix split hanging on Error
Posted by mjwall <gi...@git.apache.org>.
Github user mjwall commented on the issue:
https://github.com/apache/accumulo/pull/243
@keith-turner is this ready or are there still outstanding questions?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] accumulo issue #243: ACCUMULO-4619 fix split hanging on Error
Posted by joshelser <gi...@git.apache.org>.
Github user joshelser commented on the issue:
https://github.com/apache/accumulo/pull/243
> What I am doing is similar to what a Future would do, except it would wrap the Throwable with an ExecutionException.
OK. If that is what a ThreadPool/Executor would do, then I agree with you. I had it in my mind that an Error should still be propagated.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] accumulo issue #243: ACCUMULO-4619 fix split hanging on Error
Posted by keith-turner <gi...@git.apache.org>.
Github user keith-turner commented on the issue:
https://github.com/apache/accumulo/pull/243
> I'm worried about masking Errors that are unrecoverable by re-throwing them as RTEs.
Do you think I should check the type and rethrow an Error?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] accumulo pull request #243: ACCUMULO-4619 fix split hanging on Error
Posted by joshelser <gi...@git.apache.org>.
Github user joshelser commented on a diff in the pull request:
https://github.com/apache/accumulo/pull/243#discussion_r110006888
--- Diff: core/src/main/java/org/apache/accumulo/core/client/impl/TableOperationsImpl.java ---
@@ -389,14 +389,17 @@ public void addSplits(String tableName, SortedSet<Text> partitionKeys) throws Ta
if (exception.get() != null) {
executor.shutdownNow();
Throwable excep = exception.get();
- if (excep instanceof TableNotFoundException)
- throw (TableNotFoundException) excep;
- else if (excep instanceof AccumuloException)
- throw (AccumuloException) excep;
- else if (excep instanceof AccumuloSecurityException)
- throw (AccumuloSecurityException) excep;
+ if (excep instanceof TableNotFoundException) {
+ TableNotFoundException tnfe = (TableNotFoundException) excep;
+ throw new TableNotFoundException(tableId, tableName, "Table not found by background thread", tnfe);
+ } else if (excep instanceof AccumuloSecurityException) {
+ // base == background accumulo security exception
+ AccumuloSecurityException base = (AccumuloSecurityException) excep;
+ throw new AccumuloSecurityException(base.getUser(), base.asThriftException().getCode(), base.getTableInfo(), excep);
+ } else if (excep instanceof Error)
+ throw new Error(excep);
--- End diff --
I am not yet groking the entirety of this scenario, but trust you know what you're describing. Can you leave a comment that it is intentional that the Error is re-wrapped?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] accumulo issue #243: ACCUMULO-4619 fix split hanging on Error
Posted by keith-turner <gi...@git.apache.org>.
Github user keith-turner commented on the issue:
https://github.com/apache/accumulo/pull/243
@mjwall I merged this in 8c0f03ac2d8e06b220e2882d54914f2034625e15 and forgot to close the PR
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---