You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@phoenix.apache.org by Dan Di Spaltro <da...@gmail.com> on 2014/09/01 02:27:09 UTC
Re: ManagedTests and 4.1.0-RC1
Oh and yes Ill create a JIRA today.
On Fri, Aug 29, 2014 at 5:09 PM, Dan Di Spaltro <da...@gmail.com>
wrote:
> If I disable the metrics stuff on every query everything works fine btw.
> Therefore, I don't think it's anything related to tests.
>
>
> On Fri, Aug 29, 2014 at 3:03 PM, Samarth Jain <sa...@gmail.com>
> wrote:
>
>> + Jesse
>>
>> I think Jesse probably envisioned early use of annotations to be around
>> numerical metrics, hence the String-Integer pair. But we should change it
>> to be more generic to accept things like tenantId or userName. Do you mind
>> filing an enhancement JIRA for this Dan?
>>
>> Thanks,
>> Samarth
>>
>>
>>
>> On Fri, Aug 29, 2014 at 2:51 PM, Dan Di Spaltro <da...@gmail.com>
>> wrote:
>>
>>> Okay that helps, but it still doesn't really explain that code path.
>>> Why are annotations being converted byte-wise to ints? Aren't annotations
>>> potentially opaque strings or something else?
>>>
>>> -Dan
>>>
>>>
>>> On Fri, Aug 29, 2014 at 11:27 AM, James Taylor <ja...@apache.org>
>>> wrote:
>>>
>>>> Hey Dan,
>>>> There were some changes in the test framework to make them run faster.
>>>> Our entire test suite can run in about 10-15mins instead of 60mins
>>>> now. One of the new requirements is adding the annotation that Samarth
>>>> indicated. Once JUnit releases 4.12, this will no longer be necessary,
>>>> as the annotation will automatically be inherited from the
>>>> BaseClientManagedTimeIT class.
>>>> Thanks,
>>>> James
>>>>
>>>> On Thu, Aug 28, 2014 at 10:08 PM, Dan Di Spaltro
>>>> <da...@gmail.com> wrote:
>>>> > I basically inherit from BaseClientManagedTimeIT and write a junit
>>>> tests
>>>> >
>>>> > It's been working great up until 4.1.
>>>> >
>>>> > This code just doesn't look right, why would an annotation
>>>> necessarily have
>>>> > to be an int?
>>>> >
>>>> >
>>>> https://github.com/apache/phoenix/blob/29a7be42bfa468b12d16fd0756b987f5359c45c4/phoenix-hadoop2-compat/src/main/java/org/apache/phoenix/trace/TraceMetricSource.java#L122
>>>> >
>>>> > then calls the below function, which takes bytes and makes an int
>>>> from the
>>>> > bytes...
>>>> >
>>>> >
>>>> https://github.com/apache/phoenix/blob/f99e5d8d609d326fb3571255cd8f47961b1c6860/phoenix-hadoop-compat/src/main/java/org/apache/phoenix/trace/TracingCompat.java#L56
>>>> >
>>>> >
>>>> > On Thu, Aug 28, 2014 at 11:43 AM, Samarth Jain <
>>>> samarth.jain@gmail.com>
>>>> > wrote:
>>>> >>
>>>> >> Dan,
>>>> >>
>>>> >> Can you tell me how you are running your tests? Do you have the test
>>>> class
>>>> >> annotated with the right category annotation -
>>>> >> @Category(HBaseManagedTimeTest.class). Also, can you send over your
>>>> test
>>>> >> class to see what might be causing problems?
>>>> >>
>>>> >> Thanks,
>>>> >> Samarth
>>>> >>
>>>> >>
>>>> >> On Thu, Aug 28, 2014 at 10:34 AM, Dan Di Spaltro <
>>>> dan.dispaltro@gmail.com>
>>>> >> wrote:
>>>> >>>
>>>> >>> Any idea on this, it's blocking my usage in tests and I can't tell
>>>> if I
>>>> >>> am just setting something up incorrectly? Also I am concerned that
>>>> this can
>>>> >>> affect production since this code path I would assume is used
>>>> frequently.
>>>> >>>
>>>> >>> -Dan
>>>> >>>
>>>> >>>
>>>> >>> On Tue, Aug 26, 2014 at 10:49 PM, Dan Di Spaltro
>>>> >>> <da...@gmail.com> wrote:
>>>> >>>>
>>>> >>>> I inherit from the BaseHBaseManagedTimeIT and implement my own
>>>> tests
>>>> >>>> using the infrastructure you've put together. It's worked pretty
>>>> well,
>>>> >>>> minus the fact I use an Ivy resolver which doesn't deal with
>>>> jarless pom's
>>>> >>>> well.
>>>> >>>>
>>>> >>>> So I've upgraded from 4.0 to 4.1 and ran into a single issue that
>>>> looks
>>>> >>>> related to Tracing, and I can't really figure it out. When I
>>>> start the
>>>> >>>> cluster everything works as expected but after I am done creating
>>>> tables
>>>> >>>> like clockwork I get this:
>>>> >>>>
>>>> >>>> 58062 [defaultRpcServer.handler=2,queue=0,port=53950] WARN
>>>> >>>> org.apache.hadoop.ipc.RpcServer -
>>>> >>>> defaultRpcServer.handler=2,queue=0,port=53950: caught:
>>>> >>>> java.lang.IllegalArgumentException: offset (0) + length (4) exceed
>>>> the
>>>> >>>> capacity of the array: 3
>>>> >>>> at
>>>> >>>>
>>>> org.apache.hadoop.hbase.util.Bytes.explainWrongLengthOrOffset(Bytes.java:600)
>>>> >>>> at org.apache.hadoop.hbase.util.Bytes.toInt(Bytes.java:749)
>>>> >>>> at org.apache.hadoop.hbase.util.Bytes.toInt(Bytes.java:725)
>>>> >>>> at
>>>> >>>>
>>>> org.apache.phoenix.trace.TracingCompat.readAnnotation(TracingCompat.java:56)
>>>> >>>> at
>>>> >>>>
>>>> org.apache.phoenix.trace.TraceMetricSource.receiveSpan(TraceMetricSource.java:121)
>>>> >>>> at org.cloudera.htrace.Tracer.deliver(Tracer.java:81)
>>>> >>>> at org.cloudera.htrace.impl.MilliSpan.stop(MilliSpan.java:70)
>>>> >>>> at org.cloudera.htrace.TraceScope.close(TraceScope.java:70)
>>>> >>>> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106)
>>>> >>>> at
>>>> >>>>
>>>> org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:114)
>>>> >>>> at
>>>> org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94)
>>>> >>>> at java.lang.Thread.run(Thread.java:744)
>>>> >>>>
>>>> >>>> And the test just stops, which I imagine is a byproduct of this
>>>> >>>> exception. I inspected at this point and there are two traces the
>>>> one it
>>>> >>>> throws on is the key is "user" and value is my username. It's
>>>> trying to
>>>> >>>> convert it to an int
>>>> >>>> ...
>>>> >>>> return new Pair<String, String>(new String(key),
>>>> >>>> Integer.toString(Bytes.toInt(value)));
>>>> >>>> ...
>>>> >>>>
>>>> >>>> Any ideas?
>>>> >>>>
>>>> >>>> -Dan
>>>> >>>>
>>>> >>>> --
>>>> >>>> Dan Di Spaltro
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>> --
>>>> >>> Dan Di Spaltro
>>>> >>
>>>> >>
>>>> >
>>>> >
>>>> >
>>>> > --
>>>> > Dan Di Spaltro
>>>>
>>>
>>>
>>>
>>> --
>>> Dan Di Spaltro
>>>
>>
>>
>
>
> --
> Dan Di Spaltro
>
--
Dan Di Spaltro
Re: ManagedTests and 4.1.0-RC1
Posted by Dan Di Spaltro <da...@gmail.com>.
Here's more detail on the exception and the related line where HBase is
adding non-integer KV's.
https://issues.apache.org/jira/browse/PHOENIX-1226
https://github.com/apache/hbase/blob/0.98.5/hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RequestContext.java#L105
Thanks for the help,
-Dan
On Sun, Aug 31, 2014 at 5:27 PM, Dan Di Spaltro <da...@gmail.com>
wrote:
> Oh and yes Ill create a JIRA today.
>
>
> On Fri, Aug 29, 2014 at 5:09 PM, Dan Di Spaltro <da...@gmail.com>
> wrote:
>
>> If I disable the metrics stuff on every query everything works fine btw.
>> Therefore, I don't think it's anything related to tests.
>>
>>
>> On Fri, Aug 29, 2014 at 3:03 PM, Samarth Jain <sa...@gmail.com>
>> wrote:
>>
>>> + Jesse
>>>
>>> I think Jesse probably envisioned early use of annotations to be around
>>> numerical metrics, hence the String-Integer pair. But we should change it
>>> to be more generic to accept things like tenantId or userName. Do you mind
>>> filing an enhancement JIRA for this Dan?
>>>
>>> Thanks,
>>> Samarth
>>>
>>>
>>>
>>> On Fri, Aug 29, 2014 at 2:51 PM, Dan Di Spaltro <dan.dispaltro@gmail.com
>>> > wrote:
>>>
>>>> Okay that helps, but it still doesn't really explain that code path.
>>>> Why are annotations being converted byte-wise to ints? Aren't annotations
>>>> potentially opaque strings or something else?
>>>>
>>>> -Dan
>>>>
>>>>
>>>> On Fri, Aug 29, 2014 at 11:27 AM, James Taylor <ja...@apache.org>
>>>> wrote:
>>>>
>>>>> Hey Dan,
>>>>> There were some changes in the test framework to make them run faster.
>>>>> Our entire test suite can run in about 10-15mins instead of 60mins
>>>>> now. One of the new requirements is adding the annotation that Samarth
>>>>> indicated. Once JUnit releases 4.12, this will no longer be necessary,
>>>>> as the annotation will automatically be inherited from the
>>>>> BaseClientManagedTimeIT class.
>>>>> Thanks,
>>>>> James
>>>>>
>>>>> On Thu, Aug 28, 2014 at 10:08 PM, Dan Di Spaltro
>>>>> <da...@gmail.com> wrote:
>>>>> > I basically inherit from BaseClientManagedTimeIT and write a junit
>>>>> tests
>>>>> >
>>>>> > It's been working great up until 4.1.
>>>>> >
>>>>> > This code just doesn't look right, why would an annotation
>>>>> necessarily have
>>>>> > to be an int?
>>>>> >
>>>>> >
>>>>> https://github.com/apache/phoenix/blob/29a7be42bfa468b12d16fd0756b987f5359c45c4/phoenix-hadoop2-compat/src/main/java/org/apache/phoenix/trace/TraceMetricSource.java#L122
>>>>> >
>>>>> > then calls the below function, which takes bytes and makes an int
>>>>> from the
>>>>> > bytes...
>>>>> >
>>>>> >
>>>>> https://github.com/apache/phoenix/blob/f99e5d8d609d326fb3571255cd8f47961b1c6860/phoenix-hadoop-compat/src/main/java/org/apache/phoenix/trace/TracingCompat.java#L56
>>>>> >
>>>>> >
>>>>> > On Thu, Aug 28, 2014 at 11:43 AM, Samarth Jain <
>>>>> samarth.jain@gmail.com>
>>>>> > wrote:
>>>>> >>
>>>>> >> Dan,
>>>>> >>
>>>>> >> Can you tell me how you are running your tests? Do you have the
>>>>> test class
>>>>> >> annotated with the right category annotation -
>>>>> >> @Category(HBaseManagedTimeTest.class). Also, can you send over your
>>>>> test
>>>>> >> class to see what might be causing problems?
>>>>> >>
>>>>> >> Thanks,
>>>>> >> Samarth
>>>>> >>
>>>>> >>
>>>>> >> On Thu, Aug 28, 2014 at 10:34 AM, Dan Di Spaltro <
>>>>> dan.dispaltro@gmail.com>
>>>>> >> wrote:
>>>>> >>>
>>>>> >>> Any idea on this, it's blocking my usage in tests and I can't tell
>>>>> if I
>>>>> >>> am just setting something up incorrectly? Also I am concerned
>>>>> that this can
>>>>> >>> affect production since this code path I would assume is used
>>>>> frequently.
>>>>> >>>
>>>>> >>> -Dan
>>>>> >>>
>>>>> >>>
>>>>> >>> On Tue, Aug 26, 2014 at 10:49 PM, Dan Di Spaltro
>>>>> >>> <da...@gmail.com> wrote:
>>>>> >>>>
>>>>> >>>> I inherit from the BaseHBaseManagedTimeIT and implement my own
>>>>> tests
>>>>> >>>> using the infrastructure you've put together. It's worked pretty
>>>>> well,
>>>>> >>>> minus the fact I use an Ivy resolver which doesn't deal with
>>>>> jarless pom's
>>>>> >>>> well.
>>>>> >>>>
>>>>> >>>> So I've upgraded from 4.0 to 4.1 and ran into a single issue that
>>>>> looks
>>>>> >>>> related to Tracing, and I can't really figure it out. When I
>>>>> start the
>>>>> >>>> cluster everything works as expected but after I am done creating
>>>>> tables
>>>>> >>>> like clockwork I get this:
>>>>> >>>>
>>>>> >>>> 58062 [defaultRpcServer.handler=2,queue=0,port=53950] WARN
>>>>> >>>> org.apache.hadoop.ipc.RpcServer -
>>>>> >>>> defaultRpcServer.handler=2,queue=0,port=53950: caught:
>>>>> >>>> java.lang.IllegalArgumentException: offset (0) + length (4)
>>>>> exceed the
>>>>> >>>> capacity of the array: 3
>>>>> >>>> at
>>>>> >>>>
>>>>> org.apache.hadoop.hbase.util.Bytes.explainWrongLengthOrOffset(Bytes.java:600)
>>>>> >>>> at org.apache.hadoop.hbase.util.Bytes.toInt(Bytes.java:749)
>>>>> >>>> at org.apache.hadoop.hbase.util.Bytes.toInt(Bytes.java:725)
>>>>> >>>> at
>>>>> >>>>
>>>>> org.apache.phoenix.trace.TracingCompat.readAnnotation(TracingCompat.java:56)
>>>>> >>>> at
>>>>> >>>>
>>>>> org.apache.phoenix.trace.TraceMetricSource.receiveSpan(TraceMetricSource.java:121)
>>>>> >>>> at org.cloudera.htrace.Tracer.deliver(Tracer.java:81)
>>>>> >>>> at org.cloudera.htrace.impl.MilliSpan.stop(MilliSpan.java:70)
>>>>> >>>> at org.cloudera.htrace.TraceScope.close(TraceScope.java:70)
>>>>> >>>> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:106)
>>>>> >>>> at
>>>>> >>>>
>>>>> org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:114)
>>>>> >>>> at
>>>>> org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94)
>>>>> >>>> at java.lang.Thread.run(Thread.java:744)
>>>>> >>>>
>>>>> >>>> And the test just stops, which I imagine is a byproduct of this
>>>>> >>>> exception. I inspected at this point and there are two traces
>>>>> the one it
>>>>> >>>> throws on is the key is "user" and value is my username. It's
>>>>> trying to
>>>>> >>>> convert it to an int
>>>>> >>>> ...
>>>>> >>>> return new Pair<String, String>(new String(key),
>>>>> >>>> Integer.toString(Bytes.toInt(value)));
>>>>> >>>> ...
>>>>> >>>>
>>>>> >>>> Any ideas?
>>>>> >>>>
>>>>> >>>> -Dan
>>>>> >>>>
>>>>> >>>> --
>>>>> >>>> Dan Di Spaltro
>>>>> >>>
>>>>> >>>
>>>>> >>>
>>>>> >>>
>>>>> >>> --
>>>>> >>> Dan Di Spaltro
>>>>> >>
>>>>> >>
>>>>> >
>>>>> >
>>>>> >
>>>>> > --
>>>>> > Dan Di Spaltro
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Dan Di Spaltro
>>>>
>>>
>>>
>>
>>
>> --
>> Dan Di Spaltro
>>
>
>
>
> --
> Dan Di Spaltro
>
--
Dan Di Spaltro