You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@phoenix.apache.org by Kristoffer Sjögren <st...@gmail.com> on 2014/09/30 00:17:16 UTC
Error in readFields
Hi
Our SELECT queries went broken just now and we're seeing the following
exception [1] using Apache Phoenix 2.2.3-incubating with HBase 0.94.6
(0.94.6-cdh4.4.0).
After a some testing I realized that sqlline.sh from the 2.2.3-incubating
tarball is also broken all of a sudden (with same error)?
Seems like SELECT ... WHERE queries are problematic whereas basic queries
like "SELECT * FROM COUNTERS LIMIT 1;" works fine. Simple INSERT statements
also seems to be working.
Is upgrading to phoenix-3.0.0-incubating an option? Or is there a way to
debug this further? I tried to enable DEBUG in log4j.properties using the
'console' in sqlline but without success.
Our production servers are pretty much down at the moment...
Cheers,
-Kristoffer
[1] http://pastebin.com/H2a6xpBX
Re: Error in readFields
Posted by Kristoffer Sjögren <st...@gmail.com>.
We have found and fixed the problem now. It was related faulty upgrade
script where the server started before the phoenix jar was put in the lib
directory.
I suspect range queries and inserts that spanned rows belonging to this
particular region server failed, while other queries succeeded.
Thanks for your help James!
Cheers,
-Kristoffer
On Tue, Sep 30, 2014 at 1:20 AM, James Taylor <ja...@apache.org>
wrote:
> Hi Kristoffer,
> Did something change on your cluster prior to the breaking of your
> SELECT queries? Did it ever work?
> Thanks,
> James
>
> On Mon, Sep 29, 2014 at 4:06 PM, Kristoffer Sjögren <st...@gmail.com>
> wrote:
> > Thank for quick reply. Yes, we will try to upgrade as soon as we can get
> the
> > servers up on 2.2.3. If we can't then we go straight to upgrade.
> >
> > The region servers are running phoenix-2.2.3-incubating.jar found in [1].
> > Are the clients suppose to use the exact same jar? The jar files have
> same
> > name but are indeed slightly different in size between client (1542522
> > bytes) and server (1540202 bytes). Bummer that the jar MANIFEST does not
> > provide enough build information.
> >
> > I'm still puzzled why sqlline.sh [1] have the same problem though? Maybe
> > something with phoenix-2.2.3-incubating-client.jar?
> >
> > [1]
> >
> http://apache.mirrors.spacedump.net/incubator/phoenix/phoenix-2.2.3-incubating/bin/phoenix-2.2.3-incubating.tar.gz
> >
> >
> > On Tue, Sep 30, 2014 at 12:34 AM, James Taylor <ja...@apache.org>
> > wrote:
> >>
> >> Hi Kristoffer,
> >> Did something change on your cluster prior to the breaking of your
> >> SELECT queries? An upgrade of something?
> >>
> >> From the look of the exceptions, it seems like the Phoenix client jar
> >> and server jar are out-of-sync, because the client is sending
> >> information in a format that's no longer understood by the server.
> >>
> >> I do encourage you to upgrade, though, once you get your production
> >> servers functioning again.
> >>
> >> Thanks,
> >> James
> >>
> >> On Mon, Sep 29, 2014 at 3:17 PM, Kristoffer Sjögren <st...@gmail.com>
> >> wrote:
> >> > Hi
> >> >
> >> > Our SELECT queries went broken just now and we're seeing the following
> >> > exception [1] using Apache Phoenix 2.2.3-incubating with HBase 0.94.6
> >> > (0.94.6-cdh4.4.0).
> >> >
> >> > After a some testing I realized that sqlline.sh from the
> >> > 2.2.3-incubating
> >> > tarball is also broken all of a sudden (with same error)?
> >> >
> >> > Seems like SELECT ... WHERE queries are problematic whereas basic
> >> > queries
> >> > like "SELECT * FROM COUNTERS LIMIT 1;" works fine. Simple INSERT
> >> > statements
> >> > also seems to be working.
> >> >
> >> > Is upgrading to phoenix-3.0.0-incubating an option? Or is there a way
> to
> >> > debug this further? I tried to enable DEBUG in log4j.properties using
> >> > the
> >> > 'console' in sqlline but without success.
> >> >
> >> > Our production servers are pretty much down at the moment...
> >> >
> >> > Cheers,
> >> > -Kristoffer
> >> >
> >> > [1] http://pastebin.com/H2a6xpBX
> >> >
> >
> >
>
Re: Error in readFields
Posted by James Taylor <ja...@apache.org>.
Hi Kristoffer,
Did something change on your cluster prior to the breaking of your
SELECT queries? Did it ever work?
Thanks,
James
On Mon, Sep 29, 2014 at 4:06 PM, Kristoffer Sjögren <st...@gmail.com> wrote:
> Thank for quick reply. Yes, we will try to upgrade as soon as we can get the
> servers up on 2.2.3. If we can't then we go straight to upgrade.
>
> The region servers are running phoenix-2.2.3-incubating.jar found in [1].
> Are the clients suppose to use the exact same jar? The jar files have same
> name but are indeed slightly different in size between client (1542522
> bytes) and server (1540202 bytes). Bummer that the jar MANIFEST does not
> provide enough build information.
>
> I'm still puzzled why sqlline.sh [1] have the same problem though? Maybe
> something with phoenix-2.2.3-incubating-client.jar?
>
> [1]
> http://apache.mirrors.spacedump.net/incubator/phoenix/phoenix-2.2.3-incubating/bin/phoenix-2.2.3-incubating.tar.gz
>
>
> On Tue, Sep 30, 2014 at 12:34 AM, James Taylor <ja...@apache.org>
> wrote:
>>
>> Hi Kristoffer,
>> Did something change on your cluster prior to the breaking of your
>> SELECT queries? An upgrade of something?
>>
>> From the look of the exceptions, it seems like the Phoenix client jar
>> and server jar are out-of-sync, because the client is sending
>> information in a format that's no longer understood by the server.
>>
>> I do encourage you to upgrade, though, once you get your production
>> servers functioning again.
>>
>> Thanks,
>> James
>>
>> On Mon, Sep 29, 2014 at 3:17 PM, Kristoffer Sjögren <st...@gmail.com>
>> wrote:
>> > Hi
>> >
>> > Our SELECT queries went broken just now and we're seeing the following
>> > exception [1] using Apache Phoenix 2.2.3-incubating with HBase 0.94.6
>> > (0.94.6-cdh4.4.0).
>> >
>> > After a some testing I realized that sqlline.sh from the
>> > 2.2.3-incubating
>> > tarball is also broken all of a sudden (with same error)?
>> >
>> > Seems like SELECT ... WHERE queries are problematic whereas basic
>> > queries
>> > like "SELECT * FROM COUNTERS LIMIT 1;" works fine. Simple INSERT
>> > statements
>> > also seems to be working.
>> >
>> > Is upgrading to phoenix-3.0.0-incubating an option? Or is there a way to
>> > debug this further? I tried to enable DEBUG in log4j.properties using
>> > the
>> > 'console' in sqlline but without success.
>> >
>> > Our production servers are pretty much down at the moment...
>> >
>> > Cheers,
>> > -Kristoffer
>> >
>> > [1] http://pastebin.com/H2a6xpBX
>> >
>
>
Re: Error in readFields
Posted by Kristoffer Sjögren <st...@gmail.com>.
Thank for quick reply. Yes, we will try to upgrade as soon as we can get
the servers up on 2.2.3. If we can't then we go straight to upgrade.
The region servers are running phoenix-2.2.3-incubating.jar found in [1].
Are the clients suppose to use the exact same jar? The jar files have same
name but are indeed slightly different in size between client (1542522
bytes) and server (1540202 bytes). Bummer that the jar MANIFEST does not
provide enough build information.
I'm still puzzled why sqlline.sh [1] have the same problem though? Maybe
something with phoenix-2.2.3-incubating-client.jar?
[1]
http://apache.mirrors.spacedump.net/incubator/phoenix/phoenix-2.2.3-incubating/bin/phoenix-2.2.3-incubating.tar.gz
On Tue, Sep 30, 2014 at 12:34 AM, James Taylor <ja...@apache.org>
wrote:
> Hi Kristoffer,
> Did something change on your cluster prior to the breaking of your
> SELECT queries? An upgrade of something?
>
> From the look of the exceptions, it seems like the Phoenix client jar
> and server jar are out-of-sync, because the client is sending
> information in a format that's no longer understood by the server.
>
> I do encourage you to upgrade, though, once you get your production
> servers functioning again.
>
> Thanks,
> James
>
> On Mon, Sep 29, 2014 at 3:17 PM, Kristoffer Sjögren <st...@gmail.com>
> wrote:
> > Hi
> >
> > Our SELECT queries went broken just now and we're seeing the following
> > exception [1] using Apache Phoenix 2.2.3-incubating with HBase 0.94.6
> > (0.94.6-cdh4.4.0).
> >
> > After a some testing I realized that sqlline.sh from the 2.2.3-incubating
> > tarball is also broken all of a sudden (with same error)?
> >
> > Seems like SELECT ... WHERE queries are problematic whereas basic queries
> > like "SELECT * FROM COUNTERS LIMIT 1;" works fine. Simple INSERT
> statements
> > also seems to be working.
> >
> > Is upgrading to phoenix-3.0.0-incubating an option? Or is there a way to
> > debug this further? I tried to enable DEBUG in log4j.properties using the
> > 'console' in sqlline but without success.
> >
> > Our production servers are pretty much down at the moment...
> >
> > Cheers,
> > -Kristoffer
> >
> > [1] http://pastebin.com/H2a6xpBX
> >
>
Re: Error in readFields
Posted by James Taylor <ja...@apache.org>.
Hi Kristoffer,
Did something change on your cluster prior to the breaking of your
SELECT queries? An upgrade of something?
>From the look of the exceptions, it seems like the Phoenix client jar
and server jar are out-of-sync, because the client is sending
information in a format that's no longer understood by the server.
I do encourage you to upgrade, though, once you get your production
servers functioning again.
Thanks,
James
On Mon, Sep 29, 2014 at 3:17 PM, Kristoffer Sjögren <st...@gmail.com> wrote:
> Hi
>
> Our SELECT queries went broken just now and we're seeing the following
> exception [1] using Apache Phoenix 2.2.3-incubating with HBase 0.94.6
> (0.94.6-cdh4.4.0).
>
> After a some testing I realized that sqlline.sh from the 2.2.3-incubating
> tarball is also broken all of a sudden (with same error)?
>
> Seems like SELECT ... WHERE queries are problematic whereas basic queries
> like "SELECT * FROM COUNTERS LIMIT 1;" works fine. Simple INSERT statements
> also seems to be working.
>
> Is upgrading to phoenix-3.0.0-incubating an option? Or is there a way to
> debug this further? I tried to enable DEBUG in log4j.properties using the
> 'console' in sqlline but without success.
>
> Our production servers are pretty much down at the moment...
>
> Cheers,
> -Kristoffer
>
> [1] http://pastebin.com/H2a6xpBX
>