You are viewing a plain text version of this content. The canonical link for it is here.
Posted to server-dev@james.apache.org by Ioan Eugen Stan <st...@gmail.com> on 2011/08/20 20:50:22 UTC
hbase mailbox alpha 2
Hello,
Things are looking better and better. I have refactored a lot of code, mainly:
- reduced number of columns to 3 in MESSAGES table. I have plans to
get them down to 2, but it's ok for now.
- created Convertor classes for Flags and Properties, as Eric once
suggested. this also lead to the following benefit:
- HBaseUtils is much more clean. Another sweep is around the corner.
As it stands now, the test results are:
Tests run: 291, Failures: 0, Errors: 80, Skipped: 0
I get a lot of java.nio.BufferOverflowException and I got some other
stuff to fix in the flags area, but I think things are going well. One
thing that kept me was the fact that I was returning mssage UID's in
the wrong order. The server was expecting them in ascending order and
because I store them using Long.MAX_VALUE - uid i was getting them in
descending order. Now it's fixed. I'm using Collections.reverse() to
switch them.
The test runs for about 50 min. I don't know why it takes so much but
I think it has to do something with the large amount of debugging
output that's coming out of HBase.
Stay tuned for more information and check out he latest version in the repo:
http://code.google.com/a/apache-extras.org/p/mailbox-hdfs/
and:
http://code.google.com/a/apache-extras.org/p/hbase-mailbox-integration-test/
p.s. a quick code review will be appreciated.
Thanks,
--
Ioan Eugen Stan
http://ieugen.blogspot.com/
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: hbase mailbox alpha 2
Posted by Robert Burrell Donkin <ro...@gmail.com>.
On Sun, Aug 21, 2011 at 5:55 PM, Eric Charles <er...@apache.org> wrote:
> On 21/08/11 17:23, Ioan Eugen Stan wrote:
>
>> Latest commit: tests pass: 279, fail 12. Not much until it's all green.
>>
>
> Impressed ;)
:-)
Robert
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: hbase mailbox alpha 2
Posted by Eric Charles <er...@apache.org>.
On 21/08/11 17:23, Ioan Eugen Stan wrote:
> Latest commit: tests pass: 279, fail 12. Not much until it's all green.
>
Impressed ;)
- Eric
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: hbase mailbox alpha 2
Posted by Ioan Eugen Stan <st...@gmail.com>.
2011/8/21 Eric Charles <er...@apache.org>:
> On 20/08/11 20:50, Ioan Eugen Stan wrote:
>>
>> Hello,
>>
>> Things are looking better and better. I have refactored a lot of code,
>> mainly:
>> - reduced number of columns to 3 in MESSAGES table. I have plans to
>> get them down to 2, but it's ok for now.
>> - created Convertor classes for Flags and Properties, as Eric once
>> suggested. this also lead to the following benefit:
>> - HBaseUtils is much more clean. Another sweep is around the corner.
>>
>> As it stands now, the test results are:
>>
>> Tests run: 291, Failures: 0, Errors: 80, Skipped: 0
>>
Latest commit: tests pass: 279, fail 12. Not much until it's all green.
> Well done :)
>
>> I get a lot of java.nio.BufferOverflowException and I got some other
>> stuff to fix in the flags area, but I think things are going well. One
>> thing that kept me was the fact that I was returning mssage UID's in
>> the wrong order. The server was expecting them in ascending order and
>> because I store them using Long.MAX_VALUE - uid i was getting them in
>> descending order. Now it's fixed. I'm using Collections.reverse() to
>> switch them.
>>
>
> Why about storing them directly in correct order?
Well I decided to store them like this because usually the user
accesses the latest 100 or so messages from the mailbox, so this was
an optimization. I will check to see if this still stands.
>> The test runs for about 50 min. I don't know why it takes so much but
>> I think it has to do something with the large amount of debugging
>> output that's coming out of HBase.
>>
>
> Cluster start/stop and table enable on each test takes time.
> You can mitigate it via a junit suite that start hbase cluster once (don't
> forget to fix the maven-surefire-plugin in pom.xml to only include
> **/*SuiteTest.java.
> You will have to delete/create the table before each test (or delete
> everything). See also open thread [2] on hbase mailing list to further
> optimize the process.
>
> [1]
> http://svn.apache.org/repos/asf/james/server/trunk/hbase/src/test/java/org/apache/james/JamesServerHBaseSuiteTest.java
> [2] http://markmail.org/message/o77ldtza4ojbyoxv
>
>> Stay tuned for more information and check out he latest version in the
>> repo:
>>
>> http://code.google.com/a/apache-extras.org/p/mailbox-hdfs/
>> and:
>>
>> http://code.google.com/a/apache-extras.org/p/hbase-mailbox-integration-test/
>>
>> p.s. a quick code review will be appreciated.
>>
Yeah, I will look at that after all tests pass.
> I went through the last version, but don't have much time today...
> Quick feedback:
>
> - deleteAllMemberships and deleteAllMailboxes: you delete/recreate the
> tables instead of removing the content. why not. you could centralize the
> table creation code somewhere in a lazy singleton.
Yeah, I thought it will be faster if there are many messages. I forgot
they are used just in tests. I will modify them soon when I'm not
lazy.
> - HBaseUtils: I would separate the constants definitions in to a new
> interface and leave the methods in the class. Also, you can use inner
> interface to make it still more readable.
Done.
> - the mm file should not be under src/main/resources, but under
> src/site/resource for example.
Done, updated to reflect the current Schema.
--
Ioan Eugen Stan
http://ieugen.blogspot.com/
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: hbase mailbox alpha 2
Posted by Eric Charles <er...@apache.org>.
On 20/08/11 20:50, Ioan Eugen Stan wrote:
> Hello,
>
> Things are looking better and better. I have refactored a lot of code, mainly:
> - reduced number of columns to 3 in MESSAGES table. I have plans to
> get them down to 2, but it's ok for now.
> - created Convertor classes for Flags and Properties, as Eric once
> suggested. this also lead to the following benefit:
> - HBaseUtils is much more clean. Another sweep is around the corner.
>
> As it stands now, the test results are:
>
> Tests run: 291, Failures: 0, Errors: 80, Skipped: 0
>
Well done :)
> I get a lot of java.nio.BufferOverflowException and I got some other
> stuff to fix in the flags area, but I think things are going well. One
> thing that kept me was the fact that I was returning mssage UID's in
> the wrong order. The server was expecting them in ascending order and
> because I store them using Long.MAX_VALUE - uid i was getting them in
> descending order. Now it's fixed. I'm using Collections.reverse() to
> switch them.
>
Why about storing them directly in correct order?
> The test runs for about 50 min. I don't know why it takes so much but
> I think it has to do something with the large amount of debugging
> output that's coming out of HBase.
>
Cluster start/stop and table enable on each test takes time.
You can mitigate it via a junit suite that start hbase cluster once
(don't forget to fix the maven-surefire-plugin in pom.xml to only
include **/*SuiteTest.java.
You will have to delete/create the table before each test (or delete
everything). See also open thread [2] on hbase mailing list to further
optimize the process.
[1]
http://svn.apache.org/repos/asf/james/server/trunk/hbase/src/test/java/org/apache/james/JamesServerHBaseSuiteTest.java
[2] http://markmail.org/message/o77ldtza4ojbyoxv
> Stay tuned for more information and check out he latest version in the repo:
>
> http://code.google.com/a/apache-extras.org/p/mailbox-hdfs/
> and:
> http://code.google.com/a/apache-extras.org/p/hbase-mailbox-integration-test/
>
> p.s. a quick code review will be appreciated.
>
I went through the last version, but don't have much time today...
Quick feedback:
- deleteAllMemberships and deleteAllMailboxes: you delete/recreate the
tables instead of removing the content. why not. you could centralize
the table creation code somewhere in a lazy singleton.
- HBaseUtils: I would separate the constants definitions in to a new
interface and leave the methods in the class. Also, you can use inner
interface to make it still more readable.
- the mm file should not be under src/main/resources, but under
src/site/resource for example.
> Thanks,
--
Eric
http://about.echarles.net
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org