You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Frank Luo <jl...@merkleinc.com> on 2016/07/13 15:20:28 UTC
is possible to create multiple TableSplit per region?
We have mapper only jobs operating on a result of a Scan. Because of heavy work downstream, the mapper runs fairly slowly. So I am wondering if there is a way to create multiple TableSplit on one region hence multiple mappers can be created to work on different piece of date on the region.
I am aware of MultithreadedTableMapper class, which could be my solution, but I hesitate to use it as my code is not thread safe.
So any suggestions, or code to share?
Download the latest installment of our annual Marketing Imperatives, “Winning with People-Based Marketing”<http://www2.merkleinc.com/l/47252/2016-04-26/3lbfd1>
This email and any attachments transmitted with it are intended for use by the intended recipient(s) only. If you have received this email in error, please notify the sender immediately and then delete it. If you are not the intended recipient, you must not keep, use, disclose, copy or distribute this email without the author’s prior permission. We take precautions to minimize the risk of transmitting software viruses, but we advise you to perform your own virus checks on any attachment to this message. We cannot accept liability for any loss or damage caused by software viruses. The information contained in this communication may be confidential and may be subject to the attorney-client privilege.
Re: is possible to create multiple TableSplit per region?
Posted by Billy Watson <wi...@gmail.com>.
I agree. I'm not an expert though. I do more pig jobs than anything.
Any one else on the thread have more experience creating MR jobs on HBase
data?
On Wednesday, July 13, 2016, Frank Luo <jl...@merkleinc.com> wrote:
> It will work, but it is pretty awkward way to create more mappers.
>
>
>
> *From:* Billy Watson [mailto:williamrwatson@gmail.com
> <javascript:_e(%7B%7D,'cvml','williamrwatson@gmail.com');>]
> *Sent:* Wednesday, July 13, 2016 3:57 PM
> *To:* Frank Luo <jluo@merkleinc.com
> <javascript:_e(%7B%7D,'cvml','jluo@merkleinc.com');>>
> *Cc:* user@hbase.apache.org
> <javascript:_e(%7B%7D,'cvml','user@hbase.apache.org');>
> *Subject:* Re: Re:is possible to create multiple TableSplit per region?
>
>
>
> It seems like it might be faster then to consider a map job followed by
> another map job. Or, depending on the web service calls, maybe a combine
> step?
>
>
> William Watson
> Lead Software Engineer
>
>
>
> On Wed, Jul 13, 2016 at 4:40 PM, Frank Luo <jluo@merkleinc.com
> <javascript:_e(%7B%7D,'cvml','jluo@merkleinc.com');>> wrote:
>
> It makes a number of web-service calls.
>
>
>
> *From:* Billy Watson [mailto:williamrwatson@gmail.com
> <javascript:_e(%7B%7D,'cvml','williamrwatson@gmail.com');>]
> *Sent:* Wednesday, July 13, 2016 3:27 PM
> *To:* user@hbase.apache.org
> <javascript:_e(%7B%7D,'cvml','user@hbase.apache.org');>
> *Cc:* Frank Luo <jluo@merkleinc.com
> <javascript:_e(%7B%7D,'cvml','jluo@merkleinc.com');>>
> *Subject:* Re: Re:is possible to create multiple TableSplit per region?
>
>
>
> What do you mean by "heavy work downstream"?
>
>
>
> I think the mailing list might need a *few* more details to help out
> better.
>
>
> William Watson
>
>
>
> On Wed, Jul 13, 2016 at 12:32 PM, Frank Luo <jluo@merkleinc.com
> <javascript:_e(%7B%7D,'cvml','jluo@merkleinc.com');>> wrote:
>
> Thanks for the prompt reply, Lu.
>
> It is true that having a smaller region file size can solve the problem.
> But it also have side effects. For example, total number of regions can be
> easily doubled/tripled, and I am already facing a challenge of having too
> many regions per server. So I cannot go to that route.
>
> From: 陆巍 [mailto:luwei114@163.com
> <javascript:_e(%7B%7D,'cvml','luwei114@163.com');>]
> Sent: Wednesday, July 13, 2016 11:24 AM
> To: user@hbase.apache.org
> <javascript:_e(%7B%7D,'cvml','user@hbase.apache.org');>; Frank Luo <
> jluo@merkleinc.com <javascript:_e(%7B%7D,'cvml','jluo@merkleinc.com');>>
> Subject: Re:is possible to create multiple TableSplit per region?
>
> here is an archived mail:
> http://mail-archives.apache.org/mod_mbox/hbase-user/201303.mbox/%3CBLU0-SMTP19115A8967869D6CF0D49EF8FD70@phx.gbl%3E
>
> At 2016-07-13 23:20:28, "Frank Luo" <jluo@merkleinc.com
> <javascript:_e(%7B%7D,'cvml','jluo@merkleinc.com');><mailto:
> jluo@merkleinc.com <javascript:_e(%7B%7D,'cvml','jluo@merkleinc.com');>>>
> wrote:
>
> >We have mapper only jobs operating on a result of a Scan. Because of
> heavy work downstream, the mapper runs fairly slowly. So I am wondering if
> there is a way to create multiple TableSplit on one region hence multiple
> mappers can be created to work on different piece of date on the region.
>
> >
>
> >I am aware of MultithreadedTableMapper class, which could be my solution,
> but I hesitate to use it as my code is not thread safe.
>
> >
>
> >So any suggestions, or code to share?
>
> >
>
> >
>
> >
>
> >Download the latest installment of our annual Marketing Imperatives,
> “Winning with People-Based Marketing”<
> http://www2.merkleinc.com/l/47252/2016-04-26/3lbfd1>
>
> >
>
> >This email and any attachments transmitted with it are intended for use
> by the intended recipient(s) only. If you have received this email in
> error, please notify the sender immediately and then delete it. If you are
> not the intended recipient, you must not keep, use, disclose, copy or
> distribute this email without the author’s prior permission. We take
> precautions to minimize the risk of transmitting software viruses, but we
> advise you to perform your own virus checks on any attachment to this
> message. We cannot accept liability for any loss or damage caused by
> software viruses. The information contained in this communication may be
> confidential and may be subject to the attorney-client privilege.
>
>
>
>
> Download the latest installment of our annual Marketing Imperatives,
> “Winning with People-Based Marketing”<
> http://www2.merkleinc.com/l/47252/2016-04-26/3lbfd1>
>
> This email and any attachments transmitted with it are intended for use by
> the intended recipient(s) only. If you have received this email in error,
> please notify the sender immediately and then delete it. If you are not the
> intended recipient, you must not keep, use, disclose, copy or distribute
> this email without the author’s prior permission. We take precautions to
> minimize the risk of transmitting software viruses, but we advise you to
> perform your own virus checks on any attachment to this message. We cannot
> accept liability for any loss or damage caused by software viruses. The
> information contained in this communication may be confidential and may be
> subject to the attorney-client privilege.
>
>
>
> *Download the latest installment of our annual Marketing Imperatives,
> “Winning with People-Based Marketing”*
> <http://www2.merkleinc.com/l/47252/2016-04-26/3lbfd1>
>
> This email and any attachments transmitted with it are intended for use by
> the intended recipient(s) only. If you have received this email in error,
> please notify the sender immediately and then delete it. If you are not the
> intended recipient, you must not keep, use, disclose, copy or distribute
> this email without the author’s prior permission. We take precautions to
> minimize the risk of transmitting software viruses, but we advise you to
> perform your own virus checks on any attachment to this message. We cannot
> accept liability for any loss or damage caused by software viruses. The
> information contained in this communication may be confidential and may be
> subject to the attorney-client privilege.
>
>
>
> *Download the latest installment of our annual Marketing Imperatives,
> “Winning with People-Based Marketing”*
> <http://www2.merkleinc.com/l/47252/2016-04-26/3lbfd1>
>
> This email and any attachments transmitted with it are intended for use by
> the intended recipient(s) only. If you have received this email in error,
> please notify the sender immediately and then delete it. If you are not the
> intended recipient, you must not keep, use, disclose, copy or distribute
> this email without the author’s prior permission. We take precautions to
> minimize the risk of transmitting software viruses, but we advise you to
> perform your own virus checks on any attachment to this message. We cannot
> accept liability for any loss or damage caused by software viruses. The
> information contained in this communication may be confidential and may be
> subject to the attorney-client privilege.
>
--
William Watson
Lead Software Engineer
RE: Re:is possible to create multiple TableSplit per region?
Posted by Frank Luo <jl...@merkleinc.com>.
It will work, but it is pretty awkward way to create more mappers.
From: Billy Watson [mailto:williamrwatson@gmail.com]
Sent: Wednesday, July 13, 2016 3:57 PM
To: Frank Luo <jl...@merkleinc.com>
Cc: user@hbase.apache.org
Subject: Re: Re:is possible to create multiple TableSplit per region?
It seems like it might be faster then to consider a map job followed by another map job. Or, depending on the web service calls, maybe a combine step?
William Watson
Lead Software Engineer
On Wed, Jul 13, 2016 at 4:40 PM, Frank Luo <jl...@merkleinc.com>> wrote:
It makes a number of web-service calls.
From: Billy Watson [mailto:williamrwatson@gmail.com<ma...@gmail.com>]
Sent: Wednesday, July 13, 2016 3:27 PM
To: user@hbase.apache.org<ma...@hbase.apache.org>
Cc: Frank Luo <jl...@merkleinc.com>>
Subject: Re: Re:is possible to create multiple TableSplit per region?
What do you mean by "heavy work downstream"?
I think the mailing list might need a *few* more details to help out better.
William Watson
On Wed, Jul 13, 2016 at 12:32 PM, Frank Luo <jl...@merkleinc.com>> wrote:
Thanks for the prompt reply, Lu.
It is true that having a smaller region file size can solve the problem. But it also have side effects. For example, total number of regions can be easily doubled/tripled, and I am already facing a challenge of having too many regions per server. So I cannot go to that route.
From: 陆巍 [mailto:luwei114@163.com<ma...@163.com>]
Sent: Wednesday, July 13, 2016 11:24 AM
To: user@hbase.apache.org<ma...@hbase.apache.org>; Frank Luo <jl...@merkleinc.com>>
Subject: Re:is possible to create multiple TableSplit per region?
here is an archived mail: http://mail-archives.apache.org/mod_mbox/hbase-user/201303.mbox/%3CBLU0-SMTP19115A8967869D6CF0D49EF8FD70@phx.gbl%3E
At 2016-07-13 23:20:28, "Frank Luo" <jl...@merkleinc.com>>> wrote:
>We have mapper only jobs operating on a result of a Scan. Because of heavy work downstream, the mapper runs fairly slowly. So I am wondering if there is a way to create multiple TableSplit on one region hence multiple mappers can be created to work on different piece of date on the region.
>
>I am aware of MultithreadedTableMapper class, which could be my solution, but I hesitate to use it as my code is not thread safe.
>
>So any suggestions, or code to share?
>
>
>
>Download the latest installment of our annual Marketing Imperatives, “Winning with People-Based Marketing”<http://www2.merkleinc.com/l/47252/2016-04-26/3lbfd1>
>
>This email and any attachments transmitted with it are intended for use by the intended recipient(s) only. If you have received this email in error, please notify the sender immediately and then delete it. If you are not the intended recipient, you must not keep, use, disclose, copy or distribute this email without the author’s prior permission. We take precautions to minimize the risk of transmitting software viruses, but we advise you to perform your own virus checks on any attachment to this message. We cannot accept liability for any loss or damage caused by software viruses. The information contained in this communication may be confidential and may be subject to the attorney-client privilege.
Download the latest installment of our annual Marketing Imperatives, “Winning with People-Based Marketing”<http://www2.merkleinc.com/l/47252/2016-04-26/3lbfd1>
This email and any attachments transmitted with it are intended for use by the intended recipient(s) only. If you have received this email in error, please notify the sender immediately and then delete it. If you are not the intended recipient, you must not keep, use, disclose, copy or distribute this email without the author’s prior permission. We take precautions to minimize the risk of transmitting software viruses, but we advise you to perform your own virus checks on any attachment to this message. We cannot accept liability for any loss or damage caused by software viruses. The information contained in this communication may be confidential and may be subject to the attorney-client privilege.
Download the latest installment of our annual Marketing Imperatives, “Winning with People-Based Marketing”<http://www2.merkleinc.com/l/47252/2016-04-26/3lbfd1>
This email and any attachments transmitted with it are intended for use by the intended recipient(s) only. If you have received this email in error, please notify the sender immediately and then delete it. If you are not the intended recipient, you must not keep, use, disclose, copy or distribute this email without the author’s prior permission. We take precautions to minimize the risk of transmitting software viruses, but we advise you to perform your own virus checks on any attachment to this message. We cannot accept liability for any loss or damage caused by software viruses. The information contained in this communication may be confidential and may be subject to the attorney-client privilege.
Download the latest installment of our annual Marketing Imperatives, “Winning with People-Based Marketing”<http://www2.merkleinc.com/l/47252/2016-04-26/3lbfd1>
This email and any attachments transmitted with it are intended for use by the intended recipient(s) only. If you have received this email in error, please notify the sender immediately and then delete it. If you are not the intended recipient, you must not keep, use, disclose, copy or distribute this email without the author’s prior permission. We take precautions to minimize the risk of transmitting software viruses, but we advise you to perform your own virus checks on any attachment to this message. We cannot accept liability for any loss or damage caused by software viruses. The information contained in this communication may be confidential and may be subject to the attorney-client privilege.
Re: Re:is possible to create multiple TableSplit per region?
Posted by Billy Watson <wi...@gmail.com>.
It seems like it might be faster then to consider a map job followed by
another map job. Or, depending on the web service calls, maybe a combine
step?
William Watson
Lead Software Engineer
On Wed, Jul 13, 2016 at 4:40 PM, Frank Luo <jl...@merkleinc.com> wrote:
> It makes a number of web-service calls.
>
>
>
> *From:* Billy Watson [mailto:williamrwatson@gmail.com]
> *Sent:* Wednesday, July 13, 2016 3:27 PM
> *To:* user@hbase.apache.org
> *Cc:* Frank Luo <jl...@merkleinc.com>
> *Subject:* Re: Re:is possible to create multiple TableSplit per region?
>
>
>
> What do you mean by "heavy work downstream"?
>
>
>
> I think the mailing list might need a *few* more details to help out
> better.
>
>
> William Watson
>
>
>
> On Wed, Jul 13, 2016 at 12:32 PM, Frank Luo <jl...@merkleinc.com> wrote:
>
> Thanks for the prompt reply, Lu.
>
> It is true that having a smaller region file size can solve the problem.
> But it also have side effects. For example, total number of regions can be
> easily doubled/tripled, and I am already facing a challenge of having too
> many regions per server. So I cannot go to that route.
>
> From: 陆巍 [mailto:luwei114@163.com]
> Sent: Wednesday, July 13, 2016 11:24 AM
> To: user@hbase.apache.org; Frank Luo <jl...@merkleinc.com>
> Subject: Re:is possible to create multiple TableSplit per region?
>
> here is an archived mail:
> http://mail-archives.apache.org/mod_mbox/hbase-user/201303.mbox/%3CBLU0-SMTP19115A8967869D6CF0D49EF8FD70@phx.gbl%3E
>
>
> At 2016-07-13 23:20:28, "Frank Luo" <jluo@merkleinc.com<mailto:
> jluo@merkleinc.com>> wrote:
>
> >We have mapper only jobs operating on a result of a Scan. Because of
> heavy work downstream, the mapper runs fairly slowly. So I am wondering if
> there is a way to create multiple TableSplit on one region hence multiple
> mappers can be created to work on different piece of date on the region.
>
> >
>
> >I am aware of MultithreadedTableMapper class, which could be my solution,
> but I hesitate to use it as my code is not thread safe.
>
> >
>
> >So any suggestions, or code to share?
>
> >
>
> >
>
> >
>
> >Download the latest installment of our annual Marketing Imperatives,
> “Winning with People-Based Marketing”<
> http://www2.merkleinc.com/l/47252/2016-04-26/3lbfd1>
>
> >
>
> >This email and any attachments transmitted with it are intended for use
> by the intended recipient(s) only. If you have received this email in
> error, please notify the sender immediately and then delete it. If you are
> not the intended recipient, you must not keep, use, disclose, copy or
> distribute this email without the author’s prior permission. We take
> precautions to minimize the risk of transmitting software viruses, but we
> advise you to perform your own virus checks on any attachment to this
> message. We cannot accept liability for any loss or damage caused by
> software viruses. The information contained in this communication may be
> confidential and may be subject to the attorney-client privilege.
>
>
>
>
> Download the latest installment of our annual Marketing Imperatives,
> “Winning with People-Based Marketing”<
> http://www2.merkleinc.com/l/47252/2016-04-26/3lbfd1>
>
> This email and any attachments transmitted with it are intended for use by
> the intended recipient(s) only. If you have received this email in error,
> please notify the sender immediately and then delete it. If you are not the
> intended recipient, you must not keep, use, disclose, copy or distribute
> this email without the author’s prior permission. We take precautions to
> minimize the risk of transmitting software viruses, but we advise you to
> perform your own virus checks on any attachment to this message. We cannot
> accept liability for any loss or damage caused by software viruses. The
> information contained in this communication may be confidential and may be
> subject to the attorney-client privilege.
>
>
>
> *Download the latest installment of our annual Marketing Imperatives,
> “Winning with People-Based Marketing”*
> <http://www2.merkleinc.com/l/47252/2016-04-26/3lbfd1>
>
> This email and any attachments transmitted with it are intended for use by
> the intended recipient(s) only. If you have received this email in error,
> please notify the sender immediately and then delete it. If you are not the
> intended recipient, you must not keep, use, disclose, copy or distribute
> this email without the author’s prior permission. We take precautions to
> minimize the risk of transmitting software viruses, but we advise you to
> perform your own virus checks on any attachment to this message. We cannot
> accept liability for any loss or damage caused by software viruses. The
> information contained in this communication may be confidential and may be
> subject to the attorney-client privilege.
>
RE: Re:is possible to create multiple TableSplit per region?
Posted by Frank Luo <jl...@merkleinc.com>.
It makes a number of web-service calls.
From: Billy Watson [mailto:williamrwatson@gmail.com]
Sent: Wednesday, July 13, 2016 3:27 PM
To: user@hbase.apache.org
Cc: Frank Luo <jl...@merkleinc.com>
Subject: Re: Re:is possible to create multiple TableSplit per region?
What do you mean by "heavy work downstream"?
I think the mailing list might need a *few* more details to help out better.
William Watson
On Wed, Jul 13, 2016 at 12:32 PM, Frank Luo <jl...@merkleinc.com>> wrote:
Thanks for the prompt reply, Lu.
It is true that having a smaller region file size can solve the problem. But it also have side effects. For example, total number of regions can be easily doubled/tripled, and I am already facing a challenge of having too many regions per server. So I cannot go to that route.
From: 陆巍 [mailto:luwei114@163.com<ma...@163.com>]
Sent: Wednesday, July 13, 2016 11:24 AM
To: user@hbase.apache.org<ma...@hbase.apache.org>; Frank Luo <jl...@merkleinc.com>>
Subject: Re:is possible to create multiple TableSplit per region?
here is an archived mail: http://mail-archives.apache.org/mod_mbox/hbase-user/201303.mbox/%3CBLU0-SMTP19115A8967869D6CF0D49EF8FD70@phx.gbl%3E
At 2016-07-13 23:20:28, "Frank Luo" <jl...@merkleinc.com>>> wrote:
>We have mapper only jobs operating on a result of a Scan. Because of heavy work downstream, the mapper runs fairly slowly. So I am wondering if there is a way to create multiple TableSplit on one region hence multiple mappers can be created to work on different piece of date on the region.
>
>I am aware of MultithreadedTableMapper class, which could be my solution, but I hesitate to use it as my code is not thread safe.
>
>So any suggestions, or code to share?
>
>
>
>Download the latest installment of our annual Marketing Imperatives, “Winning with People-Based Marketing”<http://www2.merkleinc.com/l/47252/2016-04-26/3lbfd1>
>
>This email and any attachments transmitted with it are intended for use by the intended recipient(s) only. If you have received this email in error, please notify the sender immediately and then delete it. If you are not the intended recipient, you must not keep, use, disclose, copy or distribute this email without the author’s prior permission. We take precautions to minimize the risk of transmitting software viruses, but we advise you to perform your own virus checks on any attachment to this message. We cannot accept liability for any loss or damage caused by software viruses. The information contained in this communication may be confidential and may be subject to the attorney-client privilege.
Download the latest installment of our annual Marketing Imperatives, “Winning with People-Based Marketing”<http://www2.merkleinc.com/l/47252/2016-04-26/3lbfd1>
This email and any attachments transmitted with it are intended for use by the intended recipient(s) only. If you have received this email in error, please notify the sender immediately and then delete it. If you are not the intended recipient, you must not keep, use, disclose, copy or distribute this email without the author’s prior permission. We take precautions to minimize the risk of transmitting software viruses, but we advise you to perform your own virus checks on any attachment to this message. We cannot accept liability for any loss or damage caused by software viruses. The information contained in this communication may be confidential and may be subject to the attorney-client privilege.
Download the latest installment of our annual Marketing Imperatives, “Winning with People-Based Marketing”<http://www2.merkleinc.com/l/47252/2016-04-26/3lbfd1>
This email and any attachments transmitted with it are intended for use by the intended recipient(s) only. If you have received this email in error, please notify the sender immediately and then delete it. If you are not the intended recipient, you must not keep, use, disclose, copy or distribute this email without the author’s prior permission. We take precautions to minimize the risk of transmitting software viruses, but we advise you to perform your own virus checks on any attachment to this message. We cannot accept liability for any loss or damage caused by software viruses. The information contained in this communication may be confidential and may be subject to the attorney-client privilege.
Re: Re:is possible to create multiple TableSplit per region?
Posted by Billy Watson <wi...@gmail.com>.
What do you mean by "heavy work downstream"?
I think the mailing list might need a *few* more details to help out better.
William Watson
On Wed, Jul 13, 2016 at 12:32 PM, Frank Luo <jl...@merkleinc.com> wrote:
> Thanks for the prompt reply, Lu.
>
> It is true that having a smaller region file size can solve the problem.
> But it also have side effects. For example, total number of regions can be
> easily doubled/tripled, and I am already facing a challenge of having too
> many regions per server. So I cannot go to that route.
>
> From: 陆巍 [mailto:luwei114@163.com]
> Sent: Wednesday, July 13, 2016 11:24 AM
> To: user@hbase.apache.org; Frank Luo <jl...@merkleinc.com>
> Subject: Re:is possible to create multiple TableSplit per region?
>
> here is an archived mail:
> http://mail-archives.apache.org/mod_mbox/hbase-user/201303.mbox/%3CBLU0-SMTP19115A8967869D6CF0D49EF8FD70@phx.gbl%3E
>
>
>
> At 2016-07-13 23:20:28, "Frank Luo" <jluo@merkleinc.com<mailto:
> jluo@merkleinc.com>> wrote:
>
> >We have mapper only jobs operating on a result of a Scan. Because of
> heavy work downstream, the mapper runs fairly slowly. So I am wondering if
> there is a way to create multiple TableSplit on one region hence multiple
> mappers can be created to work on different piece of date on the region.
>
> >
>
> >I am aware of MultithreadedTableMapper class, which could be my solution,
> but I hesitate to use it as my code is not thread safe.
>
> >
>
> >So any suggestions, or code to share?
>
> >
>
> >
>
> >
>
> >Download the latest installment of our annual Marketing Imperatives,
> “Winning with People-Based Marketing”<
> http://www2.merkleinc.com/l/47252/2016-04-26/3lbfd1>
>
> >
>
> >This email and any attachments transmitted with it are intended for use
> by the intended recipient(s) only. If you have received this email in
> error, please notify the sender immediately and then delete it. If you are
> not the intended recipient, you must not keep, use, disclose, copy or
> distribute this email without the author’s prior permission. We take
> precautions to minimize the risk of transmitting software viruses, but we
> advise you to perform your own virus checks on any attachment to this
> message. We cannot accept liability for any loss or damage caused by
> software viruses. The information contained in this communication may be
> confidential and may be subject to the attorney-client privilege.
>
>
>
>
> Download the latest installment of our annual Marketing Imperatives,
> “Winning with People-Based Marketing”<
> http://www2.merkleinc.com/l/47252/2016-04-26/3lbfd1>
>
> This email and any attachments transmitted with it are intended for use by
> the intended recipient(s) only. If you have received this email in error,
> please notify the sender immediately and then delete it. If you are not the
> intended recipient, you must not keep, use, disclose, copy or distribute
> this email without the author’s prior permission. We take precautions to
> minimize the risk of transmitting software viruses, but we advise you to
> perform your own virus checks on any attachment to this message. We cannot
> accept liability for any loss or damage caused by software viruses. The
> information contained in this communication may be confidential and may be
> subject to the attorney-client privilege.
>
RE: Re:is possible to create multiple TableSplit per region?
Posted by Frank Luo <jl...@merkleinc.com>.
Thanks for the prompt reply, Lu.
It is true that having a smaller region file size can solve the problem. But it also have side effects. For example, total number of regions can be easily doubled/tripled, and I am already facing a challenge of having too many regions per server. So I cannot go to that route.
From: 陆巍 [mailto:luwei114@163.com]
Sent: Wednesday, July 13, 2016 11:24 AM
To: user@hbase.apache.org; Frank Luo <jl...@merkleinc.com>
Subject: Re:is possible to create multiple TableSplit per region?
here is an archived mail: http://mail-archives.apache.org/mod_mbox/hbase-user/201303.mbox/%3CBLU0-SMTP19115A8967869D6CF0D49EF8FD70@phx.gbl%3E
At 2016-07-13 23:20:28, "Frank Luo" <jl...@merkleinc.com>> wrote:
>We have mapper only jobs operating on a result of a Scan. Because of heavy work downstream, the mapper runs fairly slowly. So I am wondering if there is a way to create multiple TableSplit on one region hence multiple mappers can be created to work on different piece of date on the region.
>
>I am aware of MultithreadedTableMapper class, which could be my solution, but I hesitate to use it as my code is not thread safe.
>
>So any suggestions, or code to share?
>
>
>
>Download the latest installment of our annual Marketing Imperatives, “Winning with People-Based Marketing”<http://www2.merkleinc.com/l/47252/2016-04-26/3lbfd1>
>
>This email and any attachments transmitted with it are intended for use by the intended recipient(s) only. If you have received this email in error, please notify the sender immediately and then delete it. If you are not the intended recipient, you must not keep, use, disclose, copy or distribute this email without the author’s prior permission. We take precautions to minimize the risk of transmitting software viruses, but we advise you to perform your own virus checks on any attachment to this message. We cannot accept liability for any loss or damage caused by software viruses. The information contained in this communication may be confidential and may be subject to the attorney-client privilege.
Download the latest installment of our annual Marketing Imperatives, “Winning with People-Based Marketing”<http://www2.merkleinc.com/l/47252/2016-04-26/3lbfd1>
This email and any attachments transmitted with it are intended for use by the intended recipient(s) only. If you have received this email in error, please notify the sender immediately and then delete it. If you are not the intended recipient, you must not keep, use, disclose, copy or distribute this email without the author’s prior permission. We take precautions to minimize the risk of transmitting software viruses, but we advise you to perform your own virus checks on any attachment to this message. We cannot accept liability for any loss or damage caused by software viruses. The information contained in this communication may be confidential and may be subject to the attorney-client privilege.
Re:is possible to create multiple TableSplit per region?
Posted by 陆巍 <lu...@163.com>.
here is an archived mail: http://mail-archives.apache.org/mod_mbox/hbase-user/201303.mbox/%3CBLU0-SMTP19115A8967869D6CF0D49EF8FD70@phx.gbl%3E
At 2016-07-13 23:20:28, "Frank Luo" <jl...@merkleinc.com> wrote:
>We have mapper only jobs operating on a result of a Scan. Because of heavy work downstream, the mapper runs fairly slowly. So I am wondering if there is a way to create multiple TableSplit on one region hence multiple mappers can be created to work on different piece of date on the region.
>
>I am aware of MultithreadedTableMapper class, which could be my solution, but I hesitate to use it as my code is not thread safe.
>
>So any suggestions, or code to share?
>
>
>
>Download the latest installment of our annual Marketing Imperatives, “Winning with People-Based Marketing”<http://www2.merkleinc.com/l/47252/2016-04-26/3lbfd1>
>
>This email and any attachments transmitted with it are intended for use by the intended recipient(s) only. If you have received this email in error, please notify the sender immediately and then delete it. If you are not the intended recipient, you must not keep, use, disclose, copy or distribute this email without the author’s prior permission. We take precautions to minimize the risk of transmitting software viruses, but we advise you to perform your own virus checks on any attachment to this message. We cannot accept liability for any loss or damage caused by software viruses. The information contained in this communication may be confidential and may be subject to the attorney-client privilege.