You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by GitBox <gi...@apache.org> on 2019/12/26 20:38:47 UTC
[GitHub] [hbase] wchevreuil opened a new pull request #966: HBASE-23620 KeyPrefixRegionSplitPolicy may cause split to fail when s…
wchevreuil opened a new pull request #966: HBASE-23620 KeyPrefixRegionSplitPolicy may cause split to fail when s…
URL: https://github.com/apache/hbase/pull/966
…pecified prefix length is short enough that resulting split point ends up with same value of the parent start key.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] Apache-HBase commented on issue #966: HBASE-23620 KeyPrefixRegionSplitPolicy may cause split to fail when s…
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on issue #966: HBASE-23620 KeyPrefixRegionSplitPolicy may cause split to fail when s…
URL: https://github.com/apache/hbase/pull/966#issuecomment-569154001
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 58s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 5m 33s | master passed |
| +1 :green_heart: | compile | 0m 57s | master passed |
| +1 :green_heart: | checkstyle | 1m 20s | master passed |
| +1 :green_heart: | shadedjars | 4m 43s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 37s | master passed |
| +0 :ok: | spotbugs | 4m 31s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 4m 30s | master passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 5m 3s | the patch passed |
| +1 :green_heart: | compile | 0m 58s | the patch passed |
| +1 :green_heart: | javac | 0m 58s | the patch passed |
| +1 :green_heart: | checkstyle | 1m 17s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | shadedjars | 4m 36s | patch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | hadoopcheck | 16m 2s | Patch does not cause any errors with Hadoop 2.8.5 2.9.2 or 3.1.2. |
| +1 :green_heart: | javadoc | 0m 36s | the patch passed |
| +1 :green_heart: | findbugs | 4m 35s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 151m 34s | hbase-server in the patch passed. |
| +1 :green_heart: | asflicense | 0m 32s | The patch does not generate ASF License warnings. |
| | | 211m 21s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-966/1/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/966 |
| Optional Tests | dupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 77e2299bf25a 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /home/jenkins/jenkins-slave/workspace/HBase-PreCommit-GitHub-PR_PR-966/out/precommit/personality/provided.sh |
| git revision | master / 94346d8623 |
| Default Java | 1.8.0_181 |
| Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-966/1/testReport/ |
| Max. process+thread count | 4383 (vs. ulimit of 10000) |
| modules | C: hbase-server U: hbase-server |
| Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-966/1/console |
| versions | git=2.11.0 maven=2018-06-17T18:33:14Z) findbugs=3.1.11 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] wchevreuil commented on issue #966: HBASE-23620 KeyPrefixRegionSplitPolicy may cause split to fail when s…
Posted by GitBox <gi...@apache.org>.
wchevreuil commented on issue #966: HBASE-23620 KeyPrefixRegionSplitPolicy may cause split to fail when s…
URL: https://github.com/apache/hbase/pull/966#issuecomment-569285071
> For example, I add user id as the prefix of the row key, and I want to do local secondary index on the user data, then it is very important to me that all the data belong to a user are in the same region.
Yeah, most of the time the assumption would still be guaranteed. And some cases might not be so strict on the specific prefix value. But if the given row key prefix range can't fit in a region, what then, simply leave it grow indefinitely, and keep attempting split it even knowing it will fail?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] Apache9 commented on issue #966: HBASE-23620 KeyPrefixRegionSplitPolicy may cause split to fail when s…
Posted by GitBox <gi...@apache.org>.
Apache9 commented on issue #966: HBASE-23620 KeyPrefixRegionSplitPolicy may cause split to fail when s…
URL: https://github.com/apache/hbase/pull/966#issuecomment-569286840
Assumption means always right, there is no 'mostly right' assumption... Users choose this split policy means they want this assumption.
I think for an end user, the most important thing is monitoring, you need to know that a region is too large with this split policy, you need check whether this is expected, and try to fix it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] wchevreuil commented on issue #966: HBASE-23620 KeyPrefixRegionSplitPolicy may cause split to fail when s…
Posted by GitBox <gi...@apache.org>.
wchevreuil commented on issue #966: HBASE-23620 KeyPrefixRegionSplitPolicy may cause split to fail when s…
URL: https://github.com/apache/hbase/pull/966#issuecomment-573623324
This specific case was a customer using [yarn timeline service v2](https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/TimelineServiceV2.html). Internally, it defines a KPRSP with prefix length of 4, which in this case had problems to split regions for rows representing entries for 'hdfs' user. YARN-10077 has been filled to increase the prefix length in the table definition.
> If you do not need this assumption then just do not use this policy?
Some applications may not need it as a strict rule. @Apache9 arguments, though, are valid. Maybe a better solution then is to implement this flexible behaviour as another policy (like @saintstack suggested). But for KPRSP, I think it should at least apply the validation and throw an exception if it can't get a valid split point, other than leaving the procedure moving forward and failing further to perform the split.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] Apache9 commented on issue #966: HBASE-23620 KeyPrefixRegionSplitPolicy may cause split to fail when s…
Posted by GitBox <gi...@apache.org>.
Apache9 commented on issue #966: HBASE-23620 KeyPrefixRegionSplitPolicy may cause split to fail when s…
URL: https://github.com/apache/hbase/pull/966#issuecomment-573315577
> On the one hand, this fix is good; it caters for an issue the operator did not foresee. On the otherhand, as per @Apache9 , it does the 'fix' in violation of this KPRSP. Minimally I would think the class comment would change and we'd log the policy violations. Better would be a new policy that subclasses this one that is explicit that it will violate KPRSP if it fails to find an obliging split?
If you do not need this assumption then just do not use this policy? It is not the default one I'd say...
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] joshelser commented on issue #966: HBASE-23620 KeyPrefixRegionSplitPolicy may cause split to fail when s…
Posted by GitBox <gi...@apache.org>.
joshelser commented on issue #966: HBASE-23620 KeyPrefixRegionSplitPolicy may cause split to fail when s…
URL: https://github.com/apache/hbase/pull/966#issuecomment-574802297
> Leaving it as it is now, where it's completely breaking split seems the worse option to me.
> But for KPRSP, I think it should at least apply the validation and throw an exception if it can't get a valid split point, other than leaving the procedure moving forward and failing further to perform the split.
So, I can see the exception on Jira, but how does this manifest? Is this a side-effect of pv2 where the split procedure just keeps getting retried? Is it a simple thing for a SplitPolicy to "defer" making a split when we detect the case where we'd generate an invalid split point?
Agree with Duo that we can't change KPRSP's default functionality as that could seriously break users. I could see a new SplitPolicy which encapsulates your change, Wellington.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] wchevreuil commented on issue #966: HBASE-23620 KeyPrefixRegionSplitPolicy may cause split to fail when s…
Posted by GitBox <gi...@apache.org>.
wchevreuil commented on issue #966: HBASE-23620 KeyPrefixRegionSplitPolicy may cause split to fail when s…
URL: https://github.com/apache/hbase/pull/966#issuecomment-569279432
> This breaks the assumption of this policy? The length of the split point could be greater than the prefixLength?
Yeah, maybe worth log a WARN? Or fallback to the default policy in such condition? Leaving it as it is now, where it's completely breaking split seems the worse option to me.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] Apache9 commented on issue #966: HBASE-23620 KeyPrefixRegionSplitPolicy may cause split to fail when s…
Posted by GitBox <gi...@apache.org>.
Apache9 commented on issue #966: HBASE-23620 KeyPrefixRegionSplitPolicy may cause split to fail when s…
URL: https://github.com/apache/hbase/pull/966#issuecomment-569162626
This breaks the assumption of this policy? The length of the split point could be greater than the prefixLength?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] Apache9 commented on issue #966: HBASE-23620 KeyPrefixRegionSplitPolicy may cause split to fail when s…
Posted by GitBox <gi...@apache.org>.
Apache9 commented on issue #966: HBASE-23620 KeyPrefixRegionSplitPolicy may cause split to fail when s…
URL: https://github.com/apache/hbase/pull/966#issuecomment-569280428
This is a very important assumption, as it make sure that the rows with same prefix are in the same region. For example, I add user id as the prefix of the row key, and I want to do local secondary index on the user data, then it is very important to me that all the data belong to a user are in the same region.
So if you do not need this assumption, just do not use this split policy?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] saintstack commented on issue #966: HBASE-23620 KeyPrefixRegionSplitPolicy may cause split to fail when s…
Posted by GitBox <gi...@apache.org>.
saintstack commented on issue #966: HBASE-23620 KeyPrefixRegionSplitPolicy may cause split to fail when s…
URL: https://github.com/apache/hbase/pull/966#issuecomment-570363974
On the one hand, this fix is good; it caters for an issue the operator did not foresee. On the otherhand, as per @Apache9 , it does the 'fix' in violation of this KPRSP. Minimally I would think the class comment would change and we'd log the policy violations. Better would be a new policy that subclasses this one that is explicit that it will violate KPRSP if it fails to find an obliging split?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services