You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by GitBox <gi...@apache.org> on 2020/07/29 07:50:09 UTC
[GitHub] [hbase] utf7 opened a new pull request #2167: HBASE-24791 Improve HFileOutputFormat2 to avoid always call getTableRelativePath method
utf7 opened a new pull request #2167:
URL: https://github.com/apache/hbase/pull/2167
HBASE-24791 Improve HFileOutputFormat2 to avoid always call getTableRelativePath method
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] anoopsjohn merged pull request #2167: HBASE-24791 Improve HFileOutputFormat2 to avoid always call getTableRelativePath method
Posted by GitBox <gi...@apache.org>.
anoopsjohn merged pull request #2167:
URL: https://github.com/apache/hbase/pull/2167
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2167: HBASE-24791 Improve HFileOutputFormat2 to avoid always call getTableRelativePath method
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #2167:
URL: https://github.com/apache/hbase/pull/2167#issuecomment-665571205
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2167: HBASE-24791 Improve HFileOutputFormat2 to avoid always call getTableRelativePath method
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #2167:
URL: https://github.com/apache/hbase/pull/2167#issuecomment-667837704
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 30s | Docker mode activated. |
| -0 :warning: | yetus | 0m 4s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 4m 13s | master passed |
| +1 :green_heart: | compile | 0m 28s | master passed |
| +1 :green_heart: | shadedjars | 5m 47s | branch has no errors when building our shaded downstream artifacts. |
| -0 :warning: | javadoc | 0m 23s | hbase-mapreduce in master failed. |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 4m 3s | the patch passed |
| +1 :green_heart: | compile | 0m 28s | the patch passed |
| +1 :green_heart: | javac | 0m 28s | the patch passed |
| +1 :green_heart: | shadedjars | 5m 48s | patch has no errors when building our shaded downstream artifacts. |
| -0 :warning: | javadoc | 0m 20s | hbase-mapreduce in the patch failed. |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 9m 16s | hbase-mapreduce in the patch passed. |
| | | 32m 29s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.12 Server=19.03.12 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2167/4/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/2167 |
| Optional Tests | javac javadoc unit shadedjars compile |
| uname | Linux 7d6c473a9149 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 9a1bad84bf |
| Default Java | 2020-01-14 |
| javadoc | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2167/4/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-mapreduce.txt |
| javadoc | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2167/4/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-mapreduce.txt |
| Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2167/4/testReport/ |
| Max. process+thread count | 4874 (vs. ulimit of 12500) |
| modules | C: hbase-mapreduce U: hbase-mapreduce |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2167/4/console |
| versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] pankaj72981 commented on a change in pull request #2167: HBASE-24791 Improve HFileOutputFormat2 to avoid always call getTableRelativePath method
Posted by GitBox <gi...@apache.org>.
pankaj72981 commented on a change in pull request #2167:
URL: https://github.com/apache/hbase/pull/2167#discussion_r462084885
##########
File path: hbase-server/src/main/java/org/apache/hadoop/hbase/SplitLogCounters.java
##########
@@ -32,8 +32,6 @@
public final static LongAdder tot_mgr_log_split_batch_start = new LongAdder();
public final static LongAdder tot_mgr_log_split_batch_success = new LongAdder();
public final static LongAdder tot_mgr_log_split_batch_err = new LongAdder();
- public final static LongAdder tot_mgr_new_unexpected_wals = new LongAdder();
Review comment:
Remove HBASE-24790 related change in this PR.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] anoopsjohn commented on a change in pull request #2167: HBASE-24791 Improve HFileOutputFormat2 to avoid always call getTableRelativePath method
Posted by GitBox <gi...@apache.org>.
anoopsjohn commented on a change in pull request #2167:
URL: https://github.com/apache/hbase/pull/2167#discussion_r464174854
##########
File path: hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat2.java
##########
@@ -222,6 +222,7 @@ public RegionLocator getRegionLocator() {
private final Map<byte[], WriterLength> writers = new TreeMap<>(Bytes.BYTES_COMPARATOR);
private final Map<byte[], byte[]> previousRows = new TreeMap<>(Bytes.BYTES_COMPARATOR);
private final long now = EnvironmentEdgeManager.currentTime();
+ private byte[] tableNameBytes = Bytes.toBytes(writeTableNames);
Review comment:
Should we do this under writeMultipleTables check?
private byte[] tableNameBytes = (writeMultipleTables)? null: Bytes.toBytes(writeTableNames);
##########
File path: hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat2.java
##########
@@ -274,39 +270,36 @@ public void write(ImmutableBytesWritable row, V cell) throws IOException {
// create a new WAL writer, if necessary
if (wl == null || wl.writer == null) {
+ InetSocketAddress[] favoredNodes = null;
if (conf.getBoolean(LOCALITY_SENSITIVE_CONF_KEY, DEFAULT_LOCALITY_SENSITIVE)) {
HRegionLocation loc = null;
-
+ String tableName = Bytes.toString(tableNameBytes);
if (tableName != null) {
try (Connection connection = ConnectionFactory.createConnection(conf);
- RegionLocator locator =
- connection.getRegionLocator(TableName.valueOf(tableName))) {
+ RegionLocator locator = connection.getRegionLocator(TableName.valueOf(tableName))) {
loc = locator.getRegionLocation(rowKey);
} catch (Throwable e) {
- LOG.warn("Something wrong locating rowkey {} in {}",
- Bytes.toString(rowKey), tableName, e);
+ LOG.warn("Something wrong locating rowkey {} in {}", Bytes.toString(rowKey),
+ tableName, e);
loc = null;
- } }
-
+ }
+ }
Review comment:
Here also.
##########
File path: hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat2.java
##########
@@ -274,39 +270,36 @@ public void write(ImmutableBytesWritable row, V cell) throws IOException {
// create a new WAL writer, if necessary
if (wl == null || wl.writer == null) {
+ InetSocketAddress[] favoredNodes = null;
if (conf.getBoolean(LOCALITY_SENSITIVE_CONF_KEY, DEFAULT_LOCALITY_SENSITIVE)) {
HRegionLocation loc = null;
-
+ String tableName = Bytes.toString(tableNameBytes);
Review comment:
Looks like a format issue here?
##########
File path: hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat2.java
##########
@@ -376,16 +369,15 @@ private WriterLength getNewWriter(byte[] tableName, byte[] family, Configuration
DataBlockEncoding encoding = overriddenEncoding;
encoding = encoding == null ? datablockEncodingMap.get(tableAndFamily) : encoding;
encoding = encoding == null ? DataBlockEncoding.NONE : encoding;
- HFileContextBuilder contextBuilder = new HFileContextBuilder()
- .withCompression(compression).withChecksumType(HStore.getChecksumType(conf))
- .withBytesPerCheckSum(HStore.getBytesPerChecksum(conf)).withBlockSize(blockSize)
- .withColumnFamily(family).withTableName(tableName);
+ HFileContextBuilder contextBuilder = new HFileContextBuilder().withCompression(compression)
Review comment:
Pls check format issue at these changed/added lines once.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] utf7 commented on a change in pull request #2167: HBASE-24791 Improve HFileOutputFormat2 to avoid always call getTableRelativePath method
Posted by GitBox <gi...@apache.org>.
utf7 commented on a change in pull request #2167:
URL: https://github.com/apache/hbase/pull/2167#discussion_r462120907
##########
File path: hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat2.java
##########
@@ -248,7 +248,6 @@ public void write(ImmutableBytesWritable row, V cell) throws IOException {
tableNameBytes = Bytes.toBytes(writeTableNames);
}
String tableName = Bytes.toString(tableNameBytes);
- Path tableRelPath = getTableRelativePath(tableNameBytes);
Review comment:
tableName split 3 times int getTableRelativePath method has fixed in this PR
i will do some other fixes in next commit , the code there is a little mess
##########
File path: hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat2.java
##########
@@ -222,6 +222,7 @@ public RegionLocator getRegionLocator() {
private final Map<byte[], WriterLength> writers = new TreeMap<>(Bytes.BYTES_COMPARATOR);
private final Map<byte[], byte[]> previousRows = new TreeMap<>(Bytes.BYTES_COMPARATOR);
private final long now = EnvironmentEdgeManager.currentTime();
+ private byte[] tableNameBytes = Bytes.toBytes(writeTableNames);;
Review comment:
ok,thanks for ponit it
##########
File path: hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat2.java
##########
@@ -222,6 +222,7 @@ public RegionLocator getRegionLocator() {
private final Map<byte[], WriterLength> writers = new TreeMap<>(Bytes.BYTES_COMPARATOR);
private final Map<byte[], byte[]> previousRows = new TreeMap<>(Bytes.BYTES_COMPARATOR);
private final long now = EnvironmentEdgeManager.currentTime();
+ private byte[] tableNameBytes = Bytes.toBytes(writeTableNames);;
Review comment:
has finished
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] utf7 commented on a change in pull request #2167: HBASE-24791 Improve HFileOutputFormat2 to avoid always call getTableRelativePath method
Posted by GitBox <gi...@apache.org>.
utf7 commented on a change in pull request #2167:
URL: https://github.com/apache/hbase/pull/2167#discussion_r464209802
##########
File path: hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat2.java
##########
@@ -376,16 +369,15 @@ private WriterLength getNewWriter(byte[] tableName, byte[] family, Configuration
DataBlockEncoding encoding = overriddenEncoding;
encoding = encoding == null ? datablockEncodingMap.get(tableAndFamily) : encoding;
encoding = encoding == null ? DataBlockEncoding.NONE : encoding;
- HFileContextBuilder contextBuilder = new HFileContextBuilder()
- .withCompression(compression).withChecksumType(HStore.getChecksumType(conf))
- .withBytesPerCheckSum(HStore.getBytesPerChecksum(conf)).withBlockSize(blockSize)
- .withColumnFamily(family).withTableName(tableName);
+ HFileContextBuilder contextBuilder = new HFileContextBuilder().withCompression(compression)
Review comment:
i move the compression set config to here and has been use hbase-eclipse-format format the code here
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] pankaj72981 commented on a change in pull request #2167: HBASE-24791 Improve HFileOutputFormat2 to avoid always call getTableRelativePath method
Posted by GitBox <gi...@apache.org>.
pankaj72981 commented on a change in pull request #2167:
URL: https://github.com/apache/hbase/pull/2167#discussion_r462203737
##########
File path: hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat2.java
##########
@@ -222,6 +222,7 @@ public RegionLocator getRegionLocator() {
private final Map<byte[], WriterLength> writers = new TreeMap<>(Bytes.BYTES_COMPARATOR);
private final Map<byte[], byte[]> previousRows = new TreeMap<>(Bytes.BYTES_COMPARATOR);
private final long now = EnvironmentEdgeManager.currentTime();
+ private byte[] tableNameBytes = Bytes.toBytes(writeTableNames);;
Review comment:
Remove extra semicolon at end.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] tedyu commented on a change in pull request #2167: HBASE-24791 Improve HFileOutputFormat2 to avoid always call getTableRelativePath method
Posted by GitBox <gi...@apache.org>.
tedyu commented on a change in pull request #2167:
URL: https://github.com/apache/hbase/pull/2167#discussion_r462583864
##########
File path: hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat2.java
##########
@@ -274,39 +270,36 @@ public void write(ImmutableBytesWritable row, V cell) throws IOException {
// create a new WAL writer, if necessary
if (wl == null || wl.writer == null) {
+ InetSocketAddress[] favoredNodes = null;
if (conf.getBoolean(LOCALITY_SENSITIVE_CONF_KEY, DEFAULT_LOCALITY_SENSITIVE)) {
HRegionLocation loc = null;
-
+ String tableName = Bytes.toString(tableNameBytes);
if (tableName != null) {
try (Connection connection = ConnectionFactory.createConnection(conf);
- RegionLocator locator =
- connection.getRegionLocator(TableName.valueOf(tableName))) {
+ RegionLocator locator = connection.getRegionLocator(TableName.valueOf(tableName))) {
loc = locator.getRegionLocation(rowKey);
} catch (Throwable e) {
- LOG.warn("Something wrong locating rowkey {} in {}",
- Bytes.toString(rowKey), tableName, e);
+ LOG.warn("Something wrong locating rowkey {} in {}", Bytes.toString(rowKey),
+ tableName, e);
loc = null;
- } }
-
+ }
+ }
if (null == loc) {
LOG.trace("Failed get of location, use default writer {}", Bytes.toString(rowKey));
- wl = getNewWriter(tableNameBytes, family, conf, null);
} else {
LOG.debug("First rowkey: [{}]", Bytes.toString(rowKey));
InetSocketAddress initialIsa =
new InetSocketAddress(loc.getHostname(), loc.getPort());
if (initialIsa.isUnresolved()) {
LOG.trace("Failed resolve address {}, use default writer", loc.getHostnamePort());
- wl = getNewWriter(tableNameBytes, family, conf, null);
} else {
LOG.debug("Use favored nodes writer: {}", initialIsa.getHostString());
- wl = getNewWriter(tableNameBytes, family, conf, new InetSocketAddress[] { initialIsa
- });
+ favoredNodes = new InetSocketAddress[] { initialIsa};
Review comment:
Does this apply when LOCALITY_SENSITIVE_CONF_KEY check on line 274 is false ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] utf7 commented on pull request #2167: HBASE-24791 Improve HFileOutputFormat2 to avoid always call getTableRelativePath method
Posted by GitBox <gi...@apache.org>.
utf7 commented on pull request #2167:
URL: https://github.com/apache/hbase/pull/2167#issuecomment-666112861
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] utf7 commented on pull request #2167: HBASE-24791 Improve HFileOutputFormat2 to avoid always call getTableRelativePath method
Posted by GitBox <gi...@apache.org>.
utf7 commented on pull request #2167:
URL: https://github.com/apache/hbase/pull/2167#issuecomment-667767166
The code hash finished , mind have a look, thanks @anoopsjohn @pankaj72981
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] utf7 closed pull request #2167: HBASE-24791 Improve HFileOutputFormat2 to avoid always call getTableRelativePath method
Posted by GitBox <gi...@apache.org>.
utf7 closed pull request #2167:
URL: https://github.com/apache/hbase/pull/2167
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] utf7 edited a comment on pull request #2167: HBASE-24791 Improve HFileOutputFormat2 to avoid always call getTableRelativePath method
Posted by GitBox <gi...@apache.org>.
utf7 edited a comment on pull request #2167:
URL: https://github.com/apache/hbase/pull/2167#issuecomment-665471339
This can imporve a lot
A simple test case
before improve cost: 8566118066 7954087533 7990385173 8121662923 8094065619 8107938198 8045528125 8116824329 8122912440 8108117395
after improve cost: 1536854 50156 1097601 33953 9907 10927 10445 13076 8596 10386
before all cost: 81227.639801 ms
after all cost : 2.781901 ms
before vs after: 29198.60908098455:1
The test code in
https://github.com/utf7/hbase-client-example/blob/master/src/test/java/TestGetTableRelativePathImprove.java
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2167: HBASE-24791 Improve HFileOutputFormat2 to avoid always call getTableRelativePath method
Posted by GitBox <gi...@apache.org>.
Apache-HBase commented on pull request #2167:
URL: https://github.com/apache/hbase/pull/2167#issuecomment-667838016
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 8s | Docker mode activated. |
| -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck |
||| _ Prechecks _ |
||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 3m 50s | master passed |
| +1 :green_heart: | compile | 0m 27s | master passed |
| +1 :green_heart: | shadedjars | 5m 59s | branch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 19s | master passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 3m 39s | the patch passed |
| +1 :green_heart: | compile | 0m 26s | the patch passed |
| +1 :green_heart: | javac | 0m 26s | the patch passed |
| +1 :green_heart: | shadedjars | 5m 30s | patch has no errors when building our shaded downstream artifacts. |
| +1 :green_heart: | javadoc | 0m 18s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 10m 33s | hbase-mapreduce in the patch passed. |
| | | 33m 17s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.12 Server=19.03.12 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2167/4/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/2167 |
| Optional Tests | javac javadoc unit shadedjars compile |
| uname | Linux 16248f5deec7 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 9a1bad84bf |
| Default Java | 1.8.0_232 |
| Test Results | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2167/4/testReport/ |
| Max. process+thread count | 4751 (vs. ulimit of 12500) |
| modules | C: hbase-mapreduce U: hbase-mapreduce |
| Console output | https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2167/4/console |
| versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] utf7 commented on pull request #2167: HBASE-24791 Improve HFileOutputFormat2 to avoid always call getTableRelativePath method
Posted by GitBox <gi...@apache.org>.
utf7 commented on pull request #2167:
URL: https://github.com/apache/hbase/pull/2167#issuecomment-666207403
> > ok, i will retry in jdk 11 and find it why failed
> > thanks @tedyu @Apache9
>
> This could be another issue. Do not need to address it in this PR. It is not a blocker.
Yes,use jdk 11 build with doc always failed ,it's not related this pr
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] utf7 commented on pull request #2167: HBASE-24791 Improve HFileOutputFormat2 to avoid always call getTableRelativePath method
Posted by GitBox <gi...@apache.org>.
utf7 commented on pull request #2167:
URL: https://github.com/apache/hbase/pull/2167#issuecomment-665549289
remove the HBASE-24790 code and do more code clean by review
@anoopsjohn @pankaj72981 please have a look, thanks
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] utf7 commented on pull request #2167: HBASE-24791 Improve HFileOutputFormat2 to avoid always call getTableRelativePath method
Posted by GitBox <gi...@apache.org>.
utf7 commented on pull request #2167:
URL: https://github.com/apache/hbase/pull/2167#issuecomment-665471339
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] tedyu commented on pull request #2167: HBASE-24791 Improve HFileOutputFormat2 to avoid always call getTableRelativePath method
Posted by GitBox <gi...@apache.org>.
tedyu commented on pull request #2167:
URL: https://github.com/apache/hbase/pull/2167#issuecomment-666071103
Can you see why javadoc for hbase-mapreduce didn't pass ?
Thanks
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] utf7 commented on a change in pull request #2167: HBASE-24791 Improve HFileOutputFormat2 to avoid always call getTableRelativePath method
Posted by GitBox <gi...@apache.org>.
utf7 commented on a change in pull request #2167:
URL: https://github.com/apache/hbase/pull/2167#discussion_r464207581
##########
File path: hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat2.java
##########
@@ -222,6 +222,7 @@ public RegionLocator getRegionLocator() {
private final Map<byte[], WriterLength> writers = new TreeMap<>(Bytes.BYTES_COMPARATOR);
private final Map<byte[], byte[]> previousRows = new TreeMap<>(Bytes.BYTES_COMPARATOR);
private final long now = EnvironmentEdgeManager.currentTime();
+ private byte[] tableNameBytes = Bytes.toBytes(writeTableNames);
Review comment:
yes，i want to make code simple, so just keep init no matter multiple or not
It is a good ponit to use use writeMultipleTables check,will address it
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] utf7 commented on a change in pull request #2167: HBASE-24791 Improve HFileOutputFormat2 to avoid always call getTableRelativePath method
Posted by GitBox <gi...@apache.org>.
utf7 commented on a change in pull request #2167:
URL: https://github.com/apache/hbase/pull/2167#discussion_r462672401
##########
File path: hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat2.java
##########
@@ -274,39 +270,36 @@ public void write(ImmutableBytesWritable row, V cell) throws IOException {
// create a new WAL writer, if necessary
if (wl == null || wl.writer == null) {
+ InetSocketAddress[] favoredNodes = null;
if (conf.getBoolean(LOCALITY_SENSITIVE_CONF_KEY, DEFAULT_LOCALITY_SENSITIVE)) {
HRegionLocation loc = null;
-
+ String tableName = Bytes.toString(tableNameBytes);
if (tableName != null) {
try (Connection connection = ConnectionFactory.createConnection(conf);
- RegionLocator locator =
- connection.getRegionLocator(TableName.valueOf(tableName))) {
+ RegionLocator locator = connection.getRegionLocator(TableName.valueOf(tableName))) {
loc = locator.getRegionLocation(rowKey);
} catch (Throwable e) {
- LOG.warn("Something wrong locating rowkey {} in {}",
- Bytes.toString(rowKey), tableName, e);
+ LOG.warn("Something wrong locating rowkey {} in {}", Bytes.toString(rowKey),
+ tableName, e);
loc = null;
- } }
-
+ }
+ }
if (null == loc) {
LOG.trace("Failed get of location, use default writer {}", Bytes.toString(rowKey));
- wl = getNewWriter(tableNameBytes, family, conf, null);
} else {
LOG.debug("First rowkey: [{}]", Bytes.toString(rowKey));
InetSocketAddress initialIsa =
new InetSocketAddress(loc.getHostname(), loc.getPort());
if (initialIsa.isUnresolved()) {
LOG.trace("Failed resolve address {}, use default writer", loc.getHostnamePort());
- wl = getNewWriter(tableNameBytes, family, conf, null);
} else {
LOG.debug("Use favored nodes writer: {}", initialIsa.getHostString());
- wl = getNewWriter(tableNameBytes, family, conf, new InetSocketAddress[] { initialIsa
- });
+ favoredNodes = new InetSocketAddress[] { initialIsa};
Review comment:
Yes, if the LOCALITY_SENSITIVE_CONF_KEY is false or get locality failed,the favoredNodes will be null
if LOCALITY_SENSITIVE_CONF_KEY = true and get localicy success ,the favoredNodes will be not null
same logic with before , just code clean
before this pr, too much `wl = getNewWriter` in the code
##########
File path: hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat2.java
##########
@@ -274,39 +270,36 @@ public void write(ImmutableBytesWritable row, V cell) throws IOException {
// create a new WAL writer, if necessary
if (wl == null || wl.writer == null) {
+ InetSocketAddress[] favoredNodes = null;
if (conf.getBoolean(LOCALITY_SENSITIVE_CONF_KEY, DEFAULT_LOCALITY_SENSITIVE)) {
HRegionLocation loc = null;
-
+ String tableName = Bytes.toString(tableNameBytes);
if (tableName != null) {
try (Connection connection = ConnectionFactory.createConnection(conf);
- RegionLocator locator =
- connection.getRegionLocator(TableName.valueOf(tableName))) {
+ RegionLocator locator = connection.getRegionLocator(TableName.valueOf(tableName))) {
loc = locator.getRegionLocation(rowKey);
} catch (Throwable e) {
- LOG.warn("Something wrong locating rowkey {} in {}",
- Bytes.toString(rowKey), tableName, e);
+ LOG.warn("Something wrong locating rowkey {} in {}", Bytes.toString(rowKey),
+ tableName, e);
loc = null;
- } }
-
+ }
+ }
if (null == loc) {
LOG.trace("Failed get of location, use default writer {}", Bytes.toString(rowKey));
- wl = getNewWriter(tableNameBytes, family, conf, null);
} else {
LOG.debug("First rowkey: [{}]", Bytes.toString(rowKey));
InetSocketAddress initialIsa =
new InetSocketAddress(loc.getHostname(), loc.getPort());
if (initialIsa.isUnresolved()) {
LOG.trace("Failed resolve address {}, use default writer", loc.getHostnamePort());
- wl = getNewWriter(tableNameBytes, family, conf, null);
} else {
LOG.debug("Use favored nodes writer: {}", initialIsa.getHostString());
- wl = getNewWriter(tableNameBytes, family, conf, new InetSocketAddress[] { initialIsa
- });
+ favoredNodes = new InetSocketAddress[] { initialIsa};
Review comment:
Yes, if the `LOCALITY_SENSITIVE_CONF_KEY` is false or get locality failed,the `favoredNodes` will be null
if `LOCALITY_SENSITIVE_CONF_KEY` = true and get localicy success ,the `favoredNodes` will be not null
same logic with before , just code clean
before this pr, too much `wl = getNewWriter` in the code
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hbase] tedyu commented on pull request #2167: HBASE-24791 Improve HFileOutputFormat2 to avoid always call getTableRelativePath method
Posted by GitBox <gi...@apache.org>.
tedyu commented on pull request #2167:
URL: https://github.com/apache/hbase/pull/2167#issuecomment-666330350
+1
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org