You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@accumulo.apache.org by "cshannon (via GitHub)" <gi...@apache.org> on 2023/06/16 12:38:06 UTC

[GitHub] [accumulo] cshannon opened a new pull request, #3502: StoredTabletFile now lazy loads ReferencedTabletFile

cshannon opened a new pull request, #3502:
URL: https://github.com/apache/accumulo/pull/3502

   This improves performance because creating a `ReferencedTabletFile` does a lot of validation which can be slow due to the calls to Path.getParent() if there are several files created. Filename validation was also removed from the constructor as well to further improve performance.
   
   Removing/delaying the validation checks should be safe because a `StoredTabletFile` references an existing Tablet file that was already stored in Metadata so the Path has already been validated before. Also, the `ReferencedTabletFile` is not needed often so it makes sense to delay the creation/validation until it's actually needed.
   
   It turns out that the refactoring I just did in #3417 made this super easy because the bottleneck is primarily the validation and calling Path.getParent() (not the actual creation of the Path object) which is now only done when accessing a ReferencedTabletFile. This would be harder to fix in 2.1.x due to the inheritance.
   
   #### Benchmark _before_ this change running against main:
   
   ```
   Warmup Iteration   1: 0.017 ms/op
   Iteration   1: 0.012 ms/op
   Iteration   2: 0.012 ms/op
   Iteration   3: 0.011 ms/op
   
   0.012 ±(99.9%) 0.008 ms/op [Average]
   (min, avg, max) = (0.011, 0.012, 0.012), stdev = 0.001
   CI (99.9%): [0.003, 0.020] (assumes normal distribution)
   ```
   
   #### Benchmark _after_ this change running against main:
   
   ```
   Warmup Iteration   1: 0.007 ms/op
   Iteration   1: 0.004 ms/op
   Iteration   2: 0.004 ms/op
   Iteration   3: 0.004 ms/op
   
   0.004 ±(99.9%) 0.002 ms/op [Average]
   (min, avg, max) = (0.004, 0.004, 0.004), stdev = 0.001
   CI (99.9%): [0.002, 0.007] (assumes normal distribution)
   ```
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [accumulo] cshannon commented on pull request #3502: StoredTabletFile should lazy load ReferencedTabletFile

Posted by "cshannon (via GitHub)" <gi...@apache.org>.
cshannon commented on PR #3502:
URL: https://github.com/apache/accumulo/pull/3502#issuecomment-1601134294

   I am going to revert this change as the changes in #3509 will increase performance quite a bit and be nearly as fast as the change here but is safer as we can still validate on object creation and not potentially run into problems later. See https://github.com/apache/accumulo/pull/3509#issuecomment-1599677302 for performance numbers.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [accumulo] cshannon merged pull request #3502: StoredTabletFile should lazy load ReferencedTabletFile

Posted by "cshannon (via GitHub)" <gi...@apache.org>.
cshannon merged PR #3502:
URL: https://github.com/apache/accumulo/pull/3502


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [accumulo] cshannon commented on pull request #3502: StoredTabletFile should lazy load ReferencedTabletFile

Posted by "cshannon (via GitHub)" <gi...@apache.org>.
cshannon commented on PR #3502:
URL: https://github.com/apache/accumulo/pull/3502#issuecomment-1601142557

   @ctubbsii - is there anything else to do here since this has been reverted other than remove it from the 3.0.0 project which I did? I don't see a good way to mark this as no longer merged.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [accumulo] cshannon commented on pull request #3502: StoredTabletFile should lazy load ReferencedTabletFile

Posted by "cshannon (via GitHub)" <gi...@apache.org>.
cshannon commented on PR #3502:
URL: https://github.com/apache/accumulo/pull/3502#issuecomment-1594642012

   Something else too is that even if the performance when creating a `ReferencedTabletFile` is improved in some other way, this change to lazy load is probably still a good one as we can reduce memory by not having to create the `ReferencedTabletFile` unless it is needed as it looks like in a lot of cases in the code when creating a `StoredTabletFile` we won't actually need the `ReferencedTabletFile`.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [accumulo] cshannon commented on pull request #3502: StoredTabletFile should lazy load ReferencedTabletFile

Posted by "cshannon (via GitHub)" <gi...@apache.org>.
cshannon commented on PR #3502:
URL: https://github.com/apache/accumulo/pull/3502#issuecomment-1594713706

   > LGTM. It does not seem to be needed, but the changes in #3417 would also make it pretty simple to lazily create the path object.
   
   I considered that but since the performance is much improved without it and because the Path object is used more often anyways it would probably get created at some point in many cases so I left it alone. It is something we could change later if needed though.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [accumulo] cshannon commented on a diff in pull request #3502: StoredTabletFile should lazy load ReferencedTabletFile

Posted by "cshannon (via GitHub)" <gi...@apache.org>.
cshannon commented on code in PR #3502:
URL: https://github.com/apache/accumulo/pull/3502#discussion_r1232392823


##########
core/src/main/java/org/apache/accumulo/core/metadata/StoredTabletFile.java:
##########
@@ -46,7 +49,7 @@ public class StoredTabletFile extends AbstractTabletFile<StoredTabletFile> {
   public StoredTabletFile(String metadataEntry) {
     super(new Path(metadataEntry));
     this.metadataEntry = metadataEntry;

Review Comment:
   Well (in theory) we should be only ever be passing in a String that was read from metadata so that means it was already stored before so it should not need validation as it would have already passed validation when stored originally. 
   
   When we store new files the metadata gets generated from the[ insert() ](https://github.com/apache/accumulo/blob/6a1431b773c49f422e109268dccd460661d5e02f/core/src/main/java/org/apache/accumulo/core/metadata/ReferencedTabletFile.java#L114)method of ReferencedTabletFile.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [accumulo] ctubbsii commented on pull request #3502: StoredTabletFile should lazy load ReferencedTabletFile

Posted by "ctubbsii (via GitHub)" <gi...@apache.org>.
ctubbsii commented on PR #3502:
URL: https://github.com/apache/accumulo/pull/3502#issuecomment-1601621283

   > I don't see a good way to mark this as no longer merged.
   
   There isn't one. It's fine. Your comment and removing the 3.0.0 project should suffice.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [accumulo] cshannon commented on a diff in pull request #3502: StoredTabletFile should lazy load ReferencedTabletFile

Posted by "cshannon (via GitHub)" <gi...@apache.org>.
cshannon commented on code in PR #3502:
URL: https://github.com/apache/accumulo/pull/3502#discussion_r1232392823


##########
core/src/main/java/org/apache/accumulo/core/metadata/StoredTabletFile.java:
##########
@@ -46,7 +49,7 @@ public class StoredTabletFile extends AbstractTabletFile<StoredTabletFile> {
   public StoredTabletFile(String metadataEntry) {
     super(new Path(metadataEntry));
     this.metadataEntry = metadataEntry;

Review Comment:
   Well (in theory) we should be only ever be passing in a String that was read from metadata so that means it was already stored before so it should not need validation as it would have already passed validation when stored originally because ReferencedTabletFile will have already validated the path.
   
   When we store new files the metadata gets generated from the[ insert() ](https://github.com/apache/accumulo/blob/6a1431b773c49f422e109268dccd460661d5e02f/core/src/main/java/org/apache/accumulo/core/metadata/ReferencedTabletFile.java#L114)method of ReferencedTabletFile.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [accumulo] cshannon commented on pull request #3502: StoredTabletFile should lazy load ReferencedTabletFile

Posted by "cshannon (via GitHub)" <gi...@apache.org>.
cshannon commented on PR #3502:
URL: https://github.com/apache/accumulo/pull/3502#issuecomment-1595482052

   I created a follow on issue #3504 to make sure we still do validation in MetadataConstraints now that we lazy load the object that does the path validation


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [accumulo] cshannon commented on a diff in pull request #3502: StoredTabletFile should lazy load ReferencedTabletFile

Posted by "cshannon (via GitHub)" <gi...@apache.org>.
cshannon commented on code in PR #3502:
URL: https://github.com/apache/accumulo/pull/3502#discussion_r1232394767


##########
core/src/main/java/org/apache/accumulo/core/metadata/StoredTabletFile.java:
##########
@@ -46,7 +49,7 @@ public class StoredTabletFile extends AbstractTabletFile<StoredTabletFile> {
   public StoredTabletFile(String metadataEntry) {
     super(new Path(metadataEntry));
     this.metadataEntry = metadataEntry;

Review Comment:
   Of course there could be a mistake in the future but as of now it's not an issue when I search the code. I took out that validate call because it was pointed out as an issue in the original Issue description #3500. 
   
   Leaving the call the performance was 0.005 ms/op but with it removed I get 0.004 ms/op so a little better.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [accumulo] dlmarion commented on a diff in pull request #3502: StoredTabletFile should lazy load ReferencedTabletFile

Posted by "dlmarion (via GitHub)" <gi...@apache.org>.
dlmarion commented on code in PR #3502:
URL: https://github.com/apache/accumulo/pull/3502#discussion_r1232357446


##########
core/src/main/java/org/apache/accumulo/core/metadata/StoredTabletFile.java:
##########
@@ -46,7 +49,7 @@ public class StoredTabletFile extends AbstractTabletFile<StoredTabletFile> {
   public StoredTabletFile(String metadataEntry) {
     super(new Path(metadataEntry));
     this.metadataEntry = metadataEntry;

Review Comment:
   `metadataEntry` can be used without `ValidationUtil.validateFileName` being called via `getMetaUpdateDelete`. Is that an issue?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@accumulo.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org