You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "Tom-Newton (via GitHub)" <gi...@apache.org> on 2026/01/28 14:34:28 UTC

[I] [C++][FS][Azure] `GetFileInfo` occasionally resports NOT_FOUND on directories that do exist [arrow]

Tom-Newton opened a new issue, #49043:
URL: https://github.com/apache/arrow/issues/49043

   ### Describe the bug, including details regarding any error messages, version, and platform.
   
   We found an obscure edgecase in a production usecase where the Arrow `AzureFileSystem::GetFileInfo` incorrectly returns `FileType::NotFound`. 
   
   I think this is going to be very hard to create a reproduce for that doesn't need to read our Azure storage account, but I have idenitified the root cause. 
   
   `GetFileInfo` on flat namespace Azure blob storage accounts does the following
   ```
       Blobs::ListBlobsOptions options;
       options.Prefix = internal::RemoveTrailingSlash(location.path);
       options.PageSizeHint = 1;
   
       auto list_response = container_client.ListBlobsByHierarchy("/", options);
       if (!list_response.BlobPrefixes.empty()) {
           ...
       }
       if (!list_response.Blobs.empty()) {
           ...
       }
       info.set_type(FileType::NotFound);
       return info;
   ```
   `list_response` contains the results of the first page, but to see all the results requires calling `MoveToNextPage`. In the above code we only check the first page, because order is guaranteed and we only want the first result. However, it seems that in some rare cases `container_client.ListBlobsByHierarchy` can return an empty first page while other pages are not empty. Therefore we incorrectly get `FileType::NotFound`. 
   
   I think I can probably make a PR to fix this, but I don't know how to write a test for it. The issue only occurs when Azure does a somewhat strange thing, that may well not exist in the azurite emulator. 
   
   ### Component(s)
   
   C++


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [C++][FS][Azure] `GetFileInfo` occasionally resports NOT_FOUND on directories that do exist [arrow]

Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on issue #49043:
URL: https://github.com/apache/arrow/issues/49043#issuecomment-3822534952

   Issue resolved by pull request 49049
   https://github.com/apache/arrow/pull/49049


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [C++][FS][Azure] `GetFileInfo` occasionally resports NOT_FOUND on directories that do exist [arrow]

Posted by "Tom-Newton (via GitHub)" <gi...@apache.org>.
Tom-Newton commented on issue #49043:
URL: https://github.com/apache/arrow/issues/49043#issuecomment-3812401981

   There are actually quite a lot of places in the `AzureFileSystem` that assume if the first page is empty then the listing result is empty. I'm thinking to update all of these to skip empty pages. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [C++][FS][Azure] `GetFileInfo` occasionally resports NOT_FOUND on directories that do exist [arrow]

Posted by "kou (via GitHub)" <gi...@apache.org>.
kou closed issue #49043: [C++][FS][Azure] `GetFileInfo` occasionally resports NOT_FOUND on directories that do exist
URL: https://github.com/apache/arrow/issues/49043


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org