You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "Ivan Dimitrov (JIRA)" <ji...@apache.org> on 2019/05/16 21:20:00 UTC
[jira] [Updated] (ARROW-5318) pyarrow hdfs reader overrequests
[ https://issues.apache.org/jira/browse/ARROW-5318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ivan Dimitrov updated ARROW-5318:
---------------------------------
Description:
When reading using HdfsFilesystem's read method, the amount is not constant and flactuates between extra 0% to 300%.
Example code
{code:java}
fs = hdfs.connect(hostname, driver='libhdfs')
f = fs.open(dataset_path)
f.read(nbytes=3500000){code}
In this case, the read can send back up to 15 M bytes. The issue is true with 'libhdfs3' as well. Also present in newer versions of pyarrow.
was:
When reading using HdfsFilesystem's read method, the amount is not constant and flactuates between extra 0% to 300%.
Example code
{code:java}
fs = hdfs.connect(hostname, driver='libhdfs')
f = fs.open(dataset_path)
f.read(nbytes=3500000){code}
In this case, the read can send back up to 15 M bytes. The issue is true with 'libhdfs3' as well. Also present in newer versions of pyarrow.
> pyarrow hdfs reader overrequests
> ----------------------------------
>
> Key: ARROW-5318
> URL: https://issues.apache.org/jira/browse/ARROW-5318
> Project: Apache Arrow
> Issue Type: Bug
> Components: Python
> Affects Versions: 0.10.0
> Reporter: Ivan Dimitrov
> Priority: Blocker
>
> When reading using HdfsFilesystem's read method, the amount is not constant and flactuates between extra 0% to 300%.
> Example code
> {code:java}
> fs = hdfs.connect(hostname, driver='libhdfs')
> f = fs.open(dataset_path)
> f.read(nbytes=3500000){code}
> In this case, the read can send back up to 15 M bytes. The issue is true with 'libhdfs3' as well. Also present in newer versions of pyarrow.
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)