You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@beam.apache.org by "Fangyuan Zhou (Jira)" <ji...@apache.org> on 2020/01/20 09:09:00 UTC
[jira] [Updated] (BEAM-9152) Hadoop Downloader Range not correct
[ https://issues.apache.org/jira/browse/BEAM-9152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Fangyuan Zhou updated BEAM-9152:
--------------------------------
Description:
I found that the `HdfsDownloader.get_range(self, start, end)' get range {color:#ff0000}[start, end]{color} rather than {color:#ff0000}[start, end){color}. This will cause an error while reading hdfs file.
File "/data/anaconda3/envs/tfdv1.15/lib/python3.7/site-packages/apache_beam/io/tfrecordio.py", line 127, in read_record
buf = file_handle.read(buf_length_expected)
File "/data/anaconda3/envs/tfdv1.15/lib/python3.7/site-packages/apache_beam/io/filesystemio.py", line 123, in readinto
b[:len(data)] = data
ValueError: memoryview assignment: lvalue and rvalue have different structures
was:
I found that the `HdfsDownloader.get_range(self, start, end)' get range {color:#FF0000}[start, end]{color} rather than {color:#FF0000}[start, end){color}. This will cause an error while reading hdfs file.
File "/data/zhoufangyuan/anaconda3/envs/tfdv1.15/lib/python3.7/site-packages/apache_beam/io/tfrecordio.py", line 127, in read_record
buf = file_handle.read(buf_length_expected)
File "/data/zhoufangyuan/anaconda3/envs/tfdv1.15/lib/python3.7/site-packages/apache_beam/io/filesystemio.py", line 123, in readinto
b[:len(data)] = data
ValueError: memoryview assignment: lvalue and rvalue have different structures
> Hadoop Downloader Range not correct
> -----------------------------------
>
> Key: BEAM-9152
> URL: https://issues.apache.org/jira/browse/BEAM-9152
> Project: Beam
> Issue Type: Bug
> Components: io-py-tfrecord
> Reporter: Fangyuan Zhou
> Priority: Major
>
> I found that the `HdfsDownloader.get_range(self, start, end)' get range {color:#ff0000}[start, end]{color} rather than {color:#ff0000}[start, end){color}. This will cause an error while reading hdfs file.
>
> File "/data/anaconda3/envs/tfdv1.15/lib/python3.7/site-packages/apache_beam/io/tfrecordio.py", line 127, in read_record
> buf = file_handle.read(buf_length_expected)
> File "/data/anaconda3/envs/tfdv1.15/lib/python3.7/site-packages/apache_beam/io/filesystemio.py", line 123, in readinto
> b[:len(data)] = data
> ValueError: memoryview assignment: lvalue and rvalue have different structures
--
This message was sent by Atlassian Jira
(v8.3.4#803005)