You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@beam.apache.org by "ASF GitHub Bot (JIRA)" <ji...@apache.org> on 2019/04/01 16:28:02 UTC
[jira] [Work logged] (BEAM-6952) concatenated compressed files bug
with python sdk
[ https://issues.apache.org/jira/browse/BEAM-6952?focusedWorklogId=221344&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-221344 ]
ASF GitHub Bot logged work on BEAM-6952:
----------------------------------------
Author: ASF GitHub Bot
Created on: 01/Apr/19 16:27
Start Date: 01/Apr/19 16:27
Worklog Time Spent: 10m
Work Description: dlesco commented on pull request #8187: [BEAM-6952] concatenated compressed files bug with python sdk
URL: https://github.com/apache/beam/pull/8187
The Python apache_beam.io.filesystem module has a bug handling concatenated compressed files.
This PR has two commits:
1. a new unit test that shows the problem
2. a fix to the problem.
The unit test is added to the apache_beam.io.filesystem_test module. It was added to this module because the test: apache_beam.io.textio_test.test_read_gzip_concat does not encounter the problem in the Beam 2.11 and earlier code base because the test data is too small: the data is smaller than read_size, so it goes through logic in the code that avoids the problem in the code. So, this test sets read_size smaller and test data bigger, in order to encounter the problem. It would be difficult to test in the textio_test module, because you'd need very large test data because default read_size is 16MiB, and the ReadFromText interface does not allow you to modify the read_size.
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`).
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
Post-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
--- | --- | --- | --- | --- | --- | --- | ---
Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | --- | --- | --- | ---
Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)
Python | [](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python3_Verify/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/) <br> [](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Python_PVR_Flink_Cron/lastCompletedBuild/) | --- | --- | ---
Pre-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
--- |Java | Python | Go | Website
--- | --- | --- | --- | ---
Non-portable | [](https://builds.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/)
Portable | --- | [](https://builds.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/) | --- | ---
See [.test-infra/jenkins/README](https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md) for trigger phrase, status and link of all Jenkins jobs.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Issue Time Tracking
-------------------
Worklog Id: (was: 221344)
Time Spent: 10m
Remaining Estimate: 0h
> concatenated compressed files bug with python sdk
> -------------------------------------------------
>
> Key: BEAM-6952
> URL: https://issues.apache.org/jira/browse/BEAM-6952
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core
> Affects Versions: 2.11.0
> Reporter: Daniel Lescohier
> Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> The Python apache_beam.io.filesystem module has a bug handling concatenated compressed files.
> The PR I will create has two commits:
> # a new unit test that shows the problem
> # a fix to the problem.
> The unit test is added to the apache_beam.io.filesystem_test module. It was added to this module because the test: apache_beam.io.textio_test.test_read_gzip_concat does not encounter the problem in the Beam 2.11 and earlier code base because the test data is too small: the data is smaller than read_size, so it goes through logic in the code that avoids the problem in the code. So, this test sets read_size smaller and test data bigger, in order to encounter the problem. It would be difficult to test in the textio_test module, because you'd need very large test data because default read_size is 16MiB, and the ReadFromText interface does not allow you to modify the read_size.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)