[jira] [Work logged] (BEAM-11256) Dataframe writers send pickled methods to workers

["ASF GitHub Bot (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/BEAM-11256?focusedWorklogId=511044&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-511044 ]

ASF GitHub Bot logged work on BEAM-11256:
-----------------------------------------

                Author: ASF GitHub Bot
            Created on: 12/Nov/20 20:53
            Start Date: 12/Nov/20 20:53
    Worklog Time Spent: 10m 
      Work Description: TheNeuralBit commented on pull request #13322:
URL: https://github.com/apache/beam/pull/13322#issuecomment-726336504


   Run Python PreCommit


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Issue Time Tracking
-------------------

            Worklog Id:     (was: 511044)
    Remaining Estimate: 0h
            Time Spent: 10m

> Dataframe writers send pickled methods to workers
> -------------------------------------------------
>
>                 Key: BEAM-11256
>                 URL: https://issues.apache.org/jira/browse/BEAM-11256
>             Project: Beam
>          Issue Type: Bug
>          Components: sdk-py-core
>    Affects Versions: 2.25.0
>            Reporter: Brian Hulette
>            Assignee: Brian Hulette
>            Priority: P2
>          Time Spent: 10m
>  Remaining Estimate: 0h
>
> This can lead to errors when there's a mismatch between the pandas version used at construction time, and the one used in workers at execution time. For example to_csv constructed with 1.1.4, executed with 0.25.3:
> {code}
>   File "apache_beam/runners/common.py", line 1213, in apache_beam.runners.common.DoFnRunner.process
>   File "apache_beam/runners/common.py", line 742, in apache_beam.runners.common.PerWindowInvoker.invoke_process
>   File "apache_beam/runners/common.py", line 867, in apache_beam.runners.common.PerWindowInvoker._invoke_process_per_window
>   File "/usr/local/lib/python3.7/site-packages/apache_beam/io/fileio.py", line 695, in process
>     sink.write(record)
>   File "/usr/local/google/home/bhulette/working_dir/beam/sdks/python/apache_beam/dataframe/io.py", line 266, in write_record_incremental
>     self.empty = self.write_to(value[:0])
>   File "/usr/local/lib/python3.7/site-packages/apache_beam/dataframe/io.py", line 260, in write_to
>     self.writer(df, non_none_handle, *self.args, **self.kwargs)
>   File "/usr/local/google/home/bhulette/.pyenv/versions/beam/lib/python3.7/site-packages/pandas/core/generic.py", line 3168, in to_csv
>     .. versionchanged:: 0.24.0
> TypeError: __init__() got an unexpected keyword argument 'errors'
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)