[jira] [Created] (BEAM-11979) Can't use ReadFromMongoDB with a datetime in filter

["Gaël (Jira)" <ji...@apache.org>]

Gaël created BEAM-11979:
---------------------------

             Summary: Can't use ReadFromMongoDB with a datetime in filter
                 Key: BEAM-11979
                 URL: https://issues.apache.org/jira/browse/BEAM-11979
             Project: Beam
          Issue Type: Bug
          Components: io-py-mongodb
    Affects Versions: 2.28.0
            Reporter: Gaël
            Assignee: Yichi Zhang


I'm having an issue while using a filter containing a datetime.

This filter works directly in pymongo but not in ReadFromMongoDB.
_BoundedMongoSource.display_data() seems to be the source of the issue.
 
Fixing the line 267:
{code:java}
res['filter'] = json.dumps(self.filter){code}
by using: 

 
{code:java}
# from bson import json_util
res['filter'] = json.dumps(self.filter, default=json_util.default){code}
Here is an example of my code : 
{code:java}
import apache_beam as beam
from apache_beam.io import ReadFromMongoDB
import datetime

inputs_query = {"created_at": { "$gte": datetime.datetime.now() } } 

with beam.Pipeline() as p:
    p_inputs = (p  | 'Read Mongo Inputs' >> ReadFromMongoDB(uri=mongo_db_uri, 
                                                            db=db, 
                                                            coll=input_coll, 
                                                            filter=inputs_query
                                                            )
                    | 'Count all Inputs' >> beam.combiners.Count.Globally()
                    | 'Print Inputs' >> beam.Map(print)
                )
{code}
I get the following error : 
{code:java}
mongomicrotest.py:19: FutureWarning: ReadFromMongoDB is experimental.
  | 'Print Inputs' >> beam.Map(print)
Traceback (most recent call last):
  File "mongomicrotest.py", line 19, in <module>
    | 'Print Inputs' >> beam.Map(print)
  File "/Users/gael/venv/venv36/lib/python3.6/site-packages/apache_beam/transforms/ptransform.py", line 1058, in __ror__
    return self.transform.__ror__(pvalueish, self.label)
  File "/Users/gael/venv/venv36/lib/python3.6/site-packages/apache_beam/transforms/ptransform.py", line 573, in __ror__
    result = p.apply(self, pvalueish, label)
  File "/Users/gael/venv/venv36/lib/python3.6/site-packages/apache_beam/pipeline.py", line 646, in apply
    return self.apply(transform, pvalueish)
  File "/Users/gael/venv/venv36/lib/python3.6/site-packages/apache_beam/pipeline.py", line 689, in apply
    pvalueish_result = self.runner.apply(transform, pvalueish, self._options)
  File "/Users/gael/venv/venv36/lib/python3.6/site-packages/apache_beam/runners/runner.py", line 188, in apply
    return m(transform, input, options)
  File "/Users/gael/venv/venv36/lib/python3.6/site-packages/apache_beam/runners/runner.py", line 218, in apply_PTransform
    return transform.expand(input)
  File "/Users/gael/venv/venv36/lib/python3.6/site-packages/apache_beam/io/mongodbio.py", line 163, in expand
    return pcoll | iobase.Read(self._mongo_source)
  File "/Users/gael/venv/venv36/lib/python3.6/site-packages/apache_beam/pvalue.py", line 141, in __or__
    return self.pipeline.apply(ptransform, self)
  File "/Users/gael/venv/venv36/lib/python3.6/site-packages/apache_beam/pipeline.py", line 689, in apply
    pvalueish_result = self.runner.apply(transform, pvalueish, self._options)
  File "/Users/gael/venv/venv36/lib/python3.6/site-packages/apache_beam/runners/runner.py", line 188, in apply
    return m(transform, input, options)
  File "/Users/gael/venv/venv36/lib/python3.6/site-packages/apache_beam/runners/runner.py", line 218, in apply_PTransform
    return transform.expand(input)
  File "/Users/gael/venv/venv36/lib/python3.6/site-packages/apache_beam/io/iobase.py", line 894, in expand
    display_data = self.source.display_data() or {}
  File "/Users/gael/venv/venv36/lib/python3.6/site-packages/apache_beam/io/mongodbio.py", line 267, in display_data
    res['filter'] = json.dumps(self.filter)
  File "/Users/gael/.pyenv/versions/3.6.11/lib/python3.6/json/__init__.py", line 231, in dumps
    return _default_encoder.encode(obj)
  File "/Users/gael/.pyenv/versions/3.6.11/lib/python3.6/json/encoder.py", line 199, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/Users/gael/.pyenv/versions/3.6.11/lib/python3.6/json/encoder.py", line 257, in iterencode
    return _iterencode(o, 0)
  File "/Users/gael/.pyenv/versions/3.6.11/lib/python3.6/json/encoder.py", line 180, in default
    o.__class__.__name__)
TypeError: Object of type 'datetime' is not JSON serializable
{code}
Maybe there is a way to correctly pass a datetime in the filter ?
This is a blocker in our company project.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)