You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "adriangb (via GitHub)" <gi...@apache.org> on 2024/05/15 18:05:28 UTC
[I] Missing pushdowns for pyarrow datasets [datafusion-python]
adriangb opened a new issue, #703:
URL: https://github.com/apache/datafusion-python/issues/703
Some predicates are not pushed down to pyarrow datasets correctly. I'm guessing it's just not implemented but I couldn't find any issue tracking what is and what isn't implemented.
```python
from datetime import datetime, timezone
import shutil
import datafusion
import pyarrow
import pyarrow.compute
import pyarrow.dataset
import pyarrow.fs
shutil.rmtree('data', ignore_errors=True) # proof that there's no data in the data directory
format = pyarrow.dataset.ParquetFileFormat()
filesystem = pyarrow.fs.SubTreeFileSystem('data', pyarrow.fs.LocalFileSystem())
fragments = [
# note that the partition_expression is totally wrong
format.make_fragment(
'1.parquet',
filesystem=filesystem,
partition_expression=(pyarrow.dataset.field('a') <= pyarrow.compute.scalar(1)),
)
]
dataset = pyarrow.dataset.FileSystemDataset(
fragments, pyarrow.schema([pyarrow.field('a', pyarrow.int64())]), format, filesystem
)
fragments = list(dataset.get_fragments(pyarrow.dataset.field('a') == pyarrow.scalar(2)))
assert fragments == []
ctx = datafusion.SessionContext()
ctx.register_dataset('dataset', dataset)
df = ctx.sql('SELECT * FROM dataset WHERE a = 2')
assert df.collect() == [], fragments
format = pyarrow.dataset.ParquetFileFormat()
filesystem = pyarrow.fs.SubTreeFileSystem('data', pyarrow.fs.LocalFileSystem())
fragments = [
# note that the partition_expression is totally wrong
format.make_fragment(
'1.parquet',
filesystem=filesystem,
partition_expression=(
pyarrow.dataset.field('a') == pyarrow.scalar(datetime(2000, 1, 1, tzinfo=timezone.utc), pyarrow.timestamp('ns', '+00:00'))
),
)
]
dataset = pyarrow.dataset.FileSystemDataset(
fragments, pyarrow.schema([pyarrow.field('a', pyarrow.timestamp('ns', '+00:00'))]), format, filesystem
)
fragments = list(
dataset.get_fragments(
pyarrow.dataset.field('a')
> pyarrow.scalar(datetime(2024, 1, 1, tzinfo=timezone.utc), pyarrow.timestamp('ns', '+00:00'))
)
)
assert fragments == [], fragments
ctx = datafusion.SessionContext()
ctx.register_dataset('dataset', dataset)
df = ctx.sql("SELECT * FROM dataset WHERE a = '2024-01-01T00:00:00+00:00'")
assert df.collect() == []
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Missing pushdowns for pyarrow datasets [datafusion-python]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on issue #703:
URL: https://github.com/apache/datafusion-python/issues/703#issuecomment-2113374523
I would expect that for the timestamp case the result is empty and the file is never accessed, just like the integer case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Missing pushdowns for pyarrow datasets [datafusion-python]
Posted by "Michael-J-Ward (via GitHub)" <gi...@apache.org>.
Michael-J-Ward commented on issue #703:
URL: https://github.com/apache/datafusion-python/issues/703#issuecomment-2176499997
Sorry for the delay.
Have a PR fix up. Hopefully will get this in next release.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Missing pushdowns for pyarrow datasets [datafusion-python]
Posted by "Michael-J-Ward (via GitHub)" <gi...@apache.org>.
Michael-J-Ward commented on issue #703:
URL: https://github.com/apache/datafusion-python/issues/703#issuecomment-2113354453
Hi @adriangb,
Thanks for the issue.
Could you highlight a little more clearly where the behavior in your script deviates from expectations and how you'd expect it to behave?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org