You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "timsaucer (via GitHub)" <gi...@apache.org> on 2026/04/08 19:07:12 UTC
[I] Expose Spark functions [datafusion-python]
timsaucer opened a new issue, #1482:
URL: https://github.com/apache/datafusion-python/issues/1482
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
In the upstream repository we have functions that have been built for Spark compatibility. If we make these functions available in datafusion-python then we will both expand the usability of our project and also make it easier for new users to convert their projects from spark -> datafusion-python workflows.
**Describe the solution you'd like**
- Evaluate all of the features that are in the upstream repository and create a list of what features we wish to expose. I have not spent a lot of time reviewing the upstream spark code so whoever takes ownership of this issue should spend time understanding all that is available and in progress.
- Create a method to register all the spark functions with the session context.
- Expose all of the functions via the DataFrame API.
- Add test coverage
- Add documentation for the user site about how to enable the spark functions and a description of why they are not enabled by default.
**Describe alternatives you've considered**
We could release these as a separate Python package.
**Additional context**
Some of these functions replace the default functions and some are unique. For users of the DataFrame API this is likely not a problem because we can put them under a spark module, such as `datafusion.spark.functions` or `datafusion.functions.spark`. The big difference will be for those who use the SQL approach. We will probably want a function that updates the session context to register the spark functions on top of the default functions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Expose Spark functions [datafusion-python]
Posted by "djouallah (via GitHub)" <gi...@apache.org>.
djouallah commented on issue #1482:
URL: https://github.com/apache/datafusion-python/issues/1482#issuecomment-4230855225
I was referring more to the syntax, since it makes it easier for both humans and agents to write familiar code. To be clear, I don’t think matching Spark’s output is a good idea, especially since even Spark is moving toward ANSI compliance by default.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Expose Spark functions [datafusion-python]
Posted by "djouallah (via GitHub)" <gi...@apache.org>.
djouallah commented on issue #1482:
URL: https://github.com/apache/datafusion-python/issues/1482#issuecomment-4231611870
it will be nice maybe to use the same spark api syntax, even if the output does not 100 % match the expected behavior of Spark
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Expose Spark functions [datafusion-python]
Posted by "ntjohnson1 (via GitHub)" <gi...@apache.org>.
ntjohnson1 commented on issue #1482:
URL: https://github.com/apache/datafusion-python/issues/1482#issuecomment-4209497842
Big fan of this for making datafusion-python more accessible to pyspark users. Especially if this means we potentially push for mllib coverage from spark. I often wish for more operations across arrays etc element wise diff of array/list, lp norms, ...
I don't think I have bandwidth in the near term to push on it heavily though
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Expose Spark functions [datafusion-python]
Posted by "timsaucer (via GitHub)" <gi...@apache.org>.
timsaucer commented on issue #1482:
URL: https://github.com/apache/datafusion-python/issues/1482#issuecomment-4214360456
After some back and for with an agent, we came up with this plan. This does not need to be the final plan, but it looks good to me as a starting point.
# Plan: Expose `datafusion-spark` Functions to Python
## Context
The `datafusion-spark` Rust crate (in `apache/datafusion` at `datafusion/spark/`) provides ~74 scalar functions and 4 aggregate functions with Apache Spark-compatible semantics. We want to expose these in `datafusion-python` so users can opt into Spark-compatible behavior.
**The core challenge:** ~14-15 spark functions share names with existing DataFusion built-ins but have **different behavioral semantics**. This affects both SQL (where functions are resolved by name from a registry) and the DataFrame API (where functions are bound at expression construction time).
---
## Name Collisions Inventory
### High-Impact Collisions (different results for same input)
| Function | Spark Behavior | DataFusion Behavior |
|----------|---------------|-------------------|
| `concat` | Returns NULL if any input is NULL | Skips NULLs |
| `round` | HALF_UP rounding (2.5 → 3) | Banker's rounding (2.5 → 2) |
| `date_part` | 1-indexed day-of-week (Sun=1) | 0-indexed day-of-week |
| `date_trunc` | Converts to session TZ, always returns microseconds | Preserves original precision, no TZ conversion |
| `substring`/`substr` | Negative start counts from end | Different negative-index handling |
| `trunc` | Date truncation function | Math truncation (toward zero) — **completely different** |
| `abs` | Returns min value on signed overflow | Errors on overflow |
| `array_contains` | Three-valued NULL logic (NULL if element missing but array has NULLs) | Returns false |
### Moderate-Impact Collisions
| Function | Difference |
|----------|-----------|
| `avg` (aggregate) | Internal count uses i64 vs u64 |
| `ascii` | Spark accepts numeric types via implicit cast |
| `factorial` | Spark restricts to Int32, returns NULL for out-of-range |
| `length`/`char_length` | Spark always returns Int32 |
| `array_repeat` | Spark propagates NULL from count argument |
| `soundex` | Minor implementation differences |
| `array` | Different type coercion and null handling |
### Non-Colliding Functions (~60, safe to add)
`add_months`, `base64`, `bin`, `bit_count`, `bit_get`, `bitmap_*`, `bitwise_not`, `char`, `collect_list`, `collect_set`, `crc32`, `csc`, `date_add`, `date_diff`, `date_sub`, `elt`, `expm1`, `format_string`/`printf`, `from_utc_timestamp`, `hex`, `hour`, `if`, `ilike`, `json_tuple`, `last_day`, `like`, `luhn_check`, `make_dt_interval`, `make_interval`, `map_from_entries`, `minute`, `mod`, `negative`, `next_day`, `parse_url`, `pmod`, `rint`, `sec`, `second`, `sha1`/`sha`, `sha2`, `shiftleft`, `shiftright`, `shiftrightunsigned`, `shuffle`, `size`, `slice`, `space`, `spark_cast`, `str_to_map`, `time_trunc`, `to_utc_timestamp`, `try_parse_url`, `try_sum`, `try_url_decode`, `unbase64`, `unhex`, `unix_date`, `unix_micros`, `unix_millis`, `unix_seconds`, `url_decode`, `url_encode`, `width_bucket`
---
## Part 1: SQL Integration
For SQL queries, function resolution happens through the SessionContext's function registry. The `datafusion-spark` crate provides `register_all(&mut FunctionRegistry)` which registers all spark functions, **silently overwriting** any existing function with the same name.
### Approach: `ctx.enable_spark_functions()` + selective `ctx.register_spark_function(name)`
Add an explicit method on `SessionContext`, following the existing `enable_url_table()` pattern, plus a selective registration method for users who want fine-grained control:
```python
ctx = SessionContext()
# Option 1: Register ALL spark functions (overwrites collisions)
ctx.enable_spark_functions()
# Option 2: Register a single spark function by name
ctx.register_spark_function("round")
# SQL now uses Spark semantics for registered colliding names
result = ctx.sql("SELECT concat('a', NULL, 'b')").collect()
# Returns NULL (Spark) instead of 'ab' (DataFusion)
# Original DataFusion version still accessible via df_ prefix
result = ctx.sql("SELECT df_concat('a', NULL, 'b')").collect()
# Returns 'ab' (DataFusion)
```
**Implementation:**
- **Rust** (`context.rs`): Add `enable_spark_functions(&self)` that registers all spark functions, and `register_spark_function(&self, name: &str)` for selective registration. Both preserve originals under `df_` prefix before overwriting.
- **Python** (`context.py`): Thin wrappers calling the Rust methods.
**Preserving originals for SQL:** Before overwriting a colliding function, re-register the original DataFusion function under a `df_` prefix (e.g., `df_concat`, `df_round`). Users who need both semantics in one SQL query can use the prefixed name:
```sql
SELECT concat('a', NULL, 'b'), -- Spark: NULL
df_concat('a', NULL, 'b') -- DataFusion: 'ab'
```
**Selective registration** is useful for users who only want specific spark behaviors (e.g., Spark's `round` semantics) without changing all colliding functions. The selective method also registers non-colliding spark-only functions (e.g., `ctx.register_spark_function("crc32")`).
---
## Part 2: DataFrame API Integration — Approaches
For the DataFrame API, the situation is fundamentally different: Python functions like `f.round(col("x"))` bind directly to a function implementation at expression construction time (not via name lookup). So even if spark functions are registered on the context for SQL, `datafusion.functions.round` still uses the DataFusion implementation.
### Approach A: Separate `datafusion.spark.functions` Module (Recommended)
All spark functions live in a dedicated namespace:
```python
from datafusion import functions as f
from datafusion.spark import functions as sf
# DataFusion semantics
df.select(f.round(col("x"), lit(2))) # banker's rounding
# Spark semantics
df.select(sf.round(col("x"), lit(2))) # HALF_UP rounding
# Spark-only function (no collision)
df.select(sf.crc32(col("data")))
```
**Pros:**
- Zero breaking changes — `datafusion.functions` is untouched
- Explicit about which semantics you're using — no ambiguity
- Clean IDE autocompletion: `sf.` shows all spark functions, `f.` shows all DF functions
- Familiar to PySpark users (`pyspark.sql.functions` → `datafusion.spark.functions`)
- Future-proof: same pattern works for `datafusion.duckdb.functions` etc.
- All spark functions (colliding and non-colliding) live together consistently
- Matches how the Rust crate is organized (`datafusion-spark` is its own crate)
**Cons:**
- Users who want spark-only functions (no collision) must import from the spark module
- Puts "spark" in the module path — see naming discussion below
- The ~60 non-colliding spark functions are less discoverable than if they were in `datafusion.functions`
### Approach B: Merge Everything into `datafusion.functions`
Add all spark functions into the existing module. For collisions, use prefixed names (`spark_round`, `spark_concat`). Non-colliding functions are added without prefix.
```python
from datafusion import functions as f
f.round(col("x"), lit(2)) # DataFusion
f.spark_round(col("x"), lit(2)) # Spark
f.crc32(col("data")) # Spark-only, no prefix
```
**Pros:**
- Single import — everything under `f.`
- Non-colliding spark functions immediately discoverable
- No new module to learn about
**Cons:**
- Inconsistent naming: spark-only functions have no prefix, colliding ones do. Is `f.crc32` a "spark function"? Is `f.round` not?
- Pollutes autocomplete with `spark_*` variants interleaved with regular functions
- Already a 6231-line file — grows further
- `spark_` prefix is ugly and doesn't match how the functions are named in Spark itself
- No clean way to "import all spark functions" — they're mixed in
### Approach C: Config-Based Switching
A global/session config changes what `f.round(...)` resolves to:
```python
ctx = SessionContext()
ctx.set_function_variant("spark")
df.select(f.round(col("x"), lit(2))) # now uses Spark semantics
```
**Pros:**
- Minimal API change for users migrating from PySpark wholesale
- Same code works for both variants
**Cons:**
- **Dangerous implicit behavior**: same code produces different results depending on hidden state
- Makes code reviews/debugging harder — `f.round(...)` has no visible indication of which implementation
- Thread safety concerns with global state
- Mixed usage (needing both semantics in one query) is awkward
- Violates "explicit is better than implicit" (Python Zen)
- IDE/type checkers can't distinguish variants
---
## Naming Concern: "spark" in the codebase
The user raised whether putting `spark` in module paths sends the wrong message.
**Considerations:**
- The upstream Rust crate is officially named `datafusion-spark` in the Apache DataFusion project — this isn't our invention
- Users looking for Spark compatibility will search for "spark" — discoverability matters
- Alternative names (`extended`, `compat`, `extras`) are vague and don't convey what the functions actually do
- The `substrait` module already establishes a precedent for naming submodules after the external standard they integrate with
**Recommendation:** Use `spark` in the module name. It's accurate, discoverable, and matches the upstream crate name.
---
## Recommended Approach Summary
**SQL:** `ctx.enable_spark_functions()` — all-or-nothing, explicit opt-in, preserves originals under `df_` prefix.
**DataFrame API:** Approach A — `datafusion.spark.functions` as a separate module. This is the cleanest separation and matches Python idioms.
---
## Implementation Plan
### Step 1: Add `datafusion-spark` as a default-on dependency
**Files:**
- `/Cargo.toml` (workspace): Add `datafusion-spark = { version = "53" }` to `[workspace.dependencies]`
- `/crates/core/Cargo.toml`: Add `datafusion-spark = { workspace = true, optional = true }` and feature `spark = ["dep:datafusion-spark"]`. Add `"spark"` to the `default` feature list: `default = ["mimalloc", "spark"]`
- `/pyproject.toml`: Add `"spark"` to `features = ["substrait"]` → `features = ["substrait", "spark"]`
The `spark` feature is enabled by default, matching the `substrait` precedent. Users building custom wheels can opt out with `--no-default-features` if needed.
### Step 2: Rust — Add `enable_spark_functions()` and `register_spark_function()` to `PySessionContext`
**File:** `crates/core/src/context.rs`
Add methods (gated with `#[cfg(feature = "spark")]`):
```rust
#[cfg(feature = "spark")]
const SPARK_COLLISIONS: &[&str] = &[
"abs", "ascii", "avg", "concat", "date_part", "date_trunc",
"factorial", "length", "round", "soundex", "substring", "trunc",
"array_contains", "array_repeat", "array",
];
#[cfg(feature = "spark")]
fn preserve_original_if_collision(&self, name: &str) -> PyDataFusionResult<()> {
if SPARK_COLLISIONS.contains(&name) {
if let Ok(original) = self.ctx.udf(name) {
let renamed = original.as_ref().clone().with_name(format!("df_{name}"));
self.ctx.register_udf(renamed);
}
}
Ok(())
}
#[cfg(feature = "spark")]
pub fn enable_spark_functions(&self) -> PyDataFusionResult<()> {
// Save originals under df_ prefix for all colliding functions
for name in SPARK_COLLISIONS {
self.preserve_original_if_collision(name)?;
}
// Register all spark functions
for udf in datafusion_spark::all_default_scalar_functions() {
self.ctx.register_udf((*udf).clone());
}
for udaf in datafusion_spark::all_default_aggregate_functions() {
self.ctx.register_udaf((*udaf).clone());
}
Ok(())
}
#[cfg(feature = "spark")]
pub fn register_spark_function(&self, name: &str) -> PyDataFusionResult<()> {
// Look up the spark function by name across all function types
let all_scalars = datafusion_spark::all_default_scalar_functions();
if let Some(udf) = all_scalars.iter().find(|f| f.name() == name) {
self.preserve_original_if_collision(name)?;
self.ctx.register_udf((**udf).clone());
return Ok(());
}
let all_aggs = datafusion_spark::all_default_aggregate_functions();
if let Some(udaf) = all_aggs.iter().find(|f| f.name() == name) {
self.preserve_original_if_collision(name)?;
self.ctx.register_udaf((**udaf).clone());
return Ok(());
}
Err(PyValueError::new_err(
format!("No spark function named '{name}' found")
).into())
}
```
**File:** `python/datafusion/context.py` — Add Python wrappers:
```python
def enable_spark_functions(self) -> None:
"""Register all Spark-compatible functions on this context.
Colliding functions are overwritten with Spark semantics.
Original DataFusion versions remain accessible with a ``df_`` prefix
(e.g., ``df_round``, ``df_concat``).
"""
self.ctx.enable_spark_functions()
def register_spark_function(self, name: str) -> None:
"""Register a single Spark-compatible function by name.
If the function collides with an existing DataFusion built-in,
the original is preserved with a ``df_`` prefix.
Args:
name: The name of the spark function to register (e.g., "round", "crc32").
Raises:
ValueError: If no spark function with the given name exists.
"""
self.ctx.register_spark_function(name)
```
### Step 3: Rust — Create `spark_functions.rs` module
**File:** `crates/core/src/spark_functions.rs` (new)
Define `#[pyfunction]` wrappers for each spark function. For each of the ~74 scalar functions:
```rust
#[pyfunction]
fn round(value: PyExpr, decimal_places: Option<PyExpr>) -> PyExpr {
// Use the spark UDF directly to create the expression
let spark_round_udf = datafusion_spark::spark_round(); // or equivalent accessor
Expr::ScalarFunction(ScalarFunction::new_udf(spark_round_udf, args)).into()
}
```
For the 4 aggregate functions, follow the `add_builder_fns_to_aggregate` pattern from `functions.rs`.
Define `pub(crate) fn init_module(m: &Bound<'_, PyModule>) -> PyResult<()>` that registers all functions.
**Note:** The exact API for constructing expressions from `datafusion-spark` UDFs needs verification against the crate's public API. It may provide `expr_fn::round(...)` convenience functions or require `ScalarFunction::new_udf()`.
### Step 4: Rust — Register spark_functions submodule
**File:** `crates/core/src/lib.rs`
```rust
#[cfg(feature = "spark")]
pub mod spark_functions;
// In init_module:
#[cfg(feature = "spark")]
{
let spark_funcs = PyModule::new(py, "spark_functions")?;
spark_functions::init_module(&spark_funcs)?;
m.add_submodule(&spark_funcs)?;
}
```
### Step 5: Python — Create `datafusion.spark` package
**Files:**
- `python/datafusion/spark/__init__.py` — Minimal, exports `functions`
- `python/datafusion/spark/functions.py` — Python wrappers for all spark functions
Pattern for each function:
```python
from datafusion._internal import spark_functions as sf
def round(value: Expr, decimal_places: Expr | None = None) -> Expr:
"""Round using Spark HALF_UP semantics.
Unlike :py:func:`datafusion.functions.round` which uses banker's rounding,
this rounds half values away from zero (2.5 → 3).
Examples:
>>> ...
"""
return Expr(sf.round(value.expr, decimal_places.expr if decimal_places else None))
```
Each colliding function's docstring should note the behavioral difference from the DataFusion built-in.
**File:** `python/datafusion/__init__.py` — Add `spark` to imports and `__all__`.
### Step 6: Handle feature flag gracefully in Python
In `python/datafusion/spark/__init__.py`:
```python
try:
from datafusion._internal import spark_functions as _sf
SPARK_AVAILABLE = True
except ImportError:
SPARK_AVAILABLE = False
```
Functions in `spark/functions.py` raise a clear error if the feature isn't compiled.
### Step 7: Testing
**File:** `python/tests/test_spark_functions.py` (new)
- Test all spark functions for correctness
- For colliding functions, add **comparison tests** demonstrating the behavioral difference:
```python
def test_round_spark_vs_default():
# DataFusion uses banker's rounding
assert f.round(lit(2.5)).collect() == 2 # round to even
# Spark uses HALF_UP
assert sf.round(lit(2.5)).collect() == 3 # round away from zero
```
- Test `enable_spark_functions()` works for SQL queries
- Test `register_spark_function("round")` registers only that one function
- Test that `register_spark_function("nonexistent")` raises ValueError
- Test that `df_` prefixed originals are accessible after enabling spark functions
### Step 8: Documentation
- Add docs page at `docs/source/user-guide/spark-functions.rst` explaining the module
- Document all collisions with behavioral differences
- Add migration guide for PySpark users
---
## Verification Plan
1. **Build:** `maturin develop` compiles successfully (spark enabled by default)
2. **Import test:** `from datafusion.spark import functions as sf` works
3. **SQL test (enable all):** After `ctx.enable_spark_functions()`, `SELECT round(2.5)` returns 3 (Spark HALF_UP)
4. **SQL test (selective):** After `ctx.register_spark_function("round")`, `SELECT round(2.5)` returns 3 but `SELECT concat('a', NULL, 'b')` still returns `'ab'` (DataFusion)
5. **SQL test (selective error):** `ctx.register_spark_function("nonexistent")` raises ValueError
6. **DataFrame test:** `sf.round(lit(2.5))` returns 3, `f.round(lit(2.5))` returns 2
7. **Originals preserved:** After enabling, `SELECT df_round(2.5)` returns 2
8. **Non-colliding functions:** `sf.crc32(lit(b"hello"))` works
9. **Feature flag off:** Without `spark` feature, `from datafusion.spark import functions` raises clear ImportError
10. **Existing tests pass:** No regressions in `python/tests/`
---
## Key Files to Modify/Create
| File | Action |
|------|--------|
| `/Cargo.toml` | Add `datafusion-spark` to workspace deps |
| `/crates/core/Cargo.toml` | Add optional dep + `spark` feature |
| `/pyproject.toml` | Add `"spark"` to maturin features |
| `/crates/core/src/context.rs` | Add `enable_spark_functions()` |
| `/crates/core/src/spark_functions.rs` | **New** — Rust PyO3 wrappers for all spark functions |
| `/crates/core/src/lib.rs` | Register spark_functions submodule |
| `/python/datafusion/spark/__init__.py` | **New** — Package init |
| `/python/datafusion/spark/functions.py` | **New** — Python wrappers with docstrings |
| `/python/datafusion/__init__.py` | Add spark to exports |
| `/python/datafusion/context.py` | Add `enable_spark_functions()` wrapper |
| `/python/tests/test_spark_functions.py` | **New** — Tests |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Expose Spark functions [datafusion-python]
Posted by "timsaucer (via GitHub)" <gi...@apache.org>.
timsaucer commented on issue #1482:
URL: https://github.com/apache/datafusion-python/issues/1482#issuecomment-4231587691
> I was referring more to the syntax, since it makes it easier for both humans and agents to write familiar code. To be clear, I don’t think matching Spark’s output is a good idea, especially since even Spark is moving toward ANSI compliance by default.
I'm sorry, but I don't understand what you're specifically recommending.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Expose Spark functions [datafusion-python]
Posted by "timsaucer (via GitHub)" <gi...@apache.org>.
timsaucer closed issue #1482: Expose Spark functions
URL: https://github.com/apache/datafusion-python/issues/1482
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Expose Spark functions [datafusion-python]
Posted by "djouallah (via GitHub)" <gi...@apache.org>.
djouallah commented on issue #1482:
URL: https://github.com/apache/datafusion-python/issues/1482#issuecomment-4227962007
How about standardizing on spark API as the default, there is really no point in the long term to have two API ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Expose Spark functions [datafusion-python]
Posted by "timsaucer (via GitHub)" <gi...@apache.org>.
timsaucer commented on issue #1482:
URL: https://github.com/apache/datafusion-python/issues/1482#issuecomment-4229589307
> How about standardizing on spark API as the default, there is really no point in the long term to have two API ?
I don't think we'd want to standardize on spark as default, especially since some functions return differently. Then anyone who is expected PostgreSQL-like queries to work would fail / return unexpected results. Do you disagree?
It feels to me like enabling spark functions in the SQL syntax needs to be an opt-in feature.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org