You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "dariocurr (via GitHub)" <gi...@apache.org> on 2026/07/22 10:40:27 UTC

[PR] feat: allow configuring datafusion-cli history file location [datafusion]

dariocurr opened a new pull request, #23797:
URL: https://github.com/apache/datafusion/pull/23797

   ## Which issue does this PR close?
   
   - Closes #23796.
   
   ## Rationale for this change
   
   `datafusion-cli`'s interactive REPL hardcodes its rustyline history file as
   the relative path `.history`. Because the path is relative, the history file
   is created (and looked up) in whatever directory the CLI happens to be
   launched from, so every working directory silently accumulates its own
   hidden `.history` file with no way to point it somewhere else.
   
   ## What changes are included in this PR?
   
   - Add a `--history-file <PATH>` CLI flag to `datafusion-cli`, validated at
     parse time (rejects a path that is an existing directory, or whose parent
     directory doesn't exist).
   - Add a `DATAFUSION_HISTORY_FILE` environment variable as an alternative way
     to set it (e.g. for scripted/containerized setups). The `--history-file`
     flag takes precedence over the environment variable.
   - The existing default behavior (`.history`, relative to the current
     directory) is unchanged when neither is set, so this is non-breaking.
   - `exec_from_repl` now takes `history_file: Option<&Path>` instead of
     hardcoding `.history` twice; the `.history` default now lives in one place
     inside `exec_from_repl` itself.
   - Updated `datafusion-cli/examples/cli-session-context.rs` (the only other
     caller of `exec_from_repl`) and `docs/source/user-guide/cli/usage.md`
     accordingly.
   
   ## Are these changes tested?
   
   This is a small, mostly mechanical CLI argument-plumbing change with no new
   branching logic worth unit-testing in isolation (the added validator mirrors
   the existing `parse_valid_file`/`parse_valid_data_dir` helpers, which aren't
   separately unit-tested either). I manually verified end-to-end via the built
   binary:
   
   - Default (`.history` in cwd, unchanged behavior)
   - `--history-file <path>` override
   - `DATAFUSION_HISTORY_FILE=<path>` override
   - `--history-file` taking precedence over `DATAFUSION_HISTORY_FILE` when
     both are set
   - `--history-file` pointing at a non-existent parent directory fails fast
     with a clear error instead of a late rustyline error
   
   Existing `datafusion-cli` unit tests pass (`cargo test -p datafusion-cli --lib --bins`).
   
   ## Are there any user-facing changes?
   
   Yes: a new `--history-file` CLI flag and `DATAFUSION_HISTORY_FILE`
   environment variable, documented in `docs/source/user-guide/cli/usage.md`.
   No breaking changes; default behavior is unchanged.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: allow configuring datafusion-cli history file location [datafusion]

Posted by "dariocurr (via GitHub)" <gi...@apache.org>.
dariocurr commented on PR #23797:
URL: https://github.com/apache/datafusion/pull/23797#issuecomment-5131142810

   Is there any news on that?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: allow configuring datafusion-cli history file location [datafusion]

Posted by "dariocurr (via GitHub)" <gi...@apache.org>.
dariocurr commented on code in PR #23797:
URL: https://github.com/apache/datafusion/pull/23797#discussion_r3650045330


##########
datafusion-cli/src/exec.rs:
##########
@@ -129,13 +130,15 @@ pub async fn exec_from_files(
 pub async fn exec_from_repl(
     ctx: &dyn CliSessionContext,
     print_options: &mut PrintOptions,
+    history_file: Option<&Path>,

Review Comment:
   Took this suggestion and pushed an update: `exec_from_repl` now takes a `ReplOptions` struct (in its own new module, `datafusion-cli/src/repl_options.rs`, mirroring how `PrintOptions` gets its own module) instead of the raw `Option<&Path>`.
   
   I kept it as a plain, exhaustively-constructed struct rather than adding `#[non_exhaustive]` + partial-construction-via-`Default`, so it stays consistent with `PrintOptions`'s existing convention in this same crate (plain struct, named-field construction at each call site) rather than introducing a second, different pattern for "options" structs. So this doesn't fully close the door on future signature churn the way your suggestion would, but it does give us a dedicated place to add REPL-specific options later without touching `PrintOptions`, and it let me centralize both the `.history` default and the `DATAFUSION_HISTORY_FILE` env var resolution into `ReplOptions::default()` instead of duplicating that logic across callers. Thanks for the pointer to the pattern either way!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: allow configuring datafusion-cli history file location [datafusion]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #23797:
URL: https://github.com/apache/datafusion/pull/23797#issuecomment-5045410843

   <!-- semver-check-comment -->
   Thank you for opening this pull request!
   
   Reviewer note: [cargo-semver-checks](https://github.com/obi1kenobi/cargo-semver-checks) reported the current version number is not SemVer-compatible with the changes in this pull request (compared against the base branch).
   
   <details>
   <summary>Details</summary>
   
   ```
        Cloning apache/main
       Building datafusion-cli v54.1.0 (current)
          Built [ 198.130s] (current)
        Parsing datafusion-cli v54.1.0 (current)
         Parsed [   0.035s] (current)
       Building datafusion-cli v54.1.0 (baseline)
          Built [ 190.643s] (baseline)
        Parsing datafusion-cli v54.1.0 (baseline)
         Parsed [   0.033s] (baseline)
       Checking datafusion-cli v54.1.0 -> v54.1.0 (no change; assume patch)
        Checked [   0.120s] 223 checks: 222 pass, 1 fail, 0 warn, 30 skip
   
   --- failure function_parameter_count_changed: pub fn parameter count changed ---
   
   Description:
   A publicly-visible function now takes a different number of parameters.
           ref: https://doc.rust-lang.org/cargo/reference/semver.html#fn-change-arity
          impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.49.0/src/lints/function_parameter_count_changed.ron
   
   Failed in:
     datafusion_cli::exec::exec_from_repl now takes 3 parameters instead of 2, in /home/runner/work/datafusion/datafusion/datafusion-cli/src/exec.rs:130
   
        Summary semver requires new major version: 1 major and 0 minor checks failed
       Finished [ 392.162s] datafusion-cli
   ```
   
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: allow configuring datafusion-cli history file location [datafusion]

Posted by "ratmice (via GitHub)" <gi...@apache.org>.
ratmice commented on code in PR #23797:
URL: https://github.com/apache/datafusion/pull/23797#discussion_r3646501978


##########
datafusion-cli/src/exec.rs:
##########
@@ -129,13 +130,15 @@ pub async fn exec_from_files(
 pub async fn exec_from_repl(
     ctx: &dyn CliSessionContext,
     print_options: &mut PrintOptions,
+    history_file: Option<&Path>,

Review Comment:
   I should clarify that I'm just a drive-by reviewer, not really a maintainer or anything so take my suggestions for what their worth, perhaps the maintainers will have conflicting ideas.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: allow configuring datafusion-cli history file location [datafusion]

Posted by "dariocurr (via GitHub)" <gi...@apache.org>.
dariocurr commented on PR #23797:
URL: https://github.com/apache/datafusion/pull/23797#issuecomment-5426317381

   any update on this?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: allow configuring datafusion-cli history file location [datafusion]

Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #23797:
URL: https://github.com/apache/datafusion/pull/23797#issuecomment-5078267873

   ## [Codecov](https://app.codecov.io/gh/apache/datafusion/pull/23797?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
   :white_check_mark: All modified and coverable lines are covered by tests.
   :white_check_mark: Project coverage is 80.63%. Comparing base ([`dfe0f26`](https://app.codecov.io/gh/apache/datafusion/commit/dfe0f26b5f944c2ca7dfa748313f8eac00232775?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`55c1045`](https://app.codecov.io/gh/apache/datafusion/commit/55c1045b039a838405c341cae7517c1825b848e0?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
   :warning: Report is 58 commits behind head on main.
   
   <details><summary>Additional details and impacted files</summary>
   
   
   
   ```diff
   @@            Coverage Diff             @@
   ##             main   #23797      +/-   ##
   ==========================================
   - Coverage   80.71%   80.63%   -0.09%     
   ==========================================
     Files        1089     1091       +2     
     Lines      368760   370720    +1960     
     Branches   368760   370720    +1960     
   ==========================================
   + Hits       297636   298918    +1282     
   - Misses      53377    53959     +582     
   - Partials    17747    17843      +96     
   ```
   </details>
   
   [:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/datafusion/pull/23797?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).   
   :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
   <details><summary> :rocket: New features to boost your workflow: </summary>
   
   - :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
   - :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: allow configuring datafusion-cli history file location [datafusion]

Posted by "dariocurr (via GitHub)" <gi...@apache.org>.
dariocurr commented on PR #23797:
URL: https://github.com/apache/datafusion/pull/23797#issuecomment-5848604683

   any update on this?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: allow configuring datafusion-cli history file location [datafusion]

Posted by "ratmice (via GitHub)" <gi...@apache.org>.
ratmice commented on PR #23797:
URL: https://github.com/apache/datafusion/pull/23797#issuecomment-6253394001

   Don't really have anything to add, as you've addressed the comments in my initial review.
   I'm just a drive-by community member who also has some datafusion-cli PRs open.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: allow configuring datafusion-cli history file location [datafusion]

Posted by "dariocurr (via GitHub)" <gi...@apache.org>.
dariocurr commented on PR #23797:
URL: https://github.com/apache/datafusion/pull/23797#issuecomment-5045514511

   Re the cargo-semver-checks note above: this is an expected, intentional minor breaking change. `exec_from_repl` gained a required `history_file: Option<&Path>` parameter so the interactive shell's history location can be configured (this PR's whole purpose); the only in-tree caller besides `main.rs` is `datafusion-cli/examples/cli-session-context.rs`, which has been updated accordingly.
   
   Per the [API health policy](https://datafusion.apache.org/contributor-guide/api-health.html), could a committer add the `api change` label to this PR? I don't have push access to add it myself.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: allow configuring datafusion-cli history file location [datafusion]

Posted by "ratmice (via GitHub)" <gi...@apache.org>.
ratmice commented on code in PR #23797:
URL: https://github.com/apache/datafusion/pull/23797#discussion_r3646429473


##########
datafusion-cli/src/exec.rs:
##########
@@ -129,13 +130,15 @@ pub async fn exec_from_files(
 pub async fn exec_from_repl(
     ctx: &dyn CliSessionContext,
     print_options: &mut PrintOptions,
+    history_file: Option<&Path>,

Review Comment:
   I think it might be better instead of just adding an `Option<&Path>`,
   to declare some struct like perhaps
   
   ```
   #[non_exhaustive]
   #[derive(Default)]
   struct OtherOptions {
       history_path: Option<PathBuf>,
   }
   ```
   
   Or manually implement default to default to ".history" here in the struct than in the function.
   Then if any new options are needed which have default values, we don't have to change the abi again to add them, because they should just get picked up by the non_exhaustive/default combo.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: allow configuring datafusion-cli history file location [datafusion]

Posted by "ratmice (via GitHub)" <gi...@apache.org>.
ratmice commented on code in PR #23797:
URL: https://github.com/apache/datafusion/pull/23797#discussion_r3646501978


##########
datafusion-cli/src/exec.rs:
##########
@@ -129,13 +130,15 @@ pub async fn exec_from_files(
 pub async fn exec_from_repl(
     ctx: &dyn CliSessionContext,
     print_options: &mut PrintOptions,
+    history_file: Option<&Path>,

Review Comment:
   I should clarify that I'm just a drive-by reviewer, not really a maintainer or anything so take my suggestions for what it's worth.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: allow configuring datafusion-cli history file location [datafusion]

Posted by "ratmice (via GitHub)" <gi...@apache.org>.
ratmice commented on code in PR #23797:
URL: https://github.com/apache/datafusion/pull/23797#discussion_r3650143910


##########
datafusion-cli/src/exec.rs:
##########
@@ -129,13 +130,15 @@ pub async fn exec_from_files(
 pub async fn exec_from_repl(
     ctx: &dyn CliSessionContext,
     print_options: &mut PrintOptions,
+    history_file: Option<&Path>,

Review Comment:
   Good thinking to keep it consistent, we could always add `#[non_exhaustive]` to both in follow up patch if that is desired.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org