You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "alamb (via GitHub)" <gi...@apache.org> on 2023/08/18 16:29:00 UTC

[GitHub] [arrow-datafusion] alamb opened a new pull request, #7333: Add `SQLOptions` for controlling allowed SQL statements, update docs

alamb opened a new pull request, #7333:
URL: https://github.com/apache/arrow-datafusion/pull/7333

   ## Which issue does this PR close?
   Closes https://github.com/apache/arrow-datafusion/issues/7328
   
   ## Rationale for this change
   
   See https://github.com/apache/arrow-datafusion/issues/7328
   
   Basically it is hard to understand how to control what SQL statements DataFusion will allow
   
   ## What changes are included in this PR?
   1. New `SQLOptions` and `SessionContext::sql_with_options` apis
   2. Update docs
   ## Are these changes tested?
   
   <!--
   We typically require tests for all PRs in order to:
   1. Prevent the code from being accidentally broken by subsequent changes
   2. Serve as another way to document the expected behavior of the code
   
   If tests are not included in your PR, please explain why (for example, are they covered by existing tests)?
   -->
   
   ## Are there any user-facing changes?
   New `SQLOptions` and `SessionContext::sql_with_options` apis
   
   
   Everything should be backwards compatible


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow-datafusion] UlfarErl commented on a diff in pull request #7333: Add `SQLOptions` for controlling allowed SQL statements, update docs

Posted by "UlfarErl (via GitHub)" <gi...@apache.org>.
UlfarErl commented on code in PR #7333:
URL: https://github.com/apache/arrow-datafusion/pull/7333#discussion_r1298742405


##########
datafusion/core/src/execution/context.rs:
##########
@@ -2086,6 +2190,92 @@ impl SerializerRegistry for EmptySerializerRegistry {
     }
 }
 
+/// Describes which SQL statements can be run.
+///
+/// See [`SessionContext::sql_with_options`] for more details.
+#[derive(Clone, Debug, Copy)]
+pub struct SQLOptions {
+    /// See [`Self::with_allow_ddl`]
+    allow_ddl: bool,
+    /// See [`Self::with_allow_dml`]
+    allow_dml: bool,
+    /// See [`Self::with_allow_statements`]
+    allow_statements: bool,
+}
+
+impl Default for SQLOptions {
+    fn default() -> Self {
+        Self {
+            allow_ddl: true,
+            allow_dml: true,
+            allow_statements: true,
+        }
+    }
+}
+
+impl SQLOptions {
+    /// Create a new `SQLOptions` with default values
+    pub fn new() -> Self {
+        Default::default()
+    }
+
+    /// Should DML data modification commands  (e.g. `INSERT and COPY`) be run? Defaults to `true`.
+    pub fn with_allow_ddl(mut self, allow: bool) -> Self {
+        self.allow_ddl = allow;
+        self
+    }
+
+    /// Should DML data modification commands (e.g. `INSERT and COPY`) be run? Defaults to `true`
+    pub fn with_allow_dml(mut self, allow: bool) -> Self {
+        self.allow_dml = allow;
+        self
+    }
+
+    /// Should Statements such as (e.g. `SET VARIABLE and `BEGIN TRANSACTION` ...`) be run?. Defaults to `true`
+    pub fn with_allow_statements(mut self, allow: bool) -> Self {
+        self.allow_statements = allow;
+        self
+    }
+
+    /// Return an error if the [`LogicalPlan`] has any nodes that are
+    /// incompatible with this [`SQLOptions`].
+    pub fn verify_plan(&self, plan: &LogicalPlan) -> Result<()> {
+        plan.visit(&mut BadPlanVisitor::new(self))?;
+        Ok(())
+    }
+}
+
+struct BadPlanVisitor<'a> {
+    options: &'a SQLOptions,
+}
+impl<'a> BadPlanVisitor<'a> {
+    fn new(options: &'a SQLOptions) -> Self {
+        Self { options }
+    }
+}
+
+impl<'a> TreeNodeVisitor for BadPlanVisitor<'a> {
+    type N = LogicalPlan;
+
+    fn pre_visit(&mut self, node: &Self::N) -> Result<VisitRecursion> {
+        match node {
+            LogicalPlan::Ddl(ddl) if !self.options.allow_ddl => {
+                plan_err!("DDL not supported: {}", ddl.name())
+            }
+            LogicalPlan::Dml(dml) if !self.options.allow_dml => {
+                plan_err!("DML not supported: {}", dml.op)
+            }
+            LogicalPlan::Copy(_) if !self.options.allow_dml => {

Review Comment:
   Thanks so much @alamb for adding this to the core DF APIs.  The addition of the Copy operator is a great example of why it belongs here: we already have a visitor that does similar filtering, but we would probably have failed to update it to include Copy when we migrate to the latest DF version.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow-datafusion] alamb commented on a diff in pull request #7333: Add `SQLOptions` for controlling allowed SQL statements, update docs

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #7333:
URL: https://github.com/apache/arrow-datafusion/pull/7333#discussion_r1298645402


##########
datafusion/core/src/execution/context.rs:
##########
@@ -2637,42 +2827,6 @@ mod tests {
         Ok(())
     }
 
-    #[tokio::test]

Review Comment:
   I moved this into core_integration as it seemed like a test of the public API rather than an internal unit test



##########
datafusion/core/src/execution/context.rs:
##########
@@ -163,35 +167,64 @@ where
 /// * Register a custom data source that can be referenced from a SQL query.
 /// * Execution a SQL query
 ///
+/// # Example: DataFrame API
+///
 /// The following example demonstrates how to use the context to execute a query against a CSV
 /// data source using the DataFrame API:
 ///
 /// ```
 /// use datafusion::prelude::*;
-/// # use datafusion::error::Result;
+/// # use datafusion::{error::Result, assert_batches_eq};
 /// # #[tokio::main]
 /// # async fn main() -> Result<()> {
 /// let ctx = SessionContext::new();
 /// let df = ctx.read_csv("tests/data/example.csv", CsvReadOptions::new()).await?;
 /// let df = df.filter(col("a").lt_eq(col("b")))?
 ///            .aggregate(vec![col("a")], vec![min(col("b"))])?
 ///            .limit(0, Some(100))?;
-/// let results = df.collect();
+/// let results = df
+///   .collect()
+///   .await?;

Review Comment:
   since the results aren't checked it turns out this code was not actually running the plan (missing an `await` so I fixed that)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow-datafusion] alamb merged pull request #7333: Add `SQLOptions` for controlling allowed SQL statements, update docs

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb merged PR #7333:
URL: https://github.com/apache/arrow-datafusion/pull/7333


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org