You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "hknlof (via GitHub)" <gi...@apache.org> on 2025/06/20 12:30:29 UTC

[I] Field Naming Collisions [datafusion]

hknlof opened a new issue, #16478:
URL: https://github.com/apache/datafusion/issues/16478

   ### Describe the bug
   
   Hi team,
   
   I am analyzing some CVE data. Data is being read from an NdJson and as far as I can tell, they have the same NULLABLE fields and no variances in casing.
   
   ```console
   ❯ grep AssignedTo all.json | wc -l                                                                                                             
   57029
   ❯ grep -i assignedto all.json | wc -l
   57029
   ❯ grep assignedto all.json | wc -l
   0
   ```
   
   Hence, I would expect `df.describe()` to not run into any conversion problems.
   But, I get:
   
   ```console
   Error: Custom { kind: Other, error: Execution("Schema error: No field named assignedto. Did you mean '?table?.AssignedTo'?.") }
   ```
   
   <details>
   <summary>Code and more elaborate Output</summary>
   
   ```rust
   async fn main() -> Result<(), std::io::Error> {
       let ctx = SessionContext::new();
       let df = ctx.read_json("./all.json", NdJsonReadOptions::default()).await?;
   
       for field in df.schema().fields().iter() {
           println!("{:?}", field.name());
       }
       df.describe().await?;
   
       Ok(())
   }
   ```
   **Output**:
   ```console
   "AssignedTo"
   "Bugs"
   "CRD"
   "Candidate"
   "Description"
   "DiscoveredBy"
   "Notes"
   "Patches"
   "Priority"
   "PublicDate"
   "PublicDateAtUSN"
   "References"
   "UbuntuDescription"
   "UpstreamLinks"
   Error: Custom { kind: Other, error: Execution("Schema error: No field named assignedto. Did you mean '?table?.AssignedTo'?.") }
   ```
   </details>
   
   ### To Reproduce
   
   Read a NdJson and call describe on the resulting DataFrame.
   
   ### Expected behavior
   
   Describe should work on consistently named fields in JSON.
   
   ### Additional context
   
   _No response_


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Field Naming Collisions [datafusion]

Posted by "kosiew (via GitHub)" <gi...@apache.org>.
kosiew commented on issue #16478:
URL: https://github.com/apache/datafusion/issues/16478#issuecomment-2999357989

   hi @hknlof 
   
   I could not reproduce this with main commit ` fb01049d7`
   
   all.json
   ```
   {"TaskId":1,"Title":"Design homepage","AssignedTo":"Alice"}
   {"TaskId":2,"Title":"Implement login","AssignedTo":"Bob"}
   {"TaskId":3,"Title":"Set up database","AssignedTo":"Carol"}
   {"TaskId":4,"Title":"Write API docs","AssignedTo":"Dave"}
   {"TaskId":5,"Title":"Create unit tests","AssignedTo":"Eve"}
   {"TaskId":6,"Title":"Optimize performance","AssignedTo":"Frank"}
   {"TaskId":7,"Title":"Fix bugs","AssignedTo":"Grace"}
   {"TaskId":8,"Title":"Update dependencies","AssignedTo":"Heidi"}
   {"TaskId":9,"Title":"Deploy to staging","AssignedTo":"Ivan"}
   {"TaskId":10,"Title":"Conduct code review","AssignedTo":"Judy"}
   ```
   
   ```rust
   use datafusion::error::Result;
   use datafusion::prelude::*;
   
   
   #[tokio::main]
   async fn main() -> Result<()> {
       let ctx = SessionContext::new();
       let df = ctx
           .read_json(
               "/Users/kosiew/GitHub/datafusion/datafusion-examples/examples/all.json",
               NdJsonReadOptions::default(),
           )
           .await?;
   
       for field in df.schema().fields().iter() {
           println!("{:?}", field.name());
       }
   
       df.describe().await?;
   
       Ok(())
   }
   ```
   
   ```
   ❯ c run --example read_json
       Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.48s
        Running `/Users/kosiew/.cargo/target/debug/examples/read_json`
   "AssignedTo"
   "TaskId"
   "Title"
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Field Naming Collisions [datafusion]

Posted by "hknlof (via GitHub)" <gi...@apache.org>.
hknlof closed issue #16478: Field Naming Collisions
URL: https://github.com/apache/datafusion/issues/16478


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Field Naming Collisions [datafusion]

Posted by "hknlof (via GitHub)" <gi...@apache.org>.
hknlof commented on issue #16478:
URL: https://github.com/apache/datafusion/issues/16478#issuecomment-2999702499

   Hi @kosiew, thanks for your response. Updating from `47.0.0` to `48.0.0` resolves the issue. Your example had issues with `47.0.0`, as well. Apologies, if this is a duplicate of a previous error. Closing :)
   
   ```console
   "AssignedTo"
   "TaskId"
   "Title"
   Error: Custom { kind: Other, error: Execution("Schema error: No field named assignedto. Did you mean '?table?.AssignedTo'?.") }
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org