You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "jsai28 (via GitHub)" <gi...@apache.org> on 2025/03/22 22:09:59 UTC
[PR] Add "end to end parquet reading test" for WASM [datafusion]
jsai28 opened a new pull request, #15362:
URL: https://github.com/apache/datafusion/pull/15362
## Which issue does this PR close?
- Closes #15357.
## Rationale for this change
End to end parquet reading test for WASM.
## What changes are included in this PR?
## Are these changes tested?
## Are there any user-facing changes?
No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add "end to end parquet reading test" for WASM [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #15362:
URL: https://github.com/apache/datafusion/pull/15362#discussion_r2009204444
##########
datafusion/wasmtest/src/lib.rs:
##########
@@ -185,26 +206,56 @@ mod test {
#[wasm_bindgen_test(unsupported = tokio::test)]
async fn test_parquet_write() {
- let schema = Arc::new(Schema::new(vec![
- Field::new("id", DataType::Int32, false),
- Field::new("value", DataType::Utf8, false),
- ]));
+ let (schema, batch) = create_test_data();
+ let mut buffer = Vec::new();
+ let mut writer = datafusion::parquet::arrow::ArrowWriter::try_new(
+ &mut buffer,
+ schema.clone(),
+ None,
+ )
+ .unwrap();
- let data: Vec<ArrayRef> = vec![
- Arc::new(Int32Array::from(vec![1])),
- Arc::new(StringArray::from(vec!["a"])),
- ];
+ writer.write(&batch).unwrap();
+ writer.close().unwrap();
+ }
- let batch = RecordBatch::try_new(schema.clone(), data).unwrap();
+ #[wasm_bindgen_test(unsupported = tokio::test)]
+ async fn test_parquet_read_and_write() {
+ let (schema, batch) = create_test_data();
let mut buffer = Vec::new();
let mut writer = datafusion::parquet::arrow::ArrowWriter::try_new(
&mut buffer,
schema.clone(),
None,
)
.unwrap();
-
writer.write(&batch).unwrap();
writer.close().unwrap();
+
+ let session_ctx = SessionContext::new();
+ let store = InMemory::new();
+
+ let path = Path::from("a.parquet");
+ store.put(&path, buffer.into()).await.unwrap();
+
+ let url = Url::parse("memory://").unwrap();
+ session_ctx.register_object_store(&url, Arc::new(store));
+
+ let df = session_ctx
Review Comment:
love it
##########
datafusion/wasmtest/src/lib.rs:
##########
@@ -185,26 +206,56 @@ mod test {
#[wasm_bindgen_test(unsupported = tokio::test)]
async fn test_parquet_write() {
Review Comment:
It seems to me that this test is now redundant with "test_read_and_write" but looks good to me
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add "end to end parquet reading test" for WASM [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb merged PR #15362:
URL: https://github.com/apache/datafusion/pull/15362
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add "end to end parquet reading test" for WASM [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #15362:
URL: https://github.com/apache/datafusion/pull/15362#issuecomment-2749062757
Thanks again @jsai28 and @XiangpengHao
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add "end to end parquet reading test" for WASM [datafusion]
Posted by "XiangpengHao (via GitHub)" <gi...@apache.org>.
XiangpengHao commented on PR #15362:
URL: https://github.com/apache/datafusion/pull/15362#issuecomment-2746395748
LGTM this is really nice, thank you @jsai28
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org