You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "oznur-synnada (via GitHub)" <gi...@apache.org> on 2025/05/13 10:41:52 UTC
[D] DISCUSSION: May 27, 2025 DataFusion Meetup in Amsterdam [datafusion]
GitHub user oznur-synnada created a discussion: DISCUSSION: May 27, 2025 DataFusion Meetup in Amsterdam
Hey all,
Thanks to the support of @ohommos, his company Adyen volunteered to be the venue of our next meetup in Amsterdam. I've opened this discussion to initiate the event organization. Please feel free to share ideas - we'd love to see who's willing to participate or take the stage (event details below).
@alamb @ozankabak @v-kessler any comments or suggestions?
- Event's name: Apache DataFusion Meet-up Amsterdam
- Event's date: May 27th 2025, from 17:30 to 21:00
- Event's location: Rokin Office - Basement event space (Max 100)
- Address: Amsterdam. Rokin 49; 1012 KK, The Netherlands
GitHub link: https://github.com/apache/datafusion/discussions/16038
----
This is an automatically sent email for github@datafusion.apache.org.
To unsubscribe, please send an email to: github-unsubscribe@datafusion.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [D] DISCUSSION: May 27, 2025 DataFusion Meetup in Amsterdam [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
GitHub user alamb added a comment to the discussion: DISCUSSION: May 27, 2025 DataFusion Meetup in Amsterdam
@Dandandan is local, perhaps he knows of some locals who might be interested
GitHub link: https://github.com/apache/datafusion/discussions/16038#discussioncomment-13132786
----
This is an automatically sent email for github@datafusion.apache.org.
To unsubscribe, please send an email to: github-unsubscribe@datafusion.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [D] DISCUSSION: May 27, 2025 DataFusion Meetup in Amsterdam [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
GitHub user alamb added a comment to the discussion: DISCUSSION: May 27, 2025 DataFusion Meetup in Amsterdam
I think this event was postponed, FWIW
GitHub link: https://github.com/apache/datafusion/discussions/16038#discussioncomment-13378346
----
This is an automatically sent email for github@datafusion.apache.org.
To unsubscribe, please send an email to: github-unsubscribe@datafusion.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [D] DISCUSSION: May 27, 2025 DataFusion Meetup in Amsterdam [datafusion]
Posted by "roeap (via GitHub)" <gi...@apache.org>.
GitHub user roeap added a comment to the discussion: DISCUSSION: May 27, 2025 DataFusion Meetup in Amsterdam
If folks are interested I am happy to see if I can come to Amsterdam and talk about it a bit more :). Currently focussing on proving some end to end integrations and get a sense for performance. I did however start to look at the best integration points in iceberg-rust to extract the information needed and think there is an avenue to at least have a PoC.
GitHub link: https://github.com/apache/datafusion/discussions/16038#discussioncomment-13153871
----
This is an automatically sent email for github@datafusion.apache.org.
To unsubscribe, please send an email to: github-unsubscribe@datafusion.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [D] DISCUSSION: May 27, 2025 DataFusion Meetup in Amsterdam [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
GitHub user alamb added a comment to the discussion: DISCUSSION: May 27, 2025 DataFusion Meetup in Amsterdam
I think those topics would be great personally
BTW In my mind the dream integration scenario is something like @roeap is actively working in https://github.com/delta-incubator/deltalake-datafusion/blob/0adfc493b351da6e937e16d0c68a5b20266a8db0/crates/datafusion/examples/simple_read.rs#L11-L33
(that is for delta, but he has some plans for iceberg too)
```rust
#[tokio::main(flavor = "multi_thread", worker_threads = 4)]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
let ctx = SessionContext::new().enable_delta_kernel(None);
let base = Url::from_directory_path(std::fs::canonicalize(PATH).unwrap()).unwrap();
let path = format!("{}column_mapping/delta", base.to_string());
// build a table and get the latest snapshot from it
let table = Table::try_from_uri(path)?;
ctx.register_delta("delta_table", table.location()).await?;
let df = ctx.sql("SELECT * FROM delta_table").await?;
let df = df.collect().await?;
print_batches(&df)?;
let log_provider = DeltaLogTableProvider::try_new(table.into())?;
ctx.register_table("delta_log", Arc::new(log_provider))?;
let df = ctx
.sql("SELECT add['path'] as path FROM delta_log WHERE add['path'] IS NOT NULL")
.await?;
let df = df.collect().await?;
print_batches(&df)?;
Ok(())
}
```
GitHub link: https://github.com/apache/datafusion/discussions/16038#discussioncomment-13149779
----
This is an automatically sent email for github@datafusion.apache.org.
To unsubscribe, please send an email to: github-unsubscribe@datafusion.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [D] DISCUSSION: May 27, 2025 DataFusion Meetup in Amsterdam [datafusion]
Posted by "roeap (via GitHub)" <gi...@apache.org>.
GitHub user roeap edited a comment on the discussion: DISCUSSION: May 27, 2025 DataFusion Meetup in Amsterdam
If folks are interested I am happy to see if I can come to Amsterdam and talk about it a bit more :). Currently focussing on proving some end to end integrations and get a sense for performance. I did however start to look at the best integration points in iceberg-rust to extract the information needed and think there is an avenue to at least have a PoC.
The general idea is to make it work and have the minimal API shown above - then work with the community to make things a bit smoother in the backend ..
GitHub link: https://github.com/apache/datafusion/discussions/16038#discussioncomment-13153871
----
This is an automatically sent email for github@datafusion.apache.org.
To unsubscribe, please send an email to: github-unsubscribe@datafusion.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [D] DISCUSSION: May 27, 2025 DataFusion Meetup in Amsterdam [datafusion]
Posted by "v-kessler (via GitHub)" <gi...@apache.org>.
GitHub user v-kessler added a comment to the discussion: DISCUSSION: May 27, 2025 DataFusion Meetup in Amsterdam
Currently, the iceberg-rust implementation is still under active development and not yet ready for use with DataFusion. I expect it may take a couple more months before it’s in a usable state. For now, the only available option is the implementation from Jan Kaul.
That said, I’d be happy to attend and present Apache Iceberg and our Lakekeeper catalog to help raise awareness and motivate the DataFusion community to engage with the Iceberg integration.
What do you think about this approach?
GitHub link: https://github.com/apache/datafusion/discussions/16038#discussioncomment-13140763
----
This is an automatically sent email for github@datafusion.apache.org.
To unsubscribe, please send an email to: github-unsubscribe@datafusion.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org