You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "alamb (via GitHub)" <gi...@apache.org> on 2023/11/30 11:37:42 UTC
[I] Finalize SIGMOD 2024 paper (if accepted) [arrow-datafusion]
alamb opened a new issue, #8373:
URL: https://github.com/apache/arrow-datafusion/issues/8373
### Is your feature request related to a problem or challenge?
@JayjeetAtGithub @Dandandan @yjshen @ozankabak @sunchao and @viirya and submitted a paper to the [SIGMOD 2024 conference](https://2024.sigmod.org/calls_papers_important_dates.shtml), which was tracked by https://github.com/apache/arrow-datafusion/issues/6782
If our paper is accepted, this ticket tracks follow on work items to complete prior to the final copy
For the [Industrial Track](https://2024.sigmod.org/calls_industrial_track_papers.shtml) the dates are:
* All deadlines below are 11:59 PM Pacific Time.
* Paper submission: Thursday, November 30, 2023
* Notification of accept/reject: Wednesday, January 31, 2024
* Camera-ready deadline: Thursday, March 28, 2024
### Describe the solution you'd like
Here are the items I know so far:
- [ ] Update benchmark scripts: https://github.com/JayjeetAtGithub/datafusion-duckdb-benchmark/pull/25
- [ ] Rerun the benchmarks with versions of datafusion and duckdb that have been released since our initial runs (e.g. `datafusion 33` and `duckdb 0.9.2`) https://github.com/apache/arrow-datafusion/issues/6782#issuecomment-1827504952)
- [ ] Update the `Results` section with the new results, updating the query textual description if needed
- [ ] Fix the (currently) non working email for @JayjeetAtGithub (jchakraborty@influxdata.com currently does not work)
### Describe alternatives you've considered
_No response_
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Finalize SIGMOD 2024 paper (if accepted) [arrow-datafusion]
Posted by "JayjeetAtGithub (via GitHub)" <gi...@apache.org>.
JayjeetAtGithub commented on issue #8373:
URL: https://github.com/apache/arrow-datafusion/issues/8373#issuecomment-1925943047
Congratulations everyone !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Finalize SIGMOD 2024 paper (if accepted) [arrow-datafusion]
Posted by "vertexclique (via GitHub)" <gi...@apache.org>.
vertexclique commented on issue #8373:
URL: https://github.com/apache/arrow-datafusion/issues/8373#issuecomment-1883247750
I can check results and update them, please assign it to me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Finalize SIGMOD 2024 paper ~(if accepted)~ [arrow-datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #8373:
URL: https://github.com/apache/arrow-datafusion/issues/8373#issuecomment-2025133714
Here is the digital rights form that was submitted: [13731_1_1.pdf](https://github.com/apache/arrow-datafusion/files/14789392/13731_1_1.pdf)
Here is the final draft:
[DataFusion_Query_Engine___SIGMOD_2024-FINAL.pdf](https://github.com/apache/arrow-datafusion/files/14789704/DataFusion_Query_Engine___SIGMOD_2024-FINAL.pdf)
Here are the source files:
[DataFusion Query Engine - SIGMOD 2024-SOURCE.zip](https://github.com/apache/arrow-datafusion/files/14789721/DataFusion.Query.Engine.-.SIGMOD.2024-SOURCE.zip)
I am working to upload this to the CMT tool -- once I get confirmation they got it and it is accepted I'll (finally!) close this issue. Thanks all
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Finalize SIGMOD 2024 paper ~(if accepted)~ [arrow-datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #8373:
URL: https://github.com/apache/arrow-datafusion/issues/8373#issuecomment-1997452950
This morning, I started working on the first page
> Add more examples / better explanation of systems built on DataFusion (we have some good new examples I know of since -- Arroyo, Comet, and LanceDB comes to mind)
(that looks pretty much done now to me)
> The main criticism / weakness cited is that DataFusion doesn't demonstrate sufficient technical novelty other than integration of various existing ideas. I think this is a very valid point, and maybe we should re-emphasize the point more that it isn't technical novelty of any part, but the overall system.
I reworded the abstract to try and make the "not novel" point more explicitly. Here is what I came up with:
"Apache Arrow DataFusion\cite{DataFusion} is a fast, embeddable, and extensible query engine written in Rust\cite{Rust} that uses Apache Arrow\cite{Arrow} as its memory model. While the individual techniques used by DataFusion have been previously described, it differs from other industrial strength engines by providing competitive performance \textit{and} an open architecture that can be customized using over 10 major extension APIs. This flexibility has led to its use in many commercial and open source databases, machine learning pipelines, and other data-intensive systems. We anticipate that the accessibility and versatility of DataFusion, along with its competitive performance, will further enable the proliferation of high-performance custom data infrastructures tailored to specific needs."
> Please move the figure out of the first page, or to the bottom of the first page. It is distracting to read the caption of Figure 1 before the abstract.
I personally like the visual impact of the figure at the beginning so I would prefer keeping its location where it is. However, as the reviewer points out, the extended caption on the figure was duplicative / repetitive with the abstract. I thus reduced the caption to the following, which I think captures the essence with less distraction
"When building with DataFusion, system designers implement domain-specific features via extension APIs (blue), rather than re-implementing standard OLAP query engine technology (green)."
I also updated the figure with the new DataFusion logo https://github.com/apache/arrow-datafusion/issues/8788 (thanks @pinarbayata)
I think the first page is now looking quite good
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Finalize SIGMOD 2024 paper ~(if accepted)~ [arrow-datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #8373:
URL: https://github.com/apache/arrow-datafusion/issues/8373#issuecomment-2009307557
I also took a pas through the bibliography to make the style consistent
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Finalize SIGMOD 2024 paper ~(if accepted)~ [arrow-datafusion]
Posted by "gruuya (via GitHub)" <gi...@apache.org>.
gruuya commented on issue #8373:
URL: https://github.com/apache/arrow-datafusion/issues/8373#issuecomment-1933996862
> which I think could potentially be described as a postgres analytics acclerator (aka it is to postgres what comet is to spark)
Yeah, basically that's what we strive for, thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Finalize SIGMOD 2024 paper ~(if accepted)~ [arrow-datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #8373:
URL: https://github.com/apache/arrow-datafusion/issues/8373#issuecomment-2006812825
Update: I plan to complete the final outstanding item from the reviewers (adding some technical details about memory pool and related APIs) tomorrow, and then I will move on to wordsmithing / honing for a few days. I am sure we (at least I) could obsess over the content indefinitely, but I think we need to just "ship it" eventually and we are getting very close
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Finalize SIGMOD 2024 paper ~(if accepted)~ [arrow-datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #8373:
URL: https://github.com/apache/arrow-datafusion/issues/8373#issuecomment-2022565623
I am on the home stretch -- I plan to do a final proofreading of Section 8, 9, 10, and 11 and then submit the draft tomorrow
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Finalize SIGMOD 2024 paper (if accepted) [arrow-datafusion]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on issue #8373:
URL: https://github.com/apache/arrow-datafusion/issues/8373#issuecomment-1924533820
Thank you @alamb
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Finalize SIGMOD 2024 paper (if accepted) [arrow-datafusion]
Posted by "liurenjie1024 (via GitHub)" <gi...@apache.org>.
liurenjie1024 commented on issue #8373:
URL: https://github.com/apache/arrow-datafusion/issues/8373#issuecomment-1926362595
Congratulations everyone !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Finalize SIGMOD 2024 paper (if accepted) [arrow-datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #8373:
URL: https://github.com/apache/arrow-datafusion/issues/8373#issuecomment-1883174960
Here is what we submitted:
[DataFusion_Query_Engine___SIGMOD_2024.pdf](https://github.com/apache/arrow-datafusion/files/13875108/DataFusion_Query_Engine___SIGMOD_2024.pdf)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Finalize SIGMOD 2024 paper ~(if accepted)~ [arrow-datafusion]
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on issue #8373:
URL: https://github.com/apache/arrow-datafusion/issues/8373#issuecomment-2022599701
Thank you Andrew! Sorry I couldn't spent more time on the paper.
I'll try to review the draft this evening.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Finalize SIGMOD 2024 paper ~(if accepted)~ [arrow-datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #8373:
URL: https://github.com/apache/arrow-datafusion/issues/8373#issuecomment-2040798545
We needed to make a few tweaks on the final manuscript to conform to the publisher's rules
```
---------- Forwarded message ---------
From: New Form Needed <su...@scomminc.com>
Date: Fri, Apr 5, 2024 at 9:39 AM
Subject: ACM Proceedings (sigmodpods NF) - Submission Follow Up, ID No: modIP001
To: <al...@apache.org>
Cc: <yj...@apache.org>, <dh...@apache.org>, <ja...@ucsc.edu>, <oz...@apache.org>, <vi...@apache.org>, <su...@apache.org>, <Jy...@microsoft.com>, <ve...@microsoft.com>
Dear Author(s),
For the submission of "Apache Arrow DataFusion: a Fast, Embeddable, Modular Analytic Query Engine" to the proceedings and ACM Digital Library.
Please fix the following on your submission and re-submit on or before April 12th
1. A. The completed ACM rightsreview form was voided, per ACM policy and instructions provided all authors names, accents, initials, authorship order, dept/labs, affiliations, titles must match precisely on the PDF and ACM form: https://www.scomminc.com/pp/acmsig/sigmod-pods2024.htm#AO
https://www.scomminc.com/pp/acmsig/sigmod-pods2024.htm#GA
The following on your PDF and the rightsreview form do not match (see the following to see the differences). The ACM rightsreview form link will be sent to the designed contact author again.
Form -- Liang-Chi Hsieh: Apple;
Chao Sun: Apple
Files -- Chao Sun: Apple;
Liang-Chi Hsieh: Apple
B. To update the rightsreview form is best done on a computer by the designated contact author, not a mobile device; C. Update the rightsreview form data accordingly to match your final version; D. Then at the bottom click the check box, then click ''Save Paper and Author Details''; E. Then click ''Proceed to eRights Form''; F. Then complete the form entirely until the designated contact/corresponding author receives a new ACM confirmation email.
2. Authors 6 & 7. Per ACM policy, we can not allow bundled/bunched or grouped authors in a long string: https://www.scomminc.com/pp/acmsig/sigmod-pods2024.htm#GA and [www.scomminc.com/pp/acmsig/ACM-sample-sigconf-section6.pdf](http://www.scomminc.com/pp/acmsig/ACM-sample-sigconf-section6.pdf)
3. To make the ACM even page headers consistent for this year's conference publication (per the instructions: https://www.scomminc.com/pp/acmsig/sigmod.htm#Le), kindly use the following after authors and affiliation info in your .tex file:
\renewcommand{\shortauthors}{Andrew Lamb et al.}
%% No italics
4. Due to a recent change in the ACM guidelines, the $15.00 within the ACM CR block is no longer needed & not included in the ACM code provided, please comment this out on your new version.
%\acmPrice{15.00}
Then review the new version of your submission for any typos and then submit/upload all the files (required pdf, required source, etc) to update your pdf/submission on or before deadline above to the following link:
<REDACTED>
Thank you,
New Form Needed
Sheridan Communications
```
I made the requested edits (layouts of author names, remove $15, and update the short authors and submitted a new draft:
Here is the next draft
[DataFusion Query Engine - SIGMOD 2024-SOURCE-mk2.zip](https://github.com/apache/arrow-datafusion/files/14891700/DataFusion.Query.Engine.-.SIGMOD.2024-SOURCE-mk2.zip)
[DataFusion_Query_Engine___SIGMOD_2024-FINAL-mk2.pdf](https://github.com/apache/arrow-datafusion/files/14891701/DataFusion_Query_Engine___SIGMOD_2024-FINAL-mk2.pdf)
(BTW I am sharing this not because I think anyone really cares, but because I thought others might be interested in how this process works)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Finalize SIGMOD 2024 paper (if accepted) [arrow-datafusion]
Posted by "ozankabak (via GitHub)" <gi...@apache.org>.
ozankabak commented on issue #8373:
URL: https://github.com/apache/arrow-datafusion/issues/8373#issuecomment-1923583707
Thanks for the update 🚀
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Finalize SIGMOD 2024 paper (if accepted) [arrow-datafusion]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on issue #8373:
URL: https://github.com/apache/arrow-datafusion/issues/8373#issuecomment-1925916375
Cool! Congrats to all!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Finalize SIGMOD 2024 paper (if accepted) [arrow-datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #8373:
URL: https://github.com/apache/arrow-datafusion/issues/8373#issuecomment-1921052257
FWIW the notification deadline was yesterday but I have not heard anything one way or the other (and the CMT tool doesn't say one way or the other). I will email the chairs tomorrow if we haven't heard by then
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Finalize SIGMOD 2024 paper ~(if accepted)~ [arrow-datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb closed issue #8373: Finalize SIGMOD 2024 paper ~(if accepted)~
URL: https://github.com/apache/arrow-datafusion/issues/8373
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Finalize SIGMOD 2024 paper ~(if accepted)~ [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #8373:
URL: https://github.com/apache/datafusion/issues/8373#issuecomment-2082793044
Needed to tweak the title to have a `A` rather than `a`
```
---------- Forwarded message ---------
From: Tim Pollitt <su...@scomminc.com>
Date: Mon, Apr 29, 2024 at 8:47 AM
Subject: ACM Proceedings (sigmodpods TP) - Submission Follow Up, ID No: modIP001
To: <al...@apache.org>
Cc: <yj...@apache.org>, <dh...@apache.org>, <ja...@ucsc.edu>, <oz...@apache.org>, <vi...@apache.org>, <su...@apache.org>
Dear Author(s),
For the re-submission of "Apache Arrow DataFusion: A Fast, Embeddable, Modular Analytic Query Engine" to the proceedings and ACM Digital Library.
Please fix the following on your submission and re-submit on or before May 1st
1. Kindly make the ''A'' after the colon in the title a capital letter ''A''. This is a valid exception to the rule.
*Set reminders for the chairs' requested optional presentation video (see original prep instructions link https://www.scomminc.com/pp/acmsig/sigmod.htm).
Then review your PDF submission for any typos and then submit/upload all the files (required pdf, required source, optional thumbnail etc) to update your pdf/submission on or before deadline provided above to:
http://www.scomminc.com/acm/submissions/submission.cfm?grid=sigmodpods&eid=5E5C547F61070302&eid2=6702
Thank you,
```
[DataFusion Query Engine - SIGMOD 2024-FINAL-mk4.zip](https://github.com/apache/datafusion/files/15149987/DataFusion.Query.Engine.-.SIGMOD.2024-FINAL-mk4.zip)
[DataFusion_Query_Engine___SIGMOD_2024-FINAL-mk4.pdf](https://github.com/apache/datafusion/files/15149988/DataFusion_Query_Engine___SIGMOD_2024-FINAL-mk4.pdf)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Finalize SIGMOD 2024 paper ~(if accepted)~ [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #8373:
URL: https://github.com/apache/datafusion/issues/8373#issuecomment-2072745195
"third time's the charm"
<img width="626" alt="Screenshot 2024-04-23 at 11 34 59 AM" src="https://github.com/apache/datafusion/assets/490673/76350044-eabf-4407-83a5-c8cedaa6c69a">
[DataFusion Query Engine - SIGMOD 2024-FINAL-mk3.zip](https://github.com/apache/datafusion/files/15079259/DataFusion.Query.Engine.-.SIGMOD.2024-FINAL-mk3.zip)
[DataFusion_Query_Engine___SIGMOD_2024-FINAL-mk3.pdf](https://github.com/apache/datafusion/files/15079260/DataFusion_Query_Engine___SIGMOD_2024-FINAL-mk3.pdf)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Finalize SIGMOD 2024 paper ~(if accepted)~ [arrow-datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #8373:
URL: https://github.com/apache/arrow-datafusion/issues/8373#issuecomment-2016612420
I took another pass through the paper. In addition to some word smithing and whitespace engineering, I increased the size of the abstract both so the front page didn't look as empty but also to summarize the content of the paper (in addition to its conclusion / main point) to help readers decide if the paper was interesting to them
Here is the current text
> Apache Arrow DataFusion\cite{DataFusion} is a fast, embeddable, and extensible query engine written in Rust\cite{Rust} that uses Apache Arrow\cite{Arrow} as its memory model. In this paper we describe the technologies on which it is built, and how it fits in long term database implementation trends. We then enumerate the features of a modern OLAP engine, and outline optimizations required for high performance. Next we describe DataFusion's architecture and extension APIs to illustrate the interfaces used in modular query engines to integrate with the systems built on them. Finally, we demonstrate open standards and extensible design do not preclude state-of-the-art performance using a series of experimental comparisons to DuckDB\cite{DuckDB}.
>
> While the individual techniques used in DataFusion have been previously described many times, it differs from other industrial strength engines by providing competitive performance \textit{and} an open architecture that can be customized using more than 10 major extension APIs. This flexibility has led to use in many commercial and open source databases, machine learning pipelines, and other data-intensive systems. We anticipate that the accessibility and versatility of DataFusion, along with its competitive performance, will further the proliferation of high-performance custom data infrastructures tailored to specific needs assembled from modular components\cite{ComposableManifesto, ComposableCodex}.
Here is what it looks like
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Finalize SIGMOD 2024 paper ~(if accepted)~ [arrow-datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #8373:
URL: https://github.com/apache/arrow-datafusion/issues/8373#issuecomment-1928607468
Here is the reviewer feedback
Reviewer #2
Questions
1. Is the paper readable and well organized?
Definitely - very clear
2. Does this paper present a significant addition to the body of work in the area of data management research?
Definitely - a significant addition
3. Is the paper likely to have a broad impact on the data management community?
SIGMOD attendees will learn something interesting from the paper
The paper is likely to influence research in the community
4. Overall rating
Accept
5. Reviewer’s confidence
Expert
6. Strong points
1. Good presentation of the Apache Arrow DataFusion open-source project.
2. DataFusion efficiently implements operators that can be used by various other data systems, avoiding their cumbersome re-implementation.
3. Good experimental results versus DuckDB (which is an extremely well optimized embeddable analytics database).
4. I really appreciate how the DataFusion community was involved even in writing this paper. See here: https://github.com/apache/arrow-datafusion/issues/6782
7. Weak points
1. Minor: although well-engineered, the algorithms behind the supported operators are not new. DataFusion implements well-known techniques.
8. Overall comments
The paper describes the functionality of DataFusion, a very well-designed and implemented library based on Apache Arrow, which implements a variety of operators used in SQL. Similar to Arrow, DataFusion is an embeddable library (built in Rust), which can easily be embedded in broader data systems that require analytical operations. The paper includes a nice experimental evaluation versus DuckDB, demonstrating good results.
Reviewer #5
Questions
1. Is the paper readable and well organized?
Definitely - very clear
2. Does this paper present a significant addition to the body of work in the area of data management research?
Mostly - the contributions are above the bar
3. Is the paper likely to have a broad impact on the data management community?
SIGMOD attendees will learn something interesting from the paper
4. Overall rating
Accept
5. Reviewer’s confidence
Expert
6. Strong points
1. The paper is well written.
2. Extensive evaluation using 3 popular benchmarks.
3. An active community-driven project.
7. Weak points
1. The DataFusion project is a combination and integration of other well-known components/systems; as such, its overall technical novelty is limited.
2. The experimental evaluation didn't compare against many other popular OLAP systems in the field.
3. The support for complex analytical queries (e.g., multi-way join as those found in TPC-DS) is limited.
8. Overall comments
This paper is well written and the DataFusion project has a good momentum in the community. The idea of building an OLAP engine using a decoupled, component-based approach is interesting (versus tightly coupled designs). The paper has described most elements in DataFusion, but didn't offer enough details to demonstrate sufficient technical novelty (that goes beyond integration of various existing componentshe). How to better suit the cloud environment where most OLAP engines are running on nowadays is also not discussed in the paper.
Reviewer #7
Questions
1. Is the paper readable and well organized?
Mostly - the presentation has minor issues, but is acceptable
2. Does this paper present a significant addition to the body of work in the area of data management research?
Mostly - the contributions are above the bar
3. Is the paper likely to have a broad impact on the data management community?
SIGMOD attendees will learn something interesting from the paper
4. Overall rating
Reject
5. Reviewer’s confidence
Knowledgeable
6. Strong points
- Presents the technologies that power DataFusion and provides motivating use cases for using DataFusion, making a compelling argument over reuse in analytic systems using commodity OLAP engines and a paradigm shift in that direction.
- Provides extensive evaluation of DataFusion's performance.
- Presents DataFusion's architecture, extension APIs and features.
7. Weak points
- One of the main claims of the paper is that DataFusion is catalyzing the development of new data systems. The presentation and the evaluation of the paper would benefit from elaborating further on this claim.
- Section 5.1 Engine overview and Figure 2 need to be more extensive to be able to follow the rest of Section 5.
- The LLVM analogy distracts from the paper.
- The paper claims in section 7.4 that "..DataFusion can be customized for these different environments using the MemoryPool trait to control memory allocations, the DiskManager trait for managing temporary files (if any), and a CacheManager for caching information such as directory contents and per-file metadata.". More technical details on this topic would be helpful.
- The "Single Core Efficiency" section could benefit from running TPC-H across multiple threads and configuration settings. The authors mention a caveat of restricting duckDB performance for some benchmarks by using single thread.
8. Overall comments
I would like to thank the authors for their work. Please find some additional minor comments below:
- Please move the figure out of the first page, or to the bottom of the first page. It is distracting to read the caption of Figure 1 before the abstract.
- Please update the axes in Figure 7 to be legible.
- One of the main topics of the paper is that DataFusion catalyzes the development of new data systems. Evaluation in that direction would help support the claims in the paper further. One related angle could be the ease of developing systems (applications) on top of DataFusion, potentially including the overhead in terms of lines of code or engineering hours in developing a simple system/application with DataFusion and using a different stack or being customly built. Similarly, performance evaluation of systems relying on DataFusion could help in this direction as well.
- Similarly, the content of the paper would benefit from doing a deep dive into the query engine and a limited set of features based on how they are used by systems developed on DataFusion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Finalize SIGMOD 2024 paper ~(if accepted)~ [arrow-datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #8373:
URL: https://github.com/apache/arrow-datafusion/issues/8373#issuecomment-1933968867
Also, we maybe can add some other users like [Seafowl](https://seafowl.io/) (now part of enterprise DB), which I think could potentially be described as a postgres analytics acclerator (aka it is to postgres what comet is to spark). Maybe @gruuya can correct me if I got that wrong
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Finalize SIGMOD 2024 paper (if accepted) [arrow-datafusion]
Posted by "matthewmturner (via GitHub)" <gi...@apache.org>.
matthewmturner commented on issue #8373:
URL: https://github.com/apache/arrow-datafusion/issues/8373#issuecomment-1926071915
Great news! Congratulations to all involved!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Finalize SIGMOD 2024 paper ~(if accepted)~ [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #8373:
URL: https://github.com/apache/datafusion/issues/8373#issuecomment-2085360550
> > Needed to tweak the title to have a `A` rather than `a`
>
> I remember we have discussed this `A`/`a` issue before in emails. We got the answer now. 😄
>
> Thanks for dealing with the tweak.
No problem -- it seems like this process is still quite manual 😆 -- hopefully we have it all right now 🤞
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Finalize SIGMOD 2024 paper ~(if accepted)~ [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #8373:
URL: https://github.com/apache/datafusion/issues/8373#issuecomment-2088935249
I think it is finally accepted 🎉
```
---------- Forwarded message ---------
From: Tim Pollitt <su...@scomminc.com>
Date: Wed, May 1, 2024 at 2:44 PM
Subject: ACM Proceedings (sigmodpods TP) - Submission Follow Up, ID No: modIP001
To: <al...@apache.org>
Cc: <yj...@apache.org>, <dh...@apache.org>, <ja...@ucsc.edu>, <oz...@apache.org>, <vi...@apache.org>, <su...@apache.org>
Dear Author(s),
For the re-submission of "Apache Arrow DataFusion: A Fast, Embeddable, Modular Analytic Query Engine" to the proceedings and ACM Digital Library.
See below **** for info related to your final pdf.
In addition, authors are requested by the chairs to submit an optional presentation video.
(1) For more info, see instructions for details, and inquiry link to chairs: https://www.scomminc.com/pp/acmsig/sigmod-video.htm by May 6th.
(2) You will need the DOI assigned to your submission: 10.1145/3626246.3653368
(3) Video descriptions over the 1,024 ACM character count will be deleted/removed by ACM.
*************
We received notice of your completed ACM form, so everything appears to be in order with this submission, and will be utilized for the ACM DL publication. No revisions are needed, nor will be accepted. This submission is being moved onto the next stages of production (but kindly note the signed ACM forms still go through a double-check by ACM for various permissions).
Thank you,
Tim Pollitt
Sheridan Communications
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Finalize SIGMOD 2024 paper ~(if accepted)~ [arrow-datafusion]
Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao commented on issue #8373:
URL: https://github.com/apache/arrow-datafusion/issues/8373#issuecomment-1932681299
To update the draft I'm assuming we can just reuse the same overleaf project? we'd be happy to touch a bit more on the Comet side, and update the sentence 😂
> DataFusion is used by several Spark native runtimes, including Blaze[ 10] and at least one project that is not yet open-source.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Finalize SIGMOD 2024 paper ~(if accepted)~ [arrow-datafusion]
Posted by "JayjeetAtGithub (via GitHub)" <gi...@apache.org>.
JayjeetAtGithub commented on issue #8373:
URL: https://github.com/apache/arrow-datafusion/issues/8373#issuecomment-1934902253
@alamb I updated my affiliation and email to that of UC Santa Cruz, my university.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Finalize SIGMOD 2024 paper ~(if accepted)~ [arrow-datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #8373:
URL: https://github.com/apache/arrow-datafusion/issues/8373#issuecomment-2002411631
> "The LLVM analogy distracts from the paper." - I happen to like this analogy (and I think @ozankabak does too), but maybe we can make this section shorter / more concise.
I reduced the space taken by the LLVM section by distilling, while still leaving the section and content in place. I think we could probably reduce it a bit more if anyone wanted to wordsmith some more
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Finalize SIGMOD 2024 paper ~(if accepted)~ [arrow-datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #8373:
URL: https://github.com/apache/arrow-datafusion/issues/8373#issuecomment-2003639538
> ?The authors mention a caveat of restricting duckDB performance for some benchmarks by using single thread." -- We should make what we measured clearer
I moved the discussion of restricting per-core performance into the "Single Core Efficineny" sub section, out of the experimental evaluation introduction, which I think makes it clearer the restriction was only used for the single-core efficiency experiments
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Finalize SIGMOD 2024 paper ~(if accepted)~ [arrow-datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #8373:
URL: https://github.com/apache/arrow-datafusion/issues/8373#issuecomment-2015863171
In order to keep consistent emails I changed the author emails to `@apache.org` for committers:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Finalize SIGMOD 2024 paper ~(if accepted)~ [arrow-datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #8373:
URL: https://github.com/apache/arrow-datafusion/issues/8373#issuecomment-1951826679
An update here: I plan to take a pass through the draft the week of March 4 and implement the bulk of any feedback that was not yet implemented. After that week I'll likely take a few proofreading passes, but I don't expect to do any major revisions
I also don't plan to rerun benchmarks again due to lack of time. While the benchmark runs themselves are nicely automated thanks to @JayjeetAtGithub, analyizing the results takes significant time and research.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Finalize SIGMOD 2024 paper ~(if accepted)~ [arrow-datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #8373:
URL: https://github.com/apache/arrow-datafusion/issues/8373#issuecomment-1933952700
@JayjeetAtGithub is there any chance you can update your email address to one that works (rather than the influxdata one that does not)?
Also, it would be great if someone could work on cleaning up the bibliography.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Finalize SIGMOD 2024 paper (if accepted) [arrow-datafusion]
Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao commented on issue #8373:
URL: https://github.com/apache/arrow-datafusion/issues/8373#issuecomment-1925925325
This is great news! congrats all!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Finalize SIGMOD 2024 paper ~(if accepted)~ [arrow-datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #8373:
URL: https://github.com/apache/arrow-datafusion/issues/8373#issuecomment-2025193938
<img width="717" alt="Screenshot 2024-03-28 at 9 30 05 AM" src="https://github.com/apache/arrow-datafusion/assets/490673/4138ef54-ccd5-4945-a6d1-8a8f086cca2a">
😅
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Finalize SIGMOD 2024 paper ~(if accepted)~ [datafusion]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on issue #8373:
URL: https://github.com/apache/datafusion/issues/8373#issuecomment-2083163913
> Needed to tweak the title to have a `A` rather than `a`
I remember we have discussed this `A`/`a` issue before in emails. We got the answer now. 😄
Thanks for dealing with the tweak.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Finalize SIGMOD 2024 paper ~(if accepted)~ [arrow-datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #8373:
URL: https://github.com/apache/arrow-datafusion/issues/8373#issuecomment-2002433708
> "Section 5.1 Engine overview and Figure 2 need to be more extensive to be able to follow the rest of Section 5."
I took a pass through the start of section 5 and figure 2 as well. I moved the content of the Figure 2 caption into the first paragraph of Section 5 as a bullet point, and tried to better align it to the section headers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Finalize SIGMOD 2024 paper ~(if accepted)~ [arrow-datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #8373:
URL: https://github.com/apache/arrow-datafusion/issues/8373#issuecomment-2006755328
> work in "One related angle could be the ease of developing systems (applications) on top of DataFusion, potentially including the overhead in terms of lines of code or engineering hours in developing a simple system/application with DataFusion and using a different stack or being customly built. Similarly, performance evaluation of systems relying on DataFusion could help in this direction as well."
> Similarly, the content of the paper would benefit from doing a deep dive into the query engine and a limited set of features based on how they are used by systems developed on DataFusion.
I added these (great) ideas to the "future work" section which I think would make very excellent papers
<img width="328" alt="Screenshot 2024-03-19 at 6 34 30 AM" src="https://github.com/apache/arrow-datafusion/assets/490673/5828b787-d75d-437f-a30b-a33d86ba4c9b">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Finalize SIGMOD 2024 paper ~(if accepted)~ [arrow-datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #8373:
URL: https://github.com/apache/arrow-datafusion/issues/8373#issuecomment-2016612462
I made it through Section 6 today, and I plan to start at Section 7 tomorrow for a final read through / polish.
Starting Monday I just plan to do whitespace engineering / proofreading
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Finalize SIGMOD 2024 paper (if accepted) [arrow-datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #8373:
URL: https://github.com/apache/arrow-datafusion/issues/8373#issuecomment-1923582382
I emailed the chairs today and they said the notification will be delayed a few days. Will post updates here as I have them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Finalize SIGMOD 2024 paper ~(if accepted)~ [arrow-datafusion]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on issue #8373:
URL: https://github.com/apache/arrow-datafusion/issues/8373#issuecomment-1932694963
Yea, as Comet now is open sourced, we can explicitly mention the project (with project link) and more details about it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Finalize SIGMOD 2024 paper ~(if accepted)~ [arrow-datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #8373:
URL: https://github.com/apache/arrow-datafusion/issues/8373#issuecomment-2040802101
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Finalize SIGMOD 2024 paper ~(if accepted)~ [arrow-datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #8373:
URL: https://github.com/apache/arrow-datafusion/issues/8373#issuecomment-2009335704
Here is an email about ACM formatting guidelines in case anyone following along is interested in how this process works
-----
Dear Author,
Please remember to submit "Apache Arrow DataFusion: A Fast, Embeddable, Modular Analytic Query Engine," for publication in the proceedings and ACM DL.
********************************************************
SIGMOD'24 Authors. You are receiving this initial email request to format and submit your final version (paper and promo video) per the information below (on or before EoD April 12th)
1. IMPORTANT: Only the first designated contact/corresponding author will be sent the ACM rightsreview form email in about 24 hours.
2. Review https://www.scomminc.com/pp/acmsig/sigmod-pods2024.htm carefully as ACM is imposing all data and author fields must match on the PDF and ACM rightsreview form (stricter compliance to the template has been requested by ACM).
Kindly coordinate the following with your co-authors to avoid having to repeat the process. Any discrepancies will void a completed form.
A. Title in Initial Caps;
B. How each authors' full name with middle initials or names should appear (on both PDF & ACM form) & update the form to include all author names, dept/lab, affiliations, and email addresses;
C. How your ''dept/labs'' and ''institution/corp affiliations'' should be listed consistently (on both PDF & ACM form);
D. Any funding or grants to be acknowledged.
E. Upon completing the rightsreview form, the contact author will receive the ACM copyright block, conference data, ISBN, and DOI to include on the first page with the other required ACM sections per the ACM SIG (sigconf style) template https://www.scomminc.com/pp/acmsig/sigmod.htm
3. Submit your final version only (pdf, source file, promo video, & optional thumbnail) to the link below on or before April 12th. Material submitted before the deadline will be processed and can be finalized for the ACM DL.
********************************************************
The following is a direct link to submit based on your unique submission ID: (REMOVED)
Thank you,
Sheridan Communications
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Finalize SIGMOD 2024 paper ~(if accepted)~ [arrow-datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #8373:
URL: https://github.com/apache/arrow-datafusion/issues/8373#issuecomment-2002486534
> How to better suit the cloud environment where most OLAP engines are running on nowadays is also not discussed in the paper.
I took a shot at rewording the introduction to mention cloud databases (mainly as an example of the forces that drive the need for new databases with OLAP engines)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Finalize SIGMOD 2024 paper ~(if accepted)~ [arrow-datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #8373:
URL: https://github.com/apache/arrow-datafusion/issues/8373#issuecomment-2022673189
No problem! I think we are all quite busy -- I also want to submit the paper so I can let it go (my OCD tendencies would be to edit it indefinitely, which is not good :) )
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Finalize SIGMOD 2024 paper ~(if accepted)~ [arrow-datafusion]
Posted by "ozankabak (via GitHub)" <gi...@apache.org>.
ozankabak commented on issue #8373:
URL: https://github.com/apache/arrow-datafusion/issues/8373#issuecomment-1933802859
> "The LLVM analogy distracts from the paper." - I happen to like this analogy (and I think @ozankabak does too), but maybe we can make this section shorter / more concise.
I think this analogy is very useful -- let's keep it. In my experience it also resonates with technical folks very well. Since this feedback seems like an outlier in terms of reception, I suggest we improve other aspects of the paper.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Finalize SIGMOD 2024 paper ~(if accepted)~ [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #8373:
URL: https://github.com/apache/datafusion/issues/8373#issuecomment-2072704562
And apparently I still didn't get it entirely correct:
```
---------- Forwarded message ---------
From: Tim Pollitt <su...@scomminc.com>
Date: Tue, Apr 23, 2024 at 8:46 AM
Subject: ACM Proceedings (sigmodpods TP) - Submission Follow Up, ID No: modIP001
To: <al...@apache.org>
Cc: <yj...@apache.org>, <dh...@apache.org>, <ja...@ucsc.edu>, <oz...@apache.org>, <vi...@apache.org>, <su...@apache.org>
Dear Author(s),
For the re-submission of "Apache Arrow DataFusion: a Fast, Embeddable, Modular Analytic Query Engine" to the proceedings and ACM Digital Library.
Please fix the following on your submission and re-submit on or before April 25th
1. Please update the author order on your PDF to match what was listed on your ACM Rightsreview form:
Andrew Lamb, Yijie Shen, Daniël Heres, Jayjeet Chakraborty, Mehmet Ozan Kabak, Liang-Chi Hsieh, & Chao Sun
*Kindly note that per ACM policy, this data must match in both locations.
**Set reminders for the chairs' requested optional presentation video (see original prep instructions link https://www.scomminc.com/pp/acmsig/sigmod.htm).
Then review your PDF submission for any typos and then submit/upload all the files (required pdf, required source, optional thumbnail etc) to update your pdf/submission on or before deadline provided above to:
<LINK>
Thank you,
Tim Pollitt
Sheridan Communications
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Finalize SIGMOD 2024 paper ~(if accepted)~ [arrow-datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #8373:
URL: https://github.com/apache/arrow-datafusion/issues/8373#issuecomment-2016931577
Ok, I did a final read / wordsmith / whitespace engineering on the last few sections. I will plan to do a proofreading pass or two over the next few days, but don't plan to change anything unless there is some grammatical issue.
The current copy of the paper is here:
[DataFusion_Query_Engine___SIGMOD_2024 (1).pdf](https://github.com/apache/arrow-datafusion/files/14736731/DataFusion_Query_Engine___SIGMOD_2024.1.pdf)
(getting very close)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Finalize SIGMOD 2024 paper (if accepted) [arrow-datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #8373:
URL: https://github.com/apache/arrow-datafusion/issues/8373#issuecomment-1925913783
The paper was accepted to SIGMOD! 🎉
I'll spend some time reviewing the comments later this week and we can organize action items for the final draft
```
From: Microsoft CMT <em...@msr-cmt.org>
Date: Sun, Feb 4, 2024 at 11:28 AM
Subject: SIGMOD 2024 Industry Track decision for Paper 1
To: Andrew Lamb
Dear Andrew Lamb,
It is our great pleasure to inform you that your paper #1 "Apache Arrow DataFusion: A Fast, Embeddable, Modular Analytic Query Engine" has been Accepted to the conference. Congratulations!
The papers will be presented at SIGMOD 2024, in Santiago, Chile in June, so please plan for at least one author of the paper to attend the conference. This might require a visa, so please consult the following page at your earliest convenience https://2024.sigmod.org/visa_chile.shtml .
We hope that you will find the reviews helpful in revising accordingly the camera-ready version of the manuscript. Please note that the papers will appear in PACMMOD, like the Research Track papers of SIGMOD 2024. The formatting guidelines for the camera-ready papers are available at: https://dl.acm.org/journal/pacmmod/author-guidelines#formatting , under the "Length and Format for Camera-Ready Papers" section.
Congratulations again and looking forward to seeing you at SIGMOD 2024!
Danica and Ippokratis.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Finalize SIGMOD 2024 paper ~(if accepted)~ [arrow-datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #8373:
URL: https://github.com/apache/arrow-datafusion/issues/8373#issuecomment-1931982085
It appears we have about 2 months to complete the final draft
> Camera-ready deadline: Thursday, March 28, 2024
Here is a summary of action items based on the reviewer feeback above
- [ ] Add more examples / better explanation of systems built on DataFusion (we have some good new examples I know of since -- Arroyo, Comet, and LanceDB comes to mind)
- [ ] "Section 5.1 Engine overview and Figure 2 need to be more extensive to be able to follow the rest of Section 5."
- [ ] "The LLVM analogy distracts from the paper." - I happen to like this analogy (and I think @ozankabak does too), but maybe we can make this section shorter / more concise.
- [ ] Extend section 7.4's description with technical details about `MemoryPool`, `DiskManager`, and `CacheManager` for caching information such as directory contents and per-file metadata."
- [ ] ?The authors mention a caveat of restricting duckDB performance for some benchmarks by using single thread." -- We should make what we measured clearer
Here are some other notes I have
The main criticism / weakness cited is that DataFusion doesn't demonstrate sufficient technical novelty other than integration of various existing ideas. I think this is a very valid point, and maybe we should re-emphasize the point more that it isn't technical novelty of any part, but the overall system.
> How to better suit the cloud environment where most OLAP engines are running on nowadays is also not discussed in the paper.
This is a good point that would be good to work in
> Similarly, the content of the paper would benefit from doing a deep dive into the query engine and a limited set of features based on how they are used by systems developed on DataFusion.
I agree this would be an interesting point, but given that we are already at the 12 page limit I am not sure how to do so in this particular paper. Maybe these would make good follow on papers or blog posts (@appletreeisyellow and I could potentially write one on how InfluxDB uses PruningPredicates 🤔 )
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Finalize SIGMOD 2024 paper ~(if accepted)~ [arrow-datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #8373:
URL: https://github.com/apache/arrow-datafusion/issues/8373#issuecomment-1933207828
> To update the draft I'm assuming we can just reuse the same overleaf project? we'd be happy to touch a bit more on the Comet side, and update the sentence 😂
Yes, please, let's use the same overleaf project
> Yea, as Comet now is open sourced, we can explicitly mention the project (with project link) and more details about it.
Yes please that would be great -- and it will also address some of the reviewer feedback suggesting more details on usecases
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Finalize SIGMOD 2024 paper ~(if accepted)~ [arrow-datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #8373:
URL: https://github.com/apache/arrow-datafusion/issues/8373#issuecomment-1997451436
I started getting back into paper writing mode today. I plan to take a linear pass through the sections over the next few days addressing reviewer feedback.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Finalize SIGMOD 2024 paper ~(if accepted)~ [arrow-datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #8373:
URL: https://github.com/apache/arrow-datafusion/issues/8373#issuecomment-1997695571
I plan to work on the other feedback items noted in https://github.com/apache/arrow-datafusion/issues/8373#issuecomment-1931982085 over the next few days and will keep that checklist updated with my progress
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Finalize SIGMOD 2024 paper ~(if accepted)~ [arrow-datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #8373:
URL: https://github.com/apache/arrow-datafusion/issues/8373#issuecomment-2003809769
I also make the labels and series colors consistent in the scalability chart: https://github.com/JayjeetAtGithub/datafusion-duckdb-benchmark/pull/26
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Finalize SIGMOD 2024 paper ~(if accepted)~ [arrow-datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #8373:
URL: https://github.com/apache/arrow-datafusion/issues/8373#issuecomment-2009321040
TLDR -- please make any updates you would like / need by this weekend (Sunday March 24, 2024).
Also, fellow authors, please ensure your names / affiliations are correct on the paper as I will submit that as part of the paper as well
Update:
1. The current copy of the paper (with all edits) can be downloaded here: [DataFusion_Query_Engine___SIGMOD_2024 (9).pdf](https://github.com/apache/arrow-datafusion/files/14665354/DataFusion_Query_Engine___SIGMOD_2024.9.pdf)
2. I received an email (below) which suggests the deadline is actually April 12.
3. I still plan to submit a draft / final manuscript on March 28.
4. I plan to take one lass editing pass through the paper this week/weekend to tighten up the language and make it more concise
5. Starting next Monday, March 23 I plan to just read / update for typos and required formatting for submission on the 28th
> Dear Andrew Lamb,
>
> We are writing to you regarding the camera ready submission process. Thank you for your patience as we worked with ACM SIGMOD officers and Sheridan Communications, our publishers, to agree on the process and formatting. We apologize for the previously communicated change in formatting - upon extensive discussion we agreed that camera ready versions of the paper should use the same 2-column format as the initial submissions.
>
> By now you should have received initial email from Sheridan including the information regarding how to submit the camera-ready publication and how to review and edit the paper metadata. You will also be receiving the ACM rightsreview form soon. The deadline for submitting the camera-ready manuscript and the ACM rightsreview form is April 12th. Please let us know if you didn't receive the email from Sheridan or have any other concerns about the process.
>
> We apologize for the confusion and are looking forward to seeing you at SIGMOD 2024!
>
> Danica and Ippokratis.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Finalize SIGMOD 2024 paper ~(if accepted)~ [arrow-datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #8373:
URL: https://github.com/apache/arrow-datafusion/issues/8373#issuecomment-2009247796
Update: @viirya cleaned up some of the language, and I tweaked the images to make the internal margins even:
<img width="593" alt="Screenshot 2024-03-20 at 6 03 07 AM" src="https://github.com/apache/arrow-datafusion/assets/490673/8c5d58c4-7b75-4bd6-80c1-f190696022b5">
<img width="609" alt="Screenshot 2024-03-20 at 6 03 37 AM" src="https://github.com/apache/arrow-datafusion/assets/490673/060d6d11-b5ab-40e5-8357-5cc8d4c82d01">
I also received a bunch of instructions from the ACM, so changed the title so the "A" wasn't capitalized
<img width="578" alt="Screenshot 2024-03-20 at 6 04 18 AM" src="https://github.com/apache/arrow-datafusion/assets/490673/57d80fb7-e693-4620-b183-c7eac594c8cc">
> Extend section 7.4's description with technical details about MemoryPool, DiskManager, and CacheManager for caching information such as directory contents and per-file metadata."
I reworked this section with more details / references:
<img width="319" alt="Screenshot 2024-03-20 at 6 40 45 AM" src="https://github.com/apache/arrow-datafusion/assets/490673/655d0ab9-e91c-4436-a3da-8c702d31469c">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org