[GitHub] [arrow-datafusion] alamb commented on a diff in pull request #7005: Add support for ClickBench in bench.sh

["alamb (via GitHub)" <gi...@apache.org>]

alamb commented on code in PR #7005:
URL: https://github.com/apache/arrow-datafusion/pull/7005#discussion_r1269840443


##########
benchmarks/bench.sh:
##########
@@ -318,6 +338,62 @@ run_sort() {
     $CARGO_COMMAND --bin parquet -- sort --path "${DATA_DIR}" --scale-factor 1.0 --iterations 10 -o ${RESULTS_FILE}
 }
 
+
+# Downloads the single file hits.parquet ClickBench datasets from
+# https://github.com/ClickHouse/ClickBench/tree/main#data-loading
+#
+# Creates data in $DATA_DIR/hits.parquet
+data_clickbench_1() {
+    pushd "${DATA_DIR}" > /dev/null
+
+    # Avoid downloading if it already exists and is the right size
+    OUTPUT_SIZE=`wc -c hits.parquet  2>/dev/null  | awk '{print $1}' || true`
+    echo -n "Checking hits.parquet..."
+    if test "${OUTPUT_SIZE}" = "14779976446"; then
+        echo -n "... found ${OUTPUT_SIZE} bytes ..."
+    else
+        URL="https://datasets.clickhouse.com/hits_compatible/hits.parquet"
+        echo -n "... downloading ${URL} (14GB) ... "

Review Comment:
   Yes, you are right, they should both by bytes (KB was left over from a previous implementation). Nice eyes



##########
benchmarks/bench.sh:
##########
@@ -318,6 +338,62 @@ run_sort() {
     $CARGO_COMMAND --bin parquet -- sort --path "${DATA_DIR}" --scale-factor 1.0 --iterations 10 -o ${RESULTS_FILE}
 }
 
+
+# Downloads the single file hits.parquet ClickBench datasets from
+# https://github.com/ClickHouse/ClickBench/tree/main#data-loading
+#
+# Creates data in $DATA_DIR/hits.parquet
+data_clickbench_1() {
+    pushd "${DATA_DIR}" > /dev/null
+
+    # Avoid downloading if it already exists and is the right size
+    OUTPUT_SIZE=`wc -c hits.parquet  2>/dev/null  | awk '{print $1}' || true`
+    echo -n "Checking hits.parquet..."
+    if test "${OUTPUT_SIZE}" = "14779976446"; then
+        echo -n "... found ${OUTPUT_SIZE} bytes ..."
+    else
+        URL="https://datasets.clickhouse.com/hits_compatible/hits.parquet"
+        echo -n "... downloading ${URL} (14GB) ... "
+        wget --continue ${URL}
+    fi
+    echo " Done"
+    popd > /dev/null
+}
+
+# Downloads the 100 file partitioned ClickBench datasets from
+# https://github.com/ClickHouse/ClickBench/tree/main#data-loading
+#
+# Creates data in $DATA_DIR/hits_partitioned
+data_clickbench_partitioned() {
+    MAX_CONCURRENT_DOWNLOADS=10
+
+    mkdir -p "${DATA_DIR}/hits_partitioned"
+    pushd "${DATA_DIR}/hits_partitioned" > /dev/null
+
+    echo -n "Checking hits_partitioned..."
+    OUTPUT_SIZE=`wc -c * 2>/dev/null | tail -n 1  | awk '{print $1}' || true`
+    if test "${OUTPUT_SIZE}" = "14737666736"; then
+        echo -n "... found ${OUTPUT_SIZE} KB ..."

Review Comment:
   ```suggestion
           echo -n "... found ${OUTPUT_SIZE} bytes ..."
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org