You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "xylaaaaa (via GitHub)" <gi...@apache.org> on 2026/03/24 03:47:19 UTC
[PR] [improvement](iceberg) remove extra spark-shell bootstrap [doris]
xylaaaaa opened a new pull request, #61649:
URL: https://github.com/apache/doris/pull/61649
## What\n- remove the extra spark-shell bootstrap from the spark-iceberg container startup\n- move deletion-vector seed data into a new aggregated spark-sql script\n- delete the unused scala bootstrap script\n\n## Why\n- external regression pays this spark-iceberg startup cost on every environment bootstrap\n- keeping the deletion-vector seed data inside the existing spark-sql batch avoids one extra heavyweight Spark session\n\n## Testing\n- bash -n docker/thirdparties/docker-compose/iceberg/entrypoint.sh.tpl\n- not run docker-based regression tests in this environment\n
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](iceberg) remove extra spark-shell bootstrap [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #61649:
URL: https://github.com/apache/doris/pull/61649#issuecomment-4123653182
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](iceberg) remove extra spark-shell bootstrap [doris]
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman merged PR #61649:
URL: https://github.com/apache/doris/pull/61649
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](iceberg) remove extra spark-shell bootstrap [doris]
Posted by "xylaaaaa (via GitHub)" <gi...@apache.org>.
xylaaaaa commented on PR #61649:
URL: https://github.com/apache/doris/pull/61649#issuecomment-4115654918
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](iceberg) remove extra spark-shell bootstrap [doris]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #61649:
URL: https://github.com/apache/doris/pull/61649#discussion_r2978827893
##########
docker/thirdparties/docker-compose/iceberg/entrypoint.sh.tpl:
##########
@@ -55,16 +55,6 @@ EXECUTION_TIME2=$((END_TIME2 - START_TIME2))
echo "Script paimon total: {} executed in $EXECUTION_TIME2 seconds"
-
-ls /mnt/scripts/create_preinstalled_scripts/iceberg_scala/*.scala | xargs -n 1 -I {} bash -c '
- START_TIME=$(date +%s)
- spark-shell --conf spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions -I {}
- END_TIME=$(date +%s)
- EXECUTION_TIME=$((END_TIME - START_TIME))
- echo "Script: {} executed in $EXECUTION_TIME seconds"
-'
-
-
START_TIME3=$(date +%s)
find /mnt/scripts/create_preinstalled_scripts/iceberg_load -name '*.sql' | sed 's|^|source |' | sed 's|$|;|'> iceberg_load_total.sql
spark-sql --master spark://doris--spark-iceberg:7077 --conf spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions -f iceberg_load_total.sql
Review Comment:
`find ... -name '*.sql'` does not guarantee a stable ordering. Several existing `runXX.sql` scripts appear to rely on earlier scripts having already run (e.g. `run27.sql` only does `use demo.test_db;` and doesn’t create the DB), so a non-deterministic execution order can cause intermittent bootstrap failures or objects being created in the wrong database. Please sort the file list before generating the aggregated `*_total.sql` (and apply the same fix to the other `find ... > *_total.sql` pipelines in this script).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](iceberg) remove extra spark-shell bootstrap [doris]
Posted by "Thearas (via GitHub)" <gi...@apache.org>.
Thearas commented on PR #61649:
URL: https://github.com/apache/doris/pull/61649#issuecomment-4115203102
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR).
Please clearly describe your PR:
1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](iceberg) remove extra spark-shell bootstrap [doris]
Posted by "xylaaaaa (via GitHub)" <gi...@apache.org>.
xylaaaaa commented on PR #61649:
URL: https://github.com/apache/doris/pull/61649#issuecomment-4116675079
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](iceberg) remove extra spark-shell bootstrap [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #61649:
URL: https://github.com/apache/doris/pull/61649#issuecomment-4123652802
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org