You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "milenkovicm (via GitHub)" <gi...@apache.org> on 2025/05/21 19:27:28 UTC
[PR] feat: job id is incremental [datafusion-ballista]
milenkovicm opened a new pull request, #1267:
URL: https://github.com/apache/datafusion-ballista/pull/1267
# Which issue does this PR close?
Closes #.
# Rationale for this change
previously `job id` is generated randomly, without any ordering guarantees, which make is rather hard to determine ordering of jobs, and returning totally random results on rest api. Spark is generating job id incrementing atomic int.
# What changes are included in this PR?
- use atomic integer to generate job id, i do not see valid reason not to do it
- move job id generation from task manager to scheduler (if we later want to make it configurable it would be easier to do)
# Are there any user-facing changes?
- no, they might get confused with new job ids
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: job id is incremental [datafusion-ballista]
Posted by "milenkovicm (via GitHub)" <gi...@apache.org>.
milenkovicm commented on PR #1267:
URL: https://github.com/apache/datafusion-ballista/pull/1267#issuecomment-2913196348
hey @Dandandan it does not, the change is focused more on aligning with spark, which to my knowledge does not have multi scheduler setup.
I'm consider using ulid-s which may be better fit, they are random, and sortable. Issues with ulid-s they are quite big strings. Would work with multi scheduler.
wdyt ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: job id is incremental [datafusion-ballista]
Posted by "milenkovicm (via GitHub)" <gi...@apache.org>.
milenkovicm commented on PR #1267:
URL: https://github.com/apache/datafusion-ballista/pull/1267#issuecomment-2926963201
when going through logs, for example, it makes it easier to reason about.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: job id is incremental [datafusion-ballista]
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on PR #1267:
URL: https://github.com/apache/datafusion-ballista/pull/1267#issuecomment-2926982786
Hm yeah. Not anything against it, but just my thoughts :). I think `ULID` would be preferable over an atomic id.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: job id is incremental [datafusion-ballista]
Posted by "milenkovicm (via GitHub)" <gi...@apache.org>.
milenkovicm commented on PR #1267:
URL: https://github.com/apache/datafusion-ballista/pull/1267#issuecomment-2926982414
there is issue with having job id tied to physical directory, which may make mess when scheduler is restarted without restarting executors, making possibility to overlap job data directories (shuffle readers may read wrong spill in current implementation) thus just using incremental id in current implementation may make problems.
I was looking at the `ULID` they generate random but sortable IDs, downside is that they are too big (IMHO), not easiest to reason about ordering either.
snowflake like ids, are sortable and unique; for them machine id should be exposed as a configuration parameter.
Current approach is not sortable, now there is slight chance to get name collision (with very low probability).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: job id is incremental [datafusion-ballista]
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on PR #1267:
URL: https://github.com/apache/datafusion-ballista/pull/1267#issuecomment-2926944302
> hey @Dandandan it does not, the change is focused more on aligning with spark, which to my knowledge does not have multi scheduler setup.
>
> I'm consider using ulid-s which may be better fit, they are random, and sortable. Issues with ulid-s they are quite big strings. Would work with multi scheduler.
>
> wdyt ?
In the API, it's possible to sort by time (e.g. job start time) as well based on the job information.
Why would it need to be in the job id?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: job id is incremental [datafusion-ballista]
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on PR #1267:
URL: https://github.com/apache/datafusion-ballista/pull/1267#issuecomment-2912726820
How would this work in a multi-scheduler setup? Maybe add the scheduler id as a first item?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: job id is incremental [datafusion-ballista]
Posted by "milenkovicm (via GitHub)" <gi...@apache.org>.
milenkovicm commented on PR #1267:
URL: https://github.com/apache/datafusion-ballista/pull/1267#issuecomment-3828716363
closed as stale, may revisit it later
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: job id is incremental [datafusion-ballista]
Posted by "milenkovicm (via GitHub)" <gi...@apache.org>.
milenkovicm closed pull request #1267: feat: job id is incremental
URL: https://github.com/apache/datafusion-ballista/pull/1267
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org