You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@beam.apache.org by gabrywu <ga...@apache.org> on 2023/10/23 01:34:38 UTC

UnboundedSource.split too early

Hi, folks,
I noticed that this code snippet of UnboundedSourceWrapper tells us we
`need` splitting early.

// get the splits early. we assume that the generated splits are stable,
// this is necessary so that the mapping of state to source is correct
// when restoring
splitSources = source.split(parallelism, pipelineOptions);

However, what should we do in this case?
1. submit a job running a flink runner.
2. flink runner setup a kerberos auth
3. submitting client doesn't have that kerberos auth info

In this case, we get the splits too early when submitting client can't pass
the kerberos auth.