[jira] [Created] (BEAM-671) Update Read PTransform to implicitly use maxNumRecords and maxReadTime

["Jean-Baptiste Onofré (JIRA)" <ji...@apache.org>]

Jean-Baptiste Onofré created BEAM-671:
-----------------------------------------

             Summary: Update Read PTransform to implicitly use maxNumRecords and maxReadTime
                 Key: BEAM-671
                 URL: https://issues.apache.org/jira/browse/BEAM-671
             Project: Beam
          Issue Type: Improvement
          Components: sdk-java-core
            Reporter: Jean-Baptiste Onofré
            Assignee: Jean-Baptiste Onofré


Today, almost all IOs {{Read}} {{PTransform}}, when using an {{UnboundedSource}}, do the following in the {{apply()}}:

{code}
      org.apache.beam.sdk.io.Read.Unbounded<byte[]> unbounded =
          org.apache.beam.sdk.io.Read.from(getSource());

      PTransform<PBegin, PCollection<byte[]>> transform = unbounded;

      if (maxNumRecords != Long.MAX_VALUE) {
        transform = unbounded.withMaxNumRecords(maxNumRecords);
      } else if (maxReadTime != null) {
        transform = unbounded.withMaxReadTime(maxReadTime);
      }

      return input.getPipeline().apply(transform);
{code}

To avoid to duplicate this code bunch of time, it would make sense to do it by default in {{org.apache.beam.sdk.io.Read}}.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)