[jira] [Comment Edited] (BEAM-3587) User reports TextIO failure in FlinkRunner on master

["Jean-Baptiste Onofré (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/BEAM-3587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16352587#comment-16352587 ] 

Jean-Baptiste Onofré edited comment on BEAM-3587 at 2/5/18 4:10 PM:
--------------------------------------------------------------------

As we have:

{code}
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-shade-plugin</artifactId>
                <version>${maven-shade-plugin.version}</version>
                <configuration>
                    <createDependencyReducedPom>false</createDependencyReducedPom>
                    <filters>
                        <filter>
                            <artifact>*:*</artifact>
                            <excludes>
                                <exclude>META-INF/*.SF</exclude>
                                <exclude>META-INF/*.DSA</exclude>
                                <exclude>META-INF/*.RSA</exclude>
                            </excludes>
                        </filter>
                    </filters>
                </configuration>
                <executions>
                    <execution>
                        <phase>package</phase>
                        <goals>
                            <goal>shade</goal>
                        </goals>
                        <configuration>
                            <shadedArtifactAttached>true</shadedArtifactAttached>
                            <shadedClassifierName>shaded</shadedClassifierName>
                            <transformers>
                                <transformer implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer"/>
                            </transformers>
                        </configuration>
                    </execution>
                </executions>
            </plugin>
{code}

for a Maven build, the same has to be applied to Gradle:

{code}
shadowJar {
    transform(AppendingTransformer) {
        resource = 'reference.conf'
    }
    relocate 'com.google.protobuf', 'shaded.protobuf'
    mergeServiceFiles()
}
{code}

I'm adding a note in the documentation.


was (Author: jbonofre):
As we have:

{code}
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-shade-plugin</artifactId>
                <version>${maven-shade-plugin.version}</version>
                <configuration>
                    <createDependencyReducedPom>false</createDependencyReducedPom>
                    <filters>
                        <filter>
                            <artifact>*:*</artifact>
                            <excludes>
                                <exclude>META-INF/*.SF</exclude>
                                <exclude>META-INF/*.DSA</exclude>
                                <exclude>META-INF/*.RSA</exclude>
                            </excludes>
                        </filter>
                    </filters>
                </configuration>
                <executions>
                    <execution>
                        <phase>package</phase>
                        <goals>
                            <goal>shade</goal>
                        </goals>
                        <configuration>
                            <shadedArtifactAttached>true</shadedArtifactAttached>
                            <shadedClassifierName>shaded</shadedClassifierName>
                            <transformers>
                                <transformer implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer"/>
                            </transformers>
                        </configuration>
                    </execution>
                </executions>
            </plugin>
{code}

for a Maven build, the same has to be applied to Gradle.

I'm adding a note in the documentation.

> User reports TextIO failure in FlinkRunner on master
> ----------------------------------------------------
>
>                 Key: BEAM-3587
>                 URL: https://issues.apache.org/jira/browse/BEAM-3587
>             Project: Beam
>          Issue Type: Bug
>          Components: runner-flink
>            Reporter: Kenneth Knowles
>            Assignee: Jean-Baptiste Onofré
>            Priority: Blocker
>             Fix For: 2.3.0
>
>         Attachments: screen1.png, screen2.png
>
>
> Reported here: [https://lists.apache.org/thread.html/47b16c94032392782505415e010970fd2a9480891c55c2f7b5de92bd@%3Cuser.beam.apache.org%3E]
> "I'm trying to run a pipeline containing just a TextIO.read() step on a Flink cluster, using the latest Beam git revision (ff37337). The job fails to start with the Exception:
>   {{java.lang.UnsupportedOperationException: The transform  is currently not supported.}}
> It does work with Beam 2.2.0 though. All code, logs, and reproduction steps  [https://github.com/pelletier/beam-flink-example]"
> My initial thoughts: I have a guess that this has to do with switching to running from a portable pipeline representation, and it looks like there's a non-composite transform with an empty URN and it threw a bad error message. We can try to root cause but may also mitigate short-term by removing the round-trip through pipeline proto for now.
> What is curious is that the ValidatesRunner and WordCountIT are working - they only run on a local Flink, yet this seems to be a translation issue that would occur for local or distributed runs.
> We need to certainly run this repro on the RC if we don't totally get to the bottom of it quickly.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)