You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "PrabhuJoseph (via GitHub)" <gi...@apache.org> on 2023/10/31 07:57:27 UTC

[PR] [FLINK-33358][sql] Fix Flink SQL Client fail to start in Flink on YARN [flink]

PrabhuJoseph opened a new pull request, #23629:
URL: https://github.com/apache/flink/pull/23629

   ## What is the purpose of the change
   
   Moves the flink sql client jar precedence before the Hadoop classpath to avoid conflicts with the older jline version from the Hadoop classpath.
   
   
   ## Brief change log
   
   1. Changed the order of sql client jar and hadoop classpath in sql-client.sh
   
   
   ## Verifying this change
   
   Launched the Flink SQL Session and verified with Flink sql examples.
   
   This change is a trivial rework / code cleanup without any test coverage.
   
   
   ## Does this pull request potentially affect one of the following parts:
   
     - Dependencies (does it add or upgrade a dependency): no
     - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no
     - The serializers: no
     - The runtime per-record code paths (performance sensitive): no
     - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
     - The S3 file system connector: no
   
   ## Documentation
   
     - Does this pull request introduce a new feature? no
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-33358][sql] Fix Flink SQL Client fail to start in Flink on YARN [flink]

Posted by "PrabhuJoseph (via GitHub)" <gi...@apache.org>.
PrabhuJoseph commented on PR #23629:
URL: https://github.com/apache/flink/pull/23629#issuecomment-1920497139

   Thanks @fsk119 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-33358][sql] Fix Flink SQL Client fail to start in Flink on YARN [flink]

Posted by "rmoff (via GitHub)" <gi...@apache.org>.
rmoff commented on PR #23629:
URL: https://github.com/apache/flink/pull/23629#issuecomment-1917077150

   I can confirm that using the fix in this PR resolves the issue for me. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-33358][sql] Fix Flink SQL Client fail to start in Flink on YARN [flink]

Posted by "PrabhuJoseph (via GitHub)" <gi...@apache.org>.
PrabhuJoseph commented on PR #23629:
URL: https://github.com/apache/flink/pull/23629#issuecomment-1787531140

   @Samrat002 @1996fanrui Could you review this patch when you have time? Thanks.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-33358][sql] Fix Flink SQL Client fail to start in Flink on YARN [flink]

Posted by "PrabhuJoseph (via GitHub)" <gi...@apache.org>.
PrabhuJoseph commented on PR #23629:
URL: https://github.com/apache/flink/pull/23629#issuecomment-1825118817

   @1996fanrui Could you help in committing this change. Thanks.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-33358][sql] Fix Flink SQL Client fail to start in Flink on YARN [flink]

Posted by "PrabhuJoseph (via GitHub)" <gi...@apache.org>.
PrabhuJoseph commented on PR #23629:
URL: https://github.com/apache/flink/pull/23629#issuecomment-1917491233

   Thanks @rmoff for the verification. @dannycranmer Could you please assist with reviewing this patch? Thank you.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-33358][sql] Fix Flink SQL Client fail to start in Flink on YARN [flink]

Posted by "flinkbot (via GitHub)" <gi...@apache.org>.
flinkbot commented on PR #23629:
URL: https://github.com/apache/flink/pull/23629#issuecomment-1786695694

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "2e3b8fc85814d0bf35d1816cb45e2e2ee46d8872",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "2e3b8fc85814d0bf35d1816cb45e2e2ee46d8872",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 2e3b8fc85814d0bf35d1816cb45e2e2ee46d8872 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-33358][sql] Fix Flink SQL Client fail to start in Flink on YARN [flink]

Posted by "Samrat002 (via GitHub)" <gi...@apache.org>.
Samrat002 commented on PR #23629:
URL: https://github.com/apache/flink/pull/23629#issuecomment-1787106053

   @flinkbot run azure


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-33358][sql] Fix Flink SQL Client fail to start in Flink on YARN [flink]

Posted by "fsk119 (via GitHub)" <gi...@apache.org>.
fsk119 merged PR #23629:
URL: https://github.com/apache/flink/pull/23629


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-33358][sql] Fix Flink SQL Client fail to start in Flink on YARN [flink]

Posted by "PrabhuJoseph (via GitHub)" <gi...@apache.org>.
PrabhuJoseph commented on PR #23629:
URL: https://github.com/apache/flink/pull/23629#issuecomment-1803208444

   @fsk119 Please review this patch when you have time. Thanks.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-33358][sql] Fix Flink SQL Client fail to start in Flink on YARN [flink]

Posted by "tchivs (via GitHub)" <gi...@apache.org>.
tchivs commented on PR #23629:
URL: https://github.com/apache/flink/pull/23629#issuecomment-2076205254

   Sure enough it can be used


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-33358][sql] Fix Flink SQL Client fail to start in Flink on YARN [flink]

Posted by "lsyldliu (via GitHub)" <gi...@apache.org>.
lsyldliu commented on PR #23629:
URL: https://github.com/apache/flink/pull/23629#issuecomment-1788343978

   > In my humble opinion, this change is fine. But I don't know much about the details of flink sql, and I'm not sure whether this change introduces additional risks. It would be better if there are flink sql experts to help and confirm.
   > 
   > Hi @lsyldliu , would you mind helping take a look this PR in your free time? thanks~
   
   cc @fsk119 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-33358][sql] Fix Flink SQL Client fail to start in Flink on YARN [flink]

Posted by "rmoff (via GitHub)" <gi...@apache.org>.
rmoff commented on PR #23629:
URL: https://github.com/apache/flink/pull/23629#issuecomment-1917053501

   I am hitting the same problem, with Flink 1.18.1. 
   
   ## Setup 
   
   ```
   # Unpack clean 1.18.1 download
   cd ~/flink
   tar xf flink-1.18.1-bin-scala_2.12.tgz
   
   # Unpack clean Hadoop download
   cd ~/hadoop
   tar xf hadoop-3.3.2.tar.gz
   
   # Set `HADOOP_CLASSPATH` per instructions at 
   # https://nightlies.apache.org/flink/flink-docs-release-1.18/docs/connectors/table/hive/overview/#dependencies
   export HADOOP_CLASSPATH=`~/hadoop/hadoop-3.3.2/bin/hadoop classpath`
   ```
   
   ## Check env
   
   ```
   ❯ echo $HADOOP_CLASSPATH
   /Users/rmoff/hadoop/hadoop-3.3.2/etc/hadoop:/Users/rmoff/hadoop/hadoop-3.3.2/share/hadoop/common/lib/*:/Users/rmoff/hadoop/hadoop-3.3.2/share/hadoop/common/*:/Users/rmoff/hadoop/hadoop-3.3.2/share/hadoop/hdfs:/Users/rmoff/hadoop/hadoop-3.3.2/share/hadoop/hdfs/lib/*:/Users/rmoff/hadoop/hadoop-3.3.2/share/hadoop/hdfs/*:/Users/rmoff/hadoop/hadoop-3.3.2/share/hadoop/mapreduce/*:/Users/rmoff/hadoop/hadoop-3.3.2/share/hadoop/yarn:/Users/rmoff/hadoop/hadoop-3.3.2/share/hadoop/yarn/lib/*:/Users/rmoff/hadoop/hadoop-3.3.2/share/hadoop/yarn/*
   ```
   
   ## Run SQL Client
   
   ```
   ❯ ./bin/sql-client.sh
   SLF4J: Class path contains multiple SLF4J bindings.
   SLF4J: Found binding in [jar:file:/Users/rmoff/flink/flink-1.18.1/lib/log4j-slf4j-impl-2.17.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
   SLF4J: Found binding in [jar:file:/Users/rmoff/hadoop/hadoop-3.3.2/share/hadoop/common/lib/slf4j-log4j12-1.7.30.jar!/org/slf4j/impl/StaticLoggerBinder.class]
   SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
   SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
   
                                      ▒▓██▓██▒
                                  ▓████▒▒█▓▒▓███▓▒
                               ▓███▓░░        ▒▒▒▓██▒  ▒
                             ░██▒   ▒▒▓▓█▓▓▒░      ▒████
                             ██▒         ░▒▓███▒    ▒█▒█▒
                               ░▓█            ███   ▓░▒██
                                 ▓█       ▒▒▒▒▒▓██▓░▒░▓▓█
                               █░ █   ▒▒░       ███▓▓█ ▒█▒▒▒
                               ████░   ▒▓█▓      ██▒▒▒ ▓███▒
                            ░▒█▓▓██       ▓█▒    ▓█▒▓██▓ ░█░
                      ▓░▒▓████▒ ██         ▒█    █▓░▒█▒░▒█▒
                     ███▓░██▓  ▓█           █   █▓ ▒▓█▓▓█▒
                   ░██▓  ░█░            █  █▒ ▒█████▓▒ ██▓░▒
                  ███░ ░ █░          ▓ ░█ █████▒░░    ░█░▓  ▓░
                 ██▓█ ▒▒▓▒          ▓███████▓░       ▒█▒ ▒▓ ▓██▓
              ▒██▓ ▓█ █▓█       ░▒█████▓▓▒░         ██▒▒  █ ▒  ▓█▒
              ▓█▓  ▓█ ██▓ ░▓▓▓▓▓▓▓▒              ▒██▓           ░█▒
              ▓█    █ ▓███▓▒░              ░▓▓▓███▓          ░▒░ ▓█
              ██▓    ██▒    ░▒▓▓███▓▓▓▓▓██████▓▒            ▓███  █
             ▓███▒ ███   ░▓▓▒░░   ░▓████▓░                  ░▒▓▒  █▓
             █▓▒▒▓▓██  ░▒▒░░░▒▒▒▒▓██▓░                            █▓
             ██ ▓░▒█   ▓▓▓▓▒░░  ▒█▓       ▒▓▓██▓    ▓▒          ▒▒▓
             ▓█▓ ▓▒█  █▓░  ░▒▓▓██▒            ░▓█▒   ▒▒▒░▒▒▓█████▒
              ██░ ▓█▒█▒  ▒▓▓▒  ▓█                █░      ░░░░   ░█▒
              ▓█   ▒█▓   ░     █░                ▒█              █▓
               █▓   ██         █░                 ▓▓        ▒█▓▓▓▒█░
                █▓ ░▓██░       ▓▒                  ▓█▓▒░░░▒▓█░    ▒█
                 ██   ▓█▓░      ▒                    ░▒█▒██▒      ▓▓
                  ▓█▒   ▒█▓▒░                         ▒▒ █▒█▓▒▒░░▒██
                   ░██▒    ▒▓▓▒                     ▓██▓▒█▒ ░▓▓▓▓▒█▓
                     ░▓██▒                          ▓░  ▒█▓█  ░░▒▒▒
                         ▒▓▓▓▓▓▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒░░▓▓  ▓░▒█░
   
       ______ _ _       _       _____  ____  _         _____ _ _            _  BETA
      |  ____| (_)     | |     / ____|/ __ \| |       / ____| (_)          | |
      | |__  | |_ _ __ | | __ | (___ | |  | | |      | |    | |_  ___ _ __ | |_
      |  __| | | | '_ \| |/ /  \___ \| |  | | |      | |    | | |/ _ \ '_ \| __|
      | |    | | | | | |   <   ____) | |__| | |____  | |____| | |  __/ | | | |_
      |_|    |_|_|_| |_|_|\_\ |_____/ \___\_\______|  \_____|_|_|\___|_| |_|\__|
   
           Welcome! Enter 'HELP;' to list all available commands. 'QUIT;' to exit.
   
   Command history file path: /Users/rmoff/.flink-sql-history
   
   
   
   Exception in thread "main" org.apache.flink.table.client.SqlClientException: Could not read from command line.
           at org.apache.flink.table.client.cli.CliClient.getAndExecuteStatements(CliClient.java:221)
           at org.apache.flink.table.client.cli.CliClient.executeInteractive(CliClient.java:179)
           at org.apache.flink.table.client.cli.CliClient.executeInInteractiveMode(CliClient.java:121)
           at org.apache.flink.table.client.cli.CliClient.executeInInteractiveMode(CliClient.java:114)
           at org.apache.flink.table.client.SqlClient.openCli(SqlClient.java:169)
           at org.apache.flink.table.client.SqlClient.start(SqlClient.java:118)
           at org.apache.flink.table.client.SqlClient.startClient(SqlClient.java:228)
           at org.apache.flink.table.client.SqlClient.main(SqlClient.java:179)
   Caused by: java.lang.NoClassDefFoundError: Could not initialize class org.apache.flink.table.client.config.SqlClientOptions
           at org.apache.flink.table.client.cli.parser.SqlClientSyntaxHighlighter.highlight(SqlClientSyntaxHighlighter.java:59)
           at org.jline.reader.impl.LineReaderImpl.getHighlightedBuffer(LineReaderImpl.java:3633)
           at org.jline.reader.impl.LineReaderImpl.getDisplayedBufferWithPrompts(LineReaderImpl.java:3615)
           at org.jline.reader.impl.LineReaderImpl.redisplay(LineReaderImpl.java:3554)
           at org.jline.reader.impl.LineReaderImpl.doCleanup(LineReaderImpl.java:2340)
           at org.jline.reader.impl.LineReaderImpl.cleanup(LineReaderImpl.java:2332)
           at org.jline.reader.impl.LineReaderImpl.readLine(LineReaderImpl.java:626)
           at org.apache.flink.table.client.cli.CliClient.getAndExecuteStatements(CliClient.java:194)
           ... 7 more
   
   Shutting down the session...
   done.
   
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org