You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "woj-i (via GitHub)" <gi...@apache.org> on 2023/04/21 18:42:35 UTC

[GitHub] [spark] woj-i commented on pull request #40821: [SPARK-43152][spark-structured-streaming] Parametrisable output metadata path (_spark_metadata)

woj-i commented on PR #40821:
URL: https://github.com/apache/spark/pull/40821#issuecomment-1518209816

   Surprisingly after commiting of naming improvements (no logic changes) the build failed. I think it's not related to my change. It happened at [Run / Build modules: streaming, sql-kafka-0-10, streaming-kafka-0-10, mllib-local, mllib, yarn, mesos, kubernetes, hadoop-cloud, spark-ganglia-lgpl, connect, protobuf](https://github.com/woj-i/spark/actions/runs/4762682139/jobs/8474612155#logs)
   Can you please help me with fixing the build?
   
   I extracted some logs that most likely cause failure
   ```
   2023-04-21T08:58:47.7065800Z [info] UserDefinedFunctionE2ETestSuite:
   2023-04-21T08:58:47.9937562Z [info] - Dataset typed filter (225 milliseconds)
   2023-04-21T08:58:48.0428439Z [info] - Dataset typed filter - java (48 milliseconds)
   2023-04-21T08:58:48.1508396Z [info] - Dataset typed map (98 milliseconds)
   2023-04-21T08:58:48.2447937Z [info] - filter with condition (92 milliseconds)
   2023-04-21T08:58:48.3442084Z [info] - filter with col(*) (98 milliseconds)
   2023-04-21T08:58:48.4067957Z [info] - Dataset typed map - java (55 milliseconds)
   2023-04-21T08:58:48.4836489Z [info] - Dataset typed flat map (74 milliseconds)
   2023-04-21T08:58:48.5347903Z [info] - Dataset typed flat map - java (51 milliseconds)
   2023-04-21T08:58:48.8260858Z [info] - Dataset typed map partition (288 milliseconds)
   2023-04-21T08:58:49.0958006Z [info] - Dataset typed map partition - java (270 milliseconds)
   2023-04-21T08:58:49.1921226Z [info] - Dataset foreach (92 milliseconds)
   2023-04-21T08:58:49.2289858Z [info] - Dataset foreach - java (34 milliseconds)
   2023-04-21T08:58:49.3390396Z [info] - Dataset foreachPartition (109 milliseconds)
   2023-04-21T08:58:49.4353655Z [info] - Dataset foreachPartition - java (99 milliseconds)
   2023-04-21T08:58:49.4971434Z [info] - Dataset foreach: change not visible to client (65 milliseconds)
   2023-04-21T08:58:49.5121205Z [info] ReplE2ESuite:
   2023-04-21T08:58:53.2505694Z sh: 1: cannot open /dev/tty: No such device or address
   2023-04-21T08:58:53.2692727Z sh: 1: cannot open /dev/tty: No such device or address
   2023-04-21T08:58:53.3262467Z sh: 1: cannot open /dev/tty: No such device or address
   2023-04-21T08:58:53.3544612Z sh: 1: cannot open /dev/tty: No such device or address
   2023-04-21T08:58:53.3646754Z sh: 1: cannot open /dev/tty: No such device or address
   2023-04-21T08:58:53.3896517Z sh: 1: cannot open /dev/tty: No such device or address
   2023-04-21T08:58:53.4061277Z sh: 1: cannot open /dev/tty: No such device or address
   2023-04-21T08:58:53.4268730Z sh: 1: cannot open /dev/tty: No such device or address
   2023-04-21T08:58:53.4353509Z sh: 1: cannot open /dev/tty: No such device or address
   2023-04-21T08:58:53.4546791Z sh: 1: cannot open /dev/tty: No such device or address
   2023-04-21T08:58:53.4808386Z sh: 1: cannot open /dev/tty: No such device or address
   2023-04-21T08:58:53.4964638Z sh: 1: cannot open /dev/tty: No such device or address
   2023-04-21T08:58:53.5198604Z sh: 1: cannot open /dev/tty: No such device or address
   2023-04-21T08:58:53.5300108Z sh: 1: cannot open /dev/tty: No such device or address
   2023-04-21T08:58:53.5383343Z sh: 1: cannot open /dev/tty: No such device or address
   2023-04-21T08:58:53.5494860Z sh: 1: cannot open /dev/tty: No such device or address
   2023-04-21T08:58:53.5631736Z sh: 1: cannot open /dev/tty: No such device or address
   2023-04-21T08:58:53.5913690Z sh: 1: cannot open /dev/tty: No such device or address
   2023-04-21T08:58:53.6397803Z sh: 1: cannot open /dev/tty: No such device or address
   2023-04-21T08:58:53.6769519Z sh: 1: cannot open /dev/tty: No such device or address
   2023-04-21T08:58:53.7221075Z sh: 1: cannot open /dev/tty: No such device or address
   2023-04-21T08:58:53.7631252Z sh: 1: cannot open /dev/tty: No such device or address
   2023-04-21T08:58:53.8129701Z sh: 1: cannot open /dev/tty: No such device or address
   2023-04-21T08:58:53.8289367Z sh: 1: cannot open /dev/tty: No such device or address
   2023-04-21T08:58:53.8541965Z sh: 1: cannot open /dev/tty: No such device or address
   2023-04-21T08:58:53.8848448Z sh: 1: cannot open /dev/tty: No such device or address
   2023-04-21T08:58:53.9060169Z sh: 1: cannot open /dev/tty: No such device or address
   2023-04-21T08:58:53.9152965Z sh: 1: cannot open /dev/tty: No such device or address
   2023-04-21T08:58:53.9257073Z sh: 1: cannot create /dev/tty: No such device or address
   2023-04-21T08:58:53.9294986Z sh: 1: cannot open /dev/tty: No such device or address
   2023-04-21T08:58:53.9379865Z sh: 1: cannot open /dev/tty: No such device or address
   2023-04-21T08:58:53.9483798Z sh: 1: cannot open /dev/tty: No such device or address
   2023-04-21T08:58:53.9579639Z sh: 1: cannot open /dev/tty: No such device or address
   2023-04-21T08:58:53.9644462Z sh: 1: cannot open /dev/tty: No such device or address
   2023-04-21T08:58:53.9701922Z sh: 1: cannot open /dev/tty: No such device or address
   2023-04-21T08:58:53.9739622Z sh: 1: cannot open /dev/tty: No such device or address
   2023-04-21T08:58:53.9775897Z sh: 1: cannot open /dev/tty: No such device or address
   2023-04-21T08:58:53.9830054Z sh: 1: cannot open /dev/tty: No such device or address
   2023-04-21T08:58:53.9870461Z sh: 1: cannot open /dev/tty: No such device or address
   2023-04-21T08:58:53.9910296Z sh: 1: cannot open /dev/tty: No such device or address
   2023-04-21T08:58:53.9947627Z sh: 1: cannot open /dev/tty: No such device or address
   ```
   ...
   
   ```
   2023-04-21T08:58:59.6221061Z sh: 1: cannot open /dev/tty: No such device or address
   2023-04-21T08:58:59.6255708Z sh: 1: cannot open /dev/tty: No such device or address
   2023-04-21T08:58:59.6282881Z sh: 1: cannot open /dev/tty: No such device or address
   2023-04-21T08:58:59.6349493Z sh: 1: cannot open /dev/tty: No such device or address
   2023-04-21T08:58:59.6627557Z sh: 1: cannot open /dev/tty: No such device or address
   2023-04-21T08:58:59.6787449Z sh: 1: cannot open /dev/tty: No such device or address
   2023-04-21T08:58:59.7065904Z [info] - Simple query *** FAILED *** (10 seconds, 75 milliseconds)
   2023-04-21T08:58:59.7248880Z sh: 1: cannot open /dev/tty: No such device or address
   2023-04-21T08:58:59.7820090Z sh: 1: cannot open /dev/tty: No such device or address
   2023-04-21T08:58:59.7821918Z sh: 1: cannot open /dev/tty: No such device or address
   2023-04-21T08:58:59.7870724Z sh: 1: cannot open /dev/tty: No such device or address
   2023-04-21T08:58:59.7941758Z sh: 1: cannot open /dev/tty: No such device or address
   2023-04-21T08:58:59.8165642Z sh: 1: cannot open /dev/tty: No such device or address
   2023-04-21T08:58:59.8305760Z sh: 1: cannot open /dev/tty: No such device or address
   2023-04-21T08:58:59.8376055Z sh: 1: cannot open /dev/tty: No such device or address
   2023-04-21T08:58:59.8407369Z sh: 1: cannot open /dev/tty: No such device or address
   2023-04-21T08:58:59.8439122Z sh: 1: cannot open /dev/tty: No such device or address
   2023-04-21T08:58:59.8541147Z sh: 1: cannot open /dev/tty: No such device or address
   2023-04-21T08:58:59.8651384Z sh: 1: cannot open /dev/tty: No such device or address
   2023-04-21T08:58:59.8652177Z [info]   java.lang.RuntimeException: REPL Timed out while running command: 
   2023-04-21T08:58:59.8653000Z [info] spark.sql("select 1").collect()
   2023-04-21T08:58:59.8653943Z [info]       
   2023-04-21T08:58:59.8654470Z [info] Console output: 
   2023-04-21T08:58:59.8655695Z [info] Spark session available as 'spark'.
   2023-04-21T08:58:59.8656304Z [info]    _____                  __      ______                            __
   2023-04-21T08:58:59.8657167Z [info]   / ___/____  ____ ______/ /__   / ____/___  ____  ____  ___  _____/ /_
   2023-04-21T08:58:59.8657763Z [info]   \__ \/ __ \/ __ `/ ___/ //_/  / /   / __ \/ __ \/ __ \/ _ \/ ___/ __/
   2023-04-21T08:58:59.8663578Z [info]  ___/ / /_/ / /_/ / /  / ,<    / /___/ /_/ / / / / / / /  __/ /__/ /_
   2023-04-21T08:58:59.8664196Z [info] /____/ .___/\__,_/_/  /_/|_|   \____/\____/_/ /_/_/ /_/\___/\___/\__/
   2023-04-21T08:58:59.8664916Z [info]     /_/
   2023-04-21T08:58:59.8665558Z [info] 
   2023-04-21T08:58:59.8666266Z [info] 
   2023-04-21T08:58:59.8666734Z [info] 
   2023-04-21T08:58:59.8667377Z [info] 
   2023-04-21T08:58:59.8667825Z [info] 
   2023-04-21T08:58:59.8668458Z [info] 
   2023-04-21T08:58:59.8668901Z [info] 
   2023-04-21T08:58:59.8669534Z [info] 
   2023-04-21T08:58:59.8669960Z [info] 
   2023-04-21T08:58:59.8670722Z [info] 
   2023-04-21T08:58:59.8671179Z [info] 
   ```
   ...
   
   ```
   2023-04-21T08:58:59.9409151Z [info] Error output: Compiling (synthetic)/ammonite/predef/ArgsPredef.sc
   2023-04-21T08:58:59.9409484Z [info] Compiling /home/runner/work/spark/spark/connector/connect/client/jvm/(console)
   2023-04-21T08:58:59.9409764Z [info] java.lang.RuntimeException: Nonzero exit value: 2
   2023-04-21T08:58:59.9410063Z [info]   scala.sys.package$.error(package.scala:30)
   2023-04-21T08:58:59.9410464Z [info]   scala.sys.process.ProcessBuilderImpl$AbstractBuilder.slurp(ProcessBuilderImpl.scala:138)
   2023-04-21T08:58:59.9410846Z [info]   scala.sys.process.ProcessBuilderImpl$AbstractBuilder.$bang$bang(ProcessBuilderImpl.scala:108)
   2023-04-21T08:58:59.9411103Z [info]   ammonite.terminal.TTY$.stty(Utils.scala:103)
   2023-04-21T08:58:59.9411387Z [info]   ammonite.terminal.TTY$.withSttyOverride(Utils.scala:114)
   2023-04-21T08:58:59.9411672Z [info]   ammonite.terminal.Terminal$.readLine(Terminal.scala:41)
   2023-04-21T08:58:59.9412007Z [info]   ammonite.repl.AmmoniteFrontEnd.readLine(AmmoniteFrontEnd.scala:133)
   2023-04-21T08:58:59.9412374Z [info]   ammonite.repl.AmmoniteFrontEnd.action(AmmoniteFrontEnd.scala:28)
   2023-04-21T08:58:59.9412650Z [info]   ammonite.repl.Repl.$anonfun$action$4(Repl.scala:194)
   2023-04-21T08:58:59.9412936Z [info]   ammonite.repl.Scoped.$anonfun$flatMap$1(Signaller.scala:45)
   2023-04-21T08:58:59.9413217Z [info]   ammonite.repl.Signaller.apply(Signaller.scala:28)
   2023-04-21T08:58:59.9413496Z [info]   ammonite.repl.Scoped.flatMap(Signaller.scala:45)
   2023-04-21T08:58:59.9413765Z [info]   ammonite.repl.Scoped.flatMap$(Signaller.scala:45)
   2023-04-21T08:58:59.9414044Z [info]   ammonite.repl.Signaller.flatMap(Signaller.scala:16)
   2023-04-21T08:58:59.9414308Z [info]   ammonite.repl.Repl.$anonfun$action$2(Repl.scala:178)
   2023-04-21T08:58:59.9414621Z [info]   ammonite.util.Catching.flatMap(Res.scala:115)
   2023-04-21T08:58:59.9414875Z [info]   ammonite.repl.Repl.action(Repl.scala:170)
   2023-04-21T08:58:59.9415122Z [info]   ammonite.repl.Repl.loop$1(Repl.scala:212)
   2023-04-21T08:58:59.9415366Z [info]   ammonite.repl.Repl.run(Repl.scala:227)
   2023-04-21T08:58:59.9415618Z [info]   ammonite.Main.$anonfun$run$1(Main.scala:236)
   2023-04-21T08:58:59.9415866Z [info]   scala.Option.getOrElse(Option.scala:189)
   2023-04-21T08:58:59.9416095Z [info]   ammonite.Main.run(Main.scala:224)
   2023-04-21T08:58:59.9416446Z [info]   org.apache.spark.sql.application.ConnectRepl$.doMain(ConnectRepl.scala:100)
   2023-04-21T08:58:59.9416845Z [info]   org.apache.spark.sql.application.ReplE2ESuite$$anon$1.run(ReplE2ESuite.scala:59)
   2023-04-21T08:58:59.9417169Z [info]   java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
   2023-04-21T08:58:59.9417461Z [info]   java.util.concurrent.FutureTask.run(FutureTask.java:266)
   2023-04-21T08:58:59.9417825Z [info]   java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
   2023-04-21T08:58:59.9418212Z [info]   java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   2023-04-21T08:58:59.9445731Z [info]   java.lang.Thread.run(Thread.java:750)
   2023-04-21T08:58:59.9446013Z [info] 
   2023-04-21T08:58:59.9446291Z [info] java.lang.RuntimeException: Nonzero exit value: 2
   2023-04-21T08:58:59.9446689Z [info]   scala.sys.package$.error(package.scala:30)
   2023-04-21T08:58:59.9447082Z [info]   scala.sys.process.ProcessBuilderImpl$AbstractBuilder.slurp(ProcessBuilderImpl.scala:138)
   2023-04-21T08:58:59.9447463Z [info]   scala.sys.process.ProcessBuilderImpl$AbstractBuilder.$bang$bang(ProcessBuilderImpl.scala:108)
   2023-04-21T08:58:59.9447714Z [info]   ammonite.terminal.TTY$.stty(Utils.scala:103)
   2023-04-21T08:58:59.9448002Z [info]   ammonite.terminal.TTY$.withSttyOverride(Utils.scala:114)
   2023-04-21T08:58:59.9448285Z [info]   ammonite.terminal.Terminal$.readLine(Terminal.scala:41)
   2023-04-21T08:58:59.9448611Z [info]   ammonite.repl.AmmoniteFrontEnd.readLine(AmmoniteFrontEnd.scala:133)
   2023-04-21T08:58:59.9448994Z [info]   ammonite.repl.AmmoniteFrontEnd.action(AmmoniteFrontEnd.scala:28)
   2023-04-21T08:58:59.9449261Z [info]   ammonite.repl.Repl.$anonfun$action$4(Repl.scala:194)
   2023-04-21T08:58:59.9449548Z [info]   ammonite.repl.Scoped.$anonfun$flatMap$1(Signaller.scala:45)
   2023-04-21T08:58:59.9449820Z [info]   ammonite.repl.Signaller.apply(Signaller.scala:28)
   2023-04-21T08:58:59.9450091Z [info]   ammonite.repl.Scoped.flatMap(Signaller.scala:45)
   2023-04-21T08:58:59.9450362Z [info]   ammonite.repl.Scoped.flatMap$(Signaller.scala:45)
   2023-04-21T08:58:59.9450642Z [info]   ammonite.repl.Signaller.flatMap(Signaller.scala:16)
   2023-04-21T08:58:59.9451184Z [info]   ammonite.repl.Repl.$anonfun$action$2(Repl.scala:178)
   2023-04-21T08:58:59.9451519Z [info]   ammonite.util.Catching.flatMap(Res.scala:115)
   2023-04-21T08:58:59.9451770Z [info]   ammonite.repl.Repl.action(Repl.scala:170)
   2023-04-21T08:58:59.9452017Z [info]   ammonite.repl.Repl.loop$1(Repl.scala:212)
   2023-04-21T08:58:59.9452260Z [info]   ammonite.repl.Repl.run(Repl.scala:227)
   2023-04-21T08:58:59.9452503Z [info]   ammonite.Main.$anonfun$run$1(Main.scala:236)
   2023-04-21T08:58:59.9452752Z [info]   scala.Option.getOrElse(Option.scala:189)
   2023-04-21T08:58:59.9452981Z [info]   ammonite.Main.run(Main.scala:224)
   2023-04-21T08:58:59.9453327Z [info]   org.apache.spark.sql.application.ConnectRepl$.doMain(ConnectRepl.scala:100)
   2023-04-21T08:58:59.9453685Z [info]   org.apache.spark.sql.application.ReplE2ESuite$$anon$1.run(ReplE2ESuite.scala:59)
   2023-04-21T08:58:59.9454018Z [info]   java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
   2023-04-21T08:58:59.9454310Z [info]   java.util.concurrent.FutureTask.run(FutureTask.java:266)
   2023-04-21T08:58:59.9454671Z [info]   java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
   2023-04-21T08:58:59.9455071Z [info]   java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   2023-04-21T08:58:59.9455325Z [info]   java.lang.Thread.run(Thread.java:750)
   2023-04-21T08:58:59.9455486Z [info] 
   
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org