[GitHub] [incubator-seatunnel] superzhang0929 opened a new issue, #2200: [Bug] [seatunnel-connectors-spark] The files I submitted through --files can't be obtained in the code of sink

[GitBox <gi...@apache.org>]

superzhang0929 opened a new issue, #2200:
URL: https://github.com/apache/incubator-seatunnel/issues/2200

   ### Search before asking
   
   - [X] I had searched in the [issues](https://github.com/apache/incubator-seatunnel/issues?q=is%3Aissue+label%3A%22bug%22) and found no similar issues.
   
   
   ### What happened
   
   I want to obtained the file in output method.
   
   ### SeaTunnel Version
   
   2.1.1
   
   ### SeaTunnel Config
   
   ```conf
   env {
     spark.app.name = "test"
     spark.executor.instances =5
     spark.executor.cores =2
     spark.executor.memory = "2G"
     files = "/tmp/exec/process/0/3277/127844/12128295/metrics.json"
   }
   ```
   
   
   ### Running Command
   
   ```shell
   ./bin/start-seatunnel-spark.sh --master yarn --deploy-mode client --config application.config
   ```
   
   
   ### Error Exception
   
   ```log
   I can‘t get the file in the output method.
   
    code like this:
     
   override def output(data: Dataset[Row], environment: SparkEnvironment): Unit = {
       val file = new File("metrics.json")
       val dfFields = data.schema.fieldNames
       val bulkSize = config.getInt(BULK_SIZE)
       val retry = config.getInt(RETRY)
   
       if (!config.hasPath(FIELDS)) {
         fields = dfFields.toList
         initSQL = initPrepareSQL()
       }
       val count=data.count()
        ………………………………………………
   ```
   
   
   ### Flink or Spark Version
   
   _No response_
   
   ### Java or Scala Version
   
   _No response_
   
   ### Screenshots
   
   _No response_
   
   ### Are you willing to submit PR?
   
   - [ ] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-seatunnel] CalvinKirs closed issue #2200: [Bug] [seatunnel-connectors-spark] The files I submitted through --files can't be obtained in the code of sink

[GitBox <gi...@apache.org>]

CalvinKirs closed issue #2200: [Bug] [seatunnel-connectors-spark] The files I submitted through --files can't be obtained in the code of sink
URL: https://github.com/apache/incubator-seatunnel/issues/2200


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-seatunnel] CalvinKirs commented on issue #2200: [Bug] [seatunnel-connectors-spark] The files I submitted through --files can't be obtained in the code of sink

[GitBox <gi...@apache.org>]

CalvinKirs commented on issue #2200:
URL: https://github.com/apache/incubator-seatunnel/issues/2200#issuecomment-1188536384

   Duplicated #2202 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org