You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by rajat kumar <ku...@gmail.com> on 2022/02/26 16:10:25 UTC
Issue while creating spark app
Hello Users,
I am trying to create spark application using Scala(Intellij).
I have installed Scala plugin in intelliJ still getting below error:-
Cannot find project Scala library 2.12.12 for module SparkSimpleApp
Could anyone please help what I am doing wrong?
Thanks
Rajat
Re: Issue while creating spark app
Posted by "ashok34668@yahoo.com.INVALID" <as...@yahoo.com.INVALID>.
Thanks for all these useful info
Hi all
What is the current trend. Is it Spark on Scala with intellij or Spark on python with pycharm.
I am curious because I have moderate experience with Spark on both Scala and python and want to focus on Scala OR python going forward with the intention of joining new companies in Big Data.
thanking you
On Sunday, 27 February 2022, 21:32:05 GMT, Mich Talebzadeh <mi...@gmail.com> wrote:
Might as well update the artefacts to the correct versions hopefully. Downloaded scala 2.12.8
scala -version
Scala code runner version 2.12.8 -- Copyright 2002-2018, LAMP/EPFL and Lightbend, Inc.
Edited the pom.xml as below
<?xml version="1.0" encoding="UTF-8"?><project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"> <modelVersion>4.0.0</modelVersion>
<groupId>spark</groupId> <artifactId>MichTest</artifactId> <version>1.0</version>
<properties> <maven.compiler.source>8</maven.compiler.source> <maven.compiler.target>8</maven.compiler.target> </properties> <dependencies> <dependency> <groupId>org.scala-lang</groupId> <artifactId>scala-library</artifactId> <version>2.12.8</version> </dependency> <dependency> <groupId>org.apache.spark</groupId> <artifactId>spark-core_2.13</artifactId> <version>3.2.1</version> </dependency> <dependency> <groupId>org.apache.spark</groupId> <artifactId>spark-sql_2.13</artifactId> <version>3.2.1</version> </dependency> </dependencies></project>
and built with maven. All good
view my Linkedin profile
https://en.everybodywiki.com/Mich_Talebzadeh
Disclaimer: Use it at your own risk. Any and all responsibility for any loss, damage or destructionof data or any other property which may arise from relying on this email's technical content is explicitly disclaimed.The author will in no case be liable for any monetary damages arising from suchloss, damage or destruction.
On Sun, 27 Feb 2022 at 20:16, Mich Talebzadeh <mi...@gmail.com> wrote:
Thanks Bjorn. I am aware of that. I just really wanted to create the uber jar files with both sbt and maven in Intellij.
cheers
view my Linkedin profile
https://en.everybodywiki.com/Mich_Talebzadeh
Disclaimer: Use it at your own risk. Any and all responsibility for any loss, damage or destructionof data or any other property which may arise from relying on this email's technical content is explicitly disclaimed.The author will in no case be liable for any monetary damages arising from suchloss, damage or destruction.
On Sun, 27 Feb 2022 at 20:12, Bjørn Jørgensen <bj...@gmail.com> wrote:
Mitch: You are using scala 2.11 to do this. Have a look at Building Spark "Spark requires Scala 2.12/2.13; support for Scala 2.11 was removed in Spark 3.0.0."
søn. 27. feb. 2022 kl. 20:55 skrev Mich Talebzadeh <mi...@gmail.com>:
OK I decided to give a try to maven.
Downloaded maven and unzipped the file WSL-Ubuntu terminal as unzip apache-maven-3.8.4-bin.zip
Then added to Windows env variable as MVN_HOME and added the bin directory to path in windows. Restart intellij to pick up the correct path.
Again on the command line in intellij do
mvn -vApache Maven 3.8.4 (9b656c72d54e5bacbed989b64718c159fe39b537)Maven home: d:\temp\apache-maven-3.8.4Java version: 1.8.0_73, vendor: Oracle Corporation, runtime: C:\Program Files\Java\jdk1.8.0_73\jreDefault locale: en_GB, platform encoding: Cp1252OS name: "windows 10", version: "10.0", arch: "amd64", family: "windows"
in Intellij add maven support to your project. Follow this link Add Maven support to an existing project
There will be a pom.xml file under project directory
Edit that pom.xml file and add the following<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>spark</groupId>
<artifactId>MichTest</artifactId>
<version>1.0</version>
<properties>
<maven.compiler.source>8</maven.compiler.source>
<maven.compiler.target>8</maven.compiler.target>
</properties>
<dependencies>
<dependency>
<groupId>org.scala-lang</groupId>
<artifactId>scala-library</artifactId>
<version>2.11.7</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.10</artifactId>
<version>2.0.0</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_2.10</artifactId>
<version>2.0.0</version>
</dependency>
</dependencies>
</project>
In intellij open a Terminal under project sub-directory where the pom file is created and you edited.
mvn clean
[INFO] Scanning for projects...
[INFO]
[INFO] ---------------------------< spark:MichTest >---------------------------
[INFO] Building MichTest 1.0
[INFO] --------------------------------[ jar ]---------------------------------
[INFO]
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ MichTest ---
[INFO] Deleting D:\temp\intellij\MichTest\target
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 4.451 s
[INFO] Finished at: 2022-02-27T19:37:57Z
[INFO] ------------------------------------------------------------------------
mvn compile
[INFO] Scanning for projects...
[INFO]
[INFO] ---------------------------< spark:MichTest >---------------------------
[INFO] Building MichTest 1.0
[INFO] --------------------------------[ jar ]---------------------------------
[INFO]
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ MichTest ---
[WARNING] Using platform encoding (Cp1252 actually) to copy filtered resources, i.e. build is platform dependent!
[INFO] Copying 0 resource
[INFO]
[INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @ MichTest ---
[INFO] Nothing to compile - all classes are up to date
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 1.242 s
[INFO] Finished at: 2022-02-27T19:38:58Z
[INFO] ------------------------------------------------------------------------
Now create the package
mvn package[INFO] Scanning for projects...[INFO] [INFO] ---------------------------< spark:MichTest >---------------------------[INFO] Building MichTest 1.0[INFO] --------------------------------[ jar ]---------------------------------[INFO] [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ MichTest ---[WARNING] Using platform encoding (Cp1252 actually) to copy filtered resources, i.e. build is platform dependent![INFO] Copying 0 resource[INFO][INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @ MichTest ---[INFO] Nothing to compile - all classes are up to date[INFO][INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ MichTest ---[WARNING] Using platform encoding (Cp1252 actually) to copy filtered resources, i.e. build is platform dependent![INFO] skip non existing resourceDirectory D:\temp\intellij\MichTest\src\test\resources[INFO][INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) @ MichTest ---[INFO] Nothing to compile - all classes are up to date[INFO][INFO] --- maven-surefire-plugin:2.12.4:test (default-test) @ MichTest ---[INFO] No tests to run.[INFO][INFO] --- maven-jar-plugin:2.4:jar (default-jar) @ MichTest ---[INFO] Building jar: D:\temp\intellij\MichTest\target\scala-2.11\MichTest-1.0.jar[INFO] ------------------------------------------------------------------------[INFO] BUILD SUCCESS[INFO] ------------------------------------------------------------------------[INFO] Total time: 1.511 s[INFO] Finished at: 2022-02-27T19:40:22Z[INFO] ------------------------------------------------------------------------
sbt should work from the same directory as well.
I find it easier to package from the command line in Intellij Terminal.
Let me know if this is a wrong approach or if there are better ways of doing it.
HTH
view my Linkedin profile
https://en.everybodywiki.com/Mich_Talebzadeh
Disclaimer: Use it at your own risk. Any and all responsibility for any loss, damage or destructionof data or any other property which may arise from relying on this email's technical content is explicitly disclaimed.The author will in no case be liable for any monetary damages arising from suchloss, damage or destruction.
On Sun, 27 Feb 2022 at 10:05, Mich Talebzadeh <mi...@gmail.com> wrote:
| |
Got curious with this intellij stuff.
I recall using sbt rather than MVN so go to terminal in your intellij and verify what is installed
sbt -versionsbt version in this project: 1.3.4sbt script version: 1.3.4
scala -version
Scala code runner version 2.11.7 -- Copyright 2002-2013, LAMP/EPFL
java --versionopenjdk 11.0.7 2020-04-14OpenJDK Runtime Environment AdoptOpenJDK (build 11.0.7+10)OpenJDK 64-Bit Server VM AdoptOpenJDK (build 11.0.7+10, mixed mode)
For now in the directory where you have Main.scala create build.sbt file
// The simplest possible sbt build file is just one line:
scalaVersion := "2.11.7"
// That is, to create a valid sbt build, all you've got to do is define the
// version of Scala you'd like your project to use.
/ To learn more about multi-project builds, head over to the official sbt
// documentation at http://www.scala-sbt.org/documentation.html
libraryDependencies += "org.scala-lang.modules" %% "scala-parser-combinators" % "1.1.2"
libraryDependencies += "org.apache.spark" %% "spark-sql" % "2.1.0"
This is my Main.scala file a copy from sparkbyexample
package org.example
import org.apache.spark.sql.SparkSession
object SparkSessionTest extends App{
val spark = SparkSession.builder()
.master("local[1]")
.appName("SparkByExample")
.getOrCreate();
println("First SparkContext:")
println("APP Name :"+spark.sparkContext.appName);
println("Deploy Mode :"+spark.sparkContext.deployMode);
println("Master :"+spark.sparkContext.master);
val sparkSession2 = SparkSession.builder()
.master("local[1]")
.appName("SparkByExample-test")
.getOrCreate();
println("Second SparkContext:")
println("APP Name :"+sparkSession2.sparkContext.appName);
println("Deploy Mode :"+sparkSession2.sparkContext.deployMode);
println("Master :"+sparkSession2.sparkContext.master);
}
Go back to Terminal under directory where you have both files built.sbt and Main.scala
sbt clean
[info] Loading global plugins from C:\Users\admin\.sbt\1.0\plugins
[info] Loading project definition from D:\temp\intellij\MichTest\src\main\scala\com\ctp\training\scala\project
[info] Loading settings for project scala from build.sbt ...
[info] Set current project to MichTest (in build file:/D:/temp/intellij/MichTest/src/main/scala/com/ctp/training/scala/)
[success] Total time: 0 s, completed Feb 27, 2022 9:54:10 AM
sbt compile[info] Loading global plugins from C:\Users\admin\.sbt\1.0\plugins[info] Loading project definition from D:\temp\intellij\MichTest\src\main\scala\com\ctp\training\scala\project[info] Loading settings for project scala from build.sbt ...[info] Set current project to MichTest (in build file:/D:/temp/intellij/MichTest/src/main/scala/com/ctp/training/scala/)[info] Executing in batch mode. For better performance use sbt's shell[warn] There may be incompatibilities among your library dependencies; run 'evicted' to see detailed eviction warnings.[info] Compiling 1 Scala source to D:\temp\intellij\MichTest\src\main\scala\com\ctp\training\scala\target\scala-2.11\classes ...[success] Total time: 5 s, completed Feb 27, 2022 9:55:10 AM
sbt package
[info] Loading global plugins from C:\Users\admin\.sbt\1.0\plugins
[info] Loading project definition from D:\temp\intellij\MichTest\src\main\scala\com\ctp\training\scala\project
[info] Loading settings for project scala from build.sbt ...
[info] Set current project to MichTest (in build file:/D:/temp/intellij/MichTest/src/main/scala/com/ctp/training/scala/)
[success] Total time: 1 s, completed Feb 27, 2022 9:56:48 AM
ls
Directory: D:\temp\intellij\MichTest\src\main\scala\com\ctp\training\scala
Mode LastWriteTime Length Name
---- ------------- ------ ----
d----- 2/27/2022 7:04 AM null
d----- 2/27/2022 8:33 AM project
d----- 2/27/2022 9:08 AM spark-warehouse
d----- 2/27/2022 9:55 AM target
-a---- 2/27/2022 9:17 AM 3511 build.sbt
Note that you have target directory and underneath scala-2.11 (in my case) and the uber jar file michtest_2.11-1.0.jar
ls Directory: D:\temp\intellij\MichTest\src\main\scala\com\ctp\training\scala\target\scala-2.11
Mode LastWriteTime Length Name
---- ------------- ------ ----d----- 2/27/2022 9:55 AM classesd----- 2/27/2022 9:55 AM update-a---- 2/27/2022 9:56 AM 3938 michtest_2.11-1.0.jar
These are old stuff but still shows how to create a jar file with sbt
HTH
view my Linkedin profile
https://en.everybodywiki.com/Mich_Talebzadeh
Disclaimer: Use it at your own risk. Any and all responsibility for any loss, damage or destructionof data or any other property which may arise from relying on this email's technical content is explicitly disclaimed.The author will in no case be liable for any monetary damages arising from suchloss, damage or destruction.
On Sat, 26 Feb 2022 at 22:48, Sean Owen <sr...@gmail.com> wrote:
I don't think any of that is related, no.How are you dependencies set up? manually with IJ, or in a build file (Maven, Gradle)? Normally you do the latter and dependencies are taken care of for you, but you app would definitely have to express a dependency on Scala libs.
On Sat, Feb 26, 2022 at 4:25 PM Bitfox <bi...@bitfox.top> wrote:
Java SDK installed?
On Sun, Feb 27, 2022 at 5:39 AM Sachit Murarka <co...@gmail.com> wrote:
Hello ,
Thanks for replying. I have installed Scala plugin in IntelliJ first then also it's giving same error
Cannot find project Scala library 2.12.12 for module SparkSimpleApp
ThanksRajat
On Sun, Feb 27, 2022, 00:52 Bitfox <bi...@bitfox.top> wrote:
You need to install scala first, the current version for spark is 2.12.15I would suggest you install scala by sdk which works great.
Thanks
On Sun, Feb 27, 2022 at 12:10 AM rajat kumar <ku...@gmail.com> wrote:
Hello Users,
I am trying to create spark application using Scala(Intellij).
I have installed Scala plugin in intelliJ still getting below error:-
Cannot find project Scala library 2.12.12 for module SparkSimpleApp
Could anyone please help what I am doing wrong?
Thanks
Rajat
--
Bjørn Jørgensen
Vestre Aspehaug 4, 6010 Ålesund
Norge
+47 480 94 297
Re: Issue while creating spark app
Posted by Mich Talebzadeh <mi...@gmail.com>.
Might as well update the artefacts to the correct versions hopefully.
Downloaded scala 2.12.8
scala -version
Scala code runner version 2.12.8 -- Copyright 2002-2018, LAMP/EPFL and
Lightbend, Inc.
Edited the pom.xml as below
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>spark</groupId>
<artifactId>MichTest</artifactId>
<version>1.0</version>
<properties>
<maven.compiler.source>8</maven.compiler.source>
<maven.compiler.target>8</maven.compiler.target>
</properties>
<dependencies>
<dependency>
<groupId>org.scala-lang</groupId>
<artifactId>scala-library</artifactId>
*<version>2.12.8</version>*
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
*<artifactId>spark-core_2.13</artifactId>*
* <version>3.2.1</version>*
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
* <artifactId>spark-sql_2.13</artifactId>*
* <version>3.2.1</version>*
</dependency>
</dependencies>
</project>
and built with maven. All good
view my Linkedin profile
<https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
https://en.everybodywiki.com/Mich_Talebzadeh
*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.
On Sun, 27 Feb 2022 at 20:16, Mich Talebzadeh <mi...@gmail.com>
wrote:
> Thanks Bjorn. I am aware of that. I just really wanted to create the uber
> jar files with both sbt and maven in Intellij.
>
>
> cheers
>
>
>
> view my Linkedin profile
> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>
>
> https://en.everybodywiki.com/Mich_Talebzadeh
>
>
>
> *Disclaimer:* Use it at your own risk. Any and all responsibility for any
> loss, damage or destruction of data or any other property which may arise
> from relying on this email's technical content is explicitly disclaimed.
> The author will in no case be liable for any monetary damages arising from
> such loss, damage or destruction.
>
>
>
>
> On Sun, 27 Feb 2022 at 20:12, Bjørn Jørgensen <bj...@gmail.com>
> wrote:
>
>> Mitch: You are using scala 2.11 to do this. Have a look at Building Spark
>> <https://spark.apache.org/docs/latest/building-spark.html> "Spark
>> requires Scala 2.12/2.13; support for Scala 2.11 was removed in Spark
>> 3.0.0."
>>
>> søn. 27. feb. 2022 kl. 20:55 skrev Mich Talebzadeh <
>> mich.talebzadeh@gmail.com>:
>>
>>> OK I decided to give a try to maven.
>>>
>>> Downloaded maven and unzipped the file WSL-Ubuntu terminal as unzip
>>> apache-maven-3.8.4-bin.zip
>>>
>>> Then added to Windows env variable as MVN_HOME and added the bin
>>> directory to path in windows. Restart intellij to pick up the correct path.
>>>
>>> Again on the command line in intellij do
>>>
>>> *mvn -v*
>>> Apache Maven 3.8.4 (9b656c72d54e5bacbed989b64718c159fe39b537)
>>> Maven home: d:\temp\apache-maven-3.8.4
>>> Java version: 1.8.0_73, vendor: Oracle Corporation, runtime: C:\Program
>>> Files\Java\jdk1.8.0_73\jre
>>> Default locale: en_GB, platform encoding: Cp1252
>>> OS name: "windows 10", version: "10.0", arch: "amd64", family: "windows"
>>>
>>> in Intellij add maven support to your project. Follow this link Add
>>> Maven support to an existing project
>>> <https://www.jetbrains.com/help/idea/convert-a-regular-project-into-a-maven-project.html>
>>>
>>> There will be a pom.xml file under project directory
>>>
>>> [image: image.png]
>>>
>>> Edit that pom.xml file and add the following
>>>
>>> <?xml version="1.0" encoding="UTF-8"?>
>>> <project xmlns="http://maven.apache.org/POM/4.0.0"
>>> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
>>> xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
>>> <modelVersion>4.0.0</modelVersion>
>>>
>>> <groupId>spark</groupId>
>>> <artifactId>MichTest</artifactId>
>>> <version>1.0</version>
>>>
>>> <properties>
>>> <maven.compiler.source>8</maven.compiler.source>
>>> <maven.compiler.target>8</maven.compiler.target>
>>> </properties>
>>> <dependencies>
>>> <dependency>
>>> <groupId>org.scala-lang</groupId>
>>> <artifactId>scala-library</artifactId>
>>> <version>2.11.7</version>
>>> </dependency>
>>> <dependency>
>>> <groupId>org.apache.spark</groupId>
>>> <artifactId>spark-core_2.10</artifactId>
>>> <version>2.0.0</version>
>>> </dependency>
>>> <dependency>
>>> <groupId>org.apache.spark</groupId>
>>> <artifactId>spark-sql_2.10</artifactId>
>>> <version>2.0.0</version>
>>> </dependency>
>>> </dependencies>
>>> </project>
>>>
>>> In intellij open a Terminal under project sub-directory where the pom
>>> file is created and you edited.
>>>
>>>
>>> *mvn clean*
>>>
>>> [INFO] Scanning for projects...
>>>
>>> [INFO]
>>>
>>> [INFO] ---------------------------< spark:MichTest
>>> >---------------------------
>>>
>>> [INFO] Building MichTest 1.0
>>>
>>> [INFO] --------------------------------[ jar
>>> ]---------------------------------
>>>
>>> [INFO]
>>>
>>> [INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ MichTest ---
>>>
>>> [INFO] Deleting D:\temp\intellij\MichTest\target
>>>
>>> [INFO]
>>> ------------------------------------------------------------------------
>>>
>>> [INFO] BUILD SUCCESS
>>>
>>> [INFO]
>>> ------------------------------------------------------------------------
>>>
>>> [INFO] Total time: 4.451 s
>>>
>>> [INFO] Finished at: 2022-02-27T19:37:57Z
>>>
>>> [INFO]
>>> ------------------------------------------------------------------------
>>>
>>> *mvn compile*
>>>
>>> [INFO] Scanning for projects...
>>>
>>> [INFO]
>>>
>>> [INFO] ---------------------------< spark:MichTest
>>> >---------------------------
>>>
>>> [INFO] Building MichTest 1.0
>>>
>>> [INFO] --------------------------------[ jar
>>> ]---------------------------------
>>>
>>> [INFO]
>>>
>>> [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @
>>> MichTest ---
>>>
>>> [WARNING] Using platform encoding (Cp1252 actually) to copy filtered
>>> resources, i.e. build is platform dependent!
>>>
>>> [INFO] Copying 0 resource
>>>
>>> [INFO]
>>>
>>> [INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @
>>> MichTest ---
>>>
>>> [INFO] Nothing to compile - all classes are up to date
>>>
>>> [INFO]
>>> ------------------------------------------------------------------------
>>>
>>> [INFO] BUILD SUCCESS
>>>
>>> [INFO]
>>> ------------------------------------------------------------------------
>>>
>>> [INFO] Total time: 1.242 s
>>>
>>> [INFO] Finished at: 2022-02-27T19:38:58Z
>>>
>>> [INFO]
>>> ------------------------------------------------------------------------
>>>
>>> Now create the package
>>>
>>> *mvn package*
>>> [INFO] Scanning for projects...
>>> [INFO]
>>> [INFO] ---------------------------< spark:MichTest
>>> >---------------------------
>>> [INFO] Building MichTest 1.0
>>> [INFO] --------------------------------[ jar
>>> ]---------------------------------
>>> [INFO]
>>> [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @
>>> MichTest ---
>>> [WARNING] Using platform encoding (Cp1252 actually) to copy filtered
>>> resources, i.e. build is platform dependent!
>>> [INFO] Copying 0 resource
>>> [INFO]
>>> [INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @
>>> MichTest ---
>>> [INFO] Nothing to compile - all classes are up to date
>>> [INFO]
>>> [INFO] --- maven-resources-plugin:2.6:testResources
>>> (default-testResources) @ MichTest ---
>>> [WARNING] Using platform encoding (Cp1252 actually) to copy filtered
>>> resources, i.e. build is platform dependent!
>>> [INFO] skip non existing resourceDirectory
>>> D:\temp\intellij\MichTest\src\test\resources
>>> [INFO]
>>> [INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) @
>>> MichTest ---
>>> [INFO] Nothing to compile - all classes are up to date
>>> [INFO]
>>> [INFO] --- maven-surefire-plugin:2.12.4:test (default-test) @ MichTest
>>> ---
>>> [INFO] No tests to run.
>>> [INFO]
>>> [INFO] --- maven-jar-plugin:2.4:jar (default-jar) @ MichTest ---
>>> [INFO] *Building jar:
>>> D:\temp\intellij\MichTest\target\scala-2.11\MichTest-1.0.jar*
>>> [INFO]
>>> ------------------------------------------------------------------------
>>> [INFO] BUILD SUCCESS
>>> [INFO]
>>> ------------------------------------------------------------------------
>>> [INFO] Total time: 1.511 s
>>> [INFO] Finished at: 2022-02-27T19:40:22Z
>>> [INFO]
>>> ------------------------------------------------------------------------
>>>
>>> sbt should work from the same directory as well.
>>>
>>> I find it easier* to package from the command line in Intellij
>>> Terminal. *
>>>
>>> Let me know if this is a wrong approach or if there are better ways of
>>> doing it.
>>>
>>> HTH
>>>
>>>
>>> view my Linkedin profile
>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>>
>>>
>>> https://en.everybodywiki.com/Mich_Talebzadeh
>>>
>>>
>>>
>>> *Disclaimer:* Use it at your own risk. Any and all responsibility for
>>> any loss, damage or destruction of data or any other property which may
>>> arise from relying on this email's technical content is explicitly
>>> disclaimed. The author will in no case be liable for any monetary damages
>>> arising from such loss, damage or destruction.
>>>
>>>
>>>
>>>
>>> On Sun, 27 Feb 2022 at 10:05, Mich Talebzadeh <mi...@gmail.com>
>>> wrote:
>>>
>>>> Got curious with this intellij stuff.
>>>>
>>>> I recall using sbt rather than MVN so go to terminal in your intellij
>>>> and verify what is installed
>>>>
>>>> sbt -version
>>>> sbt version in this project: 1.3.4
>>>> sbt script version: 1.3.4
>>>>
>>>> scala -version
>>>>
>>>> Scala code runner version 2.11.7 -- Copyright 2002-2013, LAMP/EPFL
>>>>
>>>> java --version
>>>> openjdk 11.0.7 2020-04-14
>>>> OpenJDK Runtime Environment AdoptOpenJDK (build 11.0.7+10)
>>>> OpenJDK 64-Bit Server VM AdoptOpenJDK (build 11.0.7+10, mixed mode)
>>>>
>>>> For now in the directory where you have Main.scala create build.sbt file
>>>>
>>>> // The simplest possible sbt build file is just one line:
>>>>
>>>> scalaVersion := "2.11.7"
>>>> // That is, to create a valid sbt build, all you've got to do is define the
>>>> // version of Scala you'd like your project to use.
>>>>
>>>> / To learn more about multi-project builds, head over to the official sbt
>>>> // documentation at http://www.scala-sbt.org/documentation.html
>>>> libraryDependencies += "org.scala-lang.modules" %% "scala-parser-combinators" % "1.1.2"
>>>> libraryDependencies += "org.apache.spark" %% "spark-sql" % "2.1.0"
>>>>
>>>>
>>>> This is my Main.scala file a copy from sparkbyexample
>>>> <https://sparkbyexamples.com/spark/spark-setup-run-with-scala-intellij/>
>>>>
>>>>
>>>> package org.example
>>>> import org.apache.spark.sql.SparkSession
>>>> object SparkSessionTest extends App{
>>>> val spark = SparkSession.builder()
>>>> .master("local[1]")
>>>> .appName("SparkByExample")
>>>> .getOrCreate();
>>>>
>>>> println("First SparkContext:")
>>>> println("APP Name :"+spark.sparkContext.appName);
>>>> println("Deploy Mode :"+spark.sparkContext.deployMode);
>>>> println("Master :"+spark.sparkContext.master);
>>>>
>>>> val sparkSession2 = SparkSession.builder()
>>>> .master("local[1]")
>>>> .appName("SparkByExample-test")
>>>> .getOrCreate();
>>>>
>>>> println("Second SparkContext:")
>>>> println("APP Name :"+sparkSession2.sparkContext.appName);
>>>> println("Deploy Mode :"+sparkSession2.sparkContext.deployMode);
>>>> println("Master :"+sparkSession2.sparkContext.master);
>>>> }
>>>>
>>>> Go back to Terminal under directory where you have both files
>>>> built.sbt and Main.scala
>>>>
>>>>
>>>> *sbt clean*
>>>>
>>>> [info] Loading global plugins from C:\Users\admin\.sbt\1.0\plugins
>>>>
>>>> [info] Loading project definition from
>>>> D:\temp\intellij\MichTest\src\main\scala\com\ctp\training\scala\project
>>>>
>>>> [info] Loading settings for project scala from build.sbt ...
>>>>
>>>> [info] Set current project to MichTest (in build
>>>> file:/D:/temp/intellij/MichTest/src/main/scala/com/ctp/training/scala/)
>>>>
>>>> [success] Total time: 0 s, completed Feb 27, 2022 9:54:10 AM
>>>>
>>>> *sbt compile*
>>>> [info] Loading global plugins from C:\Users\admin\.sbt\1.0\plugins
>>>> [info] Loading project definition from
>>>> D:\temp\intellij\MichTest\src\main\scala\com\ctp\training\scala\project
>>>> [info] Loading settings for project scala from build.sbt ...
>>>> [info] Set current project to MichTest (in build
>>>> file:/D:/temp/intellij/MichTest/src/main/scala/com/ctp/training/scala/)
>>>> [info] Executing in batch mode. For better performance use sbt's shell
>>>> [warn] There may be incompatibilities among your library dependencies;
>>>> run 'evicted' to see detailed eviction warnings.
>>>> [info] Compiling 1 Scala source to
>>>> D:\temp\intellij\MichTest\src\main\scala\com\ctp\training\scala\target\scala-2.11\classes
>>>> ...
>>>> [success] Total time: 5 s, completed Feb 27, 2022 9:55:10 AM
>>>>
>>>> *sbt package*
>>>>
>>>> [info] Loading global plugins from C:\Users\admin\.sbt\1.0\plugins
>>>>
>>>> [info] Loading project definition from
>>>> D:\temp\intellij\MichTest\src\main\scala\com\ctp\training\scala\project
>>>>
>>>> [info] Loading settings for project scala from build.sbt ...
>>>>
>>>> [info] Set current project to MichTest (in build
>>>> file:/D:/temp/intellij/MichTest/src/main/scala/com/ctp/training/scala/)
>>>>
>>>> [success] Total time: 1 s, completed Feb 27, 2022 9:56:48 AM
>>>>
>>>> *ls*
>>>>
>>>>
>>>> Directory:
>>>> D:\temp\intellij\MichTest\src\main\scala\com\ctp\training\scala
>>>>
>>>>
>>>> Mode LastWriteTime Length Name
>>>>
>>>> ---- ------------- ------ ----
>>>>
>>>> d----- 2/27/2022 7:04 AM null
>>>>
>>>> d----- 2/27/2022 8:33 AM project
>>>>
>>>> d----- 2/27/2022 9:08 AM spark-warehouse
>>>>
>>>> d----- 2/27/2022 9:55 AM target
>>>>
>>>> -a---- 2/27/2022 9:17 AM 3511 build.sbt
>>>>
>>>> Note that you have target directory and underneath scala-2.11 (in my
>>>> case) and the uber jar file michtest_2.11-1.0.jar
>>>>
>>>> *ls*
>>>> Directory:
>>>> D:\temp\intellij\MichTest\src\main\scala\com\ctp\training\scala\target\scala-2.11
>>>> Mode LastWriteTime Length Name
>>>> ---- ------------- ------ ----
>>>> d----- 2/27/2022 9:55 AM classes
>>>> d----- 2/27/2022 9:55 AM update
>>>> -a---- 2/27/2022 9:56 AM 3938
>>>> *michtest_2.11-1.0.jar*
>>>>
>>>>
>>>> These are old stuff but still shows how to create a jar file with sbt
>>>>
>>>> HTH
>>>>
>>>>
>>>> view my Linkedin profile
>>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>>>
>>>>
>>>>
>>>> https://en.everybodywiki.com/Mich_Talebzadeh
>>>>
>>>>
>>>>
>>>> *Disclaimer:* Use it at your own risk. Any and all responsibility for
>>>> any loss, damage or destruction of data or any other property which may
>>>> arise from relying on this email's technical content is explicitly
>>>> disclaimed. The author will in no case be liable for any monetary damages
>>>> arising from such loss, damage or destruction.
>>>>
>>>>
>>>>
>>>>
>>>> On Sat, 26 Feb 2022 at 22:48, Sean Owen <sr...@gmail.com> wrote:
>>>>
>>>>> I don't think any of that is related, no.
>>>>> How are you dependencies set up? manually with IJ, or in a build file
>>>>> (Maven, Gradle)? Normally you do the latter and dependencies are taken care
>>>>> of for you, but you app would definitely have to express a dependency on
>>>>> Scala libs.
>>>>>
>>>>> On Sat, Feb 26, 2022 at 4:25 PM Bitfox <bi...@bitfox.top> wrote:
>>>>>
>>>>>> Java SDK installed?
>>>>>>
>>>>>> On Sun, Feb 27, 2022 at 5:39 AM Sachit Murarka <
>>>>>> connectsachit@gmail.com> wrote:
>>>>>>
>>>>>>> Hello ,
>>>>>>>
>>>>>>> Thanks for replying. I have installed Scala plugin in IntelliJ
>>>>>>> first then also it's giving same error
>>>>>>>
>>>>>>> Cannot find project Scala library 2.12.12 for module SparkSimpleApp
>>>>>>>
>>>>>>> Thanks
>>>>>>> Rajat
>>>>>>>
>>>>>>> On Sun, Feb 27, 2022, 00:52 Bitfox <bi...@bitfox.top> wrote:
>>>>>>>
>>>>>>>> You need to install scala first, the current version for spark is
>>>>>>>> 2.12.15
>>>>>>>> I would suggest you install scala by sdk which works great.
>>>>>>>>
>>>>>>>> Thanks
>>>>>>>>
>>>>>>>> On Sun, Feb 27, 2022 at 12:10 AM rajat kumar <
>>>>>>>> kumar.rajat20del@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> Hello Users,
>>>>>>>>>
>>>>>>>>> I am trying to create spark application using Scala(Intellij).
>>>>>>>>> I have installed Scala plugin in intelliJ still getting below
>>>>>>>>> error:-
>>>>>>>>>
>>>>>>>>> Cannot find project Scala library 2.12.12 for module SparkSimpleApp
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Could anyone please help what I am doing wrong?
>>>>>>>>>
>>>>>>>>> Thanks
>>>>>>>>>
>>>>>>>>> Rajat
>>>>>>>>>
>>>>>>>>
>>
>> --
>> Bjørn Jørgensen
>> Vestre Aspehaug 4, 6010 Ålesund
>> Norge
>>
>> +47 480 94 297
>>
>
Re: Issue while creating spark app
Posted by Bjørn Jørgensen <bj...@gmail.com>.
Rajat Kumer wrote: "Cannot find project Scala library 2.12.12 for module
SparkSimpleApp"
So when I google this error message I find scala project maven sync failed
<https://youtrack.jetbrains.com/issue/SCL-18941>
søn. 27. feb. 2022 kl. 21:59 skrev Mich Talebzadeh <
mich.talebzadeh@gmail.com>:
> sorry which error?
>
>
>
> view my Linkedin profile
> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>
>
> https://en.everybodywiki.com/Mich_Talebzadeh
>
>
>
> *Disclaimer:* Use it at your own risk. Any and all responsibility for any
> loss, damage or destruction of data or any other property which may arise
> from relying on this email's technical content is explicitly disclaimed.
> The author will in no case be liable for any monetary damages arising from
> such loss, damage or destruction.
>
>
>
>
> On Sun, 27 Feb 2022 at 20:52, Bjørn Jørgensen <bj...@gmail.com>
> wrote:
>
>> Anyway I did google on your error and found this scala project maven
>> sync failed <https://youtrack.jetbrains.com/issue/SCL-18941>
>> Is this the same as the one you are getting?
>>
>>
>> søn. 27. feb. 2022 kl. 21:16 skrev Mich Talebzadeh <
>> mich.talebzadeh@gmail.com>:
>>
>>> Thanks Bjorn. I am aware of that. I just really wanted to create the
>>> uber jar files with both sbt and maven in Intellij.
>>>
>>>
>>> cheers
>>>
>>>
>>>
>>> view my Linkedin profile
>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>>
>>>
>>> https://en.everybodywiki.com/Mich_Talebzadeh
>>>
>>>
>>>
>>> *Disclaimer:* Use it at your own risk. Any and all responsibility for
>>> any loss, damage or destruction of data or any other property which may
>>> arise from relying on this email's technical content is explicitly
>>> disclaimed. The author will in no case be liable for any monetary damages
>>> arising from such loss, damage or destruction.
>>>
>>>
>>>
>>>
>>> On Sun, 27 Feb 2022 at 20:12, Bjørn Jørgensen <bj...@gmail.com>
>>> wrote:
>>>
>>>> Mitch: You are using scala 2.11 to do this. Have a look at Building
>>>> Spark <https://spark.apache.org/docs/latest/building-spark.html> "Spark
>>>> requires Scala 2.12/2.13; support for Scala 2.11 was removed in Spark
>>>> 3.0.0."
>>>>
>>>> søn. 27. feb. 2022 kl. 20:55 skrev Mich Talebzadeh <
>>>> mich.talebzadeh@gmail.com>:
>>>>
>>>>> OK I decided to give a try to maven.
>>>>>
>>>>> Downloaded maven and unzipped the file WSL-Ubuntu terminal as unzip
>>>>> apache-maven-3.8.4-bin.zip
>>>>>
>>>>> Then added to Windows env variable as MVN_HOME and added the bin
>>>>> directory to path in windows. Restart intellij to pick up the correct path.
>>>>>
>>>>> Again on the command line in intellij do
>>>>>
>>>>> *mvn -v*
>>>>> Apache Maven 3.8.4 (9b656c72d54e5bacbed989b64718c159fe39b537)
>>>>> Maven home: d:\temp\apache-maven-3.8.4
>>>>> Java version: 1.8.0_73, vendor: Oracle Corporation, runtime:
>>>>> C:\Program Files\Java\jdk1.8.0_73\jre
>>>>> Default locale: en_GB, platform encoding: Cp1252
>>>>> OS name: "windows 10", version: "10.0", arch: "amd64", family:
>>>>> "windows"
>>>>>
>>>>> in Intellij add maven support to your project. Follow this link Add
>>>>> Maven support to an existing project
>>>>> <https://www.jetbrains.com/help/idea/convert-a-regular-project-into-a-maven-project.html>
>>>>>
>>>>> There will be a pom.xml file under project directory
>>>>>
>>>>> [image: image.png]
>>>>>
>>>>> Edit that pom.xml file and add the following
>>>>>
>>>>> <?xml version="1.0" encoding="UTF-8"?>
>>>>> <project xmlns="http://maven.apache.org/POM/4.0.0"
>>>>> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
>>>>> xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
>>>>> <modelVersion>4.0.0</modelVersion>
>>>>>
>>>>> <groupId>spark</groupId>
>>>>> <artifactId>MichTest</artifactId>
>>>>> <version>1.0</version>
>>>>>
>>>>> <properties>
>>>>> <maven.compiler.source>8</maven.compiler.source>
>>>>> <maven.compiler.target>8</maven.compiler.target>
>>>>> </properties>
>>>>> <dependencies>
>>>>> <dependency>
>>>>> <groupId>org.scala-lang</groupId>
>>>>> <artifactId>scala-library</artifactId>
>>>>> <version>2.11.7</version>
>>>>> </dependency>
>>>>> <dependency>
>>>>> <groupId>org.apache.spark</groupId>
>>>>> <artifactId>spark-core_2.10</artifactId>
>>>>> <version>2.0.0</version>
>>>>> </dependency>
>>>>> <dependency>
>>>>> <groupId>org.apache.spark</groupId>
>>>>> <artifactId>spark-sql_2.10</artifactId>
>>>>> <version>2.0.0</version>
>>>>> </dependency>
>>>>> </dependencies>
>>>>> </project>
>>>>>
>>>>> In intellij open a Terminal under project sub-directory where the pom
>>>>> file is created and you edited.
>>>>>
>>>>>
>>>>> *mvn clean*
>>>>>
>>>>> [INFO] Scanning for projects...
>>>>>
>>>>> [INFO]
>>>>>
>>>>> [INFO] ---------------------------< spark:MichTest
>>>>> >---------------------------
>>>>>
>>>>> [INFO] Building MichTest 1.0
>>>>>
>>>>> [INFO] --------------------------------[ jar
>>>>> ]---------------------------------
>>>>>
>>>>> [INFO]
>>>>>
>>>>> [INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ MichTest ---
>>>>>
>>>>> [INFO] Deleting D:\temp\intellij\MichTest\target
>>>>>
>>>>> [INFO]
>>>>> ------------------------------------------------------------------------
>>>>>
>>>>> [INFO] BUILD SUCCESS
>>>>>
>>>>> [INFO]
>>>>> ------------------------------------------------------------------------
>>>>>
>>>>> [INFO] Total time: 4.451 s
>>>>>
>>>>> [INFO] Finished at: 2022-02-27T19:37:57Z
>>>>>
>>>>> [INFO]
>>>>> ------------------------------------------------------------------------
>>>>>
>>>>> *mvn compile*
>>>>>
>>>>> [INFO] Scanning for projects...
>>>>>
>>>>> [INFO]
>>>>>
>>>>> [INFO] ---------------------------< spark:MichTest
>>>>> >---------------------------
>>>>>
>>>>> [INFO] Building MichTest 1.0
>>>>>
>>>>> [INFO] --------------------------------[ jar
>>>>> ]---------------------------------
>>>>>
>>>>> [INFO]
>>>>>
>>>>> [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @
>>>>> MichTest ---
>>>>>
>>>>> [WARNING] Using platform encoding (Cp1252 actually) to copy filtered
>>>>> resources, i.e. build is platform dependent!
>>>>>
>>>>> [INFO] Copying 0 resource
>>>>>
>>>>> [INFO]
>>>>>
>>>>> [INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @
>>>>> MichTest ---
>>>>>
>>>>> [INFO] Nothing to compile - all classes are up to date
>>>>>
>>>>> [INFO]
>>>>> ------------------------------------------------------------------------
>>>>>
>>>>> [INFO] BUILD SUCCESS
>>>>>
>>>>> [INFO]
>>>>> ------------------------------------------------------------------------
>>>>>
>>>>> [INFO] Total time: 1.242 s
>>>>>
>>>>> [INFO] Finished at: 2022-02-27T19:38:58Z
>>>>>
>>>>> [INFO]
>>>>> ------------------------------------------------------------------------
>>>>>
>>>>> Now create the package
>>>>>
>>>>> *mvn package*
>>>>> [INFO] Scanning for projects...
>>>>> [INFO]
>>>>> [INFO] ---------------------------< spark:MichTest
>>>>> >---------------------------
>>>>> [INFO] Building MichTest 1.0
>>>>> [INFO] --------------------------------[ jar
>>>>> ]---------------------------------
>>>>> [INFO]
>>>>> [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @
>>>>> MichTest ---
>>>>> [WARNING] Using platform encoding (Cp1252 actually) to copy filtered
>>>>> resources, i.e. build is platform dependent!
>>>>> [INFO] Copying 0 resource
>>>>> [INFO]
>>>>> [INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @
>>>>> MichTest ---
>>>>> [INFO] Nothing to compile - all classes are up to date
>>>>> [INFO]
>>>>> [INFO] --- maven-resources-plugin:2.6:testResources
>>>>> (default-testResources) @ MichTest ---
>>>>> [WARNING] Using platform encoding (Cp1252 actually) to copy filtered
>>>>> resources, i.e. build is platform dependent!
>>>>> [INFO] skip non existing resourceDirectory
>>>>> D:\temp\intellij\MichTest\src\test\resources
>>>>> [INFO]
>>>>> [INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile)
>>>>> @ MichTest ---
>>>>> [INFO] Nothing to compile - all classes are up to date
>>>>> [INFO]
>>>>> [INFO] --- maven-surefire-plugin:2.12.4:test (default-test) @ MichTest
>>>>> ---
>>>>> [INFO] No tests to run.
>>>>> [INFO]
>>>>> [INFO] --- maven-jar-plugin:2.4:jar (default-jar) @ MichTest ---
>>>>> [INFO] *Building jar:
>>>>> D:\temp\intellij\MichTest\target\scala-2.11\MichTest-1.0.jar*
>>>>> [INFO]
>>>>> ------------------------------------------------------------------------
>>>>> [INFO] BUILD SUCCESS
>>>>> [INFO]
>>>>> ------------------------------------------------------------------------
>>>>> [INFO] Total time: 1.511 s
>>>>> [INFO] Finished at: 2022-02-27T19:40:22Z
>>>>> [INFO]
>>>>> ------------------------------------------------------------------------
>>>>>
>>>>> sbt should work from the same directory as well.
>>>>>
>>>>> I find it easier* to package from the command line in Intellij
>>>>> Terminal. *
>>>>>
>>>>> Let me know if this is a wrong approach or if there are better ways of
>>>>> doing it.
>>>>>
>>>>> HTH
>>>>>
>>>>>
>>>>> view my Linkedin profile
>>>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>>>>
>>>>>
>>>>> https://en.everybodywiki.com/Mich_Talebzadeh
>>>>>
>>>>>
>>>>>
>>>>> *Disclaimer:* Use it at your own risk. Any and all responsibility for
>>>>> any loss, damage or destruction of data or any other property which may
>>>>> arise from relying on this email's technical content is explicitly
>>>>> disclaimed. The author will in no case be liable for any monetary damages
>>>>> arising from such loss, damage or destruction.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Sun, 27 Feb 2022 at 10:05, Mich Talebzadeh <
>>>>> mich.talebzadeh@gmail.com> wrote:
>>>>>
>>>>>> Got curious with this intellij stuff.
>>>>>>
>>>>>> I recall using sbt rather than MVN so go to terminal in your intellij
>>>>>> and verify what is installed
>>>>>>
>>>>>> sbt -version
>>>>>> sbt version in this project: 1.3.4
>>>>>> sbt script version: 1.3.4
>>>>>>
>>>>>> scala -version
>>>>>>
>>>>>> Scala code runner version 2.11.7 -- Copyright 2002-2013, LAMP/EPFL
>>>>>>
>>>>>> java --version
>>>>>> openjdk 11.0.7 2020-04-14
>>>>>> OpenJDK Runtime Environment AdoptOpenJDK (build 11.0.7+10)
>>>>>> OpenJDK 64-Bit Server VM AdoptOpenJDK (build 11.0.7+10, mixed mode)
>>>>>>
>>>>>> For now in the directory where you have Main.scala create build.sbt
>>>>>> file
>>>>>>
>>>>>> // The simplest possible sbt build file is just one line:
>>>>>>
>>>>>> scalaVersion := "2.11.7"
>>>>>> // That is, to create a valid sbt build, all you've got to do is define the
>>>>>> // version of Scala you'd like your project to use.
>>>>>>
>>>>>> / To learn more about multi-project builds, head over to the official sbt
>>>>>> // documentation at http://www.scala-sbt.org/documentation.html
>>>>>> libraryDependencies += "org.scala-lang.modules" %% "scala-parser-combinators" % "1.1.2"
>>>>>> libraryDependencies += "org.apache.spark" %% "spark-sql" % "2.1.0"
>>>>>>
>>>>>>
>>>>>> This is my Main.scala file a copy from sparkbyexample
>>>>>> <https://sparkbyexamples.com/spark/spark-setup-run-with-scala-intellij/>
>>>>>>
>>>>>>
>>>>>> package org.example
>>>>>> import org.apache.spark.sql.SparkSession
>>>>>> object SparkSessionTest extends App{
>>>>>> val spark = SparkSession.builder()
>>>>>> .master("local[1]")
>>>>>> .appName("SparkByExample")
>>>>>> .getOrCreate();
>>>>>>
>>>>>> println("First SparkContext:")
>>>>>> println("APP Name :"+spark.sparkContext.appName);
>>>>>> println("Deploy Mode :"+spark.sparkContext.deployMode);
>>>>>> println("Master :"+spark.sparkContext.master);
>>>>>>
>>>>>> val sparkSession2 = SparkSession.builder()
>>>>>> .master("local[1]")
>>>>>> .appName("SparkByExample-test")
>>>>>> .getOrCreate();
>>>>>>
>>>>>> println("Second SparkContext:")
>>>>>> println("APP Name :"+sparkSession2.sparkContext.appName);
>>>>>> println("Deploy Mode :"+sparkSession2.sparkContext.deployMode);
>>>>>> println("Master :"+sparkSession2.sparkContext.master);
>>>>>> }
>>>>>>
>>>>>> Go back to Terminal under directory where you have both files
>>>>>> built.sbt and Main.scala
>>>>>>
>>>>>>
>>>>>> *sbt clean*
>>>>>>
>>>>>> [info] Loading global plugins from C:\Users\admin\.sbt\1.0\plugins
>>>>>>
>>>>>> [info] Loading project definition from
>>>>>> D:\temp\intellij\MichTest\src\main\scala\com\ctp\training\scala\project
>>>>>>
>>>>>> [info] Loading settings for project scala from build.sbt ...
>>>>>>
>>>>>> [info] Set current project to MichTest (in build
>>>>>> file:/D:/temp/intellij/MichTest/src/main/scala/com/ctp/training/scala/)
>>>>>>
>>>>>> [success] Total time: 0 s, completed Feb 27, 2022 9:54:10 AM
>>>>>>
>>>>>> *sbt compile*
>>>>>> [info] Loading global plugins from C:\Users\admin\.sbt\1.0\plugins
>>>>>> [info] Loading project definition from
>>>>>> D:\temp\intellij\MichTest\src\main\scala\com\ctp\training\scala\project
>>>>>> [info] Loading settings for project scala from build.sbt ...
>>>>>> [info] Set current project to MichTest (in build
>>>>>> file:/D:/temp/intellij/MichTest/src/main/scala/com/ctp/training/scala/)
>>>>>> [info] Executing in batch mode. For better performance use sbt's shell
>>>>>> [warn] There may be incompatibilities among your library
>>>>>> dependencies; run 'evicted' to see detailed eviction warnings.
>>>>>> [info] Compiling 1 Scala source to
>>>>>> D:\temp\intellij\MichTest\src\main\scala\com\ctp\training\scala\target\scala-2.11\classes
>>>>>> ...
>>>>>> [success] Total time: 5 s, completed Feb 27, 2022 9:55:10 AM
>>>>>>
>>>>>> *sbt package*
>>>>>>
>>>>>> [info] Loading global plugins from C:\Users\admin\.sbt\1.0\plugins
>>>>>>
>>>>>> [info] Loading project definition from
>>>>>> D:\temp\intellij\MichTest\src\main\scala\com\ctp\training\scala\project
>>>>>>
>>>>>> [info] Loading settings for project scala from build.sbt ...
>>>>>>
>>>>>> [info] Set current project to MichTest (in build
>>>>>> file:/D:/temp/intellij/MichTest/src/main/scala/com/ctp/training/scala/)
>>>>>>
>>>>>> [success] Total time: 1 s, completed Feb 27, 2022 9:56:48 AM
>>>>>>
>>>>>> *ls*
>>>>>>
>>>>>>
>>>>>> Directory:
>>>>>> D:\temp\intellij\MichTest\src\main\scala\com\ctp\training\scala
>>>>>>
>>>>>>
>>>>>> Mode LastWriteTime Length Name
>>>>>>
>>>>>> ---- ------------- ------ ----
>>>>>>
>>>>>> d----- 2/27/2022 7:04 AM null
>>>>>>
>>>>>> d----- 2/27/2022 8:33 AM project
>>>>>>
>>>>>> d----- 2/27/2022 9:08 AM spark-warehouse
>>>>>>
>>>>>> d----- 2/27/2022 9:55 AM target
>>>>>>
>>>>>> -a---- 2/27/2022 9:17 AM 3511 build.sbt
>>>>>>
>>>>>> Note that you have target directory and underneath scala-2.11 (in my
>>>>>> case) and the uber jar file michtest_2.11-1.0.jar
>>>>>>
>>>>>> *ls*
>>>>>> Directory:
>>>>>> D:\temp\intellij\MichTest\src\main\scala\com\ctp\training\scala\target\scala-2.11
>>>>>> Mode LastWriteTime Length Name
>>>>>> ---- ------------- ------ ----
>>>>>> d----- 2/27/2022 9:55 AM classes
>>>>>> d----- 2/27/2022 9:55 AM update
>>>>>> -a---- 2/27/2022 9:56 AM 3938
>>>>>> *michtest_2.11-1.0.jar*
>>>>>>
>>>>>>
>>>>>> These are old stuff but still shows how to create a jar file with sbt
>>>>>>
>>>>>> HTH
>>>>>>
>>>>>>
>>>>>> view my Linkedin profile
>>>>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>>>>>
>>>>>>
>>>>>>
>>>>>> https://en.everybodywiki.com/Mich_Talebzadeh
>>>>>>
>>>>>>
>>>>>>
>>>>>> *Disclaimer:* Use it at your own risk. Any and all responsibility
>>>>>> for any loss, damage or destruction of data or any other property which may
>>>>>> arise from relying on this email's technical content is explicitly
>>>>>> disclaimed. The author will in no case be liable for any monetary damages
>>>>>> arising from such loss, damage or destruction.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Sat, 26 Feb 2022 at 22:48, Sean Owen <sr...@gmail.com> wrote:
>>>>>>
>>>>>>> I don't think any of that is related, no.
>>>>>>> How are you dependencies set up? manually with IJ, or in a build
>>>>>>> file (Maven, Gradle)? Normally you do the latter and dependencies are taken
>>>>>>> care of for you, but you app would definitely have to express a dependency
>>>>>>> on Scala libs.
>>>>>>>
>>>>>>> On Sat, Feb 26, 2022 at 4:25 PM Bitfox <bi...@bitfox.top> wrote:
>>>>>>>
>>>>>>>> Java SDK installed?
>>>>>>>>
>>>>>>>> On Sun, Feb 27, 2022 at 5:39 AM Sachit Murarka <
>>>>>>>> connectsachit@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> Hello ,
>>>>>>>>>
>>>>>>>>> Thanks for replying. I have installed Scala plugin in IntelliJ
>>>>>>>>> first then also it's giving same error
>>>>>>>>>
>>>>>>>>> Cannot find project Scala library 2.12.12 for module SparkSimpleApp
>>>>>>>>>
>>>>>>>>> Thanks
>>>>>>>>> Rajat
>>>>>>>>>
>>>>>>>>> On Sun, Feb 27, 2022, 00:52 Bitfox <bi...@bitfox.top> wrote:
>>>>>>>>>
>>>>>>>>>> You need to install scala first, the current version for spark is
>>>>>>>>>> 2.12.15
>>>>>>>>>> I would suggest you install scala by sdk which works great.
>>>>>>>>>>
>>>>>>>>>> Thanks
>>>>>>>>>>
>>>>>>>>>> On Sun, Feb 27, 2022 at 12:10 AM rajat kumar <
>>>>>>>>>> kumar.rajat20del@gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hello Users,
>>>>>>>>>>>
>>>>>>>>>>> I am trying to create spark application using Scala(Intellij).
>>>>>>>>>>> I have installed Scala plugin in intelliJ still getting below
>>>>>>>>>>> error:-
>>>>>>>>>>>
>>>>>>>>>>> Cannot find project Scala library 2.12.12 for module
>>>>>>>>>>> SparkSimpleApp
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Could anyone please help what I am doing wrong?
>>>>>>>>>>>
>>>>>>>>>>> Thanks
>>>>>>>>>>>
>>>>>>>>>>> Rajat
>>>>>>>>>>>
>>>>>>>>>>
>>>>
>>>> --
>>>> Bjørn Jørgensen
>>>> Vestre Aspehaug 4, 6010 Ålesund
>>>> Norge
>>>>
>>>> +47 480 94 297
>>>>
>>>
>>
>> --
>> Bjørn Jørgensen
>> Vestre Aspehaug 4, 6010 Ålesund
>> Norge
>>
>> +47 480 94 297
>>
>
--
Bjørn Jørgensen
Vestre Aspehaug 4, 6010 Ålesund
Norge
+47 480 94 297
Re: Issue while creating spark app
Posted by Mich Talebzadeh <mi...@gmail.com>.
sorry which error?
view my Linkedin profile
<https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
https://en.everybodywiki.com/Mich_Talebzadeh
*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.
On Sun, 27 Feb 2022 at 20:52, Bjørn Jørgensen <bj...@gmail.com>
wrote:
> Anyway I did google on your error and found this scala project maven sync
> failed <https://youtrack.jetbrains.com/issue/SCL-18941>
> Is this the same as the one you are getting?
>
>
> søn. 27. feb. 2022 kl. 21:16 skrev Mich Talebzadeh <
> mich.talebzadeh@gmail.com>:
>
>> Thanks Bjorn. I am aware of that. I just really wanted to create the
>> uber jar files with both sbt and maven in Intellij.
>>
>>
>> cheers
>>
>>
>>
>> view my Linkedin profile
>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>
>>
>> https://en.everybodywiki.com/Mich_Talebzadeh
>>
>>
>>
>> *Disclaimer:* Use it at your own risk. Any and all responsibility for
>> any loss, damage or destruction of data or any other property which may
>> arise from relying on this email's technical content is explicitly
>> disclaimed. The author will in no case be liable for any monetary damages
>> arising from such loss, damage or destruction.
>>
>>
>>
>>
>> On Sun, 27 Feb 2022 at 20:12, Bjørn Jørgensen <bj...@gmail.com>
>> wrote:
>>
>>> Mitch: You are using scala 2.11 to do this. Have a look at Building
>>> Spark <https://spark.apache.org/docs/latest/building-spark.html> "Spark
>>> requires Scala 2.12/2.13; support for Scala 2.11 was removed in Spark
>>> 3.0.0."
>>>
>>> søn. 27. feb. 2022 kl. 20:55 skrev Mich Talebzadeh <
>>> mich.talebzadeh@gmail.com>:
>>>
>>>> OK I decided to give a try to maven.
>>>>
>>>> Downloaded maven and unzipped the file WSL-Ubuntu terminal as unzip
>>>> apache-maven-3.8.4-bin.zip
>>>>
>>>> Then added to Windows env variable as MVN_HOME and added the bin
>>>> directory to path in windows. Restart intellij to pick up the correct path.
>>>>
>>>> Again on the command line in intellij do
>>>>
>>>> *mvn -v*
>>>> Apache Maven 3.8.4 (9b656c72d54e5bacbed989b64718c159fe39b537)
>>>> Maven home: d:\temp\apache-maven-3.8.4
>>>> Java version: 1.8.0_73, vendor: Oracle Corporation, runtime: C:\Program
>>>> Files\Java\jdk1.8.0_73\jre
>>>> Default locale: en_GB, platform encoding: Cp1252
>>>> OS name: "windows 10", version: "10.0", arch: "amd64", family: "windows"
>>>>
>>>> in Intellij add maven support to your project. Follow this link Add
>>>> Maven support to an existing project
>>>> <https://www.jetbrains.com/help/idea/convert-a-regular-project-into-a-maven-project.html>
>>>>
>>>> There will be a pom.xml file under project directory
>>>>
>>>> [image: image.png]
>>>>
>>>> Edit that pom.xml file and add the following
>>>>
>>>> <?xml version="1.0" encoding="UTF-8"?>
>>>> <project xmlns="http://maven.apache.org/POM/4.0.0"
>>>> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
>>>> xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
>>>> <modelVersion>4.0.0</modelVersion>
>>>>
>>>> <groupId>spark</groupId>
>>>> <artifactId>MichTest</artifactId>
>>>> <version>1.0</version>
>>>>
>>>> <properties>
>>>> <maven.compiler.source>8</maven.compiler.source>
>>>> <maven.compiler.target>8</maven.compiler.target>
>>>> </properties>
>>>> <dependencies>
>>>> <dependency>
>>>> <groupId>org.scala-lang</groupId>
>>>> <artifactId>scala-library</artifactId>
>>>> <version>2.11.7</version>
>>>> </dependency>
>>>> <dependency>
>>>> <groupId>org.apache.spark</groupId>
>>>> <artifactId>spark-core_2.10</artifactId>
>>>> <version>2.0.0</version>
>>>> </dependency>
>>>> <dependency>
>>>> <groupId>org.apache.spark</groupId>
>>>> <artifactId>spark-sql_2.10</artifactId>
>>>> <version>2.0.0</version>
>>>> </dependency>
>>>> </dependencies>
>>>> </project>
>>>>
>>>> In intellij open a Terminal under project sub-directory where the pom
>>>> file is created and you edited.
>>>>
>>>>
>>>> *mvn clean*
>>>>
>>>> [INFO] Scanning for projects...
>>>>
>>>> [INFO]
>>>>
>>>> [INFO] ---------------------------< spark:MichTest
>>>> >---------------------------
>>>>
>>>> [INFO] Building MichTest 1.0
>>>>
>>>> [INFO] --------------------------------[ jar
>>>> ]---------------------------------
>>>>
>>>> [INFO]
>>>>
>>>> [INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ MichTest ---
>>>>
>>>> [INFO] Deleting D:\temp\intellij\MichTest\target
>>>>
>>>> [INFO]
>>>> ------------------------------------------------------------------------
>>>>
>>>> [INFO] BUILD SUCCESS
>>>>
>>>> [INFO]
>>>> ------------------------------------------------------------------------
>>>>
>>>> [INFO] Total time: 4.451 s
>>>>
>>>> [INFO] Finished at: 2022-02-27T19:37:57Z
>>>>
>>>> [INFO]
>>>> ------------------------------------------------------------------------
>>>>
>>>> *mvn compile*
>>>>
>>>> [INFO] Scanning for projects...
>>>>
>>>> [INFO]
>>>>
>>>> [INFO] ---------------------------< spark:MichTest
>>>> >---------------------------
>>>>
>>>> [INFO] Building MichTest 1.0
>>>>
>>>> [INFO] --------------------------------[ jar
>>>> ]---------------------------------
>>>>
>>>> [INFO]
>>>>
>>>> [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @
>>>> MichTest ---
>>>>
>>>> [WARNING] Using platform encoding (Cp1252 actually) to copy filtered
>>>> resources, i.e. build is platform dependent!
>>>>
>>>> [INFO] Copying 0 resource
>>>>
>>>> [INFO]
>>>>
>>>> [INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @
>>>> MichTest ---
>>>>
>>>> [INFO] Nothing to compile - all classes are up to date
>>>>
>>>> [INFO]
>>>> ------------------------------------------------------------------------
>>>>
>>>> [INFO] BUILD SUCCESS
>>>>
>>>> [INFO]
>>>> ------------------------------------------------------------------------
>>>>
>>>> [INFO] Total time: 1.242 s
>>>>
>>>> [INFO] Finished at: 2022-02-27T19:38:58Z
>>>>
>>>> [INFO]
>>>> ------------------------------------------------------------------------
>>>>
>>>> Now create the package
>>>>
>>>> *mvn package*
>>>> [INFO] Scanning for projects...
>>>> [INFO]
>>>> [INFO] ---------------------------< spark:MichTest
>>>> >---------------------------
>>>> [INFO] Building MichTest 1.0
>>>> [INFO] --------------------------------[ jar
>>>> ]---------------------------------
>>>> [INFO]
>>>> [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @
>>>> MichTest ---
>>>> [WARNING] Using platform encoding (Cp1252 actually) to copy filtered
>>>> resources, i.e. build is platform dependent!
>>>> [INFO] Copying 0 resource
>>>> [INFO]
>>>> [INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @
>>>> MichTest ---
>>>> [INFO] Nothing to compile - all classes are up to date
>>>> [INFO]
>>>> [INFO] --- maven-resources-plugin:2.6:testResources
>>>> (default-testResources) @ MichTest ---
>>>> [WARNING] Using platform encoding (Cp1252 actually) to copy filtered
>>>> resources, i.e. build is platform dependent!
>>>> [INFO] skip non existing resourceDirectory
>>>> D:\temp\intellij\MichTest\src\test\resources
>>>> [INFO]
>>>> [INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile)
>>>> @ MichTest ---
>>>> [INFO] Nothing to compile - all classes are up to date
>>>> [INFO]
>>>> [INFO] --- maven-surefire-plugin:2.12.4:test (default-test) @ MichTest
>>>> ---
>>>> [INFO] No tests to run.
>>>> [INFO]
>>>> [INFO] --- maven-jar-plugin:2.4:jar (default-jar) @ MichTest ---
>>>> [INFO] *Building jar:
>>>> D:\temp\intellij\MichTest\target\scala-2.11\MichTest-1.0.jar*
>>>> [INFO]
>>>> ------------------------------------------------------------------------
>>>> [INFO] BUILD SUCCESS
>>>> [INFO]
>>>> ------------------------------------------------------------------------
>>>> [INFO] Total time: 1.511 s
>>>> [INFO] Finished at: 2022-02-27T19:40:22Z
>>>> [INFO]
>>>> ------------------------------------------------------------------------
>>>>
>>>> sbt should work from the same directory as well.
>>>>
>>>> I find it easier* to package from the command line in Intellij
>>>> Terminal. *
>>>>
>>>> Let me know if this is a wrong approach or if there are better ways of
>>>> doing it.
>>>>
>>>> HTH
>>>>
>>>>
>>>> view my Linkedin profile
>>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>>>
>>>>
>>>> https://en.everybodywiki.com/Mich_Talebzadeh
>>>>
>>>>
>>>>
>>>> *Disclaimer:* Use it at your own risk. Any and all responsibility for
>>>> any loss, damage or destruction of data or any other property which may
>>>> arise from relying on this email's technical content is explicitly
>>>> disclaimed. The author will in no case be liable for any monetary damages
>>>> arising from such loss, damage or destruction.
>>>>
>>>>
>>>>
>>>>
>>>> On Sun, 27 Feb 2022 at 10:05, Mich Talebzadeh <
>>>> mich.talebzadeh@gmail.com> wrote:
>>>>
>>>>> Got curious with this intellij stuff.
>>>>>
>>>>> I recall using sbt rather than MVN so go to terminal in your intellij
>>>>> and verify what is installed
>>>>>
>>>>> sbt -version
>>>>> sbt version in this project: 1.3.4
>>>>> sbt script version: 1.3.4
>>>>>
>>>>> scala -version
>>>>>
>>>>> Scala code runner version 2.11.7 -- Copyright 2002-2013, LAMP/EPFL
>>>>>
>>>>> java --version
>>>>> openjdk 11.0.7 2020-04-14
>>>>> OpenJDK Runtime Environment AdoptOpenJDK (build 11.0.7+10)
>>>>> OpenJDK 64-Bit Server VM AdoptOpenJDK (build 11.0.7+10, mixed mode)
>>>>>
>>>>> For now in the directory where you have Main.scala create build.sbt
>>>>> file
>>>>>
>>>>> // The simplest possible sbt build file is just one line:
>>>>>
>>>>> scalaVersion := "2.11.7"
>>>>> // That is, to create a valid sbt build, all you've got to do is define the
>>>>> // version of Scala you'd like your project to use.
>>>>>
>>>>> / To learn more about multi-project builds, head over to the official sbt
>>>>> // documentation at http://www.scala-sbt.org/documentation.html
>>>>> libraryDependencies += "org.scala-lang.modules" %% "scala-parser-combinators" % "1.1.2"
>>>>> libraryDependencies += "org.apache.spark" %% "spark-sql" % "2.1.0"
>>>>>
>>>>>
>>>>> This is my Main.scala file a copy from sparkbyexample
>>>>> <https://sparkbyexamples.com/spark/spark-setup-run-with-scala-intellij/>
>>>>>
>>>>>
>>>>> package org.example
>>>>> import org.apache.spark.sql.SparkSession
>>>>> object SparkSessionTest extends App{
>>>>> val spark = SparkSession.builder()
>>>>> .master("local[1]")
>>>>> .appName("SparkByExample")
>>>>> .getOrCreate();
>>>>>
>>>>> println("First SparkContext:")
>>>>> println("APP Name :"+spark.sparkContext.appName);
>>>>> println("Deploy Mode :"+spark.sparkContext.deployMode);
>>>>> println("Master :"+spark.sparkContext.master);
>>>>>
>>>>> val sparkSession2 = SparkSession.builder()
>>>>> .master("local[1]")
>>>>> .appName("SparkByExample-test")
>>>>> .getOrCreate();
>>>>>
>>>>> println("Second SparkContext:")
>>>>> println("APP Name :"+sparkSession2.sparkContext.appName);
>>>>> println("Deploy Mode :"+sparkSession2.sparkContext.deployMode);
>>>>> println("Master :"+sparkSession2.sparkContext.master);
>>>>> }
>>>>>
>>>>> Go back to Terminal under directory where you have both files
>>>>> built.sbt and Main.scala
>>>>>
>>>>>
>>>>> *sbt clean*
>>>>>
>>>>> [info] Loading global plugins from C:\Users\admin\.sbt\1.0\plugins
>>>>>
>>>>> [info] Loading project definition from
>>>>> D:\temp\intellij\MichTest\src\main\scala\com\ctp\training\scala\project
>>>>>
>>>>> [info] Loading settings for project scala from build.sbt ...
>>>>>
>>>>> [info] Set current project to MichTest (in build
>>>>> file:/D:/temp/intellij/MichTest/src/main/scala/com/ctp/training/scala/)
>>>>>
>>>>> [success] Total time: 0 s, completed Feb 27, 2022 9:54:10 AM
>>>>>
>>>>> *sbt compile*
>>>>> [info] Loading global plugins from C:\Users\admin\.sbt\1.0\plugins
>>>>> [info] Loading project definition from
>>>>> D:\temp\intellij\MichTest\src\main\scala\com\ctp\training\scala\project
>>>>> [info] Loading settings for project scala from build.sbt ...
>>>>> [info] Set current project to MichTest (in build
>>>>> file:/D:/temp/intellij/MichTest/src/main/scala/com/ctp/training/scala/)
>>>>> [info] Executing in batch mode. For better performance use sbt's shell
>>>>> [warn] There may be incompatibilities among your library dependencies;
>>>>> run 'evicted' to see detailed eviction warnings.
>>>>> [info] Compiling 1 Scala source to
>>>>> D:\temp\intellij\MichTest\src\main\scala\com\ctp\training\scala\target\scala-2.11\classes
>>>>> ...
>>>>> [success] Total time: 5 s, completed Feb 27, 2022 9:55:10 AM
>>>>>
>>>>> *sbt package*
>>>>>
>>>>> [info] Loading global plugins from C:\Users\admin\.sbt\1.0\plugins
>>>>>
>>>>> [info] Loading project definition from
>>>>> D:\temp\intellij\MichTest\src\main\scala\com\ctp\training\scala\project
>>>>>
>>>>> [info] Loading settings for project scala from build.sbt ...
>>>>>
>>>>> [info] Set current project to MichTest (in build
>>>>> file:/D:/temp/intellij/MichTest/src/main/scala/com/ctp/training/scala/)
>>>>>
>>>>> [success] Total time: 1 s, completed Feb 27, 2022 9:56:48 AM
>>>>>
>>>>> *ls*
>>>>>
>>>>>
>>>>> Directory:
>>>>> D:\temp\intellij\MichTest\src\main\scala\com\ctp\training\scala
>>>>>
>>>>>
>>>>> Mode LastWriteTime Length Name
>>>>>
>>>>> ---- ------------- ------ ----
>>>>>
>>>>> d----- 2/27/2022 7:04 AM null
>>>>>
>>>>> d----- 2/27/2022 8:33 AM project
>>>>>
>>>>> d----- 2/27/2022 9:08 AM spark-warehouse
>>>>>
>>>>> d----- 2/27/2022 9:55 AM target
>>>>>
>>>>> -a---- 2/27/2022 9:17 AM 3511 build.sbt
>>>>>
>>>>> Note that you have target directory and underneath scala-2.11 (in my
>>>>> case) and the uber jar file michtest_2.11-1.0.jar
>>>>>
>>>>> *ls*
>>>>> Directory:
>>>>> D:\temp\intellij\MichTest\src\main\scala\com\ctp\training\scala\target\scala-2.11
>>>>> Mode LastWriteTime Length Name
>>>>> ---- ------------- ------ ----
>>>>> d----- 2/27/2022 9:55 AM classes
>>>>> d----- 2/27/2022 9:55 AM update
>>>>> -a---- 2/27/2022 9:56 AM 3938
>>>>> *michtest_2.11-1.0.jar*
>>>>>
>>>>>
>>>>> These are old stuff but still shows how to create a jar file with sbt
>>>>>
>>>>> HTH
>>>>>
>>>>>
>>>>> view my Linkedin profile
>>>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>>>>
>>>>>
>>>>>
>>>>> https://en.everybodywiki.com/Mich_Talebzadeh
>>>>>
>>>>>
>>>>>
>>>>> *Disclaimer:* Use it at your own risk. Any and all responsibility for
>>>>> any loss, damage or destruction of data or any other property which may
>>>>> arise from relying on this email's technical content is explicitly
>>>>> disclaimed. The author will in no case be liable for any monetary damages
>>>>> arising from such loss, damage or destruction.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Sat, 26 Feb 2022 at 22:48, Sean Owen <sr...@gmail.com> wrote:
>>>>>
>>>>>> I don't think any of that is related, no.
>>>>>> How are you dependencies set up? manually with IJ, or in a build file
>>>>>> (Maven, Gradle)? Normally you do the latter and dependencies are taken care
>>>>>> of for you, but you app would definitely have to express a dependency on
>>>>>> Scala libs.
>>>>>>
>>>>>> On Sat, Feb 26, 2022 at 4:25 PM Bitfox <bi...@bitfox.top> wrote:
>>>>>>
>>>>>>> Java SDK installed?
>>>>>>>
>>>>>>> On Sun, Feb 27, 2022 at 5:39 AM Sachit Murarka <
>>>>>>> connectsachit@gmail.com> wrote:
>>>>>>>
>>>>>>>> Hello ,
>>>>>>>>
>>>>>>>> Thanks for replying. I have installed Scala plugin in IntelliJ
>>>>>>>> first then also it's giving same error
>>>>>>>>
>>>>>>>> Cannot find project Scala library 2.12.12 for module SparkSimpleApp
>>>>>>>>
>>>>>>>> Thanks
>>>>>>>> Rajat
>>>>>>>>
>>>>>>>> On Sun, Feb 27, 2022, 00:52 Bitfox <bi...@bitfox.top> wrote:
>>>>>>>>
>>>>>>>>> You need to install scala first, the current version for spark is
>>>>>>>>> 2.12.15
>>>>>>>>> I would suggest you install scala by sdk which works great.
>>>>>>>>>
>>>>>>>>> Thanks
>>>>>>>>>
>>>>>>>>> On Sun, Feb 27, 2022 at 12:10 AM rajat kumar <
>>>>>>>>> kumar.rajat20del@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> Hello Users,
>>>>>>>>>>
>>>>>>>>>> I am trying to create spark application using Scala(Intellij).
>>>>>>>>>> I have installed Scala plugin in intelliJ still getting below
>>>>>>>>>> error:-
>>>>>>>>>>
>>>>>>>>>> Cannot find project Scala library 2.12.12 for module
>>>>>>>>>> SparkSimpleApp
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Could anyone please help what I am doing wrong?
>>>>>>>>>>
>>>>>>>>>> Thanks
>>>>>>>>>>
>>>>>>>>>> Rajat
>>>>>>>>>>
>>>>>>>>>
>>>
>>> --
>>> Bjørn Jørgensen
>>> Vestre Aspehaug 4, 6010 Ålesund
>>> Norge
>>>
>>> +47 480 94 297
>>>
>>
>
> --
> Bjørn Jørgensen
> Vestre Aspehaug 4, 6010 Ålesund
> Norge
>
> +47 480 94 297
>
Re: Issue while creating spark app
Posted by Bjørn Jørgensen <bj...@gmail.com>.
Anyway I did google on your error and found this scala project maven sync
failed <https://youtrack.jetbrains.com/issue/SCL-18941>
Is this the same as the one you are getting?
søn. 27. feb. 2022 kl. 21:16 skrev Mich Talebzadeh <
mich.talebzadeh@gmail.com>:
> Thanks Bjorn. I am aware of that. I just really wanted to create the uber
> jar files with both sbt and maven in Intellij.
>
>
> cheers
>
>
>
> view my Linkedin profile
> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>
>
> https://en.everybodywiki.com/Mich_Talebzadeh
>
>
>
> *Disclaimer:* Use it at your own risk. Any and all responsibility for any
> loss, damage or destruction of data or any other property which may arise
> from relying on this email's technical content is explicitly disclaimed.
> The author will in no case be liable for any monetary damages arising from
> such loss, damage or destruction.
>
>
>
>
> On Sun, 27 Feb 2022 at 20:12, Bjørn Jørgensen <bj...@gmail.com>
> wrote:
>
>> Mitch: You are using scala 2.11 to do this. Have a look at Building Spark
>> <https://spark.apache.org/docs/latest/building-spark.html> "Spark
>> requires Scala 2.12/2.13; support for Scala 2.11 was removed in Spark
>> 3.0.0."
>>
>> søn. 27. feb. 2022 kl. 20:55 skrev Mich Talebzadeh <
>> mich.talebzadeh@gmail.com>:
>>
>>> OK I decided to give a try to maven.
>>>
>>> Downloaded maven and unzipped the file WSL-Ubuntu terminal as unzip
>>> apache-maven-3.8.4-bin.zip
>>>
>>> Then added to Windows env variable as MVN_HOME and added the bin
>>> directory to path in windows. Restart intellij to pick up the correct path.
>>>
>>> Again on the command line in intellij do
>>>
>>> *mvn -v*
>>> Apache Maven 3.8.4 (9b656c72d54e5bacbed989b64718c159fe39b537)
>>> Maven home: d:\temp\apache-maven-3.8.4
>>> Java version: 1.8.0_73, vendor: Oracle Corporation, runtime: C:\Program
>>> Files\Java\jdk1.8.0_73\jre
>>> Default locale: en_GB, platform encoding: Cp1252
>>> OS name: "windows 10", version: "10.0", arch: "amd64", family: "windows"
>>>
>>> in Intellij add maven support to your project. Follow this link Add
>>> Maven support to an existing project
>>> <https://www.jetbrains.com/help/idea/convert-a-regular-project-into-a-maven-project.html>
>>>
>>> There will be a pom.xml file under project directory
>>>
>>> [image: image.png]
>>>
>>> Edit that pom.xml file and add the following
>>>
>>> <?xml version="1.0" encoding="UTF-8"?>
>>> <project xmlns="http://maven.apache.org/POM/4.0.0"
>>> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
>>> xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
>>> <modelVersion>4.0.0</modelVersion>
>>>
>>> <groupId>spark</groupId>
>>> <artifactId>MichTest</artifactId>
>>> <version>1.0</version>
>>>
>>> <properties>
>>> <maven.compiler.source>8</maven.compiler.source>
>>> <maven.compiler.target>8</maven.compiler.target>
>>> </properties>
>>> <dependencies>
>>> <dependency>
>>> <groupId>org.scala-lang</groupId>
>>> <artifactId>scala-library</artifactId>
>>> <version>2.11.7</version>
>>> </dependency>
>>> <dependency>
>>> <groupId>org.apache.spark</groupId>
>>> <artifactId>spark-core_2.10</artifactId>
>>> <version>2.0.0</version>
>>> </dependency>
>>> <dependency>
>>> <groupId>org.apache.spark</groupId>
>>> <artifactId>spark-sql_2.10</artifactId>
>>> <version>2.0.0</version>
>>> </dependency>
>>> </dependencies>
>>> </project>
>>>
>>> In intellij open a Terminal under project sub-directory where the pom
>>> file is created and you edited.
>>>
>>>
>>> *mvn clean*
>>>
>>> [INFO] Scanning for projects...
>>>
>>> [INFO]
>>>
>>> [INFO] ---------------------------< spark:MichTest
>>> >---------------------------
>>>
>>> [INFO] Building MichTest 1.0
>>>
>>> [INFO] --------------------------------[ jar
>>> ]---------------------------------
>>>
>>> [INFO]
>>>
>>> [INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ MichTest ---
>>>
>>> [INFO] Deleting D:\temp\intellij\MichTest\target
>>>
>>> [INFO]
>>> ------------------------------------------------------------------------
>>>
>>> [INFO] BUILD SUCCESS
>>>
>>> [INFO]
>>> ------------------------------------------------------------------------
>>>
>>> [INFO] Total time: 4.451 s
>>>
>>> [INFO] Finished at: 2022-02-27T19:37:57Z
>>>
>>> [INFO]
>>> ------------------------------------------------------------------------
>>>
>>> *mvn compile*
>>>
>>> [INFO] Scanning for projects...
>>>
>>> [INFO]
>>>
>>> [INFO] ---------------------------< spark:MichTest
>>> >---------------------------
>>>
>>> [INFO] Building MichTest 1.0
>>>
>>> [INFO] --------------------------------[ jar
>>> ]---------------------------------
>>>
>>> [INFO]
>>>
>>> [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @
>>> MichTest ---
>>>
>>> [WARNING] Using platform encoding (Cp1252 actually) to copy filtered
>>> resources, i.e. build is platform dependent!
>>>
>>> [INFO] Copying 0 resource
>>>
>>> [INFO]
>>>
>>> [INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @
>>> MichTest ---
>>>
>>> [INFO] Nothing to compile - all classes are up to date
>>>
>>> [INFO]
>>> ------------------------------------------------------------------------
>>>
>>> [INFO] BUILD SUCCESS
>>>
>>> [INFO]
>>> ------------------------------------------------------------------------
>>>
>>> [INFO] Total time: 1.242 s
>>>
>>> [INFO] Finished at: 2022-02-27T19:38:58Z
>>>
>>> [INFO]
>>> ------------------------------------------------------------------------
>>>
>>> Now create the package
>>>
>>> *mvn package*
>>> [INFO] Scanning for projects...
>>> [INFO]
>>> [INFO] ---------------------------< spark:MichTest
>>> >---------------------------
>>> [INFO] Building MichTest 1.0
>>> [INFO] --------------------------------[ jar
>>> ]---------------------------------
>>> [INFO]
>>> [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @
>>> MichTest ---
>>> [WARNING] Using platform encoding (Cp1252 actually) to copy filtered
>>> resources, i.e. build is platform dependent!
>>> [INFO] Copying 0 resource
>>> [INFO]
>>> [INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @
>>> MichTest ---
>>> [INFO] Nothing to compile - all classes are up to date
>>> [INFO]
>>> [INFO] --- maven-resources-plugin:2.6:testResources
>>> (default-testResources) @ MichTest ---
>>> [WARNING] Using platform encoding (Cp1252 actually) to copy filtered
>>> resources, i.e. build is platform dependent!
>>> [INFO] skip non existing resourceDirectory
>>> D:\temp\intellij\MichTest\src\test\resources
>>> [INFO]
>>> [INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) @
>>> MichTest ---
>>> [INFO] Nothing to compile - all classes are up to date
>>> [INFO]
>>> [INFO] --- maven-surefire-plugin:2.12.4:test (default-test) @ MichTest
>>> ---
>>> [INFO] No tests to run.
>>> [INFO]
>>> [INFO] --- maven-jar-plugin:2.4:jar (default-jar) @ MichTest ---
>>> [INFO] *Building jar:
>>> D:\temp\intellij\MichTest\target\scala-2.11\MichTest-1.0.jar*
>>> [INFO]
>>> ------------------------------------------------------------------------
>>> [INFO] BUILD SUCCESS
>>> [INFO]
>>> ------------------------------------------------------------------------
>>> [INFO] Total time: 1.511 s
>>> [INFO] Finished at: 2022-02-27T19:40:22Z
>>> [INFO]
>>> ------------------------------------------------------------------------
>>>
>>> sbt should work from the same directory as well.
>>>
>>> I find it easier* to package from the command line in Intellij
>>> Terminal. *
>>>
>>> Let me know if this is a wrong approach or if there are better ways of
>>> doing it.
>>>
>>> HTH
>>>
>>>
>>> view my Linkedin profile
>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>>
>>>
>>> https://en.everybodywiki.com/Mich_Talebzadeh
>>>
>>>
>>>
>>> *Disclaimer:* Use it at your own risk. Any and all responsibility for
>>> any loss, damage or destruction of data or any other property which may
>>> arise from relying on this email's technical content is explicitly
>>> disclaimed. The author will in no case be liable for any monetary damages
>>> arising from such loss, damage or destruction.
>>>
>>>
>>>
>>>
>>> On Sun, 27 Feb 2022 at 10:05, Mich Talebzadeh <mi...@gmail.com>
>>> wrote:
>>>
>>>> Got curious with this intellij stuff.
>>>>
>>>> I recall using sbt rather than MVN so go to terminal in your intellij
>>>> and verify what is installed
>>>>
>>>> sbt -version
>>>> sbt version in this project: 1.3.4
>>>> sbt script version: 1.3.4
>>>>
>>>> scala -version
>>>>
>>>> Scala code runner version 2.11.7 -- Copyright 2002-2013, LAMP/EPFL
>>>>
>>>> java --version
>>>> openjdk 11.0.7 2020-04-14
>>>> OpenJDK Runtime Environment AdoptOpenJDK (build 11.0.7+10)
>>>> OpenJDK 64-Bit Server VM AdoptOpenJDK (build 11.0.7+10, mixed mode)
>>>>
>>>> For now in the directory where you have Main.scala create build.sbt file
>>>>
>>>> // The simplest possible sbt build file is just one line:
>>>>
>>>> scalaVersion := "2.11.7"
>>>> // That is, to create a valid sbt build, all you've got to do is define the
>>>> // version of Scala you'd like your project to use.
>>>>
>>>> / To learn more about multi-project builds, head over to the official sbt
>>>> // documentation at http://www.scala-sbt.org/documentation.html
>>>> libraryDependencies += "org.scala-lang.modules" %% "scala-parser-combinators" % "1.1.2"
>>>> libraryDependencies += "org.apache.spark" %% "spark-sql" % "2.1.0"
>>>>
>>>>
>>>> This is my Main.scala file a copy from sparkbyexample
>>>> <https://sparkbyexamples.com/spark/spark-setup-run-with-scala-intellij/>
>>>>
>>>>
>>>> package org.example
>>>> import org.apache.spark.sql.SparkSession
>>>> object SparkSessionTest extends App{
>>>> val spark = SparkSession.builder()
>>>> .master("local[1]")
>>>> .appName("SparkByExample")
>>>> .getOrCreate();
>>>>
>>>> println("First SparkContext:")
>>>> println("APP Name :"+spark.sparkContext.appName);
>>>> println("Deploy Mode :"+spark.sparkContext.deployMode);
>>>> println("Master :"+spark.sparkContext.master);
>>>>
>>>> val sparkSession2 = SparkSession.builder()
>>>> .master("local[1]")
>>>> .appName("SparkByExample-test")
>>>> .getOrCreate();
>>>>
>>>> println("Second SparkContext:")
>>>> println("APP Name :"+sparkSession2.sparkContext.appName);
>>>> println("Deploy Mode :"+sparkSession2.sparkContext.deployMode);
>>>> println("Master :"+sparkSession2.sparkContext.master);
>>>> }
>>>>
>>>> Go back to Terminal under directory where you have both files
>>>> built.sbt and Main.scala
>>>>
>>>>
>>>> *sbt clean*
>>>>
>>>> [info] Loading global plugins from C:\Users\admin\.sbt\1.0\plugins
>>>>
>>>> [info] Loading project definition from
>>>> D:\temp\intellij\MichTest\src\main\scala\com\ctp\training\scala\project
>>>>
>>>> [info] Loading settings for project scala from build.sbt ...
>>>>
>>>> [info] Set current project to MichTest (in build
>>>> file:/D:/temp/intellij/MichTest/src/main/scala/com/ctp/training/scala/)
>>>>
>>>> [success] Total time: 0 s, completed Feb 27, 2022 9:54:10 AM
>>>>
>>>> *sbt compile*
>>>> [info] Loading global plugins from C:\Users\admin\.sbt\1.0\plugins
>>>> [info] Loading project definition from
>>>> D:\temp\intellij\MichTest\src\main\scala\com\ctp\training\scala\project
>>>> [info] Loading settings for project scala from build.sbt ...
>>>> [info] Set current project to MichTest (in build
>>>> file:/D:/temp/intellij/MichTest/src/main/scala/com/ctp/training/scala/)
>>>> [info] Executing in batch mode. For better performance use sbt's shell
>>>> [warn] There may be incompatibilities among your library dependencies;
>>>> run 'evicted' to see detailed eviction warnings.
>>>> [info] Compiling 1 Scala source to
>>>> D:\temp\intellij\MichTest\src\main\scala\com\ctp\training\scala\target\scala-2.11\classes
>>>> ...
>>>> [success] Total time: 5 s, completed Feb 27, 2022 9:55:10 AM
>>>>
>>>> *sbt package*
>>>>
>>>> [info] Loading global plugins from C:\Users\admin\.sbt\1.0\plugins
>>>>
>>>> [info] Loading project definition from
>>>> D:\temp\intellij\MichTest\src\main\scala\com\ctp\training\scala\project
>>>>
>>>> [info] Loading settings for project scala from build.sbt ...
>>>>
>>>> [info] Set current project to MichTest (in build
>>>> file:/D:/temp/intellij/MichTest/src/main/scala/com/ctp/training/scala/)
>>>>
>>>> [success] Total time: 1 s, completed Feb 27, 2022 9:56:48 AM
>>>>
>>>> *ls*
>>>>
>>>>
>>>> Directory:
>>>> D:\temp\intellij\MichTest\src\main\scala\com\ctp\training\scala
>>>>
>>>>
>>>> Mode LastWriteTime Length Name
>>>>
>>>> ---- ------------- ------ ----
>>>>
>>>> d----- 2/27/2022 7:04 AM null
>>>>
>>>> d----- 2/27/2022 8:33 AM project
>>>>
>>>> d----- 2/27/2022 9:08 AM spark-warehouse
>>>>
>>>> d----- 2/27/2022 9:55 AM target
>>>>
>>>> -a---- 2/27/2022 9:17 AM 3511 build.sbt
>>>>
>>>> Note that you have target directory and underneath scala-2.11 (in my
>>>> case) and the uber jar file michtest_2.11-1.0.jar
>>>>
>>>> *ls*
>>>> Directory:
>>>> D:\temp\intellij\MichTest\src\main\scala\com\ctp\training\scala\target\scala-2.11
>>>> Mode LastWriteTime Length Name
>>>> ---- ------------- ------ ----
>>>> d----- 2/27/2022 9:55 AM classes
>>>> d----- 2/27/2022 9:55 AM update
>>>> -a---- 2/27/2022 9:56 AM 3938
>>>> *michtest_2.11-1.0.jar*
>>>>
>>>>
>>>> These are old stuff but still shows how to create a jar file with sbt
>>>>
>>>> HTH
>>>>
>>>>
>>>> view my Linkedin profile
>>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>>>
>>>>
>>>>
>>>> https://en.everybodywiki.com/Mich_Talebzadeh
>>>>
>>>>
>>>>
>>>> *Disclaimer:* Use it at your own risk. Any and all responsibility for
>>>> any loss, damage or destruction of data or any other property which may
>>>> arise from relying on this email's technical content is explicitly
>>>> disclaimed. The author will in no case be liable for any monetary damages
>>>> arising from such loss, damage or destruction.
>>>>
>>>>
>>>>
>>>>
>>>> On Sat, 26 Feb 2022 at 22:48, Sean Owen <sr...@gmail.com> wrote:
>>>>
>>>>> I don't think any of that is related, no.
>>>>> How are you dependencies set up? manually with IJ, or in a build file
>>>>> (Maven, Gradle)? Normally you do the latter and dependencies are taken care
>>>>> of for you, but you app would definitely have to express a dependency on
>>>>> Scala libs.
>>>>>
>>>>> On Sat, Feb 26, 2022 at 4:25 PM Bitfox <bi...@bitfox.top> wrote:
>>>>>
>>>>>> Java SDK installed?
>>>>>>
>>>>>> On Sun, Feb 27, 2022 at 5:39 AM Sachit Murarka <
>>>>>> connectsachit@gmail.com> wrote:
>>>>>>
>>>>>>> Hello ,
>>>>>>>
>>>>>>> Thanks for replying. I have installed Scala plugin in IntelliJ
>>>>>>> first then also it's giving same error
>>>>>>>
>>>>>>> Cannot find project Scala library 2.12.12 for module SparkSimpleApp
>>>>>>>
>>>>>>> Thanks
>>>>>>> Rajat
>>>>>>>
>>>>>>> On Sun, Feb 27, 2022, 00:52 Bitfox <bi...@bitfox.top> wrote:
>>>>>>>
>>>>>>>> You need to install scala first, the current version for spark is
>>>>>>>> 2.12.15
>>>>>>>> I would suggest you install scala by sdk which works great.
>>>>>>>>
>>>>>>>> Thanks
>>>>>>>>
>>>>>>>> On Sun, Feb 27, 2022 at 12:10 AM rajat kumar <
>>>>>>>> kumar.rajat20del@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> Hello Users,
>>>>>>>>>
>>>>>>>>> I am trying to create spark application using Scala(Intellij).
>>>>>>>>> I have installed Scala plugin in intelliJ still getting below
>>>>>>>>> error:-
>>>>>>>>>
>>>>>>>>> Cannot find project Scala library 2.12.12 for module SparkSimpleApp
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Could anyone please help what I am doing wrong?
>>>>>>>>>
>>>>>>>>> Thanks
>>>>>>>>>
>>>>>>>>> Rajat
>>>>>>>>>
>>>>>>>>
>>
>> --
>> Bjørn Jørgensen
>> Vestre Aspehaug 4, 6010 Ålesund
>> Norge
>>
>> +47 480 94 297
>>
>
--
Bjørn Jørgensen
Vestre Aspehaug 4, 6010 Ålesund
Norge
+47 480 94 297
Re: Issue while creating spark app
Posted by Mich Talebzadeh <mi...@gmail.com>.
Thanks Bjorn. I am aware of that. I just really wanted to create the uber
jar files with both sbt and maven in Intellij.
cheers
view my Linkedin profile
<https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
https://en.everybodywiki.com/Mich_Talebzadeh
*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.
On Sun, 27 Feb 2022 at 20:12, Bjørn Jørgensen <bj...@gmail.com>
wrote:
> Mitch: You are using scala 2.11 to do this. Have a look at Building Spark
> <https://spark.apache.org/docs/latest/building-spark.html> "Spark
> requires Scala 2.12/2.13; support for Scala 2.11 was removed in Spark
> 3.0.0."
>
> søn. 27. feb. 2022 kl. 20:55 skrev Mich Talebzadeh <
> mich.talebzadeh@gmail.com>:
>
>> OK I decided to give a try to maven.
>>
>> Downloaded maven and unzipped the file WSL-Ubuntu terminal as unzip
>> apache-maven-3.8.4-bin.zip
>>
>> Then added to Windows env variable as MVN_HOME and added the bin
>> directory to path in windows. Restart intellij to pick up the correct path.
>>
>> Again on the command line in intellij do
>>
>> *mvn -v*
>> Apache Maven 3.8.4 (9b656c72d54e5bacbed989b64718c159fe39b537)
>> Maven home: d:\temp\apache-maven-3.8.4
>> Java version: 1.8.0_73, vendor: Oracle Corporation, runtime: C:\Program
>> Files\Java\jdk1.8.0_73\jre
>> Default locale: en_GB, platform encoding: Cp1252
>> OS name: "windows 10", version: "10.0", arch: "amd64", family: "windows"
>>
>> in Intellij add maven support to your project. Follow this link Add
>> Maven support to an existing project
>> <https://www.jetbrains.com/help/idea/convert-a-regular-project-into-a-maven-project.html>
>>
>> There will be a pom.xml file under project directory
>>
>> [image: image.png]
>>
>> Edit that pom.xml file and add the following
>>
>> <?xml version="1.0" encoding="UTF-8"?>
>> <project xmlns="http://maven.apache.org/POM/4.0.0"
>> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
>> xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
>> <modelVersion>4.0.0</modelVersion>
>>
>> <groupId>spark</groupId>
>> <artifactId>MichTest</artifactId>
>> <version>1.0</version>
>>
>> <properties>
>> <maven.compiler.source>8</maven.compiler.source>
>> <maven.compiler.target>8</maven.compiler.target>
>> </properties>
>> <dependencies>
>> <dependency>
>> <groupId>org.scala-lang</groupId>
>> <artifactId>scala-library</artifactId>
>> <version>2.11.7</version>
>> </dependency>
>> <dependency>
>> <groupId>org.apache.spark</groupId>
>> <artifactId>spark-core_2.10</artifactId>
>> <version>2.0.0</version>
>> </dependency>
>> <dependency>
>> <groupId>org.apache.spark</groupId>
>> <artifactId>spark-sql_2.10</artifactId>
>> <version>2.0.0</version>
>> </dependency>
>> </dependencies>
>> </project>
>>
>> In intellij open a Terminal under project sub-directory where the pom
>> file is created and you edited.
>>
>>
>> *mvn clean*
>>
>> [INFO] Scanning for projects...
>>
>> [INFO]
>>
>> [INFO] ---------------------------< spark:MichTest
>> >---------------------------
>>
>> [INFO] Building MichTest 1.0
>>
>> [INFO] --------------------------------[ jar
>> ]---------------------------------
>>
>> [INFO]
>>
>> [INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ MichTest ---
>>
>> [INFO] Deleting D:\temp\intellij\MichTest\target
>>
>> [INFO]
>> ------------------------------------------------------------------------
>>
>> [INFO] BUILD SUCCESS
>>
>> [INFO]
>> ------------------------------------------------------------------------
>>
>> [INFO] Total time: 4.451 s
>>
>> [INFO] Finished at: 2022-02-27T19:37:57Z
>>
>> [INFO]
>> ------------------------------------------------------------------------
>>
>> *mvn compile*
>>
>> [INFO] Scanning for projects...
>>
>> [INFO]
>>
>> [INFO] ---------------------------< spark:MichTest
>> >---------------------------
>>
>> [INFO] Building MichTest 1.0
>>
>> [INFO] --------------------------------[ jar
>> ]---------------------------------
>>
>> [INFO]
>>
>> [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @
>> MichTest ---
>>
>> [WARNING] Using platform encoding (Cp1252 actually) to copy filtered
>> resources, i.e. build is platform dependent!
>>
>> [INFO] Copying 0 resource
>>
>> [INFO]
>>
>> [INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @ MichTest
>> ---
>>
>> [INFO] Nothing to compile - all classes are up to date
>>
>> [INFO]
>> ------------------------------------------------------------------------
>>
>> [INFO] BUILD SUCCESS
>>
>> [INFO]
>> ------------------------------------------------------------------------
>>
>> [INFO] Total time: 1.242 s
>>
>> [INFO] Finished at: 2022-02-27T19:38:58Z
>>
>> [INFO]
>> ------------------------------------------------------------------------
>>
>> Now create the package
>>
>> *mvn package*
>> [INFO] Scanning for projects...
>> [INFO]
>> [INFO] ---------------------------< spark:MichTest
>> >---------------------------
>> [INFO] Building MichTest 1.0
>> [INFO] --------------------------------[ jar
>> ]---------------------------------
>> [INFO]
>> [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @
>> MichTest ---
>> [WARNING] Using platform encoding (Cp1252 actually) to copy filtered
>> resources, i.e. build is platform dependent!
>> [INFO] Copying 0 resource
>> [INFO]
>> [INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @ MichTest
>> ---
>> [INFO] Nothing to compile - all classes are up to date
>> [INFO]
>> [INFO] --- maven-resources-plugin:2.6:testResources
>> (default-testResources) @ MichTest ---
>> [WARNING] Using platform encoding (Cp1252 actually) to copy filtered
>> resources, i.e. build is platform dependent!
>> [INFO] skip non existing resourceDirectory
>> D:\temp\intellij\MichTest\src\test\resources
>> [INFO]
>> [INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) @
>> MichTest ---
>> [INFO] Nothing to compile - all classes are up to date
>> [INFO]
>> [INFO] --- maven-surefire-plugin:2.12.4:test (default-test) @ MichTest ---
>> [INFO] No tests to run.
>> [INFO]
>> [INFO] --- maven-jar-plugin:2.4:jar (default-jar) @ MichTest ---
>> [INFO] *Building jar:
>> D:\temp\intellij\MichTest\target\scala-2.11\MichTest-1.0.jar*
>> [INFO]
>> ------------------------------------------------------------------------
>> [INFO] BUILD SUCCESS
>> [INFO]
>> ------------------------------------------------------------------------
>> [INFO] Total time: 1.511 s
>> [INFO] Finished at: 2022-02-27T19:40:22Z
>> [INFO]
>> ------------------------------------------------------------------------
>>
>> sbt should work from the same directory as well.
>>
>> I find it easier* to package from the command line in Intellij
>> Terminal. *
>>
>> Let me know if this is a wrong approach or if there are better ways of
>> doing it.
>>
>> HTH
>>
>>
>> view my Linkedin profile
>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>
>>
>> https://en.everybodywiki.com/Mich_Talebzadeh
>>
>>
>>
>> *Disclaimer:* Use it at your own risk. Any and all responsibility for
>> any loss, damage or destruction of data or any other property which may
>> arise from relying on this email's technical content is explicitly
>> disclaimed. The author will in no case be liable for any monetary damages
>> arising from such loss, damage or destruction.
>>
>>
>>
>>
>> On Sun, 27 Feb 2022 at 10:05, Mich Talebzadeh <mi...@gmail.com>
>> wrote:
>>
>>> Got curious with this intellij stuff.
>>>
>>> I recall using sbt rather than MVN so go to terminal in your intellij
>>> and verify what is installed
>>>
>>> sbt -version
>>> sbt version in this project: 1.3.4
>>> sbt script version: 1.3.4
>>>
>>> scala -version
>>>
>>> Scala code runner version 2.11.7 -- Copyright 2002-2013, LAMP/EPFL
>>>
>>> java --version
>>> openjdk 11.0.7 2020-04-14
>>> OpenJDK Runtime Environment AdoptOpenJDK (build 11.0.7+10)
>>> OpenJDK 64-Bit Server VM AdoptOpenJDK (build 11.0.7+10, mixed mode)
>>>
>>> For now in the directory where you have Main.scala create build.sbt file
>>>
>>> // The simplest possible sbt build file is just one line:
>>>
>>> scalaVersion := "2.11.7"
>>> // That is, to create a valid sbt build, all you've got to do is define the
>>> // version of Scala you'd like your project to use.
>>>
>>> / To learn more about multi-project builds, head over to the official sbt
>>> // documentation at http://www.scala-sbt.org/documentation.html
>>> libraryDependencies += "org.scala-lang.modules" %% "scala-parser-combinators" % "1.1.2"
>>> libraryDependencies += "org.apache.spark" %% "spark-sql" % "2.1.0"
>>>
>>>
>>> This is my Main.scala file a copy from sparkbyexample
>>> <https://sparkbyexamples.com/spark/spark-setup-run-with-scala-intellij/>
>>>
>>>
>>> package org.example
>>> import org.apache.spark.sql.SparkSession
>>> object SparkSessionTest extends App{
>>> val spark = SparkSession.builder()
>>> .master("local[1]")
>>> .appName("SparkByExample")
>>> .getOrCreate();
>>>
>>> println("First SparkContext:")
>>> println("APP Name :"+spark.sparkContext.appName);
>>> println("Deploy Mode :"+spark.sparkContext.deployMode);
>>> println("Master :"+spark.sparkContext.master);
>>>
>>> val sparkSession2 = SparkSession.builder()
>>> .master("local[1]")
>>> .appName("SparkByExample-test")
>>> .getOrCreate();
>>>
>>> println("Second SparkContext:")
>>> println("APP Name :"+sparkSession2.sparkContext.appName);
>>> println("Deploy Mode :"+sparkSession2.sparkContext.deployMode);
>>> println("Master :"+sparkSession2.sparkContext.master);
>>> }
>>>
>>> Go back to Terminal under directory where you have both files built.sbt
>>> and Main.scala
>>>
>>>
>>> *sbt clean*
>>>
>>> [info] Loading global plugins from C:\Users\admin\.sbt\1.0\plugins
>>>
>>> [info] Loading project definition from
>>> D:\temp\intellij\MichTest\src\main\scala\com\ctp\training\scala\project
>>>
>>> [info] Loading settings for project scala from build.sbt ...
>>>
>>> [info] Set current project to MichTest (in build
>>> file:/D:/temp/intellij/MichTest/src/main/scala/com/ctp/training/scala/)
>>>
>>> [success] Total time: 0 s, completed Feb 27, 2022 9:54:10 AM
>>>
>>> *sbt compile*
>>> [info] Loading global plugins from C:\Users\admin\.sbt\1.0\plugins
>>> [info] Loading project definition from
>>> D:\temp\intellij\MichTest\src\main\scala\com\ctp\training\scala\project
>>> [info] Loading settings for project scala from build.sbt ...
>>> [info] Set current project to MichTest (in build
>>> file:/D:/temp/intellij/MichTest/src/main/scala/com/ctp/training/scala/)
>>> [info] Executing in batch mode. For better performance use sbt's shell
>>> [warn] There may be incompatibilities among your library dependencies;
>>> run 'evicted' to see detailed eviction warnings.
>>> [info] Compiling 1 Scala source to
>>> D:\temp\intellij\MichTest\src\main\scala\com\ctp\training\scala\target\scala-2.11\classes
>>> ...
>>> [success] Total time: 5 s, completed Feb 27, 2022 9:55:10 AM
>>>
>>> *sbt package*
>>>
>>> [info] Loading global plugins from C:\Users\admin\.sbt\1.0\plugins
>>>
>>> [info] Loading project definition from
>>> D:\temp\intellij\MichTest\src\main\scala\com\ctp\training\scala\project
>>>
>>> [info] Loading settings for project scala from build.sbt ...
>>>
>>> [info] Set current project to MichTest (in build
>>> file:/D:/temp/intellij/MichTest/src/main/scala/com/ctp/training/scala/)
>>>
>>> [success] Total time: 1 s, completed Feb 27, 2022 9:56:48 AM
>>>
>>> *ls*
>>>
>>>
>>> Directory:
>>> D:\temp\intellij\MichTest\src\main\scala\com\ctp\training\scala
>>>
>>>
>>> Mode LastWriteTime Length Name
>>>
>>> ---- ------------- ------ ----
>>>
>>> d----- 2/27/2022 7:04 AM null
>>>
>>> d----- 2/27/2022 8:33 AM project
>>>
>>> d----- 2/27/2022 9:08 AM spark-warehouse
>>>
>>> d----- 2/27/2022 9:55 AM target
>>>
>>> -a---- 2/27/2022 9:17 AM 3511 build.sbt
>>>
>>> Note that you have target directory and underneath scala-2.11 (in my
>>> case) and the uber jar file michtest_2.11-1.0.jar
>>>
>>> *ls*
>>> Directory:
>>> D:\temp\intellij\MichTest\src\main\scala\com\ctp\training\scala\target\scala-2.11
>>> Mode LastWriteTime Length Name
>>> ---- ------------- ------ ----
>>> d----- 2/27/2022 9:55 AM classes
>>> d----- 2/27/2022 9:55 AM update
>>> -a---- 2/27/2022 9:56 AM 3938
>>> *michtest_2.11-1.0.jar*
>>>
>>>
>>> These are old stuff but still shows how to create a jar file with sbt
>>>
>>> HTH
>>>
>>>
>>> view my Linkedin profile
>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>>
>>>
>>>
>>> https://en.everybodywiki.com/Mich_Talebzadeh
>>>
>>>
>>>
>>> *Disclaimer:* Use it at your own risk. Any and all responsibility for
>>> any loss, damage or destruction of data or any other property which may
>>> arise from relying on this email's technical content is explicitly
>>> disclaimed. The author will in no case be liable for any monetary damages
>>> arising from such loss, damage or destruction.
>>>
>>>
>>>
>>>
>>> On Sat, 26 Feb 2022 at 22:48, Sean Owen <sr...@gmail.com> wrote:
>>>
>>>> I don't think any of that is related, no.
>>>> How are you dependencies set up? manually with IJ, or in a build file
>>>> (Maven, Gradle)? Normally you do the latter and dependencies are taken care
>>>> of for you, but you app would definitely have to express a dependency on
>>>> Scala libs.
>>>>
>>>> On Sat, Feb 26, 2022 at 4:25 PM Bitfox <bi...@bitfox.top> wrote:
>>>>
>>>>> Java SDK installed?
>>>>>
>>>>> On Sun, Feb 27, 2022 at 5:39 AM Sachit Murarka <
>>>>> connectsachit@gmail.com> wrote:
>>>>>
>>>>>> Hello ,
>>>>>>
>>>>>> Thanks for replying. I have installed Scala plugin in IntelliJ first
>>>>>> then also it's giving same error
>>>>>>
>>>>>> Cannot find project Scala library 2.12.12 for module SparkSimpleApp
>>>>>>
>>>>>> Thanks
>>>>>> Rajat
>>>>>>
>>>>>> On Sun, Feb 27, 2022, 00:52 Bitfox <bi...@bitfox.top> wrote:
>>>>>>
>>>>>>> You need to install scala first, the current version for spark is
>>>>>>> 2.12.15
>>>>>>> I would suggest you install scala by sdk which works great.
>>>>>>>
>>>>>>> Thanks
>>>>>>>
>>>>>>> On Sun, Feb 27, 2022 at 12:10 AM rajat kumar <
>>>>>>> kumar.rajat20del@gmail.com> wrote:
>>>>>>>
>>>>>>>> Hello Users,
>>>>>>>>
>>>>>>>> I am trying to create spark application using Scala(Intellij).
>>>>>>>> I have installed Scala plugin in intelliJ still getting below
>>>>>>>> error:-
>>>>>>>>
>>>>>>>> Cannot find project Scala library 2.12.12 for module SparkSimpleApp
>>>>>>>>
>>>>>>>>
>>>>>>>> Could anyone please help what I am doing wrong?
>>>>>>>>
>>>>>>>> Thanks
>>>>>>>>
>>>>>>>> Rajat
>>>>>>>>
>>>>>>>
>
> --
> Bjørn Jørgensen
> Vestre Aspehaug 4, 6010 Ålesund
> Norge
>
> +47 480 94 297
>
Re: Issue while creating spark app
Posted by Bjørn Jørgensen <bj...@gmail.com>.
Mitch: You are using scala 2.11 to do this. Have a look at Building Spark
<https://spark.apache.org/docs/latest/building-spark.html> "Spark requires
Scala 2.12/2.13; support for Scala 2.11 was removed in Spark 3.0.0."
søn. 27. feb. 2022 kl. 20:55 skrev Mich Talebzadeh <
mich.talebzadeh@gmail.com>:
> OK I decided to give a try to maven.
>
> Downloaded maven and unzipped the file WSL-Ubuntu terminal as unzip
> apache-maven-3.8.4-bin.zip
>
> Then added to Windows env variable as MVN_HOME and added the bin directory
> to path in windows. Restart intellij to pick up the correct path.
>
> Again on the command line in intellij do
>
> *mvn -v*
> Apache Maven 3.8.4 (9b656c72d54e5bacbed989b64718c159fe39b537)
> Maven home: d:\temp\apache-maven-3.8.4
> Java version: 1.8.0_73, vendor: Oracle Corporation, runtime: C:\Program
> Files\Java\jdk1.8.0_73\jre
> Default locale: en_GB, platform encoding: Cp1252
> OS name: "windows 10", version: "10.0", arch: "amd64", family: "windows"
>
> in Intellij add maven support to your project. Follow this link Add Maven
> support to an existing project
> <https://www.jetbrains.com/help/idea/convert-a-regular-project-into-a-maven-project.html>
>
> There will be a pom.xml file under project directory
>
> [image: image.png]
>
> Edit that pom.xml file and add the following
>
> <?xml version="1.0" encoding="UTF-8"?>
> <project xmlns="http://maven.apache.org/POM/4.0.0"
> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
> xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
> <modelVersion>4.0.0</modelVersion>
>
> <groupId>spark</groupId>
> <artifactId>MichTest</artifactId>
> <version>1.0</version>
>
> <properties>
> <maven.compiler.source>8</maven.compiler.source>
> <maven.compiler.target>8</maven.compiler.target>
> </properties>
> <dependencies>
> <dependency>
> <groupId>org.scala-lang</groupId>
> <artifactId>scala-library</artifactId>
> <version>2.11.7</version>
> </dependency>
> <dependency>
> <groupId>org.apache.spark</groupId>
> <artifactId>spark-core_2.10</artifactId>
> <version>2.0.0</version>
> </dependency>
> <dependency>
> <groupId>org.apache.spark</groupId>
> <artifactId>spark-sql_2.10</artifactId>
> <version>2.0.0</version>
> </dependency>
> </dependencies>
> </project>
>
> In intellij open a Terminal under project sub-directory where the pom file
> is created and you edited.
>
>
> *mvn clean*
>
> [INFO] Scanning for projects...
>
> [INFO]
>
> [INFO] ---------------------------< spark:MichTest
> >---------------------------
>
> [INFO] Building MichTest 1.0
>
> [INFO] --------------------------------[ jar
> ]---------------------------------
>
> [INFO]
>
> [INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ MichTest ---
>
> [INFO] Deleting D:\temp\intellij\MichTest\target
>
> [INFO]
> ------------------------------------------------------------------------
>
> [INFO] BUILD SUCCESS
>
> [INFO]
> ------------------------------------------------------------------------
>
> [INFO] Total time: 4.451 s
>
> [INFO] Finished at: 2022-02-27T19:37:57Z
>
> [INFO]
> ------------------------------------------------------------------------
>
> *mvn compile*
>
> [INFO] Scanning for projects...
>
> [INFO]
>
> [INFO] ---------------------------< spark:MichTest
> >---------------------------
>
> [INFO] Building MichTest 1.0
>
> [INFO] --------------------------------[ jar
> ]---------------------------------
>
> [INFO]
>
> [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @
> MichTest ---
>
> [WARNING] Using platform encoding (Cp1252 actually) to copy filtered
> resources, i.e. build is platform dependent!
>
> [INFO] Copying 0 resource
>
> [INFO]
>
> [INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @ MichTest
> ---
>
> [INFO] Nothing to compile - all classes are up to date
>
> [INFO]
> ------------------------------------------------------------------------
>
> [INFO] BUILD SUCCESS
>
> [INFO]
> ------------------------------------------------------------------------
>
> [INFO] Total time: 1.242 s
>
> [INFO] Finished at: 2022-02-27T19:38:58Z
>
> [INFO]
> ------------------------------------------------------------------------
>
> Now create the package
>
> *mvn package*
> [INFO] Scanning for projects...
> [INFO]
> [INFO] ---------------------------< spark:MichTest
> >---------------------------
> [INFO] Building MichTest 1.0
> [INFO] --------------------------------[ jar
> ]---------------------------------
> [INFO]
> [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @
> MichTest ---
> [WARNING] Using platform encoding (Cp1252 actually) to copy filtered
> resources, i.e. build is platform dependent!
> [INFO] Copying 0 resource
> [INFO]
> [INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @ MichTest
> ---
> [INFO] Nothing to compile - all classes are up to date
> [INFO]
> [INFO] --- maven-resources-plugin:2.6:testResources
> (default-testResources) @ MichTest ---
> [WARNING] Using platform encoding (Cp1252 actually) to copy filtered
> resources, i.e. build is platform dependent!
> [INFO] skip non existing resourceDirectory
> D:\temp\intellij\MichTest\src\test\resources
> [INFO]
> [INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) @
> MichTest ---
> [INFO] Nothing to compile - all classes are up to date
> [INFO]
> [INFO] --- maven-surefire-plugin:2.12.4:test (default-test) @ MichTest ---
> [INFO] No tests to run.
> [INFO]
> [INFO] --- maven-jar-plugin:2.4:jar (default-jar) @ MichTest ---
> [INFO] *Building jar:
> D:\temp\intellij\MichTest\target\scala-2.11\MichTest-1.0.jar*
> [INFO]
> ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO]
> ------------------------------------------------------------------------
> [INFO] Total time: 1.511 s
> [INFO] Finished at: 2022-02-27T19:40:22Z
> [INFO]
> ------------------------------------------------------------------------
>
> sbt should work from the same directory as well.
>
> I find it easier* to package from the command line in Intellij Terminal. *
>
> Let me know if this is a wrong approach or if there are better ways of
> doing it.
>
> HTH
>
>
> view my Linkedin profile
> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>
>
> https://en.everybodywiki.com/Mich_Talebzadeh
>
>
>
> *Disclaimer:* Use it at your own risk. Any and all responsibility for any
> loss, damage or destruction of data or any other property which may arise
> from relying on this email's technical content is explicitly disclaimed.
> The author will in no case be liable for any monetary damages arising from
> such loss, damage or destruction.
>
>
>
>
> On Sun, 27 Feb 2022 at 10:05, Mich Talebzadeh <mi...@gmail.com>
> wrote:
>
>> Got curious with this intellij stuff.
>>
>> I recall using sbt rather than MVN so go to terminal in your intellij and
>> verify what is installed
>>
>> sbt -version
>> sbt version in this project: 1.3.4
>> sbt script version: 1.3.4
>>
>> scala -version
>>
>> Scala code runner version 2.11.7 -- Copyright 2002-2013, LAMP/EPFL
>>
>> java --version
>> openjdk 11.0.7 2020-04-14
>> OpenJDK Runtime Environment AdoptOpenJDK (build 11.0.7+10)
>> OpenJDK 64-Bit Server VM AdoptOpenJDK (build 11.0.7+10, mixed mode)
>>
>> For now in the directory where you have Main.scala create build.sbt file
>>
>> // The simplest possible sbt build file is just one line:
>>
>> scalaVersion := "2.11.7"
>> // That is, to create a valid sbt build, all you've got to do is define the
>> // version of Scala you'd like your project to use.
>>
>> / To learn more about multi-project builds, head over to the official sbt
>> // documentation at http://www.scala-sbt.org/documentation.html
>> libraryDependencies += "org.scala-lang.modules" %% "scala-parser-combinators" % "1.1.2"
>> libraryDependencies += "org.apache.spark" %% "spark-sql" % "2.1.0"
>>
>>
>> This is my Main.scala file a copy from sparkbyexample
>> <https://sparkbyexamples.com/spark/spark-setup-run-with-scala-intellij/>
>>
>>
>> package org.example
>> import org.apache.spark.sql.SparkSession
>> object SparkSessionTest extends App{
>> val spark = SparkSession.builder()
>> .master("local[1]")
>> .appName("SparkByExample")
>> .getOrCreate();
>>
>> println("First SparkContext:")
>> println("APP Name :"+spark.sparkContext.appName);
>> println("Deploy Mode :"+spark.sparkContext.deployMode);
>> println("Master :"+spark.sparkContext.master);
>>
>> val sparkSession2 = SparkSession.builder()
>> .master("local[1]")
>> .appName("SparkByExample-test")
>> .getOrCreate();
>>
>> println("Second SparkContext:")
>> println("APP Name :"+sparkSession2.sparkContext.appName);
>> println("Deploy Mode :"+sparkSession2.sparkContext.deployMode);
>> println("Master :"+sparkSession2.sparkContext.master);
>> }
>>
>> Go back to Terminal under directory where you have both files built.sbt
>> and Main.scala
>>
>>
>> *sbt clean*
>>
>> [info] Loading global plugins from C:\Users\admin\.sbt\1.0\plugins
>>
>> [info] Loading project definition from
>> D:\temp\intellij\MichTest\src\main\scala\com\ctp\training\scala\project
>>
>> [info] Loading settings for project scala from build.sbt ...
>>
>> [info] Set current project to MichTest (in build
>> file:/D:/temp/intellij/MichTest/src/main/scala/com/ctp/training/scala/)
>>
>> [success] Total time: 0 s, completed Feb 27, 2022 9:54:10 AM
>>
>> *sbt compile*
>> [info] Loading global plugins from C:\Users\admin\.sbt\1.0\plugins
>> [info] Loading project definition from
>> D:\temp\intellij\MichTest\src\main\scala\com\ctp\training\scala\project
>> [info] Loading settings for project scala from build.sbt ...
>> [info] Set current project to MichTest (in build
>> file:/D:/temp/intellij/MichTest/src/main/scala/com/ctp/training/scala/)
>> [info] Executing in batch mode. For better performance use sbt's shell
>> [warn] There may be incompatibilities among your library dependencies;
>> run 'evicted' to see detailed eviction warnings.
>> [info] Compiling 1 Scala source to
>> D:\temp\intellij\MichTest\src\main\scala\com\ctp\training\scala\target\scala-2.11\classes
>> ...
>> [success] Total time: 5 s, completed Feb 27, 2022 9:55:10 AM
>>
>> *sbt package*
>>
>> [info] Loading global plugins from C:\Users\admin\.sbt\1.0\plugins
>>
>> [info] Loading project definition from
>> D:\temp\intellij\MichTest\src\main\scala\com\ctp\training\scala\project
>>
>> [info] Loading settings for project scala from build.sbt ...
>>
>> [info] Set current project to MichTest (in build
>> file:/D:/temp/intellij/MichTest/src/main/scala/com/ctp/training/scala/)
>>
>> [success] Total time: 1 s, completed Feb 27, 2022 9:56:48 AM
>>
>> *ls*
>>
>>
>> Directory:
>> D:\temp\intellij\MichTest\src\main\scala\com\ctp\training\scala
>>
>>
>> Mode LastWriteTime Length Name
>>
>> ---- ------------- ------ ----
>>
>> d----- 2/27/2022 7:04 AM null
>>
>> d----- 2/27/2022 8:33 AM project
>>
>> d----- 2/27/2022 9:08 AM spark-warehouse
>>
>> d----- 2/27/2022 9:55 AM target
>>
>> -a---- 2/27/2022 9:17 AM 3511 build.sbt
>>
>> Note that you have target directory and underneath scala-2.11 (in my
>> case) and the uber jar file michtest_2.11-1.0.jar
>>
>> *ls*
>> Directory:
>> D:\temp\intellij\MichTest\src\main\scala\com\ctp\training\scala\target\scala-2.11
>> Mode LastWriteTime Length Name
>> ---- ------------- ------ ----
>> d----- 2/27/2022 9:55 AM classes
>> d----- 2/27/2022 9:55 AM update
>> -a---- 2/27/2022 9:56 AM 3938 *michtest_2.11-1.0.jar*
>>
>>
>> These are old stuff but still shows how to create a jar file with sbt
>>
>> HTH
>>
>>
>> view my Linkedin profile
>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>
>>
>>
>> https://en.everybodywiki.com/Mich_Talebzadeh
>>
>>
>>
>> *Disclaimer:* Use it at your own risk. Any and all responsibility for
>> any loss, damage or destruction of data or any other property which may
>> arise from relying on this email's technical content is explicitly
>> disclaimed. The author will in no case be liable for any monetary damages
>> arising from such loss, damage or destruction.
>>
>>
>>
>>
>> On Sat, 26 Feb 2022 at 22:48, Sean Owen <sr...@gmail.com> wrote:
>>
>>> I don't think any of that is related, no.
>>> How are you dependencies set up? manually with IJ, or in a build file
>>> (Maven, Gradle)? Normally you do the latter and dependencies are taken care
>>> of for you, but you app would definitely have to express a dependency on
>>> Scala libs.
>>>
>>> On Sat, Feb 26, 2022 at 4:25 PM Bitfox <bi...@bitfox.top> wrote:
>>>
>>>> Java SDK installed?
>>>>
>>>> On Sun, Feb 27, 2022 at 5:39 AM Sachit Murarka <co...@gmail.com>
>>>> wrote:
>>>>
>>>>> Hello ,
>>>>>
>>>>> Thanks for replying. I have installed Scala plugin in IntelliJ first
>>>>> then also it's giving same error
>>>>>
>>>>> Cannot find project Scala library 2.12.12 for module SparkSimpleApp
>>>>>
>>>>> Thanks
>>>>> Rajat
>>>>>
>>>>> On Sun, Feb 27, 2022, 00:52 Bitfox <bi...@bitfox.top> wrote:
>>>>>
>>>>>> You need to install scala first, the current version for spark is
>>>>>> 2.12.15
>>>>>> I would suggest you install scala by sdk which works great.
>>>>>>
>>>>>> Thanks
>>>>>>
>>>>>> On Sun, Feb 27, 2022 at 12:10 AM rajat kumar <
>>>>>> kumar.rajat20del@gmail.com> wrote:
>>>>>>
>>>>>>> Hello Users,
>>>>>>>
>>>>>>> I am trying to create spark application using Scala(Intellij).
>>>>>>> I have installed Scala plugin in intelliJ still getting below error:-
>>>>>>>
>>>>>>> Cannot find project Scala library 2.12.12 for module SparkSimpleApp
>>>>>>>
>>>>>>>
>>>>>>> Could anyone please help what I am doing wrong?
>>>>>>>
>>>>>>> Thanks
>>>>>>>
>>>>>>> Rajat
>>>>>>>
>>>>>>
--
Bjørn Jørgensen
Vestre Aspehaug 4, 6010 Ålesund
Norge
+47 480 94 297
Re: Issue while creating spark app
Posted by Mich Talebzadeh <mi...@gmail.com>.
OK I decided to give a try to maven.
Downloaded maven and unzipped the file WSL-Ubuntu terminal as unzip
apache-maven-3.8.4-bin.zip
Then added to Windows env variable as MVN_HOME and added the bin directory
to path in windows. Restart intellij to pick up the correct path.
Again on the command line in intellij do
*mvn -v*
Apache Maven 3.8.4 (9b656c72d54e5bacbed989b64718c159fe39b537)
Maven home: d:\temp\apache-maven-3.8.4
Java version: 1.8.0_73, vendor: Oracle Corporation, runtime: C:\Program
Files\Java\jdk1.8.0_73\jre
Default locale: en_GB, platform encoding: Cp1252
OS name: "windows 10", version: "10.0", arch: "amd64", family: "windows"
in Intellij add maven support to your project. Follow this link Add Maven
support to an existing project
<https://www.jetbrains.com/help/idea/convert-a-regular-project-into-a-maven-project.html>
There will be a pom.xml file under project directory
[image: image.png]
Edit that pom.xml file and add the following
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>spark</groupId>
<artifactId>MichTest</artifactId>
<version>1.0</version>
<properties>
<maven.compiler.source>8</maven.compiler.source>
<maven.compiler.target>8</maven.compiler.target>
</properties>
<dependencies>
<dependency>
<groupId>org.scala-lang</groupId>
<artifactId>scala-library</artifactId>
<version>2.11.7</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.10</artifactId>
<version>2.0.0</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_2.10</artifactId>
<version>2.0.0</version>
</dependency>
</dependencies>
</project>
In intellij open a Terminal under project sub-directory where the pom file
is created and you edited.
*mvn clean*
[INFO] Scanning for projects...
[INFO]
[INFO] ---------------------------< spark:MichTest
>---------------------------
[INFO] Building MichTest 1.0
[INFO] --------------------------------[ jar
]---------------------------------
[INFO]
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ MichTest ---
[INFO] Deleting D:\temp\intellij\MichTest\target
[INFO]
------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO]
------------------------------------------------------------------------
[INFO] Total time: 4.451 s
[INFO] Finished at: 2022-02-27T19:37:57Z
[INFO]
------------------------------------------------------------------------
*mvn compile*
[INFO] Scanning for projects...
[INFO]
[INFO] ---------------------------< spark:MichTest
>---------------------------
[INFO] Building MichTest 1.0
[INFO] --------------------------------[ jar
]---------------------------------
[INFO]
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @
MichTest ---
[WARNING] Using platform encoding (Cp1252 actually) to copy filtered
resources, i.e. build is platform dependent!
[INFO] Copying 0 resource
[INFO]
[INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @ MichTest
---
[INFO] Nothing to compile - all classes are up to date
[INFO]
------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO]
------------------------------------------------------------------------
[INFO] Total time: 1.242 s
[INFO] Finished at: 2022-02-27T19:38:58Z
[INFO]
------------------------------------------------------------------------
Now create the package
*mvn package*
[INFO] Scanning for projects...
[INFO]
[INFO] ---------------------------< spark:MichTest
>---------------------------
[INFO] Building MichTest 1.0
[INFO] --------------------------------[ jar
]---------------------------------
[INFO]
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @
MichTest ---
[WARNING] Using platform encoding (Cp1252 actually) to copy filtered
resources, i.e. build is platform dependent!
[INFO] Copying 0 resource
[INFO]
[INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @ MichTest
---
[INFO] Nothing to compile - all classes are up to date
[INFO]
[INFO] --- maven-resources-plugin:2.6:testResources (default-testResources)
@ MichTest ---
[WARNING] Using platform encoding (Cp1252 actually) to copy filtered
resources, i.e. build is platform dependent!
[INFO] skip non existing resourceDirectory
D:\temp\intellij\MichTest\src\test\resources
[INFO]
[INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) @
MichTest ---
[INFO] Nothing to compile - all classes are up to date
[INFO]
[INFO] --- maven-surefire-plugin:2.12.4:test (default-test) @ MichTest ---
[INFO] No tests to run.
[INFO]
[INFO] --- maven-jar-plugin:2.4:jar (default-jar) @ MichTest ---
[INFO] *Building jar:
D:\temp\intellij\MichTest\target\scala-2.11\MichTest-1.0.jar*
[INFO]
------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO]
------------------------------------------------------------------------
[INFO] Total time: 1.511 s
[INFO] Finished at: 2022-02-27T19:40:22Z
[INFO]
------------------------------------------------------------------------
sbt should work from the same directory as well.
I find it easier* to package from the command line in Intellij Terminal. *
Let me know if this is a wrong approach or if there are better ways of
doing it.
HTH
view my Linkedin profile
<https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
https://en.everybodywiki.com/Mich_Talebzadeh
*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.
On Sun, 27 Feb 2022 at 10:05, Mich Talebzadeh <mi...@gmail.com>
wrote:
> Got curious with this intellij stuff.
>
> I recall using sbt rather than MVN so go to terminal in your intellij and
> verify what is installed
>
> sbt -version
> sbt version in this project: 1.3.4
> sbt script version: 1.3.4
>
> scala -version
>
> Scala code runner version 2.11.7 -- Copyright 2002-2013, LAMP/EPFL
>
> java --version
> openjdk 11.0.7 2020-04-14
> OpenJDK Runtime Environment AdoptOpenJDK (build 11.0.7+10)
> OpenJDK 64-Bit Server VM AdoptOpenJDK (build 11.0.7+10, mixed mode)
>
> For now in the directory where you have Main.scala create build.sbt file
>
> // The simplest possible sbt build file is just one line:
>
> scalaVersion := "2.11.7"
> // That is, to create a valid sbt build, all you've got to do is define the
> // version of Scala you'd like your project to use.
>
> / To learn more about multi-project builds, head over to the official sbt
> // documentation at http://www.scala-sbt.org/documentation.html
> libraryDependencies += "org.scala-lang.modules" %% "scala-parser-combinators" % "1.1.2"
> libraryDependencies += "org.apache.spark" %% "spark-sql" % "2.1.0"
>
>
> This is my Main.scala file a copy from sparkbyexample
> <https://sparkbyexamples.com/spark/spark-setup-run-with-scala-intellij/>
>
>
> package org.example
> import org.apache.spark.sql.SparkSession
> object SparkSessionTest extends App{
> val spark = SparkSession.builder()
> .master("local[1]")
> .appName("SparkByExample")
> .getOrCreate();
>
> println("First SparkContext:")
> println("APP Name :"+spark.sparkContext.appName);
> println("Deploy Mode :"+spark.sparkContext.deployMode);
> println("Master :"+spark.sparkContext.master);
>
> val sparkSession2 = SparkSession.builder()
> .master("local[1]")
> .appName("SparkByExample-test")
> .getOrCreate();
>
> println("Second SparkContext:")
> println("APP Name :"+sparkSession2.sparkContext.appName);
> println("Deploy Mode :"+sparkSession2.sparkContext.deployMode);
> println("Master :"+sparkSession2.sparkContext.master);
> }
>
> Go back to Terminal under directory where you have both files built.sbt
> and Main.scala
>
>
> *sbt clean*
>
> [info] Loading global plugins from C:\Users\admin\.sbt\1.0\plugins
>
> [info] Loading project definition from
> D:\temp\intellij\MichTest\src\main\scala\com\ctp\training\scala\project
>
> [info] Loading settings for project scala from build.sbt ...
>
> [info] Set current project to MichTest (in build
> file:/D:/temp/intellij/MichTest/src/main/scala/com/ctp/training/scala/)
>
> [success] Total time: 0 s, completed Feb 27, 2022 9:54:10 AM
>
> *sbt compile*
> [info] Loading global plugins from C:\Users\admin\.sbt\1.0\plugins
> [info] Loading project definition from
> D:\temp\intellij\MichTest\src\main\scala\com\ctp\training\scala\project
> [info] Loading settings for project scala from build.sbt ...
> [info] Set current project to MichTest (in build
> file:/D:/temp/intellij/MichTest/src/main/scala/com/ctp/training/scala/)
> [info] Executing in batch mode. For better performance use sbt's shell
> [warn] There may be incompatibilities among your library dependencies; run
> 'evicted' to see detailed eviction warnings.
> [info] Compiling 1 Scala source to
> D:\temp\intellij\MichTest\src\main\scala\com\ctp\training\scala\target\scala-2.11\classes
> ...
> [success] Total time: 5 s, completed Feb 27, 2022 9:55:10 AM
>
> *sbt package*
>
> [info] Loading global plugins from C:\Users\admin\.sbt\1.0\plugins
>
> [info] Loading project definition from
> D:\temp\intellij\MichTest\src\main\scala\com\ctp\training\scala\project
>
> [info] Loading settings for project scala from build.sbt ...
>
> [info] Set current project to MichTest (in build
> file:/D:/temp/intellij/MichTest/src/main/scala/com/ctp/training/scala/)
>
> [success] Total time: 1 s, completed Feb 27, 2022 9:56:48 AM
>
> *ls*
>
>
> Directory:
> D:\temp\intellij\MichTest\src\main\scala\com\ctp\training\scala
>
>
> Mode LastWriteTime Length Name
>
> ---- ------------- ------ ----
>
> d----- 2/27/2022 7:04 AM null
>
> d----- 2/27/2022 8:33 AM project
>
> d----- 2/27/2022 9:08 AM spark-warehouse
>
> d----- 2/27/2022 9:55 AM target
>
> -a---- 2/27/2022 9:17 AM 3511 build.sbt
>
> Note that you have target directory and underneath scala-2.11 (in my
> case) and the uber jar file michtest_2.11-1.0.jar
>
> *ls*
> Directory:
> D:\temp\intellij\MichTest\src\main\scala\com\ctp\training\scala\target\scala-2.11
> Mode LastWriteTime Length Name
> ---- ------------- ------ ----
> d----- 2/27/2022 9:55 AM classes
> d----- 2/27/2022 9:55 AM update
> -a---- 2/27/2022 9:56 AM 3938 *michtest_2.11-1.0.jar*
>
>
> These are old stuff but still shows how to create a jar file with sbt
>
> HTH
>
>
> view my Linkedin profile
> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>
>
>
> https://en.everybodywiki.com/Mich_Talebzadeh
>
>
>
> *Disclaimer:* Use it at your own risk. Any and all responsibility for any
> loss, damage or destruction of data or any other property which may arise
> from relying on this email's technical content is explicitly disclaimed.
> The author will in no case be liable for any monetary damages arising from
> such loss, damage or destruction.
>
>
>
>
> On Sat, 26 Feb 2022 at 22:48, Sean Owen <sr...@gmail.com> wrote:
>
>> I don't think any of that is related, no.
>> How are you dependencies set up? manually with IJ, or in a build file
>> (Maven, Gradle)? Normally you do the latter and dependencies are taken care
>> of for you, but you app would definitely have to express a dependency on
>> Scala libs.
>>
>> On Sat, Feb 26, 2022 at 4:25 PM Bitfox <bi...@bitfox.top> wrote:
>>
>>> Java SDK installed?
>>>
>>> On Sun, Feb 27, 2022 at 5:39 AM Sachit Murarka <co...@gmail.com>
>>> wrote:
>>>
>>>> Hello ,
>>>>
>>>> Thanks for replying. I have installed Scala plugin in IntelliJ first
>>>> then also it's giving same error
>>>>
>>>> Cannot find project Scala library 2.12.12 for module SparkSimpleApp
>>>>
>>>> Thanks
>>>> Rajat
>>>>
>>>> On Sun, Feb 27, 2022, 00:52 Bitfox <bi...@bitfox.top> wrote:
>>>>
>>>>> You need to install scala first, the current version for spark is
>>>>> 2.12.15
>>>>> I would suggest you install scala by sdk which works great.
>>>>>
>>>>> Thanks
>>>>>
>>>>> On Sun, Feb 27, 2022 at 12:10 AM rajat kumar <
>>>>> kumar.rajat20del@gmail.com> wrote:
>>>>>
>>>>>> Hello Users,
>>>>>>
>>>>>> I am trying to create spark application using Scala(Intellij).
>>>>>> I have installed Scala plugin in intelliJ still getting below error:-
>>>>>>
>>>>>> Cannot find project Scala library 2.12.12 for module SparkSimpleApp
>>>>>>
>>>>>>
>>>>>> Could anyone please help what I am doing wrong?
>>>>>>
>>>>>> Thanks
>>>>>>
>>>>>> Rajat
>>>>>>
>>>>>
Re: Issue while creating spark app
Posted by Mich Talebzadeh <mi...@gmail.com>.
Got curious with this intellij stuff.
I recall using sbt rather than MVN so go to terminal in your intellij and
verify what is installed
sbt -version
sbt version in this project: 1.3.4
sbt script version: 1.3.4
scala -version
Scala code runner version 2.11.7 -- Copyright 2002-2013, LAMP/EPFL
java --version
openjdk 11.0.7 2020-04-14
OpenJDK Runtime Environment AdoptOpenJDK (build 11.0.7+10)
OpenJDK 64-Bit Server VM AdoptOpenJDK (build 11.0.7+10, mixed mode)
For now in the directory where you have Main.scala create build.sbt file
// The simplest possible sbt build file is just one line:
scalaVersion := "2.11.7"
// That is, to create a valid sbt build, all you've got to do is define the
// version of Scala you'd like your project to use.
/ To learn more about multi-project builds, head over to the official sbt
// documentation at http://www.scala-sbt.org/documentation.html
libraryDependencies += "org.scala-lang.modules" %%
"scala-parser-combinators" % "1.1.2"
libraryDependencies += "org.apache.spark" %% "spark-sql" % "2.1.0"
This is my Main.scala file a copy from sparkbyexample
<https://sparkbyexamples.com/spark/spark-setup-run-with-scala-intellij/>
package org.example
import org.apache.spark.sql.SparkSession
object SparkSessionTest extends App{
val spark = SparkSession.builder()
.master("local[1]")
.appName("SparkByExample")
.getOrCreate();
println("First SparkContext:")
println("APP Name :"+spark.sparkContext.appName);
println("Deploy Mode :"+spark.sparkContext.deployMode);
println("Master :"+spark.sparkContext.master);
val sparkSession2 = SparkSession.builder()
.master("local[1]")
.appName("SparkByExample-test")
.getOrCreate();
println("Second SparkContext:")
println("APP Name :"+sparkSession2.sparkContext.appName);
println("Deploy Mode :"+sparkSession2.sparkContext.deployMode);
println("Master :"+sparkSession2.sparkContext.master);
}
Go back to Terminal under directory where you have both files built.sbt
and Main.scala
*sbt clean*
[info] Loading global plugins from C:\Users\admin\.sbt\1.0\plugins
[info] Loading project definition from
D:\temp\intellij\MichTest\src\main\scala\com\ctp\training\scala\project
[info] Loading settings for project scala from build.sbt ...
[info] Set current project to MichTest (in build
file:/D:/temp/intellij/MichTest/src/main/scala/com/ctp/training/scala/)
[success] Total time: 0 s, completed Feb 27, 2022 9:54:10 AM
*sbt compile*
[info] Loading global plugins from C:\Users\admin\.sbt\1.0\plugins
[info] Loading project definition from
D:\temp\intellij\MichTest\src\main\scala\com\ctp\training\scala\project
[info] Loading settings for project scala from build.sbt ...
[info] Set current project to MichTest (in build
file:/D:/temp/intellij/MichTest/src/main/scala/com/ctp/training/scala/)
[info] Executing in batch mode. For better performance use sbt's shell
[warn] There may be incompatibilities among your library dependencies; run
'evicted' to see detailed eviction warnings.
[info] Compiling 1 Scala source to
D:\temp\intellij\MichTest\src\main\scala\com\ctp\training\scala\target\scala-2.11\classes
...
[success] Total time: 5 s, completed Feb 27, 2022 9:55:10 AM
*sbt package*
[info] Loading global plugins from C:\Users\admin\.sbt\1.0\plugins
[info] Loading project definition from
D:\temp\intellij\MichTest\src\main\scala\com\ctp\training\scala\project
[info] Loading settings for project scala from build.sbt ...
[info] Set current project to MichTest (in build
file:/D:/temp/intellij/MichTest/src/main/scala/com/ctp/training/scala/)
[success] Total time: 1 s, completed Feb 27, 2022 9:56:48 AM
*ls*
Directory:
D:\temp\intellij\MichTest\src\main\scala\com\ctp\training\scala
Mode LastWriteTime Length Name
---- ------------- ------ ----
d----- 2/27/2022 7:04 AM null
d----- 2/27/2022 8:33 AM project
d----- 2/27/2022 9:08 AM spark-warehouse
d----- 2/27/2022 9:55 AM target
-a---- 2/27/2022 9:17 AM 3511 build.sbt
Note that you have target directory and underneath scala-2.11 (in my case)
and the uber jar file michtest_2.11-1.0.jar
*ls*
Directory:
D:\temp\intellij\MichTest\src\main\scala\com\ctp\training\scala\target\scala-2.11
Mode LastWriteTime Length Name
---- ------------- ------ ----
d----- 2/27/2022 9:55 AM classes
d----- 2/27/2022 9:55 AM update
-a---- 2/27/2022 9:56 AM 3938 *michtest_2.11-1.0.jar*
These are old stuff but still shows how to create a jar file with sbt
HTH
view my Linkedin profile
<https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
https://en.everybodywiki.com/Mich_Talebzadeh
*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.
On Sat, 26 Feb 2022 at 22:48, Sean Owen <sr...@gmail.com> wrote:
> I don't think any of that is related, no.
> How are you dependencies set up? manually with IJ, or in a build file
> (Maven, Gradle)? Normally you do the latter and dependencies are taken care
> of for you, but you app would definitely have to express a dependency on
> Scala libs.
>
> On Sat, Feb 26, 2022 at 4:25 PM Bitfox <bi...@bitfox.top> wrote:
>
>> Java SDK installed?
>>
>> On Sun, Feb 27, 2022 at 5:39 AM Sachit Murarka <co...@gmail.com>
>> wrote:
>>
>>> Hello ,
>>>
>>> Thanks for replying. I have installed Scala plugin in IntelliJ first
>>> then also it's giving same error
>>>
>>> Cannot find project Scala library 2.12.12 for module SparkSimpleApp
>>>
>>> Thanks
>>> Rajat
>>>
>>> On Sun, Feb 27, 2022, 00:52 Bitfox <bi...@bitfox.top> wrote:
>>>
>>>> You need to install scala first, the current version for spark is
>>>> 2.12.15
>>>> I would suggest you install scala by sdk which works great.
>>>>
>>>> Thanks
>>>>
>>>> On Sun, Feb 27, 2022 at 12:10 AM rajat kumar <
>>>> kumar.rajat20del@gmail.com> wrote:
>>>>
>>>>> Hello Users,
>>>>>
>>>>> I am trying to create spark application using Scala(Intellij).
>>>>> I have installed Scala plugin in intelliJ still getting below error:-
>>>>>
>>>>> Cannot find project Scala library 2.12.12 for module SparkSimpleApp
>>>>>
>>>>>
>>>>> Could anyone please help what I am doing wrong?
>>>>>
>>>>> Thanks
>>>>>
>>>>> Rajat
>>>>>
>>>>
Re: Issue while creating spark app
Posted by Sean Owen <sr...@gmail.com>.
I don't think any of that is related, no.
How are you dependencies set up? manually with IJ, or in a build file
(Maven, Gradle)? Normally you do the latter and dependencies are taken care
of for you, but you app would definitely have to express a dependency on
Scala libs.
On Sat, Feb 26, 2022 at 4:25 PM Bitfox <bi...@bitfox.top> wrote:
> Java SDK installed?
>
> On Sun, Feb 27, 2022 at 5:39 AM Sachit Murarka <co...@gmail.com>
> wrote:
>
>> Hello ,
>>
>> Thanks for replying. I have installed Scala plugin in IntelliJ first
>> then also it's giving same error
>>
>> Cannot find project Scala library 2.12.12 for module SparkSimpleApp
>>
>> Thanks
>> Rajat
>>
>> On Sun, Feb 27, 2022, 00:52 Bitfox <bi...@bitfox.top> wrote:
>>
>>> You need to install scala first, the current version for spark is 2.12.15
>>> I would suggest you install scala by sdk which works great.
>>>
>>> Thanks
>>>
>>> On Sun, Feb 27, 2022 at 12:10 AM rajat kumar <ku...@gmail.com>
>>> wrote:
>>>
>>>> Hello Users,
>>>>
>>>> I am trying to create spark application using Scala(Intellij).
>>>> I have installed Scala plugin in intelliJ still getting below error:-
>>>>
>>>> Cannot find project Scala library 2.12.12 for module SparkSimpleApp
>>>>
>>>>
>>>> Could anyone please help what I am doing wrong?
>>>>
>>>> Thanks
>>>>
>>>> Rajat
>>>>
>>>
Re: Issue while creating spark app
Posted by Bitfox <bi...@bitfox.top>.
Java SDK installed?
On Sun, Feb 27, 2022 at 5:39 AM Sachit Murarka <co...@gmail.com>
wrote:
> Hello ,
>
> Thanks for replying. I have installed Scala plugin in IntelliJ first then
> also it's giving same error
>
> Cannot find project Scala library 2.12.12 for module SparkSimpleApp
>
> Thanks
> Rajat
>
> On Sun, Feb 27, 2022, 00:52 Bitfox <bi...@bitfox.top> wrote:
>
>> You need to install scala first, the current version for spark is 2.12.15
>> I would suggest you install scala by sdk which works great.
>>
>> Thanks
>>
>> On Sun, Feb 27, 2022 at 12:10 AM rajat kumar <ku...@gmail.com>
>> wrote:
>>
>>> Hello Users,
>>>
>>> I am trying to create spark application using Scala(Intellij).
>>> I have installed Scala plugin in intelliJ still getting below error:-
>>>
>>> Cannot find project Scala library 2.12.12 for module SparkSimpleApp
>>>
>>>
>>> Could anyone please help what I am doing wrong?
>>>
>>> Thanks
>>>
>>> Rajat
>>>
>>
Re: Issue while creating spark app
Posted by Sachit Murarka <co...@gmail.com>.
Hello ,
Thanks for replying. I have installed Scala plugin in IntelliJ first then
also it's giving same error
Cannot find project Scala library 2.12.12 for module SparkSimpleApp
Thanks
Rajat
On Sun, Feb 27, 2022, 00:52 Bitfox <bi...@bitfox.top> wrote:
> You need to install scala first, the current version for spark is 2.12.15
> I would suggest you install scala by sdk which works great.
>
> Thanks
>
> On Sun, Feb 27, 2022 at 12:10 AM rajat kumar <ku...@gmail.com>
> wrote:
>
>> Hello Users,
>>
>> I am trying to create spark application using Scala(Intellij).
>> I have installed Scala plugin in intelliJ still getting below error:-
>>
>> Cannot find project Scala library 2.12.12 for module SparkSimpleApp
>>
>>
>> Could anyone please help what I am doing wrong?
>>
>> Thanks
>>
>> Rajat
>>
>
Re: Issue while creating spark app
Posted by Bitfox <bi...@bitfox.top>.
You need to install scala first, the current version for spark is 2.12.15
I would suggest you install scala by sdk which works great.
Thanks
On Sun, Feb 27, 2022 at 12:10 AM rajat kumar <ku...@gmail.com>
wrote:
> Hello Users,
>
> I am trying to create spark application using Scala(Intellij).
> I have installed Scala plugin in intelliJ still getting below error:-
>
> Cannot find project Scala library 2.12.12 for module SparkSimpleApp
>
>
> Could anyone please help what I am doing wrong?
>
> Thanks
>
> Rajat
>