Toree 0.1.0-incubating을 사용하는 경우 Spark 2.x.x를 지원하지 않는다

이는 Spark 2.x.x가 지원하는 Scala 버젼이 2.11.x이상인데 Toree 0.1.0-incubating은 Scala 2.10.4 버젼으로 building되었기 때문에 아래와 같은 NoSuchMethodError가 발생한다.

이런경우 Toree를 0.2.0-incubating을 사용하여 설치해주면 Scala 2.11.x이상을 지원하게되어 아래와 같은 오류는 없어진다.

1. spark2.2.0지원 가능한 버젼의 toree설치

pip3 install https://dist.apache.org/repos/dist/dev/incubator/toree/0.2.0/snapshots/dev1/toree-pip/toree-0.2.0.dev1.tar.gz

2. toree설치

pip3 install toree

3. jupyter의 kernel설정

jupyter toree install --spark_home=$SPARK_HOME --interpreters=Scala,PySpark,SparkR,SQL

------------------오류내용 예시#1---------------------

18/04/20 17:35:23 INFO Main$$anon$1: Kernel version: 0.1.0-incubating

18/04/20 17:35:23 INFO Main$$anon$1: Scala version: Some(2.10.4)

18/04/20 17:35:23 INFO Main$$anon$1: ZeroMQ (JeroMQ) version: 3.2.5

18/04/20 17:35:23 INFO Main$$anon$1: Initializing internal actor system

Exception in thread "main" java.lang.NoSuchMethodError: scala.collection.immutable.HashSet$.empty()Lscala/collection/immutable/HashSet;

        at akka.actor.ActorCell$.<init>(ActorCell.scala:336)

        at akka.actor.ActorCell$.<clinit>(ActorCell.scala)

        at akka.actor.RootActorPath.$div(ActorPath.scala:185)

        at akka.actor.LocalActorRefProvider.<init>(ActorRefProvider.scala:465)

        at akka.actor.LocalActorRefProvider.<init>(ActorRefProvider.scala:453)

        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)

        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)

        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)

        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)

        at akka.actor.ReflectiveDynamicAccess$$anonfun$createInstanceFor$2.apply(DynamicAccess.scala:78)

        at scala.util.Try$.apply(Try.scala:192)

        at akka.actor.ReflectiveDynamicAccess.createInstanceFor(DynamicAccess.scala:73)

        at akka.actor.ReflectiveDynamicAccess$$anonfun$createInstanceFor$3.apply(DynamicAccess.scala:84)

        at akka.actor.ReflectiveDynamicAccess$$anonfun$createInstanceFor$3.apply(DynamicAccess.scala:84)

        at scala.util.Success.flatMap(Try.scala:231)

        at akka.actor.ReflectiveDynamicAccess.createInstanceFor(DynamicAccess.scala:84)

        at akka.actor.ActorSystemImpl.liftedTree1$1(ActorSystem.scala:585)

        at akka.actor.ActorSystemImpl.<init>(ActorSystem.scala:578)

        at akka.actor.ActorSystem$.apply(ActorSystem.scala:142)

        at akka.actor.ActorSystem$.apply(ActorSystem.scala:109)

        at org.apache.toree.boot.layer.StandardBareInitialization$class.createActorSystem(BareInitialization.scala:71)

        at org.apache.toree.Main$$anon$1.createActorSystem(Main.scala:34)

        at org.apache.toree.boot.layer.StandardBareInitialization$class.initializeBare(BareInitialization.scala:60)

        at org.apache.toree.Main$$anon$1.initializeBare(Main.scala:34)

        at org.apache.toree.boot.KernelBootstrap.initialize(KernelBootstrap.scala:70)

        at org.apache.toree.Main$delayedInit$body.apply(Main.scala:39)

        at scala.Function0$class.apply$mcV$sp(Function0.scala:34)

        at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:12)

        at scala.App$$anonfun$main$1.apply(App.scala:76)

        at scala.App$$anonfun$main$1.apply(App.scala:76)

        at scala.collection.immutable.List.foreach(List.scala:381)

        at scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:35)

        at scala.App$class.main(App.scala:76)

        at org.apache.toree.Main$.main(Main.scala:23)

        at org.apache.toree.Main.main(Main.scala)

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

        at java.lang.reflect.Method.invoke(Method.java:498)

        at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:738)

        at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:187)

        at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:212)

        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:126)

        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

[W 17:35:31.691 NotebookApp] Timeout waiting for kernel_info reply from 6f5493bd-5123-4b25-b2a4-e856cdef51ee