無法訪問包中的AnyRef Scala

我正在使用apache toree（來自github的版本）。當我試圖對postgresql表執行查詢時，我得到了間歇性的scala編譯器錯誤（當我運行同一個單元兩次，錯誤消失，代碼運行良好）。無法訪問包中的AnyRef Scala

我在尋找如何調試這些錯誤的建議。錯誤看起來很奇怪（它們出現在標準輸出的筆記本上）。

error: missing or invalid dependency detected while loading class file 'QualifiedTableName.class'. 
Could not access type AnyRef in package scala, 
because it (or its dependencies) are missing. Check your build definition for 
missing or conflicting dependencies. (Re-run with `-Ylog-classpath` to see the problematic classpath.) 
A full rebuild may help if 'QualifiedTableName.class' was compiled against an incompatible version of scala. 
error: missing or invalid dependency detected while loading class file 'FunctionIdentifier.class'. 
Could not access type AnyRef in package scala, 
because it (or its dependencies) are missing. Check your build definition for 
missing or conflicting dependencies. (Re-run with `-Ylog-classpath` to see the problematic classpath.) 
A full rebuild may help if 'FunctionIdentifier.class' was compiled against an incompatible version of scala. 
error: missing or invalid dependency detected while loading class file 'DefinedByConstructorParams.class'. 
...

代碼簡單：從一個postgres表中提取一個數據集：

%AddDeps org.postgresql postgresql 42.1.4 --transitive 
val props = new java.util.Properties(); 
props.setProperty("driver","org.postgresql.Driver"); 
val df = spark.read.jdbc(url = "jdbc:postgresql://postgresql/database?user=user&password=password", 
       table = "table", predicates = Array("1=1"), connectionProperties = props) 
df.show()

我檢查爲明顯（二者toree和Apache火花使用階2.11.8，我建立阿帕奇toree與APACHE_SPARK_VERSION = 2.2.0這是相同的我donwloaded火花）

作爲參考，這是我用於設置toree和火花Dockerfile的一部分：

RUN wget https://d3kbcqa49mib13.cloudfront.net/spark-2.2.0-bin-hadoop2.7.tgz && tar -zxf spark-2.2.0-bin-hadoop2.7.tgz && chmod -R og+rw /opt/spark-2.2.0-bin-hadoop2.7 && chown -R a1414.a1414 /opt/spark-2.2.0-bin-hadoop2.7 
RUN (curl https://bintray.com/sbt/rpm/rpm > /etc/yum.repos.d/bintray-sbt-rpm.repo) 
RUN yum -y install --nogpgcheck sbt 
RUN (unset http_proxy; unset https_proxy; yum -y install --nogpgcheck java-1.8.0-openjdk-devel.i686) 
RUN (git clone https://github.com/apache/incubator-toree && cd incubator-toree && make clean release APACHE_SPARK_VERSION=2.2.0 ; exit 0) 
RUN (. /opt/rh/rh-python35/enable; cd /opt/incubator-toree/dist/toree-pip ;python setup.py install) 
RUN (. /opt/rh/rh-python35/enable; jupyter toree install --spark_home=/opt/spark-2.2.0-bin-hadoop2.7 --interpreters=Scala)

來源

2017-08-29 kervel

可能取決於不同的版本的Scala的（IIb）的 – cchantep

即庫之間的衝突我也喜歡，但奇怪的是，它有時可以工作。有沒有辦法找出答案？我試圖用sbt製作一個依賴樹，它沒有顯示多個版本的scala lib：https：//gist.github.com/anonymous/1ea6f24a30ac77a2252884227b88d522 – kervel

檢查依賴關係樹 – cchantep

正如在cchantep的評論中所說的，您可能使用的是不同於用於構建Spark的Scala版本。

的最簡單的解決方案是檢查哪一個被用於由火花，並切換到這一個，例如，在Mac：

brew switch scala 2.11.8

來源

2018-03-03 11:38:43

無法訪問包中的AnyRef Scala

回答

相關問題