錯誤，當我嘗試使用文本文件中sparkR

我試圖按照教程使用sparkR獲得「README.md」的單詞計數。星火版本：1.6.1 [R版本：3.3.0 Ubuntu的版本：14 然而，當我做lines <- sc.textFile("README.md")，我得到了錯誤這裏錯誤，當我嘗試使用文本文件中sparkR

Error: could not find function "sc.textFile"

然後，我試過lines <- SparkR:::textFile(sc, "./README.md")相反，我得到這個錯誤：

16/07/01 00:06:26 INFO MemoryStore: Block broadcast_4 stored as values in memory (estimated size 88.5 KB, free 258.9 KB) 
16/07/01 00:06:26 INFO MemoryStore: Block broadcast_4_piece0 stored as bytes in memory (estimated size 19.6 KB, free 278.5 KB) 
16/07/01 00:06:26 INFO BlockManagerInfo: Added broadcast_4_piece0 in memory on localhost:40985 (size: 19.6 KB, free: 511.5 MB) 
16/07/01 00:06:26 INFO SparkContext: Created broadcast 4 from textFile at NativeMethodAccessorImpl.java:-2 
16/07/01 00:06:26 ERROR RBackendHandler: collect on 44 failed 
Error in invokeJava(isStatic = FALSE, objId$id, methodName, ...) : 
    java.net.ConnectException: Call From ubuntu/127.0.1.1 to localhost:9000 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) 
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) 
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) 
    at java.lang.reflect.Constructor.newInstance(Constructor.java:526) 
    at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:791) 
    at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:731) 
    at org.apache.hadoop.ipc.Client.call(Client.java:1473) 
    at org.apache.hadoop.ipc.Client.call(Client.java:1400) 
    at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232) 
    at com.sun.proxy.$Proxy22.getFileInfo(Unknown Source) 
    at org.apache.hadoop.hdfs.proto 
16/07/01 00:15:59 INFO BlockManagerInfo: Removed broadcast_3_piece0 on localhost:40985 in memory (size: 14.1 KB, free: 511.5 MB)

更重要的是，我試圖在PySpark shell中做同樣的事情，但仍然沒有像教程那樣工作。

>>> lines = sc.textFile("README.md") 
16/07/01 00:27:22 INFO MemoryStore: Block broadcast_2 stored as values in memory (estimated size 227.6 KB, free 506.2 KB) 
16/07/01 00:27:22 INFO MemoryStore: Block broadcast_2_piece0 stored as bytes in memory (estimated size 19.6 KB, free 525.8 KB) 
16/07/01 00:27:22 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on localhost:33186 (size: 19.6 KB, free: 511.4 MB) 
16/07/01 00:27:22 INFO SparkContext: Created broadcast 2 from textFile at NativeMethodAccessorImpl.java:-2 
>>> lines.count() 
Traceback (most recent call last): 
    File "<stdin>", line 1, in <module> 
    File "/usr/local/spark/python/pyspark/rdd.py", line 1004, in count 
    return self.mapPartitions(lambda i: [sum(1 for _ in i)]).sum() 
    File "/usr/local/spark/python/pyspark/rdd.py", line 995, in sum 
    return self.mapPartitions(lambda x: [sum(x)]).fold(0, operator.add) 
    File "/usr/local/spark/python/pyspark/rdd.py", line 869, in fold 
    vals = self.mapPartitions(func).collect() 
    File "/usr/local/spark/python/pyspark/rdd.py", line 771, in collect 
    port = self.ctx._jvm.PythonRDD.collectAndServe(self._jrdd.rdd()) 
    File "/usr/local/spark/python/lib/py4j-0.9-src.zip/py4j/java_gateway.py", line 813, in __call__ 
    File "/usr/local/spark/python/pyspark/sql/utils.py", line 45, in deco 
    return f(*a, **kw) 
    File "/usr/local/spark/python/lib/py4j-0.9-src.zip/py4j/protocol.py", line 308, in get_return_value 
py4j.protocol.Py4JJavaError: An error occurred while calling z:org.apache.spark.api.python.PythonRDD.collectAndServe. 
: java.net.ConnectException: Call From ubuntu/127.0.1.1 to localhost:9000 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused 
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) 
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) 
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) 
    at java.lang.reflect.Constructor.newInstance(Constructor.java:526) 
    at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:791) 
    at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:731) 
    at org.apache.hadoop.ipc.Client.call(Client.java:1473) 
    at org.apache.hadoop.ipc.Client.call(Client.java:1400) 
    at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232) 
    at com.sun.proxy.$Proxy19.getFileInfo(Unknown Source) 
    at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:752) 
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) 
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 
    at java.lang.reflect.Method.invoke(Method.java:606) 
    at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187) 
    at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) 
    at com.sun.proxy.$Proxy20.getFileInfo(Unknown Source) 
    at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1977) 
    at org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1118) 
    at org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1114) 
    at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) 
    at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1114) 
    at org.apache.hadoop.fs.Globber.getFileStatus(Globber.java:57) 
    at org.apache.hadoop.fs.Globber.glob(Globber.java:252) 
    at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:1644) 
    at org.apache.hadoop.mapred.FileInputFormat.singleThreadedListStatus(FileInputFormat.java:257) 
    at org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:228) 
    at org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:313) 
    at org.apache.spark.rdd.HadoopRDD.getPartitions(HadoopRDD.scala:199) 
    at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:239) 
    at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:237) 
    at scala.Option.getOrElse(Option.scala:120) 
    at org.apache.spark.rdd.RDD.partitions(RDD.scala:237) 
    at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:35) 
    at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:239) 
    at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:237) 
    at scala.Option.getOrElse(Option.scala:120) 
    at org.apache.spark.rdd.RDD.partitions(RDD.scala:237) 
    at org.apache.spark.api.python.PythonRDD.getPartitions(PythonRDD.scala:58) 
    at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:239) 
    at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:237) 
    at scala.Option.getOrElse(Option.scala:120) 
    at org.apache.spark.rdd.RDD.partitions(RDD.scala:237) 
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:1929) 
    at org.apache.spark.rdd.RDD$$anonfun$collect$1.apply(RDD.scala:927) 
    at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:150) 
    at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:111) 
    at org.apache.spark.rdd.RDD.withScope(RDD.scala:316) 
    at org.apache.spark.rdd.RDD.collect(RDD.scala:926) 
    at org.apache.spark.api.python.PythonRDD$.collectAndServe(PythonRDD.scala:405) 
    at org.apache.spark.api.python.PythonRDD.collectAndServe(PythonRDD.scala) 
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) 
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 
    at java.lang.reflect.Method.invoke(Method.java:606) 
    at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:231) 
    at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:381) 
    at py4j.Gateway.invoke(Gateway.java:259) 
    at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:133) 
    at py4j.commands.CallCommand.execute(CallCommand.java:79) 
    at py4j.GatewayConnection.run(GatewayConnection.java:209) 
    at java.lang.Thread.run(Thread.java:745) 
Caused by: java.net.ConnectException: Connection refused 
    at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) 
    at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:744) 
    at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) 
    at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:530) 
    at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:494) 
    at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:608) 
    at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:706) 
    at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:369) 
    at org.apache.hadoop.ipc.Client.getConnection(Client.java:1522) 
    at org.apache.hadoop.ipc.Client.call(Client.java:1439) 
    ... 56 more

sparkR停止支持textFile函數嗎？或者是我的JAVA_HOME設置錯誤（儘管我能夠通過java -version獲得正確的響應）？

另一個微小的問題是，我必須初始化SPARK_HOME和JAVA_HOME之前，我進口sparkR包，就像這樣：

if (nchar(Sys.getenv("SPARK_HOME")) < 1) { 
    Sys.setenv(SPARK_HOME = "/usr/local/spark") 
} 
if (nchar(Sys.getenv("JAVA_HOME")) < 1) { 
    Sys.setenv(JAVA_HOME = "/usr/lib/jvm/java-7-openjdk-amd64") 
} 
library(SparkR, lib.loc = c(file.path(Sys.getenv("SPARK_HOME"), "R", "lib"))) 
sc <- sparkR.init(master = "local", appName = "test", sparkHome = Sys.getenv("SPARK_HOME")) 
lines <- sc.textFile("README.md")

如果我不這樣做，我會當我做library(sparkR)出現錯誤。我必須這樣做嗎？大多數教程並不需要在線。非常感謝。

來源

2016-07-01 Benson Chen

我看起來像HDFS問題建立試試這個'線= sc.textFile（「文件：/// /README.md」）'讓我們知道它的工作與否？ – WoodChopper

優秀!!當我遵循@WoodChopper上面所說的時候，它很有用！非常感謝！我的根字典的火花是/ usr/local/spark，所以我嘗試了'lines = sc.textFile（「file：///usr/local/spark/README.md」）'，那麼它終於有效！ –

問題是HDFS不運行或安裝問題，你的火花生成默認眺望HDFS路徑，

例如，當您提供，

lines = sc.textFile("README.md")

它將被轉換到，

lines = sc.textFile("hdfs://ip:port/README.md")

因此你的錯誤應該是fileNotFound因爲你的hdfs沒有運行/問題是拋出連接拒絕錯誤。

嘗試，如果這需要得到您的HDFS運行，

cd HADDOP_HOME/sbin 
./stop-all.sh 
./start-all.sh

來源

2016-07-01 08:36:54 WoodChopper

錯誤，當我嘗試使用文本文件中sparkR

回答

相關問題