導入錯誤：無法導入名稱sqlContext

Question

我使用pyspark讀一些CSV數據引發數據幀

，當我嘗試導入pyspark模塊如下： 從pyspark.sql進口sqlContext

爲什麼我得到以下錯誤？如何解決它？ 導入錯誤：無法導入名稱sqlContext

我使用Python 2.7和2.0.1星火

非常感謝！

Answer 1

這可能是因爲你沒有正確設置python pat。在配置我的Python環境時，我發現以下功能很有用。

def configure_spark(spark_home=None, pyspark_python=None, conf_dir=None): 
    """Configures the Python path for importing pyspark 

    Sets the SPARK_HOME and PYSPARK_PYTHON environment variables and modifies 
    the Python PATH so the pyspark package can be imported. 

    Args: 
     spark_home (str): Path of SPARK_HOME. Defaults to SPARK_HOME module 
      variable. 
     pyspark_python (str): Path to Python binary to use in PySpark. Defaults 
      to the currently executing Python binary. 
     conf_dir (str): Path to configuration directory 
    """ 

    # Set the configuration directory with some basic sanity checks: 
    if conf_dir: 
     if not os.path.isdir(conf_dir): 
      raise OSError("Spark config directory not found: %s" % conf_dir) 

     expected_conf = {'spark-env.sh', 'spark-defaults.conf'} 
     found_conf = expected_conf - set(os.listdir(conf_dir)) 
     if found_conf: 
      warnings.warn("Some configuration files were not found: %s" % found_conf) 

     os.environ['SPARK_CONF_DIR'] = conf_dir 

    spark_home = spark_home or SPARK_HOME 
    os.environ['SPARK_HOME'] = spark_home 

    if not os.path.isdir(spark_home): 
     raise OSError("Specified SPARK_HOME is not a valid directory: %s" % spark_home) 

    # Add the PySpark directories to the Python path: 
    libs = glob(os.path.join(spark_home, 'python', 'lib', '*.zip')) 
    if len(libs) < 2: 
     raise OSError("Pyspark libraries not found in %s" % spark_home) 
    for lib in libs: 
     sys.path.insert(1, lib) 

    # If PYSPARK_PYTHON isn't specified, use currently running Python binary: 
    pyspark_python = pyspark_python or sys.executable 
    os.environ['PYSPARK_PYTHON'] = pyspark_python