如何從地圖函數（Python）執行spark sql查詢？

如何執行不是程序的驅動程序部分的例程的sql查詢？如何從地圖函數（Python）執行spark sql查詢？

from pyspark import SparkContext 
from pyspark.sql import SQLContext 
from pyspark.sql.types import * 


def doWork(rec): 
    data = SQLContext.sql("select * from zip_data where STATEFP ='{sfp}' and COUNTYFP = '{cfp}' ".format(sfp=rec[0], cfp=rec[1])) 
    for item in data.collect(): 
     print(item) 
    # do something 
    return (rec[0], rec[1]) 

if __name__ == "__main__": 
    sc = SparkContext(appName="Some app") 
    print("Starting some app") 

    SQLContext = SQLContext(sc) 

    parquetFile = SQLContext.read.parquet("/path/to/data/") 
    parquetFile.registerTempTable("zip_data") 


    df = SQLContext.sql("select distinct STATEFP,COUNTYFP from zip_data where STATEFP IN ('12') ") 
    rslts = df.map(doWork) 

    for rslt in rslts.collect(): 
     print(rslt)

在這個例子中，我試圖查詢相同的表，但想查詢的SQL星火註冊過其他表。

來源

2016-02-05 Chris Townsend