如何通過Python訪問Hive？

Question

https://cwiki.apache.org/confluence/display/Hive/HiveClient#HiveClient-Python似乎已過時。

當我添加這對/ etc/profile文件：

export PYTHONPATH=$PYTHONPATH:/usr/lib/hive/lib/py 

然後我可以做進口爲鏈接中列出，與from hive import ThriftHive除外，它實際上需要：

from hive_service import ThriftHive 

接下來示例中的端口是10000，當我嘗試導致程序掛起時。默認的Hive Thrift端口是9083，它停止掛起。

所以我把它就像這樣：

from thrift import Thrift 
from thrift.transport import TSocket 
from thrift.transport import TTransport 
from thrift.protocol import TBinaryProtocol 
try: 
    transport = TSocket.TSocket('<node-with-metastore>', 9083) 
    transport = TTransport.TBufferedTransport(transport) 
    protocol = TBinaryProtocol.TBinaryProtocol(transport) 
    client = ThriftHive.Client(protocol) 
    transport.open() 
    client.execute("CREATE TABLE test(c1 int)") 

    transport.close() 
except Thrift.TException, tx: 
    print '%s' % (tx.message) 

我收到以下錯誤：

Traceback (most recent call last): 
File "<stdin>", line 1, in <module> 
File "/usr/lib/hive/lib/py/hive_service/ThriftHive.py", line 68, in execute 
self.recv_execute() 
File "/usr/lib/hive/lib/py/hive_service/ThriftHive.py", line 84, in recv_execute 
raise x 
thrift.Thrift.TApplicationException: Invalid method name: 'execute' 

但檢查ThriftHive.py文件揭示了法客戶端類中執行。

我該如何使用Python訪問Hive？

Answer 1

您可以使用蜂巢庫，你想從蜂巢進口導入蜂巢類ThriftHive

嘗試這個例子：

import sys 

from hive import ThriftHive 
from hive.ttypes import HiveServerException 

from thrift import Thrift 
from thrift.transport import TSocket 
from thrift.transport import TTransport 
from thrift.protocol import TBinaryProtocol 

try: 
    transport = TSocket.TSocket('localhost', 10000) 
    transport = TTransport.TBufferedTransport(transport) 
    protocol = TBinaryProtocol.TBinaryProtocol(transport) 
    client = ThriftHive.Client(protocol) 
    transport.open() 
    client.execute("CREATE TABLE r(a STRING, b INT, c DOUBLE)") 
    client.execute("LOAD TABLE LOCAL INPATH '/path' INTO TABLE r") 
    client.execute("SELECT * FROM r") 
    while (1): 
    row = client.fetchOne() 
    if (row == None): 
     break 
    print row 

    client.execute("SELECT * FROM r") 
    print client.fetchAll() 
    transport.close() 
except Thrift.TException, tx: 
    print '%s' % (tx.message) 

Answer 2

我斷言，你正在使用HiveServer2，這是原因使代碼不起作用。

您可以使用pyhs2正確地訪問您的蜂巢和示例代碼那樣：

import pyhs2 

with pyhs2.connect(host='localhost', 
       port=10000, 
       authMechanism="PLAIN", 
       user='root', 
       password='test', 
       database='default') as conn: 
    with conn.cursor() as cur: 
     #Show databases 
     print cur.getDatabases() 

     #Execute query 
     cur.execute("select * from table") 

     #Return column info from query 
     print cur.getSchema() 

     #Fetch table results 
     for i in cur.fetch(): 
      print i 

注意，你可以用PIP安裝pyhs2之前安裝python-devel.x86_64賽勒斯-SASL-devel.x86_64。

希望這可以幫助你。

參考：https://cwiki.apache.org/confluence/display/Hive/Setting+Up+HiveServer2#SettingUpHiveServer2-PythonClientDriver

Answer 3

上面的例子是有點過時。 一個新的例子是在這裏：

import pyhs2 as hive 
import getpass 
DEFAULT_DB = 'default' 
DEFAULT_SERVER = '10.37.40.1' 
DEFAULT_PORT = 10000 
DEFAULT_DOMAIN = 'PAM01-PRD01.IBM.COM' 

u = raw_input('Enter PAM username: ') 
s = getpass.getpass() 
connection = hive.connect(host=DEFAULT_SERVER, port= DEFAULT_PORT, authMechanism='LDAP', user=u + '@' + DEFAULT_DOMAIN, password=s) 
statement = "select * from user_yuti.Temp_CredCard where pir_post_dt = '2014-05-01' limit 100" 
cur = connection.cursor() 

cur.execute(statement) 
df = cur.fetchall() 

除了標準的Python程序，一些庫需要安裝，讓Python來建立到Hadoop的databae的連接。

1.Pyhs2，Python的蜂房服務器2客戶端驅動程序

2.Sasl，賽勒斯-SASL綁定用於Python

3.Thrift，Python綁定爲Apache節儉RPC系統

4。 PyHive，Python接口攆

記住更改可執行文件的權限

使用chmod + X test_hive2.py ./test_hive2。py

希望它可以幫助你。 參考：https://sites.google.com/site/tingyusz/home/blogs/hiveinpython

Answer 4

下面的Python程序應努力從蟒蛇訪問蜂巢表：

import commands 

    cmd = "hive -S -e 'SELECT * FROM db_name.table_name LIMIT 1;' " 

    status, output = commands.getstatusoutput(cmd) 

    if status == 0: 
     print output 
    else: 
     print "error" 

Answer 5

我認爲最簡單的方法是使用PyHive。

要安裝你需要這些庫：

pip install sasl 
pip install thrift 
pip install thrift-sasl 
pip install PyHive 

請注意，儘管安裝庫爲PyHive，導入模塊pyhive，全部小寫。

如果您使用的是Linux，則可能需要在運行上述步驟之前單獨安裝SASL。安裝軟件包libsasl2-dev使用apt-get或yum或任何你的發行包管理器。對於Windows，在GNU.org上有一些選項，你可以下載一個二進制安裝程序。在Mac SASL應可如果你已經安裝了Xcode開發工具（在終端xcode-select --install）

安裝完畢後，就可以連接到攆這樣的：

from pyhive import hive 
conn = hive.Connection(host="YOUR_HIVE_HOST", port=PORT, username="YOU") 

現在，你有蜂巢連接，你有選擇如何使用它。你可以只直線上升查詢：

cursor = conn.cursor() 
cursor.execute("SELECT cool_stuff FROM hive_table") 
for result in cursor.fetchall(): 
    use_result(result) 

...或使用該連接撥打大熊貓數據幀：

import pandas as pd 
df = pd.read_sql("SELECT cool_stuff FROM hive_table", conn) 

Answer 6

pyhs2不再維持。更好的選擇是impyla

https://github.com/cloudera/impyla

它擁有超過pyhs2還有更多的功能，例如，它有Kerberos身份驗證，這對我們來說是必須的。

from impala.dbapi import connect 
conn = connect(host='my.host.com', port=10000) 
cursor = conn.cursor() 
cursor.execute('SELECT * FROM mytable LIMIT 100') 
print cursor.description # prints the result set's schema 
results = cursor.fetchall() 

## 
cursor.execute('SELECT * FROM mytable LIMIT 100') 
for row in cursor: 
    process(row) 

Cloudera的現在把更多的精力HS2客戶https://github.com/cloudera/hs2client 其是C/C++ HiveServer2 /帕拉客戶端上。如果你向python推送大量數據或許是更好的選擇。 （有Python綁定太 - https://github.com/cloudera/hs2client/tree/master/python）

上impyla一些詳細信息：

• http://blog.cloudera.com/blog/2014/04/a-new-python-client-for-impala/
• https://github.com/cloudera/impyla/blob/master/README.md

不要混淆，一些上面的例子說說黑斑羚;只需將HiveServer2的端口更改爲10000（默認值）即可，它的工作方式與Impala示例相同。這是用於Impala和Hive的協議（Thrift）。

Answer 7

這可能是一個快速黑客連接蜂房，蟒蛇，

from pyhive import hive 
cursor = hive.connect('YOUR_HOST_NAME').cursor() 
cursor.execute('SELECT * from table_name LIMIT 5',async=True) 
print cursor.fetchall() 

輸出：元組

名單

Answer 8

你可以使用Python JayDeBeApi包來創建從蜂巢或黑斑羚JDBC驅動程序DB-API接口然後將連接傳遞給pandas.read_sql函數以在pandas數據框中返回數據。

import jaydebeapi 
# Apparently need to load the jar files for the first time for impala jdbc driver to work 
conn = jaydebeapi.connect('com.cloudera.hive.jdbc41.HS2Driver', 
['jdbc:hive2://host:10000/db;AuthMech=1;KrbHostFQDN=xxx.com;KrbServiceName=hive;KrbRealm=xxx.COM', "",""], 
jars=['/hadp/opt/jdbc/hive_jdbc_2.5.18.1050/2.5.18.1050 GA/Cloudera_HiveJDBC41_2.5.18.1050/HiveJDBC41.jar', 
'/hadp/opt/jdbc/hive_jdbc_2.5.18.1050/2.5.18.1050 GA/Cloudera_HiveJDBC41_2.5.18.1050/TCLIServiceClient.jar', 
'/hadp/opt/jdbc/hive_jdbc_2.5.18.1050/2.5.18.1050 GA/Cloudera_HiveJDBC41_2.5.18.1050/commons-codec-1.3.jar', 
'/hadp/opt/jdbc/hive_jdbc_2.5.18.1050/2.5.18.1050 GA/Cloudera_HiveJDBC41_2.5.18.1050/commons-logging-1.1.1.jar', 
'/hadp/opt/jdbc/hive_jdbc_2.5.18.1050/2.5.18.1050 GA/Cloudera_HiveJDBC41_2.5.18.1050/hive_metastore.jar', 
'/hadp/opt/jdbc/hive_jdbc_2.5.18.1050/2.5.18.1050 GA/Cloudera_HiveJDBC41_2.5.18.1050/hive_service.jar', 
'/hadp/opt/jdbc/hive_jdbc_2.5.18.1050/2.5.18.1050 GA/Cloudera_HiveJDBC41_2.5.18.1050/httpclient-4.1.3.jar', 
'/hadp/opt/jdbc/hive_jdbc_2.5.18.1050/2.5.18.1050 GA/Cloudera_HiveJDBC41_2.5.18.1050/httpcore-4.1.3.jar', 
'/hadp/opt/jdbc/hive_jdbc_2.5.18.1050/2.5.18.1050 GA/Cloudera_HiveJDBC41_2.5.18.1050/libfb303-0.9.0.jar', 
'/hadp/opt/jdbc/hive_jdbc_2.5.18.1050/2.5.18.1050 GA/Cloudera_HiveJDBC41_2.5.18.1050/libthrift-0.9.0.jar', 
'/hadp/opt/jdbc/hive_jdbc_2.5.18.1050/2.5.18.1050 GA/Cloudera_HiveJDBC41_2.5.18.1050/log4j-1.2.14.jar', 
'/hadp/opt/jdbc/hive_jdbc_2.5.18.1050/2.5.18.1050 GA/Cloudera_HiveJDBC41_2.5.18.1050/ql.jar', 
'/hadp/opt/jdbc/hive_jdbc_2.5.18.1050/2.5.18.1050 GA/Cloudera_HiveJDBC41_2.5.18.1050/slf4j-api-1.5.11.jar', 
'/hadp/opt/jdbc/hive_jdbc_2.5.18.1050/2.5.18.1050 GA/Cloudera_HiveJDBC41_2.5.18.1050/slf4j-log4j12-1.5.11.jar', 
'/hadp/opt/jdbc/hive_jdbc_2.5.18.1050/2.5.18.1050 GA/Cloudera_HiveJDBC41_2.5.18.1050/zookeeper-3.4.6.jar', 
'/hadp/opt/jdbc/impala_jdbc_2.5.35/2.5.35.1055 GA/Cloudera_ImpalaJDBC41_2.5.35/ImpalaJDBC41.jar', 
'/hadp/opt/jdbc/impala_jdbc_2.5.35/2.5.35.1055 GA/Cloudera_ImpalaJDBC41_2.5.35/TCLIServiceClient.jar', 
'/hadp/opt/jdbc/impala_jdbc_2.5.35/2.5.35.1055 GA/Cloudera_ImpalaJDBC41_2.5.35/commons-codec-1.3.jar', 
'/hadp/opt/jdbc/impala_jdbc_2.5.35/2.5.35.1055 GA/Cloudera_ImpalaJDBC41_2.5.35/commons-logging-1.1.1.jar', 
'/hadp/opt/jdbc/impala_jdbc_2.5.35/2.5.35.1055 GA/Cloudera_ImpalaJDBC41_2.5.35/hive_metastore.jar', 
'/hadp/opt/jdbc/impala_jdbc_2.5.35/2.5.35.1055 GA/Cloudera_ImpalaJDBC41_2.5.35/hive_service.jar', 
'/hadp/opt/jdbc/impala_jdbc_2.5.35/2.5.35.1055 GA/Cloudera_ImpalaJDBC41_2.5.35/httpclient-4.1.3.jar', 
'/hadp/opt/jdbc/impala_jdbc_2.5.35/2.5.35.1055 GA/Cloudera_ImpalaJDBC41_2.5.35/httpcore-4.1.3.jar', 
'/hadp/opt/jdbc/impala_jdbc_2.5.35/2.5.35.1055 GA/Cloudera_ImpalaJDBC41_2.5.35/libfb303-0.9.0.jar', 
'/hadp/opt/jdbc/impala_jdbc_2.5.35/2.5.35.1055 GA/Cloudera_ImpalaJDBC41_2.5.35/libthrift-0.9.0.jar', 
'/hadp/opt/jdbc/impala_jdbc_2.5.35/2.5.35.1055 GA/Cloudera_ImpalaJDBC41_2.5.35/log4j-1.2.14.jar', 
'/hadp/opt/jdbc/impala_jdbc_2.5.35/2.5.35.1055 GA/Cloudera_ImpalaJDBC41_2.5.35/ql.jar', 
'/hadp/opt/jdbc/impala_jdbc_2.5.35/2.5.35.1055 GA/Cloudera_ImpalaJDBC41_2.5.35/slf4j-api-1.5.11.jar', 
'/hadp/opt/jdbc/impala_jdbc_2.5.35/2.5.35.1055 GA/Cloudera_ImpalaJDBC41_2.5.35/slf4j-log4j12-1.5.11.jar', 
'/hadp/opt/jdbc/impala_jdbc_2.5.35/2.5.35.1055 GA/Cloudera_ImpalaJDBC41_2.5.35/zookeeper-3.4.6.jar' 
]) 

# the previous call have initialized the jar files, technically this call needs not include the required jar files 
impala_conn = jaydebeapi.connect('com.cloudera.impala.jdbc41.Driver', 
['jdbc:impala://host:21050/db;AuthMech=1;KrbHostFQDN=xxx.com;KrbServiceName=impala;KrbRealm=xxx.COM',"",""], 
jars=['/hadp/opt/jdbc/hive_jdbc_2.5.18.1050/2.5.18.1050 GA/Cloudera_HiveJDBC41_2.5.18.1050/HiveJDBC41.jar', 
'/hadp/opt/jdbc/hive_jdbc_2.5.18.1050/2.5.18.1050 GA/Cloudera_HiveJDBC41_2.5.18.1050/TCLIServiceClient.jar', 
'/hadp/opt/jdbc/hive_jdbc_2.5.18.1050/2.5.18.1050 GA/Cloudera_HiveJDBC41_2.5.18.1050/commons-codec-1.3.jar', 
'/hadp/opt/jdbc/hive_jdbc_2.5.18.1050/2.5.18.1050 GA/Cloudera_HiveJDBC41_2.5.18.1050/commons-logging-1.1.1.jar', 
'/hadp/opt/jdbc/hive_jdbc_2.5.18.1050/2.5.18.1050 GA/Cloudera_HiveJDBC41_2.5.18.1050/hive_metastore.jar', 
'/hadp/opt/jdbc/hive_jdbc_2.5.18.1050/2.5.18.1050 GA/Cloudera_HiveJDBC41_2.5.18.1050/hive_service.jar', 
'/hadp/opt/jdbc/hive_jdbc_2.5.18.1050/2.5.18.1050 GA/Cloudera_HiveJDBC41_2.5.18.1050/httpclient-4.1.3.jar', 
'/hadp/opt/jdbc/hive_jdbc_2.5.18.1050/2.5.18.1050 GA/Cloudera_HiveJDBC41_2.5.18.1050/httpcore-4.1.3.jar', 
'/hadp/opt/jdbc/hive_jdbc_2.5.18.1050/2.5.18.1050 GA/Cloudera_HiveJDBC41_2.5.18.1050/libfb303-0.9.0.jar', 
'/hadp/opt/jdbc/hive_jdbc_2.5.18.1050/2.5.18.1050 GA/Cloudera_HiveJDBC41_2.5.18.1050/libthrift-0.9.0.jar', 
'/hadp/opt/jdbc/hive_jdbc_2.5.18.1050/2.5.18.1050 GA/Cloudera_HiveJDBC41_2.5.18.1050/log4j-1.2.14.jar', 
'/hadp/opt/jdbc/hive_jdbc_2.5.18.1050/2.5.18.1050 GA/Cloudera_HiveJDBC41_2.5.18.1050/ql.jar', 
'/hadp/opt/jdbc/hive_jdbc_2.5.18.1050/2.5.18.1050 GA/Cloudera_HiveJDBC41_2.5.18.1050/slf4j-api-1.5.11.jar', 
'/hadp/opt/jdbc/hive_jdbc_2.5.18.1050/2.5.18.1050 GA/Cloudera_HiveJDBC41_2.5.18.1050/slf4j-log4j12-1.5.11.jar', 
'/hadp/opt/jdbc/hive_jdbc_2.5.18.1050/2.5.18.1050 GA/Cloudera_HiveJDBC41_2.5.18.1050/zookeeper-3.4.6.jar', 
'/hadp/opt/jdbc/impala_jdbc_2.5.35/2.5.35.1055 GA/Cloudera_ImpalaJDBC41_2.5.35/ImpalaJDBC41.jar', 
'/hadp/opt/jdbc/impala_jdbc_2.5.35/2.5.35.1055 GA/Cloudera_ImpalaJDBC41_2.5.35/TCLIServiceClient.jar', 
'/hadp/opt/jdbc/impala_jdbc_2.5.35/2.5.35.1055 GA/Cloudera_ImpalaJDBC41_2.5.35/commons-codec-1.3.jar', 
'/hadp/opt/jdbc/impala_jdbc_2.5.35/2.5.35.1055 GA/Cloudera_ImpalaJDBC41_2.5.35/commons-logging-1.1.1.jar', 
'/hadp/opt/jdbc/impala_jdbc_2.5.35/2.5.35.1055 GA/Cloudera_ImpalaJDBC41_2.5.35/hive_metastore.jar', 
'/hadp/opt/jdbc/impala_jdbc_2.5.35/2.5.35.1055 GA/Cloudera_ImpalaJDBC41_2.5.35/hive_service.jar', 
'/hadp/opt/jdbc/impala_jdbc_2.5.35/2.5.35.1055 GA/Cloudera_ImpalaJDBC41_2.5.35/httpclient-4.1.3.jar', 
'/hadp/opt/jdbc/impala_jdbc_2.5.35/2.5.35.1055 GA/Cloudera_ImpalaJDBC41_2.5.35/httpcore-4.1.3.jar', 
'/hadp/opt/jdbc/impala_jdbc_2.5.35/2.5.35.1055 GA/Cloudera_ImpalaJDBC41_2.5.35/libfb303-0.9.0.jar', 
'/hadp/opt/jdbc/impala_jdbc_2.5.35/2.5.35.1055 GA/Cloudera_ImpalaJDBC41_2.5.35/libthrift-0.9.0.jar', 
'/hadp/opt/jdbc/impala_jdbc_2.5.35/2.5.35.1055 GA/Cloudera_ImpalaJDBC41_2.5.35/log4j-1.2.14.jar', 
'/hadp/opt/jdbc/impala_jdbc_2.5.35/2.5.35.1055 GA/Cloudera_ImpalaJDBC41_2.5.35/ql.jar', 
'/hadp/opt/jdbc/impala_jdbc_2.5.35/2.5.35.1055 GA/Cloudera_ImpalaJDBC41_2.5.35/slf4j-api-1.5.11.jar', 
'/hadp/opt/jdbc/impala_jdbc_2.5.35/2.5.35.1055 GA/Cloudera_ImpalaJDBC41_2.5.35/slf4j-log4j12-1.5.11.jar', 
'/hadp/opt/jdbc/impala_jdbc_2.5.35/2.5.35.1055 GA/Cloudera_ImpalaJDBC41_2.5.35/zookeeper-3.4.6.jar' 
]) 

import pandas as pd 
df1 = pd.read_sql("SELECT * FROM tablename", conn) 
df2 = pd.read_sql("SELECT * FROM tablename", impala_conn) 

conn.close() 
impala_conn.close() 

Answer 9

要使用用戶名/密碼，並指定端口連接，代碼如下所示：

from pyhive import presto 

cursor = presto.connect(host='host.example.com', 
        port=8081, 
        username='USERNAME:PASSWORD').cursor() 

sql = 'select * from table limit 10' 

cursor.execute(sql) 

print(cursor.fetchone()) 
print(cursor.fetchall()) 

Answer 10

到eycheu的解決方案類似，但多了幾分細緻。

下面是一個替代的解決方案專門爲hive2該不要求PyHive或安裝全系統的包。我在Linux環境中Tristin的文章中提到我沒有那麼安裝SASL依賴的root訪問權限的工作是不是一種選擇，對我來說：

If you're on Linux, you may need to install SASL separately before running the above. Install the package libsasl2-dev using apt-get or yum or whatever package manager for your distribution.

具體而言，該解決方案側重於利用Python包： JayDeBeApi。根據我的經驗，在python Anaconda 2.7安裝上安裝這個額外的軟件包是我所需要的。該軟件包利用Java（JDK）。我假設已經建立。

1步：安裝JayDeBeApi

pip install jaydebeap 

第2步：下載相應的驅動程序爲您的環境：

• 這裏是一個企業CDH環境
• 需要link to the jarsAnother post討論在哪裏可以找到Apac的jdbc驅動程序他蜂巢

將所有.jar文件存儲在一個目錄中。我將這個目錄稱爲/ path/to/jar/files /。

3步：確定你的系統的身份驗證機制：

在pyhive列​​出的解決方案我見過平原列爲認證機制以及Kerberos的。 請注意，您的jdbc連接URL將取決於您使用的身份驗證機制。我會解釋Kerberos解決方案沒有通過用戶名/密碼。 Here is more information Kerberos authentication and options.

創建一個Kerberos票據，如果尚未創建

$ kinit 

門票可通過klist被視爲一個。

您現在可以進行通過Python連接：

import jaydebeapi 
import glob 
# Creates a list of jar files in the /path/to/jar/files/ directory 
jar_files = glob.glob('/path/to/jar/files/*.jar') 

host='localhost' 
port='10000' 
database='default' 

# note: your driver will depend on your environment and drivers you've 
# downloaded in step 2 
# this is the driver for my environment (jdbc3, hive2, cloudera enterprise) 
driver='com.cloudera.hive.jdbc3.HS2Driver' 

conn_hive = jaydebeapi.connect(driver, 
     'jdbc:hive2://'+host+':' +port+'/'+database+';AuthMech=1;KrbHostFQDN='+host+';KrbServiceName=hive' 
          ,jars=jar_files) 

如果你只關心讀書，那麼你可以通過eycheu的解決方案讀取它直接進入熊貓的輕鬆數據框：

import pandas as pd 
df = pd.read_sql("select * from table", conn_hive) 

否則，這裏是一個更靈活的通信選項：

cursor = conn_hive.cursor() 
sql_expression = "select * from table" 
cursor.execute(sql_expression) 
results = cursor.fetchall() 

你可以想象，如果你想創建一個表，你不需要「獲取」結果，而是可以提交一個創建表查詢。

Answer 11

類似於@ python-starter解決方案。但是，在python3.x上，命令包並不具有可用性。所以替代解決方案是在python3.x中使用子進程

import subprocess 

cmd = "hive -S -e 'SELECT * FROM db_name.table_name LIMIT 1;' " 

status, output = subprocess.getstatusoutput(cmd) 

if status == 0: 
    print output 
else: 
    print "error"