4
我加入了HIVE中的3個巨大表（億行表）。所有統計數據都收集起來，但仍然表現非常糟糕（查詢需要40分鐘）。配置單元查詢性能不佳
是否有任何參數可以在HIVE提示中設置以獲得更好的性能？
當我試圖執行我看到像
Sep 4, 2015 7:40:23 AM INFO: parquet.hadoop.ParquetInputFormat: Total input paths to process : 1
Sep 4, 2015 7:40:23 AM INFO: parquet.hadoop.ParquetFileReader: reading another 1 footers
所有的表都在BigSql創建與存儲參數爲「存儲爲PARQUETFILE」
我怎麼能抑制作業進度的詳細信息時，信息HIVE查詢正在運行？
關於HIVE版本
hive> set system:sun.java.command;
system:sun.java.command=org.apache.hadoop.util.RunJar /opt/ibm/biginsights/hive/lib/hive-cli-0.12.0.jar org.apache.hadoop.hive.cli.CliDriver -hiveconf hive.aux.jars.path=file:///opt/ibm/biginsights/hive/lib/hive-hbase-handler-0.12.0.jar,file:///opt/ibm/biginsights/hive/lib/hive-contrib-0.12.0.jar,file:///opt/ibm/biginsights/hive/lib/hbase-client-0.96.0.jar,file:///opt/ibm/biginsights/hive/lib/hbase-common-0.96.0.jar,file:///opt/ibm/biginsights/hive/lib/hbase-hadoop2-compat-0.96.0.jar,file:///opt/ibm/biginsights/hive/lib/hbase-prefix-tree-0.96.0.jar,file:///opt/ibm/biginsights/hive/lib/hbase-protocol-0.96.0.jar,file:///opt/ibm/biginsights/hive/lib/hbase-server-0.96.0.jar,file:///opt/ibm/biginsights/hive/lib/htrace-core-2.01.jar,file:///opt/ibm/biginsights/hive/lib/zookeeper-3.4.5.jar,file:///opt/ibm/biginsights/sheets/libext/piggybank.jar,file:///opt/ibm/biginsights/sheets/libext/pig-0.11.1.jar,file:///opt/ibm/biginsights/sheets/libext/avro-1.7.4.jar,file:///opt/ibm/biginsights/sheets/libext/opencsv-1.8.jar,file:///opt/ibm/biginsights/sheets/libext/json-simple-1.1.jar,file:///opt/ibm/biginsights/sheets/libext/joda-time-1.6.jar,file:///opt/ibm/biginsights/sheets/libext/bigsheets.jar,file:///opt/ibm/biginsights/sheets/libext/bigsheets-serdes-1.0.0.jar,file:///opt/ibm/biginsights/lib/parquet/parquet-mr/parquet-column-1.3.2.jar,file:///opt/ibm/biginsights/lib/parquet/parquet-mr/parquet-common-1.3.2.jar,file:///opt/ibm/biginsights/lib/parquet/parquet-mr/parquet-encoding-1.3.2.jar,file:///opt/ibm/biginsights/lib/parquet/parquet-mr/parquet-generator-1.3.2.jar,file:///opt/ibm/biginsights/lib/parquet/parquet-mr/parquet-hadoop-bundle-1.3.2.jar,file:///opt/ibm/biginsights/lib/parquet/parquet-mr/parquet-hive-bundle-1.3.2.jar,file:///opt/ibm/biginsights/lib/parquet/parquet-mr/parquet-thrift-1.3.2.jar,file:///opt/ibm/biginsights/hive/lib/guava-11.0.2.jar
如果您加入所有3個表中的記錄的100％，那麼統計數據不會有太大的差異。 40分鐘的大規模Hive查詢並不令人意外。一些興趣點可能是：MapReduce或Tez？什麼樣的連接*（參考EXPLAIN）*？每個連接步驟有多少個映射器*（參見MR或Tez日誌）*？大量的數據傾斜*（比較一些mappers咀嚼數據比其他更多，並放慢步伐）*？壓縮中間結果？等等等等等等等等等等。 –
當我使用STORED PARAMETER作爲ORC而不是PARQUET時，我的性能得到了很好的改善。你能讓我爲什麼這樣嗎？ –
你爲什麼選擇Hive + Parquet？由於Cloudera營銷炒作？看到我對那篇文章的評論（和BTW後）http://stackoverflow.com/questions/32350664/why-is-parquet-slower-for-me-against-text-file-format-in-hive#comment52629679_32350664 –