如何使用Eclipse和Java從Google BigQuery接收數據？

Question

我試圖從Google BigQuery接收一些數據，並在我的eclipse控制檯中顯示結果以開始使用BigQuery和Java。基本上，我跟着谷歌提供的說明，我在這裏找到： https://cloud.google.com/bigquery/docs/reference/libraries

所以，我的例子中的Java代碼看起來像谷歌在快速入門示例：

import com.google.cloud.bigquery.BigQuery; 
import com.google.cloud.bigquery.BigQueryOptions; 
import com.google.cloud.bigquery.Dataset; 
import com.google.cloud.bigquery.DatasetInfo; 

public class QuickstartSample { 
    public static void main(String... args){ 
     // Instantiates a client 
     BigQuery bigquery = BigQueryOptions.getDefaultInstance().getService(); 

     // The name for the new dataset 
     String datasetName = "bigquery-public-data:hacker_news.comments"; 

     // Prepares a new dataset 
     Dataset dataset = null; 
     DatasetInfo datasetInfo = DatasetInfo.newBuilder(datasetName).build(); 

     // Creates the dataset 
     dataset = bigquery.create(datasetInfo); 

     System.out.printf("Dataset %s created.%n", dataset.getDatasetId().getDataset()); 
    } 
} 

但是當我運行此代碼我出現以下錯誤：

HTTP ERROR 500 

Problem accessing /getbigquery. Reason: 
    Could not get the access token. 


Caused by: 
com.google.cloud.bigquery.BigQueryException: Could not get the access token. 
    at com.google.cloud.bigquery.spi.v2.HttpBigQueryRpc.translate(HttpBigQueryRpc.java:86) 
    at com.google.cloud.bigquery.spi.v2.HttpBigQueryRpc.create(HttpBigQueryRpc.java:141) 
    at com.google.cloud.bigquery.BigQueryImpl$1.call(BigQueryImpl.java:172) 
    at com.google.cloud.bigquery.BigQueryImpl$1.call(BigQueryImpl.java:169) 
    at com.google.api.gax.retrying.DirectRetryingExecutor.submit(DirectRetryingExecutor.java:94) 
    at com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:54) 
    at com.google.cloud.bigquery.BigQueryImpl.create(BigQueryImpl.java:169) 
    at testpackage.dto.QuickstartSample.main(QuickstartSample.java:27) 
    at testpackage.getbigquery.doPost(getbigquery.java:27) 
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:637) 
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:717) 
    at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511) 
    at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1166) 
    at com.google.appengine.api.socket.dev.DevSocketFilter.doFilter(DevSocketFilter.java:74) 
    at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157) 
    at com.google.appengine.tools.development.ResponseRewriterFilter.doFilter(ResponseRewriterFilter.java:134) 
    at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157) 
    at com.google.appengine.tools.development.HeaderVerificationFilter.doFilter(HeaderVerificationFilter.java:34) 
    at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157) 
    at com.google.appengine.api.blobstore.dev.ServeBlobFilter.doFilter(ServeBlobFilter.java:63) 
    at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157) 
    at com.google.apphosting.utils.servlet.TransactionCleanupFilter.doFilter(TransactionCleanupFilter.java:48) 
    at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157) 
    at com.google.appengine.tools.development.StaticFileFilter.doFilter(StaticFileFilter.java:122) 
    at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157) 

我試圖用默認證書進行驗證，所以我在我的谷歌SDK控制檯跑「gcloud身份驗證的應用程序默認登錄」。儘管我可以在SDK中登錄，但eclipse給了我以前顯示的錯誤。

然後我嘗試了其他一些東西，例如將我的令牌下載爲Json文件，並在我的應用程序中使用java切換到它或手動設置我的環境變量，但它不起作用。

有沒有人有想法我做錯了什麼？任何幫助表示讚賞。

Answer 1

這聽起來像是你打a bug in the App Engine SDK，阻止服務使用gcloud應用程序默認憑證。有a workaround：重命名（或刪除）google-cloud-sdk/platform/google_appengine/google/appengine/tools/java/lib/shared/appengine-remote-api.jar。

您還需要配置GCP項目以用於存儲和訪問BigQuery數據集。該項目可以通過選項卡設置com.google.appengine.application.id系統屬性中啓動配置的參數（通過運行>運行配置然後添加-Dcom.google.appengine.application.id=projectID到參數> VM參數部分）進行配置。但如果你從一個不同的項目訪問的數據集（例如，bigquery-public-data數據集），你將要明確配置BigQuery建設者如下：

BigQueryOptions.newBuilder() 
    .setProjectId("projectID") 
    .build().getService(); 

最後，你還必須包括appengine-api-1.0-sdk依賴通過增加下面您pom.xml：

<dependency> 
    <groupId>com.google.appengine</groupId> 
    <artifactId>appengine-api-1.0-sdk</artifactId> 
    <version>1.9.50</version> 
</dependency> 

（或更新版本）

這就是說，雲工具的Eclipse應使指定默認應用程序憑證和項目更容易，所以我提交了ticket to improve this situation。

Answer 2

應將以下內容添加到您的Maven項目

<!--Google Bigquery--> 
    <dependency> 
     <groupId>com.google.cloud</groupId> 
     <artifactId>google-cloud-bigquery</artifactId> 
     <version>0.8.0-beta</version> 
    </dependency> 

建立在你的谷歌雲平臺控制檯 - 服務帳戶> IAM->必要的許可服務帳戶的pom.xml。您將被允許下載JSON密鑰文件。

import com.google.api.client.googleapis.auth.oauth2.GoogleCredential; 
    import com.google.api.client.http.javanet.NetHttpTransport; 
    import com.google.api.client.json.jackson2.JacksonFactory; 
    import com.google.api.client.util.Data; 
    import com.google.api.services.bigquery.model.*; 
    import com.google.api.services.bigquery.Bigquery; 
    import com.google.api.services.bigquery.BigqueryScopes; 
    import java.io.ByteArrayInputStream; 


    //Read the content of the service account key file in following variable. Please take appropriate security measures into the account. 
    String credentials = "Content of JSON secret file"; 

    //Create bigquery client 
    NetHttpTransport transport = new NetHttpTransport(); 
    JacksonFactory jsonFactory = new JacksonFactory(); 
    GoogleCredential credential = GoogleCredential.fromStream(new ByteArrayInputStream(credentials.getBytes())); 
    if (credential.createScopedRequired) credential = credential.createScoped(BigqueryScopes.all); 
    Bigquery bigqueryClient = new Bigquery.Builder(transport, jsonFactory, credential).setApplicationName("Your Application Name").build(); 

    String projectId = "Your-project-id"; 
    String datasetId = "NewDatasetName"; 

    Dataset dataset = new Dataset(); 

    DatasetReference datasetRef = new DatasetReference(); 
    datasetRef.setProjectId(projectId); 
    datasetRef.setDatasetId(datasetId); 

    dataset.setDatasetReference(datasetRef); 

    try { 
     bigqueryClient.datasets().insert(projectId, dataset).execute(); 
    } catch (Exception e){ 
    //Handle exceptions 
    }