Elasticsearch獲取帶有過濾器的所有數據

Question

我想從不帶可分頁的過濾器獲取elasticsearch的所有數據。最好的方法是什麼？我`v得到了默認限制設置爲2000.我讀我應該使用掃描，但我不知道我應該如何使用它。我應該如何使用掃描和滾動來獲取所有數據？

public Map searchByIndexParams(AuctionIndexSearchParams searchParams, Pageable pageable) { 
     final List<FilterBuilder> filters = Lists.newArrayList(); 
     final NativeSearchQueryBuilder searchQuery = new NativeSearchQueryBuilder().withQuery(matchAllQuery()); 



     Optional.ofNullable(searchParams.getCategoryId()).ifPresent(v -> filters.add(boolFilter().must(termFilter("cat", v)))); 
     Optional.ofNullable(searchParams.getCurrency()).ifPresent(v -> filters.add(boolFilter().must(termFilter("curr", v)))); 
     Optional.ofNullable(searchParams.getTreeCategoryId()).ifPresent(v -> filters.add(boolFilter().must(termFilter("tcat", v)))); 
     Optional.ofNullable(searchParams.getUid()).ifPresent(v -> filters.add(boolFilter().must(termFilter("uid", v)))); 


     //access for many uids 
     if(searchParams.getUids() != null){ 
      Optional.ofNullable(searchParams.getUids().split(",")).ifPresent(v -> { 
       filters.add(boolFilter().must(termsFilter("uid", v))); 
      }); 
     } 


     //access for many categories 
     if(searchParams.getCategories() != null){ 
      Optional.ofNullable(searchParams.getCategories().split(",")).ifPresent(v -> { 
       filters.add(boolFilter().must(termsFilter("cat", v))); 
      }); 
     } 

     final BoolQueryBuilder boolQueryBuilder = new BoolQueryBuilder(); 

     if (Optional.ofNullable(searchParams.getTitle()).isPresent()) { 
      boolQueryBuilder.should(queryStringQuery(searchParams.getTitle()).analyzeWildcard(true).field("title")); 
     } 

     if (Optional.ofNullable(searchParams.getStartDateFrom()).isPresent() 
       || Optional.ofNullable(searchParams.getStartDateTo()).isPresent()) { 
      filters.add(rangeFilter("start_date").from(searchParams.getStartDateFrom()).to(searchParams.getStartDateTo())); 
     } 

     if (Optional.ofNullable(searchParams.getEndDateFrom()).isPresent() 
       || Optional.ofNullable(searchParams.getEndDateTo()).isPresent()) { 
      filters.add(rangeFilter("end_date").from(searchParams.getEndDateFrom()).to(searchParams.getEndDateTo())); 
     } 

     if (Optional.ofNullable(searchParams.getPriceFrom()).isPresent() 
       || Optional.ofNullable(searchParams.getPriceTo()).isPresent()) { 
      filters.add(rangeFilter("price").from(searchParams.getPriceFrom()).to(searchParams.getPriceTo())); 
     } 


     searchQuery.withQuery(boolQueryBuilder); 

     FilterBuilder[] filterArr = new FilterBuilder[filters.size()]; 
     filterArr = filters.toArray(filterArr); 
     searchQuery.withFilter(andFilter(filterArr)); 


     final FacetedPage<AuctionIndex> search = auctionIndexRepository.search(searchQuery.build()); 


     response.put("content", search.map(index ->auctionRepository 
       .findAuctionById(Long.valueOf(index.getId()))) 
       .getContent()); 

     return response; 
    } 

編輯：

I`v有：

String scrollId = searchTemplate.scan(searchQuery.build(), 1000, false); 

     Page<AuctionIndex> page = searchTemplate.scroll(scrollId, 15000L, AuctionIndex.class); 
     Integer i = 0; 
     if (page != null && page.hasContent()) { 

      while(page.hasContent()){ 

       page = searchTemplate.scroll(scrollId, 15000L, AuctionIndex.class); 

       if(page.hasContent()){ 
        System.out.println(i); 
        i++; 
       } 


      } 

     } 

但迭代到166，並停止什麼`錯了嗎？

Answer 1

Scroll API是以最有效的方式瀏覽所有文檔的最佳方式。使用scroll_id您可以找到存儲在服務器上的特定滾動請求的會話。

下面是一個示例，您可以在代碼中使用elasticsearch java scroll api獲取與查詢匹配的所有結果。

SearchResponse searchResponse = client.prepareSearch(<INDEX>) 
      .setQuery(<QUERY>) 
      .setSearchType(SearchType.SCAN) 
      .setScroll(SCROLL_TIMEOUT) 
      .setSize(SCROLL_SIZE) 
      .execute() 
      .actionGet(); 

while (true) { 
     searchResponse = client 
       .prepareSearchScroll(searchResponse.getScrollId()) 
       .setScroll(SCROLL_TIMEOUT) 
       .execute().actionGet(); 

     if (searchResponse.getHits().getHits().length == 0) { 
      break; //Break condition: No hits are returned 
     } 

     for (SearchHit hit : searchResponse.getHits()) { 
      // process response 
     } 
    } 

樣品使用Spring的數據elasticsearch

@Autowired 
private ElasticsearchTemplate searchTemplate; 

String scrollId = searchTemplate.scan(<SEARCH_QUERY>, 1000, false); 

Page<ExampleItem> page = searchTemplate.scroll(scrollId, 5000L, ExampleItem.class); 
if (page != null && page.hasContent()) { 
// process first batch 
    while (page != null && page.hasContent()) { 
     page = searchTemplate.scroll(scrollId, 5000L, ExampleItem.class); 
     if (page != null && page.hasContent()) { 
      // process remaining batches 
     } 
    } 
} 

這裏，ExampleItem指定是要獲取的實體。