2015-10-30 69 views
16

我有許多以logstash-Year-Week格式索引的日誌。那就是如果我想刪除超過幾周的索引,那麼我怎樣才能在elasticsearch中實現這一點。有沒有一種簡單,無縫的方式來做到這一點?刪除elasticsearch中的舊索引

回答

19

館長將是這裏的理想匹配。 您可以在這裏找到鏈接 - https://github.com/elastic/curator

像下面的命令應該只是罰款 -

curator --host <IP> delete indices --older-than 30 --prefix "twitter-" --time-unit days --timestring '%Y-%m-%d' 

你可以保持這個在CRON用於偶爾去除指數。

你可以找到一些例子和文檔在這裏 - https://www.elastic.co/guide/en/elasticsearch/client/curator/current/examples.html

+0

這正是我一直在尋找。你有關於館長應用的文件嗎? –

+4

這不適用於策展人v4或更新版本。它需要一個配置文件和一個動作文件,其中描述了策展人的動作。 –

+0

查看@ sachchit-bansal作爲策展人的回答4.2範例 – chrisan

6

請看Curator,這是專門爲這種用例開發的工具。

的樣本命令,對於文檔:

curator --host 10.0.0.2 delete indices --older-than 30 --time-unit days \ 
    --timestring '%Y.%m.%d' 
10

我使用bash腳本,只是改變了30你要保持

#!/bin/bash 

# Zero padded days using %d instead of %e 
DAYSAGO=`date --date="30 days ago" +%Y%m%d` 
ALLLINES=`/usr/bin/curl -s -XGET http://127.0.0.1:9200/_cat/indices?v | egrep logstash` 

echo 
echo "THIS IS WHAT SHOULD BE DELETED FOR ELK:" 
echo 

echo "$ALLLINES" | while read LINE 
do 
    FORMATEDLINE=`echo $LINE | awk '{ print $3 }' | awk -F'-' '{ print $2 }' | sed 's/\.//g' ` 
    if [ "$FORMATEDLINE" -lt "$DAYSAGO" ] 
    then 
    TODELETE=`echo $LINE | awk '{ print $3 }'` 
    echo "http://127.0.0.1:9200/$TODELETE" 
    fi 
done 

echo 
echo -n "if this make sence, Y to continue N to exit [Y/N]:" 
read INPUT 
if [ "$INPUT" == "Y" ] || [ "$INPUT" == "y" ] || [ "$INPUT" == "yes" ] || [ "$INPUT" == "YES" ] 
then 
    echo "$ALLLINES" | while read LINE 
    do 
    FORMATEDLINE=`echo $LINE | awk '{ print $3 }' | awk -F'-' '{ print $2 }' | sed 's/\.//g' ` 
    if [ "$FORMATEDLINE" -lt "$DAYSAGO" ] 
    then 
     TODELETE=`echo $LINE | awk '{ print $3 }'` 
     /usr/bin/curl -XDELETE http://127.0.0.1:9200/$TODELETE 
     sleep 1 
     fi 
    done 
else 
    echo SCRIPT CLOSED BY USER, BYE ... 
    echo 
    exit 
fi 
0

yanb(另一個bash)的天#

#!/bin/bash 
searchIndex=logstash-monitor 
elastic_url=logging.core.k94.kvk.nl 
elastic_port=9200 

date2stamp() { 
    date --utc --date "$1" +%s 
} 

dateDiff(){ 
    case $1 in 
     -s) sec=1;  shift;; 
     -m) sec=60;  shift;; 
     -h) sec=3600; shift;; 
     -d) sec=86400; shift;; 
     *) sec=86400;; 
    esac 
    dte1=$(date2stamp $1) 
    dte2=$(date2stamp $2) 
    diffSec=$((dte2-dte1)) 
    if ((diffSec < 0)); then abs=-1; else abs=1; fi 
    echo $((diffSec/sec*abs)) 
} 

for index in $(curl -s "${elastic_url}:${elastic_port}/_cat/indices?v" |  grep -E " ${searchIndex}-20[0-9][0-9]\.[0-1][0-9]\.[0-3][0-9]" | awk '{  print $3 }'); do 
    date=$(echo ${index: -10} | sed 's/\./-/g') 
    cond=$(date +%Y-%m-%d) 
    diff=$(dateDiff -d $date $cond) 
    echo -n "${index} (${diff})" 
    if [ $diff -gt 1 ]; then 
    echo "/DELETE" 
    # curl -XDELETE "${elastic_url}:${elastic_port}/${index}?pretty" 
    else 
    echo "" 
    fi 
done  
13

如果您使用elasticsearch版本5.x,則需要安裝策展人版本4.x。 您可以從documentation

中看到版本兼容性和安裝步驟。安裝完成。然後,只需運行命令

curator --config path/config_file.yml [--dry-run] path/action_file.yml 

Curator提供幹運行標誌以輸出Curator會執行的操作。輸出將位於您在config.yml文件中定義的日誌文件中。如果沒有在config_file.yml中定義記錄密鑰,那麼currator將輸出到控制檯。要刪除索引,而不--dry-運行標誌運行上述命令

配置文件config_file.yml是

--- 
client: 
    hosts: 
    - 127.0.0.1 
    port: 9200 
logging: 
    loglevel: INFO 
    logfile: "/root/curator/logs/actions.log" 
    logformat: default 
    blacklist: ['elasticsearch', 'urllib3'] 

操作文件action_file.yml是

--- 
actions: 
    1: 
    action: delete_indices 
    description: >- 
     Delete indices older than 7 days (based on index name), for logstash- 
     prefixed indices. Ignore the error if the filter does not result in an 
     actionable list of indices (ignore_empty_list) and exit cleanly. 
    options: 
     ignore_empty_list: True 
     timeout_override: 
     continue_if_exception: False 
     disable_action: False 
    filters: 
    - filtertype: pattern 
     kind: prefix 
     value: logstash- 
     exclude: 
    - filtertype: age 
     source: name 
     direction: older 
     timestring: '%Y.%m.%d' 
     unit: days 
     unit_count: 7 
     exclude: 

如果你想自動刪除指數每週,每月等。然後就寫這樣

#!/bin/bash 
# Script to delete the log event indices of the elasticsearch weekly 

#This will delete the indices of the last 7 days 
curator --config /path/config_file.yml /path/action_file.yml 

的bash腳本把shell腳本這些文件夾中的一個:/etc/cron.daily, /etc/cron.hourly, /etc/cron.monthly or /etc/cron.weekly和你的工作就完成了。

注意:請確保在您的配置和操作文件中使用正確的縮進。否則它將無法工作。

+1

謝謝,這是針對策展人4.2這個答案的當前(2017)版本:) – chrisan

+0

這就是策展人工作的方式! [Vineeth Mohan](https://stackoverflow.com/users/976646/vineeth-mohan)的答案已經過時,並且目前應該適用於大多數彈性搜索安裝(其中5.x是最新的)。 – jonashackt

0
curator_cli delete_indices --filter_list '{"filtertype":"none"}' 

將刪除所有或過濾器:

--filter_list '[{"filtertype":"age","source":"creation_date","direction":"older","unit":"days","unit_count":13},{"filtertype":"pattern","kind":"prefix","value":"logstash"}]' 
相關問題