一、ELK搭建篇

ELK是Elasticsearch、Logstash、Kibana的簡(jiǎn)稱，這三者是核心套件，但并非全部。

Elasticsearch是實(shí)時(shí)全文搜索和分析引擎，提供搜集、分析、存儲(chǔ)數(shù)據(jù)三大功能；是一套開放REST和JAVA API等結(jié)構(gòu)提供高效搜索功能，可擴(kuò)展的分布式系統(tǒng)。它構(gòu)建于Apache Lucene搜索引擎庫(kù)之上。

Logstash是一個(gè)用來搜集、分析、過濾日志的工具。它支持幾乎任何類型的日志，包括系統(tǒng)日志、錯(cuò)誤日志和自定義應(yīng)用程序日志。它可以從許多來源接收日志，這些來源包括 syslog、消息傳遞（例如 RabbitMQ）和JMX，它能夠以多種方式輸出數(shù)據(jù)，包括電子郵件、websockets和Elasticsearch。

Kibana是一個(gè)基于Web的圖形界面，用于搜索、分析和可視化存儲(chǔ)在 Elasticsearch指標(biāo)中的日志數(shù)據(jù)。它利用Elasticsearch的REST接口來檢索數(shù)據(jù)，不僅允許用戶創(chuàng)建他們自己的數(shù)據(jù)的定制儀表板視圖，還允許他們以特殊的方式查詢和過濾數(shù)據(jù)

環(huán)境

Centos6.5 兩臺(tái)IP：192.168.1.202 安裝： elasticsearch、logstash、Kibana、Nginx、Http、Redis 192.168.1.201 安裝: logstash

安裝

安裝elasticsearch的yum源的密鑰（這個(gè)需要在所有服務(wù)器上都配置）# rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch配置elasticsearch的yum源# vim /etc/yum.repos.d/elasticsearch.repo在elasticsearch.repo文件中添加如下內(nèi)容[elasticsearch-5.x]name=Elasticsearch repository for 5.x packagesbaseurl=https://artifacts.elastic.co/packages/5.x/yumgpgcheck=1gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearchenabled=1autorefresh=1type=rpm-md

安裝elasticsearch的環(huán)境

安裝elasticsearch# yum install -y elasticsearch安裝java環(huán)境(java環(huán)境必須是1.8版本以上的)wget http://download.oracle.com/otn-pub/java/jdk/8u131-b11/d54c1d3a095b4ff2b6607d096fa80163/jdk-8u131-linux-x64.rpmrpm -ivh jdk-8u131-linux-x64.rpm 驗(yàn)證java安裝成功java -versionjava version "1.8.0_131"Java(TM) SE Runtime Environment (build 1.8.0_131-b11)Java HotSpot(TM) 64-Bit Server VM (build 25.131-b11, mixed mode)

創(chuàng)建elasticsearch data的存放目錄，并修改該目錄的屬主屬組

# mkdir -p /data/es-data (自定義用于存放data數(shù)據(jù)的目錄)# chown -R elasticsearch:elasticsearch /data/es-data

修改elasticsearch的日志屬主屬組

# chown -R elasticsearch:elasticsearch /var/log/elasticsearch/

修改elasticsearch的配置文件

# vim /etc/elasticsearch/elasticsearch.yml找到配置文件中的cluster.name，打開該配置并設(shè)置集群名稱cluster.name: demon找到配置文件中的node.name，打開該配置并設(shè)置節(jié)點(diǎn)名稱node.name: elk-1修改data存放的路徑path.data: /data/es-data修改logs日志的路徑path.logs: /var/log/elasticsearch/配置內(nèi)存使用用交換分區(qū)bootstrap.memory_lock: true監(jiān)聽的網(wǎng)絡(luò)地址network.host: 0.0.0.0開啟監(jiān)聽的端口http.port: 9200增加新的參數(shù)，這樣head插件可以訪問es (5.x版本，如果沒有可以自己手動(dòng)加)http.cors.enabled: truehttp.cors.allow-origin: "*"啟動(dòng)elasticsearch服務(wù)

啟動(dòng)服務(wù)

/etc/init.d/elasticsearch startStarting elasticsearch: Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x0000000085330000, 2060255232, 0) failed; error='Cannot allocate memory' (errno=12)## There is insufficient memory for the Java Runtime Environment to continue.# Native memory allocation (mmap) failed to map 2060255232 bytes for committing reserved memory.# An error report file with more information is saved as:# /tmp/hs_err_pid2616.log [FAILED]這個(gè)報(bào)錯(cuò)是因?yàn)槟J(rèn)使用的內(nèi)存大小為2G，虛擬機(jī)沒有那么多的空間修改參數(shù)：vim /etc/elasticsearch/jvm.options-Xms512m-Xmx512m再次啟動(dòng)/etc/init.d/elasticsearch start查看服務(wù)狀態(tài)，如果有報(bào)錯(cuò)可以去看錯(cuò)誤日志 less /var/log/elasticsearch/demon.log(日志的名稱是以集群名稱命名的)創(chuàng)建開機(jī)自啟動(dòng)服務(wù)# chkconfig elasticsearch on

注意事項(xiàng)

需要修改幾個(gè)參數(shù)，不然啟動(dòng)會(huì)報(bào)錯(cuò)vim /etc/security/limits.conf在末尾追加以下內(nèi)容（elk為啟動(dòng)用戶，當(dāng)然也可以指定為*）elk soft nofile 65536elk hard nofile 65536elk soft nproc 2048elk hard nproc 2048elk soft memlock unlimitedelk hard memlock unlimited繼續(xù)再修改一個(gè)參數(shù)vim /etc/security/limits.d/90-nproc.conf將里面的1024改為2048（ES最少要求為2048）* soft nproc 2048另外還需注意一個(gè)問題（在日志發(fā)現(xiàn)如下內(nèi)容，這樣也會(huì)導(dǎo)致啟動(dòng)失敗，這一問題困擾了很久）[2017-06-14T19:19:01,641][INFO ][o.e.b.BootstrapChecks ] [elk-1] bound or publishing to a non-loopback or non-link-local address, enforcing bootstrap checks[2017-06-14T19:19:01,658][ERROR][o.e.b.Bootstrap ] [elk-1] node validation exception[1] bootstrap checks failed[1]: system call filters failed to install; check the logs and fix your configuration or disable system call filters at your own risk 解決：修改配置文件，在配置文件添加一項(xiàng)參數(shù)（目前還沒明白此參數(shù)的作用）vim /etc/elasticsearch/elasticsearch.yml bootstrap.system_call_filter: false

通過瀏覽器請(qǐng)求下9200的端口，看下是否成功

先檢查9200端口是否起來netstat -antp |grep 9200tcp 0 0 :::9200 :::* LISTEN 2934/java瀏覽器訪問測(cè)試是否正常（以下為正常）# curl http://127.0.0.1:9200/{ "name" : "linux-node1", "cluster_name" : "demon", "cluster_uuid" : "kM0GMFrsQ8K_cl5Fn7BF-g", "version" : { "number" : "5.4.0", "build_hash" : "780f8c4", "build_date" : "2017-04-28T17:43:27.229Z", "build_snapshot" : false, "lucene_version" : "6.5.0" }, "tagline" : "You Know, for Search"}

如何和elasticsearch交互

JavaAPIRESTful APIJavascript,.Net,PHP,Perl,Python利用API查看狀態(tài)# curl -i -XGET 'localhost:9200/_count?pretty' HTTP/1.1 200 OK content-type: application/json; charset=UTF-8 content-length: 95 { "count" : 0, "_shards" : { "total" : 0, "successful" : 0, "failed" : 0 } }

安裝插件

安裝elasticsearch-head插件安裝docker鏡像或者通過github下載elasticsearch-head項(xiàng)目都是可以的，1或者2兩種方式選擇一種安裝使用即可1. 使用docker的集成好的elasticsearch-head # docker run -p 9100:9100 mobz/elasticsearch-head:5 docker容器下載成功并啟動(dòng)以后，運(yùn)行瀏覽器打開http://localhost:9100/2. 使用git安裝elasticsearch-head # yum install -y npm # git clone git://github.com/mobz/elasticsearch-head.git # cd elasticsearch-head # npm install # npm run start 檢查端口是否起來 netstat -antp |grep 9100 瀏覽器訪問測(cè)試是否正常 http://IP:9100/

LogStash的使用

安裝Logstash環(huán)境：官方安裝手冊(cè):https://www.elastic.co/guide/en/logstash/current/installing-logstash.html下載yum源的密鑰認(rèn)證：# rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch利用yum安裝logstash# yum install -y logstash查看下logstash的安裝目錄# rpm -ql logstash創(chuàng)建一個(gè)軟連接，每次執(zhí)行命令的時(shí)候不用在寫安裝路勁（默認(rèn)安裝在/usr/share下）ln -s /usr/share/logstash/bin/logstash /bin/執(zhí)行l(wèi)ogstash的命令# logstash -e 'input { stdin { } } output { stdout {} }'運(yùn)行成功以后輸入:nihaostdout返回的結(jié)果:

注: -e 執(zhí)行操作 input 標(biāo)準(zhǔn)輸入 { input } 插件 output 標(biāo)準(zhǔn)輸出 { stdout } 插件 通過rubydebug來輸出下更詳細(xì)的信息# logstash -e 'input { stdin { } } output { stdout {codec => rubydebug} }'執(zhí)行成功輸入:nihaostdout輸出的結(jié)果:

如果標(biāo)準(zhǔn)輸出還有elasticsearch中都需要保留應(yīng)該怎么玩，看下面# /usr/share/logstash/bin/logstash -e 'input { stdin { } } output { elasticsearch { hosts => ["192.168.1.202:9200"] } stdout { codec => rubydebug }}'運(yùn)行成功以后輸入:I am elk返回的結(jié)果(標(biāo)準(zhǔn)輸出中的結(jié)果):

logstash使用配置文件

官方指南:https://www.elastic.co/guide/en/logstash/current/configuration.html創(chuàng)建配置文件01-logstash.conf# vim /etc/logstash/conf.d/elk.conf文件中添加以下內(nèi)容input { stdin { } }output { elasticsearch { hosts => ["192.168.1.202:9200"] } stdout { codec => rubydebug }}使用配置文件運(yùn)行l(wèi)ogstash# logstash -f ./elk.conf運(yùn)行成功以后輸入以及標(biāo)準(zhǔn)輸出結(jié)果

logstash的數(shù)據(jù)庫(kù)類型

1. Input插件 權(quán)威指南:https://www.elastic.co/guide/en/logstash/current/input-plugins.html file插件的使用 # vim /etc/logstash/conf.d/elk.conf 添加如下配置 input { file { path => "/var/log/messages" type => "system" start_position => "beginning" } } output { elasticsearch { hosts => ["192.168.1.202:9200"] index => "system-%{+YYYY.MM.dd}" } }運(yùn)行l(wèi)ogstash指定elk.conf配置文件，進(jìn)行過濾匹配#logstash -f /etc/logstash/conf.d/elk.conf

來一發(fā)配置安全日志的并且把日志的索引按類型做存放，繼續(xù)編輯elk.conf文件

# vim /etc/logstash/conf.d/elk.conf添加secure日志的路徑input { file { path => "/var/log/messages" type => "system" start_position => "beginning" } file { path => "/var/log/secure" type => "secure" start_position => "beginning" }}output { if [type] == "system" { elasticsearch { hosts => ["192.168.1.202:9200"] index => "nagios-system-%{+YYYY.MM.dd}" } } if [type] == "secure" { elasticsearch { hosts => ["192.168.1.202:9200"] index => "nagios-secure-%{+YYYY.MM.dd}" } }}運(yùn)行l(wèi)ogstash指定elk.conf配置文件，進(jìn)行過濾匹配# logstash -f ./elk.conf

這些設(shè)置都沒有問題之后，接下來安裝下kibana，可以讓在前臺(tái)展示

Kibana的安裝及使用

安裝kibana環(huán)境官方安裝手冊(cè):https://www.elastic.co/guide/en/kibana/current/install.html下載kibana的tar.gz的軟件包# wget https://artifacts.elastic.co/downloads/kibana/kibana-5.4.0-linux-x86_64.tar.gz解壓kibana的tar包# tar -xzf kibana-5.4.0-linux-x86_64.tar.gz進(jìn)入解壓好的kibana# mv kibana-5.4.0-linux-x86_64 /usr/local創(chuàng)建kibana的軟連接# ln -s /usr/local/kibana-5.4.0-linux-x86_64/ /usr/local/kibana 編輯kibana的配置文件# vim /usr/local/kibana/config/kibana.yml修改配置文件如下，開啟以下的配置server.port: 5601server.host: "0.0.0.0"elasticsearch.url: "http://192.168.1.202:9200"kibana.index: ".kibana" 安裝screen,以便于kibana在后臺(tái)運(yùn)行（當(dāng)然也可以不用安裝，用其他方式進(jìn)行后臺(tái)啟動(dòng)）# yum -y install screen# screen# /usr/local/kibana/bin/kibananetstat -antp |grep 5601tcp 0 0 0.0.0.0:5601 0.0.0.0:* LISTEN 17007/node 打開瀏覽器并設(shè)置對(duì)應(yīng)的indexhttp://IP:5601

二、ELK實(shí)戰(zhàn)篇

好，現(xiàn)在索引也可以創(chuàng)建了，現(xiàn)在可以來輸出nginx、apache、message、secrue的日志到前臺(tái)展示（Nginx有的話直接修改，沒有自行安裝）

編輯nginx配置文件，修改以下內(nèi)容（在http模塊下添加）log_format json '{"@timestamp":"$time_iso8601",' '"@version":"1",' '"client":"$remote_addr",' '"url":"$uri",' '"status":"$status",' '"domian":"$host",' '"host":"$server_addr",' '"size":"$body_bytes_sent",' '"responsetime":"$request_time",' '"referer":"$http_referer",' '"ua":"$http_user_agent"' '}';修改access_log的輸出格式為剛才定義的json access_log logs/elk.access.log json;繼續(xù)修改apache的配置文件LogFormat "{ \ "@timestamp": "%{%Y-%m-%dT%H:%M:%S%z}t", \ "@version": "1", \ "tags":["apache"], \ "message": "%h %l %u %t \\"%r\\" %>s %b", \ "clientip": "%a", \ "duration": %D, \ "status": %>s, \ "request": "%U%q", \ "urlpath": "%U", \ "urlquery": "%q", \ "bytes": %B, \ "method": "%m", \ "site": "%{Host}i", \ "referer": "%{Referer}i", \ "useragent": "%{User-agent}i" \ }" ls_apache_json一樣修改輸出格式為上面定義的json格式CustomLog logs/access_log ls_apache_json編輯logstash配置文件，進(jìn)行日志收集vim /etc/logstash/conf.d/full.conf input { file { path => "/var/log/messages" type => "system" start_position => "beginning" } file { path => "/var/log/secure" type => "secure" start_position => "beginning" } file { path => "/var/log/httpd/access_log" type => "http" start_position => "beginning" } file { path => "/usr/local/nginx/logs/elk.access.log" type => "nginx" start_position => "beginning" } } output { if [type] == "system" { elasticsearch { hosts => ["192.168.1.202:9200"] index => "nagios-system-%{+YYYY.MM.dd}" } } if [type] == "secure" { elasticsearch { hosts => ["192.168.1.202:9200"] index => "nagios-secure-%{+YYYY.MM.dd}" } } if [type] == "http" { elasticsearch { hosts => ["192.168.1.202:9200"] index => "nagios-http-%{+YYYY.MM.dd}" } } if [type] == "nginx" { elasticsearch { hosts => ["192.168.1.202:9200"] index => "nagios-nginx-%{+YYYY.MM.dd}" } }}運(yùn)行看看效果如何logstash -f /etc/logstash/conf.d/full.conf

可以發(fā)現(xiàn)所有創(chuàng)建日志的索引都已存在，接下來就去Kibana創(chuàng)建日志索引，進(jìn)行展示（按照上面的方法進(jìn)行創(chuàng)建索引即可），看下展示的效果

接下來再來一發(fā)MySQL慢日志的展示

由于MySQL的慢日志查詢格式比較特殊，所以需要用正則進(jìn)行匹配，并使用multiline能夠進(jìn)行多行匹配（看具體配置）input { file { path => "/var/log/messages" type => "system" start_position => "beginning" } file { path => "/var/log/secure" type => "secure" start_position => "beginning" } file { path => "/var/log/httpd/access_log" type => "http" start_position => "beginning" } file { path => "/usr/local/nginx/logs/elk.access.log" type => "nginx" start_position => "beginning" } file { path => "/var/log/mysql/mysql.slow.log" type => "mysql" start_position => "beginning" codec => multiline { pattern => "^# User@Host:" negate => true what => "previous" } }}filter { grok { match => { "message" => "SELECT SLEEP" } add_tag => [ "sleep_drop" ] tag_on_failure => [] } if "sleep_drop" in [tags] { drop {} } grok { match => { "message" => "(?m)^# User@Host: %{USER:User}\[[^\]]+\] @ (?:(?\S*) )?\[(?:%{IP:Client_IP})?\]\s.*# Query_time: %{NUMBER:Query_Time:float}\s+Lock_time: %{NUMBER:Lock_Time:float}\s+Rows_sent: %{NUMBER:Rows_Sent:int}\s+Rows_examined: %{NUMBER:Rows_Examined:int}\s*(?:use %{DATA:Database};\s*)?SET timestamp=%{NUMBER:timestamp};\s*(?(?\w+)\s+.*)\n# Time:.*$" } } date { match => [ "timestamp", "UNIX" ] remove_field => [ "timestamp" ] } } output { if [type] == "system" { elasticsearch { hosts => ["192.168.1.202:9200"] index => "nagios-system-%{+YYYY.MM.dd}" } } if [type] == "secure" { elasticsearch { hosts => ["192.168.1.202:9200"] index => "nagios-secure-%{+YYYY.MM.dd}" } } if [type] == "http" { elasticsearch { hosts => ["192.168.1.202:9200"] index => "nagios-http-%{+YYYY.MM.dd}" } } if [type] == "nginx" { elasticsearch { hosts => ["192.168.1.202:9200"] index => "nagios-nginx-%{+YYYY.MM.dd}" } } if [type] == "mysql" { elasticsearch { hosts => ["192.168.1.202:9200"] index => "nagios-mysql-slow-%{+YYYY.MM.dd}" } }}

查看效果(一條慢日志查詢會(huì)顯示一條，如果不進(jìn)行正則匹配，那么一行就會(huì)顯示一條)

具體的日志輸出需求，進(jìn)行具體的分析

三：ELK終極篇

安裝reids # yum install -y redis修改redis的配置文件# vim /etc/redis.conf修改內(nèi)容如下daemonize yesbind 192.168.1.202啟動(dòng)redis服務(wù)# /etc/init.d/redis restart測(cè)試redis的是否啟用成功# redis-cli -h 192.168.1.202輸入info如果有不報(bào)錯(cuò)即可redis 192.168.1.202:6379> inforedis_version:2.4.10....編輯配置redis-out.conf配置文件，把標(biāo)準(zhǔn)輸入的數(shù)據(jù)存儲(chǔ)到redis中# vim /etc/logstash/conf.d/redis-out.conf添加如下內(nèi)容input { stdin {}}output { redis { host => "192.168.1.202" port => "6379" password => 'test' db => '1' data_type => "list" key => 'elk-test' }} 運(yùn)行l(wèi)ogstash指定redis-out.conf的配置文件# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/redis-out.conf

運(yùn)行成功以后，在logstash中輸入內(nèi)容（查看下效果）

編輯配置redis-in.conf配置文件，把reids的存儲(chǔ)的數(shù)據(jù)輸出到elasticsearch中# vim /etc/logstash/conf.d/redis-out.conf添加如下內(nèi)容input{ redis { host => "192.168.1.202" port => "6379" password => 'test' db => '1' data_type => "list" key => 'elk-test' batch_count => 1 #這個(gè)值是指從隊(duì)列中讀取數(shù)據(jù)時(shí)，一次性取出多少條，默認(rèn)125條（如果redis中沒有125條，就會(huì)報(bào)錯(cuò)，所以在測(cè)試期間加上這個(gè)值） }}output { elasticsearch { hosts => ['192.168.1.202:9200'] index => 'redis-test-%{+YYYY.MM.dd}' }}運(yùn)行l(wèi)ogstash指定redis-in.conf的配置文件# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/redis-out.conf

把之前的配置文件修改一下，變成所有的日志監(jiān)控的來源文件都存放到redis中，然后通過redis在輸出到elasticsearch中更改為如下，編輯full.confinput { file { path => "/var/log/httpd/access_log" type => "http" start_position => "beginning" } file { path => "/usr/local/nginx/logs/elk.access.log" type => "nginx" start_position => "beginning" } file { path => "/var/log/secure" type => "secure" start_position => "beginning" } file { path => "/var/log/messages" type => "system" start_position => "beginning" }}output { if [type] == "http" { redis { host => "192.168.1.202" password => 'test' port => "6379" db => "6" data_type => "list" key => 'nagios_http' } } if [type] == "nginx" { redis { host => "192.168.1.202" password => 'test' port => "6379" db => "6" data_type => "list" key => 'nagios_nginx' } } if [type] == "secure" { redis { host => "192.168.1.202" password => 'test' port => "6379" db => "6" data_type => "list" key => 'nagios_secure' } } if [type] == "system" { redis { host => "192.168.1.202" password => 'test' port => "6379" db => "6" data_type => "list" key => 'nagios_system' } }} 運(yùn)行l(wèi)ogstash指定shipper.conf的配置文件# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/full.conf在redis中查看是否已經(jīng)將數(shù)據(jù)寫到里面(有時(shí)候輸入的日志文件不產(chǎn)生日志，會(huì)導(dǎo)致redis里面也沒有寫入日志)

把redis中的數(shù)據(jù)讀取出來，寫入到elasticsearch中(需要另外一臺(tái)主機(jī)做實(shí)驗(yàn))編輯配置文件# vim /etc/logstash/conf.d/redis-out.conf添加如下內(nèi)容input { redis { type => "system" host => "192.168.1.202" password => 'test' port => "6379" db => "6" data_type => "list" key => 'nagios_system' batch_count => 1 } redis { type => "http" host => "192.168.1.202" password => 'test' port => "6379" db => "6" data_type => "list" key => 'nagios_http' batch_count => 1 } redis { type => "nginx" host => "192.168.1.202" password => 'test' port => "6379" db => "6" data_type => "list" key => 'nagios_nginx' batch_count => 1 } redis { type => "secure" host => "192.168.1.202" password => 'test' port => "6379" db => "6" data_type => "list" key => 'nagios_secure' batch_count => 1 }} output { if [type] == "system" { elasticsearch { hosts => ["192.168.1.202:9200"] index => "nagios-system-%{+YYYY.MM.dd}" } } if [type] == "http" { elasticsearch { hosts => ["192.168.1.202:9200"] index => "nagios-http-%{+YYYY.MM.dd}" } } if [type] == "nginx" { elasticsearch { hosts => ["192.168.1.202:9200"] index => "nagios-nginx-%{+YYYY.MM.dd}" } } if [type] == "secure" { elasticsearch { hosts => ["192.168.1.202:9200"] index => "nagios-secure-%{+YYYY.MM.dd}" } } }注意:input是從客戶端收集的output是同樣也保存到192.168.1.202中的elasticsearch中，如果要保存到當(dāng)前的主機(jī)上，可以把output中的hosts修改成localhost，如果還需要在kibana中顯示，需要在本機(jī)上部署kabana，為何要這樣做，起到一個(gè)松耦合的目的說白了，就是在客戶端收集日志，寫到服務(wù)端的redis里或是本地的redis里面，輸出的時(shí)候?qū)覧S服務(wù)器即可運(yùn)行命令看看效果# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/redis-out.conf

效果是和直接往ES服務(wù)器輸出一樣的（這樣是先將日志存到redis數(shù)據(jù)庫(kù)，然后再?gòu)膔edis數(shù)據(jù)庫(kù)里取出日志）

上線ELK

１.日志分類 系統(tǒng)日志 rsyslog logstash syslog插件 訪問日志 nginx logstash codec json 錯(cuò)誤日志 file logstash mulitline 運(yùn)行日志 file logstash codec json 設(shè)備日志 syslog logstash syslog插件 Debug日志 file logstash json 或者 mulitline 2. 日志標(biāo)準(zhǔn)化 路徑 固定 格式 盡量json3. 系統(tǒng)個(gè)日志開始-->錯(cuò)誤日志-->運(yùn)行日志-->訪問日志

因?yàn)镋S保存日志是永久保存，所以需要定期刪除一下日志，下面命令為刪除指定時(shí)間前的日志

curl -X DELETE http://xx.xx.com:9200/logstash-*-`date +%Y-%m-%d -d "-$n days"`