Kibana+Logstash+Elasticsearch 日志查詢系統，kibanalogstash

Kibana+Logstash+Elasticsearch 日志查詢系統，kibanalogstash

　　搭建該平台的目的就是為了運維、研發很方便的進行日志的查詢。Kibana一個免費的web殼；Logstash集成各種收集日志插件，還是一個比較優秀的正則切割日志工具；Elasticsearch一個開源的搜索引擎框架（支持群集架構方式）。

 

1 安裝需求

1.1 理論拓撲

 

1.2 安裝環境

1.2.1 硬件環境

192.168.50.62（HP DL 385 G7 、RAM：12G、CPU：AMD 6128、DISK：SAS 146*4）

192.168.50.98（HP DL 385 G7 、RAM：12G、CPU：AMD 6128、DISK：SAS 146*6）

192.168.10.42 (Xen虛擬機、RAM：8G、CPU：×4、DISK：100G)

1.2.2 操作系統

CentOS 5.6 X64

1.2.3 Web-server基礎環境

Nginx+php（安裝過程略過）

1.2.4 軟件列表

JDK 1.6.0_25

logstash-1.1.0-monolithic.jar

elasticsearch-0.18.7.zip

redis-2.4.12.tar.gz

kibana

1.3 獲取方法

1.3.1 Jdk獲取路徑

http://www.oracle.com/technetwork/java/javase/downloads/jdk-6u25-download-346242.html

1.3.2 Logstash獲取路徑

http://semicomplete.com/files/logstash/logstash-1.1.0-monolithic.jar

1.3.3 Elasticsearch獲取路徑

https://github.com/downloads/elasticsearch/elasticsearch/ elasticsearch-0.18.7.zip

1.3.4 Kibana獲取路徑

http://github.com/rashidkpc/Kibana/tarball/master

2 安裝步驟

2.1 JDK的下載及安裝

基礎安裝

wget http://download.oracle.com/otn-pub/java/jdk/6u25-b06/jdk-6u25-linux-x64.bin

sh jdk-6u25-linux-x64.bin

mkdir -p /usr/java

mv ./jdk1.6.0_25 /usr/java

ln –s /usr/java/jdk1.6.0_25 /usr/java/default

編輯 /etc/profile文件，加入以下行內容

export JAVA_HOME=/usr/java/default

export PATH=$JAVA_HOME/bin:$PATH

export CLASSPATH=.:$JAVA_HOME/lib/tools.jar:$JAVA_HOME/lib/dt.jar:$CLASSPATH

刷新環境變量

source /etc/profile

2.2 Redis下載及安裝

wget http://redis.googlecode.com/files/redis-2.4.14.tar.gz

make –j24

make install

mkdir -p /data/redis

cd /data/redis/

mkdir {db,log,etc}

2.3 Elasticsearch下載及安裝

cd /data/

mkdir –p elasticsearch && cd elasticsearch

wget --no-check-certificate https://github.com/downloads/elasticsearch/elasticsearch/ elasticsearch-0.18.7.zip

unzip elasticsearch-0.18.7.zip

2.4 Logstash下載及安裝

mkdir –p /data/logstash/ && cd /data/logstash

wget http://semicomplete.com/files/logstash/logstash-1.1.0-monolithic.jar

2.5 Kibana下載及安裝

wget http://github.com/rashidkpc/Kibana/tarball/master --no-check-certificate

tar zxvf master

3 相關配置及啟動

3.1 Redis配置及啟動

3.1.1 配置文件

vim /data/redis/etc/redis.conf

#----------------------------------------------------

#this is the config file for redis

pidfile /var/run/redis.pid

port 6379

timeout 0

loglevel verbose

logfile /data/redis/log/redis.log

databases 16

save 900 1

save 300 10

save 60 10000

rdbcompression yes

dbfilename dump.rdb

dir /data/redis/db/

slave-serve-stale-data yes

appendonly no

appendfsync everysec

no-appendfsync-on-rewrite no

auto-aof-rewrite-percentage 100

auto-aof-rewrite-min-size 64mb

slowlog-log-slower-than 10000

slowlog-max-len 128

vm-enabled no

vm-swap-file /tmp/redis.swap

vm-max-memory 0

vm-page-size 32

vm-pages 134217728

vm-max-threads 4

hash-max-zipmap-entries 512

hash-max-zipmap-value 64

list-max-ziplist-entries 512

list-max-ziplist-value 64

set-max-intset-entries 512

zset-max-ziplist-entries 128

zset-max-ziplist-value 64

activerehashing yes

3.1.2 Redis啟動

[logstash@Logstash_2 redis]# redis-server /data/redis/etc/redis.conf &

3.2 Elasticsearch 配置及啟動

3.2.1 Elasticsearch啟動

[logstash@Logstash_2 redis]# /data/elasticsearch/elasticsearch-0.18.7/bin/elasticsearch –p ../esearch.pid &

3.2.2 Elasticsearch 群集配置

curl 127.0.0.1:9200/_cluster/nodes/192.168.50.62

3.3 Logstash配置及啟動

3.3.1 Logstash配置文件

input {

redis {

host => "192.168.50.98"

data_type =>"list"

key => "logstash:redis"

type => "redis-input"

}

}

filter {

grok {

type => "linux-syslog"

pattern => "%{SYSLOGLINE}"

}

grok {

type => "nginx-access"

pattern => "%{NGINXACCESSLOG}"

}

}

output {

elasticsearch {

host =>"192.168.50.62"

}

}

3.3.2 Logstash啟動為Index

java -jar logstash.jar agent -f my.conf &

3.3.3 Logstash啟動為agent

配置文件

input {

file{

type => "linux-syslog"

path => [ "/var/log/*.log", "/var/log/messages", "/var/log/syslog" ]

}

file {

type => "nginx-access"

path => "/usr/local/nginx/logs/access.log"

}

file {

type => "nginx-error"

path => "/usr/local/nginx/logs/error.log"

}

}

output {

redis {

host => "192.168.50.98"

data_type =>"list"

key => "logstash:redis"

}

}

Agent 啟動

java -jar logstash-1.1.0-monolithic.jar agent -f shipper.conf &

3.3.4 kibana配置

首先在nginx添加站點配置

server {

listen 80;

server_name logstash.test.com;

index index.php;

root /usr/local/nginx/html;

#charset koi8-r;

#access_log logs/host.access.log main;

location ~ .*\.(php|php5)$

{

#fastcgi_pass unix:/tmp/php-cgi.sock;

fastcgi_pass 127.0.0.1:9000;

fastcgi_index index.php;

include fastcgi.conf;

}

}

4 性能調優

4.1 Elasticsearch調優

4.1.1 JVM調優

編輯Elasticsearch.in.sh文件

ES_CLASSPATH=$ES_CLASSPATH:$ES_HOME/lib/*:$ES_HOME/lib/sigar/*

if [ "x$ES_MIN_MEM" = "x" ]; then

ES_MIN_MEM=4g

fi

if [ "x$ES_MAX_MEM" = "x" ]; then

ES_MAX_MEM=4g

fi

4.1.2 Elasticsearch索引壓縮

vim index_elastic.sh

#!/bin/bash

#comperssion the data for elasticsearch now

date=` date +%Y.%m.%d `

# compression the new index;

/usr/bin/curl -XPUT http://localhost:9200/logstash-$date/nginx-access/_mapping -d '{"nginx-access" : {"_source" : { "compress" : true }}}'

echo ""

/usr/bin/curl -XPUT http://localhost:9200/logstash-$date/nginx-error/_mapping -d '{"nginx-error" : {"_source" : { "compress" : true }}}'

echo ""

/usr/bin/curl -XPUT http://localhost:9200/logstash-$date/linux-syslog/_mapping -d '{"linux-syslog" : {"_source" : { "compress" : true }}}'

echo ""

保存該腳本並執行

sh index_elastic.sh

5 使用

5.1 Logstash查詢頁

使用火狐浏覽器或者谷歌浏覽器訪問 http://logstash.test.com