分布式日志框架ELK搭建(下)

/ 冷姿势 / 0 条评论 / 372人围观

安装kibana

rpm安装

在57,master-node主节点安装kibana

[root@master-node ~]# wget httpss://artifacts.elastic.co/downloads/kibana/kibana-6.0.0-x86_64.rpm
[root@master-node ~]# rpm -ivh kibana-6.0.0-x86_64.rpm

配置kibana

[root@master-node ~]# vim /etc/kibana/kibana.yml  # 增加以下内容
server.port: 5601  # 配置kibana的端口
server.host: 10.167.201.57  # 配置监听ip
elasticsearch.url: "https://10.167.201.57:9200"  # 配置es服务器的ip,如果是集群则配置该集群中主节点的ip
logging.dest: /var/log/kibana.log  # 配置kibana的日志文件路径,不然默认是messages里记录日志

创建日志文件

创建日志文件并授权

[root@master-node ~]# touch /var/log/kibana.log; chmod 777 /var/log/kibana.log

启动服务检查监听端口

[root@master-node ~]# systemctl start kibana
[root@master-node ~]# ps aux |grep kibana
kibana     3083 36.8  2.9 1118668 112352 ?      Ssl  17:14   0:03 /usr/share/kibana/bin/../node/bin/node --no-warnings /usr/share/kibana/bin/../src/cli -c /etc/kibana/kibana.yml
root       3095  0.0  0.0 112660   964 pts/0    S+   17:14   0:00 grep --color=auto kibana
[root@master-node ~]# netstat -lntp |grep 5601
tcp        0      0 10.167.201.57:5601     0.0.0.0:*               LISTEN      3083/node    
[root@master-node ~]# 

由于kibana是使用node.js开发的,所以进程名称为node

然后在浏览器里进行访问,如:https://10.167.201.57:5601/ ,由于没有安装x-pack,所以此时是没有用户名和密码的,可以直接访问的:

请输入图片描述

安装logstash

在58,data-node1数据节点上安装logstash,但是要注意的是目前6.0 logstash不支持JDK1.9。

rpm方式安装

[root@data-node1 ~]# wget httpss://artifacts.elastic.co/downloads/logstash/logstash-6.0.0.rpm
[root@data-node1 ~]# rpm -ivh logstash-6.0.0.rpm

配置logstash收集syslog日志

[root@data-node1 ~]# vim /etc/logstash/conf.d/syslog.conf  # 加入如下内容
input {  # 定义日志源
  syslog {
    type => "system-syslog"  # 定义类型
    port => 10514    # 定义监听端口
  }
}
output {  # 定义日志输出
  stdout {
    codec => rubydebug  # 将日志输出到当前的终端上显示
  }
}

检测配置文件

[root@data-node1 ~]# cd /usr/share/logstash/bin
[root@data-node1 /usr/share/logstash/bin]# ./logstash --path.settings /etc/logstash/ -f /etc/logstash/conf.d/syslog.conf --config.test_and_exit
Sending Logstash's logs to /var/log/logstash which is now configured via log4j2.properties
Configuration OK  # 为ok则代表配置文件没有问题
[root@data-node1 /usr/share/logstash/bin]# 

命令说明:

配置kibana服务器的ip以及配置的监听端口:

[root@data-node1 ~]# vim /etc/rsyslog.conf
#### RULES ####
*.* @10.167.201.58:10514

重启rsyslog,让配置生效:

[root@data-node1 ~]# systemctl restart rsyslog

指定配置文件,启动logstash:

[root@data-node1 ~]# cd /usr/share/logstash/bin
[root@data-node1 /usr/share/logstash/bin]# ./logstash --path.settings /etc/logstash/ -f /etc/logstash/conf.d/syslog.conf
Sending Logstash's logs to /var/log/logstash which is now configured via log4j2.properties
# 这时终端会停留在这里,因为我们在配置文件中定义的是将信息输出到当前终端

打开新终端检查一下10514端口是否已被监听:

[root@data-node1 ~]# netstat -lntp |grep 10514
tcp6       0      0 :::10514                :::*                    LISTEN      4312/java 
[root@data-node1 ~]# 

然后在别的机器ssh登录到这台机器上,测试一下有没有日志输出:

[root@data-node1 /usr/share/logstash/bin]# ./logstash --path.settings /etc/logstash/ -f /etc/logstash/conf.d/syslog.conf
Sending Logstash's logs to /var/log/logstash which is now configured via log4j2.properties
{
          "severity" => 6,
               "pid" => "4575",
           "program" => "sshd",
           "message" => "Accepted password for root from 10.167.201.57 port 58336 ssh2\n",
              "type" => "system-syslog",
          "priority" => 86,
         "logsource" => "data-node1",
        "@timestamp" => 2018-09-03T18:12:27.000Z,
          "@version" => "1",
              "host" => "10.167.201.58",
          "facility" => 10,
    "severity_label" => "Informational",
         "timestamp" => "Sep  4 02:12:27",
    "facility_label" => "security/authorization"
}
{
          "severity" => 6,
           "program" => "systemd",
           "message" => "Started Session 42 of user root.\n",
              "type" => "system-syslog",
          "priority" => 30,
         "logsource" => "data-node1",
        "@timestamp" => 2018-09-03T18:12:27.000Z,
          "@version" => "1",
              "host" => "10.167.201.58",
          "facility" => 3,
    "severity_label" => "Informational",
         "timestamp" => "Sep  4 02:12:27",
    "facility_label" => "system"
}
、、、、、、、

如上,可以看到,终端中以JSON的格式打印了收集到的日志,测试成功。

配置logstash

重新改一下配置文件,让收集的日志信息输出到es服务器中,而不是当前终端:

[root@data-node1 ~]# vim /etc/logstash/conf.d/syslog.conf # 更改为如下内容
input {
  syslog {
    type => "system-syslog"
    port => 10514
  }
}
output {
  elasticsearch {
    hosts => ["10.167.201.57:9200"]  # 定义es服务器的ip
    index => "system-syslog-%{+YYYY.MM}" # 定义索引
  }
}

检测配置文件有没有错:

[root@data-node1 ~]# cd /usr/share/logstash/bin
[root@data-node1 /usr/share/logstash/bin]# ./logstash --path.settings /etc/logstash/ -f /etc/logstash/conf.d/syslog.conf --config.test_and_exit
Sending Logstash's logs to /var/log/logstash which is now configured via log4j2.properties
Configuration OK
[root@data-node1 /usr/share/logstash/bin]# 

没问题后,启动logstash服务,并检查进程以及监听端口:

[root@data-node1 ~]# systemctl start logstash
[root@data-node1 ~]# ps aux |grep logstash
logstash   5364  285 20.1 3757012 376260 ?      SNsl 04:36   0:34 /bin/java -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+DisableExplicitGC -Djava.awt.headless=true -Dfile.encoding=UTF-8 -XX:+HeapDumpOnOutOfMemoryError -Xmx1g -Xms256m -Xss2048k -Djffi.boot.library.path=/usr/share/logstash/vendor/jruby/lib/jni -Xbootclasspath/a:/usr/share/logstash/vendor/jruby/lib/jruby.jar -classpath : -Djruby.home=/usr/share/logstash/vendor/jruby -Djruby.lib=/usr/share/logstash/vendor/jruby/lib -Djruby.script=jruby -Djruby.shell=/bin/sh org.jruby.Main /usr/share/logstash/lib/bootstrap/environment.rb logstas/runner.rb --path.settings /etc/logstash
root       5400  0.0  0.0 112652   964 pts/0    S+   04:36   0:00 grep --color=auto logstash

错误解决:

启动logstash后,进程是正常存在的,但是9600以及10514端口却没有被监听。于是查看logstash的日志看看有没有错误信息的输出,但是发现没有记录日志信息,那就只能转而去查看messages的日志,发现错误信息如下:

请输入图片描述 这是因为权限不够,既然是权限不够,那就设置权限即可:

[root@data-node1 ~]# chown logstash /var/log/logstash/logstash-plain.log 
[root@data-node1 ~]# ll !$
ll /var/log/logstash/logstash-plain.log
-rw-r--r-- 1 logstash root 7597 Mar  4 04:35 /var/log/logstash/logstash-plain.log
[root@data-node1 ~]# systemctl restart logstash

设置完权限重启服务之后,发现还是没有监听端口,查看logstash-plain.log文件记录的错误日志信息如下: 请输入图片描述 可以看到,依旧是权限的问题,这是因为之前我们以root的身份在终端启动过logstash,所以产生的相关文件的属组属主都是root,同样的,也是设置一下权限即可:

[root@data-node1 ~]# ll /var/lib/logstash/
total 4
drwxr-xr-x 2 root root  6 Mar  4 01:50 dead_letter_queue
drwxr-xr-x 2 root root  6 Mar  4 01:50 queue
-rw-r--r-- 1 root root 36 Mar  4 01:58 uuid
[root@data-node1 ~]# chown -R logstash /var/lib/logstash/
[root@data-node1 ~]# systemctl restart logstash

端口正常监听了,服务就启动成功了:

[root@data-node1 ~]# netstat -lntp |grep 9600
tcp6       0      0 127.0.0.1:9600          :::*                    LISTEN      9905/java
[root@data-node1 ~]# netstat -lntp |grep 10514
tcp6       0      0 :::10514                :::*                    LISTEN      9905/java
[root@data-node1 ~]# 

但是可以看到,logstash的监听ip是127.0.0.1这个本地ip,本地ip无法远程通信,所以需要修改一下配置文件,配置一下监听的ip:

[root@data-node1 ~]# vim /etc/logstash/logstash.yml
https.host: "10.167.201.58"
[root@data-node1 ~]# systemctl restart logstash
[root@data-node1 ~]# netstat -lntp |grep 9600
tcp6       0      0 10.167.201.58:9600     :::*                    LISTEN      10091/java          
[root@data-node1 ~]# 

kibana上查看日志

完成了logstash服务器的搭建之后,回到kibana服务器上查看日志,执行以下命令可以获取索引信息:

[root@master-node ~]# curl '10.167.201.57:9200/_cat/indices?v'
health status index                 uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   .kibana               6JfXc0gFSPOWq9gJI1ZX2g   1   1          1            0      6.9kb          3.4kb
green  open   system-syslog-2018.09 bUXmEDskTh6fjGD3JgyHcA   5   1         61            0    591.7kb        296.7kb
[root@master-node ~]# 

如上,可以看到,在logstash配置文件中定义的system-syslog索引成功获取到了,证明配置没问题,logstash与es通信正常。

[root@master-node ~]# curl -XGET '10.167.201.57:9200/system-syslog-2018.09?pretty'
{
  "system-syslog-2018.09" : {
    "aliases" : { },
    "mappings" : {
      "system-syslog" : {
        "properties" : {
          "@timestamp" : {
            "type" : "date"
          },
          "@version" : {
            "type" : "text",
            "fields" : {
              "keyword" : {
                "type" : "keyword",
                "ignore_above" : 256
              }
            }
          },
          "facility" : {
            "type" : "long"
          },
          "facility_label" : {
            "type" : "text",
            "fields" : {
              "keyword" : {
                "type" : "keyword",
                "ignore_above" : 256
              }
            }
          },
          "host" : {
            "type" : "text",
            "fields" : {
              "keyword" : {
                "type" : "keyword",
                "ignore_above" : 256
              }
            }
          },
          "logsource" : {
            "type" : "text",
            "fields" : {
              "keyword" : {
                "type" : "keyword",
                "ignore_above" : 256
              }
            }
          },
          "message" : {
            "type" : "text",
            "fields" : {
              "keyword" : {
                "type" : "keyword",
                "ignore_above" : 256
              }
            }
          },
          "pid" : {
            "type" : "text",
            "fields" : {
              "keyword" : {
                "type" : "keyword",
                "ignore_above" : 256
              }
            }
          },
          "priority" : {
            "type" : "long"
          },
          "program" : {
            "type" : "text",
            "fields" : {
              "keyword" : {
                "type" : "keyword",
                "ignore_above" : 256
              }
            }
          },
          "severity" : {
            "type" : "long"
          },
          "severity_label" : {
            "type" : "text",
            "fields" : {
              "keyword" : {
                "type" : "keyword",
                "ignore_above" : 256
              }
            }
          },
          "timestamp" : {
            "type" : "text",
            "fields" : {
              "keyword" : {
                "type" : "keyword",
                "ignore_above" : 256
              }
            }
          },
          "type" : {
            "type" : "text",
            "fields" : {
              "keyword" : {
                "type" : "keyword",
                "ignore_above" : 256
              }
            }
          }
        }
      }
    },
    "settings" : {
      "index" : {
        "creation_date" : "1520082481446",
        "number_of_shards" : "5",
        "number_of_replicas" : "1",
        "uuid" : "bUXmEDskTh6fjGD3JgyHcA",
        "version" : {
          "created" : "6020299"
        },
        "provided_name" : "system-syslog-2018.09"
      }
    }
  }
}
[root@master-node ~]#

如果日后需要删除索引的话,使用以下命令可以删除指定索引:

curl -XDELETE 'localhost:9200/system-syslog-2018.09'

es与logstash能够正常通信后就可以去配置kibana了,浏览器访问10.167.201.57:5601,到kibana页面上配置索引:

请输入图片描述

配置成功后点击 “Discover” :

请输入图片描述

如果出现找不到日志的情况一般是时间的问题,点击右上角切换成查看当天的日志信息即可!

以上这就是如何使用logstash收集系统日志,输出到es服务器上,并在kibana的页面上进行查看。

logstash收集nginx日志

和收集syslog一样,首先需要编辑配置文件,这一步在logstash服务器上完成:

[root@data-node1 ~]# vim /etc/logstash/conf.d/nginx.conf  # 增加如下内容
input {
  file {  # 指定一个文件作为输入源
    path => "/tmp/elk_access.log"  # 指定文件的路径
    start_position => "beginning"  # 指定何时开始收集
    type => "nginx"  # 定义日志类型,可自定义
  }
}
filter {  # 配置过滤器
    grok {
        match => { "message" => "%{IPORHOST:https_host} %{IPORHOST:clientip} - %{USERNAME:remote_user} \[%{HTTPDATE:timestamp}\] \"(?:%{WORD:https_verb} %{NOTSPACE:https_request}(?: HTTP/%{NUMBER:https_version})?|%{DATA:raw_https_request})\" %{NUMBER:response} (?:%{NUMBER:bytes_read}|-) %{QS:referrer} %{QS:agent} %{QS:xforwardedfor} %{NUMBER:request_time:float}"}  # 定义日志的输出格式
    }
    geoip {
        source => "clientip"
    }
}
output {
    stdout { codec => rubydebug }
    elasticsearch {
        hosts => ["10.167.201.57:9200"]
        index => "nginx-test-%{+YYYY.MM.dd}"
  }
}

同样的编辑完配置文件之后,还需要检测配置文件是否有错:

[root@data-node1 ~]# cd /usr/share/logstash/bin
[root@data-node1 /usr/share/logstash/bin]# ./logstash --path.settings /etc/logstash/ -f /etc/logstash/conf.d/nginx.conf --config.test_and_exit
Sending Logstash's logs to /var/log/logstash which is now configured via log4j2.properties
Configuration OK
[root@data-node1 /usr/share/logstash/bin]# 

检查完毕之后,进入你的nginx虚拟主机配置文件所在的目录中,新建一个虚拟主机配置文件:

[root@data-node1 ~]# cd /usr/local/nginx/conf/vhost/
[root@data-node1 /usr/local/nginx/conf/vhost]# vim elk.conf
server {
      listen 80;
      server_name elk.test.com;

      location / {
          proxy_pass      https://10.167.201.57:5601;
          proxy_set_header Host   $host;
          proxy_set_header X-Real-IP      $remote_addr;
          proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
      }
      access_log  /tmp/elk_access.log main2;
}

配置nginx的主配置文件,因为需要配置日志格式,在 log_format 那一行的下面增加以下内容:

[root@data-node1 ~]# vim /usr/local/nginx/conf/nginx.conf
log_format main2 '$https_host $remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$https_referer" '
                      '"$https_user_agent" "$upstream_addr" $request_time';

完成以上配置文件的编辑之后,检测配置文件有没有错误,没有的话就reload重新加载:

[root@data-node1 ~]# /usr/local/nginx/sbin/nginx -s reload

重启logstash服务,生成日志的索引:

systemctl restart logstash

重启完成后,在es服务器上检查是否有nginx-test开头的索引生成:

[root@master-node ~]# curl '10.167.201.57:9200/_cat/indices?v' 
health status index                 uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   .kibana               6JfXc0gFSPOWq9gJI1ZX2g   1   1          2            0     14.4kb          7.2kb
green  open   system-syslog-2018.09 bUXmEDskTh6fjGD3JgyHcA   5   1        902            0      1.1mb        608.9kb
green  open   nginx-test-2018.09.04 GdKYa6gBRke7mNgrh2PBUA   5   1         45            0      199kb         99.5kb
[root@master-node ~]# 

配置成功后,访问https://10.167.201.57:5601,重复上面的配置索引步骤即可

最终的日志收集页面是这样,ok,基本完成了: 请输入图片描述

使用beats采集日志

beats是ELK体系中新增的一个工具,它属于一个轻量的日志采集器,以上我们使用的日志采集工具是logstash,但是logstash占用的资源比较大,没有beats轻量,所以官方也推荐使用beats来作为日志采集工具。而且beats可扩展,支持自定义构建。

在 10.167.201.59 上安装filebeat,filebeat是beats体系中用于收集日志信息的工具:

[root@data-node2 ~]# wget httpss://artifacts.elastic.co/downloads/beats/filebeat/filebeat-6.0.0-x86_64.rpm
[root@data-node2 ~]# rpm -ivh  filebeat-6.0.0-x86_64.rpm

修改配置文件,将filebeat作为一个服务启动:

[root@data-node2 ~]# vim /etc/filebeat/filebeat.yml
filebeat.prospectors:
- type: log
   #enabled: false 这一句要注释掉,这句一定得注释掉
   paths:
      - /var/log/messages  # 指定需要收集的日志文件的路径

output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["10.167.201.57:9200"]  # 并配置es服务器的ip地址

修改完成后就可以启动filebeat服务了:

[root@data-node2 ~]# systemctl start filebeat
[root@data-node2 ~]# ps axu |grep filebeat
root       3021  0.3  2.3 296360 11288 ?        Ssl  22:27   0:00 /usr/share/filebeat/bin/filebeat -c /etc/filebeat/filebeat.yml -path.home /usr/share/filebeat -path.config /etc/filebeat -path.data /var/lib/filebeat -path.logs /var/log/filebeat
root       3030  0.0  0.1 112660   960 pts/0    S+   22:27   0:00 grep --color=auto filebeat

启动成功后,到es服务器上查看索引,可以看到新增了一个以filebeat-6.0.0开头的索引,这就代表filesbeat和es能够正常通信了:

[root@master-node ~]# curl '10.167.201.57:9200/_cat/indices?v' 
health status index                     uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   system-syslog-2018.03     bUXmEDskTh6fjGD3JgyHcA   5   1      73076            0     24.8mb         11.6mb
green  open   nginx-test-2018.03.04     GdKYa6gBRke7mNgrh2PBUA   5   1         91            0        1mb        544.8kb
green  open   .kibana                   6JfXc0gFSPOWq9gJI1ZX2g   1   1          3            0     26.9kb         13.4kb
green  open   filebeat-6.0.0-2018.03.04 MqQJMUNHS_OiVmO26NEWTw   3   1         66            0     64.5kb         39.1kb
[root@master-node ~]# 

es服务器能够正常获取到索引后,就可以到kibana上配置这个索引了,以后的基本步骤参考上面的内容来配置index索引,推荐使用beats来作为日志采集工具。


ok,到这里ELK的简单搭建终于完成了(还有好多好多细节没搞)。