日志收集系统PLG（Promtail+Loki+Grafana）介绍及部署
一、简 介
Loki是受Prometheus启发由Grafana Labs团队开源的水平可扩展，高度可用的多租户日志聚合系统。 开发语言: Google Go。它的设计具有很高的成本效益，并且易于操作。使用标签来作为索引，而不是对全文进行检索，也就是说，你通过这些标签既可以查询日志的内容也可以查询到监控的数据签，极大地降低了日志索引的存储。系统架构十分简单，由以下3个部分组成 ：
Promtail 是代理，负责收集日志并将其发送给 loki 。
Loki 是主服务器，负责存储日志和处理查询 。
Grafana 用于 UI 展示。
只要在应用程序服务器上安装promtail来收集日志然后发送给Loki存储，就可以在Grafana UI界面通过添加Loki为数据源进行日志查询（如果Loki服务器性能不够，可以部署多个Loki进行存储及查询）。作为一个日志系统不光只有查询分析日志的能力，还能对日志进行监控和报警。
二、原理分析
promtail收集并将日志发送给loki的 Distributor 组件 Distributor会对接收到的日志流进行正确性校验，并将验证后的日志分批并行发送到Ingester Ingester 接受日志流并构建数据块，压缩后存放到所连接的存储后端 Querier 收到HTTP查询请求，并将请求发送至Ingester 用以获取内存数据 ，Ingester 收到请求后返回符合条件的数据 ； 如果 Ingester 没有返回数据，Querier 会从后端存储加载数据并遍历去重执行查询 ，通过HTTP返回查询结果
三、对比
PLG与ELK比较优势ELK虽然功能丰富，但规模复杂，资源占用高，操作苦难，很多功能往往用不上，有点杀鸡用牛刀的感觉。 Loki 不对日志进行全文索引。通过存储压缩非结构化日志和索引元数据，Loki 操作起来会更简单，更省成本。 通过使用与 Prometheus 相同的标签记录流对日志进行索引和分组，这使得日志的扩展和操作效率更高。 安装部署简单快速，且受 Grafana 原生支持。
四、下载
curl -O -L "https://github.com/grafana/loki/releases/download/v1.5.0/loki-linux-amd64.zip" curl -O -L "https://github.com/grafana/loki/releases/download/v1.5.0/promtail-linux-amd64.zip" wget https://dl.grafana.com/enterprise/release/grafana-enterprise-8.3.3.linux-amd64.tar.gz
五、 安装loki、grafana、promtail
安装loki
先将loki-linux-amd64.zip和grafana-enterprise-8.3.3.linux-amd64.tar.gz上传到日志服务器（10.60.134.55）；再将promtail-linux-amd64.zip上传到应用服务器（10.60.134.60、10.60.134.48、10.60.134.56）。
在日志服务器 上创建目录 /usr/local/logsCollect/loki用于安装loki，创建目录/usr/local/logsCollect/grafana用于安装grafana，如下：
mkdir /data/loki mkdir /data/loki/{chunks,index}
在日志服务器 上创建目录，用于文件存储和索引存储。
解压unzip loki-linux-amd64.zip，并配置config.yaml
auth_enabled: false server: http_listen_port: 8094 ingester: lifecycler: address: 10.60.134.55 #日志服务器ip，即本机地址 ring: kvstore: store: inmemory replication_factor: 1 final_sleep: 0s chunk_idle_period: 5m chunk_retain_period: 30s schema_config: configs: - from: 2024-04-01 store: boltdb object_store: filesystem schema: v11 index: prefix: index_ period: 168h #每张表的时间范围7天 storage_config: boltdb: directory: /data/loki/index #索引文件存储地址 filesystem: directory: /data/loki/chunks #块存储地址 limits_config: enforce_metric_name: false reject_old_samples: true reject_old_samples_max_age: 168h chunk_store_config: # 最大可查询历史日期 7天,这个时间必须是schema_config中的period的倍数，否则报错。 max_look_back_period: 168h # 表的保留期7天 table_manager: retention_deletes_enabled: true retention_period: 168h
并创建启动脚本start.sh和关闭脚本shutdown.sh
#!/bin/bash nohup ./loki-linux-amd64 -config.file=./config.yaml >./server.log 2>&1 & echo "$!" > pid
#!/bin/bash kill -9 `cat pid` echo "关闭成功!"
启动loki命令：sh start.sh。如下日志表示启动成功。
安装 grafana
首先解压grafana-enterprise-8.3.3.linux-amd64.tar.gz，创建启动脚本start.sh和关闭脚本shutdown.sh。注意conf/defaults.ini为grafana的配置文件，其中http_port为浏览器访问端口。
启动grafana命令：sh start.sh。如下日志表示启动成功。
在应用服务器安装 promtail
在应用服务器 上创建目录 /usr/local/logsCollect/promtail用于安装promtail。
解压promtail-linux-amd64.zip，并配置promtail.yaml
server: http_listen_port: 9080 grpc_listen_port: 0 positions: filename: ./positions.yaml clients: - url: http://10.60.134.55:8094/loki/api/v1/push #日志服务器loki地址和端口 scrape_configs: #ucenter1 - job_name: zcbackend-172.29.21.22-1 static_configs: - targets: - 10.60.134.60 - labels: job: zcbachend-172.29.21.22-1 host: 10.60.134.60 __path__: /jiuqi/zichan/zichanyitihuazhenghexiangmu-8084/backend/server.log #本机日志路径
并创建启动脚本start.sh和关闭脚本shutdown.sh
启动promtail命令：sh start.sh。如下日志表示启动成功。
六、 测试
打开grafana。默认密码admin/admin。
添加数据源
测试连接
查询日志
也能通过导入的方式添加面板，json文件如下：
{
"annotations": {
"list": [
{
"builtIn": 1,
"datasource": {
"type": "datasource",
"uid": "grafana"
},
"enable": true,
"hide": true,
"iconColor": "rgba(0, 211, 255, 1)",
"name": "Annotations & Alerts",
"target": {
"limit": 100,
"matchAny": false,
"tags": [],
"type": "dashboard"
},
"type": "dashboard"
}
]
},
"editable": true,
"fiscalYearStartMonth": 0,
"graphTooltip": 0,
"id": 3,
"iteration": 1671275851848,
"links": [],
"liveNow": false,
"panels": [
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": {
"type": "loki",
"uid": "${ENV}"
},
"fill": 1,
"fillGradient": 0,
"gridPos": {
"h": 5,
"w": 24,
"x": 0,
"y": 0
},
"hiddenSeries": false,
"id": 4,
"legend": {
"alignAsTable": true,
"avg": false,
"current": false,
"max": false,
"min": false,
"rightSide": true,
"show": true,
"total": false,
"values": false
},
"lines": true,
"linewidth": 1,
"nullPointMode": "null",
"options": {
"alertThreshold": true
},
"percentage": false,
"pluginVersion": "8.5.10",
"pointradius": 2,
"points": false,
"renderer": "flot",
"seriesOverrides": [],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"datasource": {
"type": "loki",
"uid": "hJ7iGN-Sz"
},
"expr": "sum (count_over_time({filename=~\"$log\",host=~\"$host\"}[2m] )) by (host,filename)",
"hide": false,
"legendFormat": "{{host}}=={{filename}}",
"refId": "B"
}
],
"thresholds": [],
"timeRegions": [],
"title": "日志量统计",
"tooltip": {
"shared": true,
"sort": 0,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"mode": "time",
"show": true,
"values": []
},
"yaxes": [
{
"$$hashKey": "object:319",
"format": "short",
"logBase": 1,
"show": true
},
{
"$$hashKey": "object:320",
"format": "short",
"logBase": 1,
"show": true
}
],
"yaxis": {
"align": false
}
},
{
"datasource": {
"type": "loki",
"uid": "hJ7iGN-Sz"
},
"description": "",
"gridPos": {
"h": 21,
"w": 24,
"x": 0,
"y": 5
},
"id": 2,
"options": {
"dedupStrategy": "exact",
"enableLogDetails": false,
"prettifyLogMessage": false,
"showCommonLabels": false,
"showLabels": false,
"showTime": true,
"sortOrder": "Descending",
"wrapLogMessage": true
},
"pluginVersion": "7.4.3",
"targets": [
{
"datasource": {
"type": "loki",
"uid": "hJ7iGN-Sz"
},
"expr": "{filename=~\"$log\",host=~\"$host\"}|~ \"(?i)$log_level\"",
"hide": false,
"maxLines": 500,
"refId": "C"
}
],
"title": "日志",
"transparent": true,
"type": "logs"
}
],
"refresh": "",
"schemaVersion": 36,
"style": "dark",
"tags": [
"PLG"
],
"templating": {
"list": [
{
"current": {
"selected": false,
"text": "Loki",
"value": "Loki"
},
"hide": 0,
"includeAll": false,
"label": "选择环境",
"multi": false,
"name": "ENV",
"options": [],
"query": "loki",
"queryValue": "",
"refresh": 1,
"regex": "",
"skipUrlSync": false,
"type": "datasource"
},
{
"current": {
"selected": true,
"text": "资产server",
"value": "资产server"
},
"datasource": {
"type": "loki",
"uid": "hJ7iGN-Sz"
},
"definition": "label_values({},filename)",
"hide": 0,
"includeAll": false,
"label": "日志",
"multi": false,
"name": "log",
"options": [
{
"selected": true,
"text": "资产server-8084",
"value": "/jiuqi/zichan/zichanyitihuazhenghexiangmu-8084/backend/server.log"
},
{
"selected": true,
"text": "资产server-8085",
"value": "/jiuqi/zichan/zichanyitihuazhenghexiangmu-8085/backend/server.log"
},
{
"selected": true,
"text": "资产server-8086",
"value": "/jiuqi/zichan/zichanyitihuazhenghexiangmu-8086/backend/server.log"
},
{
"selected": true,
"text": "公车server-8087",
"value": "/jiuqi/gongche/yitihuagongchehouduan-8087/backend/server.log"
},
{
"selected": true,
"text": "公车server-8088",
"value": "/jiuqi/gongche/yitihuagongchehouduan-8088/backend/server.log"
}
],
"query": "label_values({},filename)",
"refresh": 1,
"regex": "",
"skipUrlSync": false,
"sort": 1,
"type": "query"
},
{
"current": {
"selected": true,
"text": "172.29.21.22",
"value": "10.60.134.60"
},
"datasource": {
"type": "loki",
"uid": "hJ7iGN-Sz"
},
"definition": "label_values({},host)",
"hide": 0,
"includeAll": false,
"label": "主机IP",
"multi": false,
"name": "host",
"options": [
{
"selected": true,
"text": "资产-172.29.21.22",
"value": "10.60.134.60"
},
{
"selected": true,
"text": "资产-172.29.21.164",
"value": "10.60.134.48"
},
{
"selected": true,
"text": "公车-172.29.21.36",
"value": "10.60.134.56"
}
],
"query": "label_values({},host)",
"refresh": 1,
"regex": "",
"skipUrlSync": false,
"sort": 1,
"type": "query"
},
{
"allValue": "(^\\\\S|^\\\\s)",
"current": {
"selected": false,
"text": "All",
"value": "$__all"
},
"description": "可以直接输入搜索的关键字进行过滤",
"hide": 0,
"includeAll": true,
"label": "关键字过滤",
"multi": false,
"name": "log_level",
"options": [
{
"selected": true,
"text": "All",
"value": "$__all"
},
{
"selected": false,
"text": "INFO",
"value": "INFO"
},
{
"selected": false,
"text": "DEBUG",
"value": "DEBUG"
},
{
"selected": false,
"text": "WARN",
"value": "WARN"
},
{
"selected": false,
"text": "ERROR",
"value": "ERROR"
},
{
"selected": false,
"text": "UNKNOWN",
"value": "UNKNOWN"
},
{
"selected": false,
"text": "直接输入关键字搜索",
"value": "直接输入关键字搜索"
}
],
"query": "INFO,DEBUG,WARN,ERROR,UNKNOWN,直接输入关键字搜索",
"queryValue": "",
"skipUrlSync": false,
"type": "custom"
}
]
},
"time": {
"from": "now-1h",
"to": "now"
},
"timepicker": {},
"timezone": "",
"title": "Loki日志收集",
"uid": "NlV_8QD7k",
"version": 19,
"weekStart": ""
}
yian