瓜瓜头条

日志收集系统PLG（Promtail+Loki+Grafana）部署

2024-04-11 09:53 由 怡安 发表于 #企业信息化

一、简 介

Loki是受Prometheus启发由Grafana Labs团队开源的水平可扩展，高度可用的多租户日志聚合系统。 开发语言: Google Go。它的设计具有很高的成本效益，并且易于操作。使用标签来作为索引，而不是对全文进行检索，也就是说，你通过这些标签既可以查询日志的内容也可以查询到监控的数据签，极大地降低了日志索引的存储。系统架构十分简单，由以下3个部分组成 ：

Promtail 是代理，负责收集日志并将其发送给 loki 。

Loki 是主服务器，负责存储日志和处理查询 。

Grafana 用于 UI 展示。

只要在应用程序服务器上安装promtail来收集日志然后发送给Loki存储，就可以在Grafana UI界面通过添加Loki为数据源进行日志查询（如果Loki服务器性能不够，可以部署多个Loki进行存储及查询）。作为一个日志系统不光只有查询分析日志的能力，还能对日志进行监控和报警。

二、原理分析

promtail收集并将日志发送给loki的 Distributor 组件 Distributor会对接收到的日志流进行正确性校验，并将验证后的日志分批并行发送到Ingester Ingester 接受日志流并构建数据块，压缩后存放到所连接的存储后端 Querier 收到HTTP查询请求，并将请求发送至Ingester 用以获取内存数据 ，Ingester 收到请求后返回符合条件的数据 ； 如果 Ingester 没有返回数据，Querier 会从后端存储加载数据并遍历去重执行查询 ，通过HTTP返回查询结果

三、对比

PLG与ELK比较优势ELK虽然功能丰富，但规模复杂，资源占用高，操作苦难，很多功能往往用不上，有点杀鸡用牛刀的感觉。 Loki 不对日志进行全文索引。通过存储压缩非结构化日志和索引元数据，Loki 操作起来会更简单，更省成本。 通过使用与 Prometheus 相同的标签记录流对日志进行索引和分组，这使得日志的扩展和操作效率更高。 安装部署简单快速，且受 Grafana 原生支持。

四、下载

curl -O -L "https://github.com/grafana/loki/releases/download/v1.5.0/loki-linux-amd64.zip" 
curl -O -L "https://github.com/grafana/loki/releases/download/v1.5.0/promtail-linux-amd64.zip"
wget https://dl.grafana.com/enterprise/release/grafana-enterprise-8.3.3.linux-amd64.tar.gz

五、 安装loki、grafana、promtail

安装loki

先将loki-linux-amd64.zip和grafana-enterprise-8.3.3.linux-amd64.tar.gz上传到日志服务器（10.60.134.55）；再将promtail-linux-amd64.zip上传到应用服务器（10.60.134.60、10.60.134.48、10.60.134.56）。

在日志服务器 上创建目录 /usr/local/logsCollect/loki用于安装loki，创建目录/usr/local/logsCollect/grafana用于安装grafana，如下：

mkdir  /data/loki
mkdir  /data/loki/{chunks,index}

在日志服务器 上创建目录，用于文件存储和索引存储。

解压unzip loki-linux-amd64.zip，并配置config.yaml

auth_enabled: false
server:
  http_listen_port: 8094
ingester:
  lifecycler:
    address: 10.60.134.55 #日志服务器ip，即本机地址
    ring:
      kvstore:
        store: inmemory
      replication_factor: 1
    final_sleep: 0s
  chunk_idle_period: 5m
  chunk_retain_period: 30s
schema_config:
  configs:
  - from: 2024-04-01
    store: boltdb
    object_store: filesystem
    schema: v11
    index:
      prefix: index_
      period: 168h   #每张表的时间范围7天
storage_config:
  boltdb:
    directory: /data/loki/index   #索引文件存储地址
  filesystem:
    directory: /data/loki/chunks  #块存储地址
limits_config:
  enforce_metric_name: false
  reject_old_samples: true
  reject_old_samples_max_age: 168h
chunk_store_config:
# 最大可查询历史日期 7天,这个时间必须是schema_config中的period的倍数，否则报错。
  max_look_back_period: 168h
# 表的保留期7天
table_manager:
  retention_deletes_enabled: true
  retention_period: 168h

并创建启动脚本start.sh和关闭脚本shutdown.sh

#!/bin/bash
nohup ./loki-linux-amd64 -config.file=./config.yaml >./server.log 2>&1 &
echo "$!" > pid
#!/bin/bash
kill -9 `cat pid`
echo "关闭成功!"

启动loki命令：sh start.sh。如下日志表示启动成功。

 

 

安装 grafana

首先解压grafana-enterprise-8.3.3.linux-amd64.tar.gz，创建启动脚本start.sh和关闭脚本shutdown.sh。注意conf/defaults.ini为grafana的配置文件，其中http_port为浏览器访问端口。

 启动grafana命令：sh start.sh。如下日志表示启动成功。

 

在应用服务器安装 promtail

在应用服务器 上创建目录 /usr/local/logsCollect/promtail用于安装promtail。

解压promtail-linux-amd64.zip，并配置promtail.yaml

server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: ./positions.yaml

clients:
  - url: http://10.60.134.55:8094/loki/api/v1/push #日志服务器loki地址和端口

scrape_configs:
#ucenter1
 - job_name: zcbackend-172.29.21.22-1
   static_configs:
   - targets:
       - 10.60.134.60
   - labels:
      job: zcbachend-172.29.21.22-1
      host: 10.60.134.60
      __path__: /jiuqi/zichan/zichanyitihuazhenghexiangmu-8084/backend/server.log  #本机日志路径

并创建启动脚本start.sh和关闭脚本shutdown.sh

 启动promtail命令：sh start.sh。如下日志表示启动成功。

 

六、 测试

打开grafana。默认密码admin/admin。

 添加数据源

 测试连接

 查询日志

也能通过导入的方式添加面板，json文件如下：

{ 
  "annotations": {
    "list": [
      {
        "builtIn": 1,
        "datasource": {
          "type": "datasource",
          "uid": "grafana"
        },
        "enable": true,
        "hide": true,
        "iconColor": "rgba(0, 211, 255, 1)",
        "name": "Annotations & Alerts",
        "target": {
          "limit": 100,
          "matchAny": false,
          "tags": [],
          "type": "dashboard"
        },
        "type": "dashboard"
      }
    ]
  },
  "editable": true,
  "fiscalYearStartMonth": 0,
  "graphTooltip": 0,
  "id": 3,
  "iteration": 1671275851848,
  "links": [],
  "liveNow": false,
  "panels": [
    {
      "aliasColors": {},
      "bars": false,
      "dashLength": 10,
      "dashes": false,
      "datasource": {
        "type": "loki",
        "uid": "${ENV}"
      },
      "fill": 1,
      "fillGradient": 0,
      "gridPos": {
        "h": 5,
        "w": 24,
        "x": 0,
        "y": 0
      },
      "hiddenSeries": false,
      "id": 4,
      "legend": {
        "alignAsTable": true,
        "avg": false,
        "current": false,
        "max": false,
        "min": false,
        "rightSide": true,
        "show": true,
        "total": false,
        "values": false
      },
      "lines": true,
      "linewidth": 1,
      "nullPointMode": "null",
      "options": {
        "alertThreshold": true
      },
      "percentage": false,
      "pluginVersion": "8.5.10",
      "pointradius": 2,
      "points": false,
      "renderer": "flot",
      "seriesOverrides": [],
      "spaceLength": 10,
      "stack": false,
      "steppedLine": false,
      "targets": [
        {
          "datasource": {
            "type": "loki",
            "uid": "hJ7iGN-Sz"
          },
          "expr": "sum (count_over_time({filename=~\"$log\",host=~\"$host\"}[2m] )) by (host,filename)",
          "hide": false,
          "legendFormat": "{{host}}=={{filename}}",
          "refId": "B"
        }
      ],
      "thresholds": [],
      "timeRegions": [],
      "title": "日志量统计",
      "tooltip": {
        "shared": true,
        "sort": 0,
        "value_type": "individual"
      },
      "type": "graph",
      "xaxis": {
        "mode": "time",
        "show": true,
        "values": []
      },
      "yaxes": [
        {
          "$$hashKey": "object:319",
          "format": "short",
          "logBase": 1,
          "show": true
        },
        {
          "$$hashKey": "object:320",
          "format": "short",
          "logBase": 1,
          "show": true
        }
      ],
      "yaxis": {
        "align": false
      }
    },
    {
      "datasource": {
        "type": "loki",
        "uid": "hJ7iGN-Sz"
      },
      "description": "",
      "gridPos": {
        "h": 21,
        "w": 24,
        "x": 0,
        "y": 5
      },
      "id": 2,
      "options": {
        "dedupStrategy": "exact",
        "enableLogDetails": false,
        "prettifyLogMessage": false,
        "showCommonLabels": false,
        "showLabels": false,
        "showTime": true,
        "sortOrder": "Descending",
        "wrapLogMessage": true
      },
      "pluginVersion": "7.4.3",
      "targets": [
        {
          "datasource": {
            "type": "loki",
            "uid": "hJ7iGN-Sz"
          },
          "expr": "{filename=~\"$log\",host=~\"$host\"}|~ \"(?i)$log_level\"",
          "hide": false,
          "maxLines": 500,
          "refId": "C"
        }
      ],
      "title": "日志",
      "transparent": true,
      "type": "logs"
    }
  ],
  "refresh": "",
  "schemaVersion": 36,
  "style": "dark",
  "tags": [
    "PLG"
  ],
  "templating": {
    "list": [
      {
        "current": {
          "selected": false,
          "text": "Loki",
          "value": "Loki"
        },
        "hide": 0,
        "includeAll": false,
        "label": "选择环境",
        "multi": false,
        "name": "ENV",
        "options": [],
        "query": "loki",
        "queryValue": "",
        "refresh": 1,
        "regex": "",
        "skipUrlSync": false,
        "type": "datasource"
      },
      {
        "current": {
          "selected": true,
          "text": "资产server",
          "value": "资产server"
        },
        "datasource": {
          "type": "loki",
          "uid": "hJ7iGN-Sz"
        },
        "definition": "label_values({},filename)",
        "hide": 0,
        "includeAll": false,
        "label": "日志",
        "multi": false,
        "name": "log",
        "options": [
          {
            "selected": true,
            "text": "资产server-8084",
            "value": "/jiuqi/zichan/zichanyitihuazhenghexiangmu-8084/backend/server.log"
          },
          {
            "selected": true,
            "text": "资产server-8085",
            "value": "/jiuqi/zichan/zichanyitihuazhenghexiangmu-8085/backend/server.log"
          },
          {
            "selected": true,
            "text": "资产server-8086",
            "value": "/jiuqi/zichan/zichanyitihuazhenghexiangmu-8086/backend/server.log"
          },
          {
            "selected": true,
            "text": "公车server-8087",
            "value": "/jiuqi/gongche/yitihuagongchehouduan-8087/backend/server.log"
          },
          {
            "selected": true,
            "text": "公车server-8088",
            "value": "/jiuqi/gongche/yitihuagongchehouduan-8088/backend/server.log"
          }
          
         ],
        "query": "label_values({},filename)",
        "refresh": 1,
        "regex": "",
        "skipUrlSync": false,
        "sort": 1,
        "type": "query"
      },
      {
        "current": {
          "selected": true,
          "text": "172.29.21.22",
          "value": "10.60.134.60"
        },
        "datasource": {
          "type": "loki",
          "uid": "hJ7iGN-Sz"
        },
        "definition": "label_values({},host)",
        "hide": 0,
        "includeAll": false,
        "label": "主机IP",
        "multi": false,
        "name": "host",
        "options": [
         {
            "selected": true,
            "text": "资产-172.29.21.22",
            "value": "10.60.134.60"
          },
          {
            "selected": true,
            "text": "资产-172.29.21.164",
            "value": "10.60.134.48"
          },
          {
            "selected": true,
            "text": "公车-172.29.21.36",
            "value": "10.60.134.56"
          }
        ],
        "query": "label_values({},host)",
        "refresh": 1,
        "regex": "",
        "skipUrlSync": false,
        "sort": 1,
        "type": "query"
      },
      {
        "allValue": "(^\\\\S|^\\\\s)",
        "current": {
          "selected": false,
          "text": "All",
          "value": "$__all"
        },
        "description": "可以直接输入搜索的关键字进行过滤",
        "hide": 0,
        "includeAll": true,
        "label": "关键字过滤",
        "multi": false,
        "name": "log_level",
        "options": [
          {
            "selected": true,
            "text": "All",
            "value": "$__all"
          },
          {
            "selected": false,
            "text": "INFO",
            "value": "INFO"
          },
          {
            "selected": false,
            "text": "DEBUG",
            "value": "DEBUG"
          },
          {
            "selected": false,
            "text": "WARN",
            "value": "WARN"
          },
          {
            "selected": false,
            "text": "ERROR",
            "value": "ERROR"
          },
          {
            "selected": false,
            "text": "UNKNOWN",
            "value": "UNKNOWN"
          },
          {
            "selected": false,
            "text": "直接输入关键字搜索",
            "value": "直接输入关键字搜索"
          }
        ],
        "query": "INFO,DEBUG,WARN,ERROR,UNKNOWN,直接输入关键字搜索",
        "queryValue": "",
        "skipUrlSync": false,
        "type": "custom"
      }
    ]
  },
  "time": {
    "from": "now-1h",
    "to": "now"
  },
  "timepicker": {},
  "timezone": "",
  "title": "Loki日志收集",
  "uid": "NlV_8QD7k",
  "version": 19,
  "weekStart": ""
}
logcollect.json

 

热门相关：甜蜜婚令：首长的影后娇妻   嫡女谋：逆天三小姐   我的学姐会魔法   来自异世界的诺诺   闪婚总裁很惧内