Skip to main content
Version: 5.0 🚧

Prometheus Exporter

The observability of SRS is about metrics(Prometheus Exporter), tracing(APM) and logging(Cloud Logging).

Introduction​

For detail specs, please read OpenTelemetry.

Note: Please see Metrics, tracing, and logging

The architecture for Prometheus exporter:

+-----+               +-----------+     +---------+
| SRS +--Exporter-->--| Promethus +-->--+ Grafana +
+-----+ (HTTP) +-----------+ +---------+

There is special config for exporter.

Config​

The config for exporter is bellow. Highly recommend using environment variables to enable it:

# Prometheus exporter config.
# See https://prometheus.io/docs/instrumenting/exporters
exporter {
# Whether exporter is enabled.
# Overwrite by env SRS_EXPORTER_ENABLED
# Default: off
enabled off;
# The http api listen port for exporter metrics.
# Overwrite by env SRS_EXPORTER_LISTEN
# Default: 9972
# See https://github.com/prometheus/prometheus/wiki/Default-port-allocations
listen 9972;
# The logging label to category the cluster servers.
# Overwrite by env SRS_EXPORTER_LABEL
label cn-beijing;
# The logging tag to category the cluster servers.
# Overwrite by env SRS_EXPORTER_TAG
tag cn-edge;
}

Let's start SRS exporter to export metrics to Prometheus.

Usage for SRS Exporter​

Build and start SRS 5.0.86+:

./configure && make
env SRS_ENV_ONLY=on SRS_EXPORTER_ENABLED=on SRS_LISTEN=1935 \
./objs/srs -e

Note: We use envrionment variables to config SRS, without config file. However, you're able to use config file conf/prometheus.conf to start the demo.

Note: Please open http://localhost:9972/metrics to verify SRS.

Then, use FFmpeg to push a live stream to SRS:

docker run --rm -it ossrs/srs:encoder ffmpeg -re -i doc/source.flv -c copy \
-f flv rtmp://host.docker.internal/live/livestream

Next, run node_exporter to collect the node data:

docker run --rm -p 9100:9100 prom/node-exporter

Note: Highly recommend downloading from here and startting by binary file.

Note: Please open http://localhost:9100/metrics to verify it.

Finally, create a prometheus.yml for prometheus:

scrape_configs:
- job_name: "node"
metrics_path: "/metrics"
scrape_interval: 5s
static_configs:
- targets: ["host.docker.internal:9100"]
- job_name: "srs"
metrics_path: "/metrics"
scrape_interval: 5s
static_configs:
- targets: ["host.docker.internal:9972"]

Note: We set the scrape_interval to 5s, which is default to 1m or one minute.

Start Prometheus by:

docker run --rm -v $(pwd)/prometheus.yml:/etc/prometheus/prometheus.yml \
-p 9090:9090 prom/prometheus

Please ope Prometheus: Targets, or Prometheus: Graph to query the input bitrate:

rate(srs_receive_bytes_total[10s])*8

This query is used to query the input bitrate, which is the bitrate of stream:

Normally we use Grafana to render the graph.

Usage for Grafana​

First, start Grafana in docker:

docker run --rm -it -p 3000:3000 \
-e GF_SECURITY_ADMIN_USER=admin \
-e GF_SECURITY_ADMIN_PASSWORD=12345678 \
-e GF_USERS_DEFAULT_THEME=light \
grafana/grafana

Please access Grafana console by http://localhost:3000/

Note: Please input username admin and password 12345678 then click login.

Run command to add a Prometheus DataSource:

curl -s -H "Content-Type: application/json" \
-XPOST http://admin:12345678@localhost:3000/api/datasources \
-d '{
"name": "prometheus",
"type": "prometheus",
"access": "proxy", "isDefault": true,
"url": "http://host.docker.internal:9090"
}'

Run command to import the HelloWorld dashboard:

data=$(curl https://raw.githubusercontent.com/ossrs/srs-grafana/main/dashboards/helloworld-import.json 2>/dev/null)
curl -s -H "Content-Type: application/json" \
-XPOST http://admin:12345678@localhost:3000/api/dashboards/db \
--data-binary "{\"dashboard\":${data},\"overwrite\":true,\"inputs\":[],\"folderId\":0}"

Note: For other dashboards, please see srs-grafana.

Then open Dashboards in browser, you will see the imported dashboard:

There are more other dashboards, please get them in srs-grafana.

Any patch is welcome.