Preparation

Note: In this doc, we assume you already know how to operate linux, we based on Ubuntu 20.4 and use the root user to execute, so maybe some step need update on your environment if you are use a different environment.
You can change all path in this doc according to your own preferences, but please remember to make the same changes in the configuration later

From the BlockPi official docs, we need some 3rd services to implement the service monitor:

• Prometheus: https://github.com/prometheus/prometheus
• Prometheus node_exporter: https://github.com/prometheus/node_exporter
• Prometheus alertmanager: https://github.com/prometheus/alertmanager
• Grafana: https://github.com/grafana/grafana

Install the Prometheus

create /etc/prometheus path for install and configure Prometheus:

mkdir /etc/prometheus && cd /etc/prometheus/
wget https://github.com/prometheus/prometheus/releases/download/v2.36.2/prometheus-2.36.2.linux-amd64.tar.gz
tar -zxvf prometheus-2.36.2.linux-amd64.tar.gz -C /etc/prometheus/

create the prometheus.yml in the /etc/prometheus:

scrape_configs:
  - job_name: 'prometheus'
    scrape_interval: 5s
    static_configs:
      - targets: ['localhost:9090']

Run Prometheus from systemd file:

vim /usr/lib/systemd/system/prometheus.service

[Unit]
Description=https://prometheus.io
[Service]
Restart=on-failure
ExecStart=/etc/prometheus/prometheus/prometheus --config.file=/etc/prometheus/prometheus.yml
[Install]                        
WantedBy=multi-user.target

# reload the service
systemctl daemon-reload
# boot-up
systemctl enable prometheus.service

# start/stop and check the service status
systemctl stop prometheus.service
systemctl start prometheus.service
systemctl status prometheus.service

you can check your Prometheus service use http://ip:9090 on browser, remember open the 9090 port.

Install alertmanager

wget https://github.com/prometheus/alertmanager/releases/download/v0.24.0/alertmanager-0.24.0.linux-amd64.tar.gz &&  tar -zxvf alertmanager-0.24.0.linux-amd64.tar.gz
mv alertmanager-0.24.0.linux-amd64/ alertmanager

Run alertmanager from systemd file:

vim /lib/systemd/system/alertmanager.service

[Unit]
Description=alertmanager System
Documentation=alertmanager System
[Service]
ExecStart=/etc/prometheus/alertmanager/alertmanager --config.file=/etc/prometheus/alertmanager/alertmanager.yml
[Install]
WantedBy=multi-user.target

# boot-up
systemctl enable alertmanager
# start/stop and check the service status
systemctl start alertmanager
systemctl stop alertmanager
systemctl status alertmanager

update the prometheus.yml file, add the alerting part:

alerting:
  alertmanagers:
  - static_configs:
    - targets: ["localhost:9093"]

you can check your Alertmanager service use http://your-ip:9093 on browser, remember open the 9093 port.

Install the Grafana

Install the Grafana use apt, add the repository for stable releases:

apt-get install -y apt-transport-https
apt-get install -y software-properties-common wget
wget -q -O - https://packages.grafana.com/gpg.key |  apt-key add -

echo "deb https://packages.grafana.com/oss/deb stable main" |  tee -a /etc/apt/sources.list.d/grafana.list

After you add the repository:

 apt-get update
 apt-get install grafana

To start the service and verify that the service has started:

systemctl daemon-reload
systemctl start grafana-server
systemctl status grafana-server

Configure the Grafana server to start at boot:

systemctl enable grafana-server.service

you can check your Grafana use http://ip:3000 on browser, remember open the 3000 port.

for more install and configuration info: 

• https://grafana.com/docs/grafana/next/getting-started/build-first-dashboard/
• https://grafana.com/docs/grafana/next/setup-grafana/configure-grafana/

Monitoring system status

Basic info from the BlockPi doc: Monitoring system status

To implement this monitoring, you need install the Prometheus node_exporter service on the host which you install the HyperNode, follow these steps:

Install and Configure the node_exporter

Download and install the node_exporter

mkdir ~/node_exporter && cd ~/node_exporter
wget https://github.com/prometheus/node_exporter/releases/download/v1.3.1/node_exporter-1.3.1.linux-amd64.tar.gz
tar -zxvf node_exporter-1.3.1.linux-amd64.tar.gz && rm -rf node_exporter-1.3.1.linux-amd64.tar.gz
mv node_exporter-1.3.1.linux-amd64/node_exporter /usr/local/bin/node_exporter

Run node_exporter from systemd file:

vim /etc/systemd/system/node_exporter.service

[Unit]
Description=node_exporter
Documentation=https://prometheus.io/
After=network.target
[Service]
Type=simple
User=root
ExecStart=/usr/local/bin/node_exporter
Restart=on-failure
[Install]
WantedBy=multi-user.target

# reload the service
systemctl daemon-reload
# boot-up
systemctl enable node_exporter.service

# start/stop service, check the service status
systemctl start node_exporter
systemctl stop node_exporter
systemctl status node_exporter

after start the node_exporter, you can check the status use http://ip:9100/metrics on browser, remember open the 9100 port.

now configure the node_exporter in Prometheus service, you need switch to the host which install the Prometheus and edit the configuration file prometheus.yml, change the targets value to your ip:port:

scrape_configs:
  - job_name: 'prometheus'
    scrape_interval: 5s
    static_configs:
      - targets: ['localhost:9090']
  - job_name: 'node'
    scrape_interval: 10s
    static_configs:
      - targets: ['your-ip:9100']
        labels:
          class: system
          name: klaytn
          region: Finland
          service_provider: Hetzner

restart Prometheus service

systemctl restart prometheus.service

check it on http://ip:9090, open the [Status-Targets] menu, if you are configure correct you can see what like this:

Import the monitoring data to Grafana

After the Prometheus was configured correct, we can import the data to Grafana and view, follow these steps:

1.add a Prometheus data source

input the Prometheus url:port in the URL field.

2.Import the dashboard configuration from the url: https://grafana.com/grafana/dashboards/1860, you can find this url in BlockPi official guide

click the Load button, enter the next step, and modify the Name according to your own preferences

3.after successful import, enter the node in the [Job] input field, which must corresponds to the value of job_name in the prometheus.yml configuration file, if it's successful, you can this: 

Monitoring HyperNode status

Basic info from the BlockPi doc: Monitoring HyperNode status

The steps here are basically similar to Monitoring system status, so instead of going into detail, here are the main steps.

Configure the Prometheus

add the monitoring job to prometheus.yml, change the targets value to your ip:port, then restart service (remember the class value, you need it in the alertmanager part).

 - job_name: 'hypernode'
    scrape_interval: 10s
    static_configs:
      - targets: ['your-ip:8899']
        labels:
          class: HyperNode
          name: hypernode-1
          region: Finland
          service_provider: Hetzner

Configure the Grafana

1.import the dashboard configuration from the json file, you can find this file in BlockPi official guide or click here direct

2.after successful import, modify the variables in Dashboard, change the datasource to Prometheus. 

3.change the query in the view

input the query parameters in Metrics browser field: 

increase(grpc_server_started_total{class="$class", grpc_method="ServerReflectionInfo", job="$job", instance="$node"}[1m])

4.repeat the above steps to modify another view

go_goroutines{class="$class", job="$job", instance="$node"}

Add alert use alertmanager

add alert rule

update the prometheus.yml file, add the rule_files part:

rule_files:
  - /etc/prometheus/rules/*.rules

then add a hyper.rules file in /etc/prometheus/rules/ , you can get the content from BlockPi official doc: click here.
but you need change the class value same to the class value in prometheus.yml file

restart the prometheus service, you can check it on Prometheus page.

receive alert use Slack

you need to register a Slack and add a group and channel for alert.

add a App in the channel detail, follow these steps:

after you add it, you can get a Webhook url.

update the alertmanager configuration file alertmanager.yml in /etc/prometheus/alertmanager, change the `api_url` to your slack webhook url.

route:
  group_by: ['alertname']
  group_wait: 30s
  group_interval: 1m
  repeat_interval: 30m
  receiver: 'my-slack'
receivers:
  - name: 'my-slack'
    slack_configs:
      - send_resolved: true
        api_url: your-slack-hook
        channel: '#blockpi-alert'

restart the alertmanager service, now you can get alert by Slack if your service was down.

Monitoring Klaytn Endpoint Node status

Basic info from the BlockPi doc: Monitoring Klaytn Endpoint Node status

The steps here are basically similar to Monitoring HyperNode status, so instead of going into detail, here are the main steps.

Configure the Prometheus

add the monitoring job to prometheus.yml and restart service (remember the class value, you need it in the alertmanager part).

- job_name: 'klaytn'
    scrape_interval: 10s
    static_configs:
      - targets: ['your-ip:31275']
        labels:
          class: klaytn-rpc-node
          name: klaytn-rpc-node-1
          region: Finland
          service_provider: Hetzner

Configure the Grafana

1.import the dashboard configuration from the json file, you can find this file in BlockPi official guide or click here direct

2.after successful import, modify the variables in Dashboard, change the datasource to Prometheus, reference HyperNode part

3.change the query in the view,

input the query parameters in `Metrics browser` field: 

up{job="$job"}

change to another view, update the Format and Instant, the query parameters:

klaytn_blockchain_head_blocknumber{class="$class"}

change to another view, update the Format and Instant, the query parameters:

process_virtual_memory_bytes{job="$job", class="$class"}/1024/1024/1024

the finall view:

Add alert use alertmanager

the steps is similar to Monitoring HyperNode status, you nedd add a klaytn.rules file in /etc/prometheus/rules/ , you can get the content from BlockPi official doc: click here.
but you need change the class value same to the class value in prometheus.yml file as in the Klaytn part

after added, restart the Prometheus service.

that's all ,enjoy yourself !

License: Apache-2.0
Copyright (c) 2022-present, TestRardar.app
Conatct: https://t.me/testradar_official