Observability
Metrics
Linkurious Enterprise can be monitored using Prometheus, which is an open source
monitoring tool.
When metrics are enabled, Linkurious Enterprise will provide an API endpoint on /metrics
that can be
scrapped by Prometheus.
Configuration
Metrics can be enabled and configured in linkurious/data/config/production.json
.
They are found under the metrics
key, which contains the following options:
enabled
(default:false
): Whether to enable metrics.listenPort
(default:9400
): Port where the metrics server will listen for incoming requests.
You will then need to configure Prometheus to scrap this endpoint. If you are not familiar with Prometheus, please refer to their "getting started" guide.
Dashboard
Once metrics are ingested by Prometheus, you could set up a dashboard using Grafana to visualize these metrics. An example Grafana dashboard is provided here.
This dashboard can be imported by following the official grafana documentation and using the link above.
Example data
As an illustration of the available metrics and data format, this is an example response from
the /metrics
API endpoint:
# HELP process_cpu_user_seconds_total Total user CPU time spent in seconds.
# TYPE process_cpu_user_seconds_total counter
process_cpu_user_seconds_total 82.296112
# HELP process_cpu_system_seconds_total Total system CPU time spent in seconds.
# TYPE process_cpu_system_seconds_total counter
process_cpu_system_seconds_total 30.634486
# HELP process_cpu_seconds_total Total user and system CPU time spent in seconds.
# TYPE process_cpu_seconds_total counter
process_cpu_seconds_total 112.930598
# HELP process_start_time_seconds Start time of the process since unix epoch in seconds.
# TYPE process_start_time_seconds gauge
process_start_time_seconds 1693901797
# HELP process_resident_memory_bytes Resident memory size in bytes.
# TYPE process_resident_memory_bytes gauge
process_resident_memory_bytes 182259712
# HELP process_virtual_memory_bytes Virtual memory size in bytes.
# TYPE process_virtual_memory_bytes gauge
process_virtual_memory_bytes 11938127872
# HELP process_heap_bytes Process heap size in bytes.
# TYPE process_heap_bytes gauge
process_heap_bytes 246452224
# HELP process_open_fds Number of open file descriptors.
# TYPE process_open_fds gauge
process_open_fds 47
# HELP process_max_fds Maximum number of open file descriptors.
# TYPE process_max_fds gauge
process_max_fds 1048576
# HELP nodejs_eventloop_lag_seconds Lag of event loop in seconds.
# TYPE nodejs_eventloop_lag_seconds gauge
nodejs_eventloop_lag_seconds 0
# HELP nodejs_eventloop_lag_min_seconds The minimum recorded event loop delay.
# TYPE nodejs_eventloop_lag_min_seconds gauge
nodejs_eventloop_lag_min_seconds 0.006279168
# HELP nodejs_eventloop_lag_max_seconds The maximum recorded event loop delay.
# TYPE nodejs_eventloop_lag_max_seconds gauge
nodejs_eventloop_lag_max_seconds 58.988691455
# HELP nodejs_eventloop_lag_mean_seconds The mean of the recorded event loop delays.
# TYPE nodejs_eventloop_lag_mean_seconds gauge
nodejs_eventloop_lag_mean_seconds 0.010528417698386331
# HELP nodejs_eventloop_lag_stddev_seconds The standard deviation of the recorded event loop delays.
# TYPE nodejs_eventloop_lag_stddev_seconds gauge
nodejs_eventloop_lag_stddev_seconds 0.0949220590624985
# HELP nodejs_eventloop_lag_p50_seconds The 50th percentile of the recorded event loop delays.
# TYPE nodejs_eventloop_lag_p50_seconds gauge
nodejs_eventloop_lag_p50_seconds 0.010330111
# HELP nodejs_eventloop_lag_p90_seconds The 90th percentile of the recorded event loop delays.
# TYPE nodejs_eventloop_lag_p90_seconds gauge
nodejs_eventloop_lag_p90_seconds 0.010510335
# HELP nodejs_eventloop_lag_p99_seconds The 99th percentile of the recorded event loop delays.
# TYPE nodejs_eventloop_lag_p99_seconds gauge
nodejs_eventloop_lag_p99_seconds 0.011575295
# HELP nodejs_active_resources Number of active resources that are currently keeping the event loop alive, grouped by async resource type.
# TYPE nodejs_active_resources gauge
nodejs_active_resources{type="PipeWrap"} 12
nodejs_active_resources{type="ProcessWrap"} 10
nodejs_active_resources{type="TCPServerWrap"} 2
nodejs_active_resources{type="TCPSocketWrap"} 5
nodejs_active_resources{type="Timeout"} 11
nodejs_active_resources{type="Immediate"} 1
# HELP nodejs_active_resources_total Total number of active resources.
# TYPE nodejs_active_resources_total gauge
nodejs_active_resources_total 41
# HELP nodejs_active_handles Number of active libuv handles grouped by handle type. Every handle type is C++ class name.
# TYPE nodejs_active_handles gauge
nodejs_active_handles{type="Socket"} 7
nodejs_active_handles{type="Pipe"} 10
nodejs_active_handles{type="ChildProcess"} 10
nodejs_active_handles{type="Server"} 2
# HELP nodejs_active_handles_total Total number of active handles.
# TYPE nodejs_active_handles_total gauge
nodejs_active_handles_total 29
# HELP nodejs_active_requests Number of active libuv requests grouped by request type. Every request type is C++ class name.
# TYPE nodejs_active_requests gauge
# HELP nodejs_active_requests_total Total number of active requests.
# TYPE nodejs_active_requests_total gauge
nodejs_active_requests_total 0
# HELP nodejs_heap_size_total_bytes Process heap size from Node.js in bytes.
# TYPE nodejs_heap_size_total_bytes gauge
nodejs_heap_size_total_bytes 89440256
# HELP nodejs_heap_size_used_bytes Process heap size used from Node.js in bytes.
# TYPE nodejs_heap_size_used_bytes gauge
nodejs_heap_size_used_bytes 81652264
# HELP nodejs_external_memory_bytes Node.js external memory size in bytes.
# TYPE nodejs_external_memory_bytes gauge
nodejs_external_memory_bytes 5334109
# HELP nodejs_heap_space_size_total_bytes Process heap space size total from Node.js in bytes.
# TYPE nodejs_heap_space_size_total_bytes gauge
nodejs_heap_space_size_total_bytes{space="read_only"} 0
nodejs_heap_space_size_total_bytes{space="old"} 62533632
nodejs_heap_space_size_total_bytes{space="code"} 7024640
nodejs_heap_space_size_total_bytes{space="map"} 2891776
nodejs_heap_space_size_total_bytes{space="large_object"} 14950400
nodejs_heap_space_size_total_bytes{space="code_large_object"} 991232
nodejs_heap_space_size_total_bytes{space="new_large_object"} 0
nodejs_heap_space_size_total_bytes{space="new"} 1048576
# HELP nodejs_heap_space_size_used_bytes Process heap space size used from Node.js in bytes.
# TYPE nodejs_heap_space_size_used_bytes gauge
nodejs_heap_space_size_used_bytes{space="read_only"} 0
nodejs_heap_space_size_used_bytes{space="old"} 57058280
nodejs_heap_space_size_used_bytes{space="code"} 5979200
nodejs_heap_space_size_used_bytes{space="map"} 2025864
nodejs_heap_space_size_used_bytes{space="large_object"} 14768216
nodejs_heap_space_size_used_bytes{space="code_large_object"} 972544
nodejs_heap_space_size_used_bytes{space="new_large_object"} 0
nodejs_heap_space_size_used_bytes{space="new"} 861432
# HELP nodejs_heap_space_size_available_bytes Process heap space size available from Node.js in bytes.
# TYPE nodejs_heap_space_size_available_bytes gauge
nodejs_heap_space_size_available_bytes{space="read_only"} 0
nodejs_heap_space_size_available_bytes{space="old"} 4319168
nodejs_heap_space_size_available_bytes{space="code"} 603072
nodejs_heap_space_size_available_bytes{space="map"} 813112
nodejs_heap_space_size_available_bytes{space="large_object"} 0
nodejs_heap_space_size_available_bytes{space="code_large_object"} 0
nodejs_heap_space_size_available_bytes{space="new_large_object"} 1030976
nodejs_heap_space_size_available_bytes{space="new"} 169544
# HELP nodejs_version_info Node.js version info.
# TYPE nodejs_version_info gauge
nodejs_version_info{version="v18.16.1",major="18",minor="16",patch="1"} 1
# HELP nodejs_gc_duration_seconds Garbage collection duration by kind, one of major, minor, incremental or weakcb.
# TYPE nodejs_gc_duration_seconds histogram
nodejs_gc_duration_seconds_bucket{le="0.001",kind="minor"} 145
nodejs_gc_duration_seconds_bucket{le="0.01",kind="minor"} 553
nodejs_gc_duration_seconds_bucket{le="0.1",kind="minor"} 560
nodejs_gc_duration_seconds_bucket{le="1",kind="minor"} 560
nodejs_gc_duration_seconds_bucket{le="2",kind="minor"} 560
nodejs_gc_duration_seconds_bucket{le="5",kind="minor"} 560
nodejs_gc_duration_seconds_bucket{le="+Inf",kind="minor"} 560
nodejs_gc_duration_seconds_sum{kind="minor"} 0.9594195639983442
nodejs_gc_duration_seconds_count{kind="minor"} 560
nodejs_gc_duration_seconds_bucket{le="0.001",kind="incremental"} 28
nodejs_gc_duration_seconds_bucket{le="0.01",kind="incremental"} 34
nodejs_gc_duration_seconds_bucket{le="0.1",kind="incremental"} 35
nodejs_gc_duration_seconds_bucket{le="1",kind="incremental"} 35
nodejs_gc_duration_seconds_bucket{le="2",kind="incremental"} 35
nodejs_gc_duration_seconds_bucket{le="5",kind="incremental"} 35
nodejs_gc_duration_seconds_bucket{le="+Inf",kind="incremental"} 35
nodejs_gc_duration_seconds_sum{kind="incremental"} 0.027089656999247376
nodejs_gc_duration_seconds_count{kind="incremental"} 35
nodejs_gc_duration_seconds_bucket{le="0.001",kind="major"} 0
nodejs_gc_duration_seconds_bucket{le="0.01",kind="major"} 4
nodejs_gc_duration_seconds_bucket{le="0.1",kind="major"} 21
nodejs_gc_duration_seconds_bucket{le="1",kind="major"} 21
nodejs_gc_duration_seconds_bucket{le="2",kind="major"} 21
nodejs_gc_duration_seconds_bucket{le="5",kind="major"} 21
nodejs_gc_duration_seconds_bucket{le="+Inf",kind="major"} 21
nodejs_gc_duration_seconds_sum{kind="major"} 0.42418163500053924
nodejs_gc_duration_seconds_count{kind="major"} 21
# HELP http_request_duration_seconds duration histogram of http responses labeled with: status_code, method, path
# TYPE http_request_duration_seconds histogram
http_request_duration_seconds_bucket{le="0.003",status_code="304",method="GET",path="/"} 0
http_request_duration_seconds_bucket{le="0.03",status_code="304",method="GET",path="/"} 3
http_request_duration_seconds_bucket{le="0.1",status_code="304",method="GET",path="/"} 3
http_request_duration_seconds_bucket{le="0.3",status_code="304",method="GET",path="/"} 4
http_request_duration_seconds_bucket{le="1.5",status_code="304",method="GET",path="/"} 4
http_request_duration_seconds_bucket{le="10",status_code="304",method="GET",path="/"} 4
http_request_duration_seconds_bucket{le="+Inf",status_code="304",method="GET",path="/"} 4
# HELP up 1 = up, 0 = not up
# TYPE up gauge
up 1