All versions of this manual
X
Welcome to Linkurious administration manual v4.3.15. Use the menu to access other documentations.
  1. Introduction
  2. Feature Availability
  3. FAQ
  4. Getting started
    1. Version support policy
    2. Release notes
    3. Client requirements
    4. Graph DB Feature Map
  5. Deploying on premise
    1. Technical requirements
    2. Downloading
    3. Installing
    4. Starting Linkurious
    5. Stopping Linkurious
    6. Cluster mode
  6. Configuring
  7. Troubleshooting
    1. Checking Linkurious status
    2. Reading the logs
    3. Getting support
    4. Managing licenses
    5. Vulnerabilities
  8. Importing graph data
    1. Neo4j
    2. Memgraph
    3. Amazon Neptune
    4. Azure Cosmos DB
    5. Google Spanner Graph
    6. Google BigQuery Graph
    7. CSV Import
    8. Updating your graph data
  9. Updating Linkurious
    1. Checking for updates
    2. Backing up your data
    3. Update procedure
  10. Configuring data-sources
    1. Neo4j
    2. Amazon Neptune
    3. Memgraph
    4. Google Spanner Graph
    5. Google BigQuery Graph
    6. Azure Cosmos DB
    7. Alternative IDs
    8. Merging data-sources
    9. Advanced settings
  11. Resource management
    1. Managing spaces
    2. Managing tags
  12. Search index
    1. Search Option Comparison
    2. Neo4j Search
    3. Elasticsearch
    4. Amazon Neptune Search
    5. Google Spanner Graph Search
    6. Google BigQuery Graph Search
    7. Azure Search
    8. Incremental Indexing
    9. Numerical/Date search
  13. User-data store
    1. Migrating to an external database
    2. Backing-up your store
  14. Web server
  15. Authentication
    1. Getting started
    2. LDAP / Active Directory
    3. SSO with Azure
    4. SSO with Google
    5. SSO with OpenID Connect
    6. SSO with SAML2 / ADFS
    7. Configuration
  16. Access control
    1. Managing users
    2. Managing groups
    3. Standard access-rights
    4. Property-key access-rights
  17. Guest mode
    1. Guest mode
  18. Data Schema
    1. Property types
    2. Hiding data
    3. Strict mode
  19. Alerts
    1. Configuration
  20. Entity resolution
    1. Requirements
    2. Installing
    3. Configuring
    4. Accessing Entity Resolution
    5. Managing mappings
    6. Running Entity Resolution
    7. Viewing Results and explanations
    8. Node Grouping and Collapsing
    9. Managing the license
    10. Troubleshooting tips
  21. Email Notifications
    1. Configuration
  22. Copilot
    1. Configuration
  23. Visualizations appearance
    1. Default styles
    2. Default captions
    3. Default properties order
    4. Geographic tiles
  24. Audit trail
    1. Log format
  25. Observability
  26. Plugins
    1. Query AI
  27. Integrations
    1. Deep link
    2. Webhooks
    3. API Applications
 

Observability

Metrics and traces

Linkurious Enterprise is instrumented with OpenTelemetry, the open standard for observability. When enabled, Linkurious Enterprise produces:

  • Metrics: process and Node.js runtime metrics, HTTP metrics, and Linkurious Enterprise specific metrics (see Available metrics).
  • Traces: spans for incoming and outgoing HTTP requests, and for the other libraries supported by OpenTelemetry auto-instrumentation.
  • Log correlation: when a log is written during a traced operation, the Linkurious Enterprise log record includes its trace_id and span_id, so you can match logs with traces in your observability backend. Logs themselves are not exported through OpenTelemetry.

Linkurious Enterprise also still supports the deprecated legacy Prometheus metrics configured through the metrics key of linkurious/data/config/production.json. This mechanism is deprecated and will be removed in a future version, use OpenTelemetry instead. OpenTelemetry and the legacy Prometheus metrics cannot be enabled at the same time: if both are enabled, Linkurious Enterprise fails to start.

Configuration

OpenTelemetry is disabled by default. It is configured only through environment variables, not through linkurious/data/config/production.json.

To enable it, set LKE_OTEL_ENABLED=true. Then use the standard OpenTelemetry SDK environment variables to configure exporters and sampling. The defaults differ between the Docker image and other installations:

Variable Docker image default Default otherwise
OTEL_METRICS_EXPORTER prometheus otlp
OTEL_EXPORTER_PROMETHEUS_PORT 9400 9464
OTEL_TRACES_EXPORTER otlp otlp
OTEL_TRACES_SAMPLER parentbased_traceidratio parentbased_always_on
OTEL_TRACES_SAMPLER_ARG 0.1 (10% of traces) -
OTEL_EXPORTER_OTLP_ENDPOINT http://localhost:4318 http://localhost:4318

If you do not run an OTLP collector, set OTEL_TRACES_EXPORTER=none. Outside the Docker image, also set OTEL_METRICS_EXPORTER=prometheus to expose a Prometheus endpoint (on port 9464 by default), or OTEL_METRICS_EXPORTER=none to disable metrics. Otherwise Linkurious Enterprise will keep trying to send telemetry to an endpoint that does not exist.

When OpenTelemetry starts, Linkurious Enterprise logs OpenTelemetry SDK started. If it fails to start, Linkurious Enterprise logs an error and keeps running without telemetry.

Example: Prometheus scraping (Docker)

docker run -e LKE_OTEL_ENABLED=true -e OTEL_TRACES_EXPORTER=none -p 9400:9400 ...

Metrics are then available at http://<host>:9400/metrics. Configure Prometheus to scrape this endpoint. If you are new to Prometheus, see their "getting started" guide.

Example: OTLP collector

Set the following environment variables (see how to set environment variables):

LKE_OTEL_ENABLED=true
OTEL_METRICS_EXPORTER=otlp
OTEL_TRACES_EXPORTER=otlp
OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4318

Available metrics

Data-sources

Metric Type Attributes
linkurious_graph_query_duration_seconds Histogram source_key, graph_vendor, dialect, operation, origin, outcome
linkurious_graph_query_total Counter Same as above
linkurious_datasource_state Gauge source_key, graph_vendor, index_vendor, state
  • operation is read or write. outcome is success, error or timeout.
  • linkurious_datasource_state reports one series per data-source and possible state. The value is 1 for the current state and 0 for the others. For example, alert on linkurious_datasource_state{state="offline"} == 1.

Process

process_cpu_seconds_total, process_cpu_user_seconds_total, process_cpu_system_seconds_total, process_resident_memory_bytes, process_heap_bytes, process_start_time_seconds, nodejs_active_handles_total and nodejs_active_requests_total.

Node.js runtime and HTTP

The OpenTelemetry auto-instrumentation also produces these metrics:

  • Node.js runtime: event loop delay and utilization, garbage collection duration, heap usage by space.
  • HTTP: request durations.

Their names follow the OpenTelemetry semantic conventions.

Legacy Prometheus metrics

This mechanism is deprecated and will be removed in a future version of Linkurious Enterprise. Use OpenTelemetry instead.

When legacy metrics are enabled, Linkurious Enterprise provides an API endpoint on /metrics that can be scraped by Prometheus.

Legacy configuration

Legacy metrics can be enabled and configured in linkurious/data/config/production.json. They are found under the metrics key, which contains the following options:

  • enabled (default: false): Whether to enable metrics.
  • listenPort (default: 9400): Port where the metrics server will listen for incoming requests.

You will then need to configure Prometheus to scrape this endpoint. If you are not familiar with Prometheus, please refer to their "getting started" guide.

Migrating to OpenTelemetry

  1. Set metrics.enabled to false in linkurious/data/config/production.json.
  2. Enable OpenTelemetry as described in Configuration.
  3. Update your dashboards and alerts: most metric names differ between the legacy metrics and OpenTelemetry (see Available metrics).

With the Docker image, the OpenTelemetry Prometheus exporter listens on port 9400 and path /metrics by default, like the legacy metrics, so the Prometheus scrape target does not change.

Dashboard

Once metrics are ingested by Prometheus, you could set up a dashboard using Grafana to visualize these metrics. An example Grafana dashboard for the legacy metrics is provided here.

This dashboard can be imported by following the official grafana documentation and using the link above.

Example data

As an illustration of the available legacy metrics and data format, this is an example response from the /metrics API endpoint:

# HELP process_cpu_user_seconds_total Total user CPU time spent in seconds.
# TYPE process_cpu_user_seconds_total counter
process_cpu_user_seconds_total 82.296112

# HELP process_cpu_system_seconds_total Total system CPU time spent in seconds.
# TYPE process_cpu_system_seconds_total counter
process_cpu_system_seconds_total 30.634486

# HELP process_cpu_seconds_total Total user and system CPU time spent in seconds.
# TYPE process_cpu_seconds_total counter
process_cpu_seconds_total 112.930598

# HELP process_start_time_seconds Start time of the process since unix epoch in seconds.
# TYPE process_start_time_seconds gauge
process_start_time_seconds 1693901797

# HELP process_resident_memory_bytes Resident memory size in bytes.
# TYPE process_resident_memory_bytes gauge
process_resident_memory_bytes 182259712

# HELP process_virtual_memory_bytes Virtual memory size in bytes.
# TYPE process_virtual_memory_bytes gauge
process_virtual_memory_bytes 11938127872

# HELP process_heap_bytes Process heap size in bytes.
# TYPE process_heap_bytes gauge
process_heap_bytes 246452224

# HELP process_open_fds Number of open file descriptors.
# TYPE process_open_fds gauge
process_open_fds 47

# HELP process_max_fds Maximum number of open file descriptors.
# TYPE process_max_fds gauge
process_max_fds 1048576

# HELP nodejs_eventloop_lag_seconds Lag of event loop in seconds.
# TYPE nodejs_eventloop_lag_seconds gauge
nodejs_eventloop_lag_seconds 0

# HELP nodejs_eventloop_lag_min_seconds The minimum recorded event loop delay.
# TYPE nodejs_eventloop_lag_min_seconds gauge
nodejs_eventloop_lag_min_seconds 0.006279168

# HELP nodejs_eventloop_lag_max_seconds The maximum recorded event loop delay.
# TYPE nodejs_eventloop_lag_max_seconds gauge
nodejs_eventloop_lag_max_seconds 58.988691455

# HELP nodejs_eventloop_lag_mean_seconds The mean of the recorded event loop delays.
# TYPE nodejs_eventloop_lag_mean_seconds gauge
nodejs_eventloop_lag_mean_seconds 0.010528417698386331

# HELP nodejs_eventloop_lag_stddev_seconds The standard deviation of the recorded event loop delays.
# TYPE nodejs_eventloop_lag_stddev_seconds gauge
nodejs_eventloop_lag_stddev_seconds 0.0949220590624985

# HELP nodejs_eventloop_lag_p50_seconds The 50th percentile of the recorded event loop delays.
# TYPE nodejs_eventloop_lag_p50_seconds gauge
nodejs_eventloop_lag_p50_seconds 0.010330111

# HELP nodejs_eventloop_lag_p90_seconds The 90th percentile of the recorded event loop delays.
# TYPE nodejs_eventloop_lag_p90_seconds gauge
nodejs_eventloop_lag_p90_seconds 0.010510335

# HELP nodejs_eventloop_lag_p99_seconds The 99th percentile of the recorded event loop delays.
# TYPE nodejs_eventloop_lag_p99_seconds gauge
nodejs_eventloop_lag_p99_seconds 0.011575295

# HELP nodejs_active_resources Number of active resources that are currently keeping the event loop alive, grouped by async resource type.
# TYPE nodejs_active_resources gauge
nodejs_active_resources{type="PipeWrap"} 12
nodejs_active_resources{type="ProcessWrap"} 10
nodejs_active_resources{type="TCPServerWrap"} 2
nodejs_active_resources{type="TCPSocketWrap"} 5
nodejs_active_resources{type="Timeout"} 11
nodejs_active_resources{type="Immediate"} 1

# HELP nodejs_active_resources_total Total number of active resources.
# TYPE nodejs_active_resources_total gauge
nodejs_active_resources_total 41

# HELP nodejs_active_handles Number of active libuv handles grouped by handle type. Every handle type is C++ class name.
# TYPE nodejs_active_handles gauge
nodejs_active_handles{type="Socket"} 7
nodejs_active_handles{type="Pipe"} 10
nodejs_active_handles{type="ChildProcess"} 10
nodejs_active_handles{type="Server"} 2

# HELP nodejs_active_handles_total Total number of active handles.
# TYPE nodejs_active_handles_total gauge
nodejs_active_handles_total 29

# HELP nodejs_active_requests Number of active libuv requests grouped by request type. Every request type is C++ class name.
# TYPE nodejs_active_requests gauge

# HELP nodejs_active_requests_total Total number of active requests.
# TYPE nodejs_active_requests_total gauge
nodejs_active_requests_total 0

# HELP nodejs_heap_size_total_bytes Process heap size from Node.js in bytes.
# TYPE nodejs_heap_size_total_bytes gauge
nodejs_heap_size_total_bytes 89440256

# HELP nodejs_heap_size_used_bytes Process heap size used from Node.js in bytes.
# TYPE nodejs_heap_size_used_bytes gauge
nodejs_heap_size_used_bytes 81652264

# HELP nodejs_external_memory_bytes Node.js external memory size in bytes.
# TYPE nodejs_external_memory_bytes gauge
nodejs_external_memory_bytes 5334109

# HELP nodejs_heap_space_size_total_bytes Process heap space size total from Node.js in bytes.
# TYPE nodejs_heap_space_size_total_bytes gauge
nodejs_heap_space_size_total_bytes{space="read_only"} 0
nodejs_heap_space_size_total_bytes{space="old"} 62533632
nodejs_heap_space_size_total_bytes{space="code"} 7024640
nodejs_heap_space_size_total_bytes{space="map"} 2891776
nodejs_heap_space_size_total_bytes{space="large_object"} 14950400
nodejs_heap_space_size_total_bytes{space="code_large_object"} 991232
nodejs_heap_space_size_total_bytes{space="new_large_object"} 0
nodejs_heap_space_size_total_bytes{space="new"} 1048576

# HELP nodejs_heap_space_size_used_bytes Process heap space size used from Node.js in bytes.
# TYPE nodejs_heap_space_size_used_bytes gauge
nodejs_heap_space_size_used_bytes{space="read_only"} 0
nodejs_heap_space_size_used_bytes{space="old"} 57058280
nodejs_heap_space_size_used_bytes{space="code"} 5979200
nodejs_heap_space_size_used_bytes{space="map"} 2025864
nodejs_heap_space_size_used_bytes{space="large_object"} 14768216
nodejs_heap_space_size_used_bytes{space="code_large_object"} 972544
nodejs_heap_space_size_used_bytes{space="new_large_object"} 0
nodejs_heap_space_size_used_bytes{space="new"} 861432

# HELP nodejs_heap_space_size_available_bytes Process heap space size available from Node.js in bytes.
# TYPE nodejs_heap_space_size_available_bytes gauge
nodejs_heap_space_size_available_bytes{space="read_only"} 0
nodejs_heap_space_size_available_bytes{space="old"} 4319168
nodejs_heap_space_size_available_bytes{space="code"} 603072
nodejs_heap_space_size_available_bytes{space="map"} 813112
nodejs_heap_space_size_available_bytes{space="large_object"} 0
nodejs_heap_space_size_available_bytes{space="code_large_object"} 0
nodejs_heap_space_size_available_bytes{space="new_large_object"} 1030976
nodejs_heap_space_size_available_bytes{space="new"} 169544

# HELP nodejs_version_info Node.js version info.
# TYPE nodejs_version_info gauge
nodejs_version_info{version="v18.16.1",major="18",minor="16",patch="1"} 1

# HELP nodejs_gc_duration_seconds Garbage collection duration by kind, one of major, minor, incremental or weakcb.
# TYPE nodejs_gc_duration_seconds histogram
nodejs_gc_duration_seconds_bucket{le="0.001",kind="minor"} 145
nodejs_gc_duration_seconds_bucket{le="0.01",kind="minor"} 553
nodejs_gc_duration_seconds_bucket{le="0.1",kind="minor"} 560
nodejs_gc_duration_seconds_bucket{le="1",kind="minor"} 560
nodejs_gc_duration_seconds_bucket{le="2",kind="minor"} 560
nodejs_gc_duration_seconds_bucket{le="5",kind="minor"} 560
nodejs_gc_duration_seconds_bucket{le="+Inf",kind="minor"} 560
nodejs_gc_duration_seconds_sum{kind="minor"} 0.9594195639983442
nodejs_gc_duration_seconds_count{kind="minor"} 560
nodejs_gc_duration_seconds_bucket{le="0.001",kind="incremental"} 28
nodejs_gc_duration_seconds_bucket{le="0.01",kind="incremental"} 34
nodejs_gc_duration_seconds_bucket{le="0.1",kind="incremental"} 35
nodejs_gc_duration_seconds_bucket{le="1",kind="incremental"} 35
nodejs_gc_duration_seconds_bucket{le="2",kind="incremental"} 35
nodejs_gc_duration_seconds_bucket{le="5",kind="incremental"} 35
nodejs_gc_duration_seconds_bucket{le="+Inf",kind="incremental"} 35
nodejs_gc_duration_seconds_sum{kind="incremental"} 0.027089656999247376
nodejs_gc_duration_seconds_count{kind="incremental"} 35
nodejs_gc_duration_seconds_bucket{le="0.001",kind="major"} 0
nodejs_gc_duration_seconds_bucket{le="0.01",kind="major"} 4
nodejs_gc_duration_seconds_bucket{le="0.1",kind="major"} 21
nodejs_gc_duration_seconds_bucket{le="1",kind="major"} 21
nodejs_gc_duration_seconds_bucket{le="2",kind="major"} 21
nodejs_gc_duration_seconds_bucket{le="5",kind="major"} 21
nodejs_gc_duration_seconds_bucket{le="+Inf",kind="major"} 21
nodejs_gc_duration_seconds_sum{kind="major"} 0.42418163500053924
nodejs_gc_duration_seconds_count{kind="major"} 21

# HELP http_request_duration_seconds duration histogram of http responses labeled with: status_code, method, path
# TYPE http_request_duration_seconds histogram
http_request_duration_seconds_bucket{le="0.003",status_code="304",method="GET",path="/"} 0
http_request_duration_seconds_bucket{le="0.03",status_code="304",method="GET",path="/"} 3
http_request_duration_seconds_bucket{le="0.1",status_code="304",method="GET",path="/"} 3
http_request_duration_seconds_bucket{le="0.3",status_code="304",method="GET",path="/"} 4
http_request_duration_seconds_bucket{le="1.5",status_code="304",method="GET",path="/"} 4
http_request_duration_seconds_bucket{le="10",status_code="304",method="GET",path="/"} 4
http_request_duration_seconds_bucket{le="+Inf",status_code="304",method="GET",path="/"} 4

# HELP up 1 = up, 0 = not up
# TYPE up gauge
up 1