All versions of this manual
X
Welcome to Linkurious administration manual v4.1.9. Use the menu to access other documentations.
  1. Introduction
  2. FAQ
  3. Getting started
    1. Version support policy
    2. Release notes
    3. Client requirements
    4. Graph DB Feature Map
  4. Deploying on premise
    1. Technical requirements
    2. Downloading
    3. Installing
    4. Starting Linkurious
    5. Stopping Linkurious
  5. Configuring
  6. Troubleshooting
    1. Checking Linkurious status
    2. Reading the logs
    3. Getting support
    4. Managing licenses
    5. Vulnerabilities
  7. Importing graph data
    1. Neo4j
    2. Memgraph
    3. Amazon Neptune
    4. Cosmos DB
    5. Updating your graph data
  8. Updating Linkurious
    1. Checking for updates
    2. Backing up your data
    3. Update procedure
  9. Configuring data-sources
    1. Neo4j
    2. Amazon Neptune
    3. Memgraph
    4. Cosmos DB
    5. Alternative IDs
    6. Merging data-sources
    7. Advanced settings
  10. Resource management
    1. Managing spaces
    2. Managing tags
  11. Search index
    1. Search Option Comparison
    2. Neo4j Search
    3. Elasticsearch
    4. OpenSearch for Amazon Neptune
    5. Azure Search
    6. Incremental Indexing
    7. Numerical/Date search
  12. User-data store
    1. Migrating to an external database
    2. Backing-up your store
  13. Web server
  14. Authentication
    1. Getting started
    2. LDAP / Active Directory
    3. SSO with Azure AD
    4. SSO with Google
    5. SSO with OpenID Connect
    6. SSO with SAML2 / ADFS
    7. Configuration
  15. Access control
    1. Managing users
    2. Managing groups
    3. Standard access-rights
    4. Property-key access-rights
  16. Guest mode
    1. Guest mode
  17. Data Schema
    1. Property types
    2. Hiding data
    3. Strict mode
  18. Alerts
    1. Configuration
  19. Email Notifications
    1. Configuration
  20. Visualizations appearance
    1. Default styles
    2. Default captions
    3. Default properties order
    4. Geographic tiles
  21. Audit trail
    1. Log format
  22. Webhooks
  23. Observability
  24. Plugins
  25. Deep link
 

Search index: Incremental Indexing

Incremental indexing allows you to keep in sync your Elasticsearch index and your Neo4j graph database.

Linkurious Enterprise will index at a regular interval new and updated items from your database. This way, you avoid a complete reindex every time you update your data.

This is achieved by keeping track of a timestamp on every node and edge in the database, hereby allowing the indexer to only consider the nodes with newer timestamps and consequently reducing the indexing time.

You should consider this option if your database holds a significant number of items and needs to be updated frequently.

Requirements

1. Compatible Neo4j & Linkurious Enterprise versions

Here some important considerations when choosing incremental indexing:

  • Up to Linkurious Enterprise 3.1.4: incremental indexing was available from Neo4j v3.1.5 and above.
  • Since Linkurious Enterprise 3.1.5, incremental indexing is only available on Neo4j v4.3.0 and above.

2. APOC

Linkurious Enterprise relies on APOC triggers to ensure that every node and edge created or updated has a timestamp.

You need to make sure that you have installed APOC correctly and enabled APOC triggers. You can find all the information you need to install APOC from Neo4j documentation.

You can quickly verify that you have installed everything correctly by executing the following command from your Neo4j browser:

CALL apoc.trigger.list()

3. A property name

Your need to carefully choose the property that will hold the timestamp on every node/edge of your database. The consequence of this choice is that Linkurious Enterprise will create triggers that will store a timestamp on this property for all new and updated nodes/edges.

This means that any information stored on that property will be overwritten by the trigger

Enabling incremental indexing with Elasticsearch

After you have installed and configured APOC, you can enable incremental indexing from the data-source configuration page or by editing the configuration file with the following options:

  • incrementalIndexing You can enable incremental indexing by switching this option to true.
  • timestampPropertyName You must then provide the timestamp property name to keep track of during incremental indexing. E.g: "lastEditTimestamp".

After you have enabled incremental indexing for the first time, Linkurious Enterprise requires to perform a complete re-indexing of the data-source to ensure that every item has been indexed. You simply need to click on the "Start indexing" button to complete the configuration.

Linkurious Enterprise will index your data-source incrementally from that point forward using the timestamps generated by the APOC triggers.

Scheduling incremental indexing

Once you have set up the incremental indexing, you can configure the frequency at which it will be triggered. You can customize this schedule by adding a cron expression to your Elasticsearch configuration.

You can make this change from the configuration file located at linkurious/data/config/production.json under datasources.index for each dataSource.

By default, we have set all incremental indexing to be launched every Sunday at 12PM, but you can change it to the frequency that most suits your needs. We advise that you schedule your increments to run after you have updated your database with new information. Here are some examples of cron expressions:

  • Update index every 5 minutes: "incrementalIndexingCron": "*/5 * * * *"
  • Update index every 30 minutes: "incrementalIndexingCron": "*/30 * * * *"
  • Update index every hour: "incrementalIndexingCron": "0 * * * *"
  • Update index every day at 3PM: "incrementalIndexingCron": "00 15 * * *"
  • Update index every 2 weeks (on the 1st and 15th): "incrementalIndexingCron": "00 00 1,15 * *"
  • Update index every month: "incrementalIndexingCron": "00 00 1 * *"

If you need to index your data-source at non-regular intervals:

  • you can trigger it manually using the "Update index" button in the data-source configuration page.
  • you can trigger it automatically using the update index API endpoint.

Indexing by clicking the "Update index" button (or using the equivalent update index API endpoint) only performs an incremental update of the index. Note that you can also rebuild your index from scratch by clicking the "Re-create index" button. It might be useful if the index is inconsistent with the data-source schema (because the latter has changed).

You can disable automatic indexing with the following configuration: "incrementalIndexingCron": "none"