All versions of this manual
X
Welcome to Linkurious administration manual v3.0.14. Use the menu to access other documentations.
  1. Introduction
  2. FAQ
  3. Getting started
    1. Technical requirements
    2. Downloading
    3. Installing
    4. Starting Linkurious
    5. Stopping Linkurious
    6. Configure Linkurious
    7. Release notes
    8. Graph DB Feature Map
  4. Troubleshooting
    1. Checking Linkurious status
    2. Reading the logs
    3. Getting support
    4. Managing licenses
    5. Vulnerabilities
  5. Importing graph data
    1. Neo4j
    2. Cosmos db
  6. Updating Linkurious
    1. Checking for updates
    2. Backing up your data
    3. Update procedure
  7. Configuring data-sources
    1. Neo4j
    2. Cosmos DB
    3. Alternative IDs
    4. Merging data-sources
    5. Advanced settings
  8. Search index
    1. Search Option Comparison
    2. Neo4j
    3. Neo4j to Elasticsearch
    4. Azure search
    5. Configuring Elasticsearch
    6. Using Elasticsearch on AWS
    7. Incremental Indexing
    8. Numerical/Date search
  9. User-data store
    1. Migrating to an external database
    2. Backing-up your store
  10. Web server
  11. Authentication
    1. Getting started
    2. LDAP / Active Directory
    3. SSO with Azure AD
    4. SSO with Google
    5. SSO with OpenID Connect
    6. SSO with SAML2 / ADFS
    7. Configuration
  12. Access control
    1. Managing users
    2. Managing groups
    3. Standard access-rights
    4. Property-key access-rights
  13. Guest mode
    1. Guest mode
  14. Data Schema
    1. Property types
    2. Hiding data
    3. Strict mode
  15. Alerts
    1. Configuration
  16. Visualizations appearance
    1. Default styles
    2. Default captions
    3. Default properties order
    4. Geographic tiles
  17. Audit trail
    1. Log format
  18. Plugins
 

Search index: Incremental Indexing

Incremental indexing allows you to keep in sync your Elasticsearch index and your Neo4j graph database.

Linkurious Enterprise will index at a regular interval new and updated items from your database. This way, you avoid a complete reindex every time you update you data.

This is achieved by keeping track of a timestamp on every node and edge in the database, hereby allowing the indexer to only consider the nodes with newer timestamps and consequently reducing the indexing time.

You should consider this option if your database holds a significant number of items and needs to be updated frequently.

Requirements

1. APOC

Linkurious Enterprise relies on APOC triggers to ensure that every node and edge created or updated has a timestamp.

You need to make sure that you have installed APOC correctly and enabled APOC triggers. You can find all the information you need to install APOC from Neo4j documentation.

You can quickly verify that you have installed everything correctly by executing the following command from your Neo4j browser:

CALL apoc.trigger.list()

2. A property name

Your need to carefully choose the property that will hold the timestamp on every node/edge of your database. The consequence of this choice is that Linkurious Enterprise will create triggers that will store a timestamp on this property for all new and updated nodes/edge.

This means that any information stored on that property will be overwritten by the trigger

Enabling incremental indexing with Elasticsearch

After you have installed and configured APOC, you can enable incremental indexing from the data-source configuration page or by editing the configuration file with the following options:

  • incrementalIndexation You can enable incremental indexing by switching this option to true.
  • timestampPropertyName You must then provide the timestamp property name to keep track of during incremental indexing. E.g: “lastEditTimestamp”.

After you have enabled incremental indexing for the first time, Linkurious Enterprise requires to perform a complete re-indexing of the data-source to ensure that every item has been indexed. You simply need to click on the "Start indexing" button to complete the configuration.

Linkurious Enterprise will index your data-source incrementally from that point forward using the timestamps generated by the APOC triggers.

Scheduling incremental indexing

Once you have set up the incremental indexation, you can configure the frequency at which it will be triggered. You can customize this schedule by adding a cron expression to your Elasticsearch configuration.

You can make this change from the configuration file located at linkurious/data/config/production.json under datasources.index for each dataSource.

By default, we have set all incremental indexing to be launched every Sunday at 12PM, but you can change it to the frequency that most suits your needs. We advise that you schedule your increments to run after you have updated your database with new information. Here are some examples of cron expressions:

  • Update index every 5 minutes: "incrementalIndexationCron": "*/5 * * * *"
  • Update index every 30 minutes: "incrementalIndexationCron": "*/30 * * * *"
  • Update index every hour: "incrementalIndexationCron": "0 * * * *"
  • Update index every day at 3PM: "incrementalIndexationCron": "00 15 * * *"
  • Update index every 2 weeks (on the 1st and 15th): "incrementalIndexationCron": "00 00 1,15 * *"
  • Update index every month: "incrementalIndexationCron": "00 00 1 * *"

If you need to index your data-source on a non regular interval, you can trigger it manually with the “Start indexation” button in the data-source configuration page or trigger it automatically with the start indexation api.

You can disable automatic indexing with the following configuration: "incrementalIndexationCron": "none"

Limitations

Here some important considerations when choosing incremental indexing.

Incremental indexing is only available on Neo4j v3.5.1 and above.