Documentation

Running the service

The Letarette service depends on the NATS high performance messaging system. All service components connect to and communicate via a NATS server, running somewhere on the service network.

For a basic setup, it is enough to just launch a NATS server instance and point the Letarette service and clients to it. For more advanced NATS setups, please check the NATS documentation.

With a NATS server up and running, we can now launch the Letarette service. The default settings are probably enough to get started, but there are a few common settings for customizing the service to your needs.

Common settings

The Letarette service tries to be a nice 12-factor app, and is configured by environment variables. These are the most common settings you might want to change:

  • LETARETTE_DB_PATH

    Path to the SQLite database file for the index. By default, Letarette uses the file letarette.db in the current directory.

  • LETARETTE_INDEX_SPACES

    A comma-separated list of the spaces that the service should separate the index into. By default, the service starts up with a single docs space.

  • LETARETTE_STEMMER_LANGUAGES

    A comma-separated list of language algorithms for the stemmer. If more than one stemmer algorithm is set, the stemmer will try them one at a time until one succeeds. This selection is done word by word. The default stemmer language is english.

  • LETARETTE_NATS_URLS

    The default NATS URL list is nats://localhost:4222. If needed, set this to the list of initial NATS servers to connect to.

  • LOG_LEVEL

    The service logs to standard output at default level INFO. Set to DEBUG, INFO, WARN or ERROR for less or more detailed logging.

For example, launching a Letarette service for indexing Scandinavian music artists and tracks might look like this:

export LOG_LEVEL=DEBUG
export LETARETTE_NATS_URL=nats://my-nats-server:4222
export LETARETTE_DB_PATH=/var/letarette/music.db
export LETARETTE_INDEX_SPACES=artists,tracks
export LETARETTE_STEMMER_LANGUAGES=swedish,norwegian,danish
./letarette

Load balancing and sharding

If your index grows so big that a single search request becomes too slow, you can split your index into shards. To split an index in three, you need at least three service instances, covering shards 1/3, 2/3 and 3/3. Each service instance serves one shard only.

The set of shards with the same denominator defines a shard group. The default shard group is /1 — effectively a non-sharded index.

Multiple instances for each shard can be launched for redundance, or to handle higher loads. When there are more than one service running for the same shard, search requests will be randomly distributed between them.

The client libraries handle all interaction with the search cluster, regardless of setup. Nothing needs to be changed on the client side for different sharding or redundancy configurations.

To turn the example above to a setup with two shards, start two instances of the service with the same settings, except for the LETARETTE_SHARD setting, which should be set to 1/2 for the first instance and 2/2 for the second.

Shard initialization

When the Letarette service launches, it monitors the cluster to find the current configuration. If the service finds that it is behind other service instances, it will try to rebuild its index by cloning them, instead of fetching all documents from the Document Manager.

This reduces Document Manager load and cuts startup time for new service instances. Note that cloning works in setups with mixed shard group sizes, which simplifies transitions between different sharded setups without downtime.

Authentication and encryption

Authentication in Letarette means NATS authentication.

Letarette supports specifying username and password or token as part of the NATS URL: nats://user:pass@server:port.

Nkey - based authentication is specified by providing a path to a seed file using the LETARETTE_NATS_SEEDFILE variable.

Encryption is mostly set up on the NATS server side. If additional root CA files are needed to verify the server certificate, they can be specified using the LETARETTE_NATS_ROOTCAS variable.

Service settings reference

Giving any command-line argument to letarette will print a description of all common settings and their compile-time default values. The list of settings can also be printed using lrcli env. Adding the -v flag to lrcli env will list advanced settings as well.

All durations can be specified with common time unit suffixes, such as 1ms or 20s.

You probably don't need to change or know about any of the advanced settings.

Service settings reference