Documentation

Integrating with Letarette

There are two parts to connecting your application to Letarette: searching and feeding the index. The search side is maybe the most straightforward — you obviously need to send queries to Letarette and present the results. But to have anything to search for, you also need to feed your documents to the index.

Searching

To integrate Letarette search functionality into your application, you use a Search Agent from one of the client libraries. The Search Agent connects to NATS, sends search requests to the Letarette service and collects the results.

A minimal Go example:

agent, err := client.NewSearchAgent([]string{"nats://localhost:4222"})
if err != nil {
    fmt.Printf("NATS connection failed: %v", err)
    return
}
defer agent.Close()

spaces := []string{"haystack"}
pageLimit := 10
pageOffset := 0

res, err := agent.Search("needle", spaces, pageLimit, pageOffset)
if err != nil {
    fmt.Printf("Search request failed: %v", err)
    return
}

for _, hit := range res.Result.Hits {
    fmt.Println(hit.Snippet)
}

Feeding the index

To get your documents into Letarette, you need a Document Manager. The Letarette service talks to the Document Manager to update its index to changes in the primary storage.

Connecting a SQL database to Letarette

If your primary storage is a SQL database, the easiest way to connect to Letarette is to use the SQL Document Manager, letarette.sql.

With letarette.sql, you only need to provide two queries and a database connection string to get up and running.

With the two queries in the default indexrequest.sql and documentrequest.sql files, the Document Manager is launched like this:

export LRSQL_DB_DRIVER="postgres"
export LRSQL_DB_CONNECTION="postgres://user:password@localhost/testdb"
./lrsql

letarette.sql currently supports PostgreSQL, MySQL, SQLite and SQL Server.

Check out letarette.sql for the structure of the queries and a complete example.

Note that you still need to use a client library (or the lrcli utility) to query the index.

Connecting using a client library

If your primary storage is something else than a SQL database, or if you need more control of how to transform your documents for Letarette, each client library provides a Document Manager implementation for that purpose.

Similarly to the letarette.sql case, there are two callback functions that needs to be provided for the Document Manager to handle index and document requests.

If you want to jump right into a full example, check out the letarette.js example project!

Index Request Handler

This handler responds to index update requests:

type IndexUpdateRequest struct {
    Space         string
    FromTime      time.Time
    AfterDocument DocumentID
    Limit         uint16
}

An index update request asks for a list of documents updated after a given document, the AfterDocument. That document is the current update position of the requesting index.

The handler has to take these two cases into account:

• The AfterDocument is unchanged in primary storage since it was last sent to the index

  In this case, the handler responds with the IDs of the oldest documents that come after the given document.

• The AfterDocument has been updated since it was sent to the index

  In this case, the handler responds with the IDs of the oldest documents that are updated at or after FromTime.

Handlers must follow a strict ordering of updated timestamp primarily and document ID secondarily to make sure that no document updates are missed.

In other words, documents that have the exact same updated timestamp are sorted by ID.

In both cases, at most Limit document IDs are returned.

Document Request Handler

When the Letarette service has received an index update response, it will start to request the corresponding documents by sending document requests:

type DocumentRequest struct {
    Space  string
    Wanted []DocumentID
}

The handler responds to these requests by finding the requested documents and returning them in the shape of Letarette documents:

type Document struct {
    ID      DocumentID
    Updated time.Time
    Title   string
    Text    string
    Alive   bool
}

Next steps

If you haven't already, check out letarette.sql and letarette.js for more details and examples of integrating. Or read more about how to set up the Letarette service.