Query Domino data and faceted search with Domino JNA (part 1): setup, sorting and pagination

Every web application I build includes lists (or as Domino people like to call them: views) in one form of another. And then users start asking question like “But can it do sorting?” “How about a search function (on just these fields?)” “And filtering?”. With <insert-you-favorite-framework-here> that’s not too hard on short lists. I’m an Angular guy and can easily give them sorting and filtering.

But then the data grows. And you need to find a solution for paging. Or build an infinite scroll. So the fun starts: how do I get paged/ sorted/ filtered data from my Domino database? My favorite goto place for this always used to be the XPages REST controls. But they have limitations. Luckily I recently discovered an alternative: the Domino JNA project by Karsten Lehmann.

In short: it gives you an API you can use to access Domino data in ways you never did before. It does that by surfacing low-level Domino APIs that you normally don’t have access to. You’ll be amazed at all the gems hidden away in the product. One of the things I particularly like about the Domino JNA API is being able to work with Domino view data, perform JOIN-like queries  (yes, you read that right), and use the built-in sorting and pagination functions. That allowed me to build a faceted search. How I did that is the topic of this series of articles.

Note: the demos for this post are created using XPages as the front end. Everything that I write here will also benefit you when you’re creating a REST API on Domino. I might even do a follow-up article describing that.

In this first post I cover the basics: installing the plugin, setup a structure to read view data and create a first basic example of showing a sorted list of entries (from a view) using JNA.

Installation of JNA

Download the latest version of Domino JNA from the Github repo. Domino JNA is delivered as an OSGi plugin and needs to be installed on both the server and client. Restart the server and client and you’re good to go.

In the application properties of the database you want to use Domino JNA in, you need to enable it on the Page Generation tab:

Fake data database with 50,000 contacts

For testing I created a database with 50.000 contacts. The fake contacts were created by the Fake Name Generator and imported into an NSF that can be downloaded here. The NSF just contains the data and a view.

A basic list with pagination

Lets start with the basics: accessing a view and show the entries in a list. The demo can be viewed here (source code on Github) and consists of an XPage with a data table and a Java (controller) class to drive the backend. In the controller class you’ll find this:

 

Let’s walk through the code.

Opening a view (in JNA called a ‘collection’) is done similar as with the standard API: open the database and then the view/collection (lines 4 and 5).

Once you have the collection, you start to read entries from it. That’s done a little different then what you’re used to. You start of by specifying how to ‘walk’ the view entries. The simplest form is just move from one entry to the next using Navigate.NEXT (line 11). JNA also offers alternatives like moving to the next category or next ‘unread’ entry. More on that in another post.

Normally you retrieve a ViewEntry from a view (or view navigator) that includes all column data as well as metadata like the Note ID. You can’t control what data to read or ignore.  With JNA you can specify exactly what data to retrieve for every column. That allows you to code for performance: if you don’t need the column data, you just don’t read it. In the controller class in line 15 you can see that I’m reading the Note ID and all column values (ReadMask.SUMMARYVALUES) for each entry.

The API call on lines 21 to 24 is where the entries are actually read from the view. The skipEntries parameter allows me to start reading from a certain location in the view (useful for pagination or an infinite scrolling list) and with the NUM_PER_PAGE paramater I’m telling the API the number of entries to read in one request. The final parameter is the callback function that specifies what to do with every entry (a NotesViewEntryData object) retrieved from the view. In the example I’m using the built-in EntriesAsListCallback that returns a Java list containing the entries from the view. You can also write you own callback function to convert every entry in (for example) your own Java models.

Since I’m using XPages for the demo, I need the controller class to be serializable. The list of NotesViewEntryData object isn’t serializable, so in the last lines I’m converting the 15 entries read from the view to a list of Java maps, containing only the primitive values from the columns.

Pagination

If you look at the full code of the ListController class, you’ll see that I’m keeping track of a ‘skipEntries‘ variable. If a user navigates to the next page of data, I update this variable (incrementing it by the number of entries per page) and by adding that variable to the getAllEntries() method call I start reading at the new index. That, combined with the number of entries shown on every page allow me to add a pager.

Sorting

To enable sorting on the dataset, you need to make a change to the view design. For every column that should be sortable you enable the option “Click on column header to sort”. With that in place you can change the sort order of the view data with this API call:

Combine that with a click handler on the view columns and two variables in the controller class to store the current sort order and direction and this is what you get.

Finally

This article describes how you can start using Domino JNA to work view Domino view data. In the next article I’ll show you how you can use the built-in entry selection methods to filter the data (based on search results in the current or a different view), while maintaining the sort order and pagination functions.

(UPDATE: don’t forget to read part 2 and part 3 of this Domino JNA series)

Convenience at a cost: comparing Domino Java APIs performance (standard, ODA, JNA)

I’m into performance at the moment, trying to solve some nasty issues in one of the applications I work on. While researching the topic I came across Karsten Lehmann’s Domino JNA project that allows you to use some low-level C-API methods using Java. It contains functions that are very useful in my scenario, but I also wondered how the library performed. So I wrote some basic tests, comparing JNA with the ‘standard’ (or ‘legacy’) Domino API and, while I was at it, the org.openntf.domino (ODA) API.

So I wrote a couple of tests using the 3 APIs that traverse a view from the well known ‘Fakenames’ database. It is based on the Domino Name & Address book template and contains 40,000 documents. The tests will loop through a view called ‘People’ containing all the documents and will read a value from one of the columns.

At this point I will be taking bets:

What do you think is the fastest?

View Results

Loading ... Loading ...

Time to see for yourself! I created a simple application that allows you to test the different methods and view the results (latest 10 are shown only): take a look here.

The source code of the application and test code can be found here. The environment: CentOS 6 (64 bit), SSD, 1GB RAM, Domino 901FP5 (64 bit), ExtLib 17, ODA 3.2.1, JNA 0.9.5. Please let me know if I made any errors in the tests.

I don’t know what about you, but the results surprised me! I didn’t expect the overhead of the non-standard APIs to be that big. In every day use I guess this won’t affect your applications a lot, but it’s something you definitely have to be aware of if you needs to squeeze just a bit more performance out of your application.

UPDATE

– Upgraded the JNA project to 0.9.5
– Based on Andy Cunliffe’s comment I’ve added a new test using a ‘manual’ Java loop in the ODA:

ViewEntry ve = nav.getFirst();
while (null != ve) {

// code here

ve = nav.getNext();
}

That code seems to run about 30% faster that a standard Java loop in ODA:

for (ViewEntry ve : nav) {

// code here

}

Escaping the yellow bubble at ICON UK

IMAG1819
Last Friday I was in Londen for what was yet again an excellent ICON UK. Tim Clark really did a great job in organising it and I hope he can now relax again. Kudos also to Tony Holder for organising the speaker dinner at Wilton’s. Didn’t know you can eat that well in the UK!

The venue (IBM Client Centre) added a nice touch to the conference and was conveniently located at the center of Londen. That allowed me to try out a Boris Bike and see what cycling on the left side of the road is like. Straight sections didn’t cause me any problems, but sweat ran down my back at every crossing. I’m glad I’m still alive to write this. But hey: no guts, no glory!

I was once again allowed to host a session and decided this time to NOT do it about XPages or any other IBM technology, but share my experiences with an alternative web development stack called MEAN (that’s MongoDb, Express, AngularJS and Node). Thanks to anyone who was in the room for attending. For all of you who weren’t, here’s the deck: