Mindoo Blog - Cutting edge technologies - About Java, Lotus Notes and iPhone

  • Now on OpenNTF: Open Eclipse Update Site - based on IBM’s template but with extended functionality

    Karsten Lehmann  6 December 2013 19:16:01
    I just created a new project on OpenNTF called "Open Eclipse Update Site".

    The Open Eclipse Update Site database is based on the OpenNTF project "Eclipse Update Site (updatesite.ntf)" from IBM with additional functionality, e.g.
    • Mac Notes Client support (no SWT error messages like in the original template)
    • View action to delete selected features from the database (no need to delete all like in the original template)
    • Support for headless builds (automatic generation of update site, e.g. in a Jenkins server):
      database contains an agent called "(API)" that can be called to delete all content and import an update site from the local disk

    Now on OpenNTF: Mindoo XPages2Eclipse - Eclipse APIs for XPiNC applications!

    Karsten Lehmann  28 November 2013 19:45:49
    I just created a new project on OpenNTF: Mindoo XPages2Eclipse.
    Our toolkit, which provides extensive Eclipse APIs to XPages developers in the IBM Notes Client (XPiNC),

    is now available for free!


    Here is the project description:

    XPages2Eclipse is a language extension for XPages-development within the Lotus Notes Client

    Find out how XPages2Eclipse simplifies the development of XPages-applications for the IBM Lotus Notes Client considerably. With the help of this extensive toolkit you will be able to develop applications, which exhaust the full potential of the local client.

    XPages is the new technology of the hour for the notes/domino platform. It makes possible the development of modern and attractive applications in an up-to-date integrated development environment - the Domino Designer.

    For newcomers it is often difficult to master the XPages technology due to its extensive set of features - the change from classical Notes development with LotusScript is not to be accomplished within a couple of days.
    In addition to getting familiar with the usual Web standards like HTML, CSS and JavaScript, one also has to get to know the ui-library Dojo, as well as IBM specific additions like server side JavaScript (SSJS), themes or components from the Domino Extension Library.

    Yet, the result at the end of this steep learning curve is quite impressive:
    dynamic web user interfaces that can join data of multiple Lotus Notes databases or other data sources, if needed, or even an application for mobile devices – no traces left of the antiquated user interfaces that used to be created with classic Lotus Notes development.

    Thanks to the Lotus Notes Standard Client XPages applications can also be used locally and even offline.


    Expanding the boundaries of local XPages applications

    Unfortunately - from the perspective of developers - it is difficult or even impossible, to offer a set of features users are familiar with when executing XPages applications locally. That is, if developers stay within the boundaries of the XPages standard.
    There are hardly any standard APIs available to interact with the Lotus Notes Client or any other locally installed software.

    Missing are for example:

    • Integration of existing Notes applications: filling in Notes forms and Emails with data from XPages applications, accessing documents selected in classical Notes views, running existing LotusScript code
    • Import or export of data from IBM Lotus Symphony , supporting documents, spreadsheets and presentations
    • Executing document attachments with associated desktop-applications (for Windows, Linux and Mac OS)
    • Accessing the clipboard to store HTML, text, images or files
    • Executing long-running operations in the background, displaying their progress and cancelling the operation if necessary
    • Convenient features like file selection, including multi-selection, and folder selection
    But it doesn't have to be this way - XPages2Eclipse comes to your rescue. It enables developers to use functions of the Rich Client, without the need to become experts in either plugin- or Java development.
    Every feature of XPages2Eclipse can be used from within server side JavaScript.

    XPages2Eclipse offers something for everyone

    The requirements listed above are all met by XPages2Eclipse - and more!
    Your users will experience a real Rich Client-feeling for XPages applications within the IBM Lotus Notes Standard Client.
    As a developer you will save time and will be spared a lot of hard work to implement these functions, while enhancing consumer acceptance of your product.


    More information and sample code

    You can find lots of sample code and API documentation in the XPages2Eclipse wiki on the XPages2Eclipse website.


    New on OpenNTF: Plugin to close XPiNC applications from CSJS code / to detect Notes ID changes

    Karsten Lehmann  9 September 2013 22:06:28
    There is a new project on OpenNTF that I created a few days ago: Mindoo Xulrunner Prefs.js Management Plugin.
    It's nothing big, only a small Eclipse plugin that can be installed in the Notes Client to manage the preferences of the Xulrunner engine that renders XPages in the Notes Client (XPiNC).

    The main purpose for this is to set the property "dom.allow_scripts_to_close_windows" to false on a number of machines (the plugin can be deployed via policy). This enables XPages applications to close their own tab in client-side JavaScript (CSJS), something that is not possible by default yet (last tested version: Notes Client R9).

    But the even more interesting part, at least for Eclipse plugin developers, is that the project demonstrates how to run code before and after the password prompt of IBM Notes.
    We use the following Extension Point:

    <Image:New on OpenNTF: Plugin to close XPiNC applications from CSJS code / to detect Notes ID changes?xml version="1.0" encoding="UTF-8"?Image:New on OpenNTF: Plugin to close XPiNC applications from CSJS code / to detect Notes ID changes>
    <Image:New on OpenNTF: Plugin to close XPiNC applications from CSJS code / to detect Notes ID changes?eclipse version="3.4"?Image:New on OpenNTF: Plugin to close XPiNC applications from CSJS code / to detect Notes ID changes>
    <Image:New on OpenNTF: Plugin to close XPiNC applications from CSJS code / to detect Notes ID changespluginImage:New on OpenNTF: Plugin to close XPiNC applications from CSJS code / to detect Notes ID changes>
       <Image:New on OpenNTF: Plugin to close XPiNC applications from CSJS code / to detect Notes ID changesextension
             point="com.ibm.rcp.lifecycle.application.startBundles"Image:New on OpenNTF: Plugin to close XPiNC applications from CSJS code / to detect Notes ID changes>
          <Image:New on OpenNTF: Plugin to close XPiNC applications from CSJS code / to detect Notes ID changesapplication
                id="com.ibm.rcp.personality.framework.RCPApplication"Image:New on OpenNTF: Plugin to close XPiNC applications from CSJS code / to detect Notes ID changes>
             <Image:New on OpenNTF: Plugin to close XPiNC applications from CSJS code / to detect Notes ID changesbundle
                   id="com.mindoo.xpinc.changeprefs"Image:New on OpenNTF: Plugin to close XPiNC applications from CSJS code / to detect Notes ID changes>
             <Image:New on OpenNTF: Plugin to close XPiNC applications from CSJS code / to detect Notes ID changes/bundleImage:New on OpenNTF: Plugin to close XPiNC applications from CSJS code / to detect Notes ID changes>
          <Image:New on OpenNTF: Plugin to close XPiNC applications from CSJS code / to detect Notes ID changes/applicationImage:New on OpenNTF: Plugin to close XPiNC applications from CSJS code / to detect Notes ID changes>
       <Image:New on OpenNTF: Plugin to close XPiNC applications from CSJS code / to detect Notes ID changes/extensionImage:New on OpenNTF: Plugin to close XPiNC applications from CSJS code / to detect Notes ID changes>
    <Image:New on OpenNTF: Plugin to close XPiNC applications from CSJS code / to detect Notes ID changes/pluginImage:New on OpenNTF: Plugin to close XPiNC applications from CSJS code / to detect Notes ID changes>


    The plugin code will actually get started a bit too early for our specific use case, because the user still has to enter his password, so no Notes session is available yet to read the Notes.ini content (or even find out where the Notes.ini is located).

    So we register an ILoginContextEventListener to get notified as soon as the user has logged in:

    public void start(BundleContext bundleContext) throws Exception {
            XPiNCChangePrefsActivator.context = bundleContext;
                   
            //we register a ILoginContextEventListener to get notified when the
            //user has logged into the platform
            ILoginContextService service = SecurePlatform.getLoginContext();
            service.addListener(new ILoginContextEventListener() {
                    boolean hasTweakedPrefs = false;
                           
                    public void handleLogin(LoginContextEvent event) {
                    if (event.type == LoginContextEvent.MethodEnd && !event.hasException) {

                            synchronized (XPiNCChangePrefsActivator.class) {
                                    if (!hasTweakedPrefs) {
                                            //we use a flag here, because the
                                            //method is called twice
                                            XPiNCPrefs.tweakXulrunnerPrefs();
                                            hasTweakedPrefs=true;
                                    }
                            }
                    }
                }

                public void handleLogout(LoginContextEvent event) {  }
            });
    }


    This technique can also be used to detect Notes ID changes and is inspired by a blog article of Hunter Medney.

    Please note that there is another Extension Point in the Eclipse platform to launch code on startup, which is called after the user has logged in:

    <Image:New on OpenNTF: Plugin to close XPiNC applications from CSJS code / to detect Notes ID changesextension point="org.eclipse.ui.startup"Image:New on OpenNTF: Plugin to close XPiNC applications from CSJS code / to detect Notes ID changes>
        <Image:New on OpenNTF: Plugin to close XPiNC applications from CSJS code / to detect Notes ID changesstartup class="com.mindoo.startuptest.MyStartupHandler" /Image:New on OpenNTF: Plugin to close XPiNC applications from CSJS code / to detect Notes ID changes>
    <Image:New on OpenNTF: Plugin to close XPiNC applications from CSJS code / to detect Notes ID changes/extensionImage:New on OpenNTF: Plugin to close XPiNC applications from CSJS code / to detect Notes ID changes>

    But the Extension Point we used for the plugin seems to be called a bit earlier, which reduces the risk that any sidebar panel or open tab that uses Xulrunner is opened before, which would prevent us from changing the prefs.js content permanently.

    New on OpenNTF: Geospatial indexing for IBM Notes/Domino data

    Karsten Lehmann  30 July 2013 23:11:37
    Last weekend I created a new project on OpenNTF.org, which is part of a pretty big "pet project" that I have been working on for several month and that will hopefully be ready for primetime someday.

    My original plan was to submit this pet project for the last XPages development contest, either in addition to or instead of the Mindoo FTP Server, but the project got bigger and bigger over time - and an FTP server was finally easier to polish and explain than my other idea.

    This idea has to do with alternative indexing techniques for IBM Notes/Domino data, something like "Notes Views on steroids":
    Building an external indexer for IBM Notes/Domino that is more powerful than classic Notes Views, but still easy to use and scalable for large amounts of data.

    And while I was investigating different open source indexers and database engines, I once again came across the topic "Geospatial Indexing", which I had already discussed in the article XPages series #14: Using MongoDB’s geo-spatial indexing in XPages apps

    Geospatial indexing basically solves the task to find locations stored in a database that are close to a given set of coordinates, specified as latitude/longitude pair and to sort the results by distance.
    With all those smartphones out there that carry a GPS chip, the requirement nowadays is pretty often to "find the next Italian restaurant" or "find friends nearby" that all can be solved with Geospatial Indexing.

    In my XPages series article I demonstrated how to use an external MongoDB database to do these kind of searches from XPages applications, but this stuff gets even more interesting and realistic if we can solve it with pure Notes/Domino technologies - and it is possible.

    There are a few obvious ways how Geospatial searches could be implemented with Notes/Domino APIs, e.g. Database.search(String), fulltext searching or just manually scanning through all view entries to find the relevant documents.
    The main problem is, that they either do not scale very well, because all documents of a database have to be scanned or they require the creation of a fulltext index, which I personally try to avoid for this kind of lookups (takes a lot of disk space, is often not up to date, sometimes issues with date searches, when Domino thinks a field is not a date/time, but a text).

    The solution: Geohashes

    After a few hours of searching, I found a document that explains how MongoDB has implemented Geospatial Indexes.
    They convert latitude/longitude pairs to a single string value, a so called Geohash.

    This way, a single prefix lookup is enough to search for both values. All you have to do is to compute the list of Geohash boxes that intersect the search area and find view entries that start with the right Geohash prefix:

    Image:New on OpenNTF: Geospatial indexing for IBM Notes/Domino data
    (screenshot taken from the Geohash demonstrator website)


    Mindoo Geohash Demo

    The new project on OpenNTF that demonstrates the Geohash technique is called "Mindoo Geohash Demo" and it looks like this:

    Image:New on OpenNTF: Geospatial indexing for IBM Notes/Domino data


    Project description

    The sample database can be used to store and search real-world locations. A location document consists of a name, a type (e.g. "Restaurant" or "Supermarket"), address information with street/zip/city/country and a field for other custom data.

    When entered via the web interface, we use the Google Geocoding API  to retrieve geo coordinates (latitude/longitude) for the address.
    These coordinates are stored alongside the other location data in the database.
    Location documents can also be created via a REST API call.

    Image:New on OpenNTF: Geospatial indexing for IBM Notes/Domino data

    The database also provides search functionality via web UI and REST API to quickly find the nearest locations for a given point (either entered as address or latitude/longitude pair), sorted in ascending distance.

    To get started, simply sign the database, copy it to your IBM Domino R9 server and open it in a browser.
    The database contains a sample dataset (all Starbucks stores in New York and Berlin, all Apple Stores in Germany) as a starting point, but this data can be deleted to start from scratch.
    To search for locations, enter an address (e.g. "Brandenburger Tor, Berlin, Germany") and the maximum distance in meters (e.g. 1000) in the search form and click the search button.

    You can further restrict the result set by specifying a location type (e.g. "Coffee"). Just select a type and leave the address field empty to see all locations with that type in the database.

    Image:New on OpenNTF: Geospatial indexing for IBM Notes/Domino data

    For a visual representation of the search results, select up to 25 rows in the result list and they will get displayed via the Google Maps API.

    Hope you like the demo! All code and required libraries are available under Apache 2.0 license.

    XSS security fix in Domino R9 HTTP server may break existing web applications

    Karsten Lehmann  3 June 2013 21:56:04
    Last week we noticed that two of our web applications did not work as expected after upgrading our servers to Domino R9.
    We tracked down the issue and found the problem: In one REST API call, we have a query string parameter that contains a Domino fulltext query to filter the entries of a Notes view.
    Domino now reported that the query syntax was wrong. The same code had worked in 8.5.3.

    The reason is that the Domino R9 HTTP server contains a security fix to prevent applications from being vulnerable to cross site scripting attacks (XSS).
    IBM picked the brute force solution here: All occurences of "<" and ">" in the url automatically get replaced. "<" becomes "-lt" and ">" becomes "-gt".

    Let's take the following simple XPage as an example:

    <Image:XSS security fix in Domino R9 HTTP server may break existing web applications?xml version="1.0" encoding="UTF-8"?Image:XSS security fix in Domino R9 HTTP server may break existing web applications>
    <Image:XSS security fix in Domino R9 HTTP server may break existing web applicationsxp:view xmlns:xp="http://www.ibm.com/xsp/core"Image:XSS security fix in Domino R9 HTTP server may break existing web applications>
    Content of query parameter param1:<Image:XSS security fix in Domino R9 HTTP server may break existing web applicationsxp:brImage:XSS security fix in Domino R9 HTTP server may break existing web applications><Image:XSS security fix in Domino R9 HTTP server may break existing web applications/xp:brImage:XSS security fix in Domino R9 HTTP server may break existing web applications>
            <Image:XSS security fix in Domino R9 HTTP server may break existing web applicationsxp:inputTextarea id="inputTextarea1"
                    style="width:600.0px;height:200.0px" value="#{javascript:param.param1}"Image:XSS security fix in Domino R9 HTTP server may break existing web applications>
            <Image:XSS security fix in Domino R9 HTTP server may break existing web applications/xp:inputTextareaImage:XSS security fix in Domino R9 HTTP server may break existing web applications>
    <Image:XSS security fix in Domino R9 HTTP server may break existing web applications/xp:viewImage:XSS security fix in Domino R9 HTTP server may break existing web applications>


    When I call this XPage with a URL like

    http://localhost/urltest.nsf/params.xsp?param1=[date]%3E%3D01.01.2008%20AND%20[date]%3C%3D31.12.2008

    we get the following result in Domino 8.5.3:

    Content of query parameter param1:
    [date]>=01.01.2008 AND [date]<=31.12.2008


    With Domino R9, we get this instead:

    Content of query parameter param1:
    [date]-gt=01.01.2008 AND [date]-lt=31.12.2008


    You can see that the operators "<" and ">" got replaced and the ft query is no longer valid.

    The big surprise here was that the Domino server even replaces these characters if they are correctly escaped as hex codes like %3C and %3E.
    This way, IBM wants to prevent web developers from writing the query string content as part of a HTML page without properly escaping "dangerous" characters, e.g. to tell the user that a passed search query

    "<Image:XSS security fix in Domino R9 HTTP server may break existing web applicationsscript type='text/javascript'Image:XSS security fix in Domino R9 HTTP server may break existing web applications>alert('it works!');<Image:XSS security fix in Domino R9 HTTP server may break existing web applications/scriptImage:XSS security fix in Domino R9 HTTP server may break existing web applications>"

    is not understandable, which would immediately execute the script block in the browser and could cause a lot worse effects than just a simple alert box.

    And this is not a theoretical threat. It has been done before.

    Workarounds / solutions
    We asked IBM if the current implementation, which also replaces the hex encoded characters, is working as designed and they confirmed. They said they had seen too many XSS attacks in different areas of the product and customer applications, so they picked the "big hammer" as a solution. It's unlikely that this behavior will change anytime soon, but if somebody has a better idea, they are open for discussion.

    Here are a few things that you might try to make your application work again, if you've got the same issues:

    1. Change the URL parameters
    In our sample, we could change the query parameters so that we only pass the min and max dates as query string arguments. The whole FT query can then get computed on the server side. Depending on the kind of query string parameter, replacing "-lt" and "-gt" with the correct values also could be an option. But this might lead into trouble if someone wants to send "-lt" or "-gt" as part of a query string parameter on purpose.

    2. Use POST instead of GET
    If you put the ft query in the payload of a POST request, the parameters do not get replaced.

    3. Disable the XSS fix (not recommended)
    IBM dev told us that the XSS security fix can be disabled by setting the Notes.ini variable

    HTTP_QUERY_STRING_SCRUB=0

    Of course, then the server will be more vulnerable to XSS attacks. So use it at your own risk and try to find a better solution.

    Entwicklercamp 2013 slides about "Dojo 1.8 and AMD" now available in English

    Karsten Lehmann  24 May 2013 20:48:25
    My slides about "Dojo 1.8 and AMD" are now available in English. I have updated my original blog article:


    My EntwicklerCamp 2013 slides: Dojo 1.8 and AMD

    Karsten Lehmann  13 March 2013 14:53:59
    Here are the slides for my EntwicklerCamp 2013 session about "Dojo 1.8 and AMD" in English:

    Image:My EntwicklerCamp 2013 slides: Dojo 1.8 and AMD


    and in German

    Image:My EntwicklerCamp 2013 slides: Dojo 1.8 and AMD


    Download archive with both versions:

    Slides as ZIP-Archive

    Quick tip: Fixing Dojo drag and drop issues in a Windows 8 VMWare on the Mac

    Karsten Lehmann  7 March 2013 07:33:56
    I am currently setting up a new dev environment with Windows 8 and Notes/Domino 9 to work on demos for my Dojo 1.8/AMD session at Entwicklercamp next week.

    To my surprise, I noticed yesterday, that drag and drop operations on Dojo widgets did not work as expected. For example, I could not drag the splitters of a BorderContainer layout widget and the columns of a LazyTreeGrid could not get resized.
    It seemed as if mouse events got lost, but I only got that effect in Firefox and Chrome. In IE, everything was working fine.

    After a bit of Googling, I found out that VMWare Fusion emulates some kind of touch device for Windows 8 and that this can get disabled by setting

    touchscreen.vusb.present = "FALSE"

    in the VMX file of the VMWare image. I tried it and it helped, drag and drop is now working again.

    Download links for IBM Connect 2013 session slides

    Karsten Lehmann  31 January 2013 09:12:53
    As in previous years, I copied the download URLs of all the already available session slides from the socialbizonline.com website.
    Unfortunately, a lot of slides are still missing and some are only provided in black and white mode with 2-on-1 page.

    Use your preferred download manager to download the files. I use DownThemAll for this purpose. You need to be logged in to the website to access the files.

    Here are the download links:

    Connect2013_PDFs.html

    And here is the spreadsheet I used to format the links:

    Connect2013_PDFs.ods

    Have fun with it on your flight back home!


    Fast Notes view reading via Java API:New ViewNavigator cache methods in 8.5.3

    Karsten Lehmann  17 January 2013 08:53:08
    Preface

    A posting about new API methods in 8.5.3 may look a bit weird, now that the Notes/Domino R9 beta is already out for one month. I wanted to blog about this topic for one year now, actually since last year's Lotusphere conference, where the new API methods got presented by IBM (in session "AD112 What's new in the IBM Lotus Domino Objects: Version 8.5.3 in Demos"), but could not find the time and have always expected that IBM dev would write a wiki article about it - which they haven't so far.

    Let's get started!


    This blog article is about getting better view reading performance for both Java and LotusScript based applications. As discussed back in 2010 in two blog posts

    Fast Notes view reading via C API: A comparison C vs. Notes.jar
    Fast Notes view reading via C API Part 2: How to read views 99.7% faster than Notes.jar

    it is possible to get an extreme performance boost traversing view data by using raw C API methods instead of Java/LotusScript, which both bypass ECL checks that cost performance and in addition they are able to fetch multiple view entries at once (in one server transaction) instead of one at a time.

    Not everyone likes developing C code and compiling it for each platform an application should run on. So IBM decided to add a few methods to the 8.5.2 Java API so that everyone can get better performance.

    Most of the improvements have been documented in the Designer wiki article Fast Retrieval of View Data Using the ViewNavigator Cache - V8.52.
    Improvements have been made in the ViewNavigator class so that the methods ViewNavigator.getNext() and ViewNavigator.getPrev() can use an internal cache to preload up to 400 view entries.

    All you have to do is:

    //decide how many entries should be preloaded
    int numCacheEntries=400;

    //get a view from a database
    View lookupView=db.getView("myview");

    //disable auto updating
    lookupView.setAutoUpdate(false);

    //get a ViewNavigator instance for all the view entries
    ViewNavigator nav=lookupView.createViewNav();

    //and set the size of the preloading cache:
    nav.setBufferMaxEntries(numCacheEntries);

    //and then traverse the view:
    ViewEntry currEntry=nav.getFirst();
    while (currEntry!=null) {
           if (currEntry.isValid()) {
                   doSomethingWithTheEntry(currEntry);
           }
           ViewEntry nextEntry=nav.getNext();
           //don't forget to recycle!
           currEntry.recycle();
           currEntry=nextEntry;
    }

    nav.recycle();
    lookupView.recycle();


    This sample code reads all the entries of a view from start to end. For categorized views, entries get read as if they all were expanded, so we read parent entries first, then their child entries, next the children's child entries.

    Reading subsets of a view


    If reading the whole view is too much, you might just as well read a specific view category:

    ViewNavigator subCategoryNav=lookupView.
                            createViewNavFromCategory("topcategory\\subcategory_level1");


    or read unread documents:

    ViewNavigator unreadEntriesNav=lookupView.createViewNavFromAllUnread("Firstname Lastname/Company");

    or start reading document entries beginning with a specific letter (first column must be sorted, e.g. by lastname):

    ViewEntry firstEntry=lookupView.getEntryByKey("L", false);
    if (firstEntry!=null) {
            ViewNavigator subEntriesNav=lookupView.createViewNavFrom(firstEntry);
           
            //we use a cache size of 50; use more if you expect more results
           subEntriesNav.setBufferMaxEntries(50);

            //traverse the view:
           //...

            firstEntry.recycle();
            subEntriesNav.recycle();
    }


    Undocumented feature: ViewNavigatort.skip(int entriesToSkip)


    There is one more new method in 8.5.2 that is missing in IBM's wiki article:

    int start=50;

    //try to skip 50 entries and return the entries actually skipped
    int skippedEntries = viewNav.skip(start);

    if (skippedEntries==start) {
            //read the current entry after the skip operation
            ViewEntry currEntry=viewNav.getCurrent();

            //work with currEntry
            //...

            currEntry.recycle();
    }


    This method skips a number of entries from the current position (e.g. from the first entry) and is very fast, because view traversal is done in C and no column data is decoded. It returns the number of entries that were actually skipped.
    If skippedEntries is less than start, the view does not contain enough rows, so there is nothing relevant to read.

    The skip method is used in the XPages Extension Library to find relevant data for a view navigator based REST service: the code first skips "start" entries and then reads the next "count" entries from the view and returns their data as JSON.

    Another use case for this method is counting view entries:
    The funny thing is, that both ViewNavigator.getCount() and ViewNavigator.skip(Integer.MAX_VALUE) both the return the same count, but the latter is much faster.

    New in 8.5.3: Better performance for other traversal methods

    While 8.5.2 only provided improved performance for ViewNavigator.getNext() and ViewNavigator.getPrev(), 8.5.3 adds better performance for all the other traversal methods, e.g.

    public ViewEntry getNextSibling();
    public ViewEntry getPrevSibling();
    public ViewEntry getNextCategory();
    public ViewEntry getPrevCategory();
    public ViewEntry getNextDocument();
    public ViewEntry getPrevDocument();
    public ViewEntry getParent();
    public ViewEntry getChild();


    To activate the cache, there is a new method in ViewNavigator that defines the cache size and the cache filling strategy:

    public void setCacheGuidance(int maxEntries, int readMode);

    which replaces the previous setBufferMaxEntries method (you can still use the old method, but the new method can do the same and more).

    readMode can either be the constant

    ViewNavigator.VN_CACHEGUIDANCE_READALL
    or
    ViewNavigator.VN_CACHEGUIDANCE_READSELECTIVE.

    With the first constant, the ViewNavigator fills the cache like it did in 8.5.2, that means it preloads the next maxEntries entries of the view as if a categorized view would be totally expanded.

    The second constant however fills the cache with entries based on the getter method that you call next (see examples below; you can reset the cache size and mode at any time).

    Reading category entries

    So to quickly read all categories of a view, you could do

    //decide how many entries should be preloaded
    int numCacheEntries=400;

    //get a view from a database
    View lookupView=db.getView("myview");

    //disable auto updating
    lookupView.setAutoUpdate(false);

    //get a ViewNavigator instance for all the view entries
    ViewNavigator nav=lookupView.createViewNav();

    ViewEntry currEntry=nav.getFirst();

    if (currEntry!=null) {
            if (!currEntry.isCategory()) {
                    //make sure currEntry is a category or null
                    ViewEntry firstCategory=nav.getNextCategory();
                    currEntry.recycle();
                    currEntry=firstCategory;
            }

            //set cache strategy to optimize getNextCategory() calls,
            //fills the cache with the result of 400 getNextCategory() calls
            nav.setCacheGuidance(numCacheEntries, ViewNavigator.VN_CACHEGUIDANCE_READSELECTIVE);

            while (currEntry!=null)  {
                    doSomethingWithTheCategory(currEntry);

                    //move to next category; refills the cache after 400 categories
                    ViewEntry nextEntry=nav.getNextCategory();
                    currEntry.recycle();
                    currEntry = nextEntry;
            }
    }
    nav.recycle();
    lookupView.recycle();


    Reading direct child entries

    In our web development projects, the most important use case for the new cache support in 8.5.3 is to read the direct child entries in a view.
    We often do this to populate a lazy loading JavaScript tree or tree table: When the user expands a tree entry, the next tree level gets loaded and displayed.

    This is demonstrated in the following snippet. parentEntryPos would contain the position of the expanded parent entry.

    //get a view from a database
    View lookupView=db.getView("myview");

    //disable auto updating
    lookupView.setAutoUpdate(false);

    //position of parent view entry to read direct child entries or null to load the first level
    String parentEntryPos="3.1.2";

    //get a ViewNavigator instance for all the view entries
    ViewNavigator nav=lookupView.createViewNav();

    boolean foundFirstEntry=false;
    if (parentEntryPos!=null && !"".equals(parentEntryPos)  {
            if (nav.gotoPos(parentEntryPos, '.')) {
                    if (nav.gotoFirstChild()) {
                            foundFirstEntry=true;
                    }
            }
    }
    else {
            if (nav.gotoFirst())
                    foundFirstEntry=true;
    }

    if (foundFirstEntry) {
            //get the first entry to read:
            ViewEntry currEntry=nav.getCurrent();

            //get the maximum number of siblings to find the right cache size:
            int maxNumSiblings=currEntry.getSiblingCount();

            //set cache strategy to optimize getNextSibling() calls
            //fills the cache with the result of maxNumSiblings getNextSibling() calls
            nav.setCacheGuidance(maxNumSiblings, ViewNavigator.VN_CACHEGUIDANCE_READSELECTIVE);

            if (currEntry!=null) {
                    while (currEntry!=null)  {
                            //not 100% sure that the user can see the child entries;
                            //currChildCount may contain hidden documents:
                            int currChildCount=currEntry.getChildCount();
                            boolean hasChildren=false;
                            if (currChildCount > 0)
                                    hasChildren=true;
                           
                            Read the position of the current entry in the view:
                            String currEntryPos=currEntry.getPosition('.');

                            doSomethingWithTheSibling(currEntry, currEntryPos, hasChildren);

                            //move to next sibling
                            ViewEntry nextEntry=nav.getNextSibling();
                            currEntry.recycle();
                            currEntry = nextEntry;
                    }
            }
    }

    nav.recycle();
    lookupView.recycle();


    Of course, the code can be further improved to ignore the first start entries and stop after reading count entries.

    Unfortunately, there is no method skipSiblings(int siblingsToSkip) in ViewNavigator to quickly skip sibling entries in a view level. We requested this as a feature, but things are not moving very fast in this area of the product. ;-)

    Child and sibling counts in Domino: an eye opener

    When you look at that last snippet, you might wonder if reading the sibling or child count and the entry position is a wise decision.
    Same did I at first. I thought reading count information of ViewEntries would probably take a long time, because Domino has to go through all the entries to calculate the right amount.

    The good news is:
    Reading child and sibling counts in Domino is actually very fast.

    The bad news is:
    The count returned is not exactly what you might expect. :-)

    I guess a picture is worth a thousand words:

    Image:Fast Notes view reading via Java API:New ViewNavigator cache methods in 8.5.3

    What you see is a view in Sean Cull's Fakenames database with 40.000 fake user records. I added columns for both the child count and the entry positions.
    Domino returns a child count of 8, but we can only see 6 entries and there seem to be gaps in the view: the entries 1.4 and 1.5 are missing.

    The reason is that I added reader fields to those documents, so that I'm not allowed to see them anymore.

    And since checking if a user is allowed to see a document is a pretty heavy operation of comparing reader fields, the user's roles in the database's ACL and the group documents in the server directory, Domino does not do this just to calculate a count.
    Domino simply returns a count value instead, that is stored in the view index and is only a maximum of available child/sibling entries.
    It's the amount of entries the server can see.

    Please note that reading count values is not always a cheap operation. When working with ViewEntryCollection, DocumentCollection or NoteCollection, Domino will go through all entries in many cases, which takes a lot of time.

    Troubleshooting cache usage

    If you are not sure if your cache settings are actually used, you can enable client_clock debugging via Notes.ini as described here or here (in German language).
    While traversing view entries, you should see much less RPC calls to the server when using the cache than without using it.

    Backward compatibility

    If you want to use the new cache guidance functionality, but your code needs to run on older Domino servers as well, you should not call the new method directly. Your code would not compile in older versions and throw an exception when it runs.

    Use Java reflection instead to check if setCacheGuidance actually exists:

    public static boolean setCacheGuidance(ViewNavigator viewNav, int maxEntries, int readMode) throws NotesException {
            try {
                    Method setCacheGuidance=ViewNavigator.class.getMethod("setCacheGuidance", Integer.TYPE,
                                                    Integer.TYPE);
                    setCacheGuidance.invoke(viewNav, maxEntries, readMode);
                    return true;
            }
            catch (SecurityException e) {
                    //ignore, must be 8.5.2 or below
            }
            catch (NoSuchMethodException e) {
                    //ignore, must be 8.5.2 or below
            }
            catch (IllegalArgumentException e) {
                    //ignore, must be 8.5.2 or below
            }
            catch (IllegalAccessException e) {
                    //ignore, must be 8.5.2 or below
            }
            catch (InvocationTargetException e) {
                    if (e.getCause() instanceof NotesException)
                            throw (NotesException) e.getCause();
                    //ignore, must be 8.5.2 or below
            }
            return false;
    }


    ViewEntryCollection improvements?

    Talking about ViewEntryCollection, there have been plans to add cache support in this class as well. This would be useful to quickly read the results of a fulltext search (e.g. up to 5000 entries by default).
    Unfortunately, the current R9 beta does not contain anything new in this area, so there is no big hope that it will get added in R9.

    More on this topic

    If this topic is interesting for you and you attend IBM Connect 2013, mark this session in your schedule:

    AD211 - What's New In The IBM Domino Objects?
    Elizabeth Sawyer - IBM; James Cooper - IBM

    The Domino Objects for Release 9.0 debuts a major set of classes for Calendar & Scheduling as well as additional performance features for View Navigation.  Join us as we explore the powerful new capabilities coming soon to application developers through discussion and demonstration of  representative use cases.


    I'm pretty sure that they will also demonstrate something new in the R9 API, for example public Vector getColumnValues(int columnNumber) in the View class that you can find in the beta version and that should work like @DbColumn. :-)


    That's it for today. I hope this article is helpful and you can speed up your applications!