Mindoo Blog Cutting edge technologies - About Java, Lotus Notes and iPhone http://www.mindoo.de/web/blog.nsf/ en-us Wed, 13 Mar 2013 14:53:59 +0200 My EntwicklerCamp 2013 slides: Dojo 1.8 and AMD (German) Wed, 13 Mar 2013 14:53:59 +0200 http://www.mindoo.com/web/blog.nsf/dx/13.03.2013145359KLEJBB.htm Entwicklercamp 2013 Karsten Lehmann http://www.mindoo.com/web/blog.nsf/dx/13.03.2013145359KLEJBB.htm?opendocument&comments http://www.mindoo.com/web/blog.nsf/dx/13.03.2013145359KLEJBB.htm
Here are the slides for my EntwicklerCamp 2013 session about "Dojo 1.8 and AMD":




Download archive:
Slides as ZIP-Archive


To all English speaking readers: Google Translate is your friend - the slides are in German :-)

My plan is to translate them to English though and publish my demos as soon as possible. ]]> http://www.mindoo.com/web/blog.nsf/dxcomments/13.03.2013145359KLEJBB.htm http://www.mindoo.com/web/blog.nsf/dx/13.03.2013145359KLEJBB.htm?opendocument&comments Quick tip: Fixing Dojo drag and drop issues in a Windows 8 VMWare on the Mac Thu, 7 Mar 2013 07:33:56 +0200 http://www.mindoo.com/web/blog.nsf/dx/07.03.2013073356KLE9QA.htm Webdev Karsten Lehmann http://www.mindoo.com/web/blog.nsf/dx/07.03.2013073356KLE9QA.htm?opendocument&comments http://www.mindoo.com/web/blog.nsf/dx/07.03.2013073356KLE9QA.htm Dojo 1.8/AMD session at Entwicklercamp next week.

To my surprise, I noticed yesterday, that drag and drop operations on Dojo widgets did not work as expected. For example, I could not drag the splitters of a BorderContainer layout widget and the columns of a LazyTreeGrid could not get resized.
It seemed as if mouse events got lost, but I only got that effect in Firefox and Chrome. In IE, everything was working fine.

After a bit of Googling, I found out that VMWare Fusion emulates some kind of touch device for Windows 8 and that this can get disabled by setting

touchscreen.vusb.present = "FALSE"

in the VMX file of the VMWare image. I tried it and it helped, drag and drop is now working again.
]]> http://www.mindoo.com/web/blog.nsf/dxcomments/07.03.2013073356KLE9QA.htm http://www.mindoo.com/web/blog.nsf/dx/07.03.2013073356KLE9QA.htm?opendocument&comments Download links for IBM Connect 2013 session slides Thu, 31 Jan 2013 09:12:53 +0200 http://www.mindoo.com/web/blog.nsf/dx/31.01.2013091253KLEBN5.htm Lotusphere 2013 Karsten Lehmann http://www.mindoo.com/web/blog.nsf/dx/31.01.2013091253KLEBN5.htm?opendocument&comments http://www.mindoo.com/web/blog.nsf/dx/31.01.2013091253KLEBN5.htm socialbizonline.com website.
Unfortunately, a lot of slides are still missing and some are only provided in black and white mode with 2-on-1 page.

Use your preferred download manager to download the files. I use DownThemAll for this purpose. You need to be logged in to the website to access the files.

Here are the download links:

Connect2013_PDFs.html

And here is the spreadsheet I used to format the links:

Connect2013_PDFs.ods

Have fun with it on your flight back home!

]]> http://www.mindoo.com/web/blog.nsf/dxcomments/31.01.2013091253KLEBN5.htm http://www.mindoo.com/web/blog.nsf/dx/31.01.2013091253KLEBN5.htm?opendocument&comments Fast Notes view reading via Java API:New ViewNavigator cache methods in 8.5.3 Thu, 17 Jan 2013 08:53:08 +0200 http://www.mindoo.com/web/blog.nsf/dx/17.01.2013085308KLEB9S.htm Notesdev Karsten Lehmann http://www.mindoo.com/web/blog.nsf/dx/17.01.2013085308KLEB9S.htm?opendocument&comments http://www.mindoo.com/web/blog.nsf/dx/17.01.2013085308KLEB9S.htm Preface

A posting about new API methods in 8.5.3 may look a bit weird, now that the Notes/Domino R9 beta is already out for one month. I wanted to blog about this topic for one year now, actually since last year's Lotusphere conference, where the new API methods got presented by IBM (in session "AD112 What's new in the IBM Lotus Domino Objects: Version 8.5.3 in Demos"), but could not find the time and have always expected that IBM dev would write a wiki article about it - which they haven't so far.

Let's get started!

This blog article is about getting better view reading performance for both Java and LotusScript based applications. As discussed back in 2010 in two blog posts

Fast Notes view reading via C API: A comparison C vs. Notes.jar
Fast Notes view reading via C API Part 2: How to read views 99.7% faster than Notes.jar

it is possible to get an extreme performance boost traversing view data by using raw C API methods instead of Java/LotusScript, which both bypass ECL checks that cost performance and in addition they are able to fetch multiple view entries at once (in one server transaction) instead of one at a time.

Not everyone likes developing C code and compiling it for each platform an application should run on. So IBM decided to add a few methods to the 8.5.2 Java API so that everyone can get better performance.

Most of the improvements have been documented in the Designer wiki article Fast Retrieval of View Data Using the ViewNavigator Cache - V8.52.
Improvements have been made in the ViewNavigator class so that the methods ViewNavigator.getNext() and ViewNavigator.getPrev() can use an internal cache to preload up to 400 view entries.

All you have to do is:

//decide how many entries should be preloaded
int numCacheEntries=400;

//get a view from a database
View lookupView=db.getView("myview");

//disable auto updating
lookupView.setAutoUpdate(false);

//get a ViewNavigator instance for all the view entries
ViewNavigator nav=lookupView.createViewNav();

//and set the size of the preloading cache:
nav.setBufferMaxEntries(numCacheEntries);

//and then traverse the view:
ViewEntry currEntry=nav.getFirst();
while (currEntry!=null) {
       if (currEntry.isValid()) {
               doSomethingWithTheEntry(currEntry);
       }
       ViewEntry nextEntry=nav.getNext();
       //don't forget to recycle!
       currEntry.recycle();
       currEntry=nextEntry;
}

nav.recycle();
lookupView.recycle();

This sample code reads all the entries of a view from start to end. For categorized views, entries get read as if they all were expanded, so we read parent entries first, then their child entries, next the children's child entries.

Reading subsets of a view

If reading the whole view is too much, you might just as well read a specific view category:

ViewNavigator subCategoryNav=lookupView.
                        createViewNavFromCategory("topcategory\\subcategory_level1");

or read unread documents:

ViewNavigator unreadEntriesNav=lookupView.createViewNavFromAllUnread("Firstname Lastname/Company");

or start reading document entries beginning with a specific letter (first column must be sorted, e.g. by lastname):

ViewEntry firstEntry=lookupView.getEntryByKey("L", false);
if (firstEntry!=null) {
        ViewNavigator subEntriesNav=lookupView.createViewNavFrom(firstEntry);
       
        //we use a cache size of 50; use more if you expect more results
       subEntriesNav.setBufferMaxEntries(50);

        //traverse the view:
       //...

        firstEntry.recycle();
        subEntriesNav.recycle();
}

Undocumented feature: ViewNavigatort.skip(int entriesToSkip)

There is one more new method in 8.5.2 that is missing in IBM's wiki article:

int start=50;

//try to skip 50 entries and return the entries actually skipped
int skippedEntries = viewNav.skip(start);

if (skippedEntries==start) {
        //read the current entry after the skip operation
        ViewEntry currEntry=viewNav.getCurrent();

        //work with currEntry
        //...

        currEntry.recycle();
}

This method skips a number of entries from the current position (e.g. from the first entry) and is very fast, because view traversal is done in C and no column data is decoded. It returns the number of entries that were actually skipped.
If skippedEntries is less than start, the view does not contain enough rows, so there is nothing relevant to read.

The skip method is used in the XPages Extension Library to find relevant data for a view navigator based REST service: the code first skips "start" entries and then reads the next "count" entries from the view and returns their data as JSON.

Another use case for this method is counting view entries:
The funny thing is, that both ViewNavigator.getCount() and ViewNavigator.skip(Integer.MAX_VALUE) both the return the same count, but the latter is much faster.

New in 8.5.3: Better performance for other traversal methods

While 8.5.2 only provided improved performance for ViewNavigator.getNext() and ViewNavigator.getPrev(), 8.5.3 adds better performance for all the other traversal methods, e.g.

public ViewEntry getNextSibling();
public ViewEntry getPrevSibling();
public ViewEntry getNextCategory();
public ViewEntry getPrevCategory();
public ViewEntry getNextDocument();
public ViewEntry getPrevDocument();
public ViewEntry getParent();
public ViewEntry getChild();

To activate the cache, there is a new method in ViewNavigator that defines the cache size and the cache filling strategy:

public void setCacheGuidance(int maxEntries, int readMode);

which replaces the previous setBufferMaxEntries method (you can still use the old method, but the new method can do the same and more).

readMode can either be the constant

ViewNavigator.VN_CACHEGUIDANCE_READALL
or
ViewNavigator.VN_CACHEGUIDANCE_READSELECTIVE.

With the first constant, the ViewNavigator fills the cache like it did in 8.5.2, that means it preloads the next maxEntries entries of the view as if a categorized view would be totally expanded.

The second constant however fills the cache with entries based on the getter method that you call next (see examples below; you can reset the cache size and mode at any time).

Reading category entries

So to quickly read all categories of a view, you could do

//decide how many entries should be preloaded
int numCacheEntries=400;

//get a view from a database
View lookupView=db.getView("myview");

//disable auto updating
lookupView.setAutoUpdate(false);

//get a ViewNavigator instance for all the view entries
ViewNavigator nav=lookupView.createViewNav();

ViewEntry currEntry=nav.getFirst();

if (currEntry!=null) {
        if (!currEntry.isCategory()) {
                //make sure currEntry is a category or null
                ViewEntry firstCategory=nav.getNextCategory();
                currEntry.recycle();
                currEntry=firstCategory;
        }

        //set cache strategy to optimize getNextCategory() calls,
        //fills the cache with the result of 400 getNextCategory() calls
        nav.setCacheGuidance(numCacheEntries, ViewNavigator.VN_CACHEGUIDANCE_READSELECTIVE);

        while (currEntry!=null)  {
                doSomethingWithTheCategory(currEntry);

                //move to next category; refills the cache after 400 categories
                ViewEntry nextEntry=nav.getNextCategory();
                currEntry.recycle();
                currEntry = nextEntry;
        }
}
nav.recycle();
lookupView.recycle();


Reading direct child entries

In our web development projects, the most important use case for the new cache support in 8.5.3 is to read the direct child entries in a view.
We often do this to populate a lazy loading JavaScript tree or tree table: When the user expands a tree entry, the next tree level gets loaded and displayed.

This is demonstrated in the following snippet. parentEntryPos would contain the position of the expanded parent entry.

//get a view from a database
View lookupView=db.getView("myview");

//disable auto updating
lookupView.setAutoUpdate(false);

//position of parent view entry to read direct child entries or null to load the first level
String parentEntryPos="3.1.2";

//get a ViewNavigator instance for all the view entries
ViewNavigator nav=lookupView.createViewNav();

boolean foundFirstEntry=false;
if (parentEntryPos!=null && !"".equals(parentEntryPos)  {
        if (nav.gotoPos(parentEntryPos, '.')) {
                if (nav.gotoFirstChild()) {
                        foundFirstEntry=true;
                }
        }
}
else {
        if (nav.gotoFirst())
                foundFirstEntry=true;
}

if (foundFirstEntry) {
        //get the first entry to read:
        ViewEntry currEntry=nav.getCurrent();

        //get the maximum number of siblings to find the right cache size:
        int maxNumSiblings=currEntry.getSiblingCount();

        //set cache strategy to optimize getNextSibling() calls
        //fills the cache with the result of maxNumSiblings getNextSibling() calls
        nav.setCacheGuidance(maxNumSiblings, ViewNavigator.VN_CACHEGUIDANCE_READSELECTIVE);

        if (currEntry!=null) {
                while (currEntry!=null)  {
                        //not 100% sure that the user can see the child entries;
                        //currChildCount may contain hidden documents:
                        int currChildCount=currEntry.getChildCount();
                        boolean hasChildren=false;
                        if (currChildCount > 0)
                                hasChildren=true;
                       
                        Read the position of the current entry in the view:
                        String currEntryPos=currEntry.getPosition('.');

                        doSomethingWithTheSibling(currEntry, currEntryPos, hasChildren);

                        //move to next sibling
                        ViewEntry nextEntry=nav.getNextSibling();
                        currEntry.recycle();
                        currEntry = nextEntry;
                }
        }
}

nav.recycle();
lookupView.recycle();


Of course, the code can be further improved to ignore the first start entries and stop after reading count entries.

Unfortunately, there is no method skipSiblings(int siblingsToSkip) in ViewNavigator to quickly skip sibling entries in a view level. We requested this as a feature, but things are not moving very fast in this area of the product. ;-)

Child and sibling counts in Domino: an eye opener

When you look at that last snippet, you might wonder if reading the sibling or child count and the entry position is a wise decision.
Same did I at first. I thought reading count information of ViewEntries would probably take a long time, because Domino has to go through all the entries to calculate the right amount.

The good news is:
Reading child and sibling counts in Domino is actually very fast.

The bad news is:
The count returned is not exactly what you might expect. :-)

I guess a picture is worth a thousand words:

Image:Fast Notes view reading via Java API:New ViewNavigator cache methods in 8.5.3

What you see is a view in Sean Cull's Fakenames database with 40.000 fake user records. I added columns for both the child count and the entry positions.
Domino returns a child count of 8, but we can only see 6 entries and there seem to be gaps in the view: the entries 1.4 and 1.5 are missing.

The reason is that I added reader fields to those documents, so that I'm not allowed to see them anymore.

And since checking if a user is allowed to see a document is a pretty heavy operation of comparing reader fields, the user's roles in the database's ACL and the group documents in the server directory, Domino does not do this just to calculate a count.
Domino simply returns a count value instead, that is stored in the view index and is only a maximum of available child/sibling entries.
It's the amount of entries the server can see.

Please note that reading count values is not always a cheap operation. When working with ViewEntryCollection, DocumentCollection or NoteCollection, Domino will go through all entries in many cases, which takes a lot of time.

Troubleshooting cache usage

If you are not sure if your cache settings are actually used, you can enable client_clock debugging via Notes.ini as described here or here (in German language).
While traversing view entries, you should see much less RPC calls to the server when using the cache than without using it.

Backward compatibility

If you want to use the new cache guidance functionality, but your code needs to run on older Domino servers as well, you should not call the new method directly. Your code would not compile in older versions and throw an exception when it runs.

Use Java reflection instead to check if setCacheGuidance actually exists:

public static boolean setCacheGuidance(ViewNavigator viewNav, int maxEntries, int readMode) throws NotesException {
        try {
                Method setCacheGuidance=ViewNavigator.class.getMethod("setCacheGuidance", Integer.TYPE,
                                                Integer.TYPE);
                setCacheGuidance.invoke(viewNav, maxEntries, readMode);
                return true;
        }
        catch (SecurityException e) {
                //ignore, must be 8.5.2 or below
        }
        catch (NoSuchMethodException e) {
                //ignore, must be 8.5.2 or below
        }
        catch (IllegalArgumentException e) {
                //ignore, must be 8.5.2 or below
        }
        catch (IllegalAccessException e) {
                //ignore, must be 8.5.2 or below
        }
        catch (InvocationTargetException e) {
                if (e.getCause() instanceof NotesException)
                        throw (NotesException) e.getCause();
                //ignore, must be 8.5.2 or below
        }
        return false;
}

ViewEntryCollection improvements?

Talking about ViewEntryCollection, there have been plans to add cache support in this class as well. This would be useful to quickly read the results of a fulltext search (e.g. up to 5000 entries by default).
Unfortunately, the current R9 beta does not contain anything new in this area, so there is no big hope that it will get added in R9.

More on this topic

If this topic is interesting for you and you attend IBM Connect 2013, mark this session in your schedule:

AD211 - What's New In The IBM Domino Objects?
Elizabeth Sawyer - IBM; James Cooper - IBM

The Domino Objects for Release 9.0 debuts a major set of classes for Calendar & Scheduling as well as additional performance features for View Navigation.  Join us as we explore the powerful new capabilities coming soon to application developers through discussion and demonstration of  representative use cases.


I'm pretty sure that they will also demonstrate something new in the R9 API, for example public Vector getColumnValues(int columnNumber) in the View class that you can find in the beta version and that should work like @DbColumn. :-)


That's it for today. I hope this article is helpful and you can speed up your applications!

]]> http://www.mindoo.com/web/blog.nsf/dxcomments/17.01.2013085308KLEB9S.htm http://www.mindoo.com/web/blog.nsf/dx/17.01.2013085308KLEB9S.htm?opendocument&comments Our session got accepted for IBM Connect: BOF211 Leverage OSGi plug-ins in Your XPages Applications! Wed, 9 Jan 2013 08:20:41 +0200 http://www.mindoo.com/web/blog.nsf/dx/09.01.2013082041KLEAMH.htm IBM Connect Karsten Lehmann http://www.mindoo.com/web/blog.nsf/dx/09.01.2013082041KLEAMH.htm?opendocument&comments http://www.mindoo.com/web/blog.nsf/dx/09.01.2013082041KLEAMH.htm
Here is the abstract:

BOF211 Leverage OSGi plug-ins in Your XPages Applications!
Tammo Riedinger, Mindoo GmbH; Karsten Lehmann, Mindoo GmbH

Hear how XPages apps can be extended with your own visual controls and data can be leveraged from external databases! We want to talk about code sharing of libraries between multiple XPages apps, Notes Client plugins and standalone applications.We also want to discuss, how custom server processes and even complete applications can be added to IBM Lotus Domino on top of OSGI, due to its highly modular and extensible nature. The only limit is your imagination!


That's quite similar to the description that we had submitted as a normal session, which unfortunately did not get accepted by the track managers. BOF sessions however get voted by the conference attendees.

What you don't get in our BOF session compared to the normal one:
  • slides
  • demos

What you will get in our BOF session compared to the normal one:
  • discussion
  • exchange of experiences

and hopefully a small breakfast buffet with coffee in front of the room, in case we get a 7am slot for the BOF (hear us IBM? We once did a BOF with no breakfast, that was hard :-)  ).

This is how IBM describes Birds of a Feather sessions:
  • A Connect 2013 BOF is a relaxed,  interactive discussion, with no media or AV. This is a 'chalk-talk' style meeting, and the purpose of the discussion is to gather input, feedback, and learn from each other. Again, no room AV is to be used, no exceptions please.
  • The role of the facilitator is to do just that -- facilitate discussion. All in attendance should be given an opportunity to become engaged. This is not a 'one to many' session. Remember -- at Connect, "Everyone has a voice" !
  • Flip-charts and markers will be provided upon request. Any other session tools, props, etc are the responsibility of the facilitator(s).

Great to be part of the conference! See you in Orlando!
]]> http://www.mindoo.com/web/blog.nsf/dxcomments/09.01.2013082041KLEAMH.htm http://www.mindoo.com/web/blog.nsf/dx/09.01.2013082041KLEAMH.htm?opendocument&comments XPages series #15: Free FTP server on top of Domino’s OSGi framework Sat, 3 Nov 2012 14:14:02 +0200 http://www.mindoo.com/web/blog.nsf/dx/03.11.2012141402KLEHJC.htm Free Tools Karsten Lehmann http://www.mindoo.com/web/blog.nsf/dx/03.11.2012141402KLEHJC.htm?opendocument&comments http://www.mindoo.com/web/blog.nsf/dx/03.11.2012141402KLEHJC.htm Renovations company.
Those images very pretty large, with several GB's of data and the fastest and easiest way to get our hands on the images was to set up an FTP server and have the IBM'ers upload the files.

That was a good occasion to build a small proof-of-concept that I had already thought of 1-2 years ago:

An FTP server running on top of Domino

The implementation is actually quite easy: We simply take an existing FTP server and integrate it into the OSGi framework of the Domino server as a plug-in.
I selected the Apache FtpServer for this purpose, because it's available as open source with Apache License and easy to set up.

The first task was to find all necessary libraries to run the server. The FtpServer project is available via Apache's Maven repositories. Maven is a powerful build system that takes care of all code dependencies.
Although Maven can often be a real PITA, creating a build means basically just to download the source code and use the command

mvn clean install

or

mvn clean install -DskipTests

The option "-DskipTests" skips the automatic test case execution during the build process and reduces build time.

As tweeted a few months ago, there is a way to tweak Maven's pom.xml file to let it generate all required Jar files and copy them to a target directory.
You just need to add this snippet to the pom.xml file:

<plugin>
      <artifactId>maven-dependency-plugin</artifactId>
      <executions>
              <execution>
                      <phase>install</phase>
                      <goals>
                              <goal>copy-dependencies</goal>
                      </goals>
                      <configuration>
                              <outputDirectory>${project.build.directory}/lib</outputDirectory>
                      </configuration>
              </execution>
      </executions>
</plugin>

I then copied all generated Jar files into a new plug-in:

Image:XPages series #15: Free FTP server on top of Domino’s OSGi framework




and exported the relevant packages in the plug-in's manifest file:

Image:XPages series #15: Free FTP server on top of Domino’s OSGi framework



Next, I created a plug-in "com.mindoo.ftp" that uses the code from "org.apache.ftpserver" and creates the actual FTP server instance.
It reads the server configuration and user account data from a small Notes database and writes log documents into another one.

For now, the FTP server reads and writes its files from disk and not from document attachments in NSF databases, although that would be technically possible and might be an option for the future.

I decided not to use NSF for the files, because a) this is just a proof-of-concept and b) the FTP server should be able to resume uploaded and downloaded files. The standard Notes Java API does not provide any real streaming functionality (EmbeddedObject.getInputStream() internally just writes the file to disk first, which would kill performance).

Streaming of attachments out of NSF is possible via NSFNoteExtractWithCallback in the C API (we used this in the MindPlan application to let the user cancel attachment extraction) and there may be a similar method to import attachments via stream.

Server Configuration

Our configuration database looks like this - nothing fancy yet:

Image:XPages series #15: Free FTP server on top of Domino’s OSGi framework



I added fields for the most relevant server settings to the configuration form, e.g. the server's TCP/IP port, a blacklist to block IP addresses and subnets as well as parameters to use the server behind a NAT router (I tried running it from inside a VMware image).

Image:XPages series #15: Free FTP server on top of Domino’s OSGi framework



For users, we can configure their home directory, set a password (we only save a one-way hash generated by @Password) and set a few options to limit the user's access to the server, e.g. define an access period and remove write permissions.

Image:XPages series #15: Free FTP server on top of Domino’s OSGi framework


Running the server

The FTP server plugins should be deployed into the Domino server's HTTP task OSGi framework and start when the HTTP server gets started.
Unfortunately, in 8.5.3 this is not that easy. There is no such extension point that can be used to automatically start a plug-in on HTTP task startup, like org.eclipse.ui.startup for RCP applications (we have an open feature request for 8.5.4). Everything is started lazily when it is used for the first time. Bad for a server that needs to listen for incoming connections on a TCP/IP port.

However, the XPages runtime provides one extension point that we can use:

<plugin>
       <extension
       point="com.ibm.commons.Extension">
               <service
               class="com.mindoo.ftp.startup.StartupHandler"
               type="com.ibm.designer.runtime.extensions.RuntimeInitializationEvent">
               </service>
       </extension>
</plugin>

This way, our plugin gets started the first time an XPage is called. That's a first step, but we need to automate this XPage invocation somehow, to be sure that the server is running after the HTTP task or the whole server is restarted.

Fortunately, tweeting about this issue helped. Thanks to René Winkelmeyer, I could come up with a solution that leverages the XPages preload feature of Domino 8.5.3 to automatically start the FTP server plugin.
All you have to do is add the following lines to the server's Notes.ini:

XPagesPreload=1
XPagesPreloadDB=mindoo-ftp-config.nsf/ftplauncher.xsp

"ftplauncher.xsp" is a small dummy XPage in our configuration database. You can use any other XPage instead, since the purpose is just to start the OSGi framework.

DOTS

There is an alternative OSGi implementation that can run as part of the Domino server installation: DOTS, also known as Domino Tasklet Service for IBM Lotus Domino.
In constrast to the HTTP OSGi framework, DOTS contains an extension point to launch plug-ins on framework startup.

I did not choose DOTS to run the FTP server, because as of 8.5.3, DOTS is not part of the Domino server and needs to be deployed manually. In addition, by using the HTTP OSGi framework instead of DOTS, we can directly call plug-in methods from XPages applications, which might be useful for some applications.

Server console commands

I added three server console commands to control the FTP server from the server console:

tell http osgi mftp restart  -  stops and starts the server
tell http osgi mftp stop  -  stops the server
tell http osgi mftp start  -  starts the server

And of course, the FTP server gets restarted on

restart task http

which restarts the whole HTTP task.

Adding and modifying users does not require a server restart. The lists of blocked IP's and subnets in the server configurations does also get automatically re-read when somebody tries to log into the server.
All other options currently require a restart of the FTP server.

Free as in beer

This proof-of-concept is made available for free. I am sure there are other cool use cases for OSGi based background tasks in Domino. In particular, a scheduler plug-in might be a good thing.

Setup instructions

The download archive contains an update site with the plugin code that needs to be deployed onto the Domino server just like the XPages Extension Library : create an NSF update site database, import the plug-ins and features, configure the Notes.ini variable OSGI_HTTP_DYNAMIC_BUNDLES and restart the HTTP task.

The default paths for the configuration database is "mindoo-ftp-config.nsf" and "mindoo-ftp-config-log.nsf" for the log database.

An alternative configuration database path can be configured via the Notes.ini variable "$mndFTPConfig=configdb.nsf". For the log database, the variable is called "$mndFTPLog=logdb.nsf".

And finally, the base directory of the FTP server can be set via "$mndFTPBaseDir=c:\temp\mftp".

Hope this tool is useful!

Here is the download link:
mftp.zip

]]> http://www.mindoo.com/web/blog.nsf/dxcomments/03.11.2012141402KLEHJC.htm http://www.mindoo.com/web/blog.nsf/dx/03.11.2012141402KLEHJC.htm?opendocument&comments Session accepted for Entwicklercamp 2013: Dojo 1.8 and AMD Sun, 30 Sep 2012 21:28:24 +0200 http://www.mindoo.com/web/blog.nsf/dx/30.09.2012212824KLEQUC.htm Entwicklercamp 2013 Karsten Lehmann http://www.mindoo.com/web/blog.nsf/dx/30.09.2012212824KLEQUC.htm?opendocument&comments http://www.mindoo.com/web/blog.nsf/dx/30.09.2012212824KLEQUC.htm Entwicklercamp 2013 in Gelsenkirchen, Germany (11th - 13th of March) got accepted.

The session is about the new Dojo toolkit 1.8 that we have already used in a customer project since its release in August 2012.
Sessions at Entwicklercamp are held in German language, so here is the translated abstract and the original one in German.

Dojo 1.8 and AMD

The session provides an introduction to web application development with version 1.8 of the Dojo toolkit.

It covers available UI elements for web and mobile development, declarative and programmatic web ui design as well as the new AMD module system, that can be used to build highly modular web applications.

With short demos, additional tools will be presented, that can be used to impress customers and colleagues.


Dojo 1.8 und AMD

Die Session gibt einen Einstieg in die Webapplikationsentwicklung mit dem Dojo Toolkit in der Version 1.8.

Behandelt werden die zur Verfügung stehenden grafischen Elemente für Web und Mobile, deklarative und programmatische Weboberflächengestaltung sowie das neue AMD-Modulsystem, mit dem sich äußerst modulare Applikationen aufbauen lassen.

Im Rahmen von Kurzdemos werden darüber hinaus noch weitere Tools vorgestellt, mit denen man Kunden und Kollegen zum Staunen bringen kann.
]]> http://www.mindoo.com/web/blog.nsf/dxcomments/30.09.2012212824KLEQUC.htm http://www.mindoo.com/web/blog.nsf/dx/30.09.2012212824KLEQUC.htm?opendocument&comments Customer project: Dojo 1.8 based portal on top of XPages and Domino 8.5.3 Tue, 18 Sep 2012 00:13:00 +0200 http://www.mindoo.com/web/blog.nsf/dx/18.09.2012001300KLEU38.htm Webdev Karsten Lehmann http://www.mindoo.com/web/blog.nsf/dx/18.09.2012001300KLEU38.htm?opendocument&comments http://www.mindoo.com/web/blog.nsf/dx/18.09.2012001300KLEU38.htm Tweeting about my findings on the web is just so much easier than to write a complete blog article.

We spent most of our time working on web applications for desktop browsers and the iPad (including Retina support), based on our favorite web toolkits Ext.JS from Sencha as well as the Dojo toolkit.
Since we like cutting edge development, we prefer to use the latest versions of toolkits. For Sencha Touch, this means 2.0.x, Ext.JS is available in version 4.1.x and for Dojo it's the brand new version 1.8 that we already used for a portal framework development during the past 3 weeks.

Portal framework

This portal framework is a customer project, so there is not much code I can share (I will try to write blog entries about specific technologies).
But I can describe the basic ideas in this blog post and provide some links to useful toolkits that helped us building the app.

The portal leverages Dojo's dojox.layout.GridContainer (click here for a portal sample app) to build the portal UI. Portlets can have two different render modes: IFrame and inline rendering. In IFrame rendering mode, the portlet content is completely independent from the parent page; in inline rendering mode, the portlet code can directly render its content into the container node of a dojox.widget.Portlet within the portal page.

The portal page acts as a communication mediator between the portlets (both with IFrame and inline rendering). Portlets can send JSON based messages to the portal, which - depending on the message content - can send a response to the portlet (e.g. to return loaded portlet state information), forward the message to another portlet (to update another portlets state) or send out a broadcast message to all portlets in the page (e.g. to publish selection changes in a Dojo grid).

More or less, the portal acts like a Composite Application on steroids. :-)
 
Cross-domain security

While sending messages between portlets with inline rendering is quite easy, we needed to find a way to cross the IFrame boundaries in order to send data from IFrame content to the main page back and forth.
This is not that easy, because browsers prevent this kind of communication to avoid cross-domain security issues.

Fortunately, there is easyxdm, a Javascript library that contains several solutions for cross-domain communication and automatically picks the right one for the current browser.

Portlet content rendering

Loading the portlet content for IFrame rendering mode is quite easy. All you need to do is create an IFRAME node with JavaScript and set its src attribute to the remote content URL.
This way, portlet content can get generated in external databases or even on different servers (including servers that are not Domino based).

This works quite well on desktop browsers. They have a good internet connection and plenty of RAM, so it's not a problem to load newer or older Dojo versions or other toolkits as part of the IFrame content.
Scrolling content that is larger than the portlet viewport is quite easy as well. This is all handled automatically by the desktop browser.

Unfortunately, this approach did not work well on mobile devices. Scrolling content with a Swipe gesture did not work at all and we noticed rendering issues: in many cases, the IFrame content did not render properly.
That lead us to the development of inline rendering mode: The whole portal content is stored in one DOM tree and shares the portal's Dojo 1.8 framework.
This is good and bad at the same time: Page load is faster, but portlets needs to be developed in a way that they don't interfere with each other and only load required resources once (like JavaScript code or CSS files).

In addition to adding the inline rendering mode, additional tweaks had to be applied to the portal rendering in order to support drag&drop of portlets on the iPad (Dojo's GridContainer layout is pretty hardcoded to mouse events) and change the portal column width with a pinch gesture.
To drag&drop portlets, we implemented a special edit mode, comparable to the "jiggle mode" in the springboard on iOS devices: Portlets in edit mode get a CSS based jiggle animation to illustrate that they can get moved with a swipe gesture (tap on the portlet and drag it to the target location).

Dynamic classloading

To implement the inline rendering mode of portlets in a clean way, we built the whole portal framework on top of Dojo's new AMD classloader architecture. This lets us define classes, implement class inheritance and lazily load portlet code so that it gets loaded when it is required for the first time, speeding up overall page load.
By following a tutorial on the Dojo website, we could also load specific classes from different locations than the portal page itself. This enables us to separate the portal code from the actual portlet code, making the application reusable and easy to extend.

In addition, we used the Ensure library to coordinate the loading of resources that are required for the portlet rendering process: Portlets send a JSON message to the portal with a list of required HTML, JS and CSS files and the portal only loads files that have not been fetched for other portlets before.
When all resources are loaded, Ensure calls our callback function which then notifies the portlet that all resources have been loaded and content rendering can start.

Modality

Portlets can be configured to be modal. A modal portlet gets displayed in a Dojo dialog on desktop browsers (including dialog-in-dialog) and gets opened in a separate dojox.mobile.View with transition effect on the iPad in order to use all the real estate on the iPad screen. Since portlets can exchange data via the portal page mediator, this can be used to provide special settings portlets to configure one portlet from another one.
Our first 'real life' portlets use this modality feature to display a list of documents in a document library database. A click on a list entry displays detail information of the entry in a separate dialog / mobile view including the full document richtext with inline images and attachments (leveraging a nice hidden url syntax of Domino).
Both the list and detail portlet use Dojo's powerful Django template language to generate the HTML content.

Backend

The portal and portlet code and data are stored in a Lotus Domino 8.5.3 server.

As you might know, Lotus Domino 8.5.3 contains Dojo version 1.6.x as part of the XPages runtime environment. That version is already pretty outdated and not sufficient for our portal framework usecase (e.g. no dojox.mobile.ScrollablePane).
There is some hope that 8.5.4 will contain Dojo 1.8, but we could not wait until Q1/2013 for the 8.5.4 release (the world out there does not wait for IBM dev).
Replacing the existing Dojo version of the server with a newer one is more or less a hack and not supported as well, so we decided to render the whole portal UI in our own code and develop a set of REST services to communicate between frontend and backend code.

Content rendering is done in an XAgent style XPage. The page loads one or more files from the database design (added in the Java perspective of Domino Designer), parses placeholders in the text content with a regular expression (e.g. to include other files or insert paths and multilingual text phrases coming from property files in the database design) and sends the output to the browser.

We've got REST services to load/store information about all the portlets on the portal page. Each portlet has a state and appearance information as a JSON object (the latter e.g. to store in which column of the portal a portlet should be displayed).
The REST services are implemented with a Java control similar to IBM's rest service in the Extension Library (using the XPages extensibility API), but we removed the SSJS functionality from the control and directly call Java code (that's much cleaner than writing SSJS code that then calls our Java classes).

For each portlet instance, we create a Notes document in the database. There is a document to store the overall state of the portal page itself and there are configuration documents in the database to configure which portlets can be added to the page by the current user.
By replacing the backend storage code, portlet data can optionally be stored temporarily in the user session (sessionScope, viewScope). And since the whole framework is developed in a modular way, the portal UI can also be replaced by something else, e.g. to display new tabs instead of portlets. Actually, we don't use the terms "portal" and "portlet" in the framework. They are called "WorkbenchPage" and "ViewPart", similar to Eclipse terminology and displaying them in a portal layout is just a single use case.  

Summary

Developing this application was really fun, especially the iPad and CSS animation part.
The only thing that was a real PITA was to get the whole stuff working on Internet Explorer 7 (fortunately, only the desktop part, not iPad mode).

Hopefully, some of the concepts and links in this article are helpful for others as well, although there is not a single line of code in it. :-)
]]> http://www.mindoo.com/web/blog.nsf/dxcomments/18.09.2012001300KLEU38.htm http://www.mindoo.com/web/blog.nsf/dx/18.09.2012001300KLEU38.htm?opendocument&comments New address of Mindoo GmbH in Karlsruhe, Germany Sat, 30 Jun 2012 14:14:23 +0200 http://www.mindoo.com/web/blog.nsf/dx/30.06.2012141423KLEGD2.htm Mindoo Karsten Lehmann http://www.mindoo.com/web/blog.nsf/dx/30.06.2012141423KLEGD2.htm?opendocument&comments http://www.mindoo.com/web/blog.nsf/dx/30.06.2012141423KLEGD2.htm As of 1st of July 2012, our new company address in Karlsruhe, Germany is

Mindoo GmbH
Haid-und-Neu-Straße 7
76131 Karlsruhe
Germany ]]> http://www.mindoo.com/web/blog.nsf/dxcomments/30.06.2012141423KLEGD2.htm http://www.mindoo.com/web/blog.nsf/dx/30.06.2012141423KLEGD2.htm?opendocument&comments XPages series #14: Using MongoDB’s geo-spatial indexing in XPages apps part 2 Fri, 27 Apr 2012 19:00:09 +0200 http://www.mindoo.com/web/blog.nsf/dx/27.04.2012190009KLEMXN.htm XPages Karsten Lehmann http://www.mindoo.com/web/blog.nsf/dx/27.04.2012190009KLEMXN.htm?opendocument&comments http://www.mindoo.com/web/blog.nsf/dx/27.04.2012190009KLEMXN.htm Click here for part 1.


Diving into the code: client side

The UI stuff is stored in an NSF database with basic page layout defined in an XPage and the client-side application logic code stored as Dojo class file in the Java perspective of Domino Designer.



The Dojo class location is defined and the class is loaded with the following code in the XPage "start.xsp":

<xp:this.resources>
        <xp:dojoModulePath prefix="mongo">
                <xp:this.url><![CDATA[#{javascript:'/dojo/mongo';}]]></xp:this.url>
        </xp:dojoModulePath>
                <xp:dojoModule name="mongo.test.PageHandler">
        </xp:dojoModule>
</this.resources>

After that, the class can be instantiated in a script block:

<xp:scriptBlock type="text/javascript">
        <xp:this.value><![CDATA[
        dojo.addOnLoad(function() {
                window.pageHandler=new mongo.test.PageHandler();
                window.pageHandler.init();
        });
        ]]></xp:this.value>
</xp:scriptBlock>

In the init() method, the page handler class registers event code for the form fields and buttons in the UI and creates the Dojo grid components (dojox.grid.LazyTreeGrid) to display the database content.

Now let's take a look at two UI operations in detail, to see how the UI communicates with the server-side code.

Adding data to the database

When places get added through the Administration tab, we first use the servlet "/mongogeo" to convert the addresses into coordinates:



What we get back is the JSON response of  the Google Geocoding API. It does not only contain the coordinates of an address, but other quite useful information like the name of the federal state, the country code and coordinates of a bounding box (for general addresses like city names that don't have exact coordinates).

Next, a POST operation to "/mongotest/addplaces" creates the actual place documents in the database followed by a reload operation of our Dojo grid to reflect the new data in the UI:



Query the database

The REST service "/mongotest/queryplaces" is used both for reading all places (on the Administration tab) and finding the nearest places for a certain position (on the Search tab). For the latter, we specify additional longitude/latitude arguments in the URL:

http://localhost/dev/ec12/nosql/mongodb/geo.nsf/mongotest/queryplaces?distance=3&longitude=8.3799444&latitude=49.009148&type=Shop&start=0&count=100

Dynamic sorting can be applied to the result with and without longitude/latitude arguments by specifying a sort parameter.
If no sort parameter and no position is specified, places are sorted by name as default sorting. For queries with longitude/latitude parameters, the distance between both points is used as default sorting.

http://localhost/dev/ec12/nosql/mongodb/geo.nsf/mongotest/queryplaces?distance=3&longitude=8.3799444&latitude=49.009148&type=Shop&start=0&count=100&sort=name

In the UI, data can be sorted by clicking one of the grid column headers.




Enough technical details, let's come to an end with this long blog article! :-)

If you would like to try out the sample in your own environment, here are the setup instructions:


Setup instructions - Prerequisites

I expect that you have downloaded and installed the latest version of MongoDB for your operating system. The sample was built for version 2.0.3 and I just found out that 2.0.4 is already available.

The server code is using localhost and port 27017 by default, but this can be changed by setting the environment variable NOSQL_MONGO_SERVER to something like "hostname1,hostname2:27123,hostname3", a comma separated list with hostname and optional port. The Mongo driver will use the list of servers for failover in case a server goes down.

Plugin installation

Download and extract the archive file from the specified download link.

Then follow the instructions in the Domino wiki article XPages Extension Library Deployment in Domino 8.5.3 and IBM XWork Server to create an NSF based update site database on the Domino server and import the update site from the download archive into the update site database, followed by a HTTP task restart (restart task http).

Database installation

Copy the sample database from the download archive to your Domino server and sign it with your Notes ID. Now open the database in the browser.

There is a section "Initialization" on the Administration tab which lets you import a set of default addresses (from Mindoo's home town Karlsruhe, Germany). This step is optional.

Download link

Click here to download the sample application



Phew... That was a long text. Thanks for reading until here!

And stay tuned for an article about my second demo from Entwicklercamp 2012:

Using Neo4J to solve the travelling salesman problem in XPages apps

]]> http://www.mindoo.com/web/blog.nsf/dxcomments/27.04.2012190009KLEMXN.htm http://www.mindoo.com/web/blog.nsf/dx/27.04.2012190009KLEMXN.htm?opendocument&comments XPages series #14: Using MongoDB’s geo-spatial indexing in XPages apps part 1 Fri, 27 Apr 2012 18:59:38 +0200 http://www.mindoo.com/web/blog.nsf/dx/27.04.2012185938KLEMXB.htm XPages Karsten Lehmann http://www.mindoo.com/web/blog.nsf/dx/27.04.2012185938KLEMXB.htm?opendocument&comments http://www.mindoo.com/web/blog.nsf/dx/27.04.2012185938KLEMXB.htm session about NoSQL databases at the German Entwicklercamp conference in March 2012. It demonstrates how the document-oriented NoSQL database MongoDB can be used in an XPages web application for the IBM Lotus Domino server.

The rise of location based services

Location based services have become quite popular in the past years: Most of the smartphones carry a GPS sensor, and there are a lot of popular apps out there (e.g. Foursquare, Yelp) that use cloud services to find people, restaurants and places nearby.

For a number of reasons (mostly scalability/performance related), the NSF database of Lotus Notes/Domino is not the best choice to implement these kind of applications/services. MongoDB is much more suited for this use case and the article shows how you can integrate MongoDB into Notes/Domino to combine both worlds and get the best from both of them.

The Dojo based web application that I am going to discuss in detail uses MongoDB's geospatial indexing feature to easily find the nearest points of interest for a mobile web user.

Here is how our application looks like:



The "Administration" tab provides a user interface to add/remove places to/from the MongoDB database. A place is stored as a document in a MongoDB document collection and consists of a name, a type (e.g. shop/gas station) and the position information as [longitude, latitude] value pair.

The database can be queried on the "Search" tab by entering an address, a distance in kilometers and an optional type. The address will automatically get converted to [longitude, latitude] coordinates by using the Google Geocoding API, which is then used to find places within the specified distance.

Our MongoDB document collection is indexed in a way so that we can quickly query the database for places that surround the current user's position and sort them by ascending distance between both points.



Since geospatial indexing is a built-in feature of MongoDB, the solution is very easy to implement.

Limitations of NSF

While it is not impossible to build this application in pure Lotus Notes/Domino technology with an NSF database, the missing geo index and limited scalability of NSF would make it difficult to keep response times low with large data sets, e.g. millions or even billions of places worldwide. MongoDB instead provides a sharding feature to distribute the data evenly to several servers and reduce server load.

Additional indexer like Apache Lucene in combination with the OSGi tasklet service plugin's extension hooks might help to improve Domino's limited indexing support (we already used a Lucene index with Domino data in a big customer project) , but using MongoDB for this use case is just so much easier.

Architecture

The application consists of client-side and server-side code:
Client-side code is written in JavaScript language as a Dojo class using the class declaration system of Dojo 1.6. This Dojo class handles all user interface operations.

Server-side code is written in Java as a servlet and is exposed to the client-side via REST API's (here: GET/POST requests sent via dojo.xhrGet / dojo.xhrPost).

We developed two servlets for this application. The first servlet handles init/add/remove/query operations in MongoDB. The second servlet simply acts as a proxy to access the Google Geocoding API from client-side Javascript in order to convert between addresses and geo coordinates (longitude/latitude).

Since the REST protocol is completely stateless, the server-side code is very easy to test. With a bit of refactoring, it's even possible to run the whole application in alternative servlet containers like Jetty, Tomcat or directly within the Eclipse IDE instead of the Domino server.

Personally, I really like this REST API based approach because it's clean, transparent and keeps technologies separated.
It enables you to replace the server environment, the database system and the client-side UI toolkit at any time. You can even build multiple UIs (e.g. based on Dojo 1.6, 1.7 and Sencha's ExtJS) or mobile clients that work with the same REST API.

Diving into the code: server side
Our sample comes as an Eclipse plugin "com.mindoo.mongo.test" to be run on Domino's OSGi framework (tested with Domino 8.5.3 GA without any fixpacks or XPages extension library):



As you can see above, I added the Mongo API classes to the plugin's classpath.
Adding the Mongo API classes to an NSF and using them directly from SSJS code might also be possible, but I prefer the plugin solution, because the MongoDB access is supposed to be a central service on the Domino server. In addition, by using a plugin we don't have any issues with Domino's restricted security manager that prevents operations to run properly in an XPages context (e.g. a lot of drivers are using Log4J for logging, which does not run well within an XPages application).

Our two servlets are defined in the plugin.xml file of the plugin:

<?xml version="1.0" encoding="UTF-8"?>
<?eclipse version="3.4"?>
<plugin>
        <extension point="org.eclipse.equinox.http.registry.servlets">
                <servlet alias="/mongotest" class="com.mindoo.mongo.test.servlet.MongoTestServlet"
                        load-on-startup="true">
                </servlet>
        </extension>
        <extension point="org.eclipse.equinox.http.registry.servlets">
                <servlet alias="/mongogeo" class="com.mindoo.mongo.test.servlet.GeoQueryServlet"
                        load-on-startup="true">
                </servlet>
        </extension>
</plugin>

That's all you need to declare your own servlet on Domino.
After that, the servlet can either be accessed on the URL base level (http://server/mongogeo) or in the context of a Domino database (http://server/path/to/db.nsf/mongotest).

If called in the context of a database, the Domino HTTP server first makes sure that the current user is allowed to access the database and if not, it displays a login prompt. A utility class (com.ibm.domino.osgi.core.context.ContextInfo) can be used to get session/database objects for the current HTTP request.

In our sample, we use the first format for our Geo API query, because it's an open service of the server without access restriction. Anyone can use it and the URL is simple to remember.

The second URL format (/path/to/db.nsf/mongotest) is used to access MongoDB data. This enables us to check the user against the Notes database ACL to see if he is allowed to read or write MongoDB data.

Here is the method of our MongoDB access servlet that handles POST requests:

protected void doPost(HttpServletRequest req, HttpServletResponse resp) throws ServletException, IOException {
        Session session=ContextInfo.getUserSession();
        Database dbContext=ContextInfo.getUserDatabase();
        if (dbContext==null) {
                ServletUtils.sendHttpError(resp, HttpServletResponse.SC_FORBIDDEN,
                        "Servlet must be run in the context of a Domino database", null);
                return;
        }

        try {
                String pathInfo=req.getPathInfo();
                if ("/deleteplaces".equals(pathInfo)) {
                        if (!isUserAuthenticated(MongoAccessMode.Write, session, dbContext)) {
                                ServletUtils.sendHttpError(resp, HttpServletResponse.SC_FORBIDDEN,
                                        "Current user is not allowed to write data", null);
                                return;
                        }
                        doDeleteGridData(session, dbContext, req, resp);
                }
                else if ("/addplaces".equals(pathInfo)) {
                        if (!isUserAuthenticated(MongoAccessMode.Write, session, dbContext)) {
                                ServletUtils.sendHttpError(resp, HttpServletResponse.SC_FORBIDDEN,
                                        "Current user is not allowed to write data", null);
                                return;
                        }
                        doAddGridData(session, dbContext, req, resp);
                }
                else if ("/queryplaces".equals(pathInfo)) {
                        if (!isUserAuthenticated(MongoAccessMode.Read, session, dbContext)) {
                                ServletUtils.sendHttpError(resp, HttpServletResponse.SC_FORBIDDEN,
                                        "Current user is not allowed to read data", null);
                                return;
                        }
                        doQueryPlaces(session, dbContext, req, resp);
                }
                else if ("/initdb".equals(pathInfo)) {
                        if (!isUserAuthenticated(MongoAccessMode.Write, session, dbContext)) {
                                ServletUtils.sendHttpError(resp, HttpServletResponse.SC_FORBIDDEN,
                                        "Current user is not allowed to write data", null);
                                return;
                        }
                        doInitDbWithValues(session, dbContext, req, resp);
                }
                else if ("/cleardb".equals(pathInfo)) {
                        if (!isUserAuthenticated(MongoAccessMode.Write, session, dbContext)) {
                                ServletUtils.sendHttpError(resp, HttpServletResponse.SC_FORBIDDEN,
                                        "Current user is not allowed to write data", null);
                                return;
                        }
                        doClearDb(session, dbContext, req, resp);
                }
                else {
                        ServletUtils.sendHttpError(resp, 500, "Unsupported command "+pathInfo, null);
                }
        }
        catch (Throwable e1) {
                e1.printStackTrace();
                //avoid posting the full stacktrace to the user (for security reasons)
                ServletUtils.sendHttpError(resp, 500, "An error occurred processing the request. The error has been logged. Please refer to the server log files/console for details", null);
                return;
        }
}

The code is pretty simple: We check if the user is allowed to call the specific service (read operations require the role [MongoDBRead], write operations the role [MongoDBWrite] ) and then forward the request to helper methods.

Note that I am not going into full detail in this blog article how we actually access data in MongoDB. You can find that in the provided download archive and there is a great tutorial at the MongoDB website about using the Java API.

Believe me, this stuff is really easy to use!


Click here for part 2 of this article! ]]> http://www.mindoo.com/web/blog.nsf/dxcomments/27.04.2012185938KLEMXB.htm http://www.mindoo.com/web/blog.nsf/dx/27.04.2012185938KLEMXB.htm?opendocument&comments English slides for my Entwicklercamp 2012 session about NoSQL databases Thu, 5 Apr 2012 00:07:41 +0200 http://www.mindoo.com/web/blog.nsf/dx/05.04.2012000741KLETXV.htm NoSQL Karsten Lehmann http://www.mindoo.com/web/blog.nsf/dx/05.04.2012000741KLETXV.htm?opendocument&comments http://www.mindoo.com/web/blog.nsf/dx/05.04.2012000741KLETXV.htm NoSQL session at this year's Entwicklercamp (developer camp) conference in Gelsenkirchen, Germany.

Image:English slides for my Entwicklercamp 2012 session about NoSQL databases

I am still searching for time to polish the two demos: MongoDb integration in XPages apps and NoSQL integration in XPages apps.
Next week is blocked for German holidays. Need to work on the demos afterwards.
]]> http://www.mindoo.com/web/blog.nsf/dxcomments/05.04.2012000741KLETXV.htm http://www.mindoo.com/web/blog.nsf/dx/05.04.2012000741KLETXV.htm?opendocument&comments Download link for my Entwicklercamp 2012 session slides about NoSQL databases (German) Tue, 27 Mar 2012 19:00:43 +0200 http://www.mindoo.com/web/blog.nsf/dx/27.03.2012190043KLEMY2.htm NoSQL Karsten Lehmann http://www.mindoo.com/web/blog.nsf/dx/27.03.2012190043KLEMY2.htm?opendocument&comments http://www.mindoo.com/web/blog.nsf/dx/27.03.2012190043KLEMY2.htm I plan to translate them to English, but this may take some time.

Image:Download link for my Entwicklercamp 2012 session slides about NoSQL databases (German)


I will also write blog entries about my two demos: leveraging the MongoDb geo index feature from an XPages application and embedding the Neo4J graph databases in XPages (travelling salesman algorithm to find the best route in a graph).

Update:
The English slides of the session are now online.
]]> http://www.mindoo.com/web/blog.nsf/dxcomments/27.03.2012190043KLEMY2.htm http://www.mindoo.com/web/blog.nsf/dx/27.03.2012190043KLEMY2.htm?opendocument&comments Speaking at Entwicklercamp 2012 about NoSQL databases Mon, 19 Mar 2012 09:08:18 +0200 http://www.mindoo.com/web/blog.nsf/dx/19.03.2012090818KLEBKA.htm NoSQL Karsten Lehmann http://www.mindoo.com/web/blog.nsf/dx/19.03.2012090818KLEBKA.htm?opendocument&comments http://www.mindoo.com/web/blog.nsf/dx/19.03.2012090818KLEBKA.htm Entwicklercamp in Gelsenkirchen, Germany. My session will provide an overview of the NoSQL market for the database types
  • key value stores
  • column oriented databases
  • document oriented databases
  • and graph database

It discusses Redis, Apache Cassandra, MongoDB and Neo4j in detail and I will be demo'ing, how to integrate MongoDB's geo-spatial indexing feature and Neo4j's graph search algorithms in XPages applications.

Finally, the session compares the feature sets of Lotus Notes with other NoSQL to see how compatitive Lotus Notes, one of the oldest NoSQL database systems, is today in this moving market.

Working on this topic was quite challenging, because there is A LOT of material to check out on the web, but it also was a lot of fun playing with the databases and building the demos.
]]> http://www.mindoo.com/web/blog.nsf/dxcomments/19.03.2012090818KLEBKA.htm http://www.mindoo.com/web/blog.nsf/dx/19.03.2012090818KLEBKA.htm?opendocument&comments Icon resources on the web for commercial and non-commercial use Wed, 8 Feb 2012 08:44:18 +0200 http://www.mindoo.com/web/blog.nsf/dx/08.02.2012084418KLEB4A.htm Notesdev Karsten Lehmann http://www.mindoo.com/web/blog.nsf/dx/08.02.2012084418KLEB4A.htm?opendocument&comments http://www.mindoo.com/web/blog.nsf/dx/08.02.2012084418KLEB4A.htm Since I found a new massive resource of free icons yesterday, I thought it might be a good idea so share some URLs.

For commercial applications, we bought the Icon Experience library a few years ago, which contains about 2500+ icons in 7 sizes at a very affordable price:
http://www.iconexperience.com

Free icons with varying licenses can be found here:

http://openiconlibrary.sourceforge.net
http://www.freeiconsweb.com
http://www.famfamfam.com
http://p.yusukekamiyamane.com

For the latter, licenses reach from GPL, LGPL, Creative Commons to Public Domain. I guess for pure inhouse development, this is not that important, but if you plan to sell your application or publish it on OpenNTF, you should doublecheck if the icon license is ok for your use case.

Feel free to add your own favorites by commenting on this article! ]]> http://www.mindoo.com/web/blog.nsf/dxcomments/08.02.2012084418KLEB4A.htm http://www.mindoo.com/web/blog.nsf/dx/08.02.2012084418KLEB4A.htm?opendocument&comments XPages series #13: XPiNC app development tool Thu, 2 Feb 2012 16:24:12 +0200 http://www.mindoo.com/web/blog.nsf/dx/02.02.2012162412KLEL3Q.htm XPages Karsten Lehmann http://www.mindoo.com/web/blog.nsf/dx/02.02.2012162412KLEL3Q.htm?opendocument&comments http://www.mindoo.com/web/blog.nsf/dx/02.02.2012162412KLEL3Q.htm Sencha's Ext JS for the UI and is completely based on our own web app development framework.
The framework does not use XPages design elements at all, but follows a simple REST API architecture with standard servlets to produce the data and user interface.

The benefit of this approach is that web applications can be developed, run and debugged from a pure Eclipse environment. They can even run on an different servlet engine than the Domino server's http task and - thanks to our data abstraction layer - they can even store the whole app data in a non-Domino database and mix/merge data between different database types.

Another benefit is that almost all the code has been written by ourselves. So it's not a kind of blackbox, made by IBM, where it's hard to work around occuring issues, but we are able to track issues down right until the service(HttpServletRequest req, HttpServletResponse response) call coming from the web container.

The downside is of course, that apps developed with the framework do not make use of IBM's XPages Extension Library, that contains various powerful UI controls. So we needed to create them ourselves.

And regarding XPiNC development, there is another downside: JavaScript errors do not get logged/displayed in the Notes Client, unless you register your own global error handler (providing a function for window.onerror).
Normal XPiNC applications do already contain such an error handler (somewhere within the XSP API object), which uses an internal bridge to post the error content to the Notes Client's status bar.

The development tool

To make development of XPiNC applications easier, I have created a small Eclipse plugin that displays three icons in the Client's toolbar:
Image:XPages series #13: XPiNC app development tool


The third icon lauches a piece of code that injects Firebug lite into the currently visible XPage:

        public void run(IAction action) {
                IWorkbenchPart part =
                  PlatformUI.getWorkbench().getActiveWorkbenchWindow().getPartService().getActivePart();

                if (part instanceof XspViewPart) {
                        XspViewPart xPart=(XspViewPart)part;
                        XspXulRunnerBrowser browser=xPart.getWebBrowser();
                       
                        HTMLDocument doc=browser.getDocument();
                        NodeList headNodes=doc.getElementsByTagName("head");
                        if (headNodes!=null && headNodes.getLength()>0
                           && headNodes.item(0) instanceof Element) {

                                //add this snippet to the HTML DOM tree:
                                //< script type="text/javascript"
                                // src="https://getfirebug.com/firebug-lite.js" > < /script >
                                String firebugUrl="https://getfirebug.com/firebug-lite-debug.js";
                                Element headNode=(Element) headNodes.item(0);
                                Element scriptNode=doc.createElement("script");
                                scriptNode.setAttribute("type", "text/javascript");
                                scriptNode.setAttribute("src", firebugUrl);
                                headNode.appendChild(scriptNode);
                        }
                }
                else {
                        MessageDialog.openError(Display.getDefault().getActiveShell(), "Error",
                          "This action does only work within an XPiNC application!");
                }
        }

The result looks like this:

Image:XPages series #13: XPiNC app development tool


Using Firebug Lite, you can easily inspect the HTML DOM tree and CSS attributes of the current page. Another very useful feature is the Console API, which you may know already from classic browser development:
it lets you write log/debug messages to the browser console (by calling console.log('...')) and has other nice features like stacktrace dumping of JavaScript calls.

To our surprise we found out that the pure Xulrunner engine, that is used to display XPages in the Notes Client, does not register the global "console" at all. So Firebug lite came to the rescue. You can find all your log messages in the Console tab.

The other two toolbar actions are even more powerful. One lets you define a custom script library URL, e.g. to a script library design element on a public or intranet web server. The other actions will then create a script tag in the current XPage that points to the library. That way, you can inject any code you like into the XPiNC application.

Download

Finally, here is the download link with the Eclipse plugin project, feature project and update site:

xpinc-firebuglite-helper.zip

Hope this helps!
]]> http://www.mindoo.com/web/blog.nsf/dxcomments/02.02.2012162412KLEL3Q.htm http://www.mindoo.com/web/blog.nsf/dx/02.02.2012162412KLEL3Q.htm?opendocument&comments Lotusphere 2012: Download links for 105 additional session slides Sun, 22 Jan 2012 22:59:10 +0200 http://www.mindoo.com/web/blog.nsf/dx/22.01.2012225910KLETSK.htm Lotusphere 2012 Karsten Lehmann http://www.mindoo.com/web/blog.nsf/dx/22.01.2012225910KLETSK.htm?opendocument&comments http://www.mindoo.com/web/blog.nsf/dx/22.01.2012225910KLETSK.htm Lotusphere 2012 website with my previously released list of session slide download links.
I found 105 new slide downloads:

ls12_20120122.csv
ls12_20120122.html

There are still a few missing slide decks, but it's a big step forward. ]]> http://www.mindoo.com/web/blog.nsf/dxcomments/22.01.2012225910KLETSK.htm http://www.mindoo.com/web/blog.nsf/dx/22.01.2012225910KLETSK.htm?opendocument&comments Lotusphere 2012: download links for session slides Tue, 17 Jan 2012 10:50:09 -0400 http://www.mindoo.com/web/blog.nsf/dx/17.01.2012105009KLELKX.htm Lotusphere 2012 Karsten Lehmann http://www.mindoo.com/web/blog.nsf/dx/17.01.2012105009KLELKX.htm?opendocument&comments http://www.mindoo.com/web/blog.nsf/dx/17.01.2012105009KLELKX.htm Lotusphere 2012 website. Since my Macbook Pro could not get a proper connection to the wireless network in Dolphin (network is quite bad both at Lotusphere and in the Yacht Club we are staying in), I had to do this on the iPhone, because it was the only device that got an IP address. That was fun. ;-)

So here are the links to the slides that are available so far, sorted by session ID. Unfortunately, many are still missing. From our last years experience, the speakers are not the one to blame here. All had to submit their slides back in December.

Hopefully, the remaining session slides will follow shortly.

ls12_20120117.csv
ls12_20120117.html

Use your preferred download utility (e.g. DownThemAll) for download. You need to be logged in on the LS12 website before downloading. ]]> http://www.mindoo.com/web/blog.nsf/dxcomments/17.01.2012105009KLELKX.htm http://www.mindoo.com/web/blog.nsf/dx/17.01.2012105009KLELKX.htm?opendocument&comments Status report / no session submission for LS12 Mon, 7 Nov 2011 10:27:35 +0200 http://www.mindoo.com/web/blog.nsf/dx/07.11.2011102731KLED4S.htm Lotusphere 2012 Karsten Lehmann http://www.mindoo.com/web/blog.nsf/dx/07.11.2011102731KLED4S.htm?opendocument&comments http://www.mindoo.com/web/blog.nsf/dx/07.11.2011102731KLED4S.htm To give you an impression, here are a few things that we've been working on:

Development of an OSGi based web application framework with Ext.js UI
Abstraction layer for web applications that unifies data access across document-oriented and relational database systems, allows for tracking of data object changes (old/new values, group multi-object changes as transactions), different kinds of data serialization like XML or JSON and visualizing data with an Ext.js based, dynamically created web UI (using Jamon template language) that reads and writes contents through REST APIs. The framework can run from pure Eclipse and also in Domino's OSGi container.

Development of an OSGi based XPages application framework with Dojo UI
XPages extensibility API add-on to develop the backend code of XPages directly in Java instead of writing SSJS code or Expression Language to bridge between UI element events/properties and application code.
The framework uses Java Annotations and Java Reflection APIs for UI/backend code weaving.

Development of an IBM Websphere portal server applications
Custom development of market-analysis application for consumer goods industry.

Development of dynamic web application based on Glassfish and JPA, with jQuery and Highcharts
Data analysis tool for the buying department of a German automotive company.

Plugin development for the Lotus Notes Client
Several productivity enhancement plugins for the Lotus Notes Client

Plugin deployment support
Autodetection routine for XPages applications in Client/Server to automatically check for required Eclipse features/plugins when an application is opened (implemented in Lotusscript) so that the XPages runtime does not display those confusing error messages about missing XPages extensions. Instead, errors are properly handled without user interaction and missing features are automatically deployed on the machine.

XPiNC development framework
Spent time to work on enhancements for the XPages2Eclipse toolkit that provides Eclipse APIs to XPages applications in the Lotus Notes Client.

Classic Domino web application development
Improved round tripping quality of richtext editing between web browser and Notes Client including paste optimization from MS Word. Solution uses a combination of CKeditor add-ons and HTML/DXL conversions developed in Lotussript code for R7.  Customer: print industry

Classic Notes Client development
Built a Notes application to automatically deploy and update a list of mail folders to iOS users on Traveler.


A word about Lotusphere 2012
Since time is so limited, we decided not to submit a session proposal for Lotusphere 2012.

We will however participate with full conference pass and I just booked hotel and flight two days ago. In contrast to previous years, we will arrive one day earlier, on Friday, 13th of January 2012 (hopefully not a bad day for travel, at least it's a direct flight :-)).
Are there any plans yet for a blogger meeting for a beer or two on Saturday 14th like in previous years?

Another reason not to submit a session is that things get more complicated year after year. To learn about OSGi development or XPages extensibility APIs in a hands-on session for a day or two does make much more sense than watching a one hour presentation.

We could have submitted our plugin development session from last year another time (overview of plugin development for Notes Client and DDE with 14 demos), because slides and demos are already done. But that session (which got very good feedback at LS11) has now been presented four times at two conferences (Lotusphere and German Entwicklercamp 2010/2011). It's hard to get motivated for the 5th time :-).


So, to sum up this long blog entry, I am really looking forward to seeing many of you guys at LS12 - and to having more time to work on concepts and product prototypes in the next weeks. ]]> http://www.mindoo.com/web/blog.nsf/dxcomments/07.11.2011102731KLED4S.htm http://www.mindoo.com/web/blog.nsf/dx/07.11.2011102731KLED4S.htm?opendocument&comments XPages series #12: XAgents and performance bottlenecks Sun, 17 Jul 2011 10:18:55 +0200 http://www.mindoo.com/web/blog.nsf/dx/17.07.2011101855KLEBRW.htm XPages Karsten Lehmann http://www.mindoo.com/web/blog.nsf/dx/17.07.2011101855KLEBRW.htm?opendocument&comments http://www.mindoo.com/web/blog.nsf/dx/17.07.2011101855KLEBRW.htm XAgent is a term that describes the equivalent of a classic Notes web agent in XPages technology: an XPage is called via URL and produces any kind of data (e.g. HTML, dynamic images, data in JSON format or ODF documents) by sending Strings or even raw bytes directly to the browser.
 
Chris Toohey and Stephan Wissel have already blogged about this topic a few years ago and discussed some use cases:

IBM Lotus Notes Domino REST Web Services via XPage XAgents
Web Agents XPages style

How to write an XAgent
To write an XAgent, you basically need to do two things: first you add the attribute rendered="false" to the xp:view tag of an XPage to prevent the XPages engine from rendering any output. Second, you need to write the code that produces the data and add it to the beforeRenderResponse event of an XPage.

Here are two simple example XAgents. The first one writes XML content to the writer object of the servlet response (which is used to return character data):

<?xml version="1.0" encoding="UTF-8"?>
<xp:view xmlns:xp="http://www.ibm.com/xsp/core" rendered="false">

        <xp:this.beforeRenderResponse>><![CDATA[#{javascript:try {
        var uName = session.createName(session.getEffectiveUserName());
        var exCon = facesContext.getExternalContext();
        var response = exCon.getResponse();
        var writer = response.getWriter();
        response.setContentType("text/xml");
        response.setHeader("Cache-Control", "no-cache");
        writer.write("<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n");
        writer.write("<test>Hello "+uName.getAbbreviated()+"</test>");
        facesContext.responseComplete();
}
catch (e) {
        _dump(e);
}}]]></xp:this.beforeRenderResponse></xp:view>

The second example demonstrates how you can load a file resource from the database design and write it to the output stream object of the servlet response:

<?xml version="1.0" encoding="UTF-8"?>
<xp:view xmlns:xp="http://www.ibm.com/xsp/core" rendered="false">

        <xp:this.beforeRenderResponse><![CDATA[#{javascript:try {
        var exCon = facesContext.getExternalContext();
        var response = exCon.getResponse();
        var outStream = response.getOutputStream();
       
        //load a file resource from db design and write it to the output stream
        var cl=com.ibm.domino.xsp.module.nsf.NotesContext.getCurrent().getModule().getModuleClassLoader();
        var buf=new byte[16384];
        var len;
        var filePath="/test/document.docx";
        var inStream=cl.getResourceAsStream(filePath);
        if (inStream==null) {
                var errMsg="File "+filePath+" not found";
                _dump(errMsg);
                response.sendError(404, errMsg);
        }
        else {
                //set mime type according to file type
                response.setContentType("application/vnd.openxmlformats-officedocument.wordprocessingml.document");
                response.setHeader("Cache-Control", "no-cache");
                //send http header with filename (needed so that the browser does not use the XPage's name as filename)
                response.setHeader("Content-Disposition", 'attachment; filename="document.docx"');
               
                //read bytes from inStream into the buffer
                while ((len=inStream.read(buf))>-1) {
                        //and write the data to the output stream
                        outStream.write(buf, 0, len);
                }
                inStream.close();
                outStream.close();
        }
        facesContext.responseComplete();
}
catch (e) {
        _dump(e);
}}]]></xp:this.beforeRenderResponse></xp:view>

The loaded file resource must be part of the NSF project's build path to make the XPage code find it (that's where the module classloader is looking for files).
Please refer to part two of this XPages series to get more information about how to add a folder to the build path.


Image:XPages series #12: XAgents and performance bottlenecks

If you just want to offer a file for download, developing an XAgent to stream the data would probably be the wrong solution.

In that case, you should add your files to the database design in the Java Perspective of Domino Designer instead and reference them directly in a URL. This is something that is used in the OpenNTF project XPages Mobile Controls to add a newer Dojo version (e.g. 1.6 with advanced mobile device support) to a Domino server than the default one (e.g. version 1.4.3 for Lotus Domino 8.5.2).

But the second code example from above could of course be modified to do something more useful, for example to replace placeholders in the file, create a zip file from multiple files on the fly using Java's ZipOutputStream or render a captcha image for spam protection.

Sounds great?
Well, before you start thinking about building large applications based on this approach, you should read on to get to know about it's limitations.

The limitations
What I'm going to show you is not only a limitation of XAgents, but of XPages technology in general. For XPages producing the user interface of an application, this limitation may not be critical, but it can become a show stopper for using them the XAgent way.

I've created a small test application with three XPages: Start.xsp, Frame1.xsp and Frame2.xsp (download link). The first one (Start.xsp) contains two iframes and a button. By clicking on the button, the other two XPages are loaded as iframe contents:

var now=new Date();
var nowTime=now.getTime();
var iframe1=document.getElementById("iframe1");
var iframe2=document.getElementById("iframe2");
iframe1.src=document.location.pathname+"/Frame1.xsp?startTime="+nowTime;
iframe2.src=document.location.pathname+"/Frame2.xsp?startTime="+nowTime;

Frame1.xsp contains beforeRenderResponse event code that delays the rendering for 5000 milliseconds (java.lang.Thread.sleep(5000)), while Frame2.xsp waits for 2500 milliseconds.

You would expect that both XPages are opened and rendered concurrently. Because of the delay, Frame2.xsp should be visible first, then Frame1.xsp.

Unfortunately, the output looks like this instead (click on the image to open the test application):

Image:XPages series #12: XAgents and performance bottlenecks

Frame1.xsp is displayed first, followed by Frame2.xsp. Frame2.xsp takes about 7600 milliseconds to be rendered, which is the sum of both delays, because the XPages engine does not render multiple XPages in an application at the same time for a single user.

It was a real surprise when we noticed this effect in a customer project. Our UI was based on the Ext JS toolkit which is heavily using REST services to read the UI component data. Our plan was to use XAgents for those REST services, but we quickly noticed that all UI elements only received data one at a time, which was a no-go for the application UI performance. Nothing was loading in parallel.

There are technical reasons for such a restriction, as Philippe Riand, XPages Chief Architect at IBM, explained to me in April 2011.
As you may know, the XPages runtime is based on JSF technology, which uses a tree of components on the server side that represents the structure of a web page and its current state. The tree is created/restored when a HTTP request comes in and saved/discarded after it is completed.
Unfortunately, this tree does not support being accessed by two concurrent threads, which would break many things like data sources and component states.

To prevent this to happen, XPages synchronizes on the user session object (HttpSession), so that only one thread can access the tree at a time. But by synchronizing on the user session (and not just on the component tree instance), the XPages dev crew also prevented any other XPage in an application from running (the Domino server seems to assign session objects on a per database basis).

The consequence is that you should think twice about your XPages application architecture, if you have many concurrent HTTP requests or if some of them take a lot of time to be processed. An XAgent may be the easiest solution to deploy, but may not produce the best user experience in all cases.

Workarounds and solution
There is some hope that the situation will improve in Domino 8.5.3.

As Philippe said, the next XPages engine will be smarter in synchronization and synchronize less than previously.
In addition, there will be a new database property called xsp.session.transient. This flag means that unique session objects will be created per request to the server and discarded right after the request ended. This is a first attempt to provide session less mode.
If you use this option, then you can create one database with all the services and no synchronization will happen, as each request will have its own session object.

I have requested a more granular option so that you can activate session less mode for single XPages in an application, but I'm not sure if this will make 8.5.3.

For Domino 8.5.2, I can see three things that you can do if you can't live with this performance issue:

1. Move XPages that run for a long time to separate databases
This does not help a lot, but if you have an XPages application with an XPage that needs some time to process, you could move it to different database. In that case, the rest of the application would still be responsive, although your long running XPage for example produces a 100 MB log output in XML format that you want to download.

2.  Write your own servlet in Java
By writing your own servlet as an Eclipse plugin running in Domino 8.5.2's OSGi framework, your code can run without any performance bottleneck caused by synchronization. See this article in the OpenNTF blog for implementation details.

3. There is an app for that :-)
As you may know, we have developed XPages2Eclipse, an extensive toolkit for XPages development, which is currently in beta-testing (June 2011). The articles Building servlets in JavaScript and Building servlets in Java demonstrate how you can write your own servlet implementation in JavaScript or Java language.
This solution reads all your code from the database design, so that you don't have to deploy every servlet individually. Just install the toolkit on a Domino server once, then you can create any number of servlets by writing code in Domino Designer and replicate them to the server.

]]> http://www.mindoo.com/web/blog.nsf/dxcomments/17.07.2011101855KLEBRW.htm http://www.mindoo.com/web/blog.nsf/dx/17.07.2011101855KLEBRW.htm?opendocument&comments