1. The Big Blob

Seen from a developer's perspective early CMSs came along as one big piece of software . If you wanted to develop custom applications with them you usually had to develop on top of the CMS API. This API for WCMS would contain abstractions like "paragraph", "user" or "workflow" and expose actions like "publish a page", i.e. the abstraction level was the level of the WCMS business logic (I use the term "business logic" as in "the business of managing content"). For the purpose of this argument, let's call these APIs the "business level API".

2. Splitting off the repository

It was obvious that the content had to be also accessible for developers outside of the context of the business-layer API. Thus, the repository layer was abstracted: a number of CMSs created an API that allowed direct access to the content, either through JCR/JSR-170 or some proprietary API. This gave developers two choices how to access the content: either in the context of the business API or the "raw" access through, say, JCR.

Introducing the repository-level API can be seen as a clean architectural cut for a CMS in order to improve separation of concerns.

3. Splitting off the web infrastructure

In this next evolutionary step we take what is left of the WCMS after the repository is split off and separate off another layer: the web infrastrucure (Sling). This layer contains the basic plumbing required to build web apps, e.g. request handling, selecting content, script execution, authentication, filters etc. This leaves the business layer with the business logic only.

One effect of splitting off the web infrastructure is that developers can write web applications that sit "next to" the WCMS application rather than running on top of the WCMS. This means that these developers do not need to be concerned with (and learn about) the particular CMS business logic because they have access to the web application infrastructure provided by the CMS.

This is a second architectural cut and separation of concerns that acknowledges that there are different types of content centric applications and WCMS is only one of them (consider forums, blogs, media asset management, wikis, etc). These different content-centric apps share requirements for repositories and web infrastructure, though.

It should be noted that compared to the second step above this additional separation of concerns does not really allow you to do anything that was not possible before. However, it is such a big advancement in terms of developer productivity that I consider it to be a qualitative change. (If you disagree: Try building a web app starting with just a connection to your repository and another one on top of Sling. You will understand.)

Some clarifications

There are of course myriads of WCMSs already that let developers write extensions or modules or similar. That approach is different though because these applications run "within" the CMS context, i.e. the developer is still interfacing the business layer. On top of Sling, your app is a first-class citizen just like CQ5.

This CMS-oriented view of the world and its history might seem strange if you look from an application server angle: in the app server world the persistence layer, the application framework (Struts, PHP, ...) and the application were always neatly separated - so Sling would be nothing new. However, this is not quite the same situation as step 3 in the CMS world because in the classical web infrastructure world the application framework knows nothing about the underlying repository - usually it just handles database connection pooling or similar. Sling does know about the repository and thus it can provide additional functionality. Therefore, if you want to stay in the classical web infrastructure picture you could regard Sling as a very specialized application framework (specialized for content-centric applications). AFAIK this is a new piece of infrastructure that did not exist before.

The abstracts:

Was ist Content?

Bilder, Videos, Texte – man erkennt Content, wenn man ihn sieht. Aber hat Content charakteristische Eigenschaften, die ihn von reinen Daten unterscheiden? In Gesprächen über Content und Content-Management ist gelegentlich festzustellen, dass Content mit relationalen Datenbanken oder Dateisystemen gleich gesetzt wird. Obwohl Content letzten Endes oft im Dateisystem oder einer Datenbank gespeichert wird, wird eine Reduktion auf diese Low-Level-Sicht der Thematik nicht gerecht. Content ist eine spezielle Form von Daten, denen eine eigene Qualität innewohnt. Insofern ist auch Content-Management mehr als nur eine Datenbankmaske.
David Nüscheler, Michael Marth

JCR und Apache Jackrabbit

Der JCR-Standard (Content Repository API for Java) beschreibt eine klare Schnittstelle zwischen Anwendung und Content-Ablage. Das hierdurch definierte Content Repository stellt die dem Content typischen Eigenschaften und Funktionalitäten in standardisierter Weise zur Verfügung. Dies erleichtert erheblich die systemübergreifende Nutzung von Content und erschließt damit ein bisher nur mit viel Mühen auszuschöpfendes Potenzial.
Michael Marth, Gerd Handke, David Nüscheler, Carsten Ziegeler

I have recently provided a Sling example application called Notes which I want to use to demonstrate the implementation of OpenSearch. Import the Notes application into your CRX Quickstart (download Quickstart if you do not have it, yet). For importing navigate to http://localhost:7402/crx/index.jsp and start the Package Manager.

In OpenSearch search results can be in different formats. Let's start with producing results in XHTML. Add a new file at /apps/notes/opensearch.jsp. It shall contain:

<?xml version="1.0" encoding="UTF-8"?>
<%@page import="javax.jcr.query.*, javax.jcr.*"%>
<%@taglib prefix="sling" uri="http://sling.apache.org/taglibs/sling/1.0"%>
<sling:defineObjects/>
<%@ include file="/apps/notes/util.jsp" %>
<%
String q = "/jcr:root/content/notes//*["+ SearchUtils.parameterToQuery(request.getParameter("qt")) +"]";
Query query = currentNode.getSession().getWorkspace().getQueryManager().createQuery(q, "xpath");
NodeIterator result = query.execute().getNodes();
%>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
  <head profile="http://a9.com/-/spec/opensearch/1.1/" >
    <title>Notes app search results</title>
    <link rel="search" type="application/opensearchdescription+xml" href="http://localhost:7402/apps/notes/opensearch" title="Notes Search" />
    <meta name="<%=result.getSize()%>" content="4230000"/>
    <link rel="stylesheet" href="/apps/notes/static/blue.css">
  </head><body>
    <div id="Header"><a href="/content/notes.html">< back to thread overview</a></div><div id="Content">
    <h1>search results for query <%=request.getParameter("qt") %>, hits: <%=result.getSize()%></h1><ul><%@ include file="/apps/notes/searchBox.jsp" %><ul>
<%
while(result.hasNext()) {
  Node n = result.nextNode();
  String t = n.getProperty("body").getValue().getString();
%>		
    <li>
    <a href="http://localhost:7402<%=n.getPath()%>.thread.html"><%=t.length() > 50 ? t.substring(0,50) + "..." : t%></a>
    <div><p><%=t%></p></div>
    </li>
<%}%>		
</ul></div></body></html>

There is only a few OpenSearch-related lines like the link to the OpenSearch descriptor file (explained further below)

<link rel="search" type="application/opensearchdescription+xml" href="http://localhost:7402/apps/notes/opensearch" title="Notes Search" />

and the META tag that describes the result set:

<meta name="<%=result.getSize()%>" content="4230000"/>

The rest of this jsp is just using the standard JCR-provided query functionality to produce standard XHTML. That's it. This is your shiny new OpenSearch interface (I need to stress this once more because it's really cool: no search engine crawling or or other setup was needed. Neither was there any JCR-JSP-wiring or other web app configuration). Point your browser to: http://localhost:7402/content/notes.opensearch.html?qt=JCR to have a look (the request parameter qt denotes the query term).

The above mentioned descriptor file describes the app's search capabilities to external parties. Given the link element from above it needs to be in /apps/notes/opensearch and shall contain:

<?xml version="1.0" encoding="UTF-8"?>
<OpenSearchDescription xmlns="http://a9.com/-/spec/opensearch/1.1/" xmlns:moz="http://www.mozilla.org/2006/browser/search/">
  <ShortName>Notes app</ShortName>
  <Description>Notes app search interface</Description>
  <Url type="text/html" template="http://localhost:7402/content/notes.opensearch.html?qt={searchTerms}"/>
  <Url type="application/rss+xml" template="http://localhost:7402/content/notes.opensearchrss.xml?qt={searchTerms}"/>
  <Url type="application/x-suggestions+json" template="http://localhost:7402/content/notes.opensearchsuggestions.json?qt={searchTerms}"/>
</OpenSearchDescription>

The first URL element describes the XHTML-based output implemented discussed above. OpenSearch also allows RSS-based output which is described in the second URL-element. The implementation of RSS-based OpenSearch results is even more succinct because there is less boiler-plate code. Create the renderer file /apps/notes/opensearchrss.xml.jsp and let it contain:

<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:opensearch="http://a9.com/-/spec/opensearch/1.1/">
<%@page import="javax.jcr.query.*, javax.jcr.*"%>
<%@taglib prefix="sling" uri="http://sling.apache.org/taglibs/sling/1.0"%>
<sling:defineObjects/>
<%@ include file="/apps/notes/util.jsp" %>
<%
String q = "/jcr:root/content/notes//*["+ SearchUtils.parameterToQuery(request.getParameter("qt")) +"]";
Query query = currentNode.getSession().getWorkspace().getQueryManager().createQuery(q, "xpath");
NodeIterator result = query.execute().getNodes();
%>
<channel>
  <title>Local Notes app</title>
  <link>http://localhost:7402/content/notes.html</link>
  <description>Search results for "<%=request.getParameter("qt") %>" at the local Notes app</description>
  <opensearch:totalResults><%=result.getSize()%></opensearch:totalResults>
<%
while(result.hasNext()) {
	Node n = result.nextNode();
%>
  <item>
    <% String t = n.getProperty("body").getValue().getString(); %>
    <title><%=t.length() > 50 ? t.substring(0,50) + "..." : t%></title>
    <link>http://localhost:7402<%=n.getPath()%>.thread.html</link>
    <description><%=t%></description>
  </item>
<%}%>
</channel></rss>

Hit http://localhost:7402/content/notes.opensearchrss.xml?qt=JCR with your browser or any other RSS viewer to retrieve the search results in RSS format.

OK, this is simple to implement, but what is it good for? For example, recent browsers support autodiscovery of OpenSearch engines and allow users to add them to the browser's upper right search box. Simply add a link in the HTML page's header:

<link rel="search" type="application/opensearchdescription+xml" title="Notes" href="http://localhost:7402/apps/notes/opensearch">

I have tested this in IE7 and FF2:

Search Suggestions

There is an enhancement suggested for upcoming versions of OpenSearch that would standardize search suggestions. In Firefox, however, a similar feature is already implemented. The third URL in the descriptor file describes the interface for search suggestions in FF (i.e. this will not work in IE). The corresponding implementation at /apps/notes/opensearchsuggestions.json.jsp looks like this:

<%@page import="javax.jcr.query.*, javax.jcr.*, java.util.*"%>
<%@taglib prefix="sling" uri="http://sling.apache.org/taglibs/sling/1.0"%>
<sling:defineObjects/>
<%@ include file="/apps/notes/util.jsp" %>
<%
// Hashmap to collect and count full words that contain the query term
TreeSet<String> collector = new TreeSet();

// only accept min 2 char strings and does not end with a blank
if((request.getParameter("qt").length() > 1) && !(request.getParameter("qt").endsWith(" "))) {

// the actual search
String q = "/jcr:root/content/notes//*[" + SearchUtils.parameterToSuggestionQuery(request.getParameter("qt")) + "]";
Query query = currentNode.getSession().getWorkspace().getQueryManager().createQuery(q, "xpath");
NodeIterator result = query.execute().getNodes();

// the constant part of the suggestions
String constantPart = "";
String[] requestTokens = request.getParameter("qt").split("\\s");  
for (int x=0; x<requestTokens.length - 1; x++) constantPart += requestTokens[x] + " ";

// we need to see what the actual word was where the search result occurred
while(result.hasNext()) {
  Node n = result.nextNode();
  String[] tokens = n.getProperty("body").getValue().getString().split("\\s");  
  for (int i=0; i<tokens.length; i++) if(tokens[i].toLowerCase().contains(requestTokens[requestTokens.length-1].toLowerCase())) collector.add(tokens[i]);
}

// print result
// format is e.g. ["fi", ["firefox", "first", "fist"]]
%>["<%=request.getParameter("qt")%>", [<%
boolean first = true;
Iterator results = collector.iterator();
while(results.hasNext()) { %>
  <%=!first?", ":""%>"<%=constantPart + results.next()%>"
  <% first = false; }%> ]]
<%} else { // if the query term is only 1 char long%>
["<%=request.getParameter("qt")%>",[]]
<%}%>

As the jsp's file name suggests the jsp returns json-formatted data. The returned data is a collection of suggestions based on the actual content (i.e. suggesting words that would produce hits). The format for a query "fi" must be of the form ["fi", ["firefox", "first", "fist"]]. Most of the code deals with String manipulation which is a bit verbose in Java. Also, I wanted to have multi-word queries like "jcr re" return suggestions for the last word only (i.e. ["jcr re", ["jcr repository", "jcr renderer"]]) which requires a couple of additional lines of code.

As usual, the full application including the complete sources is attached to this post in CRX Package Manager format.

Paul Mabray: So tell me a bit about each of you - your background, hobbies, etc.

Matej Knopp: Hi, I am Matej. I’m a key independent consultant from Slovakia. My number one project for Inertia is working on Brix and enjoying every minute of it. I’m also an Apache Wicket enthusiast and committer. Obviously I have a severe lack of free time.

Paul: Do you do anything else?

Matej: Whatever little time I find away from the keyboard I try to read as much as I can.

Patrick Angeles: I guess, aside from you Paul, I am the veteran at Inertia. I am the VP of Technology and lead our SaaS development for a software solution that connects wineries to their customers. I’ve been in tech for over 10 years working for big companies like Bankers Trust and Acquire Systems. Food, wine and my family are the only things I have time to do outside of work.

Igor Vaynberg: I’m the Principal Software Engineer at Inertia, residing in sunny Sacramento, California. I’m the guy your brought in to help build the next-generation enterprise web application that help will more easily bridge the gap between wineries and their customers.

Paul: Tell me more

Igor: Well, my love for writing software was sparked when I received a Sinclair Z80 on my tenth birthday and I’ve been doing it ever since. I do have the same hobbies as you such as playing with my kids, taking my beautiful wife to fancy dinners - that should get me points at home - and fragging my friends at Halo. I also still find time to work on software. My number one outlet is the Apache Wicket framework - I really believe to be the best way to build web applications.

Paul: And I am the founder of Inertia and now Chief Strategy Officer in charge of product development and business development. Just like Patrick I only have time for the family outside of work. But I love what I do and when this team brought forth the concept to make Brix open source, I was 200% in support of the initiative.

Igor/Patrick: COME ON

Paul: OK, I frag a few people on Halo when I get a spare minute.

Paul: So Michael asked us a question - Brix uses Apache Jackrabbit as its content repository. Back in the days when you made this decision what potential choices were you considering and what influenced your decision towards Jackrabbit?

Igor: From the start, one of our major requirements was the ability for our users to edit content using their favorite desktop editor. We picked SVN and JCR as two possible solutions: SVN with its local check-in/check-out, and JCR with its WebDAV support. We started off with SVN because it had more powerful versioning features, such as merging. But we quickly ran into features we needed that were missing and would be too expensive to implement ourselves: referential integrity and ASLv2 java connector library being the two that jump to mind (we were planning to open-source Brix from the start under ASLv2).

Patrick: That is why we switched to JCR. Jackrabbit seemed to be the most actively developed and complete JCR implementation, and since it was already released under ASLv2 it was not a very hard choice. A bonus for us is that two of our Brix developers are also Apache committers, and since Inertia is not shy about contributing to open source we saw an opportunity to improve Jackrabbit and give back to its community as we went along.

Q: Did your expectations regarding the JCR come true or did you have to overcome some difficulties you did not expect? Were there any pleasant or not so pleasant surprises after working with Jackrabbit for a while?

Matej: The most pleasant surprise was how much functionality Jackrabbit/JCR brought to the table: indexing, referential integrity, WebDAV, workspaces, versioning, and the list keeps going on. The API is consistent and easy to use.

Igor: Don’t forget our experience with Jackrabbit/JCR was not perfect. It seems that the modus operandi of the JCR community is to use a repository with a set small number of workspaces, which did not mesh well with our needs. Brix makes an extensive use of JCR workspaces for multi-site support, site snapshots, and publishing workflow. This means that we needed to create and delete workspaces on the fly as well as have an easy way to search across them via some index.

The first problem we ran into was that workspace creation event was not replicated across the Jackrabbit cluster. We have already patched Jackrabbit to support this feature, the patch is available in Jackrabbit’s issue tracker under JCR-1677.

Patrick: The second, and more severe, problem was the way Jackrabbit manages database connections. Currently each Jackrabbit workspace keeps an open connection to the database. We are planning to host over 300 client sites on Brix in the near future. Brix’s publishing workflow requires 3 workspaces per site: development, staging, and production. Without even counting workspaces created for snapshots, we are up to 900 workspaces and thus 900 open database connections. This simply does not scale. Unfortunately, this does not look like a high priority for the Jackrabbit community, because seeminly the most common usecase is a repository with a small number of workspaces. Nonetheless, we are working on a patch that would allow for connection pooling. A first pass of the patch is available in the Jackrabbit issue tracker under JCR-1456.

Q: One often discussed aspect of building content management systems on top of JCRs is the question if object-content-mapping should be used to wrap collections of JCR nodes into application-level objects. What is your view on this question?

Igor: Brix’s object model is very simple, so using some kind of object-content mapping would have been overkill for us. We do have a wrapper for the JCR api, but that was meant for other things (see below).

Q: In Brix you have wrapped JCR nodes and sessions in Brix' own wrapper classes. In the wrappers API calls are intercepted and events are generated. Why was this design decision made rather than using JCRs native Observation mechanisms?

Patrick: Our JCR API wrapper is two levels deep.

The bottom layer is an event layer. It provides events for before and after a change, as well as an ability to queue the events in a list and batch process them later. We also group and normalize the events so it is possible to know what is about to be changed before the change is applied. JCR’s native observation mechanism is limited to only observing changes that have already happened.

Matej: The top wrapper has two basic jobs:

Translate JCR’s checked exception model into unchecked while providing a pluggable and centralized way for exception handling strategies. In OOP JCR code is sprinkled among many classes, the checked exception model becomes extremely viral. Also, since a lot of exceptions are unrecoverable from Brix’s perspective there is very little point of having them be checked.

The second job of this wrapper is to allow node wrapping. Instead of using a full blown and complex object-content-mapping system we provide a simple mechanism for having nodes wrapped based on their brix:nodeType property. This makes it possible to write code like this:

Node n = session.getRootNode().getNode("page.html");
PageNode page = (PageNode) n;
page.setTitle("Page Title");
page.setMarkup("Page Content");

Q: Brix' WebDAV server has the nice feature that uploaded resources are automatically converted into Brix artifacts. How does that work?

Igor: This is done by intercepting node events using the aforementioned wrapper and modifying the nodes before they are saved based on rules defined in Brix. There is also some magic to handle varying WebDAV client behaviors, for example: when a new file is created via Coda it will first create an Untitled file and immediately rename it to its given name.

Q:If you had one wish regarding JCR and the JCR community, what would it be?

Patrick: We would wish the JCR community a very long and productive future.

Paul: I hope that the JCR community sees Brix as useful piece of software and contributes to its health and success.

As it was laid out in a previous post there are two iterations we need to go through before we release CQ5: the tech preview phase and the beta program. I am very happy to report that the tech preview has been successfully completed. By now, the beta program has started. I'm excited to see the shine in the developers eyes when they are allowed to take this baby into their hands.

The customer who completed the tech preview with us is now in production with CQ5. There are about 80 editors producing content already. This number is expected to increase into several hundred very soon. This is a fantastic success, but there are also other ("softer") factors that we have experienced during the tech preview project. Both, the editors as well as the developers perceive working with CQ5 to be very pleasant and exciting compared to the competing products they knew. I am extremely happy about this feedback as it goes to show that our efforts on user experience and software architecture were well invested.

There is also another interesting by-product of the beta program: by the time the program is finished most of our service and support team will have real-life experience with CQ5 from real-life customers in production. So from day one of the product launch there will be experienced service and support resources available from Day.

One essential question for determining a fitting integration strategy is if the db rows shall be exposed as resources (in the REST sense). Let us start with the case that they shall not be exposed as resources. In such a scenario a db row does not have its own URL. A typical Sling application in such a scenario would expose the JCR-based content as resources, but might add data from the db to the presentation of the resources.

On a technical level this could be implemented by creating an OSGi bundle that uses JDBC to connect to the database. The bundle would then be called by the rendering script through an OSGi service interface (as described here).

However, this approach cannot be used if the db rows shall be exposed as resources and have their own URL.

For data that changes on low frequencies one can create a JCR node for each db row. This requires the ability to keep the db and the JCR in sync. The JCR nodes could contain the primary key of the db data. Synching can be achieved by any programm capable of JCR and JDBC.

One seemingly obvious way to avoid having to synch is to create a dummy node that represents all of the db data. However, the resulting URLs would not be resource-oriented at all since you need to pass the resource identifier as a request parameter.

The full monty and best solution for making your db URLs RESTful is to create a Sling ResourceProvider. This would be an OSGi bundle that claims a part of the URL space in Sling and maps the URLs to resources of whatever kind. In the Sling samples there is an example implementation for file systems. In this example a tree of the local file system is mapped onto the URL space and all files are given Sling resource types. As such, the normal mechanisms for rendering requests are used. The same can be achieved with db rows, of course.

It should be noted that there is a generic problem when the relational data needs to be mapped onto the hierarchical URL space. You could go the brute force way of using /content/tablename/id. However, if your data is somehow categorized using many-to-one relationships consider "denormalizing" the data (mentally - you do not need to actually change the tables). This would yield nicer URLs like /content/category1/category2/data (e.g. like /content/wagons/mercedes/e-class).

But once in a while one can actually look at a previously hidden JCR application: a couple of days ago the JCR-based CMS "Brix" has been open-sourced. It is based on Wicket and Jackrabbit. From what I understand its origins are in online wine sales (note to self: next time ask for a content sample before posting).

In their own words:

Thanks for making this available, guys.