PlanetDay

Meta-Programming with Scala Part III: Partial function application

2008-08-27T22:35:06Z

In my previous post about Meta-Programming with Scala I suspected that there was no way to express partial function application in Scala’s type system. However Matt Hellige proofed me wrong in his comment.

His solution uses a trait for partially applying a function to some of its arguments. An abstract type exposed by the trait represents the resulting function which takes the remaining arguments.

object Partial {
  // Partial application of f2 to x
  trait papply[f2[_, _], x] {
    type f1[y] = f2[x, y]
  }

  // apply f to x
  type apply[f[_], x] = f[x]

  trait X
  trait Y
  trait F[A1, A2]

  // Test whether applying the partial application of
  // F to X to Y equals in the type F[X, Y]
  case class Equals[A >: B <: B, B]
  Equals[apply[papply[F, X]#f1, Y], F[X, Y]]
}

Having this solved we can define a type which encodes multiplication on the Church Numerals.

    trait curry[n[_[_], _], s[_]] {
      type f[z] = n[s, z]
    }

    // Multiplication for this encoding
    type mult[m[_[_], _], n[_[_], _], s[_], z] = m[curry[n, s]#f, z]

A full working example is available from my code page. Note, the code takes forever (i.e. some minutes) to compile. Matt also noted an issue with squares. With my version of the compiler (Ecipse plugin 2.7.2.r15874-b20080821120313) the issue does not show up however.

Flex Open Source? Right...

2008-08-26T11:50:13Z

There is a bug in the Flash player that makes it impossible to implement a RESTful architecture with a Flex or Flash backend. In essence, it is a buggy implementation of an http client - the client cannot get to the response body unless the reponse code is 200.. This is really bad. But what is an embarrassment for Adobe is that:

That bug was opened more than a year ago
It is marked as priority C only
It was closed and had to be reopened after the community complained
Adobe engineers suggested as a workaround to purchase LCDS Server

So, lessons learned (actually: lessons that proof in practice what was clear beforehand):

It does not matter too much if Flex is open-sourced as long as the Flash player is not
It does not matter at all if Flex or Flash are open-sourced if the community cannot apply patches and get them into the deployed player

Flex being open source is completely irrelevant.

[LOTD] JCR in the sky

2008-08-25T14:00:06Z

Storing data within the cloud (formerly known as SaaS, the elderly might even remember ASP) is the latest craze. If you want to jump on that bandwaggon or simply have a lot of files check out Scott Dietrich's Jackrabbit DataStore for Amazon S3. Scott has posted a link to the sources on the Jackrabbit user list.

Sling is an evolutionary step for WCMSs

2008-08-18T08:00:10Z

Communique 5 is currently being built on top of Apache Sling. I believe this is a step forward in the overall architectural evolution of web content management systems because it exposes a new type of interface to developers of content centric applications: the web infrastructure layer. Let me explain this idea in the context of CMS history:

1. The Big Blob

Seen from a developer's perspective early CMSs came along as one big piece of software . If you wanted to develop custom applications with them you usually had to develop on top of the CMS API. This API for WCMS would contain abstractions like "paragraph", "user" or "workflow" and expose actions like "publish a page", i.e. the abstraction level was the level of the WCMS business logic (I use the term "business logic" as in "the business of managing content"). For the purpose of this argument, let's call these APIs the "business level API".

2. Splitting off the repository

It was obvious that the content had to be also accessible for developers outside of the context of the business-layer API. Thus, the repository layer was abstracted: a number of CMSs created an API that allowed direct access to the content, either through JCR/JSR-170 or some proprietary API. This gave developers two choices how to access the content: either in the context of the business API or the "raw" access through, say, JCR.

Introducing the repository-level API can be seen as a clean architectural cut for a CMS in order to improve separation of concerns.

3. Splitting off the web infrastructure

In this next evolutionary step we take what is left of the WCMS after the repository is split off and separate off another layer: the web infrastrucure (Sling). This layer contains the basic plumbing required to build web apps, e.g. request handling, selecting content, script execution, authentication, filters etc. This leaves the business layer with the business logic only.

One effect of splitting off the web infrastructure is that developers can write web applications that sit "next to" the WCMS application rather than running on top of the WCMS. This means that these developers do not need to be concerned with (and learn about) the particular CMS business logic because they have access to the web application infrastructure provided by the CMS.

This is a second architectural cut and separation of concerns that acknowledges that there are different types of content centric applications and WCMS is only one of them (consider forums, blogs, media asset management, wikis, etc). These different content-centric apps share requirements for repositories and web infrastructure, though.

It should be noted that compared to the second step above this additional separation of concerns does not really allow you to do anything that was not possible before. However, it is such a big advancement in terms of developer productivity that I consider it to be a qualitative change. (If you disagree: Try building a web app starting with just a connection to your repository and another one on top of Sling. You will understand.)

Some clarifications

There are of course myriads of WCMSs already that let developers write extensions or modules or similar. That approach is different though because these applications run "within" the CMS context, i.e. the developer is still interfacing the business layer. On top of Sling, your app is a first-class citizen just like CQ5.

This CMS-oriented view of the world and its history might seem strange if you look from an application server angle: in the app server world the persistence layer, the application framework (Struts, PHP, ...) and the application were always neatly separated - so Sling would be nothing new. However, this is not quite the same situation as step 3 in the CMS world because in the classical web infrastructure world the application framework knows nothing about the underlying repository - usually it just handles database connection pooling or similar. Sling does know about the repository and thus it can provide additional functionality. Therefore, if you want to stay in the classical web infrastructure picture you could regard Sling as a very specialized application framework (specialized for content-centric applications). AFAIK this is a new piece of infrastructure that did not exist before.

Happy Birthday Madonna

2008-08-16T07:06:00Z

Today is a great day to celebrate a birthday…

Javamagazin

2008-08-14T08:30:13Z

If you happen to live in Germany, Switzerland or Austria (and speak German) make sure you do not miss this month's Javamagazin. The title story is "Was ist Content?" ("what is content?") and there is also an introductory article about JCR and Apache Jackrabbit.

The abstracts:

Was ist Content?

Bilder, Videos, Texte – man erkennt Content, wenn man ihn sieht. Aber hat Content charakteristische Eigenschaften, die ihn von reinen Daten unterscheiden? In Gesprächen über Content und Content-Management ist gelegentlich festzustellen, dass Content mit relationalen Datenbanken oder Dateisystemen gleich gesetzt wird. Obwohl Content letzten Endes oft im Dateisystem oder einer Datenbank gespeichert wird, wird eine Reduktion auf diese Low-Level-Sicht der Thematik nicht gerecht. Content ist eine spezielle Form von Daten, denen eine eigene Qualität innewohnt. Insofern ist auch Content-Management mehr als nur eine Datenbankmaske.
David Nüscheler, Michael Marth

JCR und Apache Jackrabbit

Der JCR-Standard (Content Repository API for Java) beschreibt eine klare Schnittstelle zwischen Anwendung und Content-Ablage. Das hierdurch definierte Content Repository stellt die dem Content typischen Eigenschaften und Funktionalitäten in standardisierter Weise zur Verfügung. Dies erleichtert erheblich die systemübergreifende Nutzung von Content und erschließt damit ein bisher nur mit viel Mühen auszuschöpfendes Potenzial.
Michael Marth, Gerd Handke, David Nüscheler, Carsten Ziegeler

Type-safe Builder Pattern in Java

2008-08-13T21:00:26Z

In this post I deviate a bit from the topic of my recent posts about Meta-Programming with Scala. I will have more to say about the latter topic in upcoming posts however.

Recently I read this rather fascinating post about a Type-safe Builder Pattern in Scala. When Heinz Kabutz mentioned the builder pattern in his latest issues of the The Java Specialists’ Newsletter I decided to try to come up with a type safe version for Java.

What I finally came up with is not strictly a builder but something I rather call an initializer. The initializer contains the initial state required by the target object. The state is accumulated within the initializer. Only when the state is complete can it be passed to the targets object’s constructor. Java’s type system prevents passing a initializer with an incomplete state to the target class’s constructor.

public class Foo {
  private final int a;
  private final int b;

  public Foo(Initializer initializer) {
    this(initializer.a, initializer.b);
  }

  private Foo(int a, int b) {
    super();
    this.a = a;
    this.b = b;
  }

  public String toString() {
    return "a = " + a + ", b = " + b;
  }

  public static class Initializer {
    private int a;
    private int b;

    private Initializer() {
      super();
    }

    private Initializer(int a, int b) {
      super();
      this.a = a;
      this.b = b;
    }

    public static Initializercreate() {
      return new Initializer();
    }

    public Initializer setA(int a) {
      this.a = a;
      return new Initializer(a, this.b);
    }

    public Initializer setB(int b) {
      this.b = b;
      return new Initializer(this.a, b);
    }

    static abstract class TRUE {}
    static abstract class FALSE {}
  }

}

The basic technique is the same as for the Type-safe Builder Pattern in Scala: the phantom types TRUE and FALSE are used to keep track of the state. Only a complete state will result in a Initialiter instance which subsequently can be passed to Foo’s constructor.

Here is how this is used:

public class Main {
  public static void main(String[] args) {
    Initializer initializer = Initializer.create();

    // Foo.create(initializer);           // won't compile
    // Foo.create(initializer.setB(1));   // won't compile
    // Foo.create(initializer.setA(1));   // won't compile

    Foo foo = new Foo(initializer.setA(1).setB(2));
    System.out.println(foo);
  }
}

Implementing OpenSearch in 10 Minutes

2008-08-12T13:30:08Z

One of the productivity-boosting features of JCR is the included search engine (which is Apache Lucene in case of Jackrabbit and CRX). This feature can be used to very quickly develop an OpenSearch interface for a Sling-based application.

I have recently provided a Sling example application called Notes which I want to use to demonstrate the implementation of OpenSearch. Import the Notes application into your CRX Quickstart (download Quickstart if you do not have it, yet). For importing navigate to http://localhost:7402/crx/index.jsp and start the Package Manager.

In OpenSearch search results can be in different formats. Let's start with producing results in XHTML. Add a new file at /apps/notes/opensearch.jsp. It shall contain:


<%@page import="javax.jcr.query.*, javax.jcr.*"%>
<%@taglib prefix="sling" uri="http://sling.apache.org/taglibs/sling/1.0"%>

<%@ include file="/apps/notes/util.jsp" %>
<%
String q = "/jcr:root/content/notes//*["+ SearchUtils.parameterToQuery(request.getParameter("qt")) +"]";
Query query = currentNode.getSession().getWorkspace().getQueryManager().createQuery(q, "xpath");
NodeIterator result = query.execute().getNodes();
%>


  
    Notes app search results
    
    
    
  
    < back to thread overview

    search results for query <%=request.getParameter("qt") %>, hits: <%=result.getSize()%>
<%@ include file="/apps/notes/searchBox.jsp" %>
<%
while(result.hasNext()) {
  Node n = result.nextNode();
  String t = n.getProperty("body").getValue().getString();
%>		
    
    <%=t.length() > 50 ? t.substring(0,50) + "..." : t%>
    <%=t%>
    
<%}%>

There is only a few OpenSearch-related lines like the link to the OpenSearch descriptor file (explained further below)

and the META tag that describes the result set:

The rest of this jsp is just using the standard JCR-provided query functionality to produce standard XHTML. That's it. This is your shiny new OpenSearch interface (I need to stress this once more because it's really cool: no search engine crawling or or other setup was needed. Neither was there any JCR-JSP-wiring or other web app configuration). Point your browser to: http://localhost:7402/content/notes.opensearch.html?qt=JCR to have a look (the request parameter qt denotes the query term).

The above mentioned descriptor file describes the app's search capabilities to external parties. Given the link element from above it needs to be in /apps/notes/opensearch and shall contain:



  Notes app
  Notes app search interface

The first URL element describes the XHTML-based output implemented discussed above. OpenSearch also allows RSS-based output which is described in the second URL-element. The implementation of RSS-based OpenSearch results is even more succinct because there is less boiler-plate code. Create the renderer file /apps/notes/opensearchrss.xml.jsp and let it contain:



<%@page import="javax.jcr.query.*, javax.jcr.*"%>
<%@taglib prefix="sling" uri="http://sling.apache.org/taglibs/sling/1.0"%>

<%@ include file="/apps/notes/util.jsp" %>
<%
String q = "/jcr:root/content/notes//*["+ SearchUtils.parameterToQuery(request.getParameter("qt")) +"]";
Query query = currentNode.getSession().getWorkspace().getQueryManager().createQuery(q, "xpath");
NodeIterator result = query.execute().getNodes();
%>

  Local Notes app
  http://localhost:7402/content/notes.html
  Search results for "<%=request.getParameter("qt") %>" at the local Notes app
  <%=result.getSize()%>
<%
while(result.hasNext()) {
	Node n = result.nextNode();
%>
  
    <% String t = n.getProperty("body").getValue().getString(); %>
    <%=t.length() > 50 ? t.substring(0,50) + "..." : t%>
    http://localhost:7402<%=n.getPath()%>.thread.html
    <%=t%>
  
<%}%>

Hit http://localhost:7402/content/notes.opensearchrss.xml?qt=JCR with your browser or any other RSS viewer to retrieve the search results in RSS format.

OK, this is simple to implement, but what is it good for? For example, recent browsers support autodiscovery of OpenSearch engines and allow users to add them to the browser's upper right search box. Simply add a link in the HTML page's header:

I have tested this in IE7 and FF2:

Search Suggestions

There is an enhancement suggested for upcoming versions of OpenSearch that would standardize search suggestions. In Firefox, however, a similar feature is already implemented. The third URL in the descriptor file describes the interface for search suggestions in FF (i.e. this will not work in IE). The corresponding implementation at /apps/notes/opensearchsuggestions.json.jsp looks like this:

<%@page import="javax.jcr.query.*, javax.jcr.*, java.util.*"%>
<%@taglib prefix="sling" uri="http://sling.apache.org/taglibs/sling/1.0"%>

<%@ include file="/apps/notes/util.jsp" %>
<%
// Hashmap to collect and count full words that contain the query term
TreeSet collector = new TreeSet();

// only accept min 2 char strings and does not end with a blank
if((request.getParameter("qt").length() > 1) && !(request.getParameter("qt").endsWith(" "))) {

// the actual search
String q = "/jcr:root/content/notes//*[" + SearchUtils.parameterToSuggestionQuery(request.getParameter("qt")) + "]";
Query query = currentNode.getSession().getWorkspace().getQueryManager().createQuery(q, "xpath");
NodeIterator result = query.execute().getNodes();

// the constant part of the suggestions
String constantPart = "";
String[] requestTokens = request.getParameter("qt").split("\\s");  
for (int x=0; x["<%=request.getParameter("qt")%>", [<%
boolean first = true;
Iterator results = collector.iterator();
while(results.hasNext()) { %>
  <%=!first?", ":""%>"<%=constantPart + results.next()%>"
  <% first = false; }%> ]]
<%} else { // if the query term is only 1 char long%>
["<%=request.getParameter("qt")%>",[]]
<%}%>

As the jsp's file name suggests the jsp returns json-formatted data. The returned data is a collection of suggestions based on the actual content (i.e. suggesting words that would produce hits). The format for a query "fi" must be of the form ["fi", ["firefox", "first", "fist"]]. Most of the code deals with String manipulation which is a bit verbose in Java. Also, I wanted to have multi-word queries like "jcr re" return suggestions for the last word only (i.e. ["jcr re", ["jcr repository", "jcr renderer"]]) which requires a couple of additional lines of code.

As usual, the full application including the complete sources is attached to this post in CRX Package Manager format.

Disappointed with Scriptaculous

2008-08-12T10:57:03Z

The Autocompleter looks nice, but why the heck does it send a POST to retrieve suggestions? I had to walk away from it.

Better specified than sorry

2008-08-07T19:29:02Z

Long time no posts here, thanks to Mento I have switched very much to link blogging, but if you happen to be interested in the stuff I write, feel free to follow me at FriendFeed.

Today I learnt an interesting fact about interaction with developers: if you as a product manager want a feature implemented, do not talk to the developer first, write a specification and a mockup first. If you skip writing the specification first (which can and will be changed afterwards) things can turn against you.

In our new web content management product we will have quite powerful support for tagging (that can be used in a folksonomy-like way, as categories, as a taxonomy and even as a thesaurus, as I said, quite powerful). I did specify the conceptual model, the content model, created some mockups (for another product that will use the tagging feature) and discussed with the developer how to implement the tagging user interface.

Together we refined the original idea I had for the user interface, given my general experience what works in AJAX web UI, and his more recent experience with the particular widget framework we are using, examined a set of model application that already implemented some of the metaphors used and came up with a simple, but powerful (and extensible) solution. I knew what the developer would do, the developer knew what he should do, everything was fine and I rushed to the next item in my busy schedule.

Then the unpredictable happened: Schedules were rearranged, priorities shifted, the developer switched to another task, went to vacation, schedules were rearranged again, priorities shifted again, another developer picked up - and knew nothing about our previous discussion and our common mental specification, so he started coming up with his own solution, because he could not use the specification I skipped writing, ran into problems and I had to troubleshoot.

Needless to say, the troubleshooting took more time than writing a helpful specification would have taken, and a written specification would have helped our initial concept discussion as I had to explain less and we could work in a more well-structured way.

Lesson learnt: better specify than sorry!

Germany’s most anticipated Web 2.0 sites - one year later

2008-08-06T08:49:11Z

I just came across a post about Germany's most anticipated Web2.0 sites that dates back to last August. Since this is almost exactly one year ago I decided to spend half an hour of my life and conduct a completely unscientific survey to see how many of these startups are still around. For this reason I simply looked at the web sites to see if they are still here and checked if there are recent posts in the company blog to check if the site is a zombie (if there is no blog I checked the press releases). The result is: 51% are still around, 36% are gone (or not started one year later). The rest seems shaky (last blog post 6 months ago) or unknown (state cannot be determined by looking at the blog).

If you consider that these startups were above the rest ("highly anticipated") and that checking if there are even online one year later is a very low barrier I was surprised by the low survival ratio. But one can regard this as positive: "quick failure" is a good strategy for testing ideas.

Hindsight is 20/20

2008-08-05T14:46:02Z

I have a fondness for retro computers, history of computing and famous discussions (like the Torvalds/Tanenbaum debate) . In this context I just came across this W3C mailing list debate regarding REST and SOAP. Nice read.

Roy Fielding's OSCON talk about Open Architecture

2008-08-05T13:00:11Z

Day's Chief Scientist Roy Fielding gave a talk about Open Architecture at this year's OSCON. Roy specifically looks at Peyman Oreizy's thesis titled "Open Architecture Software: A Flexible Approach to Decentralized Software Evolution". Find the slides below:

git clones of Apache codebases

2008-07-31T10:33:05Z

In the past few months we’ve been discussing on the infrastructure-dev mailing list about various ways of extending or improving the version control functionality available to Apache projects. One of the main themes of the discussion has been making it easier to access Apache codebases using git or other distributed SCM tools.

The new svn.eu.apache.org mirror that was recently announced supports git-svn use when accessed as an authenticated user over https. Unfortunately that access is limited to Apache committers and git-svn can be notoriously slow when making initial clones of complex codebases in svn.

To work around these issues I set up a collection of git mirrors of selected Apache codebases on my server. You can find these unofficial mirrors at http://jukka.zitting.name/git/. The mirrors are automatically updated daily.

The mirrors work pretty much like normal git repositories in that you don’t need git-svn or any other svn tools to work with them. The only significant difference to normal git repositories is that svn tags are mapped to branches named “tags/…” in the mirrors due to the different way git and svn handle tags. Also, settings like svn:ignore, svn:eol-style, etc. are not replicated in these git mirrors.

Let me know if you’re interested in seeing other Apache codebases mirrored. I’m also interested in other feedback or ideas related to these git mirrors.

Locales vs. consistent builds

2008-07-31T09:11:56Z

[MOJO-1193] Locale support for formatting dates and times

I wanted to use the Maven buildnumber plugin to generate a date string like “31-Jul-2008″ for inclusion in an automatically generated release notes document. This way I could minimize the amount of manual tweaking I need to do when publishing the release.

This was easy enough (though quite verbose) to achieve:

    
      org.codehaus.mojo
      buildnumber-maven-plugin
      
        {0,date,dd-MMM-yyyy}
        
          timestamp
        
      
      
        
          validate
          
            create

Unfortunately, the resulting date string would be the expected “31-Jul-2008″ on one computer, but “31-heinä-2008″ (in Finnish) on my laptop. Instead of changing the settings of my laptop I wanted to explicitly set the locale that the buildnumber plugin should use when formatting the date. I wasn’t too surprised that the required functionality didn’t yet exist (it’s quite common for people to forget about locales), but luckily it wasn’t too hard to add. Thus MOJO-1193.

With my patch the required configuration becomes:

      
        {0,date,dd-MMM-yyyy}
        en_US
        
          timestamp

Consistency is good.

QOTD - about dwarfs

2008-07-31T07:45:54Z

Being considered the best speaker in a computer science department is like being known as the tallest of the Seven Dwarfs.

This is from Randy Pausch The Last Lecture book, I got it yesterday and find it really hard to put it down.

CQ5 status update: tech preview completed

2008-07-31T00:00:01Z

Here's a brief update on Communique 5's delivery: we are in excellent shape and the quality gated approach is paying of for our customers.

As it was laid out in a previous post there are two iterations we need to go through before we release CQ5: the tech preview phase and the beta program. I am very happy to report that the tech preview has been successfully completed. By now, the beta program has started. I'm excited to see the shine in the developers eyes when they are allowed to take this baby into their hands.

The customer who completed the tech preview with us is now in production with CQ5. There are about 80 editors producing content already. This number is expected to increase into several hundred very soon. This is a fantastic success, but there are also other ("softer") factors that we have experienced during the tech preview project. Both, the editors as well as the developers perceive working with CQ5 to be very pleasant and exciting compared to the competing products they knew. I am extremely happy about this feedback as it goes to show that our efforts on user experience and software architecture were well invested.

There is also another interesting by-product of the beta program: by the time the program is finished most of our service and support team will have real-life experience with CQ5 from real-life customers in production. So from day one of the product launch there will be experienced service and support resources available from Day.

Ask the community: Brix

2008-07-31T00:00:01Z

Recently, I blogged about Brix, which is a new open source cms based on Apache Wicket and Jackrabbit. Brix has been developed by the Inertia Beverage Group. Inertia's Chief Strategy Officer Paul Mabray sat down with the dev team and passed on my questions about JCR and Jackrabbit:

Paul Mabray: So tell me a bit about each of you - your background, hobbies, etc.

Matej Knopp: Hi, I am Matej. I’m a key independent consultant from Slovakia. My number one project for Inertia is working on Brix and enjoying every minute of it. I’m also an Apache Wicket enthusiast and committer. Obviously I have a severe lack of free time.

Paul: Do you do anything else?

Matej: Whatever little time I find away from the keyboard I try to read as much as I can.

Patrick Angeles: I guess, aside from you Paul, I am the veteran at Inertia. I am the VP of Technology and lead our SaaS development for a software solution that connects wineries to their customers. I’ve been in tech for over 10 years working for big companies like Bankers Trust and Acquire Systems. Food, wine and my family are the only things I have time to do outside of work.

Igor Vaynberg: I’m the Principal Software Engineer at Inertia, residing in sunny Sacramento, California. I’m the guy your brought in to help build the next-generation enterprise web application that help will more easily bridge the gap between wineries and their customers.

Paul: Tell me more

Igor: Well, my love for writing software was sparked when I received a Sinclair Z80 on my tenth birthday and I’ve been doing it ever since. I do have the same hobbies as you such as playing with my kids, taking my beautiful wife to fancy dinners - that should get me points at home - and fragging my friends at Halo. I also still find time to work on software. My number one outlet is the Apache Wicket framework - I really believe to be the best way to build web applications.

Paul: And I am the founder of Inertia and now Chief Strategy Officer in charge of product development and business development. Just like Patrick I only have time for the family outside of work. But I love what I do and when this team brought forth the concept to make Brix open source, I was 200% in support of the initiative.

Igor/Patrick: COME ON

Paul: OK, I frag a few people on Halo when I get a spare minute.

Paul: So Michael asked us a question - Brix uses Apache Jackrabbit as its content repository. Back in the days when you made this decision what potential choices were you considering and what influenced your decision towards Jackrabbit?

Igor: From the start, one of our major requirements was the ability for our users to edit content using their favorite desktop editor. We picked SVN and JCR as two possible solutions: SVN with its local check-in/check-out, and JCR with its WebDAV support. We started off with SVN because it had more powerful versioning features, such as merging. But we quickly ran into features we needed that were missing and would be too expensive to implement ourselves: referential integrity and ASLv2 java connector library being the two that jump to mind (we were planning to open-source Brix from the start under ASLv2).

Patrick: That is why we switched to JCR. Jackrabbit seemed to be the most actively developed and complete JCR implementation, and since it was already released under ASLv2 it was not a very hard choice. A bonus for us is that two of our Brix developers are also Apache committers, and since Inertia is not shy about contributing to open source we saw an opportunity to improve Jackrabbit and give back to its community as we went along.

Q: Did your expectations regarding the JCR come true or did you have to overcome some difficulties you did not expect? Were there any pleasant or not so pleasant surprises after working with Jackrabbit for a while?

Matej: The most pleasant surprise was how much functionality Jackrabbit/JCR brought to the table: indexing, referential integrity, WebDAV, workspaces, versioning, and the list keeps going on. The API is consistent and easy to use.

Igor: Don’t forget our experience with Jackrabbit/JCR was not perfect. It seems that the modus operandi of the JCR community is to use a repository with a set small number of workspaces, which did not mesh well with our needs. Brix makes an extensive use of JCR workspaces for multi-site support, site snapshots, and publishing workflow. This means that we needed to create and delete workspaces on the fly as well as have an easy way to search across them via some index.

The first problem we ran into was that workspace creation event was not replicated across the Jackrabbit cluster. We have already patched Jackrabbit to support this feature, the patch is available in Jackrabbit’s issue tracker under JCR-1677.

Patrick: The second, and more severe, problem was the way Jackrabbit manages database connections. Currently each Jackrabbit workspace keeps an open connection to the database. We are planning to host over 300 client sites on Brix in the near future. Brix’s publishing workflow requires 3 workspaces per site: development, staging, and production. Without even counting workspaces created for snapshots, we are up to 900 workspaces and thus 900 open database connections. This simply does not scale. Unfortunately, this does not look like a high priority for the Jackrabbit community, because seeminly the most common usecase is a repository with a small number of workspaces. Nonetheless, we are working on a patch that would allow for connection pooling. A first pass of the patch is available in the Jackrabbit issue tracker under JCR-1456.

Q: One often discussed aspect of building content management systems on top of JCRs is the question if object-content-mapping should be used to wrap collections of JCR nodes into application-level objects. What is your view on this question?

Igor: Brix’s object model is very simple, so using some kind of object-content mapping would have been overkill for us. We do have a wrapper for the JCR api, but that was meant for other things (see below).

Q: In Brix you have wrapped JCR nodes and sessions in Brix' own wrapper classes. In the wrappers API calls are intercepted and events are generated. Why was this design decision made rather than using JCRs native Observation mechanisms?

Patrick: Our JCR API wrapper is two levels deep.

The bottom layer is an event layer. It provides events for before and after a change, as well as an ability to queue the events in a list and batch process them later. We also group and normalize the events so it is possible to know what is about to be changed before the change is applied. JCR’s native observation mechanism is limited to only observing changes that have already happened.

Matej: The top wrapper has two basic jobs:

Translate JCR’s checked exception model into unchecked while providing a pluggable and centralized way for exception handling strategies. In OOP JCR code is sprinkled among many classes, the checked exception model becomes extremely viral. Also, since a lot of exceptions are unrecoverable from Brix’s perspective there is very little point of having them be checked.

The second job of this wrapper is to allow node wrapping. Instead of using a full blown and complex object-content-mapping system we provide a simple mechanism for having nodes wrapped based on their brix:nodeType property. This makes it possible to write code like this:

Node n = session.getRootNode().getNode("page.html");
PageNode page = (PageNode) n;
page.setTitle("Page Title");
page.setMarkup("Page Content");

Q: Brix' WebDAV server has the nice feature that uploaded resources are automatically converted into Brix artifacts. How does that work?

Igor: This is done by intercepting node events using the aforementioned wrapper and modifying the nodes before they are saved based on rules defined in Brix. There is also some magic to handle varying WebDAV client behaviors, for example: when a new file is created via Coda it will first create an Untitled file and immediately rename it to its given name.

Q:If you had one wish regarding JCR and the JCR community, what would it be?

Patrick: We would wish the JCR community a very long and productive future.

Paul: I hope that the JCR community sees Brix as useful piece of software and contributes to its health and success.

Meta-Programming with Scala Part II: Multiplication

2008-07-30T21:07:57Z

This was sitting on my desk for quite a while now while I was busy with other things. Finally I came around to write things up.

In my last post I showed how to encode the Church numerals and addition on them with Scala’s type system. I mentioned that this approach does not seem to scale. In this post I show the problems I faced while I tried to extend the approach to multiplication. This particular example shows that Scala’s type system does not seem to support partial function application which in general is crucial for defining more complex functions base on simpler ones. But before delving into Scala, let’s review the church numerals in lambda calculus.

Define a lambda term for each natural number n

where stands for the n-fold application of s. Think of s standing for successor and z for zero. Then the number n is simply the n-fold successor of zero.

Addition can now be define as

Here the n-fold successor of zero is used as zero element for m. Again this can be thought of as the m-fold successor of the n-fold successor of zero.

Multiplications is similar but instead of using a different value for zero, a different successor function is used:

This boils down to the m-fold application of the n-fold successor function.

As in my previous post the Church numerals are encoded in Scala’s type system as follows

type _0[s[_], z] = z
type _1[s[_], z] = s[z]
type _2[s[_], z] = s[s[z]]
type _3[s[_], z] = s[s[s[z]]]
type _4[s[_], z] = s[s[s[s[z]]]]

I did not succeed encoding multiplication however. The most straight forward encoding - which closely follows the one for addition - looks like this

type mul[m[s[_], z], n[s[_], z], s[_], z] = m[n[s[_], z], z]

However the Scala compiler complains:

  s[_$1] forSome { type _$1 } takes no type parameters,
  expected: one

Unlike addition, we need to pass a partially applied function here - namely the n-fold successor function. Since the above does not work, I’m not sure if there even is a syntax for expressing partial function application in Scala’s type system.

In another approach I tried to introducing a formal parameter for the n-fold successor function:

type mul[m[n[s[_], z], z], n[s[_], z], s[_], z] = m[n, z]

The compiler does not complain here which is encouraging. However this breaks on application

abstract class Zero
abstract class Succ[T]

type zero = mul[_0, _0, Succ, Zero]
type one = mul[_1, _3, Succ, Zero]

The first application results in a kind error

the kinds of the type arguments (Meta._0,Meta._0,Meta.Succ,Meta.Zero) do not conform to
the expected kinds of the type parameters (type m,type n,type s,type z).
Meta._0's type parameters do not match type m's expected parameters:
type s has one type parameter, but type n has two

The second application causes an assertion failure of the Scala compiler (see Ticket #742).

For a better understanding let’s analyze what mul[_1, _3, Succ, Zero] expands to:

mul[_1, _3, Succ, Zero] =
_1[_3, Zero] =
_3[Zero]

While this looks somewhat correct, it is certainly not. _3 takes two arguments but gets one. This is exactly what the Scala compiler was complaining about.

Sling Applications and Relational Databases

2008-07-30T00:00:01Z

One topic that periodically comes up is how to integrate data that resides relational databases with a Sling application. The use case might be that there is legacy data that cannot be migrated but needs to displayed in a new (Sling-based) web application. In this post I would like to discuss some aspects and possible strategies to accomplish this goal.

One essential question for determining a fitting integration strategy is if the db rows shall be exposed as resources (in the REST sense). Let us start with the case that they shall not be exposed as resources. In such a scenario a db row does not have its own URL. A typical Sling application in such a scenario would expose the JCR-based content as resources, but might add data from the db to the presentation of the resources.

On a technical level this could be implemented by creating an OSGi bundle that uses JDBC to connect to the database. The bundle would then be called by the rendering script through an OSGi service interface (as described here).

However, this approach cannot be used if the db rows shall be exposed as resources and have their own URL.

For data that changes on low frequencies one can create a JCR node for each db row. This requires the ability to keep the db and the JCR in sync. The JCR nodes could contain the primary key of the db data. Synching can be achieved by any programm capable of JCR and JDBC.

One seemingly obvious way to avoid having to synch is to create a dummy node that represents all of the db data. However, the resulting URLs would not be resource-oriented at all since you need to pass the resource identifier as a request parameter.

The full monty and best solution for making your db URLs RESTful is to create a Sling ResourceProvider. This would be an OSGi bundle that claims a part of the URL space in Sling and maps the URLs to resources of whatever kind. In the Sling samples there is an example implementation for file systems. In this example a tree of the local file system is mapped onto the URL space and all files are given Sling resource types. As such, the normal mechanisms for rendering requests are used. The same can be achieved with db rows, of course.

It should be noted that there is a generic problem when the relational data needs to be mapped onto the hierarchical URL space. You could go the brute force way of using /content/tablename/id. However, if your data is somehow categorized using many-to-one relationships consider "denormalizing" the data (mentally - you do not need to actually change the tables). This would yield nicer URLs like /content/category1/category2/data (e.g. like /content/wagons/mercedes/e-class).

Red Red Wine (JCR Edition)

2008-07-28T00:00:01Z

David Nuescheler has a slide that compares JCR-based applications to an iceberg (see e.g here, first presentation, slides 18 and 19). No, not in the sense that it will sink the Titanic, but in the sense that the visible part is only a small fraction. Well, proving things that are hidden by definition is often a bit tricky. In the case of JCR-based apps one can refer to the traffic on the Jackrabbit user list or the number of Jackrabbit downloads.

But once in a while one can actually look at a previously hidden JCR application: a couple of days ago the JCR-based CMS "Brix" has been open-sourced. It is based on Wicket and Jackrabbit. From what I understand its origins are in online wine sales (note to self: next time ask for a content sample before posting).

In their own words:

Using Apache Wicket as the technology to serve the content makes it very easy to embed custom, stateful Wicket components into any CMS page, allowing rich integration with existing Wicket web application. Using Apache Jackrabbit allows Brix to easily integrate full text search, versioning, and WebDav access.

Thanks for making this available, guys.

R.I.P. Randy Pausch

2008-07-26T13:24:55Z

In French we say “the best often leave first”. I’m not teaching in schools anymore these days, but if I was I would watch the last lecture with my students.

Update: here’s another shorter video recorded little more than two months ago at Carnegie Mellon’s Commencement - I guess that one sums it all. Via Tristan Nitot.

GHOP Grand Prize Winners @google

2008-07-26T12:19:17Z

Two weeks ago I was at Google headquarters to represent the ASF for the GHOP Awards Ceremony.

Visiting the Googleplex was very interesting as an example of a modern working environment, free of most or all corporate annoyances that used to be present in too many companies.

The legends about the great free food are not exaggerated, Cafe MOMA and Charlie’s Cafe lived more than lived up to my expectations. To me the Googleplex looks like a environment that would appeal very much to young engineers, as you can more or less live on site, with food, laundry, massage, swimming pool, Friday afternoon music, TGIF and other perks. If you have a family, on the other hand, you might need some discipline to escape the campus at the appropriate time…

In the afternoon of the awards ceremony day, we were treated to great jetlag-proof presentations on AppEngine (by Guido Van Rossum, Mr.Python himself), Android by Romain Guy (nice to see a young French guy make his way there), GFS/BigTable by Jeff Dean (employee #20) and testing by Bharat Mediratta and Mike Bland (readers know how passionate I am about testing, and these guys seem to have a similar vision than mine, so /me happy).

Great stuff, and all this for ten students, their parents and mentors! Big thanks to Leslie Hawthorn, Chris DiBona and team for setting this up so nicely.

Here’s the (shortish) interview with the ASF’s Grand Prize winner, Spencer Davis. For me, meeting these bright young folks has been the highlight of all this, and the motivation to continue contributing to these student programs. Seems like there is hope for humanity after all ;-)

Sandy Armstrong, mentor for GNOME, and Peter Cawley, winner for Drupal, have more details on the awards ceremony day. The “exotic” ASF presentation that I mentioned happened in Kathmandu, Nepal, as mentioned in a previous post.

Hey, El Reg - Microsoft is not becoming an ASF member!

2008-07-26T08:46:51Z

Contrary to what The Register reports, Microsoft is not paying $100,000 annual membership to the Apache Software Foundation.

ASF membership cannot be bought: people earn their individual membership by merit, and there’s no such thing as ASF member companies.

As with any other sponsor of the ASF, Microsoft’s sponsorship only means that they’re giving money to the ASF, money that the ASF can use freely, as the ASF does not accept directed donations.

I am very pleased to see this happening. It won’t make me love Microsoft’s current products much more (although, as my son notes, the XBox is a nifty piece of kit), but it is great to see more and more people inside Microsoft understand the importance of open source in today’s IT landscape.

Update: as I write this, the Google Number for +apache +microsoft +sponsor* is 682,000

Update: it’s been at least 24 hours since I posted a comment to the Register article linked above, indicating their error. Surprisingly, that comment hasn’t been moderated in yet…

Welcome, Microsoft!

2008-07-25T19:46:58Z

Microsoft just announced their platinum level sponsorship of the Apache Software Foundation.

history.forward(), indeed!

Fluid userstyle for FriendFeed take two

2008-07-24T13:32:35Z

The topic of user styles for Fluid came up on FriendFeed recently and Alexander Kucera asked me to share my userstyle I am using for FriendFeed. Here you are:

Userstyle

The result looks like this:

The black background is inspired by Twitteriffic and allows for using a semi-transparent window, which does not look as good with a white background. Additionally I stripped everything from the display that is not feed, including the sidebar, the share-dialog, the friendfeed logo and so on. I am using a dark grey logo for the Fluid application that fits the background of the SSB window.

Screencast: JCR Connector for MS Sharepoint

2008-07-24T00:00:01Z

Here's a 3 minute screencast about Day's JCR Connector for MS Sharepoint (which makes available Sharepoint servers as JCR-compliant repositories). If you need your Java apps to retrieve content from your corporation's Sharepoint servers: this is for you.

Sling Cheat Sheet

2008-07-22T00:00:01Z

My approach to learning a new technology (or-relearning something that changed several times since I last spend time with it) is writing documentation. As I wanted to write a small Sling application using the newly released Apache Sling 1.0, I decided to create a one-page documentation of the Script-resolution process in Sling. This is the frontside of my cheat sheet. It shows you how to get from HTTP request to content node, from content node to resource type, from resource type to script and what scripting variables are available.

On the flipside of the cheat sheet I documented all the hidden, but powerful request parameters you can use when dealing with the SlingPostServlet, the default handler for all POST requests that gives you endless options for creating, modifying, deleting, copying and moving nodes in your repository.

Click on the thumbnails to get a high-resolution PDF version of either side (3,7 and 1,7 MB) or download the PDF version (300 KB) here.

Gmail superstars

2008-07-16T13:31:44Z

From the very useful department: the Gmail superstars option augments the message starring feature with different colors and shapes.

Very useful if like me you keep your (small of course) to-do list in your inbox.

Craviotto drums - wow

2008-07-16T09:01:46Z

In my quest for a new snare drum last week in Silicon Valley, I ended up at Gelb Music, where Don suggested that I try his favorite one, the Craviotto one-ply snare (nice store by the way, friendly staff and lots of instruments and accessories to try out).

What a drum - I couldn’t afford it as I had other stuff to buy as well, but the sound quality and dynamics of this snare are absolutely amazing. It was not that expensive, about $700 if I remember correctly, which is much less than I would pay here in Switzerland for anything similar. Craviotto also makes complete drumsets, but the only price indication that I could find by browsing around is $8000 for a four-piece kit, which is not exactly in my price range at this time.

In the end I got a nice sounding Yamaha maple snare, it’s miles better than my current 20-years old brass one and was only a bit over the budget that I had initially set. I guess life has to be made of compromises, and this is a more than decent one.