PlanetDay

CMS vendors now and then

2010-03-12T16:56:58Z

CMS analyst Janus Boye has just published a post on CMS vendors that discontinue their products (because they get bought out or similar)

During the past 10 years, a number of software products used by online professionals have been discontinued

That sentence reminded me that I had given a talk almost 10 years ago (it was in 2001 exactly) that contained a slide on the CMS market at that time:

The circles denote vendors that were part of CMS market overview articles by popular German IT magazines in that year (I wanted to show how differently the market place could be perceived). A vendor placed in any of the circles had enough attention to be part of at least one evaluation. The vendors outside of the circles were not part of any of these overview articles, but somehow present in the market place - at least I knew their names back then.

It is interesting to look at the landscape from that time. Of course there are a number of well-known vendors that got bought (Vignette, Obtree, Gauss), but the majority still seems to linger on - at least, a web site still exists, for example iRacer, Schema Text, or Contens.

On the other hand, one can ask how many vendors that were important enough to make it into a (German) market overview are still relevant in the market place today. I have used Janus Boye's spreadsheet of relevant European CMS vendors as a benchmark and checked which vendor's of today's list were already in 2001's presentation: Day, Coremedia and Open Text were "in the circles". Tridion was there, but outside of the circles. The rest of the vendors that Janus considers relevant today were not on my radar in 2001.

The end of my presentation involved a couple of CMS-related predictions. Let's see how I did. I predicted:

product borders between CMS, DMS and app servers will blur further - my take now: wrong. I do not think that these border are more blurry than they were in 2001
more standards and standards-based software (Java, JSP/ASP, XML, XSL) - true. The underlying technologies of CMSs are more homogeneous than they were at that time. Remember TCL?
But no true compatibility. True. Nothing more to say.
Improved Personalization. Improved Multi-Channel support. Both not really true, but rather fads of those days.
Improved DMS features and Office integration. Don't ask me why I said that.
No quick market consolidation in sight. Right on the money here.

Mostly correct on general market considerations, mostly wrong on features.

The new BASIC

2010-01-29T22:40:07Z

I’m seeing many posts that worry about computing devices like iPhones and the new iPad preventing people from having direct control over the hardware. Mark is telling us about a Ctrl+Reset and a BASIC prompt. Nowadays you get started with the following on an HTML page:

And you can do anything! Don’t tell me the days of tinkering are over.

Scala type level encoding of the SKI calculus

2010-01-29T17:58:41Z

In one of my posts on type level meta programming in Scala the question of Turing completeness came up already. The question is whether Scala’s type system can be used to force the Scala compiler to carry out any calculation which a Turing machine is capable of. Various of my older posts show how Scala’s type system can be used to encode addition and multiplication on natural numbers and how to encode conditions and bounded loops.

Motivated by the blog post More Scala Typehackery which shows how to encode a version of the Lambda calculus which is limited to abstraction over a single variable in Scala’s type system I set out to further explore the topic.

The SKI combinator calculus

Looking for a calculus which is relatively small, easily encoded in Scala’s type system and known to be Turing complete I came across the SKI combinator calculus. The SKI combinators are defined as follows:

,
,
.

They can be used to encode arbitrary calculations. For example reversal of arguments. Let . Then

Self application is used to find fixed points. Let for some combinator . Then . That is, is a fixed point of . This can be used to achieve recursion. Let be the reversal combinator from above. Further define

for some combinator and
.

That is, combinator is the combinator obtained by applying its argument to the combinator . (There is a bit of cheating here: I should actually show that such combinators exist. However since the SKI calculus is Turing complete, I take this for granted.) Now let be in from above (That is we have now). Then

and by induction

Type level SKI in Scala

Encoding the SKI combinator calculus in Scala’s type system seems not too difficult at first. It turns out however that some care has to be taken regarding the order of evaluation. To guarantee that for all terms which have a normal form, that normal form is actually found, a lazy evaluation order has to be employed.

Here is a Scala type level encoding of the SKI calculus:

trait Term {
  type ap[x <: Term] <: Term
  type eval <: Term
}

// The S combinator
trait S extends Term {
  type ap[x <: Term] = S1[x]
  type eval = S
}
trait S1[x <: Term] extends Term {
  type ap[y <: Term] = S2[x, y]
  type eval = S1[x]
}
trait S2[x <: Term, y <: Term] extends Term {
  type ap[z <: Term] = S3[x, y, z]
  type eval = S2[x, y]
}
trait S3[x <: Term, y <: Term, z <: Term] extends Term {
  type ap[v <: Term] = eval#ap[v]
  type eval = x#ap[z]#ap[y#ap[z]]#eval
}

// The K combinator
trait K extends Term {
  type ap[x <: Term] = K1[x]
  type eval = K
}
trait K1[x <: Term] extends Term {
  type ap[y <: Term] = K2[x, y]
  type eval = K1[x]
}
trait K2[x <: Term, y <: Term] extends Term {
  type ap[z <: Term] = eval#ap[z]
  type eval = x#eval
}

// The I combinator
trait I extends Term {
  type ap[x <: Term] = I1[x]
  type eval = I
}
trait I1[x <: Term] extends Term {
  type ap[y <: Term] = eval#ap[y]
  type eval = x#eval
}

Further lets define some constants to act upon. These are used to test whether the calculus actually works.

trait c extends Term {
  type ap[x <: Term] = c
  type eval = c
}
trait d extends Term {
  type ap[x <: Term] = d
  type eval = d
}
trait e extends Term {
  type ap[x <: Term] = e
  type eval = e
}

Eventually the following definition of Equals lets us check types for equality:

case class Equals[A >: B <:B , B]()

Equals[Int, Int]     // compiles fine
Equals[String, Int] // won't compile

Now lets see whether we can evaluate some combinators.

  // Ic -> c
  Equals[I#ap[c]#eval, c]

  // Kcd -> c
  Equals[K#ap[c]#ap[d]#eval, c]

  // KKcde -> d
  Equals[K#ap[K]#ap[c]#ap[d]#ap[e]#eval, d]

  // SIIIc -> Ic
  Equals[S#ap[I]#ap[I]#ap[I]#ap[c]#eval, c]

  // SKKc -> Ic
  Equals[S#ap[K]#ap[K]#ap[c]#eval, c]

  // SIIKc -> KKc
  Equals[S#ap[I]#ap[I]#ap[K]#ap[c]#eval, K#ap[K]#ap[c]#eval]

  // SIKKc -> K(KK)c
  Equals[S#ap[I]#ap[K]#ap[K]#ap[c]#eval, K#ap[K#ap[K]]#ap[c]#eval]

  // SIKIc -> KIc
  Equals[S#ap[I]#ap[K]#ap[I]#ap[c]#eval, K#ap[I]#ap[c]#eval]

  // SKIc -> Ic
  Equals[S#ap[K]#ap[I]#ap[c]#eval, c]

  // R = S(K(SI))K  (reverse)
  type R = S#ap[K#ap[S#ap[I]]]#ap[K]
  Equals[R#ap[c]#ap[d]#eval, d#ap[c]#eval]

Finally lets check whether we can do recursion using the fixed point operator from above. First lets define .

  // b(a) = S(Ka)(SII)
  type b[a <: Term] = S#ap[K#ap[a]]#ap[S#ap[I]#ap[I]]

Further lets define some of the s from above.

trait A0 extends Term {
  type ap[x <: Term] = c
  type eval = A0
}
trait A1 extends Term {
  type ap[x <: Term] = x#ap[A0]#eval
  type eval = A1
}
trait A2 extends Term {
  type ap[x <: Term] = x#ap[A1]#eval
  type eval = A2
}

Now we can do iteration on the type level using a fixed point combinator:

  // Single iteration
  type NN1 = b[R]#ap[b[R]]#ap[A0]
  Equals[NN1#eval, c]

  // Double iteration
  type NN2 = b[R]#ap[b[R]]#ap[A1]
  Equals[NN2#eval, c]

  // Triple iteration
  type NN3 = b[R]#ap[b[R]]#ap[A2]
  Equals[NN3#eval, c]

Finally lets check whether we can do ‘unbounded’ iteration.

trait An extends Term {
  type ap[x <: Term] = x#ap[An]#eval
  type eval = An
}
// Infinite iteration: Smashes scalac's stack
  type NNn = b[R]#ap[b[R]]#ap[An]
  Equals[NNn#eval, c]

Well, we can

$ scalac SKI.scala
Exception in thread "main" java.lang.StackOverflowError
        at scala.tools.nsc.symtab.Types$SubstMap.apply(Types.scala:3165)
        at scala.tools.nsc.symtab.Types$SubstMap.apply(Types.scala:3136)
        at scala.tools.nsc.symtab.Types$TypeMap.mapOver(Types.scala:2735)

Filed under: Uncategorized Tagged: Meta-Programming, Scala

[LOTD] Standards Diagram for Content Management

2010-01-13T00:00:01Z

Following up on Jon Marks' post on standards relevant for content management Justin Cormack has put together a "Standards Diagram for Content Management" Prezi landscape. Nice work!

The part "structuring" in Justin's presentation contains Docbook and DITA. Theresa compared these two standards a while ago:

| View | Upload your own

Declarative Services: Configuration

2010-01-13T00:00:01Z

OSGi Declarative Services components are configured by properties defined by the component developer in the component descriptor and by configuration managed by system administrators using the OSGi Configuration Admin Service. The combined set of properties is traditionally made available to the component as a Dictionary calling the ComponentContext.getProperties() method. The component context is provided to the activate method called when the component is activated.

This makes configuration of components very simple, since the component itself does not have to care where configuration comes from and how it is maintained. In addition, a component always knows it is starting from scratch when the activate method is called. This is because a component instance is never reused and a component reconfiguration means the component is deactivated and activated with the new configuration.

There are some drawbacks to this solution, though:

Reactivation of a component may be an expensive operation. Particularly if the component provides a service which is heavily used and relied upon, such as for example the SlingRepository service in Communiqu&eactue 5.
If a component is keeping internal state, e.g. gathering statistics, reactivation of the component will cause loss of this state.

This is where the new Declarative Services specification version 1.1 kicks in and improves much. First of all, configuration may be updated dynamically without reactivating the component. Second a component may declare itself as not needing configuration or even as requiring configuration thus only activating the component if configuration is actually available from the Configuration Admin service.

To use dynamic reconfiguration you have to declare a method to take this updated configuration in the modified attribute of the component element. When using the Apache Felix Maven SCR Plugin, use the modified attribute of the @scr.component tag:

 @scr.component modified="modified"

and define a method taking the configuration, for example:

 private void modified(Map config) {
     // apply the configurtion dynamically
 }

Unless the configuration may cause the component's references to be modified the component configuration is now dynamically provided without reactivating the component.

The method defined in the modified attribute has the same requirements as the activate method: It may be have any access modifier (but should not be public) and it may take any combination of ComponentContext, BundleContext, and Map arguments. Of course for a modified the primary useful type Map as in the example above<.

If you know your component cannot be configured or if you absolutely need configuration of your component, you can declare this desire using the configuration-policy attribute of the component element (or the policy attribute of the @scr.component JavaDoc tag):

optional: Configuration from the Configuration Admin Service is provided to the component if available. This is the default setting and is the same as in the previous Declarative Services specification.
require: Configuration from the Configuration Admin Service is required for the component to be activated. If the configuration is deleted, the component will be deactivated. This setting allows for a component to be controlled by the existence of configuration.
ignore: Configuration from the Configuration Admin Service is never retrieved on behalf of and provided to the component. If your component has no configurable properties, this setting makes sense.

Note: To use the functionality described in this article, you have to use a Declarative Services 1.1 implementation such as Apache Felix SCR 1.2.0 or newer.

Declarative Services: Delayed Components

2010-01-12T00:00:01Z

The OSGi Declarative services specification defines three types of components:

Immediate Components are immediately created when the providing bundle is started and may or may not provide services
Delayed Components provide services but are only created when used by a service consumer.
Factory Components are created on demand by calling the ComponentFactory.newInstance(Dictionary) method of the Component Factory service registered for the component.

This blog is about a special behaviour of delayed components which may seem unexpected in the first place: Delayed components are created (activated) on-demand (when they are first requested) and have to be deleted (deactivated) as soon as there is no user any longer (Chapter 112.5.4, Delayed Component, in the Compendium Spec; see also FELIX-1825)

The intent of this behaviour is to reduce the system (or bundle) startup time in that delayed components are only instantiated when really used. In addition memory consumption may be reduced at times when the service is not used. The drawback is, that the service is activated and deactivated if there is a single consumer which uses the service for short periods of times only. An example of such an oft used service is an OSGi EventHandler service, which is got by the Event Admin service when ever an event must be delivered and released after the delivery.

How does a Component become a delayed component ?

By default a component providing a service is a delayed component unless the component is explicitly declared as an immediate component. If you are using the Apache Felix Maven SCR plugin, a component is (by default) delayed if the @scr.service tag is used. Thus the above activation and deactivation rules apply. To turn a service component into an immediate component, you have to set the immediate attribute to, as in :

 @scr.component immediate="true"

Shall I change all my components to be immediate components ?

The short answer is: It depends.

Here are some general rules of thumb:

If your service can be expected to be immediately used and not released until system shutdown, defining the component as delayed does not make much sense. In this case, it is probably better to explicitly define the component as immediate. An example of such a component is a Servlet service in Sling, which is immediately used by the Sling Servlet Registry.
If your service is in fact a service factory (using the servicefactory attribute, you cannot declare the component immediate, because service factory components are always delayed
If your component is used a lot for short periods of time, you should probably define your service as an immediate component. An example of such a service is an OSGi EventHandler service.
If you want to maintain state in your component and make that state available to clients by registering a service, the component should be defined as immediate. An example of such a service might be statistics provider, which gathers statistics and provides them through its service API.
If your component is only seldom used it would be best to define it as a delayed component. An example of such a component in Communiqué 5 might be a workflow step service, which implements very specific behaviour.

mp3tagger on GitHub

2010-01-09T15:25:12Z

On the mp3 tagger post I have received quite a bit of feedback and feature requests. Therefore, I thought it might be a good idea to do "social coding" and put the code on GitHub where it can easily be forked (and the forks can be watched).

Other than that, the latest version of the tagger contains these improvements:

the Last.fm keys and secret are not stored in the code anymore, but entered on the first run and stored in ~/.mp3tagger.cfg
you can run the script in two additional modes: simulation and ask. In simulation mode no changes to mp3s will be saved, in ask mode you will be asked to save each change. Start the script with flags "-m simulation" or "-m ask", respectively.
It is now possible to specify a list of genre tags that will be considered (additionally to the mp3 default genre tags). The list needs to be stored in a config file at ~/.mp3tagger_genres.cfg (in the "generic" section of the file). The full format this file needs to have is shown below.
The last improvement is a tricky one: after tagging all my mp3s I ended up with hundreds of albums tagged with genre Electronic or Indie. I wanted to refine these genres into sub-genres. This again works by putting a list of possible sub-genres into ~/.mp3tagger_genres.cfg and running the tagger with flag "-r genre", e.g. "-r Electronic". You would run this option when you find that you have too many albums of one genre and want to split them up.

So in summary my config file ~/.mp3tagger_genres.cfg looks like:


[generic]
genres=Shoegaze,Dubstep,Grime,Dub,Drum And Bass
[refinements]
Electronic=Idm,Turntableism,Techno,Minimal,Dub,Big Beat,Ambient,Breakbeat,House,Lounge,Electroclash,Drum And Bass,Chillout
Indie=Indie Rock,Indie Pop,Singer-Songwriter,Indie Pop,Shoegaze,Post-Rock,Americana,New Wave,Alt-Country
Reggae=Dancehall,Dub,Ska

Identifications of OSGi Bundles

2010-01-07T00:00:01Z

Looking at the Bundle details in the Web Console you will notice a number of entries providing information about the bundles. In this article I will explain a bit more about the information regarding the identification of bundles.

Looking at the following screen dump you will notice a number of identifaction details of a bundle:

The bundle identification number. This is a number uniquely identifying the bundle. This number assigned to the bundle at the installation time of the bundle and never changes. These numbers are also unique within the framework. Never will a bundle ever be assigned the same number, even if the currently installed bundle is uninstalled.
The bundle name. This is a descriptive name of the bundle which is ignored by the OSGi framework. This name is provided by the bundle developer as the contents of the Bundle-Name manifest header.
The bundle symbolic name. This is a symbolic name of the bundle, which is used by the OSGi framework together with the bundle version number to identify the bundle. That is, no two bundles with the same symbolic name and version may be installed in a single OSGi framework. But multiple bundles may be installed which have the same symbolic name but differ in their version number. This symbolic name is provided by the bundle developer as the contents of the Bundle-SymbolicName manifest header.
The bundle version. The version of the bundle is used to convey the development state of the bundle as a whole. Together with the bundle symbolic name the version number must be unique within an OSGi framework (see the description of the bundle symbolic name). Note that the OSGi framework places only synthatic restrictions on version numbers. This version is provided by the bundle developer as the contents of the Bundle-Version manifest header.
The bundle location. The bundle location is basically just a string. It is recommended that this string follows the syntax of an URL and it may even be used as an URL to update the bundle from: The Bundle.update() method uses the bundle location to try to access an updated bundle version. This is not used as such though in Communiqué 5 where the bundle location merely indicates where the bundle has initially been installed from. This value is provided by the administrator as the location parameter when installing the bundle through the BundleContext.installBundle(String location) or BundleContext.installBundle(String location, InputStream input) method. This value will not changed when the bundle is updated.
The last modification time. The last modification time is set by the framework when the bundle is modified. A bundle is modified when it is installed, updated or uninstalled. Starting and stopping the bundle does not change the last modification time. This information can be used to verify that a bundle update has really been executed by the OSGi framework.

As can be seen a number of identification details are set on installation time and never change afterwards and some details may change over time when a bundle is updated.

So here are the rules regarding these details:

The bundle identification number is assigned by the framework and is never reused.
The bundle location is assigned from the location parameter when the bundle is installed and never changes during the existence of the bundle. No two bundles will ever be installed at the same time with the same bundle location. As such this location is also a unique identifier for bundles. But in contrast to the bundle identification number, the location may be "reused". That is, once a bundle with a given location has been uninstalled another bundle may be installed with the same location.
The bundle symbolic name and version together also uniquely identify an installed bundle. These values are taken from the bundle manifest and therefore may change when a bundle is updated. But it is not allowed for multiple bundles to be installed with the same symbolic name and version at the same time.

So, whenever you update a bundle, either placing it into an install folder in the repository or through the Web Console, expect the bundle version and/or symbolic name to change. But both the bundle identification number and the bundle location will not be modified. The bundle last modification time will reflect the time at which the bundle has actually been updated by the framework.

Custom CQ5 workflow step that integrates Twitter and Jabber

2009-12-28T00:00:01Z

As part of the IKS project each CMS vendor completes a couple of benchmarks in order to establish a baseline against which future semantic improvements can be measured. For benchmark 3 "Workflow Service" Bertrand and I chose to implement the task "Create a multi-channel (email, SMS, instant messaging, Twitter,...) notification service for workflow transitions". We have created an automated workflow step that can be inserted into a custom workflow and either send an e-mail, send a direct message on Twitter or send a chat message on GTalk/Jabber. The corresponding message's payload is the path to the content node in the workflow plus an optional custom text.

Below follows a description how this functionality was implemented in CQ5. The complete code is attached to this post as a CQ5 package. I will outline of some of the considerations and gotchas regarding this particular feature, but some issues apply to CQ5 development in general as well. The environment I used for development was CRXDE Lite (the web-based IDE available at /crxde of your CQ5 installation) and a beta version of the upcoming CQ5 release 5.3. It is probably helpful to install the package (see setup section below) and read the code alongside with this post.

OSGi services

A good way to hook up external services like Twitter etc. is to create a custom OSGi service that exposes only the business functionality and hides the internal classes. Moreover, it is good practice to provide a Java interface and the separate the implementation of the service (allowing the replacement of the implementation without affecting relying parties). The services will show up in the Sling configuration console at /system/console/configMgr. This allows the administrator to configure the service's private parameters at deployment time (in our case Twitter account credentials and Jabber user credentials). The config is consumed by the service like e.g.:

/** @scr.property */
public static final String GTALK_USER = "gtalk.service.user"; 
/** @scr.property */
public static final String GTALK_PASSWORD = "gtalk.service.password";   

protected void activate(ComponentContext context) {
    Dictionary config = context.getProperties();
    user = (String) config.get(GTALK_USER);
    password = (String) config.get(GTALK_PASSWORD);
}

3rd party libraries

In order to use Twitter and Jabber I utilized the open source libraries Twitter4J and Smack, respectively. With CRXDE (Lite) it is very simple to include such 3rd party jars in a custom OSGi bundle: just drop them in the bundle's /libs folder. When building the bundle CRXDE will embed them. Compilation and deployment is done by executing "Build Bundle" (right-click on the .bnd file in the bundle root).

A note on 3rd party jar's dependencies

It might well be that the bundle compiles and deploys, but does not start. Check the OSGi console at /system/console/bundles to find out if your bundle's state is "Active" (good) or just "Installed" (not good). The latter happens e.g. when the embedded jar has dependencies on other jars that are not embedded. In such a case check the bundle's details page in the Sling console to find out which dependencies are missing and either add them to /libs as well or take them out of the OSGi imports. That is achieved by editing the .bnd file's import directive, e.g.

Import-Package: !com.sun.syndication.*, !dalvik.system, *

Workflow action

The last needed piece is a workflow step that can be added into a custom workflow. For that purpose one simply needs to create a class that implements the Interface JavaProcessExt. The method execute will receive the workflow's payload - from there is is trivial to obtain the services described above and pass them the content. CQ Workflow Actions can be customized for each particular workflow they are used in. I use this feature to customize the accounts to which a message shall be sent (the custom format is explained in the setup section below). The customization string is passed to the execute method as well: comma-separated values will arrive as a String[] array.

Setting up the package

To get this up and running download the attached CQ5 package and install it throught the package manager. In the Sling console configure the the services com.day.cq.mailer.impl.MailerService, com.day.iks.service.impl.TwitterServiceImpl and com.day.iks.service.impl.GtalkServiceImpl. For Twitter and Gtalk you need to supply the credentials of the (technical) user that shall send the DMs or chat messages, respectively. In the case of e-mail you need to configure your mail server.

Next, create a custom workflow in the CQ5 workflow section and add the workflow action (name). The configuration options are:

for sending an e-mail: email,user@mydomain.com,some_message
for sending a direct message on Twitter: dm,twitter_user,some_message
for sending a chat message on Gtalk: gtalk,user@gmail.com,some_message

The (optional) message will be appended with the content item's path.

Here is an example for GTalk:

In the cases of Twitter DM and GTalk make sure that the recipient has opted-in to receive messages from the technical user you have configured as a sender.

JCR and Rails revisited

2009-12-21T00:00:01Z

Pengchao Wang of Thoughtworks has published his approach to using JCR as a backend for a JRuby-based Rails app. Interestingly, Pengchao uses unstructured nodes (at least, that is what I gather from reading the code), so he can take full advantage of the schema-free side of JCR and freely add properties to JCR-persisted Ruby objects. The code is on GitHub, have a look at jcr-rails-demo/lib/jcr/record_base.rb from which the models classes inherit. If Rails and JCR is of interest to you: also have a look at what Ngoc Dao wrote on how to use JCR in Rails applications

Rails + JCR

View more presentations from shen liu.

[LOTD] The Skinny on JCR, CMIS and OSGi

2009-12-18T00:00:01Z

Remember Jon Mark's overview picture of content technologies I mentioned the other day? Jon has just published his first column on CMSWire where he delivers a brilliant textual description of his diagram, spanning across JCR, CMIS and OSGi.

Jon rightfully points out that CMIS is of very limited value for managing web content and explains that further in the comments section. It is at best tricky to store in a CMIS repository an HTML document that contains references to other items in the same repository, i.e. you will struggle to use CSS, images or hyperlinks.

Differences between JCR 1 and JCR 2 on API level

2009-12-17T00:00:01Z

My colleague Sebastian Hoogenberk has run JDiff over the JCR 1 and JCR 2 Javadocs. The results are useful to get a clear overview over the changes on API level. Note that JDiff unfortunately seems to get confused with some methods and marks them both as "added" and "replaced".

Open Innovation in Software means Open Source

2009-12-07T10:38:33Z

I’m giving a talk today at the Open Source, Open Development, Open Innovation workshop in Oxford:

Open source software is more than just a licence, it is also a software development methodology that allows companies to share resources and collaborate on non-core parts of their software/service offering. When managed well, open development enables a reduction in cost, and an increase in innovation as a result of the convergence of the best minds in the problem space. In this presentation Bertrand Delacretaz will describe how Day Software has embraced open development by positioning itself as the leaders in both open standards and open source software. We will examine how Day’s active engagement with 25 open source projects and numerous standards groups has enabled the company to become a world leader in their market and in the open source projects they participate in.

The funny thing is that the above abstract was written by Ross Gardler while waiting for my own version of it – and it says exactly what I was trying to say, only better ;-)

The event is covered by a live blog, and you can ask questions there.

To put it simply, my conclusion is that quick feedback from users and customers is key to open innovation – and open source. if done right, provides lots of feedback, fast.

Daily Shoot, week 3

2009-12-07T09:17:04Z

Another week of @dailyshoot:

PS. Check out the updated dailyshoot.com web site.

Top 10 Features in the Upcoming CQ 5.3 Release

2009-12-02T00:00:01Z

Here is a short presentation with my personal top ten features in our upcoming release. Of course, this would ideally be accompanied by short fast paced demos, so if you are interested in getting personalized demo or a video, please reach out to us.

What's new in CQ 5.3? Top 10 features.

View more documents from David Nuescheler.

Daily Shoot, week 2

2009-11-29T16:38:51Z

As I mentioned last week, I’ve been following @dailyshoot for a series of daily photo assignments. Here’s what I shot this week:

Sling over HTTP

2009-11-28T21:18:06Z

A few days ago I posted about Jackrabbit, and now it’s time to follow up with Sling as a means of accessing a content repository over HTTP. Apache Sling is a web framework based on JCR content repositories like Jackrabbit and among other things it adds some pretty nice ways of accessing manipulating content over HTTP.

The easiest way to get started with Sling is to download the “Sling Standalone Application” from the Sling downloads page. Unpack the distribution package and start the Sling application with “java -jar org.apache.sling.launchpad.app-5-incubator.jar”. Like Jackrabbit, Sling can by default be accessed at http://localhost:8080/. There’s a 15 minute tutorial that you can check out to learn more about Sling.

Since Sling comes with an embedded Jackrabbit repository, it also supports much of the WebDAV functionality covered in my previous post. Instead of rehashing those points, this post takes a look at the additional HTTP content access features in Sling.

CR1: Create a document

Like with Jackrabbit, all documents in Sling have a path that is used to identify and locate the document. Sling solves the problem of having to come up with the document name by supporting a virtual “star resource” that’ll automatically generate a unique name for a new document. Thus instead of having to think of a URL like “http://localhost:8080/hello” in advance, the new document can be created by simply posting to the star resource at “http://localhost:8080/*”.

The Sling POST servlet is a pretty versatile tool, and can be used to perform many content manipulation operations using normal HTTP POST requests and the application/x-www-form-urlencoded format used by normal HTML forms. With the POST servlet, the example document can be created like this:

$ curl --data 'title=Hello, World!' --data 'date=2009-11-17T12:00:00.000Z' \
       --data 'date@TypeHint=Date' --user admin:admin \

http://localhost:8080/*

The 201 Created response will contain a Location header that points to the newly created document. In this case the returned URL is “http://localhost:8080/hello_world_” based on some document title heuristics included in Sling. If you run the command again you’ll get a different URL since the Sling star resource will automatically avoid overwriting existing content.

Pros:

A single standard POST request is enough
The HTML form format is used for the POST body
Automatically generated clean and readable document URL

Cons:

The star resource URL pattern is fixed and creates an unnecessarily tight binding between the client and the server

CR2: Read a document

Sling contains multiple ways of accessing the document content in different renderings. In fact much of the power of Sling comes from the extensive support for rendering underlying content in various different and easily customizable ways.

Unfortunately at least the latest 5-incubator version of the Sling Application doesn’t support any reasonable default rendering at the previously returned document URL. The client needs to explicitly know to add a “.json” or “.xml” suffix to the document URL to get a JSON or XML rendering of the document.

$ curl http://localhost:8080/hello_world_.json
{
  "title":           "Hello, World!",
  "date":            "Tue Nov 17 2009 12:00:00 GMT+0100",
  "jcr:primaryType": "nt:unstructured"
}
$ curl http://localhost:8080/hello_world_.xml

The JCR document view format is used for the XML rendering.

Pros:

A single GET request is enough
Both the JSON and XML formats are easy to consume

Cons:

Simply GETting the document URL doesn’t return anything useful
The “.json” and “.xml” URL patterns create an unnecessary binding between the client and the server
Neither rendering contains property type information
The XML rendering contains unnecessary namespace declarations

CR3: Update a document

The Sling POST servlet supports also document updates, so we can just POST the updated properties to the document URL:

$ curl --data 'history=Document date updated' \
       --data 'date=2009-11-18T12:00:00.000Z' \
       --data 'date@TypeHint=Date' --user admin:admin \

http://localhost:8080/hello_world_

Pros:

A single standard POST request is enough
The HTML form format is used for the POST body

Cons:

None.

CR4: Delete a document

You can either use the special “:operation=delete” feature of the Sling POST servlet or a standard DELETE request to delete a document:

$ curl --data ':operation=delete' --user admin:admin \

http://localhost:8080/hello_world_

$ curl --request DELETE --user admin:admin \

http://localhost:8080/hello_world_

Pros:

A standard DELETE or POST request is all that’s needed

Cons:

None.

Increasing Interest in JCR and Apache Sling

2009-11-26T15:07:46Z

I’m attending some tech conferences in the Java space for several years now; for the last years I’m trying to push JCR, Apache Jackrabbit, and Apache Sling at various occasions – mostly through several talks. It seems to me that there is a change/increase in interest for these topics.
While two years ago, people have not been really that interested in JCR and usually asked questions along the line of “why should i use this? I have a database, it works fine” etc., this has definitly changed now. There are more and more people interested in alternatives to a POD (plain old database). It seems to me that the pain with traditional dbs is now too much and they’re searching for nosql solutions. Don’t get me wrong, JCR is not the golden hammer for data storage – there are valid use cases for PODs, but there are also many use cases where an alternative like JCR is much better suited.
Today people ask questions like “I looked at jcr, I have this problem, and I think I could do it this way. What do you think?” and of course variations on this theme.
I don’t think that this is motivated through the NOSQL hype, I ratherthink that this is a parallel movement which has the same origin. In addition, these people seems to know a lot about Apache Jackrabbit and usually ask deep going questions. As I’m not working on Jackrabbit itself – I’m a user of Jackrabbit – these questions usually give me a hard time
While I have a hard time with Jackrabbit questions, people seem often to have a hard time understanding the needs for Apache Sling. For one this might be cause they are still living in their POD driven world and know how to handle applications based on databases. Building applications on top of NOSQL solutions is a slightly different thing. When it comes to Apache Sling which is a web framework, it is immediately compared against web application frameworks with all the nice ui features, widget libraries and whatnot. And as Sling does not provide a UI library it is often immediately discarded.
But on the bright side as soon as people realize that they need something like JCR and now need a way to get all this content stored in the repository out to the users in a nice, elegant and flexible way, they also realize that Apache Sling helps a lot.
So my hope is that the adaption of JCR is increasing (and I think that is already the case) and that this drives the adoption of Apache Sling as well
And of course we are not day dreaming in Apache Sling – we will continue it’s development and add missing pieces or provide bridges etc.

[LOTD] CMIS, JCR and OSGi for Idiots

2009-11-26T00:00:01Z

Drinking with David has inspired Jon Marks (aka @McBoof) to draw a brilliant drawing of the landscape of content technologies. Beer :)

Jackrabbit over HTTP

2009-11-24T01:25:34Z

Last week I posted a simple set of operations that a “RESTful content repository” should support over HTTP. Here’s a quick look at how Apache Jackrabbit meets this challenge.

To get started I first downloaded the standalone jar file from the Jackrabbit downloads page, and started it with “java -jar jackrabbit-standalone-1.6.0.jar”. This is a quick and easy way to get a Jackrabbit repository up and running. Just point your browser to http://localhost:8080/ to check that the repository is there.

Jackrabbit comes with a built-in advanced WebDAV feature that gives you pretty good control over your content. The root URL for the default workspace is http://localhost:8080/server/default/jcr:root/ and by default Jackrabbit grants full write access if you specify any username and password.

Note that Jackrabbit also has another, filesystem-oriented WebDAV feature that you can access at http://localhost:8080/repository/default/. This entry point is great for dealing with simple things like normal files and folders, but for more fine-grained content you’ll want to use the advanced WebDAV feature as outlined below.

CR1: Create a document

All documents (nodes) in Jackrabbit have a pathname just like files in a normal file system. Thus to create a new document, we first need to come up with a name and a location for it. Let’s call the example document “hello” and place it at the root of the default workspace, so we can later address it at the path “/hello”. The related WebDAV URL is http://localhost:8080/server/default/jcr:root/hello/.

You can use the MKCOL method to create a new node in Jackrabbit. An MKCOL request without a body will create a new empty node, but you can specify the initial contents of the node by including a snippet of JCR system view XML that describes your content. In our case we want to specify the “title” and “date” properties. Note that JCR does not support date-only properties, so we need to store the date value as a more accurate timestamp.

The full request looks like this:

$ curl --request MKCOL --data @- --user name:pass \
       http://localhost:8080/server/default/jcr:root/hello/ <
  
    Hello, World!
  
  
    2009-11-17T12:00:00.000Z
  

END

The resulting document is available at the URL we already constructed above, i.e. http://localhost:8080/server/default/jcr:root/hello/.

Pros:

A single standard WebDAV MKCOL request is enough
The standard JCR system view XML format is used for the MKCOL body
The XML format is easy to produce

Cons:

We need to decide the name and location of the document before it can be created
The name of the document is duplicated, once in the URL and once in the sv:name attribute
The date property must be specified down to the millisecond
While standardized, the MKCOL method is not as well known as PUT or POST
While standardized, the JCR system view format is not as well known as JSON, Atom or generic XML
The system view XML format is quite verbose

CR2: Read a document

Now that the document is created, we can read it with a standard GET request:

$ curl --user name:pass http://localhost:8080/server/default/jcr:root/hello/


  
    nt:unstructured
  
  
    2009-11-17T12:00:00.000Z
  
  
    Hello, World!

Note that the result includes the standard jcr:primaryType property that is always included in all JCR nodes. Also all namespaces registered in the repository are included even though strictly speaking they add little value to the response.

Pros:

A single GET request is enough
The XML format is easy to consume

Cons:

The system view format is a bit verbose and generally not that well known

CR3: Update a document

The WebDAV feature in Jackrabbit does not support setting multiple properties in a single request, so we need to use separate requests for each property change. The easiest way to update a property is to PUT the new value to the property URL. The only tricky part is that unless the node type explicitly says otherwise the new value is by default stored as a binary stream. You need to specify a custom jcr-value/… content type to override that default.

$ curl --request PUT --header "Content-Type: jcr-value/date" \
       --data "2009-11-18T12:00:00.000Z"  --user name:pass \

http://localhost:8080/server/default/jcr:root/hello/date

$ curl --request PUT --header "Content-Type: jcr-value/string" \
       --data "Document date updated"  --user name:pass \

http://localhost:8080/server/default/jcr:root/hello/history

GETting the document after these changes will give you the updated property values.

Pros:

Standard PUT requests are used
No XML or other wrapper format needed, just send the raw value as the request body

Cons:

More than one request needed
Need to use non-standard jcr-value/… media types for non-binary values

CR4: Delete a document

Deleting a document is easy with the DELETE method:

$ curl --request DELETE --user name:pass \

http://localhost:8080/server/default/jcr:root/hello/

That’s it. Trying to GET the document after it’s been deleted gives a 404 response, just as expected.

Pros:

A standard DELETE request is all that’s needed

Cons:

None.

The perfect monitoring

2009-11-23T00:00:01Z

I often thought about how a perfect monitoring solution should like; then the following requirements come up:

1) consistent: When the monitoring indicates a problem, there really is a problem with the application.

2) reliable: When there is a problem that hinders the users from properly working on the application, the monitoring will indicate this, before the support line is overwhelmed by calls reporting that your application has a problem.

3) informative: When a monitoring indicates a problem, there should a recommendation how this problem can be fixed with minimal impact. This recommendation can be either documented offline (operation guidelines) or in the monitoring itself.

4) proactive: a monitoring resource should detect problems before they happen. Sounds strange, but many problems (excessive use of memory) can be reported before they actually have a big impact.

The requirement 2) is a hard one; essentially it requires that all problem situations are pre-known and the presence of such a problem situation can be indicated by the monitoring resource. In fact most situations are not known to anybody, until they happen and are fully analyzed and understood.

1) can fulfilled very easily: never report a problem. From a purely academical view this requirement is fulfilled then, but actually it is not really usable. A more usable approach is to report only when it's 99,99% clear that no user can work anymore (e.g. on a CQ authoring system when a vital OSGI service isn't available any more). But the more subtile problems cannot be caught that way.

3) requires a certain amount of experience with the application and the will to write good documentation, that is kept up to date. And 4) requires knowledge about typical problems and the signs of problems.

All these requirements often need an extension to the application to provide an interface to the monitoring, where the monitoring can fetch the data from and decide what to do with it. But monitoring is the poor child of application development; it is neither a functional requirement nor a non-functional requirement with the importance of usability, performance or availability (aah, by the way: how should we measure that? No-one cares, as long you guarantee 99,9% ...), but only a requirement of the operations team - no-one spends time or money on it, until operations ask for it. "Ups, we already spent our budget on other things". Only a few operations teams have the management standing, that they can deny such applications to run on their machines then, in most cases they are just overruled by management. Then, the operations team usually comes up with some thoughts and tries to fill the gap themselves, but in most times that does not work very well. Especially requirement 2) is then often violated and 3) and 4) are not implemented at all. But operations can show some green bulbs in the monitoring system to the management.

So, as a last requirement (which should be the very first requirement, though):

5) there should be a proper monitoring at all. A monitoring that only watches for a running process cannot detect that this process already internally has deadlocked and isn't working anymore.

Finally, when you need to implement a complex application, make sure that some of its internal state can be exposed to an external monitoring solution, which helps to operate your application. Treat this as a normal "must-have" feature with specification, implementation and test. You will make your operations team really happy.

If you do not do that, your monitoring system are actually the users, when they complain about non-working functionality, which must be fixed by your operations team then. And that brings costs (service calls) and negative management attention. Nothing one wants to have.

Daily Shoot, week 1

2009-11-22T23:12:58Z

A week ago James Duncan Davidson and Mike Clark launched @dailyshoot, a Twitter feed that posts daily photo assignments. The idea is to encourage people who want to learn photography to practice it every day with the help of a simple assignment that fits a single tweet. I’m following Duncan’s blog, so I found out about Daily Shoot the day it was launched.

So far I’ve completed all the assignments and I’ve already learned quite a bit doing so. It’s very interesting to see how other people interpret the same assignments. I avoid looking at other responses before completing an assignment so that I don’t end up just copying someone else’s approach. Once I’m done I look at what other’s have done for some nice insight on what I could have done differently. The process is quite educational.

Here’s what I’ve shot this week:

You can click on the pictures for more background on each assignment and how I approached it. For more information on Daily Shoot, see the recently launched website.

Ignite slides

2009-11-21T00:00:01Z

RT @joannekh:

Day Ignite presentations now available at www.day.com/ignite

(dayignite on slideshare)

What is a content repository

2009-11-19T00:00:01Z

Joint post of Henri Bergius and Michael Marth cross-posted here and here.

Web Content Repositories are more than just plain old relational databases. In fact, the requirements that arise when managing web content have led to a class of content repository implementations that are comparable on a conceptual level. During the IKS community workshop in Rome we got together to compare JCR (the Jackrabbit implementation) and Midgard's content repository. While in some cases the terminology might be different, many of the underlying ideas are identical. So we came up with a list of common traits and features of our content repositories. For comparison, there is also Apache CouchDB.

So, why use a Content Repository for your application instead of the old familiar RDBMS? Repositories provide several advantages:

Common rules for data access mean that multiple applications can work with same content without breaking consistency of the data
Signals about changes let applications know when another application using the repository modifies something, enabling collaborative data management between apps
Objects instead of SQL mean that developers can deal with data using APIs more compatible with the rest of their desktop programming environment, and without having to fear issues like SQL injection
Data model is scriptable when you use a content repository, meaning that users can easily write Python or PHP scripts to perform batch operations on their data without having to learn your storage format
Synchronization and sharing features can be implemented on the content repository level meaning that you gain these features without having to worry about them

feature	JCR / Jackrabbit	Midgard	CouchDB
content type system	In JCR structured or unstructured nodes are supported and can be mixed at will in a content tree.	Content types are defined in MgdSchema types. All content must be stored to an MgdSchema type, but types can be extended on content instance level using the "parameter" triplets	Type-free
type hierarchy	Structured node types support inheritence of types, additional cross-cutting aspects can be added with "mixins". Node types can define allowed node types for child nodes in the content hierarchy.	MgdSchemas allow inheritance, and an extended type can be instantiated either using the extended type or the base type	Type-free
IDs	Nodes with mixin "referenceable" have ~~GUID~~ a UUID. In practice the node path is often used to reference nodes.	Every object has a GUID used for referencing. Objects located in trees that have a "name" property can also be referred to using the path	All objects can be accessed via a UUID
References	Nodes can reference each other with hard link (special property type) or soft link (by referring to the node path)	MgdSchema types can have properties linking to other objects of same or different type. A link of "parentfield" type places an MgdSchema type in a tree.	No reference support built-in
content hierarchy	All content is hierarchical / in a tree	Content can exist in tree, or independently of it depending on the MgdSchema type definition	flat structure
interesting property types	Multi-valued (like an array), binary properties (e.g. for files), nodes have an implicit sort-order	Binary properties stored using the Midgard Attachment system	Support for binary properties
transactions	Multiple content modifications are written in transactions.	Transactions can be used optionally.
events	JCR Observers can register for content changes on different paths and/or for different node types and/or CRUD, receive notification of changes as serialized node	All transactions cause both process-internal GObject signals, and interprocess DBus signals	Support for one external event notification shell script
workspaces	Workspaces provide separate root trees.	No workspaces support in Midgard 9.03, coming in next version	Multiple databases within one CouchDB instance
import and export	nodes or parts of the repository (or the whole repo) can be imported or exported in XML. 2 formats: docview for human-frindly representation, sysview including all technical aspects	Objects can be exported and imported in XML format. There are tools supporting replication via HTTP, tarballs, XMPP, and the CouchDB replication protocol	JSON serialization is the standard way of accessing the repository. CouchDB replication protocol supports full synchronization between instances
versioning	Checkin/checkout model to create new versions of nodes, optionally versions complete sub-trees, supports branching of versions.	No versioning	All versions of content are stored and accessible separately, no branching
locking	Nodes can be locked and unlocked	Objects can be locked and unlocked
object mapping	Not in standard, but implemented in Jackrabbit. Rarely used in practice.	Object mapping is the standard way of accessing the repository	All content is accessed via JSON objects
queries	In JCR1 Sql or XPath, in JCR2 also QueryBuilder.	Query Builder	Javascript map/reduce
access control	Done on repository level, i.e. all access control is independent of application. In Jackrabbit: pluggable authentication/authorization handlers.	No access control in Midgard repository, usually implemented on application level. Midgard proves a user authentication API	No access control
persistence	In Jackrabbit different Persistence Managers can be plugged in (RDBMS, tar file, ...)	libgda allows storage to different RDBMS like MySQL, SQLite and Postgres	CouchDB has its own storage
architecture	Jackrabbit: library (jar), JEE resource, OSGi bundle or standalone server	Library	Erlang-based daemon
APIs	Standard: Java-based, PHP coming up. In Jackrabbit: also WebDAV and HTTP-based API	C, Objective-C, PHP, Python	HTTP+JSON
full-text search	Included in repository. In Jackrabbit: Lucene bundled	No (SOLR used on application level)	Plugin for using Lucene, not installed by default
standard metadata	All nodes have access rights, jcr:primaryType and jcr:mixinTypes properties. JCR 2.0 standardizes a set of optional metadata properties.	All objects have a set of standard metadata including creator, revisor, timestamps etc	No standard properties

Content Repository over HTTP

2009-11-18T01:45:11Z

Two weeks ago during the BarCamp at the ApacheCon US I chaired a short session titled “The RESTful Content Repository”. The idea of the session was to discuss the various ways that existing content repositories support RESTful access over HTTP and to perhaps find some common ground from which a generic content repository protocol could be formulated.

The REST architectural style was generally accepted as a useful set of constraints for the architecture of distributed content-based applications, but as an architectural style it doesn’t define what the bits on the wire should look like. This is what we set out to define with the HTTP protocol as a baseline. We didn’t get too far, but see below for some collected thoughts and a useful set of “test cases” that I hope to use to further investigate this idea.

Existing solutions

Many existing content repositories and related products already support one or more HTTP-based access patterns: Apache Jackrabbit exposes two slightly different WebDAV-based access points. Apache Sling adds the SlingPostServlet and default JSON and XML renderings of content. Apache CouchDB uses JSON over HTTP as the primary access protocol. Apache Solr uses XML over HTTP. Midgard doesn’t have a built-in HTTP binding for content, but makes it very easy to implement such bindings. This list just scratches the surface…

There are even existing generic protocols that match at least parts of what we wanted to achieve. WebDAV has been around for ten years already, but the way it extends HTTP with extra methods makes it harder to use with existing HTTP clients and libraries. The AtomPub protocol solves that issue, but being based on the Atom format and leaving much of the server behaviour undefined, AtomPub may not be the best solution for generic content repositories.

Content repository operations over HTTP

To better understand the needs and capabilities of existing solutions, we should come up with a simple set of content operations and find out if and how different systems support those operations over HTTP. The most basic such set of operations is CRUD, i.e. how to create, read, update, and delete a document, so let’s start with that. I’m giving each operation a key (CRn, as in “Content Repository operation N”) and a brief description of what’s expected. In later posts I hope to explore how these operations can be implemented with curl or some other simple HTTP client accessing various kinds of content repositories. I’m also planning to extend the set of required operations to cover features like search, linking, versioning, transactions, etc.

CR1: Create a document

Documents with simple properties like strings and dates are basic building blocks of all content applications. How can I create a new document with the following properties?

title = “Hello, World!” (string)
date = 2009-11-17 (date)

At the end of this operation I should have a URL that I can use to access the created document.

CR2: Read a document

Given the URL of a document (see CR1), how do I read the properties of that document?

The retrieved property values should match the values given when the document was created.

CR3: Update a document

Given the URL of a document (see CR1), how do update the properties of that document? For example, I want to update the existing date property and add a new string property:

date = 2009-11-18 (date)
history = “Document date updated” (string)

When the document is read (see CR2) after this update, the retrieved information should contain the original title and the above updated date and history values.

CR4: Delete a document

Given the URL of a document (see CR1), how do I delete that document?

Once deleted, it should no longer be possible to read (see CR2) or update (see CR3) the document.

[LOTD] IKS in the press

2009-11-18T00:00:01Z

French IT mag LeMagIT has published an article about the IKS project including quotations from Bertrand Delacretaz. Bertrand emphasizes the need for concrete results:

pour décoller, les technologies sémantiques ont besoin de cas d'utilisateur concrets

In the comments section Bertrand mentions his tag line for semantic technologies that I can very well relate to:

La sémantique "sous le capot" oui, la sémantique "dans la figure", non

This roughly translates as: "semantics under the hood yes, semantics in your face, no".

In Computerworld UK open source blogger Glyn Moody has described his first hands impressions from the IKS workshop in Rome. He comes to a similar conclusion:

Paradoxically, semantic search will only ever really take off once it has receded so far into the fabric of computing that people aren't even aware it's there.

What Makes Apache Tick?

2009-11-17T09:19:57Z

Looking at the diversity of Apache Software Foundation communities, one can see a recipe for failure: people from different cultural backgrounds, different mother tongues, different employers, different timezones…all working together to create some of the best software on the planet? You must be kidding.

How can this very loose collage of disparate people pump out dozens of high-quality releases every year, often working better than more structured corporate teams? This “mystery” has been on my mind for a while, and I have identified four drivers that influence the way we use collaboration tools that play a major part in our success.

The first driver is a common vision amongst project members. The Biblical saying, “Without a vision, my people perish” is quite valid for our projects. Both using a central development mailing list for each project, and spending time to collectively define our project’s charter, helps us foster this common vision amongst project members. Every member should have the same answer to the “what are our goals?” question, so it’s important to get them to talk in a central place, where they all get the same information, as opposed to undocumented, one-to-one discussions.

Secondly, providing real-time status updates to project members is key in helping them stay on track. At Apache, this is implemented by the many events generated by our collaboration tools: commit events to indicate code changes, issue tracker events to provide updates about the status of bugs and new features, success/failure events from continuous build systems, and standardized ways of announcing releases so that other projects are informed. Project members subscribe freely to as few or as many event channels that they want to, so as to stay on top of things in near real-time, and without having to actively ask others about what happened. Status meetings? No need for those, as the information is flowing all the time.

The third success driver lies in enabling real-time help requests. In an immediate crisis of the “we need to deliver this by tomorrow” type, especially when working with a big team, you need to be able to ask for help without necessarily 1) knowing who specifically will help, and 2) bothering others with direct person-to-person requests, especially if they work in a different timezone. The key here is using issue trackers, where one web page stores key data and parts of the dialog that leads to resolving an issue. Posting an issue on the tracker, with sufficiently detailed instructions about how to reproduce the problem, along with attributes such as severity level, affected modules, etc., is the best way to expose a problem to the group quickly and with precision. Using an issue tracker also allows you to quickly and efficiently change priorities as well as re-assign issues and tasks – key elements that make all the difference in a crisis.

Finally, having searchable archives of this information allows new project members, or those returning after a period of absence, to learn what transpired and why things have been done in a certain way. Without self-service archives, new participants would have to talk to everybody to find out about the project’s history, past decisions, conventions, etc., which is neither efficient nor scalable. Most of our archives are automatically built as project activities progress: mailing lists are archived, source code control history is kept forever, and issue trackers write the full history of the project’s micro-decisions.

Combined with Apache’s principles of meritocracy and consensus-based decision making, these four collaboration drivers allow our project teams to work very efficiently, and, in many cases, even more so than structured teams that do not establish those central hubs of information exchange.

Does your project team foster a common vision and provide tools for real-time status updates, real-time help requests and self-service archives to its members? If yes, congratulations: you’re on a good track to becoming as effective as an Apache project!

Many thanks to Sally Khudairi for reviewing and copy editing.

ApacheCon US 2009 picture by
Ted Leung / Creative Commons License (CC BY-NC-SA 2.0).

See also my Open Source Tools are Good For You presentation, which discusses the tools that Apache projects use to implement this.

What does Apache provide that other code repositories don’t?

2009-11-16T16:46:19Z

People thinking about creating an open source project might rightly consider hosting on one of the various hosting services available: Google Code, SourceForge, kenai, bitbucket and github come to mind. Quick and easy, create a repository or request some resources and you’re in business.

Incubating a project at the Apache Software Foundation (ASF) takes a lot more effort than just requesting a hosting space on one of those services, so why would you do that? One can perfectly host code on one of those services with an Apache License, so what’s the difference?

I think the big difference lies in the governance model, and in fact calling the ASF just a code repository is very wrong. Let’s discuss some key elements of that.

The Apache voting process has been tried and tested since 1999, or even earlier. This is one of the things that projects coming through the incubator have to learn, led by their mentors. Learning is usually very easy as people quickly see the benefits of those simple no-nonsense rules.

The ASF also provides a well-defined structure for managing projects, and the foundation as a whole, in a fair and consensus-driven way. One could argue that structure gets in the way, and sometimes it does, but when things go wrong having a well-defined way of getting back on track helps tremendously. And this structure leaves a lot of freedom to the project’s management committee (PMC), there’s a lot of room for adapting a project’s way of working to its community and goals.

Creating an Apache project is certainly not required for all open source projects (and the foundation couldn’t scale to thousands of projects right now anyway), but for the critical infrastructure parts of one’s business (what’s sometimes called “open core”), having an established governance model makes all the difference.

The governance model is just one of the benefits that Apache projects get – there’s also the visibility, brand recognition, nice build services, and other tools, and, last but not least, the many friends that you make along the way! As everybody now knows, there are no jerks at Apache!

Running the iTunes genre tagger script with OS X Automator

2009-11-15T14:37:08Z

Due to public demand here's a little recipe how to run last post's mp3 tagger without using the command line on OS X:

Open Automator
Start a new "Application" project
Drag the "Run Shell Script" action into the right workflow panel, set the "pass input" drop-down to "as arguments" and edit the script to (see screenshot below):

for f in "$@"
do
/opt/local/bin/python /Users/michaelmarth/Development/Code/mp3tagger/tag_groupings.py -d "$f"
done

(you will have to adapt the paths to your local setup)

Save the application and happily start dropping mp3 folders onto the application's icon.

Back from W-JAX 09

2009-11-13T00:00:01Z

This year's WJAX in Munich has been (again) a great success. The conference area was crowded up to the maximum capacity of the hotel I guess. Around 150 talks, different special days covering topics like persistence, OSGi, Scala, and the never dying SOA. My two talks about JCR and Apache Sling have been well attended, some interesting questions came up and I could spread the interest in these cool technologies. Now looking forward to JAX 2010 :)

JCR - Java Content Repositories

View more documents from Carsten Ziegeler.

OSGi, Scripting and REST, Building Webapps With Apache Sling

View more documents from Carsten Ziegeler.

Update: The IKS semantic engine - a pragmatist's view

2009-11-13T00:00:01Z

Update to "The IKS semantic engine - a pragmatist's view": here are the slides:

Title The IKS RESTful semantic engine - let's get started!

View more documents from bdelacretaz.

The presentation went well, and will hopefully lead to a sprint to actually implement something along these lines. The two demos that used UIMA at the workshop made me think that UIMA should be part of that picture, at least as a plugin for semantic lifting. And I did the presentation in less than 8 minutes out of the 10 that were allocated. Bonus points?

IKS Search Benchmark

2009-11-12T00:00:01Z

CQ5 search comes with some improvements over JCR's search capabilities, e.g. adapting result rankings to what users choose or faceted search. Within the IKS project Bertrand and I have experimented with another possibility: link-based ranking, i.e. adjusting search results based on the content of link tags. For example: if page A links to page B with the link text "lorem ipsum" then page B should get a higher ranking when a user searches for "lorem ipsum". This is essentially what Google does, but we wanted to apply it to internal links (within the same site) only.

To give away the results right away: for many web sites the results will probably not improve dramatically, because there are not enough internal links. However, it might help for some projects so our implementation approach is described below in case you want to give it a try in your project.

In order to extract links from a node we opted for parsing the complete rendered HTML presentation of a node rather than looking only at the Rich Text properties of one node. In that way we could also catch programmatically generated links from templates. So we ended up by setting up a little spider on the publish server that retrieves HTML representations of all pages. The spider is deployed as an OSGi bundle within the server so it gets the locations of all pages from an internal repository query. For each page the HTML is retrieved and parsed. The found links are stored as child nodes below the page that is linked to. In the example from above: if page A links to page B with the link text "lorem ipsum" then page B gets a child node with properties source=A and text="lorem ipsum". Implemented in that way we could basically use the Jackrabbit indexer without further changes.

We have also implemented a JCR Observer that catches changes to pages and fixes the corresponding links. Template updates are not caught, yet.

The sources are attached to this post. The Java program can be used as a standalone application or deployed as an OSGi bundle. The standalone program takes a couple of optional arguments for running a full upfront spidering, deleting all found link nodes etc. In case you want to give it a try please be aware:

The standalone program requires RMI to be enabled on the repository which is not the case by default (in the code port 1235 is used).
The searches must take into account the new properties of the link nodes. One possibility is to re-configure the Jackrabbit indexing, which in CQ5 is done in the crx-quickstart/server/runtime/0/_crx/WEB-INF/classes/indexing_config.xml file, by adding:
```
  linkedText
```

The boost factor in this configuration can be adjusted to give links a proper weight relative to the other properties of a node
For reindexing delete these directories:
crx-quickstart/repository/repository/index
crx-quickstart/repository/workspaces/crx.default/index
crx-quickstart/repository/workspaces/crx.system/index

Results

We tested the approach on the content of our corporate website (a rather small content corpus). Overall, the search results improved slightly, but not much (although we did not spend a lot of time on tweaking the boost factor). As stated above I believe that corporate websites in general will not benefit from link-based ranking very much as the majority of links in them are often reflecting the navigation (i.e. the hierarchical structure of the site) so they provide little additional information. Of course, on the other side there is no harm in using links for search relevance either.

Alternative approach

Marcel Reutegger (the MAN when it comes to JCR searches) gave a lot of great input to our experiment (thanks a lot for this). He also hinted how an alternative implementation could look like: using an output filter, which can process HTML content as it's being generated. In CQ5 the validity of links is already checked that way, so storing them would naturally fit there. Also, he suggested storing the links not below the pages themselves, but in a separate part of the repository. In a background processing job these links could be aggregated and the most relevant key words would eventually be written into the page nodes.

Ignite Chicago

2009-11-11T00:00:01Z

After the Ignite in Zurich, there came Ignite in Chicago, where our American customers, prospects, partners, and Day staff met to share information, experiences, to network, and simply have a very good time. The event itself was slightly bigger than the event in Zurich, both in terms of number of participants and available room.

Ignite was hosted by Day customers, in more than one way: by the City of Chicago itself, and by the grand Intercontinental Hotel, of the IHG Group, on Chicago's famous shopping avenue, the Magnificent Mile.

Again, we had a lot of great presentations, panels, Q&A sessions, as well as informal chats. And the Foreigner concert at the end was the icing on the cake.

Be sure to check out the conference hashtag was #dayignite and here are some Ignite pictures on Flickr, with lots of new coverage from Chicago:

Looking forward to next year's Day Customer Summit!

On Version Numbers

2009-11-09T15:21:46Z

I have been thinking about using version numbers lately while working on some API extension of the Sling Engine bundle. So here is what I think versions are all about and that we all should be very careful when changing code and assigning versions to it.

On a high level versions have various aspects:

Syntax: There is no global agreement on the correct syntax of versions. I tend to like the OSGi syntax specification: The version has four parts separated by dots. The first three parts are numbers, called the major, minor and micro version. The fourth part is a plain (reduced character set) string which may be used to describe a particular version. Version numbers are compared as you would expect, except that the fourth part is employs case-sensitive string comparison comparing the actual Unicode codepoints of the characters.
Semantic: The semantics of a version define what it means to increment each place of a version. In the world of software development there is even less agreement on the semantics of version numbers than there is agreement on the syntax. The OSGi specification just defines suggested semantics.
Expectations: When seeing product version numbers people tend to have expectations towards the products. For example when Firefox went from 2.x to 3.0 we expected a major change. Likewise when Day upgraded the version number to 5 for the newest version of Communiqué the expectation is correct, that it is a major new version of the product. In fact we completely rewrote Communiqué for the 5.0 release.
Version Items: When it comes to apply version numbers to things there are quite a number of things in a single product, which may be numbered. Take for example Day Communiqué 5. There is a product - the thing you take out of the box and install on your server. Then there are OSGi bundles. Finally there are Java packages shared between the bundles and used by the application scripts.

So here are my definitions of the version number aspects layed out above.

Syntax
IMHO the Syntax for version numbers as defined in the OSGi Core specification (Section 3.2.4, Version) is good enough and clear for most uses. The nice thing about this specification is that in Section 3.2.5, Version Ranges, a syntax is defined to define ranges of versions. Such ranges are of great use when depending on other items. Most importantly of course this would be list of imported Java packages.

Semantics
As for the semantics the main problem comes from the fact, that not all versioned items understand version numbers in the same way. For example on a product level, c.f. Day Communiqué, the version number of a release is generally defined by marketing and/or product management.

I will not dive into how product numbers are to be defined. This is outside of my working knowledge and beyond may abilities ;-)

On the OSGi bundle level on the other hand and even more so on the Java package level (for OSGi package exports), the version number is more a call of the developer. Version numbers on this level are intended to convey to other developers something about the evolution of the bundle and/or package.

Let's start with exported Java packages. I tend to attribute the following semantics to the parts of a a version number:

Increasing the major version number means the API has been modified in an incompatible way. Mostly this means public classes, interfaces, methods, fields have been removed or renamed. As a consequence code using and implementing the API will break and has to be modified.

Increasing the minor version number means the API has just been enhanced in a way that is compatible for use. Increasing the minor version number, though, means that code implementing the API might have to be modified to comply with the added API like the definition of new methods.

Increasing the micro version number means that there have been some bug fixes. Generally, a pure API consisting of just interfaces has little chance for bugs which do not ammount to minor or even major version number increase. If the exported packages of a bundle happen to contain concrete or abstract classes with implementation code, bugs cannot be excluded. As such it is conceivable that a the micro version number of an exported package might be increased.

As for the qualifier part, as the fourth part of a version number is called by the OSGi specification, this meaning of this part is completely free. On a package export level, I would go as far as to say, it should not generally be used. The qualifier part may be interesting on an OSGi bundle level to create inter-release builds.

Expectations
Peoples expections as it comes to version numbers is not ease to convey. Most people expect different things. But I think one thing is common to all: If there is a version number increase something must have changed.

So, I think to use developers it is important to understand, that we only increase the version number of an item if there is a change -- not sure whether a fixed spelling error in some Java comment is change enough. Again, your mileage may vary if you happen to be product manager for a product to be soled ....

Recommendations
Based on how I understand the version number parts in terms of exported packages, here are my recommendations for package imports and bundle versions.

If you implement the exported API of another bundle, import the API package using a version range of the form [x.y,x.y+1). This means accept any increment in the micro and qualifier parts. But as soon as the minor version number changes, consider this an incompatibility.

If you use an exported API, import the API package using a version range of the form [x.y,x+1). This means accept any version starting with a minimum number upto the next breaking API chnage identified by a new major version number.

Don't increase the version number of an API package if nothing in that package has changed at all.

Bundles should be versions following versioning of exported packages. So if at least one of the exported packages has a major version number increase, the bundle's version should also have a major version number increase. Likewise for the minor number. The use of qualifiers is optional and sometimes helpful.

Apart from being driven by versioning of exported packages, bundle versions may also be increased depending on the extent of changes in the bundle. For example in the case of a pure implementation bundle, greatly increasing the functionality might give rise to a major version number increase of the bundle.

If you are using Maven to build your projects, always depend on the lowest version of a dependent module which has the API functionality you need.

Link
The Eclipse site contains a very interesting and IMHO very reality proven text about versioning of products, bundles and packages: Version Numbering

[LOTD] How Day Software stumbled upon an open source business strategy

2009-11-09T00:00:01Z

Day's CMO Kevin Cochrane has been interviewed by Matthew Aslet of the 451 group about Day's open source strategy. I particularly liked:

While many other vendors have chosen to retain control over their open source projects for commercial reasons, Day opted to relinquish control with the aim of ubiquity.

Full interview here.

The IKS semantic engine - a pragmatist's view

2009-11-09T00:00:01Z

As work on the IKS project progresses, my (extremely) pragmatic mind keeps going back to the how can we make this simpler? question.

One of the major goals of IKS is to create semantic extensions for content management systems, but what does that mean? The exact use cases are still vague, and in such a situation it is too easy to over-engineer things, just in case.

We have been talking about RESTful interfaces to IKS components for a while now, but what does this mean exactly? How can we make a concrete step towards defining such interfaces?

I'm a big fan of small concrete steps that lead us towards pragmatic solutions, so let's try to take one such step.

Machine-level use cases

Let's start by defining a few simple use cases, at the "machine level": a content management system is the client, and the IKS semantic engine the server. We have discussed this already within IKS, here's a synthetic summary:

Semantic lifting: Let IKS extract semantic information from (multimedia) content: person and place names, structured links between content items, etc. Optionally make this information editable/confirmable by the client system, as a human user might have to refine the system's suggestions.
At the machine interface level, this requires registering content with the IKS semantic engine, reading the resulting semantically lifted document, and optionally modifying it.
Classification and auto-tagging: Let IKS suggest categories and/or tags for pieces of multimedia content. If an author validates the suggestions, inform IKS of what choices were made.
From the machine interface point of view, this is very similar to semantic lifting.
Query building assistance: Let IKS assist users in formulating search queries, interactively.
From the machine interface point of view, this is very similar to semantic lifting.
Similarities, correlation: Let IKS find similarities between pieces of multimedia content. The axes on which those similarities are found can vary: images, for example, can be graphically similar, or similar in terms of the real world entities that they display.
At the machine interface level, this requires registering content with the IKS semantic engine, and later running queries against this content.

This simple list already hides significant complexity, yet those use cases should be understandable by Joe Author.

Enabling those four use cases could add a lot of value to existing and future content management systems, depending on the quality of the semantic components.

RESTful interface

Let's design a RESTful interface based on the machine interactions required to implement the above use cases.

Remember that, in what follows, client designates a content management system that wants to use the IKS engine.

Register content with IKS

To build knowledge about our content, IKS needs to be able to find it. In RESTful terms this means providing IKS with an URL that points to said content, so we have:

Rule #1: Content is registered with the IKS server by HTTP POST requests, containing lists of URLs that point to (created or modified) content items.

Rule #2: IKS reads content by making HTTP GET requests to registered pieces of content. Those URLs must return Content-Types that IKS understands. Some Content-Types are preferred and allow IKS to better understand the content.

Semantic Lifting

Once content is registered, the client can request a semantic view of that content from IKS. That view lists semantic entities that have been extracted from the content.

Depending on the IKS implementation, the semantic view can be editable. It is retrieved by a GET request that contains the IKS identifier (provided by IKS when content is registered) of the content item, and modified using an HTTP PUT request.

The Content-Type and data formats use existing standards, as far as possible.

The semantic view includes IKS-specific metadata, for example to indicate that some parts of the semantic view are still being computed.

Rule #3: The semantic view of a content item is retrieved with a GET request, and if editable can be modified by a PUT request of the modified version.

Semantic queries

Semantic queries are implemented using GET methods on various query URLs, that define how the query is interpreted.

Results are returned with similar Content-Types and data formats as used for semantic lifting.

Rule #4: Semantic queries are executed via GET requests, and return the identifiers (URLs) of the selected content items, optionally with some contextual info to display on query result pages.

IKS engine status

Semantic lifting and indexing operations might take some time, so it's useful for the client to have information on the engine's status, in machine-readable form.

Rule #5: The IKS server reserves part of its URL space for system status information, and provides status information in a structured format.

Is that it?

I think that's it - these simple RESTful interactions should be sufficient to implement our use cases.

What's left is to define the Content-Types used, and for this we can most certainly use existing formats, no need to reinvent any wheel here.

RESTful IKS framework

The proof of the pudding is in the eating, and if we wait too long the pudding might lose its taste...so why not start buiding this right away?

Purists might (rightly) argue that the above is not a design, just a somewhat vague set of principles. Yet, combined with a prototype implementation, this might be a very good way of making a step in the right direction, and of clarifying requirements and interfaces.

My suggestion for the next steps is as follows:

Implement the above interface, using dummy semantic components.
Provide system interfaces to integrate actual semantic components (semantic lifting, classification, auto-tagging, querying) as plugins.
Researchers can work on the semantic lifting components, and integrate them without requiring significant changes on the client side.

Conclusion

The best way to go forward with this is probably to create an open source project to collaborate on this RESTful IKS framework.

Even if that framework is thrown away later as the IKS architecture progresses, if would allow IKS consortium members to build a much better understanding of what's actually needed to add "semantic value" to existing and future content management systems.

Slides from the NoSQL Meetup and ApacheCon US 09

2009-11-09T00:00:01Z

ApacheCon US 09 in Oakland and the NoSQL meetup are over. Find below the slides of the talks given by Day's engineers covering Apache projects Sling, Jackrabbit, Tika and POI as well as OSGi.

Bertrand Delacretaz: JCR In 10 Minutes

View more documents from bdelacretaz.

Bertrand Delacretaz: Life in Open Source Communities

View more documents from bdelacretaz.

Carsten Ziegeler: JCR In Action

View more documents from Carsten Ziegeler.

Carsten Ziegeler: Embrace Change - Embrace OSGi

View more documents from Carsten Ziegeler.

Felix Meschberger: Rapid JCR Applications Development with Sling

View more presentations from Felix Meschberger.

Jukka Zitting: Mime Magic With Apache Tika

View more presentations from Jukka Zitting.

Paolo Mottadelli: Apache Poi Recipes

View more documents from Paolo Mottadelli.

JBoye Presentation: WCM Trends for 2010

2009-11-05T00:00:01Z

Today I had the opportunity to speak at the JBoye conference in Aarhus. It was a pleasure as every year since the audience and speakers really constitutes a who-is-who of WCM visionaries and insiders. I am definitely looking forward to coming back next year.

JBoye Presentation: WCM Trends for 2010

View more documents from David Nuescheler.

Ask the community: TibTec

2009-11-04T00:00:01Z

The Tuberculosis Project project is one of the Sling users registered on the Sling user wiki page. This is an interview with developer Audrey Colbrant who worked on the project.

Audrey, can you please tell us a bit about the TibTec Tuberculosis Project? What are the project's aims and background?

The TB project is developed by Tibtec, a nonprofit technology center based in Dharamsala (India) and directed by M. Phuntsok DORJEE. The aim of the project was to build a system to monitor the tuberculosis among tibetan communities in India, Nepal and Bhutan. Thanks to technology advances in mobile and web computing, it is now possible to design a recording and reporting web portal supporting the WHO DOTS protocol.

The project of monitoring the tuberculosis among tibetan communities in India was born 1 year ago thanks to four actors: the DoH (Department of Health, Tibetan Government in Exile), Tibetan Delek Hospital (Gangchen Kyishong - India), AISPO (Italian Association for Solidarity Of Persons), and the Johns Hopkins University (USA). TibTec is working on a system for the above four actors.

The main goal of the project is to build a simple, low-cost and versatile framework so that communities all over the world could benefit from it. The system could be easily customized for other works as well since it based on open source software.

If you want to take a look at the architecture, follow the guide.

So how did you end up using Sling? Did you compare Sling against some other frameworks?

The implementation of the TB project was part of the master project in computer science of my university. Jacques Lemordant, researcher in the WAM project at INRIA was in contact with M. Dorjee, CEO of TibTec, since several years. Together they have defined headlines of the project and chosen the more efficient technologies to be used.

Sling was chosen because we are very familiar with XML technologies (RELAX NG, XPATH, XSLT...) and hierarchical representation of data.

Another point was the fact that we wanted to access data from Android (Apache http client) and a full REST API was the simplest way to access a JCR and manipulate data represented as trees. XML being very well supported in Android, Sling is a perfect match with Android to design agile mobile web framework.

Sling is also part of a course in mobile and web technologies as the master level of the University Joseph Fourier of Grenoble.

Now that you have completed an implementation project with Sling are there any lessons learned you would like to share with the community?

The Sling approach is fairly new and I haven’t seen any other same kind of approach before. The concept is simple but it takes a little bit time to be used to the utilization. So never give up, solutions come slowly with perseverance.

If you had one free wish from the Sling committers...

Sling is a very interesting and powerful way to work with resources but difficult to handle for Sling beginners when you have a full and composite website to implement, mostly because of the lack of information on the internet.

The harder thing that gives me a lot of headaches was to find a good syntax to use that changes according to the technology you mix up.

So I think it could be helpful to have more tutorials on the syntax to use in each different case, what is better to do or not, and advice on choices to take in programming (for example I have met choices for protecting the access to the repository; choices about which kind of link is better to use: reference or path, etc).

It could be also good to finalize all links of this useful webpage

Python script to set genre in iTunes with Last.fm tags

2009-11-03T21:22:58Z

Now that I have started to seriously use iTunes I figured it might be nice to have the genre tag set in a meaningful way. Since I have a reasonably large collection of mp3s doing that manually was out of question - I wrote me a Python script to do that. There seems to be a large demand for such a functionality (at least I found a lot of questions on how to automatically set the genre tag) so maybe someone else finds the script useful. It is pasted below.

General Strategy

The basic idea is to use Last.fm's tags for genre tagging. In iTunes the genre tag is IMO best used when it only contains one single genre, i.e. something like "Electronica", not something like "Electronica / Dance". On the other hand dropping all but one tag would lose a lot of information, so I decided to use the groupings tag for additional information that is contained in the list of tags that an artist has on Last.fm. In the example above that would be something like "Electronica, Dance, 80s, German". In that way it is simple to use iTunes' Smart Playlist feature to create play lists of all, say, dance music. This approach is probably not suitable for classical music..

The ID3 field that is exposed in iTunes' UI as "grouping" is defined in the ID3v2 spec as:

TIT1: The 'Content group description' frame is used if the sound belongs to a larger category of sounds/music. For example, classical music is often sorted in different musical sections (e.g. "Piano Concerto", "Weather - Hurricane").

So, the strategy I described above seems to be kind of in line with the spec. In general, it is a good idea to have a look at the ID3v2 spec if you consider dabbling with mp3 tags.

Practical Considerations

If one would just take an artist's highest-rated Last.fm tag for the genre one would end up with pretty inconsistent genre tags (think "hip-hop", "hip hop", and "hiphop"). Therefore, I chose to use a fixed set of values for genre. In a previous version of ID3 the list of possible genres was fixed. While this is clearly a terrible idea to start with it came along handy in this case. I used his as a fixed list for genres.

The second practical consideration was which Last.fm tags to include. In Last.fm parlance each artist tag comes with a weight (values form 0 to 100). Selecting only the tags with weight larger than 50 worked out fine for me (usually I had 1-5 tags per artist).

A third thing you might want to be aware of: if you programmatically change tags in an mp3 iTunes will not pick up these changes automatically. A simple way of letting it know: select the "Get Info" command on these items. This will trigger a reload of the new tag values.

Script

To run the script you will need the Python libraries mutagen and pylast installed. Run it with the option
-d directory_with_mp3s
The script will walk along this directory and modify all mp3s it finds. Also, you will need a Last.fm API key and set your API_KEY and API_SECRET accordingly in the script.


#!/usr/bin/env python
# encoding: utf-8
"""
tag_groupings.py

Created by Michael Marth on 2009-11-02.
Copyright (c) 2009 marth.software.services. All rights reserved.
"""

import sys
import getopt
import pylast
import os.path
from mutagen.id3 import TCON, ID3, TIT1

help_message = '''
Adds ID3 tags to mp3 files for genre and groupings. Tag values are retrieved from Last.FM. Usage:
-d mp3_directory
'''

class Usage(Exception):
   def __init__(self, msg):
       self.msg = msg

all_genres = TCON.GENRES
genre_cache = {}
groupings_cache = {}
API_KEY = "your key here"
API_SECRET = "your secret here"
network = pylast.get_lastfm_network(api_key = API_KEY, api_secret = API_SECRET)

def artist_to_genre(artist):
   if genre_cache.has_key(artist):
       return genre_cache[artist]
   else:
       tags = network.get_artist(artist).get_top_tags()      
       for tag in tags:
           if all_genres.__contains__(tag[0].name.title()):
               genre_cache[artist] = tag[0].name.title()
               print "%20s %s" % (artist,tag[0].name.title())
               return tag[0].name.title()

def artist_to_groupings(artist):
   if groupings_cache.has_key(artist):
       return groupings_cache[artist]
   else:
       tags = network.get_artist(artist).get_top_tags()
       relevant_tags = []
       for tag in tags:
           if int(tag[1]) >= 50:
               relevant_tags.append(tag[0].name.title())
       groupings = ", ".join(relevant_tags)
       groupings_cache[artist] = groupings
       print "%20s %s" % (artist,groupings)
       return groupings

def walk_mp3s():
   for root, dirs, files in os.walk('.'):
       for name in files:
           if name.endswith(".mp3"):
               audio = ID3(os.path.join(root, name))
               artist = audio["TPE1"]
               genre = artist_to_genre(artist[0])
               grouping = artist_to_groupings(artist[0])
               if genre != None:
                   audio["TCON"] = TCON(encoding=3, text=genre)
               if grouping != None:
                   audio["TIT1"] = TIT1(encoding=3, text=grouping)
               audio.save()

def main(argv=None):
   if argv is None:
       argv = sys.argv
   try:
       try:
           opts, args = getopt.getopt(argv[1:], "ho:vd:", ["help", "output="])
       except getopt.error, msg:
           raise Usage(msg)
 
       # option processing
       for option, value in opts:
           if option == "-v":
               verbose = True
           if option in ("-h", "--help"):
               raise Usage(help_message)
           if option in ("-o", "--output"):
               output = value
           if option in ("-d"):
               try:
                   os.chdir(value)
               except Exception,e:
                   print "error with directory " + value
                   print e
       walk_mp3s()      
 
   except Usage, err:
       print >> sys.stderr, sys.argv[0].split("/")[-1] + ": " + str(err.msg)
       print >> sys.stderr, "\t for help use --help"
       return 2

if __name__ == "__main__":
   sys.exit(main())

JCR in 15 minutes

2009-11-03T18:02:23Z

We had a great NoSQL meeting yesterday evening colocated with ApacheCon. Thanks Jukka for organizing!

I was in track B for the second part, and found it very interesting to compare three different approaches to non-relational content storage: MarkLogic server, JCR and Pier Fumagalli’s Lucene+DAV technique.

I also quite liked Steve Yen’s “horseless carriage” way of looking at NoSQL. Defining things by what they are, as opposed to what they are not, sounds like a good idea.

I gave a short talk about JCR, find the slides below. Of course, as usual, they’re not as good as when I’m here to talk about them ;-)

Interviewed by Internet Briefing Blog

2009-11-03T07:18:19Z

Reto Hartinger of the Internet Briefing group has interviewed me about what I work on these days. The interview is in German.

NoSQL Meetup and ApacheCon 09 in Oakland

2009-11-02T00:00:01Z

ApacheCon US 09 starts today in Oakland. A couple of Day's engineers will give talks, not just about the usual suspects Sling and Jackrabbit, but also Tika and POI (details below).

Also, Jukka Zitting has helped organize a NoSQL meetup in Oakland starting tonight where Bertrand Delacretaz will talk about JCR.

Bertrand Delacretaz: Life in Open Source communities: Open Source communities often seem to have their own unwritten rules of operation and communication, their own jargon and their own etiquette, which sometimes make them appear obscure and closed to outsiders. In this talk, we'll provide recommendations on how to get touch with, and how to join, Open Source communities. Based on ten years of experience in various Open Source projects, we will provide practical information on how to communicate effectively on mailing lists, how to formulate questions in an effective way, how to contribute in ways that add value to the project, and generally how to interact with Open Source communities in ways that are mutually beneficial. This talk will help Open Source beginners get closer to the communities that matter to them, and help more experienced community members understand how to welcome and guide newcomers.

Carsten Ziegeler: JCR in Action - Content-based Applications with Jackrabbit: The Java Content Repository API (JCR) is the ideal solution to store hierarchical structured content, and to develop content-oriented applications. This session provides a practical introduction to help you get started using JCR in your own application. To demonstrate the basic architecture of such applications, a sample content-based application will be developed during the session. Basic techniques will be explained, including navigation, searching, and observations, using the Apache Jackrabbit project.

Embrace OSGi - A Developer's Quickstart: In theory, the first choice for highly modular, dynamic, and extensible applications is OSGi technology. The theory sounds very tempting, but what about the real world? Starting with the basics of OSGi, this session is focused on practical examples, tools, and procedures for a rapid adoption of OSGi in your own projects. Learn how to avoid the typical traps and how to get the most out of OSGi

Felix Meschberger: Rapid JCR applications development with Sling: Apache Sling is an OSGi-based, scriptable applications layer, using REST principles, that runs on top of a JCR content repository. In this talk, we'll see how Sling enables rapid development of JCR-based content applications, by leveraging the JSR 223 scripting framework. We'll also look at the rich set of OSGi components provided by Sling. We will create a simple application from scratch in a few minutes, and explain a more complex multimedia application that does a lot with just a few lines of code. This talk will help you get started with Sling and understand how the different components fit together.

Jukka Zitting: MIME Magic with Apache Tika: Apache Tika aims to make it easier to extract metadata and structured text content from all kinds of files. Tika is a subproject of Apache Lucene, and leverages libraries like Apache POI and Apache PDFBox to provide a powerful yet simple interface for parsing dozens of document formats. This makes Tika an ideal companion for Apache Lucene, or for any search engine that needs to be able to index metadata and content from many different types of files. This presentation introduces Apache Tika and shows how it's being used in projects like Apache Solr and Apache Jackrabbit. You will learn how to integrate Tika with your application and how to configure and extend Tika to best suit your needs. The presentation also summarizes the key characteristics of the more widely used file formats and metadata standards, and shows how Tika can help deal with that complexity. The audience is expected to have basic understanding of Java programming and MIME media types.

Paolo Mottadelli: Apache POI recipes: The Apache POI project provides Open Source Java APIs for the manipulation of Microsoft Office format files. It was developed to provide OLE2 Compound Document format support. POI support for the new format was necessitated by the proliferation of new Office Open XML (OOXML) documents, due to its standardization. As a result, a common challenge emerged for projects that leverage POI to read and write Excel, Word, and PowerPoint documents: supporting the new format while maintaining backward compatibility with the earlier one. This session provides an overview of how the new POI architecture makes that challenge easier, using the common interfaces package and their double implementation. Participants will also learn about the main new features provided by POI towards support of the new OOXML format. To demonstrate POI's features, this session will also drive through a collection of practical recipes to solve the tough problems of integrating Office documents in your enterprise applications.

Life in Open Source Communities, live at ApacheCon!

2009-10-30T10:18:26Z

I have just finished my slides for next week at ApacheCon. Though the topic of how to “survive” in our open source communities has been on my mind for a while, this is a totally new presentation, which is both great (in the blank slate sense) and a lot of work.

Having recently read Presentation Zen (very recommended if you do presentations and/or like beautiful books), I started adding full-screen pictures to the first few slides, and couldn’t stop! The presentation will then consist of me ad-libbing (or more precisely trying to tell stories) on a series of nice pictures grabbed from morguefile.com (don’t worry about that name).

I’ll post the slides here later, for now they are super secret, so you’ll just get the teasers…images courtesy of morguefile.com (update: slides added now).

Hope to see you next week! In any case I have collected a number of useful links in my delicious bookmarks, I’ll point people to them in the presentation.

Bertrand on the telly (brush up your French)

2009-10-28T00:00:01Z

Bertrand Delacretaz was interviewed at the OpenWorld Forum in Paris about the Apache Software Foundation. It is an ASF primer - apparently the ASF and its ways of working are relatively unknown in France.

Author-centric feature scoping and integration

2009-10-28T00:00:01Z

Won't somebody please think of the ~~children~~ authors?
(almost Helen Lovejoy)

Have you seen the component for multi-variate testing (MVT, aka A/B testing) in the upcoming CQ5.3 release? I saw it demoed by David at Ignite and was completely blown away: CMS users (authors) can simply drag a couple of alternative banners onto the component right from within their regular editing interface (the CMS will then show the different versions to different users and count the click-through rates so that eventually the best performing banner is determined).

The MVT feature reminded me of two other CQ5 features: personalisation and analytics. All three are truly and seamlessly integrated into the user interface of the authors and they all provide less functionality than full-blown standalone solutions. To give you a concrete example: for each page the editors see right in the site admin (i.e. in their daily user interface) how many views each page got in the last 30 days. Clearly, this is no match for the kinds of reports you get on, say, Google Analytics, not even the same ballpark. Yet, I still think the authors get something that is valuable for them: they see it right away what is of interest to their audience.

(there is screencast available that demonstrates personalisation features from an author's perspective, registration needed)

When I compare this author-centric evaluation of functionality with my usual point of view as a system architect the business value of a feature for an author might be determined by:

ease of use rather than feature richness and
seamless integration into the UI

This is probably true for most systems that have non-technical users, but I believe the effect is amplified in CMSs because many CMS users use the system only once in a while rather than regularly.

Of course, this author-centric view on features should not necessarily dictate the underlaying systems architecture especially when you look at a complete content management solution encompassing analytics, personalization etc. The architecture might still be full-stack or best-of-breed and I do not want to postulate one being better than the other. However, I believe that one way of knowing that you got the author's user experience right is when you cannot see the system architecture reflected in the UI anymore. Or to put it the other way round: the UI should not enable you to guess which box which feature is running on.(*)

Thanks to Lars for providing discussions and ideas on this post.

(*) This idea is adjacent to a pet subject of mine: the user interface for basic content management (CRUD stuff) should not enable you to guess the underlying content/data schema. Sadly, up to today not few CMSs UIs resemble ERP-style data entry masks.

How well does the french-speaking world know the Apache Software Foundation?

2009-10-27T10:03:32Z

I was at OpenWorldForum in Paris a few weeks ago, together with fellow Apache members Sander Striker and Emmanuel Lécharny.

My first impressions (apart from the fact that Paris is always nice – I knew that already) were that the French tend to wear suits and say “vous” (polite form of “you”) instead of “tu” (the familiar form) which I would tend to use in geeky circles. Cultural differences…

But more seriously, how well does the french-speaking world know the Apache Software Foundation? Not well, it seems to me. In most of our discussions people could associate the ASF with the Apache HTTP Server project, but not much more. 2′000 committers? 300 members? Really?

To help improve this, I hope that the ASF can take a more active role in the conference next year, I’ll bring this up next week at ApacheCon with our conference people.

In the meantime french-speaking folks are welcome to learn a bit more about it thanks to TiViPRO’s interviews of Emmanuel and myself, shot during the conference.

NoSQL interests

2009-10-27T10:01:12Z

We’re organizing a NoSQL meetup in Oakland on Monday next week. In addition to helping set the meetup agenda, the “Topics you are interested in” question in the sign up form provides some interesting insight on the current interests of the NoSQL community. Here’s a quick breakdown of the key terms distilled from the 88 signups we’ve received so far.

Note that the data is biased towards Apache projects due to the meetup being organized at ApacheCon US 2009.

Projects

The following open source projects were mentioned. The list is in alphabetical order, as the data set is too small to make any reasonable ordering by popularity.

Topics

Many responses were about the “big data” aspect of the NoSQL movement. Some frequent keywords: distributed storage, large transactional data, consistency, failover, availability, reliability, stability, failure detection, failed node replacement, (petabyte) scalability, consistency levels, storage technology, performance, benchmarks, optimization, backup and recovery, map/reduce

Another common theme were the various database types and the NoSQL “development model”. Keywods: document stores, key/value stores, consistent hashing, graph databases, object databases, persistent queues, content modeling, migration from the relational model, social graphs, streaming, software as a service, offline applications, full text search, natural language processing

Beyond the above big themes, I found it interesting that the following technologies were specifically named: Erlang, Java, WebSimpleDB, WebDAV

In addition to specific topics, many people were asking for case studies or “lessons learned” -type presentations.

Colayer's approach to collaboration software

2009-10-24T14:23:21Z

Chances are you have not heard of Colayer, a Swiss-Indian company producing a SaaS-based collaboration software. I did a small project with the guys, that is how I got to know. When I first saw their product I immediately thought the guys are onto something good, so it is worthwhile to share a bit of their application concepts. They follow an approach I have not seen anywhere else.

On first look Colayer seems to be a mixture between wikis and forums: the logical data structure resembles a hierarchical web-based forum and the forum entries are editable and versioned like in a wiki. But there is more: presence and real-time. All users that are currently logged in are visible and one can have real-time chats within the context of the page one is in or see updates to the page in real-time (similar as in Google Docs). These chats are treated as atomic page elements (called sems in Colayer parlance) just like the forum entries or other texts. Through this mechanism, all communication around one topic stays on one page and in the same context.

There are two more crucial elements: time and semantics. All sem's visibility is controlled by their age and their importance. As such, a simple chat is given less weight that a project decision and will fade out of view after some time. All new items from all pages (i.e. discussions or topics) are aggregated on a personal home page and shown within the context where they occurred.

Below is a screenshot of such different sems in one page. One page corresponds to one topic or forum or wiki page. You can see the hierarchical model and the different semantics (denoted by the colors).

Here is an example screen shot that aggregates different recent sems on one page (essentially a context-aware display of new items including time and context in the same display). Note that this way of displaying new items manages to map importance, time and context into a two-dimensional page, which I find a very cool achievement.

The funny thing about Colayer's product (especially when compared to Google Wave) is that one "gets it" when first looking at it. It solves a problem I am facing in my work on a daily basis: where to put or find crucial information - on an internal mailing list or on the wiki?

The Colayer application is delivered as a browser-based SaaS solution (mainly targeted towards company-internal collaboration). This limits potential usage scenarios outside of the firewall. It would be cool if Colayer found a way of opening up their application to other data sources or consumers. It would be worth it, the app rocks.

Ignite Zurich

2009-10-19T00:00:01Z

Ignite in Zurich was a blast. Lots of good presentations (which I will hopefully be able to share later) and excellent discussions on content management. My favourite quote was from Newsweek's Meshach Jackson. Their editors on the CMS selection process:

If you don't select CQ we're all quitting.

I also enjoyed what David Nuescheler had to say about separation of content and layout. The standard CMS architecture thinking goes like this
a) we need to separate content and layout
b) therefore, the user interface for the authors cannot look anything like the rendered pages and we will make it look like a database entry mask

David demonstrated that a) simply does not lead to b). Or in his own words:

Separating content and layout does not mean you have to confuse the authors

To get a taste of the event: The conference hashtag was #dayignite and here are some Ignite pictures on Flickr:

So, in case you missed Ignite Zurich there is still Ignite Chicago coming up...

Putting POI on a diet

2009-10-16T22:39:21Z

The Apache POI team is doing an amazing job at making Microsoft Office file formats more accessible to the open source Java world. One of the projects that benefits from their work is Apache Tika that uses POI to extract text content and metadata from all sorts of Office documents.

However, there’s one problem with POI that I’d like to see fixed: It’s too big.

More specifically, the ooxml-schemas jar used by POI for the pre-generated XMLBeans bindings for the Office Open XML schemas is taking up over 50% of the 25MB size of the current Tika application. The pie chart below illustrates the relative sizes of the different parser library dependencies of Tika:

Both PDF and the Microsoft Office formats are pretty big and complex, so one can expect the relevant parser libraries to be large. But the 14MB size of the ooxml-schemas jar seems excessive, especially since the standard OOXML schema package from which the ooxml-schemas jar is built is only 220KB in size.

Does anyone have good ideas on how to best trim down this OOXML dependency?

[ANN] Sling Users List now available

2009-10-07T00:00:01Z

Apache Sling keeps on growing up. Today's step: Felix Meschberger announced a users mailing list (before there was only one mailing list for developers and users). The new list is also available in the Discussion Groups section of dev.day.com.

If you use Sling please consider adding your project to the Sling users wiki page.

Re: RESTful daydream #4

2009-10-07T00:00:01Z

Justin Cormack has written a noteable blog post titled "RESTful daydream #4" (pun intended?) about RESTful content repositories. I disagree with some of what Justin writes about JCR and Sling. However, I completely share his vision about REST's role in content management and I am with him regarding the overall theme of his post.

Essentially, Justin asks for a "web content repository":

The odd thing is that a web content repository alone surely lends itself to a simple REST architecture. Content is after all lots of small resources with relations. [...] It takes content, relates it to other content, and serves it back, with authentication and versioning. Everything else is in other system layers, transforming it and so on. Not simple, but well defined; lower level than JCR + Sling say.

Apache Sling provides a RESTful interface onto content, but being a web application framework it provides much more, especially scripting. So I can understand well why Justin dismisses Sling as being too powerful. However, I believe that Jackrabbit's native http layer is pretty much on the mark (Justin also dismisses it in the comments as being not RESTful. I do not know why.). As far as I know the Jackrabbit http remoting layer is quite work-in-progress so one might argue about the details regarding its RESTfulness, but overall it fits what I understand to be a RESTful web content repository.

There is one aspect regarding a web content repository I would like to add to the discussion: I think it is crucial that the representations of resources and be consumed by browsers. HTML forms for writing and presenting resources in JSON should be part of the equation. Adding models or semantics on top of that ~~will~~ might do more harm than good.

Do we need a formal standard for web content repositories? From my perspective: not yet. At the moment we still need to learn more, i.e. more repository implementations and users that have built a CMS on top of a web content repository.

Found out about FlexEvent.UPDATE_COMPLETE

2009-10-01T08:45:03Z

Here is a small bit about the asynchronous nature of the Flex layout mechanisms that I learned while slapping together a presales demo yesterday:

When changing properties of UIComponents listen for FlexEvent.UPDATE_COMPLETE events. They get fired when the change is actually done. In my case I needed to get the textWidth of a Label after changing the label text. Right after calling the setter of text the getter of textWidth will still return the old value.

[Some background reading]

[LOTD] Why JCR is good for Content Management?

2009-10-01T00:00:01Z

eXo's Peter Nedonosko discusses on his blog why JCR is good for content management. I could not agree more with his conclusion: it is useful for cms developers.

JCR standard is useful for CMS [...]

Talk at Java User Group

2009-09-30T06:12:27Z

Yesterday, I gave a talk at the Java User Group Switzerland (JUGS) titled "Agile RESTful Web Development". It was about the REST style in principle and hands-on RESTful development with Apache Sling. I enjoyed giving the talk and think that it was well received. Here's the slide deck:

Agile RESTful Web Development

View more documents from mmarth.

[ANN] Talking at openworldforum in Paris this Friday

2009-09-29T00:00:01Z

I'll be talking at Open World Forum in Paris later this week, presenting a condensed version of my Open Source Collaboration Tools are Good for You talk, this Friday at 13:40 (and no, the topic is not "Coming Soon" as indicated on the program ;-)

Later on Friday afternoon, I'll take part in a forum (in french) on the future of open source forges. I haven't seen the list of participants yet, but the topic looks promising.

Sander Striker, Executive Vice President of the Apache Software Foundation, will also be there for a roundtable this Thursday after the 16:30 keynote.

And I'll be staying with fellow Apache member Emmanuel Lecharny - meeting the locals is always nice, and Emmanuel's a city cyclist as well, so we'll be able to trade city jungle tricks I guess.

The conference looks quite busy with lots of interesting presentations, roundtables and workshops.

Looking forward to it, and make sure to say bonjour if you're around!

Data First in Cloud Persistence

2009-09-25T00:00:01Z

My colleague Cedric Huesler gave a talk "Data First in Cloud Persistence" at yesterday's CloudCamp in London. Missed it? You'll have another chance next week at the Frankfurt CloudCamp. Meanwhile, here's the slide deck (I love the second slide):

Data First in Cloud Persistence

3 weeks to go - next CQ5 at Customer Summit

2009-09-24T00:00:01Z

In 3 weeks from today we are going to kick-off our annual Customer Summit in Zurich (and two weeks later in Chicago). Going through the office in Basel feels like everybody is busy making sure the events are going to be awesome.

The engineering team is putting the final touch on what has been in the making for multiple years - yes - I'm talking about JCR 2.0. It is exciting to witness the implementation of the next generation content infrastructure API and see what it does for the user facing content applications. The next version of our WCM, DAM and Social Collaboration platform CQ5 - to be unveiled at the Customer Summits - is taking full advantage of JCR 2.0 (JSR-283). This for sure is great news for developers. They get an extended rock solid API to build content applications and integration services.

Another mission - with focus to reduce the distance between the author and the consumer (visitor) - resulted in a slew of new CQ5 features. We want content authors to have a better insight on how their content is consumed and provide feedback what worked and what not. With this, we are closing an important loop in providing targeted content to an apparently unknown audience.

If that is not enough, I'm sure you are interested to learn how the City of Chicago, Adobe, New York University (in Chicago), Telecom Italia, Boehringer Ingelheim and Investec (in Zurich) are using the Day products. That's only an excerpt of companies and partners that will participate - we are finalizing the agenda this week and will make it available starting next week.

For all of you already signed-up for the Summit - we are looking forward to share with you the latest awesomeness in Zurich or Chicago. If you haven't - it's not too late.

Some graphics work for a change

2009-09-23T17:11:59Z

I’ve recently spent some effort in improving the look of the Apache Jackrabbit website. I’m no designer, so the results aren’t that great, but it’s been a nice break from the regular project work. And I got to brush up my Photoshop and Gimp skills.

One part of the effort was creating an icon for the site. Previously the site used the feather icon used as the default on all Apache project sites, but I wanted a Jackrabbit-specific icon that helps me to quickly identify and access Jackrabbit pages among the numerous tabs I usually have open in my browser. The work is a good example of incremental improvements in action:

I started with a copy of the Jackrabbit logo with nice alpha-layered transparent background. It looked great until I noticed that some browsers lost the smooth alpha layer and instead resulted in a rather badly aliased icon seen above.

The straightforward solution was to add a white background as can be seen in step 2. That worked already pretty well in all browsers.

After a few days of watching the icon I found it a bit too blocky to my taste, so I tried to restore some of the nice transparency effect by rounding the corners a bit. I’m pretty happy with the result.

Of course, if you have design talent and think you can do better, go for it!

Release time

2009-09-19T11:02:43Z

There’s lots of upcoming release activity at the Apache projects I’m more or less involved with:

The incubating Apache PDFBox project is just about to release the eagerly anticipated 0.8.0 release. I’m expecting to see the release announcement on Tuesday next week. PDFBox is a Java library for working with PDF documents.
Another incubating project, Apache UIMA, is working towards the 2.3.0 release. I’m looking forward to seeing both UIMA and PDFBox graduating from the Apache Incubator shortly after the respective releases. UIMA is a framework and a set of components for analyzing large volumes of unstructured information.
The Apache Sling project is a component-based project like Apache Felix, so there is no clear project-wide release cycle. Instead Sling is about to start releasing new versions of most of the components changed since the all-inclusive incubator releases. Sling is a JCR-based web framework.
Apache Tika uses PDFBox for extracting text content from PDF documents. I’m hoping to see a Tika 0.5 release soon with the latest PDFBox dependency and the design improvements I’ve been working on. Tika is a toolkit for extracting text and metadata from all kinds of documents.
Apache Solr is about to enter code freeze in preparation for the 1.4 release that will include the “Solar Cell” feature based on Tika. Solr is a search server based on Lucene.
The Commons IO project has been upgraded to use Java 5 features and I’m starting to push it towards a 2.0 release. Commons IO is a library of Java IO utilities.
Lucene Java is gearing up for the 2.9 release, and will soon follow up with the 3.0 release. The trie range feature is an especially welcome addition for many use cases. Lucene is a feature-rich high performance search engine.
And last but not least, Apache Jackrabbit is getting ready to release the 2.0 version based on the recently approved JCR 2.0 standard. Jackrabbit is a feature-complete JCR content repository implementation.

I’m hoping to see most of these releases happening in time for the ApacheCon US 2009 conference in early November.

JSR-283 builds on JSR-170

2009-09-18T00:00:01Z

And of course, JSR-283 builds on the work done in JSR-170:

734	issues
662	issues in JSR 170.
1456	days
1213	days from start to finish of JSR 170.
72	individual members of the expert group
1/2	as many members in the 170 expert group.
43	organizations represented
1/2	as many organizations represented.
277	pages of spec
297	pages in 170. Yet 283 contains so much more! :-)
87	interfaces and classes
59	interfaces and classes in 170.
522	fields and methods
334	fields and methods in 170.

Ruby script for generating Google sitemaps

2009-09-17T13:05:15Z

I just wrote a small Ruby script to generate a Google sitemap out of file directory. I thought that it might come handy as a quick start for someone, so here is the code (requires the builder gem)

require 'builder'
htmlfiles = Dir.glob("*.html")
x = Builder::XmlMarkup.new(:target => $stdout, :indent => 1)
x.instruct!
x.urlset( "xmlns" => "http://www.sitemaps.org/schemas/sitemap/0.9" ) {
for file in htmlfiles do
x.url{x.loc "http://www.example.com/#{file}"; x.lastmod "2009-09-17"; x.changefreq "monthly"; x.priority "0.8"}
end
}

JSR-283 passes its final Test

2009-09-14T00:00:01Z

734	issues
1456	days
72	individual members of the expert group
43	organizations represented
3	ballots of the executive committee of the JCP
41	yes votes
0	abstentions
0	no votes
5	face-to-face meetings
76	telephone conferences
277	pages of spec
87	interfaces and classes
522	fields and methods
1895	testcases in the official TCK
100%	signature coverage of the TCK
2.0	Content Repository for Java Technology API

[LOTD] Apache Projects: last 10 years, next 10 years

2009-09-03T00:00:01Z

eWeek has looked back 10 years of Apache Software Foundation (see also this interview). They have identified 11 Apache projects that they think have changed computing in the last 10 years (an awesome lineup!) as well as 10 technologies that will change computing in the next 10 years. Amongst the latter: Apache Jackrabbit and Sling.

OpenJCR

2009-09-01T00:00:01Z

A new effort has been started over at java.net to implement

an open source community project building a free and comprehensive next generation content repository defined by the Java Content Repository API specification (JSR-170 and JSR-283)

The project is called OpenJCR. Its aims are

OpenJCR is designed to address large deployments, to provide high performance, to be highly extensible, secure, and easy to deploy in a server or embedded environment.

Not much to see so far, but then the project seems to be really young. In any case, it is good to see another JCR open source implementation project popping up.

SimPCoRe: PHP-based content repository

2009-08-31T00:00:01Z

Via @georgwaechter I have come across SimPCoRe which is a content repository for PHP. Judging from the description and this discussion SimPCoRe does not fully implement JSR-170, but only a smaller feature set. It does, however, seem to implement the basic ideas and concepts of a (Java) content repository.

It is great to see the ideas and concepts of the "content repository" spreading - be it standards-based or an adaption/evolution or a minimal feature set. Anything that helps with the implementation of better content management systems is a good trend (see also this previous post). Especially, given the large amount of PHP-based CMSs I am happy to see more PHP developers working on content repository implementations (others are Jackalope and Flow3).

Jackrabbit 1.6 comes with native http access layer

2009-08-31T00:00:01Z

Jackrabbit 1.6 has been released a couple of days ago and it comes with a very cool, but somewhat hidden feature: it is now possible to use the Content Repository through http. There is direct read/write access to the repository even from the browser.

Of course, through Sling this was possible before, but now http-based access is native to the Content Repository itself. However, having Jackrabbit-native http access makes it much easier to use JCR from whichever framework or language you happen to use to write applications. In that sense, I tend to think of this as a game changing feature for Jackrabbit and JCR.

Unfortunately, the name of this access capability is not something one would expect like "RESTful this" or "Web that", but "JCR Remoting". Anyway, to check it out run the Jackrabbit 1.6 standalone server and find the description and some examples at http://localhost:8080/webdav-jcr.jsp

Nodes can be read through a GET request for /path/to/node.json , whole trees can be read in one request through a request to /path/to/node.treedepth.json (which is the format known from Sling). Adding, deleting or moving nodes is possible by sending POSTs in a specific format defined in the tutorials at http://localhost:8080/remoting/write.jsp

When I read the JCR Remoting documentation I got very confused by two concepts: WebDAV extensions and the JCR Remoting Client. Here is what I found out:

The intention of JCR Remoting is to make available the complete JCR functionality. At the moment, not all of that is implemented in the protocol. For this reason WebDAV is used as a fallback (for now). This makes sense, because WebDAV is http-based as well.

The JCR Remoting Client is a set of Java classes that implement the JCR Remoting protocol and provide clients with a Java-based JCR session. Essentially, this is an alternative to an RMI-based connection to a Java Content Repository. It might be of interest to you if you intend to use Java to connect to the repository, but you can safely ignore it if you do not intend to do so.

David Nuescheler on Commercial Open Source Software

2009-08-31T00:00:01Z

Italian open source blogger Roberto Galoppini has recently joined in the discussion about open source business models that got started by Eric Barroca and Dirk Riehle. For his post Roberto interviewed Day's CTO David Nuescheler and published parts of it. Please find the complete interview below.

On standard participation costs

It definitely depends on the level of participation in a particular standardization process. Simple participation in an expert group or a technical committee can require only a small effort such as a couple of hours of work per month or week. An active participant in a standardization can easily rack up numerous hours per week ranging up to a couple of FTEs. When we talk about leading a specification effort, like we did in JSR-170 or JSR-283, there are a number of subject matter experts more or less constantly involved.

The burden on editor or specification lead also varies by standardization body. With some standardization bodies there is no requirement to produce a testkit to certify or measure compliance of an implementation with a standard.

However, for example in the JCP the specification lead must provide a testkit (TCK) and even a reference implementation the passes 100% of the testkit to prove that the entire specification does not just look good on paper but is completely implementable. Providing a testkit and a reference implementation certainly makes it much harder to complete a standard but it of course also makes the final result a much more solid specification.

In our specific case we did not really see the standards participation as a cost but much more as a part of a our leading edge research and development efforts that we would incur anyway. The choice that we have is whether to do that research behind closed doors and receive our first input from our customer base once the product is shipped or if we want to discuss these topics with the brightest minds in the industry by leading such a standardization. We very much view the expert group in a specification as our extended R&D-team that we can leverage for architecture discussions that matter to the entire industry.

On symbiotic costs (aka competition costs within Apache projects)

We generally feel that through developing most of our infrastructure code in public there are various benefits for us as a commercial vendor.

To ensure that we gained maximum benefit we chose to collaborate with the Apace Software Foundation. In particular because it is renowned for its clean cut meritocracy and true-spirited open source, from its very liberal licensing to the diversity-oriented community and foundation processes.

Traditionally I think that people view community participation in terms of contributing code patches or feature donations. In our experience the most valuable contributions from the community comes in completely unexpected forms. These include the use of a very public active mailing list as a sounding board and self-help archive, issues reporting, the fact that the community is building the software around-the-clock, having a very agile test harness of the latest features and changes in the form of the community and ultimately the prospect of being able to then hire the brightest minds in the industry having seen their great community work on “our technology”.

Most importantly though I think that learning to develop in the “Apache Way” has greatly improved our internal communication skills and has greatly enhanced the overall quality of our products.

We feel that only on rare occasions do we compete against “our” Apache open source projects and we have definitely been able to keep our open source projects very competitive within the open source world.

On code production tracking costs

The production of code that ends up in our products happens in various ways already. I would argue that every software vendor leverages open source and closed source libraries. Some vendors may rely more on open source libraries than others.

At Day we believe in producing a lot of our code ourselves. If we find ourselves in a position where we need to develop commodity code that is generally applicable and reusable we try to open source as much as possible.

This results in our active participation in a lot open source projects (see e.g. here)

Keeping your code open of course also allows to use public, 3rd party tracking tools and statistics about open source projects and allows to compare with peers (e.g. on Ohloh)

Implementing Java Interfaces and Generics

2009-08-30T19:20:42Z

In an earlier post I asked for an implementation of the following Java interface in Scala:

public interface Iterator2 extends java.util.Iterator {}

The solution was to use a combination of self types and existential types. However, some comments made me aware of Ticket #1737 where a more general form of the problem is discussed. Given the following Java code:

public interface A {
    void run(T t);
    T get();
}

public abstract class B implements A {}

How could B be implemented in Scala? My previous approach is not sufficient here since for implementing the run() method, one needs to be able to refer to the existential type in run()’s parameter list:

class C extends B {
  this: A[_] =>

  def run(t:???) {} // What should go here for t's type?
  def get = "hello world"
}

So what we need is a way to name the existential type used in the self type (that is the type parameter to A) such that we can refer to it in the parameter list of the run() method. Here is my shot:

class D {
  type Q = X forSome { type X; }
}

class C extends B {
  this: A[D#Q] =>

  def run(t:D#Q ) {}
  def get: D#Q = "hello world"
}

Unfortunately this results in a compilation error with Scala 2.7.5

error: class C needs to be abstract, since method run in trait A of type (T)Unit is not defined

and in a AssertionError for Scala 2.8.0.r18604-b20090830020201

Exception in thread "main" java.lang.AssertionError: assertion failed: michid.iterator.A[T]
at scala.Predef$.assert(Predef.scala:107)
at scala.tools.nsc.symtab.Types$TypeRef.transform(Types.scala:1417)
at scala.tools.nsc.symtab.Types$TypeRef$$anonfun$baseTypeSeq$3.apply(Types.scala:1588)
...

I created Ticket #2091 for these problems.

Posted in Uncategorized Tagged: Scala

Are you using Sling?

2009-08-27T00:00:01Z

Are you using Apache Sling for internal projects or commercial software or anything else? Let the world know and add yourself to the wiki page of Sling users that got just started by Sling committer Ian Boston (the wiki page also contains an interesting summary on Sling's impact on the code base of the Sakai project).

Eclipse unresolved compilation problems vs. Maven

2009-08-21T12:43:23Z

The Eclipse Java compiler is built so that it almost always creates a valid class file - even if there are problems like unresolved imports. In that case the problematic method will contain a statement throwing an exception, literally:

public void myMethod() {
    throw new Error("Unresolved compilation problems: \n\t
                     The method isA(Resource, String) is
                     undefined for the type ResourceUtil");
}

The original code of the method gets removed and the class file is valid, although the source is not compilable! This is really weird stuff when you see that error on a running system first and scratch your head how such an exception gets into the system. At least it gets (surprisingly) clearer when you look at the decompiled class file. This "feature" is seen as a good thing by the Eclipse devs, it seems to be required for the incremental compilation style of Eclipse and cannot be disabled apart from completely disabling the Java builder for a project, which turns Eclipse into a not-so-good text editor.

Anyway, this leads to a really annoying problem in conjunction with our favorite Java build system: if the Maven build and Eclipse share the same output directory (eg. target/classes, which is the default if you use mvn eclipse:eclipse), a Maven build that creates a jar from those classes will pick them up without noticing that there is a compilation problem. Because the class file is up-to-date compared to its source file, Maven won't recompile it and simply puts it into your jar. Then it gets deployed, executed and your system throws that nasty error. And you wonder "how could a compilation error get unnoticed into a running system?"

The only solution I found is to avoid sharing of the output directory for classes between Maven and Eclipse. To do that with the maven eclipse plugin, use the "outputDirectory" parameter:

mvn eclipse:eclipse -DoutputDirectory=target/eclipse-classes

That should ensure you don't accidentally run across that problem again.

Update: As noted by the commenters, always using mvn clean as part of your build process helps as well, although that slows down development a bit. For release builds you should always do a clean.

Apache Jackrabbit 1.6.0 released

2009-08-11T15:11:03Z

The Apache Jackrabbit project has just released Jackrabbit version 1.6.0. This release will most likely be the latest JCR 1.0 -based Jackrabbit 1.x minor release before the upcoming Jackrabbit 2.0 and the upgrade to JCR version 2.0. The purpose goal of this release is to push out as many of the recent Jackrabbit trunk improvements as possible so that the number of new things in Jackrabbit 2.0 remains manageable.

The most notable changes and new features in this release are:

The RepositoryCopier tool makes it easy to backup and migrate repositories (JCR-442). There is also improved support for selectively copying content and version histories between repositories (JCR-1972).
A new WebDAV-based JCR remoting layer has been added to complement the existing JCR-RMI layer (JCR-1877, JCR-1958).
Query performance has been further optimized (JCR-1820, JCR-1855 and JCR-2025).
Added support for Ingres and MaxDB/SapDB databases (JCR-1960, JCR-1527).
Session.refresh() can now be used to synchronize a cluster node with changes from the other nodes in the cluster (JCR-1753).
Unreferenced version histories are now automatically removed once all the contained versions have been removed (JCR-134).
Standalone components like the JCR-RMI layer and the OCM framework have been moved to a separate JCR Commons subproject of Jackrabbit, and are not included in this release. Updates to those components will be distributed as separate releases.
Development preview: There are even more JSR 283 features in Jackrabbit 1.6 than were included in the 1.5 version. These new features are accessible through special “jsr283″ interfaces in the Jackrabbit API. Note that none of these features are ready for production use, and will be replaced with final JCR 2.0 versions in Jackrabbit 2.0.

This release is the result of contributions from quite a few people. Thanks to everyone involved, this is open source in action!

Shared Library with iPhoto and Lightroom

2009-08-10T16:58:43Z

I recently started using Adobe Photoshop Lightroom 2 for advanced RAW photo editing while using Apple iPhoto 09 for quick edits and as main library to keep everything organized.

By default, Lightroom can't access to original photos managed by iPhoto, because they are within a bundle. I found a nice way to workaround this limitation and allow iPhoto and Lightroom work of the same library.

Short Description: Create a symbolic link (symlink) in Pictures to the iPhoto Library "Originals" folder.

Step-by-step:

1. Install SymbolicLinker from Nick Zitzmann
2. Use the Finder and go to /Pictures/iPhoto Library
3. on the file - use context menu action "Show Package Contents"
4. Go to "Originals" folder and use the "Make Symbolic Link" action in More
5. Move the Symbolic Link into /Pictures (and rename if you like)
6. Start Lightroom and use the "Import Photos from Disk..."
7. Navigate to the Symbolic Link
8. Use "Add photos to catalog without moving" for file handling drop-down (should be default)

Credits: Moving Your Images From iPhoto to Lightroom from Gene McCullagh. He is using an "Alias" and recommends to copy the files - but that duplicates all photos.

links for 2009-07-30

2009-07-31T01:07:17Z

BootXChanger

change boot image for macs

(tags: graphics boot image mac freeware software hacks tools osx apple)

Day Customer Summit - Registration open

2009-07-30T00:00:01Z

Same procedure as every year... not exactly! This year's Customer Summit - happening in October 2009 - will be held in two location. For the folks in Europe and Asia we meet in Zurich and for our friends in US and Canada we meet in beautiful Chicago.

Further, we combined the business and technical gathering - the two-day conference agenda is packed with keynotes in the morning and split business and technical sessions in the afternoon.

We also use the opportunity of the Customer Summit to present the latest product advancements and announce new releases.

This year's product highlights:

In-depth presentation of version 5.3 of the CQ5 platform (that includes WCM, DAM and Social Collaboration)
Unveiling of CRX 2.0: the JCR 2.0 (JSR-283) content repository
Presentation of the 2010 roadmap
and one more thing

Do you like to share your CQ5 story? We have reserved speaking slots for partners, developers and customers to share their experience. Check out the Call for Speakers page to learn more.

The Customer Summit is a great opportunity to meet the community, exchange ideas and network with implementation partners and Day employees. We look forward to see you in Zurich and Chicago.

Visit the Summit Site for detailed agenda and registration.

links for 2009-07-28

2009-07-29T01:06:50Z

Developer Evangelism - home of the Developer Evangelist Handbook

(tags: webdev handbook ebook development tutorial evangelism developer communication evangelist)

[LOTD] Polyglott Persistence

2009-07-29T00:00:01Z

A couple of days ago, I wrote about the NoSQL movement and my conviction that data storage models will soon be better fitted to the data at hand (rather than have any data shoehorned into a relational model). It turns out that Scott Leberknight has come up with a nice buzz word for this phenomenon: "Polyglot Persistence". Quote from a presentation abstract of his:

Polyglot persistence is all about considering your persistence requirements and selecting a persistence mechanism that best mets those requirements, as opposed to selecting an RDBMS as the default choice.

InfoQ has more information on his talk. From the article:

The types of data managed in the applications is very different as well. It can be either Structured (relational data), Semi-Structured (for example, documents in a medical records system) or Unstructured (audio/video stream).

(SCNR: if you have all of the above you might want to look at content repositories).

Related to this: For an introduction to ACID vs BASE you might also enjoy the talk "Drop ACID and think about data" from PyCon.

[LOTD] eXo JCR implementation

2009-07-28T00:00:01Z

Congratulations to the eXo folks for their latest release of the eXo JCR implementation. Apart from the standardized JSR-170 features they provide an Extensions API that looks interesting. Extensions can be used to trigger actions when the content changes. That sounds like JCR Observations, but the extensions get called when the changes happen in the session, whereas observations only fire for persisted content. Therefore, this allows for "before update"-like functionality. Nice!

links for 2009-07-25

2009-07-26T01:11:14Z

Developer Color Picker for Mac

(tags: utility development cocoa colorpicker mac software tools tool osx color)

Version 5.2.1 available - thanks to customers and partners for phenomenal adoption and feedback on CQ5

2009-07-23T00:00:01Z

I'd like to thank the dozens of customer for joining us immediately after the release of CQ5's version 5.2 and with whom we've been able to work with. It has been and is a fantastic collaboration that spans across all continents. Readers of this blog know the brands we have communicated in the past and more are to be announced. Our customers have provided us with excellent feedback, good challenges and excellent collaboration that helped us to increase CQ5's already high level of fun-of-use, stability, scalability, and performance. Fun-of-use, not just for the authors and content editors, but also for our developers and system administrators, in line with our 4 target groups for whom we strive to deliver excellence.

Version 5.2.1 is now available for download. It is a maintenance and convenience release. The most important improvements according to your feedback have been rolled up into it. As such version 5.2.1 delivers on of the pillars of our product strategy: "tight customer and partner focus".

We continue working on the other two aspects of our product strategy: technology & product leadership and standards & open source leadership for the content industry with our planned CQ 5.3 release as well as JCR 2.0 (JSR-283).

For more on those two, please join us at the "Ingnite 2009" customer summits Europe and US. The sign-up page will be up very soon.

[LOTD] Sling for Pythonistas

2009-07-21T00:00:01Z

Yesterday, a Python framework for GAE development called Pyxer got updated to version 0.7.2. I mention this because in a well-hidden corner of Pyxer is a Python wrapper for Sling development: check out /src/pyxer/sling/sling.py. This class encapsulates the Sling HTTP API in a Python class (at least some of the API, but it gives you a start).

If you want to give it a spin be aware that the Pyxer class puts the default Sling port to 7777, instead of 8888.

[LOTD] State of the JSR-283 implementation in Jackrabbit

2009-07-20T00:00:01Z

Jukka Zitting, committer in the Apache Jackrabbit project, has published a post on the status of the JSR-283 implementation in Jackrabbit. For a quick summary of JSR-283 see spec lead David Nuescheler's recent post.

JCR 2.0 implementation progress

2009-07-18T17:38:30Z

The JCR 2.0 API specified by JSR 283 has been in Proposed Final Draft (PFD) stage since March, and Apache Jackrabbit developers have been busy implementing all the specified new features and adding compliance test cases for them.

Both the Reference Implementation (RI) and the Technology Compatibility Kit (TCK) of JSR 283 will be based on Jackrabbit code, and we expect the final version of the specification to be released shortly after Jackrabbit trunk becomes feature-complete and the API coverage of the TCK reaches 100%. The following two graphs illustrate our progress on both these fronts.

First a track of all the JCR 2.0 implementation tasks we’ve filed under the JCR-1104 collection issue. The amount of work per each sub-task is not uniform, so this graph only shows the general trend and does not suggest any specific completion date.

The second graph tracks the TCK API coverage. We started with the JCR 1.0 TCK, so the first 300-400 method signatures were already covered with few changes to existing test code. Based on Julian’s API coverage reports in JSR-2085, this graph tracks progress in covering the 100+ new method signatures introduced in JCR 2.0. Again, the graph is meant to show just a general trend and should not be used to extrapolate future progress.

Wan’t to see JCR 2.0 in action? The latest Jackrabbit 2.0 alpha releases are available for download!

links for 2009-07-17

2009-07-18T00:12:32Z

Chocomoko - lastfm player for mac

lastfm player for mac

(tags: lastfm streaming music player mac software last.fm osx apple)

IKS project, months 4 to 6: a community is born!

2009-07-17T15:53:12Z

For me the major IKS project event this quarter is without a doubt the Workshop in Salzburg at the end of May that I blogged about before.

Talk is cheap and implementation is another story – we now have to implement the ideas discussed there, in my case mostly the search engine and user stories. We’ve got started on those, but most of the work is still ahead of us.

We have also received the first benchmark requirements, where we’ll have to evaluate our product’s semantic search abilities, and create some prototype extensions that allow us to better cover this area.

With so many CMS luminaries at the Salzburg workshop, we have started to create an IKS community – at least when it comes to agreeing on what IKS should be, and that’s great already. Now’s the time to move to the next level and actually start creating useful software for and with this community.

NoSQL: A long-time relation(ship) comes to an end

2009-07-16T14:21:58Z

(cross-posting from here)

OK, I admit it, declaring that "the RDBMS is dead" is a meme that has been going around the software industry for a while. Remember object-oriented data bases that were supposed to replace the relational ones? Well, guess who is still here. However, despite the RDBMS's amazing survival skills I would like to propose a related prediction:

I believe that the year 2009 will go down in history as the year when the "relational model default" ended. The term "relational model default" was coined by me to describe a peculiar thing that goes on in application development: start talking to your average application developer about some arbitrary business requirement and chances are that simultaneously he mentally constructs a relational model to fit those requirements.
That relational approach to modeling your problem may or may not be suitable. The real problem is that all too often this default does not get challenged. As a consequence, whatever the fitting data model might be, it gets shoehorned into tables and relations.
This default "thinking" has not yet changed for the masses, but I believe that it has changed for the early adopters (which means that invariably it will change for the masses in some years).
I see the default to change from:
"I need to store some data i.e. I need a relational database"
to:
"I need to store something, let me see the data to decide how to store it."
The most concrete and visible manifestation of the rising interest in non-relational data store is the "NoSQL" movement. NoSQL denotes a group of people interested in exploring and comparing alternatives to the traditional relational data storages like MySQL or Postgres. The inaugural get-together has been covered in Computerworld, see also Johan Oskarsson's post and there is, of course, a Hashtag.
Other than the NoSQL group I have a second data point to offer: there is a Cambrian Explosion happening in terms of projects exploring non-relational data stores. During the Cambrian Explosion a major diversification of organisms took place. Similarly a plethora of new projects that explore alternatives to relational models continue to gain interest. Here is an incomplete list:

AllegroGraph, Amazon's SimpleDB, Cassandra, CouchDB, Dynomite, Google's App Engine datastore, HBase, Hypertable, Kai, MemcacheDB, Mongo DB, Neo4J, OpenRDF, Project Voldemort, Redis, Ringo, Scalaris , ThruDB, Tokyo Cabinet (and Tokyo Tyrant and LightCloud)
Last, but certainly not least, there are Apache Jackrabbit and Apache Sling.
From my perspective there are three main areas of innovation in this Cambrian Explosion of data stores:
1. Models

In the relational model you break down your data into tables and relations. This model implies that the data is somewhat tabular. However, in some cases the data simply is not tabular.
Consider web content, which is hierarchical and mixes fine-granular data with binary files (this model is implemented in Jackrabbit). Other (not mutually exclusive) alternative models are document-oriented, key-value pairs, or Graphs/RDF.
One very important aspect of many alternative models is that they are schemaless. That means that they accommodate for Data First approaches where it is not required to define the data structure before one can actually store any data. This enables agile approaches to software development in the short term as well as more flexibility in the long term evolution of business requirements.
Without defining a data structure first it is not possible to store anything at all in an RDBMS. This fact is probably one of the root causes of the relational default thinking. An RDBMS-based developer simply cannot develop anything without thinking about table structure.

2. Scalability

A second area of innovation is scalability. This can be split down into two sections: One is scalability achieved by distributing the data store across separate machines, the approach pioneered by Google. Opposed to classical clustering of RDBMSs the order of magnitude of machines that are considered is hundreds rather than ten. Obviously, different trade-offs regarding consistency and availability of individual cluster nodes must be taken when architecting for such a high number of cluster nodes. Eventual consistency is one of the interesting concepts invented in this space.

While the commoditization of server hardware triggered this first approach to scalability, a second area is related to the rise of multi-core processors. For a number of years CPUs have not gotten faster, but rather the number of cores has increased. There is no explicit contradiction in running a classical RDBMS on a multi-core machine and even having the RDBMS take advantage of them. However, it seems to me that the SQL language is a poor fit for queries in a multi-core environment when compared with alternatives such as Map/Reduce which are parallel by design.

3. Web

The third area of innovation revolves around the fact that the web is the dominant paradigm for computing in our time. This is also acknowledged by the two considerations discussed above. However, a third one is that HTTP is used for accessing the data. Other types of connectivity that were typically implemented as JDBC or ODBC drivers are not needed/used anymore. In many cases the data store exposes its resources in a RESTful API. An obvious benefit is the ubiquitous availability of clients including the browser itself. The classical RDBMS approach involving a dedicated driver looks like a client-server architecture mindset in comparison (I wrote about this 1.5 years ago).
At this point let me re-iterate that RDBMSs are here to stay, just like mainframes never went away. Moreover, a couple of the innovation areas cited above are not that new at all, especially, when it comes to non-relational data models (for example, I recently dug into the foundations of the Lotus Notes document store and came out very impressed). However, it is only now that the relational model default will disappear.

What about content management systems?

Considering the content management system industry as a whole I am extremely happy about this shift away from RDBMSs. Especially the model aspect is crucial: RDBMSs embody a fundamentally wrong model for content. There are varying opinions in the industry about what "content" really is, but one thing is more or less universally accepted: it is (at least partially) unstructured. Well, RDBMSs are designed for structured data. Duh.
So why are there one gazillion LAMP-based CMSs? I blame the relational model default. But as this default vanishes we will see more and more CMSs that are not based on an RDBMS (see the Jackrabbit wiki for a list of JCR-based ones, as well as the recent PHP-based JCR implementations Jackalope or for Typo3 or the Midgard content repository).
Don't laugh, but I truly envision a better (CMS) world once more CMSs are built upon proper tools and not forced into a relational model anymore. It will be a better world for developers and consequently for the CMS users.
What about Day?

REST and content repositories were invented and evangelized by Day's Chief Scientist Roy and Day's CTO David years ago already. So it is no surprise that Day's content management systems are in an excellent shape with respect to these considerations. CQ5 is built upon Apache Jackrabbit, i.e. a data store that implements a content-centric model, and Apache Sling, a web framework designed to be RESTful right from the start.
When it comes to scaling: a week ago we gave a live demonstration on how to install and cluster CQ5 on Amazon's EC2 service. But, expect even more exciting news in this area.

NoSQL: A long-time relation(ship) comes to an end

2009-07-16T00:00:01Z

OK, I admit it, declaring that "the RDBMS is dead" is a meme that has been going around the software industry for a while. Remember object-oriented data bases that were supposed to replace the relational ones? Well, guess who is still here. However, despite the RDBMS's amazing survival skills I would like to propose a related prediction:

I believe that the year 2009 will go down in history as the year when the "relational model default" ended. The term "relational model default" was coined by me to describe a peculiar thing that goes on in application development: start talking to your average application developer about some arbitrary business requirement and chances are that simultaneously he mentally constructs a relational model to fit those requirements.

That relational approach to modeling your problem may or may not be suitable. The real problem is that all too often this default does not get challenged. As a consequence, whatever the fitting data model might be, it gets shoehorned into tables and relations.

This default "thinking" has not yet changed for the masses, but I believe that it has changed for the early adopters (which means that invariably it will change for the masses in some years).

I see the default to change from:

"I need to store some data i.e. I need a relational database"

to:

"I need to store something, let me see the data to decide how to store it."

The most concrete and visible manifestation of the rising interest in non-relational data store is the "NoSQL" movement. NoSQL denotes a group of people interested in exploring and comparing alternatives to the traditional relational data storages like MySQL or Postgres. The inaugural get-together has been covered in Computerworld, see also Johan Oskarsson's post and there is, of course, a Hashtag.

Other than the NoSQL group I have a second data point to offer: there is a Cambrian Explosion happening in terms of projects exploring non-relational data stores. During the Cambrian Explosion a major diversification of organisms took place. Similarly a plethora of new projects that explore alternatives to relational models continue to gain interest. Here is an incomplete list:

AllegroGraph, Amazon's SimpleDB, Cassandra, CouchDB, Dynomite, Google's App Engine datastore, HBase, Hypertable, Kai, MemcacheDB, Mongo DB, Neo4J, OpenRDF, Project Voldemort, Redis, Ringo, Scalaris , ThruDB, Tokyo Cabinet (and Tokyo Tyrant and LightCloud)

Last, but certainly not least, there are Apache Jackrabbit and Apache Sling.

From my perspective there are three main areas of innovation in this Cambrian Explosion of data stores:

1. Models
In the relational model you break down your data into tables and relations. This model implies that the data is somewhat tabular. However, in some cases the data simply is not tabular.

Consider web content, which is hierarchical and mixes fine-granular data with binary files (this model is implemented in Jackrabbit). Other (not mutually exclusive) alternative models are document-oriented, key-value pairs, or Graphs/RDF.

One very important aspect of many alternative models is that they are schemaless. That means that they accommodate for Data First approaches where it is not required to define the data structure before one can actually store any data. This enables agile approaches to software development in the short term as well as more flexibility in the long term evolution of business requirements.

Without defining a data structure first it is not possible to store anything at all in an RDBMS. This fact is probably one of the root causes of the relational default thinking. An RDBMS-based developer simply cannot develop anything without thinking about table structure.

2. Scalability
A second area of innovation is scalability. This can be split down into two sections: One is scalability achieved by distributing the data store across separate machines, the approach pioneered by Google. Opposed to classical clustering of RDBMSs the order of magnitude of machines that are considered is hundreds rather than ten. Obviously, different trade-offs regarding consistency and availability of individual cluster nodes must be taken when architecting for such a high number of cluster nodes. Eventual consistency is one of the interesting concepts invented in this space.

While the commoditization of server hardware triggered this first approach to scalability, a second area is related to the rise of multi-core processors. For a number of years CPUs have not gotten faster, but rather the number of cores has increased. There is no explicit contradiction in running a classical RDBMS on a multi-core machine and even having the RDBMS take advantage of them. However, it seems to me that the SQL language is a poor fit for queries in a multi-core environment when compared with alternatives such as Map/Reduce which are parallel by design.

3. Web
The third area of innovation revolves around the fact that the web is the dominant paradigm for computing in our time. This is also acknowledged by the two considerations discussed above. However, a third one is that HTTP is used for accessing the data. Other types of connectivity that were typically implemented as JDBC or ODBC drivers are not needed/used anymore. In many cases the data store exposes its resources in a RESTful API. An obvious benefit is the ubiquitous availability of clients including the browser itself. The classical RDBMS approach involving a dedicated driver looks like a client-server architecture mindset in comparison (I wrote about this 1.5 years ago).

At this point let me re-iterate that RDBMSs are here to stay, just like mainframes never went away. Moreover, a couple of the innovation areas cited above are not that new at all, especially, when it comes to non-relational data models (for example, I recently dug into the foundations of the Lotus Notes document store and came out very impressed). However, it is only now that the relational model default will disappear.

What about content management systems?

Considering the content management system industry as a whole I am extremely happy about this shift away from RDBMSs. Especially the model aspect is crucial: RDBMSs embody a fundamentally wrong model for content. There are varying opinions in the industry about what "content" really is, but one thing is more or less universally accepted: it is (at least partially) unstructured. Well, RDBMSs are designed for structured data. Duh.

So why are there one gazillion LAMP-based CMSs? I blame the relational model default. But as this default vanishes we will see more and more CMSs that are not based on an RDBMS (see the Jackrabbit wiki for a list of JCR-based ones, as well as the recent PHP-based JCR implementations Jackalope or for Typo3 or the Midgard content repository).

Don't laugh, but I truly envision a better (CMS) world once more CMSs are built upon proper tools and not forced into a relational model anymore. It will be a better world for developers and consequently for the CMS users.

What about Day?

REST and content repositories were invented and evangelized by Day's Chief Scientist Roy and Day's CTO David years ago already. So it is no surprise that Day's content management systems are in an excellent shape with respect to these considerations. CQ5 is built upon Apache Jackrabbit, i.e. a data store that implements a content-centric model, and Apache Sling, a web framework designed to be RESTful right from the start.

When it comes to scaling: a week ago we gave a live demonstration on how to install and cluster CQ5 on Amazon's EC2 service. But, expect even more exciting news in this area.

Roy Fielding elected into ASF Board of Directors

2009-07-13T00:00:01Z

Day's Chief Scientist Roy Fielding has been elected as one of the new members of the Apache Software Foundation's a new Board of Directors. Congratulations! Roy had been member of the Board before, see Shane Cucuru's nice timeline of the Board members.

On the occasion of the ASF's 10th birthday we published an interview with Roy and Bertrand Delacretaz (who served on the Board during the last term) about the foundation's past and future.

links for 2009-07-10

2009-07-11T00:15:08Z

The Ultimate Beginner’s Guide To AppleScript « AppStorm

(tags: programming applescript mac tutorial tutorials howto scripting reference osx apple)

The Board is dead, long live the Board!

2009-07-09T16:42:52Z

Well, not exactly dead, but today the Apache Software Foundation elected a new board of directors:

2009 board: Shane Curcuru, Doug Cutting, Justin Erenkrantz, Roy T. Fielding, Jim Jagielski, Geir Magnusson, Jr., Brian McCallister, Brett Porter, Greg Stein

Congrats to all! There were a lot of excellent candidates this time, and the new board looks like a nice mix of great people. Looking forward to what this year brings.

And thanks to the outgoing board – being part of it has been a rich learning experience for me. Hard work at times, but it’s great to have more insight about how the foundation works and to be able to help.

I’ll list the names of the 2008 board here, so that I can find them in 50 years when we talk about the good times:

2008 board: Justin Erenkrantz, J Aaron Farr, Jim Jagielski (chairman), Geir Magnusson Jr, William Rowe Jr, Sam Ruby, Henning Schmiedehausen, Greg Stein, Bertrand Delacretaz

Shane Curcuru has created a nice timeline of all board members from the beginning to 2008.

links for 2009-07-08

2009-07-09T00:37:44Z

Flexibits | Cameras

manage multiple cameras

(tags: iphoto camera iphone management cameras mac photography software free)

Happy coders visualized

2009-07-08T00:00:01Z

My colleague Lars Trieloff has gone through a fun excercise: he used Nat Pryce's software Project Painter and Synesketch to visualize how happy the developers of our software are. Project Painter analyzes the emotions expressed in comments of the software and generates an image from that. The results: they are happy and in the case of Sling a bit surprised as well (see the Synesketch wiki for how to interpret the images).

Apache Felix

Apache Jackrabbit

Apache Sling

CQ5

Reviewing code quality of Apache Sling using Sonar

2009-07-01T00:00:01Z

This is a cross-post of Freddy's analysis at the Sonar site. We use Sonar internally at Day to track and improve the quality of all our software. Also check out Nemo which is Sonar's platform for analysing various other FOSS projects.

A few weeks ago Michael Marth, who runs dev.day.com (Day’s developer portal), asked us if we could put together our impressions on the code quality of Apache Sling using Sonar. We thought it would be valuable to share the result of this exercise with the community.

Apache Sling in a few words

“Apache Sling is an innovative web framework that is intended to bring back the fun to web development. It uses all those nice cool and new technologies that make up a state-of-the-art framework. This is Apache Sling in five bullets:

REST based web framework
Content-driven, using a JCR content repository
Powered by OSGi
Scripting inside, multiple languages
Apache Open Source project

Some size indications of the project

40 Maven modules
70,707 lines of code
731 Java classes
and 23,043 lines of Javadoc

The strengths in terms of quality

A project that you get and compile with no difficulty by running two commands:
1. svn checkout https://svn.apache.org/repos/asf/sling/trunk/
2. mvn clean install
This sounds like an evidence but is not always the case :-)
Amongst 130,172 physical lines, only 0.9% are involved in a duplication
46.4% of public API are commented with a Javadoc block

The weaknesses

Only 9% of the source code is covered by 338 unit tests
Average cyclomatic complexity by method (excluding getters and setters) is greater than 3 (3.2).
That is kind of a warning saying “your methods are taking too much responsibilities and should be re-factored”. This warning is confirmed by others metrics : 394 methods have a complexity greater than 7 and 86 methods have more than 50 statements. What is true at method level gets also partially confirmed at class level as 60 classes have a Fan Out Complexity greater than 20 (The number of other classes referenced by a class)

Bad programming practices that should be improved

198 times, method parameters are reassigned in the core of the method
68 times, local variables are defined and hide class fields
28 times, NullPointerException are thrown when an IllegalParameterException would be more suitable

Potential bugs that should be quickly analyzed

Correctness - An apparent infinite recursive loop : there is an apparent infinite recursive loop in org.apache.sling.scripting.jsp.jasper.runtime. JspContextWrapper.include(String, boolean)
Multithreaded correctness - Unsynchronized get method, synchronized set method : org.apache.sling.scripting.jsp.jasper.compiler. JspRuntimeContext.getJspReloadCount() is unsynchronized, org.apache.sling.scripting.jsp.jasper.compiler. JspRuntimeContext.setJspReloadCount(int) is synchronized
Multithreaded correctness - Method calls Thread.sleep() with a lock held : org.apache.sling.event.impl. JobEventHandler.runJobQueue(String, JobBlockingQueue) calls Thread.sleep() with a lock held
Malicious code vulnerability - Field is a mutable array : org.apache.sling.jcr.webdav.impl.servlets. SlingWebDavServlet.COLLECTION_TYPES_DEFAULT is a mutable array

This analysis was done with the intention of giving a synthetic overview of the current state of the project. Where should you start from if tomorrow you wake up with a single idea in mind : “Improving quality of the Apache Sling project !” ?

With respectively a cyclomatic complexity of 428, 385 and 343, classes Generator, Parser and XMLEndoginDetector should be first refactored. With no surprise, the Generator.java file has the greatest number of duplicated lines (154) and rules violations (109)
With its 43 cyclomatic complexity and no unit tests, the method ModifyAceServlet.handleOperation(..) is what we call “a crappy method” :-)

More information on the code quality of the project is available on Nemo.

[LOTD] Content Structure in a CMS

2009-06-30T00:00:01Z

Via Seth Gottlieb I have found this really good presentation on content modelling for content management systems by Deane Barker:

"Just Put That In The Zip Code Field..."

View more documents from gadgetopia.

Cleve Gibbon's recent well-written series on content modelling is closely related to Deane's presentation. Make sure to have a look if you are interested in that area.

I really like Deane's presentation, at least when I look at it from the paradigm that proper content management needs a-priori content modelling. Lately, I have come to question this idea, but that shall be the topic of a different post.

Jazoon 09 slides

2009-06-29T10:16:33Z

The slides shown in my Jazoon talk are now online:

Jazoon 2009

View more presentations from mmarth.

Jazoon 09 Slides

2009-06-29T00:00:01Z

In case you missed the Day talks at Jazoon conference please find the slides below:

Thomas Mueller: Testing Zen

View more documents from day.

Thomas Mueller: Java Persistence Frameworks

View more documents from day.

Michael Marth and Michael Dürig: Scalable Agile Web Development: REST meets JCR meets OSGI

View more presentations from mmarth.

Michael Dürig and Michael Marth: Building RESTful Web Applications with Scala for Sling

View more presentations from day.

[ANN] Upcoming Cloud-Computing Events (July 7 and 9)

2009-06-29T00:00:01Z

The introduction of the simplified clustering in CRX 1.4.1 (back in January 2009) kick-started the efforts to make CRX and CQ5 easily deployable into the Amazon cloud computing infrastructure.

A lot happend since then - most importantly - we got in touch with customers that have exactly the challenges we were looking to solve with cloud-based deployments. One of them being the ability to scale infrastructure for peak usage, without the cost of running all servers all the time. Of course, to set a good example, we run our own sites - such as www.day.com - on EC2.

Next week, we would like to give you an update of what we did so far and share the plans for the future.

On July 7th (Tuesday) we have a half day seminar in London. Sarah Burnett from Butler Group and myself will discuss the advantages and use cases to make best use of cloud computing infrastructure. This is a great event to get yourself familiar with the cloud computing topic and learn how you can apply that to your content management initiatives. Join the free seminar in London. Sign-up here.

On July 9th (Thursday) I'm going to broadcast my speech and discussions from London via a Webinar using WebEx. Feel free to sign-up to get the details to join the Webinar.

Personally, I believe the most exciting part of the cloud computing era are the new ways to solve challenges accepting the fact that almost unlimited computing resources are at your disposal (at a fairly decent price).

Scala for Sling @ Jazzon 09

2009-06-26T13:38:06Z

Yesterday I gave a presentation at Jazoon 09 about using Scala for scripting RESTful web applications with Apache Sling.

In the session I showed how to take advantage of Scala to create RESTful web applications with Apache Sling. I demonstrated how to uses its DSL capability and support for XML literals to create type safe web site templates. In contrast to conventional web site template mechanisms (e.g. JSP), this does not rely on a pre-processor but rather uses pure Scala code.

There are Session slides and support material available here. The support material contains a fully workable demo application. A Scala scripting bundle for Sling is also included.

Posted in Uncategorized Tagged: JCR, Scala, Sling

Puzzle: implement this (solution)

2009-06-24T20:51:35Z

Well, I wasn’t aware of Ticket #1737 when I was trying to find a solution to the problem from my previous post. Thanks to Jorge Ortiz for pointing this out. However, I reviewed my approach to solving this and didn’t find sever limitations. Maybe someone else does…

When I initially stumbled on this, I remembered that existential types where introduced into Scala for coping with Java’s raw types. But there is an additional twist here, we need to tell the compiler that our MyIterator implementation actually ‘is an instance of a raw type’. So combining existential types with self types led me to the following solution:

class MyIterator extends Iterator2 {
  this: java.util.Iterator[_] =>
  def hasNext = true
  def remove = throw new Error
  def next = "infinity"
}

We can now safely use instances of MyIterator.

  def test1(it: MyIterator) = {
    println(it.next)
  }

  def test2(it: java.util.Iterator[_]) = {
    println(it.next)
  }

  val it = new MyIterator
  val v: String = it.next
  println(v)

  test1(it)
  test2(it)

The approach using existential types in combination with self types makes sure that values returned from the next method always are typed correctly.

Posted in Uncategorized Tagged: Puzzle, Scala