This post is a write-down of my experiences building a XML/XSLT driven simple CMS system. In it I will show you the required technologies, my approaches and my solutions to problems I've encountered. You'll need a solid understanding of XML at least, understanding of Ant and XSLT helps a lot, too.

This post contains tons of XML and I tried my best to format it in a readable way — in fact I've spent hours to get everything nicely on the screen.

Requirements to the CMS

I had several objectives when I built my system:

• All mundane and repetitive tasks should be automated, including thumbnail generation, creation of a global navigation, synchronizing with a live site
• Flexible enough to support new "content types" and new views
• Versionable with Git, Subversion or other SCMs
• Creation of output artifacts should be a single command

Technologies

The most prominent and important technology I used is XSLT 2.0. If you don't know what XSLT is, here is XSLT in a nutshell.

XSLT stands for eXtensible Stylesheet Language Transformations. What it does is surprisingly simple. It takes a piece of XML as input (aka source tree or input tree), applies some transformations on it and outputs a new piece of XML. Sounds simple in theory, however in practice it is usually somewhat more complicated because XSLT is a functional language and sometimes requires you to solve problems backwards and don't even get me started on XML namespaces...

Traditionally you'd have one input XML file, one XSL stylesheet and accordingly one single XML output file. However with the new xsl:result-document I can suddenly output into several files.

As I'm using XSLT the next technology used is XML of course. Not much to say about this except that it is involved in pretty much every step. To bind everything together I'm using Apache Ant, an incredibly flexible tool. Interestingly enough Ant uses XML files for its build files which comes in handy later on. Also it solves one of my requirements: it is just plain text and therefor versionable.

Last there is image manipulation. I've experimented a bit with the Ant image tasks that use JAI to perform manipulation but quickly dropped that approach as this task won't even let you specify a quality parameter for image manipulations. I've ended up using an old friend: ImageMagick. ImageMagick is an amazing toolkit of versatile command line tools to modify images. It allows to modify images in many different ways from resizing to color correction or combining them.

Basic Structure

The basic structure of an XML/XSLT driven CMS contains one or more XML files that contain the "raw" data for the website, XSLT to transform it into XHTML and assets. More on assets later.

The XML Part

The XML file used in my project is a very simple one. It defines "pages" that get translated into real HTML pages. Each page has a section called contents which in turn contain the actual page contents. Let's have a look at a short sample:

Well, so far nothing impressive. Actually it looks like more work compared to typing out the actual pages themselves. But behold, here comes the stylesheet to actually turn the above XML into nice websites (XHTML):

Now it get's more interesting. Let's dissect what the above XSLT does. The blocks in line 10 to 12 are a template that matches the root level element in the source XML tree. All it does is to call the next matching stylesheets which are in lines 14 to 19. This template does exactly the same and passes on. Then we reach line 18 with the template matching page elements.

Line 18 defines a variable called filename which is prefilled with with the filename attribute from the XML page element. The next line sports the so much praised (at least by me) xsl:result-document element. If you read the line it gets clear very fast that it should output a new HTML file (format="html" href="$filename"), nicely formatted (indent="yes") and with the appropriate DOCTYPE declaration. The  next lines are just an HTML boilerplate with head and content type. Please note that this is a very minimalistic example. Usually you would include stylesheets, JavaScript includes and other additional headers there. For simplicity they are omitted.

Lines 28 to 31 output a new title tag populated with the title attribute from our source attribute. Convenient. The CMS is starting to take shape.

And finally lines 33 to 35 will call another template lines 41 to 46 which just copies the HTML into the output document.

So far so good. What have we achieved? The stylesheet creates a new page based on boilerplate HTML - which can be arbitrarily complex with CSS stylesheets, JavaScript and more - and pre-fills the title and the actual body. Now let's add more functionality. Every good website needs a navigation. We can achieve that easily by adding another template that matches all page elements. We add a few new lines to the boilerplate HTML:

Looks simple enough. This will output a new DIV, fill it with a unordered list and calls another template.  The interesting part is the select attribute: it tells the XSLT processor to select all root level page elements. Now let's have a look at how the called template looks like:

The XSLT processor will translate this into global navigation of our site. All top level pages will be listed with the correct title and the correct filename.

The Ant Part

Now that we have the stylesheets and the input XML files in place, we'll need an elegant way to transform the whole site with on simple command. As discussed earlier Ant is a good choice for this. I won't delve into the actual ANT buildfile as it is pretty straightforward. What I want to show is how to use XSLT to generate ANT buildfiles from other XML. This is actually a pretty powerful technique as I can custom tailor ANT files to whatever need I have. In this particular example I want to show how to create image thumbnails.

Here is what we have. In our input XML file, we reference images. For those images, let's say, product shots, we have the originals lying in some location in the filesystem. Those images need to be scaled down to a high res view (800×600) and to a thumbnail (128×128). We could do that with an ANT pathmatcher, but that would just take all the images it can find, not just the referenced ones. Bad and slow. And how do we keep the build directory free of old and unused pictures then?

What we'll do instead is we'll take the input XML file, create a XSLT stylesheet that will grab all the image references and put them into a custom ANT build file which in turn calls an image manipulation program (ImageMagick) to scale down the pictures. There's a few other goodies in here too, e.g. automatic file name cleanup. Here is the images.xsl:

Google Sitemaps

Most people that have used Google's Webmaster Tools know about the sitemap. No need to write them manually any more, you can easily use the XML representation and a XSLT stylesheet to generate it for you:

Issues

So with all the technologies in place, I ran into some issues but all of them could be solved quickly. I'm listing them here hoping they'll help other people.

XSLT 2.0 with Ant

This one was most annoying. Per default Ant has an xslt-task which allows you to transform XML but it is limited to XSLT 1.0. If you read the documentation you'll find out that you can plug in different XSLT processors but you are still quite limited. If you want to use XSLT 2.0 with Ant get the amazing Saxon processor and use their Ant task which is a bit hidden on their page. Get it from the Sourceforge download site. This task lets you leverage the full functionality of XSLT 2.0 including the xsl:result-document.

All you have to do to get it working is to copy the required JARs into the classpath, declare the custom task and use it:

Now you can use the saxon-xslt task:

XML Namespaces in HTML

This was a tricky one. The problem was that in my custom XML I had snippets of HTML which I wanted to mirror into the output documents. I used the xsl:copy-of function which basically did what it was supposed to. It copied the (X)HTML over but by doing so, it had to adjust the namespaces for the XML. The input document had no namespace declarations at all and the output document was bound to the XHTML namespace. So the processor did exactly what it is supposed to: it added null namespace  declarations to the copied elements which looked like this:

Not a problem per se but unfortunately invalid XHTML. The solution to this problem is surprisingly simple: Add namespace declarations to the input document:

The above lines will put the input document per default into the XHTML namespace and everything else in my own, personal namespace. Regular (X)HTML markup however does not get any namespace prefixes, so it will be put into the XHTML namespace:

Conclusion

The one big problem I still have with this system is that it does not support hierarchical pages yet. But if the need arises it could be added without too much effort. Another moose with the system is that it still requires editing XML which is neither user-friendly nor easy for non technical users. But one could easily create a Flex or AIR app to modify and edit those files. Thanks to Flex' really good XML support it should not be hard to do that. I could even think of a Grails Webapp generating the XML for this (using the brilliant XML markup builder).

In general it is a really powerful and flexible system. As I am running this on a UNIX system I can harness all the tools and goodies that are there. Ant and Java give me whatever power and libraries I need. So I can steer this thing in whichever direction I want.

If you have any questions feel free to comment below or drop me an email.

So for all that don't know the game here's Settlers II in a nutshell: every player starts out with a headquarter, a limited amount of building materials, goods, tools, settlers with different skills and a small strip of land. The surroundings offer everything you need to build up your small empire: woods that can be cut down by lumberjacks, granite that can be cut by stone cutters, mountains that offer coal, ore and gold and flat land that can be used to grow crops. On of the major challenges of this game is to get all the production chains right: let's take for example a lumberjack. A lumberjack cuts down trees. The logs then needs to be transported to a sawmill where they get cut into handy boards. (Picture 1 shows such a production chain.) These boards in turn then need to get to a stock or to construction sites. And if you don't build a forester your lumberjack will run out of trees very soon. That was one of the easier production chains. Let's look at something more sophisticated: production of meat. You'll start of with a farm that grows crops. Then you need a pigfarm that takes the crops grown by the farm and water from a well to breed pigs. And finally, to get to your tasty meat, you'll need a slaughterhouse. So, all in all, getting those production chains right tricky and very rewarding. Nothing better than seeing wares running smoothly form source to destination (at least for me ).

Picture 1: A production chain for lumber. On the bottom left there is the forrester. The guy living in that building just keeps planting trees. The building at the top left and to the right are lumberjacks.They cut down trees. Those trees then get transported to the building to the top right, the sawmill where they get cut up.

The other big challenge, and actually the reason why I'm blogging about this, is the way system. All the production facilities mentioned above are connected by a road system. Each of those buildings has a flag in front it and whenever a building "finishes" producing an item, the item gets dropped at the flag (I won't go into the hygienically implications of dropping food on flag on a dusty road... ). Connected to the flag is a road that leads to another flag and standing on that road is a carrier, a poor guy who's job it is to just carry stuff from the one flag to the other flag. See Picture 2 to get a better understanding. So eventually this road leads to another building that actually needs what has been produced. In the picture below that would be the small pig on the flag to the right that needs to get to the slaughterhouse to the left (Hey, it's biological after all!).

Picture 2: Flags, roads and items. Notice the pig on the flag on the right.

So getting the way system right is crucial to your success in that game. If your roads are too long, it will take forever to complete things. If you don't set enough flags, the actual capacity of a given road segment will drop - after all there is only one guy standing between two flags and he can carry only one item at once (an exception are the donkeys that you get after a road has been in heavy use but that still only gives you two items per segment).

Now, you have an understanding of what that game is about. If you don't, get yourself a copy and play it a bit. But be careful, it's addictive.

So while I was playing a bit I started to wonder, how all this worked. How do wares get routed from one flag to another flag. How to flags eventually reach their destination? How is ensured that wares don't run in circles? How is ensured that best possible route is taken? All those thoughts kind of reminded me of my excellent algorithms and data structures lessons at the Hochschule München, where graphs and different algorithms on graphs where a big subject. So I got really curious how to implement something like that. After all, at university we implemented a lot of the data structures and algorithms but never for a "real" problem.

Later that night I started up Eclipse and got churned out some code to actually represent a hexgonal map (Settlers 2 uses a hexagonal grid as opposed to most modern games that just use isometric rectangular maps), render out that map to the screen and some classes to represent flags and roads. And then the fun started...

My first goal was to implement a pathfinding algorithm that actually finds a good route from one given hexagon to another hexagon. Sounds easy but is a really hard problem. There are a couple of algorithms out there that actually solve that problem but I decided to go with the simplest one, Dijsktras algorithm that is guaranteed to produce the best solution. However implementing that algorithm wasn't as easy as I anticipated. My first test run produced some rather... unexpected results:

Picture 3: Pathfinding done wrong. The goal was to find the shortest possible route from the one green flag to the other green flag... The gray zig-zag line is the path my implementation found...

Obviously that was not the best possible solution! Eventually it took me a couple of hours to get my implementation right. But then I got it right:

Picture 4: Pathfinding done right! Made the paths red in that screenshot for better visibility. Note how the red paths connecting the green flags are the shortest possible.

My conclusion after a couple of hours trying to get that right: It was a lot of fun and kudos to the developers of Settlers 2! I did not spend too much time on that but it became quickly obvious that building a game as "simple" as this old DOS game is all but easy. Building even simple pathfinding imposed quite a challenge on me. Building a whole game like that seems one hell of a task...

The next step is now to build a pathfinder that is actually aware of terrain (you can't build roads on water!) and won't allow to cross roads. With that in place i can finally start to play around with the actual routing of goods in this graph. The technical more adept might argue that I actually don't need all the stuff that I wrote so far to calculate some spanning trees on a graph. And they're right. But otherwise it wouldn't be as much fun!

Depending on how much spare time I have I'll keep playing around with this and keep you posted...

At this point I want to thank all the great people that worked so hard with me to get this project done.

Thank you.

If I find some time I will blog about my experiences integrating all those technologies. Might be some time though.

Update 19/02/2009: We made it to the Marketing Mag! Yeay!

Songbird features a real plugin system, it's skinnable and has lot's of neat features! Go for it!

To get going with the application we are planning to perform tests with some real data. The problem at this point is that entering real data into a database that might be wiped due to changes in the domain model is a very tedious task, at least for the person responsible for entering the data. And the reason for running such a test is to find problems with the domain model which lead to the dreaded changes in the domain model. This cat bites her own tail...

As wiping out the database and re-entering the data is not an option at this point, I decided to look into LiquiBase which as been sitting on my delicious list for quite a while.

This post is an attempt to write down my experiences and learned lessons with LiquiBase. The documentation of LiquiBase is technically complete and extensive but lacks more information about how to use it in a real project. This is an attempt to create a guide on how to leverage LiquiBase in a real world project.

What is this LiquiBase anyways?

For starters LiquiBase is a tool that helps developers to track and apply changes to database schemas. Basically what it does is storing all changes that are made to a database in a special XML file, called a database changelog file. This changelog file can be used to determine if there are any changes between the schema in the changelog file and the schema found in a database. The nice thing about LiquiBase is that it is database neutral so you can use with any database Java is able to connect to, and that are a lot...

But there is more to LiquiBase that just tracking changes. The outstanding feature of LiquiBase is that it is able to migrate data using the information stored in the database changelog file. This doesn't sound like very much when you are starting a project from scratch, but as soon as you start entering data in your evolving application, you'll understand. And as every application evolves, this is a feature handy for every developer, especially for Grails developers as Hibernate as ORM basically supports only two modes of operation to handle the database schema (configured via the hibernate.hbm2ddl.auto option): wiping out the database and applying a new schema (called create or create-drop) or attempting to update the schema (simply called update). The update option looks like a good way to go at first but as it turns out it has its drawbacks, especially if there is data in the database.

How does it work?

The hub of all activity in LiquiBase is the changelog file. Basically there are two possible ways to create and maintain the changelogs, the first is to use the generated changelogs as far as possible and the other is to write changelogs by hand. I'm sticking mostly to the first one as the generated changelogs are sufficient most of the time and writing changelogs manually is way to tedious though easy as there is a complete XSD.

Each entry in the changelog file is called a changeset and has a unique id. The id is used to refer to specific datasets. But how does LiquiBase know which changesets have been applied to the database? The answer is simple, it creates a table called DATABASECHANGELOG into which it will insert a row for each applied changeset. So make sure you start using LiquiBase early because you really want to have these tables in your database when changes have been incorporated into a running system.

Applying changesets against the database is called migration but more on this later.

Setting the Scene

The work flow to push in changes may depend on your project. The work flow I'm describing can be applied to team of couple of Grails developers and has been tested on a real world project.

As an example for the rest of the post let's assume a project that is developed by a couple of code monkeys. The change the domain classes and to monkey stuff. On the other hand there is a data entry system deployed so real data can be inserted into the database and the software can be demoed. I will refer to this system as the demo system. The demo system and the data entered into it has several purposes. First to see if the domain model is adequate for real data and second to provide the developers with real testing data.

Problems arise quickly in this scenario, for example changes are made to the domain model and the database schema of the demo system must be updated, preferably without wiping out all the data that has been entered. At this point LiquiBase comes in.

Grails and LiquiBase - Getting Started

Ok, now how to use LiquiBase in a Grails Project? Easy: there is a LiquiBase grails plugin for grails which can easily be installed using the following command:

grails install-plugin liquibase

This will install a copy of LiquiBase in your application and set up a couple of new commands for the grails command line interface. Before you can do anything useful with LiquiBase you have to create a database changelog file for your current database, preferably using the generate-changelog command:

grails generate-changelog grails-app/migrations/changelog.xml

Note: if you omit the filename the changelog will be dumped to the console. You can dump the changelog into any file but grails-app/migrations/changelog.xml is the default and LiquiBase will keep looking for this file. With the changelog file in place you should add it to your version control system.

Tracking Changes

So the next step is to make some changes to the domain model which result in a change of the database schema. You now have two possibilities for updating your changelog file. Either manually or using some of the neat tools LiquiBase offers. I won't go into the details for creating the files manually, the LiquiBase manual does a good job at this. So let's talk about the tools.

LiquiBase ships with a very powerful command: diff. What this does is it looks at the current database and at the changelog (I have to correct myself here, see next section! Sorry about this!) another database and outputs the difference between them. As LiquiBase changelog XML! So this is awesome because you can make changes to your Grails domain model, run it against a database and pull out the changes. All that is then to do is to add the newly generated changesets to your database changelog file.

The db-diff Tool

The db-diff tool in the Grails plugin is kind of mysterious. The LiquiBase documentation for the Grails plugin does not even mention it and I got it wrong in the first run. I had to reading the section about the database diff provided by LiquiBase and dig through the Groovy scripts of the LiquiBase Grails plugin to understand what it does.

The diff utility of LiquiBase compares two databases with each other, which kind of explains its name (doh). When using the LiquiBase command line interface you have to provide two different connection strings for two databases. When using the Grails plugin however, you cannot specify two databases (not yet?); so what happens is that: the database for the current environment ist compared to the database of the test environment. This behaviour is hardcoded into plugins/liquibase-1.8.0.0/scripts/DbDiff.groovy and limits the usefulness of the tool. So what I did is every time a database has been migrated the new schema gets propagated to the testing database. If the domain model changes the schema of the dev database, the changes can easily be determined by diff'ing it against the testing database.

This is not exactly a perfect solution as you manually have to keep the testing database in sync with the changelog. So I am currently playing around with a modification of the Grails LiquiBase plugin which adds a compareToChangelog command which populates a separate database from the current changelog and then compares this to the database of the currently used environment. Using this tool makes updating the changelog really easy as you get the changesets just for those changes you made to the domain model. If there is interest I will publish this to the LiquiBase plugin.

Migrating a Database

Finally your changelog has been updated and you now need to update your demo database. The first thing you should do: Make a backup of your database! Although LiquiBase has matured to some extent, migrating a database full of data is a risky endeavor. So make sure you are on the safe side before touching a database with real data.

The easiest way to migrate the database is to set up a copy of your project and modify the datasource to point at the demo database. The first thing you might want to do is to check which changesets have to be executed by running the following command:

grails status

This will generate a list of changesets which have to be applied, assuming there is a DATABASECHANGELOG table in the database. If you are happy with what you see, you can either migrate the database directly or do a dry run. I prefer to do a dry run before touching the database by running:

grails migrate-sql

This will output a sequence of SQL statements that LiquiBase will issue to migrate the database. This is quite handy as you can see what will happen. Then, finally the database is migrated by issuing the following command:

grails migrate

When everything goes fine your database has just been updated. If something goes wrong, you can either use the rollback commands provided or fall back to the backup of your database.

Gotchas

The following points present pitfalls you want to avoid (its enough that I hit them . Please note that this is a living section, I keep adding new points I encounter.

Use LiquiBase early!

Make sure you start using LiquiBase early. Insert the DATABASECHANGELOG table as soon as possible and ensure that changesets are run against the databases.

Backups, Backups, Backups...

Backup your database before migrating! Really, do it!

Stick to one Database Flavor

Do not switch database flavors, e.g. from HSQLDB to MySQL. I tried this and I've encountered  a couple of issues with this. For example a changelog created from a HSQLDB may not run on a MySQL and vice versa due to different named datatypes (e.g. MySQL MEDIUMBLOB vs. HSQLDB VARBINARY). Further the changelog generated from a MySQL can cause problems on other databases as it names all primary keys "PRIMARY" per default. This leads to erroneous SQL that cannot be run.

Those problemes are nothing that couldn't be solved with a little bit of search and replace and even might be necessary to create the schema for a production database; but it is definitively nothing you want to do very often.

How to Start Using LiquiBase in the Middle of a Project

When you start to use LiquiBase in the middle of a project where there already is a database filled with precious production data, things can easily become complicated. The key to success here is to freeze the project or at least the part which affects the database schema for some time, generate a database changelog file from the current state and propagate this to all database instances. The reason why this is so important is that you need the DATABASECHANGELOG table on those databases; without this table and its entries  LiquiBase does not know which changes to apply. You could use the db-diff tool here, but the far more elegant approach is to simply propagate those changes to all databases once.

As soon as you have the DATABASECHANGELOG on all database instances, updating the schema becomes so easy that you will almost get addicted to it!

So, step by step:

• freeze your project for half an hour, maybe you can do this in the evening
• create a new changelog file from the current schema with the grails generate-changelog command
• mark all those changesets as imported with the grails changelog-sync command
• push those changes to ALL databases associated with the project. If you fail to do so, migrating the schema of a database without those tables is almost impossible. Depending on the size of your database this requires a massive amount of time.
• With the DATABASECHANGELOG table in place, migrating LiquiBase to the next version is as simple as entering grails migrate.

Conclusion

LiquiBase is a great addition to Grails and you should really really think about using it. Otherwise managing changes to the database can easily become a big problem and im talking out of experience here. Hopefuly this post helps you to get started with LiquiBase. As always if you have any comments please feel to post.

LEGO

Gestern Abend habe ich eine großartige Software gefunden, den Lego Digital Creator (LDD), den man kostenlos herunterladen kann. Der LDD ist eine virtuelle Lego-Box mit sehr sehr vielen unterschiedlichen Teilen (obwohl ich doch schon einige nicht finden konnte, z.B. Differentialgetriebe oder diese Schutzblechdingens die alles in allem 5 oder 6 Noppen lang waren und die Radladerschaufeln sowieso... ) und einem unerschöpflichen Vorrat an Steinen in alle Farben -- ich hab schon versucht sie alle mal aufzubrauchen, aber es sind einfach nicht weniger geworden . Der LDD kommt mit einer schicken 3D-Oberfläche daher, die Steuerung ist intuitiv und bietet neben dem eigentlichen Bauspaß mit der Explosionsanimation, bei der das eben Erbaute in alle Einzelteile explodiert und sich dann wieder zusammenfügt, auch Befriedigung für die eher destruktiv veranlagten. Und ja, es gibt eine Mac-Version! Danke Lego!

Wenn man mit dem was man erschaffen hat zufrieden ist, kann man sogar sein Modell bei Lego bestellen; bei fast allen Modellen die ich bis jetzt gebaut habe, wurde ich dann aber darauf hingewiesen das bestimmte Steine nicht bestellt werden können, und somit mein Modell nicht bestellbar ist. Aber das ist ja nicht so tragisch -- das zusammenbauen alleine macht schon so viel Spaß!

Darüber hinaus gibts hier noch das heutige Bild!

Den Reactable habe ich das erste mal in einem kurzen Bericht im Fernsehen gesehen und war sofort fasziniert; ein Tisch mit einer reaktiven Oberfläche der bei Auflegen von "Bausteinen" Geräusche erzeugt. Die Videopräsentationen muss man sich einfach ansehen: Basic Demo #1, Live in Berlin. Das ist so cool!

Interessanterweise stellen die Macher des Reactable ihr Framework für die "Benutzeroberfläche" zur Verfügung. Damit ließen sich wirklich tolle Sachen machen, man stelle sich einfach die Möglichkeiten eines absolut intuitiven und Hilfsmittelfreien Systems vor! Wo bekomme ich jetzt so einen Tisch her?