So, what's the idea behind this script? Everybody that uses Liquibase (if you are not, check out my blog post about what Liquibase is and how to use it) knows how painful it is to update your changelog.xml. Liquibase already gives you a great tool to start with, the db-diff command. However, it is hardcoded to diff the current environment's database against the test database. This is annoying as you'll have to modify your datasources and keep the database schema updated manually. But fear not, here is my (surprisingly) easy solution.

I've created a simple script (or rather, I took the db-diff script and hacked it), which I've called incremental DB diff (all the other cool names are already taken...) and what it does is the following:

1. update the schema in the db-diff (or whatever you call it) database using the changelog.xml
2. diff the database of the current environment against this database and output the diff as Liquibase XML

So, how is this a good thing? Easy: this little script will automatically create a target database to diff against using the changelog.xml which is good, as the changelog.xml represents your last migration status. And as it uses a separate database it doesn't influence the test database.

After running the script just put the output in the changelog.xml and you're done.

All you have to do to use this is to drop the file into your /scripts/ directory, create a new database and an according datasource entry for the environment "dbdiff" like this:

	dbdiff {
		dataSource {
			driverClassName = "com.mysql.jdbc.Driver"
			dbCreate = "create-drop"
			url = "jdbc:mysql://localhost/foo_dbdiff"
			username = "foo"
			password = "bar"
		}
	}

Download DbDiffIncremental.groovy. Drop me a mail or a comment if you have questions.

It took me some time to figure it out, hopefully I can save you from going through all this.

The Flex web tier compiler usually gives me this error:

Error: Could not resolve <namespace:Component> to a component implementation.

 <namespace:Component/>

error during compilation Could not resolve <namespace:Component> to a component implementation.

Initial research confirmed what I thought, this error is caused by a mismatch between a XML namespace declaration and the actual location of a component. However the location and the namespace declaration looked right and the issue occurred only after I had moved the questionable component into a separate module.

After two painful hours I realized that the Flex webtier compiler, as configured with GraniteDS, resolves namespaces relative to modules, not relative to the source folder. D'oh!

The solution I came up with was simple, might be naïve though. I figured that if resolution is relative to the Module and not to the source folder, there might be no such source folder as I would expect it. A quick look into the flex-config.xml found in <GRAILS_PROJ>/web-app/WEB-INF/flex/flex-config.xml confirmed this:

<!-- List of path elements that form the roots of ActionScript class hierarchies. -->
 <!-- not set -->
 <!--
 <source-path>
 <path-element>string</path-element>
 </source-path>
 -->

So there is no explicit source path set. So I changed it so it would point to <GRAILS_PROJ>/grails-app/views/flex:

<!-- List of path elements that form the roots of ActionScript class hierarchies. -->
 <!-- not set -->
 <source-path>
 <path-element>../../../grails-app/views/flex</path-element>
 </source-path>

Voilá - it works. However, as mentioned earlier, this might be a very limited solution, but for now it works.

At this point I want to thank all the great people that worked so hard with me to get this project done.

Thank you.

If I find some time I will blog about my experiences integrating all those technologies. Might be some time though.

Update 19/02/2009: We made it to the Marketing Mag! Yeay!

Scaffolding in a Nutshell

I assume that you know what Grails and its scaffolding mechanism is about, but I'll just summarize it here.

In a nutshell scaffolding is Grails ability to create controllers and views for domain classes. It analyzes the structure of the domain objects and creates all the files and logic required for simple CRUD functionality. If you have some domain classes all you have to do is type the following from within your project directory:

$ grails generate-all

This will create controllers and views for all domain classes found. Note that there are generate-views and generate-controller commands as well.

However you'll soon find out that the default templates which are used to create all the files may not fit your needs. Grails offers a separate command to copy the default templates into your project directory:

$ grails install-templates

This will create a new folder src/templates in your project, holding an copy of the default templates, all yours to modify and adapt.

Grails Scaffolding Templates

Before we dive into the details of scaffolding, you need to understand the different templates and what they are for. Running the install-templates command will give you the following files withing your project directory:

src/templates
|-- artifacts
|   |-- Controller.groovy
|   |-- DomainClass.groovy
|   |-- Script.groovy
|   |-- Service.groovy
|   |-- TagLib.groovy
|   |-- Tests.groovy
|   `-- WebTest.groovy
|-- scaffolding
|   |-- Controller.groovy
|   |-- create.gsp
|   |-- edit.gsp
|   |-- list.gsp
|   |-- renderEditor.template
|   `-- show.gsp
`-- war
    `-- web.xml

The files relevant for the scaffolding process are in the scaffolding directory and have been highlighted with bold font. So, what are those files for? Let's examine them one by one.

The first one is Controller.groovy. This file is used to create the controllers for your domain classes. The syntax in this file may appear a bit weird in the beginning but after all it's just a groovy template (I really encourage you to read this!). With this in mind reading the templates gets very easy. In the Controller.groovy you basically will find all the actions you would expect from your scaffolded controllers and some groovy template markup.

Next are the GSP files, create.gsp, edit.gsp, list.gsp and show.gsp. Those are obviously used to render the GSP markup required for your CRUD views. At a first glance lots of magic happen in those files. We'll cover the details in just a minute.

Finally there is the renderEditor.template. This file is used when rendering the HTML forms. Basically what happens in here is a translation of a type like Date or byte[] to their according form elements like a date picker or a file upload button.

The Scaffolding Process

Alright, so what happens when you issue generate-all for a domain class? There are a couple of players involved in this game. As most obvious besides the templates we just saw there are the actual scripts,  generate-all, generate-views or generate-controller which reside under $GRAILS_HOME/scripts. These scripts instantiate a DefaultGrailsTemplateGenerator which basically is a wrapper around Groovy's Template and TemplateEngine and binds together all the pieces we've encountered so far. You won't find any Javadocs about this class, so you'll have to browse the Grails sourcecode if you're interested. The DefaultGrailsTemplateGenerator then calls the actual templates.

One key aspect for creating really powerful scaffolding code is to understand what is handed into the templates by the DefaultGrailsTemplateGenerator. It took me quite some time and some source code browsing to figure this out. First there is a variable called propertyName. There isn't anything magical about it, just the name for a variable. However you will encounter this variable all over the templates. For example if scaffolding is run for a class called "User" propertyName would be "user" (note the lower case "u") and all variables of type "User" would be called "user" in the scaffolded output.

Then there is an instance of DefaultGrailsDomainClass which is handed into the template as a variable called domainClass. This class variable plays a major role in scaffolding as it allows you to access all kinds of information about the domain class the scaffolding process is handling. You can get information about the GORM mappings, the associations the class has and even the other side of the associations, which is crucial when building more complex forms and more. And yes, i really encourage you to read the Javadocs for this class.

Depending on whether views or controllers are generated either the create.gsp, list.gsp, edit.gsp and show.gsp or Controller.groovy are used as template.

Generating a Controller

When generating controllers the Controller.groovy file is used as template. What you will end up with is a new file named after your Class containing the usual actions required for CRUD. There is not much magic happening in here; the template is pretty straight forward and just creates calls for the different GORM methods. Read it to get a feeling for this.

Generating Views

However when generating views, things get more interesting. In the views and the renderEditor.template the domainClass property is actually used to determine information about collections.

So let's dive into the details of how the views are generated. If you look at the templates you'll see that all of them create a list of properties available for a domain class by querying the domainClass variable:

props = domainClass.properties.findAll { !excludedProps.contains(it.name) }

All that remains to do now is to iterate over those properties, determine whether they should be visible and call the renderEditor to actually render the required code. The following block of code has been copied from the edit template and has been commented by me to clarify:

props.each { p ->
    cp = domainClass.constrainedProperties[p.name]
    display = (cp ? cp.display : true) // Determine visibility
    if(display) { %>
    <tr class="prop">
        <td valign="top" class="name">
            <label for="${p.name}">${p.naturalName}:</label>
        </td>
        <td valign="top" class="value \${hasErrors(bean:${domainClass.propertyName},field:'${p.name}','errors')}">
            ${renderEditor(p)}  // Call render editor to render correct form element
        </td>
    </tr>

This is basically what happens in all the templates. A list of properties is generated and iterated over. One piece is still missing: the renderEditor.template.

As stated earlier, the renderEditor.template translates the type of a property into the appropriate form control. The way this happens is disappointingly simple; open up the file in your favorite editor and you will find a big block of if-statements like this:

<%  if(property.type == Boolean.class || property.type == boolean.class)
        out << renderBooleanEditor(domainClass,property)
    else if(Number.class.isAssignableFrom(property.type) || (property.type.isPrimitive() && property.type != boolean.class))
        out << renderNumberEditor(domainClass,property)
    else if(property.type == String.class)
        out << renderStringEditor(domainClass,property)
    else if(property.type == Date.class || property.type == java.sql.Date.class || property.type == java.sql.Time.class)
        out << renderDateEditor(domainClass,property)
...

and further below the methods called by the if-statements:

...
    private renderByteArrayEditor(domainClass,property) {
        return "<input type=\"file\" id=\"${property.name}\" name=\"${property.name}\" />"
    }

    private renderManyToOne(domainClass,property) {
        if(property.association) {
            return "<g:select optionKey=\"id\" from=\"\${${property.type.name}.list()}\" name=\"${property.name}.id\" value=\"\${${domainClass.propertyName}?.${property.name}?.id}\" ${renderNoSelection(property)}></g:select>"
        }
    }
...

So in the rendering process for each property the type will be matched in the if statements and the appropriate function is called to render out some HTML/GSP code.

Conclusion

Creating scaffolding templates is not hard. It requires some knowledge about the involved technologies and concepts but I hope this post gives you a fair understanding of what is going on. If you still have any questions, feel free to comment or drop me an email.

As always, comments are welcome.

Ugly but simple example: I have two domain classes, say Book and Author and a Book has an Author. How Author looks like is not important. Book looks like this

class Book {
  Author author
}

Due to changes in the requirements the author needs to be optional thus nullable. With updated constraints the Book class looks like this:

class Book {
  Author author
  static constraints = { author(nullable:true) }
}

So upgrading the database with LiquiBase should be as easy as adding a couple of lines to the changelog file:

<changeSet author="jakob" id="foo-1">
  <dropForeignKeyConstraint baseTableName="BOOK" constraintName="FK2E3AE9CD85EDFA"/>
  <dropNotNullConstraint tableName="BOOK" columnName="AUTHOR_ID"/>
  <addForeignKeyConstraint baseColumnNames="AUTHOR_ID"
    baseTableName="BOOK" constraintName="FK2E3AE9CD85EDFA"
    deferrable="false" initiallyDeferred="false"
    referencedColumnNames="ID" referencedTableName="AUTHOR"/>
</changeSet>

Basically this changeset drops the foreign key constraint on AUTHOR_ID, drops the not-null-constraints and re-adds the foreign key constraint which is required to handle the association between the Book and the Author class.

Unfortunately the addForeignKeyConstraint will cause an error, something like this:

Caused by: java.sql.SQLException: Column types do not match in statement [ALTER TABLE...

The reason why this happens is that: when dropping the not-null constraint the type of the AUTHOR_ID column is re-set to something unusable. Inspecting the database with the HSQLDB Databasemanager showed that the type of the is set to NULL which is weird. The solution to this is easy however. All you need to do is to provide LiquiBase the type with the columnDataType attribute of the column in the dropNotNullConstraint element:

<changeSet author="jakob" id="foo-1">
  <dropForeignKeyConstraint baseTableName="BOOK" constraintName="FK2E3AE9CD85EDFA"/>
  <dropNotNullConstraint tableName="BOOK" columnName="AUTHOR_ID"/>
  <addForeignKeyConstraint baseColumnNames="AUTHOR_ID" columnDataType="BIGINT" 
    baseTableName="BOOK" constraintName="FK2E3AE9CD85EDFA"
    deferrable="false" initiallyDeferred="false"
    referencedColumnNames="ID" referencedTableName="AUTHOR"/>
</changeSet>

With this in place it will work like a charm.

To get going with the application we are planning to perform tests with some real data. The problem at this point is that entering real data into a database that might be wiped due to changes in the domain model is a very tedious task, at least for the person responsible for entering the data. And the reason for running such a test is to find problems with the domain model which lead to the dreaded changes in the domain model. This cat bites her own tail...

As wiping out the database and re-entering the data is not an option at this point, I decided to look into LiquiBase which as been sitting on my delicious list for quite a while.

This post is an attempt to write down my experiences and learned lessons with LiquiBase. The documentation of LiquiBase is technically complete and extensive but lacks more information about how to use it in a real project. This is an attempt to create a guide on how to leverage LiquiBase in a real world project.

What is this LiquiBase anyways?

For starters LiquiBase is a tool that helps developers to track and apply changes to database schemas. Basically what it does is storing all changes that are made to a database in a special XML file, called a database changelog file. This changelog file can be used to determine if there are any changes between the schema in the changelog file and the schema found in a database. The nice thing about LiquiBase is that it is database neutral so you can use with any database Java is able to connect to, and that are a lot...

But there is more to LiquiBase that just tracking changes. The outstanding feature of LiquiBase is that it is able to migrate data using the information stored in the database changelog file. This doesn't sound like very much when you are starting a project from scratch, but as soon as you start entering data in your evolving application, you'll understand. And as every application evolves, this is a feature handy for every developer, especially for Grails developers as Hibernate as ORM basically supports only two modes of operation to handle the database schema (configured via the hibernate.hbm2ddl.auto option): wiping out the database and applying a new schema (called create or create-drop) or attempting to update the schema (simply called update). The update option looks like a good way to go at first but as it turns out it has its drawbacks, especially if there is data in the database.

How does it work?

The hub of all activity in LiquiBase is the changelog file. Basically there are two possible ways to create and maintain the changelogs, the first is to use the generated changelogs as far as possible and the other is to write changelogs by hand. I'm sticking mostly to the first one as the generated changelogs are sufficient most of the time and writing changelogs manually is way to tedious though easy as there is a complete XSD.

Each entry in the changelog file is called a changeset and has a unique id. The id is used to refer to specific datasets. But how does LiquiBase know which changesets have been applied to the database? The answer is simple, it creates a table called DATABASECHANGELOG into which it will insert a row for each applied changeset. So make sure you start using LiquiBase early because you really want to have these tables in your database when changes have been incorporated into a running system.

Applying changesets against the database is called migration but more on this later.

Setting the Scene

The work flow to push in changes may depend on your project. The work flow I'm describing can be applied to team of couple of Grails developers and has been tested on a real world project.

As an example for the rest of the post let's assume a project that is developed by a couple of code monkeys. The change the domain classes and to monkey stuff. On the other hand there is a data entry system deployed so real data can be inserted into the database and the software can be demoed. I will refer to this system as the demo system. The demo system and the data entered into it has several purposes. First to see if the domain model is adequate for real data and second to provide the developers with real testing data.

Problems arise quickly in this scenario, for example changes are made to the domain model and the database schema of the demo system must be updated, preferably without wiping out all the data that has been entered. At this point LiquiBase comes in.

Grails and LiquiBase - Getting Started

Ok, now how to use LiquiBase in a Grails Project? Easy: there is a LiquiBase grails plugin for grails which can easily be installed using the following command:

grails install-plugin liquibase

This will install a copy of LiquiBase in your application and set up a couple of new commands for the grails command line interface. Before you can do anything useful with LiquiBase you have to create a database changelog file for your current database, preferably using the generate-changelog command:

grails generate-changelog grails-app/migrations/changelog.xml

Note: if you omit the filename the changelog will be dumped to the console. You can dump the changelog into any file but grails-app/migrations/changelog.xml is the default and LiquiBase will keep looking for this file. With the changelog file in place you should add it to your version control system.

Tracking Changes

So the next step is to make some changes to the domain model which result in a change of the database schema. You now have two possibilities for updating your changelog file. Either manually or using some of the neat tools LiquiBase offers. I won't go into the details for creating the files manually, the LiquiBase manual does a good job at this. So let's talk about the tools.

LiquiBase ships with a very powerful command: diff. What this does is it looks at the current database and at the changelog (I have to correct myself here, see next section! Sorry about this!) another database and outputs the difference between them. As LiquiBase changelog XML! So this is awesome because you can make changes to your Grails domain model, run it against a database and pull out the changes. All that is then to do is to add the newly generated changesets to your database changelog file.

The db-diff Tool

The db-diff tool in the Grails plugin is kind of mysterious. The LiquiBase documentation for the Grails plugin does not even mention it and I got it wrong in the first run. I had to reading the section about the database diff provided by LiquiBase and dig through the Groovy scripts of the LiquiBase Grails plugin to understand what it does.

The diff utility of LiquiBase compares two databases with each other, which kind of explains its name (doh). When using the LiquiBase command line interface you have to provide two different connection strings for two databases. When using the Grails plugin however, you cannot specify two databases (not yet?); so what happens is that: the database for the current environment ist compared to the database of the test environment. This behaviour is hardcoded into plugins/liquibase-1.8.0.0/scripts/DbDiff.groovy and limits the usefulness of the tool. So what I did is every time a database has been migrated the new schema gets propagated to the testing database. If the domain model changes the schema of the dev database, the changes can easily be determined by diff'ing it against the testing database.

This is not exactly a perfect solution as you manually have to keep the testing database in sync with the changelog. So I am currently playing around with a modification of the Grails LiquiBase plugin which adds a compareToChangelog command which populates a separate database from the current changelog and then compares this to the database of the currently used environment. Using this tool makes updating the changelog really easy as you get the changesets just for those changes you made to the domain model. If there is interest I will publish this to the LiquiBase plugin.

Migrating a Database

Finally your changelog has been updated and you now need to update your demo database. The first thing you should do: Make a backup of your database! Although LiquiBase has matured to some extent, migrating a database full of data is a risky endeavor. So make sure you are on the safe side before touching a database with real data.

The easiest way to migrate the database is to set up a copy of your project and modify the datasource to point at the demo database. The first thing you might want to do is to check which changesets have to be executed by running the following command:

grails status

This will generate a list of changesets which have to be applied, assuming there is a DATABASECHANGELOG table in the database. If you are happy with what you see, you can either migrate the database directly or do a dry run. I prefer to do a dry run before touching the database by running:

grails migrate-sql

This will output a sequence of SQL statements that LiquiBase will issue to migrate the database. This is quite handy as you can see what will happen. Then, finally the database is migrated by issuing the following command:

grails migrate

When everything goes fine your database has just been updated. If something goes wrong, you can either use the rollback commands provided or fall back to the backup of your database.

Gotchas

The following points present pitfalls you want to avoid (its enough that I hit them . Please note that this is a living section, I keep adding new points I encounter.

Use LiquiBase early!

Make sure you start using LiquiBase early. Insert the DATABASECHANGELOG table as soon as possible and ensure that changesets are run against the databases.

Backups, Backups, Backups...

Backup your database before migrating! Really, do it!

Stick to one Database Flavor

Do not switch database flavors, e.g. from HSQLDB to MySQL. I tried this and I've encountered  a couple of issues with this. For example a changelog created from a HSQLDB may not run on a MySQL and vice versa due to different named datatypes (e.g. MySQL MEDIUMBLOB vs. HSQLDB VARBINARY). Further the changelog generated from a MySQL can cause problems on other databases as it names all primary keys "PRIMARY" per default. This leads to erroneous SQL that cannot be run.

Those problemes are nothing that couldn't be solved with a little bit of search and replace and even might be necessary to create the schema for a production database; but it is definitively nothing you want to do very often.

How to Start Using LiquiBase in the Middle of a Project

When you start to use LiquiBase in the middle of a project where there already is a database filled with precious production data, things can easily become complicated. The key to success here is to freeze the project or at least the part which affects the database schema for some time, generate a database changelog file from the current state and propagate this to all database instances. The reason why this is so important is that you need the DATABASECHANGELOG table on those databases; without this table and its entries  LiquiBase does not know which changes to apply. You could use the db-diff tool here, but the far more elegant approach is to simply propagate those changes to all databases once.

As soon as you have the DATABASECHANGELOG on all database instances, updating the schema becomes so easy that you will almost get addicted to it!

So, step by step:

• freeze your project for half an hour, maybe you can do this in the evening
• create a new changelog file from the current schema with the grails generate-changelog command
• mark all those changesets as imported with the grails changelog-sync command
• push those changes to ALL databases associated with the project. If you fail to do so, migrating the schema of a database without those tables is almost impossible. Depending on the size of your database this requires a massive amount of time.
• With the DATABASECHANGELOG table in place, migrating LiquiBase to the next version is as simple as entering grails migrate.

Conclusion

LiquiBase is a great addition to Grails and you should really really think about using it. Otherwise managing changes to the database can easily become a big problem and im talking out of experience here. Hopefuly this post helps you to get started with LiquiBase. As always if you have any comments please feel to post.

grails test war

Everything is fine when you stick to them. But if you created another environment, e.g. "foo", the same line wont work anymore:

grails foo war

Welcome to Grails 1.0.3 - http://grails.org/
Licensed under Apache Standard License 2.0
Grails home is set to: /home/jakob/grails-1.0.3

Base Directory: /home/jakob/workspace/fooproject
Script Foo not found.
Run 'grails help' for a complete list of available scripts.

This is annoying but easy to solve. All you have to do is to provide the name of the environment as a variable:

grails -Dgrails.env=foo war

Thats it.

Using value objects makes developing applications easy as your objects get very structured as you do not clutter classes with non domain properties. On the other hand this imposes several problems when constructing user interfaces, especially with the scaffolding plugin due to the fact that the scaffolding plugin only handles basic values like Strings or Integers. And Date objects. That made me curious as the scaffolding plugin creates a nice date selector for you. What is even more amazing is that the date selector consists of several HTML inputs and their input is converted transparently into a Date object.

And this is the solution to the problems described above. However, documentation on this specific topic is scarce. So I digged through the Grails source code to find out how the conversion of dates and other datatypes it is done.

Setting the Scene

For this example I will use the following, really simple value object:

public class Time implements Serializable {
  Integer hour
  Integer minute
}

Understanding Binding in Grails

After digging through the source code for quite a while I finally found the package that does the magic. Grails relies on some Spring-magic here by using the ServletRequestDataBinder facility. This is basically a class that does the hard work of converting request parameters into the required format with the help of good old PropertyEditors from the Java Beans specification. With this knowledge in place, the whole thing boils down on creating an appropriate property editor and registering it with Grails.

Understanding PropertyEditor

If you are new to the concept of PropertyEditors I recommend reading the javadocs both for PropertyEditor and PropertyEditorSupport and some example code. In the Spring manual there is a section about validation that gives a good introduction.

In a nutshell a PropertyEditor is a class that is responsible for converting a string representation of some sort into a more complex structure.

Using your PropertyEditor with Grails

"Registering" the PropertyEditor is easy, name it according to the convention "classNameEditor" and drop it into the same package as the value class it is for. If the Time class is in the package app.domain.common, the editor goes to the same package. The editor must be called TimeEditor then it will be used transparently.

Using it is easy, too. Take the following example HTML code, that is backed by a bean with a property of type Time:

<input type="text" name="time" value="${bean.time}" />

Submitting this form will automatically cause the PropertyEditor to convert the value entered in the textbox to whatever value object it has been written to.

Caveat

Creating your own PropertyEditor you will get a very flexible tool for easily creating nice UIs but there are some things you should keep in mind. First of all, you have to do all input validation yourself. Depending on the input you are building this can be a complex task. You should invest some time to ensure your PropertyEditor only accepts values it should, usually a regular expression is a good way to ensure this.

Second, you should try to keep your PropertyEditor simple, do not bloat it. It should only perform operations required to convert the value and nothing more.

Conclusion

The solution presented in this post is an easy way to create custom input elements but has one major drawback: it only supports input from a single text field. Applied to our time example this means that there is only one field to enter hour and minutes which seems acceptable for this simple value object but for complex objects like an address that consists of various fields this might be insufficient. I'll blog on this in the next couple of days if time permits.

I hope my post is of use to you and you now know how to create your own, powerful taglibs. As always, comments are welcome.