Nomenclature:

• [Local] Client: your local computer. In fact, if I say local, I mean the client.
• [Remote] Server: the server you connect to. If I say remote, I mean server.

Forward Tunnels

Your standard tunnel, allows you to take a local port and redirect it to a remote port on the server:

$ ssh -L REMOTEPORT:client:CLIENTPORT user@server

Now, that by opening a tunnel in this way:

$ ssh -L 4040:localhost:8080 user@server

you can connect to localhost:8080 and you'll get, through the tunnel, what's on server:4040.

Remote Tunnels

Regular forward tunnels are neat if you want to access a service that runs on a forbidden port through a firewall. But what if you need to map an incoming port on a remote server to a local port? Maybe because your ISP doesn't allow you NAT arbitrary ports?  Do not fear, remote tunnels are here for your rescue. They work pretty much the same way as regular tunnels, just the other way round. Instead of opening a listening socket on your local box, they open a port on the remote server. So, to  open a port on your remote box and forward it to a local port, use the following:

$ ssh -R REMOTEPORT:client:CLIENTPORT user@server

That's it. Well, almost. There's a catch. Per default sshd will now bind the listening socket to the loopback device. Kind of useless if you want to forward incoming connections through the tunnel. If you study the man page, you'll notice that you can specify a binding port for the remote tunnel:

$ ssh -R BIND_ADDRESS:REMOTEPORT:client:CLIENTPORT user@server

So, if your server's IP was 12.23.34.45, you would specify:

$ ssh -R 12.23.34.45:8080:localhost:8080 user@server

However, depending on your sshd configuration it might or might not work. Per default sshd won't let you just bind sockets to anything. You'll need to allow that, by specifying the GatewayPorts option in your sshd_config and restart sshd:

GatewayPorts yes

I got slightly confused, as the man page states that you can specify a value "clientspecified" to allow the client to select a specific interface, but it didn't work for me.

I am currently working on a large legacy system that is full of old code and to a certain extend full of comments. But sadly, most of these comments are not helpful at all:

Wow. Useless. Why document a getter (or setter)? But that's ok, that's something I could live with. However, consider the following snippet:

Ain't this helpful? Unfortunately not. However, a lot of developers seem to have the urge to document the obvious. That applies not only to real code statements, but to higher level entities as well:

As pretty as it looks, it's absolutely useless and a waste of screen estate. Everybody who is versed in Java (or your language of choice) will know that import statements are at the top of the class. Also, a range of lines that all start with the word "import" should give you a hint. Then there's the block indicating that a class definition will follow. Let this sink in. Is it that surprising and unconventional to find a class definition in a Java file that it has to be documented? I don't believe so. It is another example of wasted time and screen real estate.

Then there's another type of comments I run into a lot that make my life harder:

Commented out code is another big mystery to me. The developer who commented out the code, forgot about it by now. And everybody else just doesn't know what to make of it. Why is commented out? Why wasn't it removed completely? All these questions pop up in a developers head. And because he doesn't have a good answer, he'll just leave that snippet sit there. In fact, I strongly believe that the best way to preserve code for eternity is to put it into a comment, as most developers won't touch it. Why so many developers keep doing that in the age of source code management tools is beyond me. Whenever I find a commented piece of code, I delete it. And so should you. If the code were important, it wouldn't be commented. And you know, it's not that those lines are contributing to the actual application. They are just landmines of confusion. Remove them.

And then we get to my personal nemesis, comments where the developers actually tried to explain something but only manage to confuse the reader even more:

Ehr... what? I can see what happened in the developers head: "Hmmm... this if statement... it's big. Yes, it's pretty ugly, it took my 10 minutes to get that right. I'll better write a comment... ". Unfortunately, he just made it even harder to understand.

After reading these examples, one might argue "Complicated code is hard to explain!" or "It is really hard to write good comments!". While these points are all valid problems, writing code is the wrong solution in my opinion. Consider the following listing, which is a revised version of the previous abomination:

This might not be perfect, and the sample doesn't make a lot of sense, but after reading those 5 lines you should be able to understand what it is that is going on. So what did I change? First of all, instead of cramming everything into one line, the code has been split up. Instead of having everything in the if statement, it is split up. Separation of concerns. Variables with telling names make it really easy to comprehend the intention: "if (wildcardSearch ... ) - aaah, that makes sense". It might be more to type, but I dare to say that a comment is not necessary any more.

Code should be self explanatory. Code should be readable like a book. If you are feeling an urge to comment the lines you just wrote, your code probably needs some refactoring. Maybe simple things like renaming variables, reformatting, maybe more complex operations like abstracting functionality away. If it really is something complex, explain the WHY this approach was taken and explain HOW it works. Be precise, be concise. Don't waste words (sample from class FooBar):

This is one of my favorite examples. Almost an entire line of comments wasted. Empty words and phrases that don't add any value. First of all, after opening the file FooBar.java, you'll expect nothing else than class FooBar. So, the first words are unnecessary. Second, the phrase "The purpose of this class is to" doesn't carry any meaning. It's empty wordage. If you cut all the ballast away, you end up with the core of the comment:

Notice how the original comment of almost an entire line got boiled down to the core of the statement. Rule #17 from Elements of Style comes to mind: Omit needless words! To quote: "A sentence should contain no unnecessary words, a paragraph no unnecessary sentences, for the same reason that a drawing should have no unnecessary lines and a machine no unnecessary parts. " (p. 23, Elements of Style, Strunk and White). I think documenting the purpose of a class is important, but get to the point.

Comments Are Still Good

The above are all examples of useless comments. However, there are situations where a comment is good and necessary. If you have a public API, you'll want to explain the public methods and constructors. Consider the following questions: WHY would I use a certain constructor? WHAT is the return value of a method? HOW can I get the class to do XYZ? But as previously stated, be concise and short.

If you have a package for a certain purpose, write a package level package-info.java file to explain the package contents and maybe give a short sample of how to use the contained classes.

Final Advice

After a lot of ranting, here is a condensed list of advice.

What NOT to comment:

• The obvious. This includes getters, setters, default constructors, import statements, loggers, variable assignments,
• Code. Remove it or keep it.

What to comment:

• Public APIs including classes and methods.
• High level packages.
• If something is done in an unusual way, for example swallowing an exception: document WHY this exception doesn't need to be handled.
• Complex algorithms (if you really are implementing one).

How to avoid the necessity to write comments:

• Telling variable names
• Separation of concerns
• Use CONSTANTS!
• Don't try to be smart. Write simple solutions. If it's so complicated that you need to document it, refactor!

How to write GOOD comments:

• Be precise, short and concise. Don't babble.
• Try to answer the questions that a user of your API might ask: what, why, how, when?
• Prefer to explaining WHY something is done over WHAT or is done. WHAT is done is usually obvious.
• If necessary, give short samples of how to use.
• Minimize markup.
• Get yourself a copy of Elements of Style.
• Get yourself a copy of Clean Code.
• Check your spelling and punctuation.

But back to the actual problem: I had a lot of these files to look through and no tool to help me format them. So my first approach was to simply hack together a simple Java app (Yes, Java. Not Perl. Just felt like it. , doing some String manipulation and indent it. However I quickly realized that my simple formatter was thrown off by multiple elements in on line. The basic format is something like this:

# comment
\group {
    \optionOne "value"
    \optionTwo 3.1415
    \subgroup { ... }
}

Nothing to fancy. However my simple app was thrown off by something like this:

\group { \anotherGroup { ...

So I realized, yes, that's easy. I just need to tokenize the input stream, and when I'm at that, I can actually check for syntactically correct input. Then I remembered from my compiler classes in university that building lexers and parsers was not fun. Interesting, but just not fun. Then I remembered reading about ANLTR. A tool that builds lexers and parsers from a grammar. Something I always wanted to use but never found a reason to. And formatting these files, just gave me a really good reason dive in.

Lexers and Parsers in a Nutshell

If you are not familiar with the terms Lexers and Parsers, here's a quick rundown. Both terms are usually closely associated with compilers, because they solve a big part of the problems a compiler has to solve. A lexer, short for Lexical Analyser, is a program that splits an input stream based on certain rules into smaller chunks, called tokens. They are usually implemented using a regular language (Chomsky 3 to be precise) and can "recognize" simple constructs. For example, a hypothetical lexer for Java might read the following input:

int i;
i = 1 + 2;

and produce the following tokens (in this notation that I just made up - please note that the tokens are UPPERCASE):

[ INT ] [ WHITESPACE] [ IDENTIFIER('i') ] [ SEMICOLON ] [ NEWLINE ] [ IDENTIFIER('i') ] [ WHITESPACE ] [ ASSIGNMENT ] ...

Notice how every "piece" get's its own token. Comparing the tokens and the original input, it is obvious that  working with tokens is already so much easier that working with the raw text. So, now that we have tokens, what does the parser actually do? In very generic terms, the parser takes the tokens and "understands" the token stream. Or at least, he'll try to. Most parsers spend most of their lives trying to understand input and then failing because of some syntactic error. For example, you forgot a bracket in your program, or a semicolon. Take the slightly modified code from the first sample:

inmt i
i = 1  2;

Wow. Lot's of input that doesn't make sense, right? "The word "inmt" is not a valid type identifier!" you say! "And the semicolons! Where are they?" The experienced and weathered developer will scream in disgust: "This is invalid input!"   And this is where the parser comes in. A parser has a formal description of what the incoming token stream should look like. This formal description is called a grammar. A grammar is usually described in forms of "productions" which look like this very simplified sample (another notation I made up but heavily influenced by EBNF):

assignment -> IDENTIFIER ASSIGN expression;
expression -> IDENTIFIER | addition | multiplication;
addition -> VALUE + VALUE;
...

So, these productions map a generic concept to a more concrete concept. When the parser starts, it will match the first rule against the input, and slowly working it's way down to the most precise rules he can find (or not, if the input doesn't match the grammar). By consulting the grammar, the parser always knows what the input can contain. Usually the parser takes every production that matches and puts all of the input into a tree, the so called AST (Abstract Syntax Tree), a tree like representation of the input. For our initial sample, the (simplified) AST could look like this:

           .- variable(i)
          /
program -|                  .- variable(i)
          \                /
           `- assignment -|                .- NUMERAL(1)
                           \              /
                            `- addition -| 
                                          \
                                           `- NUMERAL(2)

Wow, that took me a time to "draw".

So, from this (simplified) AST you can see the structure of the program, right away. It is almost the pure essence of the meaning of the input code. Quite a useful and flexible mechanism. After the parser finished parsing the input tree and assuming the input was correct according to the grammar, the parser spits out a complete AST for the input. And with the AST, you can do a lot of fun and awesome things. To close the circle to my introduction, you can easily output a nicely formatted version of the input file without having to worry you might break the file. It can act as a verifier, testing whether you forgot some important input (semicolons, brackets, single or double quotes come to mind) or even translate it into something different.

So now you should have a really basic understanding of what a lexer and a parser does. There's a lot more to this highly interesting field. I can recommend the ANTLR Reference Guide or any of the dragon books about compilers (For some reason compiler books almost always have a dragon on them. Some say, it is because of the inherent complexity of compilers... which is probably true).

ANTLR

Back to my formatter/verifier for proprietary input. ANLTR takes off a lot of work from the developer because it will generate a complete lexer and parser (LL(*) parser to be precise) from a grammar for you. So all I had to do was to write a grammar for the proprietary format. Of course I don't want to keep that from you:

grammar MyGrammar;
options {
    language=Java;
    output=AST;
    ASTLabelType=CommonTree;
}
WS	:	( ' ' |'\t' |'\r' )+ { $channel=HIDDEN; };
NL	:	'\r'? '\n' { $channel=HIDDEN; };
DIGITS	:	'0'..'9'+;
EXTRACHARS
:	( '/' | '\\' | '_' | '.' | ':' | '-' | '|' | '<' | '>' );
CHARS	:	('a'..'z' | 'A'..'Z')+;
STRING	:	'"' (CHARS | EXTRACHARS | DIGITS | WS)* '"';
GROUPSTART
:	'{';
GROUPEND:	'}';
dialog	:	entry* EOF;
entry	:	comment | option;
option	:	'/' (DIGITS | CHARS)+ (ident | group);
ident	:	STRING | DIGITS;
group	:	GROUPSTART entry* GROUPEND;
comment	:	'#' (CHARS | DIGITS | EXTRACHARS )*;

In total it's a 33 lines with generous white space, a few comments and extra options I removed here. There's a few extra lines to load the input, and a single class that outputs a formatted file based on the generated AST. Everything else has been generated by ANTLR.

And suddenly I could verify and format ALL of our files using this tool. Awesome, isn't it?

Final Thoughts

The learning curve for an approach like this is steep. Grammars are nothing that you can pick up over night, it will require some effort to understand them and the pitfalls you can encounter (left recursive grammars, ambiguous grammars, etc etc) but the payoffs are enormous. Instead of manually formatting and manually counting opening and closing brackets in every file that I open, I can now run a simple oneliner that'll find all files and verify them for me:

$ find ./ -iname '*.xxx' -exec verifier {} \;

Done. All files. We have a few hundred of those, so there's some significant time savings in this little project. Also, developers are more willing to work with them now, as this little tool will always tell them if they screwed up the syntax or not.

All in all, I found this little exercise quite transforming. I was somewhat scared of parsers and compilers, just because it's hard to build them in a good way. But now, given the power of ANTLR and the grammar driven approach, I start to see new solutions for a lot of problems.

Therefore my advice: go and learn about grammars and play with ANTLR. It can make your life so much easier.

In the first SWF I just tried to get the basic code working, randomly moving particles, multiple colors and a nice blur effect. Looks cool, but turns black as the blur effect sums up (reload if its black already):

My next attempt is better. I used a ColorTransform to slowly increase the color values which gives this nice animation:

Amazing what can be done in an hour of playing around with Flash's drawing API and bitmap manipulation.

Hier gibt es die Code Samples!

This post is a write-down of my experiences building a XML/XSLT driven simple CMS system. In it I will show you the required technologies, my approaches and my solutions to problems I've encountered. You'll need a solid understanding of XML at least, understanding of Ant and XSLT helps a lot, too.

This post contains tons of XML and I tried my best to format it in a readable way — in fact I've spent hours to get everything nicely on the screen.

Requirements to the CMS

I had several objectives when I built my system:

• All mundane and repetitive tasks should be automated, including thumbnail generation, creation of a global navigation, synchronizing with a live site
• Flexible enough to support new "content types" and new views
• Versionable with Git, Subversion or other SCMs
• Creation of output artifacts should be a single command

Technologies

The most prominent and important technology I used is XSLT 2.0. If you don't know what XSLT is, here is XSLT in a nutshell.

XSLT stands for eXtensible Stylesheet Language Transformations. What it does is surprisingly simple. It takes a piece of XML as input (aka source tree or input tree), applies some transformations on it and outputs a new piece of XML. Sounds simple in theory, however in practice it is usually somewhat more complicated because XSLT is a functional language and sometimes requires you to solve problems backwards and don't even get me started on XML namespaces...

Traditionally you'd have one input XML file, one XSL stylesheet and accordingly one single XML output file. However with the new xsl:result-document I can suddenly output into several files.

As I'm using XSLT the next technology used is XML of course. Not much to say about this except that it is involved in pretty much every step. To bind everything together I'm using Apache Ant, an incredibly flexible tool. Interestingly enough Ant uses XML files for its build files which comes in handy later on. Also it solves one of my requirements: it is just plain text and therefor versionable.

Last there is image manipulation. I've experimented a bit with the Ant image tasks that use JAI to perform manipulation but quickly dropped that approach as this task won't even let you specify a quality parameter for image manipulations. I've ended up using an old friend: ImageMagick. ImageMagick is an amazing toolkit of versatile command line tools to modify images. It allows to modify images in many different ways from resizing to color correction or combining them.

Basic Structure

The basic structure of an XML/XSLT driven CMS contains one or more XML files that contain the "raw" data for the website, XSLT to transform it into XHTML and assets. More on assets later.

The XML Part

The XML file used in my project is a very simple one. It defines "pages" that get translated into real HTML pages. Each page has a section called contents which in turn contain the actual page contents. Let's have a look at a short sample:

Well, so far nothing impressive. Actually it looks like more work compared to typing out the actual pages themselves. But behold, here comes the stylesheet to actually turn the above XML into nice websites (XHTML):

Now it get's more interesting. Let's dissect what the above XSLT does. The blocks in line 10 to 12 are a template that matches the root level element in the source XML tree. All it does is to call the next matching stylesheets which are in lines 14 to 19. This template does exactly the same and passes on. Then we reach line 18 with the template matching page elements.

Line 18 defines a variable called filename which is prefilled with with the filename attribute from the XML page element. The next line sports the so much praised (at least by me) xsl:result-document element. If you read the line it gets clear very fast that it should output a new HTML file (format="html" href="$filename"), nicely formatted (indent="yes") and with the appropriate DOCTYPE declaration. The  next lines are just an HTML boilerplate with head and content type. Please note that this is a very minimalistic example. Usually you would include stylesheets, JavaScript includes and other additional headers there. For simplicity they are omitted.

Lines 28 to 31 output a new title tag populated with the title attribute from our source attribute. Convenient. The CMS is starting to take shape.

And finally lines 33 to 35 will call another template lines 41 to 46 which just copies the HTML into the output document.

So far so good. What have we achieved? The stylesheet creates a new page based on boilerplate HTML - which can be arbitrarily complex with CSS stylesheets, JavaScript and more - and pre-fills the title and the actual body. Now let's add more functionality. Every good website needs a navigation. We can achieve that easily by adding another template that matches all page elements. We add a few new lines to the boilerplate HTML:

Looks simple enough. This will output a new DIV, fill it with a unordered list and calls another template.  The interesting part is the select attribute: it tells the XSLT processor to select all root level page elements. Now let's have a look at how the called template looks like:

The XSLT processor will translate this into global navigation of our site. All top level pages will be listed with the correct title and the correct filename.

The Ant Part

Now that we have the stylesheets and the input XML files in place, we'll need an elegant way to transform the whole site with on simple command. As discussed earlier Ant is a good choice for this. I won't delve into the actual ANT buildfile as it is pretty straightforward. What I want to show is how to use XSLT to generate ANT buildfiles from other XML. This is actually a pretty powerful technique as I can custom tailor ANT files to whatever need I have. In this particular example I want to show how to create image thumbnails.

Here is what we have. In our input XML file, we reference images. For those images, let's say, product shots, we have the originals lying in some location in the filesystem. Those images need to be scaled down to a high res view (800×600) and to a thumbnail (128×128). We could do that with an ANT pathmatcher, but that would just take all the images it can find, not just the referenced ones. Bad and slow. And how do we keep the build directory free of old and unused pictures then?

What we'll do instead is we'll take the input XML file, create a XSLT stylesheet that will grab all the image references and put them into a custom ANT build file which in turn calls an image manipulation program (ImageMagick) to scale down the pictures. There's a few other goodies in here too, e.g. automatic file name cleanup. Here is the images.xsl:

Google Sitemaps

Most people that have used Google's Webmaster Tools know about the sitemap. No need to write them manually any more, you can easily use the XML representation and a XSLT stylesheet to generate it for you:

Issues

So with all the technologies in place, I ran into some issues but all of them could be solved quickly. I'm listing them here hoping they'll help other people.

XSLT 2.0 with Ant

This one was most annoying. Per default Ant has an xslt-task which allows you to transform XML but it is limited to XSLT 1.0. If you read the documentation you'll find out that you can plug in different XSLT processors but you are still quite limited. If you want to use XSLT 2.0 with Ant get the amazing Saxon processor and use their Ant task which is a bit hidden on their page. Get it from the Sourceforge download site. This task lets you leverage the full functionality of XSLT 2.0 including the xsl:result-document.

All you have to do to get it working is to copy the required JARs into the classpath, declare the custom task and use it:

Now you can use the saxon-xslt task:

XML Namespaces in HTML

This was a tricky one. The problem was that in my custom XML I had snippets of HTML which I wanted to mirror into the output documents. I used the xsl:copy-of function which basically did what it was supposed to. It copied the (X)HTML over but by doing so, it had to adjust the namespaces for the XML. The input document had no namespace declarations at all and the output document was bound to the XHTML namespace. So the processor did exactly what it is supposed to: it added null namespace  declarations to the copied elements which looked like this:

Not a problem per se but unfortunately invalid XHTML. The solution to this problem is surprisingly simple: Add namespace declarations to the input document:

The above lines will put the input document per default into the XHTML namespace and everything else in my own, personal namespace. Regular (X)HTML markup however does not get any namespace prefixes, so it will be put into the XHTML namespace:

Conclusion

The one big problem I still have with this system is that it does not support hierarchical pages yet. But if the need arises it could be added without too much effort. Another moose with the system is that it still requires editing XML which is neither user-friendly nor easy for non technical users. But one could easily create a Flex or AIR app to modify and edit those files. Thanks to Flex' really good XML support it should not be hard to do that. I could even think of a Grails Webapp generating the XML for this (using the brilliant XML markup builder).

In general it is a really powerful and flexible system. As I am running this on a UNIX system I can harness all the tools and goodies that are there. Ant and Java give me whatever power and libraries I need. So I can steer this thing in whichever direction I want.

If you have any questions feel free to comment below or drop me an email.

So, what's the idea behind this script? Everybody that uses Liquibase (if you are not, check out my blog post about what Liquibase is and how to use it) knows how painful it is to update your changelog.xml. Liquibase already gives you a great tool to start with, the db-diff command. However, it is hardcoded to diff the current environment's database against the test database. This is annoying as you'll have to modify your datasources and keep the database schema updated manually. But fear not, here is my (surprisingly) easy solution.

I've created a simple script (or rather, I took the db-diff script and hacked it), which I've called incremental DB diff (all the other cool names are already taken...) and what it does is the following:

1. update the schema in the db-diff (or whatever you call it) database using the changelog.xml
2. diff the database of the current environment against this database and output the diff as Liquibase XML

So, how is this a good thing? Easy: this little script will automatically create a target database to diff against using the changelog.xml which is good, as the changelog.xml represents your last migration status. And as it uses a separate database it doesn't influence the test database.

After running the script just put the output in the changelog.xml and you're done.

All you have to do to use this is to drop the file into your /scripts/ directory, create a new database and an according datasource entry for the environment "dbdiff" like this:

	dbdiff {
		dataSource {
			driverClassName = "com.mysql.jdbc.Driver"
			dbCreate = "create-drop"
			url = "jdbc:mysql://localhost/foo_dbdiff"
			username = "foo"
			password = "bar"
		}
	}

Download DbDiffIncremental.groovy. Drop me a mail or a comment if you have questions.

It took me some time to figure it out, hopefully I can save you from going through all this.

The Flex web tier compiler usually gives me this error:

Error: Could not resolve <namespace:Component> to a component implementation.

 <namespace:Component/>

error during compilation Could not resolve <namespace:Component> to a component implementation.

Initial research confirmed what I thought, this error is caused by a mismatch between a XML namespace declaration and the actual location of a component. However the location and the namespace declaration looked right and the issue occurred only after I had moved the questionable component into a separate module.

After two painful hours I realized that the Flex webtier compiler, as configured with GraniteDS, resolves namespaces relative to modules, not relative to the source folder. D'oh!

The solution I came up with was simple, might be naïve though. I figured that if resolution is relative to the Module and not to the source folder, there might be no such source folder as I would expect it. A quick look into the flex-config.xml found in <GRAILS_PROJ>/web-app/WEB-INF/flex/flex-config.xml confirmed this:

<!-- List of path elements that form the roots of ActionScript class hierarchies. -->
 <!-- not set -->
 <!--
 <source-path>
 <path-element>string</path-element>
 </source-path>
 -->

So there is no explicit source path set. So I changed it so it would point to <GRAILS_PROJ>/grails-app/views/flex:

<!-- List of path elements that form the roots of ActionScript class hierarchies. -->
 <!-- not set -->
 <source-path>
 <path-element>../../../grails-app/views/flex</path-element>
 </source-path>

Voilá - it works. However, as mentioned earlier, this might be a very limited solution, but for now it works.