Groovy at Twin Cities Code Camp April 4

Twin Cities Code Camp is a free, one day conference in Minneapolis this Saturday, April 4th. If you missed my and Scott's Groovy Metaprogramming a few weeks ago, never fear! We're booked in the second session of TCCC. The sessions look fun, so come on out. This is truly cutting edge stuff as I'll be showing off the new AST Browser that I added to Groovy 1.7 just this week. w00t!

For anyone down in Wisconsin, I'll be giving the same presentation at Capital City JUG on June 9th. See you there!

Why Dynamic Languages Metaobject Protocol?

Simula 67 introduced object inheritance in 1967. Thus has born late binding: the program does not know which service implementation on an object is invoked until runtime. Inheritance turned out pretty useful. In the late 1990s inheritance evolved into a preference for interface inheritance rather than implementation inheritance and we now live with the mantra of "composition over inheritance" for software design. Nevertheless, polymorphism lives on in most newer languages, with Joe Armstrong's Why OO Sucks rant an entertaining and notable exception from the creator of Erlang. By the way, besides inheritance, Simula 67 also introduced objects, classes, subclasses, virtual methods, coroutines, and discrete event simulation. The '60s were wild.

In 1980 Smalltalk introduced the doesNotUnderstand: message. In many programming languages, some sort of MethodMissing exception occurs when you invoke a service on an object that does not exist. In Smalltalk, instead of failing, the VM sends the doesNotUnderstand: message to the object along with the selector of the message and its arguments. If you want, you can implement any arbitrary logic you want within the message handler, and the callers don't know any different that this particular message wasn't part of your declared API. Thus was born even later binding: the program does not know which services an object provides until runtime.

Sometime after its first release in 2003, Groovy introduced invokeMethod and getProperty. In a language built around the idea of messages, like Smalltalk is, the doesNotUnderstand: message is a simple way to expose method selection to the user. The VM doesn't locate the method for you, it simply tells you what was asked to be invoked and then provides you with easy hooks to respond. The Java platform is different. The compiler does some of this work for you and the runtime does the rest. The method resolution rules aren't simple, and the result of finding a method to invoke is currently baked into the class' bytecode (invokeDynamic is coming though). Groovy works around this with the invokeMethod and invokeProperty hooks. Any object in the Groovy platform can override these methods to provide runtime service creation. Your object doesn't have a findByCityAndState(String, String) method? No problem, override invokeMethod and provide one at runtime. (Note, GORM actually uses ExpandoMetaclass for dynamic finders but the intent is the same). Groovy wrote their own runtime method dispatch algorithm that sits on top of that of Java's, and they let you hook into it. I mentioned the concept was simple in Smalltalk... here is a diagram of the Groovy dispatch logic courtesy of Venkat Subramaniam.
Simple? I say no. Is it the same way methods are dispatched as in Rhino, JRuby, or Jython, all dynamic languages on the JVM? No. A lot of coding in dynamic JVM languages occurs in a mixed environment where Java and dynamic code coexists in the same process. What happens when Java invokes findByCityAndState? Trick question, it can't be done. Synthesized methods within invokeMethod do not appear in the public API of the class bytecode, so the Java compiler can't even see the method and won't even compile. Even if the method were in the class, the Java code would call straight into the method implementation and avoid all the indirection. What happens when a Jython script calls into a Groovy script? Again, the call would be filtered through the Java layer so no cross language metaprogramming would be possible unless a custom Groovy-aware adapter were written that replicated the Groovy method dispatch.

Clearly, users want control over how services on objects are located, and a cross language solution would be best. Enter 2007 and Atilla Szegedi's JVM Dynamic Languages Metaobject Protocol Library. Instead of baking method dispatch into the language runtime, it is externalized into a MetaObjectProtocol. So in Rhino, the JVM JavaScript engine, instead of interacting with the Scriptable object to get a property like this:

((Scriptable)obj).get("foo");

You'd interact with the MOP, like this:

metaobjectProtocol.get(obj, "foo");

I believe the IRubyObject serves the same purpose in JRuby. In fact, most languages have a concept of a parent adapter class that would get replaced by the MOP, which Atilla calls a "Navigator" instead of the adapter. The power of this is a universal invocation interface. Cross language compatibility is ensured by writing method invocations through the MOP, and each language would come with its own implementation of the MOP.

The coolness of this is the composibility of the MOPs. The composite runtime MOP might be composed of a Groovy MOP, a JRuby MOP, and a Java MOP. When a method is invoked, the chain of MOPs is consulted. If the first MOP can handle the invocation then it does so, but if it cannot then it replies with "No Authority" and the next MOP in the chain is called. The CompositeMetaobjectProtocol class is just the plain old Composite Pattern. Who says design patterns are dead? And while each language would have its own MOP, you could certainly have your own for your library or framework. This puts the method resolution algorithm in the hands of the user.

The history of programming languages doesn't seem to be about making the right fundamental decisions and having further work build on it. It seems to be about finding ways to safely put more and more flexibility in the hands of the programmers. I relish the idea of one day being able to safely modify method invocation in a language, and the Dynamic MOP seems like a good step forward.

Visualizing Groovy AST

So you want to write a Groovy AST transformation... it couldn't be simpler to read a compiling program than with the new local transformation capabilities. The hard part, for me at least, is figuring out how to modify that program. To do that you're going to have to hand write something into the abstract syntax tree (AST).

This post explains Groovy AST by example, starting simple and ending... well, still pretty simple.

An AST is a tree representation of your source code. The tree is composed of leafs and branches, most of which are subtypes of ASTNode. The two main classes of node are Expression objects (like MethodCallExpression and BinaryExpression) and StatementObjects (like BlockStatement and ReturnStatement).

Here is the AST for 4. Just plain old 4.

4 is a BlockStatement with one expression attached. That ExpressionStatement has one ConstantExpression, the Integer 4, attached.

The AST for 4+2 is more interesting:

Again, there's a BlockStatement with one ExpressionStatement attached. But now that expression has a BinaryExpression attached. The BinaryExpression is made up of 4:Integer, the + Token, and the 2:Integer expression.

The next AST shows some variable declarations from this code: int x = 4; x*x
This time the BlockStatement is made up of two expressions, a DeclarationExpression and a BinaryExpression.

The last example is for the code "println 4". It shows a method invocation:
Hopefully I've given you a flavor of what AST looks like and a feeling for the size of the task you're facing when you're thinking about writing some. It's not all that difficult but there are a lot of different types involved and there's not exactly great tooling built around it.

Global AST Transformations in Groovy 1.6

Groovy 1.6 offers offers several approaches to transforming the AST of code within the compiler. You can write a custom AST visitor, you can use annotations and a local AST transformation, or you can use a global AST transformation.

This blog posts explains how to write and debug a global AST transformation.

Sticking with the naive and simple example from the local transformation post, consider wanting to provide console output at the start and stop of method calls within your code. The following "Hello World" example would actually print "Hello World" along with a start and stop message:

def greet() {
   println "Hello World"
}

greet()

Admittedly not a great use case, but it is useful to explain the mechanics of global transformations.

A global transformation requires four steps: 1) write an ASTTransformation subclass, 2) create a Jar metadata file containing the name of your ASTTransformation, 3) create a Jar containing the class and metadata, and 4) invoke groovyc with that Jar on your classpath.

Writing an ASTTransformation

This is almost exactly the same step you'll need if writing a local transformation. You must define an ASTTransformation subclass that reads, and possibly rewrites, the syntax tree of the compiling code. Here is the transformation that will add a console start message and end message to all method invocations:

@GroovyASTTransformation(phase=CompilePhase.CONVERSION)
public class LoggingASTTransformation implements ASTTransformation {

   static final def TARGET = WithLogging.getName()

   public void visit(ASTNode[] astNodes, SourceUnit sourceUnit) {
       List methods = sourceUnit.getAST()?.getMethods()
       methods?.each { MethodNode method ->
           Statement startMessage = createPrintlnAst("Starting $method.name")
           Statement endMessage = createPrintlnAst("Ending $method.name")

           List existingStatements = method.getCode().getStatements()
           existingStatements.add(0, startMessage)
           existingStatements.add(endMessage)
       }
   }

   private Statement createPrintlnAst(String message) {
       return new ExpressionStatement(
           new MethodCallExpression(
               new VariableExpression("this"),
               new ConstantExpression("println"),
               new ArgumentListExpression(
                   new ConstantExpression(message)
               )
           )
       )
   }
}

The first line (@GroovyASTTransformation) line tells the Groovy compiler that this is an AST transformation that should occur in the conversion CompilePhase. Unlike local transformations, global transformations can occur in any phase.

The public visit(ASTNode[], SourceUnit) method is invoked for each source unit compiled. In this example, I'm just pulling out all the methods defined in the source. A method to the compiler is simply a list of Statement objects, so I'm adding a statement zero logging the start message and appending a statement to the end of the list with the end message.

Notice how hard it is to create a simple println Statement in the createPrintlnAst method. A method call has a target(this), a name(println), and an argument list(the message). As stated previously, I find the easiest way to create AST is to write the Groovy code I expect to create, then observe what AST the compiler generates within the IDE's debugger. This requires a test harness with a custom GroovyClassLoader and an AST Visitor. Perhaps this gets easier with time?

Writing Jar Metadata

The Groovy compiler discovers your ASTTransformation through a file named "org.codehaus.groovy.transform.ASTTransformation". This file must contain the fully qualified package and name of your transformation. In my example, the file simply has one line:

gep.LoggingASTTransformation

Creating the Jar

The ASTtransformation and the metadata must be packaged into a single Jar file. The org.codehaus.groovy.transform.ASTTransformation file must be in the META-INF/services directory. The Jar layout for this example follows:

LogMethodTransform.jar
--gep
----LoggingASTTransformation.class
----LoggingASTTransformation$_visit_closure1.class
--META-INF
----services
------org.codehaus.groovy.transform.ASTTransformation

Compiling the Example

The new Jar must be put on the groovyc classpath for the transformation to be invoked. If the sample script at the top of the post is in a file named "LoggingExample.groovy", then the command line to compile this is:

groovyc -cp LogMethodTransform.jar LoggingExample.groovy

This generates a LoggingExample.class that, when run with Java, produces:

Starting greet
Hello World
Ending greet

Mindblowing, right?

Debugging Global Transformations

Local transformations are simple to debug: the IDE (at least IDEA) supports it with no extra effort. Global transformations are not so easy. To test this I wrote a test harness that invoked LoggingASTTransformation on a file explicitly. The test harness source is available and could easily be modified to fit your needs. Let me know if you know an easier way to debug this!

Enjoy, and (as always) full source is available at: http://svn.assembla.com/svn/SampleCode/gep/src/gep/transform/global/

Local AST Transformations in Groovy 1.6

It's been baking for so long that I forgot Groovy 1.6 was still in release candidate mode! Well, it's here at last and Guillaume wrote a good overview on InfoQ.

Groovy 1.6 provides two options for hooking into the Groovy compiler for compile-time metaprogramming: local and global AST transformations. This post explains how to write and debug a local AST transformation.

As a naive and simple example, consider wanting to write a @WithLogging annotation that would add console messages at the start and end of a method invocation. So the following "Hello World" example would actually print "Hello World" along with a start and stop message:

@WithLogging
def greet() {
  println "Hello World"
}

greet()

An incredibly poor man's aspect oriented programming, if you will.

A local AST transformation is an easy way to do this. It requires two things: a definition of the @WithLogging annotation and an implementation of ASTTransformation that adds the logging expressions to the method.

An ASTTransformation is a callback that gives you access to the SourceUnit, through which you can get a reference to the AST. The AST is a tree structure of Expression objects that the source code has been transformed into. An easy way to learn about the AST is to explore it in a debugger, which will be shown shortly. Once you have the AST, you can analyze it to find out information about the code or rewrite it to add new functionality.

The local transformation annotation is the simple part. Here is the @WithLogging one:

import org.codehaus.groovy.transform.GroovyASTTransformationClass

@Retention(RetentionPolicy.SOURCE)
@Target([ElementType.METHOD])
@GroovyASTTransformationClass(["gep.LoggingASTTransformation"])
public @interface WithLogging {
}

The annotation retention can be SOURCE, you won't need the annotation past that. The element type here is METHOD, the @WithLogging annotation applies to methods. But the most important part is the @GroovyASTTransformationClass annotation. This links the @WithLogging annotation to the ASTTransformation subclass you will write. gep.LoggingTransformation is the full package and class of my ASTTransformation. This line wires the annotation to the transformation.

With this in place, the Groovy compiler is going to try to invoke gep.LoggingASTTransformation every time an @WithLogging is found in a source unit. Any breakpoint set within LoggingASTTransformation will now be hit within the IDE when running the sample script.

The ASTTransformation subclass is a little more complex. Here is the very simple, and very naive, transformation to add a method start and stop message for @WithLogging:

@GroovyASTTransformation(phase=CompilePhase.SEMANTIC_ANALYSIS)
public class LoggingASTTransformation implements ASTTransformation {

  public void visit(ASTNode[] nodes, SourceUnit sourceUnit) {
      List methods = sourceUnit.getAST()?.getMethods()
      // find all methods annotated with @WithLogging
      methods.findAll { MethodNode method ->
          method.getAnnotations(new ClassNode(WithLogging))
      }.each { MethodNode method ->
          Statement startMessage = createPrintlnAst("Starting $method.name")
          Statement endMessage = createPrintlnAst("Ending $method.name")

          List existingStatements = method.getCode().getStatements()
          existingStatements.add(0, startMessage)
          existingStatements.add(endMessage)
      }
  }

  private Statement createPrintlnAst(String message) {
      return new ExpressionStatement(
          new MethodCallExpression(
              new VariableExpression("this"),
              new ConstantExpression("println"),
              new ArgumentListExpression(
                  new ConstantExpression(message)
              )
          )
      )
  }
}

Starting at the top...

The @GroovyASTTransformation(phase=CompilePhase.SEMANTIC_ANALYSIS) line tells the Groovy compiler that this is a local transformation that applies to the SEMANTIC_ANALYSIS CompilePhase. Local transformations can only be applied at semantic analysis or later phases, and this line is required!

The public visit(ASTNode[], SourceUnit) method is invoked for each source unit that contains your target annotation. The AST you receive is not for the @WithLogging annotated method, it is for the entire file that contains @WithLogging. In this example, I'm just using findAll to locate methods that are annotated with @WithLogging, then using an each statement to wrap any annotated method with print lines. A method to the compiler is simply a list of Statement objects, so I'm adding a statement zero logging the start message and appending a statement to the list with the end message.

Note the creation of the new println statements in the createPrintlnAst(String) method. Creating AST for code is not simple. In this case I need to construct a new method call, passing in the receiver/variable, the name of the method, and an argument list. When creating AST I like to write the code I'm trying to create in a Groovy file and then inspect the AST of that code in the debugger to learn what I want to create. Then I write a function like createPrintlnAst using what I learned through the debugger. Cumbersome, for me at least.

The final result:

@WithLogging
def greet() {
  println "Hello World"
}

greet()

Produces:

Starting greet
Hello World
Ending greet

Enjoy! Full code available at: http://svn.assembla.com/svn/SampleCode/gep/src/gep/transform/local/