The System Metaphor is Still Lame

The eXtreme Programming System Metaphor practice makes a pretty easy target. I've seen large development teams grind to a halt because of persistent broken builds without any continuous integration. I've seen teams' productivity drop to a crawl under the burden of technical debt and lack of test driven development. But what happens when a team lacks a good metaphor for how the system works? Can you really name a reasonable, realistic negative impact from a lack of system metaphor? Just last week I was dismissing it in an internal agility training as an unneeded XP practice. So I was surprised to see Joshua Kerievsky's and Brain Foote's "System Metaphor Revisited" talk at Agile 2009. Seriously, the guy who wrote Refactoring to Patterns is also hyping the System Metaphor? We'll see...

The thesis of the talk was that a System Metaphor provides illumination, inspiration, and integrity (...and alliteration). Briefly and in my own words, it illuminates an unfamiliar design by providing a reference point in the familiar. It inspires unexpected design ideas by providing a frame of reference in which to apply your creativity. And it builds integrity by suggesting a single, coherent, and consistent framework and set of names into the system.

And the Industrial Logic case study is just so cool. I've loved their marketing since they started with the "Let the Good Designs Roll" idea. I want to be a rock star and I want to take their training classes. And now their trainings are all hinged around the ideas of albums, tracks, and playlists. Very cool. I guess it helps to have a great product.

I have no doubt that the guys presenting have deemed their system metaphor a qualitative success for the project. But I still think the practice is totally lame.

A metaphor embedded in code is rigid. Having a great marketing metaphor is fine; marketing campaigns sell products. But they're also easy to change and refine, while an IT system is not. How many defects will be introduced if you change your marketing metaphor from a rock and roll concert to a book store? Not many. How many defects are introduced by changing the code to reflect this? Is none really the answer?

A metaphor is a 2nd ubiquitous language. One universal domain language is of clear benefit to a project. And that language should be the language of your users, domain experts, and programmers. A system metaphor is usually not this same language. For my projects, it's hard enough to try to get all the stakeholders to agree on terms of use. Introducing another layer of abstraction and the associated verbiage isn't going to help things. Ubiquitous language is important; but create it out of your domain not your metaphor.

A metaphor illuminates in the large, but confuses in the small. A metaphor guides you towards understanding when contemplating projects as a whole. It provides familiar signposts and a framework on which to hang ideas. But a lot of programming involves making small changes to systems you only partially understand. When this is the case, the metaphor becomes accidental complexity in the system. Having postage stamp, postage meter, and postage machine objects in your system helps the original design team, but doesn't help the contractor being asked to figure out why reports aren't printing.

A metaphor will likely become a mixed metaphor over time: consider a system that treats all data as streams. It's consistent and elegant. But eventually the Enterprise knocks on your door and you develop some sort of messaging service. Now your streams have mailboxes and can be put in message queues. This undermines the benefit of integrity via consistency. We live with this sort of ambiguity every day, but it doesn't mean we should accept it.

Metaphors do not decompose well. You system will grow, it will take on technical debt, and eventually it will need to be refactored, decomposed, and modularized. This is the natural rhythm of agile systems. When it comes time to break out components, will your modularization boundaries be natural boundaries within your metaphor? At a minimum, you'll probably end up discussing how the two don't line up correctly, and now you're spending time solving a self-created problem... surely a bad process smell. It seems like only the most dependency-obsessed team leads could foresee and prevent this. A YAGNI approach may not be your ally here.

In the end, I see a metaphor as a set of constraints. Humans can do their best and most creative thinking when provided rigid constraints. But constraints limit design ideas as much as suggest new ones. I'm sure some people have had success with a system metaphor, but I find success because of system metaphor a much harder pill to swallow. To all those espousing the use of system metaphor as a useful practice, I quote a famous and apocryphal doctor and say, "Clearly I can see you're nuts".

And just for fun, I used eight metaphors in this blogpost. There is no prize for finding them all.

Counter-intuitive Agile Coaching Tips

Just finished the "What Do Agile Coaches Do?" session from Rachel Davies and Liz Sedley at Agile 2009. It was a great workshop based on their new book and J. Richard Hackman's "Leading Teams".

Hackman defines three types of coaching intervention:

• Motivational - intended to improve team or team member's effort
• Consultative - intended to improve a team's process
• Educational - intended to improve a team's understanding
  

Today's first insight is that I personally focus on educational and consultative interventions, to the almost complete exclusion of motivational intervention. I'm not interested in giving people pep talks, so clearly I'm lacking some motivational skills and techniques. Anyway, Hackman defines three key times when these interventions should be used within a project:

• Beginning - focus on motivational intervention
• Midpoint - focus on consultative intervention
• End - focus on educational intervention
  

Delaying education until the end of a project sounds ludicrous. Wouldn't the begining be better, when people need it? The answer is to think of a project as an iteration, not a full release. Iteration start is when the team needs energy to commit to work and kick off the sprint. Makes sense that the "sprint" start is when you want to get a good start out of the blocks, huh? So the beginning is a good time for motivation after all. The iteration midpoint is when the team is neck deep in implementation and sees that not all their practices are working in their best interest. What's the solution? A small process tweak via consultative intervention. The end is the retrospective is where the whole team has a forum and opportunity to make big changes to how they work. That's the time where education on a new and unknown practice can make the biggest impact, because the team can commit to it and take action immediately. Hackman's description of when the interventions are most effective sounded fishy at first, but when viewed against an iterative process it makes sense!

But non of this is counter-intuitive coaching. The most interesting part of the workshop was our group's breakout session where we discussed coaching in the context of some manufactured scenarios from the speakers. We came up with these unique coaching strategies:

1) Coaching Through Absurdity
In my group was Alex Chin, and in order to get his team to estimate at the task level he added huge estimates to each task. These grossly absurd estimates forced the team to come back and add better estimates. Awesome idea. Next time my team won't make a decision I'm going to make an absurd one for them and wait for a reaction.
2) Letting People Fail
We spent a lot of time discussing how to guide a team down the correct path. If your team insists on walking precariously close to a metaphorical cliff, then don't waste your time each week shepherding them off it. Let them fall of the cliff once. Surely that will teach them to avoid it next time. Right?
3) Doing Nothing is Doing Something
You can lead a horse to water but you can't make him drink. Likewise, you can't force a feedback loop. Coaching is not just about knowing when to intervene, but when not to. If there is a team issue that needs resolving, sometimes you just need to get the people together and wait. Do nothing and let the team decide. The most effective way to coach is to set the stage for team emergence and then get out of the way! Agile isn't about the coach solving your problems, it's about the coach getting you to solve your own.

Agile 2009 Day 0: Kanban Bag Stuffing

Spent about 4 hours today packing 1500 conference bags for Agile 2009. Luckily, several guys from the Kanban-Dev mailing list showed up to turn a scene from Modern Times into an unique exercise in Lean production.

I won't explain much about the "Pull" system or queuing theory backing Lean production; rather, I'll just explain what we did.

We had about 1500 bags to pack with more than 25 items each: a lot of flyers, several books (including past volunteer Ahmed Sidky's Becoming Agile), and even some agile hand lotion (wtf, right? But it beats the face creme I got at a triathlon called, I kid you not, "every manjack"). Anyway, it all needed to be bagged, carted, and trolleyed to the reception area. Uff da.

The group split into two teams. Our goal was obviously to get the bags packed as quickly as possible. The Kanban experiment was used to create a system of flow, in which each group had a steady stream of product through their production line. Bottlenecks meant the upstream production unit should slow down and not having work to do meant the upstream unit had to speed up. But it wasn't about working more or working less to achieve flow. It was about tweaking the process so that everyone working hard would result in the flow. The group had to self organize and evolve so that this steady state existed. Over two shifts of people this did all happen to a certain extent. We didn't have measurements to see who was fastest, but each process did improve over time. Here are some observations, along with how they might apply to software teams:

• Having no leader meant that every team member contributed new ideas to process improvement. Does the leader of your software team sometimes stifle innovation?
• Having two teams meant that competition fueled more frequent improvements. What external forces are driving your software team to evolve?
• Having no roles created a system where shouting "We're out of Agile Hand Lotion" meant that the person most capable of grabbing more lotion at that point in time immediately pitched in to do the job. Are the roles in your organization lightweight enough to allow the right person to contribute?
• Having one set of workers out-of-sight (running trolleys through elevators and hotels) meant that it was more difficult for the group to self correct their entire process. Finished bags did pile up and we were at a loss as to how to make the trolley's move faster. What parts of your SDLC are out-of-sight and out-of-mind?
• Not producing a Value Stream Map meant it was difficult to see the whole and fix the real problem, which was slow elevators. VSMs are much lower ceremony than most people realize, and we could have done one. Can your team truly see the whole?
• Many improvements were small and non-obvious in retrospect - "Why don't we pack the books last because they are heaviest?" and "Open the bags more quickly by pulling the handle like this." Does your company have a process group that focuses on the macro view to the exclusion of the micro view?
  

At the end of the day, pro facilitator Jean Tabaka of Collaboration Explained fame hosted our retrospective for us. The format was ORID:

• Objective – Focus on the facts, hard evidence data
• Reflective – Focus on how that made people feel or other associations
• Interpretive – Focus on the impact and significance
• Decisive – Focus on next steps
  

It was a typical retrospective in that we gathered data and created a shared understanding of the event in the Objective phase, talked about our feelings and personal experiences in the Reflective phase, generated insights in the Interpretive phase, and voted with dots in the Decisive phase to create a list of action items for next year. She and the rest of the team did a great job, and both Kanban and the retrospective were superb ways to kick off the volunteer sessions!

And just because I get a kick out of having foreign characters in my posts: 看板 is the symbol for Kanban. Whee!

Exploring Immutability

When did immutable objects become idiomatic Java? I suppose it depends on your frame of reference and what your experience brings to coding. It may not be idiomatic Java to you. In fact, Java Beans and fluent interfaces seem to rely on in-place side effects and mutability to function properly. Exactly what place do immutable objects have in the Java world?

The immutability of String surprised me when I was learning Java. One of the first programs I wrote assumed String#replace did an in-place edit of the String, one of the first debugger sessions I conducted clearly showed me that String#replace was not editing the String, and one of the first Sun Forum questions I posted was about String#replace being broken in Java. BROKEN, I say! This represents the one time when an answer to a Sun Forum question clearly solved the poster's problem without a lengthy chain of "please post your code" responses. The point is that immutability has been with Java forever. And now that the days of Object Pools and persistent performance concerns are behind us, it's time to embrace immutability in our own code.

The benefits of immutability are many and well-documented. Effective Java (either edition) is a good start. I'll touch the highlights with the example of a Person object:

class Person {
 def firstName
 def lastName
}

HA! Tricked you, it's Groovy. In the Groovy compiler, the Person class gains two methods for each property defined:

public void setFirstName(Object)
public Object getFirstName()
...

The problem with the Class (as the JVM sees it), is that many threads may access Person and see the same instance in different states. It is difficult to share this object correctly across threads. The problem with the Java source is the cruft of writing getters and setters. The problem with the API of Person is that working effectively with the object requires many calls to setX and setY mutators just to get simple interactions working. The problem with maintaining this code is that the object is somewhat non-deterministic and difficult to reason about. Groovy uniquely solves only one of these issues: the source level cruft. Sure, you don't have to write out getters and setters, but that is the least important problem to solve.

Fluent APIs solve the issue of having to munge up the source code with repetitive calls to setX and setY. Instead of following the Bean pattern of setters being void methods, instead have all your setters return a 'this' reference:

class Person {

 def firstName
 def lastName

 def setFirstName(firstName) {
   this.firstName = firstName
   this
 }

 def setLastName(lastName) {
   this.lastName = lastName
   this
 }
}

Now you have a nicer API to work with at the expense of not having a standard Java bean. Remember the Java language change proposal to make void methods always implicitly return a 'this' reference? I feel like I'm the only guy wanting that in Project Coin. Oh well. Here is what consuming the API now looks like:

def person = new Person()
   .setFirstName('hamlet')
   .setLastName('darcy')
assert 'hamlet darcy' ==  "$person.firstName $person.lastName"

Groovy has a solution to this same problem: the with block. All object have a with method, and when inside the closure parameter, all method and field references are resolved back to the object having with invoked. Visual Basic had this feature and I always missed it in Java:

def person = new Person()
person.with {
   firstName = 'hamlet'
   lastName = 'darcy'
}
assert 'hamlet darcy' ==  "$person.firstName $person.lastName"

So both Groovy and Java have an answer for the boilerplate involved with writing and consuming APIs. But again, these aren't the big issue with mutable objects: non-determinism and thread safety are. To help solve that you can use immutable objects.

Can we get nice, fluent interfaces in Java without sacrificing mutability? The go-to solution in Java is the Builder idiom. Make your Person object immutable (all final fields), and the constructor private. Then define a mutable Builder class that wraps the constructor in a better API. In the end, creating Person instances looks like this:

def person = new PersonBuilder()
       .setFirstName('hamlet')
       .setLastName('darcy')
       .build()
assert 'hamlet darcy' ==  "$person.firstName $person.lastName"

You instantiate the Builder and the builder instantiates the Person. This is nice... I guess maybe. Remember the idea of a "meaningless abstraction"? PersonBuilder is definitely a meaningless abstraction. It does not exist to solve any of the problems the system was designed to do. It exists solely to solve a problem in the implementation. Suspect. Very suspect.

The Builder has by far the longest implementation, and the DRY principle is glaringly violated in the duplicate fields:

class Person {
   private final def firstName
   private final def lastName

   Person(firstName, lastName) {
       this.firstName = firstName
       this.lastName = lastName
   }
}

class PersonBuilder {
   def firstName
   def lastName

   def build() {
       new Person(firstName, lastName)
   }

   def setFirstName(firstName) {
     this.firstName = firstName
     this
   }

   def setLastName(lastName) {
     this.lastName = lastName
     this
   }   
}

It should be said, though, that this idiom does address the determinism and concurrency issues with mutability. Person can now be shared freely across threads, cached safely, and behaves in a more deterministic manner.

The best approach has not yet been mentioned. Simply create an immutable object without a builder, and also provide a nice fluent API around it. The calling code can look exactly the same as the previous fluent API example:

def person = new Person()
       .setFirstName('hamlet')
       .setLastName('darcy')
assert 'hamlet darcy' ==  "$person.firstName $person.lastName"

You can do this by slightly tweaking the fluent API example. Don't return a 'this' reference from setters, just return an entirely new object. And don't worry about all those little objects you're creating and tossing away.

class Person {

   private final def firstName
   private final def lastName

   Person() { }

   Person(firstName, lastName) {
       this.firstName = firstName
       this.lastName = lastName
   }
  
   def setFirstName(firstName) {
       new Person(firstName, lastName)
   }

   def setLastName(lastName) {
       new Person(firstName, lastName)
   }
}

Is a Person instance threadsafe? Yes. It is immutable.
Is a Person instance deterministic? Yes. No race conditions here.
Is a Person instance API easy to consume? Yup, it has a fluent API.
Does the Person class introduce a meaningless abstraction? Nope, and good thing!
Does the Person class require an exotic programming language to implement? No, this works fine in Java.
Does the Person implementation require tons of boilerplate? Maybe. The good part is that every implementation of a setter is exactly the same. The bad part is that every implementation of a setter is exactly the same. Not very DRY, but it could be automated in Groovy at runtime with a little metaprogramming.

That's it, immutable exploration over. And what place should immutability have in the Java world? It should be front and center. I hope I've shown that immutability and good APIs do not need to conflict with each other. You can get both with a tiny bit of extra work.

How to Tell Your Co-Workers Have Discovered Functional Programming

Three ways to tell that your co-workers are learning a Functional Programming language at home:

1. You've found the number of .class file in your project quadrupled in the last month.

Here is an output directory with 17 .class files, only 2 of which are not inner classes, most of which are anonymous.



See all those dollar signs indicating inner/anonymous classes? Uh oh, looks like someone discovered shitty lambdas. Your chaos meter should be increasing... will your project be maintainable of it's not idiomatic Java?

2. You've found a class named Pair

public class Pair<T, U> {

   private final T left;
   private final U right;

   public Pair(T left, U right) {
       this.left = left;
       this.right = right;
   }

   T getLeft() { return left; }
   U getRight() { return right; }

   static <T, U> Pair from(T left, U right) {
       return new Pair<T, U>(left, right);
   }
}

It's not a tuple, but it's about 2/3 of the way there! There's already jfun.util.Pair in JFun and gnu.lists.Pair in Gnu Lists, and now you've got your own too. Hooray. But wait, how will you maintain your code base if the declared name of your data structures don't tell you how it's meant to be used? Chaos... level... rising...

3. You've found a widely implemented one-method highly generic interface

An interface like this has over 100 implementations, many of which are anonymous:

public interface Transformer<T, U> {
   U transform(T input);
}

This is equivalent to F in Functional Java and Transformer5 in JConch.

At this point your chaos meter should be off the chart. These functional programmers are surely ruining your system and livelihood. So, I'm hesitant to mention what happens when you combine all of them together:

Transformer identityfunction =
   new Transformer<Pair<String, String>, Pair<String, String>>() {
       public Pair<String, String> transform(Pair<String, String> input>) {
           return input;
       }
   }
}

Voila, an anonymous function object that uses the Pair class to make a 1-arity method into a 2-arity method, returning multiple values.

These are the trials and tribulations of finding the appropriate use of new programming paradigms in old languages. Looking at version control history, the person that committed all these sins was... me. Will this be the Java code I write in two years time? Certainly not. But which of these example is the growing pains of learning FP and which of these will still be with me in the future?

Comments are open!

Testing OSGi Bundles with Pax Exam, Groovy and Gradle

You can't be an expert on everything. There are a few topics in life I've given up on, forever to be ignorant:

• Botany - Don't ask me the name of any plant or tree. I won't know.
• Hardware Specs - Telling me the model number and speed on your new laptop just makes me confused. I stopped reading the Computer Shopper years ago.
• Maven - I'm happy to use it IF SOMEONE ELSE CREATES THE POM. Life's too short for that much XML.
  

So what was I to do with Pax Exam? This framework allows you to write JUnit or TestNG tests and then executes them in the OSGi container of your choosing. It's a cool framework but much of the documentation assumes a willingness to scroll through lines and lines of Maven XML. Ugh. Life's too short to grapple with build systems.

So the nut is cracked. I've got Groovy unit tests running across Equinox, Felix, and Knopflerfish from both Gradle and IntelliJ IDEA. No Maven in sight.

Writing Pax Exam Tests in Groovy

Once it was all configured, writing Pax Exam tests were simple. Start with a plain old JUnit 4 test and do two things:

1. Annotate it to run with the Pax Exam JUnit runner
2. Configure it to provision the Groovy-all JAR
   

Here it is, complete with imports and ready for a little copy-paste action.

import org.junit.runner.RunWith
import org.ops4j.pax.exam.junit.JUnit4TestRunner
import org.junit.Test
import org.ops4j.pax.exam.junit.Configuration
import org.ops4j.pax.exam.Option
import static org.ops4j.pax.exam.CoreOptions.*

@RunWith (JUnit4TestRunner)
class GroovyIntegrationTest {

  @Configuration
  public Option[] configure() {
      [
          provision(
              mavenBundle().groupId('org.codehaus.groovy').artifactId('groovy-all').version('1.6.4')
          )
      ] as Option[]
  }

  @Test
  public void testFramework() {
      println 'Hello from Pax-Groovy!'
  }
}

A few things to notice... CoreOptions was statically imported, so the methods provision() and mavenBundle() are resolved. The configure() method returns an Array of Option objects, so the "as Option[]" cast is needed. And by default, the tests are going to run on Felix 1.8 (or thereabouts).

Before Pax Exam becomes useful you're going to want to provision your bundle as the system under test, get access to the BundleContext it was loaded with, and specify more containers to run on.

Provisioning your bundle and running on multiple containers is a matter of configuring the test differently:

@Configuration
public Option[] configure() {
  [equinox(),
  felix(),
  knopflerfish(),
  provision(
      bundle(new File('./../out/production/Filter4osgi.jar').toURI().toString()),
      mavenBundle().groupId('org.codehaus.groovy').artifactId('groovy-all').version('1.6.4')
  )] as Option[]
}

Here I'm specifying Equinox, Felix, and Knopflerfish. There's also options for allEquinoxVersions, allFelixVersions, and allKnopflerfishVersions, as well as allFrameworks and allFrameworksVersions to test on the world. That options takes a few minutes to execute! Provisioning my own bundle can be done by loading it as a file URL. This example isn't very robust but it proves the concept.

The BundleContext is available as an injectable bean from Pax Exam. If you want the BundleContext in your unit test, to perhaps test that services were registered correctly, then add this as a field:

@Inject
BundleContext bundleContext

And here is a simple test to make sure that my filter4osgi library was correctly installed:

@Test
public void test_BundleIsLoaded() {

  def found = bundleContext.bundles.find {
      'filter4osgi' == it.symbolicName
  }
  Assert.assertNotNull('filter4osgi bundle not loaded!', found)
}

Running Pax Exam Tests from Gradle

The Gradle build was slick. Tell Gradle to load your bundle as a file URL and the rest is just boilerplate dependency management:

usePlugin 'groovy'

repositories {
  mavenCentral()
  flatDir dirs: [
      './../out/production',   // filter4osgi jar built earlier
  ]
}

dependencies {
  groovy group: 'org.codehaus.groovy', name: 'groovy-all', version: '1.6.4'
  compile(
      'org.ops4j.pax.exam:pax-exam:1.0.0',
      'org.ops4j.pax.exam:pax-exam-container-default:1.0.0',
      'org.ops4j.pax.exam:pax-exam-junit:1.0.0',
      'junit:junit:4.5',
      'org.osgi:org.osgi.core:4.0.1',
      ':filter4osgi:',
  )
}

"gradle test" is the only target you'll ever need.

Running Pax Exam Tests from IntelliJ IDEA

Your unit tests will show up once for each container you're targeting. So my two tests run against 3 containers shows up as 6 tests in the test runner window, plus a sweet ASCII art logo:

One nice side effect of Maven is that IDEA can generate files off it, so you don't need to worry about the transative dependencies in the IDE, it's all configured for you. When using Gradle you do need to set up the IDE manually. I added the following JARs as dependencies and it all worked fine (with one exception):

commons-discovery-0.4.jar
commons-logging-1.1.jar
easymock-2.4.jar
junit-4.4.jar
log4j-1.2.12.jar
ops4j-base-lang-1.0.0.jar
ops4j-base-monitors-1.0.0.jar
ops4j-base-net-1.0.0.jar
org.osgi.core-4.0.1.jar
osgi-3.4.0.jar
pax-exam-1.1.0-SNAPSHOT.jar
pax-exam-container-default-1.1.0-SNAPSHOT.jar
pax-exam-container-rbc-1.1.0-SNAPSHOT.jar
pax-exam-container-rbc-client-1.1.0-SNAPSHOT.jar
pax-exam-junit-1.1.0-SNAPSHOT.jar
pax-exam-junit-extender-1.1.0-SNAPSHOT.jar
pax-exam-junit-extender-impl-1.1.0-SNAPSHOT.jar
pax-exam-runtime-1.1.0-SNAPSHOT.jar
pax-exam-spi-1.1.0-SNAPSHOT.jar
pax-exam-testng-1.1.0-SNAPSHOT.jar
pax-exam-tutorial-1.1.0-SNAPSHOT.jar
pax-runner-no-jcl-1.1.0.jar

You can surely discard a lot of these JARs. But I added the list of completeness. It's just the IDE setup. I did have one issue with running the tests in the IDEA. Between runs I needed to manually delete my $TEMP_DIR/paxexam_runner_[user] folder. It sounds like no one else is experiencing this issue and I'm using a snapshot I built myself from source. The mailing list has been pretty responsive but it's a mystery error for now.

That's the end of it, folks. I'm going to the back yard to sit in the kiddie pool with my daughter. It's hot as a mutha in my office.

Groovy + OSGi: Gradle Makes it Easy

I'm exploring the tooling around Groovy and OSGi for the September Groovy.MN meeting, where I'm presenting Groovy+OSGi Jumpstart (and then on to 2GX!).

In the end, creating OSGi bundles from Groovy classes was pretty darn easy with Gradle. The User Guide documentation wasn't exactly clear, and the current 0.7 sample project didn't do much of anything... so here's the goods while you wait for the updates to released.

A good background on using Groovy and OSGi is the Beginner's Guide to OSGi on the Desktop. It explains bundles and Activators are, and what the Import-Package and Export-Package metadata specifies. In a nutshell: an OSGi bundle is a JAR file with extra data in the MANIFEST.MF. It's dependencies are specified in the Import-Package attribute and it's exported API is specified in the Export-Package attribute. On bundle startup, a method called Activator.start() is called, and on shutdown Activator.shutdown() is called. The bundle replaces the JAR as a unit of deployment, the dependency management metadata replaces JAR Hell, and the Activator replaces "public static void main".

This post will show you a few things:

• how to implement an Activator in Groovy
• how valid bundle metadata looks
• how to create the bundle from Gradle
• how to load and execute the bundle in Eclipse Equinox
  

A Groovy Activator
This couldn't be much simpler. Implement the org.osgi.framework.BundleActivator interface in Groovy. You now have knowledge of 4% of the OSGi API (it's only 27 classes, you know).

package org.gradle

import org.osgi.framework.BundleActivator
import org.osgi.framework.BundleContext

public class GradleActivator implements BundleActivator {

 public void start(BundleContext context) {
     println "Hello from a Groovy Gradle Activator"
 }

 public void stop(BundleContext context) {
 }
}

Notice, I put this in the org.gradle package (it's important later). Now think for a moment about all the public methods on this object... can you name all 16 methods Groovy adds?.

Valid Bundle Metadata
The bundle metadata in the JAR's MANIFEST.MF file needs to specify, at a minimum, a unique SymbolicName, all the ImportPackages that the class will need to run, and all the ExportPackages that other classes will need to run it.

What does this activator need for ImportPackage? It needs to import org.gradle (itself), and it needs to import org.osgi.framework (the interface). You also need to import groovy.lang and a few other Groovy packages so that Groovy method dispatch works correctly.

What does the activator need for ExportPackage? Certainly org.gradle, so that the container can invoke the Activator. What else? This hidden dependencies are in the silent API of Groovy objects. All Groovy objects have a get/setMetaClass that operates on type groovy.lang.MetaClass. So to use the Activator you need to export groovy.lang and a few other packages, even if you don't invoke those methods!

A Gradle Generated MANIFEST.MF
Luckily, the Gradle OSGi plugin handles all of the ImportPackage and ExportPackage dependencies for you. Here is the full MANIFEST that Gradle generates:

Manifest-Version: 1.0
Ant-Version: Apache Ant 1.7.0
Created-By: 1.6.0_04 (Sun Microsystems Inc.)
Export-Package: org.gradle;uses:="groovy.lang,org.codehaus.groovy.refl
 ection,org.codehaus.groovy.runtime,org.osgi.framework,org.codehaus.gr
 oovy.runtime.callsite";version="1.0"
Bundle-Version: 1.0
Tool: Bnd-0.0.255
Bnd-LastModified: 1250128067203
Bundle-Name: Example Gradle Activator
Bundle-ManifestVersion: 2
Bundle-Activator: org.gradle.GradleActivator
Import-Package: groovy.lang;version="1.6",org.codehaus.groovy.reflecti
 on;version="1.6",org.codehaus.groovy.runtime;version="1.6",org.codeha
 us.groovy.runtime.callsite;version="1.6",org.gradle;version="1.0",org
 .osgi.framework;version="1.4"
Bundle-SymbolicName: gradle_tooling.osgi

It's a mouthful! Gotta love the wrapping rules on manifest files. Did you know the wrapping is defined not on the number of characters in a line but the number of bytes? Ugh.

Anway, as you can see, the Export-Package and Import-Package are correct. Groovy was discovered and specified. You can also see in the Tool attribute that bnd is used under the covers.

In Gradle, you use the OSGi plugin to simplify the generation of the manifest. The following build file specifies the Equinox dependency (to resolve the OSGi classes) and the Groovy dependency (to compile the Groovy). The "configure(jar.osgi)" begins the OSGi configuration. By specifying the * for import and export packages, bnd is invoked to determine the dependencies. 22 lines of build script gives you a "gradle jar" target that creates a valid bundle:

group = 'gradle_tooling'
version = '1.0'

usePlugin 'groovy'
usePlugin 'osgi'

repositories {
  mavenRepo(urls: 'http://repository.jboss.org/maven2/')
}

dependencies {
  groovy group: 'org.codehaus.groovy', name: 'groovy-all', version: '1.6.0'
  compile( 'org.eclipse:osgi:3.4.3.R34x_v20081215-1030' )
}

configure(jar.osgi) {
  version = '1.0'
  name = 'Example Gradle Activator'
  instruction 'Bundle-Activator', 'org.gradle.GradleActivator'
  instruction 'Import-Package', '*'
  instruction 'Export-Package', '*'
}

Proof Positive
It's a hello world bundle, right? So let's run it!

Invoke the OSGi container in console mode:

java -jar %osgi_jar% -console

The Gradle script will stick the OSGi jar in the .gradle cache. Mine is at $home/.gradle/cache/org.eclipse/osgi/jars/osgi-3.4.3.R34x_v20081215-1030.jar

Now just install and start the bundle:

osgi> ss

Framework is launched.

id      State       Bundle
0       ACTIVE      org.eclipse.osgi_3.4.0.v20080605-1900

osgi> install file:/d:/libs/groovy-all-1.6-beta-2.jar
Bundle id is 27

osgi> install file:/d:/libs/osgi-1.0.jar
Bundle id is 28

osgi> ss

Framework is launched.

id      State       Bundle
0       ACTIVE      org.eclipse.osgi_3.4.0.v20080605-1900
27      INSTALLED   groovy-all_1.6.0.beta-2
28      INSTALLED   gradle_tooling.osgi_1.0.0

osgi> start 28
Hello from a Groovy Gradle Activator

osgi>

Wheee! It works; hooray for Gradle, Groovy and OSGi... a trifecta of un-suck.

30 Days Locked in a Team Room

Spent the last 30 days in a conference room converted into a shared work area for me and four of my (now) closest co-workers. We had asked to be moved to a shared space but none of us actually expected it to happen. After a month of it, all I can say is:

I do not want work in a cubicle ever again for as long as I live. Ever. Period.

One dude whistles. One dude crunches mini carrots all day long. One person is negotiating a house sale on the phone. And I pop my gum incessantly (if I don't I get an overwhelming urge for tobacco products). Yet it is the best work environment I've had in years.

Today I had to write up a summary of the team's past 8 retrospectives. Almost every single action item from the retrospectives over 4 months had to do with improving communication in some form, talking to each other more frequently, or getting better feedback. Until we got the team room. After moving to a shared space, the action items had nothing to do with communication. Empirically speaking, the team room unequivocally solved our project's biggest problems.

A team room has absolutely not lead to more interruptions. Side conversations are very easy to ignore. And any question you have is usually answered within seconds. Compare that to sending someone an IM seeing if they are available and then wandering over to talk. Speaking of, it's been wonderful not having to run some stupid instant messenger program in the background all day.

Cubicles are inhumane. Solitary confinement is punishment under most systems. Closing ourselves off from human contact and than holding a meeting to discuss ways to improve collaboration is laughable if we didn't actually hold those meetings every few months.

The insanity ended for me 30 days ago. I never, ever want to go back to working in a cubicle.

IntelliJ IDEA 9 OSGi Support Explored

IntelliJ IDEA 9 (Maia) purports to support OSGi bundles... so I dug in the weekend to find out just what this means.

This blog posts provides a quick start guide for two things: 1) getting the IDEA 9 EAP to publish your library as an OSGi bundle, and 2) getting a second bundle published and running to test your library.

General Impressions
The IDEA 9 OSGi support is a port/conversion of the Osmorc plugin which was available in version 8. In general, the functionality seems stable but sparsely documented. I could find neither a discussion group nor documentation for Osmorc. A lot of the configuration was a process of trial and error. But it does all seem to work, and after investing a few hours it has started to seem straight-forward in hindsight. You'll have to judge for yourself how it compares to Eclipse (that's next on my list!). Osmorc tries to do a lot of auto-magic generation for you, and when I stepped outside the norm (like including Groovy as a test-only dependency) I was confronted with hard to fix failures. However, this may not be a fair criticism. For any serious project, you will want to write a bnd script, not simply use an IDE, and IDEA lets you use a bnd script to drive the OSGi configuration. So at points where the config wizard started to get difficult, I really should have stepped back and created a bnd script.

Getting a Bundle Published
While Eclipse seems to be focused on creating a Plugin (ie Bundle) from scratch using a wizard, IDEA is more focused on including OSGi support on one of your existing libraries using an OSGi facet. There is no "Create OSGi Project" wizard. You simply need to add an OSGi facet to one of your projects. I find this approach more reasonable. But let's not get too far before describing the basics of publishing a bundle:

1. Installing IntelliJ IDEA 9 and OSGi

• Download and install the Maia EAP
  
• Download and install an OSGi SDK (not just the framework jar). I had trouble with Eclipse Equinox, but KnopplerFish worked just fine

2. Configuring IntelliJ IDEA for OSGi
The next step is to tell IDEA about your OSGi SDK. In Settings (Ctrl+Alt+S), go to OSGi->IDE Settings->Framework Definitions and add your framework:
Now tell IDEA to use the framework you just registered. In Settings (Ctrl+Alt+S), go to OSGi->Project Settings and select your framework in the dropdown:

Done!

3. Publishing Your Library
Now let's publish your Java library as an OSGi bundle. In this example, I'm using my filter4osgi library that provides a nice API for LDAP style filters (shameless plug: I'll pay you for bug reports). This step will create the OSGi .jar file and required manifests.

• Open an existing IDEA project... the fewer dependencies the easier it will be to publish
• Add the OSGi facet to the project module. Clicking the little plus sign and picking OSGi should do the trick.
  

A note about the OSGi facet... currently, you're not able to add more than one OSGi facet to a module. For me, this forced me to create a multi-module project in order to produce several OSGi bundles from the same codebase. This seems like a considerable limitation to me but maybe my projects are structured strangely.

The Manifest Generation tab allows you to specify the exported packages for the bundle. I needed to manually add the package I wanted to export, Osmorc did not seem to discover this for me. Again, using bnd is probably the best option here.


There are plenty of other options, but they aren't as important right now. And note, I didn't add an Activator, because my bundle is just a library.

To verify that you published your library correctly, rebuild the application. The jar file should be written to disk. The manifest file with the OSGi headers is pretty simple:

Manifest-Version: 1.0
Export-Package: org.filter4osgi.builder;version="0.0.1"
Bundle-Version: 0.0.1
Tool: Bnd-0.0.337
Bnd-LastModified: 1249412068156
Bundle-Name: filter4osgi
Bundle-ManifestVersion: 2
Created-By: 1.6.0_12 (Sun Microsystems Inc.)
Import-Package: org.filter4osgi.builder;version="0.0"
Bundle-SymbolicName: filter4osgi
Include-Resource: ..\Filter4osgi

4. Running Your Bundle
Running the bundle is simple. Create a new Run/Debug Configuration using the little plus sign:
Just use the Add button to specify your bundle. You can also specify dependent bundles too.
On the Parameters tab, you'll need to specify the OSGi framework to run within:

Now just click Run, and your console output window should have something like this written to it:

Knopflerfish OSGi framework, version 4.1.7
Copyright 2003-2009 Knopflerfish. All Rights Reserved.

See http://www.knopflerfish.org for more information.
Failed to remove existing fwdir C:\Documents and Settings\hdarcy\.IntelliJIdea90\system\osmorc\runtmp1249413066328
Framework launched
Installed: file:///D:/dev/filter4osgi/out/production/Filter4osgi.jar (id#1)
Started: file:///D:/dev/filter4osgi/out/production/Filter4osgi.jar (id#1)

It worked! (Although I have no idea why there is a failure removing "fwdir", whatever that means). Alternately, you can load the jar file into any other container and interact with it that way. One feature that is missing is the ability to interact with the container from IDEA. You cannot issue the stop or uninstall commands. In fact, the only way I found to stop the container is with the big red Stop button. It would be better if you could issue text commands to the container. This doesn't seem like it would be difficult to add.

5. Testing Your Bundle with an OSGi Test Harness
There are a few test utilities for OSGi, such as Pax Exam and the Knopflerfish Junit Support, but all I wanted to do was write a bundle that imported filter4osgi and made sure the classes were available. So I simply added a new module to my project called "filter4osgiIntegrationTest". To this I added the Knopflerfish framework.jar since I needed to create an Activator. Take note, the prior example of exposing a library as a bundle required zero design or runtime dependencies on any OSGi implementation.

Without adding the OSGi facet, I created a new Java class that implemented BundleActivator. IDEA flagged me with a missing facet error and asked if I'd like to create a facet, which I did. Nice!

The Manifest Generation tab was mostly filled in, but again I needed to hand edit the MANIFEST.MF data so that it imported the package from the original filter4osgi bundle and added the root '.' to the bundle classpath (otherwise there were ClassNotFound exceptions).

Now I just implemented the activator to construct a new filter object:

public class ActivatorForTesting implements BundleActivator {
 @Override
 public void start(BundleContext context) throws Exception {
   FilterBuilder filter = or(
     and(
       eq("mailbox", "default"),
       eq("lang", "EN_US")
     ),
     and(
       eq("mailbox", "dflt"),
       eq("lang", "EN_CA")
     )
   );
   assert filter.toString().equals("(|(&(mailbox=default) (lang=EN_US)) (&(mailbox=dflt) (lang=EN_CA)))");
 }

 @Override
 public void stop(BundleContext context) throws Exception { }
}

Hey, that's not a bad API for working with LDAP filters, is it?

And now it's time to create the Run/Debug Configuration, which was probably the hardest step in all this. In the Run Config, I needed to specify that both the filter4osgi and filter4osgiIntegrationTest need to be started up, and that the filter4osgi must be started first. You do this in the Bundles tab, and ordering is done by indicating a Start Level:


Now, when I run the config, I should see that both bundles were started. And if I don't get a ClassNotFound error then I know that the filter4osgi exports lined up correctly with the filter4osgiIntegrationTest imports. Which they do:

Knopflerfish OSGi framework, version 4.1.7
Copyright 2003-2009 Knopflerfish. All Rights Reserved.

See http://www.knopflerfish.org for more information.
Failed to remove existing fwdir C:\Documents and Settings\hdarcy\.IntelliJIdea90\system\osmorc\runtmp1249414635312
Framework launched
Installed: file:///D:/dev/filter4osgi/out/production/Filter4osgi.jar (id#1)
Started: file:///D:/dev/filter4osgi/out/production/Filter4osgi.jar (id#1)
Installed: file:///D:/dev/filter4osgi/out/production/filter4osgiIntegrationTest.jar (id#2)
Started: file:///D:/dev/filter4osgi/out/production/filter4osgiIntegrationTest.jar (id#2)

Perfect!

All in all, the support was quite usable. I think starting with a bnd file would have made things easier for me, as well as if IDEA had let me interact directly with the container.Definitely looking forward to the official release!