Monday, August 1, 2011

Should Static Type-Checked Compilation Come to Groovy?

Please note: As always, I speak for myself alone and not as a representative for any project or company. These are my opinions and reading of an ever-changing situation.

Update: This was a pretty stupid thing to post, because everything is in flux and these are just some thoughts.

Why Should Groovy have a Static Mode?

There were some internal discussions about "Grumpy mode" among some of the Groovy committers. I had a lot of reservations about introducing static-typing or static-mode into Groovy. When I thought about the idea I come up with many unanswerable questions. Is it a good idea or not? Only time will tell. Here are some of the questions that I wish I had answers to.

What exactly do users want?

Do people want a static language with a dynamic backdoor, such as the way C#/.NET has a dynamic keyword or the way you can use Groovy++ by default with the .gpp file extension and then sprinkle in @Dynamic annotations? This is one approach, where the core of your language is statically typed but you have an option to escape into dynamic-land. Or do people want a dynamic language with a static backdoor, which Groovy will be (as far as I know) one of the few examples of. This use case is the same as adding an @Typed keyword to your Groovy code to enable static compilation, or adding an @Grumpy keyword. Or do people just want better performance from Groovy, end of story?

If you want a statically typed language without the verbosity of Java then use Scala. Yes, Scala is complex. But Groovy is just as complex. Does anyone really know Groovy without reading a few books? Yes you can get things done, but understanding metaprogramming, understanding method dispatch, and understanding all the closure options (delegates, delegate-first, trampoline) is just as complex. I am excited about Kotlin and think it looks like a great statically-typed language, but at this point it is nothing more than a slidedeck and a wiki for anyone that isn't on the IntelliJ IDEA development team. It's not available to anyone.

I honestly don't think a Grumpy mode will boost my productivity much. Will it aide in language adoption? I say probably not. In my opinion there are three large barriers to Groovy adoption in the mainstream: performance, tooling, and bugs. Not being statically typed is a connoisseurs criticism, and connoisseurs generally chase after things that are interesting rather than good (it's been said about me as well). Groovy is a language that appeals to people that just want to get something done; it's about productivity not about programming language theory. Adding static typing might please some high-end connoisseurs if it is done in a good way, but I have trouble believing it will please many real users. Search the web for "Groovy sucks". You won't find pages about dynamic type systems, you'll find performance benchmarks. In my personal experience, people I talk to about Groovy++ are asking about performance. Sure, traits are clever. Sure, the type-checking is an interesting idea. But the interest is in performance. Alex's blog posts are mostly about x-thousand transactions per second, or beating Scala performance. I think the number 1 priority of Groovy++ is performance, and type checking is a 2nd nice to have benefit.

And what about people that want a static language with a dynamic backdoor (for builder objects perhaps)? Other languages have this. C# has a "dynamic" variable, right? Boo has a dynamic variable as well, and even lowly Flex has dynamic XML support. This is useful and valuable. I would love to see an @Groovy annotation in Java, where you escape from Java into Groovy, rather than an @Grumpy annotation for Groovy. The benefit of @Groovy for Java is that tool support is the best in the world (without a doubt) and the language has a specification, is well known, and is well tested. Of course, an @Groovy annotation would only ever work with core Java language grammar changes or very ugly hacks.

Lastly, do users just want static compile-time typing for tooling support to catch silly errors as early as possible? Well yes they want that. Who wouldn't? But who is writing Groovy code without testing it? And we already do a great job of recommending suggestions when methods fail ("hashode not found. Did you mean hashcode()"). And the tools exist to do this. IntelliJ IDEA is very reliable in this regard and GroovyLint does the same thing on a limited scale. These features exist today but since they aren't in Eclipse by default then I suspect they are underused. Catching silly errors is entirely possible without the burden of static typing. In fact, some might say that the majority of CodeNarc errors fall into the silly category!

Can Groovy Afford the Complexity?

The Java world talks about a "complexity budget" where a language can only absorb so much complexity before developers leave for something simpler. The term "complexity budget" is poorly defined and often used as a scare tactic in hyperbolic arguments, but I often fear that Groovy could become the next C++ and overrun this arbitrary "complexity budget". C is a good and simple language. Java is a good and simple language. Neither C nor Java are good for all tasks, but they are both successful and capable of having system written in them. C++ started building on C and had every feature known to man added. Nowadays the criticism of C++ is that only 20% is safe to use, but no two C++ programmers can agree on which 20%. Groovy is a multi-paradigm language. You can script, you can OO, and you can FP. Already I am advising teams to find their standard. If you like FP then embrace curry, trampoline, and functions. If you don't then steer clear of those features. Adding a static mode is going to make Groovy more complex. The same code will most likely behave differently between Grumpy mode and not Gumpy. This is one more thing to think about when programming, even if we work to minimize the differences.

Looking at Groovy++, they have done some very cool things. But sometimes the basic features of Groovy++ and Groovy differ, and this cannot be allowed to happen because it is too complex. Already Groovy developers need to know Java and the JVM. I don't know how you could us Groovy outside of Grails without knowing Java. Introducing a new mode that is subtly different could be too much complexity.

Time and Budget Trade-offs

Adding static-mode seems to me like a big undertaking in terms of developer hours. Groovy++ is a good proof of concept for the idea, but the implementation won't be merged into core-groovy. The approach used by Groovy++ is to subclass ASTNodes with AST Transformations and then write out different bytecode for the node, fighting with groovyc to make sure the correct node is invoked. It is better to implement this feature as a part of the core compiler than as an AST transformation, and that is the route the team are currently taking.

It is wonderful that VMWare supports Groovy financially by giving several people jobs. But remember there are finite resources for the project. Any time spent on static-mode means there is less time spent on other features. There aren't enough hours in the day to do everything. So developing a high-performing and well-behaving @Grumpy mode means there are less resources to solve the performance problems of plain old Groovy. I already asserted that performance was a big barrier to wider adoption. Optimized ints are good for benchmarks and came out in Groovy 1.8, but I reckon most method calls are on user defined objects. How many ints are used in a Grails app? Maybe there is a way to do everything at once, but I'm a little sad to see performance improvements not come faster to the language.

My Wishlist

Here is my wishlist of what @Grumpy would do for me and how the feature should roll out.

Some social wishes...

* GEP - The GEP process for Groovy enhancements is supposed to help us formalize features and provide feedback, but it's underused. Please, let's start a GEP as soon as possible and move this big change into the public debate.

* Dev-List Discussion - The dev-list exists to give developers a forum to communicate with each other. There will always be internal conference calls and IM Chats as well, but please lets move more discussion back into the public domain.

* Research - Are there any other dynamic languages with Static backdoor that we learn from them? I don't know of one, but would love to see more examples of what works and what doesn't

Some type inference and static typing wishes...

* Closure Return Type Type-Inference - Closures should have their return type inferred. This is non trivial, and you can have closures that return the values of closure.apply(). At some point type inference breaks. Where will Groovy's fail? How far will the type inferrer go? And how will you specify a return type on a closure when it does fail?

* Closure Arity - The type checker should check the number of parameters a closure has. You should not be able to pass a one method closure to something like Collection.inject() that requires two parameters. All closures need to specify at their type declaration how many parameters it takes.

* Closure Parameter Types - The parameter types of a closure should be type checked. You shouldn't be able to pass a closure that takes a String argument to something that requires a closure that takes an Event argument.

* Closure casting to interfaces - When you cast a Closure into a Single-Method-Interface then the arity, return type, and parameter types should all be checked and verified.

* Interfaces Implemented as a Map - When you implement an interface using the Map notation [ methodName: { /* body */ } ] then the closure arity, parameter types, and return type should be checked, and the compiler should make sure all interface methods are implemented.

* curry/ncurry type issues- Functions like curry, lCurry, rCurry, and nCurry should work in a correct, statically-typed way. For example, if you curry of a 2-arg closure then you're returned a 1-arg closure. Curry a 4-arg closure and you're returned a 3-arg closure. This is a fun edge case that can't be implemented with just overloaded methods!

* Calling groovy from Java - The semantics for calling Grumpy Groovy from Java must remain sane. I'm interested to see some examples of what the APIs look like once all the new types are added. For instance, what is the Closure class hierarchy? Will there be Closure0, Closure1, Closure2 types to model the arity of closures?

* A Grumpy GDK - All of the libraries that ship with Groovy should be as Grumpy as possible. That way a Grumpy developer never has a problem just because a Groovy GDK method was not Grumpy as well. For instance, you should be able to easily make a Grumpy subclass of GroovyTestCase, so any dynamic method on GroovyTestCase should be updated to be as static as possible. Only really use dynamic features when they are necessary.

* Generic by Default - The type inference mechanism should create the correct generics for methods, similar to how F#/Ocaml creates generic methods by default. For instance, the closure: def wrap = { println "logging..."; it() } should have a generic type signature that returns the same type that the 'it' parameter returns.

* AST Transforms Can Read Types - Transforms should be easily able to read the type information out of the AST, and it should be there as early as possible in the compilation process. So the type inference should happen early. The types don't need to be checked early, but they need to be inferred early. Also, when the type of an object is unknown then an AST transformation should be able to read this. Currently, an unknown type is Object to Groovy, but there should be a difference between Object and Unknown.

* AST Transforms Can Write Types - Transforms should be able to write type information. Transforms should be able to participate fully in this process if they are to remain powerful and flexible.

* Type Verification Happens at the End - Type checking and verification should occur at the end of all the compiler phases. This enables AST transforms to actively participate in the type inferring and type specifying process. If type verification runs and fails too early then transforms that could fix the problem won't have a chance to run

So that is my wishlist, perhaps it is a connoisseur's wishlist. What is it that you want to see in Grumpy mode? What do you not want to see?

You can leave feedback here are on the Groovy dev mailing list. We'll see where this goes!


Peter Gromov said...

A note on compilation errors: writing tests is necessary but having the compiler catch your typos is much faster. In our project, the tests take 3-4 hours to execute. And they don't test everything, not the UI code, for example. Thats's why we don't use Groovy even in non-performance-critical areas. And that's why we could use the static mode there, if it provided the same level of compile-time safety.

johnrellis said...

Great article. It will be interesting to see where this goes but I am not convinced a static backdoor is a required language feature. I do think the term Grumpy should be scrapped in favour of something more boring but maybe I am just a buzzkill :)

In all seriousness though, the term Grumpy adds more complexity i.e. I need to explain to someone what Grumpy means as well as why I keep referring to GString's :) Static or Typed are self explanatory. You cited increased complexity as a possible drawback and this adds to it.

Jochen "blackdrag" Theodorou said...

I think it came out enough in the post, that Grumpy is no static compilation, it is simply a static type check. Static compilation could be a next step maybe. I am saying this just for being absolutely clear about this ;)

I would say the initial idea of Groovy++ to use annotations to say this is static, is the right way from the perspective of Groovy. For me personally a variable that suddenly changes to dynamic semantics is not visible enough.

Then there are some things I should correct I guess. We don't get an additional man for this, his hiring is independent of any static things happening in Groovy. The eclipse team is helping us a little for a short time. You have to consider that much of the code you would need for a static type checking system already exists in the eclipse plugin and much of the code we would develop would make itself useful there as well. So it is just a natural combination. We also wouldn't have needed the green light from VMware, to do everything on our own. Only for making additional use of the eclipse plugin guys resources we need an ok.

Then some points to your wishlist:
* I have an idea how to check Closure Arity and not using the groovy++ idea of Func0-Funcn or Closure0-Closuren.
* lcurry, ncurry and rcurry... lcurry and rcurry are no problem as long as the number of parameters used and their types is known. If not, this would lead to a compilation error. ncurry would be no problem for the case that we know the position, but in general we won't know that one. I guess you can count on ncurry causing either a compilation error all the time or it causing the types vanishing and thus maybe later a compilation error. I know no type system in the world that could support ncurry to 100%
* as for calling Groovy from Java... I don't intend to do any changes here actually. If we would use Closure0 and such, then they would be interfaces, not classes, thus making no difference to Javacode, unless you want to.

And @johnrellis. Grumpy is lilke the project name, for the annotation we would use something else like @StaticTyped, @Strict or something like this. Maybe we will even use @Typed like groovy++. We have not yet decided on this.

csanders said...

I've actually been waiting to come _back_ to groovy for just this reason.

Static type checking is not about performance for me, it's about catching as many bugs as possible. It's about solid, clean, reusable code that is easily testable, easily readable, and groovy makes it fun to use.

pred said...

I would love to see a direct call syntax btw like
object->call() or whatever, to allow callsite bypassing like with the AST method : setMethodTarget( ... ).

It would be making optimization work for framework developers a bit simpler :)

Martin said...

Really interesting.

When programming in Groovy, but I want static checking, I do that bit in Java.

The problem is Java's lack of (groovy) features - Closures etc.

I think that Groovy does what it does well, and that instead of adding static support to Groovy we just need a better Java.

phil swenson said...

IMO, if Groovy had been designed with static typing and type inference it would already have become the fabled "".

For 90% of the code people write in groovy, static typing would work fine. For the other 10% (builders, dynamic methods for database results, etc) a @Dynamic annotation would suffice.

r donovan said...

Great article, the game changer may be java 7/8's invoke dynamic. From the performance side we will have to see how Groovy 2's use of invoke dynamic will enhance the performance.

Eric Weimer said...

IMHO the global economic crisis has more to do with Groovy's penetration in Java shops than anything else.

I don't buy the argument that the lack of compile time syntax checking affects productivity. If tests run for hours, the problem is the test architecture. Groovy & Grails fit well into the modern TDD, automated test/continuous integration environments we use. Some Groovy complaints are likely coming from those who have yet to adopt modern best practices.

A great benefit Groovy brings is that it is more concise while being more readable. Adding static capabilities, if it makes the language more complex, is something I do not need nor want.

peter north said...

Great article. Hopefully people will find the solutions of there problems.

Poker Online

Lauraine said...

Thanks for sharing your info. I really appreciate your efforts and I will be waiting for your further write ups thanks once again.
html5 music player| html5 media player

Dagdgsd Dffbd said...

The article posted was very informative and useful
thanks for sharing..
jaring futsal | jaring golf | jaring kassa / jaring polynet | jaring pengaman proyek | jaring pengaman bangunan | jaring pengaman gedung | jaring gawang | jaring paranet / jaring tanaman | rumput sintetis / rumput futsal |
tangga darurat | jaring cargo | agen jaring | jaring outbound | jaring truk | tali tambang

Okoren Antonio said...

Doing exactly what is required and by entering into the static mode to done things up in some quick manner is an ideal way to think about paraphrasing about contexts this will indeed followed by everything what has been emphasized. paraphrase example

Mac Topan said...

Thanks for Sharing That... Sucses for You












حسام داود said...

levelشركة تسليك مجارى بالرياض
شركة تنظيف بالرياض
شركة تنظيف شقق بالرياض
شركة تنظيف منازل بالرياض
شركة تنظيف خزنات بالرياض
شركة مكافحة حشرات بالرياض
شركة رش مبيدات بالرياض
شركة تخزين اثاث بالرياض
شركة تنظيف مجالس بالرياض
شركة تنظيف فلل بالرياض