After a few years of development I think Jar Jar Links is in the home stretch, up to version 1.0rc3 now.

Part of the problem was deciding how to handle manifest files in merged jars. Thankfully while I sat and did nothing the Ant team added a "filesetmanifest" attribute to the Jar task which for now should handle most needs (re-signing jars is a still a known issue).

Maven support also came up from time to time, so eventually I broke down and added a plugin. It is pretty simple but has almost all the features of the Ant task. Look for the "plugin" jar on the download page. I'm not sure if anyone has used this yet so feedback would be appreciated.

Finally, in the 11th hour I've added a new feature. Previously you could exclude certain classes from the resulting jar file using the "zap" rule. However, this was a course-grained and somewhat error-prone approach to shrinking your jar file. On some projects of mine I've avoided the use of "zap" and instead followed jarjar with ProGuard. While this can work well it is a heavyweight solution and configuring ProGuard can sometimes be a pain.

The solution is a new "keep" rule. You can have multiple keep rules and they each take a wildcard pattern which defines which classes must be retained in the final jar (the roots of the dependency tree). It then transitively computes all the dependencies and throws everything else away. Because jarjar already had a lot of dependency checking code built-in it was a relatively minor feature to add. Unlike ProGuard this only works on the class level, but for a lot of jarjar uses there will still be significant savings. For example, in the new Guice project adding a single <keep pattern="com.google.**"/> rule reduced the jar size from 500K to 376K.

To give credit where it is due, this feature was partly inspired by Torsten Curdt's minijar project (which in turn has a repackaging feature which seems to have been inspired by jarjar?). I guess jarjar and minijar are pretty similar now, except for license differences and that minijar is Maven-specific. The more the merrier :)

Update: Over the last month or two I've probably given the same in-person demo 20 times so I decided it would behoove me to make a screencast of the editor in action (Flash 8 required, sorry). It's a little rough but hopefully gets the message across.

For anyone who's curious what I've been up to the last six months, there's finally something to point to: http://beta.tonicpoint.com/. I know, it is just some screenshots, but there really is a site hidden underneath!

Most of the site was pulling together pieces of our existing technology—PowerPoint file format I/O, PNG/PDF/SWF/SVG generation, etc. The biggest new piece of code is the editor. While it's not intended to replace PowerPoint, it nonetheless should be a showcase for how rich standards-based webapps really can be (it heavily uses SVG on browsers that support it, and eventually VML on IE). There's a lot of fodder for future posts here, so now I have no excuse not to blog. For now I will just say that MochiKit does make JavaScript suck less, but suckiness is still greater than zero.

I'll also take this time to plug the Amazon Web Services. We're currently using both S3 and EC2, which we expect will make it quite easy for us to scale up as we need to. I hope companies with the infrastructure will follow Amazon's lead—a competitive marketplace for storage and computing services would be awesome. Perhaps overly optimistically we've already structured our internal interfaces to allow us to easily switch providers :)

p.s. I'll be at the Office 2.0 Conference next week, drop me a line if you want to meet up.

Over the last few months I've been investigating adding support for MS Office 2007 file formats (aka Office Open XML) to our enterprise products. Now compared to supporting the binary file formats it is almost trivial, but there is still a fair bit of code involved!

The first step is to support the underlying packaging format, which is known as the Open Packaging Conventions. It is essentially a ZIP file with some XML files which describe the relationships between the various entries, their content types, etc. After playing with it a little while it seems pretty thought out, although of course I am biased because of a scarring exposure to OLE internals. The spec is available a number of places but the version I prefer is Part 2 of the ECMA TC45 Final Draft, available here.

To manipulate OPC documents Microsoft provides the .NET System.IO.Packaging API. When I first looked around there was no Java API so I created one for internal use, patterned after the .NET version. In the meantime Julien Chable posted what looks like a complete Java-based API along with an article describing its use on the http://openxmldeveloper.org/ site. I'm not sure if the code is open-source, though, because I can't find a license listed. Another useful tool is diffopc, which shows a graphical diff between the contents of two ZIP files (it can also be used for other ZIP-based formats like ODF).

One shared drawback of Julien's and my library is that they cannot update an existing document in-place. Instead, after you have made changes, you need to write out a new file. The primary reason for this (at least in my library) is that the Java ZIP APIs do not support it. In fact it is currently the #20 most-requested enhancement (see the full list). If you feel like helping me out then adding a vote to this bug couldn't hurt :)

Ant also has some ZIP code but it is basically identical to the Java API. However, it is ASL 2.0 and could be the basis for a more full-featured library. There is also the TrueZIP project, which is very full-featured, almost to a fault—it supports a bunch of other non-ZIP formats and weighs in over 200K but still does not yet support some necessary extensions (e.g. zip64). In the meantime I've decided to just stick with the java.util.zip API and live with the limitations.

p.s. my Gmail spam folder is over 34000 now (> 1000/day), a personal record—hooray.

After reading this diatribe on the use of the final modifier, I decided to implement one of the commenters' suggestions: write a tool to strip them out. Of course, by the time I was almost finished I stumbled across Xzajo's BCEL-based implementation, which was only posted a couple of weeks ago.

My implementation has a couple of advantages:

• It has no dependencies (ASM is Jar Jar'd).
• It removes final modifiers from methods, too.

Here is all the code:

package com.sixlegs.definalizer;

import java.lang.instrument.*;
import java.security.ProtectionDomain;
import org.objectweb.asm.*;
import static org.objectweb.asm.Opcodes.ACC_FINAL;

public class Main implements ClassFileTransformer
{
  public static void premain(String args, Instrumentation inst) {
    inst.addTransformer(new Main());
  }

  public byte[] transform(ClassLoader loader, String className, 
    Class<?> clazz, ProtectionDomain protectionDomain, byte[] b) {
    ClassReader reader = new ClassReader(b);
    ClassWriter writer = new ClassWriter(true);
    ClassAdapter adapter = new Definalizer(writer, className);
    reader.accept(adapter, true);
    return writer.toByteArray();
  }

  static class Definalizer extends ClassAdapter 
  {
    public Definalizer(ClassVisitor cv, String className) {
      super(cv);
    }

    @Override public void visit(int version, int access, /* ... */) {
      access &= ~ACC_FINAL;
      cv.visit(version, access, name, signature, superName, interfaces);
    }

    @Override public MethodVisitor visitMethod(int access, String name, /* ... */) {
      access &= ~ACC_FINAL;
      return cv.visitMethod(access, name, desc, signature, exceptions);
    }
  }
}

Compile this code into a jar and specify a "Premain-Class" manifest entry:

Premain-Class: com.sixlegs.definalizer.Main

Now the jar can be specified as an argument to -javaagent. Here is an example which demonstrates using CGLIB to create a subclass at runtime:

import net.sf.cglib.proxy.*;
import java.lang.reflect.Method;

public class Test
{
  public static class FinalExample {
    @Override final public String toString()  {
      return "final method";
    }
  }
  
  public static void main(String[] args) throws Exception {
    Enhancer e = new Enhancer();
    e.setSuperclass(FinalExample.class);
    e.setCallback(new MethodInterceptor(){
      public Object intercept(Object obj, Method method,
        Object[] args, MethodProxy proxy) throws Throwable {
        System.err.println("calling " + method);
        return proxy.invokeSuper(obj, args);
      }
    });
    System.err.println("RESULT: " + (FinalExample)e.create());
  }
}

The command line I used to run it was:

CLASSPATH=.:cglib.jar java -javaagent:definalizer.jar Test

Normally this would fail because the class and method are final, but because definalizer has stripped all of the final modifiers, the method is successfully intercepted:

calling public java.lang.String Test$FinalExample.toString()

The entire source distribution can be downloaded here, or just the binary jar (use at your own risk). If someone is interested in integrating this into their own project, let me know.

Today we are releasing a free Java-based desktop application which can view Microsoft PowerPoint 97-2003 presentations. Download it here, or look at the screenshots. It uses the same underlying code as our enterprise Java libraries and should run on any platform where the 1.5 JRE is available.

OS X and Linux users should find it especially useful as Microsoft does not provide a free viewer for those platforms (the last viewer for the Mac was for Office 98 and requires OS 9). It should also start up faster and have better overall fidelity than free office suites like OpenOffice. There is no true "slide show" mode yet (we hope to add that eventually), but the slide navigator view and incremental search feature make browsing nicer.

Although fidelity is one of our top goals, there are likely a few things that are "not quite right" because our knowledge of the file format is necessarily reverse-engineered. If you find a presentation which demonstrates a bug, please visit our forums. Of course feature suggestions are welcome too!

On the technical side, there are a bunch of libraries and tools which we used in creating the viewer, besides the standard ones like Ant and Emacs :). Here are most of them:

• Batik (for linear and radial gradients)
• ImageInfo (to determine image dimensions)
• Sixlegs PNG (to decode PNG images)
• BrowserLauncher (to open browser to help page)
• JGoodies Forms/Looks (for layout and non-Mac L&F)
• Jar Jar Links (to pack all libraries into a single jar)
• ProGuard (to compress the bytecode)
• JarBundler (to build OS X app bundle)
• Launch4j (to build Windows EXE)
• NSIS (to build Windows installer)
• Amazon S3 (to host the downloads)

The code is structured so that the viewer could be launched via Java Web Start, although the experience isn't quite as nice as a desktop app. If anyone really wants to run this via Web Start, let me know.

Classpath wildcards are a long-awaited feature which are finally making their appearance in Mustang. You can read more about them here and here.

Now that the syntax is nailed down I've added the same capabilities to the latest release of the Jar Jar Links library. In particular the dependency finding feature is much easier to use now:

$ java -jar jarjar.jar --find --level=jar 'lib/*'

This will parse all of the jar files in the lib directory and print out all of the inter-jar dependencies. Note that the asterisk normally must be quoted in order to prevent expansion by the shell.

One good use I've found for the dependency code is in building WAR files. We have a number of web applications built from the same source tree, and each one depends on a slightly different set of libraries. The simple approach is to just dump every library into every WAR, but it is not too much harder to use Jar Jar to determine exactly which libraries should be included.

Because the format of the --find command to Jar Jar is not exactly what we need, we can use the programmatic API instead:

import com.tonicsystems.jarjar.*;
public class DepDump
{
    public static void main(String[] args) throws Exception {
        DepHandler handler = new AbstractDepHandler(DepHandler.LEVEL_JAR){
            public void handle(String from, String to) {
                System.out.println(new File(to).getName());
            }
        };
        new DepFind().run(args[0], args[1], handler);
    }
}

This will only print out the dependencies in the second classpath argument. For example:

$ java DepDump build/main 'lib/*' > includes.txt

Assuming all of your class files are in build/main and your libraries in lib, this will only print out the names of the libraries that your code actually uses. Redirecting the output to includes.txt is useful because in our Ant task we can use the includesfile option to only add those particular libraries to our WAR:

<zipfileset dir="lib" prefix="WEB-INF/lib" includesfile="includes.txt"/>

If you are using the JarJarTask to embed other libraries, using the DepFind API can also be useful to verify that the resulting library has zero dependencies.

I've been working on an application which can be deployed via Java Web Start. One nice think about Web Start is that it will use content negotiation with the server to download highly compressed jar files when they are available.

The documentation includes sample Servlet code to handle the negotiation, which is nice, but I didn't want to run a Servlet container just for this. Thankfully I ran across Keith Lea's blog entry Guide: How to use Pack200 for your Java Web Start applications on Apache Web Server which shows how to use Apache's own content negotiation modules and eliminates the need for any Servlets. However, it seemed like it could still be made a little simpler. In particular, I didn't like having to create a "type map" for each compressed jar. What follows is what I think is the simplest possible configuration for getting this to work on Apache 2.

Add the following to your global httpd.conf file:

AddType application/x-java-jnlp-file .jnlp
AddType application/x-java-archive .jar
<Files *.jar.gz>
  AddEncoding x-gzip .jar
  ForceType application/x-java-archive
</Files>
<Files *.jar.pack.gz>
  AddEncoding pack200-gzip .jar
  ForceType application/x-java-archive
</Files>

In each folder where you will serve jar files from, add an .htaccess file with these lines:

Options +MultiViews
MultiviewsMatch Any

That's it for configuration! Now you just have to drop the right jars into the folder with the .htaccess file. If your original jar is named foo.jar, you should put the following into the folder:

foo.jar.unpacked

original jar, just renamed

foo.jar.gz

gzipped

foo.jar.pack.gz

compressed with Pack200

For my application these three jars were 823, 698, and 234 kilobytes, respectively. Because Web Start auto-updates those bytes can really add up in saved bandwidth costs over time.

Here is a simple shell script which will take a single unpacked jar file and create the three necessary files:

#!/bin/sh
gzip -c $1 > $1.gz
pack200 $1.pack.gz $1
mv $1 $1.unpacked

Note that renaming foo.jar to foo.jar.unpacked is a critical step. The Apache MultiViews feature will disregard the compressed jars and always serve foo.jar if it actually exists. You don't have to use the .unpacked suffix, just make sure to tack on something.

I recently came across Matthias Ernst's blog entry on compiling the JSP expression language. Back in the day this was something that I spent a bit of time on myself, but the code has been languishing ever since we decided to focus on PowerPoint solutions, so Matthias has inspired me to dust it off and release it as an open source project.

This zip file includes the entire project: Ant build file, JUnit tests, and all the necessary libraries. It uses ASM 3.0 (beta) for bytecode generation, and Jar Jar Links to embed the ASM libraries when building the library Jar file. The source uses the same license as GNU Classpath (GPL with library exception).

The library needs some more documentation, but essentially it is just an implementation of the javax.servlet.jsp.el.ExpressionEvaluator interface:

ExpressionEvaluator eval = new com.tonicsystems.el.Evaluator();
Expression e = eval.parseExpression("${8 * 8}", String.class, null);
assertEquals("64", e.evaluate(null));

The main benefit of byte-compiling is of course performance. In my very simple tests the performance ranges from about 5-10x faster than the Apache Commons EL, which uses an interpreter. However there are some areas of the EL spec which cause big problems for byte-compiling. In particular, the resulting type when dereferencing variables, Maps, and Lists cannot be known until runtime.

The general solution to the unknown type issue is to compile as much of an expression as you can, and lazily compile the rest once you have enough information (i.e. when the variables have been defined). The problem with this is it is possible to design some degenerate expressions which have to be split up into many different subexpressions, all compiled separately. To help counter this, I've added an extension to the parsing interface whereby you can optionally specify a prototype for your variables. This is basically just a Map which "resembles" the variables you will actually end up using. The compiler ignores the actual values, but uses the types of the objects to generate more efficient bytecode:

Map prototypes = new HashMap();
prototypes.put("a", new Integer(0));
Expression e = eval.parseExpression("${a + 3}", String.class, null, prototypes);

Another bytecode-specific issue is that the JSP EL interfaces only provide a way to parse one expression at a time, but you really do not want to compile each expression into a new class. For this I've added a Precompiler class which can take multiple expressions and ensure that they are all compiled into a single underlying class:

Precompiler.process(new Expression[]{ e1, e2, e3 });

BTW, I recently changed the code to use a custom ClassLoader instead of injecting the classes into the parent, in order to facilitate garbage collection compiled expressions. However, this seems to have a negative effect on the runtime performance--the expressions run about twice as slow. Does the use of custom ClassLoaders affect HotSpot somehow?

Because the code is relatively old and was never completely finished, there are a few things on the TODO list:

• Support for JSP 2.1
• Use generic type information to improve the bytecode, when available
• Support for the implicit objects defined by JSP
• More tests (or get access to the official tests)
• Documentation

If people are interested in working on the code, let me know and I'll register a project somewhere like SourceForge.

I just came across a this article on CGLIB. It's the kind of thing I've meant to write for a long time, so I'm glad someone has done it. It provides a nice overview of the proxy side of CGLIB, including using a CallbackFilter and multiple interceptors.

As part of our Java PowerPoint renderer I've had to decode all of the image formats that can be embedded within PowerPoint, including the three metafile formats: WMF, EMF, and PICT. Now I could write a whole essay on the how ugly these formats are, but it wouldn't really be fair. All three are mostly just straight dumps of API method calls and in-memory records to disk. A fine choice at the time, I'm sure, but it means that to accurately render them you have to reimplement a big chunk of Windows GDI and Color QuickDraw.

On the Windows platform, GDI+ promised to make a lot of the headaches of GDI go away. First and foremost, GDI+ is much more device-independent than GDI. For example, GDI+ did away with raster operations, which are particularly difficult to deal with when converting to modern vector formats like PDF or SVG (although people have made valiant efforts). GDI+ also introduced a new metafile format, EMF+. Because EMF+ uses the new GDI+ records, it should be much nicer to work with. Some other advantages of EMF+ over EMF:

• Gradient fills are represented mathematically, instead of by hundreds of tiny polygons (such "fake" gradients are very common in EMF files, and prevent them from being anti-aliased without serious artifacts)
• Embedded images are stored as PNG or JPEG, instead of BMP
• Support for full transparency, instead of faking binary transparency using bitmasks and raster operations

Unfortunately, EMF+ is almost completely undocumented. People ask about it on the newsgroups from time to time, but this is the closest thing I found to an answer from Microsoft, three years ago. Luckily, in my searches I did find Jeremy Todd, who as it turned out had reverse-engineered a good chunk of the EMF+ records himself. There are still a number of missing pieces, some of them critical, but it should be a very good base on which to build. One of the reasons for this blog entry is to get the search engines to index his page!

What does this have to do with PowerPoint? There is another variation of EMF+, called EMF+ Dual. Dual metafiles are EMF files with EMF+ records hidden inside EMF comments. This lets applications that understand EMF render them as normal. Applications which know to look for the special EMF+ comments can extract the EMF+ records and use them to render instead. In recent version of Microsoft Office, all generated EMFs are actually EMF+ Dual metafiles, including all the charts in Excel and PowerPoint.

The upshot is that currently only Microsoft can leverage their knowledge of the EMF+ records in order to produce better looking and more compact output when converting documents to PDF. Even after transitioning to their new XML formats, image objects, including EMF, will exist as binary files within the container ZIP. Hopefully Microsoft will see fit to finally document EMF+, because otherwise I don't see how the presence of secret binary data can be reconciled with the stated goal of an open format.