While working with a customer on a migration from Ant to Maven we needed to maintain the same build performance in the converted Maven build. Some ingenious work had been done in the existing Ant tasks to provide a degree of incremental behaviour which we had to match or do better. We have had a form of incremental behaviour in Eclipse m2e for quite a while, but working on this project provided the impetus for creating a proper incremental build API. The incremental build API we created is not Maven specific but our first goal was to make this API work in Maven. Our long term goals are to have parity between the incremental behaviour in Eclipse and Maven on the command line, as well as provide an incremental build API that can easily be integrated into any tool. At very high level, an incremental process takes some inputs and configuration options and produces some outputs. Between two different runs of the same build the delta in changes to the inputs is generally small, by processing just the delta it is possible to drastically reduce the amount of work needed to produce the outputs. In order for the build to behave correctly you also need to account for stale outputs: the case where the source inputs have been deleted or modified in such a way that the they are not produced.

We looked at many existing tools, found many patterns of how inputs are used to create outputs and have come up with approach to deal with these patterns incrementally in the form a library.

Associations between inputs and outputs

The library internally deals with associated inputs to outputs with a general provides/requires model, but in practice there are two common scenarios we want to outline. The first is where a single input is used to create many outputs, an example of which might be a single Antlr grammar file used to produce a set of files. In this case each output is produced from one, and only one input. During each build we only need to process new or modified inputs. We also need to delete outputs that correspond to deleted inputs. Inputs that have not changed from the previous build do not need to be reprocessed, therefore making the build faster.

The second is where many inputs are used to produce a single output, an example of which might be the processing of a Maven plugin where the sources are scanned for implementations of the @Mojo annotation along with @Parameter annotations found within to produce a single Maven plugin.xml descriptor. In this case the single output is produced from multiple inputs and the number of inputs can change from one run to the next. The output needs to be regenerated when there are new inputs, changed inputs, and deleted inputs that contributed to the output.

There is support for a general provides/requires model that is beyond the scope of this article, but is used to implement our incremental command line Java compiler. We will talk about this in a separate article.

Configuration management

The incremental library also handles looking at the configuration that is used by a tool to generate its outputs. When the configuration changes, even if the inputs have not changed, you may need to regenerate the outputs. For example, if you change the debug flag from off to on, the compiler needs to re-compile the classes in order to add the debug information even if the source inputs have not changed.

We currently have a Maven specific implementation that provides reasonable handling of changes to Mojo configurations, but can be customized using @Incremental annotation. For example, you may have a Mojo @Parameter whose value is never going to affect the output generation and therefore you want to ignore this configuration during incremental input processing.

For those interested in this effort please join the Takari development group. We’d love your feedback on this initial release, and will be happy to try and help you integrate the library into your existing Maven plugins, and Ant tasks. We will even try to help you with Gradle plugins!

You can find the source to the library in our Github repo, and if you want to try integrating the library into your project you can find it in Maven Central with the following coordinates:

<dependency>
  <groupId>io.takari</groupId>
  <artifactId>incrementalbuild</artifactId>
  <version>0.0.1</version>
</dependency>

Remember that our incremental build support is in active development and is rapidly changing. We are happy to answer questions but we are integrating this library into our own plugins and as we find things we need to change we change the APIs if necessary. We’d really like people to take a look at our incremental support but expect some short term flux.

In our next post we’ll start providing some example of how to use the incremental build support in Maven plugins.

Comments