16 November, 2009

Gonzo Camp II: Writing Ruby

Thanks to Mark Briggs, the Gonzo Camp II event was well-run and very informative. The event brought together a variety of experts, practitioners and students of journalism. Mark's guidance and inspiration created a fertile environment for the development of a variety of entrepreneurial ideas for journalism.

This camp was less productive in terms of running code demonstrated than the inaugural event, probably because a large fraction of the coders who signed up didn't attend. The poor internet connectivity was also an impediment – the prototype that I developed could have had more functions if I'd been able to test faster and download more code to integrate. Still, it was surprising that I was on the only team (out of 5) to demonstrate a working prototype.

The prototype I built for our team implemented a low-cost paywall targeting community news outlets. It implemented a basic HTTP gateway. I built it by extending Mongrel and adding an entry to my local hosts file to simulate a DNS entry. The result was a paywall erected for lodinews.com that allowed viewing of the paper's home page without registration, required confirmation to view any local news article, and allowed unfettered access to AP articles.

It was disappointing that I didn't get to see how others would approach building a prototype within 3-4 hours. On the other hand, it was very interesting to learn about the ideas that were developed with the influence of students and young journalists.

Thanks to Mark's organization and the event sponsors, there were plentiful tasty refreshments, nutritious lunch and a bountiful after-party. Gonzo Camp was definitely a valuable experience.

14 September, 2009

Bahn.de is awesome!

I've appreciated the clean bahn.de website and its complete and accurate railway itinerary creation ever since I first used it in 2003. The site continues to impress with additions of comparative travel charts like this one: Bahn.de comparative energy consumption graph

It's pretty cool to see how much less impact travel by rail will be compared to a car or airline.

While I'm sure more rigorous users have some complaints, the bahn.de service seems much more helpful in getting around Europe than anything else. I must also observe there are no such comprehensive travel solutions for planning a trip in the US. Then again, the only comparable options for transportation in the US are bus and airline – train travel is only possible in certain areas.

In Europe, reservations are only available from the national carriers, so only certain itineraries can be reserved from the Bahn.de website. On the other hand, since I'm traveling with a Eurail pass, it would be necessary for me to make the reservations in person anyway. (It would be nice if the Eurail pass could develop a website for pass holders to make reservations, but I'm sure that's more easily said than done.)

01 September, 2009

Eclipse update sites for XText incubation

I've been fumbling around a bit trying to get XText 0.7.2 working in my Eclipse 3.5 installations, so I thought it might be helpful to write down what I've found.

There are 3 update sites that were needed in order to complete the update today.

  1. MWE SDK from EMFT→ http://download.eclipse.org/modeling/emft/updates/releases/
  2. The Xtext plugin from TMF→ http://download.eclipse.org/modeling/tmf/updates/
  3. The Xtext ANTLR Support plugin (0.7.2) from Itemis→ http://download.itemis.com/updates/
The last is optional, though there are warnings that the capability is limited without ANTLR. It is omitted due to the requirement for EPL in Eclipse projects; ANTLR's current license prevents it from being offered from the Eclipse.org update sites. These all appear in the list of Eclipse update sites on ekkes-corner.

The Itemis update site provides a convenient collection of all the related modeling plugins. I turns out that my difficulty resulted from the fact that I didn't realize p2 (Eclipse Provisioning Platform) wasn't loading the current contents of the update site, so I started to search for update sites that had the 0.7.2 release of the plugins. What I learned is that I can get the current contents of the update sites by opening the Install/Update Preferences and invoking 'Test Connection' on each. Perhaps there's a better way...

Hopefully not too many others will encounter the difficulty I had finding the right versions, particularly for MWE.

01 August, 2009

Groovy Eclipse Plugin alpha zero-day review

In a multi-language project, Eclipse is a crucial tool for a number of reasons that might be apparent to readers of other posts here. One of the languages that is important in this particular project is Groovy. Until the end of July 2009, the Groovy Eclipse plugin was a rather cursory integration that was generally effective in getting Groovy projects to build together with other Eclipse projects, but didn't provide a full set of IDE capability for Groovy.

Great news arrived as July ended: the alpha for V2 of the Groovy eclipse plugin was published for Eclipse 3.5. (The website is currently only referring to Eclipse 3.4.2 support, but there's also a Groovy Eclipse update site for Eclipse 3.5.) After a few hours of exploration, a few bugs and areas needing polish are apparent, but these are so far no more of an impediment than the issues encountered with the old dev-release plugin for Groovy 1.6 (the previous Groovy 1.5 plugin was a bit more useable than 1.6, but not so much as to prevent migration to the newer version).

The basic functions at this stage of the V2 development will be very relevant to this project: the integration of Java with core Groovy is exactly the level of capability at the foundation of the plug-in. Grails developers may not find as much relevance until the plug-in gets more features. Pure Groovy developers might have a more robust IDE than Eclipse. But the support for multiple languages – C, Java & Groovy in this case – makes Eclipse a uniquely comprehensive and extensible solution.

This new plugin offers many more IDE features than previously available, though they remain more basic that the features in Eclipse Java (JDT) environment.

  • The outline view is much improved.
  • Performance is snappy – no observable delays while editing.
  • Compile errors handled properly – previously, java-level errors like missing implementation of abstract method would only appear after a save. Tests would launch and run against the old code, causing consternation if one didn't watch the Problems view.

The biggest downside of installing this plugin so far is the appearance of warnings when using Java generic types in Groovy. While adding the type specifiers to Map and Collection might not be a bad thing, there seem to be a few problems – with the result one may end up specifying <Object> where a more specific type would be accurate and correct in Java. A few editor glitches appear sporadically, though they're not yet pervasive enough to warrant a bug report.

This is an important milestone for Eclipse & Groovy. The team at SpringSource, particularly Andy Clement and Andrew Eisenberg, have done a great job with this alpha release of a promising V2 product!

21 July, 2009

Xtext Linking one step at a time

The Eclipse environment with Xtext provides an astoundingly powerful toolset. The EMF/TMF folks are doing a great job providing good capabilities with straightforward means of tailoring and extension. That said, there's a lot to the environment that can make some tasks appear more daunting than need be.

After reading Sven Efftinge's post on an experimental Xtext feature that automates linking among models in a project, I was definitely inclined to make use of it. On the other hand, it's a solution much bigger than the problem that I currently face: linking to references within the same model (program, actually) while assuming that references without a local definition are external.

My grammar allows references to imported constructs in a manner similar to Java's implied import of all class names that reside in the same package. The approach in Sven's article on integrating with the EMF index will ultimately be implemented if time allows in this project. Presently, I don't want the parser to actually load all the models partly because the main goal is to analyze individual models and the actual content of imported models is extraneous.

The simple way keep errors from being flagged where implicit imports can't be resolved is to extend the DefaultLinkingService. To start with, the ILinkingService binding has be be replaced by adding an override in the LanguageRuntimeModule class in the main parser project's directory:

 public Class<? extends ILinkingService> bindILinkingService() {
  return PdlLinkingService.class;

The custom linking service is a minimal extension of DefaultLinkingService. Thanks to Sebastian for guidance on optimizing the integration to Xtext. Knut's pointers for dealing with the ECore resources was crucial to implementing a proper solution. Here's the complete class (I'm sure there's still room for improvement):

 * Provide the linking semantics for PDL. The references to 'imported' table
 * definitions are stubbed out by this class until the Indexer is implemented.
 * @author John Bito
public class PdlLinkingService extends DefaultLinkingService {

 private PdlFactory factoryInstance = null;
  * Keep stubs so that new ones aren't created for each linking pass.
 private final Map<String, PhysicalFileName> stubbedRefs;
 private Resource stubsResource = null;

 public PdlLinkingService() {
  stubbedRefs = new Hashtable<String, PhysicalFileName>();

  * Retrieve the factory for model elements
  * @return the factory for the model defined by the language
 private PdlFactory getFactory() {
  if (null == factoryInstance)
   factoryInstance = PdlPackage.eINSTANCE.getPdlFactory();
  return factoryInstance;

  * Use a temporary 'child' resource to hold created stubs. The real resource
  * URI is used to generate a 'temporary' resource to be the container for
  * stub EObjects.
  * @param source
  *            the real resource that is being parsed
  * @return the cached reference to a resource named by the real resource
  *         with the added extension 'xmi'
 private Resource makeResource(Resource source) {
  if (null != stubsResource)
   return stubsResource;
  URI stubURI = source.getURI();
  stubURI = stubURI.appendFileExtension("xmi");
  stubsResource = source.getResourceSet().getResource(stubURI, false);
  if (null == stubsResource)
   // TODO find out if this should be cleaned up so as not to clutter
   // the project.
  return stubsResource;

  * Override default in order to supply a stub object. If the default
  * implementation isn't able to resolve the link, assume it to be a local
  * resource.
  * @param context
  *            the model element containing the reference
  * @param ref
  *            the reference defining the type that must be resolved
  * @param node
  *            the parse tree node containing the text of the reference (ID)
  * @return the default implementation's return if non-empty or else an
  *         internally-generated PhysicalFileName
  * @throws IllegalNodeException
  *             if detected by the default implementation
  * @see org.eclipse.xtext.linking.impl.DefaultLinkingService#getLinkedObjects(org.eclipse.emf.ecore.EObject,
  *      org.eclipse.emf.ecore.EReference,
  *      org.eclipse.xtext.parsetree.AbstractNode)
 public List<EObject> getLinkedObjects(EObject context, EReference ref,
   AbstractNode node) throws IllegalNodeException {
  List<EObject> result = super.getLinkedObjects(context, ref, node);
  // If the default implementation resolved the link, return it
  if (null != result && !result.isEmpty())
   return result;
  // Is this a reference to be stubbed?
  if (PdlPackage.Literals.FILE_NAME
    .isSuperTypeOf(ref.getEReferenceType())) {
   // Get the stub's name from the text of the parse tree node.
   String name = getCrossRefNodeAsString(node);
   FileName stub = stubbedRefs.get(name);
   if (null == stub) {
    // Create the model element instance using the factory
    stub = getFactory().createPhysicalFileName();
    // Attach the stub to the resource that's being parsed
   result = Collections.singletonList((EObject) stub);
  return result;

Of course, this is a stop-gap that's not appropriate for complete language processing, but it's illustrative of one simple way semantics may be adjusted within the Xtext framework. There are two likely problems making this implementation sub-optimal:

  • The 'temporary' resource lifetime isn't managed, so the local cache of EObject references can be out of sync. Intuitively, one would expect the resource to exist longer than the LinkingService, so it will be surprising to find the LinkingService using a reference to an EObject that's not stored in the resource.
  • Since the code doesn't try to find EObject instances that are already in the resource, there can be a memory leak if the LinkingService is created multiple times during a session. It may make sense to use the delete method on the Resource, but it looks a bit dangerous and I don't have the wherewithal to test that at the moment.
I searched a bit for examples of code managing ECore resources; my search terms weren't very fruitful. Pointers would be greatly appreciated.

11 July, 2009

Embedding Java in a C language application

The Java Native Interface is a powerful API that allows Java to call native libraries and also allows native executables to invoke Java (or any JVM-based) classes.

An important addition to the Java Native Interface in Java 1.4 was support for java.nio.ByteBuffer parameters. This allows the native code (C in my case) to efficiently share data with classes running in the JVM. Often, when C code is invoking Java methods, it has to create objects, particularly Strings, to pass to the methods. While this provides some safety to the Java side, it has some performance implications.

With Java 1.4, we have the option to pass data across the JNI in a single buffer shared by the C and Java code referenced by a java.nio.ByteBuffer. The use of this technique comes with many caveats, particularly if the application runs multiple threads. The ideal application for this approach is when a fixed-size message with an unvarying or self-defining format is passed across the interface. The performance advantage comes because both the C and Java code can change the data, so the ByteBuffer object only has to be created once. The C code can change the data in the buffer and pass the same ByteBuffer object to multiple method invocations and then make use of changes the Java code makes to the buffer contents.

JNIEnv *jni = get_jniEnv();
jclass cls = getClassRef();
jmethodID getdata_mid = getMethodID(DATA_LOADER);
static jobject jbuff = NULL;
static char *databuff = NULL;

if (!databuff)
  if (databuff = malloc(cbuff))
    memset(databuff, '\0', cbuff);

if (jni && !jbuff && databuff)
  jbuff = (*jni)->NewDirectByteBuffer(jni, databuff, cbuff);
if (!(jni && cls && getdata_mid && jbuff))
  //quit unless all references are available
  return NULL;
(*jni)->CallStaticVoidMethod(jni, cls, getdata_mid, jbuff);
if (check_exception())
  return NULL;  
return databuff;
The code above assumes that all the work to initialize the JVM, load the class and lookup the method is handled elsewhere. The caller can examine and change the data retrieved by the Java code. It can then modify the contents of the buffer returned and call the function again; then Java code can use the changes made in buffer between the calls. This saves object creation and destruction as well as the time copying data from one buffer to another.

On the Java side, there are some limitations on the use of the ByteBuffer. It's not backed by an Array object, so some of the operations declared by the abstract class are unsupported when the object is created by the C code.

JRE Internal Error – "exception happened outside interpreter, nmethods and vtable stubs (1)"?!

When calling Java via JNI, there's no checking of the method parameters. As a result, if your code passes parameters that don't match the method signature in a Call...Method JNI call, the JVM will likely panic and terminate with a message like "exception happened outside interpreter, nmethods and vtable stubs (1)" if you're lucky (and your JVM is at least 1.6). If you're really lucky, the JVM will write an error log that includes a stack trace pointing to the C code that made the JNI call.

Unfortunately, the 'Internal Error' reported by the JVM is really a generic panic message, so it's also likely to result from more insidious problems like your code writing into the Java heap. In that case you could try valgrind or similar tool to track down stray memory usage, but I can't say whether it'll work when calling into the JNI.

07 July, 2009

Adding function to Xtext-generated plugins

The Xtext parser generator is generating much more than a parser—otherwise ANTLR by itself would be a simpler solution. As noted in my post on migrating the Xtext grammar, just by deploying the plugins generated by Xtext, a basic outline is provided while editing the defined language.

For my analyzer, I have to add checks that flag errors for programs that contain multiple definitions for certain types of structures. The legacy language allows a single name to be associated with multiple, different structure definitions as the program executes, but part of the upgrade will cause this to be illegal, so it's the job of the static analyzer (parser) to detect this case so the programmers don't have to search out and test each program manually.

The Xtext manual section on Validation includes Custom Validation describes the classes involved in implementing validations for the language. In order to get the Xtext customization framework for validation, add

<fragment class="org.eclipse.xtext.generator.validation.JavaValidatorFragment"/>
to the GenerateLanguageName.mwe file in the parser project. Unfortunately, the Java-based checks are not so easy to write as the oAW Check language. One of the nice things about oAW was the content assist—the editor for Check knows the AST model and makes it easy to write checks that work. There's another fragment to add to the MWE file in order to get the Check capability:
<fragment class="org.eclipse.xtext.generator.validation.CheckFragment"/>
Since the TMF Xtext plugins don't seem to include a Check editor with content assist, it's actually more convenient to write the validations in Java. Referring to the LanguageName.ecore file in the visual editor (should open when you double-click the ecore file) makes it pretty easy to see the types and their features when working on validations.

I just needed to add two validations (so far). The Java is pretty small, but it took me a little while to get comfortable with the CST classes.

 * Issue warning if the View object is using the dynamic resolution syntax
 * @param view object to be validated
public void checkViewNotDynamic(View view) {
  if (null != view.getDynamic())
    warning("View defined with dynamic target", PdlPackage.VIEW__DYNAMIC );
 * Issue error if there there exist multiple Join objects identified by the same name
 * @param join one of the Join objects in the current ProcessDefinition 
public void checkJoinUnique(final Join join) {
 final String name = join.getName(); // This name must not be used to identify any other Join object
  // Now loop on all the other elements that are in the ProcessDefintion
  for (EObject sibling : ((ProcessDefinition) join.eContainer()).getCommands()) {
    // Check for another object instance that's a Join with the same name as the one being validated
    if (join != sibling && sibling instanceof Join && name.equals(((Join) sibling).getName()))
      error("duplicate join '" + name + "'", PdlPackage.JOIN__NAME);
These two validators are picked up by the Xtext framework and invoked by the editor and the generator. The warnings and errors show up in the editor as well as the Package Explorer, Problems and other views.

With the parser doing the right thing, I copied the Xpand template that I wrote under oAW into the generator project as templates/Template.xpt. The workflow/LanguageNameGenerator.mwe had a problem—it got a NoClassDefFoundError because it specified a class name in the register element with an initial lowercase letter. I don't know if that was because of something I did or a bug in the project wizard. Changing the MWE file was all that was needed to test the template. It's the same as oAW, which is to say it's a convenient way to traverse the CST and emit code.

«DEFINE main FOR ProcessDefinition»
  «FILE "test"»
    «EXPAND viewDef FOREACH commands.typeSelect(View)»
    «EXPAND joinDef FOREACH commands.typeSelect(Join)»

//In the output file, there will be one line for each instance of Join
//Within the tempate, the properties of the instance are accessed like
//local variables.
«DEFINE joinDef FOR Join »
// Pay no attention to the man behind the curtain named «name»

«DEFINE viewDef FOR View »
 «IF null!=name && null!=source -» «REM» Generate mappings for statically-defined views «ENDREM»
 views["«name.toUpperCase()»"] = joins.createView("«source.toUpperCase()»");
 «ENDIF -»
You'll notice that the template language uses guillemets to enclose the executable instructions. To make sure that you read all of the documentation, the authors only tell you that the characters are bound to CTRL-< and CTRL-> at the end. The Xpand editor was pretty nice to use in oAW. I haven't tried it yet in TMF, partly because it wasn't installed. I'm not quite sure how I got the runtime plugin for Xpand installed—perhaps p2 resolved a dependency. I'm pretty sure that once I install the Xpand UI from the M2T Xpand update site, I'll get the content assist that I used when I was originally developing the template.

The Software Life: Antlr Frustrations

Andrew McKinlay's post The Software Life: Antlr Frustrations on "no start rule" warnings gives a succinct explanation of a problem that can be somewhat evasive. The ANTLR warning "no start rule (no rule can obviously be followed by EOF)" may come and go as other issues with the grammar are resolved. One answer is to always start the grammar with a rule like:

prog : expr ;
The prog rule doesn't change anything about the language that can be parsed. It can be used as an entry point by the code calling the parser, like an interface that can be stable even as the grammar changes underneath.

03 July, 2009

Migrating from oAW XText to Eclipse TMF XText

Part of my project is analyzing the structure of programs processed by the interpreter that I'm working to enhance. Xtext and the related tools from oAW allowed me to define a subset of the language for analysis and code generation. As there's a need for more function (and hopefully more speed), it's great that a new Xtext is available for Eclipse Galileo.

The reference for Xtext includes a section on changes from oAW to TMF that's quite informative, but doesn't include any advice on how to change the existing Eclipse projects. That's in the Eclipsepedia wiki section on Xtext Migration.

The guide is a bit ominous-sounding when it talks about specifying the language name in the Xtext project wizard. Since the previous name for my package wasn't very good, and there is very little extension work in the previous project—the main purpose is the generator—the approach for the moment is to copy the text into the shells created by the project wizard. The wizard doesn't give much advice about the naming, but the editor complains if the last element of the language name is lowercase.

In the old oAW system, the plugin added a context menu named 'generate Xtext artifacts'. This doesn't appear anymore. Since I'm building my eclipse configuration from the 'Classic SDK', it's necessary to install the Modeling Workflow Engine (MWE) separately. The MWE is required to run the workflow GenerateLanguageName.mwe in the parser project.

As the documentation says, unlike oAW, TMF Xtext doesn't enable parser backtracking by default. I was able to restructure the grammar rules to get rid of many of the warning(200)Decision can match input such as "'whatever'" using multiple alternatives messages. (These aren't really warnings, since the resulting parser is ignoring portions of the grammar that I'm specifying). In the end, I found that there is a construct in the legacy language accepted by its recursive interpreter that's indeterminate. So I modified the Xtext MWE file as recommended by Sebastian Zarnekow to replace AntlrDelegatingFragment with XtextAntlrGeneratorFragment and DelegatingGeneratorFragment with XtextAntlrUiGeneratorFragment adding the child <options backtrack="true"/> element to each (don't remove the JavaBasedContentAssistFragment). I wonder if Xtext provides (or will provide) a way to enable backtracking for a subset of the grammar rules.

Once the Grammar was compiling again, I was able to check it out quickly by following the steps suggested in the Getting Started with Xtext post by Peter Friese. Invoking 'Run As → Eclipse Application' from the context menu of the parser project brings up a new IDE in a separate, initially empty, workspace. From there, create a new project and within that a new file with the extension named in the Xtext Wizard. Copying an existing program populated the outline view—a vast improvement over oAW which ate all the heap so the outline had to be disabled. One caveat: with the generated plugin configured, the editor is sure to fail with an NPE if you open a file named with the extension associated with your editor outside of a project (using File → Open).

Now, on to working with linkages in the AST!

02 July, 2009

Jevopi's Developer Blog: User Report: Migrate from oAW Xtext to TMF Xtext

Jevopi's Developer Blog: User Report: Migrate from oAW Xtext to TMF Xtext

Just starting to work on the conversion of the Xtext grammar and code generator to the new technology developed by TMF. My case is a bit different as the language definition is for an existing general-purpose programming language. The current scope of the processing is limited to analyzing and extracting specific constructs in the language.

It's nice that Javopi provided such a thorough explanation of the process required to migrate that language processor from oAW XText to the Eclipse TMF XText. I'll be working on some additional details like analyzing cross references and code generation.

More posts to come.

22 June, 2009

Unit test framework for legacy C application code

Adding a bunch of function to a big application (a 4GL) that was developed long ago and over the course of 10 years (1987 - 1997) is a risky business. Particularly when the only extant testing code is written in the 4GL – equivalent to only being able to test an HTML browser by getting it to process HTML.

I've certainly learned a lot since I took on my first product development job. One of the practices I'm still learning is good unit testing. I've learned to think in terms of the JUnit unit testing framework. Such a framework is important to focus the testing code on testing instead of housekeeping and to define standard reports. There are several unit testing frameworks for C listed on OpenSourceTesting.org.

The following list contains the subset of the test frameworks that I've considered for my project. As noted below, I've picked up the Cgreen framework with good results.

This framework is in active use, but it's primarily used for C++ applications. The API looks simple enough that it could be reasonably used to test C functions.
C Unit Test System
Defunct project – there's a note on the sourceforge project forum from 2003 asking if there are any users.
Unit testing strictly for C with a facility for mocking functions.
C Unit
Abandoned project. No response to inquiries or code updates since 2006.
A very simple implementation of unit test framework. Some activity on the project. No documentation found.
Abandoned project. No activity since 0.9 in 2006
Simple C Test
Abandoned project. No activity since 2005. Some documentation is provided.
Test soon
Abandoned project. No activity since 2007. Seems to focus on flexibility to structure tests and not so much on enabling easy authoring.
TUT test framework
Interesting framework in C++ compiled into each test suite. Not clear how failures should be detected, not designed to automatically recover from faults (SEGV) so further test cases can run.
Abandoned project. No activity since 2006.
Unity & CMock
Nice looking unit test and mock tools for C targeting embedded systems developers. Cmock requires Ruby and Rake.

The other tools listed were apparently only for C++ or objective C.

After some examination and experimentation, the Cgreen package seems to be the best fit. The mock function capability is a useful stand-in for the mock objects one would use for an object-oriented environment.

Updating the logging code with log4c

Having been delighted at my introduction to log4j in 2001, I was anxious to find a flexible logging and tracing framework suitable for introduction in the C program I'm updating. All the Google searches I tried produced results that pointed at Java, Python or PHP tools. I'd begun to despair and was thinking about putting something together on my own. It was Wikipedia's Log4J that finally pointed me to the Log4c logging and tracing framework.

I'm still working on getting it fully integrated, but the Log4c download was easy to compile on Sun's Solaris 10 (binaries for Linux are provided). One thing to note is that the CVS repository only has sources for automake, so the installation instructions are a couple steps removed.

It's not so easy to find documentation, though some effort seems to have gone into preparing some. The links for documentation go to a SourceForge site that yields an entry saying 1.0.0 with no content. I was eventually able to find a good guide to the use of Log4C on slideshare.

I haven't tried all the options, yet, but Log4C provides an elegant API that looks like it will be quite efficient. It will allow logging and tracing calls to be added with little impact on the application. It also promises to support generating syslog entries and rolling log files.


Copyright 2009-2010 John Bito. Creative Commons License
This work is licensed under a Creative Commons Attribution-NoDerivs 3.0 Unported License.