Architecture

This article shows some architectural aspects of the MT4j framework. It is intended to help users and developers to better understand the structure and concepts behind MT4j.

Overview

The functionality of the MT4j platform architecture is divided into different layers communicating through events sent from one layer to the next. The emphasis on input layers represents the importance of a flexible input architecture.

Input Hardware Abstraction Layer

By using a hardware abstraction layer, MT4j can support various input hardware with only minimal adjustments in the input hardware abstraction layer. In this abstraction layer, the different raw input data is converted into unified input events. The only step to be taken in order to support a new type of input hardware is to extend the abstract super class of all the input sources and add the functionality specific to this type of input.
MT4j comes with a set of implemented input providers supporting mouse, keyboard and two multi-touch input protocols, including the TUIO-protocol. All of these input sources can be used synchronously and in combination without the risk of undeterministic behaviour.

Input Processing Layer

In MT4j, the input processing occurs at two different stages in the input event flow.
The first stage is the global input processing stage where a number of input processors can be registered which subsequently listen directly to the input of the various input sources. This stage is used when all input has to be processed. It also allows to modify input before it is passed up to the next layers. For example, every newly created scene in MT4j automatically registers a global input processor which checks if there is a component at the position of the input and sets that component as the target of that input event.
The second input processing stage is located at the component level. It allows processing of input that was targeted at one component only. Here, multi-touch gestures like the rotate- and scale gestures can be found, for example. These component input processors can be registered modularly with any component. Both global- and component input processors are extensible and can be customized. If the criteria for a multi-touch gesture are met, the input processor fires a gesture event carrying the information about the recognized gesture to the corresponding component which passes the event on to its gesture listeners. The action taken when a gesture event is received is determined by the attached gesture listeners which can modify the components behaviour or appearance.

Presentation Layer

MT4j provides a flexible way to create customizable and media rich user interfaces. For this purpose, MT4j contains graphical components ranging from graphical primitives to more complex user interface widgets and is inherently designed to allow development of 2D and 3D applications.

Scenes

In order to organize different aspects of an MT4j application the concept of “scenes” was introduced. Scenes encapsulate and cleanly separate the input processing and presentation of one aspect of an application from another. An example of using scenes would be a game that has a menu scene and a gameplay scene which contain different business logic and interfaces. To navigate between scenes, a scene change can be triggered.

Components

In MT4j, graphical user interfaces are based on a hierarchic component structure which allows the composition of user interface components in a tree structure often referred to as a 'scene graph'.
The base component class provides basic functionality like the composition of components and has no direct visible representation. This base class is often used as a group component. Subclasses can then add a visible representation by overriding the components drawing method. Included visible components are primitive shape components (e.g. rectangle, polygon, ellipse and line) as well as a set of more complex user interface components. More complex user interface components are often based on primitive shapes and provide functionality like image rendering with support for common image file formats, rendering of scalable vector graphics or rendering of 3D models.
When creating a custom user interface component, a lot of functionality can be reused from available components. Custom components can be built upon already available functionality (e.g. hit detection, gesture processing, positioning) by composing primitive shapes and available user interface components to more complex components.

Canvas

The root component of every scene in MT4j is the canvas component. It is a component with special functionality and acts as the link between the global input processing layer and the presentation layer. All input events pass through the canvas component which then further propagates the event to its destination. It also contains methods for checking which components are located at a specified screen position and it is responsible for recursively drawing the canvas with all its child elements.

Rendering

For rendering of MT4j components, the processing toolkit is used. Processing is an open source java toolkit aimed at the creation of data visualizations, interactions and computational art and has a very active community. It provides an easy syntax for accessing drawing and visualization functionality and contains many useful utilities. By using a rendering abstraction layer, it is possible to choose between different renderers. There are software- and hardware accelerated renderers available.

Event Propagation and Processing

All input events originate from one of the registered input sources. These input events are then passed on to the currently active scene's global input processors. Here, all input can be parsed, modified and analyzed. Global gestures that dont belong to any special component can also be detected. A scene can register a new global input processor by calling registerGlobalInputProcessor(..)
If a global input processor is registered, the Canvas of the registering scene is added as a listener to that processor automatically and recieves all events fired by the input processors. Custom listener can also be added. When recieving such an input event, the Canvas checks which target the event has and propagates the event to it by invoking the target's processInputEvent(MTInputEvent evt) method. If the event has no target, it isn't propagated further. When recieved, the target component then fires the input event to its registered input listeners. All input events a component recieves can thus be listened to by using a component's addInputListener(inputListener) method.
Component input processors, which can parse and analyze these input events can be registered with a component by calling a component's registerInputProcessor(inputProcessor) method. The component input processor will subsequently listen to all input events the component recieves. The component input processors are used in MT4j to detect most multi-touch gestures like the drag- or rotate gesture.
These input processors can fire gesture events if a special input pattern or condition is met. These gesture events are fired to the components processGestureEvent(gestureEvent) method. From there, the component distributes the gesture events further to the component's gesture listeners.
All the gesture events a component recieves can be listened to by calling the addGestureListener(gestureEvtSender, listener) method that takes the component input processor's class as the first argument and a IGestureEventListner instance as a second parameter.

To enable the functionality of an input processor on a component, the following two steps have to be taken:

1. register the input processor (e.g. DragProcessor) with the component
2. add a gesture event listener to the component listenting to the input processor's gesture events

It is assumed, that the application has at least one input source registered already.

As an example, setting up multi-touch drag functionality for a component can be achieved by following code:

component.registerInputProcessor(new DragProcessor(mtApplication));
 
component.addGestureListener(DragProcessor.class, new IGestureEventListener() {
        public boolean processGestureEvent(MTGestureEvent ge) {
                DragEvent de = (DragEvent)ge;
                de.getTargetComponent().translateGlobal(de.getTranslationVect());
                return false;
        }
});

Most components have the drag, rotate and scale processors in combination with the according listeners set up already.