jebishop.blog

Multi-touch Gesture Recognition in Silverlight 3

November 5, 2009

Tags: multitouch, nui, Silverlight, touch

Silverlight 3 provides all the info you need to receive and interpret multi-touch input on Windows 7, but it takes some work to translate these events into standard gestures – like “pinching” to zoom in or out (a list of other common gestures, and some good background info, can be found on the Engineering Windows 7 blog).

Tim Heuer gave a great introduction to the basics of Silverlight 3 multi-touch, covering hardware/software requirements and how to listen to the application-wide Touch.FrameReported event to get touch information. Building on these basics, I’d like to give a short example of how you can add support for some common gestures – such as Translation, Rotation, and Scaling – to your touch-aware Silverlight 3 application on a per-element basis.

At a high level, the steps to accomplish this are:

Receive touch events at an app level.
Route touch events to the target element(s) being manipulated.
Interpret touch events on a target element as gestures.
Hook up gestures to manipulate the element.

To make things a bit simpler and more reusable, I’d like to create a static class to accomplish #1 and #2 – I’ll call it TouchProcessor.

For #3 and #4, I would also like to be able to reuse this logic – so I’ll create a GestureController class to receive events on behalf of an element, and define an ITouchElement interface that it can use to manipulate elements.

Simplified class diagram:

Delving into more detail on each of these:

1. TouchProcessor

Static class that manages registering for the global Touch.FrameReported event, and passing events on to the GestureControllers of specific touched elements. It exposes a single public property:

public static class TouchProcessor
{
    public static bool IsTouchEnabled { get; set; }
}

2. ITouchElement

An interface describing the available gestures that an element supports. In this case, we would like to support the following gestures :

Translate
- This would result from a “panning” gesture – touching an element with a single finger and dragging it around.
Rotate an element
- This would be accomplished by placing two fingers on an element and rotating them circularly in either direction.
Scale an element
- This would allow resizing an element by placing two fingers on an element and “pinching” in or out.
Bring to front
- This would cause an element to be shown overtop all other elements, and would result from tapping an element with one or more fingers, or from starting any of the above gestures.

As you can imagine, these gestures are all readily implemented for any UIElement using the RenderTransform and ZIndex properties.

public interface ITouchElement
{
    void Translate(double x, double y);
    void Rotate(double angle);
    void Scale(double scale);
    void BringToFront();
}

3. GestureController

This class is the “brains” behind gesture recognition. There will be a GestureController instance for each ITouchElement: it will track all touch input a user directs to the element, interpret it, and invoke the appropriate ITouchElement method depending on the gesture. Since a code listing is worth a thousand words, here’s a snapshot of all the members of this class:

public class GestureController
{
    private ITouchElement _element;
    private Dictionary<int, Point> _points;
    private int _primaryContactId = -1;
 
    public GestureController(ITouchElement element);
    public void TouchPointReported(TouchPoint touchPoint);
    private void InterpretSingleTouchGesture(Point oldPosition, Point newPosition);
    private void InterpretMultiTouchGesture(int Id, Point oldPosition, Point newPosition);
 
    private void TranslateElement(Point oldPosition, Point newPosition);
    private void ScaleElement(Point primaryPosition, Point oldPosition, Point newPosition);
    private void RotateElement(Point primaryPosition, Point oldPosition, Point newPosition);
    private double GetAngleDeltaBetweenPoints(Point referencePoint, Point startPoint, Point endPoint);
    private double GetDistanceBetweenPoints(Point point1, Point point2);
}

You can see it keeps track of a specific target element, a set of active touch points, and which touch point is the “primary” point (for single-touch gestures).

First, some terminology:

Touch device
- A finger.
Touch Contact
- The location of a touch device while it is touching an element on the screen.
“Primary” device
- The first touch device (finger) that hits an element. This can be reset by lifting all fingers off an element, and touching down again.
“Secondary” device
- Any touch device (finger) that hit an element after the primary one was already down. These points put the “multi” in “multi-touch”!

Now, let’s take a look at some of the methods in the GestureController class:

TouchPointReported

This is GestureController’s only public method, which is invoked by the TouchProcessor whenever a global touch event is directed at its attached element. The snippet below shows the implementation of this method:

public void TouchPointReported(TouchPoint touchPoint)
{
    switch (touchPoint.Action)
    {
        case TouchAction.Down:
            // start tracking device position
            _points.Add(touchPoint.TouchDevice.Id, new Point(touchPoint.Position.X, touchPoint.Position.Y));
 
            // if the new point is the only one down, it's the new primary point
            if (_points.Count == 1)
            {
                _primaryContactId = touchPoint.TouchDevice.Id;
            }
 
            _element.BringToFront();
            break;
        case TouchAction.Move:
            int Id = touchPoint.TouchDevice.Id;
            Point newPoint = new Point(touchPoint.Position.X, touchPoint.Position.Y);
            Point oldPoint = _points[Id];
 
            if (_points.Count == 1)
            {
                InterpretSingleTouchGesture(oldPoint, newPoint);
            }
            else
            {
                InterpretMultiTouchGesture(Id, oldPoint, newPoint);
            }
 
            _points[Id] = newPoint;
            break;
        case TouchAction.Up:
            // stop tracking device position
            _points.Remove(touchPoint.TouchDevice.Id);
            break;
    };
}

The operation here depends on which type of action was reported for the given touch point:

Down
- A new touch device (finger) was just placed on the element, so start tracking that device.
- If there aren’t any other active devices (i.e., only one finger is on the element), make it the new “primary” contact.
Move
- A touch device that’s already being tracked has moved, so determine what gesture this movement should be interpreted as. It may be a single-touch gesture (from the “primary” contact) such as a Translate, or a multi-touch gesture such as a Rotate or Scale.
Up
- A touch device that was being tracked has been lifted up, so stop tracking it. The same finger may touch down again, but it will be treated as a different device.

InterpretSingleTouchGesture, InterpretMultiTouchGesture

These methods will call the appropriate helper methods to translate, scale, or rotate an element when a touch device moves. A translation will be performed if the device was the primary contact, otherwise a scale or rotate will be performed.

private void InterpretSingleTouchGesture(Point oldPosition, Point newPosition)
{
    // only one contact is down, so translate element
    TranslateElement(oldPosition, newPosition);
}
 
private void InterpretMultiTouchGesture(int Id, Point oldPosition, Point newPosition)
{
    if (Id == _primaryContactId)
    {
        // the primary contact moved, so translate element
        TranslateElement(oldPosition, newPosition);
    }
    else
    {
        // a secondary contact moved, so scale and rotate if applicable
        ScaleElement(_points[_primaryContactId], oldPosition, newPosition);
        RotateElement(_points[_primaryContactId], oldPosition, newPosition);
    }
}

TranslateElement

This method calculates the distance between the old and new position of the primary contact on the X-Y plane, and tells the element to move accordingly:

private void TranslateElement(Point oldPosition, Point newPosition)
{
    // translation: change in primary contact location
    double xDelta = newPosition.X - oldPosition.X;
    double yDelta = newPosition.Y - oldPosition.Y;
 
    if (yDelta != 0 || xDelta != 0)
    {
        _element.Translate(xDelta, yDelta);
    }
}

ScaleElement

This method will tell an element to grow or shrink depending on the change in distance between the primary contact’s current location and the location of the secondary contact that just moved at two points in time – before the move event, and after the move event. As you can see, it uses some helper methods that we’ll get to shortly.

private void ScaleElement(Point primaryPosition, Point oldPosition, Point newPosition)
{
    // scaling: calculate change in distance from primary point
    double previousLength = GetDistanceBetweenPoints(primaryPosition, oldPosition);
    double newLength = GetDistanceBetweenPoints(primaryPosition, newPosition);
    double scale = (newLength - previousLength) / newLength;
    if (scale != 0)
    {
        _element.Scale(scale);
    }
}

RotateElement

Similar to ScaleElement, this method tells an element to rotate depending on the change in angle between the primary contact and where the secondary contact moved. It also uses a helper method to do some math.

private void RotateElement(Point primaryPosition, Point oldPosition, Point newPosition)
{
    // rotation: calculate change in angle relative to primary point       
    double angleDelta = GetAngleDeltaBetweenPoints(primaryPosition, oldPosition, newPosition);
    if (angleDelta != 0)
    {
        _element.Rotate(angleDelta);
    }
}

GetDistanceBetweenPoints, GetAngleDeltaBetweenPoints

These two helper methods take care of some geometry calculations.

GetDistanceBetweenPoints simply returns the distance between two points using the Pythagorean theorem:

GetAngleDeltaBetweenPoints returns the change in angle, in degrees, of a moving secondary contact relative to a fixed primary contact. It uses the two-argument inverse tangent function to find the angle in green shown below:

private double GetDistanceBetweenPoints(Point point1, Point point2)
{
    return Math.Sqrt(Math.Pow(point1.X - point2.X, 2) + Math.Pow(point1.Y - point2.Y, 2));
}
 
private double GetAngleDeltaBetweenPoints(Point referencePoint, Point startPoint, Point endPoint)
{
    // simple way to calculate angle's magnitude and direction
    // slightly naive implementation since it assumes the reference point remains fixed, but it's still functional
    Double angle = Math.Atan2(endPoint.Y - referencePoint.Y, endPoint.X - referencePoint.X) - Math.Atan2(startPoint.Y - referencePoint.Y, startPoint.X - referencePoint.X);
 
    // convert to degrees
    return angle * 180 / Math.PI;
}

4. ManipulableElement

This is just a UserControl type that implements ITouchElement. I added a rectangle to it, but you could use anything you like. As you can see, it’s pretty simple – the ITouchElement method implementations just operate on some render transforms and the z-index of the control.

XAML Content:

<UserControl.RenderTransform>
    <TransformGroup>
        <!-- note ordering is important here -->
        <ScaleTransform x:Name="scale" />
        <RotateTransform x:Name="rotate" />
        <TranslateTransform x:Name="translate" />
    </TransformGroup>
</UserControl.RenderTransform>
<Grid x:Name="LayoutRoot">
    <Rectangle Name="contentRectangle" Height="200" Width="200" />
</Grid>

Code:

public partial class  ManipulableElement : UserControl, ITouchElement
{
    // initialization omitted

    public void Translate(double deltaX, double deltaY)
    {
        this.translate.X += deltaX;
        this.translate.Y += deltaY;
    }

    public void Rotate(double angle)
    {
        this.rotate.Angle += angle;
    }

    public void Scale(double scale)
    {
        this.scale.ScaleX += scale;
        this.scale.ScaleY += scale;
    }

    public void BringToFront()
    {
         // small hack to bring a DraggableElement to the front
        this.SetValue(Canvas.ZIndexProperty, zIndexCounter++);
    }
}

And that’s it!

Full source code available from:

SilverlightMultitouchGestureSample.zip

Filed under:
Silverlight by jesse

5 Responses to “Multi-touch Gesture Recognition in Silverlight 3”

CedricSoubrie

November 12th, 2009 at 7:32 am

I’m a bit surprised by your implmentation.
What if both fingers moves ? It will be sometimes interpreted as a zoom/rotate (as secondary finger moves) and sometimes as a pan gesture (as primary finger moves)?

Or maybe when there is several fingers, only the last one still send events ? This would mean that in case of a zoom, only the secondary finger will send events, but this means that if the user only moves its first finger nothing will happen.
jesse

November 13th, 2009 at 12:17 pm

Hi Cedric,
A “secondary finger” can be any one after the first (primary). They’ll all send events, and since they’re just calculated relative to the primary, it’s actually fairly robust even with 3+ fingers on an element – e.g. putting 3 fingers on an element and rotating behaves similarly to only 2.

I wanted to keep this example fairly simple to show the concepts, but this could certainly be expanded to customize the behavior for other cases (e.g. multi-finger panning).

M	T	W	T	F	S	S

						1
2	3	4	5	6	7	8
9	10	11	12	13	14	15
16	17	18	19	20	21	22
23	24	25	26	27	28	29
30