So I’m extending my degree’s project and about to author a paper – or at least try to.

To increase the confusion that little bit further, here’s the provisional title:

Identifying a remote input sensor that best yields a ‘more’ natural/intuitive syntax or symbology while optimising character recognition

Damn that needs to be shortened and/or made more concise or something…

My university project involved examining the input sensors on the Nintendo Wii Remote, the infra-red sensor and the accelerometer.

So, the gist of the paper is essentially to carry on this comparison, to find a possible link between the co-ordinate data from the infra-red sensor and the acceleration data from the accelerometer.  Of course, I may not be able to draw a comparison so I may leave this idea to persue another avenue of exploration…

Step one is to make an application that will simultaneously record both accelerometer and infra-red data from the Wii Remote then I can start capturing (meaningful) data…

Wish me luck!

“One of the next tasks in this iteration of the gesture application is to investigate a reliable method of using the coordinate data of more than one IR blob. Not only would this method increase the sensitive area of the Wiimote, it would also make the motion much smoother (similar to my averaging method to smooth out the accelerometer readings).”

Well, I’ve managed to do this now – as you will have seen from the video in the ‘intro’ post of the same title.

First, view the video below that shows the latest iteration of the application

videohere

For the purposes of development, I’ve decided to show the positioning of the infrared ‘blobs’ that the Wii Remote’s camera tracks, along with their index numbers. These index numbers are assigned by the Wii Remote itself, along with the calculations to determine the position of each indexed blob. This in itself is actually quite interesting (believe it or not!) and is due to a limitation of Bluetooth:

“…the camera is a high resolution camera and works incredibly fast (on 100 Hz). This generates a lot of data which has to be analyzed. It is way too much to send by Bluetooth to the Wii, for Bluetooth has a maximum capacity of 1 Mbit/s. Assumed that one pixels takes one byte, sending 100 images of 1024×768 pixels take 630 Mbit/s, which is way too much for Bluetooth to handle.”

Smit, J.B.. (2008). Affordable Head Tracking. University of Groningen. (chapter 2), p17.

Consider that figure of 630 Mbit/s when you can have up to 4 Wii Remotes connected to a PC (or indeed Wii) at once – that’s an awful lot of bandwidth!

Instead of very long explanations, I’ll reel off a list of features that the application now has, associated with infrared. (Still probably long explanations!)

1. Can use 1 to 4 infrared dots to determine the position of the Wii Remote

2. When 2 dots are detected (this is where my Bluetack comes out again to cover up all but 2 LEDs) then the distance between them in pixels (relative to the resolution of the application) is calculated. From this, it’s possible to track backward and forward motion and make an approximate representation for the 3D model’s movement. This is a significant step to adding the ability to recognise gestures in three-dimensions

I’ve limited it to using only 2 dots at the moment because then it’s a simple Pythagoras calculation to get the distance between the dots. If the Wii Remote is detecting 4 blobs then it’s a question of working out the greatest distance between all 4 points. This is potentially a lot of calculations that could slow the program down.

Aside from that, if the remote is really close to a cluster of LEDs at one end of the sensor-bar then it it would appear that the remote is far away due to the small distance between the LEDs. Therefore, the LEDs must be at either end of the sensor-bar.

3. If there aren’t any IR sources detected by the Wii Remote then the 3D model shader changes to represent it as ‘off-line’. The application stops registering movement rather than registering lots of strange directions. When the remote is moved out of range then the remote freezes in it’s last position. When the remote is moved back into range then the movement from the last position continues.

Next up will be the explanation of the Levenshtein algorithm!

(higher quality available here)

After discovering the limitations of the Wiimote’s accelerometer I decided to delve into the possibilities of the other sensor in the Wii Remote – it’s camera. For those who don’t know, that sensor bar sitting on top of the television isn’t actually sensing anything at all. It’s merely emitting a number of infra-red sources. The Wii knows where the player is pointing because the camera in the Wiimote is tracking the position of the infra-red sources. In fact, the camera in the Wiimote is capable of tracking 4 infra-red points (known as ‘blobs’) at 100Hz (100 times per second).

The Director Xtra I’m using for the purposes of my project supplies the coordinates (X and Y values between 0 and 1) of each blob in a handy array. So, thankfully, the potentially fiddly process of getting to the data has been dealt with – it’s just a question of interpreting the data to get something meaningful from it.

One blob to rule them all…

Armed with my Wiimote, el-cheapo battery powered sensor bar and a few lumps of Bluetack I set to work.

Firstly, my sensor bar isn’t as good as the actual Nintendo one, but then it wouldn’t be as it cost a whole $6. It has 6 infra-red LEDs, 3 at each end of the bar and arranged in such a way that they ‘fan out’ to create a spread of infra-red – the theory being that the Wii console will appear more sensitive or reactive. The genuine Nintendo sensor-bar has a total of 10 LEDs in it making it far superior. For the purpose of the program I’m writing, I only need one (there are downfalls to this approach that I’ll explain shortly) so I covered up the rest of the LEDs with Bluetack!

I’ve adapted the program I wrote for the accelerometers to use the readings from the Wiimote’s camera instead.

The 3D model of the Wiimote is pointing forwards and doesn’t rotate with the movement of the real Wiimote. Obviously there will be a certain amount of rotation when the real-life Wiimote is moved, however one has to remember that the Wiimote’s camera must be pointed in the direction of the infra-red. This is so the application can recognise the movement. If the application relied on the accelerometer for movement as well, there would have to be some consideration taken for the occasions that the Wiimote’s camera would be out of view of the infra-red (i.e. when pointed straight up or down).

This is where the downfall of using one infra-red LED for positioning becomes apparent. LEDs in general do not have a very wide emitting angle. It’s very unlikely that you would see any light – or in this case, detect any infra-red – if looking at the LED from the side. Once the Wiimote’s camera has moved out of effective viewing range of the LED then the readings either become unreliable or non-existent (in this case the Wiimote returns a value of -1).

Accelerometer & IR side by side…

This version of the application incorporates a slightly different method of movement recognition. The versions of the application that rely on accelerometer data as a means of control use – for want of a better term – a state logic. The Wiimote has to be swung to a certain position for it to register as a different movement. For example, if the user swings the Wiimote to the right, the application will register that the Wiimote is in the ‘right’ position. Now, if the user swings the Wiimote to the left, the application won’t register any leftwards motion until the accelerometer is reading that the Wiimote is in the ‘left’ position. It is better to think of the application as recognising a series of pre-set positions that the Wiimote can be held in rather than detecting the direction of the motions themselves.

The version of the application relying on IR dot positioning works differently. There are no pre-set positions that the user can hold the Wiimote. This time, the application senses the direction of the motion. For example, the Wiimote is motionless in the user’s hand – the 3D model on the screen is also stationary. The user moves the Wiimote up – as soon as this happens, the application moves the 3D model of the Wiimote up the screen and registers this as an upward movement.

This method is much more effective and natural-feeling than the previous applications using the accelerometer data. Whenever the Wiimote is stationary this registers as an ‘idle’ state – unlike the accelerometer application where ‘idle’ meant holding the Wiimote in a specified position.

The method also makes the ‘game’ I created seem a bit easier and less frantic. When the game demanded an ‘up-down’ motion, the accelerometer-based version would see the player having to swing the Wiimote far enough to get it into the correct position. It’s confusing for the game-player who is probably wondering why it took so long for their motions to be recognised. Now just the slightest movement in the correct direction would register as a movement in a certain direction – I suppose some may argue that this takes the challenge out of the game, but it opens the door for more complex gestures that the player would not have had the time (or patience) to make.

Another advantage to this method of sensing is that it’s easier to perceive the motion one’s making when gesturing at a 2D surface such as a television or computer monitor…

As an example scenario; one of the gestures the application may recognise is a 5 pointed star. You are given a whiteboard pen. There are two whiteboards in the room. One is mounted on the wall and the other is mounted on the ceiling. You are told that you have to look in front of you while drawing, so no neck tilting! First you draw the star on the whiteboard mounted on the wall – you can see the board and the shape you’re drawing so it’s really easy. Now, when it comes to draw the star on the ceiling whiteboard it’s not as easy because you can’t really see what you’re doing and the star looks distorted.

OK, that may be quite a long way of trying to explain the benefits of the IR dot method, however it nicely highlights its benefits. In programming terms, it makes the process of designing gestures easier and enables more complex methods of determining gestures to be integrated into the application – such as the Levenshtien Distance algorithm.

Next steps…

One of the next tasks in this iteration of the gesture application is to investigate a reliable method of using the coordinate data of more than one IR blob. Not only would this method increase the sensitive area of the Wiimote, it would also make the motion much smoother (similar to my averaging method to smooth out the accelerometer readings).

Returning to the accelerometer-based version of the application. I may try to alter the code to sense change in direction rather than finite states. The accelerometer data was too ‘jittery’ the last time I tried it, however, that was before I started using averaged data over time rather than real-time values. It might be possible! However, bear in mind that the problems highlighted in my whiteboard scenario earlier will still be an issue.

Once the aspect of getting the user’s input has been dealt with, the various methods of checking for valid gestures will be the next priority. In terms of my project proposal, the investigation into forms of gesture alphabets shall begin. This will represent another iteration in the application, where we move away from the gaming aspect and more toward the place that gesture recognition has in the everyday computing world…

I have searched through forums associated with non-gaming-exclusive applications of the Wiimote and found that the general opinion of members is that it’s not possible (http://www.wiili.org/forum/tracking-distance-and-3d-position-without-a-sensor-bar-t3763.html). Another search under general applications of accelerometers (not associated in any way with the Wiimote) revealed algorithms that take accelerometer data and several other determinable variables to calculate displacement.

Technically I ‘should’ be able to swipe the Wiimote 1 metre to the right and collect the data output from the accelerometers. Then I could integrate these readings to retrieve the velocity at a given time, and therefore the displacement (or distance). The final result should have some resemblance to 1 metre.

I performed the same gesture a number of times, however the displacement calculations returned wildly different results. A slight deviation in the results could be acceptable however when moving the Wiimote over the distance of 30 centimetres I returned results between 4 and 40 centimetres.

At this point, it should be noted that the Wiimote doesn’t actually output values that can be interpreted as acceleration. The values it returns are actually measured in g’s (g-force) which is the amount of acceleration in relation to the Earth’s gravity. 1g is roughly equivalent of 9.8m/s/s or 9.8ms^-2 so the data from the Wiimote can be converted to real acceleration data. (http://en.wikipedia.org/wiki/G-force)

In the experiments I conducted using the Wiimote, the accelerometer data was captured at anything from 100 times per second to as little as 10 times per second. Therefore, 9.8ms^-2 has to be divided by the number of samples per second.

All the calculations are perfectly sound on paper, however the source of the data – the accelerometer – isn’t reliable enough to produce consistent, predictable results.

So, why not?

Well, first, I had a look at the specification of the Wiimote’s accelerometer – an Analog Devices ADXL330 (http://www.analog.com/en/mems-and-sensors/imems-accelerometers/adxl330/products/product.html). The g-force readings of the accelerometer are accurate to +/-3g. If the Wiimote is shaken hard or swung sufficiently fast then readings outside +/-3g can easily be produced. Any readings outside the specification of accelerometer should be disregarded, however, in some cases, this would mean completely disregarding an entire set of data!

Next, I found that the Wiimote can be held at a certain orientation and provided the user has a steady enough hand, they can move the Wiimote through the air without altering the output of any of the axis on the accelerometers.

Before writing off the Wiimote’s capabilities to measure displacement using its accelerometer data, it should be noted that a programmer and car-enthusiast (http://forums.vwvortex.com/zerothread?id=3362586) successfully strapped his Wiimote to his car and managed to measure his 0-60mph time, and therefore distance traveled. It should be noted that perhaps periods of smooth acceleration can be accurately recorded with the Wiimote. This is because the g-force readings from the accelerometer would be constantly increasing (or decreasing). If one was to plot the readings on a graph of acceleration against time, they would see a smooth line. If one was to plot the readings of a rapid hand gesture on a similar graph, the line would resemble more that of a ‘heart-rate monitor’. It’s obvious that the former readings would be more accurate.

The problems I have experienced don’t appear to just be the preserve of programmers using the Wiimote for non Nintendo Wii software.

It also appears that more observant gamers have noticed the shortcomings of the Wiimote as well. The ‘real-life’ movement that was promised by Nintendo with the release of the Wii console doesn’t match the now-apparent limitations of the system.

(http://www.gamasutra.com/view/feature/1494/wheres_the_wiimote_using_kalman_.php)

Using the Tennis game in Wii Sports as an example, more specifically the way the player serves the ball to the opponent, one can see that there are many ways of playing the game. The method of serving in Wii Sports Tennis is to hold the Wiimote down by your side, raise it up quickly and bring it down again. A player who may have played tennis before or at least knows what a tennis serve looks like may try to serve the ball using a real-life serving technique. From personal experience, I’ve seen adults try to serve by throwing a non-existent ball into the air, and then try to hit it using the Wiimote. Annoyingly for them, using a real-life technique in Wii Sports Tennis doesn’t actually work. The serve must be carried out using the rather childlike method described above.

Other gestures like those required in the bowling game in Wii Sports appear to be more realistic in terms of the motion the player makes with the Wiimote. However, this is purely due to the fact that the real-life bowling action is within the capabilities of the Wiimote’s accelerometers. Compare this to where the action had to be altered by the game’s developers for the tennis game to a gesture that the Wiimote can understand.

In terms of the gaming aspect of the Wiimote’s technical ability, it seems suitable to simulate some types of motion that occur on very definite axis lines – i.e. in straight lines rather than a diagonal swing across all axes. For other, more complex gestures, there is a certain amount of difference between the motion that the game’s developers tell the player to make (on-screen instructions) and the action motion that the game picks up (the code running the game). For example, a game may tell the player to perform an elaborate swishing gesture with the character’s ‘sword’, but in fact, the game’s code doesn’t care how elegant or ‘swishy’ the gesture is. It will only concern itself with a very small portion of the movement to produce a distinct gesture.

Possibly the most irritating aspect of the Wiimote for consumers is that it isn’t the all-singing all-dancing control device that Nintendo promised. All the people who go and buy the various sword fighting games out there (Red Steel etc) and games from the Star Wars franchise must have a sour taste in their mouths when they realise that the Wiimote isn’t quite the sword or Lightsaber they first thought. This is due, once more, to the Wiimote’s inability to sense it’s position in space – it’s not possible using accelerometers.

Nintendo are shortly to release a device called Wii Motion Plus that bolts onto the Wii
mote (where the Nunchuck is usually connected) that contains a gyroscope. This device will enable game developers much more scope when programming the physical aspect of their games.

“Nintendo’s upcoming Wii MotionPlus accessory for the revolutionary Wii Remote controller again redefines game control, by more quickly and accurately reflecting motions in a 3-D space.”

Hopefully, this will see less use of the IR Sensor Bar to detect distance from the TV and more use of the Wiimote on its own.

While the purpose of this article wasn’t to moan able the shortcomings and downfalls of the Wiimote, I feel that some of the text is necessary due to the fact that I myself came to the Wiimote thinking that so much more was possible! However, it’s not all bad – there are ways around most of the problems I’m encountering thus far.

Now, for the purpose of this investigation, I shall suspend the research into calculating displacement with the Wiimote. The main reason for this is that it doesn’t appear to be possible to obtain accurate readings from the types of gestures I wish to record and recognise.

What would have been version 5 was the application’s ability to recognise simple gestures. Whenever a gesture was recognised, the application played a lightsaber sound (yeh, I know, too tempting right?)

Now (for technically version 6) the application has morphed into a simple game – just for the purposes of demostration – where the player has to make the correct gesture displayed on the screen within a certain amount of time. All this is happening in time to a piece of music, so the general idea is that the player should wave the Wii Remote in the correct directions in time with the music…

…and it ‘kind of’ works!

Anyway, take a look at the video below. There isn’t any audio present at the moment due to some syncing problems I had with the screen recorder, so at some point my trusty camera will come out again and I’ll have to film myself playing it…! For the time being, picture it being played to Lady Gaga’s ‘Just Dance’.

Higher quality video available here…

That’s all for the moment…. Over n out…

(higher quality video available for viewing here!)

“What you’ll probably notice is that the movement is quite stuttery, so even when the Wiimote is in a stationary position there may still be some jitters. This is due to the fact that the 3D model of the Wiimote takes it’s rotation properties directly from the ever-changing data of the real Wiimote’s accelerometers.”

This is fixed! If you compare the movement of the Wii Remote in this video to the other 3 I’ve recorded so far, you’ll notice that the motion much smoother. It’s difficult to tell from the video, but now it’s easier to select the direction you want to point the Wii Remote. Before, if the Wii Remote was held on the cusp of a different direction, for example between ‘left’ and ‘idle’, the chances were that there would be lots of ‘left’ directions generated as the accelerometers in the Wii Remote flit between ‘left’ and ‘idle’ positions.

The smooth movement is gained by sampling the last 15 accelerometer values for each axis (x, y and z) and calculating an average of each. This average is used in place of the raw accelerometer readings. This method irons out the little ‘in-between’ vibrations from the accelerometer.

The slight (and the benefits of averaging 99.9% outweigh this) problem is that there is a slight delay between moving the Wii Remote and the program reacting. This is because the latest movement is being averaged alongside the last 14 movements could be much lower or higher – so the numbers have to effectively ‘catch up’. In fact, as the application is running at 30 frames per second, the movement is up to half a second behind. The delay can be reduced by taking fewer samples for the average.

…the work continues.

Today, I altered my Wiimote program in Director to make a 3D object move around based on the data gathered from the Wii Remote’s accelerometers. The control is actually pretty accurate. So far, you have to hold the remote in a fairly ‘wooden’ fashion. By this I mean that you can’t swish the remote left or right as one might if playing a real Wii game (Wii Sports Tennis comes to mind here).

So, after some faffing with the Lingo in Director I have the following degree of control over my 3D ball…

note: the ‘deadzone’ for movement is attained when you hold the remote straight up in the air.

Move left: swing the remote to the left
Move right: swing the remote to the right
Move down: pitch the remote forwards (the end of the remote is moved in an arc away from your body)
Move up: pitch the remote backwards (the end of the remote is moved towards your body, the amount of movement is limited for this action)

At the moment, the movement of the ball isn’t directly related to the accelerometer data in terms of the actual numbers… So if the accelerometer hints that the Wii Remote is being swung to the left then the ball is moved left by a finite amount.

That’s all for the moment, however everything is well on the way to more exciting things!

So moving the remote rapidly from side to side will cause change in the ‘x’ axis.

The sensitivity of the accelerometer in the remote (and indeed the Wii Nunchuck – they’re the same spec) generally means that it’s very difficult move along one axis at a time.

Now it’s time to work out how to get this information recorded in Director!

1. ‘Rez in Flash’
2. …but in a faux-3D world.
3. The player flies into the screen.
4. Exclusively uses a Nintendo Wii Remote as a means of control.
5. Move the player left and right by altering the ‘roll’ of the Wii Remote.
6. Aim using the Wii Remote functioning in the same way as a mouse.
7. Use of gestures – Tracing different shapes using the Wii Remote to unlock areas and proceed through the game.

That’s all folks – hope it’s easy to make!

MA