So, I’m going to be building a robot that can play Pong over the next five days.

Why am I going to spend five days building a robot that can play Pong?

Well, first of all, it’ll be totally cool to have a Pong playing robot.  You simply cannot deny the awesome factor of that.

Second, it will be fun to do.  I’ve wanted to learn about computer vision for a while, and I’ve wanted to tinker with robotics for a while, and I love video games.  So why not combine all of those into one big crazy project?

And finally, ATARI PLAYING ROBOT!  Do you really need any other reason?

I chose Pong as the first game to attempt because it has uncomplicated graphics and clearly defined rules that can be described using fairly simple mathematical models.  The ball will remain in motion at a constant speed in a straight line until it hits something else.  (Newton’s First Law of Motion, hence all that “Newtonian Motion Simulator” nonsense earlier.)  There’s no need for any kind of pathfinder or decision making AI or similar kind of complicated logic for playing the game.  If I can detect the position of the ball in two subsequent frames, and I know where the walls are, then I can predict exactly where my paddle needs to be placed in order to hit the ball back.  Computers are very good at calculating that sort of thing.  Pong is also a paddle controlled game, which means the robot only needs to be able to rotate clockwise or counterclockwise.  No complicated multi-axis positioning that a joystick would require.

(It should be noted that the fundamentals needed to play Pong are the same things that are needed to play many other games.  If you get Pong, then you get Breakout almost for free, because Breakout is Pong rotated 90 degrees.  Similar to Breakout is the frantic bomb-catching game of Kaboom!.  And, although they’re visually very different and utilize very different controls, games like Rock Band or Guitar Hero are essentially variants of Kaboom!.  Which means that I can go from building something that can play Pong and have a clear path to a 5-star performance on Green Grass and High Tides.)

I’ve seen other people make game-playing contraptions, but none of them have tried actually watching the screen and reacting to what’s there.  I’ve seen a Wii Bowling Robot that can bowl a perfect game, but it’s taking advantage of fact that if you roll perfectly straight, you get a strike.  I’ve seen several Rock Band/GH bots, but they’re hacked controllers that play back a recording.  And I’ve seen a program that can play a kick-ass game of Super Mario World, but it’s an AI built directly into the game code.  I’ve never seen anyone who has made a robot that uses a camera to play the game using the normal TV display and normal controllers.  That’s my goal here.  To build a robot that can play games on a completely unmodified Atari 2600.

For the robot, I’m going to use Lego Mindstorms.  That seemed like the easiest way to go, especially since I have no electrical engineering experience.  If this goes well, I might step it up a notch and try a raw microprocessor and motors and risk of electric shock and fire type robot, but for now, the Legos seem good.  I bought the kit about a year ago and it’s been hanging around on a shelf ever since then.  It was actually around that time that I first started thinking of building an Atari controlling robot.

Of course, all of this plan is extremely difficult if I can’t see the game.  In order to do that, I plan to use OpenCV.  It’s an open source computer vision library (with an available .Net binding), that seems to have all manner of image processing and recognition functionality in a package that looks disturbingly easy to use.  Whether or not it is, I’ll find out soon enough.  It’s the same software that’s used in some of those DARPA challenge rally cars that drive themselves across the desert, so I figure it oughta be able to distinguish a couple of big blocky rectangles moving around on a TV screen.  (Unless the raster scan on my CRT TV throws it off…)

So, the end-to-end plan is this:  I point a camera at the TV, showing a game of Atari 2600 Pong.  The camera feeds into an OpenCV based processor running on my computer.  The computer calculates where the paddle needs to be in order to hit the ball, and uses Bluetooth to tell the Mindstorms robot to rotate the Atari paddle left or right.  The paddle hits the ball back.  Because the AI on the Video Olympics version of Pong is simple “follow” logic that doesn’t move fast enough to reach the ball in time when the ball reaches a certain speed, my robot is basically guaranteed to be able to beat the AI.

Seems possible, eh?

I plan to tackle the computer vision stuff first, then move onto the robotics.

Let’s break down that spec and take a closer look at the components.

Visual Data Collector

The Visual Data Collector is a standard PC USB Webcam.  Hopefully it will meet my requirements.  If not, I have a backup plan.

Computer Vision Textbook

I plan to use OpenCV to process the input from the Visual Data Collector.  It seems fairly straightforward to use, even when the task is complex.  Of course, I haven’t actually programmed anything using it yet, so we’ll see.  I’ve skimmed the book and have some idea of where to start.

Communications Interface

The Newtonian Motion Decoder and Processor will talk to the Real Time Autonomous Interaction Device over Bluetooth.  I’ve never programmed anything using Bluetooth.  I pretty much stick to TCP/IP.  This will be an interesting change.

Real Time Autonomous Interaction Device

The Realtime Autonomous Interaction Device will be constructed using a Lego Mindstorms robotics kit.  I bought this about a year ago and only built the sample car thing that everyone builds first.  I’ve never tried anything on my own.  Until now.

Simulator Control Interface

This device houses an analog potentiometer, which will function as the interface to the Newtonian Motion Simulation.

Low-Resolution Newtonian Motion Simulator

This Video Computer System excels at running simulations of Newtonian motion, so I have chosen it as the primary Newtonian Motion Simulator for this project.

Newtonian Motion Software

The read-only memory chip on this circuit board contains fifty different kinds of Newtonian motion simulations, however, for the purposes of this project, I will only be focusing on one of them.

Hardware

Here’s a jumble of most of the equipment I’ll be using during this project.  Not pictured are the display screens for the Low-Resolution Newtonian Motion Simulator and the primary and secondary Decoder and Processor devices, as they are not very portable.

So, there you have it!