Computer Vision Processing Game Screen? Check.
Game Play Logic Predicting Ball Trajectory? Check.
Mindstorm Robot Attached To Paddle? Check.
Bluetooth Communication To Robot? Check.
Linking Trajectory Prediction to Robot Communication? Check.

Okay! Let’s go!

Maiden Voyage

Complex Systems Interacting To Produce Failure? Check.

That high pitched noise at the end after it jams is coming from the motor. Motors are not supposed to give off high pitched noises. Additionally, prior to jamming, it forced the potentiometer inside the paddle to skip past the end of its range a couple of times (The loud clicks that you heard). That can’t be a good thing to have it doing…

Even with this program, I still get beat by the CPU.  Stupid Atari.

Looks like I still have some work to do.

Anyway, major milestone!  I am now able to play a live game and have it reasonably direct me where to put the paddle in most cases.  It obviously still needs some tweaking to be flawless, but I think I can claim this as a success.

I set up the application to overlay all of its information over the real game screen, then started recording.  Then I stumbled across this:

[Click to Play Video]

I think there’s something philosophical and deep to be had here.  I’m working on all this technology and complicated calculations and processing in order to beat the game, and none of it turns out to be as effective as sitting still and doing nothing.  Awesome.

BTW, final score, 21-Zip.

 

Edit:  Here’s a YouTube video of an entire perfect game from that spot.

Had a bit of a problem with the bounce on the trajectory projection.  First, there were infinite slopes getting in the works and gumming everything up.  An infinite slope is a vertical line, and vertical lines don’t cross the paddle plane, and when the trajectory projection doesn’t intersect the paddle plane, you end up with a Forward Intersection Point list with about three million points in it, as the ball is predicted to be happily bouncing up and down over and over and over until the end of RAM.  After that, I had a bug where I wasn’t using the last calculated bounce point as the seed for the next bounce point.  This led to an angular infinite loop, bouncing between points 1 and 2 forever.

But those are both gone, and what I have now is a real-life forward trajectory projection!  It’s really exciting, because it typically completely fails to predict where the ball is actually going to go until the last second!

Here’s some video of the current status.

Two Point Slope Calculation:  Two Point Trajectory Projection

Five Point Calculation:  Five Point Trajectory Projection

You can see that the five point average is less hyperactive, but it’s still not good enough.  Hopefully the endpoint averaging will solve that problem.

But that will have to wait.  This is where I’ll leave it for the night.  Not bad for starting with nothing this morning…  Four days remain.

One thing that I haven’t mentioned regarding the “physics” of Pong is that the paddle causes a total change in the motion of the ball.  The walls reflect the ball like a real-world wall would, giving the ball a negative slope of the same value.  The paddles, however, assign a new slope, based on where the ball hits.

If the ball hits high on the paddle, the ball gets a steep upward slope.  If the ball hits toward the middle of the bottom half, the ball takes a 45 degree downward trajectory, and when it hits the middle, it will go straight across.

The diagram above is for illustrative purposes only, as I’m pretty sure there are 7 possible angles, not the five shown:  Three up, three down, and straight.  Seven makes the most sense when considering the game programming, as well.  There are seven possible angles for each side, which makes 14 possible angles in normal play.  Add in straight up and straight down, which are possible on some of the other variations of Pong on the cartridge, and you get 16 total angles, which can be stored neatly in a four-bit nibble.

Another point to note is the X on the upper half of the paddle.  That represents the location of the follow point on the CPU paddle.  The Atari will try to always seek that point to the ball, which means that most of the time, when the CPU paddle hits the ball, it’ll bounce it in an upper trajectory.  However, in order to make it possible for the player to actually beat the computer, the CPU paddle can’t move fast enough to perfectly track the ball.  The paddle is always too slow for the highest angles and will only be able to hit them if the ball reflects off a boundary wall close enough to the CPU side so it doesn’t have to chase it that far.  At higher speeds, it can be too slow for any angle, potentially even straight across.

Anyway, the relevance of this to my project is that I’m not going to attempt to calculate the trajectory after hitting the paddle.  It would be too much work to determine the location of the different reflection zones, and wouldn’t be of any real benefit.

I am Locutus of Pong.

You will be assimiliated.

 

[Click for Video]

I now have a basic recognizer working off of a video recording of the game.  It’s finding the top and bottom boundaries, and most of the time, it knows where the left and right paddles and the ball are.  There are still occasional glitches, like where an especially prominent piece of noise decides it wants to be the ball, or where for some reason, there are nearly overlapping contours found over one of the paddles, however, I think it’s at a point where I can proceed to the gameplay logic calculations.

I’m sure there’s some good reason that OpenCV is returning duplicate or near duplicate contours, and I’m sure there’s some known way to deal with the problem.  I probably should read the book a bit more…

Nah, the book’s only for when I get completely stuck.

Anyway, here’s a description of what I’ve done so far, for those of you playing at home:

1. Pull a frame from a pre-recorded video of Pong.  This frame is in the upper left video box.
2. Run the frame through a Gaussian smoothing operation twice to tone down the annoying noise in the video.
3. Turn the frame greyscale for the next operation.
4. Run a Canny edge detector on the image.  The results of this are in the upper right box.
5. Feed the Canny image into a contour finder.  The results, shown in the lower left, look pretty much the same as the Canny image, but the difference is that the Canny results are a regular bitmap image, while the contours are sets of vectors that are more easily manipulated.
6. The contours are run through in order to find the playfield elements.  The assumptions used to find the playfield elements are as follows:
   1. Anything greater than 70% of the width of the screen is either the top boundary or the bottom boundary.  If it’s above the middle of the screen, it’s the top, if it’s below the middle, it’s the bottom.
   2. The ball and paddles are between the top and bottom boundaries.  This lets me throw out the score.
   3. While I’m at it, throw out any contours that are really dark or black.  They’re probably just noise.
   4. Sort what’s left by area, descending.
   5. The two biggest things are the left and right paddles.  Whichever one is farthest left is the left paddle.
   6. The third biggest thing is the ball.
   7. Everything else is noise.
7. I package up the recognized playfield elements into a nice class containing a bunch of rectangles.  These rectangles are drawn in the window on the lower right.

Going forward, I’ll keep a list of the recognized playfield elements from the last X frames and use them to calculate the position and trajectory of the ball.  Since I know where the ball is across multiple frames and where the boundaries are, I can extrapolate where the paddle needs to be  The goal for this phase will be to get the calculations written and to display an overlay on a live game that will tell me exactly where I need to place my paddle as soon as the ball starts heading my way.  Most of this is pure algebra, but there may be complications if the recognition isn’t good enough.

The code has been checked in up to this point.  https://mathpirate.net/svn/

So, the last issue was just a sort order problem, as I suspected.  Started sorting descending and presto!  Things magically changed.

Unfortunately, things magically changed into a new set of problems.  The left and right paddles are frequently there now, but the ball is now missing.  Additionally, it seems to think that the right paddle is the ball, the left paddle is the right paddle, and I don’t know what it thinks is the left paddle, because it’s not showing up at all.

Still, it’s progress…  I guess.

I’m trying to recognize the elements on the screen.  The upper and lower boundaries were easy, but the ball and paddles aren’t behaving quite so nice.

[Click for video]

As you see, it’s consistently saying the ball is a paddle and that the left paddle isn’t even there.  Something tells me that there’s a bug here…

Yep.  Noise is going to be a problem.

(Left: Raw Video.  Right: Canny Edge Detection)