Random header image... Refresh for more!

Category — Crazy Weekend Project 4

Authorization Required

I’m stuck in Twitter API Authorization Hell.  Apparently username/password combos are too old school.

September 5, 2010   No Comments

The Holy Grail

I just found a Twitter API wrapper that also handles Yammer.  Oh the joy!  It’s exactly everything I’ve ever wanted and more!

September 5, 2010   No Comments

Merge Error

There seems to be another problem with the merging algorithm.  Now it’s corrupting the source indexers, causing some of the merged content to end up in the source as well as causing an abnormal number of terminations.  Gotta fix that before proceding.

Turns out that I was simply sharing the suffix lists of the first indexer that had a particular prefix with the merged content.  The merged content would be fine, but that source indexer would have extra, random terms thrown in.  Those random terms, since they weren’t in the original, would cause a lookup error when trying to look up possible suffixes by prefix.  That lookup error would cause an early termination.

In other words, bad things happen.

Anyway, should be fixed now.

September 5, 2010   No Comments

Late Start on Day 3.

Our automobile industry has overhauled assembly lines, increased worker productivity, and is less than half of what happened yesterday and what’ll happen tomorrow.

Many apologies.  I’d hoped to have started on this around 5-ish, but failed to do so.  Oh well.

When I left off yesterday, I’d redone the UI for the Echo Chamber application, and set it up to support multiple separate indexers.

It is now the time to turn to Twitter.  I need to start scraping tweets and feeding them into the system.  That is, after all, the core idea behind the project…  You can only go so far chopping up Reagan speeches and splicing in bits of the Kama Sutra, after all. 1

API, here I come!

  1. Good evening. Usually, I talk of tax cuts, I’m reminded that the advances on the threshold of a number of military production is putting a heavy strain on the human heart, called upon now to resolve that the man should touch her private parts, should loosen her girdle and the man whose business it is called the “inside pressing.” []

September 5, 2010   No Comments

Middle East Peace Process FTW

Israel agreed to negotiate a major agreement with Israel. That’s another accomplishment of this nation.

September 5, 2010   No Comments

Well… That could be part of the problem.

Apparently merged.SourceData has 45 million lines in it.

I’m thinking that could be where I went wrong…

foreach (Indexer toMerge in indexers)
{
    merged.Name += "|" + toMerge.Name;
    foreach (KeyValuePair<EchoToken, List<string>> record in toMerge.TokenTable)
   {
         if (merged.TokenTable.ContainsKey(record.Key))
         {
               merged.TokenTable[record.Key].AddRange(record.Value);
         }
         else
         {
              merged.TokenTable[record.Key] = record.Value;
         }

         merged.SourceData.AddRange(toMerge.SourceData);
    }
}

Well, of course I want to add the Source Data for EVERY SINGLE TOKEN IN THE TABLE.  That’s not a disaster waiting to happen or anything like that…

I’ll just move that one curly brace up a few lines before anyone notices…

September 5, 2010   No Comments

throw new IranContraException();

Hmmm…  That didn’t work.  I guess it doesn’t like merging with itself.

Although, the merge shouldn’t have been that big, so I don’t know why it had a problem.

September 5, 2010   No Comments

Three’s a Crowd

As I expected, a prefix length of three is too long for general use.  In most cases, when I generate a string of text, what comes out is a direct quote.

Although, something tells me that Reagan never actually said this:

Tonight America is stronger because of the way it tears at our lives and because it’s aimed at destroying the brightness and life of the sons and daughters of the United States.

or this

You know, Abe Lincoln, the day after his election to the presidency, gathered in his office the newsmen who had been bringing ruin down on Beirut.

Still, the occasional flashes of brilliance are too rare to be useful.  I think I’m going to stick to lengths 1 and 2 most of the time.

September 4, 2010   No Comments

Maybe that wasn’t such a good idea…

The processed Reagan speech file made a 20 MB XML file.  There were something like 72000 prefixes in the file.

I’ll have to see if this is going to be a problem…

September 4, 2010   No Comments

Slight Indexer Change

I think I’m going to have the Indexer generate an index for prefix lengths 1, 2, and 3.  That way, it’ll be much easier to switch between them at generation time, which will let me play around more.  I don’t think it’s necessary to go to lengths of 4 or more, as those lengths are unlikely to produce the level of crazy that I’m looking for here.

Furthermore, I realized that selecting the suffix based on probability vs. even distribution is something that can be handled strictly at generation time.  Just pull the list of possible suffixes in to a set, and choose from there.  It’s not as efficient as pre-calculating the two types at generation time, but it’s easier to do.

Remember, it’s not about getting it done right, it’s about getting it done right now.

September 4, 2010   No Comments