Notes on tech

As I inch my way towards completion of the notification project, I have had my own set of challenges. One of the challenges was to extract plain vanilla text out of the complex HTML that we usually come across on various sites. The html that is dished out on a typical website contains various elements like flash banners, advertisements, style sheets, java script etc. If you are out to determine whether the content of a particular page has changed since you last viewed it, you might as well extract just the content, ignoring all the other blah blew dooh daah that make up the page. Also every site has a different style of presentation as well as content updation logic. Since the past one week, I was busy trying to come up with a pluggable framework that can contain various comparators. The comparators do the filtering of content and determine whether there have been any updates to the content since the last visit (crawl?). There might be comparators that correspond to certain kind of site layouts, some might be responsible for filtering RSS feeds, PDF's, what not.

So the framework is ready, and I have already implemented a simple text comparator that basically is fairly successful in extracting the real stuff from the fluff that surrounds it. Only if we had something like this in our lives.