Ming the Mechanic:
News Aggregator

The NewsLog of Flemming Funch
 News Aggregator2003-12-28 19:55
5 comments
picture by Flemming Funch

I spent the evening making my own news aggregator, and remarkably I succeeded to my satisfaction.

For any non-techies, a news aggregator is a program that sucks up data in RSS format, served by weblogs of various kinds, and presents it all in a uniform way.

I've tried a bunch of different aggregators. What I liked best was Radio Userland, because it shows the feeds together, looking like a weblog, and it seemed to be able to mainly show me a flowing stream of new stuff. Which I liked, but I'd kind of like more options. But when my paid license expired, I hesitated to renew it, because I wasn't really using it for its weblog or other functions. So I tried a variety of other programs.

FeedReader on Windows, which was nice. Except for that I don't like having to watch postings one at a time. I like the big overview. On Mac I then used Shrook for quite a while. It still had that 3-pane thing, and crashed every couple of hours, so it ended up not running most of the time. I tried installing NewsMonster, after its website made me feel kind of stupid, as it is so superior that it apparently can do everything, including a bunch of things I don't know what are. Except for that it couldn't find Java on my computer, and messed up some of the menus in Mozilla that it was supposed to integrate with. I installed Pears which runs in Python. Worked, but was a bit too simple. I installed AmphetaDesk, which required installing a whole bunch of Perl libraries first. And, now, I like the look of it. Quite a bit like Radio. But now there's again a bunch of things I'd want it to do that it doesn't do.

So, I woke up late and thought that if I could make my own aggregator, and I could finish it so it was functional today, I'd go for it. I really have other things to do, but it is Sunday and christmas, so nobody would be missing me too much.

Somewhat reluctantly I decided to look for a library that does the basic fetching of an RSS feed. My first thought was that I could just as well write that myself too, but that is the kind of arrogance that makes me often end up with projects full of features, but not quite finished, because I try to do it all myself. So I picked up the Magpie RSS library in PHP. Which seems simple enough, and I only needed some of its features.

Now, I decided to set it all up on my server, as opposed on my local machine, so I can make the functionality available for other users of my weblog program, and so the feeds can be cached amongst those users. And what I wanted was to store feeds and postings in mysql, so they can be kept indefinitely, and to be able to keep track of which ones have been read and stuff like that. So that is what I set up. A cron job picks up all channels every hour, and figures out what are new or updated postings. And then some PHP pages show which feeds one is subscribed to, which are available from the pool that is already on the server, and allows addition of new feeds. And one can see them either one at a time, or mixed together. And I borrowed somewhat the look from AmphetaDesk. But then I added the ability to keep track of which items in each feed a given user has read, and which ones they've at all seen. Then it can avoid showing what has been marked as read, and it can mark new postings with a little NEW icon. And I made it so the postings can be grouped by feed or by date. And they can be sorted in various ways. And I made a way of saving interesting postings to a separate place before they scroll away. And I added in the 50 or so feeds that I normally watch. And this already works better for me than any of the other aggregators I've used.

I'll tinker some more with it before I'll let anybody else use it. And there are a few more things I'd like to add. It should be able, of course, to pass a post on to my weblog program, if I want to quote it. I need some ways of searching through older postings. Some more options of viewing them. Like, headings only, short excerpts, with or without pictures, etc. Maybe a way of categorizing the saved postings. But this should do for today.


[< Back] [Ming the Mechanic]

Category:  

5 comments

29 Dec 2003 @ 17:59 by maxtobin : Still doing good stuff.
Is this not the time to put it out there? As you have been discussing open source etc, Let some others play with it and it may get to be more perfect than even you could make it Flemming. What a way to spend Christmas!!! Our best wishes to the Family and for your success in a fertile New Year  


19 Dec 2014 @ 15:38 by Alejandro @186.93.4.150 : wSBosszJYrWnsJZvlq
I agree with Vince LaRue. They have not created a liinvg cell, they have replaced the DNA in a liinvg cell with an exact copy assembled artificially.It is a useful step forward, but it has been over-hyped.The danger is that crazy people might use this technology to engineer biological weapons - but that has been possible for years. The benefit is that bacteria or yeasts can be designed with custom DNA for making complex drugs, and possibly other chemicals.  


23 Dec 2014 @ 15:05 by Sigrid @115.112.106.139 : tsMUDAVoRRdDJ
Can someone pslaee explain why Israel hasn't sent its own flotilla to deliver up to date medications and cement to the Kurds in east Turkey? And/or publicly recognized the Armenian genocide? Perhaps even have a national commemoration for it? Better yet, have the Israeli navy accompany the flotilla. I wonder how Turkey might look at its own flotilla then.  


21 Feb 2016 @ 17:45 by Verle @188.143.232.32 : ReOXNAccwzGsihlvwew
 


21 Feb 2016 @ 21:19 by Evaline @188.143.232.32 : kPosFBGNKDQoyjJrFs
 


Other stories in
2014-11-01 17:33: The conversation of work
2007-02-24 14:20: Writing books in HTML/CSS
2007-02-05 15:21: Software is hard
2006-11-19 21:30: Thingamy
2005-12-14 15:15: Ruby on Rails
2005-03-19 16:04: Comment and Refererrer Spam
2005-02-23 21:34: Wikipedia
2005-02-22 17:32: Mail
2005-02-10 16:00: More Google wizardry
2005-02-04 15:14: The Six Laws of the New Software



[< Back] [Ming the Mechanic] [PermaLink]? 


Link to this article as: http://ming.tv/flemming2.php/__show_article/_a000010-001016.htm
Main Page: ming.tv