Ming the Mechanic:
News Aggregator

The NewsLog of Flemming Funch
 News Aggregator2003-12-28 19:55
picture by Flemming Funch

I spent the evening making my own news aggregator, and remarkably I succeeded to my satisfaction.

For any non-techies, a news aggregator is a program that sucks up data in RSS format, served by weblogs of various kinds, and presents it all in a uniform way.

I've tried a bunch of different aggregators. What I liked best was Radio Userland, because it shows the feeds together, looking like a weblog, and it seemed to be able to mainly show me a flowing stream of new stuff. Which I liked, but I'd kind of like more options. But when my paid license expired, I hesitated to renew it, because I wasn't really using it for its weblog or other functions. So I tried a variety of other programs.

FeedReader on Windows, which was nice. Except for that I don't like having to watch postings one at a time. I like the big overview. On Mac I then used Shrook for quite a while. It still had that 3-pane thing, and crashed every couple of hours, so it ended up not running most of the time. I tried installing NewsMonster, after its website made me feel kind of stupid, as it is so superior that it apparently can do everything, including a bunch of things I don't know what are. Except for that it couldn't find Java on my computer, and messed up some of the menus in Mozilla that it was supposed to integrate with. I installed Pears which runs in Python. Worked, but was a bit too simple. I installed AmphetaDesk, which required installing a whole bunch of Perl libraries first. And, now, I like the look of it. Quite a bit like Radio. But now there's again a bunch of things I'd want it to do that it doesn't do.

So, I woke up late and thought that if I could make my own aggregator, and I could finish it so it was functional today, I'd go for it. I really have other things to do, but it is Sunday and christmas, so nobody would be missing me too much.

Somewhat reluctantly I decided to look for a library that does the basic fetching of an RSS feed. My first thought was that I could just as well write that myself too, but that is the kind of arrogance that makes me often end up with projects full of features, but not quite finished, because I try to do it all myself. So I picked up the Magpie RSS library in PHP. Which seems simple enough, and I only needed some of its features.

Now, I decided to set it all up on my server, as opposed on my local machine, so I can make the functionality available for other users of my weblog program, and so the feeds can be cached amongst those users. And what I wanted was to store feeds and postings in mysql, so they can be kept indefinitely, and to be able to keep track of which ones have been read and stuff like that. So that is what I set up. A cron job picks up all channels every hour, and figures out what are new or updated postings. And then some PHP pages show which feeds one is subscribed to, which are available from the pool that is already on the server, and allows addition of new feeds. And one can see them either one at a time, or mixed together. And I borrowed somewhat the look from AmphetaDesk. But then I added the ability to keep track of which items in each feed a given user has read, and which ones they've at all seen. Then it can avoid showing what has been marked as read, and it can mark new postings with a little NEW icon. And I made it so the postings can be grouped by feed or by date. And they can be sorted in various ways. And I made a way of saving interesting postings to a separate place before they scroll away. And I added in the 50 or so feeds that I normally watch. And this already works better for me than any of the other aggregators I've used.

I'll tinker some more with it before I'll let anybody else use it. And there are a few more things I'd like to add. It should be able, of course, to pass a post on to my weblog program, if I want to quote it. I need some ways of searching through older postings. Some more options of viewing them. Like, headings only, short excerpts, with or without pictures, etc. Maybe a way of categorizing the saved postings. But this should do for today.

[< Back] [Ming the Mechanic]



28 Dec 2003 @ 20:57 by george dafermos @ : nice one
..can't wait to use it.

and newsmonster isn't that great, but then again, i don't really like news aggregators (can't tell why. probably because i'd use an aggregator in order to save time, but going through some 150 feeds takes quite some time). i tried newsmonster a while ago mostly because i liked having an aggreagor integrated with my browser(after installing jave runtime environment 2 for win), but i quickly lost interest in it, and i have now resorted to using (selectively and occassionally) radio again.  

29 Dec 2003 @ 17:59 by maxtobin : Still doing good stuff.
Is this not the time to put it out there? As you have been discussing open source etc, Let some others play with it and it may get to be more perfect than even you could make it Flemming. What a way to spend Christmas!!! Our best wishes to the Family and for your success in a fertile New Year  

30 Dec 2003 @ 06:44 by lugon @ : try to free it, see what happens
Best wishes to you and your readers and ... until their 6th degree friends. :-)  

19 Dec 2014 @ 15:38 by Alejandro @ : wSBosszJYrWnsJZvlq
I agree with Vince LaRue. They have not created a liinvg cell, they have replaced the DNA in a liinvg cell with an exact copy assembled artificially.It is a useful step forward, but it has been over-hyped.The danger is that crazy people might use this technology to engineer biological weapons - but that has been possible for years. The benefit is that bacteria or yeasts can be designed with custom DNA for making complex drugs, and possibly other chemicals.  

23 Dec 2014 @ 15:05 by Sigrid @ : tsMUDAVoRRdDJ
Can someone pslaee explain why Israel hasn't sent its own flotilla to deliver up to date medications and cement to the Kurds in east Turkey? And/or publicly recognized the Armenian genocide? Perhaps even have a national commemoration for it? Better yet, have the Israeli navy accompany the flotilla. I wonder how Turkey might look at its own flotilla then.  

30 Jun 2015 @ 03:15 by http://tokogreenworldindonesia.com/ @ : Way cool!
Way cool! Some very valid points! I appreciate you penning this post plus the rest of the website is very good.
Obat Herbal Sakit Ulu Hati  

21 Feb 2016 @ 17:45 by Verle @ : ReOXNAccwzGsihlvwew

21 Feb 2016 @ 21:19 by Evaline @ : kPosFBGNKDQoyjJrFs

25 Nov 2016 @ 08:59 by SejarahQQ @ : SejarahQQ.com Agen DominoQQ Online
Thanks for your post. Click to play !  

25 Nov 2016 @ 10:32 by BandarPoker @ : SejarahQQ.com Agen DominoQQ Online
I like your article  

6 Dec 2016 @ 14:09 by Baseball Betting Guide @ : very nice post
very nice post, i certainly love this website, keep on it Favorite Soccer Jersey Hockey  

Your Name:
Your URL: (or email)
For verification, please type the word you see on the left:

Other stories in
2014-11-01 17:33: The conversation of work
2007-02-24 14:20: Writing books in HTML/CSS
2007-02-05 15:21: Software is hard
2006-11-19 21:30: Thingamy
2005-12-14 15:15: Ruby on Rails
2005-03-19 16:04: Comment and Refererrer Spam
2005-02-23 21:34: Wikipedia
2005-02-22 17:32: Mail
2005-02-10 16:00: More Google wizardry
2005-02-04 15:14: The Six Laws of the New Software

[< Back] [Ming the Mechanic] [PermaLink]? 

Link to this article as: http://ming.tv/flemming2.php/__show_article/_a000010-001016.htm
Main Page: ming.tv