Enhanced Download Script for It's FOSS Community Topics

[abu (Alfred Bühler)]

@pdecker

Where’s the fun

My bad, I forgot the meat.

[Rosika (Rosika Schreck)]

Hi all.

@abu :

@abu

[Wallabag] it’s a typical LAMP application that can store entire pages…

Thanks for the expalantion.
I might look it up, just out of interest.

@abu

I won’t do it without them. I’m reading tons of blogs and news sites via RSS.

I make use of the concept of RSS feeds as well, albeit not as much as you.
If I want to get a quick overview of the news, I quickly fire up a sandboxed instance of newsboatin the terminal:

env LANG=en_US.UTF-8 firejail --ignore="shell none" newsboat -C /media/rosika/f14a27c2-0b49-4607-94ea-2e56bbf76fe1/DATEN-PARTITION/Dokumente/Ergänzungen_zu_Programmen/zu_newsbeuter/alternative_config -u /media/rosika/f14a27c2-0b49-4607-94ea-2e56bbf76fe1/DATEN-PARTITION/Dokumente/Ergänzungen_zu_Programmen/zu_newsbeuter/urls2

… the file urls2 holding a self-defined set of feeds.

newsboat 2.21.0 - Ihre Feeds (2 ungelesen, 6 gesamt)
   1         (0/0) Topthemen - SZ.de
   2 N    (79/493) MakeUseOf
   3        (0/32) Going Linux
   4         (0/0) tagesschau.de - Die Nachrichten der ARD
   5 N    (35/112) 6 Minute English
   6         (0/0) DIE ZEIT | Nachrichten, News, Hintergründe und Debatten


q:Beenden ENTER:Öffnen n:nächster Ungelesener r:neu laden R:alle neu laden A:gel

It´s done in a jiffy and provides me with all the information I need.

If you need more information on any given subject displayed in the RSS reader newsboat can trigger a browser instance with immediate redirect to the topic in question.

Many greetings from Rosika

[ernie (Ernest N. Wilcox Jr.)]

@pdecker

It is running on my old Dell XPS Studio with the original Intel Core i7 processor and exposed to the web through a Tailscale funnel, so it might not be up and running all the time.

I think your RSS aggregator is a very good idea, and I have a few questions:

1. Is it an executable or a bash script?
2. Can it run from anywhere, or is it dependent on your network?
3. If it can run from anywhere, can/will you make it, or it’s source code available for download?

I’d rather not eat up your bandwidth to aggregate the news feeds that interest me, or depend on your aggregator being online, but mostly, the former.

Ernie

[ernie (Ernest N. Wilcox Jr.)]

Very nicely done! That was a lot of work! It’s not something that’ll work for me, because I prefer RSS feeds.

I take notes from the articles I read on It’sFOSS, or from other sources/newsletters, when I find something I may want or need, and store it as an entry in a text document.

• I start each entry with the title of the article.

• I add a synopsis of any needed information/steps.

• I include the URL of the source article as an end note for each entry.

This way, I can get at the info, even if I’m offline, or the source item disappears. My methodology isn’t automated at all, but I have all the stuff I want, in a manner that works for me, when I need it, and since I’ a retiree, I have plenty of time to manage it all, at least when I’m not working on another adventure of mine,

Ernie

[nevj (Neville Jackson)]

@ernie

take notes from the articles I read on It’sFOSS, or from other sources

I do the same
I have a directory ( folder to you Win types) where I make multiple subdirectories for topics. Inside each topic directory there may be a list of url’s and some notes and maybe an occasional copy.
So I use the filesystem to index it.
Primitive but it works for me.

[pdecker]

@ernie

• Is it an executable or a bash script?
• Can it run from anywhere, or is it dependent on your network?
• If it can run from anywhere, can/will you make it, or it’s source code available for download?

I used VS Code and GitHub Copilot to help me write it in Python. I can make the source available. It’s in Azure Repos, like GitHub in many ways and also owned by Microsoft.

It has a tiny SQLite database to store the most recent contents of the RSS feeds in the same directory structure. I have a list of URLs for the RSS feeds in the code itself. It makes more sense to put those in some sort of text file, so they’re easily maintained. It’s on my To-Do list. I also have a list of words to filter out so I don’t see stories I’m not interested in.

I’ll try to get those changes made and make the script easier for others to deploy on their own computer/network. The bandwidth makes no difference to me. I have a 1Gb fiber with no limit on download/upload.

[ernie (Ernest N. Wilcox Jr.)]

@nevj

I have a directory ( folder to you Win types) where I make multiple subdirectories for topics. Inside each topic directory there may be a list of url’s and some notes and maybe an occasional copy.

Wow! You’re far more organized than I am. All I do is keep notes with headings, organized alphabetically monthly, in a single, now very large, text file.

Ernie

[ernie (Ernest N. Wilcox Jr.)]

@pdecker

The bandwidth makes no difference to me. I have a 1Gb fiber with no limit on download/upload.

Wow! You have a better connection than I do. I have ATT’s Fiber 500, that’s probably about or a bit less than half of what you have, and I have no limits on download/upload either. If bandwidth is not an issue for you, then I’ll keep using your setup, and if st some point in the future, I decide to self host my own, I’ll get the source from you then. Is that OK with you?

Ernie

[callpaul.eu (Paul)]

@pdecker

I have a 1Gb fiber with no limit on download/upload.

I have the same and just received an offer to double the speed for the same price from orange waiting to see if it makes a difference but I suspect not it’s mainly for games and TV they are offering it I believe

[nevj (Neville Jackson)]

@callpaul.eu

@pdecker

have a 1Gb fiber with no limit on download/upload.

I have the same and just received an offer to double the speed for the same price

You guys are off the planet.
I get 100Mbps at the best, and there are lots in Australia worse off than that.

[Rosika (Rosika Schreck)]

@ernie :

Hi Ernie,

thanks for your praise.

Your approch is perfectly valid, of course.
And I´m glad it works so well with you. Thanks for your contribution.

Many greetings from Rosika

[callpaul.eu (Paul)]

@nevj

lots in Australia worse

That’s because you have to share the connection with all those sheep

Ha ha ha

[pdecker]

@callpaul.eu

double the speed

I’m not sure where we should draw the line and say it’s good enough. Many times, any slowness is on the source we are hitting and not our connection speed. It’s always more than fast enough for me. Maybe at some point I’ll save some money and drop back a tier. Looks like that only saves me $10/month.

@ernie

Is that OK with you?

Feel free to hit my feed. Like I said, it may not always be up and running.

Sometimes I think about using AWS or Digital Ocean or something to run things like this in the cloud. That way it’s always available. It’s about $5/month to do that. Not much, but not really needed either.

[callpaul.eu (Paul)]

@pdecker

save some money and drop back a tier

I did that with my mobile which I only use for calls not internet

[ernie (Ernest N. Wilcox Jr.)]

@pdecker

Sometimes I think about using AWS or Digital Ocean or something to run things like this in the cloud. That way it’s always available. It’s about $5/month to do that. Not much, but not really needed either.

Since you’re offering access to your aggregator at no cost to your fellow It’sFOSSers, unless too many of us are accessing it simultaneously, and are adversely affecting your bandwidth (or causing other issues), why spend money you don’t really have to? My recommendation is to keep things as they are as long as it works for you, and if anything changes, decide what you want to do then.

Ernie

[pdecker]

I just made a change. One of the feeds is in several languages and it broke the process for that feed. I put in a filter to fix that.