Skip to main content

Feeds

A native feed reader — RSS, Atom, Tumblr, and Are.na — with its own web tab and per-user SQLite store. There is no external service; the poller, the store, and the reader are all in-tree (src/istota/feeds/).

The feeds module is on by default. Opt out per user with disabled_modules = ["feeds"].

Reading​

The Feeds tab is a masonry card grid with an image/text filter, sort by published or added, grid and list views, a navigable image lightbox, and a click-to-expand reader overlay showing a card's full un-clipped content with ←/→ navigation between posts.

The sidebar scopes the view to everything, unread only, a single feed, or a whole category. The Unread and Starred views show their totals across the whole reader, and a feed with unread entries is set in bold with an unread mark beside its name. Per-entry starring is bound to f; bulk mark-as-read (Shift-A) honours whatever scope is active rather than clearing the whole account. Entries mark themselves read after 1.5 seconds in the viewport.

Feed, category, All, Unread and Starred selections are saved in the URL and browser history. Back and Forward restore earlier selections, and reloading or sharing the URL opens the same selection. Clicking the active feed or category returns to All and adds a history entry. Returning from settings keeps the selection; a feed or category that no longer exists falls back to All.

Video embedded in a post plays inline with ordinary controls. Nothing autoplays — a grid of cards all starting at once is not what scrolling a reader asks for — and the image/text filter hides inline video along with pictures, since a filter that left clips playing would only mean "some of the media". Embeds resolve through a host allowlist (YouTube, youtube-nocookie, Vimeo) rather than passing a provider's own iframe HTML through the sanitizer.

Repeat images​

As a reblogged photo travels through the blogs you follow, the same picture arrives several times. A duplicate inside one post is dropped, and across posts an image a newer entry already showed is hidden on the older ones — the post still appears, with a note counting what was hidden.

This is a display-time decision, never a row mutation, and it is bounded two ways: to a look-back window (image_dedupe_window_days, default 14, 0 = off) and to the slice you are currently viewing. An image resurfacing months later still shows, and browsing one blog never hides a tile because of something in another.

Subscriptions​

Manage subscriptions, categories, and OPML import/export from the sprocket-icon settings page, or from the skill CLI:

istota-skill feeds list # subscribed feeds
istota-skill feeds categories # categories
istota-skill feeds entries # entries
istota-skill feeds add URL # subscribe
istota-skill feeds remove ID # unsubscribe
istota-skill feeds refresh # mark feeds due for the next poll
istota-skill feeds poll # poll everything due, now
istota-skill feeds prune [--dry-run] # apply the entry retention policy
istota-skill feeds import-opml PATH
istota-skill feeds export-opml

run-scheduled is the scheduler's entry point, not something to run by hand. The scheduler looks every five minutes and queues it only when at least one feed is due. Unlike poll, it caps its burst at 50 feeds by default; the due list is oldest-first, so a larger backlog drains over consecutive ticks rather than being dropped. --limit overrides in both directions.

Entry retention​

Stored entries are bounded by two passes, both applied by prune and run daily by the auto-seeded _module.feeds.prune job. They protect different things and are not one rule:

  • Age deletes read and removed entries older than entry_retention_days (default 90). Starred entries, unread entries and anything the feed's most recent response still returned are exempt, and a feed is never taken below a floor of 50 entries. Age counts from when the entry arrived in this reader, not from its published date.
  • Maximum trims each feed to max_entries_per_feed (default 5000), keeping the most recently added. This pass does not read status, so an unread entry can be deleted here once a feed is over its maximum. Starred entries are the only exemption, and a feed's stars can never take its budget below 50 unstarred entries.

Either pass is switched off by setting its value to 0. On a database upgraded to schema v8 both are deferred for 90 days; the envelope reports that as entry_pruning_deferred_until and every count is zero. --dry-run runs both passes and rolls them back, reporting the counts a real run would delete.

Polling and rate limits​

Requests are paced per host, not per feed. A feed's poll interval says nothing about how many requests reach one host at once, and every Are.na channel you have is one request to api.are.na (every Tumblr blog one to api.tumblr.com), so a due list of twenty channels used to go out as twenty back-to-back requests and the tail of it was reliably turned away. A minimum gap of 2 s now separates two requests to the same host. RSS pays nothing for it — dozens of distinct origins — and two RSS feeds that genuinely share an origin are paced together, which is correct for the same reason. A second cap bounds the sleeping itself at 60 s per run: the poll is a background task holding one worker slot, and on exhaustion the run stops rather than un-pacing itself, leaving the rest due for the next tick.

A run also stops trying what does not answer. A fetch that cannot connect or times out marks its host unreachable for the rest of the run, so the other feeds on that host wait for the next tick instead of each waiting out a timeout. If two hosts are unreachable and none has answered, the run takes the network to be down and stops; on a working network that also happens when two dead feeds are first in line, and they are backed off before the next run. No new fetch starts once a run is three minutes old. In each case the feeds left untried stay due, with no error recorded against them. A new user's first poll fetches the default subscriptions, so on an install with no network it ends after two fetches.

HTTP 429 is a throttle, not a feed error. It does not write last_error, does not increment error_count and does not carry a doubled interval forward once it clears — being turned away for asking too often says nothing about whether the feed is broken. It is recorded in its own right instead: feeds.last_throttled_at (added in schema v6), cleared by the next successful fetch so the column means "throttled now" rather than "throttled once". GET /api/feeds/config reports it as a throttled_feeds count beside error_feeds, and per feed as last_throttled_at; the settings page does not render either yet. last_fetched_at is written back unchanged, because nothing was fetched.

The standoff never schedules a throttled feed sooner than a healthy one would poll: it floors at the largest of the feed's own interval, the 30-minute default and a 60-minute rate-limit backoff, and a server-named Retry-After is taken only where it is longer than that floor, capped at 6 hours. Every next_poll_at — success and failure alike — is multiplied by a random factor within ±10%, so a group of feeds refreshed together drifts apart instead of bursting, failing and rescheduling in lockstep forever.

A poll run reports throttled alongside errors, and per feed rate_limited and retry_after_seconds; a run that was turned away for every feed reports status: partial_error rather than reading as a clean poll that found nothing. It does not fail the scheduled task, because the throttle has already pushed the feed past any retry.

The known gap is that none of this spans users: a poll is a per-user subprocess, so two users reach one host from one IP with no shared budget.

A feed must be on a public address. The poll runs from the daemon's own network, so a feed URL that resolves to a private, loopback or reserved address is refused before any request is sent, and so is a redirect to one. The feed shows "The feed's address is not an http or https URL on a public host." as its error. To poll a feed on your own network, grant its exact host name to feeds in the operator config's [private_hosts] table:

[private_hosts]
"rss.home.example.org" = ["feeds"]

If the host is already in the table for another consumer, add "feeds" to its existing list: a second line for the same host is a TOML error.

Plain HTTP is allowed to any public host, and a user name and password in a feed URL are sent as Basic auth to that host only. Each fetch is capped at 16 MiB and 90 seconds, and follows at most five redirects. Tumblr and Are.na feeds are fetched from their fixed API hosts and are not affected.

Are.na runs on the v3 API. Six block types have typed builders (Text, Image, Link, Embed, Attachment, Channel) over a generic fallback, so a type Are.na adds later still renders rather than breaking the poll.

Storage​

Per-user feeds.db at Config.module_db_path(user_id, "feeds"), with tables feed_categories, feeds, feed_entries, entry_images, and schema_meta (schema v8). Settings live as rows keyed under feeds_settings.* — default_poll_interval_minutes, image_dedupe_window_days, entry_retention_days and max_entries_per_feed.

A poll refreshes an entry it already holds rather than discarding it, so a provider-side fix reaches entries already on file. User state (read status, starred, starred timestamp) is never overwritten, and a field is only replaced by a non-empty value, so a sparser re-fetch cannot blank a title you already have. The "N new" count still counts only genuinely new inserts.

  • Briefings — an rss briefing source reads from the same subscriptions.
  • Web interface — where the Feeds tab sits in the nav.