#speedups

This version brings a very useful feature: chunked export. Instead of waiting for the webserver to exhaust memory and never start transfering a 15-days export, you can now ask for nearly whatever you want and see it coming feed-by-feed.

Beware though: as exporting your entire database is possible, you could get into trouble either handling the resulting JSON file, or bringing your server to its knees, memory-wise.

You probably didn't know it yet about this export feature. My bad it was totally undocumented until today. That's because the feature is still a little rough for wide use.

As of now, it is designed around the use case of administrators exporting the whole database content (or a user folder) between 2 dates, to feed external systems with any kind of web-fetched contents.

If you're interested, the procedure is not straightforward, but still simple.

This version (and the quick 0.91 published a few days ago) brings some exciting features, and a handfull of fixes:

a new “excerpt to content” processor. It's best used on RSS feeds that produce already good content, avoiding the need to fetch the full article. In a few cases of them, the RSS content is not only already full, but ad-less, while fetching the article will clutter the content with unwanted things. With this processor (internally called 1fs-content-from-excerpt) you can completely bypass the downloading phase, and even the HTML to markdown (internally called content) phase. See doc for details. This processor is already included in the default chain, and activated for some web sites on 1flow.io.

some updates to processors and chains names, refreshing them to better reflect the features they offer.

dead-feeds refreshing: suggested long ago by kouio, we check every day some feeds that are considered dead, to see if they came back to life. Coupled to auto-reopen-on-external-subscribe feature, this should keep the dead number of feeds as low as possible on the platform, while avoiding to consume too much resources on feeds that are really dead.

documentation updates. The processing framework starts to mature to my taste, and the documentation tries to reflect this.

enhanced interface for staff members, regarding web sites and processing chains. parameters icons are now in a stronger color when parameters are set, in order for the manager to quickly see which web sites / chains have parameters set, and which not. I will probably code a dedicated query filter (with a shortcut button) soon.

the load_processors management command will now reset SQL sequences at the end of the process, thus allowing you to create processors/chains without problems. Before, in some conditions, you could not create custom local processors/chains if you had already loaded the 1flow processors pack.

I pushed the ported interface this morning. You should keep in mind that there WILL be some bugs. But as far as I could test it, I didn't encounter any (or fixed them before pushing).

The great news is a major speed up in reading lists. PostgreSQL does a very good job when looking up articles in feeds / subscriptions. Even with Django Polymorphic, multiple tables and millions of rows, the result is damn fast — and stable whatever the number of articles in the feed — compared to the same code in MongoDB. and there is no optimization yet (not even a prefetch_related()…).

You should now be able to further edit your folders & subscriptions, to assign them custom images if you want.

We have roughly 8.2 millions article migrated. the machine is still experience a quite heavy load, tasks wise (we have 3 millions of them in our various queues as I write this).

The only problem that arise because of this load is that new articles will not show up in your subscriptions until these tasks are processed. Fortunately, feeds continue to be refreshed nearly real-time, and new articles recorded in the database. Only their content and the article activation in user subscriptions is late.