Some details on the architecture and future high level plans for Socialhome.
Our current and future (dotted lines) components look something like this:
At the lowest level we have the database (PostgreSQL) and Redis based cache / queue storage. Search is currently powered by Django-Haystack + Whoosh.
On top of this we have the background jobs, which are powered by RQ.
In the middle sits Django.
To provide data for the frontend we have 3 solutions - WebSockets (powered by Channels), a REST API powered by Django REST framework and Django template engine itself.
For the frontend, we will have 3 solutions. Currently everything is Django templates. We will want to keep some of the pages as Django templates. For the streams (and possibly other pages), we want to create a Vue.js app. Additionally, mobile apps would be provided.
Component and feature notes¶
What we want to do is rewrite the streams as a more modern performant JS application. For the framework, discussion has been centering around Vue.js. The rationale is that Vue has the benefits of React.js with less overhead in learning curve and development time.
Socialhome code layout is split into logical Django apps based on the feature provided. The JS code should follow this pattern and live in the respective app. For example for the
streams Vue.js application, the following code layout would make sense:
socialhome/ streams/ app/ components/ (components) App.vue main.js templates/ streams/ app.html views.py
Basically the idea is that
views.py contains a Django view that loads the template inheriting from
base.html. The template then injects the Vue app, loading the stream. To speed up rendering we provide some initial stream data in the Django template context, then continuing everything via the REST API.
All the Vue apps build configuration should be on the top level of Socialhome, set up so all the apps build using the same
npm commands. Each Vue.js app should however generate its own JS bundle file.
All code should be allowed ES7 features, using Babel to transpile.
We should use standard testing tools for the Vue apps code, for example Karma + Mocha.
Since this is a huge task which cannot be done at once, the new Vue.js based streams will be provided in addition to the current streams served by Django templates. This could be done in phases:
- Alpha, little functionality - Render using Vue.js if a parameter ?vue passed in the url.
- Beta, most of the functionality present - Allow user to go to preferences and choose whether to see the new or legacy stream.
- Final, all functionality covered - Make Vue based streams default, removing the old streams code.
Search is currently powered by django-haystack as the framework and Whoosh as the engine. Whoosh is a pure Python backend with a file based search index. As performance requirements increase (for example full text content search), we should offer the option to use Elasticsearch as an optional search backend. Django-haystack supports both backends with just configuration changes. Whoosh should still be the default since it doesn’t require extra installations like Elasticsearch does.
The global search works as follows, in this order:
- Search by profile handle:
- If direct match found -> render profile
- If remote match found -> fetch and render profile
- Search all indexes for any matches
Currently a search index only exists only for profile objects. The plan is to add the following search indexes:
Profiles and tags can be easily listed in a list or table structure. Content would make sense to be rendered in a normal grid. This would make the search results page just another (dynamic) stream. See below mockup.
There are many streams in Socialhome. The main streams are user profiles, followed and the public stream, but basically each single content view is also a stream. Opening a reply in an individual window would also create a stream for that reply content. Additionally, we want users to be able to create custom streams according to rules. For example, a stream could be “followed profiles + tag #foobar + tag #barfoo”.
A stream should automatically subscribe the user using websockets and handle any incoming messages from the server (currently in
socialhome/stream/app/main.js), notifying the user of new content and adding it to the page on request (without a page load).
This basic design should be kept in mind when touching stream related code.
To make complex streams load fast, we precache them in Redis. The precache streams are updated on content save time.
Each stream has an Ordered Set for each user with the following data:
key = sh:streams:<stream_name>:<user_id> score = <time> value = <content.id>
Additionally, each stream has a Hash for each user with the “through ID’s”. A through ID is the content which caused the cached content to be added into the stream. Normally this would be the cached content itself, but for shares, this would be the share content ID. The Hash is as follows:
key = sh:streams:<stream_name>:<user_id>:throughs field = <content.id> value = <through content.id>
Only expensive streams are precached. This includes any stream which will pull up shares (for example “Followed” and “My content (all)”). Additinally any custom streams should always be precached for fast reads. An example of a stream which is not precached is the “Public” stream.