Architecture

Some details on the architecture and future high level plans for Socialhome.

Component map

Our current and future (dotted lines) components look something like this:

_images/architecture.png

At the lowest level we have the database (PostgreSQL) and Redis based cache / queue storage. Search is currently powered by Django-Haystack + Whoosh.

On top of this we have the background jobs, which are powered by RQ.

In the middle sits Django.

To provide data for the frontend we have 3 solutions - WebSockets (powered by Channels), a REST API powered by Django REST framework and Django template engine itself.

For the frontend, we will have 3 solutions. Currently everything is Django templates. We will want to keep some of the pages as Django templates. For the streams (and possibly other pages), we want to create a Vue.js app. Additionally, mobile apps would be provided.

Component and feature notes

Vue.js app(s)

The current streams JavaScript is largely based on lots of jQuery events modifying the DOM. Since the streams can grow to be quite big, this results in very bad performance. Additionally, the code is beginning to get hard to read and difficult to modify without regressions (“spaghetti code”).

What we want to do is rewrite the streams as a more modern performant JS application. For the framework, discussion has been centering around Vue.js. The rationale is that Vue has the benefits of React.js with less overhead in learning curve and development time.

Why not replace the whole frontend with Vue.js? Simply because we want to use the best job for each software area. Some of the pages will not benefit from being rewritten in JavaScript. Django templates are powerful and fast to develop. But some parts, like the streams, will benefit hugely from features like the virtual DOM provided by Vue, in addition to allowing cleaner JavaScript code base.

Code layout

Socialhome code layout is split into logical Django apps based on the feature provided. The JS code should follow this pattern and live in the respective app. For example for the streams Vue.js application, the following code layout would make sense:

socialhome/
  streams/
    app/
      components/
        (components)
      App.vue
      main.js
    templates/
      streams/
        app.html
    views.py

Basically the idea is that views.py contains a Django view that loads the template inheriting from base.html. The template then injects the Vue app, loading the stream. To speed up rendering we provide some initial stream data in the Django template context, then continuing everything via the REST API.

All the Vue apps build configuration should be on the top level of Socialhome, set up so all the apps build using the same npm commands. Each Vue.js app should however generate its own JS bundle file.

Code style

For the new Vue.js based JavaScript we should follow the popular Airbnb guidelines with the following exceptions:

  • No semicolons. This is a Python project, we can go for more Pythonic looking JavaScript.

All code should be allowed ES7 features, using Babel to transpile.

Tests

We should use standard testing tools for the Vue apps code, for example Karma + Mocha.

Timeline

Since this is a huge task which cannot be done at once, the new Vue.js based streams will be provided in addition to the current streams served by Django templates. This could be done in phases:

  1. Alpha, little functionality - Render using Vue.js if a parameter ?vue passed in the url.
  2. Beta, most of the functionality present - Allow user to go to preferences and choose whether to see the new or legacy stream.
  3. Final, all functionality covered - Make Vue based streams default, removing the old streams code.

Tracking issue

The Vue.js streams rewrite is tracked in this issue.

Streams

There are many streams in Socialhome. The main streams are user profiles, followed and the public stream, but basically each single content view is also a stream. Opening a reply in an individual window would also create a stream for that reply content. Additionally, we want users to be able to create custom streams according to rules. For example, a stream could be “followed profiles + tag #foobar + tag #barfoo”.

A stream should automatically subscribe the user using websockets and handle any incoming messages from the server (currently in socialhome/static/js/streams.js), notifying the user of new content and adding it to the page on request (without a page load).

This basic design should be kept in mind when touching stream related code.

Stream templates

Note

This section relates to the old Django templates + jQuery stream. For the Vue.js streams, see above.

Content in streams in is visualized mainly as content grid boxes. This includes replies too, which mainly use the same template code.

There are a few locations to modify when changing how content is rendered in streams or the content detail view:

  • socialhome/streams/templates/streams/base.html - This renders the initial stream as a basic Django template on page load.
  • socialhome/streams/templates/streams/_grid_item.html - Renders actual content item in initial stream and content detial.
  • socialhome/static/js/content.js - This is the main JavaScript template which is used to insert content into the stream. This is used for both top level content and replies in content streams.

All these templates must be checked when any content rendering related tweaks are done. Note however that actual content Markdown rendering happens at save time, not in the templates.

Precaching

To make complex streams load fast, we precache them in Redis. The precache streams are updated on content save time.

Each stream has an Ordered Set for each user with the following data:

key = sh:streams:<stream_name>:<user_id>
score = <time>
value = <content.id>

Additionally, each stream has a Hash for each user with the “through ID’s”. A through ID is the content which caused the cached content to be added into the stream. Normally this would be the cached content itself, but for shares, this would be the share content ID. The Hash is as follows:

key = sh:streams:<stream_name>:<user_id>:throughs
field = <content.id>
value = <through content.id>

Only expensive streams are precached. This includes any stream which will pull up shares (for example “Followed” and “My content (all)”). Additinally any custom streams should always be precached for fast reads. An example of a stream which is not precached is the “Public” stream.