This page is a stub

The Wazimap-NG frontend uses a custom component architecture. Over time, this will likely evolve to use a more standard framework such as web components.

Observable Abstract Class

All components extend the Component abstract class which itself extends the Observable abstract class. Observableimplements the Observer design pattern. Extending it gives a class the ability to register listeners and trigger events [Note, we may move to use native CustomEvents if we adopt the web component architecture]. Events are arbitrary string identifiers although they are often namespaced to the component itself.

For instance, when a dropdown element is selected, a dropdown component might fire the dropdown.selected event.

Here's an example:

class DropDown extends Observable {
    static EVENTS = {
        selected: 'dropdown.selected' 
    }
    
    constructor() {
        super();
    }
    
    onElementSelected() {
        let payload = ...
        this.triggerEvent(DropDown.EVENTS.selected, payload)
    }
}

The payload is any arbitrary object and datatype that the class seeks to send to any listeners on that event.

Calling code might look something like this:

let dropdown = new DropDown();
dropdown.on(DropDown.EVENTS.selected, payload => alert('An element has been selected')

whenever the onElementSelected method is called, every listener will receive the payload.

Component Abstract Class

Component extends Observable and adds a parent-child relationship enabling components built from other components.

Instead of our Dropdown extending Observable, it now extends Component.

class DropDown extends Component {
    static EVENTS = {
        selected: 'dropdown.selected' 
    }
    
    constructor(parent) {
        super(parent);
    }
    
    onElementSelected() {
        ...
    }
}

The calling code would look like this:

class Application extends Component {
    constructor() {
        super()
        this._dropdown = new DropDown(this);
        
        this.registerChild(this._dropdown);
    }
}

Doing this simply stores the parent and child components. In future this may be used to bubble events or in someway communicate across the entire component hierarchy. [Note, the Component class is still being developed].

Building Components

Apart from extending Component Wazimap-NG components use a Model View Controller design. The view comprises an HTML fragment, usually created by the web designer. [These are currently embeded in index.html but may in future be separated into individual component html files.]

The model stores the state and associated logic of the component and the controller acts as the glue between model and view, as well as handling events and interactions.

Controller

Here is a basic implementation of the DropDown component.

// This class implements the controller for the DropDown component.
export class DropDownComponent extends Component {
    constructor(parent, container, values = [], defaultText = '') {
        this._container = container
        this._model = new DropDownModel(values, defaultText);
        this.prepareDomElements();
        this.prepareEvents();
    }
    
    get container() {
        return this._container;
    }
    
    get model() {
        return this._model;
    }
    
    prepareDomElements() {
        this._textArea = $(this.container).find('.textArea')[0];
        this._trigger = $(this.container).find('.trigger')[0];
        this._optionContainer = $(this.container).find('.options')[0];
        this._listItem = $(self._optionContainer).find('.list_item')[0]
    }
    
    prepareEvents() {
        const self = this;
        this._trigger.on('click', () => {
            $(self._optionContainer).show();
        })
        this.model.on(DropDownModel.EVENTS.changeValue, model => {
            self.updateText();
        })
    }
    
    setOptions(values) {
        const self = this;
        this.model.options = values;
        $(this._optionContainer).html('');
        values.array.forEach((element, idx) => {
            let li = self._listItem.clodeNode(true);
            li.text(element);
            $(self._optionContainer).html('');
            let li = listItem.cloneNode(true);
            li.on('click', () => {
                self.model.currentIndex = idx;
            })
            $(self._optionContainer).append(li);
        });
    }
    
    updateText() {
        $(this._textArea).text(this.model.currentValue)
    }
    
    reset() {
        this.model.reset();
    }
}

A few points worth noting:

constructor(parent, container, values = [], defaultText = '')

All components receive their parent component as the first argument. In many cases the second argument is the DOM element that contains the view HTML.

constructor(...) {
    ...
    this.prepareDomElements();
    ...
}
    
...

prepareDomElements() {
    this._textArea = $(this.container).find('.textArea')[0];
    this._trigger = $(this.container).find('.trigger')[0];
    this._optionContainer = $(this.container).find('.options')[0];
    this._listItem = $(self._optionContainer).find('.list_item')[0]
}
    
...

By convention, the constructor calls prepareDomElements which prepares various DOM elements used in the component. Note how each of the lines in prepareDomElements ends in [0]. This is due to the fact that JQuery's find method returns a JQuery object. By convention, DOM objects are used in models, these can easily be converted into JQuery objects using the $ function, e.g. $(this._textArea).

A prepareEvents method is often also used to wire up events from the constructor.

constructor(...) {
        ...
        this.prepareEvents();
    }
    
    prepareEvents() {
        const self = this;
        this._trigger.on('click', () => {
            $(self._optionContainer).show();
        })
        this.model.on(DropdownModel.EVENTS.changeValue, model => {
            self.updateText();
        })
    }

It's is important to note that only the web designer edits the HTML directly. If addition DOM elements are needed, the component will clone existing elements:

setOptions(values) {
        ...
        values.array.forEach((element, idx) => {
            let li = self._listItem.clodeNode(true); // <-- Note that the li element is being cloned from an existing one.
            li.text(element);
            ...
            $(self._optionContainer).append(li); // <--- Note how the li element is inserted into the DOM.
        });
    }

Model

While it is tempting to create the Component using a single class, separating the model from the view often results in cleaner, more robust, and easier-to-test code. The model only stores the state of the component and does not interact with either the view or the controller directly.

It is important that the model is agnostic of the controller. In other words, the model should not hold a reference to the controller and should not call its methods. Communication takes place using either events or callbacks. While not mandatory, events are preferred as callbacks can result in slightly less-readable code.

In the case of a DropDown component, the state that needs to be stored includes:

• Current options available in the dropdown

• The currently selected item

• Default text to display if no item is selected.

class DropDownModel extends Observable {
    static EVENTS = {
        changeValue: 'dropdown.model.changevalue'
    }
    
    constructor(options = [], defaultText = '') {
        this._options = options
        this._defaultText = defaultText;
        this._currentIndex = null;
    }
    
    get options() {
        return this._options;
    }
    
    set options(values) {
        this._options = values;
        this.reset();
    }
    
    get defaultText() {
        return this._defaultText;
    }
    
    get currentIndex() {
        return this._currentIndex;
    }
    
    set currentIndex(newIdx) {
        if (newIdx != this._currentIndex) {
            this._currentIndex = newIdx;
            this.triggerEvent(DropDownModel.EVENTS.changeValue, this)
        }
    }
    
    get currentValue() {
        if (this.currentIndex == null)
            return this.defaultText;
        return this.options[this.currentIndex];
    }
    
    reset() {
        this.currentIndex = null;
    }
}

most of the model class looks like boilerplate, the most interesting method is the setter for currentIndex

set currentIndex(newIdx) {
    if (newIdx != this._currentIndex) {
        this._currentIndex = newIdx;
        this.triggerEvent(DropDownModel.EVENTS.changeValue, this)
    }
}

Here we check if the index of the selected item has changed, if so, it is update and a DropDownModel.EVENTS.changeValue is fired. Recall from the DropDownComponent class above, we add a listener to this event:

prepareEvents() {
    ...
    this.model.on(DropDownModel.EVENTS.changeValue, model => {
        self.updateText();
    })
    ...
}

which then gets the appropriate value from the model:

updateText() {
    $(this._textArea).text(this.model.currentValue)
}

The current index value on the model is updated when the li element is clicked:

setOptions(values) {
    ...
    li.on('click', () => {
        self.model.currentIndex = idx;
    })
    ...
}

MVC Architecture

The benefit of using a decoupled MVC framework is that the resulting code is uncomplicated and reusable. Using this structure, it would be possible to have two controllers working with a single model. When a user interacts with the first component, the second one will automatically update.

To ensure that the Model is decoupled from the Controller, it is usually best to separate events into two types, user interaction events and state change events. A user clicking on a box is an example of a user interaction event. They are typically primitive events. The controller notifies the model of the event by calling a method on the model object. This method will likely modify the internal state of the model. Once the state changes, a state change event is fired from the model. An example is the changeValue event. These events usually operate at a higher semantic level. The controller receives the state change event and updates the view accordingly. In this case, the text shown.

A few points on style

Prefer using the _ prefix as a signifier of a private attribute, e.g.

constructor(options = [], defaultText = '') {
        this._options = options
        this._defaultText = defaultText;
        this._currentIndex = null;
    }

Use getters and setters

get options() {
        return this._options;
    }
    
    set options(values) {
        this._options = values;
        this.reset();
    }

Where possible provide the simplest possible arguments to methods, e.g.

doSomething(data) {
    console.log(data.key1.key2.value) // bad 
}

doSomething(value) {
    console.log(value);
}

Methods should receive only information that they need. Large application-wide state objects should be avoided.

Wazimap-NG is multi-tenanted. A single backend can host multiple profiles, e.g. and both use the same server and database.

In order to determine which profile to use, the client sends a wm-hostnameheader with this api call: /api/v1/profile_by_url/?format=jsonthis is received by the server which then matches the hostname with available profiles. You can determine which profiles are currently served by a particular backend using the following url: /api/v1/profiles/. It will return a list of profiles with their configurations, e.g.

In this case, when the server receives wm-hostname set to geo.vulekamali.gov.za, it returns profile 2. A single profile may match multiple urls.

This is the technical handbook for

This is intended for

• Profile administrators - how to configure profiles

• Developers - how we work on this project.

Profile administrators should see the .

The Django apps and database models are divided into roughly two groups: models that store data, and models that are used to present information to the end-user.

Datasets App

Dataset and DatasetData

Only the Geography, Count, and at least one additional column are required. An example table might look as follows:

When this file is uploaded, a new Dataset object is created. Each row is stored in a DatasetData object. A typical DatasetData object might look as follows:

All groups and the Count column are stored in a JSONField.

Indicators and IndicatorData

Another key concept is an Indicator. Indicators represent saved aggregations and filters on a dataset. For example, the above Dataset can be used to create an Indicator containing population per geography disaggregated by gender. The equivalent query in SQL would look something like this:

Similarly, another indicator can be created to return population disaggregated by age.

When a new indicator is created, data from DatasetData is processed to create an IndicatorData object, one per geography. A simplified version of and IndicatorData would like something like this:

Universe

Universes represent saved filters on queries and enable the Data Administrator to run a query on a subset of the database. The default Universe is the total of all the distinct observations in a geography (e.g. the total population of the geography). It is possible to create a custom Universe and apply it to an Indicator.

A Universe which creates a filter on gender can enable queries on Female exclusively. PseudoSQL to represent this operation

The Universe filters field contains a dictionary that will be used in a Django ORM filter method. Below is an example filter to extract adults 60 and older.

This filter is then passed to the Django ORM as follows:

Profile App

ProfileIndicator, ProfileKeyMetrics, and ProfileHighlight. ProfileIndicator is the most commonly used of the three.

ProfileKeyMetrics display only a single value from an Indicator. For instance, the number of youth between 15-24 living in the area.

ProfileHighlights are similar to ProfileKeyMetrics in that they display a single value from an Indicator, but are displayed in the Map View rather than the Rich Data View.

Points App

1. Do no change existing class names (unless strictly necessary to implement a request)

2. If a class name is change, ensure that change is discussed with a developer and documented in the Gitbook.

3. When making sweeping changes to an already existing component, be sure to duplicate the original component and mark the new component with a versioned name so that any changes do not break the production site. eg. ".map-options" -> ".map-options--v2".

4. Ensure that any change is documented in the Gitbook.

Below is an example of an Indicator Data object

'groups': {'race': {'Other': [{'count': 14034.3619100001,
     'region of birth': 'Other'},
    {'count': 23014.7457499993, 'region of birth': 'Rest of africa'},
    {'count': 34546.705719999, 'region of birth': 'Sadc'},
    {'count': 33844.2467799998, 'region of birth': 'South africa'},
    {'count': 8627.21382000001, 'region of birth': 'Unspecified'}],
   'White': [{'count': 17755.8348600004, 'region of birth': 'Other'},
    {'count': 1076.82088, 'region of birth': 'Rest of africa'},
    {'count': 19345.9559600004, 'region of birth': 'Sadc'},
    {'count': 1175191.32683901, 'region of birth': 'South africa'},
    {'count': 9675.18685000005, 'region of birth': 'Unspecified'}],
   'Coloured': [{'count': 1104.4229, 'region of birth': 'Other'},
    {'count': 842.120029999999, 'region of birth': 'Rest of africa'},
    {'count': 5666.85164000002, 'region of birth': 'Sadc'},
    {'count': 1575281.6271999, 'region of birth': 'South africa'},
    {'count': 1675.21038, 'region of birth': 'Unspecified'}],
   'Black african': [{'count': 9463.15047999995, 'region of birth': 'Other'},
    {'count': 68003.2315900028, 'region of birth': 'Rest of africa'},
    {'count': 810647.802930328, 'region of birth': 'Sadc'},
    {'count': 14739761.6385047, 'region of birth': 'South africa'},
    {'count': 85310.3884700055, 'region of birth': 'Unspecified'}],
   'Indian or asian': [{'count': 38132.814809999, 'region of birth': 'Other'},
    {'count': 5076.67313999999, 'region of birth': 'Rest of africa'},
    {'count': 2539.20952000001, 'region of birth': 'Sadc'},
    {'count': 414713.136599942, 'region of birth': 'South africa'},
    {'count': 5150.74462, 'region of birth': 'Unspecified'}]},
  'gender': {'Male': [{'count': 55748.008589999, 'region of birth': 'Other'},
    {'count': 72957.3858900018, 'region of birth': 'Rest of africa'},
    {'count': 517617.920130245, 'region of birth': 'Sadc'},
    {'count': 8727584.65349521, 'region of birth': 'South africa'},
    {'count': 63433.5451700032, 'region of birth': 'Unspecified'}],
   'Female': [{'count': 24742.5763700004, 'region of birth': 'Other'},
    {'count': 25056.2055000003, 'region of birth': 'Rest of africa'},
    {'count': 355128.605640082, 'region of birth': 'Sadc'},
    {'count': 9211207.32242836, 'region of birth': 'South africa'},
    {'count': 47005.1989700023, 'region of birth': 'Unspecified'}]},
  'age group': {'15-19': [{'count': 6298.26723000001,
     'region of birth': 'Other'},
    {'count': 6312.14378000002, 'region of birth': 'Rest of africa'},
    {'count': 68137.1359800096, 'region of birth': 'Sadc'},
    {'count': 4775821.0196088, 'region of birth': 'South africa'},
    {'count': 9359.01092000012, 'region of birth': 'Unspecified'}],
   '20-24': [{'count': 16982.6305099999, 'region of birth': 'Other'},
    {'count': 22846.8292099995, 'region of birth': 'Rest of africa'},
    {'count': 229155.41202007, 'region of birth': 'Sadc'},
    {'count': 4684128.15334481, 'region of birth': 'South africa'},
    {'count': 27011.4452600009, 'region of birth': 'Unspecified'}],
   '25-29': [{'count': 26769.6408399996, 'region of birth': 'Other'},
    {'count': 34202.7678200008, 'region of birth': 'Rest of africa'},
    {'count': 309901.929190191, 'region of birth': 'Sadc'},
    {'count': 4354313.53260714, 'region of birth': 'South africa'},
    {'count': 37417.0233700027, 'region of birth': 'Unspecified'}],
   '30-35': [{'count': 30440.0463799999, 'region of birth': 'Other'},
    {'count': 34651.8505800018, 'region of birth': 'Rest of africa'},
    {'count': 265552.048580056, 'region of birth': 'Sadc'},
    {'count': 4124529.27036281, 'region of birth': 'South africa'},
    {'count': 36651.2645900019, 'region of birth': 'Unspecified'}]}},
 'subindicators': {'Sadc': 872746.525770327,
  'Other': 80490.5849599994,
  'Unspecified': 110438.744140006,
  'South africa': 17938791.9759236,
  'Rest of africa': 98013.5913900021}}

Your code will be reviewed. And you can can take steps to make it easier for the reviewer and yourself.

What is the job of the Review?

The review is there to improve the code quality and to ensure architectural direction and correctness.

The big benefit of a review is to help you get better. You will learn from Code Reviews.

Everybody's code can be improved. And your code will get better with reviews. Programming is a skill that improves with training.

Code Reviews help reveal implicit knowledge that is not expressed in code.

Four eyes see more than just two. Even the best testing does not lead to 100% bug-free code. It helps, but Code Reviews are a big part of providing confidence in the code.

Who can/should approve a PR

Everyone can submit their review and we encourage everyone to do so. But only the PM and the Lead Developers can approve a PR.

Who can merge a PR?

Once the PR is approved anyone can merge the PR. And you are encouraged to merge the PR once it is approved.

How to create a good PR

Create a PR as early as possible, and push frequently.

Open a Pull Request (PR) once you decide to work on an issue.

Use the --allow-empty flag with git to create an empty commit and push it to create a PR. Use the branch naming as described in the git handbook.

Create a Draft Pull Request to signal, this is a work in progress. Change to Ready to review once you're confident that it's ready to be reviewed.

If you share your progress early on, the reviewer can take a glance at your work early, and help you move in the right direction.

Use the PR template

The template is there to help you and the reviewer to make the process as easy and fast as possible.

Write a good description

The description should reference the issue that the PR is addressing. It should include, in your own words, what you are trying to accomplish. How you understand the issue. It will help the issue creator and the reviewer to better understand and address potential misunderstandings early on.

The description should also include your thinking process. How do you approach this issue.

And It should include how you went about testing this locally. For the frontend which buttons to click on which pages, for the backend which API endpoint to call and how.

As well as anything you’d like the reviewers opinion on specifically (eg class names, time complexity of code, etc)

Run quality checks locally

Run codeclimate locally and address issues early on. It will show you issues that will be raised once you push the code.

Run the tests locally

Make sure you run the tests locally (no need to run E2E) to make sure they pass.

Only mark the PR to ready for review, when you truly think it's ready

Don't mark it ready too soon, when you still working on it or did not push the latest changes. Take a look if the code diff looks like you would expect.

Leave your own review

If you wanna be proactive leave your own review and comment on the lines that you think might be worth explaining to the reviewer. You will also find issues you didn't notice while you were coding when you take a reviewer perspective.

djfldsjfdjfdfkdsl

• read through story - what's the point of the change?

• read through PR description

  • what change is the dev trying to make?

  • what tests does this rely on to keep working? which ones are new?

• run the tests - do the tests actually pass?

• review the code

  • Does the code fit in with the context?

  • Is the code of sufficient quality?

  • Does anything look scary? Risky?

  • Does anything not look right?

  • Does it solve the problem sufficiently generally or is it too specific?

  • Have we considered the

• resolve conflicts if possible

• approve

More QA

• For all the different sorts of data, will this work?

• For all the different sorts of silly things users and admins can do, will this work?

PO review process

• Does it address the business problem?

Description

Related Issue

How to test it locally

Screenshots

Changelog

Added

Updated

Removed

Checklist

• 🚀 is the code ready to be merged and go live?

• 🛠 does it work (build) locally

Pull Request

• 📰 good title

• 📝good description

• 🔖 issue linked

• 📖 changelog filled out

Commits

• commits are clean

• commit messages are clean

Code Quality

• 🚧 no commented out code

• 🖨 no unnecessary logging

• 🎱 no magic numbers

• ⚙️ ran jslint

• 🧰 ran codeclimate locally

Testing

• ✅ added (appropriate) unit tests

• 💢 edge cases in tests were considered

• ✅ ran tests locally & are passing

The central unit of analysis in NG is a geography. These represent spatial boundaries which can be associated with data that describes it. For example, a typical geography may represent a Province. Associated data may include demographics in the area, the crime rate, economic activity, education, or any other arbitrary data.

Geographies are related to each other in a tree-like structure (strictly a directed acyclic graph) where each geography in a level is either a root node or has exactly one parent.

The most common hierarchy that we use for the South African context starts with Country at the top. This is followed by Provinces, all of which are non-overlapping and completely contained with Country. Below Province we find District, Municipality, Mainplace and Subplace. Another path may be Municipality -> Ward.

The assumption about the geographical hierarchy is as follows:

1. It is a directed acyclic graph (DAG)

2. Child geographies are completely contained by their parent.

3. Sibling geographies do not overlap.

4. Each geography has only one parent.

Forks can occur at any part of the hierarchy and are dependent on the current geographical hierarchy in use. For the purpose of illustration, here is a more extensive hierarchy:

Each Geography in a hierarchy should have the follow four fields:

Code - An identifier (not necessarily unique). Where possible, this code should follow an existing standard such as the ISO 3166 for countries.

Boundaries - A spatial boundary definition.

Parent Geography - A reference to its parent, or null if it is a root geography. This reference should use the parent’s code.

Version - For a number of reasons, boundaries may change while codes remain the same. For instance, municipal boundaries in South Africa change every 5 years based on population growth and migration patterns. In some cases, old municipalities may disappear and new ones are created. In other cases minor boundary changes are made to existing geographies. In these cases, it is important to record an identifier that reflects this ‘version’ change, even when the code remains the same.

The approach in NG is to define a geographical hierarchy which then associates the various levels. When a request is received for a particular geography, it sends a list of its children to enable drilling downwards into the hierarchy.

Hierarchical Structure

Not linear

Hierarchies are not linear but are usually considered to be tree-like although strictly directed acyclic graphs. This means that one parent can have many types of levels - e.g. Municipality may have Wards and Mainplaces. These levels overlap. Less common is for a child to have multiple parent levels, e.g. A Ward can be a child of either Metro or a Municipality.

No universal levels

There is no assumption that every parent level will have the same child levels. For example, when exploring a world geography, each country may have different levels specific to it.

No wall-to-wall coverage

Boundaries do not need to cover the entire area of their parent, holes are possible.

Multiple simultaneously display levels

Many levels are not compatible with each other as they overlap, Wards and Mainplaces are one such example. There are however instances where they do not overlap such as Districts and Metros. In each case, it is up to the client to decide how these should be rendered.

Technical Details

Backend

Treebeard is used to manage the Geography model. Then enables fast querying of hierarchies which is usually hard to do in naive SQL table structures. Spatial data is stored in GeographyBoundary objects. Geographies are serialized to GeoJSON before being served to the client.

Compression

Overly detailed geographies can result in large downloads. The GeographyBoundary models uses a CachedMultipolygonField to automatically compress boundaries everytime the model is saved. Downloads are still often large and can be reduced further by converting to TopoJSON. A custom serializer needs to be written to do this. Leaflet will also need a plugin to be able to convert TopoJSON to GeoJSON.

Frontend

Geographies are displayed on a Leaflet map drawn using Canvas. The client receives a preferred_children configuration key which provides guidance for which level to display when there are multiple options. For example:

"preferred_children": {
    "country": [
      "province"
    ],
    "district": [
      "municipality",
      "mainplace",
      "ward"
    ],
    "province": [
      "district",
      "municipality"
    ],
    "mainplace": [
      "subplace"
    ],
    "municipality": [
      "mainplace",
      "ward"
    ]
  }

every key represents a level, following by an ordered list of children that it can display. A municipality has two types of children mainplace and ward. According to this configuration, mainplaces should be preferred to wards and will be displayed as default. A toggle exists on the user interface to switch between geographies. If data for a particular level is not available then that should not be shown to the user.

See also

Frontend code merge

1. Developer creates a draft PR

2. Developer completes code and removes the draft label from the PR.

3. The Trello ticket is moved from In progress to Code Review.

4. The lead dev merges the latest staging into the PR and reviews. Once the code is ready, the PR is updated with the merged code.

5. The Trello ticket is then moved into Review (Product Owner)

6. The Product Owner reviews the Netlify deploy preview. Once the implementation is approved, the Trello ticket moves to To be deployed

7. The Lead Developer merges the PR into staging

8. The Product Owner reviews the staging server (on a scheduled, or ad hoc basis)

9. Once approved, staging is merged into master which is then pushed to the production server.

This process allows to PO to review a feature before it is merged into the staging branch thereby removing the need to rollback a PR that isn't approved by the PO. One gap in this process if a bug enters the codebase during merging into staging. This can occur in the following cases:

1. PR 1 is approved, PR 2 is approved, PR 1 is merged in staging, PR 2 is merged into staging. The PO has not had the opportunity to review PR 1 and PR 2 at the same time.

2. Resolution of conflicts.

General

Rationale for recent structure update: The latest version of wazimap has been reworked to ensure we can include the following features:

1. In-page anchor linking and page navigation

2. A component based approach to populating pages

3. An improved mobile experience

4. Filtering of sub-indicator

Styles section

Rich Data Panel

Every other element is arranged in relation to this panel (using position fixed). This is so that we can maintain in page anchor linking and page navigation.

1. Rich data nav items

These items sit on the left hand side of the rich data panel and act as anchors to jump to different content. They also allow users to know which section they are currently in.

How to implement this component:

Call .rich-data-nav__item into .rich-data-nav__list

Once you're strong enough, save the world:

1. Obtain the latest Webflow export from

2. Run path-to-webflow-export.zip

3. Check that everything works as expected

   1. Start the local dev server and try it

   2. Run automated tests

4. Add all the changes to git, commit and push to your branch.

Fixing conflicts

If you have conflicts with webflow-controlled files, these can be fixed as follows:

1. Fix any conflicts in non-webflow-controlled files as usual.

2. Import the latest webflow export.

3. Test the changes, and add all the changes to git and commit as usual.

This only works when the latest webflow export is safe to import, which should always be the case, by versioning components with breaking changes.

2022-11-22 - Powered by Wazimap

• Added "Powered by Wazimap" watermarks to map, point mapper, data mapper and rich data.

2023/06/26 - Removal of tutorial button in nav

02/15/2023 - Various mobile UI changes

• Made the map chips scroll on mobile landscape

• Adjusted positions of all panel toggles (except my views) on mobile landscape

• Adjusted max width of data mapper and point mapper to ensure they fit on mobile. (max-width: 83vw)

02/14/2023 - Rich data content max width

2022-12-05 - Data mapper content list wrapper

2022-12-02 - Powered by Wazimap Removed

2023/03/17 - Unnecessary code removal

2023/03/16 - Location tag scroll bar fix

• Adjusted the alignment of the "map-location__tags" to fix the scrolling issue on mobile

• Adjusted the width of "location-highlight" to not be a fixed width and cause wrapping issues

2023/03/15 - Various mobile changes

• Made map location chips panel overflow with items aligned right so that the current filter shows first

• Made the .map-bottom-items and map-bottoms-items have a the correct width and padding to make sure its contents does not overlap with other items on small screens. The only thing i couldnt test was whether the zoom buttons are working fine after this change. This seemed like a better way of handling the “map-options” change suggested as this would ensure no items in that parent div overlap rather than just 1.

• Positioning of the left side panel toggles is fixed on may side, they are lower and dont hit the map location chips

• Changed the position of the geo select and its z index. This should be changed to be more dynamic and responsive to the state of other items at the bottom, but for not it should make the mobile experience better.

• Set max width on rich data content as per suggestion.

10/11/2021 - Filter truncation

Changes:

• Added truncation to long filters

• Removed generic-modal

10/06/2021 - Cluster scroll fix

Changed:

Added a element .facility-tooltip__scroll inside .facility-tooltip is--cluster which has overflow: auto to accommodate a long scrolling list.

10/06/2021 - Point mapper truncation fix

Ensured that point mapper labels are correctly truncated.

10/01/2021 - Map data version support

Generic Modal:

09/23/2021 - Facilities loading state

Fixed:

• Fixed a small bug where the facilities facet number was being pushed too far left if there was not a download button visible. Now, if the download button is hidden, the facet will align correctly with the right side of the panel. This fix was imlemented by changing the display of the parent from display: grid to display: flex

Changed:

• Added loading state (hidden) for the location facilities title (.location__facilities_title--loading)

• Added loading state (hidden) for hide/show facilities button (.location-facilities__trigger--loading)

• Added loading state (hidden) for location facility block items (.location-facility__item--loading)

09/16/2021 - Point mapper dropdown icon colour fix

Changed:

• Fixed what colour is automatically inherited for the point mapper dropdown headers so that it is not overridden with grey when using a theme colour.

09/16/2021 - Cluster tooltip

Added:

• .facility-tooltip is--cluster to the styles panel for use when clustering nearby points on the map.

09/15/2021 - Point mapper dropdown and filters

Point mapper dropdown with toggle:

Point filters:

Changes:

• Added .point-mapper__h1_checkbox that is hidden by default

• Added .point-filters panel in .map-bottom-items (hidden by default)

08/31/2021 - Fix for tutorial modal height

Changes:

• Restricted max height to 90vh on the tutorial modal

08/25/2021 - Rich Data Nav Fix

Changed:

• Made the section nav scroll when height is restricted

08/12/2021 - Print CSS test

Changed:

• add the following code to the print css:

@media print {
  .page-break-before {
    break-before: page;
  }

  body, .main, .rich-data, .rich-data-content, .rich-data-content .section, .profile-indicator{
    float: none !important;
    display: block !important;
  }

  .rich-data__print, .tab-notice{
    display: none;
  }

  .rich-data[style*="display: none;"]{
    display: none !important;
  }

  .facility-info[style*="display: none;"]{
    display: none !important;
  }

  .facility-info[style*="display: flex;"]{
    display: inline-table;
    position: absolute;
    width: 100%;
    height: 100%;
    top: 0;
    left: 0;
    background-color: #fff;
    max-height: 100%;
    max-width: 100%;
  }

  .facility-info__view-google-map, .facility-info__print, .facility-info__close{
    display: none;
  }
}

07/19/2021- Point-legend, table margin, z-depth bottom-items

Changed:

• Added margin-bottom to .profile-indicator__table_inner to fix spacing issues in the rich-data panel

• z-depth: 999 added to items within the .map-bottoms-items panel

• Added .point-legend__remove to the .point-legend contained within the .map-print panel and added a .hidden class to it. It has also been moved further down the DOM so that the .point-legend that gets copied is the correct one.

• Added .hidden class to .data-category__h1_icon by default to avoid incorrect icons

• Added .slide-info__introduction to every tutorial slide

07/13/2021- Fixed rich data filter buttons to resemble map-options

Changed:

• Reverted to old implementation of .map-options__filters

• Changed .profile-indicator__new-filter to be a wide button to replicate the map-options button

07/12/2021- Remove map options tooltip and change filter buttons

Changed:

• Removed the indicator context tooltip from the bottom right of the map-options panel

• Changed the way we implement filter buttons in the map-options panel to bring it in line with the rich-data panel

• Added .mapping-options__filter-buttons with the following buttons:

  • .mapping-options__new-filter

  • .mapping-options__remove-filter (hidden by default)

07/05/2021- Disabled state for map download button

• Added .disabled state to .map-download button by default.

• Remove this class to set state to enabled.

07/05/2021- Removed filter disabled

• Removed the .disabled class from .dropdown-menu__trigger

• The disabled class is now only on .mapping-options__filter

• Added code to disabled filters that have the disabled class:

.mapping-options__filter.disabled {
	pointer-events: none;
  }

07/02/2021 - Map options filters

Changed:

• Added .map-options__filters_content back into .map-options__filters in the .map-options panel.

• .map-options__filters_content is hidden by default using the .hidden class

06/28/2021 - Table content truncation

Changed:

• Added code to force truncation on the table cells

  .profile-indicator__table_cell {
			width: 100%;
			white-space: nowrap;
			overflow: hidden;
			text-overflow: ellipsis;
	}

06/28/2021 - Location tag width

Changed:

• Set text within .map-tooltip__geography-chip to breaking: no-wrap;

06/22/2021 - SVG console error fix

Changed:

• Adjusted the svg height value for .location-facility__icon to 24px to avoid console errors.

• Removed wazimap-ng.css script call

06/21/2021 - .is--disabled to .disabled

Changed:

• renamed the disabled class on the rich-text dropdowns to .disabled from .is--disabled

06/11/2021 - Map options truncation fix

Fixed:

Fixed the short truncation on map options title. It now extends the full width of the panel.

Changed:

Removed capitalization on the map options title.

05/25/2021 - Changes to map options filters

Changed:

• The backend will need to clone the .map-options__filters_content in the styles panel for the map-options panel as needed. The hope is that this will prevent the dropdowns from breaking.

05/21/2021 - Additive filtering

Added:

These two implementation of this feature work in a similar ways

• .profile-indicator__filter-labels now contains the column labels for each filter column

• Each filter row is now within .profile-indicator__filter-row and .mapping-options__filter-row

• Possible buttons to the right of the filters (add filter and remove filter) are within .profile-indicator__filter-buttons and .mapping-options__filter-buttons

  • These buttons are .profile-indicator__remove-filter and .profile-indicator__new-filter in the rich data panel

  • These buttons are .mapping-options__remove-filter in the mapping options

  • .profile-indicator__new-filteris visible by default in the rich data panel

  • .profile-indicator__remove-filter is hidden by default in the rich data panel

Changed:

• Added border-bottom: 1px; and padding-bottom: 12px; to .profile-indicator__header

04/29/2021 - Data attributes and classes for Point Mapper and locations

Changed:

• Added class="i18n" to all instances of "Point Mapper" and "locations"

  • Point mapper panel and toggle tooltips

  • Locations section in the rich data view (includes the Locations count and the buttons)

• Added data-i18n="Point Mapper" to all instances of "Point Mapper"

• Added data-i18n="locations" to all instances of "locations"

Fixed:

• Changed the wording of "Download all facilities" to "Download all locations" in the rich data view.

04/22/2021 - Download all facilities

Changes:

• Added .location__facilities_download.location__facilities_download-all--header button to .location__facilities_header. This button is only visible on desktop. On mobile it hides and the button moves down to below the location cards. This is because downloading takes less priority on mobile devices.

• Added .location__facilities_download.location__facilities_download-all--footer. This button shows on smaller screen sizes.

• .location__facilities_header-wrapper to contain the icon and title. Please note that this may cause integrations with backend data to break.

04/22/2021 - Chart menu data attributes

Changes:

• Added data attributes to the buttons in the chart dropdown menu

• The following "data-id" attributes have been added:

  • data-id="Percentage"

  • data-id="Value"

  • data-id="csv"

  • data-id="excel"

  • data-id="json"

04/09/2021 - Rich data tables

Changes:

• Added .profile-indicator__table to the .styles section

Documentation:

• .profile-indicator__table_row.profile-indicator__table_row--header is the header row for the table and styles the row accordingly.

• At the moment, only 3 column tables are supported

• .profile-indicator__table_row's contain .profile-indicator__table_cell's

• The first .profile-indicator__table_cell in each row has the class .profile-indicator__table_cell--first applied in order to apply specific styling.

• By default the .profile-indicator__table_show-more is hidden (comprised of the "Load more rows" and "Showing" info).

• When the table exceeds a certain number of rows (dev decision), subsequent rows are not shown and can be toggled to show using the load more rows button.

• There is no functionality from the Webflow side to handle this "expansion". Please advise if there is anything on my end that would assist in getting this functionality working.

03/09/2021 - Print and google maps button for location

Added:

• Added print button to .facility-info

  • .facility-info__print is now a child of .facility-info__header

• Added .facility-info__view-google-map as a child of .facility-info.

  • Component has class .hidden applied by default and can be removed to show when available

03/01/2021 - Explicit print button in rich data

Changed

• Added .rich-data__print to the rich data panel

03/01/2021 - Rich data location buttons

Changed

• Info text changed from "This location has 0000 locations in 0000 categories." to "This area has 0000 locations in 0000 categories."

• Button colour changed to green

• Button wording changed from "facilities" to "locations"

02/18/2021 - Map legend always showing

Changed

1. Map legend always showing:

• Made .map-options and .map-point-legend children of .map-bottom-items (new div)

• .map-options and .map-point-legend have a .hidden applied by default while waiting for the user to add a point layer or choropleth to the map.

• Changes to .map-options to make it's width consistent with other items of the ui

  • .map-options now stretches to the width of its parent .map-bottom-items

  • width: 95%; max-width: 650px;

• Added .point-legend__remove to the .point-legend so that the user can remove a point layer from the map without needing to open the point mapper tab

  • This functionality needs to be added on the backend

  • If this functionality is not ready for deployment, the .point-legend__remove item can be hidden before cloning

2. Version of the point-legend without a remove button for print

02/18/2021 - Map legend

Changed

1. Map print legend:

• Added .map-print to the .map div

• Added .map-print__point-legend to .map-print

• .map-print has a .hidden class by default which is removed when required

2. Point legend:

• .point-legend contains .point-legend__color and .point-legend__text

  • .point-legend__text has a default state of loading...

  • .point-legend__color is fill: rgba(0, 0, 0, 0.06) by default

  • .point-legend__color fill must be updated to the value of the point

  • .point-legend can be duplicated for every instance required

3. Choropleth legend:

• This item has been added to map-print to accommodate functionality that was being hacked together previously.

• The functionality for this item is the same as the default choropleth legend.

• Please contact Matthew if you have any questions about the intention of this component.

02/17/2021 - Added data attributes to hover menu items

Changed

Added data attributes to hover menu components:

• Added data-element="chart-value-select" to the hover menu chart value type selector

• Added data-element="chart-download-data" to hover menu download data options

Fixed

• Removed .webflow from .hover-menu__content that was not removed before previous export

• This class is used to show items when working on them in webflow. This will mean that the hover menu will start off open. With this fix it should return to a default of closed.

02/16/2021 - Hiding and styling content in chart hover menus

Changed

1. Hover menu groups for chart-value and download-data

• Added a div for .hover-menu__chart-value and .hover-menu__download-data

• Add .hidden to remove this group from the menu

2. Changed css for active state on hover menu list items

• Changed the way the .last class is applied to .hover-menu__content_list-item to make the active class behave as intended.

  • Old method was a single class .hover-menu__content_list-item--last

  • New approach is a combo class added. eg. .hover-menu__content_list-item.last

• To make the a list-item "active", just add the active class

  • eg..hover-menu__content_list-item.last.active

02/12/2021 - Tab notice and map title

Changed

Map title:

• Changed default state for .map-title to display: block

• Added .hidden class to .block-title to hide by default.

• Remove .hidden to show

Tab notice:

• Added .tab-notice as a child of .main

• Added .hidden class to .tab-notice to hide by default.

• Remove .hidden to show

• Adjust <a> on .tab-notice__content to adjust the link of the notice

• Adjust .tab-notice__text to adjust the text within the notice (default is "loading...")

Removed hardcoded script in index.html

• On request from Emre, I have removed the second script in the screenshot above.

• UPDATE: Reuploaded a version with both the scripts in the screenshot removed.

2022-10-03

Fix to map zoom positioned items

• Changed the position of the download button

• Changed the position of the geo select panel

• Changed the position of the map options panel

2022-10-02

Move custom stylesheet code from body to head tag

Map panels update - 12/10/2021

Changes:

• Added map-bottom-items--v2 version with updated functionality and styling

• Removed various leaflet styling code from the webflow side

• Removed webflow interactions for opening and closing map option panels. Now controlled by devs using hidden classes.

Various changes - 12/03/2021

Map credit:

• Added .map-credit to .map.

• This item is shown by default and the link directs to Wazimap NG product page.

Point filters:

• Added .is--shown and .is--hidden for the .point-filters_content element.

• Backend should toggle this to open and close the modal.

• Backend needs to hide show the arrow icons on the right hand side since this was controlled by interaction before.

• The .is--hidden class is applied as default.

• Removed the interaction that used to hide and show this content.

• Added .point-filters__no-data to the .point-filters_content block for when there is no data available.

Data mapper:

• Added .map-options__no-data to the .map-options__filters_content

• .map-options__no-data is hidden by default

Class renaming - 01/18/2022

Changed class name .hdden to .hidden for .mapping-options__add-filter.

Transition CSS - 01/14/2022

Added transition styling to the v2 data mapper content items and made max-height on closed !important.

Data mapper v2 - 01/13/2022

Changes:

• Added v2 versions of all clickable triggers and content blocks:

  • data-category__h1_trigger--v2

  • data-category__h1_content--v2

  • data-category__h2_trigger--v2

  • data-category__h2_content--v2

  • data-category__h3_trigger--v2

  • data-category__h3_content--v2

• The arrow icons in .data-category__h1_trigger--v2 is controlled by adding or removing is--closed to BOTH icons. This will hide the one and show the other.

Social sharing

Added social sharing content to the export including:

• Social sharing image

• Favicon

• Social sharing description

• Renamed the site title (Wazimap NG)

Roll back tutorial content - 03/29/2022

Changelog:

• Rolled back the changes to the tutorial modal. Includes tutorial content.

Remove dummy tutorial content - 03/28/2022

Changelog:

• Replaced all dummy content in the tutorial modal with "Loading..."

• Removed the default background image for ."tutorial-slide__image".

• Left the same number of slides to not break the JS.

• Removed "municipality" dummy text from ".map-tooltip__geography-chip"

• Removed "location type" dummy text from ".location-tag__type"

Removal of dummy data charts - 03/24/2022

Changelog:

• Removed ".bar-chart" from ".indicator-chart" to prevent it from accidentally ending up in production.

• Changed all styles panel text to "Loading..." where there was any risk of it ending up in production.

Try to write tests first. You don't need to practice TDD, but if you write a test first it will make it easier for yourself to write the correct code.

red, green, refactor

Always start a test with red (failing test). If your test is green from the beginning you might not catch potential bugs. Always make sure the tests fail first, even if you wrote the correct code already. Use a different assumption then, to make sure the test fails and you see the actual output. After the test is green, try to refactor your code (make it better).

Don't write your test assertions by using the output of a failing test. You should know the output (assertion) before the test runner tells you the failure.

What to test

The ideal set of tests tends to result in ...

• small but right number of unit tests.

• large number of integration tests.

• small number of E2E tests as kind off smoke tests.

Always test edge cases.

Do not only test the obvious path but also the not so obvious. Use parameterized tests in pytest for that.

Test business logic, don't test libraries. We assume library code is tested by the developers of those libraries. If not the integration and e2e tests will catch those. Business logic: All code that is unique, that provides value to the codebase. That drives the app. CRUD is not business logic. State management is usally not business logic.

Unit tests

We want to test code in isolation. It is ok to mock most of the side effects. Although side affects are a smell and should be investigated. If you find yourself mocking a lot of parts or have a hard time writing a test: This is a signal, refactor your code!

Side effects are things like database updates or sending messages.

Integration Tests

Integration tests test at least two parts of a unit. It is ok to mock certain boundaries but be precise and document what these boundaries are. Integration tests don't test the UI. Either use an unit test or an E2E test for that.

Typical boundaries:

• API calls

• User Interface

• Third party services

In index.html adding .i18n class to an element makes the i18n library translate the text of that element on the fly. .18n class can be added using the js/webflow/import.js. Using the data-i18n attribute, we can define which property to be used from the translations config.

//sample config
translations: {
    'en': {
      'Point Mapper': 'Services'
    }
  },

<!-- sample html -->
<div data-i18n="Point Mapper" id="test" class="i18n">Point Mapper</div>

In a loop,

• We check all the elements that have .18n class

• Search their data-i18n attribute value in the translations config(i.e "Point Mapper" in the sample code above)

• Modify the element's text using the value that corresponds to the key(i.e "Services" in the sample code above)

• The modified #test element will look like :

<div id="test" class="i18n">Services</div>

File structure

• We use Cypress for GUI tests along with Cucumber

• All the GUI tests could be found in __tests__/gui folder

• The test steps are in the .feature files and the test step definitions are in the .js files in the folder that shares the same name as the .feature file

  • For example the step definitions of facility_modal.feature need to be in the facility_modalfolder(name of the js files are not important)

• We intercept all the requests and respond with local data. The data for each test are kept in json files in the folder of the corresponding test.

Running the tests

• Start the project

• In a new terminal running yarn cypress:open will open the cypress window.

• In the cypress window you can see all the e2e and gui tests and by clicking on any of it you can start testing

• Without opening the cypress window, there are 2 ways to run the gui tests

  • yarn cypress:gui runs all the gui tests and saves their result in cypress dashboard

  • yarn cypress:gui-local runs all the gui tests without saving anything to the cypress dashboard. While working locally, this script should be used to prevent exceeding dashboard monthly test quota

Cypress dashboard

• https://dashboard.cypress.io/

• The credentials for the dashboard could be found in OpenUp general credentials file

• Everytime cypress:gui script is run, the test results, logs and their videos are saved to the dashboard. Github actions runs this script too.

• By selecting WazimapNG in the Projects page, all the latest test runs could be viewed.

• By clicking on any of the test runs, you can view

  • Overview : How many tests are failed / passed / skipped.... etc

  • Test results : Details, statuses, result logs... etc

  • Specs : Screenshots, videos... etc

• More useful details(Average run duration, Top failures, Slowest tests...) could be found in Analytics menu of the dashboard.

11/04/2021 - Facilities content toggles

Changes:

• Added .is--shown and .is--hidden for the .location__facilities_content element.

• The .is--hidden class is applied as default.

• Removed the interaction that used to hide and show this content.

10/26/2021 - Point mapper icon colour fix

Changed:

Removed color: inherit styling from h1__point-mapper_trigger which was causing there to be an issue with the default colour of icons in the point mapper

10/21/2021 - Point mapper truncation

Changed:

Fixed truncation on

• Fixed truncation on point mapper themes and cat

Location pins

Google style pin

This marker is based on the look of the google pin style and is able to accommodate icons and text abbreviations easily.

Dimensions:

At all zoom levels, this pin should be 21px wide.

Usage:

• Only adjust the colour of the element with the specified colour (#4693EF)

• The stroke around the edge is defined as 8% black and does not need to be changed

• Text abbreviations within the marker should be 8.5px

White pin

This marker is based on the look of the google pin style and is able to accommodate icons and text abbreviations easily.

Dimensions:

At all zoom levels, this pin should be 21px wide.

Usage:

• Only adjust the colour of the element with the specified colour (#4693EF)

• Text abbreviations within the marker should be 8px

Proposed by:

Date:

Status: Proposed

Context

Proposed Solution 1

Benefits

Disadvantages

Proposed Solution 2

Benefits

Disadvantages

Other implications

Proposed Solution 3

Benefits

Disadvantages

Resolution

This problem was finally addressed by ...

Icon choice

When choosing an icon, the following things should be taken into consideration.

1. The icon should be generalized and not reliant on knowledge of a specific language or region.

2. The icon should follow established norms for icons of it's type in similar applications.

3. When the choice deviates from a norm, it should only be done for very good reasons.

4. Icons should be made distinct enough from others in the UI

Some examples:

Size

Icons should generally have the following size properties:

1. Icons should be no larger than 24px wide and 24px tall.

2. Icons that are smaller than 24px in size, should fit within a bounding box that is 24px.

3. This is generally the approach taken for icon sets like Material and FontAwesome.

Visual weight

Visual weight refers to the amount of prevalence an icon has when looking at it's role in the UI combined with how important the action that icon represents is to the user. When deciding on the visual weight of an icon, the following considerations need to be made.

How unique is the icon in the UI

If there is only 1 instance of this icon in the entire UI, it generally denotes it requiring a higher visual weight. eg. search, data mapper, rich data view.

How important is the action the icon represents

If the action has high importance it should generally be given more weight in the UI, either through, size, colour or containing element settings (eg. size of button, space around button etc.).

NG Proposal 2

Representing geographical hierarchies to the user.

Proposed by: Adi Eyal

Date: 2020-09-02

Status: Work in progress

Context

Read about Geography Hierarchies here.

The non-linear structure of Geography Hierarchies poses a challenge to presenting information on a map. When in the context of a Municipality, the user will either want to see Wards or Mainplaces. These cannot be shown simultaneously as they overlap with each other. A graphical toggle would be required to change between these two levels.

Forks can occur at any part of the hierarchy and are dependent on the current geographical hierarchy in use. For the purpose of illustration, here is a more extensive hierarchy:

In this document we discuss the most appropriate user interface to navigate this hierarchy.

At the time of writing, this is how Wazimap depicts the Wards in the Cape Town metro.

Cape Town Metro with Wards displayed

Cape Town Metro with Mainplaces displayed

A typical user requirement would be to switch between Wards and Mainplaces.

Approach 1

The first approach to address this is to use a contextual toggle. The select box on the right is populated with the child levels available at the current geography. When the current geography changes, the options change with it.

This approach requires minor UI changes and can be implemented relatively quickly. Users may however may not necessarily understand the geographical hierarchy and contextual changes may be hard to follow. For example, it may not be obvious that Subplaces are not available below Wards.

Approach 2

Creating a review app

1. Login to the heroku dashboard

2. Go to the wazimap pipeline

3. Click "Create review app" for your pull request

Find the URL to the deployed review app in the pull request or by clicking "View app" in heroku.

Migrations and demodata are only loaded when the app is created. If you need to recreate the database, destroy the app and recreate it.

Connecting a netlify frontend deploy preview to backend review app

Creating staging apps in heroku

This is probably not helpful any more now that real review apps work again, but we're leaving this here for now in case it's needed.

We were creating "review apps" in the staging "stage" of the wazimap pipeline in heroku while heroku review apps were not working due to the exfiltration of github oauth keys.

Create a new app

We name apps according to the pull request number, e.g. wazimap-pr-1234 for PR #1234

Steps for creating an app via CL

• heroku login

In the project directory (to update your project git remotes):

• heroku apps:create wazimap-pr-1234 --remote heroku-pr-1234

• heroku pipelines:add wazimap --app wazimap-pr-1234 --stage staging

Steps for creating an app via GUI

• Login into Heroku Web

• Go to wazimap pipeline

Steps for setting up the stack

• heroku stack:set container --app wazimap-pr-1234

• Adding Addons:

  • heroku addons:add heroku-postgresql:hobby-dev --app wazimap-pr-1234

  • heroku addons:add heroku-redis:hobby-dev --app wazimap-pr-1234

• Adding Config vars: heroku config:set DJANGO_SECRET_KEY=3423424 AWS_ACCESS_KEY_ID=xxx AWS_SECRET_ACCESS_KEY=xxx AWS_STORAGE_BUCKET_NAME=xxx AWS_S3_REGION_NAME=xxx --app wazimap-pr-1234

Deploying branch

• git add and commit everything you want to deploy

• git push heroku-pr-1234 BranchName:master

Post-release step

Start Worker dynos

Via CLI:

heroku ps:scale worker=1 --app wazimap-pr-1234

Via GUI:

• heroku run python3 manage.py migrate --app wazimap-pr-1234

• heroku run python3 manage.py loaddata demodata.json --app wazimap-pr-1234

• heroku run python3 manage.py createsuperuser --app wazimap-pr-1234 if you want to create new supruser

Usefull commands

• Check configs for review app via cli heroku config --app wazimap-pr-1234

• Define region while creating app heroku apps:create wazimap-pr-1234 --remote staging --region eu

• Checking logs

  • worker logs : heroku logs --dyno=worker --app wazimap-pr-1234 --tail

  • web logs: heroku logs --dyno=web --app wazimap-pr-1234 --tail

Connecting backend review app to deploy preview app

Destroy review apps when branches are merged

heroku apps:destroy wazimap-pr-1234

NG Proposal 4 - Format configuration

Proposed by: Adi Eyal

Date: 2020-09-14

Status: Work in progress

Context

Wazimap-NG Indicators contain different types of numbers that need to be formatted. Some Profile Administrators may want to round floating point numbers to 1 decimal place, while others prefer 2. Some numbers are percentages and should be displayed as such. Currently the Wazimap front-end does not receive sufficient information from the API to help determine the type of number received nor is it possible for administrators to configure the formatting of that number. This NGP discusses a few approaches to resolving this issue.

Proposed Solution 1

Each Indicator may require its own specialised formatting. To achieve

Proposed Solution 1

Proposed Solution 2

Benefits

Disadvantages

Other implications

Whereas indicators can now be mapped using a choropleth. Superindicators cannot since a choropleth can only display one Count variable at a time. This can be addressed by another select box allowing the user to choose the Count variable of interest or mapping multi-count variables could be prevented entirely.

Recommendation

Stub page

Whereas indicators can now be mapped using a choropleth. Superindicators cannot since a choropleth can only display one Count variable at a time. This can be addressed by another select box allowing the user to choose the Count variable of interest or mapping multi-count variables could be prevented entirely

Proposed by: JD Bothma

Date: 2022-04-21

Status: Spike probable solution

Context

We would like to be able to serve open graph metadata specific to each wazimap NG profile. Wazimap profiles are designed as stand-alone sites or sub-sites often set up as subdomains of other sites. Open graph metadata can significantly increase the chance of someone clicking a link to a wazimap profile found in social media or search engines. While Google takes Open Graph metadata updated client-side using javascript into account, we are not aware of social media or other search engines that do that. To reach any of these audiences except Google, we need to serve open graph metadata specific to a profile server-side.

The fields that would benefit from being served this way are

• Page title - often used as an emphasised heading

• Image - very effective at being eye-catching and drawing interest from potential users

• Description - this can both convince users that something might be worth clicking, and also set their expectations to reduce surprise and this bouncing from the site.

• Favicon - some tools like bookmark and link aggregators and search engines also use this. While browsers support dynamically-set favicons, it's not clear how many other tools do.

Open Graph Metadata also help SEO. Additional content that can help SEO is the introductory content currently located on the profile landing pages.

Current state

Landing pages

Most sites have a landing page made in webflow or Wordpress. These can easily serve profile-specific branded metadata to accomplish most of the above. This then links to the relevant wazimap profile with a clear call to action. This content can be quite effective for SEO.

We would like to move these into wazimap NG to be able to maintain that content from one interface, reduce the jumps users have to make, and reduce the hosting costs of a wazimap profile. If we move these into wazimap NG, we lose this ability to serve the open graph metadata need.

Hardcoded metadata

The open graph metadata is currently the same for all wazimap profiles, hardcoded on the webflow side and imported to the frontend app from there. This can be updated both in webflow and overridden in the import script in the frontend app.

Updating this to something that is at least sensible as generic content is a high priority until this content is dynamic and profile-specific.

Levels of achievement

In ascending order of value:

1. Better hardcoded metadata (get rid of the webflow favicon, ensuring webflow changes don't introduce odd things like "2021" in the title

2. Profile-specific metadata, e.g. "Youth Explorer" or "Vulekamali Geospatial data viewer" in the title, branding and screenshot in the image, etc.

3. Profile- and geography-specific metadata, e.g. "Tswhane - Youth Explorer" or "Eastern Cape - Vulekamali Geospatial".

4. Specific indicator and geography, e.g. "Multidimensional youth poverty - Tshwane - Youth Explorer" in the title, perhaps some or all of the indicator description in the open graph metadata

5. dynamic image, e.g. the selected geography on the map, perhaps dynamically-branded; perhaps a choropleth or rich data view chart image for the selected geography and indicator.

We're aiming around level 2 for now.

Level 3 depends on changing from fragment identifier (#geo:CPT) to querystring (?geo=CPT)

Proposed Solution 1 - Server-side rendering (SSR) for the frontend app using node.js in a dokku app

The frontend would be deployed as it currently is, but Javascript is executed server-side to inspect the request, fetch the profile-specific data via the API, templates the metadata into the HTML page, and serves the response.

https://www.npmjs.com/package/mustache-express

A prerequisite for next.js would be replacing webflow or horrid hacks to import our webflow export as a custom "Document".

Benefits

• a lot of people are doing this these days

• the frontend remains rather decoupled from the backend

• Very little changes in terms of developing on the frontend - it remains just a yarn start kind of process

Disadvantages

• Do we have experience of serving stuff with node.js in prod?

Proposed Solution 2 - Server-side rendering for the frontend app using node.js on netlify and similar

Same as above, but on a (not so?)-static hosting platform like netlify or Cloudflare apps instead of a dokku server.

Benefits

Disadvantages

Other implications

• Each profile hostname has to be added as a custom domain in Netlify or whatever the platform is. Netlify doesn't support more than 100 custom domains (it's not clear what the technical limit is) on an app and we have already had issues with their restriction of the apex domain being allowed only on one netlify team without manual intervention from their support. See NGP7 - Wazimap profile domain management.

Proposed Solution 3 - Serving HTML, CSS and JS using the (currently) backend django app

Index.html becomes a django template. Django templates in the data before serving /

The backend will need to respond to requests for all profile hostnames, affecting how the backend infra gets configured.

Benefits

• We have lots of experience of this

• no additional api requests to serve / - it can access django models immediately.

Disadvantages

• frontend dev can easily get coupled with running the backend, needing at least a basic backend setup to run, even if subsequent requests to the api go to another backend (e.g. prod). But this can add confusion and/or complexity (it is possible to keep them separate. Just hard.)

Resolution

Proposal:

1. Quick win with minimal interruption:

   1. Use minimal server-side templating like express-handlebars to query and render profile details

   2. enables all achievement levels but certainly 2-3

2. eventually consider full-blown server-side rendering next.js style

   1. enables pre-rendering javascript-drawn content, skipping several data roundtrips

   2. watch out: next.js is possibly not the right tool - apparently more page-content oriented than SPA.

Explanations

Good Title:

Use the Issue Number and the issue title for the title of your PR.

Description

Add a good description here. Use the issue description to inform your own understanding of this issue. See How to prepare a Pull Request

Related Issue

Create a link to the related Trello Card here. So that it can be easily referenced.

How to test it locally

Explain how you tested the changes locally and made sure that the PR does what you intend to. For example if you working on Frontend Code describe which page you should open and what steps to take to see the changes you made. For the backend, which endpoint to hit with which params, best to provide an example like a curl command.

Screenshots

If on FE, upload the desired outcome (design) and create a Screenshot, once you complete your work. Even better create a gif which shows how it works.

Changelog

Create a detailed changelog, if possible. This changelog can be informed by the commits you create (reference the commit part in the git handbook). They should contain parts that you changed. This can be a bullet list, no need to create full sentences. Should be functions, classes, API Endpoints, etc.

Added What has been added.

Updated What has been updated.

Removed What has been removed.

Checklists

This part should be checked off accordingly, when you're ready.

NG Proposal 1 - Changing data model

Proposed by: Adi Eyal

Date: 2020-08-31

Status: Proposed

Context

The data model that NG has been built around is the concept of a universe, i.e. the total number of people in a particular context. This universe can then be disaggregated by a number of attributes. For instance the country population is a universe that is divided into male and female, i.e.:

# male + # female = total population

Similarly:

# Left-handers in WC + # Right-handers in WC = total population of WC

The input file might look like this:

Table 1

That works well for census data where you are disaggregating a universe. It falls short when you would like to compare two unrelated datasets side-by-side. As an example:

Table 2

In this case, we cannot sum the two rows to get the universe, i.e. there aren’t 30,000 + 35,000 = 65,000 people with access to water in the Western Cape. We do however ever want to be able to compare these two figures.

The problem does not only apply to time-based data, e.g.

Table 3

In this case we have two universes which overlap but we still want to be able to compare them:

Proposed Solution 1

An underlying assumption of the dataset model is that there is only one count field in every file. We could change this assumption by allowing multiple counts, effectively pivoting our table.

Table 4

This solution will require significant changes to the following components:

• Data Import

• Variable Creation (data is aggregated at this level)

• API

• Front-end data model

• Front-end visualisations

We would also need to decide how the system would identify count columns. The current convention is to match by name. We could use a similar approach by requiring a standard prefix, e.g. Count: 2016. Alternatively, the administrator could identify these columns once the data has been uploaded.

An alternative approach would be to designate a pivot column, e.g. Year.

Benefits

• This is a robust approach that ensures compatibility between Count columns (compared with Solution 2 below). Every column available for the first Count column will be available to the second one.

Disadvantages

• A significant amount of effort is required to make this change.

• It limits comparisons of datasets to those that were included in the initial upload. If data from a new year becomes available, it is not possible to include it.

• Depending on implementation, may place an additional burden on the Data Administrator requiring a special naming of columns in the spreadsheet to be uploaded.

Proposed Solution 2

Using this approach we add the concept of a super indicator which ties together two or more indicators together. For instance, table 3 becomes two separate indicators:

Table 5

Table 6 These are then associated in the backend.

This solution would affect the following components:

• A new database model would be required

• The API will need to be changed

• Front-end data model

• Front-end visualisations

Benefits

• Overall fewer changes are necessary to implement this feature

• Data Administrators are able to associate arbitrary indicators without pre-planning.

• An almost identical workflow is used as the current approach.

Disadvantages

• It is possible to bind two, completely unrelated or incompatible indicators. For instance, the number of bankruptcies vs Child pregnancies.

• Less extreme but equally problematic is two related datasets with different groups, e.g. 2016 Matric passes disaggregated by gender vs 2017 Matric passes without disaggregation. In this case, we will need to decide how this will be displayed on the frontend, especially in graph filters

In the case of two incompatible datasets, we need to decide whether filters are available for missing groups.

Other implications

Whereas indicators can now be mapped using a choropleth. Superindicators cannot since a choropleth can only display one Count variable at a time. This can be addressed by another select box allowing the user to choose the Count variable of interest or mapping multi-count variables could be prevented entirely.

Proposed Solution 3

Another approach to addressing this issue is to recognise that the cause of this problem is the concept of a non-overlapping universe introduce by the Dataset model. This approach does not always make sense as in the examples above. To create an Indicator, a background process is fired that groups DatasetData objects appropriately. In cases like the ones described here, it might be easier to create the indicator directly and avoid datasets entirely.

When creating a new indicator, the admin is asked whether it should be created from a dataset (the current process) or whether it should be uploaded from a file. This latter approach would simply create the relevant IndicatorData directly. Since an indicator must link to a dataset, one can be created automatically. Creating new indicators from this dataset should not be allowed however.

Benefits

This is by far the easiest approach to implement. Very little needs to change with the exception of the file upload mechanism when creating a new indicator. Once the IndicatorData objects are created, downstream users of this data remain unchanged.

It also allows flexibility for data administrators to shape data without too many constraints on the format.

Disadvantages

Data re-use is limited. Datasets provide opportunities to create multiple indicators. The superindicator concept allows even more mixing and matching of indicators.

Resolution

This problem was finally addressed by marking columns as aggregatable or not aggregateable. If a column is not aggregateable, different values in that column need to be shown separately. This can be implemented as a dropdown filter, e.g. you always need to choose a year. Alternatively, a grouped bar chart could be used, e.g. 2016 and 2017 bars.

The change also involved removing all aggregation from the backend and sending raw data to the frontend.

Proposed by: JD Bothma

Date: April 2022

Status: Spike probable solution

Context

As wazimap scales in the number of profiles it hosts, we will need to serve on more and more hostnames. Each hostname we serve has to be valid for the TLS certificate, and traffic for that hostname has to be routed to the Wazimap NG frontend. That imposes the following requirements for adding a new profile:

1. Come up with an appropriate hostname

   • currently: usually a geo. prefix to the client's domain

   • also: a sandbox-geo. prefix for a second profile to use as sandbox

   • often: initially just a projectname.wazimap.openup.org.za subdomain to get things moving quickly

2. Relate the hostname to the relevant profile in Wazimap admin

   • A Profile Admin level user can do this. It's currently JSON but can be made simpler.

3. Configure the frontend web server to serve that hostname

   • currently: add it as a custom domain in netlify

4. Provision a new TLS certificate which includes the new domain as a Subject Alternate Name

   • currently: Click renew certificate, then verify DNS, then renew certificate, perhaps a number of times, until it's happy - in netlify

Problems with this approach:

• A networking-aware developer has to do the DNS and custom domain configuration

• Stale domains can break certificate renewal

• Netlify has a limit to the number of custom domains allowed - they have warned us not to add more than 100

• The Certificate is growing bigger and bigger. We aren't sure of the consequences of this.

Anticipated scale over the next 1-3 years:

• We are aiming for 40 profile subscriptions by the end of 2022-2023 financial year.

• Pending: Feedback from current profile owners on the importance of custom domains for their profiles.

• Based on the current profiles, we guess that at least 50% would consider custom domains important.

Relevant Let's Encrypt limits:

• Certificates per Registered Domain (50 per week) - Only really relevant if we try to have a distinct certificate per wazimap.co.za subdomain instead of a wildcard certificate for that domain. Renewal could be staggered. This would probably happen naturally by renewing daily a month before expiry as is currently done.

• Names per Certificate (100) - Relevant if all domains are on a single certificate. This (all domains added as Subject Alternative Name values on a single certificate for the app) is the usual approach dokku-letsencrypt and netlify uses for domains added to the same app.

• Failed Validation limit of 5 failures per account, per hostname, per hour. - Relevant if one certificate is used for multiple domains, and DNS for one of the domains is not resolving to us yet. We'd need to be careful about accidental or intentional denial of service here.

Limits on Netlify:

• Wildcard domains are supported on Pro, but not at the same time as custom domains

• TLS Certificates can only be handled by Netlify if Netlify DNS is used for the wildcard cart

• Only 100 custom domains are supported on a site. (Possibly due to letsencrypt Names per Cert limit)

Client wants/needs

youthexplorer.org.za

Youth explorer very much is an established brand in their circles.

gdc-projects.org.za

Could you please also let us know what the technical costs are for keeping a custom domain.

whowhatwhere.org.za

We are fine with http://whowhatwhere.wazimap.co.za/ being the domain

Similar functionality on competitors/related tools

Proposed Solution 1 - Only allow subdomains of our base domain

Requirements:

• Once-off wildcard DNS for the base domain, e.g. *.wazimap.co.za

  • Supported by netlify, dokku

• 2-monthly wildcard TLS certificate for the wildcard domain

  • Supported by netlify, dokku

• Wildcard reverse proxying to the frontend web server

  • Supported by netlify, dokku

Benefits

Disadvantages

Proposed Solution 2 - Allow subdomain of our base domain for new sites, and legacy custom domains

Requirements:

• Same as solution 1 for wildcard DNS/cert/proxying

• The solution 1 requirements for about 5 custom domains

  • Supported by dokku

  • Not supported by netlify ("You can’t use domain aliases on a site with wildcard subdomains enabled")

Benefits

Disadvantages

Other implications

Proposed Solution 3 - subdomains by default, option of custom domains on dokku added manually

Approach

• DNS

  • Wildcard DNS for subdomains of our base domain

  • Any DNS CNAME to to our frontend web server

• Virtual hosting (reverse proxy or static server)

  • Subdomains are handled automatically by dokku/nginx wicard domain

  • Custom domains: ??? manually-added or automated?

• TLS certificate

  • dokku letsencrypt - wildcard + up to 99 custom domains via dokku letsencrypt (100 30-char domains like e.g youthexplorer.wazimap.co.za is 3kb baggage per new TLS connection)

  • alternative to dokku letsencrypt: certbot + vhost per custom domain + dokku app listening on a host port for non-dokku reverse proxy

Adding/removing a domain manually would entail

1. Add the domain to the profile in Admin

2. Add CNAME record pointing to the server, or ask the client to do so

3. SSH to the server

4. Add domain to dokku/nginx vhost

5. Renew TLS certificate

For 20 custom domains, we might have to do this on average 30 times or just over once every 2 weeks.

Benefits

Disadvantages

Proposed Solution 4 - subdomains by default, option of custom domains on dokku automated

Same as Solution 3 but instead of adding/removing domains manually by SSHing to the server, we automate that, triggered by domain modifications in admin.

Approach:

• Add the domain to the profile in Admin

• Add CNAME record pointing to the server, or ask the client to do so

• ...Automation to add/remove domain config to server

• ...Automation to renew certificate

• ...Automation to let admin know it's all up and running or feed back errors

Benefits

Disadvantages

Proposed Solution 5

Benefits

Disadvantages

Resolution

Try option 3 until the effort of maintaining custom domains manually and the value to be gained by automating is worth the effort of automating handling custom domains.

Further reading

• https://discuss.httparchive.org/t/san-certificates-how-many-alt-names-are-too-many/1867

• Someone exploring many of the same questions on Netlify

Proposed by: JD Bothma

Date: 2022-04-21

Status: Spike probable solution

Context

Using webflow as the frontend framework makes concurrent frontend development error-prone and requires complicated synchronisation between team members to plan changes. It also often requires multiple round-trips between Matt making changes in Webflow, frontend devs trying to use those changes, Matt then having to make tweaks, until it is done. Each change requires a webflow export, sharing a big zip file, requiring an import which may bring surprise changes in the diff introduced by webflow, adding cognitive load to the code review.

The plan thus far has been to introduce React for the next interactive component we need to modify or introduce.

We have added a small non-interactive component successfully using custom Javascript creating requisite markup, and custom CSS in a project-specific CSS bundle.

We are now looking at introducing a frontend framework, or the frontend libraries, needed to eventually be completely free from Webflow in our frontend dev process.

Current components