Data is uploaded to Wazimap NG using Django (which can be accessed here). Once you are logged in, scroll down to the POINTS section, and click on the +Add button next to Collections (see Figure 5, below).

On the following page, do the following (refer to Figure 6, below):

1. Select (from the dropdown list) the Profile which should be associated with your point collection;

2. Give your point collection a Name;

3. Select the appropriate Permission type (Public or Private);

4. Import your point collection by clicking Choose file and selecting the .csv file you exported earlier;

5. Add the Source information; and

6. Scroll down and click Save in the bottom right hand corner.

Public & Private Datasets

Datasets can be set as Public or Private under Permission type. Public datasets, and variables derived from them, can be used on any profile in Wazimap NG. This enables reuse of valuable datasets without the need for each use case to source, and upload the data again. Private datasets and variables derived from them can only be used on the profile they are assigned to under Profile.

---

Your point collection (datatset) should now be added to Django. To display your point collection on Wazimap NG, you must first create a Theme, and then a Profile Collection (from your point collection), which will be nested under your Theme. Themes and Profile Collections are vital in displaying data using the Point Mapper on Wazimap NG.

A universe refers to the population against which the variable is being applied. Universe can also be left as blank which would then apply to everyone.

To create universes, you will need to write a bit of json.

The structure of this is:

{
"Variable Name": "Filter value",
"Another variable__in": ["array", "of", "values"]
}

See below for an example of youth age range universe.

{
  "Age Group__in": [
    "15-19",
    "20-24",
    "25-29",
    "30-35"
  ]
}

This universe when applied to a variable, would include all people within the ages of 15 to 35.

These are the steps needed to generated datasets for Africa Data Hub.

The script of ADH can be found at https://github.com/OpenUpSA/wazimap-adh-data under the folder COVID Countries and Africa Admin 1.

Collection data

Download owid-covid-latest.csv from https://github.com/owid/covid-19-data/tree/master/public/data

Download from https://github.com/dsfsi/covid19za/tree/master/data the following files: covid19za_provincial_cumulative_timeline_confirmed.csv, covid19za_provincial_cumulative_timeline_deaths.csv, covid19za_provincial_cumulative_timeline_recoveries.csv

Download from https://data.humdata.org/organization/hera-humanitarian-emergency-response-africa ALL( for various countries) csv files that end with Coronavirus (Covid-19) Subnational eg. Niger: Coronavirus (Covid-19) Subnational. All the datasets from this source should be saved in a folder named HERA

For each csv file in HERA folder, change columns name from

['CONTAMINES', 'DECES', 'GUERIS', 'CONTAMINES_FEMME', 'CONTAMINES_HOMME', 'CONTAMINES_GENRE_NON_SPECIFIE']

These dataset should be saved under the folder wazimap-adh-data-main/COVID Countries and Africa Admin 1 After downloading the datasets, run the script as in the following order

Admin0_dataTransformation_v2.ipynb

Africa_Admin1_dataTransformation.ipynb

Data_Aggregation.ipynb

Data_Aggregation.ipynb will generate cases_monthly, death_monthly .

Wazimap NG (Next Generation) provides easy access to different kinds of data through a mapped and open-source information system, designed to help non-technical users explore data, both meaningfully and in context. In other words, a Geo-Information Spatial (GIS) Tool “for the rest of us”.

This Profile Curation Handbook contains documentation on how to manage and upload datasets, as well as create and manage profiles and their associated indicators. There are two administration roles:

1. Data Administrator — responsible for sourcing, shaping, and uploading datasets and point collections to Wazimap NG.

2. Profile Administrator — responsible for defining and managing profile indicators, and site overall content. In short, responsible for how data looks on Wazimap NG.

Both these roles have a function in Wazimap NG’s three views. It is, however, possible for these roles to be performed by the same person. The three views are Point Mapper, Data Mapper, and Rich Data. The table below shows the respective logos for each view of Wazimap NG.

The rest of this Profile Curation Handbook is structured according to these three views of Wazimap NG, and the various roles that Data and Profile Administrators play for each.

The easiest tool to create or edit .csv files is Google Sheets — it can open both .csv and .xlsx files, and can export .csv files for upload to Wazimap NG. Any point collection requires the following three fields (as highlighted in red in Figure 2, below):

1. name;

2. longitude; and

3. latitude.

As shown in Figure 2, above, additional fields can be included as attributes for each point in the collection. In the Point Mapper view, these additional attributes are displayed in the More Info section of each point’s tooltip (refer to ).

The name field (column) of your point collection cannot be formatted as a number. It may be necessary to format the entire column to plain text by selecting the entire column and clicking Format > Number > Plain text (see Figure 3, below).

Additionally, column headings cannot start with a number (e.g., 2019 HDI), contain periods (e.g., SDG 3.1), or contain square brackets (e.g., HDI over the Years [Average]).

Once your point collection is complete (has the three required fields, is formatted correctly, and contains no blank cells in the first row), it can be exported as a .csv file. To do this, click File > Download > Comma-separated values (.csv) (see Figure 4, below).

To create a Profile Collection, scroll down to the POINTS section again, and this time click on the +Add button next to Profile Collections (see Figure 10, below).

On the following page, do the following (refer to Figure 11, below):

1. Select (from the dropdown list) the Profile which should be associated with your Profile Collection;

2. Select (from the dropdown list) the Theme you created;

3. Select (from the dropdown list) the Collection (point collection) you uploaded;

4. Give your Profile Collection an appropriate Label (this will be displayed on Wazimap NG); and

5. Scroll down and click Save in the bottom right hand corner.

Adding Filters to Profile Collections

As shown earlier in , it is possible to filter points based on their attributes (e.g., Responsible Institution). To add filters to your Point Collection, include the attributes you want to filter your points by in the Configuration panel using a filterable_fields array (see Figure 12, below).

Adding HTML Field Types

In addition to adding filters, you can define the field type of columns in your Profile Collection to render data according to HTML code. This is particularly useful for linking to additional, external information (e.g., linking to the source of your points).

This requires that the html code be included in your dataset when (see, for example, the code used in Figure 13, below). To define the field type, include the attribute (column) you want to define in the Configuration panel using a field_type array (refer to Figure 12, above).

The HTML code for adding links is:

<a href='url' target='_blank'>name</a>

In Figure 13, above, the url is , and the name is South African Department of Water & Sanitation. On Wazimap NG, in the More Info section of each point’s tooltip, this data will be displayed as hyperlinked text (see Figure 14, below).

In addition to the href HTML attribute, the following are also allowed: class, target, data-*, and style.

---

If all steps were followed correctly, your point collection will now display on your Wazimap NG Profile (sometimes a hard refresh of the Wazimap NG page is required for changes to reflect). The sections that follow offer some additional information for displaying point collections on your Wazimap NG profile.

It is possible to upload additional points to an existing point collection without having to replace the entire point collection. To do so, first prepare a .csv file with the new points to be added, as outlined under Shaping Data for Point Collections.

Next, in Django, scroll down to the POINTS section, and this time click on Collections (see Figure 15, below).

The page that opens will contain all point collections for all Wazimap NG profiles. There are two ways to locate an existing point collection (see Figure 16, below):

1. If you know the point collection’s name, simply enter it in the Search bar (top left), and click Search; or

2. Filter point collections by the associated Profile (top right), and locate the desired point collection in the list.

Once located, click on the point collection’s name. The same page as shown in , earlier, will open. Next, import your new points by clicking Choose file and selecting the .csv file with the new points to be added. Then, hit Save. After refreshing your Wazimap NG Profile, the new points should appear on the map.

Reordering subindicators

It might be that you want to change the order in which the sub-indicators are shown. For example, you may want to swap Agree and Disagree in the chart below

In the Admin Suite, find SubindicatorsGroups

Find the relevant indicator:

Then drag the subindicators into the desired order:

The order should now be changed on the front-end. In order to see it you will need to hard-refresh your browser - ctrl + shift + R (this will be fixed soon).

Non-aggregatable columns

Values are summed over all dimensions other than the indicator variable by default. That means an indicator on the column "financial year" from a dataset with columns "financial year" and "income source" will disaggregate by financial year, and show the sum of the different income sources, unless the user adds a filter on income source.

It sometimes doesn't make sense to sum values over a dimension. For example

• If you don't know how many years are in a dataset, the total across years doesn't mean anything

• If your data contains overlapping categories to support differing standards, e.g. ages 15-24 and 15-35, summing over these would lead to double-counting.

You can mark a column as non-aggregatable by un-checking the Can aggregate check-box. It is checked by default as most columns are fine to aggregate over.

This will mean that indicators from this dataset will automatically have a filter for this subindicator group, unless this group is used as the indicator variable (in which case it is already disaggregated).

Unlike user-added filters, filters added to disaggregate non-aggregatable columns can not be removed.

You can . E.g. to ensure that the latest financial year is the default value filtered against. Users can still choose other options if they wish.

To create a Theme, scroll down to the POINTS section again, and this time click on the +Add button next to Themes (see Figure 7, below).

On the following page, do the following (refer to Figure 8, below):

1. Select (from the dropdown list) the Profile which should be associated with your Theme;

2. Give your Theme an appropriate Name (this will be displayed on Wazimap NG);

3. Select a preferred Icon; and

4. Scroll down and click Save in the bottom right hand corner.

On Wazimap NG, the theme colour is configurable. Simply add the of your choice. The next step is to create the Profile Collection (from your point collection), that will be nested under your newly created Theme.

Long term maintenance to a point collection often involves the following tasks:

1. Removing points that are no longer relevant;

2. Adding points that are not already in the database; and

3. Updating data about points already in the database (e.g., opening hours, services provided, correcting/improving a description).

To do this, you need to be able to compare your updated data to the data you have already uploaded to a profile collection in Wazimap NG. The easiest way to do this is to use unique identifiers.

Adding Unique Identifiers

An identifier is a value which uniquely and consistently identifies an object — in this case, a point in your point collection.

It is important to include some kind of identifier in your point data to facilitate updates to the data. Users down the line will rely on your identifiers being unique and consistent to be able to incorporate updates should they download a copy of your point data.

If your data does not have an official identifier that is consistent over time, and unique per point, it will become difficult to check for any duplicates or whether newly-provided points are already in your database (e.g., by name, address, and so on). Think about what your users will be able to provide you, and include that in what is shown to them as well.

You can use the following formula in Google Sheets to create a UUID (Copy this into the appropriate cell):

Checking for Duplicates & Removing them

Google Sheets has a built-in function to check for and remove duplicates. In Google Sheets, select the data of interest, and click on Data > Data clean-up > Remove Duplicates (see Figure 20, below). In the box that appears (see Figure 21, below), be sure to select the option Data has header row if it indeed does.

If you are using Excel, see . If you are using Workbench, follow the steps below:

1. Add a deduplication step.

2. Select a column whose values probably ought to be unique.

3. Look for rows where the duplicate number is greater than 1.

Merging Updates

If you have a consistent UUID (unique identifier), then:

1. In Excel or Google Sheets, use .

2. In Workbench, might work.

If you have messy or dirty data with different capitalisation and potential spelling mistakes, try using with CSV reconciliation.

Figure 1, below, is an example of such point mapping, and shows the location of Water Treatment Works within the City of Cape Town Metro of South Africa.

The first step towards creating such a map is sourcing and understanding your data, and knowing what you are trying to display using your data. This is often the hardest part, and requires collaboration between Data and Profile Administrators. It is also important to know a dataset’s Terms of Use or Licence, and what this allows you to use the data for.

Wazimap NG uses .csv files to display point collections, and these .csv files require a particular shaping (formatting) to be properly understood by the Wazimap NG platform. This is the first step towards creating your map.

Preparing the dataset

Before a dataset can be uploaded, the data needs to be cleaned and shaped into the correct format. As a rule, the more disaggregated the dataset, the better as this allows a single dataset to be (re)used for multiple indicators and also allows for multivariable analysis.

The system accepts files in csv, xls and xlsx formats. The file needs to adhere to a specific structure and ensure it always contains the following fields:

Inside the Geography column, valid values are

• country code (ZA)

• province codes (GAU, LIM, WC, etc.)

• Municipal Demarcation Board codes (CTP, WC024)

Once the dataset has been sourced and shaped, it is ready to upload.

Uploading the dataset

Log into the backend administration section of the website and navigate to Datasets and click Add. Give the dataset file a meaningful name and select the applicable geographical boundaries and capture source information (this will be displayed to users to help them understand where the data came from). Proceed with uploading the file from your machine.

Uploading the file kicks off a background task to process the file.

The system will alert you once this is complete. You may also check on the status of the job by viewing the queue Django Q > queued tasks.

Once the file has been processed, you can proceed with .

Dataset permissions and sharing

Datasets can be marked Public or Private.

Public datasets, and variables derived from them, can be used on any profile in Wazimap. This enables reuse of valuable datasets without the need for each case to source and upload the data.

Private datasets and variables derived from them can only be used on the profile they belong to.

Qualitative datasets

Qualitative datasets can be uploaded and used to create qualitative indicators. A qualitative dataset must describe the relevant geography and provide the content. This content can be plaintext or HTML, see the example below on how the dataset should be laid out.

In the content type dropdown menu select "Qualitative". Then continue to in the same way as you would for a quantitative dataset.

It is possible to edit individual points within an existing point collection in Django. To do so, scroll down to the POINTS section, and this time click on Locations (see Figure 17, below).

The page that opens will contain all points for all Wazimap NG profiles. Points can be located in the same way as point collections:

1. If you know the point’s name, simply enter it in the Search bar (top left), and click Search;

2. Filter points by the associated Profile (top right), and locate the desired point in the list; or

3. Filter points by the point collection name.

Once located, click on the point’s name. On the page that opens, it is possible to edit a point’s name (see Figure 18, below), and its attributes (see Figure 19, below). Attributes appear in code format.

Referring to Figure 16, for each attribute associated with a point, there is a key and a value. A key corresponds to the column headings in the originally uploaded .csv file, and value to the cells in the respective column. For this reason, it is advisable NOT TO EDIT a key, but only a value.

Point collections refer to a number a different locations (points on a map) and are typically grouped by a specific subject matter. For example a dataset of schools in South Africa.

Collections are created by uploading a csv file and associating the file with a theme and sub-theme.

Preparing the points dataset

Prepare a csv file containing at least the following fields: Name, longitude, latitude. You can see an example below:

Variables are data points and can also be grouped for aggregate fields. These form the basis for the Profile Administrator to create Profile Indicators from.

Create new variable(s)

To create a new variable first ensure that the dataset file is uploaded and that it has been processed on the system. If this is the case, then proceed to Variables in the admin system and select Add to create a new one.

1. First, select the dataset this variable is found in, from the dropdown list and continue by clicking the Save and continue editing button.

2. Select the that this variable applies to (leave blank for the entire population). Note that you can also create a new universe at this point, if necessary, by clicking on the plus (+) sign next to the universe drop down.

3. Select which field(s) to group the variable by - you will notice that these are the data columns in your dataset file. These become sub-indicators. You will typically want one or perhaps two of these.

Variables are extracted as a background process and you will be alerted once they are complete.

Repeat this for as many variables as you need to create from your dataset and repeat for all your dataset files you have uploaded.

Once the variables have been created, the Profile Administrator can now proceed with creating and configuring the rest of the site.

Profile access permissions

Public profiles can be viewed by anyone on the internet.

Private profiles can only be viewed by users who are assigned permission for that profile. Users have to login to view that profile.

Categories can be created and managed by navigating to Indicator Categories in the admin site.

1. Provide a name for the category

2. Select the profile to be associated with this category (default is Youth Explorer but there might be others in the future)

Here are the steps to create a Profile Highlight (as shown in the image below):

STEP 1: Create and upload a dataset for the Profile Highlight of interest (e.g., Persons aged 60+), using the same method discussed under .

STEP 2: Create a variable from you uploaded dataset, using the same method discussed under .

STEP 3: Under the Profile Menu (in Django), select Add New Profile Highlight, and it will redirect you to the page, as show below.

STEP4:

It is important that users can tell whether a value in an indicator is zero, or missing.

Presenting the gap in the data is often just as helpful or important as presenting the data itself. It would also often misrepresent the facts to present a gap in the data as if it is zero.

On the other hand, it can be very inefficient to try to express every possible zero in a dataset. The simplest way to express zeros in a dataset is to simply have a row for every possible combination of attributes for each geographic area represented in the dataset. That would result in incredibly large datasets, where the value (Count column) of most of the rows would often tend to be zero.

Wazimap tries to support minimally-sized datasets by making some assumptions about the data, while also trying to support the presentation of missing data.

When to use

A profile indicator should be used when you want to display an indicator to the user. A profile indicator is presented as a bar chart in the rich data view and is also available under the data mapper.

If your indicator only has one subindicator - i.e. only one bar in the barchart, consider using a instead.

Dataset

In our context a dataset is a set of data with a geography attribute, a count attribute, and one or more dimension attributes. A dataset can be constructed from one or more files file structured in a specific format (see Creating Datasets) and uploaded to the dataset in wazimap. Datasets are the data source that Variables are created from. The platform allows for multiple datasets from various sources to be uploaded and displayed to users. Datasets are uploaded and managed by a Data Administrator.

Subindicator

In Wazimap a Subindicator is what we call an attribute value in one of the classifying columns or dimensions of a dataset.

In OLAP terms, a subindicator corresponds to a member of a dimension.

In statistics, a subindicator corresponds to categories of a categorical variable.

Subindicators are so named because they represent the choices offered when plotting a choropleth.

Subindicator Groups

A subindicator group represents the set of subindicators of a particular attribute or column in the original dataset.

In OLAP terms, a subindicator group corresponds to a dimension.

It corresponds directly to the columns in a dataset, other than the Geography and Count columns.

Variables

Variables are datapoints used to create profile indicators from and are created by the Data Administrator. Multiple variables can be created from the same dataset.

Most of the time, a variable simply exposes a subindicator group for use as categories in an indicator.

Variables exist for more complicated cases where the way percentages need to be calculated using a different population than simply the total of all subindicators. This is done by associating a universe to the variable.

Universe

Universe refers to the population to which the indicators are applied. There can be multiple universes if required and it can also be left blank to apply to the entire population.

Profile Indicator

Profile indicators are created by the Profile Administrator and are presented to the user on the website. Indicators belong to categories (e.g. Demographics) and can belong to sub-categories (e.g. General Population). In addition, they can also have sub-indicators (e.g. Age could take the individual age brackets as sub-indicators).

Key Metric

Key metrics are values of significance as decided upon by the Profile Administrator. These are used to showcase and callout highlighted values both in the rich data (profile) view, as well as on the map view. Key metrics can be shown as a percentage or absolute numbers as defined by the Profile Administrator.

Data Mapper

The Data Mapper provides an interface for users to plot indicators on the map. Only indicators available for plotting are shown and these might change depending on the geography level and that data available for that level. Please note that all indicators are shown in the Rich Data View.

Rich Data View

What was once referred to as a profile view on Wazimap is now the Rich Data View and provide charted exploration of the available data indicators. This view also reveals the source of each indicator along with a description for the categories and indicators (optional and set by Profile Administrators). This view also allows a chart to be downloaded (to be used elsewhere) and will soon allow for a chart to be embedded and for data to be downloaded. The Rich Data View also supports a print-friendly view allowing for easier sharing and dissemination.

Point Menu

The point menu houses point data themes and collections and allows for these to be overlaid on the map.

Point Data

Point data refers to coordinate based data rather than a dataset shaped within a geographical boundary. This allows for points to be overlaid on various other indicators.

This page is a stub - please add documentation

Point themes and Profile Collections are vital in displaying data using the Point Mapper on Wazimap. Themes are similar to the Indicator Categories in the Data Mapper while Profile Collections are similar to the Profile Indicators of the Data Mapper.

Themes

Themes can be created and managed by navigating to Themesunder the Pointssection of the Wazimap-NG admin page.

1. Select which profile you want to create the theme for

2. Provide a name for the theme

3. Select an appropriate icon to display

Theme colour is currently fixed according to the order of themes:

Theme colour will be more configurable in a future update. Please let us know what your needs are.

Profile Collections

Profile Collections can be created and managed by navigating to Themesunder the Pointssection of the Wazimap-NG admin page.

1. Select which profile you would like the Profile Collection to be associated with

2. Select which Theme you would like the new Profile Collection to fall under

3. Select the Point Collection that has the data you would like to represented by the Profile Collection

NOTE: Currently the Profile Collection Icon and Colour functionality is not working. These fields can be left empty.

Sources

• Label data sources as {{ dataset title }} - {{ organisation }}so that users can more easily find the right data source.

• Link to the official page about the actual dataset, if one exists, otherwise to the homepage of the source organisation.

• Prefer writing source names in full, rather than abbreviated. e.g. Our world in data rather than OWID

• See if they have a preferred way to be cited and try to use that.

Number formatting

Percentage raw data

• Select Value as the default between Value/Percentage presentation in the chart/table

• Disable the Value/Percentage toggle

Ratios

• Use "" as the format string - as in - don't use the SI unit formatting, because then 0.789 will look like 789m and people will think it means millions.

Decimals

If you're going to use format strings that format with SI units, ensure you configure no more than 2 decimal places or 3 significant digits, otherwise you could end up with numbers presented as 12.345k and people will misread the . as a thousand separator due to the three decimal places and think this is 12 345 000

Phone numbers in CSVs

Cells and columns in CSV don't have well-defined types so programs reading those CSVs generally infer the type from the values.

This can be a problem when opening a file with phone numbers which often start with a zero, and look like a number to programs reading CSVs.

When reading a CSV file, see if you can specify that such columns should be read as Text rather than letting the program infer the type.

After reading, if the columns were read as text, the zero-prefixes will remain. If the program read it as numbers, the zero prefix will have been lost.

To check that the file was saved correctly, you can open it in a text editor like Notepad to check that the zeros are still there:

Custom events

Formatting standards for display purposes

Number of people

• Ensure any data pertaining to people are rounded to the nearest whole number.

• Do not use "M" and "k". Use full figures with thousands-separator.

• Examples:

  • data entry refers to 104678.9 people --> this should be displayed as 104,679.

  • data entry refers to 109987789.49 people

Financial years

Full values

Prefer full financial years, as in 2016-2017 rather than 2017 or 2016-17. This is because

• Most people don't know that in the municipal sphere (unlike national) 2017 means 2016-2017

• Many people won't realise that the -17 means the next year - writing it in full is much easier to understand correctly the first time.

Subindicator ordering

Order financial years in subindicator groups in reverse chronological order - that is 2018-2019 and then 2017-2016. People are most often interested in the latest financial year.

Non-aggregatable

Always mark the financial year column as non-aggregatable. Summing over financial years usually doesn't make sense for an abritrary number of financial years and can easily lead to surprises for users.

Default filter

When the financial year column is not the variable, add a default filter to match the latest financial year. Marking financial year as non-aggregatable will already add a filter for it, but adding default filter configuration will ensure that a sensible value is selected in the filter.

Age bands

The reasoning for our preferred age bands is as follows:

• Ideally bands should align with voting age

• Ideally bands should align across datasets e.g. demographics should align with election data

• perhaps 0-18, 18-19, 20-29, 30-39, 40-49, 50-59, 60+

Sources

Label data sources as {{ dataset title }} - {{ organisation }}so that users can more easily find the right data source.

as opposed to

Geography Hierarchy = World

Africa