Wazimap NG (Next Generation) provides easy access to different kinds of data through a mapped and open-source information system, designed to help non-technical users explore data, both meaningfully and in context. In other words, a Geo-Information Spatial (GIS) Tool “for the rest of us”.

This Profile Curation Handbook contains documentation on how to manage and upload datasets, as well as create and manage profiles and their associated indicators. There are two administration roles:

1. Data Administrator — responsible for sourcing, shaping, and uploading datasets and point collections to Wazimap NG.

2. Profile Administrator — responsible for defining and managing profile indicators, and site overall content. In short, responsible for how data looks on Wazimap NG.

Both these roles have a function in Wazimap NG’s three views. It is, however, possible for these roles to be performed by the same person. The three views are Point Mapper, Data Mapper, and Rich Data. The table below shows the respective logos for each view of Wazimap NG.

The rest of this Profile Curation Handbook is structured according to these three views of Wazimap NG, and the various roles that Data and Profile Administrators play for each.

Figure 1, below, is an example of such point mapping, and shows the location of Water Treatment Works within the City of Cape Town Metro of South Africa.

The first step towards creating such a map is sourcing and understanding your data, and knowing what you are trying to display using your data. This is often the hardest part, and requires collaboration between Data and Profile Administrators. It is also important to know a dataset’s Terms of Use or Licence, and what this allows you to use the data for.

Wazimap NG uses .csv files to display point collections, and these .csv files require a particular shaping (formatting) to be properly understood by the Wazimap NG platform. This is the first step towards creating your map.

To create a Theme, scroll down to the POINTS section again, and this time click on the +Add button next to Themes (see Figure 7, below).

On the following page, do the following (refer to Figure 8, below):

1. Select (from the dropdown list) the Profile which should be associated with your Theme;

2. Give your Theme an appropriate Name (this will be displayed on Wazimap NG);

3. Select a preferred Icon; and

4. Scroll down and click Save in the bottom right hand corner.

On Wazimap NG, the theme colour is configurable. Simply add the of your choice. The next step is to create the Profile Collection (from your point collection), that will be nested under your newly created Theme.

The easiest tool to create or edit .csv files is Google Sheets — it can open both .csv and .xlsx files, and can export .csv files for upload to Wazimap NG. Any point collection requires the following three fields (as highlighted in red in Figure 2, below):

1. name;

2. longitude; and

3. latitude.

As shown in Figure 2, above, additional fields can be included as attributes for each point in the collection. In the Point Mapper view, these additional attributes are displayed in the More Info section of each point’s tooltip (refer to ).

The name field (column) of your point collection cannot be formatted as a number. It may be necessary to format the entire column to plain text by selecting the entire column and clicking Format > Number > Plain text (see Figure 3, below).

Additionally, column headings cannot start with a number (e.g., 2019 HDI), contain periods (e.g., SDG 3.1), or contain square brackets (e.g., HDI over the Years [Average]).

Once your point collection is complete (has the three required fields, is formatted correctly, and contains no blank cells in the first row), it can be exported as a .csv file. To do this, click File > Download > Comma-separated values (.csv) (see Figure 4, below).

Data is uploaded to Wazimap NG using Django (which can be accessed ). Once you are logged in, scroll down to the POINTS section, and click on the +Add button next to Collections (see Figure 5, below).

On the following page, do the following (refer to Figure 6, below):

1. Select (from the dropdown list) the Profile

Preparing the dataset

Before a dataset can be uploaded, the data needs to be cleaned and shaped into the correct format. As a rule, the more disaggregated the dataset, the better as this allows a single dataset to be (re)used for multiple indicators and also allows for multivariable analysis.

The system accepts files in csv, xls and xlsx formats. The file needs to adhere to a specific structure and ensure it always contains the following fields:

Inside the Geography column, valid values are

• country code (ZA)

• province codes (GAU, LIM, WC, etc.)

• Municipal Demarcation Board codes (CTP, WC024)

• Ward IDs (10204020 for Stellenbosch Ward 20 in 2016 demarcation)

• or the lower level numerical geography code (e.g. 160001, 175005, etc.)

Once the dataset has been sourced and shaped, it is ready to upload.

Uploading the dataset

Log into the backend administration section of the website and navigate to Datasets and click Add. Give the dataset file a meaningful name and select the applicable geographical boundaries and capture source information (this will be displayed to users to help them understand where the data came from). Proceed with uploading the file from your machine.

Uploading the file kicks off a background task to process the file.

The system will alert you once this is complete. You may also check on the status of the job by viewing the queue Django Q > queued tasks.

Once the file has been processed, you can proceed with .

Dataset permissions and sharing

Datasets can be marked Public or Private.

Public datasets, and variables derived from them, can be used on any profile in Wazimap. This enables reuse of valuable datasets without the need for each case to source and upload the data.

Private datasets and variables derived from them can only be used on the profile they belong to.

Qualitative datasets

Qualitative datasets can be uploaded and used to create qualitative indicators. A qualitative dataset must describe the relevant geography and provide the content. This content can be plaintext or HTML, see the example below on how the dataset should be laid out.

In the content type dropdown menu select "Qualitative". Then continue to in the same way as you would for a quantitative dataset.

It is possible to upload additional points to an existing point collection without having to replace the entire point collection. To do so, first prepare a .csv file with the new points to be added, as outlined under Shaping Data for Point Collections.

Next, in Django, scroll down to the POINTS section, and this time click on Collections (see Figure 15, below).

The page that opens will contain all point collections for all Wazimap NG profiles. There are two ways to locate an existing point collection (see Figure 16, below):

1. If you know the point collection’s name, simply enter it in the Search bar (top left), and click Search; or

2. Filter point collections by the associated Profile (top right), and locate the desired point collection in the list.

Once located, click on the point collection’s name. The same page as shown in , earlier, will open. Next, import your new points by clicking Choose file and selecting the .csv file with the new points to be added. Then, hit Save. After refreshing your Wazimap NG Profile, the new points should appear on the map.

To create a Profile Collection, scroll down to the POINTS section again, and this time click on the +Add button next to Profile Collections (see Figure 10, below).

On the following page, do the following (refer to Figure 11, below):

1. Select (from the dropdown list) the Profile which should be associated with your Profile Collection;

It is possible to edit individual points within an existing point collection in Django. To do so, scroll down to the POINTS section, and this time click on Locations (see Figure 17, below).

The page that opens will contain all points for all Wazimap NG profiles. Points can be located in the same way as point collections:

1. If you know the point’s name, simply enter it in the Search bar (top left), and click Search;

Long term maintenance to a point collection often involves the following tasks:

1. Removing points that are no longer relevant;

2. Adding points that are not already in the database; and

3. Updating data about points already in the database (e.g., opening hours, services provided, correcting/improving a description).

A universe refers to the population against which the variable is being applied. Universe can also be left as blank which would then apply to everyone.

To create universes, you will need to write a bit of json.

The structure of this is:

{
"Variable Name": "Filter value",
"Another variable__in": ["array", "of", "values"]
}

See below for an example of youth age range universe.

{
  "Age Group__in": [
    "15-19",
    "20-24",
    "25-29",
    "30-35"
  ]
}

This universe when applied to a variable, would include all people within the ages of 15 to 35.

Variables are data points and can also be grouped for aggregate fields. These form the basis for the Profile Administrator to create Profile Indicators from.

Create new variable(s)

To create a new variable first ensure that the dataset file is uploaded and that it has been processed on the system. If this is the case, then proceed to Variables in the admin system and select Add to create a new one.

1. First, select the dataset this variable is found in, from the dropdown list and continue by clicking the Save and continue editing button.

2. Select the that this variable applies to (leave blank for the entire population). Note that you can also create a new universe at this point, if necessary, by clicking on the plus (+) sign next to the universe drop down.

3. Select which field(s) to group the variable by - you will notice that these are the data columns in your dataset file. These become sub-indicators. You will typically want one or perhaps two of these.

4. Give your variable a meaningful name.

5. Click Save .

Variables are extracted as a background process and you will be alerted once they are complete.

Repeat this for as many variables as you need to create from your dataset and repeat for all your dataset files you have uploaded.

Once the variables have been created, the Profile Administrator can now proceed with creating and configuring the rest of the site.

Point collections refer to a number a different locations (points on a map) and are typically grouped by a specific subject matter. For example a dataset of schools in South Africa.

Collections are created by uploading a csv file and associating the file with a theme and sub-theme.

Preparing the points dataset

Prepare a csv file containing at least the following fields: Name, longitude, latitude. You can see an example below:

In addition to these fields, various other fields can also be included as attributes for the point. These are shown on the point tooltip. An example of a file with additional attributes shown below:

Format of the fields

The "Name" column of your point collection cannot be a number data type. It may be necessary to convert the column to a text/string format before uploading. This can be done easily in Microsoft Excel and is illustrated in the figure below.

NOTE: It may sometimes be more suitable to change which column is given the "Name" heading as this is what is displayed in the tooltip on the map. In the screenshot below, the "Description" column would be a more appropriate choice to be made the "Name" column.

Uploading points and assigning to themes

Once the file is ready to be uploaded, navigate to Point Collections and click Add which will allow you to name the collection and upload the file.

Select the profile which should be associated with these points and select the theme they belong to. Provide a label and the source of the data and proceed to upload.

Uploading additional Points data

Uploads to existing point collections add data without replacing existing points.

To add additional points to a point collection, prepare the file with the new points to be added, as outlined in the Preparing the points dataset section.

Select the point collection you would like to update, upload the file with the new additional point data, and proceed to save the changes.

Editing Points data

You can edit information for a specific point by selecting it under Points > Locations.

To rectify/revise/update previously updated data in bulk, delete the existing point collection and recreate the adjusted file containing all the data.

Bulk updates to a points dataset

Long term maintenance to a point dataset often involves the following tasks:

• removing points that are no longer relevant

• adding points that are not already in the database

• updating data about points already in the database (e.g. opening hours, services provided, correcting/improving a description)

To be able to do this, you need to be able to compare your updated data to the data you have already uploaded to a collection in Wazimap.

To match incoming data to your existing data, you need an identifier. Failing that, you will need to use whatever other identifying information you have available.

Identifiers

An identifier is a value which uniquely and consistently identifies an object - in this case, a point in your point collection.

It is important to include some kind of identifier in your point data to facilitate updates to the data. Downstream users may also need to be able to rely on your identifiers being unique and consistent to be able to incorporate your updates if they keep a copy of your point data.

If your data does not have an official identifier that is consistent over time, and unique per point, think about how you will check if you have any duplicates and check whether newly-provided points are already in your database - e.g. by name, address, and so on. Think about what your users will be able to provide you, and include that in what is shown to them as well.

Consider making up and maintaining your own unique consistent identifier. We suggest using UUIDs because they are globally unique, and not just unique within one table. This is important in case you want to combine tables later, e.g. if you want to merge "public schools" and "private schools" into one "schools" table and continue to use your unique identifier.

You can use the following formula in excel to create a UUID:

When filled down a column, these identifiers will look like

Make sure to copy and paste as text (not the formula) into a new column and use the text version of it going forward. You don't want it to calculate a new random UUID for existing points.

How to check for duplicates

In Excel, see

In Workbench

1. add a deduplication step

2. select a column whose values probably ought to be unique

3. look for rows where duplicate number is greater than 1

How to merge in updates

If you have a consistent unique identifier:

• In Excel, use VLOOKUP()

• In Workbench, perhaps Join Tabs might work

If you have messy data with different capitalisation and potential spelling mistakes:

• Try using OpenRefine with CSV reconcilliation

How to check for new records to import

How to check for stale records

Here are the steps to create a Profile Highlight (as shown in the image below):

STEP 1: Create and upload a dataset for the Profile Highlight of interest (e.g., Persons aged 60+), using the same method discussed under Creating Datasets.

STEP 2: Create a variable from you uploaded dataset, using the same method discussed under Creating Variables.

STEP 3: Under the Profile Menu (in Django), select Add New Profile Highlight, and it will redirect you to the page, as show below.

STEP4: Select your Profile from the drop-down list.

STEP 5: Select the Variable you previously created from your dataset.

STEP 6: Add a label or title for the Profile Highlight.

STEP 7: Select the Sub-Indicator to be displayed.

STEP 8: Under Denominator, select Absolute Value.

STEP 9: Scroll down, hit Save, and refresh the Wazi Profile. Your Profile Highlight should now show the value for the selected geography.

Categories can be created and managed by navigating to Indicator Categories in the admin site.

1. Provide a name for the category

2. Select the profile to be associated with this category (default is Youth Explorer but there might be others in the future)

3. Provide descriptive text explaining what the category contains and any other details relevant to the users. This is displayed in the Rich Data View.

Ordering

Categories, subcategories, and profile indicators can be reordered by dragging the handle in the ordering column.

The ordering handle is only available when sorted by that column.

Reordering subindicators

It might be that you want to change the order in which the sub-indicators are shown. For example, you may want to swap Agree and Disagree in the chart below

In the Admin Suite, find SubindicatorsGroups

Find the relevant indicator:

This page is a stub - please add documentation

When to use

A profile indicator should be used when you want to display an indicator to the user. A profile indicator is presented as a bar chart in the rich data view and is also available under the data mapper.

If your indicator only has one subindicator - i.e. only one bar in the barchart, consider using a Profile Key Metric instead.

New Profile Indicators

1. To create new profile indicators, log into the backend admin section with your admin account and click Add next to Profile indicators.

2. Select the profile to associate this indicator with (there might only be a single one).

3. Select the variable on which this is based, from the dropdown list.

Display configuration

Additional configuration is possible, such as how numbers are formatted. Documentation can be found in the .

Geography Hierarchy = World

Africa

Profile access permissions

Public profiles can be viewed by anyone on the internet.

Private profiles can only be viewed by users who are assigned permission for that profile. Users have to login to view that profile.

Point themes and Profile Collections are vital in displaying data using the Point Mapper on Wazimap. Themes are similar to the Indicator Categories in the Data Mapper while Profile Collections are similar to the Profile Indicators of the Data Mapper.

Themes

Themes can be created and managed by navigating to Themesunder the Pointssection of the Wazimap-NG admin page.

1. Select which profile you want to create the theme for

2. Provide a name for the theme

3. Select an appropriate icon to display

Theme colour is currently fixed according to the order of themes:

Theme colour will be more configurable in a future update. Please let us know what your needs are.

Profile Collections

Profile Collections can be created and managed by navigating to Themesunder the Pointssection of the Wazimap-NG admin page.

1. Select which profile you would like the Profile Collection to be associated with

2. Select which Theme you would like the new Profile Collection to fall under

3. Select the Point Collection that has the data you would like to represented by the Profile Collection

4. Decide on the Label for the Profile Collection (this will be the text that is displayed in the front-end)

NOTE: Currently the Profile Collection Icon and Colour functionality is not working. These fields can be left empty.

Sources

• Label data sources as {{ dataset title }} - {{ organisation }}so that users can more easily find the right data source.

• Link to the official page about the actual dataset, if one exists, otherwise to the homepage of the source organisation.

• Prefer writing source names in full, rather than abbreviated. e.g. Our world in data rather than OWID

• See if they have a preferred way to be cited and try to use that.

Number formatting

Percentage raw data

• Select Value as the default between Value/Percentage presentation in the chart/table

• Disable the Value/Percentage toggle

Ratios

• Use "" as the format string - as in - don't use the SI unit formatting, because then 0.789 will look like 789m and people will think it means millions.

Decimals

If you're going to use format strings that format with SI units, ensure you configure no more than 2 decimal places or 3 significant digits, otherwise you could end up with numbers presented as 12.345k and people will misread the . as a thousand separator due to the three decimal places and think this is 12 345 000

Phone numbers in CSVs

Cells and columns in CSV don't have well-defined types so programs reading those CSVs generally infer the type from the values.

This can be a problem when opening a file with phone numbers which often start with a zero, and look like a number to programs reading CSVs.

When reading a CSV file, see if you can specify that such columns should be read as Text rather than letting the program infer the type.

After reading, if the columns were read as text, the zero-prefixes will remain. If the program read it as numbers, the zero prefix will have been lost.

To check that the file was saved correctly, you can open it in a text editor like Notepad to check that the zeros are still there:

It is important that users can tell whether a value in an indicator is zero, or missing.

Presenting the gap in the data is often just as helpful or important as presenting the data itself. It would also often misrepresent the facts to present a gap in the data as if it is zero.

On the other hand, it can be very inefficient to try to express every possible zero in a dataset. The simplest way to express zeros in a dataset is to simply have a row for every possible combination of attributes for each geographic area represented in the dataset. That would result in incredibly large datasets, where the value (Count column) of most of the rows would often tend to be zero.

Wazimap tries to support minimally-sized datasets by making some assumptions about the data, while also trying to support the presentation of missing data.

Current behaviour

Cases presented as missing data

Wazimap presents a value for a geographic area as "missing" when there are no rows of data for that geographic area in the dataset backing an indicator. As soon as there is one or more datum for a geography in an indicator, all subindicator and filter combinations will be presented as zero instead of missing. .

On a choropleth plotting hostpital beds per 1,000 people in countries in Africa, it would be wrong to plot countries with missing data as having zero beds.

Cases not explicitly presented as zero or missing

No data available for a given subindicator for the selected geography

When a subindicator does not occur in the data for a given geographic area, it will be excluded from the chart.

For example, when years are missing from an indicator on misspending, those years are not shown for that geography.

This behaviour is important in instances where an subindicator group can have a very large variety of different values, and only a small number are applicable to a specific geographic area. For example, election results showing the votes received by a party should only show the parties that contested that geographic area. If subindicators are included that did not contest that area, the chart would include hundreds of irrelevant items.

Some rows available for a subindicator for the selected geography, but not for every combination of filters

When a dataset does not contain explicit zero rows for a certain combination of subindicators, and a filter is applied excluding the available data, the subindicators without data are excluded from the chart.

Cases presented as zero

When a row in the dataset has the Count value of zero, it is of course presented as zero.

In the data mapper, when there is some data row for a given geographic area in an indicator, every combination of filters would be presented as zero even if there was no row in the dataset for that combination of attribute values.

In the example below, the number of white Tshivenda speakers is explicitly presented as zero, even though there is no such row in the dataset. The data mapper assumes that the data for a geography is complete if there is some row for that geography - perhaps for a different subindicator, or for the selected subindicator but for a different combination of filters..

No supported yet

Due to the assumptions shown above made by Wazimap about your data, we don't currently support the following. If there is demand, we can consider adding support, perhaps by making the behaviour configurable per dataset or indicator. We can potentially also help you shape your datasets and indicators to achieve your objectives within the above behaviour.

Explicit missing data in rich data view

The rich data view currently just hides rows without data for a given subindicator or filter selection. It does not support showing a label on a chart axis with a blank space to make the lack of data for that indicator visually explicit.

Partial data in the data mapper

The data mapper currently does not support partial data for a geography. If an indicator has one datum for that geography, all combinations of subindicator and filters will show zeros for that geography where other values are not available.

As a workaround, you can show gaps in an entire subindicator by separating a dataset into a dataset and indicator per subindicator. The indicators where no values are available for a geography will then present that geography as blank (grey).

Dataset

In our context a dataset is a set of data with a geography attribute, a count attribute, and one or more dimension attributes. A dataset can be constructed from one or more files file structured in a specific format (see Creating Datasets) and uploaded to the dataset in wazimap. Datasets are the data source that Variables are created from. The platform allows for multiple datasets from various sources to be uploaded and displayed to users. Datasets are uploaded and managed by a Data Administrator.

Subindicator

In Wazimap a Subindicator is what we call an attribute value in one of the classifying columns or dimensions of a dataset.

In OLAP terms, a subindicator corresponds to a member of a dimension.

In statistics, a subindicator corresponds to categories of a categorical variable.

Subindicators are so named because they represent the choices offered when plotting a choropleth.

Subindicator Groups

A subindicator group represents the set of subindicators of a particular attribute or column in the original dataset.

In OLAP terms, a subindicator group corresponds to a dimension.

It corresponds directly to the columns in a dataset, other than the Geography and Count columns.

Variables

Variables are datapoints used to create profile indicators from and are created by the Data Administrator. Multiple variables can be created from the same dataset.

Most of the time, a variable simply exposes a subindicator group for use as categories in an indicator.

Variables exist for more complicated cases where the way percentages need to be calculated using a different population than simply the total of all subindicators. This is done by associating a universe to the variable.

Universe

Universe refers to the population to which the indicators are applied. There can be multiple universes if required and it can also be left blank to apply to the entire population.

Profile Indicator

Profile indicators are created by the Profile Administrator and are presented to the user on the website. Indicators belong to categories (e.g. Demographics) and can belong to sub-categories (e.g. General Population). In addition, they can also have sub-indicators (e.g. Age could take the individual age brackets as sub-indicators).

Key Metric

Key metrics are values of significance as decided upon by the Profile Administrator. These are used to showcase and callout highlighted values both in the rich data (profile) view, as well as on the map view. Key metrics can be shown as a percentage or absolute numbers as defined by the Profile Administrator.

Data Mapper

The Data Mapper provides an interface for users to plot indicators on the map. Only indicators available for plotting are shown and these might change depending on the geography level and that data available for that level. Please note that all indicators are shown in the Rich Data View.

Rich Data View

What was once referred to as a profile view on Wazimap is now the Rich Data View and provide charted exploration of the available data indicators. This view also reveals the source of each indicator along with a description for the categories and indicators (optional and set by Profile Administrators). This view also allows a chart to be downloaded (to be used elsewhere) and will soon allow for a chart to be embedded and for data to be downloaded. The Rich Data View also supports a print-friendly view allowing for easier sharing and dissemination.

Point Menu

The point menu houses point data themes and collections and allows for these to be overlaid on the map.

Point Data

Point data refers to coordinate based data rather than a dataset shaped within a geographical boundary. This allows for points to be overlaid on various other indicators.

Formatting standards for display purposes

Number of people

These are the steps needed to generated datasets for Africa Data Hub.

The script of ADH can be found at https://github.com/OpenUpSA/wazimap-adh-data under the folder COVID Countries and Africa Admin 1.

Collection data

Download owid-covid-latest.csv from https://github.com/owid/covid-19-data/tree/master/public/data

Download from https://github.com/dsfsi/covid19za/tree/master/data the following files: covid19za_provincial_cumulative_timeline_confirmed.csv, covid19za_provincial_cumulative_timeline_deaths.csv, covid19za_provincial_cumulative_timeline_recoveries.csv

Download from https://data.humdata.org/organization/hera-humanitarian-emergency-response-africa ALL( for various countries) csv files that end with Coronavirus (Covid-19) Subnational eg. Niger: Coronavirus (Covid-19) Subnational. All the datasets from this source should be saved in a folder named HERA

For each csv file in HERA folder, change columns name from

['CONTAMINES', 'DECES', 'GUERIS', 'CONTAMINES_FEMME', 'CONTAMINES_HOMME', 'CONTAMINES_GENRE_NON_SPECIFIE']

These dataset should be saved under the folder wazimap-adh-data-main/COVID Countries and Africa Admin 1 After downloading the datasets, run the script as in the following order

Admin0_dataTransformation_v2.ipynb

Africa_Admin1_dataTransformation.ipynb

Data_Aggregation.ipynb

Data_Aggregation.ipynb will generate cases_monthly, death_monthly .

Custom events