Metadata driven UI

Motivation

We wanted to create a unified way to add filters and controls for all our plots in the front end. Previously there were different approaches to this for each plot but we wanted to create a unified way to reduce code duplication, make the interface consistent and mean enhancements to one plot can easily be ported over to all others. The drop downs in the side bar are split into two sections, plot controls and filters.

1. controls - dropdowns that influence other dropdowns
2. filters - dropdowns that act as filters for slicing the data

The controls can run effects to modify the filters such as:

• Set specific filters as multi-selects. For example, in the barchart users can select an X Axis and Disaggregate by controls which set whatever is selected into a multi-select dropdown, while keeping the rest as single selects. See barchart screenshot
• Set selected value for a filter e.g. if users select an indicator we might want to select specific sex and age groups which have data for this indicator e.g. ANC indicators should be female and 15-49.
• Set filters e.g. ART data has a column for period, ANC data has a column for year, we want to show the period for filter for ART only and the year filter for ANC only
• Filters can be hidden if we want to simplify the UI to remove clutter for users whilst ensuring we still filter the underlying data
• Set custom effects that the plot impl can pick up on and use to control the plot UI e.g. in the table we have a custom effect which defines for each control what is displayed as a header and as a row in the table view.

Below are some examples of the plot control and filter UIs that the metadata allow us to build

Implementation overview

The backend (this repository) provides metadata JSON endpoints that tell the frontend how to construct the UI and also how to dynamically update it based on the value of the dropdowns.

For details on where metadata is currently served from see Endpoints section.

Reminder that there are two types of dropdowns that can be created via metadata:

1. Dropdowns that influence other dropdowns - we will call these controls and call their values settings
2. Dropdowns that acts as filters for slicing the data - we will call these filters and their values selections

Here is a useful visualisation of the schema that I’ll keep referring to so that we don’t get lost in the JSON nesting (note that indicators is not expanded as this just contains metadata about the indicator values such as their format which isn’t relevant to the metadata structure):

├── filterTypes[]
│   ├── id
│   ├── column_id
│   ├── use_shape_regions
│   ├── options[]
│   │   ├── label
│   │   ├── id
│   │   ├── description
├── indicators
├── plotSettingsControl{}
│   ├── defaultEffect
│   │   ├── setFilters
│   │   ├── ... (see effect below)
│   ├── plotSettings[]
│   │   ├── label
│   │   ├── id
│   │   ├── hidden
│   │   ├── value
│   │   ├── options[]
│   │   │   │   ├── id
│   │   │   │   ├── label
│   │   │   │   ├── effect
│   │   │   │   │   ├── setFilters
│   │   │   │   │   ├── setMultiple
│   │   │   │   │   ├── setFilterValues
│   │   │   │   │   ├── setHidden
│   │   │   │   │   ├── customPlotEffect

This is just a full version of the json schema.

Filters

├── filterTypes[] # here
│   ├── id # 1
│   ├── column_id # 2
│   ├── use_shape_regions # 3
│   ├── options[] # 4
│   │   ├── label
│   │   ├── id
│   │   ├── description
├── indicators
├── plotSettingsControl{}
│   ├── defaultEffect
│   │   ├── setFilters
│   │   ├── ... (see effect below)
│   ├── plotSettings[]
│   │   ├── label
│   │   ├── id
│   │   ├── hidden
│   │   ├── value
│   │   ├── options[]
│   │   │   │   ├── id
│   │   │   │   ├── label
│   │   │   │   ├── effect
│   │   │   │   │   ├── setFilters
│   │   │   │   │   ├── setMultiple
│   │   │   │   │   ├── setFilterValues
│   │   │   │   │   ├── setHidden
│   │   │   │   │   ├── customPlotEffect

The filterTypes key is the simplest. It is an array of all the filters that can be displayed in the UI with their options. Explanations for each number in the visualisation:

1. The id is just an id we can assign to the filter to refer to it later
2. The column_id this is the column id of the data which we have to filter
3. use_shape_regions is a boolean that should largely be ignored, it is a one off trick for geographical area filters only to reduce the amount of payload we send to the frontend by telling it to derive the options of a filter select with this boolean from the shape file that the user has uploaded
4. The options are the options visible to the user in the dropdown. Each option has an id by which filter selections can refer to the option, a label to display in the dropdown menu and an optional description prop for contextual information (this doesn’t get displayed to the user at all)

Plot Settings Control

├── filterTypes[]
│   ├── id
│   ├── column_id
│   ├── use_shape_regions
│   ├── options[]
│   │   ├── label
│   │   ├── id
│   │   ├── description
├── indicators
├── plotSettingsControl{} # here
│   ├── defaultEffect # 1
│   │   ├── setFilters
│   │   ├── ... (see effect below)
│   ├── plotSettings[] # 2
│   │   ├── label # 3
│   │   ├── id # 4
│   │   ├── hidden # 5
│   │   ├── value # 6
│   │   ├── options[]
│   │   │   │   ├── id # a
│   │   │   │   ├── label
│   │   │   │   ├── effect
│   │   │   │   │   ├── setFilters
│   │   │   │   │   ├── setMultiple
│   │   │   │   │   ├── setFilterValues
│   │   │   │   │   ├── setHidden
│   │   │   │   │   ├── customPlotEffect

The plot settings controls is an object with keys of the relevant plot name (e.g. choropleth) and value with the structure shown in the visualisation. Explanation of keys:

1. defaultEffect is the base effect applied to all settings (remember these are the values of the control dropdown) - more on this later
2. plotSettings is an array of all the controls a tab will have
3. label is the title of the dropdown the user will see in the UI
4. id is the id of the plot settings (this id is how the store keeps track of which setting of the control is selected)
5. hidden boolean - there are times we want the control to be invisible. The calibrate plot reuses the barchart component, however they expect an X Axis and Disaggregate by control - these controls are fixed in the calibrate plot so we set them to a constant value and hide them from the user allowing us to reuse the barchart component without any special code
6. value is an optional key to set the default value of the control as something other than the default (default in frontend for single select is the first value and for a multi-select is all the values). This is the id of the option (a in the visualisation)

Settings

├── filterTypes[]
│   ├── id
│   ├── column_id
│   ├── use_shape_regions
│   ├── options[]
│   │   ├── label
│   │   ├── id
│   │   ├── description
├── indicators
├── plotSettingsControl{}
│   ├── defaultEffect
│   │   ├── setFilters
│   │   ├── ... (see effect below)
│   ├── plotSettings[]
│   │   ├── label
│   │   ├── id
│   │   ├── hidden
│   │   ├── value
│   │   ├── options[] # here
│   │   │   │   ├── id # 1
│   │   │   │   ├── label # 2
│   │   │   │   ├── effect # 3
│   │   │   │   │   ├── setFilters
│   │   │   │   │   ├── setMultiple
│   │   │   │   │   ├── setFilterValues
│   │   │   │   │   ├── setHidden
│   │   │   │   │   ├── customPlotEffect

The settings completely determine the filters and their selections. The next section will go through each effect and how the frontend resolves it but for now, explanation of keys:

1. id for the store to record what setting is chosen
2. label displayed to the user in the dropdown menu
3. The effect key is probably the most important idea here. Every setting has an effect associated with it and aggregating all the effects along all the settings the user has chosen completely determines the filters and their initial selections that are shown to the user. So everything can be derived from just the settings that are chosen

Effects

Effects are the actions that can be undertaken when a plot control is selected. They can be used to modify the filters and set custom effects which can be used by the plot.

├── filterTypes[]
│   ├── id # a
│   ├── column_id
│   ├── use_shape_regions
│   ├── options[]
│   │   ├── label
│   │   ├── id # b
│   │   ├── description
├── indicators
├── plotSettingsControl{}
│   ├── defaultEffect # (here as well)
│   │   ├── setFilters
│   │   ├── ... (see effect below)
│   ├── plotSettings[]
│   │   ├── label
│   │   ├── id
│   │   ├── hidden
│   │   ├── value
│   │   ├── options[]
│   │   │   │   ├── id
│   │   │   │   ├── label
│   │   │   │   ├── effect # here
│   │   │   │   │   ├── setFilters
│   │   │   │   │   ├── setMultiple
│   │   │   │   │   ├── setFilterValues
│   │   │   │   │   ├── setHidden
│   │   │   │   │   ├── customPlotEffect

For this section we will need a more detailed view of the effects structure:

├── setFilters[] # 1
│   ├── filterId # 1.1
│   ├── label # 1.2
│   ├── stateFilterId # 1.3
├── setMultiple[] # 2
├── setFilterValues{} # 3
├── setHidden[] # 4
├── customPlotEffect{} # 5

Explanation of each effect and how the frontend aggregates this:

1. setFilters is how the effect specifies what filters the user sees from the filterTypes. Only one setFilters effect is used, they are not combined. So if two settings set filters, the last setFilters in the for loop would be used when going through all the settings. Keys:
   1. filterId is the id key in a filterTypeselement (a in the schema above)
   2. label is the title of the filter displayed to the user in the UI
   3. stateFilterId is the id the state uses to identify this filter. This is distinct to the filterId because in the bubble plot we have two indicators filters that are the same filter type (so have the same options and the same column_id) however they are distinct because one controls the color of the bubbles and one controls the size of the bubbles so while both their filterIds are indicator, the stateFilterIds for both are different, colorIndicator and sizeIndicator. For most cases this can be set as the same thing as the filterId. All other effects will now identify filters by stateFilterId as these uniquely correspond to what the user sees
2. setMultiple is an effect that converts a filter to a multi-select dropdown (they are all single select by default). It is an array of stateFilterIds (1.3 in the schema) This is aggregated via concatenation in the frontend so if two settings have setMultiple properties ["state_id_1", "state_id_2"] and ["state_id_3"] respectively then all three filters with those state ids will be set to multi-selects
3. setFilterValues is an effect that sets the selection of the filters in setFilters. It is an object with keys equal to stateFilterId and value an array of ids of the selections (b in the schema). Note this is still an array of length one if you want to specify a value for a single select filter. This is aggregated by the spread operator in the frontend
4. setHidden is an effect that hides filters and is usually used when we have a fixed filter that we don’t want the user to change but is still required to slice the data. This is aggregated via concatenation in the frontend
5. customPlotEffect is an effect that plots can use to build reactive UI from for a specific plot type. At the moment we are only using this in the table, so set the row and header groups. But it could be expanded to general metadata which we want a plot control to update, but does not have an effect on the filters themselves.

Typescript definition and derived frontend store

I have used the directory-structure-esque visualisation for most of this because it is simple and obscures a lot of the types and other information as that would be too much to throw. But now that you know how the metadata works, here is the full typescript definition of it with contextual types (instead of just string, it shows which type of string it refers to, e.g. whether it is stateFilterId or filterId):

type FilterId = string
type StateFilterId = string
type ColumnId = string
type SelectionId = string
type PlotControlId = string
type PlotSettingId = string
type PlotName = string

type FilterOption = {
    id: SelectionId,
    label: string,
    description: string
}

type FilterType = {
    id: FilterId,
    column_id: ColumnId,
    options: FilterOption[]
}

type FilterRef = {
    filterId: FilterId,
    label: string,
    stateFilterId: StateFilterId
}

type PlotSettingEffect = {
    setFilters?: FilterRef[],
    setMultiple?: StateFilterId[],
    setFilterValues?: Record<StateFilterId, SelectionId[]>,
    setHidden?: StateFilterId[],
    customPlotEffect?: any
}

type PlotSettingOption = {
    id: PlotSettingId,
    label: string,
    effect: PlotSettingEffect
}

type PlotSetting = {
    id: PlotControlId,
    label: string,
    hidden?: boolean,
    value?: PlotSettingId,
    options: PlotSettingOption[]
}

type PlotSettingsControl = {
    defaultEffect: PlotSettingEffect,
    plotSettings: PlotSetting[]
}

type MetadataStructure = {
    filterTypes: FilterType[],
    indicators: any, // irrelevant 
    plotSettingsControl: {
        // example of plotName can be choropleth or table, etc
        [plotName: PlotName]: PlotSettingsControl
    }
}

type FilterSelection = {
    multiple: boolean
    selection: FilterOption[],
    hidden: boolean
} & FilterRef

type ControlSelection = {
    id: PlotControlId,
    label: string,
    selection: FilterOption[]
}

type FrontendSelectionState = {
    // these plotNames keys in the type are the generated keys from
    // the json schema in Hintr
    [plotName: PlotName]: {
        controls: ControlSelection[]
        filters: FilterSelection[]
    }
} & { error: Error }

type FrontendDataState = {
    // just an array of filtered data
    [plotName: PlotName]: any[]
}

Endpoints

For each data source in the frontend UI there is a different endpoint and associated JSON schema that specifies the metadata. Note this will probably go stale but should give a starting point

(*) Kotlin backend gets the path to the duckdb output file using hintr endpoint at /calibrate/result/path/<id>. It then slices this using SQL based on the filters the user set in the front end. So we only return the slice of data the user is looking at.

Note that data and metadata can be in the same endpoint or different ones. Favour returning them from the same endpoint unless splitting them up will significantly speed up the UI e.g. returning metadata for output plots all together saves reading the output and building the filters from this multiple times. In general, I think having the data & metadata returned together is simpler.

Front end implementation

This section gives an overview of how the front end handles the metadata and where any custom logic for a specific plot should be injected. The aim of the front end impl is that the wiring up of state will be as straight forward as possible, and we get as much code use as possible.