Reconcile question numbers across waves

Handling questions that change between survey waves

Question numbers and response values sometimes change between waves of a survey. This tutorial shows how to reconcile those differences so you can compare results consistently across waves.

Ideally, question numbers and response options would carry the same meaning in every wave. New questions and response options would get new identifiers, and retired ones would never be repurposed. In practice, things aren't always that clean.

Protobi data processes give you a way to reconcile question numbers and response options programmatically, so your data stays consistent across waves. A good place to start is a worksheet that lists questions and responses as rows and maps their identifiers wave by wave. We refer to this as a question mapping. This tutorial shows how to read and apply that mapping in Protobi.

For a simpler example of stacking waves without remapping, see Combine data.

Example

In this example there are two data tables, wave1 and wave2. The project was built using the wave2 data, and the goal is to bring in all data columns from wave1. Two columns were renamed between waves — but since they represent the same questions, we want Protobi to recognize the wave1 values as belonging to the renamed wave2 columns.

Create a data process

Create a new data process and paste the full code from the example below into it. Make sure to set the process as the Primary data source for your project after running it.

Once you run the code, values from renamed wave1 columns will be correctly mapped to their corresponding wave2 columns.

How the code works

Config

The config object is where you define your column mappings. Each entry is a target-origin pair — the target (left) is the column where you want the data to appear, and the origin (right) is where it should be pulled from. Target names typically come from the most recent wave and origin names from the historical wave.

Exact substitutions

For direct one-to-one column matches, list them explicitly inside config. For example, S50r1 maps exactly to S50_1.

Pattern substitutions

If you don't have a complete column-by-column mapping, you can use RegExp patterns to match groups of columns automatically. For instance, if S40_1, S40_2, and S40_3 from wave1 should map to S20r1, S20r2, and S20r3 in wave2, you can define a pattern that handles the whole group in one rule.

Converter function

The Converter function takes your config object and returns a convertRow function that applies the mappings to each row of data.

Applying the converter

At the bottom of your code, call the converter to produce an updated version of your historical data:

let wave1converted = wave1.map(convertRow)

This creates a new version of wave1 with all column names reconciled to match wave2, which can then be stacked cleanly.

Full code example

const tables = await Protobi.get_tables(["wave2", "wave1"])
let wave2 = tables['wave2']
let wave1 = tables['wave1']

let config = {
    // Exact substitutions — new name on left, old name on right
    "S20": "S40",
    "S50r1": "S50_1",
    "S60r1": "S60_1",
    "S60r2": "S60_2",
    "S60r3": "S60_3",
    "S60r4": "S60_4",

    // Pattern substitutions — XX* matches "XX", "XX_1", "XX_2", etc.
    "S10*": "",
    "S40*": "S30",
    "S20*": "S40",
    "S30*": "",
    "S50*": "S50",
    "S60*": "S60",
    "S70*": "S40",
    "S80*": "S80",
}

let Converter = function(config) {
    return function convertRow(row) {
        let rowKeys = Object.keys(row)
        let result = {}
        for (let left in config) {
            let right = config[left]
            if (right) {
                if (left.endsWith('*')) {
                    let leftPart = left.slice(0, -1)
                    let reRight = new RegExp("^(" + right + ")(_(\\d+))?")
                    let matchKeys = rowKeys.filter(key => reRight.test(key))
                    if (matchKeys.length) {
                        for (let rightKey of matchKeys) {
                            let match = rightKey.match(reRight)
                            if (match) {
                                let r = match[3]
                                let leftKey = r ? [leftPart, "r", r, "c", 1].join('') : leftPart
                                result[leftKey] = row[rightKey]
                                details[leftKey] = rightKey
                            }
                        }
                    }
                } else {
                    result[left] = row[right]
                }
            }
        }
        return result
    }
}

let convertRow = new Converter(config)
let wave1converted = wave1.map(convertRow)
console.log("Pattern-based replacements", details)

let stacked = Protobi.stack_rows([wave2, wave1converted])

return stacked;

If you have questions about applying this to your specific project, reach out to us at support@protobi.com.