Protobi

← All help posts

View post

## Handling questions that change between survey waves

Question numbers and response values sometimes change between waves of a survey. This tutorial shows how to reconcile those differences so you can compare results consistently across waves.

Ideally, question numbers and response options would carry the same meaning in every wave. New questions and response options would get new identifiers, and retired ones would never be repurposed. In practice, things aren't always that clean.

Protobi data processes give you a way to reconcile question numbers and response options programmatically, so your data stays consistent across waves. A good place to start is a worksheet that lists questions and responses as rows and maps their identifiers wave by wave. We refer to this as a question mapping. This tutorial shows how to read and apply that mapping in Protobi.

For a simpler example of stacking waves without remapping, see <a href="/110842-under-construction-internal/combine-data" title="">Combine data</a>.

---

## Example

In this example there are two data tables, **wave1** and **wave2**. The project was built using the **wave2** data, and the goal is to bring in all data columns from **wave1**. Two columns were renamed between waves — but since they represent the same questions, we want Protobi to recognize the wave1 values as belonging to the renamed wave2 columns.

---

## Create a data process

<a href="http://help.protobi.com/adminaccess/project-data#New%20process-1" title="">Create a new data process</a> and paste the full code from the example below into it. Make sure to set the process as the **Primary** data source for your project after running it.

Once you run the code, values from renamed wave1 columns will be correctly mapped to their corresponding wave2 columns.

---

## How the code works

### Config

The `config` object is where you define your column mappings. Each entry is a target-origin pair — the target (left) is the column where you want the data to appear, and the origin (right) is where it should be pulled from. Target names typically come from the most recent wave and origin names from the historical wave.

### Exact substitutions

For direct one-to-one column matches, list them explicitly inside `config`. For example, `S50r1` maps exactly to `S50_1`.

### Pattern substitutions

If you don't have a complete column-by-column mapping, you can use RegExp patterns to match groups of columns automatically. For instance, if `S40_1`, `S40_2`, and `S40_3` from wave1 should map to `S20r1`, `S20r2`, and `S20r3` in wave2, you can define a pattern that handles the whole group in one rule.

### Converter function

The `Converter` function takes your `config` object and returns a `convertRow` function that applies the mappings to each row of data.

### Applying the converter

At the bottom of your code, call the converter to produce an updated version of your historical data:

```js
let wave1converted = wave1.map(convertRow)
```

This creates a new version of wave1 with all column names reconciled to match wave2, which can then be stacked cleanly.

---

## Full code example

```js
const tables = await Protobi.get_tables(["wave2", "wave1"])
let wave2 = tables['wave2']
let wave1 = tables['wave1']

let config = {
    // Exact substitutions — new name on left, old name on right
    "S20": "S40",
    "S50r1": "S50_1",
    "S60r1": "S60_1",
    "S60r2": "S60_2",
    "S60r3": "S60_3",
    "S60r4": "S60_4",

// Pattern substitutions — XX* matches "XX", "XX_1", "XX_2", etc.
    "S10*": "",
    "S40*": "S30",
    "S20*": "S40",
    "S30*": "",
    "S50*": "S50",
    "S60*": "S60",
    "S70*": "S40",
    "S80*": "S80",
}

let Converter = function(config) {
    return function convertRow(row) {
        let rowKeys = Object.keys(row)
        let result = {}
        for (let left in config) {
            let right = config[left]
            if (right) {
                if (left.endsWith('*')) {
                    let leftPart = left.slice(0, -1)
                    let reRight = new RegExp("^(" + right + ")(_(\\d+))?")
                    let matchKeys = rowKeys.filter(key => reRight.test(key))
                    if (matchKeys.length) {
                        for (let rightKey of matchKeys) {
                            let match = rightKey.match(reRight)
                            if (match) {
                                let r = match[3]
                                let leftKey = r ? [leftPart, "r", r, "c", 1].join('') : leftPart
                                result[leftKey] = row[rightKey]
                                details[leftKey] = rightKey
                            }
                        }
                    }
                } else {
                    result[left] = row[right]
                }
            }
        }
        return result
    }
}

let convertRow = new Converter(config)
let wave1converted = wave1.map(convertRow)
console.log("Pattern-based replacements", details)

let stacked = Protobi.stack_rows([wave2, wave1converted])

return stacked;
```

If you have questions about applying this to your specific project, reach out to us at support@protobi.com.

Publishing

Date

Status

Slug edit

Content

Thumbnail

Categories Manage

Tools

AI Database

Publishing

Content

Tools

Danger zone