Question numbers and response values sometimes change between waves of a survey.  This tutorial shows  ways you can reconcile question numbers   to compare results across waves.

Ideally, question numbers and response options would mean the same thing in each waves.If  If new questions and response options are added, they get new identifiers.  If questions or response options are dropped, those would be retired and not repurposed in later waves.

But in practice, things aren't always this simple.  So you can use Protobi data processes to reconcile question numbers and response options so they are consistent for each wave.

An effective way to start is to create a worksheet with a list of questions and responses as rows, and showing their identifiers in each wave.

We refer to this process as "mapping" questions. This tutorial demonstrates how such a mapping can be read and applied programmatically in Protobi. 

A simpler  example of stacking waves of data is in this tutorial Combine data.

Example

In this example there are two data tables, wave2 and wave1. We created a project with the wave2 data, and now we want to bring in all data columns from the first wave.

However, two data columns have been renamed in wave2. Since the questions are the same from the previous wave, we would like Protobi to recognize values from the wave 1 file as values for the two renamed data columns in the wave2 file. 

Create a data process to reconcile question numbers

See create a data process and copy the code below into the process. Note: the process needs to be set as the primary datafile for the project.

The code below will reconcile the question numbers. After you run the code, you will see that values from prior wave data columns that have been renamed will be connected to the current wave's data columns.

Config

The "config" object in the code below allows you to specify target (left) and origin (right) pairs. The target is the column where you would like the data to appear and the origin column is where the data should be pulled from.

Typically, the target column names come from the most recent wave (i.e. wave2), and the origin column names come from the historical wave (i.e. wave1).

Exact substitution

If you have exact target column to origin column matches you can include them as exact substitutions within the "config" object. For example S50r1 matches exactly to S50_1.

Pattern substitutions

If you do not have a complete and detailed list that reconciles columns between current and historical waves, you can run the code below which uses RegExp to identify exact column matches for the general patterns you specify.

For instance, if S40 has three sub-questions S40_1, S40_2 and S40_3 that should be mapped respectively to S20r1, S20r2 and S20r3 you can specify the RegExp pattern that's expected from the left and right hands sides to find the right matches.

Converter function

The function Converter takes a config object that specifies and returns a function convertRow.

Call the converter function

At the bottom of your code call the converter function to create an updated version of the historicial data (i.e. wave1converted). The updated version is the original file (i.e. wave1) with the convertRow appled. See code below.

let wave1converted = wave1.map(convertRow) 

Feel free to reach out to support@protobi.com if you have any questions on how to use this option in your specific project.

Full code example

const tables = await Protobi.get_tables( ["wave2", "wave1"])  // download data tables
let wave2 = tables['wave2']
let wave1 = tables['wave1']

let config = {
    // exact substitutions, new name on left, old name on right
    "S20": "S40",
    "S50r1": "S50_1", 
    "S60r1": "S60_1", 
    "S60r2": "S60_2",
    "S60r3": "S60_3", 
    "S60r4": "S60_4", 

   // pattern substitutions XX* matches "XX" or "XX_1", "XX_2", etc.
    "S10*": "",
    "S40*": "S30",
    "S20*": "S40",
    "S30*": "",
    "S50*": "S50",
    "S60*": "S60",
    "S70*": "S40",
    "S80*": "S80",
}

let Converter = function(config) {
    return function convertRow(row) {
        let rowKeys = Object.keys(row)
        let result = {}
        for (let left in config) {
            let right = config[left]
            
            if (right) {
                if (left.endsWith('*')) {
                    // it's a pattern based substitution
                    let leftPart = left.slice(0,-1)
                   let reRight = new RegExp("^("+right+")(_(\\d+))?")
                    let matchKeys = rowKeys.filter(key => reRight.test(key))

                    if (matchKeys.length) {
                        for (let rightKey of matchKeys) {
                             let reRight = new RegExp("^("+right+")(_(\\d+))?")
                            let match = rightKey.match(reRight)
                            
                            if (match) {
                                let base = match[1]
                                let r = match[3]
                                let c = 1
                                let leftKey = r ?  [ leftPart , "r", r, "c", 1].join('') : leftPart
                                result[leftKey] = row[rightKey]
                                details[leftKey ] = rightKey
                            }
                        }
                    }
                }
                else {
                    // its a direct substitution
                    result[left] = row[right]
                }
            }
        }
        return result
    }
}

let convertRow = new Converter(config)

let wave1converted = wave1.map(convertRow) 
console.log("Pattern-based replacements", details)

let stacked = Protobi.stack_rows([wave2,wave1converted])

return stacked;