Merge data from multiple studies

Programmatically merge data from multiple studies into one Protobi project. 

Select a combined project space

Choose a survey to use as the final combined project space. Data from other studies can be merged into the project, and the existing question order, titles, formats, etc., will still stay the same. If you merge multiple waves of a survey you may want to use the most recent wave as the combined space. When combining different country data, use the survey with the question text and formats that are in your desired language. 

Create a data process

In the data tab of project settings, create a new process. 

Set the process to asynchronous. This type of data process can retrieve data from multiple projects. 

Example code

Expand the code panel to copy the data process code seen below. 

This code appends rows of data from multiple surveys vertically, resulting in a larger combined dataset. This is also referred to as "stacked" data. 

Code

//This data process merges data from multiple studies using the unique pulseIds
var entries = [
    {label: "main", pulseId: "60bmp8gs-q4rp-njyu-dii0x63ca40y8ktr8"},
    {label: "EU",   pulseId: "7l16338n-f19m-1l93-olpd2julmbdv339if"}
]
  
Protobi.get_table("main", function(err, main) {
    if (err)  return callback(err);
    console.log(main)
    async.mapSeries(
        entries,
        function(entry, cb) {
            var url = `https://rtanalytics.sermo.com/api/v3/realtime/${entry.pulseId}/csv`
            console.log(url)

            d3.csv(url, function(rows) {
                if (!rows) return cb(404, "Data not found")
                rows.forEach(function(row) {
                    _.merge(row, entry)  // Set all entry values as row values
                })
                return cb(null, rows)
            })
        },
        function(err, results) {
            if (err)  return callback(err);
            var stacked = _.flatten(results)
            //Reassign questions that differ between surveys (ends before the $.notify)   
            //The loop below operates on each row in the final "stacked" data
            stacked.forEach(function(row) {
                //Assign "temp" name to questions that don't share same key between surveys
                temp = row["Q1"]  
                temp1 = row["Q2"] 
                //Set a condition, reassign questions if data is from EU survey
                if (row.label == "EU") {    
                    //Map the correct "temp" key to the existing question in the project
                    row["Q2"] = temp        
                    row["Q3"] = temp1         
                }
            })  
            //Below returns the stacked data, do not delete    
            $.notify("Success", "success")
            return callback(null, stacked)
        })   
})

In the entries array, add the pulseIds of each survey you want to merge. The example combines two surveys, but you can include more. On line 13 below, the function retrieves data from each survey. The method used here pulls the most recent data. When any of the individual surveys receive more responses, the combined project will also reflect the most up-to-date data. 

Identical questions that have different keys

The surveys may not use the same key for every identical question. On lines 27 - 38 is code that reassigns question keys to align with the main project. 

Notice on lines 30 and 31 temp (temporary) values are assigned to the previous question keys. This is a way to store the value for later use. For instance, in line 35 Q2 is manipulated so the temp1 value is useful in line 36 to reference the original Q2 values. 

Reminder: For data processes, "Save" and "Run" the process after you are done editing the code view. To use the result of the process as the primary data for the project, you will need to set it as "Primary".

Data processes are specific to each project, and your code may not look identical to our example.

Support

If you are having trouble merging data, please contact us at support@sermo.freshdesk.com