Combine data from multiple projects

We can Combine data using a simple process when all the data we need is housed within one project space. However, to combine data tables that are in separate projects or even non-Protobi sources we use an asynchronous data process. This is applicable when merging external data, and operating on results of other processes or data in other projects. 

Add an asynchronous data process

In the Data tab under "Project settings..." create a data process and set it to "Asynchronous" rather than "Simple":

Click "Edit/Run" to edit the code view.

Combine data from multiple projects

This example code combines data from multiple Protobi projects. “var sources” is an array containing objects of key pairs. Next to “datasetId” is a unique identifier found in the project’s URL. Next to “key” is a name that will represent the project. Next to "tables" is the name of the data table you want to use from that project (if there's more than one separate).

var sources = [
    {datasetId: "5dc40f3972a9af00046aceb3", key:"project1", tables:["main","main2"]},
    {datasetId: "5db276c8445a1c000477aa07", key:"project2", tables:["main"]}
    ]
   
async.mapSeries(
    sources,
    function(entry,cb) {
        Protobi.get_tables(entry.datasetId, entry.table, cb) 
    },
    function(err, tables) {
        if (err) {
            console.log("error message")
            return $.notify(err)
        }
        else {
            // datasets is now an array of results paralleling `sources`

            var project1 = tables[0]
            var project1 = tables[1] 
            var project2 = tables[2]                      
            var rows = Protobi.stack_rows(tables)
    $.notify("Process complete", "success")
            console.log("rows",rows)
                return callback(null,rows)
        }
       
    })

Combine results of a data process

Using an asynchronous data process, you can also combine data tables with the result of data processes. In the example below, the code combines a data table from one project with the result of a data process from another. 

var sources = [
    {datasetId: "5dc40f3972a9af00046aceb3", key:"project1", table:"main"},
    {datasetId: "5db276c8445a1c000477aa07", key:"project2", table:"process"}
    ]
   
async.mapSeries(
    sources,
    function(entry, cb) {
        Protobi.get_table(entry.datasetId, entry.table, cb)
    },
    function(err, tables) {
        if (err) return callback(err)
        else {
            // datasets is now an array of results paralleling sources
            var project1 = tables[0]
            var project2 = tables[1]
            
            console.log(tables)
                      
            var rows = Protobi.stack_rows(tables)
        
            $.notify("Process complete", "success")
            return callback(null,rows)
        }
    }
)

Retrieve project setup configuration

Asynchronous data processes can access not just data tables but can also get the elements configuration. Elements determine the setup and organization of a project, including question titles and value format. You can find the configuration for a project in settings on the "Elements" tab. The code below calls the elements of a specified project.

The following example does three things:

• gets the elements configuration for this project,
• gets the data from data table main from another project
• returns that table as the data table for this process

Protobi.get_elements(function(err, tabular) {
    if (err) return callback(err)
    
    var q1 = tabular.getDimension('q1') 

    Protobi.get_tables("5cf6a9ae9468bab783ebc277","main", function(err, rows) {
        if (err) return callback(err)
        console.log(rows)
    rows.forEach(function(row) {
        row.q1 = q1.getValue(row)   // use the element <strong>q1 </strong>to calculate a value for row.q1 
    }) 

        
        return callback(null, rows)
    })
})

Here is an example of an asynchronous data process in context:

Reminder: For data processes, "Save" and "Run" the process after you are done editing the code view. To use the result of the process as the primary data for the project, you will need to set it as "Primary".

Advanced support

Protobi can handle complex data processing cases, not limited to those that are documented within our tutorials. Our support team is ready to help you with your specific goals. Please contact us at support@protobi.com to discuss further.