Significance testing

673d23dd68cf08627e90e16be103471a

Updated at February 24th, 2020

There are two ways Protobi uses significance testing:

  1. Comparing current vs baseline scenario
  2. In Crosstabs

Current vs Baseline

When you press to query (e.g. "Excellent" in q2) Protobi shows results for just those respondents. The percentages and solid color bars reflect the current scenario, (e.g. only those in "Excellent" health). The thin black outlines reflect the baseline scenario (all respondents).

Protobi shows blue triangles wherever the current scenario is significantly different from baseline. Here, the outward pointing triangle indicates that 49.6% is significantly higher than the baseline of 30.5%. Inward pointing triangles indicate the value for the current scenario is significantly lower than the value for the baseline scenario.


Set current filters as baseline

If you want to make a strict comparison between non-overlapping groups, you can change the baseline scenario. Protobi allows you to set current filters as the base instead of all respondents (initial base).

Press the toolbar button "Set base" to make the baseline scenario equal to the current scenario (e.g. respondents in "Excellent" health). 

Now shift+press on "Excellent" to select those respondents who are NOT in "Excellent" health.

We can see above, we're now running a strong comparison between two distinct groups. The groups being those who are NOT in "Excellent" health (current scenario, solid bars) versus those who ARE in "Excellent" health (the baseline scenario, thin black outline).

Crosstabs

Crosstabs in Protobi use either Pairwise or Complement significance testing. Pairwise testing compares each column with the average of all other columns. Complement testing compares each column individually with each other column.

Admins can change crosstab significance tests in Project properties. The default mode is Complement testing.

Complement

In Complement testing, each column is compared with the average of all other columns (excluding itself). Blue cells indicate the value is significantly higher in the specified column compared to all other columns, and grey indicates significantly lower. 

Pairwise

Pairwise testing compares each column with each other individual column. This mode shows detailed superscripts like a traditional crosstab. 

Below we see that the percentage of respondents who rated their health as "Excellent" is much higher in column A (Very happy) than columns B, C or D. 

P-value 

To change the P-value for significance tests open the Project properties dialog. The default value is 0.05.

Note: Protobi limits significance testing to N>=20 to avoid testing when the sample size is too small. 


Was this article helpful?