Randomization (roles, conditions)
Participants can be randomly assigned to treatments and roles. Randomization is flexible: researchers can either let SMARTRIQS assign treatments and roles after participants are matched in groups (Method 1, recommended) or assign treatments and roles before participants are matched in groups, by using the `Randomizer’ element in the Survey Flow. While matching is significantly faster when using Method 1, Method 2 is required if participants have to read treatment- or role-specific instructions or do treatment- or role-specific tasks before they are being matched with others. (In most experiments, however, participants can read treatment-specific instructions or complete treatment-specific tasks after they have been matched.) In addition, researchers can setup experiments with pre-defined roles (Method 3), for example, when using confederates, or when participants are assigned to roles based on some demographic characteristic (e.g., gender, political affiliation).
Method 1: letting SMARTRIQS randomize (RECOMMENDED)
To let SMARTRIQS randomize the participant’s role, simply set participantRole = random before the MATCH block.
The matching algorithm will then randomly assign roles to participants.
Similarly, if there are more than 1 condition in the experiment, set set participantCondition = random before the MATCH block. The matching algorithm will then randomly assign a condition to the group.
This method guarantees very short waiting times, since when participants join a group, their roles and condition are not determined yet. Therefore, whenever there are enough participants to form a group, a new group is created, and participants are assigned roles and conditions only after that. This guarantees that there will no more than n-1 participants waiting to be matched at any given time, where n = group size, regardless of the number of conditions. For example, even if there are 4 conditions in a 4-player game, there will be no more than 3 people waiting to be matched at any given time, since whenever 4 people are available, they immediately form a new group, assigned to the same condition, and then get their roles assigned within that group.
Below is a screenshot of a study with a single condition and 4 players per group, using default randomization in the MATCH block:
Note that we set participantRole = random
Method 2: randomize manually in Qualtrics
Use the Randomizer element in the Survey Flow to manually assign roles / conditions to participants. Make sure to check “Evenly present elements”.
When treatments and roles are determined in Qualtrics before matching, participants have to wait until there is at least one participant in each role, which can cause significantly longer waiting times, especially when the group size is large or when there are multiple treatments. For example, if there are 4 roles in just a single condition, it is possible that even 9 people will be waiting at the same time (3-3 people in 3 roles, 0 in the 4th role). Unless recruitment is very efficient and there are a lot of new participants, longer waiting times significantly increase attrition rates.
Below is a screenshot of a study with a single condition and 4 players per group, using manual randomization in Qualtrics:
Note that instead of setting participantRole = random, we used a Randomizer element in the Survey Flow, and assigned participants to one of the four roles randomly.
The screenshot taken from the progress monitor below demonstrates the issue with this method:
Even though there are already 6 participants, no groups have been created yet, since there is no one in role B (there are 3 D and 2 C players). Using the ‘Evenly present elements’ option in the Randomizer element might help, but since players can drop out during matching, that still does not guarantee that groups will be created once 4 players join.
Method 3: Using pre-defined roles
In some cases, researchers might want to assign participants to pre-defined roles, for example, if all participants are assigned to one role within a group, while the other roles are populated by confederates (research assistants), or when role assignment depends on some demographic characteristic (e.g., gender, political affiliation, etc.).
To implement such a design, the easiest solution is to utilize “URL query strings” that can be attached to any URL link.
First, open the Survey Flow editor and delete “random” from the participantRole field, so that it should read “Value will be set from Panel or URL”:
Then, when distributing the survey to participants, add “?participantRole=A” after the study URL. This indicates that everyone who opens the link will be assigned to role “A”. You can use any role name in these query strings attached after an URL, just make sure to have the role name defined in the roles field in the Survey Flow (see picture above).
To allow confederates (or the other group of participants) to join, use the same study URL, but add “?participantRole=B” instead. This indicates that eveyone who opens this version of the link will be assigned to role “B”.
For example, if the original study URL is
http://chicagobooth.az1.qualtrics.com/jfe/form/SV_abcdefg123456
(not a real link, just for demonstration), then use:
http://chicagobooth.az1.qualtrics.com/jfe/form/SV_abcdefg123456?participantRole=A
to distribute your survey to participants (or participant group 1) and
http://chicagobooth.az1.qualtrics.com/jfe/form/SV_abcdefg123456?participantRole=B
to distribute your survey to confederates (or participant group 2).