2. Randomization Testing

As a part of your pre-launch testing plan, you should ensure that your project's randomization matches the selected randomization method, including arm and block weights.  

 Unsure what randomization method your project should be using? See Randomization for a complete description on:

• Basic Randomization
• Block Randomization
• Block Randomization with Stratification 

 Need help setting up your project's randomization? See How to Configure Randomization and How to Configure Strata 

 Now that you have set up your project's randomization, you are ready to test! 

• Depending on the complexity of your randomization scheme, you should plan to enroll at least 15 test participants in a row and record the outcomes.  If your study has multiple strata, arms, and weights, plan to enroll at least 30.  Why? This will allow you to detect any discrepancies in the arm allocations.  
  

Let's look at an example using basic randomization! 

1. Imagine you have a 3-arm study (control, incentive, social arms) using basic randomization.  The arm weights are as follows:
   
   1. Control- 0
   2. Social- 1
   3. Incentive- 2
      
      
      
2. What should you expect when you enroll and randomize 15 test participants?
   1. Control = 0 participants
   2. Social ~ 5 participants
   3. Incentive ~ 10 participants 
      
      
3. Once you determine how the participants should be allocated, you are ready to test! 
   1. Create the participant
   2. Complete all necessary enrollment steps
      1.  Tip! Have a lot of enrollment steps that are slowing down the randomization testing process?? Test on staging instead! Put in a JIRA ticket to have production copied to staging and then delete all enrollment steps (Be careful not to delete any steps that are required for arm assignment, strata assignment, etc.!) so that your enrollment looks like this, 
         
         
         
         
   3. View participant's arm assignment in their profile.
      
      
4. For each participant's assignment, you should track this in either a Google sheet or Excel doc.  This will help you to view the randomization pattern. 
   
   
   
   
5. Repeat for remaining participants.
   
   
6. We predicted that the control arm would have 0 participants, the social arm would have ~5 participants, and the incentive arm would have ~10.  Based on our testing, we can conclude that the randomization is working as expected! 
   
   
7.  Notice any inconsistencies in the arm allocation? Put in a JIRA Help ticket including:
   1. Randomization Method
   2. Arm weight
   3. Screen shot or link to testing 
      

Let's look an example using block randomization! 

1.  Imagine you have a 3-arm study (control, incentive, social arms) using block randomization.  
   1. The arm weights are as follows:
      
      1. Control- 1
      2. Social- 1
      3. Incentive- 1
   2. The block weights are as follows: 3
      
      
      
      
2. What should you expect when you enroll and randomize 15 test participants?
   1. Control ~ 5 participants
   2. Social ~ 5 participants
   3. Incentive ~ 5 participants 
   4.  Note! Since there is a block size of 3, you should expect the participants to be randomized in 1 of the 3 arms in blocks of 3.  What does this mean? At the start of the block, a test participant is randomized to control. The next participant in the block can only be randomized to incentive or social since control was already 'taken'.
   
   
3. Once you determine how the participants should be allocated, you are ready to test! 
   1. 1. Create the participant
      2. Complete all necessary enrollment steps
         1.  Tip! Have a lot of enrollment steps that are slowing down the randomization testing process?? Test on staging instead! Put in a JIRA ticket to have production copied to staging and then delete all enrollment steps (Be careful not to delete any steps that are required for arm assignment, strata assignment, etc.!) so that your enrollment looks like this, 
            
            
            
            
4. View participant's arm assignment in their profile.
   
   
5. For each participant's assignment, you should track this in either a Google sheet or Excel doc.  This will help you to view the randomization pattern. 
   1.  Tip! When testing block randomization, use shaded colors to indicate the different block sizes.  In this example, we want to alternate between 2 shades of orange every 3 participants.  Again, this will help you to detect patterns. 
      
      
      
      
6. Repeat for remaining participants.
   
   
7. We predicted that all arms would have exactly 5 participants and for each 3-block, there would be one participant in each arm (as indicated by the green and purple boxes above).  Based on our testing, we can conclude that the randomization is working as expected! 
   
   
8.  Notice any inconsistencies in the arm allocation? Put in a JIRA Help ticket including:
   1. Randomization Method
   2. Arm weight
   3. Screen shot or link to testing