Evenly Serve Items in Qualtrics Across Your Survey Participants

Survey Setup

Have you ever wanted to have each of your survey participants rank 5 product reviews from a set of thousands, or gather comments on a handful of images from a set of hundreds? Maybe you want to show users a response they provided in a previous survey. Well, now you can with the Dynamic Item Server (DYIS) developed by the Research Programming Team. Just email us to get started.

How It Works

The application is self-serve and all you need to get started is a CSV with header rows that include columns app_id, served and order.

  • app_id A name for the set of items. We call this an App.
  • served This is the initial served count and will almost always be set to 0.
  • order This will be a number and will be used as the ordering when serving items. See the Orderings Options section for more details.

All other columns will be delivered as the key/value payload for the item. You can now add a Qualtrics Web Service call in your survey flow to request a set of items. As items are served, their served count will be incremented and new items will be ordered first by their served count and then by their order value. The result is that all items will be served once evenly before cycling though the items again.

Orderings Options

When creating a new App you can choose from the following ordering methods.

  • None will serve items in a cycle using your provided ordering value, in ascending order, repeatedly.
  • On Next Cycle will serve items using your provided ordering value in ascending order for the first cycle, but be randomized for each subsequent cycle.
  • By Key will serve items using your provided key value, cycling as if you selected On Next Cycle ordering. You can learn more about using keys here.


With the following CSV:

app_id order served color
testing 1 0 red
testing 2 0 green
testing 3 0 blue
testing 4 0 cyan
testing 5 0 magenta
testing 6 0 yellow
testing 7 0 orange
testing 8 0 white
testing 9 0 black
testing 10 0 grey

After uploading this CSV with the ordering method set to “On Next Cycle”, follow the instruction on the DYIS’s app to add the web service call to your Qualtrics survey. Assuming you want to show 2 colors to each participant, your first 10 survey participants will receive the following colors:

  1. red, green
  2. blue, cyan
  3. magenta, yellow
  4. orange, white
  5. black, gray
  6. cyan, green
  7. green, black
  8. red, yellow
  9. orange, white
  10. blue, magenta

Since the ordering method is set to “On Next Cycle”, the first cycle order (participants 1-5) is determined by the user supplied ordering value but each subsequent cycle will be randomized. Note that participants 6-10 received the full set of colors, but with new ordering and pairing. This is helpful if you want to make sure that items are not served with the same set of other items across all cycles.

What About Keys

Assume you wanted to show users their favorite color they selected in a previous study. You can do that using the “By Key” ordering method.

Assuming you have following CSV

app_id order served color key
testing_with_keys 1 0 red user1
testing_with_keys 2 0 blue user2
testing_with_keys 3 0 yellow user3
testing_with_keys 4 0 white user4

and the ordering is set to “By Key”. And assuming you have the user’s ID somewhere in the survey, you can now use Qualtrics’s Piped Text feature to provide the user ID to the web service call for a single item. If the key matches an item in the set, it will return that item’s payload. So if the request calls for 1 item with “user2” as the key, “blue” would be returned and its served count would be incremented.

How To Get Started

We hope that you will find new uses for this service and we are happy to work with you on implementation, testing and delivery. Just shoot us an email at research-programming@wharton.upenn.edu to get started.