In this experiment, your task is to evaluate an AI model and judge whether it is correct or not.
After your decision, you may place a bet based on how certain you are.
If your decision regarding the AI system is correct, you get the bet amount but if it is
incorrect, you lose it.
Your payment is $1.0 plus whatever you accumulate during this experiment (therefore the
guaranteed minimum is $1.0).
(-/-)
Balance: $0+$1.0
Thank you for participation in our study.
Please navigate to Prolific or with the code C18F0V8M.
Thank you for testing out our study.
Note that your data has not been logged.
Please contact the authors with any questions.
Welcome to our user study. You will be trying to answer questions with the assistance of an AI system. We will first walk you through an example question with
images and afterwards you will annotate 60 examples.
Each example consists of a question, the AI answer and the AI confidence.
Your task is to decide whether the AI system is correct or incorrect, by looking at the AI confidence.
After your decision, you place a bet based on how sure you are of your decision.
If you are very sure, place a larger bet and vice versa if not.
If your decision about the correctness of the AI system is correct, you receive money and
you lose it if your decision was wrong. Therefore, for each question, you can increase your balance by up to $0.1, or decrease it by upto $0.1 (your total balance will not fall below $0 though).
The guaranteed minimum is $1.0 regardless of your performance, but you can earn more money if your bets are correct (the additional money will be paid out as bonuses).
Our AI system encounters diverse types of questions.
You are not permitted to leave this tab for the duration of the experiment. This will be
monitored and violation of this can result in your submission being rejected.
The average time of the experiment is 5 minutes.
You will not be paid if you already participated in this study before.
Click Start study if you understood all the instructions and want to start with the first
question.
Feel free to re-read the instructions again (Previous).
