Making inferences about claims we do not have direct experience with is a common feature of everyday life. In these situations, it makes sense to consult others: an apparent consensus may be a useful cue to the truth of a claim. This strategy is not without its challenges. The utility of a consensus should depend in part on the sources of evidence that underlie it. If each person based their conclusion on independent data then the fact that they agree is informative. If, instead, everyone relied on the same primary source, the consensus is less meaningful. However, the extent to which people are actually sensitive to this kind of source independence is still unclear. Here, we present the results of three experiments that examine this issue in a social media setting, by varying the sources of primary data cited via retweets. In each experiment, participants rated their agreement with 12 different claims before and after reading four tweets that were retweeted on the basis of either the same or different primary data. We found that people were sensitive to source independence only when it was clear that the tweeters had relied on the primary data to reach their conclusion. Implications for existing research are discussed.