Crowd-sourced idea filtering with Bag of Lemons: the impact of the token budget size

Identifying the best ideas from the vast volumes generated by open innovation engagements is costly and often time-consuming. One approach is to engage crowds in filtering the ideas, not just generating them. Klein and Garcia, 2015 proposed a “BOL” approach that is better (in terms of accuracy and speed) at idea filtering than other filtering methods such as a conventional Likert approach. The idea behind this approach (BOL) is that it asks the crowd to distribute a fixed budget of tokens that eliminate bad ideas rather than select good ones. In this paper, we explain why BOL works better than other filtering methods using empirical experiments (with n = 850 subjects). Also, we present the effect of the token budget size on idea-filtering engagement and found, among others, that the accuracy of a filter depends on the token budget size.