By comparing sampling with and without replacement, we present a scenario where more information does not necessarily lead to a decrease in (subjective) uncertainty. The reason is that under certain information-generating mechanisms, it is costly to obtain information. For sampling with replacement, repeated sampling provides more information and thus reduces ambiguity. However, this is not the case for sampling with no replacement. Using simulation experiments, we show that sampling without replacement affects not only subjective beliefs, but also the environment the decision-maker faces. The former has a positive effect on the decision-maker's knowledge of the true distribution, which we call the information effect. The latter changes the state probability to be estimated and thus adds ``noise'' to the decision-maker's belief update, which we call the scale effect. When the sample drawn from the population is small, the information effect dominates. When the number of samples observed becomes large, especially when it is close to the size of the population, the scale effect dominates. That is, although the improved knowledge of the population is favorable, the larger the sample, the less the decision-makers know about the data-generating mechanism for the remaining population. As a result, more information does not necessarily reduce the ambiguity confronted by decision-makers. When the two effects cancel each other, it is optimal to stop learning.