The methods for unsupervised visual inspection use algorithms that are developed, trained and evaluated on publicly available datasets. However, these datasets do not reflect genuine industrial conditions, and thus current methods are not evaluated in real-world industrial production contexts. To answer this shortcoming, we introduce AutoVI, an industrial dataset of visual defects that can be encountered on automotive assembly lines. This dataset, comprising six inspection tasks, was designed as a benchmark to assess the performance of defect detection methods under realistic acquisition conditions. We analyze the performance of current stateof-the-art methods and discuss the difficulties specifically encountered in the industrial context. Our results show that current methods leave considerable room for improvement. We make AutoVI publicly available to develop unsupervised detection methods that will be better suited to real industrial tasks.