SOSP2017
Problem
automated and systematic testing of large-scale, real-world DL systems with thousands of neurons and millions of parameters for all corner cases is extremely challenging.
如何准确评估一个DL系统的好坏？
Background
- adversarial deep learning have demonstrated that carefully crafted synthetic images by adding minimal perturbations扰动 to an existing image can fool state-of-the-art DL systems.（像素攻击）
- limit: it must limit its perturbations to tiny invisible changes or require manual checks.
- Challenges:
- how to generate inputs that trigger different parts of a DL system’s logic and uncover different types of erroneous behaviors,
- how to identify erroneous behaviors of a DL system without manual labeling/checking.
- This work
- neuron coverage (the number of neurons activated)
- multiple DL systems finish same work. (traditional software engineering technical)
- how to use gradient to solve the problem(test more neuron coverage and as many differential behaviors as possible)
Details
input: unlabeled test inputs
output: new tests that cover a large number of neurons