lecture 12 visualization

Q1 why does visualizing the weights of the filters tell you what the filter is looking for?

template match. (filter = template)

the input which maximizes that activation under a norm constraint on the input is exactly when those two vectors (input + filter) matches up.

visualization of second last layers:

L2 nearest neighbors in feature space can be same types ( tree on the left or tree on the right)

visualization of middle layers

select images with maximal activations in exact channels of exact layers (the selected images are usually same)

mask part of the image before feeding into CNN

saliency maps

• which pixels matter?
• computer gradient of output/input directly

deep dream

set feature maps as the (backpropagation) gradient to maximize the features found by DNN

neural texture synthesis

generate a big texture from a small texture(image)

• how to compute Gram Matrix(also for style transfer)?:

1. get a feature map through DNN with size ($C\times H\times W$)
2. generate $cchw$ matrix for each pair of c-dimensional vectors measuring co-occurrence ($new_{ij} = old1_iold2_j$)
3. average from cchw to cc

• using random image to generate gram matrix and compare it with input’s gram matrix( loss function to compare)

style transfer

• gram matrix is also used
• but the iteration may be too large (random image as initial image), solution: train a DNN(feedforward net in the image) ,input: raw image output: style image and the weight is updated by throwing the output into the gram matrix calculation procedure.

lecture 13 generative models

面试了两次了，去年一次，今年一次。记录一下流程和心得吧。

流程

• 跟教授套磁（个人觉得是影响最后结果最重要的因素，最好是能跟教授做research， 比如申请ra，申请fyp在cu做）
• cs 官网报名。注意early submission一般是8，9月截止。正常的是11月开始 越早报名越好，regular interview不是等到最后三月DDL后才一起面试。
• interview，了解到的eligible的同学基本都是top15水准的内地同学吧，中大，浙大，华科，清华等等。不过也有南科大top1的同学和华盛顿回港的待就业local。
  • 首先是一小时的written test。考的东西是 组合 + 概率 + 图论 + OS + data structure 等各种杂交的把。不过难度不是很大，准备其实也无从下手。内地同学可能要把单词搞明白？重要性据说也就那样，好了自然是加分项，没好也不会太扣分。
  • 面试是三位老师。常规的 自我介绍，reasons for apply for this program是一定要准备的。另外的话 就是准备好介绍自己的proj。一般来说 前期套磁很顺利的话 教授就不会太刁难的问算法 技术问题，只是发邮件“套磁”的话可能会问到比如leetcode这样的东西。
• 面试完之后，如果通过了，前期已经完全说好的就可以直接跟教授拿offer了，否则还需要找教授问要不要收我。（这个面试代表engineering falculty 认可你做phd学生，不代表某个特定的教授就要招你）

感想

• 面试的时候有说 charles kao 是自己的偶像，想做newton in physicals那样的人物。教授有点“冷笑”的意思。 再想想自己现在其实并没有什么能摆上台面的achievement，说这种话真有点“志大才疏”的意思，心比天高的人其实是我最不想成为的人。在我能说出这些话之前，先有足够的实力才不会被人质疑。不过所幸还有足够时间努力。
• eric lo 从工业界回到学术圈的原因也知道了，不喜欢编程。觉得钱不是最重要的事。很高兴其实还是有很多人这么想的。就像记者问charles kao没有光纤专利后不后悔，他说不后悔，” if everything is for making money, then I cannot absolutely invent the optical fiber”。至少我也一定不会做一个money-oriented 的人，对我来说 人生的追求不是making big money.

1
2
3
4
5
6
7
8
9
10
11

// loop over inner tile(output tile) dimension
    for ( int iktile = 0; iktile < ntiles; iktile++ ) {
  
      // loop over row tiles
      for ( int irowtile = 0; irowtile < ntiles; irowtile++ ) {
        // loop over column tiles
        for ( int icoltile = 0; icoltile < ntiles; icoltile++ ) {
…
		}
      }
    }

Details description

社交网络。

看完后浓浓的窒息感。

第一想法就是觉得人生中有许多特别重要的抉择 当时我们可能不以为然 但往后就会陷入后悔。不过仔细想想他们的结局。其实没有谁特别糟糕。即便特别糟糕了 事情既然发生 也不能怎样了 接受天命很难 却是大部分人必须学会的。

另外在争吵后，如何去调整关系，不管是伴侣，还是朋友，还是合作伙伴。也许他说的是有道理，也许我真的不适合，公司的未来发展，总之，不要盲目把自己就想成真理。

SOSP2017

Problem

automated and systematic testing of large-scale, real-world DL systems with thousands of neurons and millions of parameters for all corner cases is extremely challenging.

如何准确评估一个DL系统的好坏？

Background

• adversarial deep learning have demonstrated that carefully crafted synthetic images by adding minimal perturbations扰动 to an existing image can fool state-of-the-art DL systems.（像素攻击）
  • limit: it must limit its perturbations to tiny invisible changes or require manual checks.
• Challenges:
  • how to generate inputs that trigger different parts of a DL system’s logic and uncover different types of erroneous behaviors,
  • how to identify erroneous behaviors of a DL system without manual labeling/checking.
• This work
  • neuron coverage (the number of neurons activated)
  • multiple DL systems finish same work. (traditional software engineering technical)
  • how to use gradient to solve the problem(test more neuron coverage and as many differential behaviors as possible)

Details

input: unlabeled test inputs

output: new tests that cover a large number of neurons

SuperNeurons: Dynamic GPU Memory Management for Training Deep Neural Networks

Problem

Memory optimization of single DNN training work in single GPU

Method

• Liveness Analysis 只保存未来会用到的tensors
• Unified Tensor Pool communication tools for GPU-CPU memory using Least Recent Used tensor replacement policy and a Tensor cache. （只有conv 这种co mputation-intensive的才放到cpu去）
• Cost-Aware Re-computation 只要不高于peak memory 都不recompute，否则除了conv 都recompute（pool,activation…）
• Selective convolution algorithm 基于此刻与peak memory的差值选择convolution 方法（eg。 FFT需要额外的convolution space）

攻击神经网络

Fast Gradient Method (Explaining & Harnessing Adversarial Examples)

linear model:

$$
y’=w^Tx’ = w^Tx+w^T\sigma
$$

where $\sigma$ is the change of original input x, and we have:
$$
\sigma = \varepsilon sign(w)
$$
理解： 在保持$|\sigma|$ 不变情况下，y’ 变化最大只可能 使 $w^T\sigma = |w|_1$

DNN
$$
\sigma = \varepsilon sign (\triangledown_xJ(\theta,x,y_{true}))
$$

where J is the loss function. In most cases (soft-max as classifier layer)

理解： 要让y’变化最大，及 应向loss 变大的方向变化。 即若$(\triangledown_xJ(\theta,x,y))$ 为负数（loss 与 x 的gradient）则 x应变小 （$\sigma$需为负数）

Iterative method

Basic iterative method

update equation (1) $x_2’ = x_1’ + \sigma$

Targeted

(3) becomes:
$$
\sigma = -\varepsilon sign (\triangledown_xJ(\theta,x,y_{targeted}))
$$

Our Method

What we have known and what we can obtained now?

• Printable image size:

  $P(i,j)$ as the indexed boxes with size: Black/white box: $P_h\times P_w$

• Actual image ($100\times 100$) per frame.

  $S_I$: starting index of the billboard for each frame.

  $R_I$: range of the billboard for each frame.

  $G_I$: gradient of each pixel in the bill board/

What we need to calculate

$G_P$ :gradient of each box in P.

How?

Transform:

• Nearest

• SPP:

• Binary

EIE: Efficient Inference Engine on Compressed Deep Neural Network

Introduction

• DRAM access costs more energy than basic operations
• CNN has a lot of data reuse, which makes DRAM access not very important
• Deep compression makes the kernel sparse, able to exist on mobile

Contribution

• accelerator for sparse and weight sharing neural networks
• accelerator that exploits the dynamic sparsity of activations to save computation
• a method of both distributed storage and distributed computation to parallelize a sparsified layer across multiple PEs (processing elements)

Motivation

• FC is used widely, and memory access is usually the bottle-neck ( no reuse of the input matrix)
• network compression offers a solution of memory bottleneck
• previous SPMV accelerator can only exploit static weight sparsity.
• batching of FC is not suitable for latency reason

Model

Variation of CSC

v : value

z : offset of last non-zero value (one column as a vector) or first element

q: pointer to the fist non-zero element of each column

combination between algorithm and hardware

Python function object reference导致的bug

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

imsave('./generated_inputs/' +str(iters) +'_0'+str(i)+ '_two.png', deprocess_image(imgs[0]))

def deprocess_image(x):
    tmp = x
    tmp = tmp.reshape((100, 100, 3))
    # Remove zero-center by mean pixel
    tmp[:, :, 0] += 103.939
    tmp[:, :, 1] += 116.779
    tmp[:, :, 2] += 123.68
    # 'BGR'->'RGB'
    tmp = tmp[:, :, ::-1]
    tmp = np.clip(tmp, 0, 255).astype('uint8')
    return tmp
// 如此添加tmp。x的值仍会一直改变。则重复调用deprocess无法得到相同结果。
// 正确做法 使用x.copy()
def deprocess_image(x):
    tmp = x.copy()
    tmp = tmp.reshape((100, 100, 3))
    # Remove zero-center by mean pixel
    tmp[:, :, 0] += 103.939
    tmp[:, :, 1] += 116.779
    tmp[:, :, 2] += 123.68
    # 'BGR'->'RGB'
    tmp = tmp[:, :, ::-1]
    tmp = np.clip(tmp, 0, 255).astype('uint8')
    return tmp