ナード戦隊データマン

データサイエンスを用いて悪と戦うぞ

TensorFlowでRNNを実装しTensorBoardでサマリーやグラフを見る

TensorBoardを使えば、tensorflowに関していろいろな可視化ができます。ここでは、MNISTをRNNで解き、その過程や結果をTensorBoardで見てみます。

データの準備とプレースホルダの定義

In[1]:

import tensorflow as tf
from tensorflow.examples.tutorials.mnist import input_data
mnist = input_data.read_data_sets("/tmp/data/", one_hot=True)

element_size = 28
time_steps = 28
num_classes = 10
batch_size = 128
hidden_layer_size = 128

LOG_DIR = "logs/RNN_with_summaries"

_inputs = tf.placeholder(tf.float32, shape=[None, time_steps, element_size], name='inputs')
y = tf.placeholder(tf.float32, shape=[None, num_classes], name='labels')

サマリーを定義する関数を作成

In[2]:

def variable_summaries(var):
    with tf.name_scope('summaries'):
        mean = tf.reduce_mean(var)
        tf.summary.scalar('mean', mean)
        with tf.name_scope('stddev'):
            stddev = tf.sqrt(tf.reduce_mean(tf.square(var - mean)))
        tf.summary.scalar('stddev', stddev)
        tf.summary.scalar('max', tf.reduce_max(var))
        tf.summary.scalar('min', tf.reduce_min(var))
        tf.summary.histogram('histogram', var)

重み・バイアス・rnnを定義

In[3]:

with tf.name_scope('rnn_weights'):
    with tf.name_scope("W_x"):
        Wx = tf.Variable(tf.zeros([element_size, hidden_layer_size]))
        variable_summaries(Wx)
    with tf.name_scope("W_h"):
        Wh = tf.Variable(tf.zeros([hidden_layer_size, hidden_layer_size]))
        variable_summaries(Wh)
    with tf.name_scope("Bias"):
        b_rnn = tf.Variable(tf.zeros([hidden_layer_size]))
        variable_summaries(b_rnn)

def rnn_step(previous_hidden_state, x):
    current_hidden_state = tf.tanh(tf.matmul(previous_hidden_state, Wh)+tf.matmul(x, Wx) + b_rnn)
    return current_hidden_state

processed_input = tf.transpose(_inputs, perm=[1,0,2])
initial_hidden = tf.zeros([batch_size, hidden_layer_size])
all_hidden_states = tf.scan(rnn_step, processed_input, initializer=initial_hidden, name='states')

線形レイヤの定義

In[4]:

with tf.name_scope('linear_layer_weights') as scope:
    with tf.name_scope("W_linear"):
        Wl = tf.Variable(tf.truncated_normal([hidden_layer_size, num_classes], mean=0, stddev=.01))
        variable_summaries(Wl)
    with tf.name_scope("Bias_linear"):
        bl = tf.Variable(tf.truncated_normal([num_classes], mean=0, stddev=.01))
        variable_summaries(bl)

def get_linear_layer(hidden_state):
    return tf.matmul(hidden_state, Wl) + bl

with tf.name_scope('linear_layer_weights') as scope:
    all_outputs = tf.map_fn(get_linear_layer, all_hidden_states)
    output = all_outputs[-1]
    tf.summary.histogram('outputs', output)

オプティマイザ・コスト関数等を定義しサマリをマージ

In[5]:

with tf.name_scope('cross_entropy'):
    cross_entropy = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits=output, labels=y))
    tf.summary.scalar('cross_entropy', cross_entropy)
    
with tf.name_scope('train'):
    train_step = tf.train.RMSPropOptimizer(0.001, 0.9).minimize(cross_entropy)
    
with tf.name_scope('accuracy'):
    correct_prediction = tf.equal(tf.argmax(y, 1), tf.argmax(output, 1))
    accuracy = (tf.reduce_mean(tf.cast(correct_prediction, tf.float32)))*100
    tf.summary.scalar('accuracy', accuracy)

merged = tf.summary.merge_all()

訓練・精度の確認・ログの出力

In[6]:

test_data = mnist.test.images[:batch_size].reshape((-1, time_steps, element_size))
test_label = mnist.test.labels[:batch_size]

with tf.Session() as sess:
    train_writer = tf.summary.FileWriter(LOG_DIR + '/train', graph=tf.get_default_graph())
    test_writer = tf.summary.FileWriter(LOG_DIR + '/test', graph=tf.get_default_graph())
    sess.run(tf.global_variables_initializer())
    
    for i in range(10000):
        batch_x, batch_y = mnist.train.next_batch(batch_size)
        batch_x = batch_x.reshape((batch_size, time_steps, element_size))
        summary,_ = sess.run([merged, train_step], feed_dict={_inputs:batch_x, y:batch_y})
        train_writer.add_summary(summary, i)
        
        if i % 1000 == 0:
            acc, loss = sess.run([accuracy, cross_entropy], feed_dict={_inputs:batch_x, y:batch_y})
            print("Iter " + str(i) + ", Minibatch Loss= "+
                 "{:.6f}".format(loss) + ", Training Accuracy= " + 
                 "{:.5f}".format(acc))
        if i % 10:
            summary, acc = sess.run([merged, accuracy], feed_dict={_inputs:test_data, y:test_label})
            test_writer.add_summary(summary, i)
            
    test_acc = sess.run(accuracy, feed_dict={_inputs:test_data, y:test_label})
    print("Test Accuracy:", test_acc)

Out[6]:

Iter 0, Minibatch Loss= 2.302097, Training Accuracy= 7.81250
Iter 1000, Minibatch Loss= 1.490295, Training Accuracy= 42.18750
Iter 2000, Minibatch Loss= 0.842843, Training Accuracy= 75.00000
Iter 3000, Minibatch Loss= 0.306145, Training Accuracy= 89.06250
Iter 4000, Minibatch Loss= 0.248037, Training Accuracy= 94.53125
Iter 5000, Minibatch Loss= 0.114223, Training Accuracy= 96.87500
Iter 6000, Minibatch Loss= 0.077423, Training Accuracy= 99.21875
Iter 7000, Minibatch Loss= 0.053525, Training Accuracy= 99.21875
Iter 8000, Minibatch Loss= 0.079996, Training Accuracy= 97.65625
Iter 9000, Minibatch Loss= 0.031891, Training Accuracy= 99.21875
Test Accuracy: 98.4375

ログをtensorboardから見る

$ tensorboard --logdir=logs

f:id:mathgeekjp:20171012165813p:plain

f:id:mathgeekjp:20171012165826p:plain

f:id:mathgeekjp:20171012165909p:plain

f:id:mathgeekjp:20171012165951p:plain

参考

GitHub - Hezi-Resheff/Oreilly-Learning-TensorFlow