如何在 tensorflow 0.11 中使用 RNN 编写多维回归预测器

Question

这是我实际尝试做的玩具版本。我有大量时间步长（150,000 步）的高维输入数据（2e05 到 5e06 维）。我知道最后我可能需要对状态进行一些嵌入/压缩（参见 this question）。但是让我们暂时把它放在一边。

以这个11维的玩具输入数据为例

t  Pattern
0  0,0,0,0,0,0,0,0,0,2,1 
1  0,0,0,0,0,0,0,0,2,1,0 
2  0,0,0,0,0,0,0,2,1,0,0 
n  ...

我希望 RNN 学习将当前时间步与下一个时间步相关联，这样如果输入 (x) 为 t0，则所需输出 (y) 为 t1。

使用 RNN 的想法是，我一次只能为网络提供一个时间步长（由于我的真实数据的维数很大）。由于输入和输出的数量相同，我不确定基本的 RNN 是否合适。我稍微看了看 seq2seq 教程，但我不确定这个应用程序是否需要 encoder/decoder，而且我无法使用我的玩具数据。

以下是我所能想到的全部内容，但它根本没有收敛。我错过了什么？

import numpy as np
import tensorflow as tf

# Imports for loading CSV file
from tensorflow.python.platform import gfile 
import csv

# Input sequence
wholeSequence = [[0,0,0,0,0,0,0,0,0,2,1],
                 [0,0,0,0,0,0,0,0,2,1,0],
                 [0,0,0,0,0,0,0,2,1,0,0],
                 [0,0,0,0,0,0,2,1,0,0,0],
                 [0,0,0,0,0,2,1,0,0,0,0],
                 [0,0,0,0,2,1,0,0,0,0,0],
                 [0,0,0,2,1,0,0,0,0,0,0],
                 [0,0,2,1,0,0,0,0,0,0,0],
                 [0,2,1,0,0,0,0,0,0,0,0],
                 [2,1,0,0,0,0,0,0,0,0,0]]

data = np.array(wholeSequence[:-1], dtype=int) # all but last
target = np.array(wholeSequence[1:], dtype=int) # all but first
trainingSet = tf.contrib.learn.datasets.base.Dataset(data=data, target=target)
trainingSetDims = trainingSet.data.shape[1]

EPOCHS = 10000
PRINT_STEP = 1000

x_ = tf.placeholder(tf.float32, [None, trainingSetDims])
y_ = tf.placeholder(tf.float32, [None, trainingSetDims])

cell = tf.nn.rnn_cell.BasicRNNCell(num_units=trainingSetDims)

outputs, states = tf.nn.rnn(cell, [x_], dtype=tf.float32)
outputs = outputs[-1]

W = tf.Variable(tf.random_normal([trainingSetDims, 1]))     
b = tf.Variable(tf.random_normal([trainingSetDims]))

y = tf.matmul(outputs, W) + b

cost = tf.reduce_mean(tf.square(y - y_))
train_op = tf.train.RMSPropOptimizer(0.005, 0.2).minimize(cost)

with tf.Session() as sess:
    tf.initialize_all_variables().run()
    for i in range(EPOCHS):
        sess.run(train_op, feed_dict={x_:trainingSet.data, y_:trainingSet.target})
        if i % PRINT_STEP == 0:
            c = sess.run(cost, feed_dict={x_:trainingSet.data, y_:trainingSet.target})
            print('training cost:', c)

    response = sess.run(y, feed_dict={x_:trainingSet.data})
    print(response)

方法的来源this thread。

最后我想使用 LSTM，重点是对序列建模，以便可以通过用 t0 启动网络然后馈送来重建整个序列的 近似值预测返回作为下一个输入.

EDIT1

我现在看到成本降低了，因为我添加了以下代码以在训练前将直方图输入数据重新缩放为概率分布：

# Convert hist to probability distribution
wholeSequence = np.array(wholeSequence, dtype=float) # Convert to NP array.
pdfSequence = wholeSequence*(1./np.sum(wholeSequence)) # Normalize to PD.

data = pdfSequence[:-1] # all but last
target = pdfSequence[1:] # all but first

输出仍然不像输入，所以我肯定遗漏了一些东西：

('training cost:', 0.49993864)
('training cost:', 0.0012213766)
('training cost:', 0.0010471855)
('training cost:', 0.00094231067)
('training cost:', 0.0008385859)
('training cost:', 0.00077578216)
('training cost:', 0.00071381911)
('training cost:', 0.00063783216)
('training cost:', 0.00061271922)
('training cost:', 0.00059178629)
[[ 0.02012676  0.02383044  0.02383044  0.02383044  0.02383044  0.02383044
   0.02383044  0.02383044  0.02383044  0.01642305  0.01271933]
 [ 0.02024871  0.02395239  0.02395239  0.02395239  0.02395239  0.02395239
   0.02395239  0.02395239  0.02395239  0.016545    0.01284128]
 [ 0.02013803  0.02384171  0.02384171  0.02384171  0.02384171  0.02384171
   0.02384171  0.02384171  0.02384171  0.01643431  0.0127306 ]
 [ 0.020188    0.02389169  0.02389169  0.02389169  0.02389169  0.02389169
   0.02389169  0.02389169  0.02389169  0.01648429  0.01278058]
 [ 0.02020025  0.02390394  0.02390394  0.02390394  0.02390394  0.02390394
   0.02390394  0.02390394  0.02390394  0.01649654  0.01279283]
 [ 0.02005926  0.02376294  0.02376294  0.02376294  0.02376294  0.02376294
   0.02376294  0.02376294  0.02376294  0.01635554  0.01265183]
 [ 0.02034193  0.02404562  0.02404562  0.02404562  0.02404562  0.02404562
   0.02404562  0.02404562  0.02404562  0.01663822  0.01293451]
 [ 0.02057907  0.02428275  0.02428275  0.02428275  0.02428275  0.02428275
   0.02428275  0.02428275  0.02428275  0.01687536  0.01317164]
 [ 0.02042386  0.02412754  0.02412754  0.02412754  0.02412754  0.02412754
   0.02412754  0.02412754  0.02412754  0.01672015  0.01301643]]

Answer 1

我放弃了直接使用tensowflow，最终使用了Keras。以下是使用带有第二个密集层的单层 LSTM 学习上述玩具序列的代码：

import numpy as np

from keras.models import Sequential
from keras.layers import Dense
from keras.layers import LSTM

# Input sequence
wholeSequence = [[0,0,0,0,0,0,0,0,0,2,1],
                 [0,0,0,0,0,0,0,0,2,1,0],
                 [0,0,0,0,0,0,0,2,1,0,0],
                 [0,0,0,0,0,0,2,1,0,0,0],
                 [0,0,0,0,0,2,1,0,0,0,0],
                 [0,0,0,0,2,1,0,0,0,0,0],
                 [0,0,0,2,1,0,0,0,0,0,0],
                 [0,0,2,1,0,0,0,0,0,0,0],
                 [0,2,1,0,0,0,0,0,0,0,0],
                 [2,1,0,0,0,0,0,0,0,0,0]]

# Preprocess Data: (This does not work)
wholeSequence = np.array(wholeSequence, dtype=float) # Convert to NP array.
data = wholeSequence[:-1] # all but last
target = wholeSequence[1:] # all but first

# Reshape training data for Keras LSTM model
# The training data needs to be (batchIndex, timeStepIndex, dimentionIndex)
# Single batch, 9 time steps, 11 dimentions
data = data.reshape((1, 9, 11))
target = target.reshape((1, 9, 11))

# Build Model
model = Sequential()  
model.add(LSTM(11, input_shape=(9, 11), unroll=True, return_sequences=True))
model.add(Dense(11))
model.compile(loss='mean_absolute_error', optimizer='adam')
model.fit(data, target, nb_epoch=2000, batch_size=1, verbose=2)

如何在 tensorflow 0.11 中使用 RNN 编写多维回归预测器

How to write a multidimentional regression predictor using an RNN in tensorflow 0.11

machine-learning

time-series

prediction

tensorflow