angle-uparrow-clockwisearrow-counterclockwisearrow-down-uparrow-leftatcalendarcard-listchatcheckenvelopefolderhouseinfo-circlepencilpeoplepersonperson-fillperson-plusphoneplusquestion-circlesearchtagtrashx

Predicting the next future value using Deep Learning and LSTM

Predicting the future with Deep Learning is a hot topic. That is why I want to master this.

30 January 2022 Updated 30 January 2022
post main image
https://www.pexels.com/nl-nl/@andrew

Many problems are time related. We have collected some samples and now want to use them to predict the next value. That is what this post is about. It is not about predicting many future values, that is a different topic.
As a data scientist noob, I just work through some examples I found on the internet. I change the input sequence and see what happens. You might find this useful. I found the 'Machine Learning Mastery' website very helpful. I will mainly refer to the following article: 'How to Develop LSTM Models for Time Series Forecasting', see links below.

LSTM (Long Short-Term Memory)

LSTM is used when we want to make predictions that are time based. You have a dataset with samples. A sample can be a value like the price of a stock, but it also can be the price of a stock together with other values like social network sentiment. This is totally different from predicting a value based on non-time-related values like in my previous post 'Predicting values using Deep Learning and Keras'.

With LSTM we can include the time component but it is not necessary. LSTM is about defining a sequence in time, not about the time itself.

Univariate vs Multivariate

There are two basic types of LSTM.

Univariate Time Series:

  • one-dimensional
    Example1: Use only the stock market closing price
    Example2: Use only the energy consumption

Multivariate Time Series:

  • multidimensional
    Example1: use the stock market closing price and e.g. opening price and social media sentiment
    Example2: Use the energy consumption and temperature

Below I will investigate both Univariate LSTM and Multivariate LSTM using examples. For me the most difficult part was how to prepare the data for the model.  Once I understood this, I finally could focus on the solution.

Univariate Time Series: slope, use only y

I worked through the Univariate - Vanilla LSTM example on the page 'How to Develop LSTM Models for Time Series Forecasting', see links below, and created my own example by changing the input sequence using a function:

# define input sequence
def fx(x):
    y = 4 + 3*x
    return y

x_end = 6
raw_seq = []
for x in range(0, x_end):
    y = fx(x)
    raw_seq.append(y)

Prepare the input data. We use the above function to generate some samples:

raw_seq = [4, 7, 10, 13, 16, 19]

Choose the number of time steps:

n_steps = 3

Then we transform this into inputs and outputs like this:

[ 4,  7, 10] -> 13
[ 7, 10, 13] -> 16
[10, 13, 16] -> 19

To predict the next value we use the last inputs:

[13, 16, 19] -> ?

The inputs and outputs are converted into:

X = [
  [ 4  7 10]
  [ 7 10 13]
  [10 13 16]
]
y = [13 16 19]

And X is reshaped into:

reshaped X = [
  [ [ 4] [ 7] [10] ]
  [ [ 7] [10] [13] ]
  [ [10] [13] [16] ]
]

Note that we are predicting only the next value here. The code:

# Based on the Univariate Vanilla LSTM example from this page:
# How to Develop LSTM Models for Time Series Forecasting
# https://machinelearningmastery.com/how-to-develop-lstm-models-for-time-series-forecasting
#
# input sequence is a slope:
# - y = 4 + 3*x 
# - we are using only y
#
import numpy as np
from keras.models import Sequential
from keras.layers import Dense, LSTM

# split a univariate sequence into samples
def split_sequence(sequence, n_steps):
    X, y = list(), list()
    for i in range(len(sequence)):
        # find the end of this pattern
        end_ix = i + n_steps
        # check if we are beyond the sequence
        if end_ix > len(sequence)-1:
            break
        # gather input and output parts of the pattern
        seq_x, seq_y = sequence[i:end_ix], sequence[end_ix]
        X.append(seq_x)
        y.append(seq_y)
    return np.array(X), np.array(y)

# choose a number of time steps
n_steps = 3

# define input sequence
def fx(x):
    y = 4 + 3*x
    return y

x_end = 6
raw_seq = []
for x in range(0, x_end):
    y = fx(x)
    raw_seq.append(y)
print('raw_seq = {}'.format(raw_seq))

# predict data
x_input = raw_seq[-1*n_steps:]
print('x_input = {}'.format(x_input))
y_expected = fx(x_end)
print('y_expected = {}'.format(y_expected))

# split into samples
X, y = split_sequence(raw_seq, n_steps)
print('X = {}'.format(X))
print('y = {}'.format(y))

# reshape from [samples, timesteps] into [samples, timesteps, features]
n_features = 1
print('X.shape[0] = {}'.format(X.shape[0]))
print('X.shape[1] = {}'.format(X.shape[1]))
X = X.reshape((X.shape[0], X.shape[1], n_features))
print('reshaped X = {}'.format(X))

# define model
def get_model(m):
    if m == 'Vanilla_LSTM':
        model = Sequential(name=m)
        model.add(LSTM(50, activation='relu', input_shape=(n_steps, n_features)))
        model.add(Dense(1))
        model.compile(optimizer='adam', loss='mse')
    elif m == 'Stacked_LSTM':
        model = Sequential(name=m)
        model.add(LSTM(50, activation='relu', return_sequences=True, input_shape=(n_steps, n_features)))
        model.add(LSTM(50, activation='relu'))
        model.add(Dense(1))
        model.compile(optimizer='adam', loss='mse')
    model.summary()
    return model
#model = get_model('Vanilla_LSTM')
model = get_model('Stacked_LSTM')

# fit model
model.fit(X, y, epochs=200, verbose=0)

# show prediction
x_input = np.array(x_input)
x_input = x_input.reshape((1, n_steps, n_features))
yhat = model.predict(x_input, verbose=0)
print('prediction: for x_input = {}, yhat = {}, y_expected = {}'.format(x_input, yhat, y_expected))

And the result:

raw_seq = [4, 7, 10, 13, 16, 19]
x_input = [13, 16, 19]
y_expected = 22
X = [[ 4  7 10]
 [ 7 10 13]
 [10 13 16]]
y = [13 16 19]
X.shape[0] = 3
X.shape[1] = 3
reshaped X = [[[ 4]
  [ 7]
  [10]]

 [[ 7]
  [10]
  [13]]

 [[10]
  [13]
  [16]]]
Model: "Stacked_LSTM"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 lstm (LSTM)                 (None, 3, 50)             10400     
                                                                 
 lstm_1 (LSTM)               (None, 50)                20200     
                                                                 
 dense (Dense)               (None, 1)                 51        
                                                                 
=================================================================
Total params: 30,651
Trainable params: 30,651
Non-trainable params: 0
_________________________________________________________________
prediction: for x_input = [[[13]
  [16]
  [19]]], yhat = [[21.4401]], y_expected = 22

You can try it with Vanilla_LSTM and Stacked_LSTM models.

Multivariate Time Series: slope, use X and y

Again I worked through the Multivariate - Multiple Input Series example on the page 'How to Develop LSTM Models for Time Series Forecasting', see links below, and created my own example by changing the input sequence using a function:

# define input sequence
def fx(x):
    y = 4 + 3*x
    return y

x_end = 6
in_seq = []
out_seq = []
for x in range(0, x_end):
    y = fx(x)
    in_seq.append(x)
    out_seq.append(y)
in_seq = np.array(in_seq)
out_seq = np.array(out_seq)

Prepare the input data. We use the above function to generate some samples:

in_seq = [0 1 2 3 4 5]
out_seq = [ 4  7 10 13 16 19]

Choose the number of time steps:

n_steps = 3

Then we transform this into inputs and outputs like this:

[ [0] [1] [2] ] -> 10
[ [1] [2] [3] ] -> 13
[ [2] [3] [4] ] -> 16
[ [3] [4] [5] ] -> 19

To predict the next value we use the last inputs:

[ [ 4] [ 5] [ 6] ] -> ?

The inputs and outputs are converted into:

X = [
  [ [0] [1] [2] ]
  [ [1] [2] [3] ]
  [ [2] [3] [4] ]
  [ [3] [4] [5] ]
]
y = [10 13 16 19]

Note that we are predicting only the next value here. The code:

# Based on the multivariate LSTM Multiple Input Series example from this page:
# How to Develop LSTM Models for Time Series Forecasting
# https://machinelearningmastery.com/how-to-develop-lstm-models-for-time-series-forecasting
#
# input sequence is a slope:
# - y = 4 + 3*x 
# - we are using both x and y
#
import numpy as np
from keras.models import Sequential
from keras.layers import Dense, LSTM
 
# split a multivariate sequence into samples
def split_sequences(sequences, n_steps):
	X, y = list(), list()
	for i in range(len(sequences)):
		# find the end of this pattern
		end_ix = i + n_steps
		# check if we are beyond the dataset
		if end_ix > len(sequences):
			break
		# gather input and output parts of the pattern
		seq_x, seq_y = sequences[i:end_ix, :-1], sequences[end_ix-1, -1]
		X.append(seq_x)
		y.append(seq_y)
	return np.array(X), np.array(y)

# choose a number of time steps
n_steps = 4

# define input sequence
def fx(x):
    y = 4 + 3*x
    return y

x_end = 12
in_seq = []
out_seq = []
for x in range(0, x_end):
    y = fx(x)
    in_seq.append(x)
    out_seq.append(y)
in_seq = np.array(in_seq)
out_seq = np.array(out_seq)
print('in_seq = {}'.format(in_seq))
print('out_seq = {}'.format(out_seq))

# convert to [rows, columns] structure
in_seq = in_seq.reshape((len(in_seq), 1))
out_seq = out_seq.reshape((len(out_seq), 1))
print('after convert to [rows, columns]:')
print('in_seq = {}'.format(in_seq))
print('out_seq = {}'.format(out_seq))

# horizontally stack columns
dataset = np.hstack((in_seq, out_seq))
print('dataset = {}'.format(dataset))

# convert into input/output
X, y = split_sequences(dataset, n_steps)
print('X = {}'.format(X))
print('y = {}'.format(y))

# prediction data = last row of dataset
x_input = X[-1]
y_expected = y[-1]

# remove prediction data from dataset
X = X[:-1]
y = y[:-1]
print('X = {}'.format(X))
print('y = {}'.format(y))

# the dataset knows the number of features, e.g. 2
n_features = X.shape[2]
print('n_features = {}'.format(n_features))

# define model
def get_model(m):
    if m == 'Vanilla_LSTM':
        model = Sequential(name=m)
        model.add(LSTM(50, activation='relu', input_shape=(n_steps, n_features)))
        model.add(Dense(1))
        model.compile(optimizer='adam', loss='mse')
    elif m == 'Stacked_LSTM':
        model = Sequential(name=m)
        model.add(LSTM(50, activation='relu', return_sequences=True, input_shape=(n_steps, n_features)))
        model.add(LSTM(50, activation='relu'))
        model.add(Dense(1))
        model.compile(optimizer='adam', loss='mse')
    model.summary()
    return model
model = get_model('Vanilla_LSTM')
#model = get_model('Stacked_LSTM')

# fit model
model.fit(X, y, epochs=200, verbose=0)

# show prediction
x_input = x_input.reshape((1, n_steps, n_features))
yhat = model.predict(x_input, verbose=0)
print('prediction: for x_input = {}, yhat = {}, y_expected = {}'.format(x_input, yhat, y_expected))

And the result:

in_seq = [ 0  1  2  3  4  5  6  7  8  9 10 11]
out_seq = [ 4  7 10 13 16 19 22 25 28 31 34 37]
after convert to [rows, columns]:
in_seq = [[ 0]
 [ 1]
 [ 2]
 [ 3]
 [ 4]
 [ 5]
 [ 6]
 [ 7]
 [ 8]
 [ 9]
 [10]
 [11]]
out_seq = [[ 4]
 [ 7]
 [10]
 [13]
 [16]
 [19]
 [22]
 [25]
 [28]
 [31]
 [34]
 [37]]
dataset = [[ 0  4]
 [ 1  7]
 [ 2 10]
 [ 3 13]
 [ 4 16]
 [ 5 19]
 [ 6 22]
 [ 7 25]
 [ 8 28]
 [ 9 31]
 [10 34]
 [11 37]]
X = [[[ 0]
  [ 1]
  [ 2]
  [ 3]]

 [[ 1]
  [ 2]
  [ 3]
  [ 4]]

 [[ 2]
  [ 3]
  [ 4]
  [ 5]]

 [[ 3]
  [ 4]
  [ 5]
  [ 6]]

 [[ 4]
  [ 5]
  [ 6]
  [ 7]]

 [[ 5]
  [ 6]
  [ 7]
  [ 8]]

 [[ 6]
  [ 7]
  [ 8]
  [ 9]]

 [[ 7]
  [ 8]
  [ 9]
  [10]]

 [[ 8]
  [ 9]
  [10]
  [11]]]
y = [13 16 19 22 25 28 31 34 37]
X = [[[ 0]
  [ 1]
  [ 2]
  [ 3]]

 [[ 1]
  [ 2]
  [ 3]
  [ 4]]

 [[ 2]
  [ 3]
  [ 4]
  [ 5]]

 [[ 3]
  [ 4]
  [ 5]
  [ 6]]

 [[ 4]
  [ 5]
  [ 6]
  [ 7]]

 [[ 5]
  [ 6]
  [ 7]
  [ 8]]

 [[ 6]
  [ 7]
  [ 8]
  [ 9]]

 [[ 7]
  [ 8]
  [ 9]
  [10]]]
y = [13 16 19 22 25 28 31 34]
n_features = 1
Model: "Vanilla_LSTM"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 lstm (LSTM)                 (None, 50)                10400     
                                                                 
 dense (Dense)               (None, 1)                 51        
                                                                 
=================================================================
Total params: 10,451
Trainable params: 10,451
Non-trainable params: 0
_________________________________________________________________
prediction: for x_input = [[[ 8]
  [ 9]
  [10]
  [11]]], yhat = [[35.79921]], y_expected = 37

You can try it with Vanilla_LSTM and Stacked_LSTM models.

Summary

This was a totally different experience compared to the one described in the previous post 'Predicting values using Deep Learning and Keras'. Understanding how to prepare the data was very important, if not the most important. But there is some excellent documentation on the internet about this.
LSMT is a black box, I like this. But it may need a lot of data points (samples). The most important thing for me at the moment is the quality of the prediction. How good is it?  Also I can predict the future for the next time unit, but I also want to predict the future for many more time units. We all want this right? Very much still to learn ...

Links / credits

How to Convert a Time Series to a Supervised Learning Problem in Python
https://machinelearningmastery.com/convert-time-series-supervised-learning-problem-python

How to Develop LSTM Models for Time Series Forecasting
https://machinelearningmastery.com/how-to-develop-lstm-models-for-time-series-forecasting

How to Use the TimeseriesGenerator for Time Series Forecasting in Keras
https://machinelearningmastery.com/how-to-use-the-timeseriesgenerator-for-time-series-forecasting-in-keras

Understand Keras's RNN behind the scenes with a sin wave example - Stateful and Stateless prediction
https://fairyonice.github.io/Understand-Keras's-RNN-behind-the-scenes-with-a-sin-wave-example.html

Leave a comment

Comment anonymously or log in to comment.

Comments

Leave a reply

Reply anonymously or log in to reply.