Gradient Descent Optimizer

(1) What Is Sequential Data?

Sequential data is data arranged in sequences where order matters. Data points are dependent on other data points in this sequence.
Examples of sequential data include customer grocery purchases, patient medical records, or simply a sequence of numbers with a pattern.
In sequence data, each element in the sequence is related to the ones that come before and after it, and the way these elements are ordered provides context, meaning, and structure to the data.

Order Matters: In sequence data, the arrangement of elements is important. Changing the order of elements can lead to different meanings or interpretations. For example, in natural language processing, the order of words in a sentence conveys the sentence’s meaning.
Temporal or Sequential Relationship: Sequence data often exhibits a temporal or sequential relationship. This means that the elements are typically collected or observed in a specific order over time. Time series data such as stock price, for instance, records values at different time points.
Variable Length: Sequences can have variable lengths. In natural language, sentences can be of different lengths, and in genomics, DNA sequences can vary in length depending on the organism.

Time Series Data: This is a common type of sequential data where measurements are taken at regular intervals over time. Examples include stock prices, temperature readings, and sensor data.
Natural Language Text: Textual data, such as sentences or paragraphs, is inherently sequential. The order of words matters to convey meaning, context, and relationships between words.
Speech Signals: Audio signals from spoken language are sequential. Phonemes, syllables, and words are organized sequentially to form sentences.
Music: Musical notes in a composition are ordered sequentially to create melodies, harmonies, and rhythms.
DNA Sequences: In genetics, DNA sequences represent the arrangement of nucleotides in a specific order, carrying genetic information.
Video Frames: In video data, each frame is a snapshot of a scene at a specific time. The sequence of frames creates the illusion of motion.
User Actions / Clickstream data: User interactions on a website or application, such as clicks or keystrokes, occur sequentially and can be used to understand user behavior.

Let’s consider a sequential problem to see how well suited basic neural network might be. Suppose we have a sequence of data that contains temperature and humidity value every day.
Our goal is to build a neural network that imports the temperature and humidity values of a given day as input and output. For instance, to predict the weather for that day is sunny or rainy. This is a straightforward task for a traditional feedforward neural network.

Using our dataset, we first feed a data point into the input layer. The data then flows into the hidden layer or layers where the weights and biases are applied. Then the output layer classifies the result from the hidden layer, which ultimately produces the output of sunny or cloudy or rainy weather.

Of course, we can repeat the steps for the second day and get the results. However, it is important to know that the model is stateless, it does not remember the data that it just analyzed. All it does it to take input after input and produce an individual classification for every day.

In fact, a traditional neural network assumes the data is non-sequential, and that each data point is independent of other data points. As a result, the inputs are analyzed in isolation, which can cause problems in case there are dependencies in the data.

To see how this can be a limitation, let’s go back to the weather example again. As you can imagine when examining weather, there is often a strong correlation of the weather on one day having a strong influence on the weather on the subsequent days.
That is if it was sunny on one day in the middle of the summer, it’s not unreasonable to presume that it’ll also be sunny the following day.
A traditional neural network model does make use of this information, however, so we’d have to turn to a different type of a model like a Recurrent Neural Network model.