Demand prediction for urban air mobility using deep learning

Machine learning pipeline

A machine learning pipeline is a way of formalizing and organizing the processes required to deploy a machine learning model. Machine learning pipelines are sequential steps that handle everything from data extraction and preprocessing to model training and deployment. The one presented in this section is concerned with supervised learning, using tabular data, as illustrated in Table 1. In this table, Xj (j = 1,2,…) represents a given input feature, and y represents the output (the variable to predict), a continuous variable in our case study.

The following steps have to be followed to accomplish a demand prediction task. It starts (as shown in Fig. 1) with step 1, data acquisition that includes data collection from different sources, e.g., sensors, physical measurements, and analysis (physical, chemical, etc.). Since urban mobility uses and generates a huge amount of data, we can consider big data, real-time data, and velocity of data as critical aspects while storing this huge data.

Then, some data fusion techniques must be performed to build a dataset for the next step, 2, data preprocessing. First, we clean the data to detect and fix incorrect or missing values in the data pre-processing step. For example, we may discover many data anomalies with zero, negative, or illogical values. We can also do feature engineering in this step, which means we can perform some calculations to create a new dataset feature. This step allows putting the data in a format compatible with the machine learning algorithms used in step 3. The third step of the pipeline consists of building or using an appropriate machine learning model, e.g., (support vector machine (SVM), k-nearest neighbor (KNN), Random Forest, etc.), which depends on many parameters like type of data (image, text, voice, time-series, etc.) In step 4, these models are trained and tested by setting different tunning hyperparameters to achieve an efficient performance. For example, in this study, we will use the following hyperparameters:

Embedding size: An embedding size is a relatively low-dimensional space into which you can translate high-dimensional vectors. Embeddings make it easier to do machine learning on large inputs like sparse vectors representing words.

Hidden layers: Hidden layers are the layers between the input layer and output layer.

Batch size: Mini batch size is the number of sub-samples given to the network after which parameter update happens.

Learning rate: The learning rate defines how quickly a network updates its parameters.

Epochs: The number of epochs is the number of times the whole training data is shown to the network while training. In step 5, we used different evaluation metrics to evaluate the performance of implemented models. When evaluating machine learning models, choosing the right metric is also critical. It aims to estimate the generalization accuracy of the future (unseen/out-of-sample) data. For example, for the classification problem, we have matrics such as confusion matrix, precision, recall, F1 score, Log loss, and for the regression problem, we have mean absolute error (MAE), mean square error (MSE), and root mean square error (RMSE). This study uses RMSE as an evaluation metric to evaluate performance. The RMSE is the most commonly used metric for regression tasks (see Eq. 1). This is calculated by taking the square root of the average squared distance between the actual and predicted scores. (1)${\displaystyle RMSE\sqrt{\frac{1}{n}\sum _{i=1}^n{\left(y_i-{\hat{y}}_i\right)}^2}.}$

Here, yⁱ denotes the true score for the i^th data point, and yⁱ denotes the predicted value.

Ground taxis face longer routing and travel times due to factors such as traffic congestion, route restrictions, stop-and-go nature, terrain and geography, and limited speed. These factors can slow down travel times, especially in densely populated urban areas with congested roadways. Ground taxis also need to navigate around one-way streets, road closures, and detours, adding distance and time. They also need to stop at traffic lights, intersections, or pick-up/drop-off points, extending travel times. Geographic obstacles like rivers, mountains, or densely populated areas can limit route options and require longer detours. Ground taxis are also subject to speed limits, limiting their ability to travel quickly. Air taxis offer time savings and efficiency in urban mobility. The following block diagram shows the basic operation of UAM is illustrated in Fig. 2.

Dataset description

This section presents a detailed description of the datasets used in our experimental study. We used the New York City Taxi and Limousine Commission Trip Record Data dataset. This dataset contains records for each taxi journey, comprising pick-up and drop-off, date and time, pick-up and drop-off locations, trip lengths, categorized prices, rate kinds, payment methods, and driver-reported passenger counts (see Table 2). It should be noted that our case study is based on supervised learning. To that end, we should define, among the available variables, the one representing the output (y). In this study, the concerning output is the “demand” that we need to predict. But, in the raw data, no information regarding demand is provided. Therefore, we should define one, which is explained in the next section.

Demand calculation

In literature, Rajendran, Srinivas & Grimshaw (2021) uses the density of passengers requesting air taxi rides to travel to a specific location to calculate demand level. Therefore, we use the same columns to calculate the demand ratio based on passenger count and distance (see Eq. 2). The high ratio indicates high demand, and the low ratio indicates low demand. (2)${\displaystyle Ratio=PassengerCount\ast Distance.}$

In the next section, we prepare a detailed analysis of UAM demand prediction using different State-of-the-art deep learning (DL) approaches.

Long short-term memory (LSTM)

We implement the long short-term memory (LSTM) network, an improved variant of recurrent neural networks, to remember previously-stored information. This method was used to solve the RNN gradient problem. The LSTM method is well-suited for identifying, processing, and predicting time series with unknown time lags. We trained LSTM via backpropagation and reduced the network loss. An LSTM network has three gates see (Fig. 3).

Input gate: Discover which value to change memory from the input. The sigmoid function determines what values [0,1] should be moved. The tanh function, on the other hand, assigns weight to the values passed to assess their importance on a scale of [−1 to 1]. (3)${\displaystyle i_t=\sigma \left(W_i\ast \left[h_{t-1},x_t\right]+b_i\right)}$ (4)${\displaystyle {\tilde{C}}_t=tanh\left(W_c\ast \left[h_{t-1},x_t\right]+b_c\right)}$

Forget gate: Discover what blocks discarding information. The sigmoid function is defined as it examines the previous state and content inputs and returns a number ranging from 0 (omit this) to 1 (keep this) for each of the inputs C_(t−1) cell state number. (5)${\displaystyle f_t=\sigma \left(W_f\ast \left[h_{t-1},x_t\right]+b_f\right)}$

Output gate: Block input and memory to evaluate output. The sigmoid function defines 0,1 values. The tanh function gives weighting to pass values to determine their level of importance from −1 to 1 and multiplied with sigmoid output. (6)${\displaystyle O_t=\sigma \left(W_o\ast \left[h_{t-1},x_t\right]+b_o\right)}$ (7)${\displaystyle h_t=O_t\ast tanh\left(C_t\right).}$

Gated recurrent unit (GRU)

We used the GRU network, the enhanced version of the regular recurrent neural network (Cho et al., 2015). GRU employs gate updates and resets to solve the vanishing gradient problem of a standard RNN. These two vectors determine the information that should be transferred to the output. They are unique in that they can be trained to remember past information without being washed away over time or information unrelated to prediction being eliminated. Figure 4 demonstrates detailed gated recurrent unit (GRU) architecture.

Update gate: For time step t, the formula for calculating the update gate z_t is: (8)${\displaystyle z_t=\sigma \left(W^{\left(z\right)}{x}_t+U^{\left(z\right)}{h}_{t-1}\right).}$

Next, x_t is connected to the network unit, it is multiplied by its weight W^((z)). Similarly for h_(t−1) stores data for the preceding t-1 units then multiplied by its weight U^((z)). All outcomes are entered, and the number is squashed between 0 and 1, applying a sigmoid activation feature. Figure 5 follows the schema as mentioned earlier.

The model can use the update gate to determine how much previous knowledge (from earlier steps in time) should be transferred into the future. This is extremely useful because the model will decide to copy all previous data and eliminate the possibility of gradient fading.

Reset gate: Basically, the model uses this gate to determine how much past knowledge to forget. To measure, we applied: (9)${\displaystyle r_t=\sigma \left(W^{\left(r\right)}{x}_t+U^{\left(r\right)}{h}_{t-1}\right)}$ This formula is the same for update gates. The difference is in weight and gate use. Figure 6 below indicates where the gate is: As previously, we plug in h_(t−1) blueline and x_t purple line, multiply with their respective weights, add the outcomes, and use the sigmoid function.

Current memory content: The effect of the gates on the final product is demonstrated in this example. Then, we start using the reset gates. The reset gate stores past data and adds new memory content. The following is how it was calculated: (10)${\displaystyle h_t^{\prime }=tanh\left(W{x}_t+r_t\odot U{h}_{t-1}\right)}$

Step 1. The input x_t multiply by a weight W and h_(t−1) by a weight U.

Step 2. Hadamard (element-wise) calculates among the reset gate r_t and Uh_(t−1). It resolves to decide what measures should be removed from the previous ones. Assume we are dealing with a sentiment analysis issue to determine someone’s feelings for a book review. After a few paragraphs, the text begins with “This is a fantasy novel that explains...” and ends with “I didn’t like the book because I think it captures so many specifics.” We only need the last section of the review to determine the book’s overall satisfaction level. A neural network gets closer to the conclusion of the text; this will be learning to allocate r_t vector near to 0, trying to erase previous sentences and concentrating only on the ending sentences.

Step 3. Steps 1 and 2 should be added together.

Step 4. Use the tanh nonlinear activation function.

Figure 7 illustrates the steps. We ensure an element-wise multiplication of h_(t−1) blueline, and r_t orange line and then add the result pink line by the input x_t purple line. Lastly, tanh is used to produce $h_t^{{}^{\prime }}$ bright green line.

Final memory at current time step: Lastly, the network, represented by the, h_t vector is measured. This vector contains information about the current unit and transfers it to the network. Therefore, the upgrade gate is required. Defines what should be collected from present memory substance $h_t^{{}^{\prime }}$ and what should be collected from prior steps h_(t−1). This is how it’s done: (11)${\displaystyle h_t=z_t\odot h_{t-1}+\left(1-z_t\right)\odot h_t^{\prime }.}$

To update the gate, element-wise multiplication should be used z_t and h_(t−1). Use element-wise multiplication to 1 − z_t and $h_t^{{}^{\prime }}$. Add the outcomes from steps 1 and 2 The most critical information is now at the start of the document. It will learn the vector environment, z_t near to 1 and hold the most previous details. After that, z_t approximately 1 at this time stage, 1 − z_t approximately 0 would disregard a significant percentage of the remaining content (in this scenario, the last section of the analysis describing the book plan), insignificant to our prediction. Figure 8 highlights the above equation: As a result of this, z_t the green line was used to compute 1 − z_t and combined with $h_t^{{}^{\prime }}$, the bright green line produces a result as a dark red line. z_t is used with h_(t−1) blueline for element-wise multiplication. Lastly, h_t, the blue line results from the summation of the outputs equivalent to the bright and dark red lines. Hence, we can see how GRUs can store and filter data through their gates. The vanishing gradient problem is no longer an issue because the model does not wash out the new input every time it is run. In its place, it saves the relevant data and forwards it to the next network step. They can perform exceptionally well in complex situations if they have received the proper training and preparation.

Transformer

The “Attention Is All You Need” article introduces a new architecture known as “Transformer” (Vaswani et al., 2017). Similar to the LSTM, Transformer is a two-part architecture for transforming sequences used in machine learning (encoder and decoder). Since it does not imply recurrent networks, it differs from previous sequence-to-sequence models (GRU, LSTM, etc.). Using recurrent networks to capture sequence dependencies is one of the most effective methods available as shown in Fig. 9. Architecture is shown that on the left is the encoder, and on the right is the decoder. In the diagram, Nx shows the encoder and decoder as stackable modules. Each module is comprised of feed-forward and multi-head attention layers. Because we cannot use strings directly, we must first embed the inputs and outputs in an n-dimensional space. The model’s positional encoding of words is a minor but crucial component. We must assign a relative position to each word or part because the order of the elements in a sequence is important. Each word contains these positions (n-dimensional vector) (See Fig. 10). Let us start with the attention mechanism’s description. It is simple and can be described by the equation: (12)${\displaystyle Attention\left(Q,K,V\right)=\sigma \left(\frac{Q{K}^i}{\sqrt{d_k}}\right)V.}$

Q is a matrix containing the query (vector representation of one word in the sequence), K contains all the keys (vector representations of all words in the sequence), and V contains all the values. V and Q are the same word sequences for multi-head attention modules in encoders and decoders. V is different from Q because it is used in the attention module, which considers encoder and decoder sequences. So, we multiply and add the values in V together with certain attention weights specified as follows: (13)${\displaystyle a=softmax\left(\left(\frac{Q{K}^T}{\sqrt{d_k}}\right)\right..}$

The weights are based on how each word (Q) influences the others (represented by K). Weights between 0 and 1 are also applied to the SoftMax function. Those weights are used for all words in V. When compared to Q, the encoder and decoder vectors are the same but different from the module with both encoder and decoder inputs. It is possible to parallelize this attention mechanism into multiple mechanisms, as illustrated in the right image. The attention mechanism is recurrent, having the types Q, K, and V linear projections. As a result, the model may learn from dissimilar Q, K, and V representations. These linear representations multiply the variables Q, K, and V by learned weight matrices W. Q, K, and V of the attention modules differ dependent on whether they are located in the encoder, decoder, or somewhere between these two locations. We need to focus on also the entire encoder input sequence or a subset of it for this analysis. A multi-head attention module is used to ensure that an encoder input sequence and the decoder input sequence are considered up to a certain point in the decoding process. A pointwise feed-forward layer follows the encoder and decoder’s multi-attention heads. Using the same parameters for each position in the sequence, this small feed-forward network performs a linear transformation on each element in the given sequence.

Execution environment

We set up the following execution environment to implement the aforementioned deep learning models:

• Python 3 was used as a programming language.

• System characteristics used an Intel Core i5 processor at 2.67 GHz and 8GB of memory.

• Libraries used for implementation are TensorFlow (Dean & Monga, 2015), Scikit-learn (Vaswani et al., 2017), and Keras (Cho et al., 2015).

Results and discussion

In this section, the results of each step of the machine-learning pipeline discussed earlier in the methodology section are presented in detail.

Step 1. Data acquisition: Data acquisition is the first step in the machine learning pipeline. Since air taxi services are currently in the testing phase, we do not have the actual data for demand prediction. As a result, we predict future demand for air taxi services using a percentage of current normal taxi demand from the land taxi-based benchmark dataset. Therefore, we use the New York City Taxi, and Limousine Commission Trip Record Dataset in this study discussed earlier in the ‘Dataset description’ section, with a detailed description of each field of the dataset. This dataset contains records for each taxi journey, comprising pick-up and drop-off, date and time, pick-up and drop-off locations, trip lengths, categorized prices, rate kinds, payment methods, and driver-reported passenger counts.

Step 2. Data preprocessing:

The data preprocessing is the second step and critical step in the machine learning pipeline. We discussed the case study dataset and the description of each dataset column in the ‘Literature Review’ section. Since this dataset is a raw dataset, we found many data anomalies that need to be fixed in this step, e.g., a few columns with zero, negative, and illogical values. We also eliminate some columns in this data preprocessing steps that are no longer useful for demand prediction (rate_code id, congestion surcharge, etc.)

Step 3. Models implemented: In this step, we implement three deep learning-based models (LSTM, GRU, and Transformer) that were tested in our experiment, and the results were as follows:

LSTM trained on the training dataset and tested on the test dataset.

GRU trained on the training dataset and tested on the test dataset.

The transformer was trained on the training dataset and tested on the test dataset.

Step 4. Models training: In this step, we trained and tested the models mentioned earlier by using different hyperparameters to tune the performance of the methods discussed above in step 3.

Tables 3, 4 and 5 shows the hyperparameters for the LSTM, GRU, and transformer network.

Step 5. Models evaluation

As previously discussed in the ‘Methodology’ section, we used various evaluation metrics to assess the performance of the implemented models, including mean absolute error (MAE), mean square error (MSE), and root mean square error (RMSE) (RMSE). We used RMSE as an evaluation metric to evaluate performance in this experiment. For regression tasks, the RMSE is the most commonly used metric. The square root of the average squared distance between the actual and predicted scores is used to calculate this. As a result, we achieved a root-mean-square error of 2.06 for the LSTM network and 2.11 for the GRU network and 0.64 for the Transformer network. The experimental results show that the transformer outperformed on GRU and LSTM networks. The detailed results are summarized in Table 6. Our empirical study presented the evidence to support the demand prediction with relatively high predictive performance. Our experimental result shows that a Transformer was applied to analyse urban mobility demand and highly predictive performance (Abbasi et al., 2020; Abassi et al., 2023). Since the Transformer method performs better than GRU and LSTM, we will analyze some key insights in detail in the next section.

More results of the best model

As previously demonstrated, the transformer model outperformed LSTM and GRU. More transformer results, specifically attention loss, learning curve, and variable importance, are examined in this section.

Attention loss. Regarding attention loss for training and testing steps, we have seen in Figs. 11 and 12 that values are overlapping. It shows that the loss is relatively minimal between observed and predicted values.

Learning curve. We ran multiple experiments with different values to find the optimal learning rate and found that a 0.035 learning rate was the best (see Fig. 13).

Variable importance. Since we have many variables in our dataset and they do not influence demand equally, that is why we use variable importance to identify the most important variables. Figure 14 depicts important variables that are more useful and influential for forecasting demand, such as trip distance, passenger count, and pickup and drop-off locations. Fig. 13). It is interesting for decision-makers who want to deploy or implement UAM. Understanding and achieving optimal values for these variables is critical when investing. For example, a high passenger count indicates a high demand for long enough distances. Pick-up and drop-off locations are also important for decision-makers when deciding on new UAM infrastructure deployment, such as vertiports, etc. These significant findings indicate the possibility of utilizing an approach such as transfer learning to propagate knowledge from previously trained models to a new UAM dataset, as outlined in the following section.