{
"cells": [
{
"cell_type": "markdown",
"metadata": {
"origin_pos": 0
},
"source": [
"# Factorization Machines\n",
"\n",
"Factorization machines (FM) :cite:`Rendle.2010`, proposed by Steffen Rendle in 2010, is a supervised algorithm that can be used for classification, regression, and ranking tasks. It quickly took notice and became a popular and impactful method for making predictions and recommendations. Particularly, it is a generalization of the linear regression model and the matrix factorization model. Moreover, it is reminiscent of support vector machines with a polynomial kernel. The strengths of factorization machines over the linear regression and matrix factorization are: (1) it can model $\\chi$-way variable interactions, where $\\chi$ is the number of polynomial order and is usually set to two. (2) A fast optimization algorithm associated with factorization machines can reduce the polynomial computation time to linear complexity, making it extremely efficient especially for high dimensional sparse inputs. For these reasons, factorization machines are widely employed in modern advertisement and products recommendations. The technical details and implementations are described below.\n",
"\n",
"\n",
"## 2-Way Factorization Machines\n",
"\n",
"Formally, let $x \\in \\mathbb{R}^d$ denote the feature vectors of one sample, and $y$ denote the corresponding label which can be real-valued label or class label such as binary class \"click/non-click\". The model for a factorization machine of degree two is defined as:\n",
"\n",
"$$\n",
"\\hat{y}(x) = \\mathbf{w}_0 + \\sum_{i=1}^d \\mathbf{w}_i x_i + \\sum_{i=1}^d\\sum_{j=i+1}^d \\langle\\mathbf{v}_i, \\mathbf{v}_j\\rangle x_i x_j\n",
"$$\n",
"\n",
"where $\\mathbf{w}_0 \\in \\mathbb{R}$ is the global bias; $\\mathbf{w} \\in \\mathbb{R}^d$ denotes the weights of the i-th variable; $\\mathbf{V} \\in \\mathbb{R}^{d\\times k}$ represents the feature embeddings; $\\mathbf{v}_i$ represents the $i^\\mathrm{th}$ row of $\\mathbf{V}$; $k$ is the dimensionality of latent factors; $\\langle\\cdot, \\cdot \\rangle$ is the dot product of two vectors. $\\langle \\mathbf{v}_i, \\mathbf{v}_j \\rangle$ model the interaction between the $i^\\mathrm{th}$ and $j^\\mathrm{th}$ feature. Some feature interactions can be easily understood so they can be designed by experts. However, most other feature interactions are hidden in data and difficult to identify. So modeling feature interactions automatically can greatly reduce the efforts in feature engineering. It is obvious that the first two terms correspond to the linear regression model and the last term is an extension of the matrix factorization model. If the feature $i$ represents an item and the feature $j$ represents a user, the third term is exactly the dot product between user and item embeddings. It is worth noting that FM can also generalize to higher orders (degree > 2). Nevertheless, the numerical stability might weaken the generalization.\n",
"\n",
"\n",
"## An Efficient Optimization Criterion\n",
"\n",
"Optimizing the factorization machines in a straight forward method leads to a complexity of $\\mathcal{O}(kd^2)$ as all pairwise interactions require to be computed. To solve this inefficiency problem, we can reorganize the third term of FM which could greatly reduce the computation cost, leading to a linear time complexity ($\\mathcal{O}(kd)$). The reformulation of the pairwise interaction term is as follows:\n",
"\n",
"$$\n",
"\\begin{aligned}\n",
"&\\sum_{i=1}^d \\sum_{j=i+1}^d \\langle\\mathbf{v}_i, \\mathbf{v}_j\\rangle x_i x_j \\\\\n",
" &= \\frac{1}{2} \\sum_{i=1}^d \\sum_{j=1}^d\\langle\\mathbf{v}_i, \\mathbf{v}_j\\rangle x_i x_j - \\frac{1}{2}\\sum_{i=1}^d \\langle\\mathbf{v}_i, \\mathbf{v}_i\\rangle x_i x_i \\\\\n",
" &= \\frac{1}{2} \\big (\\sum_{i=1}^d \\sum_{j=1}^d \\sum_{l=1}^k\\mathbf{v}_{i, l} \\mathbf{v}_{j, l} x_i x_j - \\sum_{i=1}^d \\sum_{l=1}^k \\mathbf{v}_{i, l} \\mathbf{v}_{i, l} x_i x_i \\big)\\\\\n",
" &= \\frac{1}{2} \\sum_{l=1}^k \\big ((\\sum_{i=1}^d \\mathbf{v}_{i, l} x_i) (\\sum_{j=1}^d \\mathbf{v}_{j, l}x_j) - \\sum_{i=1}^d \\mathbf{v}_{i, l}^2 x_i^2 \\big ) \\\\\n",
" &= \\frac{1}{2} \\sum_{l=1}^k \\big ((\\sum_{i=1}^d \\mathbf{v}_{i, l} x_i)^2 - \\sum_{i=1}^d \\mathbf{v}_{i, l}^2 x_i^2)\n",
" \\end{aligned}\n",
"$$\n",
"\n",
"With this reformulation, the model complexity are decreased greatly. Moreover, for sparse features, only non-zero elements needs to be computed so that the overall complexity is linear to the number of non-zero features.\n",
"\n",
"To learn the FM model, we can use the MSE loss for regression task, the cross-entropy loss for classification tasks, and the BPR loss for ranking task. Standard optimizers such as stochastic gradient descent and Adam are viable for optimization.\n"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {
"attributes": {
"classes": [],
"id": "",
"n": "2"
},
"origin_pos": 1,
"tab": [
"mxnet"
]
},
"outputs": [],
"source": [
"import os\n",
"from mxnet import gluon, init, np, npx\n",
"from mxnet.gluon import nn\n",
"from d2l import mxnet as d2l\n",
"\n",
"npx.set_np()"
]
},
{
"cell_type": "markdown",
"metadata": {
"origin_pos": 2
},
"source": [
"## Model Implementation\n",
"The following code implement the factorization machines. It is clear to see that FM consists a linear regression block and an efficient feature interaction block. We apply a sigmoid function over the final score since we treat the CTR prediction as a classification task.\n"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {
"attributes": {
"classes": [],
"id": "",
"n": "2"
},
"origin_pos": 3,
"tab": [
"mxnet"
]
},
"outputs": [],
"source": [
"class FM(nn.Block):\n",
" def __init__(self, field_dims, num_factors):\n",
" super(FM, self).__init__()\n",
" num_inputs = int(sum(field_dims))\n",
" self.embedding = nn.Embedding(num_inputs, num_factors)\n",
" self.fc = nn.Embedding(num_inputs, 1)\n",
" self.linear_layer = nn.Dense(1, use_bias=True)\n",
"\n",
" def forward(self, x):\n",
" square_of_sum = np.sum(self.embedding(x), axis=1) ** 2\n",
" sum_of_square = np.sum(self.embedding(x) ** 2, axis=1)\n",
" x = self.linear_layer(self.fc(x).sum(1)) \\\n",
" + 0.5 * (square_of_sum - sum_of_square).sum(1, keepdims=True)\n",
" x = npx.sigmoid(x)\n",
" return x"
]
},
{
"cell_type": "markdown",
"metadata": {
"origin_pos": 4
},
"source": [
"## Load the Advertising Dataset\n",
"We use the CTR data wrapper from the last section to load the online advertising dataset.\n"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {
"attributes": {
"classes": [],
"id": "",
"n": "3"
},
"origin_pos": 5,
"tab": [
"mxnet"
]
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Downloading ../data/ctr.zip from http://d2l-data.s3-accelerate.amazonaws.com/ctr.zip...\n"
]
}
],
"source": [
"batch_size = 2048\n",
"data_dir = d2l.download_extract('ctr')\n",
"train_data = d2l.CTRDataset(os.path.join(data_dir, 'train.csv'))\n",
"test_data = d2l.CTRDataset(os.path.join(data_dir, 'test.csv'),\n",
" feat_mapper=train_data.feat_mapper,\n",
" defaults=train_data.defaults)\n",
"train_iter = gluon.data.DataLoader(\n",
" train_data, shuffle=True, last_batch='rollover', batch_size=batch_size,\n",
" num_workers=d2l.get_dataloader_workers())\n",
"test_iter = gluon.data.DataLoader(\n",
" test_data, shuffle=False, last_batch='rollover', batch_size=batch_size,\n",
" num_workers=d2l.get_dataloader_workers())"
]
},
{
"cell_type": "markdown",
"metadata": {
"origin_pos": 6
},
"source": [
"## Train the Model\n",
"Afterwards, we train the model. The learning rate is set to 0.02 and the embedding size is set to 20 by default. The `Adam` optimizer and the `SigmoidBinaryCrossEntropyLoss` loss are used for model training.\n"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {
"attributes": {
"classes": [],
"id": "",
"n": "5"
},
"origin_pos": 7,
"tab": [
"mxnet"
]
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"loss 0.505, train acc 0.284, test acc 0.278\n",
"156830.9 examples/sec on [gpu(0), gpu(1)]\n"
]
},
{
"data": {
"image/svg+xml": [
"\n",
"\n",
"\n"
],
"text/plain": [
""
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"devices = d2l.try_all_gpus()\n",
"net = FM(train_data.field_dims, num_factors=20)\n",
"net.initialize(init.Xavier(), ctx=devices)\n",
"lr, num_epochs, optimizer = 0.02, 30, 'adam'\n",
"trainer = gluon.Trainer(net.collect_params(), optimizer,\n",
" {'learning_rate': lr})\n",
"loss = gluon.loss.SigmoidBinaryCrossEntropyLoss()\n",
"d2l.train_ch13(net, train_iter, test_iter, loss, trainer, num_epochs, devices)"
]
},
{
"cell_type": "markdown",
"metadata": {
"origin_pos": 8
},
"source": [
"## Summary\n",
"\n",
"* FM is a general framework that can be applied on a variety of tasks such as regression, classification, and ranking.\n",
"* Feature interaction/crossing is important for prediction tasks and the 2-way interaction can be efficiently modeled with FM.\n",
"\n",
"## Exercises\n",
"\n",
"* Can you test FM on other dataset such as Avazu, MovieLens, and Criteo datasets?\n",
"* Vary the embedding size to check its impact on performance, can you observe a similar pattern as that of matrix factorization?\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"origin_pos": 9,
"tab": [
"mxnet"
]
},
"source": [
"[Discussions](https://discuss.d2l.ai/t/406)\n"
]
}
],
"metadata": {
"language_info": {
"name": "python"
}
},
"nbformat": 4,
"nbformat_minor": 4
}