60分钟吃掉三杀模型FiBiNET

神经网络的结构设计有3个主流的高级技巧：

• 1，高低融合 (将高层次特征与低层次特征融合，提升特征维度的丰富性和多样性，像人一样同时考虑整体和细节)
• 2，权值共享 (一个权值矩阵参与多个不同的计算，降低参数规模并同时缓解样本稀疏性，像人一样一条知识多处运用)
• 3，动态适应 (不同的输入样本使用不同的权值矩阵，动态地进行特征选择并赋予特征重要度解释性，像人一样聚焦重要信息排除干扰信息)

技巧应用范例：

• 1，高低融合 (DeepWide,UNet,特征金字塔FPN...)
• 2，权值共享 (CNN,RNN,FM,DeepFM,BlinearFFM...)
• 3，动态适应 (各种Attention机制...)

新浪微博广告推荐技术团队2019年发布的CTR预估模型FiBiNET同时巧妙地运用了以上3种技巧，是神经网络结构设计的教科书级的范例。

在此介绍给大家。

参考资料：

• FiBiNET论文：https://arxiv.org/pdf/1905.09433.pdf
• FiBiNET-结合特征重要性和双线性特征交互进行CTR预估：https://zhuanlan.zhihu.com/p/72931811
• 代码实现：https://github.com/xue-pai/FuxiCTR/blob/main/fuxictr/pytorch/models/FiBiNET.py
• SENet原理：https://zhuanlan.zhihu.com/p/65459972

公众号后台回复关键词: FiBiNET 获取本文全部源码和数据集下载链接。

本文目录： 一，FiBiNET原理解析 二，FiBiNET的pytorch实现 三，Criteo数据集完整范例 1，准备数据 2，定义模型 3，训练模型 4，评估模型 5，使用模型 6，保存模型

一，FiBiNET原理解析

FiBiNET全称为Feature Importance and Bilinear Interaction Network.

顾名思义，其主要的创意有2个。

第一个是Feature Importance，通过借鉴SENet（Squeeze-and-Excitation）Attention机制实现特征选择和重要度解释。

第二个是Bilinear Interaction Network，这是应用权值共享技巧对 FFM(Field-Aware FM)结构进行改进的一种结构。

同时，FiBiNET保留了DeepWide的高低融合的网络架构。

所以它综合使用了 高低融合、权值共享、动态适应 这3种神经网络结构设计的高级技巧。一个不落，Triple kill!

我们重点介绍一下 SENet Attention 和 Bilinear Interaction.

1, SENet Attention

SENet 全称为 Squeeze-and-Excitation Network，是一种通过注意力机制计算特征重要度的网络模块。

最早是在CV领域引入，通过在ResNet结构上添加SENet Attention模块，赢得了ImageNet 2017竞赛分类任务的冠军。

如何计算各个Feature Map(通道)的特征重要度(注意力权重)呢？

SENet的思想非常简洁。

step1: 通过全局池化将各个Feature Map由一个一个的矩阵汇总成一个一个的标量。此即Squeeze操作。

step2：通过一个2层MLP将汇总成得到的一个一个的标量所构成的向量进行变换，得到注意力权重。此即Excitation操作。细节一点地说，这个2层的MLP的第1层将通道数量缩减成原来的1/3, 第2层再将通道数恢复。并且每层后面都接入了激活函数。

step3：用注意力权重乘以原始的Feature Map。这个是Re-Weight操作。

图片示意如下。

pytorch代码实现如下，可能比图片更加好懂。

        
          
import torch   
from torch import nn   
class SENetAttention(nn.Module):  
    """  
    Squeeze-and-Excitation Attention  
    输入shape: [batch\_size, num\_fields, d\_embed]   #num\_fields即num\_features  
    输出shape: [batch\_size, num\_fields, d\_embed]  
    """  
    def \_\_init\_\_(self, num\_fields, reduction\_ratio=3):  
        super().__init__()  
        reduced_size = max(1, int(num_fields / reduction_ratio))  
        self.excitation = nn.Sequential(nn.Linear(num_fields, reduced_size, bias=False),  
                                        nn.ReLU(),  
                                        nn.Linear(reduced_size, num_fields, bias=False),  
                                        nn.ReLU())  
  
    def forward(self, x):  
        Z = torch.mean(x, dim=-1, out=None) #1,Sequeeze  
        A = self.excitation(Z) #2,Excitation  
        V = x * A.unsqueeze(-1) #3,Re-Weight  
        return V  

      

2, Bilinear Interaction

Bilinear Interaction实际上是FFM在权值共享思想下的一种改进，也可以称之为Bilinear FFM。

我们先说说FFM(Field-Aware FM)，再看看这个Bilinear FFM 怎么改进的。

FM用隐向量之间的点积来计算特征之间的交叉，并且一个特征用一个隐向量来表示。

FFM认为一个特征用一个隐向量来表达太粗糙了，如果这个特征和不同分组(Field)的特征来做交叉，应该用不同的隐向量。

举例来说，考虑一个广告点击预测的场景，广告类别 和 用户所在城市、用户职业之间的交叉。

在FM中 一个确定的广告类别 比如游戏广告 不论是和用户所在城市，还是用户职业交叉，都用同一个隐向量。

但是FFM认为，用户所在城市和用户职业是两类完全不同的特征(不同Field)，描述它们的向量空间应该是完全不相关的，FM用一个相同的隐向量来和它们做点积不合理。

所以，FFM引入了Field(域)的概念，和不同Field的特征做交叉，要使用不同的隐向量。

实践表明，FFM这个思路是有效的, FFM的作者阮毓钦正是凭借这个方案赢得了2015年kaggle举办的Criteo比赛的冠军。

但是FFM有个很大的缺点，就是参数量太多了。

对于FM来说，每个特征只有一个隐向量，假设有n个特征，每个隐向量维度为k，全部隐向量参数矩阵的大小 size = n k.

但是对于FFM，有过有f个不同的field，每个特征都将有f-1个隐向量，全部隐向量的参数矩阵的大小增大为 size = (f-1) n k.

通常的应用场景中，Field的数量有几十几百维，而Feature的数量有数万数百万维。

很显然，FFM将隐向量的参数规模扩大了几十几百倍。

FFM的本质思想是在做特征交叉的时候要区分不同的Field，其实现方式是和不同的Field做交叉时用不同的隐向量。

有没有办法保留FFM中区分不同Field的特性，并降低参数规模呢？

BilinearFFM说，我有办法，权重共享走起来！

BilinearFFM不直接针对不同Field设计不同的隐向量，而是引入了Field变换矩阵来区分不同的Field。

每个特征还是一个隐向量，但是和不同的Field的特征做交叉时，先乘上这个特征所在Field的变换矩阵，然后再做后面的点积。

因此，同属一个Field的特征共享一个Field变换矩阵。这种bilinear_type叫做 field_each.

Field变换矩阵的大小是k^2, 这种方式下，全部隐向量的参数大小加上共享变换矩阵的参数大小一共是 size = n k + f k^2

由于k和f远小于n，这种Bilinear方式相比FM增加的参数量可以忽略不计。

除了 同属一个Field的特征共享一个Field变换矩阵外，我们还可以更加简单粗暴一点，所有特征共享一个变换矩阵.

这种bilinear_type叫做 field_all.这种方式下，size = n k + k^2

我们也可以更加精细一点，相同的Field组合之间的交互共享一个变换矩阵，这种bilinear_type叫做field_interaction.

总共有f(f-1)/2种组合，这种方式下， size = n k + k^2 f(f-1)/2

以上就是BilinearFFM的基本思想。

FiBiNET中用到的Bilinear Interaction相比BilinearFFM, 还有一处小改动，将点积改成了哈达玛积，如下图所示。

pytorch代码实现如下，整体不难理解。作2点说明。

1，Field概念说明

在FFM相关的文章中，引入了Field的概念，以和Feature区分，一个Field中可以包括多个Feature.

实际上Field就是我们通常理解的特征，包括数值特征和类别特征，但是Feature是数值特征或者类别特征onehot后的特征。一个类别特征对应一个Field，但是对应多个Feature。

2，combinations函数说明

组合函数combinations从num_fields中任取2种作为组合，共有 num_fields*(num_fields-1)中组合方式。

所以输出的Field数量变成了 num_fields*(num_fields-1)/2。

        
          
import torch   
from torch import nn   
from itertools import combinations  
class BilinearInteraction(nn.Module):  
    """  
    双线性FFM  
    输入shape: [batch\_size, num\_fields, d\_embed] #num\_fields即num\_features  
    输出shape: [batch\_size, num\_fields*(num\_fields-1)/2, d\_embed]  
    """  
    def \_\_init\_\_(self, num\_fields, d\_embed, bilinear\_type="field\_interaction"):  
        super().__init__()  
        self.bilinear_type = bilinear_type  
        if self.bilinear_type == "field\_all":  
            self.bilinear_layer = nn.Linear(d_embed, d_embed, bias=False)  
        elif self.bilinear_type == "field\_each":  
            self.bilinear_layer = nn.ModuleList([nn.Linear(d_embed, d_embed, bias=False)  
                                                 for i in range(num_fields)])  
        elif self.bilinear_type == "field\_interaction":  
            self.bilinear_layer = nn.ModuleList([nn.Linear(d_embed, d_embed, bias=False)  
                                                 for i, j in combinations(range(num_fields), 2)])  
        else:  
            raise NotImplementedError()  
  
    def forward(self, feature\_emb):  
        feature_emb_list = torch.split(feature_emb, 1, dim=1)  
        if self.bilinear_type == "field\_all":  
            bilinear_list = [self.bilinear_layer(v_i) * v_j  
                             for v_i, v_j in combinations(feature_emb_list, 2)]  
        elif self.bilinear_type == "field\_each":  
            bilinear_list = [self.bilinear_layer[i](feature_emb_list[i]) * feature_emb_list[j]  
                             for i, j in combinations(range(len(feature_emb_list)), 2)]  
        elif self.bilinear_type == "field\_interaction":  
            bilinear_list = [self.bilinear_layer[i](v[0]) * v[1]  
                             for i, v in enumerate(combinations(feature_emb_list, 2))]  
        return torch.cat(bilinear_list, dim=1)  

      

二，FiBiNET的pytorch实现

下面是FiBiNET的一个pytorch实现。

核心代码是SENetAttention模块和BilinearInteraction模块的实现。

        
          
import torch   
from torch import nn   
from itertools import combinations  
  
class NumEmbedding(nn.Module):  
    """  
    连续特征用linear层编码  
    输入shape: [batch\_size,num\_features, d\_in], # d\_in 通常是1  
    输出shape: [batch\_size,num\_features, d\_out]  
    """  
      
    def \_\_init\_\_(self, n: int, d\_in: int, d\_out: int, bias: bool = False) -> None:  
        super().__init__()  
        self.weight = nn.Parameter(torch.Tensor(n, d_in, d_out))  
        self.bias = nn.Parameter(torch.Tensor(n, d_out)) if bias else None  
        with torch.no_grad():  
            for i in range(n):  
                layer = nn.Linear(d_in, d_out)  
                self.weight[i] = layer.weight.T  
                if self.bias is not None:  
                    self.bias[i] = layer.bias  
  
    def forward(self, x\_num):  
        assert x_num.ndim == 3  
        #x = x\_num[..., None] * self.weight[None]  
        #x = x.sum(-2)  
        x = torch.einsum("bfi,fij->bfj",x_num,self.weight)  
        if self.bias is not None:  
            x = x + self.bias[None]  
        return x  
      
class CatEmbedding(nn.Module):  
    """  
    离散特征用Embedding层编码  
    输入shape: [batch\_size, num\_features],   
    输出shape: [batch\_size, num\_features, d\_embed]  
    """  
    def \_\_init\_\_(self, categories, d\_embed):  
        super().__init__()  
        self.embedding = nn.Embedding(sum(categories), d_embed)  
        self.offsets = nn.Parameter(  
                torch.tensor([0] + categories[:-1]).cumsum(0),requires_grad=False)  
          
        nn.init.xavier_uniform_(self.embedding.weight.data)  
  
    def forward(self, x\_cat):  
        """  
        x\_cat: Long tensor of size ``(batch\_size, features\_num)``  
        """  
        x = x_cat + self.offsets[None]  
        return self.embedding(x)   
      
class CatLinear(nn.Module):  
    """  
    离散特征用Embedding实现线性层（等价于先F.onehot再nn.Linear()）  
    输入shape: [batch\_size, num\_features ],   
    输出shape: [batch\_size, d\_out]  
    """  
    def \_\_init\_\_(self, categories, d\_out=1):  
        super().__init__()  
        self.fc = nn.Embedding(sum(categories), d_out)  
        self.bias = nn.Parameter(torch.zeros((d_out,)))  
        self.offsets = nn.Parameter(  
                torch.tensor([0] + categories[:-1]).cumsum(0),requires_grad=False)  
        nn.init.xavier_uniform_(self.fc.weight.data)  
  
    def forward(self, x\_cat):  
        """  
        Long tensor of size ``(batch\_size, num\_features)``  
        """  
        x = x_cat + self.offsets[None]  
        return torch.sum(self.fc(x), dim=1) + self.bias   
      
class SENetAttention(nn.Module):  
    """  
    Squeeze-and-Excitation Attention  
    输入shape: [batch\_size, num\_fields, d\_embed]   #num\_fields即num\_features  
    输出shape: [batch\_size, num\_fields, d\_embed]  
    """  
    def \_\_init\_\_(self, num\_fields, reduction\_ratio=3):  
        super().__init__()  
        reduced_size = max(1, int(num_fields / reduction_ratio))  
        self.excitation = nn.Sequential(nn.Linear(num_fields, reduced_size, bias=False),  
                                        nn.ReLU(),  
                                        nn.Linear(reduced_size, num_fields, bias=False),  
                                        nn.ReLU())  
  
    def forward(self, x):  
        Z = torch.mean(x, dim=-1, out=None) #1,Sequeeze  
        A = self.excitation(Z) #2,Excitation  
        V = x * A.unsqueeze(-1) #3,Re-Weight  
        return V  
      
class BilinearInteraction(nn.Module):  
    """  
    双线性FFM  
    输入shape: [batch\_size, num\_fields, d\_embed] #num\_fields即num\_features  
    输出shape: [batch\_size, num\_fields*(num\_fields-1)/2, d\_embed]  
    """  
    def \_\_init\_\_(self, num\_fields, d\_embed, bilinear\_type="field\_interaction"):  
        super().__init__()  
        self.bilinear_type = bilinear_type  
        if self.bilinear_type == "field\_all":  
            self.bilinear_layer = nn.Linear(d_embed, d_embed, bias=False)  
        elif self.bilinear_type == "field\_each":  
            self.bilinear_layer = nn.ModuleList([nn.Linear(d_embed, d_embed, bias=False)  
                                                 for i in range(num_fields)])  
        elif self.bilinear_type == "field\_interaction":  
            self.bilinear_layer = nn.ModuleList([nn.Linear(d_embed, d_embed, bias=False)  
                                                 for i, j in combinations(range(num_fields), 2)])  
        else:  
            raise NotImplementedError()  
  
    def forward(self, feature\_emb):  
        feature_emb_list = torch.split(feature_emb, 1, dim=1)  
        if self.bilinear_type == "field\_all":  
            bilinear_list = [self.bilinear_layer(v_i) * v_j  
                             for v_i, v_j in combinations(feature_emb_list, 2)]  
        elif self.bilinear_type == "field\_each":  
            bilinear_list = [self.bilinear_layer[i](feature_emb_list[i]) * feature_emb_list[j]  
                             for i, j in combinations(range(len(feature_emb_list)), 2)]  
        elif self.bilinear_type == "field\_interaction":  
            bilinear_list = [self.bilinear_layer[i](v[0]) * v[1]  
                             for i, v in enumerate(combinations(feature_emb_list, 2))]  
        return torch.cat(bilinear_list, dim=1)  
      
  
#mlp  
class MultiLayerPerceptron(nn.Module):  
    def \_\_init\_\_(self, d\_in, d\_layers, dropout,   
                 d\_out = 1):  
        super().__init__()  
        layers = []  
        for d in d_layers:  
            layers.append(nn.Linear(d_in, d))  
            layers.append(nn.BatchNorm1d(d))  
            layers.append(nn.ReLU())  
            layers.append(nn.Dropout(p=dropout))  
            d_in = d  
        layers.append(nn.Linear(d_layers[-1], d_out))  
        self.mlp = nn.Sequential(*layers)  
  
    def forward(self, x):  
        """  
        float tensor of size ``(batch\_size, d\_in)``  
        """  
        return self.mlp(x)  
      
  
#fibinet   
class FiBiNET(nn.Module):  
      
    def \_\_init\_\_(self,  
                 d\_numerical,   
                 categories,   
                 d\_embed,  
                 mlp\_layers,   
                 mlp\_dropout,  
                 reduction\_ratio = 3,  
                 bilinear\_type = "field\_interaction",  
                 n\_classes = 1):  
          
        super().__init__()  
          
        if d_numerical is None:  
            d_numerical = 0  
        if categories is None:  
            categories = []  
              
        self.categories = categories  
        self.n_classes = n_classes  
          
        self.num_linear = nn.Linear(d_numerical,n_classes) if d_numerical else None  
        self.cat_linear = CatLinear(categories,n_classes) if categories else None  
          
        self.num_embedding = NumEmbedding(d_numerical,1,d_embed) if d_numerical else None  
        self.cat_embedding = CatEmbedding(categories, d_embed) if categories else None  
          
        num_fields = d_numerical+len(categories)  
          
        self.se_attention = SENetAttention(num_fields, reduction_ratio)  
        self.bilinear = BilinearInteraction(num_fields, d_embed, bilinear_type)  
          
        mlp_in = num_fields * (num_fields - 1) * d_embed  
        self.mlp = MultiLayerPerceptron(  
            d_in= mlp_in,  
            d_layers = mlp_layers,  
            dropout = mlp_dropout,  
            d_out = n_classes  
        )  
          
          
    def forward(self, x):  
        """  
        x\_num: numerical features  
        x\_cat: category features  
        """  
        x_num,x_cat = x  
          
        #一，wide部分  
        x_linear = 0.0  
        if self.num_linear:  
            x_linear = x_linear + self.num_linear(x_num)   
        if self.cat_linear:  
            x_linear = x_linear + self.cat_linear(x_cat)  
              
        #二，deep部分   
          
        #1，embedding  
        x_embedding = []  
        if self.num_embedding:  
            x_embedding.append(self.num_embedding(x_num[...,None]))  
        if self.cat_embedding:  
            x_embedding.append(self.cat_embedding(x_cat))  
        x_embedding = torch.cat(x_embedding,dim=1)  
          
        #2，interaction  
        se_embedding = self.se_attention(x_embedding)  
        ffm_out = self.bilinear(x_embedding)  
        se_ffm_out = self.bilinear(se_embedding)  
        x_interaction = torch.flatten(torch.cat([ffm_out, se_ffm_out], dim=1), start_dim=1)  
          
        #3，mlp  
        x_deep = self.mlp(x_interaction)  
          
        #三，高低融合  
        x_out = x_linear+x_deep  
        if self.n_classes==1:  
            x_out = x_out.squeeze(-1)  
        return x_out  

      

        
          
##测试 FiBiNET  
  
model = FiBiNET(d_numerical = 3, categories = [4,3,2],  
        d_embed = 4, mlp_layers = [20,20], mlp_dropout=0.25,  
        reduction_ratio = 3,  
        bilinear_type = "field\_interaction",  
        n_classes = 1)  
  
x_num = torch.randn(2,3)  
x_cat = torch.randint(0,2,(2,3))  
print(model((x_num,x_cat)))    
  

      

        
          
tensor([-0.8621,  0.6743], grad_fn=<SqueezeBackward1>)  

      

三，criteo数据集完整范例

Criteo数据集是一个经典的广告点击率CTR预测数据集。

这个数据集的目标是通过用户特征和广告特征来预测某条广告是否会为用户点击。

数据集有13维数值特征(I1-I13)和26维类别特征(C14-C39), 共39维特征, 特征中包含着许多缺失值。

训练集4000万个样本，测试集600万个样本。数据集大小超过100G.

此处使用的是采样100万个样本后的cretio_small数据集。

        
          
!pip install -U torchkeras -i https://pypi.org/simple/  

      

        
          
import numpy as np   
import pandas as pd   
import datetime   
  
from sklearn.model_selection import train_test_split   
  
import torch   
from torch import nn   
from torch.utils.data import Dataset,DataLoader    
import torch.nn.functional as F   
import torchkeras   
  
def printlog(info):  
    nowtime = datetime.datetime.now().strftime('%Y-%m-%d %H:%M:%S')  
    print("\n"+"=========="*8 + "%s"%nowtime)  
    print(info+'...\n\n')  
  

      

1，准备数据

        
          
from sklearn.preprocessing import LabelEncoder,QuantileTransformer  
from sklearn.pipeline import Pipeline   
from sklearn.impute import SimpleImputer   
  
dfdata = pd.read_csv("../input/criteo-small/train\_1m.txt",sep="\t",header=None)  
dfdata.columns = ["label"] + ["I"+str(x) for x in range(1,14)] + [  
    "C"+str(x) for x in range(14,40)]  
  
cat_cols = [x for x in dfdata.columns if x.startswith('C')]  
num_cols = [x for x in dfdata.columns if x.startswith('I')]  
num_pipe = Pipeline(steps = [('impute',SimpleImputer()),('quantile',QuantileTransformer())])  
  
for col in cat_cols:  
    dfdata[col]  = LabelEncoder().fit_transform(dfdata[col])  
  
dfdata[num_cols] = num_pipe.fit_transform(dfdata[num_cols])  
  
categories = [dfdata[col].max()+1 for col in cat_cols]  
  

      

        
          
import torch   
from torch.utils.data import Dataset,DataLoader   
  
#DataFrame转换成torch数据集Dataset, 特征分割成X\_num,X\_cat方式  
class DfDataset(Dataset):  
    def \_\_init\_\_(self,df,  
                 label\_col,  
                 num\_features,  
                 cat\_features,  
                 categories,  
                 is\_training=True):  
          
        self.X_num = torch.tensor(df[num_features].values).float() if num_features else None  
        self.X_cat = torch.tensor(df[cat_features].values).long() if cat_features else None  
        self.Y = torch.tensor(df[label_col].values).float()   
        self.categories = categories  
        self.is_training = is_training  
      
    def \_\_len\_\_(self):  
        return len(self.Y)  
      
    def \_\_getitem\_\_(self,index):  
        if self.is_training:  
            return ((self.X_num[index],self.X_cat[index]),self.Y[index])  
        else:  
            return (self.X_num[index],self.X_cat[index])  
      
    def get\_categories(self):  
        return self.categories  
      

      

        
          
dftrain_val,dftest = train_test_split(dfdata,test_size=0.2)  
dftrain,dfval = train_test_split(dftrain_val,test_size=0.2)  
  
ds_train = DfDataset(dftrain,label_col = "label",num_features = num_cols,cat_features = cat_cols,  
                    categories = categories, is_training=True)  
  
ds_val = DfDataset(dfval,label_col = "label",num_features = num_cols,cat_features = cat_cols,  
                    categories = categories, is_training=True)  
  
ds_test = DfDataset(dftest,label_col = "label",num_features = num_cols,cat_features = cat_cols,  
                    categories = categories, is_training=True)  
  

      

        
          
dl_train = DataLoader(ds_train,batch_size = 2048,shuffle=True)  
dl_val = DataLoader(ds_val,batch_size = 2048,shuffle=False)  
dl_test = DataLoader(ds_test,batch_size = 2048,shuffle=False)  
  
for features,labels in dl_train:  
    break   
      

      

2，定义模型

        
          
def create\_net():  
    net = FiBiNET(  
        d_numerical= ds_train.X_num.shape[1],  
        categories= ds_train.get_categories(),  
        d_embed = 8, mlp_layers = [128,64,32], mlp_dropout=0.25,  
        reduction_ratio = 3,  
        bilinear_type = "field\_all",  
        n_classes = 1  
          
    )  
    return net   
  
from torchkeras import summary  
  
net = create_net()  
  
  

      

3，训练模型

        
          
import os,sys,time  
import numpy as np  
import pandas as pd  
import datetime   
from tqdm import tqdm   
  
import torch  
from torch import nn   
from accelerate import Accelerator  
from copy import deepcopy  
  
  
def printlog(info):  
    nowtime = datetime.datetime.now().strftime('%Y-%m-%d %H:%M:%S')  
    print("\n"+"=========="*8 + "%s"%nowtime)  
    print(str(info)+"\n")  
      
class StepRunner:  
    def \_\_init\_\_(self, net, loss\_fn,stage = "train", metrics\_dict = None,   
                 optimizer = None, lr\_scheduler = None,  
                 accelerator = None  
                 ):  
        self.net,self.loss_fn,self.metrics_dict,self.stage = net,loss_fn,metrics_dict,stage  
        self.optimizer,self.lr_scheduler = optimizer,lr_scheduler  
        self.accelerator = accelerator  
      
    def \_\_call\_\_(self, features, labels):  
        #loss  
        preds = self.net(features)  
        loss = self.loss_fn(preds,labels)  
  
        #backward()  
        if self.optimizer is not None and self.stage=="train":  
            if self.accelerator is  None:  
                loss.backward()  
            else:  
                self.accelerator.backward(loss)  
            self.optimizer.step()  
            if self.lr_scheduler is not None:  
                self.lr_scheduler.step()  
            self.optimizer.zero_grad()  
              
        #metrics  
        step_metrics = {self.stage+"\_"+name:metric_fn(preds, labels).item()   
                        for name,metric_fn in self.metrics_dict.items()}  
        return loss.item(),step_metrics  
      
      
class EpochRunner:  
    def \_\_init\_\_(self,steprunner):  
        self.steprunner = steprunner  
        self.stage = steprunner.stage  
        self.steprunner.net.train() if self.stage=="train" else self.steprunner.net.eval()  
          
    def \_\_call\_\_(self,dataloader):  
        total_loss,step = 0,0  
        loop = tqdm(enumerate(dataloader), total =len(dataloader))  
        for i, batch in loop:  
            features,labels = batch  
            if self.stage=="train":  
                loss, step_metrics = self.steprunner(features,labels)  
            else:  
                with torch.no_grad():  
                    loss, step_metrics = self.steprunner(features,labels)  
  
            step_log = dict({self.stage+"\_loss":loss},**step_metrics)  
  
            total_loss += loss  
            step+=1  
            if i!=len(dataloader)-1:  
                loop.set_postfix(**step_log)  
            else:  
                epoch_loss = total_loss/step  
                epoch_metrics = {self.stage+"\_"+name:metric_fn.compute().item()   
                                 for name,metric_fn in self.steprunner.metrics_dict.items()}  
                epoch_log = dict({self.stage+"\_loss":epoch_loss},**epoch_metrics)  
                loop.set_postfix(**epoch_log)  
  
                for name,metric_fn in self.steprunner.metrics_dict.items():  
                    metric_fn.reset()  
        return epoch_log  
  
class KerasModel(torch.nn.Module):  
    def \_\_init\_\_(self,net,loss\_fn,metrics\_dict=None,optimizer=None,lr\_scheduler = None):  
        super().__init__()  
        self.accelerator = Accelerator()  
        self.history = {}  
          
        self.net = net  
        self.loss_fn = loss_fn  
        self.metrics_dict = nn.ModuleDict(metrics_dict)   
          
        self.optimizer = optimizer if optimizer is not None else torch.optim.Adam(  
            self.parameters(), lr=1e-2)  
        self.lr_scheduler = lr_scheduler  
          
        self.net,self.loss_fn,self.metrics_dict,self.optimizer = self.accelerator.prepare(  
            self.net,self.loss_fn,self.metrics_dict,self.optimizer)  
  
    def forward(self, x):  
        if self.net:  
            return self.net.forward(x)  
        else:  
            raise NotImplementedError  
  
  
    def fit(self, train\_data, val\_data=None, epochs=10, ckpt\_path='checkpoint.pt',   
            patience=5, monitor="val\_loss", mode="min"):  
          
        train_data = self.accelerator.prepare(train_data)  
        val_data = self.accelerator.prepare(val_data) if val_data else []  
  
        for epoch in range(1, epochs+1):  
            printlog("Epoch {0} / {1}".format(epoch, epochs))  
              
            # 1，train -------------------------------------------------    
            train_step_runner = StepRunner(net = self.net,stage="train",  
                    loss_fn = self.loss_fn,metrics_dict=deepcopy(self.metrics_dict),  
                    optimizer = self.optimizer, lr_scheduler = self.lr_scheduler,  
                    accelerator = self.accelerator)  
            train_epoch_runner = EpochRunner(train_step_runner)  
            train_metrics = train_epoch_runner(train_data)  
              
            for name, metric in train_metrics.items():  
                self.history[name] = self.history.get(name, []) + [metric]  
  
            # 2，validate -------------------------------------------------  
            if val_data:  
                val_step_runner = StepRunner(net = self.net,stage="val",  
                    loss_fn = self.loss_fn,metrics_dict=deepcopy(self.metrics_dict),  
                    accelerator = self.accelerator)  
                val_epoch_runner = EpochRunner(val_step_runner)  
                with torch.no_grad():  
                    val_metrics = val_epoch_runner(val_data)  
                val_metrics["epoch"] = epoch  
                for name, metric in val_metrics.items():  
                    self.history[name] = self.history.get(name, []) + [metric]  
              
            # 3，early-stopping -------------------------------------------------  
            arr_scores = self.history[monitor]  
            best_score_idx = np.argmax(arr_scores) if mode=="max" else np.argmin(arr_scores)  
            if best_score_idx==len(arr_scores)-1:  
                torch.save(self.net.state_dict(),ckpt_path)  
                print("<<<<<< reach best {0} : {1} >>>>>>".format(monitor,  
                     arr_scores[best_score_idx]),file=sys.stderr)  
            if len(arr_scores)-best_score_idx>patience:  
                print("<<<<<< {} without improvement in {} epoch, early stopping >>>>>>".format(  
                    monitor,patience),file=sys.stderr)  
                self.net.load_state_dict(torch.load(ckpt_path))  
                break   
              
        return pd.DataFrame(self.history)  
  
    @torch.no\_grad()  
    def evaluate(self, val\_data):  
        val_data = self.accelerator.prepare(val_data)  
        val_step_runner = StepRunner(net = self.net,stage="val",  
                    loss_fn = self.loss_fn,metrics_dict=deepcopy(self.metrics_dict),  
                    accelerator = self.accelerator)  
        val_epoch_runner = EpochRunner(val_step_runner)  
        val_metrics = val_epoch_runner(val_data)  
        return val_metrics  
          
         
    @torch.no\_grad()  
    def predict(self, dataloader):  
        dataloader = self.accelerator.prepare(dataloader)  
        result = torch.cat([self.forward(t[0]) for t in dataloader])  
        return result.data  
                

      

        
          
from torchkeras.metrics import AUC  
  
loss_fn = nn.BCEWithLogitsLoss()  
  
metrics_dict = {"auc":AUC()}  
  
optimizer = torch.optim.Adam(net.parameters(), lr=0.002, weight_decay=0.001)   
  
model = KerasModel(net,  
                   loss_fn = loss_fn,  
                   metrics_dict= metrics_dict,  
                   optimizer = optimizer  
                  )           

      

        
          
dfhistory = model.fit(train_data=dl_train,val_data=dl_val,epochs=100, patience=5,  
                      monitor = "val\_auc",mode="max",ckpt_path='checkpoint.pt')  

      

4，评估模型

        
          
%matplotlib inline  
%config InlineBackend.figure_format = 'svg'  
  
import matplotlib.pyplot as plt  
  
def plot\_metric(dfhistory, metric):  
    train_metrics = dfhistory["train\_"+metric]  
    val_metrics = dfhistory['val\_'+metric]  
    epochs = range(1, len(train_metrics) + 1)  
    plt.plot(epochs, train_metrics, 'bo--')  
    plt.plot(epochs, val_metrics, 'ro-')  
    plt.title('Training and validation '+ metric)  
    plt.xlabel("Epochs")  
    plt.ylabel(metric)  
    plt.legend(["train\_"+metric, 'val\_'+metric])  
    plt.show()  

      

        
          
plot_metric(dfhistory,"loss")  

      

        
          
plot_metric(dfhistory,"auc")  

      

5，使用模型

        
          
from sklearn.metrics import roc_auc_score  
preds = torch.sigmoid(model.predict(dl_val))  
labels = torch.cat([x[-1] for x in dl_val])  
  
val_auc = roc_auc_score(labels.cpu().numpy(),preds.cpu().numpy())  
print(val_auc)  

      

0.7806176567186112

6，保存模型

        
          
torch.save(model.net.state_dict(),"best\_fibinet.pt")  
net_clone = create_net()  
net_clone.load_state_dict(torch.load("best\_fibinet.pt"))  

      

        
          
from sklearn.metrics import roc_auc_score  
net_clone.eval()  
preds = torch.cat([torch.sigmoid(net_clone(x[0])).data for x in dl_val])   
labels = torch.cat([x[-1] for x in dl_val])  
  
val_auc = roc_auc_score(labels.cpu().numpy(),preds.cpu().numpy())  
print(val_auc)  

      

0.7806176567186112

可以看到FiBiNET在验证集的AUC得分为0.7806,相比之下DeepFM的验证集AUC为0.7803。

不能说纹丝不动, 只能说了涨了个蚊子腿大小肉的点。

并且这是以较大地牺牲模型训练预测效率为代价的。

DeepFM训练一个Epoch大约需要20s, 而FiBiNET训练一个Epoch需要大约2min.

尽管如此, FiBiNET的结构设计依然是值得我们学习和借鉴的, 集神经网络结构设计三大主流高级技巧于一体, 闪烁着穿越时空的才华与智慧光芒。

以上。

万水千山总是情，点个在看行不行？😋😋

公众号后台回复关键词: FiBiNET 获取本文全部源码和数据集下载链接。