Skip to content

Ranking Models ​

Ranking models are core components in recommendation systems, used to predict users' click-through rates or preference scores for items, thereby performing fine-grained ranking on retrieved candidates. Torch-RecHub provides various advanced ranking models covering different feature processing and modeling approaches.

1. WideDeep ​

Description ​

WideDeep is a hybrid model combining a linear model (Wide part) and a deep neural network (Deep part), designed to leverage both the memorization capability of linear models and the generalization capability of deep models.

Paper Reference ​

Cheng, Heng-Tze, et al. "Wide & deep learning for recommender systems." Proceedings of the 1st workshop on deep learning for recommender systems. 2016.

Core Principles ​

  • Wide Part: Linear model using cross features, good at capturing memorization effects
  • Deep Part: Deep neural network using Embedding and fully connected layers, good at capturing generalization effects
  • Joint Training: Wide and Deep parts are trained simultaneously, outputs combined through sigmoid function

Usage ​

python
from torch_rechub.models.ranking import WideDeep

dense_features = [DenseFeature(name="age", embed_dim=1), DenseFeature(name="income", embed_dim=1)]
sparse_features = [SparseFeature(name="city", vocab_size=100, embed_dim=16), SparseFeature(name="gender", vocab_size=3, embed_dim=8)]

model = WideDeep(
    wide_features=sparse_features,
    deep_features=sparse_features + dense_features,
    mlp_params={"dims": [256, 128, 64], "dropout": 0.2, "activation": "relu"}
)

Parameters ​

ParameterTypeDescriptionDefault
wide_featureslistFeature list for Wide partNone
deep_featureslistFeature list for Deep partNone
mlp_paramsdictDNN parameters including dims, dropout, activationNone

Use Cases ​

  • Basic ranking tasks
  • Scenarios requiring both memorization and generalization
  • Limited feature engineering resources

2. DeepFM ​

Description ​

DeepFM combines Factorization Machine (FM) and deep neural network, capable of capturing both low-order and high-order feature interactions.

Paper Reference ​

Guo, Huifeng, et al. "DeepFM: a factorization-machine based neural network for CTR prediction." Proceedings of the 26th international joint conference on artificial intelligence. 2017.

Core Principles ​

  • FM Part: Captures second-order feature interactions with linear complexity
  • Deep Part: Captures high-order feature interactions through neural network
  • Shared Embedding: FM and Deep parts share feature embeddings, reducing parameters

Usage ​

python
import os

from torch_rechub.models.ranking import DeepFM

model = DeepFM(
    deep_features=sparse_features + dense_features,
    fm_features=sparse_features,
    mlp_params={"dims": [256, 128, 64], "dropout": 0.2, "activation": "relu"}
)

Parameters ​

ParameterTypeDescriptionDefault
deep_featureslistFeature list for Deep partNone
fm_featureslistFeature list for FM partNone
mlp_paramsdictDNN parametersNone

Use Cases ​

  • Scenarios where feature interactions are important
  • Need to capture both low-order and high-order feature interactions
  • CTR prediction tasks

3. DCN ​

Description ​

DCN (Deep & Cross Network) explicitly learns feature crosses through a Cross Network while maintaining linear computational complexity.

Paper Reference ​

Wang, Ruoxi, et al. "Deep & cross network for ad click predictions." Proceedings of the ADKDD'17. 2017.

Core Principles ​

  • Cross Network: Explicitly learns high-order feature crosses, each layer output:xl+1=x0xlTwl+bl+xl
  • Deep Network: Deep neural network capturing nonlinear feature interactions
  • Joint Training: Cross and Deep networks compute in parallel, results concatenated for final output

Usage ​

python
from torch_rechub.models.ranking import DCN

model = DCN(
    features=sparse_features + dense_features,
    n_cross_layers=3,
    mlp_params={"dims": [256, 128, 64], "dropout": 0.2, "activation": "relu"},
)

Parameters ​

ParameterTypeDescriptionDefault
featureslistFeatures shared by the Cross and Deep branchesrequired
mlp_paramsdictDNN parametersNone
n_cross_layersintNumber of Cross Network layersrequired

Use Cases ​

  • Scenarios requiring explicit feature crosses
  • Limited computational resources
  • CTR prediction tasks

4. DCNv2 ​

Description ​

DCNv2 extends DCN's scalar/vector cross parameters to matrix interactions. This implementation can also use a mixture of low-rank experts to reduce the cost of a full matrix interaction.

Paper Reference ​

Wang, Ruoxi, et al. "DCN V2: Improved deep & cross network and practical lessons for web-scale learning to rank systems." Proceedings of the web conference 2021. 2021.

Core Principles ​

  • Matrix-form crossing: More expressive than DCN's vector-form crossing
  • Low-rank expert mixture: Uses multiple low-rank Cross experts when use_low_rank_mixture=True
  • Selectable structure: Supports crossnet_only, stacked, and parallel

Usage ​

python
from torch_rechub.models.ranking import DCNv2

model = DCNv2(
    features=sparse_features + dense_features,
    n_cross_layers=3,
    mlp_params={"dims": [256, 128, 64], "dropout": 0.2, "activation": "relu"},
    model_structure="parallel",       # crossnet_only / stacked / parallel
    use_low_rank_mixture=True,
    low_rank=32,
    num_experts=4,
)

Parameters ​

ParameterTypeDescriptionDefault
featureslistFeatures shared by the Cross and Deep branchesrequired
mlp_paramsdictDNN parametersNone
n_cross_layersintNumber of Cross Network layersrequired
model_structurestrcrossnet_only / stacked / parallelparallel
use_low_rank_mixtureboolWhether to use a mixture of low-rank CrossNet expertsTrue
low_rankintLow-rank dimension32
num_expertsintNumber of CrossNetMix experts4

Use Cases ​

  • Scenarios requiring more efficient feature crosses
  • Large-scale recommendation systems
  • CTR prediction tasks

5. EDCN ​

Description ​

EDCN (Enhanced Deep & Cross Network) is an enhanced cross network model that combines explicit feature crosses with deep feature extraction for improved expressiveness.

Paper Reference ​

Ma, Xiao, et al. "Enhanced Deep & Cross Network for Feature Cross Learning in Click-Through Rate Prediction." Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining. 2021.

Core Principles ​

  • Cross Network: Explicitly learns high-order feature crosses
  • Deep Network: Deep neural network capturing nonlinear feature interactions
  • Feature Importance Learning: Introduces feature importance weights for better interpretability

Usage ​

python
from torch_rechub.models.ranking import EDCN

model = EDCN(
    features=sparse_features + dense_features,
    n_cross_layers=3,
    mlp_params={"dims": [256, 128, 64], "dropout": 0.2, "activation": "relu"},
    bridge_type="hadamard_product",
    use_regulation_module=True,
)

Parameters ​

ParameterTypeDescriptionDefault
featureslistAll features used by the modelrequired
mlp_paramsdictDNN parametersNone
n_cross_layersintNumber of Cross Network layersrequired
bridge_typestrBridge between the Cross and Deep streamshadamard_product
use_regulation_moduleboolWhether to enable the feature regulation moduleTrue

Use Cases ​

  • Complex feature interaction scenarios
  • Models requiring high expressiveness
  • CTR prediction tasks

6. AFM ​

Description ​

AFM (Attention Factorization Machine) is an attention-based factorization machine that adaptively learns the importance of different feature interactions.

Paper Reference ​

Xiao, Jun, et al. "Attentional factorization machines: Learning the weight of feature interactions via attention networks." arXiv preprint arXiv:1708.04617 (2017).

Core Principles ​

  • FM Foundation: Based on factorization machine, captures second-order feature interactions
  • Attention Mechanism: Introduces attention network to assign dynamic weights to each feature interaction
  • Attention Output: Weighted sum of attention weights and feature interaction vectors

Usage ​

python
from torch_rechub.models.ranking import AFM

model = AFM(
    fm_features=sparse_features,
    embed_dim=16,  # must match the embedding width of fm_features
    t=64,
)

Parameters ​

ParameterTypeDescriptionDefault
fm_featureslistSparse features used for FM interactions; all embedding widths must matchrequired
embed_dimintFeature embedding dimensionrequired
tintAttention hidden dimension64

Use Cases ​

  • Scenarios with varying feature interaction importance
  • Need for interpretability
  • CTR prediction tasks

7. FiBiNET ​

Description ​

FiBiNET (Feature Importance and Bilinear feature Interaction NETwork) combines feature importance learning with bilinear feature interactions for more effective feature interaction capture.

Paper Reference ​

Juan, Yuchin, et al. "FiBiNET: Combining Feature Importance and Bilinear feature Interaction for Click-Through Rate Prediction." Proceedings of the 13th ACM Conference on Recommender Systems. 2019.

Core Principles ​

  • Feature Importance Network: Learns feature importance through Squeeze-and-Excitation mechanism
  • Bilinear Interaction: Uses bilinear functions to capture feature interactions
  • Feature Enhancement: Enhances input features for improved expressiveness

Usage ​

python
from torch_rechub.models.ranking import FiBiNet

model = FiBiNet(
    features=sparse_features,
    mlp_params={"dims": [256, 128, 64], "dropout": 0.2, "activation": "relu"},
    reduction_ratio=3,
    bilinear_type="field_interaction",
)

Parameters ​

ParameterTypeDescriptionDefault
featureslistFeatures used by SENET and bilinear interactionrequired
mlp_paramsdictPrediction MLP parametersrequired
reduction_ratiointSENET reduction ratio3
bilinear_typestrfield_all / field_each / field_interactionfield_interaction

Use Cases ​

  • Scenarios with varying feature importance
  • Need for complex feature interactions
  • CTR prediction tasks

8. DeepFFM ​

Description ​

DeepFFM (Deep Field-aware Factorization Machine) combines field-aware factorization machine with deep neural network, capturing field-aware high-order feature interactions.

Paper Reference ​

Xiao, Jun, et al. "Deep learning over multi-field categorical data." European conference on information retrieval. Springer, Cham, 2016.

Core Principles ​

  • FFM Foundation: Field-aware factorization machine, learns specific interaction vectors for each field pair
  • Deep Network: Deep neural network capturing high-order feature interactions
  • Joint Training: FFM and Deep parts trained jointly

Usage ​

python
from torch_rechub.models.ranking import DeepFFM, FatDeepFFM

# FFM uses a different embedding for every field pair, so cross-feature
# vocabularies must reserve one offset range per field.
ffm_linear_features = [
    SparseFeature(f.name, vocab_size=f.vocab_size, embed_dim=1) for f in sparse_features
]
ffm_cross_features = [
    SparseFeature(f.name, vocab_size=f.vocab_size * len(sparse_features), embed_dim=10)
    for f in sparse_features
]

model = DeepFFM(
    linear_features=ffm_linear_features,
    cross_features=ffm_cross_features,
    embed_dim=10,
    mlp_params={"dims": [1600, 1600], "dropout": 0.5, "activation": "relu"},
)

# FatDeepFFM (enhanced version)
fat_model = FatDeepFFM(
    linear_features=ffm_linear_features,
    cross_features=ffm_cross_features,
    embed_dim=10,
    reduction_ratio=1,
    mlp_params={"dims": [1600, 1600], "dropout": 0.5, "activation": "relu"},
)

Parameters ​

ParameterTypeDescriptionDefault
linear_featureslistFirst-order linear features, normally with embed_dim=1required
cross_featureslistFFM interaction features whose vocabularies reserve field-offset spacerequired
embed_dimintFFM embedding dimensionrequired
reduction_ratiointCEN reduction ratio, required only by FatDeepFFMrequired (FatDeepFFM)
mlp_paramsdictMLP parameters after FFM interactionsrequired

Use Cases ​

  • Scenarios where field-aware feature interactions are important
  • Complex feature interaction scenarios
  • CTR prediction tasks

9. BST ​

Description ​

BST (Behavior Sequence Transformer) uses Transformer to model user behavior sequences, capturing long-range dependencies in sequences.

Paper Reference ​

Chen, Qiwei, et al. "Behavior Sequence Transformer for E-commerce Recommendation in Alibaba." arXiv preprint arXiv:1905.06874 (2019).

Core Principles ​

  • Transformer Encoder: Uses multi-head self-attention to capture sequence dependencies
  • Positional Encoding: Adds positional information to preserve sequence order
  • Feature Fusion: Fuses sequence features with other features for final prediction

Usage ​

python
from torch_rechub.basic.features import SparseFeature, SequenceFeature
from torch_rechub.models.ranking import BST

features = [SparseFeature("user_id", vocab_size=n_users + 1, embed_dim=8)]
target_features = [
    SparseFeature("target_item_id", vocab_size=n_items + 1, embed_dim=8),
    SparseFeature("target_cate_id", vocab_size=n_cates + 1, embed_dim=8),
]
history_features = [
    SequenceFeature("hist_item_id", vocab_size=n_items + 1, embed_dim=8,
                    pooling="concat", shared_with="target_item_id"),
    SequenceFeature("hist_cate_id", vocab_size=n_cates + 1, embed_dim=8,
                    pooling="concat", shared_with="target_cate_id"),
]

model = BST(
    features=features,
    history_features=history_features,
    target_features=target_features,
    mlp_params={"dims": [256, 128]},
    nhead=8,
    dropout=0.2,
    num_layers=1,
    max_seq_len=51,
)

Parameters ​

ParameterTypeDescriptionDefault
featureslistUser-profile/context features, excluding history and targetrequired
history_featureslistHistory sequences with pooling="concat"required
target_featureslistTarget-item features corresponding to the history featuresrequired
mlp_paramsdictPrediction MLP parametersrequired
nheadintAttention heads; must divide the summed history embedding dimension8
dropoutfloatTransformer dropout0.2
num_layersintNumber of Transformer Encoder layers1
max_seq_lenintUpper bound of history length plus one target51

Use Cases ​

  • Scenarios where user behavior sequences are important
  • Long sequence modeling
  • Sequential recommendation tasks

10. DIN ​

Description ​

DIN (Deep Interest Network) is an attention-based deep interest network that dynamically captures user interests based on target items.

Paper Reference ​

Zhou, Guorui, et al. "Deep interest network for click-through rate prediction." Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining. 2018.

Core Principles ​

  • Interest Extraction: Extracts interest representations from user behavior sequences
  • Attention Mechanism: Computes attention weights for each historical behavior based on target item
  • Dynamic Interest Aggregation: Dynamically aggregates user interests based on attention weights

Usage ​

python
from torch_rechub.basic.features import SparseFeature, SequenceFeature
from torch_rechub.models.ranking import DIN

features = [SparseFeature("user_id", vocab_size=n_users + 1, embed_dim=8)]
target_features = [SparseFeature("target_item_id", vocab_size=n_items + 1, embed_dim=8)]
history_features = [
    SequenceFeature("hist_item_id", vocab_size=n_items + 1, embed_dim=8,
                    pooling="concat", shared_with="target_item_id")
]

model = DIN(
    features=features,
    history_features=history_features,
    target_features=target_features,
    mlp_params={"dims": [256, 128]},
    attention_mlp_params={"dims": [64, 32], "use_softmax": False},
)

Parameters ​

ParameterTypeDescriptionDefault
featureslistUser-profile/context featuresrequired
history_featureslistHistory sequences with pooling="concat"required
target_featureslistTarget features matching history count, order, and dimensionsrequired
mlp_paramsdictPrediction MLP parametersrequired
attention_mlp_paramsdictActivation Unit parametersrequired

Use Cases ​

  • Scenarios with dynamic user interests
  • Target item-related interest modeling
  • CTR prediction tasks

11. DIEN ​

Description ​

DIEN (Deep Interest Evolution Network), proposed by Alibaba at AAAI 2019, extends DIN with an Interest Extractor Layer (GRU + auxiliary loss) and an Interest Evolution Layer (AUGRU) to model how user interests evolve over time.

Paper Reference ​

Zhou, Guorui, et al. "Deep interest evolution network for click-through rate prediction." Proceedings of the AAAI conference on artificial intelligence. 2019.

Core Principles ​

  • Interest Extractor Layer: GRU over behaviour sequences; auxiliary loss supervises each hidden state with positive/negative next-step samples (paper Eq.7)
  • Interest Evolution Layer: AUGRU embeds attention into the update gate; attention is softmax-normalised over the full valid sequence (paper Eq.14-16)
  • Auxiliary Loss: Laux=−1N∑[log⁡σ(ht⋅et+1+)+log⁡(1−σ(ht⋅et+1−))]
  • Padding: index 0 is the padding token; padding positions are excluded from GRU, AUGRU attention, and auxiliary loss; all-padding samples keep zero hidden state

Usage ​

python
from torch_rechub.basic.features import SparseFeature, SequenceFeature
from torch_rechub.models.ranking import DIEN

# padding_idx=0 must be set on target_features — they own the embedding tables
target_features = [
    SparseFeature("target_item_id", vocab_size=n_items+1, embed_dim=8, padding_idx=0),
    SparseFeature("target_cate_id", vocab_size=n_cates+1, embed_dim=8, padding_idx=0),
]
history_features = [
    SequenceFeature("hist_item_id", vocab_size=n_items+1, embed_dim=8,
                    pooling="concat", shared_with="target_item_id", padding_idx=0),
    SequenceFeature("hist_cate_id", vocab_size=n_cates+1, embed_dim=8,
                    pooling="concat", shared_with="target_cate_id", padding_idx=0),
]
neg_history_features = [
    SequenceFeature("neg_hist_item_id", vocab_size=n_items+1, embed_dim=8,
                    pooling="concat", shared_with="target_item_id", padding_idx=0),
    SequenceFeature("neg_hist_cate_id", vocab_size=n_cates+1, embed_dim=8,
                    pooling="concat", shared_with="target_cate_id", padding_idx=0),
]

model = DIEN(
    features=features,
    history_features=history_features,
    neg_history_features=neg_history_features,
    target_features=target_features,
    mlp_params={"dims": [256, 128]},
    alpha=0.2,
)
# CTRTrainer must use loss_mode=False — forward returns (prediction, aux_loss)

Parameters ​

ParameterTypeDescription
featureslistUser profile / context features fed into the top MLP
history_featureslistPositive behaviour sequences; pooling="concat", padding_idx=0, shared_with=target_feature
neg_history_featureslistNegative-sampled sequences; same constraints as history_features; shared_with must point to target feature
target_featureslistTarget item features; padding_idx=0 so the shared embedding table's row 0 is a zero vector
mlp_paramsdictTop MLP params; activation is fixed to dice
alphafloatAuxiliary loss weight (default 0.2)

Use Cases ​

  • E-commerce / news recommendation where user interests evolve over time
  • Sequential behaviour data with temporal ordering
  • CTR prediction tasks

12. AutoInt ​

Description ​

AutoInt (Automatic Feature Interaction Learning via Self-Attentive Neural Networks) uses self-attention to automatically learn feature interactions, flexibly capturing various orders of feature interactions.

Paper Reference ​

Song, Weiping, et al. "AutoInt: Automatic Feature Interaction Learning via Self-Attentive Neural Networks." Proceedings of the 28th ACM International Conference on Information and Knowledge Management. 2019.

Core Principles ​

  • Embedding Layer: Maps discrete features to low-dimensional vector space
  • Multi-head Self-attention: Automatically learns interaction relationships between features
  • Residual Connection: Enhances training stability
  • Layer Normalization: Accelerates model convergence

Usage ​

python
from torch_rechub.models.ranking import AutoInt

model = AutoInt(
    sparse_features=sparse_features,
    dense_features=dense_features,
    num_layers=3,
    num_heads=2,
    dropout=0.2,
    mlp_params={"dims": [256, 128], "dropout": 0.2, "activation": "relu"},
)

Parameters ​

ParameterTypeDescriptionDefault
sparse_featureslistAt least one sparse feature; all must use the same embed_dimrequired
dense_featureslistDense features; may be an empty listrequired
num_layersintNumber of Interacting Layers3
num_headsintNumber of attention heads2
dropoutfloatAttention dropout0.0
mlp_paramsdict or NoneOptional Deep-branch parametersNone

Use Cases ​

  • Automatic feature interaction learning
  • Complex feature interaction scenarios
  • CTR prediction tasks

13. Model Comparison ​

ModelComplexityExpressivenessEfficiencyInterpretability
WideDeepLowMediumHighHigh
DeepFMMediumHighMediumMedium
DCN/DCNv2MediumHighHighMedium
EDCNMediumHighMediumMedium
AFMMediumMediumMediumHigh
FiBiNETMediumHighMediumMedium
DeepFFMHighHighLowMedium
BSTHighHighLowMedium
DINMediumHighMediumMedium
DIENHighHighLowMedium
AutoIntHighHighLowMedium

14. Usage Recommendations ​

  1. Choose based on data scale: For small-scale data, use simple models (WideDeep, DeepFM); for large-scale data, try more complex models
  2. Choose based on feature types: For important sequence features, use BST, DIN, DIEN; for important feature interactions, use DCN, DeepFM
  3. Choose based on computational resources: For limited resources, use efficient models (DCN, WideDeep)
  4. Try multiple models and ensemble: Different models may capture different feature interaction patterns; ensembling can improve results

15. Complete Training Example ​

python
from torch_rechub.models.ranking import DeepFM
from torch_rechub.trainers import CTRTrainer
from torch_rechub.utils.data import DataGenerator
from torch_rechub.basic.features import DenseFeature, SparseFeature

# 1. Define features
dense_features = [
    DenseFeature(name="age", embed_dim=1),
    DenseFeature(name="income", embed_dim=1)
]

sparse_features = [
    SparseFeature(name="city", vocab_size=100, embed_dim=16),
    SparseFeature(name="gender", vocab_size=3, embed_dim=16),
    SparseFeature(name="occupation", vocab_size=20, embed_dim=16)
]

# 2. Prepare data
x = {
    "age": age_data,
    "income": income_data,
    "city": city_data,
    "gender": gender_data,
    "occupation": occupation_data
}
y = label_data

# 3. Create data generator
dg = DataGenerator(x, y)
train_dl, val_dl, test_dl = dg.generate_dataloader(split_ratio=[0.7, 0.1], batch_size=256)

# 4. Create model
model = DeepFM(
    deep_features=sparse_features + dense_features,
    fm_features=sparse_features,
    mlp_params={"dims": [256, 128, 64], "dropout": 0.2, "activation": "relu"}
)

# 5. Create trainer
trainer = CTRTrainer(
    model=model,
    optimizer_params={"lr": 0.001, "weight_decay": 0.0001},
    n_epoch=50,
    earlystop_patience=10,
    device="cpu",
    model_path="saved/deepfm"
)

# 6. Train model
os.makedirs("saved/deepfm", exist_ok=True)
trainer.fit(train_dl, val_dl)

# 7. Evaluate model
auc = trainer.evaluate(trainer.model, test_dl)
print(f"Test AUC: {auc}")

# 8. Export ONNX model (first install: pip install "torch-rechub[onnx]")
trainer.export_onnx("deepfm.onnx")

16. FAQ ​

Q: How to choose the right model? ​

A: Choose based on data scale, feature types, computational resources, and business requirements. Start with simple models and gradually try more complex ones.

Q: What to do about overfitting? ​

A: Try the following:

  • Add regularization (L1/L2)
  • Increase dropout rate
  • Use early stopping
  • Add more training data
  • Simplify model structure

Q: How to handle large-scale features? ​

A: Try the following:

  • Feature selection: Keep only important features
  • Feature hashing: Map high-dimensional features to low-dimensional space
  • Hierarchical embedding: Use different embedding dimensions for different features

Q: How to speed up training? ​

A: CTRTrainer can use one GPU through device, or wrap the model in single-machine torch.nn.DataParallel through gpus=[...]. It does not provide automatic mixed precision (AMP) or multi-machine distributed training. Whether batch size can be increased depends on available memory; test it incrementally.