Skip to content

Ranking Models

Ranking models are core components in recommendation systems, used to predict users' click-through rates or preference scores for items, thereby performing fine-grained ranking on retrieved candidates. Torch-RecHub provides various advanced ranking models covering different feature processing and modeling approaches.

1. WideDeep

Description

WideDeep is a hybrid model combining a linear model (Wide part) and a deep neural network (Deep part), designed to leverage both the memorization capability of linear models and the generalization capability of deep models.

Paper Reference

Cheng, Heng-Tze, et al. "Wide & deep learning for recommender systems." Proceedings of the 1st workshop on deep learning for recommender systems. 2016.

Core Principles

  • Wide Part: Linear model using cross features, good at capturing memorization effects
  • Deep Part: Deep neural network using Embedding and fully connected layers, good at capturing generalization effects
  • Joint Training: Wide and Deep parts are trained simultaneously, outputs combined through sigmoid function

Usage

python
from torch_rechub.models.ranking import WideDeep

dense_features = [DenseFeature(name="age", embed_dim=1), DenseFeature(name="income", embed_dim=1)]
sparse_features = [SparseFeature(name="city", vocab_size=100, embed_dim=16), SparseFeature(name="gender", vocab_size=3, embed_dim=8)]

model = WideDeep(
    wide_features=sparse_features,
    deep_features=sparse_features + dense_features,
    mlp_params={"dims": [256, 128, 64], "dropout": 0.2, "activation": "relu"}
)

Parameters

ParameterTypeDescriptionDefault
wide_featureslistFeature list for Wide partNone
deep_featureslistFeature list for Deep partNone
mlp_paramsdictDNN parameters including dims, dropout, activationNone

Use Cases

  • Basic ranking tasks
  • Scenarios requiring both memorization and generalization
  • Limited feature engineering resources

2. DeepFM

Description

DeepFM combines Factorization Machine (FM) and deep neural network, capable of capturing both low-order and high-order feature interactions.

Paper Reference

Guo, Huifeng, et al. "DeepFM: a factorization-machine based neural network for CTR prediction." Proceedings of the 26th international joint conference on artificial intelligence. 2017.

Core Principles

  • FM Part: Captures second-order feature interactions with linear complexity
  • Deep Part: Captures high-order feature interactions through neural network
  • Shared Embedding: FM and Deep parts share feature embeddings, reducing parameters

Usage

python
import os

from torch_rechub.models.ranking import DeepFM

model = DeepFM(
    deep_features=sparse_features + dense_features,
    fm_features=sparse_features,
    mlp_params={"dims": [256, 128, 64], "dropout": 0.2, "activation": "relu"}
)

Parameters

ParameterTypeDescriptionDefault
deep_featureslistFeature list for Deep partNone
fm_featureslistFeature list for FM partNone
mlp_paramsdictDNN parametersNone

Use Cases

  • Scenarios where feature interactions are important
  • Need to capture both low-order and high-order feature interactions
  • CTR prediction tasks

3. DCN

Description

DCN (Deep & Cross Network) explicitly learns feature crosses through a Cross Network while maintaining linear computational complexity.

Paper Reference

Wang, Ruoxi, et al. "Deep & cross network for ad click predictions." Proceedings of the ADKDD'17. 2017.

Core Principles

  • Cross Network: Explicitly learns high-order feature crosses, each layer output:xl+1=x0xlTwl+bl+xl
  • Deep Network: Deep neural network capturing nonlinear feature interactions
  • Joint Training: Cross and Deep networks compute in parallel, results concatenated for final output

Usage

python
from torch_rechub.models.ranking import DCN

model = DCN(
    features=sparse_features + dense_features,
    n_cross_layers=3,
    mlp_params={"dims": [256, 128, 64], "dropout": 0.2, "activation": "relu"},
)

Parameters

ParameterTypeDescriptionDefault
featureslistFeatures shared by the Cross and Deep branchesrequired
mlp_paramsdictDNN parametersNone
n_cross_layersintNumber of Cross Network layersrequired

Use Cases

  • Scenarios requiring explicit feature crosses
  • Limited computational resources
  • CTR prediction tasks

4. DCNv2

Description

DCNv2 extends DCN's scalar/vector cross parameters to matrix interactions. This implementation can also use a mixture of low-rank experts to reduce the cost of a full matrix interaction.

Paper Reference

Wang, Ruoxi, et al. "DCN V2: Improved deep & cross network and practical lessons for web-scale learning to rank systems." Proceedings of the web conference 2021. 2021.

Core Principles

  • Matrix-form crossing: More expressive than DCN's vector-form crossing
  • Low-rank expert mixture: Uses multiple low-rank Cross experts when use_low_rank_mixture=True
  • Selectable structure: Supports crossnet_only, stacked, and parallel

Usage

python
from torch_rechub.models.ranking import DCNv2

model = DCNv2(
    features=sparse_features + dense_features,
    n_cross_layers=3,
    mlp_params={"dims": [256, 128, 64], "dropout": 0.2, "activation": "relu"},
    model_structure="parallel",       # crossnet_only / stacked / parallel
    use_low_rank_mixture=True,
    low_rank=32,
    num_experts=4,
)

Parameters

ParameterTypeDescriptionDefault
featureslistFeatures shared by the Cross and Deep branchesrequired
mlp_paramsdictDNN parametersNone
n_cross_layersintNumber of Cross Network layersrequired
model_structurestrcrossnet_only / stacked / parallelparallel
use_low_rank_mixtureboolWhether to use a mixture of low-rank CrossNet expertsTrue
low_rankintLow-rank dimension32
num_expertsintNumber of CrossNetMix experts4

Use Cases

  • Scenarios requiring more efficient feature crosses
  • Large-scale recommendation systems
  • CTR prediction tasks

5. EDCN

Description

EDCN (Enhanced Deep & Cross Network) is an enhanced cross network model that combines explicit feature crosses with deep feature extraction for improved expressiveness.

Paper Reference

Ma, Xiao, et al. "Enhanced Deep & Cross Network for Feature Cross Learning in Click-Through Rate Prediction." Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining. 2021.

Core Principles

  • Cross Network: Explicitly learns high-order feature crosses
  • Deep Network: Deep neural network capturing nonlinear feature interactions
  • Feature Importance Learning: Introduces feature importance weights for better interpretability

Usage

python
from torch_rechub.models.ranking import EDCN

model = EDCN(
    features=sparse_features + dense_features,
    n_cross_layers=3,
    mlp_params={"dims": [256, 128, 64], "dropout": 0.2, "activation": "relu"},
    bridge_type="hadamard_product",
    use_regulation_module=True,
)

Parameters

ParameterTypeDescriptionDefault
featureslistAll features used by the modelrequired
mlp_paramsdictDNN parametersNone
n_cross_layersintNumber of Cross Network layersrequired
bridge_typestrBridge between the Cross and Deep streamshadamard_product
use_regulation_moduleboolWhether to enable the feature regulation moduleTrue

Use Cases

  • Complex feature interaction scenarios
  • Models requiring high expressiveness
  • CTR prediction tasks

6. AFM

Description

AFM (Attention Factorization Machine) is an attention-based factorization machine that adaptively learns the importance of different feature interactions.

Paper Reference

Xiao, Jun, et al. "Attentional factorization machines: Learning the weight of feature interactions via attention networks." arXiv preprint arXiv:1708.04617 (2017).

Core Principles

  • FM Foundation: Based on factorization machine, captures second-order feature interactions
  • Attention Mechanism: Introduces attention network to assign dynamic weights to each feature interaction
  • Attention Output: Weighted sum of attention weights and feature interaction vectors

Usage

python
from torch_rechub.models.ranking import AFM

model = AFM(
    fm_features=sparse_features,
    embed_dim=16,  # must match the embedding width of fm_features
    t=64,
)

Parameters

ParameterTypeDescriptionDefault
fm_featureslistSparse features used for FM interactions; all embedding widths must matchrequired
embed_dimintFeature embedding dimensionrequired
tintAttention hidden dimension64

Use Cases

  • Scenarios with varying feature interaction importance
  • Need for interpretability
  • CTR prediction tasks

7. FiBiNET

Description

FiBiNET (Feature Importance and Bilinear feature Interaction NETwork) combines feature importance learning with bilinear feature interactions for more effective feature interaction capture.

Paper Reference

Juan, Yuchin, et al. "FiBiNET: Combining Feature Importance and Bilinear feature Interaction for Click-Through Rate Prediction." Proceedings of the 13th ACM Conference on Recommender Systems. 2019.

Core Principles

  • Feature Importance Network: Learns feature importance through Squeeze-and-Excitation mechanism
  • Bilinear Interaction: Uses bilinear functions to capture feature interactions
  • Feature Enhancement: Enhances input features for improved expressiveness

Usage

python
from torch_rechub.models.ranking import FiBiNet

model = FiBiNet(
    features=sparse_features,
    mlp_params={"dims": [256, 128, 64], "dropout": 0.2, "activation": "relu"},
    reduction_ratio=3,
    bilinear_type="field_interaction",
)

Parameters

ParameterTypeDescriptionDefault
featureslistFeatures used by SENET and bilinear interactionrequired
mlp_paramsdictPrediction MLP parametersrequired
reduction_ratiointSENET reduction ratio3
bilinear_typestrfield_all / field_each / field_interactionfield_interaction

Use Cases

  • Scenarios with varying feature importance
  • Need for complex feature interactions
  • CTR prediction tasks

8. DeepFFM

Description

DeepFFM (Deep Field-aware Factorization Machine) combines field-aware factorization machine with deep neural network, capturing field-aware high-order feature interactions.

Paper Reference

Xiao, Jun, et al. "Deep learning over multi-field categorical data." European conference on information retrieval. Springer, Cham, 2016.

Core Principles

  • FFM Foundation: Field-aware factorization machine, learns specific interaction vectors for each field pair
  • Deep Network: Deep neural network capturing high-order feature interactions
  • Joint Training: FFM and Deep parts trained jointly

Usage

python
from torch_rechub.models.ranking import DeepFFM, FatDeepFFM

# FFM uses a different embedding for every field pair, so cross-feature
# vocabularies must reserve one offset range per field.
ffm_linear_features = [
    SparseFeature(f.name, vocab_size=f.vocab_size, embed_dim=1) for f in sparse_features
]
ffm_cross_features = [
    SparseFeature(f.name, vocab_size=f.vocab_size * len(sparse_features), embed_dim=10)
    for f in sparse_features
]

model = DeepFFM(
    linear_features=ffm_linear_features,
    cross_features=ffm_cross_features,
    embed_dim=10,
    mlp_params={"dims": [1600, 1600], "dropout": 0.5, "activation": "relu"},
)

# FatDeepFFM (enhanced version)
fat_model = FatDeepFFM(
    linear_features=ffm_linear_features,
    cross_features=ffm_cross_features,
    embed_dim=10,
    reduction_ratio=1,
    mlp_params={"dims": [1600, 1600], "dropout": 0.5, "activation": "relu"},
)

Parameters

ParameterTypeDescriptionDefault
linear_featureslistFirst-order linear features, normally with embed_dim=1required
cross_featureslistFFM interaction features whose vocabularies reserve field-offset spacerequired
embed_dimintFFM embedding dimensionrequired
reduction_ratiointCEN reduction ratio, required only by FatDeepFFMrequired (FatDeepFFM)
mlp_paramsdictMLP parameters after FFM interactionsrequired

Use Cases

  • Scenarios where field-aware feature interactions are important
  • Complex feature interaction scenarios
  • CTR prediction tasks

9. BST

Description

BST (Behavior Sequence Transformer) uses Transformer to model user behavior sequences, capturing long-range dependencies in sequences.

Paper Reference

Chen, Qiwei, et al. "Behavior Sequence Transformer for E-commerce Recommendation in Alibaba." arXiv preprint arXiv:1905.06874 (2019).

Core Principles

  • Transformer Encoder: Uses multi-head self-attention to capture sequence dependencies
  • Positional Encoding: Adds positional information to preserve sequence order
  • Feature Fusion: Fuses sequence features with other features for final prediction

Usage

python
from torch_rechub.basic.features import SparseFeature, SequenceFeature
from torch_rechub.models.ranking import BST

features = [SparseFeature("user_id", vocab_size=n_users + 1, embed_dim=8)]
target_features = [
    SparseFeature("target_item_id", vocab_size=n_items + 1, embed_dim=8),
    SparseFeature("target_cate_id", vocab_size=n_cates + 1, embed_dim=8),
]
history_features = [
    SequenceFeature("hist_item_id", vocab_size=n_items + 1, embed_dim=8,
                    pooling="concat", shared_with="target_item_id"),
    SequenceFeature("hist_cate_id", vocab_size=n_cates + 1, embed_dim=8,
                    pooling="concat", shared_with="target_cate_id"),
]

model = BST(
    features=features,
    history_features=history_features,
    target_features=target_features,
    mlp_params={"dims": [256, 128]},
    nhead=8,
    dropout=0.2,
    num_layers=1,
    max_seq_len=51,
)

Parameters

ParameterTypeDescriptionDefault
featureslistUser-profile/context features, excluding history and targetrequired
history_featureslistHistory sequences with pooling="concat"required
target_featureslistTarget-item features corresponding to the history featuresrequired
mlp_paramsdictPrediction MLP parametersrequired
nheadintAttention heads; must divide the summed history embedding dimension8
dropoutfloatTransformer dropout0.2
num_layersintNumber of Transformer Encoder layers1
max_seq_lenintUpper bound of history length plus one target51

Use Cases

  • Scenarios where user behavior sequences are important
  • Long sequence modeling
  • Sequential recommendation tasks

10. DIN

Description

DIN (Deep Interest Network) is an attention-based deep interest network that dynamically captures user interests based on target items.

Paper Reference

Zhou, Guorui, et al. "Deep interest network for click-through rate prediction." Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining. 2018.

Core Principles

  • Interest Extraction: Extracts interest representations from user behavior sequences
  • Attention Mechanism: Computes attention weights for each historical behavior based on target item
  • Dynamic Interest Aggregation: Dynamically aggregates user interests based on attention weights

Usage

python
from torch_rechub.basic.features import SparseFeature, SequenceFeature
from torch_rechub.models.ranking import DIN

features = [SparseFeature("user_id", vocab_size=n_users + 1, embed_dim=8)]
target_features = [SparseFeature("target_item_id", vocab_size=n_items + 1, embed_dim=8)]
history_features = [
    SequenceFeature("hist_item_id", vocab_size=n_items + 1, embed_dim=8,
                    pooling="concat", shared_with="target_item_id")
]

model = DIN(
    features=features,
    history_features=history_features,
    target_features=target_features,
    mlp_params={"dims": [256, 128]},
    attention_mlp_params={"dims": [64, 32], "use_softmax": False},
)

Parameters

ParameterTypeDescriptionDefault
featureslistUser-profile/context featuresrequired
history_featureslistHistory sequences with pooling="concat"required
target_featureslistTarget features matching history count, order, and dimensionsrequired
mlp_paramsdictPrediction MLP parametersrequired
attention_mlp_paramsdictActivation Unit parametersrequired

Use Cases

  • Scenarios with dynamic user interests
  • Target item-related interest modeling
  • CTR prediction tasks

11. DIEN

Description

DIEN (Deep Interest Evolution Network), proposed by Alibaba at AAAI 2019, extends DIN with an Interest Extractor Layer (GRU + auxiliary loss) and an Interest Evolution Layer (AUGRU) to model how user interests evolve over time.

Paper Reference

Zhou, Guorui, et al. "Deep interest evolution network for click-through rate prediction." Proceedings of the AAAI conference on artificial intelligence. 2019.

Core Principles

  • Interest Extractor Layer: GRU over behaviour sequences; auxiliary loss supervises each hidden state with positive/negative next-step samples (paper Eq.7)
  • Interest Evolution Layer: AUGRU embeds attention into the update gate; attention is softmax-normalised over the full valid sequence (paper Eq.14-16)
  • Auxiliary Loss: Laux=1N[logσ(htet+1+)+log(1σ(htet+1))]
  • Padding: index 0 is the padding token; padding positions are excluded from GRU, AUGRU attention, and auxiliary loss; all-padding samples keep zero hidden state

Usage

python
from torch_rechub.basic.features import SparseFeature, SequenceFeature
from torch_rechub.models.ranking import DIEN

# padding_idx=0 must be set on target_features — they own the embedding tables
target_features = [
    SparseFeature("target_item_id", vocab_size=n_items+1, embed_dim=8, padding_idx=0),
    SparseFeature("target_cate_id", vocab_size=n_cates+1, embed_dim=8, padding_idx=0),
]
history_features = [
    SequenceFeature("hist_item_id", vocab_size=n_items+1, embed_dim=8,
                    pooling="concat", shared_with="target_item_id", padding_idx=0),
    SequenceFeature("hist_cate_id", vocab_size=n_cates+1, embed_dim=8,
                    pooling="concat", shared_with="target_cate_id", padding_idx=0),
]
neg_history_features = [
    SequenceFeature("neg_hist_item_id", vocab_size=n_items+1, embed_dim=8,
                    pooling="concat", shared_with="target_item_id", padding_idx=0),
    SequenceFeature("neg_hist_cate_id", vocab_size=n_cates+1, embed_dim=8,
                    pooling="concat", shared_with="target_cate_id", padding_idx=0),
]

model = DIEN(
    features=features,
    history_features=history_features,
    neg_history_features=neg_history_features,
    target_features=target_features,
    mlp_params={"dims": [256, 128]},
    alpha=0.2,
)
# CTRTrainer must use loss_mode=False — forward returns (prediction, aux_loss)

Parameters

ParameterTypeDescription
featureslistUser profile / context features fed into the top MLP
history_featureslistPositive behaviour sequences; pooling="concat", padding_idx=0, shared_with=target_feature
neg_history_featureslistNegative-sampled sequences; same constraints as history_features; shared_with must point to target feature
target_featureslistTarget item features; padding_idx=0 so the shared embedding table's row 0 is a zero vector
mlp_paramsdictTop MLP params; activation is fixed to dice
alphafloatAuxiliary loss weight (default 0.2)

Use Cases

  • E-commerce / news recommendation where user interests evolve over time
  • Sequential behaviour data with temporal ordering
  • CTR prediction tasks

12. AutoInt

Description

AutoInt (Automatic Feature Interaction Learning via Self-Attentive Neural Networks) uses self-attention to automatically learn feature interactions, flexibly capturing various orders of feature interactions.

Paper Reference

Song, Weiping, et al. "AutoInt: Automatic Feature Interaction Learning via Self-Attentive Neural Networks." Proceedings of the 28th ACM International Conference on Information and Knowledge Management. 2019.

Core Principles

  • Embedding Layer: Maps discrete features to low-dimensional vector space
  • Multi-head Self-attention: Automatically learns interaction relationships between features
  • Residual Connection: Enhances training stability
  • Layer Normalization: Accelerates model convergence

Usage

python
from torch_rechub.models.ranking import AutoInt

model = AutoInt(
    sparse_features=sparse_features,
    dense_features=dense_features,
    num_layers=3,
    num_heads=2,
    dropout=0.2,
    mlp_params={"dims": [256, 128], "dropout": 0.2, "activation": "relu"},
)

Parameters

ParameterTypeDescriptionDefault
sparse_featureslistAt least one sparse feature; all must use the same embed_dimrequired
dense_featureslistDense features; may be an empty listrequired
num_layersintNumber of Interacting Layers3
num_headsintNumber of attention heads2
dropoutfloatAttention dropout0.0
mlp_paramsdict or NoneOptional Deep-branch parametersNone

Use Cases

  • Automatic feature interaction learning
  • Complex feature interaction scenarios
  • CTR prediction tasks

13. Model Comparison

ModelComplexityExpressivenessEfficiencyInterpretability
WideDeepLowMediumHighHigh
DeepFMMediumHighMediumMedium
DCN/DCNv2MediumHighHighMedium
EDCNMediumHighMediumMedium
AFMMediumMediumMediumHigh
FiBiNETMediumHighMediumMedium
DeepFFMHighHighLowMedium
BSTHighHighLowMedium
DINMediumHighMediumMedium
DIENHighHighLowMedium
AutoIntHighHighLowMedium

14. Usage Recommendations

  1. Choose based on data scale: For small-scale data, use simple models (WideDeep, DeepFM); for large-scale data, try more complex models
  2. Choose based on feature types: For important sequence features, use BST, DIN, DIEN; for important feature interactions, use DCN, DeepFM
  3. Choose based on computational resources: For limited resources, use efficient models (DCN, WideDeep)
  4. Try multiple models and ensemble: Different models may capture different feature interaction patterns; ensembling can improve results

15. Complete Training Example

python
from torch_rechub.models.ranking import DeepFM
from torch_rechub.trainers import CTRTrainer
from torch_rechub.utils.data import DataGenerator
from torch_rechub.basic.features import DenseFeature, SparseFeature

# 1. Define features
dense_features = [
    DenseFeature(name="age", embed_dim=1),
    DenseFeature(name="income", embed_dim=1)
]

sparse_features = [
    SparseFeature(name="city", vocab_size=100, embed_dim=16),
    SparseFeature(name="gender", vocab_size=3, embed_dim=16),
    SparseFeature(name="occupation", vocab_size=20, embed_dim=16)
]

# 2. Prepare data
x = {
    "age": age_data,
    "income": income_data,
    "city": city_data,
    "gender": gender_data,
    "occupation": occupation_data
}
y = label_data

# 3. Create data generator
dg = DataGenerator(x, y)
train_dl, val_dl, test_dl = dg.generate_dataloader(split_ratio=[0.7, 0.1], batch_size=256)

# 4. Create model
model = DeepFM(
    deep_features=sparse_features + dense_features,
    fm_features=sparse_features,
    mlp_params={"dims": [256, 128, 64], "dropout": 0.2, "activation": "relu"}
)

# 5. Create trainer
trainer = CTRTrainer(
    model=model,
    optimizer_params={"lr": 0.001, "weight_decay": 0.0001},
    n_epoch=50,
    earlystop_patience=10,
    device="cpu",
    model_path="saved/deepfm"
)

# 6. Train model
os.makedirs("saved/deepfm", exist_ok=True)
trainer.fit(train_dl, val_dl)

# 7. Evaluate model
auc = trainer.evaluate(trainer.model, test_dl)
print(f"Test AUC: {auc}")

# 8. Export ONNX model (first install: pip install "torch-rechub[onnx]")
trainer.export_onnx("deepfm.onnx")

16. FAQ

Q: How to choose the right model?

A: Choose based on data scale, feature types, computational resources, and business requirements. Start with simple models and gradually try more complex ones.

Q: What to do about overfitting?

A: Try the following:

  • Add regularization (L1/L2)
  • Increase dropout rate
  • Use early stopping
  • Add more training data
  • Simplify model structure

Q: How to handle large-scale features?

A: Try the following:

  • Feature selection: Keep only important features
  • Feature hashing: Map high-dimensional features to low-dimensional space
  • Hierarchical embedding: Use different embedding dimensions for different features

Q: How to speed up training?

A: CTRTrainer can use one GPU through device, or wrap the model in single-machine torch.nn.DataParallel through gpus=[...]. It does not provide automatic mixed precision (AMP) or multi-machine distributed training. Whether batch size can be increased depends on available memory; test it incrementally.