Parameter settings for evaluations¶
(Default values are in ~/recommendation/properties/evaluation.yaml)
The benchmark provides several arguments for describing:
Basic setting of the parameters
See below for the details:
Evaluation required parameters¶
Evaluation process set ups¶
eval_step (int)
: How many epochs apart should the model be evaluated on the validation set.eval_type (str)
: Evaluation types can be chosen from [‘ranking’, ‘CTR’], with ranking being the most commonly used. Most models do not support CTR tasks.eval_batch_size (int)
: The batch size used to conduct evaluation.
Evaluation metric set ups¶
watch_metric (str)
: During training, the model should be saved for testing when the watch_metric on the validation set reaches its highest value.topk (list)
: The evaluation ranking list size list, such as [5,10, 20]store_scores (bool)
: Decide whether to save ranking scores for the post-processing (re-ranking) step.decimals (int)
: Number of decimal places retained of evaluation metrics.mmf_eval_ratio (float)
: The parameters of the MMF metric define the how many ratio of the worst-off group’s utility compared to the utilities of all groups.fairness_type (str)
: The fairness computational methods, choosed among [“Exposure”, “Utility”]. Exposure means computes fairness on the item exposure, Uility means computes fairness on the ranking scores.
Training set ups¶
device (str)
: used device (cpu or gpu) to run the codes.epoch (int)
: training epochs.batch_size (int)
: training batch size.learning_rate (float)
: learning rate of optimizing the models.