`eval.metrics`¶

class olmo_core.eval.metrics.Metric(device=None, process_group=None)[source]¶

Base class for evaluation metrics.

abstract update(*args, **kwargs)[source]¶

Update the metric.

abstract compute()[source]¶

Compute the metric.

abstract reset()[source]¶

Reset the metric.

class olmo_core.eval.metrics.MeanMetric(device=None, process_group=None)[source]¶

Computes the mean over a stream of values.

update(value, weight=1.0)[source]¶

Parameters:

value (Union[float, Tensor]) – The latest value to update the metric with. Could be a tensor of values.
weight (Union[float, Tensor], default: 1.0) – The corresponding weight(s) for the value. Should be the same shape as value.

Return type:

None

Computes the mean over the values and weights given.

Reset the metric.

eval.metrics¶