A camera looks down at a road. A truck passes underneath and the system has to say what it’s carrying. Some materials are coarse families that split into commercial grades; the grades matter for billing, and they look similar enough that a single flat classifier over every leaf class does poorly. So it’s staged: a coarse model picks the family, then a specialist refiner splits that family into products. Standard hierarchical classification. ...
A five-class material classifier reads a crop of a truck bed and calls what’s in it. Four of the five classes were fine — F1 between 0.92 and 0.97. One was hopeless: class F1 coarse classes (3 of them) 0.92 – 0.97 the coarser of two stone grades 0.720 the finer of two stone grades 0.476 macro 0.791 The two problem classes are the same crushed stone at two grain sizes. Between 13 and 18 of 30 validation frames of the finer grade came back labelled as the coarser one. A coin flip with extra steps. ...
The classifier reported a macro F1 of 0.975. It was, in the narrow sense, telling the truth: that number came out of a validation set the model never trained on, computed correctly, reproducibly. What it measured was “a new frame of a truck I have already seen.” Roughly 94% of the validation images had another frame of the same truck pass sitting in the training set. Different filename, different moment, same vehicle, same load, same lighting, half a second apart. ...