When accepting a parcel from another person, we are able to use information about that person’s movement to estimate in advance the weight of the parcel, that is, to judge its weight from observed action. Perceptual weight judgment provides a powerful method to study our interpretation of other people’s actions, but it is not known what sources of information are used in judging weight. We have manipulated full form videos to obtain precise control of the perceived kinematics of a box lifting action, and use this technique to explore the kinematic cues that affect weight judgment. We find that observers rely most on the duration of the lifting movement to judge weight, and make less use of the durations of the grasp phase, when the box is first gripped, or the place phase, when the box is put down. These findings can be compared to the kinematics of natural box lifting behaviour, where we find that the duration of the grasp component is the best predictor of true box weight. The lack of accord between the optimal cues predicted by the natural behaviour and the cues actually used in the perceptual task has implications for our understanding of action observation in terms of a motor simulation. The differences between perceptual and motor behaviour are evidence against a strong version of the motor simulation hypothesis.