Following an active task, the memory representations for used and unused objects are different. However, it is not clear whether these differences arise due to prioritizing objects that are task-relevant, objects that are physically interacted with, or a combination of the two factors. The present study allowed us to tease apart the relative importance of task-relevance and physical manipulation on object memory. A paradigm was designed in which objects were either necessary to complete a task (target), moved out of the way (obstructing, but interacted with), or simply present in the environment (background). Participants’ eye movements were recorded with a portable tracker during the task, and they received a memory test on the objects after the task was completed. Results showed that manipulating an object is sufficient to change how information is extracted and retained from fixations, compared to background objects. Task-relevance provides an additional influence: information is accumulated and retained differently for manipulated target objects than manipulated obstructing objects. These findings demonstrate that object memory is influenced both by whether we physically interact with an object, and the relevance of that object to our behavioral goals.