The purpose of karma yoga is to develop immunity from the reactions. How many people are immune to the reactions of other objects and people? You react to whatever a person says and are not immune to what is said.
If somebody says good things, you react and respond; you are elated, happy, the ego is bloated. When somebody says a harsh word, you feel dejected and depressed and your ego is deflated. You are always influenced by the inputs of others. Developing immunity from that is karma yoga, not just sweeping the floor.
Having abandoned attachment, yogis perform action by the body, mind, and intellect and also by the senses, only for the purification of the self. (5:11)