You’re right, yet many people still say ‘literally’ immediately before saying something figurative.

Colloquialisms are king, you will always lose to the masses — in this case they happen to be very ignorant.