The Cambridge Dictionary says that “just saying” is used when you complain or criticise another person in an attempt to make it less offensive. The takeaway here is that “just saying” is a way to dismiss a person's feelings and absolve the speaker of any blame for their hurtful words.
When someone says, “Just Saying,” they mean that they didn't mean to offend or annoy anyone; they were just stating a fact, an opinion, or an observation.