How easily you're understood
When a listener has to work harder to process what you said, they trust it less. This is not an opinion, in a controlled study, native listeners rated identical trivia statements as LESS TRUE when spoken with a non-native accent, and the effect got stronger the heavier the accent. The mechanism is processing fluency: hard to parse feels like not true. The researchers named job seekers and offshore work explicitly.
This is fixable, and not by 'losing your accent'. Keep reading, the fix is smaller than you think.