Buying a property, neighbor across the st is using the property's address

I own a house, and am currently thinking about buying the vacant lot next to mine to ensure that no one builds on it (beautiful view).

While talking with the seller they mentioned that the neighbor across the street (we are not on speaking terms, long story) is using the vacant lot's address as their own.

Will this cause a problem with the sale or future sales, and if I buy the property and want to receive mail there in the future, what actions do I need to take to reclaim my address?

I am located in Tennessee if that matters.