Loners

I think that your neighbors have a valid concern in that they live next to a home that can be empty for periods of time,
then be used by people they do not know.
The urge to keep an eye on the neighborhood is a natural safety concern, at least for me in my neighborhood.

Since I live in the South, my immediate neighbor makes it a point to call when she is leaving overnight or longer,
so I can "keep an eye on the place" for her, which I am glad to do, mainly because she calls, not drops by to chat about it.
and she tells me if she is expecting anyone to be at the house while she is gone, which again I appreciate, since I can ignore my dog's warning barks ( he keeps an eye on her place too).

as for friends....
one has to set clear boundaries, then repeat them if necessary in order to maintain them.
It can be done sweetly.
real friends respect boundaries, of course.