This issue has arisen. It's rather a complex one so let me explain.
Any building that offers a service (shop, tavern) is INDESTRUCTIBLE.
Houses are DESTRUCTIBLE but only from the inside, and you can only enter if you are granted permission by the householder. Broken fragments of houses, or any other buildings, cannot be taken.
The mansion is INDESTRUCTIBLE, as it is a main attraction, but it's a no-fighting zone inside it anyway.
For that matter, any PERSON that provides a service (i.e. shopkeepers) is INVULNERABLE. They cannot be hurt (but they aren't players).
Players have the option to turn battling off entirely, and if they do this they cannot attack but they cannot be hurt either. There is a cooldown going into and out of no-battle mode of one minute, during which time you cannot attack; but if you are coming into no-battle mode you can still be attacked during that minute.
((Read as: convenient way of stopping everyone dying if they're having conversations.))
Certain NPCs will be battle-able. This is generally the case if they're not providing a service. However they yield no EXP and will respawn quickly.
The "flip-world", which is NOT accessible yet btw, does not allow no-battle mode.