It's not as bad as you think Peppermint_M
However I agree with a lot of the suggestions above, especially the timbered house. A lot of the dragon sets have some wood element, so this would fit with the theme and be realistic (hoardings). I think it would look better if you made the walkway on the walls one brick lower, because the guard on the left side looks like he's standing on top, which isn't very realistic. Finally you said you were going for a realistic castle, so I would suggest making a gate of some kind as opposed to just an arch, which would make it IMO more solid looking.
Good luck!