There are signs of unease within the Government about the plan to hand over the British Indian Ocean Territory.