Bees swarming my rabbit hutch?

I've got my rabbits hutch in a shady corner of the yard a few feet from an orange tree and we usually have a fair handful of bees around that orange tree. I just went out to check the rabbits food and water while he's having run around time in the yard and there are BEES flying in and out of his sleeping area of his hutch. More than the usual one or two buzzing about. I'm concerned about them possibly moving in and trying to turn my rabbits home into a hive. Is there a way for me to shoo them away without spraying any chemicals or such that would harm the rabbit? What could I do to smoke them out? Help me!!!!