Dear Secret-Toy
It sounds to me like your boss is trying to keep your "meetings" a secret. He could be treating you differently because he’s unsure of how he should act around you without giving away your secrets to the entire club. If that’s not the case, he simply prefers you as a toy than a worker. I suggest pulling him into a private conversation to figure out his intentions. Once you do that, the choice is up to you to stay or go, but warning that things might get ugly.
xoxo
Tease