From an interview with Kevin Wing, NATAS

MGM had a strict rule that the adult Munchkins could not socialize with us child Munchkins. When we were not “on set” we were in our studio school classroom. I didn’t see anything suspicious in his persistent invitations to lunch, but a candy bar helped me see he wasn’t really on the up-‘n’-up.

The full story is in Chapter 2 in the book.