Oldie but Goodie: the answer

The answer is A--the yoyo will wind itself up.

The easiest way to see the answer is to take moments about the point of contact. The weight, the normal, and the friction force all act through the point of contact, so none of them produces a moment--just like pushing on the wrong side of a door won't work. (It doesn't matter if the friction is to the left or right, it still operates through the point C).

The only force producing a moment about C is the tension in the string. In the original drawing, as in Fig. 1, that force clearly creates a moment that causes a clockwise rotation about C. Hence the yoyo rotates clockwise, and winds itself up.

At the critical angle shown in Fig. 2, there are no forces causing moments about C. At the point, the yoyo won't want to to rotate at all--it will tend to lift off the ground or tip over. At angles steeper than the critical angle, the moment from the string will cause a counter-clockwise rotation, and the yoyo will unwind.

You can try this, it works. Just pull gently.

Exercise: show that the critical angle is θ_C = cos^-1 r/R. For the picture I drew, r = 0.5R, so θ_C = 60°.