(a) Suppose (x)s=x-1defines an automorphism of G. This map is always a bijective, so the hypothesis is really just saying (xy)-1 = x-1y-1 for all x,y in G. This implies y-1x-1 = x-1y-1, which gives x-1 = y x-1y-1, then x-1y = y x-1, y = xy x-1, and finally, yx=xy for all x,y in G. Thus $G$ is abelian.
(b) Suppose s : G ---> G is an automorphism satisfying s2 = id and (x)s is different from x for x not equal to e. Consider the function G --> G mapping x to x-1xs. We claim this map is injective. Indeed, suppose x-1xs = y-1ys. Manipulating and using that s is a homomorphism, we get yx-1 = ys (xs)-1 = (yx-1)s. Then yx-1 = e, by hypothesis, so y=x. Since the map is injective, and G is finite, the map is surjective as well. Then, for any y in G, there is an x in G such that y = x-1xs. Applying s, we get that ys = (x-1xs)s = (x-1)sxss, which equals (xs)-1x since ss = s2, and this is equal to (x-1xs)-1, which is y-1. Thus (y)s = y-1 for all y in G. Then G is abelian by part a.
This shows, for finite groups, the only possible automorphism of order two with no notrivial fixed points is the inversion mapping, in which cae G is abelian. What about infinite groups?