In pandas 0.18, the behavior of ``nth()`` changed so that Grouper no
longer can be easily used to recover group labels.
Instead of using the built-in grouper behavior, we use a groupby on two
arrays we build ourselves. This recovers the original behavior, and is
about 2x faster as a bonus.