Both Sides of the Spectrum: Learn to Recognize Which One You Are