A generalized coherence framework for detecting and characterizing nonlinear interactions in the nervous system