To this day, I primarily judge a song by two qualities: intent and lyrics. Intent simply means: what is the author trying to accomplish through the song? What is he/she trying to say? How well is this achieved?
[This post is basically some thoughts that I took out of my last one because they didn't really fit with what was being said. However, I still think they are worth saying, so here is me rambling about lyrics:]
I also judge it by the content of the lyrics. This does not mean that I think deep is better than shallow. I rather enjoy some shallow songs. But the depth of the song should be reflective of the intent of the song.
For instance, praise choruses are not for the purpose of teaching doctrine or thoughtful reflection. They are emotional or experiential inducers. One may also say mood setters. They shape one's mood or emotional state to induce a particular attitude towards God. Now I am not saying that all one feels while musically worshiping God is a product of the music. I believe that a real encounter with God can and usually takes place. But the purpose of the music is as a tool to allow one to approach God in a particular way to better enable that encounter.
As such, lyrical simplicity is better, since difficult lyrics require attention. Since the purpose is to focus on God, focusing on the lyrics is a counter-productive.
Still, that doesn't mean the lyrics are to be ignored. The lyrics still act as a guide to shaping one's attitude towards God. Thus the lyrics should still be critiqued. For instance, should I be singing about dancing as I just stand here? Is this statement actually accurate?
Lyrical analysis is more important, of course, in music which is designed for didactic, instructional or reflective purposes. This is true of hymns, but is also true of most alternative Christian music (by this, I mean Christian music which is based on secular forms), as well as much of secular music.
Lyrical analysis is also quite important with secular music. I dislike it when people say that we shouldn't listen to secular music at all. We are supposed to reach the people around us. The most important thing to reaching a people is to understand that people. The primary way to understand a people is to engage with their information media. Throughout history, this has mostly been through literature and song, though it has greatly expanded today. By scorning popular media (especially the simple form of music), we are rejecting the most effective tools in reaching the people around us.
However, I do agree that what you take in when engaging with media affects you far more than you are often aware. Therefore, discernment is needed. In music, on what basis can we discern the goodness of a song? The lyrics.