In my opinion there is a connection. I am in my mid-forties and after a little over a year of daily use I noticed the same thing. It is just a guess here - but I think it lowers testosterone by interfering somehow with cholesterol. There seems to be enough tidbits of anecdotal information on "lack of desire" along with some studies and anecdotal blood test results that show changes in cholesterol level (which is necessary for testosterone production) that I think it is at least a possibility. I personally have tried adding zinc, magnesium, vitamin D, olive oil and DHEA to my diet - all of which helped (although the DHEA gets pretty mixed results in studies).
If all that fails - go get yourself some cialis. Even if things are all good, that stuff will have the oak trees in you neighborhood jealous and your partner scared.
Great non-technical website:
http://www.artofmanliness.com/2013/01/15/how-testosterone-is-made/
He even has a section where he did a 90 day experiment to raise his T-level naturally with test results to prove it.
Good Luck!