Push button start problems

I currently own a 2009 Avalon Limited with a little over 60,000 miles on it. I have started to experience a problem with starting the car. At times, it simply will not start. Its like there is a dead battery. I have had a new battery installed on the car as well as new batteries in the remote. I had this same problem in my 2006 Avalon Limited. After many trips to the dealer, being told that they could not find anything wrong with the car, I traded it in on the 2009 I currently own. I am going to the dealer in the morning, but do not expect to get any answers. Am getting very frustrated. Is there anyone who knows what the problem is and how to get repaired.
Comments
I found a post concerning this same problem in another discussion.
http://forums.edmunds.com/discussion/comment/4045179/#Comment_4045179
" The problem was the brake switch that activates the starter button."