Speakerphone Mic on iPhone 5 Only Works Sometimes

I have an iphone 5. When I am on a call, the speakerphone mic only works sometimes. During the times that it isn't, people on the other line cannot hear me. I also cannot use the loudspeaker for siri at this time as well.

Anyone had experience with this? Might this be a hardware or software related problem? Suggestions? Thanks