There could be a number of reasons. In no particular order here are some ideas:

The audio you are trying to decode was obtained at a different sample rate than the rate used for the model; the grammar is large but the model was trained on insufficient data; the dictionary contains conflicting uses of phonemes (say you have words with the phoneme "ax" but the prompt audio actually contains a variety of "ae", "ah" and "aa").

If the sample rate is not an issue, reduce your grammar to just a few simple sentences and see if the accuracy rate goes up.