Acoustic Model Discussions

Flat
gauden_scale_densities_bwd: Assertion `finite(den[c][j][k])' failed.
User: bkbroiler
Date: 8/23/2010 7:42 pm
Views: 5865
Rating: 2

Hi,

  I am trying to use SphinxTrain to train up some English acoustic models for a project.  I have trained models with SphinxTrain on numerous prior occassions but this time I encountered a new error that I'm unfamiliar with.

  I have about 200hrs of speech data with transcripts and appropriate lexicon, etc., I initially went through a training run with a small fraction of this data (=~2%) just to confirm that things were more or less setup correctly.  This completed successfully so I started a run with the full corpus.

   The full corpus setup made it through the first several steps,

--------------------------------------

    ("$ST::CFG_SCRIPT_DIR/00.verify/verify_all.pl",
     "$ST::CFG_SCRIPT_DIR/02.falign_ci_hmm/slave_convg.pl",
     "$ST::CFG_SCRIPT_DIR/03.force_align/slave_align.pl",
     "$ST::CFG_SCRIPT_DIR/20.ci_hmm/slave_convg.pl",
     "$ST::CFG_SCRIPT_DIR/30.cd_hmm_untied/slave_convg.pl",
     "$ST::CFG_SCRIPT_DIR/40.buildtrees/slave.treebuilder.pl",
     "$ST::CFG_SCRIPT_DIR/45.prunetree/slave.state-tying.pl",

---------------------------------------

   but eventually bombed out on

     "$ST::CFG_SCRIPT_DIR/50.cd_hmm_tied/slave_convg.pl",

   on 4 gaussians, giving the following error,

----------------------------------------------------

....

INFO: corpus.c(1346): Will process 17768 utts starting at 53304
INFO: main.c(622): Reestimation: Baum-Welch
bw: gauden.c:1377: gauden_scale_densities_bwd: Assertion `finite(den[c][j][k])' failed.
....

----------------------------------------------------

 

  I tracked this down in the source code to,

gauden.c:

gauden_scale_densities_bwd(...){
....
/* BHIKSHA converged g->n_density to g->n_top; possible bugfix, APR 6 98 */
for (k = 0; k < g->n_top; k++) {
/* BHIKSHA converged g->n_density to g->n_top; possible bugfix, END */
den[c][j][k] = EXPF(den[c][j][k] - scl[j]);
assert(finite(den[c][j][k]));
...
}

Where the above assertion seems to be what is causing the failure, however I'm uncertain as to what this implies about my training data, or the process itself.

 

The only thing I'm doing that is any different from previous training exercises or projects is that I'm using Queue::POSIX and npart > 1, however it seems unlikely that this should be the cause of the problem. 

 

Any pointers as to what this error implies would be greatly appreciated!

--- (Edited on 8/23/2010 7:42 pm [GMT-0500] by bkbroiler) ---

Re: gauden_scale_densities_bwd: Assertion `finite(den[c][j][k])' failed.
User: kmaclean
Date: 9/11/2010 10:53 pm
Views: 2189
Rating: 2

cross posted from the Sphinx Forum

--- (Edited on 9/11/2010 11:53 pm [GMT-0400] by kmaclean) ---

PreviousNext