What is a development set used for?

Audio and Prompts Discussions

Flat

User: weedwind
Date: 2/9/2015 4:08 pm

Views: 9068
Rating: 1

Hi, all,

When people talked about the WSJ database, they always mentioned concepts of development set (DEV), and evaluation set (EVAL). Can anyone tell me what a DEV is, and what is it used for?

Thank you so much.

Mike

--- (Edited on 2/9/2015 4:08 pm [GMT-0600] by weedwind) ---

Re: What is a development set used for?

User: nsh
Date: 2/9/2015 4:21 pm

Views: 133
Rating: 1

There are hyperparameters used in decoding (language weight, word insertion probability, decoding beams).

Development set is used to estimate best possible hyperparameters.

The final result is verified on a test set.

You can read more about development set or, another name, validation set here:

http://stats.stackexchange.com/questions/19048/what-is-the-difference-between-test-set-and-validation-set

--- (Edited on 2/10/2015 01:21 [GMT+0300] by nsh) ---

Re: What is a development set used for?

User: weedwind
Date: 2/9/2015 6:06 pm

Views: 59
Rating: 1

Thank you so much for your answer. I read through that reference, but I still want to make it totally clear. So, let me ask a specific question.

Suppose I have a MFCC frontend for an ASR task. I want to tune the number of filters in the filterbank (though people know a good number, but just assume it's unknown). I have a training set, a validation set, and a test set, can you please show me how I should proceed?

Mike

--- (Edited on 2/9/2015 6:06 pm [GMT-0600] by weedwind) ---

Re: What is a development set used for?

User: weedwind
Date: 2/9/2015 6:13 pm

Views: 191
Rating: 1

Also, why some database have a validation set, such as WSJ, but some do not, such as TIMIT?

--- (Edited on 2/9/2015 6:13 pm [GMT-0600] by weedwind) ---

Re: What is a development set used for?

User: nsh
Date: 2/11/2015 4:01 am

Views: 3051
Rating: 1

> Suppose I have a MFCC frontend for an ASR task. I want to tune the number of filters in the filterbank (though people know a good number, but just assume it's unknown). I have a training set, a validation set, and a test set, can you please show me how I should proceed?

Train multiple models with each filterbank, optimize hyperparameters (number of tied states, language weight, search beams) for each model with error rate on development set. Test each model on a test set and select the best possible filter width.

> Also, why some database have a validation set, such as WSJ, but some do not, such as TIMIT?

Designers of TIMIT didn't consider the importance of development set, it was not a good practice those days.

Overall, development set is more useful in NIST competitions when competitors have access to development data during few months and then get access to a test set on a final evalutation. For other practical cases the usefulness of development set is limited.

--- (Edited on 2/11/2015 13:01 [GMT+0300] by nsh) ---

Previous • Next •


Username	Password