infotech

 

 

Feedback and inquiries

Center for Machine Vision Research
Department of Computer Science and Engineering P.O.Box 4500
FIN-90014 University of Oulu
Finland
Tel. +358-8-553 2525
contact-cse (at) ee.oulu.fi

University of Oulu
Outex Texture Database

Test suite types and their testing protocols

The framework contains three basic types of test suites:

The purpose of an individual test suite is to encapsulate a meaningful entity for the purpose of empirical evaluation of a candidate texture analysis algorithm. A test suite may contain a large number of classification or segmentation problems having a particular basic structure, but for example different partitioning of the image data into training and testing sets due to randomization, or different collection of textures. The motivation in packing a large number of problems with well defined variation for example in terms of textures is to properly evaluate the robustness of the algorithm with respect to its built-in parameters (any external parameters provided by the user are required to remain constant throughout the suite) and analyzed textures. Further, individual test suites may be combined into challenging ´grand suites´ assessing the performance in many different respects.

 

Texture classification ("TC")

Generation of data used in test suites.

The images used in a texture classification suite are extracted from the given set of source images (particular texture classes, illuminations, spatial resolutions, and rotation angles) by centering the sampling grid so that equally many pixels are left over on each side of the sampling grid. Thus, for example for window sizes 128x128, 64x64 and 32x32 pixels 20, 88 and 368 images in total are obtained from a given source image. To remove the effect of global first and second order gray scale properties in intensity images, each intensity image is individually normalized to have an average intensity of 128 and a standard deviation of 20. If the training and testing images of a particular texture classification problem are extracted from the same set of source images, the images are divided randomly to two halves of equal size for the purpose of obtaining an unbiased performance estimate. Within the context of a given test suite this random partitioning may be repeated N times, resulting in a test suite of N individual problems, which facilitates more reliable performance evaluation than just a single shot experiment.

Input data in an individual problem. L is a set of NL texture classes (labels); A is a set of NA training images with known class labels LA; B is a set of NB testing images; C is a set of NL cost functions containing one value for each class. for the purpose of obtaining unbiased performance estimates.

'Hidden' ground truth data in an individual problem. True class labels LB of testing images.

Required output in an individual problem. Class labels LO assigned to testing images.

Optional output in an individual problem. Total elapsed computation time T from the start of the program to the output of LO; types 'wall clock' time Twc and 'computer timer’ time Tct.

Performance metric in an individual problem. Score for an individual problem is defined as the proportion of the combined cost of correctly classified testing images of the combined cost of all classified testing images (d(i,j) is Kronecker delta function returning 1, if i and j are equal, otherwise 0):

Test suite score. If test suite contains P>1 problems, then following scores are computed (otherwise test suite score S equals Sp of the only problem included in the suite):

 

The organization of classification test suites.

The hierarchical architecture of classification test suites:

Outex_TC_?????/ images/
  /000/
  /001/
  /002/
  .
  .
  .
  /nnn /classes.txt
    /test.txt
    /train.txt

Explanation of directories/files

  • images Includes the images needed in the test suite. The indices of images are 000000.[type] ... nnnnnn.[type]

     

  • 000/ 001/ 002/... nnn/ Includes the specified problem in this test suite. Each one of these directories has three files: classes.txt, test.txt and train.txt.

     

    • classes.txt - Contains image, true class and cost information of this suite.
    • test.txt - Contains the images (names) and cost values of the test samples.
    • train.txt - Contains the images (names) and cost values of the train samples.

 

 

Segmentation test suites

The texture mosaics used in supervised and unsupervised texture segmentation suites are generated as follows.

Given a set of candidate source images (particular texture classes, illuminations, spatial resolutions, and rotation angles) and the ground truth image containing R regions (Fig.), R images are randomly drawn from the candidate images, so that they all belong to different texture classes. The image region included in the mosaic and the training image needed for supervised segmentation are randomly extracted from the chosen source images so that they do not overlap.

 

Supervised texture segmentation ("SS")

Input data in an individual problem. I is the image to be segmented having NI pixels. RI is the number of regions (textures) in I, each having a distinct class label Lr. Ar is a set of training images, one image for each region (texture) r.

Required output in an individual problem. Output label image O, which has to be of the same size as I. If algorithm produces O that is smaller in size than I, then I is enlarged a sufficient amount by reflection to image Ie prior running the algorithm so that Oe is of the same size as I. In other words, a too small O is not enlarged by padding, for example.

Optional output in an individual problem. Total elapsed computation time T from the start of the program to the output of O; types 'wall clock' time Twc and 'computer timer' time Tct.

Performance metric in an individual problem. Let Iij and Oij denote the class labels of pixel (i , j ) in I and O, respectively (we assume that the regions in O are in the same spatial order as in I). Then the score of an individual problem is the proportion of correctly labeled pixels of all pixels:

(1)
Test suite score. If test suite contains P>1 problems, then following scores are computed (otherwise test suite score S equals Sp of the only problem included in the suite):

(2)

 

Unsupervised texture segmentation ("US")

Input data in an individual problem. Same as in supervised texture segmentation, but training images Ar are not provided. Algorithms are categorized based on whether the number of regions RI is provided or not.

Required output in an individual problem. Same as in supervised texture segmentation, and also the number of regions in the output image, RO.

Optional output in an individual problem. Same as in supervised texture segmentation. Performance metric in an individual problem. See Eq.(1). If RI<>RO, then Eq.(1) is computer over all pairings of individual regions in I and O, and the smallest obtained Sp is the score for the problem.

Test suite score. If test suite contains P1 problems, then the proportion of 'successful' segmentations (Ss) is first computed as follows. If the algorithm itself determines the number of regions, then:

If the algorithm is provided with RI, then Ss is computed as follows:

Finally, scores defined in Eq.(2) are computed both for successful and unsuccessful segmentations.

 

 

The organization of segmentation test suites (SS/US).

The hierarchical architecture of segemntation test suites:

Outex_SS_?????/ ground_truth.ras
  /000/
  /001/
  /002/
  .
  .
  .
  /nnn /info.txt
    /problem.ras
    /train_01.ras
    /train_02.ras
    .
    .
    .
    /train_nn.ras

Explanation of directories/files

  • ground_truth.ras The ground truth image of this test suit.

     

  • 000/ 001/ 002/... nnn/ Includes the specified problem in this test suite. Subdirectory of each problem has the following files:

     

    • info.txt - Contains class info and the placement info of the train and test images.
    • problem.ras - The problem image.
    • train_01.ras ... train_nn.ras - The training images of the test suit. (Only in SS test suites)

 

 

Last modified: 2007-08-27