Class CrossProductTokenization

  • All Implemented Interfaces:
    java.io.Serializable, Annotatable, SymbolTokenization, Changeable

    public class CrossProductTokenization
    extends WordTokenization
    Tokenization for cross-product alphabets. This class handles the general case of tokens of the form (foo bar baz), where each element is handled by a sub-tokenization. By default, these will be the "name" tokenizations of each of the sub-alphabets, but any tokenization can be used.
    Since:
    1.2
    Author:
    Thomas Down, Greg Cox
    See Also:
    Serialized Form
    • Constructor Detail

      • CrossProductTokenization

        public CrossProductTokenization​(Alphabet alpha,
                                        java.util.List tokenizers)
    • Method Detail

      • parseToken

        public Symbol parseToken​(java.lang.String token)
                          throws IllegalSymbolException
        Description copied from interface: SymbolTokenization
        Returns the symbol for a single token.

        The Symbol will be a member of the alphabet. If the token is not recognized as mapping to a symbol, an exception will be thrown.

        Parameters:
        token - the token to retrieve a Symbol for
        Returns:
        the Symbol for that token
        Throws:
        IllegalSymbolException - if there is no Symbol for the token