How to test and understand custom analyzers in Lucene | RTFB

But now if you want to create a new JUnit test for testing the behaviour of an analyzer, you’ve got access to a new Base Class that you can extends called BaseTokenStreamTestCase. But the joy of it all is not exactly to be able to write "public class MyWonderfulTestCase extends BaseTokenStreamTestCase" and clap your hands, now you have access to a brand new class of assertions (by the way you need to enable assertions to execute your tests with the -ea parameter as VM args)

•  assertTokenStream : it allows you to specify the field on which you’re testing (otherwize "dummy" fieldName gets passed onto the analyzer) and check the token stream output;
• assertAnalyzesTo : you don’t specify the field on which you’re testing, but it has a simpler syntax.

Read full article from How to test and understand custom analyzers in Lucene | RTFB