Testing

Now that we have all of the parts of our lexer together, we should do one last thing.

At the very end of your outside while loop, add the following line:

throw new Error(`Unknown input character: ${currentToken}`)

An extremely simple error message - but should help us notice if we have anything in our input program that we can’t parse just yet.

Now, we can call our function with what we came up with above:

console.log(tokeniser(`
  new hello = 'world'
  print hello
`))

And if everything works, the output should look something like:

[
  { type: 'LineBreak' },
  { type: 'VariableDeclaration' },
  { type: 'Literal', value: 'hello' },
  { type: 'AssignmentOperator' },
  { type: 'String', value: 'world' },
  { type: 'LineBreak' },
  { type: 'ConsoleLog' },
  { type: 'Literal', value: 'hello' },
  { type: 'LineBreak' }
]

You can see that each of the parts of our program have been parsed correctly, and their values have been saved where required!