literals

The code for parsing literals is extremely similar to the string matching code.

const literalRegex = /[a-zA-Z]/
const literalRegexNext = /[a-zA-Z0-9]/

if (literalRegex.test(currentToken)) {
  const bucket = lookahead(literalRegex, literalRegexNext)
  
  out.push({
    type: TokenType.Literal,
    value: bucket.join('')
  })

  currentPosition += bucket.length

  continue
}

A quick note: This should be put after our string check, and nearly always be the last check (if) that occurs in our while loop, as we're basically "catch-all'ing" syntax we don't otherwise recognise, and assuming it's a literal.

You may notice that we're actually passing two regexs to our lookahead function here. This is because we want to differenciate what a literal is allowed to start with v what it can contain.

In our langauge (exactly the same as Javascript), a literal won't be allows to start with a number, but will be allowed to contain numbers.

This isn’t stricly necessary, as our compiler gets more advanced we'll be able to substitute variables with "valid" Javascript ones, but to start with (for both ease, and development speed) we’re just going to map them 1-1.