Matching strings & literals

Now that we have implemented checks for our known token types, it’s time to match things we aren’t aware of ahead of time, or special values like strings.

For this to work, we need another lookahead-type utility function. This one will take a regex instead of a string.

function lookahead (match: RegExp, matchNext?: RegExp): string[] {
  const bucket: string[] = []

  while (true) {
    const nextIndex = currentPosition + bucket.length
    const nextToken = input[nextIndex]
    if (!nextToken) {
      break
    }
    let m: string | RegExp = match
    if (matchNext && bucket.length) {
      m = matchNext
    }
    if (m && !m.test(nextToken)) {
      break
    }
    bucket.push(nextToken)
  }

  return bucket
}

This may look complicated to start with, but it’s basically just doing what our main while loop is!

It’s iterating over the next characters in our input, trying to see if they match either our start or end regex, and then returns them.

After implementing this function, matching both string and literal values should be fairly straight forward.