ponyfoo.com

ES6 Generators in Depth

Fix
A relevant ad will be displayed here soon. These ads help pay for my hosting.
Please consider disabling your ad blocker on Pony Foo. These ads help pay for my hosting.
You can support Pony Foo directly through Patreon or via PayPal.

This is ES6 in Depth, the longest-running article series in the history of Pony Foo! Trapped in the ES5 bubble? Welcome! Let me get you started with destructuring, template literals, arrow functions, the spread operator and rest parameters, improvements coming to object literals, the new classes sugar on top of prototypes, let, const, and the “Temporal Dead Zone”, and Iterators.

Like I did in previous articles on the series, I would love to point out that you should probably set up Babel and follow along the examples with either a REPL or the babel-node CLI and a file. That’ll make it so much easier for you to internalize the concepts discussed in the series. If you aren’t the “install things on my computer” kind of human, you might prefer to hop on CodePen and then click on the gear icon for JavaScript – they have a Babel preprocessor which makes trying out ES6 a breeze. Another alternative that’s also quite useful is to use Babel’s online REPL – it’ll show you compiled ES5 code to the right of your ES6 code for quick comparison.

Before getting into it, let me shamelessly ask for your support if you’re enjoying my ES6 in Depth series. Your contributions will go towards helping me keep up with the schedule, server bills, keeping me fed, and maintaining Pony Foo as a veritable source of JavaScript goodies.

Thanks for listening to that, and let’s go into generators now! If you haven’t yet, you should read yesterday’s article on iterators, as this article pretty much assumes that you’ve read it.

Generator Functions and Generator Objects

Generators are a new feature in ES6. You declare a generator function which returns generator objects g that can then be iterated using any of Array.from(g), [...g], or for value of g loops. Generator functions allow you to declare a special kind of iterator. These iterators can suspend execution while retaining their context. We already examined iterators in the previous article and how their .next() method is called once at a time to pull values from a sequence.

Here is an example generator function. Note the * after function. That’s not a typo, that’s how you mark a generator function as a generator.

function* generator () {
  yield 'f'
  yield 'o'
  yield 'o'
}

Generator objects conform to both the iterable protocol and the iterator protocol. This means…

var g = generator()
// a generator object g is built using the generator function
typeof g[Symbol.iterator] === 'function'
// it's an iterable because it has an @@iterator
typeof g.next === 'function'
// it's also an iterator because it has a .next method
g[Symbol.iterator]() === g
// the iterator for a generator object is the generator object itself
console.log([...g])
// <- ['f', 'o', 'o']
console.log(Array.from(g))
// <- ['f', 'o', 'o']

(This article is starting to sound an awful lot like a Math course…)

When you create a generator object (I’ll just call them “generator” from here on out), you’ll get an iterator that uses the generator to produce its sequence. Whenever a yield expression is reached, that value is emitted by the iterator and function execution is suspended.

Let’s use a different example, this time with some other statements mixed in between yield expressions. This is a simple generator but it behaves in an interesting enough way for our purposes here.

function* generator () {
  yield 'p'
  console.log('o')
  yield 'n'
  console.log('y')
  yield 'f'
  console.log('o')
  yield 'o'
  console.log('!')
}

If we use a for..of loop, this will print ponyfoo! one character at a time, as expected.

var foo = generator()
for (let pony of foo) {
  console.log(pony)
  // <- 'p'
  // <- 'o'
  // <- 'n'
  // <- 'y'
  // <- 'f'
  // <- 'o'
  // <- 'o'
  // <- '!'
}

What about using the spread [...foo] syntax? Things turn out a little different here. This might be a little unexpected, but that’s how generators work, everything that’s not yielded ends up becoming a side effect. As the sequence is being constructed, the console.log statements in between yield calls are executed, and they print characters to the console before foo is spread over an array. The previous example worked because we were printing characters as soon as they were pulled from the sequence, instead of waiting to construct a range for the entire sequence first.

var foo = generator()
console.log([...foo])
// <- 'o'
// <- 'y'
// <- 'o'
// <- '!'
// <- ['p', 'n', 'f', 'o']

A neat aspect of generator functions is that you can also use yield* to delegate to another generator function. Want a very contrived way to split 'ponyfoo' into individual characters? Since strings in ES6 adhere to the iterable protocol, you could do the following.

function* generator () {
  yield* 'ponyfoo'
}
console.log([...generator()])
// <- ['p', 'o', 'n', 'y', 'f', 'o', 'o']

Of course, in the real world you could just do [...'ponyfoo'], since spread supports iterables just fine. Just like you could yield* a string, you can yield* anything that adheres to the iterable protocol. That includes other generators, arrays, and come ES6 – just about anything.

var foo = {
  [Symbol.iterator]: () => ({
    items: ['p', 'o', 'n', 'y', 'f', 'o', 'o'],
    next: function next () {
      return {
        done: this.items.length === 0,
        value: this.items.shift()
      }
    }
  })
}
function* multiplier (value) {
  yield value * 2
  yield value * 3
  yield value * 4
  yield value * 5
}
function* trailmix () {
  yield 0
  yield* [1, 2]
  yield* [...multiplier(2)]
  yield* multiplier(3)
  yield* foo
}
console.log([...trailmix()])
// <- [0, 1, 2, 4, 6, 8, 10, 6, 9, 12, 15, 'p', 'o', 'n', 'y', 'f', 'o', 'o']

You could also iterate the sequence by hand, calling .next(). This approach gives you the most control over the iteration, but it’s also the most involved. There’s a few features you can leverage here that give you even more control over the iteration.

Iterating Over Generators by Hand

Besides iterating over trailmix as we’ve already covered, using [...trailmix()], for value of trailmix(), and Array.from(trailmix()), we could use the generator returned by trailmix() directly, and iterate over that. But trailmix was an overcomplicated showcase of yield*, let’s go back to the side-effects generator for this one.

function* generator () {
  yield 'p'
  console.log('o')
  yield 'n'
  console.log('y')
  yield 'f'
  console.log('o')
  yield 'o'
  console.log('!')
}
var g = generator()
while (true) {
  let item = g.next()
  if (item.done) {
    break
  }
  console.log(item.value)
}

Just like we learned yesterday, any items returned by an iterator will have a done property that indicates whether the sequence has reached its end, and a value indicating the current value in the sequence.

If you’re confused as to why the '!' is printed even though there are no more yield expressions after it, that’s because g.next() doesn’t know that. The way it works is that each time its called, it executes the method until a yield expression is reached, emits its value and suspends execution. The next time g.next() is called, _execution is resumed _from where it left off (the last yield expression), until the next yield expression is reached. When no yield expression is reached, the generator returns { done: true }, signaling that the sequence has ended. At this point, the console.log('!') statement has been already executed, though.

It’s also worth noting that context is preserved across suspensions and resumptions. That means generators can be stateful. Generators are, in fact, the underlying implementation for async/await semantics coming in ES7.

Whenever .next() is called on a generator, there’s four “events” that will suspend execution in the generator, returning an IteratorResult to the caller of .next().

  • A yield expression returning the next value in the sequence
  • A return statement returning the last value in the sequence
  • A throw statement halts execution in the generator entirely
  • Reaching the end of the generator function signals { done: true }

Once the g generator ended iterating over a sequence, subsequent calls to g.next() will have no effect and just return { done: true }.

function* generator () {
  yield 'only'
}
var g = generator()
console.log(g.next())
// <- { done: false, value: 'only' }
console.log(g.next())
// <- { done: true }
console.log(g.next())
// <- { done: true }

Generators: The Weird Awesome Parts

Generator objects come with a couple more methods besides .next. These are .return and .throw. We’ve already covered .next extensively, but not quite. You could also use .next(value) to send values into the generator.

Let’s make a magic 8-ball generator. First off, you’ll need some answers. Wikipedia obliges, yielding 20 possible answers for our magic 8-ball.

var answers = [
  `It is certain`, `It is decidedly so`, `Without a doubt`,
  `Yes definitely`, `You may rely on it`, `As I see it, yes`,
  `Most likely`, `Outlook good`, `Yes`, `Signs point to yes`,
  `Reply hazy try again`, `Ask again later`, `Better not tell you now`,
  `Cannot predict now`, `Concentrate and ask again`,
  `Don't count on it`, `My reply is no`, `My sources say no`,
  `Outlook not so good`, `Very doubtful`
]
function answer () {
  return answers[Math.floor(Math.random() * answers.length)]
}

The following generator function can act as a “genie” that answers any questions you might have for them. Note how we discard the first result from g.next(). That’s because the first call to .next enters the generator and there’s no yield expression waiting to capture the value from g.next(value).

function* chat () {
  while (true) {
    let question = yield '[Genie] ' + answer()
    console.log(question)
  }
}
var g = chat()
g.next()
console.log(g.next('[Me] Will ES6 die a painful death?').value)
// <- '[Me] Will ES6 die a painful death?'
// <- '[Genie] My sources say no'
console.log(g.next('[Me] How youuu doing?').value)
// <- '[Me] How youuu doing?'
// <- '[Genie] Concentrate and ask again'

Randomly dropping g.next() feels like a very dirty coding practice, though. What else could we do? We could flip responsibilities around.

Inversion of Control

We could have the Genie be in control, and have the generator ask the questions. How would that look like? At first, you might think that the code below is unconventional, but in fact, most libraries built around generators work by inverting responsibility.

function* chat () {
  yield '[Me] Will ES6 die a painful death?'
  yield '[Me] How youuu doing?'
}
var g = chat()
while (true) {
  let question = g.next()
  if (question.done) {
    break
  }
  console.log(question.value)
  console.log('[Genie] ' + answer())
  // <- '[Me] Will ES6 die a painful death?'
  // <- '[Genie] Very doubtful'
  // <- '[Me] How youuu doing?'
  // <- '[Genie] My reply is no'
}

You would expect the generator to do the heavy lifting of an iteration, but in fact generators make it easy to iterate over things by suspending execution of themselves – and deferring the heavy lifting. That’s one of the most powerful aspects of generators. Suppose now that the iterator is a genie method in a library, like so:

function genie (questions) {
  var g = questions()
  while (true) {
    let question = g.next()
    if (question.done) {
      break
    }
    console.log(question.value)
    console.log('[Genie] ' + answer())
  }
}

To use it, all you’d have to do is pass in a simple generator like the one we just made.

genie(function* questions () {
  yield '[Me] Will ES6 die a painful death?'
  yield '[Me] How youuu doing?'
})

Compare that to the generator we had before, where questions were sent to the generator instead of the other way around. See how much more complicated the logic would have to be to achieve the same goal? Letting the library deal with the flow control means you can just worry about the thing you want to iterate over, and you can delegate how to iterate over it. But yes, it does mean your code now has an asterisk in it. Weird.

Dealing with asynchronous flows

Imagine now that the genie library gets its magic 8-ball answers from an API. How does that look then? Probably something like the snippet below. Assume the xhr pseudocode call always yields JSON responses like { answer: 'No' }. Keep in mind this is a simple example that just processes each question in series. You could put together different and more complex flow control algorithms depending on what you’re looking for.

This is just a demonstration of the sheer power of generators.

function genie (questions) {
  var g = questions()
  pull()
  function pull () {
    let question = g.next()
    if (question.done) {
      return
    }
    ask(question.value, pull)
  }
  function ask (q, next) {
    xhr('https://computer.genie/?q=' + encodeURIComponent(q), got)
    function got (err, res, body) {
      if (err) {
        // todo
      }
      console.log(q)
      console.log('[Genie] ' + body.answer)
      next()
    }
  }
}

See this link for a live demo on the Babel REPL

Even though we’ve just made our genie method asynchronous and are now using an API to fetch responses to the user’s questions, the way the consumer uses the genie library by passing a questions generator function remains unchanged! That’s awesome.

We haven’t handled the case for an err coming out of the API. That’s inconvenient. What can we do about that one?

Throwing at a Generator

Now that we’ve figured out that the most important aspect of generators is actually the control flow code that decides when to call g.next(), we can look at the other two methods and actually understand their purpose. Before shifting our thinking into “the generator defines what to iterate over, not the how, we would’ve been hard pressed to find a user case for g.throw. Now however it seems immediately obvious. The flow control that leverages a generator needs to be able to tell the generator that’s yielding the sequence to be iterated when something goes wrong processing an item in the sequence.

In the case of our genie flow, that is now using xhr, we may experience network issues and be unable to continue processing items, or we may want to warn the user about unexpected errors. Here’s how, we simply add g.throw(error) in our control flow code.

function genie (questions) {
  var g = questions()
  pull()
  function pull () {
    let question = g.next()
    if (question.done) {
      return
    }
    ask(question.value, pull)
  }
  function ask (q, next) {
    xhr('https://computer.genie/?q=' + encodeURIComponent(q), got)
    function got (err, res, body) {
      if (err) {
        g.throw(err)
      }
      console.log(q)
      console.log('[Genie] ' + body.answer)
      next()
    }
  }
}

The user code is still unchanged, though. In between yield statements it may throw errors now. You could use try/catch blocks to address those issues. If you do this, execution will be able to resume. The good thing is that this is up to the user, it’s still perfectly sequential on their end, and they can leverage try/catch semantics just like in high-school.

genie(function* questions () {
  try {
    yield '[Me] Will ES6 die a painful death?'
  } catch (e) {
    console.error('Error', e.message)
  }
  try {
    yield '[Me] How youuu doing?'
  } catch (e) {
    console.error('Error', e.message)
  }
})

Returning on Behalf of a Generator

Usually not as interesting in asynchronous control flow mechanisms in general, the g.return() method allows you to resume execution inside a generator function, much like g.throw() did moments earlier. The key difference is that g.return() won’t result in an exception at the generator level, although it will end the sequence.

function* numbers () {
  yield 1
  yield 2
  yield 3
}
var g = numbers()
console.log(g.next())
// <- { done: false, value: 1 }
console.log(g.return())
// <- { done: true }
console.log(g.next())
// <- { done: true }, as we know

You could also return a value using g.return(value), and the resulting IteratorResult will contain said value. This is equivalent to having return value somewhere in the generator function. You should be careful there though – as neither for..of, [...generator()], nor Array.from(generator()) include the value in the IteratorResult that signals { done: true }.

function* numbers () {
  yield 1
  yield 2
  return 3
  yield 4
}
console.log([...numbers()])
// <- [1, 2]
console.log(Array.from(numbers()))
// <- [1, 2]
for (let n of numbers()) {
  console.log(n)
  // <- 1
  // <- 2
}
var g = numbers()
console.log(g.next())
// <- { done: false, value: 1 }
console.log(g.next())
// <- { done: false, value: 2 }
console.log(g.next())
// <- { done: true, value: 3 }
console.log(g.next())
// <- { done: true }

Using g.return is no different in this regard, think of it as the programmatic equivalent of what we just did.

function* numbers () {
  yield 1
  yield 2
  return 3
  yield 4
}
var g = numbers()
console.log(g.next())
// <- { done: false, value: 1 }
console.log(g.return(5))
// <- { done: true, value: 5 }
console.log(g.next())
// <- { done: true }

You can avoid the impending sequence termination, as Axel points out, if the code in the generator function when g.return() got called is wrapped in try/finally. Once the yield expressions in the finally block are over, the sequence will end with the value passed to g.return(value)

function* numbers () {
  yield 1
  try {
    yield 2
  } finally {
    yield 3
    yield 4
  }
  yield 5
}
var g = numbers()
console.log(g.next())
// <- { done: false, value: 1 }
console.log(g.next())
// <- { done: false, value: 2 }
console.log(g.return(6))
// <- { done: false, value: 3 }
console.log(g.next())
// <- { done: false, value: 4 }
console.log(g.next())
// <- { done: true, value: 6 }

That’s all there is to know when it comes to generators in terms of functionality.

Use Cases for ES6 Generators

At this point in the article you should feel comfortable with the concepts of iterators, iterables, and generators in ES6. If you feel like reading more on the subject, I highly recommend you go over Axel’s article on generators, as he put together an amazing write-up on use cases for generators just a few months ago.

Liked the article? Subscribe below to get an email when new articles come out! Also, follow @ponyfoo on Twitter and @ponyfoo on Facebook.
One-click unsubscribe, anytime. Learn more.

Comments (9)

Owen Densmore wrote

Heck, while I’m asking for things: could you show module loaders at some point?

I’m loving modules (no huge batch of

I’ve heard WebPack is also a loader, and a bit concerned if we all do it differently, we’ll end up with Loader Wars!

Owen Densmore wrote

[Oops, script html in earlier post confused the forum code]

Heck, while I’m asking for things: could you show module loaders at some point?

I’m loving modules (no huge batch of script tags, yay) but the whole loader thing is pretty opaque. I’m using JSPM but in nearly complete ignorance, and not using bundling yet. I also haven’t figured out how to share my es6 modules with my es5 friends.

I’ve heard WebPack is also a loader, and a bit concerned if we all do it differently, we’ll end up with Loader Wars!

Michael wrote

Hi, nice article however ‘var’ should be replaced with either ‘let’ or ‘const’ event though var still exist. Just to follow correct pattern. Best regards.

Elen wrote

Hi Nicolas and thank you for your articles! Can I ask you a question? In the following example:

function* gen() { yield ‘test1’; yield ‘test2’; yield ‘test3’; }

const res1 = gen();

Everytime I do gen().next(); I always get Object {value: “test1”, done: false} (which is wrong,the generator should move on). Whereas if I call it like res1.next() (which is EXACTLY the same thing, since res1=gen()) it runs correctly until the end. Why is this happening? Have I misunderstood something?

Thank you in advance :)

Yash Thakkar wrote

Hi Elen,

The problem is you are calling gen() function directly where as you assigned it to “res1”. So it will call the function from top everytime you call and it will yield at ‘test1’.

You should to write res1.next() and it will give you next values till you call.

like this,

function* gen() { yield ‘test1’;yield ‘test2’;yield ‘test3’; }

const res1 = gen()

console.log(res1.next(),res1.next(), res1.next())

Hope this will clear your doubt.