Generators - Ivan Habunek

# Generators

### Lazy, iterative goodness

Ivan Habunek

ZgPHP Meetup #46, Zagreb, 18.06.2015

---

## Ivan Habunek

### @ihabunek

***Big Fish Software*** Sole developer and owner

***WebCamp Zagreb*** Co-organizer

***ZgPHP meetups*** Co-organizer

Open source enthusiast

---

---

# Iterators

---

```php
interface Iterator extends Traversable
{
    abstract public current()
    abstract public key()
    abstract public next()
    abstract public rewind()
    abstract public valid()
}
```

---

```php
interface Iterator extends Traversable
{
*   abstract public current()
    abstract public key()
    abstract public next()
    abstract public rewind()
    abstract public valid()
}
```

**current()**

Returns the current element.

---

```php
interface Iterator extends Traversable
{
    abstract public current()
*   abstract public key()
    abstract public next()
    abstract public rewind()
    abstract public valid()
}
```

**key()**

Returns the key of the current element.

---

```php
interface Iterator extends Traversable
{
    abstract public current()
    abstract public key()
*   abstract public next()
    abstract public rewind()
    abstract public valid()
}
```

**next()**

Move forward to the next element.

---

```php
interface Iterator extends Traversable
{
    abstract public current()
    abstract public key()
    abstract public next()
*   abstract public rewind()
    abstract public valid()
}
```

**rewind()**

Rewind the Iterator to the first element.

---

```php
interface Iterator extends Traversable
{
    abstract public current()
    abstract public key()
    abstract public next()
    abstract public rewind()
*   abstract public valid()
}
```

**valid()**

Checks if current position is valid.

---

# Use case #1: Reading file data

---

### Approach #1: Read file into memory

```php
$lines = file('input.txt');
foreach ($lines as $line) {
    processLine($line); // Does some work
}
```

---

### Approach #1: Read file into memory

```php
function getLines($file)
{
    return file($file);
}

function processData($lines)
{
    foreach ($lines as $line) {
        processLine($line);
    }
}
```

```php
$lines = getLines('input.txt');

processData($lines);
```

---

### Approach #1: Read file into memory

---

### Approach #2: Iterate over file

```php
$fp = fopen('input.txt', 'r');
while ($line = $fp->gets()) {
    processLine($line);
}
fclose($fp);
```

---

### Approach #2: Iterate over file

---

## Solution: use iterators

---

## Approach #3: File iterator

/** File pointer. */
    protected $fp;

/** Current line (value) */
    protected $line;

/** Current line number (key) */
    protected $no;

public function __construct($file) {
        $this->fp = fopen($file, 'r');
    }

public function __destruct() {
        fclose($this->fp);
    }

public function current() {
        return $this->line;
    }
```
]]

public function key() {
        return $this->no;
    }

public function next() {
        $this->line = fgets($this->fp);
        $this->no++;
    }

public function rewind() {
        fseek($this->fp, 0);
        $this->line = fgets($this->fp);
        $this->no = 0;
    }

public function valid() {
        return false !== $this->line;
    }
}
```
]]

---

## Approach #3: File iterator

```php
class FileIterator implements Iterator {

/** File pointer. */
    protected $fp;

/** Current line (value) */
    protected $line;

/** Current line number (key) */
    protected $no;
}
```

---

## Approach #3: File iterator

```php
class FileIterator implements Iterator {

public function __construct($file) {
        $this->fp = fopen($file, 'r');
    }

public function __destruct() {
        fclose($this->fp);
    }
}
```

---

## Approach #3: File iterator

```php
class FileIterator implements Iterator {

public function next() {
        $this->line = fgets($this->fp);
        $this->no++;
    }

public function rewind() {
        fseek($this->fp, 0);
        $this->line = fgets($this->fp);
        $this->no = 0;
    }

public function valid() {
        return false !== $this->line;
    }
}
```

---

## Approach #3: File iterator

```php
class FileIterator implements Iterator {

public function current() {
        return $this->line;
    }

public function key() {
        return $this->no;
    }
}
```

---

## Approach #3: File iterator

```php
$it = new FileIterator($file);
foreach ($it as $line) {
    processLine($line);
}
```

---

## Approach #3: File iterator

```php
// Data generator
function getLines($file)
{
    return new FileIterator($file);
}

// Data consumer
function processLines(Iterator $lineIterator)
{
    foreach ($lineIterator as $line) {
        processLine($line);
    }
}
```

---

### Don't reinvent the wheel

---

### SplFileInfo & SplFileObject

```php
$fileInfo = new SplFileInfo(__FILE__);
$fileObject = $fileInfo->openFile('r');

foreach ($fileObject as $line) {
    print_r($line);
}

```

---

## The thing with interators...

## ... so much code.

---

## Introducing generators

```php
function generator() {
    yield 1;
    yield 2;
    yield 3;
}
```

```php
$gen = generator();
foreach ($gen as $item) {
    echo $item;
}
```

```sh
123
```

---

## Introducing generators

```php
function generator() {
    yield 1;
    yield 2;
*   return;
    yield 3;
}
```

---

## Introducing generators

```php
function generator() {
    yield 1;
    yield 2;
*   return 4;
    yield 3;
}
```

```sh
Fatal error:
Generators cannot return values using "return"
```

---

## Introducing generators

```php
function getLines($file)
{
    $fp = fopen($file, 'r');

while ($line = fgets($fp)) {
        yield $line;
    }

fclose($fp);
}
```

---

## Introducing generators

```php
function getLines($file)
{
    $fp = fopen($file, 'r');

$no = 0;
    while ($line = $fp->gets()) {
*       yield $no => $line;
        $no += 1;
    }

fclose($fp);
}
```

---

## Introducing generators

```php
$lineGenerator = getLines($file);
foreach ($lineGenerator as $line) {
    processLine($line);
}
```

---

## Syntax sugar, nothing more

---

## Generator advantages

- simpler than iterators
- faster than iterators*
- easy to mock

\* .text16[https://gist.github.com/nikic/2975796]

---

## Generator disadvantages

- can be traversed only once
- cannot be serialized
- cannot be indexed (`ArrayAccess`)
- cannot be used with `array_map` or `array_filter`

---

## Reverse generators

```php
function logger($file)
{
    $fp = fopen($file, 'a');

while (true) {
*       $line = yield;
        $fp->write($line);
    }
}
```

```php
$w = logger('log.txt');
$w->send("foo");
$w->send("bar");
$w->send("baz");
```

---

# Real world example

## API returning a lot of JSON data

---

### Sub-optimal approach

```php
$data = $database->loadData();
$json = json_encode($data);
echo $json;
```

---

### Generator #1: Read data from database and yield rows.

```php
function getRows(PDO $pdo)
{
    $sql = "SELECT * FROM branche";

$stmt = $pdo->prepare($sql);
    $stmt->execute();

while ($row = $stmt->fetch()) {
        yield $row;
    }
}
```
---

### Generator #1: Read data from database and yield rows.

```php
function getRows(PDO $pdo)
{
    $sql = "SELECT * FROM branche";

$stmt = $pdo->prepare($sql);
    $stmt->execute();

*   while ($row = $stmt->fetch()) {
*       yield $row;
*   }
}
```

---

### ~~Generator~~ #1: Read data from database and yield rows.

```php
function getRows(PDO $pdo)
{
    $sql = "SELECT * FROM branche";

$stmt = $pdo->prepare($sql);
    $stmt->execute();

return $stmt;
}
```

---

### ~~Generator~~ #1: Read data from database and yield rows.

```code16
[
    "id" => "1",
    "brancheTyp_id" => "branche",
    "kategorie_id" => null,
    "topBranche" => "0",
    "numCards" => "409",
    "startChar" => "T",
    "name" => "Textilwaren und Bekleidung Einzelhandel",
    "linkName" => "textilwaren-und-bekleidung-einzelhandel",
    "alternativName" => null,
    "mediaVersion_id" => null,
    "mhp_id" => null,
    "beschreibung" => null,
    "created" => "2008-03-03 13:46:50",
    "lastmodify" => "2015-03-10 13:24:40",
    "login_id" => "465"
]
```

---

### Generator #2: Iterates over rows, and yields API models.

```php
function getModels(Iterator $rows)
{
    foreach ($rows as $row) {
        yield [
            'id'   => $row['id'],
            'name' => $row['name'],
            'slug' => $row['linkName'],
        ];
    }
}
```

---

### Generator #2: Iterates over rows, and yields API models.

```code20
[
  'id' => '1',
  'name' => 'Textilwaren und Bekleidung Einzelhandel',
  'slug' => 'textilwaren-und-bekleidung-einzelhandel',
]
```

---

### Generator #3: Iterates over models, and yields JSON chunks.

```php
function getJson(Iterator $models)
{
    yield "[";

$first = true;
    foreach ($models as $model) {
        if (!$first) yield ",";
        $first = false;

yield json_encode($model);
    }

yield "]";
}

```

---

```code16
[
```

```code16
{
    "id": "1",
    "name": "Textilwaren und Bekleidung Einzelhandel",
    "slug": "textilwaren-und-bekleidung-einzelhandel"
}
```

```code16
,
```

```code16
{
    "id": "2",
    "name": "Textilwaren Einzelhandel",
    "slug": "textilwaren-einzelhandel"
}
```

```code16
,
```

```code16
{
    "id": "3",
    "name": "Gest\u00fcte",
    "slug": "gestuete"
}
```

```code16
]
```

---

### Usage

```code20
// Database
$pdo = new PDO("mysql:host=localhost;dbname=sandbox", "ihabunek");
$pdo->setAttribute(PDO::MYSQL_ATTR_USE_BUFFERED_QUERY, 0);

// Data wrangling
$rows = getRows($pdo);
$models = getModels($rows);
$json = getJson($models);

// Echo the data
foreach ($json as $chunk) {
    echo $chunk;
}

flush();
```

---

### Usage

```code20
// Database
$pdo = new PDO("mysql:host=localhost;dbname=sandbox", "ihabunek");
*$pdo->setAttribute(PDO::MYSQL_ATTR_USE_BUFFERED_QUERY, 0);

// Data wrangling
$rows = getRows($pdo);
$models = getModels($rows);
$json = getJson($models);

// Echo the data
foreach ($json as $chunk) {
    echo $chunk;
}

flush();
```

---

### Usage

```code20
// Database
$pdo = new PDO("mysql:host=localhost;dbname=sandbox", "ihabunek");
$pdo->setAttribute(PDO::MYSQL_ATTR_USE_BUFFERED_QUERY, 0);

// Data wrangling
*$rows = getRows($pdo);
$models = getModels($rows);
$json = getJson($models);

// Echo the data
foreach ($json as $chunk) {
    echo $chunk;
}

flush();
```

---

### Usage

```code20
// Database
$pdo = new PDO("mysql:host=localhost;dbname=sandbox", "ihabunek");
$pdo->setAttribute(PDO::MYSQL_ATTR_USE_BUFFERED_QUERY, 0);

// Data wrangling
$rows = getRows($pdo);
*$models = getModels($rows);
$json = getJson($models);

// Echo the data
foreach ($json as $chunk) {
    echo $chunk;
}

flush();
```

---

### Usage

```code20
// Database
$pdo = new PDO("mysql:host=localhost;dbname=sandbox", "ihabunek");
$pdo->setAttribute(PDO::MYSQL_ATTR_USE_BUFFERED_QUERY, 0);

// Data wrangling
$rows = getRows($pdo);
$models = getModels($rows);
*$json = getJson($models);

// Echo the data
foreach ($json as $chunk) {
    echo $chunk;
}

flush();
```

---

### Usage

```code20
// Database
$pdo = new PDO("mysql:host=localhost;dbname=sandbox", "ihabunek");
$pdo->setAttribute(PDO::MYSQL_ATTR_USE_BUFFERED_QUERY, 0);

// Data wrangling
$rows = getRows($pdo);
$models = getModels($rows);
$json = getJson($models);

// Echo the data
*foreach ($json as $chunk) {
*   echo $chunk;
*}

flush();
```

---

### More realistic usage

```code20
// Database
$pdo = new PDO("mysql:host=localhost;dbname=sandbox", "ihabunek");
$pdo->setAttribute(PDO::MYSQL_ATTR_USE_BUFFERED_QUERY, 0);

// Data wrangling
$rows = getRows($pdo);
$models = getModels($rows);
$json = getJson($models);

// Create a streamed response
*$response = new Symfony\Component\HttpFoundation\StreamedResponse();
*$response->setCallback(function () use ($json) {
*   foreach ($json as $chunk) {
*       echo $chunk;
*   }
*   flush();
*});

// Send
$response->send();

```
---

# Real world example

## Infinite series

---

# Real world example

## Infinite(ish) series

---

### Range function

```php
array range (
    mixed $start,
    mixed $end,
    number $step = 1
)
```

---

### Sub-optimal approach

```php
$range = range(0, 1E6);
foreach ($range as $number) {
    // Do something
}
```

---

### Sub-optimal approach

```php
$range = range(0, 1E6);
foreach ($range as $number) {
    // Do something
}
```

Memory usage (PHP 5.6): 68MB

---

### Sub-optimal approach

```php
$range = range(0, 1E6);
foreach ($range as $number) {
    // Do something
}
```

Memory usage (PHP 5.6): 68MB

Memory usage (PHP 7.0a1): 21MB

---

### Reimplement range as an Iterator

.left-column[.text16[
```php
class XRange implements Iterator
{
    private $max;
    private $min;
    private $position;
    private $step;

public function __construct(
        $min, $max, $step = 1)
    {
        $this->max = $max;
        $this->min = $min;
        $this->position = $min;
        $this->step = $step;
    }

function rewind()
    {
        $this->position = $this->min;
    }

function current()
    {
        return $this->position;
    }

```
]]

function key()
    {
        return $this->position;
    }

function next()
    {
        $this->position += $this->step;
    }

function valid()
    {
        return $this->position = $this->max;
    }
}

.
```
]]

---

### Reimplement range using a generator

```php
function xrange($min, $max, $step = 1)
{
    for ($i = $min; $i <= $max; $i += $step) {
        yield $i;
    }
}
```

---

### Reimplement range using a generator

```php
function xrange($min, $max, $step = 1)
{
    for ($i = $min; $i <= $max; $i += $step) {
        yield $i;
    }
}
```

Limited to PHP_INT_MAX (2,147,483,647)

---

### We can go further

```php
function xrange($min, $max, $step = 1)
{
    for (
        $i = $min;
        bccmop($i, $max) < 0;
        $i = bcadd($i, $step)
    ) {
        yield $i;
    }
}
```

---

### We can go further

```php
function xrange($min, $max, $step = 1)
{
    for (
        $i = $min;
        bccmop($i, $max) < 0;
        $i = bcadd($i, $step)
    ) {
        yield $i;
    }
}
```

Limited to your memory.

---

# Iterator functions

---

### iterator_apply — Call a function for every element in an iterator

```php
$fun = function (Generator $gen) {
    $item = $gen->current();
    // Do something with $item
    return true;
};

$gen = xrange(1, 100);
iterator_apply($gen, $fun, [$gen]);
```

* equivalent to `array_walk()`

---

### iterator_count — Count the elements in an iterators

```php
$gen = xrange(1, 100);

echo iterator_count($gen);
```

```sh
100
```

---

### iterator_to_array — Copy the iterator into an array

```php
$gen = xrange(1, 3);

$array = iterator_to_array($gen);

print_r($array);
```

```sh
Array
(
    [0] => 1
    [1] => 2
    [2] => 3
)
```

---

# nikic/iter

## Iteration primitives using generators

---

### range

```php
use function iter\range;

range(0, 5);
// 0, 1, 2, 3, 4, 5

range(5, 0);
// 5, 4, 3, 2, 1, 0

range(0.0, 3.0, 0.5);
// 0.0, 0.5, 1.0, 1.5, 2.0, 2.5, 3.0
```

---

### take

```php
take(5, range(0, PHP_INT_MAX));
// 0, 1, 2, 3, 4, 5
```

---

### map

```php
use function iter\fn\operator;
use function iter\map;

map(operator('*', 2), [1, 2, 3, 4, 5]);
// 2, 4, 6, 8, 10
```

---

### filter

```php
use function iter\filter;
use function iter\fn\operator;

filter(operator('<', 0), [0, -1, -10, 7, 20, -5, 7]);
// -1, -10, -5
```

---

### reduce

```php
use function iter\filter;
use function iter\fn\operator;

echo reduce(operator('+'), range(1, 5), 0);
// 15

echo reduce(operator('*'), range(1, 5), 1);
 // 120
```

---

### reindex

```code20
use function iter\fn\index;
use function iter\reindex;

$input = [
    ['id' => 10, 'name' => 'Maja'],
    ['id' => 27, 'name' => 'Ivan'],
    ['id' => 39, 'name' => 'Pero'],
    ['id' => 42, 'name' => 'Luna'],
];

*reindex(index('id'), $input);
```

```code20
[
    10 => ['id' => 10, 'name' => 'Maja'],
    27 => ['id' => 27, 'name' => 'Ivan'],
    39 => ['id' => 39, 'name' => 'Pero'],
    42 => ['id' => 42, 'name' => 'Luna'],
];
```

---

# Questions?

## Rate my talk!

https://joind.in/14684