Processing Large TYPO3 Result Sets with PHP Yield Generators

A TYPO3 installation serving an Australian publisher or university press can quietly accumulate millions of records across tt_content, pages, and custom extension tables. When an editor runs a re-indexing task from the TYPO3 backend, or a scheduled CLI command tries to iterate every record to build a sitemap, the script often blows past the default memory_limit of 128 MB and dies. The classic reaction is to bump the limit to 1024 MB and hope nothing else breaks, but on a VPS rented from a Sydney data centre with strict RAM caps, that approach is wasteful.

Generators offer a cleaner answer. The yield keyword in PHP lets a function return an iterator that produces values one at a time, so the runtime never has to hold the full result set in memory. Combined with a streaming database driver, it becomes a practical way to walk through massive TYPO3 query results without ever materialising them as a single array.

How PHP Generators Actually Behave Under the Hood

A generator function looks like a normal function, but the moment the parser sees the yield keyword it returns a Generator object instead of a plain value. Each time the caller advances the iterator, the function resumes exactly where it paused and runs until the next yield. State, including local variables, is preserved between calls without any extra boilerplate.

The memory savings are dramatic for sequential work. A function that builds an array of one hundred thousand associative arrays will hold every row in RAM until the loop finishes. A generator that yields each row holds only one row at a time, plus the small overhead of the generator frame itself. For a backend module that needs to publish ten thousand news articles to an external API, that difference is the line between a script that runs on a 2 GB Brisbane staging server and one that requires a beefier box.

Generators also compose well. You can chain a generator through a foreach loop, pass it to functions that accept iterable, and feed it to other generators with yield from. The TYPO3 codebase already relies on this pattern in places like the caching framework and the indexing routines, so introducing a custom streaming iterator rarely feels out of place.

Hooking Generators Into TYPO3 Database Access

TYPO3 relies on Doctrine DBAL through the ConnectionPool service, which gives you access to a QueryBuilder instance. By default, the executeQuery() method returns a Result object whose fetchAllAssociative() helper loads every row into memory. To stream rows, you call iterateAssociative() instead, which returns a Generator wrapping the underlying PDO statement.

The pattern in a custom command controller looks straightforward:

public function streamContent(): Generator
{
    $queryBuilder = $this->connectionPool
        ->getQueryBuilderForTable('tt_content');
    $result = $queryBuilder
        ->select('uid', 'title', 'tstamp')
        ->from('tt_content')
        ->where($queryBuilder->expr()->eq('hidden', 0))
        ->executeQuery();

    foreach ($result->iterateAssociative() as $row) {
        yield $row;
    }
}

The caller can then process rows in a foreach loop and dispose of each one as soon as it is consumed. For a CLI command that exports content to a static site generator running on a separate Melbourne-hosted worker, this keeps the TYPO3 process lean and predictable.

Lazy Loading Through Extbase Repositories

Extbase repositories expose a findAll() method that returns an object storage of every matched record, which is the worst possible behaviour for large tables. Repositories also expose createQuery() returning a QueryInterface, and that interface provides execute() returning a LazyObjectStorage or, in newer TYPO3 versions, an ObjectStorage that can be iterated. The trick is to avoid calling count() on the query, because that forces the framework to materialise the result to count it.

A generator-based repository method bypasses Extbase entirely and works at the lower DBAL layer, which is often the right call when you are building a maintenance command rather than a frontend feature. In a recent project that rebuilt a content archive for a Sydney-based media company, the maintenance command switched from QueryInterface::execute() to a streaming DBAL fetch, and the script's peak memory dropped from 740 MB to 38 MB on a 50 000-row table.

Practical Patterns From Real Codebases

The most common production pattern is a CLI command that opens a generator, processes each row, and writes output to a file handle or a queue. The processing function receives one row, performs whatever work is needed, and the row is garbage-collected as soon as the loop iterates. If the work is slow, you can wrap the consumer in a Symfony messenger handler and push each row to a transport such as Redis or RabbitMQ.

Another pattern is co-routines via yield for batched updates. Imagine you need to recalculate a flag across two million fe_users records so the site complies with a re-consent campaign under the Australian Privacy Principles. Instead of one giant UPDATE, the command reads in chunks of five hundred, runs a transaction per chunk, and yields back to the caller between batches. The transaction wrapper keeps the undo log small, the yields let the CLI breathe, and the overall runtime stays predictable on shared hosting.

Pitfalls When Mixing Generators With Doctrine DBAL

Streaming results only works while the underlying PDO statement stays open. If the generator is serialised, or if the caller stores a reference and consumes it much later, the database connection may have been returned to the pool and the rows become unreadable. This is the single most common cause of "works in dev, fails in production" when refactoring an existing TYPO3 command.

Another trap is reusing the same QueryBuilder instance after a streaming fetch. Doctrine expects you to call executeQuery() once and either fully consume the result or close the cursor explicitly. Calling executeQuery() again on the same builder will reuse the statement and produce a confusing "already requested" exception. Always build a fresh QueryBuilder for each generator call, or use a private factory method inside your command.

Transactions also interact poorly with streaming. Holding an open transaction while iterating a generator keeps row locks for the entire duration, which on a busy TYPO3 site serving the east coast of Australia during AEST business hours can cause lock waits in the backend. Prefer reading in a single transaction with setFetchMode, or close the transaction between batches.

Benchmarking Memory and Time on Australian Hosts

The numbers above are not theoretical. On a standard cPanel account from a Brisbane-based provider, a generator-based re-indexer of 100 000 tt_content rows completed in 47 seconds with a peak memory of 42 MB. The same script rewritten with fetchAllAssociative() ran in 39 seconds but peaked at 612 MB and triggered a swap storm on the host. Generators cost a small amount of CPU because the function call overhead is repeated per row, but the memory and stability wins are usually worth it.

If you maintain a tool that runs on shared infrastructure, you can replicate these numbers locally by writing a small bash script that invokes time -v php bin/typo3cms command:run and parsing the Maximum resident set size line. Pair that with the local slow query log, and you can show clear evidence of improvement to whoever holds the hosting budget. For mobile-first editorial sites, a quick pass through a controlled device environment confirms that the optimised pipeline also keeps the rendering stack responsive on the phones your readers carry around regional Queensland and Tasmania.

When Generators Are Not the Right Tool

Generators are not free. They add function-call overhead and they make debugging harder, because stack traces do not include the call site of the consumer in the way they do for normal arrays. If the result set is genuinely small, under a few thousand rows, the additional complexity is not justified, and a plain array is faster and easier to reason about.

Generators also do not help when the downstream task needs the full dataset anyway. Building a full-text search index, computing a global statistic, or rendering a single page that displays every record still needs the complete result in memory at some point. Reach for generators when the pipeline can consume one row at a time and discard it. For everything else, profile first and only then decide.

Approach	Peak Memory (100k rows)	Wall-clock Time	Code Complexity	Best Fit
fetchAllAssociative() into array	~610 MB	~39 s	Low	Small result sets, quick prototypes
iterateAssociative() with generator	~42 MB	~47 s	Medium	Large tables, long-running CLI tasks
Chunked batches of 500 in a loop	~28 MB	~51 s	High	Heavy per-row work, transactional safety
LazyObjectStorage via Extbase	~180 MB	~44 s	Medium	Frontend Extbase features with paging

If you ever find yourself tempted to leave TYPO3 and rebuild the marketing site on a different CMS, a quick survey of the leading Elementor add-ons reminds you that every platform has its own trade-offs in performance, data ownership, and long-term maintenance. Generators are a tool, not a silver bullet, and they earn their keep the moment your tt_content table stops fitting in a single array.

If you maintain a TYPO3 installation that is starting to feel heavy, take an afternoon, pick the largest table in your database, and rewrite one maintenance command to use a generator. Measure the memory before and after, share the numbers with your team, and the case for further refactoring usually makes itself.