PHP Fatal error: Allowed memory size of 268435456 bytes exhausted on NormalizerFormatter.php on line 62

I am trying to insert around 20000 records in mysql by doctrine2

    foreach ($getIds as $id){// 20000 Ids
            $twNode = new TwNode();
            $twNode->setTwId($id);
            $twNode->addAcFriend($this->twAccount);
            $this->twAccount->addFollower($twNode);
            $this->em->persist($twNode);
            $this->em->persist($this->twAccount);
            $this->em->flush();
            $this->output->write('W');
            $followersWrite++;
        }
    }

However this error happnes, how can I avoid this Allowed memory error?

PHP Fatal error:  Allowed memory size of 268435456 bytes exhausted (tried to allocate 72 bytes) in /Users/whitebear/httproot/myapp/vendor/monolog/monolog/src/Monolog/Formatter/NormalizerFormatter.php on line 62
PHP Stack trace:
PHP   1. {main}() /Users/whitebear/httproot/myapp/app/console:0
PHP   2. Symfony\Component\Console\Application->run() /Users/whitebear/httproot/myapp/app/console:22
PHP   3. Symfony\Bundle\FrameworkBundle\Console\Application->doRun() /Users/whitebear/httproot/myapp/vendor/symfony/symfony/src/Symfony/Component/Console/Application.php:106
PHP   4. Symfony\Component\Console\Application->doRun() /Users/whitebear/httproot/myapp/vendor/symfony/symfony/src/Symfony/Bundle/FrameworkBundle/Console/Application.php:83
PHP   5. Symfony\Component\Console\Command\Command->run() /Users/whitebear/httproot/myapp/vendor/symfony/symfony/src/Symfony/Component/Console/Application.php:200
PHP   6. Acme\TopBundle\Command\twCrawlerCommand->execute() /Users/whitebear/httproot/myapp/vendor/symfony/symfony/src/Symfony/Component/Console/Command/Command.php:242
PHP   7. Acme\TopBundle\Command\twCrawlerCommand->initialRun() /Users/whitebear/httproot/myapp/src/Acme/TopBundle/Command/twCrawlerCommand.php:430
PHP   8. EntityManager52ee6dcbdb2bb_546a8d27f194334ee012bfe64f629947b07e4919\__CG__\Doctrine\ORM\EntityManager->flush() /Users/whitebear/httproot/myapp/src/Acme/TopBundle/Command/twCrawlerCommand.php:284
PHP   9. Doctrine\ORM\EntityManager->flush() /Users/whitebear/httproot/myapp/app/cache/dev/jms_diextra/doctrine/EntityManager_52ee6dcbdb2bb.php:402
PHP  10. Doctrine\ORM\UnitOfWork->commit() /Users/whitebear/httproot/myapp/vendor/doctrine/orm/lib/Doctrine/ORM/EntityManager.php:389
PHP  11. Doctrine\ORM\UnitOfWork->executeInserts() /Users/whitebear/httproot/myapp/vendor/doctrine/orm/lib/Doctrine/ORM/UnitOfWork.php:335
PHP  12. Doctrine\ORM\Persisters\BasicEntityPersister->executeInserts() /Users/whitebear/httproot/myapp/vendor/doctrine/orm/lib/Doctrine/ORM/UnitOfWork.php:952
PHP  13. Doctrine\DBAL\Statement->execute() /Users/whitebear/httproot/myapp/vendor/doctrine/orm/lib/Doctrine/ORM/Persisters/BasicEntityPersister.php:279
PHP  14. Doctrine\DBAL\Logging\LoggerChain->startQuery() /Users/whitebear/httproot/myapp/vendor/doctrine/dbal/lib/Doctrine/DBAL/Statement.php:161
PHP  15. Symfony\Bridge\Doctrine\Logger\DbalLogger->startQuery() /Users/whitebear/httproot/myapp/vendor/doctrine/dbal/lib/Doctrine/DBAL/Logging/LoggerChain.php:54
PHP  16. Symfony\Bridge\Doctrine\Logger\DbalLogger->log() /Users/whitebear/httproot/myapp/vendor/symfony/symfony/src/Symfony/Bridge/Doctrine/Logger/DbalLogger.php:80
PHP  17. Monolog\Logger->debug() /Users/whitebear/httproot/myapp/vendor/symfony/symfony/src/Symfony/Bridge/Doctrine/Logger/DbalLogger.php:102
PHP  18. Monolog\Logger->addRecord() /Users/whitebear/httproot/myapp/vendor/monolog/monolog/src/Monolog/Logger.php:423
PHP  19. Monolog\Handler\AbstractProcessingHandler->handle() /Users/whitebear/httproot/myapp/vendor/monolog/monolog/src/Monolog/Logger.php:244
PHP  20. Monolog\Formatter\LineFormatter->format() /Users/whitebear/httproot/myapp/vendor/monolog/monolog/src/Monolog/Handler/AbstractProcessingHandler.php:35
PHP  21. Monolog\Formatter\NormalizerFormatter->format() /Users/whitebear/httproot/myapp/vendor/monolog/monolog/src/Monolog/Formatter/LineFormatter.php:43
PHP  22. Monolog\Formatter\LineFormatter->normalize() /Users/whitebear/httproot/myapp/vendor/monolog/monolog/src/Monolog/Formatter/NormalizerFormatter.php:40
PHP  23. Monolog\Formatter\NormalizerFormatter->normalize() /Users/whitebear/httproot/myapp/vendor/monolog/monolog/src/Monolog/Formatter/LineFormatter.php:88
PHP  24. Monolog\Formatter\LineFormatter->normalize() /Users/whitebear/httproot/myapp/vendor/monolog/monolog/src/Monolog/Formatter/NormalizerFormatter.php:70
PHP  25. Monolog\Formatter\NormalizerFormatter->normalize() /Users/whitebear/httproot/myapp/vendor/monolog/monolog/src/Monolog/Formatter/LineFormatter.php:88

Fatal error: Allowed memory size of 268435456 bytes exhausted (tried to allocate 72 bytes) in /Users/whitebear/httproot/myapp/vendor/monolog/monolog/src/Monolog/Formatter/NormalizerFormatter.php on line 62

Call Stack:
    0.0002     229968   1. {main}() /Users/whitebear/httproot/myapp/app/console:0
    0.0126    2063376   2. Symfony\Component\Console\Application->run() /Users/whitebear/httproot/myapp/app/console:22
    0.0144    2256968   3. Symfony\Bundle\FrameworkBundle\Console\Application->doRun() /Users/whitebear/httproot/myapp/vendor/symfony/symfony/src/Symfony/Component/Console/Application.php:106
    0.1503   11751600   4. Symfony\Component\Console\Application->doRun() /Users/whitebear/httproot/myapp/vendor/symfony/symfony/src/Symfony/Bundle/FrameworkBundle/Console/Application.php:83
    0.1518   11753384   5. Symfony\Component\Console\Command\Command->run() /Users/whitebear/httproot/myapp/vendor/symfony/symfony/src/Symfony/Component/Console/Application.php:200
    0.1521   11758040   6. Acme\TopBundle\Command\twCrawlerCommand->execute() /Users/whitebear/httproot/myapp/vendor/symfony/symfony/src/Symfony/Component/Console/Command/Command.php:242
    0.2409   20673112   7. Acme\TopBundle\Command\twCrawlerCommand->initialRun() /Users/whitebear/httproot/myapp/src/Acme/TopBundle/Command/twCrawlerCommand.php:430
  101.5883  235069104   8. EntityManager52ee6dcbdb2bb_546a8d27f194334ee012bfe64f629947b07e4919\__CG__\Doctrine\ORM\EntityManager->flush() /Users/whitebear/httproot/myapp/src/Acme/TopBundle/Command/twCrawlerCommand.php:284
  101.5883  235069200   9. Doctrine\ORM\EntityManager->flush() /Users/whitebear/httproot/myapp/app/cache/dev/jms_diextra/doctrine/EntityManager_52ee6dcbdb2bb.php:402
  101.5883  235069280  10. Doctrine\ORM\UnitOfWork->commit() /Users/whitebear/httproot/myapp/vendor/doctrine/orm/lib/Doctrine/ORM/EntityManager.php:389
  107.0744  259211408  11. Doctrine\ORM\UnitOfWork->executeInserts() /Users/whitebear/httproot/myapp/vendor/doctrine/orm/lib/Doctrine/ORM/UnitOfWork.php:335
  107.1128  259277872  12. Doctrine\ORM\Persisters\BasicEntityPersister->executeInserts() /Users/whitebear/httproot/myapp/vendor/doctrine/orm/lib/Doctrine/ORM/UnitOfWork.php:952
  109.1604  267877360  13. Doctrine\DBAL\Statement->execute() /Users/whitebear/httproot/myapp/vendor/doctrine/orm/lib/Doctrine/ORM/Persisters/BasicEntityPersister.php:279
  109.1604  267877408  14. Doctrine\DBAL\Logging\LoggerChain->startQuery() /Users/whitebear/httproot/myapp/vendor/doctrine/dbal/lib/Doctrine/DBAL/Statement.php:161
  109.1604  267877408  15. Symfony\Bridge\Doctrine\Logger\DbalLogger->startQuery() /Users/whitebear/httproot/myapp/vendor/doctrine/dbal/lib/Doctrine/DBAL/Logging/LoggerChain.php:54
  109.1604  267877592  16. Symfony\Bridge\Doctrine\Logger\DbalLogger->log() /Users/whitebear/httproot/myapp/vendor/symfony/symfony/src/Symfony/Bridge/Doctrine/Logger/DbalLogger.php:80
  109.1604  267877592  17. Monolog\Logger->debug() /Users/whitebear/httproot/myapp/vendor/symfony/symfony/src/Symfony/Bridge/Doctrine/Logger/DbalLogger.php:102
  109.1604  267877640  18. Monolog\Logger->addRecord() /Users/whitebear/httproot/myapp/vendor/monolog/monolog/src/Monolog/Logger.php:423
  109.1604  267878968  19. Monolog\Handler\AbstractProcessingHandler->handle() /Users/whitebear/httproot/myapp/vendor/monolog/monolog/src/Monolog/Logger.php:244
  109.1604  267878968  20. Monolog\Formatter\LineFormatter->format() /Users/whitebear/httproot/myapp/vendor/monolog/monolog/src/Monolog/Handler/AbstractProcessingHandler.php:35
  109.1604  267878968  21. Monolog\Formatter\NormalizerFormatter->format() /Users/whitebear/httproot/myapp/vendor/monolog/monolog/src/Monolog/Formatter/LineFormatter.php:43
  109.1604  267878968  22. Monolog\Formatter\LineFormatter->normalize() /Users/whitebear/httproot/myapp/vendor/monolog/monolog/src/Monolog/Formatter/NormalizerFormatter.php:40
  109.1604  267878968  23. Monolog\Formatter\NormalizerFormatter->normalize() /Users/whitebear/httproot/myapp/vendor/monolog/monolog/src/Monolog/Formatter/LineFormatter.php:88
  109.1605  267881872  24. Monolog\Formatter\LineFormatter->normalize() /Users/whitebear/httproot/myapp/vendor/monolog/monolog/src/Monolog/Formatter/NormalizerFormatter.php:70
  109.1605  267881872  25. Monolog\Formatter\NormalizerFormatter->normalize() /Users/whitebear/httproot/myapp/vendor/monolog/monolog/src/Monolog/Formatter/LineFormatter.php:88

Thanks to your comment, I understood it is relavant with the memory comsumed by $this->em

so, I put the codes to unset $this->em every 5000 times;

if ($k > 1000){
    $this->em->flush();
    unset($this->em);
    $this->em = $this->getContainer()->get('doctrine')->getManager();
}

but still in vain, I think I should unset the origin of $this->getContainer()->get('doctrine')->getManager();

is it possible? or am I right?

Answers


You may want to read Doctrine 2 Batch Processing. They are using this pattern:

// $em instanceof EntityManager
$batchSize = 20;
for ($i=1; $i<=10000; ++$i) {
     $obj = new MyEntity;
     $obj->setFoo('...');
     // ... set more data
     $em->persist($obj);
     if (($i % $batchSize) == 0) {
         $em->flush();
         $em->clear();
    }
}
$em->flush();

As I understand it - after persisting, doctrine holds all mapped objects in memory in identity map. So memory requirements grow all the time with every new object. If you do clear(), it will detach all managed entities and if you have no references to them, they will be garbage-collected.

Beware: it will detach all other entities you are using. Or you can detach only those inserted objects $em->detach($twNode); // detach from Doctrine, so that it can be GC'd immediately

Edit: added flush() after for loop to ensure saving all entities even when entity count is not divisible by batch size, as suggested in comment


You can avoid this by adding the following on the top of your script

set_time_limit(6000); 
ini_set("memory_limit", -1);

But the real issue should be fixed in the coding by adding data in chunks which is better in terms of performance.


Three approaches...

  1. Increase the amount of allowed memory ini_set("memory_limit", "512M"); or in the php.ini file itself

  2. Break your process into multiple chunks of, say, 5000 a piece.

  3. Figure out where the memory is going and try to free it using unset or by setting objects to null.

A good first place to look where your memory is going is probably here

101.5883  235069104   8. EntityManager52ee6dcbdb2bb_546a8d27f194334ee012bfe64f629947b07e4919\__CG__\Doctrine\ORM\EntityManager->flush() /Users/whitebear/httproot/myapp/src/Acme/TopBundle/Command/twCrawlerCommand.php:284

As the amount of memory jumps by about 200MB here.


Need Your Help

Angular Material md-list flexing

css angularjs angular-material

I want to use Angular Material's md-list for showing elements, using responsive design. And my task is to show these items in 1 column if media device is small and in 3 columns if it's larger than ...

add two double given wrong result

c# .net floating-point double addition

I'm using the following piece of code and under some mysterious circumstances the result of the addition is not as it's supposed to be: