流明数据库队列第一个作业总是失败允许内存耗尽

Question

我有一个非常奇怪的情况，我在我的 Lumen 数据库队列中设置了一个运行的作业，并且除了第一个作业之外的所有作业都被处理了。我确实不断收到此特定错误：

[2017-12-12 22:07:10] lumen.ERROR: Symfony\Component\Debug\Exception\FatalErrorException: Allowed memory size of 1073741824 bytes exhausted (tried to allocate 702558208 bytes) in /var/www/vhosts/XXXXXXXXX$
Stack trace:
#0 /var/www/vhosts/XXXXXXXX/vendor/laravel/lumen-framework/src/Concerns/RegistersExceptionHandlers.php(54): Laravel\Lumen\Application->handleShutdown()
#1 [internal function]: Laravel\Lumen\Application->Laravel\Lumen\Concerns\{closure}()
#2 {main}

我已经尝试允许内存限制上升，但我不断收到相同的错误，但内存耗尽的值不同。

我觉得很奇怪，它总是第一份工作，而所有其他工作运行都很好。我应该在第一份工作中寻找不良数据吗？

我的代码基本上是这样的：

这是我的命令文件

namespace App\Console\Commands;

use App\Jobs\UpdateNNNAppListJob;
use Illuminate\Console\Command;
use App\Services\MiddlewareApi;
use Illuminate\Support\Facades\DB;
use Illuminate\Support\Facades\Log;
use Mockery\Exception;
use Illuminate\Support\Facades\Queue;

class AddEmailsToAppList extends Command
{
    /**
     * The name and signature of the console command.
     *
     * @var string
     */
    protected $signature = 'addemails:nnnmobileapp';

    /**
     * The console command description.
     *
     * @var string
     */
    protected $description = 'This will add all mobile app users in the database to the nnn mobile app list.';

    /**
     * Create a new command instance.
     *
     * @return void
     */
    public function __construct()
    {
        parent::__construct();
    }

    public function handle()
    {
        $chunkSize = 500; //this is the most middleware can handle with its bulk signup call
        $emailChunks = $this->getEmailsToAdd($chunkSize);
        $jobDelay = 120; //time between queued jobs
        $jobDelayTimeKeeper = 60; //This will be the actual time delay that will be put into the later method

        foreach ($emailChunks as $emailChunk) {
            Queue::later($jobDelayTimeKeeper, new UpdateMmpAppListJob($emailChunk));
            $jobDelayTimeKeeper = $jobDelayTimeKeeper + $jobDelay;
        }
    }

    public function getEmailsToAdd($chunkSize)
    {
        $emails = DB::table('app_users')
            ->join('app_datas', 'app_datas.customer_number', '=', 'app_users.customer_number')
            ->select('app_users.email')
            ->get()
            ->chunk($chunkSize);

        return $emails;
    }
}

这是我的工作文件

<?php
namespace App\Jobs;

use App\Services\MiddlewareApi;
use Illuminate\Support\Facades\Log;
use Mockery\Exception;

class UpdateMmpAppListJob extends Job
{

    /**
     * Array of emails to update list with
     * @var array
     */
    protected $emailArray;

    /**
     * The number of times the job may be attempted.
     *
     * @var int
     */
    public $tries = 2;

    public function __construct($emailArray)
    {
        $this->emailArray = $emailArray;
    }

    public function handle()
    {
        $listCodeToAddTo = 'NNNAPP';
        $sourceId = 'NNNNNNN';

        $middlewareApi = new MiddlewareApi();

        try {
            $middlewareApi->post_add_customer_signup_bulk($listCodeToAddTo, $this->emailArray, $sourceId);
        } catch (\Exception $e) {
            Log::error('An error occurred with theUpdateMmpAppListJob: ' . $e);
            mail('djarrin@NNN.com', 'UpdateNnnAppListJob Failure', 'A failure in the UpdateNnnAppListJob, here is the exception: ' . $e);
        }

    }

    public function failed(\Exception $exception)
    {
        mail('djarrin@moneymappress.com', 'Push Processor Que Failure', 'A failure in the UpdateMmpAppListJob, here is the exception: ' . $exception);
    }
}

任何关于此问题的 help/suggestions 将不胜感激。

Answer 1

您的代码调用 ->get() 会将整个结果加载到内存中。这会导致您看到的巨大内存分配。删除它并让 ->chunk(...) 使用数据库构建器而不是 get() 返回的内存中的 Collection。您还必须为处理每个块的块提供回调。

public function handle() {
    $chunkSize = 500; //this is the most middleware can handle with its bulk signup call
    $jobDelay = 120; //time between queued jobs
    $jobDelayTimeKeeper = 60; //This will be the actual time delay that will be put into the later method

    DB::table('app_users')
        ->join('app_datas', 'app_datas.customer_number', '=', 'app_users.customer_number')
        ->select('app_users.email')
        ->chunk($chunkSize, function($emailChunk) use (&$jobDelayTimeKeeper, $jobDelay) {
            Queue::later($jobDelayTimeKeeper, new UpdateMmpAppListJob($emailChunk));
            $jobDelayTimeKeeper = $jobDelayTimeKeeper + $jobDelay;
        });
}

Answer 2

上述概念是正确的，但需要此语法才能通过

[2017-12-14 22:08:26] lumen.ERROR: RuntimeException: You must specify an orderBy clause when using this function. in /home/vagrant/sites/nnn/vendor/illuminate/database/Query/Builder.php:1877

这适用于 Lumen 5.5：

public function handle()
    {
        $chunkSize = 500; //this is the most middleware can handle with its bulk signup call
        $jobDelay = 120; //time between queued jobs
        $jobDelayTimeKeeper = 60; //This will be the actual time delay that will be put into the later method

        $emails = DB::table('app_users')
            ->join('app_datas', 'app_datas.customer_number', '=', 'app_users.customer_number')
            ->select('app_users.email')->orderBy('app_users.id', 'desc');

        $emails->chunk($chunkSize, function($emailChunk) use (&$jobDelayTimeKeeper, $jobDelay) {
            Queue::later($jobDelayTimeKeeper, new UpdateMmpAppListJob($emailChunk));
            $jobDelayTimeKeeper = $jobDelayTimeKeeper + $jobDelay;
        });
    }

流明数据库队列第一个作业总是失败允许内存耗尽

Lumen Database Queue first job always failing Allowed memory exhausted

queue

lumen