批量并行 HTTP 请求，每个请求使用异步 for 循环

Question

我正在尝试运行使用数组中的一组关键字将请求分批并行到 API。 Article by Denis Fatkhudinov.

我遇到的问题是，对于每个关键字，我需要使用不同的 page 参数再次运行请求，次数与 pages 中的数字一样多变量。

对于 chainNext 函数的 return，我一直得到 Cannot read property 'then' of undefined。

单独批量并行请求，没有 for 循环，效果很好，我正在努力将 for 循环合并到进程中。

// Parallel requests in batches
async function runBatches() {
  // The keywords to request with 
  const keywords = ['many keyword strings here...'];
  // Set max concurrent requests
  const concurrent = 5;
  // Clone keywords array
  const keywordsClone = keywords.slice()
  // Array for future resolved promises for each batch
  const promises = new Array(concurrent).fill(Promise.resolve());
  // Async for loop
  const asyncForEach = async (pages, callback) => {
    for (let page = 1; page <= pages; page++) {
      await callback(page);
    }
  };
  // Number of pages to loop for
  const pages = 2;

  // Recursively run batches
  const chainNext = (pro) => {
    // Runs itself as long as there are entries left on the array
    if (keywordsClone.length) {
      // Store the first entry and conviently also remove it from the array
      const keyword = keywordsClone.shift();
      // Run 'the promise to be' request
      return pro.then(async () => {
        // ---> Here was my problem, I am declaring the constant before running the for loop
        const promiseOperation = await asyncForEach(pages, async (page) => {
          await request(keyword, page)
        });
        // ---> The recursive invocation should also be inside the for loop
        return chainNext(promiseOperation);
      });
    }

    return pro;
  }

  return await Promise.all(promises.map(chainNext));
}

// HTTP request
async function request(keyword, page) { 
  try {
    // request API 
    const res = await apiservice(keyword, page);
    // Send data to an outer async function to process the data
    await append(res.data);
  } catch (error) {
    throw new Error(error) 
  }
}


runBatches()

Answer 1

问题只是pro没有定义，因为你还没有初始化它。

你基本上执行这段代码：

Promise.all(new Array(concurrent).fill(Promise.resolve().map(pro => {
  // pro is undefined here because the Promise.resolve had no parameter
  return pro.then(async () => {})
}));

我不完全确定你背后的想法，但这是你在更浓缩版本中的问题。

Answer 2

我通过在 for 循环中移动实际请求 promiseOperation 并在那里返回递归函数来实现它的工作

// Recursively run batches
const chainNext = async (pro) => {
  if (keywordsClone.length) {
    const keyword = keywordsClone.shift()
    return pro.then(async () => {
      await asyncForEach(pages, (page) => {
        const promiseOperation = request(keyword, page)
        return chainNext(promiseOperation)
      })
    })
  }
  return pro
}

批量并行请求的功劳归于https://itnext.io/node-js-handling-asynchronous-operations-in-parallel-69679dfae3fc

批量并行 HTTP 请求，每个请求使用异步 for 循环

Parallel HTTP requests in batches with async for loop for each request

javascript

node.js

async-await

es6-promise