Node.js

Question

我正在用节点测试流，我设置了一个程序来读取一个大文件并使用流再次写入它。问题是当运行程序时，节点的内存使用量上升到 1.3 GB，这正是正在读取的文件的大小。就像它不流式传输它，它缓冲它并一次性写入它或者垃圾收集器不会破坏内存中的 chunk 变量。这是程序：

const {  createReadStream, createWriteStream } = require('fs');

const readStream = createReadStream('../movie.mp4', {
    highWaterMark: 10000
});
const writeStream = createWriteStream('./copy.mp4', {
    highWaterMark: 10000
});

readStream.on('data', function (chunk) {
    writeStream.write(chunk);
})

readStream.on('end', function () {
    console.log("reading done");
    writeStream.end();
});

writeStream.on('close', function () {
    console.log("Writing done.");
})

奇怪的是，如果我通过管道传输这些流，它会按预期工作并且内存使用量不会超过 20 MB。像这样：

const {  createReadStream, createWriteStream } = require('fs');

const readStream = createReadStream('../movie.mp4', {
    highWaterMark: 10000
});
const writeStream = createWriteStream('./copy.mp4', {
    highWaterMark: 10000
});

readStream.pipe(writeStream);

什么会导致这种行为？

节点版本：v14.15.4

Answer 1

嗯，我找到问题了。这种情况称为背压。在我的例子中，它发生是因为读取流比写入流快得多。这是因为 readStream 在内存中缓冲读取数据，直到 writeMemory 写入它。所以解决方案是我们暂时暂停 readStream 直到 writeStream 完成写入，然后我们为它提供更多数据块。这是正确的程序：

const {  createReadStream, createWriteStream } = require('fs');

const readStream = createReadStream('../movie.mp4', {
    highWaterMark: 10000
});

const writeStream = createWriteStream('./copy.mp4', {
    highWaterMark: 10000
});


readStream.on('data', function (chunk) {
    // according to docs the value of result variable is: 
    // Returns: <boolean> false if the stream wishes for the calling code to wait for the 'drain' event to be emitted before continuing to write additional data; otherwise true.
    const result = writeStream.write(chunk);

    if(!result) {
        console.log("BACKPRESSURE");
        readStream.pause();
    }
});

writeStream.on('drain', () => {
    console.log("DREAINED");
    readStream.resume();
});

readStream.on('end', function () {
    console.log("reading done");
    writeStream.end();
});

writeStream.on('close', function () {
    console.log("Writing done.");
})

关于 drain 事件的文档是 here。

Node.js - 使用 createReadStream 和 createWriteStream 时内存使用率高

Node.js - High memory usage when using createReadStream and createWriteStream

stream