如何保存两个摄像头的数据而不影响它们的拍照速度？
How to save two camera's data but not influence their picture-acquire speed?
我正在使用多光谱相机收集数据。一种是近红外线，另一种是彩色的。不是两个摄像头，而是一个摄像头可以同时获取两种不同的图像。我可以使用一些 API 函数，例如 J_Image_OpenStream。两部分核心代码如下所示。一个是用来打开两个流（其实是在一个sample里，我必须要用，但是我不是很清楚它们的意思），设置两个avi文件的保存路径，然后开始采集。
// Open stream
retval0 = J_Image_OpenStream(m_hCam[0], 0, reinterpret_cast<J_IMG_CALLBACK_OBJECT>(this), reinterpret_cast<J_IMG_CALLBACK_FUNCTION>(&COpenCVSample1Dlg::StreamCBFunc0), &m_hThread[0], (ViewSize0.cx*ViewSize0.cy*bpp0)/8);
if (retval0 != J_ST_SUCCESS) {
AfxMessageBox(CString("Could not open stream0!"), MB_OK | MB_ICONEXCLAMATION);
return;
}
TRACE("Opening stream0 succeeded\n");
retval1 = J_Image_OpenStream(m_hCam[1], 0, reinterpret_cast<J_IMG_CALLBACK_OBJECT>(this), reinterpret_cast<J_IMG_CALLBACK_FUNCTION>(&COpenCVSample1Dlg::StreamCBFunc1), &m_hThread[1], (ViewSize1.cx*ViewSize1.cy*bpp1)/8);
if (retval1 != J_ST_SUCCESS) {
AfxMessageBox(CString("Could not open stream1!"), MB_OK | MB_ICONEXCLAMATION);
return;
}
TRACE("Opening stream1 succeeded\n");
const char *filename0 = "C:\Users\shenyang\Desktop\test0.avi";
const char *filename1 = "C:\Users\shenyang\Desktop\test1.avi";
int fps = 10; //frame per second
int codec = -1;//choose the compression method
writer0 = cvCreateVideoWriter(filename0, codec, fps, CvSize(1296,966), 1);
writer1 = cvCreateVideoWriter(filename1, codec, fps, CvSize(1296,964), 1);
// Start Acquision
retval0 = J_Camera_ExecuteCommand(m_hCam[0], NODE_NAME_ACQSTART);
retval1 = J_Camera_ExecuteCommand(m_hCam[1], NODE_NAME_ACQSTART);
// Create two OpenCV named Windows used for displaying "BGR" and "INFRARED" images
cvNamedWindow("BGR");
cvNamedWindow("INFRARED");
还有一个是两个流函数，看起来很像。
void COpenCVSample1Dlg::StreamCBFunc0(J_tIMAGE_INFO * pAqImageInfo)
{
if (m_pImg0 == NULL)
{
// Create the Image:
// We assume this is a 8-bit monochrome image in this sample
m_pImg0 = cvCreateImage(cvSize(pAqImageInfo->iSizeX, pAqImageInfo->iSizeY), IPL_DEPTH_8U, 1);
}
// Copy the data from the Acquisition engine image buffer into the OpenCV Image obejct
memcpy(m_pImg0->imageData, pAqImageInfo->pImageBuffer, m_pImg0->imageSize);
// Display in the "BGR" window
cvShowImage("INFRARED", m_pImg0);
frame0 = m_pImg0;
cvWriteFrame(writer0, frame0);
}
void COpenCVSample1Dlg::StreamCBFunc1(J_tIMAGE_INFO * pAqImageInfo)
{
if (m_pImg1 == NULL)
{
// Create the Image:
// We assume this is a 8-bit monochrome image in this sample
m_pImg1 = cvCreateImage(cvSize(pAqImageInfo->iSizeX, pAqImageInfo->iSizeY), IPL_DEPTH_8U, 1);
}
// Copy the data from the Acquisition engine image buffer into the OpenCV Image obejct
memcpy(m_pImg1->imageData, pAqImageInfo->pImageBuffer, m_pImg1->imageSize);
// Display in the "BGR" window
cvShowImage("BGR", m_pImg1);
frame1 = m_pImg1;
cvWriteFrame(writer1, frame1);
}
问题是如果我不保存 avi 文件，因为
/*writer0 = cvCreateVideoWriter(filename0, codec, fps, CvSize(1296,966), 1);
writer1 = cvCreateVideoWriter(filename1, codec, fps, CvSize(1296,964), 1);*/
//cvWriteFrame(writer0, frame0);
//cvWriteFrame(writer0, frame0);
在两个显示器windows中，拍摄的图片相似，这意味着它们是同步的。但是如果我要写数据到avi文件，由于两种图片大小不一样，而且都比较大，结果发现这会影响两台相机的采集速度，而且拍摄的图片不同步。但是我无法创建这么大的缓冲区来将整个数据存储在内存中，而且 I/O 设备相当慢。我应该怎么办？非常非常感谢。
一些 class 变量是：
public:
FACTORY_HANDLE m_hFactory; // Factory Handle
CAM_HANDLE m_hCam[MAX_CAMERAS]; // Camera Handles
THRD_HANDLE m_hThread[MAX_CAMERAS]; // Stream handles
char m_sCameraId[MAX_CAMERAS][J_CAMERA_ID_SIZE]; // Camera IDs
IplImage *m_pImg0 = NULL; // OpenCV Images
IplImage *m_pImg1 = NULL; // OpenCV Images
CvVideoWriter* writer0;
IplImage *frame0;
CvVideoWriter* writer1;
IplImage *frame1;
BOOL OpenFactoryAndCamera();
void CloseFactoryAndCamera();
void StreamCBFunc0(J_tIMAGE_INFO * pAqImageInfo);
void StreamCBFunc1(J_tIMAGE_INFO * pAqImageInfo);
void InitializeControls();
void EnableControls(BOOL bIsCameraReady, BOOL bIsImageAcquiring);
录制没有丢帧的视频的正确方法是隔离这两个任务（帧获取和帧序列化），使它们不会相互影响（特别是序列化的波动不会消失）捕获帧的时间，必须立即发生以防止帧丢失）。
这可以通过将序列化（帧的编码并将它们写入视频文件）委托给单独的线程，并使用某种同步队列将数据提供给工作线程来实现。
以下是一个简单示例，说明如何完成此操作。由于我只有一个摄像头，而不是你的那种，我将简单地使用网络摄像头并复制帧，但一般原则也适用于你的场景。
示例代码
一开始我们有一些包括：
#include <opencv2/opencv.hpp>
#include <chrono>
#include <condition_variable>
#include <iostream>
#include <mutex>
#include <queue>
#include <thread>
// ============================================================================
using std::chrono::high_resolution_clock;
using std::chrono::duration_cast;
using std::chrono::microseconds;
// ============================================================================
同步队列
第一步是定义我们的同步队列，我们将使用它与编写视频的工作线程进行通信。
我们需要的主要功能是能够：
- 将新图像推入队列
- 从队列中弹出图像，空时等待。
- 能够在我们完成后取消所有待处理的流行音乐。
我们使用 std::queue to hold the cv::Mat instances, and std::mutex to provide synchronization. A std::condition_variable 用于在图像插入队列（或设置取消标志）时通知消费者，并使用一个简单的布尔标志来通知取消。
最后，我们使用空 struct cancelled 作为从 pop() 抛出的异常，因此我们可以通过取消队列干净地终止 worker。
// ============================================================================
class frame_queue
{
public:
struct cancelled {};
public:
frame_queue();
void push(cv::Mat const& image);
cv::Mat pop();
void cancel();
private:
std::queue<cv::Mat> queue_;
std::mutex mutex_;
std::condition_variable cond_;
bool cancelled_;
};
// ----------------------------------------------------------------------------
frame_queue::frame_queue()
: cancelled_(false)
{
}
// ----------------------------------------------------------------------------
void frame_queue::cancel()
{
std::unique_lock<std::mutex> mlock(mutex_);
cancelled_ = true;
cond_.notify_all();
}
// ----------------------------------------------------------------------------
void frame_queue::push(cv::Mat const& image)
{
std::unique_lock<std::mutex> mlock(mutex_);
queue_.push(image);
cond_.notify_one();
}
// ----------------------------------------------------------------------------
cv::Mat frame_queue::pop()
{
std::unique_lock<std::mutex> mlock(mutex_);
while (queue_.empty()) {
if (cancelled_) {
throw cancelled();
}
cond_.wait(mlock);
if (cancelled_) {
throw cancelled();
}
}
cv::Mat image(queue_.front());
queue_.pop();
return image;
}
// ============================================================================
存储工人
下一步是定义一个简单的storage_worker，它将负责从同步队列中取出帧，并将它们编码成视频文件，直到队列被取消。
我添加了简单的计时，这样我们就可以了解编码帧所花费的时间，以及简单的控制台日志记录，因此我们可以了解程序中发生的事情。
// ============================================================================
class storage_worker
{
public:
storage_worker(frame_queue& queue
, int32_t id
, std::string const& file_name
, int32_t fourcc
, double fps
, cv::Size frame_size
, bool is_color = true);
void run();
double total_time_ms() const { return total_time_ / 1000.0; }
private:
frame_queue& queue_;
int32_t id_;
std::string file_name_;
int32_t fourcc_;
double fps_;
cv::Size frame_size_;
bool is_color_;
double total_time_;
};
// ----------------------------------------------------------------------------
storage_worker::storage_worker(frame_queue& queue
, int32_t id
, std::string const& file_name
, int32_t fourcc
, double fps
, cv::Size frame_size
, bool is_color)
: queue_(queue)
, id_(id)
, file_name_(file_name)
, fourcc_(fourcc)
, fps_(fps)
, frame_size_(frame_size)
, is_color_(is_color)
, total_time_(0.0)
{
}
// ----------------------------------------------------------------------------
void storage_worker::run()
{
cv::VideoWriter writer(file_name_, fourcc_, fps_, frame_size_, is_color_);
try {
int32_t frame_count(0);
for (;;) {
cv::Mat image(queue_.pop());
if (!image.empty()) {
high_resolution_clock::time_point t1(high_resolution_clock::now());
++frame_count;
writer.write(image);
high_resolution_clock::time_point t2(high_resolution_clock::now());
double dt_us(static_cast<double>(duration_cast<microseconds>(t2 - t1).count()));
total_time_ += dt_us;
std::cout << "Worker " << id_ << " stored image #" << frame_count
<< " in " << (dt_us / 1000.0) << " ms" << std::endl;
}
}
} catch (frame_queue::cancelled& /*e*/) {
// Nothing more to process, we're done
std::cout << "Queue " << id_ << " cancelled, worker finished." << std::endl;
}
}
// ============================================================================
处理中
终于，我们可以把这些放在一起了。
我们首先初始化和配置我们的视频源。然后我们创建两个 frame_queue 个实例，每个实例对应一个图像流。我们通过创建 storage_worker 的两个实例来实现这一点，每个队列一个。为了让事情变得有趣，我为每个设置了不同的编解码器。
下一步是创建并启动工作线程，它将执行每个 storage_worker 的 run() 方法。让我们的消费者准备好，我们可以开始从相机捕获帧，并将它们提供给 frame_queue 个实例。如上所述，我只有一个来源，所以我将同一帧的副本插入两个队列。
注意： 我需要使用 cv::Mat 的 clone() 方法进行深拷贝，否则我会插入对单个缓冲区的引用OpenCV VideoCapture 出于性能原因使用。这意味着工作线程将获得对这个单一图像的引用，并且不会同步访问这个共享图像缓冲区。您需要确保这种情况也不会发生在您的场景中。
一旦我们读取了适当数量的帧（您可以实现您想要的任何其他类型的停止条件），我们就会取消工作队列，并等待工作线程完成。
最后我们写一些关于不同任务所需时间的统计数据。
// ============================================================================
int main()
{
// The video source -- for me this is a webcam, you use your specific camera API instead
// I only have one camera, so I will just duplicate the frames to simulate your scenario
cv::VideoCapture capture(0);
// Let's make it decent sized, since my camera defaults to 640x480
capture.set(CV_CAP_PROP_FRAME_WIDTH, 1920);
capture.set(CV_CAP_PROP_FRAME_HEIGHT, 1080);
capture.set(CV_CAP_PROP_FPS, 20.0);
// And fetch the actual values, so we can create our video correctly
int32_t frame_width(static_cast<int32_t>(capture.get(CV_CAP_PROP_FRAME_WIDTH)));
int32_t frame_height(static_cast<int32_t>(capture.get(CV_CAP_PROP_FRAME_HEIGHT)));
double video_fps(std::max(10.0, capture.get(CV_CAP_PROP_FPS))); // Some default in case it's 0
std::cout << "Capturing images (" << frame_width << "x" << frame_height
<< ") at " << video_fps << " FPS." << std::endl;
// The synchronized queues, one per video source/storage worker pair
std::vector<frame_queue> queue(2);
// Let's create our storage workers -- let's have two, to simulate your scenario
// and to keep it interesting, have each one write a different format
std::vector <storage_worker> storage;
storage.emplace_back(std::ref(queue[0]), 0
, std::string("foo_0.avi")
, CV_FOURCC('I', 'Y', 'U', 'V')
, video_fps
, cv::Size(frame_width, frame_height)
, true);
storage.emplace_back(std::ref(queue[1]), 1
, std::string("foo_1.avi")
, CV_FOURCC('D', 'I', 'V', 'X')
, video_fps
, cv::Size(frame_width, frame_height)
, true);
// And start the worker threads for each storage worker
std::vector<std::thread> storage_thread;
for (auto& s : storage) {
storage_thread.emplace_back(&storage_worker::run, &s);
}
// Now the main capture loop
int32_t const MAX_FRAME_COUNT(10);
double total_read_time(0.0);
int32_t frame_count(0);
for (; frame_count < MAX_FRAME_COUNT; ++frame_count) {
high_resolution_clock::time_point t1(high_resolution_clock::now());
// Try to read a frame
cv::Mat image;
if (!capture.read(image)) {
std::cerr << "Failed to capture image.\n";
break;
}
// Insert a copy into all queues
for (auto& q : queue) {
q.push(image.clone());
}
high_resolution_clock::time_point t2(high_resolution_clock::now());
double dt_us(static_cast<double>(duration_cast<microseconds>(t2 - t1).count()));
total_read_time += dt_us;
std::cout << "Captured image #" << frame_count << " in "
<< (dt_us / 1000.0) << " ms" << std::endl;
}
// We're done reading, cancel all the queues
for (auto& q : queue) {
q.cancel();
}
// And join all the worker threads, waiting for them to finish
for (auto& st : storage_thread) {
st.join();
}
if (frame_count == 0) {
std::cerr << "No frames captured.\n";
return -1;
}
// Report the timings
total_read_time /= 1000.0;
double total_write_time_a(storage[0].total_time_ms());
double total_write_time_b(storage[1].total_time_ms());
std::cout << "Completed processing " << frame_count << " images:\n"
<< " average capture time = " << (total_read_time / frame_count) << " ms\n"
<< " average write time A = " << (total_write_time_a / frame_count) << " ms\n"
<< " average write time B = " << (total_write_time_b / frame_count) << " ms\n";
return 0;
}
// ============================================================================
控制台输出
运行这个小样本，我们在控制台得到如下日志输出，以及磁盘上的两个视频文件。
注意： 因为这实际上编码比捕获快得多，所以我在 storage_worker 中添加了一些等待以更好地显示分离。
Capturing images (1920x1080) at 20 FPS.
Captured image #0 in 111.009 ms
Captured image #1 in 67.066 ms
Worker 0 stored image #1 in 94.087 ms
Captured image #2 in 62.059 ms
Worker 1 stored image #1 in 193.186 ms
Captured image #3 in 60.059 ms
Worker 0 stored image #2 in 100.097 ms
Captured image #4 in 78.075 ms
Worker 0 stored image #3 in 87.085 ms
Captured image #5 in 62.061 ms
Worker 0 stored image #4 in 95.092 ms
Worker 1 stored image #2 in 193.187 ms
Captured image #6 in 75.074 ms
Worker 0 stored image #5 in 95.093 ms
Captured image #7 in 63.061 ms
Captured image #8 in 64.061 ms
Worker 0 stored image #6 in 102.098 ms
Worker 1 stored image #3 in 201.195 ms
Captured image #9 in 76.074 ms
Worker 0 stored image #7 in 90.089 ms
Worker 0 stored image #8 in 91.087 ms
Worker 1 stored image #4 in 185.18 ms
Worker 0 stored image #9 in 82.08 ms
Worker 0 stored image #10 in 94.092 ms
Queue 0 cancelled, worker finished.
Worker 1 stored image #5 in 179.174 ms
Worker 1 stored image #6 in 106.102 ms
Worker 1 stored image #7 in 105.104 ms
Worker 1 stored image #8 in 103.101 ms
Worker 1 stored image #9 in 104.102 ms
Worker 1 stored image #10 in 104.1 ms
Queue 1 cancelled, worker finished.
Completed processing 10 images:
average capture time = 71.8599 ms
average write time A = 93.09 ms
average write time B = 147.443 ms
average write time B = 176.673 ms
可能的改进
目前，在序列化根本跟不上相机生成新图像的速度的情况下，无法防止队列变得太满。为队列大小设置一些上限，并在推送帧之前检查生产者。您将需要决定要如何处理这种情况。
我正在使用多光谱相机收集数据。一种是近红外线，另一种是彩色的。不是两个摄像头，而是一个摄像头可以同时获取两种不同的图像。我可以使用一些 API 函数，例如 J_Image_OpenStream。两部分核心代码如下所示。一个是用来打开两个流（其实是在一个sample里，我必须要用，但是我不是很清楚它们的意思），设置两个avi文件的保存路径，然后开始采集。
// Open stream
retval0 = J_Image_OpenStream(m_hCam[0], 0, reinterpret_cast<J_IMG_CALLBACK_OBJECT>(this), reinterpret_cast<J_IMG_CALLBACK_FUNCTION>(&COpenCVSample1Dlg::StreamCBFunc0), &m_hThread[0], (ViewSize0.cx*ViewSize0.cy*bpp0)/8);
if (retval0 != J_ST_SUCCESS) {
AfxMessageBox(CString("Could not open stream0!"), MB_OK | MB_ICONEXCLAMATION);
return;
}
TRACE("Opening stream0 succeeded\n");
retval1 = J_Image_OpenStream(m_hCam[1], 0, reinterpret_cast<J_IMG_CALLBACK_OBJECT>(this), reinterpret_cast<J_IMG_CALLBACK_FUNCTION>(&COpenCVSample1Dlg::StreamCBFunc1), &m_hThread[1], (ViewSize1.cx*ViewSize1.cy*bpp1)/8);
if (retval1 != J_ST_SUCCESS) {
AfxMessageBox(CString("Could not open stream1!"), MB_OK | MB_ICONEXCLAMATION);
return;
}
TRACE("Opening stream1 succeeded\n");
const char *filename0 = "C:\Users\shenyang\Desktop\test0.avi";
const char *filename1 = "C:\Users\shenyang\Desktop\test1.avi";
int fps = 10; //frame per second
int codec = -1;//choose the compression method
writer0 = cvCreateVideoWriter(filename0, codec, fps, CvSize(1296,966), 1);
writer1 = cvCreateVideoWriter(filename1, codec, fps, CvSize(1296,964), 1);
// Start Acquision
retval0 = J_Camera_ExecuteCommand(m_hCam[0], NODE_NAME_ACQSTART);
retval1 = J_Camera_ExecuteCommand(m_hCam[1], NODE_NAME_ACQSTART);
// Create two OpenCV named Windows used for displaying "BGR" and "INFRARED" images
cvNamedWindow("BGR");
cvNamedWindow("INFRARED");
还有一个是两个流函数，看起来很像。
void COpenCVSample1Dlg::StreamCBFunc0(J_tIMAGE_INFO * pAqImageInfo)
{
if (m_pImg0 == NULL)
{
// Create the Image:
// We assume this is a 8-bit monochrome image in this sample
m_pImg0 = cvCreateImage(cvSize(pAqImageInfo->iSizeX, pAqImageInfo->iSizeY), IPL_DEPTH_8U, 1);
}
// Copy the data from the Acquisition engine image buffer into the OpenCV Image obejct
memcpy(m_pImg0->imageData, pAqImageInfo->pImageBuffer, m_pImg0->imageSize);
// Display in the "BGR" window
cvShowImage("INFRARED", m_pImg0);
frame0 = m_pImg0;
cvWriteFrame(writer0, frame0);
}
void COpenCVSample1Dlg::StreamCBFunc1(J_tIMAGE_INFO * pAqImageInfo)
{
if (m_pImg1 == NULL)
{
// Create the Image:
// We assume this is a 8-bit monochrome image in this sample
m_pImg1 = cvCreateImage(cvSize(pAqImageInfo->iSizeX, pAqImageInfo->iSizeY), IPL_DEPTH_8U, 1);
}
// Copy the data from the Acquisition engine image buffer into the OpenCV Image obejct
memcpy(m_pImg1->imageData, pAqImageInfo->pImageBuffer, m_pImg1->imageSize);
// Display in the "BGR" window
cvShowImage("BGR", m_pImg1);
frame1 = m_pImg1;
cvWriteFrame(writer1, frame1);
}
问题是如果我不保存 avi 文件，因为
/*writer0 = cvCreateVideoWriter(filename0, codec, fps, CvSize(1296,966), 1);
writer1 = cvCreateVideoWriter(filename1, codec, fps, CvSize(1296,964), 1);*/
//cvWriteFrame(writer0, frame0);
//cvWriteFrame(writer0, frame0);
在两个显示器windows中，拍摄的图片相似，这意味着它们是同步的。但是如果我要写数据到avi文件，由于两种图片大小不一样，而且都比较大，结果发现这会影响两台相机的采集速度，而且拍摄的图片不同步。但是我无法创建这么大的缓冲区来将整个数据存储在内存中，而且 I/O 设备相当慢。我应该怎么办？非常非常感谢。
一些 class 变量是：
public:
FACTORY_HANDLE m_hFactory; // Factory Handle
CAM_HANDLE m_hCam[MAX_CAMERAS]; // Camera Handles
THRD_HANDLE m_hThread[MAX_CAMERAS]; // Stream handles
char m_sCameraId[MAX_CAMERAS][J_CAMERA_ID_SIZE]; // Camera IDs
IplImage *m_pImg0 = NULL; // OpenCV Images
IplImage *m_pImg1 = NULL; // OpenCV Images
CvVideoWriter* writer0;
IplImage *frame0;
CvVideoWriter* writer1;
IplImage *frame1;
BOOL OpenFactoryAndCamera();
void CloseFactoryAndCamera();
void StreamCBFunc0(J_tIMAGE_INFO * pAqImageInfo);
void StreamCBFunc1(J_tIMAGE_INFO * pAqImageInfo);
void InitializeControls();
void EnableControls(BOOL bIsCameraReady, BOOL bIsImageAcquiring);
录制没有丢帧的视频的正确方法是隔离这两个任务（帧获取和帧序列化），使它们不会相互影响（特别是序列化的波动不会消失）捕获帧的时间，必须立即发生以防止帧丢失）。
这可以通过将序列化（帧的编码并将它们写入视频文件）委托给单独的线程，并使用某种同步队列将数据提供给工作线程来实现。
以下是一个简单示例，说明如何完成此操作。由于我只有一个摄像头，而不是你的那种，我将简单地使用网络摄像头并复制帧，但一般原则也适用于你的场景。
示例代码
一开始我们有一些包括：
#include <opencv2/opencv.hpp>
#include <chrono>
#include <condition_variable>
#include <iostream>
#include <mutex>
#include <queue>
#include <thread>
// ============================================================================
using std::chrono::high_resolution_clock;
using std::chrono::duration_cast;
using std::chrono::microseconds;
// ============================================================================
同步队列
第一步是定义我们的同步队列，我们将使用它与编写视频的工作线程进行通信。
我们需要的主要功能是能够：
- 将新图像推入队列
- 从队列中弹出图像，空时等待。
- 能够在我们完成后取消所有待处理的流行音乐。
我们使用 std::queue to hold the cv::Mat instances, and std::mutex to provide synchronization. A std::condition_variable 用于在图像插入队列（或设置取消标志）时通知消费者，并使用一个简单的布尔标志来通知取消。
最后，我们使用空 struct cancelled 作为从 pop() 抛出的异常，因此我们可以通过取消队列干净地终止 worker。
// ============================================================================
class frame_queue
{
public:
struct cancelled {};
public:
frame_queue();
void push(cv::Mat const& image);
cv::Mat pop();
void cancel();
private:
std::queue<cv::Mat> queue_;
std::mutex mutex_;
std::condition_variable cond_;
bool cancelled_;
};
// ----------------------------------------------------------------------------
frame_queue::frame_queue()
: cancelled_(false)
{
}
// ----------------------------------------------------------------------------
void frame_queue::cancel()
{
std::unique_lock<std::mutex> mlock(mutex_);
cancelled_ = true;
cond_.notify_all();
}
// ----------------------------------------------------------------------------
void frame_queue::push(cv::Mat const& image)
{
std::unique_lock<std::mutex> mlock(mutex_);
queue_.push(image);
cond_.notify_one();
}
// ----------------------------------------------------------------------------
cv::Mat frame_queue::pop()
{
std::unique_lock<std::mutex> mlock(mutex_);
while (queue_.empty()) {
if (cancelled_) {
throw cancelled();
}
cond_.wait(mlock);
if (cancelled_) {
throw cancelled();
}
}
cv::Mat image(queue_.front());
queue_.pop();
return image;
}
// ============================================================================
存储工人
下一步是定义一个简单的storage_worker，它将负责从同步队列中取出帧，并将它们编码成视频文件，直到队列被取消。
我添加了简单的计时，这样我们就可以了解编码帧所花费的时间，以及简单的控制台日志记录，因此我们可以了解程序中发生的事情。
// ============================================================================
class storage_worker
{
public:
storage_worker(frame_queue& queue
, int32_t id
, std::string const& file_name
, int32_t fourcc
, double fps
, cv::Size frame_size
, bool is_color = true);
void run();
double total_time_ms() const { return total_time_ / 1000.0; }
private:
frame_queue& queue_;
int32_t id_;
std::string file_name_;
int32_t fourcc_;
double fps_;
cv::Size frame_size_;
bool is_color_;
double total_time_;
};
// ----------------------------------------------------------------------------
storage_worker::storage_worker(frame_queue& queue
, int32_t id
, std::string const& file_name
, int32_t fourcc
, double fps
, cv::Size frame_size
, bool is_color)
: queue_(queue)
, id_(id)
, file_name_(file_name)
, fourcc_(fourcc)
, fps_(fps)
, frame_size_(frame_size)
, is_color_(is_color)
, total_time_(0.0)
{
}
// ----------------------------------------------------------------------------
void storage_worker::run()
{
cv::VideoWriter writer(file_name_, fourcc_, fps_, frame_size_, is_color_);
try {
int32_t frame_count(0);
for (;;) {
cv::Mat image(queue_.pop());
if (!image.empty()) {
high_resolution_clock::time_point t1(high_resolution_clock::now());
++frame_count;
writer.write(image);
high_resolution_clock::time_point t2(high_resolution_clock::now());
double dt_us(static_cast<double>(duration_cast<microseconds>(t2 - t1).count()));
total_time_ += dt_us;
std::cout << "Worker " << id_ << " stored image #" << frame_count
<< " in " << (dt_us / 1000.0) << " ms" << std::endl;
}
}
} catch (frame_queue::cancelled& /*e*/) {
// Nothing more to process, we're done
std::cout << "Queue " << id_ << " cancelled, worker finished." << std::endl;
}
}
// ============================================================================
处理中
终于，我们可以把这些放在一起了。
我们首先初始化和配置我们的视频源。然后我们创建两个 frame_queue 个实例，每个实例对应一个图像流。我们通过创建 storage_worker 的两个实例来实现这一点，每个队列一个。为了让事情变得有趣，我为每个设置了不同的编解码器。
下一步是创建并启动工作线程，它将执行每个 storage_worker 的 run() 方法。让我们的消费者准备好，我们可以开始从相机捕获帧，并将它们提供给 frame_queue 个实例。如上所述，我只有一个来源，所以我将同一帧的副本插入两个队列。
注意： 我需要使用 cv::Mat 的 clone() 方法进行深拷贝，否则我会插入对单个缓冲区的引用OpenCV VideoCapture 出于性能原因使用。这意味着工作线程将获得对这个单一图像的引用，并且不会同步访问这个共享图像缓冲区。您需要确保这种情况也不会发生在您的场景中。
一旦我们读取了适当数量的帧（您可以实现您想要的任何其他类型的停止条件），我们就会取消工作队列，并等待工作线程完成。
最后我们写一些关于不同任务所需时间的统计数据。
// ============================================================================
int main()
{
// The video source -- for me this is a webcam, you use your specific camera API instead
// I only have one camera, so I will just duplicate the frames to simulate your scenario
cv::VideoCapture capture(0);
// Let's make it decent sized, since my camera defaults to 640x480
capture.set(CV_CAP_PROP_FRAME_WIDTH, 1920);
capture.set(CV_CAP_PROP_FRAME_HEIGHT, 1080);
capture.set(CV_CAP_PROP_FPS, 20.0);
// And fetch the actual values, so we can create our video correctly
int32_t frame_width(static_cast<int32_t>(capture.get(CV_CAP_PROP_FRAME_WIDTH)));
int32_t frame_height(static_cast<int32_t>(capture.get(CV_CAP_PROP_FRAME_HEIGHT)));
double video_fps(std::max(10.0, capture.get(CV_CAP_PROP_FPS))); // Some default in case it's 0
std::cout << "Capturing images (" << frame_width << "x" << frame_height
<< ") at " << video_fps << " FPS." << std::endl;
// The synchronized queues, one per video source/storage worker pair
std::vector<frame_queue> queue(2);
// Let's create our storage workers -- let's have two, to simulate your scenario
// and to keep it interesting, have each one write a different format
std::vector <storage_worker> storage;
storage.emplace_back(std::ref(queue[0]), 0
, std::string("foo_0.avi")
, CV_FOURCC('I', 'Y', 'U', 'V')
, video_fps
, cv::Size(frame_width, frame_height)
, true);
storage.emplace_back(std::ref(queue[1]), 1
, std::string("foo_1.avi")
, CV_FOURCC('D', 'I', 'V', 'X')
, video_fps
, cv::Size(frame_width, frame_height)
, true);
// And start the worker threads for each storage worker
std::vector<std::thread> storage_thread;
for (auto& s : storage) {
storage_thread.emplace_back(&storage_worker::run, &s);
}
// Now the main capture loop
int32_t const MAX_FRAME_COUNT(10);
double total_read_time(0.0);
int32_t frame_count(0);
for (; frame_count < MAX_FRAME_COUNT; ++frame_count) {
high_resolution_clock::time_point t1(high_resolution_clock::now());
// Try to read a frame
cv::Mat image;
if (!capture.read(image)) {
std::cerr << "Failed to capture image.\n";
break;
}
// Insert a copy into all queues
for (auto& q : queue) {
q.push(image.clone());
}
high_resolution_clock::time_point t2(high_resolution_clock::now());
double dt_us(static_cast<double>(duration_cast<microseconds>(t2 - t1).count()));
total_read_time += dt_us;
std::cout << "Captured image #" << frame_count << " in "
<< (dt_us / 1000.0) << " ms" << std::endl;
}
// We're done reading, cancel all the queues
for (auto& q : queue) {
q.cancel();
}
// And join all the worker threads, waiting for them to finish
for (auto& st : storage_thread) {
st.join();
}
if (frame_count == 0) {
std::cerr << "No frames captured.\n";
return -1;
}
// Report the timings
total_read_time /= 1000.0;
double total_write_time_a(storage[0].total_time_ms());
double total_write_time_b(storage[1].total_time_ms());
std::cout << "Completed processing " << frame_count << " images:\n"
<< " average capture time = " << (total_read_time / frame_count) << " ms\n"
<< " average write time A = " << (total_write_time_a / frame_count) << " ms\n"
<< " average write time B = " << (total_write_time_b / frame_count) << " ms\n";
return 0;
}
// ============================================================================
控制台输出
运行这个小样本，我们在控制台得到如下日志输出，以及磁盘上的两个视频文件。
注意： 因为这实际上编码比捕获快得多，所以我在 storage_worker 中添加了一些等待以更好地显示分离。
Capturing images (1920x1080) at 20 FPS.
Captured image #0 in 111.009 ms
Captured image #1 in 67.066 ms
Worker 0 stored image #1 in 94.087 ms
Captured image #2 in 62.059 ms
Worker 1 stored image #1 in 193.186 ms
Captured image #3 in 60.059 ms
Worker 0 stored image #2 in 100.097 ms
Captured image #4 in 78.075 ms
Worker 0 stored image #3 in 87.085 ms
Captured image #5 in 62.061 ms
Worker 0 stored image #4 in 95.092 ms
Worker 1 stored image #2 in 193.187 ms
Captured image #6 in 75.074 ms
Worker 0 stored image #5 in 95.093 ms
Captured image #7 in 63.061 ms
Captured image #8 in 64.061 ms
Worker 0 stored image #6 in 102.098 ms
Worker 1 stored image #3 in 201.195 ms
Captured image #9 in 76.074 ms
Worker 0 stored image #7 in 90.089 ms
Worker 0 stored image #8 in 91.087 ms
Worker 1 stored image #4 in 185.18 ms
Worker 0 stored image #9 in 82.08 ms
Worker 0 stored image #10 in 94.092 ms
Queue 0 cancelled, worker finished.
Worker 1 stored image #5 in 179.174 ms
Worker 1 stored image #6 in 106.102 ms
Worker 1 stored image #7 in 105.104 ms
Worker 1 stored image #8 in 103.101 ms
Worker 1 stored image #9 in 104.102 ms
Worker 1 stored image #10 in 104.1 ms
Queue 1 cancelled, worker finished.
Completed processing 10 images:
average capture time = 71.8599 ms
average write time A = 93.09 ms
average write time B = 147.443 ms
average write time B = 176.673 ms
可能的改进
目前，在序列化根本跟不上相机生成新图像的速度的情况下，无法防止队列变得太满。为队列大小设置一些上限，并在推送帧之前检查生产者。您将需要决定要如何处理这种情况。