为什么 sqlite 在通过 valgrind 运行 时存储不同的值？
Why does sqlite store a different value when running through valgrind?
在我正在编写的应用程序中，我注意到当我尝试将双精度值插入 sqlite 数据库时，实际上存储了一个（稍微）不同的值，但只有当应用程序通过 valgrind 运行 .当直接调用程序（无需重新编译）时，所有双打都按预期存储。
此代码重现了该问题。为简洁起见，删除了一些安全检查等。
#include <sqlite3.h>
#include <iostream>
#include <vector>
#include <any>
#include <memory>
#include <cstring>
#include <iomanip>
class SqliteDB
{
sqlite3 *d_db;
bool d_ok;
public:
inline SqliteDB(std::string const &name);
inline ~SqliteDB();
inline void exec(std::string const &q, std::vector<std::vector<std::pair<std::string, std::any>>> *results);
};
inline SqliteDB::SqliteDB(std::string const &name)
:
d_db(nullptr),
d_ok(false)
{
d_ok = (sqlite3_open(name.c_str(), &d_db) == 0);
}
inline SqliteDB::~SqliteDB()
{
if (d_ok)
sqlite3_close(d_db);
}
inline void SqliteDB::exec(std::string const &q, std::vector<std::vector<std::pair<std::string, std::any>>> *results)
{
sqlite3_stmt *stmt;
if (sqlite3_prepare_v2(d_db, q.c_str(), -1, &stmt, nullptr) != SQLITE_OK)
{
std::cout << "SQL Error: " << sqlite3_errmsg(d_db) << std::endl;
return;
}
int rc;
results->clear();
while ((rc = sqlite3_step(stmt)) == SQLITE_ROW)
{
results->resize(results->size() + 1);
for (int i = 0; i < sqlite3_column_count(stmt); ++i)
{
if (sqlite3_column_type(stmt, i) == SQLITE_INTEGER)
{
results->back().emplace_back(std::make_pair(sqlite3_column_name(stmt, i), sqlite3_column_int64(stmt, i)));
}
else if (sqlite3_column_type(stmt, i) == SQLITE_FLOAT)
{
results->back().emplace_back(std::make_pair(sqlite3_column_name(stmt, i), sqlite3_column_double(stmt, i)));
}
else if (sqlite3_column_type(stmt, i) == SQLITE_TEXT)
{
results->back().emplace_back(std::make_pair(sqlite3_column_name(stmt, i), std::string(reinterpret_cast<char const *>(sqlite3_column_text(stmt, i)))));
}
else if (sqlite3_column_type(stmt, i) == SQLITE_BLOB)
{
size_t blobsize = sqlite3_column_bytes(stmt, i);
std::shared_ptr<unsigned char []> blob(new unsigned char[blobsize]);
std::memcpy(blob.get(), reinterpret_cast<unsigned char const *>(sqlite3_column_blob(stmt, i)), blobsize);
results->back().emplace_back(std::make_pair(sqlite3_column_name(stmt, i), std::make_pair(blob, blobsize)));
}
else if (sqlite3_column_type(stmt, i) == SQLITE_NULL)
{
results->back().emplace_back(std::make_pair(sqlite3_column_name(stmt, i), nullptr));
}
}
}
if (rc != SQLITE_DONE)
std::cout << "SQL Error: " << sqlite3_errmsg(d_db) << std::endl;
sqlite3_finalize(stmt);
}
inline std::string toHexString(double d)
{
unsigned char *data = reinterpret_cast<unsigned char *>(&d);
std::ostringstream oss;
oss << "(hex:) ";
for (uint i = 0; i < sizeof(d); ++i)
oss << std::hex << std::setfill('0') << std::setw(2)
<< (static_cast<int32_t>(data[i]) & 0xFF)
<< ((i == sizeof(d) - 1) ? "" : " ");
return oss.str();
}
int main()
{
SqliteDB db(":memory:");
std::vector<std::vector<std::pair<std::string, std::any>>> results;
db.exec("CREATE TABLE part (_id INTEGER PRIMARY KEY, ratio REAL)", &results);
double d = 1.4814814329147339;
std::cout << "Inserting into table: " << std::defaultfloat << std::setprecision(17) << d
<< " " << toHexString(d) << std::endl;
db.exec("INSERT INTO part VALUES (1,1.4814814329147339)", &results);
db.exec("SELECT ratio FROM part WHERE _id = 1", &results);
for (uint i = 0; i < results.size(); ++i)
for (uint j = 0; j < results[i].size(); ++j)
{
if (results[i][j].second.type() == typeid(double))
std::cout << "Retrieved from table: " << std::defaultfloat << std::setprecision(17) << std::any_cast<double>(results[i][j].second)
<< " " << toHexString(std::any_cast<double>(results[i][j].second)) << std::endl;
}
return 0;
}
我已经确认问题发生在存储值时，而不是在检索值时（也就是说，数据库实际上包含不同的双精度值）。上述程序的输出：
[~/valgrindsqlitedouble] $ g++ -std=c++2a -Wall -Wextra -Wshadow -Wold-style-cast -pedantic -fomit-frame-pointer -O1 -g -lsqlite3 main.cc
[~/valgrindsqlitedouble] $ ./a.out
Inserting into table: 1.4814814329147339 (hex:) 00 00 00 e0 25 b4 f7 3f
Retrieved from table: 1.4814814329147339 (hex:) 00 00 00 e0 25 b4 f7 3f
[~/valgrindsqlitedouble] $ valgrind ./a.out
==3340== Memcheck, a memory error detector
==3340== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==3340== Using Valgrind-3.14.0 and LibVEX; rerun with -h for copyright info
==3340== Command: ./a.out
==3340==
Inserting into table: 1.4814814329147339 (hex:) 00 00 00 e0 25 b4 f7 3f
Retrieved from table: 1.4814814329147341 (hex:) 01 00 00 e0 25 b4 f7 3f
==3340==
==3340== HEAP SUMMARY:
==3340== in use at exit: 0 bytes in 0 blocks
==3340== total heap usage: 299 allocs, 299 frees, 269,972 bytes allocated
==3340==
==3340== All heap blocks were freed -- no leaks are possible
==3340==
==3340== For counts of detected and suppressed errors, rerun with: -v
==3340== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)
[~/valgrindsqlitedouble] $
我假设当 sqlite 将字符串转换为 double 时发生了一些舍入错误，但是谁能解释为什么它只在 valgrind 为 运行 时发生？
其次，我可以摆脱这个问题吗？在最终程序中，精度甚至不会那么重要，但在开发过程中最好能有可预测的输出。我正在处理一些大的二进制文件，在测试中验证程序是否正常工作的最简单方法是将生成的输出文件（包括数据库）与已知的良好输出文件进行比较。有什么方法可以让 sqlite 插入我想要的 8 个字节吗？
谢谢！
Your program is then run on a synthetic CPU provided by the Valgrind core.
所以理论上，当 运行 在 Valgrind 下时，程序的行为有很大的变化空间。当然，在实践中我们希望情况并非如此，因为不仅我们的程序应该是可移植的，而且如果 Valgrind 改变了它们的行为，那么即使在同一平台上，它也不是真正测试我们想要它测试的东西。
但是，您的期望值已经 non-portable，开发人员在他们的 限制 部分中将其用作防御措施：
As of version 3.0.0, Valgrind has the following limitations in its implementation of x86/AMD64 floating point relative to IEEE754.
Precision: There is no support for 80 bit arithmetic. Internally, Valgrind represents all such "long double" numbers in 64 bits, and so there may be some differences in results. Whether or not this is critical remains to be seen. Note, the x86/amd64 fldt/fstpt instructions (read/write 80-bit numbers) are correctly simulated, using conversions to/from 64 bits, so that in-memory images of 80-bit numbers look correct if anyone wants to see.
The impression observed from many FP regression tests is that the accuracy differences aren't significant. Generally speaking, if a program relies on 80-bit precision, there may be difficulties porting it to non x86/amd64 platforms which only support 64-bit FP precision. Even on x86/amd64, the program may get different results depending on whether it is compiled to use SSE2 instructions (64-bits only), or x87 instructions (80-bit). The net effect is to make FP programs behave as if they had been run on a machine with 64-bit IEEE floats, for example PowerPC. On amd64 FP arithmetic is done by default on SSE2, so amd64 looks more like PowerPC than x86 from an FP perspective, and there are far fewer noticable accuracy differences than with x86.
和
As of version 3.0.0, Valgrind has the following limitations in its implementation of x86/AMD64 SSE2 FP arithmetic, relative to IEEE754.
Essentially the same: no exceptions, and limited observance of rounding mode. Also, SSE2 has control bits which make it treat denormalised numbers as zero (DAZ) and a related action, flush denormals to zero (FTZ). Both of these cause SSE2 arithmetic to be less accurate than IEEE requires. Valgrind detects, ignores, and can warn about, attempts to enable either mode.
等
我建议使用 Valgrind 来查找 low-level 错误（内存泄漏等），而不是用于执行任何功能测试。
或者，如果您有八个特定字节要放入数据库中，只需这样做，而不是通过 floating-point 往返。
在我正在编写的应用程序中，我注意到当我尝试将双精度值插入 sqlite 数据库时，实际上存储了一个（稍微）不同的值，但只有当应用程序通过 valgrind 运行 .当直接调用程序（无需重新编译）时，所有双打都按预期存储。
此代码重现了该问题。为简洁起见，删除了一些安全检查等。
#include <sqlite3.h>
#include <iostream>
#include <vector>
#include <any>
#include <memory>
#include <cstring>
#include <iomanip>
class SqliteDB
{
sqlite3 *d_db;
bool d_ok;
public:
inline SqliteDB(std::string const &name);
inline ~SqliteDB();
inline void exec(std::string const &q, std::vector<std::vector<std::pair<std::string, std::any>>> *results);
};
inline SqliteDB::SqliteDB(std::string const &name)
:
d_db(nullptr),
d_ok(false)
{
d_ok = (sqlite3_open(name.c_str(), &d_db) == 0);
}
inline SqliteDB::~SqliteDB()
{
if (d_ok)
sqlite3_close(d_db);
}
inline void SqliteDB::exec(std::string const &q, std::vector<std::vector<std::pair<std::string, std::any>>> *results)
{
sqlite3_stmt *stmt;
if (sqlite3_prepare_v2(d_db, q.c_str(), -1, &stmt, nullptr) != SQLITE_OK)
{
std::cout << "SQL Error: " << sqlite3_errmsg(d_db) << std::endl;
return;
}
int rc;
results->clear();
while ((rc = sqlite3_step(stmt)) == SQLITE_ROW)
{
results->resize(results->size() + 1);
for (int i = 0; i < sqlite3_column_count(stmt); ++i)
{
if (sqlite3_column_type(stmt, i) == SQLITE_INTEGER)
{
results->back().emplace_back(std::make_pair(sqlite3_column_name(stmt, i), sqlite3_column_int64(stmt, i)));
}
else if (sqlite3_column_type(stmt, i) == SQLITE_FLOAT)
{
results->back().emplace_back(std::make_pair(sqlite3_column_name(stmt, i), sqlite3_column_double(stmt, i)));
}
else if (sqlite3_column_type(stmt, i) == SQLITE_TEXT)
{
results->back().emplace_back(std::make_pair(sqlite3_column_name(stmt, i), std::string(reinterpret_cast<char const *>(sqlite3_column_text(stmt, i)))));
}
else if (sqlite3_column_type(stmt, i) == SQLITE_BLOB)
{
size_t blobsize = sqlite3_column_bytes(stmt, i);
std::shared_ptr<unsigned char []> blob(new unsigned char[blobsize]);
std::memcpy(blob.get(), reinterpret_cast<unsigned char const *>(sqlite3_column_blob(stmt, i)), blobsize);
results->back().emplace_back(std::make_pair(sqlite3_column_name(stmt, i), std::make_pair(blob, blobsize)));
}
else if (sqlite3_column_type(stmt, i) == SQLITE_NULL)
{
results->back().emplace_back(std::make_pair(sqlite3_column_name(stmt, i), nullptr));
}
}
}
if (rc != SQLITE_DONE)
std::cout << "SQL Error: " << sqlite3_errmsg(d_db) << std::endl;
sqlite3_finalize(stmt);
}
inline std::string toHexString(double d)
{
unsigned char *data = reinterpret_cast<unsigned char *>(&d);
std::ostringstream oss;
oss << "(hex:) ";
for (uint i = 0; i < sizeof(d); ++i)
oss << std::hex << std::setfill('0') << std::setw(2)
<< (static_cast<int32_t>(data[i]) & 0xFF)
<< ((i == sizeof(d) - 1) ? "" : " ");
return oss.str();
}
int main()
{
SqliteDB db(":memory:");
std::vector<std::vector<std::pair<std::string, std::any>>> results;
db.exec("CREATE TABLE part (_id INTEGER PRIMARY KEY, ratio REAL)", &results);
double d = 1.4814814329147339;
std::cout << "Inserting into table: " << std::defaultfloat << std::setprecision(17) << d
<< " " << toHexString(d) << std::endl;
db.exec("INSERT INTO part VALUES (1,1.4814814329147339)", &results);
db.exec("SELECT ratio FROM part WHERE _id = 1", &results);
for (uint i = 0; i < results.size(); ++i)
for (uint j = 0; j < results[i].size(); ++j)
{
if (results[i][j].second.type() == typeid(double))
std::cout << "Retrieved from table: " << std::defaultfloat << std::setprecision(17) << std::any_cast<double>(results[i][j].second)
<< " " << toHexString(std::any_cast<double>(results[i][j].second)) << std::endl;
}
return 0;
}
我已经确认问题发生在存储值时，而不是在检索值时（也就是说，数据库实际上包含不同的双精度值）。上述程序的输出：
[~/valgrindsqlitedouble] $ g++ -std=c++2a -Wall -Wextra -Wshadow -Wold-style-cast -pedantic -fomit-frame-pointer -O1 -g -lsqlite3 main.cc
[~/valgrindsqlitedouble] $ ./a.out
Inserting into table: 1.4814814329147339 (hex:) 00 00 00 e0 25 b4 f7 3f
Retrieved from table: 1.4814814329147339 (hex:) 00 00 00 e0 25 b4 f7 3f
[~/valgrindsqlitedouble] $ valgrind ./a.out
==3340== Memcheck, a memory error detector
==3340== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==3340== Using Valgrind-3.14.0 and LibVEX; rerun with -h for copyright info
==3340== Command: ./a.out
==3340==
Inserting into table: 1.4814814329147339 (hex:) 00 00 00 e0 25 b4 f7 3f
Retrieved from table: 1.4814814329147341 (hex:) 01 00 00 e0 25 b4 f7 3f
==3340==
==3340== HEAP SUMMARY:
==3340== in use at exit: 0 bytes in 0 blocks
==3340== total heap usage: 299 allocs, 299 frees, 269,972 bytes allocated
==3340==
==3340== All heap blocks were freed -- no leaks are possible
==3340==
==3340== For counts of detected and suppressed errors, rerun with: -v
==3340== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)
[~/valgrindsqlitedouble] $
我假设当 sqlite 将字符串转换为 double 时发生了一些舍入错误，但是谁能解释为什么它只在 valgrind 为 运行 时发生？
其次，我可以摆脱这个问题吗？在最终程序中，精度甚至不会那么重要，但在开发过程中最好能有可预测的输出。我正在处理一些大的二进制文件，在测试中验证程序是否正常工作的最简单方法是将生成的输出文件（包括数据库）与已知的良好输出文件进行比较。有什么方法可以让 sqlite 插入我想要的 8 个字节吗？
谢谢！
Your program is then run on a synthetic CPU provided by the Valgrind core.
所以理论上，当 运行 在 Valgrind 下时，程序的行为有很大的变化空间。当然，在实践中我们希望情况并非如此，因为不仅我们的程序应该是可移植的，而且如果 Valgrind 改变了它们的行为，那么即使在同一平台上，它也不是真正测试我们想要它测试的东西。
但是，您的期望值已经 non-portable，开发人员在他们的 限制 部分中将其用作防御措施：
As of version 3.0.0, Valgrind has the following limitations in its implementation of x86/AMD64 floating point relative to IEEE754.
Precision: There is no support for 80 bit arithmetic. Internally, Valgrind represents all such "long double" numbers in 64 bits, and so there may be some differences in results. Whether or not this is critical remains to be seen. Note, the x86/amd64 fldt/fstpt instructions (read/write 80-bit numbers) are correctly simulated, using conversions to/from 64 bits, so that in-memory images of 80-bit numbers look correct if anyone wants to see.
The impression observed from many FP regression tests is that the accuracy differences aren't significant. Generally speaking, if a program relies on 80-bit precision, there may be difficulties porting it to non x86/amd64 platforms which only support 64-bit FP precision. Even on x86/amd64, the program may get different results depending on whether it is compiled to use SSE2 instructions (64-bits only), or x87 instructions (80-bit). The net effect is to make FP programs behave as if they had been run on a machine with 64-bit IEEE floats, for example PowerPC. On amd64 FP arithmetic is done by default on SSE2, so amd64 looks more like PowerPC than x86 from an FP perspective, and there are far fewer noticable accuracy differences than with x86.
和
As of version 3.0.0, Valgrind has the following limitations in its implementation of x86/AMD64 SSE2 FP arithmetic, relative to IEEE754.
Essentially the same: no exceptions, and limited observance of rounding mode. Also, SSE2 has control bits which make it treat denormalised numbers as zero (DAZ) and a related action, flush denormals to zero (FTZ). Both of these cause SSE2 arithmetic to be less accurate than IEEE requires. Valgrind detects, ignores, and can warn about, attempts to enable either mode.
等
我建议使用 Valgrind 来查找 low-level 错误（内存泄漏等），而不是用于执行任何功能测试。
或者，如果您有八个特定字节要放入数据库中，只需这样做，而不是通过 floating-point 往返。