如果右侧 table 已根据连接键排序,则部分合并连接行为
Partial merge join behavior if the right table is already sorted according to the join key
我正在为下面描述的用例评估不同的连接算法,并在文档中寻找说明。
在解释部分合并连接算法时,文档提到:
First ClickHouse sorts the right table by join key in blocks and
creates min-max index for sorted blocks. Then it sorts parts of left
table by join key and joins them over right table. The min-max index
is also used to skip unneeded right table blocks.
这是有道理的,但是如果左右 table 都已经根据连接键在磁盘上排序了呢? Clickhouse 是否仍将正确的 table 再次排序并以相同的顺序将其转储到磁盘上?如果是,那是有意为之的行为吗?
具体用例:
左 table:
CREATE TABLE default.spans (
`project_id` UInt64,
`transaction_span_id` UInt64,
`transaction_name` LowCardinality(String),
`span_id` UInt64,
`transaction_finish_ts` DateTime,
`deleted` UInt8,
`retention_days` UInt16,
... MANY MORE COLUMNS ....
) ENGINE = ReplacingMergeTree(deleted)
PARTITION BY toMonday(transaction_finish_ts)
ORDER BY (
project_id,
toStartOfDay(transaction_finish_ts),
transaction_name,
cityHash64(transaction_span_id),
cityHash64(span_id),
)
TTL transaction_finish_ts + toIntervalDay(retention_days)
右table:
CREATE TABLE default.transactions_local (
`project_id` UInt64,
`span_id` UInt64,
`transaction_name` LowCardinality(String),
`finish_ts` DateTime,
`retention_days` UInt16,
`deleted` UInt8,
... A LOT MORE COLUMNS ...
) ENGINE = ReplacingMergeTree(deleted)
PARTITION BY (retention_days, toMonday(finish_ts))
ORDER BY (
project_id,
toStartOfDay(finish_ts),
transaction_name,
cityHash64(span_id)
)
TTL finish_ts + toIntervalDay(retention_days)
加入密钥:
spans.project_id = transact.project_id AND
toStartOfDay(spans.transaction_finish_ts) = toStartOfDay(transact.finish_ts) AND
spans.transaction_name = transact.transaction_name AND
cityHash64(spans.transaction_span_id) = cityHash64(transact.span_id)
联接将是 n:1,左边的 n 条记录 table 对应右边的一条记录 table。
因此,连接键中的四个条件与两个 table 的 ORDER BY 子句中的第一个字段相同。
这意味着 table 中的 none 应该需要重新排序。
(一周前作为问题提出 here)
谢谢,
菲利波
Which makes sense, but what if both the left and
right table are already sorted on disk according to the join key?
Clickhouse 不使用 table 的订单和表格的索引。
Which means that none of the tables should need to be re-sorted.
没关系。 CH 无法使用它。有点不可能与当前查询管道一起使用。
近期没有改进联接的计划(实施合并联接),因为它们不适合当前架构。
我正在为下面描述的用例评估不同的连接算法,并在文档中寻找说明。 在解释部分合并连接算法时,文档提到:
First ClickHouse sorts the right table by join key in blocks and creates min-max index for sorted blocks. Then it sorts parts of left table by join key and joins them over right table. The min-max index is also used to skip unneeded right table blocks.
这是有道理的,但是如果左右 table 都已经根据连接键在磁盘上排序了呢? Clickhouse 是否仍将正确的 table 再次排序并以相同的顺序将其转储到磁盘上?如果是,那是有意为之的行为吗?
具体用例: 左 table:
CREATE TABLE default.spans (
`project_id` UInt64,
`transaction_span_id` UInt64,
`transaction_name` LowCardinality(String),
`span_id` UInt64,
`transaction_finish_ts` DateTime,
`deleted` UInt8,
`retention_days` UInt16,
... MANY MORE COLUMNS ....
) ENGINE = ReplacingMergeTree(deleted)
PARTITION BY toMonday(transaction_finish_ts)
ORDER BY (
project_id,
toStartOfDay(transaction_finish_ts),
transaction_name,
cityHash64(transaction_span_id),
cityHash64(span_id),
)
TTL transaction_finish_ts + toIntervalDay(retention_days)
右table:
CREATE TABLE default.transactions_local (
`project_id` UInt64,
`span_id` UInt64,
`transaction_name` LowCardinality(String),
`finish_ts` DateTime,
`retention_days` UInt16,
`deleted` UInt8,
... A LOT MORE COLUMNS ...
) ENGINE = ReplacingMergeTree(deleted)
PARTITION BY (retention_days, toMonday(finish_ts))
ORDER BY (
project_id,
toStartOfDay(finish_ts),
transaction_name,
cityHash64(span_id)
)
TTL finish_ts + toIntervalDay(retention_days)
加入密钥:
spans.project_id = transact.project_id AND
toStartOfDay(spans.transaction_finish_ts) = toStartOfDay(transact.finish_ts) AND
spans.transaction_name = transact.transaction_name AND
cityHash64(spans.transaction_span_id) = cityHash64(transact.span_id)
联接将是 n:1,左边的 n 条记录 table 对应右边的一条记录 table。
因此,连接键中的四个条件与两个 table 的 ORDER BY 子句中的第一个字段相同。 这意味着 table 中的 none 应该需要重新排序。
(一周前作为问题提出 here)
谢谢, 菲利波
Which makes sense, but what if both the left and right table are already sorted on disk according to the join key?
Clickhouse 不使用 table 的订单和表格的索引。
Which means that none of the tables should need to be re-sorted.
没关系。 CH 无法使用它。有点不可能与当前查询管道一起使用。
近期没有改进联接的计划(实施合并联接),因为它们不适合当前架构。