Hyperledger Fabric - Peer 无法使用 Mutual TLS 连接到 (raft) Orderer

Question

我正在运行在 kubernetes 上使用 HLF -（3 个 raft 排序器和 2 个节点）

现在由于 raft 需要双向 TLS，我必须设置一些证书。

这 3 个 raft 排序器能够相互通信，因为他们正在选举一个领导者，并在我拉下那个领导者时重新选举另一个领导者。

当我设置对等点时，我使用同一个 CA 来生成证书。我能够创建频道并从同行加入。但是我必须在这些命令之前运行 CORE_PEER_MSPCONFIGPATH=$ADMIN_MSP_PATH ，否则我会收到拒绝访问错误。

我还被迫将以下标志附加到每个 peer channel x 命令 I 运行.

--tls --cafile $ORD_TLS_PATH/cacert.pem --certfile $CORE_PEER_TLS_CLIENTCERT_FILE --keyfile $CORE_PEER_TLS_CLIENTKEY_FILE --clientauth

我可以使用管理员 msp 创建、获取、加入频道。

现在一旦加入通道，peer 就无法连接到 orderer，不知何故给出了一个错误的证书。

订购者日志

使用了错误的证书？

2019-08-15 16:07:55.699 UTC [core.comm] ServerHandshake -> ERRO 221 TLS handshake failed with error remote error: tls: bad certificate server=Orderer remoteaddress=10.130.2.148:53922
2019-08-15 16:07:55.699 UTC [grpc] handleRawConn -> DEBU 222 grpc: Server.Serve failed to complete security handshake from "10.130.2.148:53922": remote error: tls: bad certificate

对等日志

这些表明它无法用 ca.crt ?

验证它

2019-08-15 16:10:17.990 UTC [grpc] DialContext -> DEBU 03a parsed scheme: ""
2019-08-15 16:10:17.990 UTC [grpc] DialContext -> DEBU 03b scheme "" not registered, fallback to default scheme
2019-08-15 16:10:17.991 UTC [grpc] watcher -> DEBU 03c ccResolverWrapper: sending new addresses to cc: [{orderer-2.hlf-orderers.svc.cluster.local:7050 0  <nil>}]
2019-08-15 16:10:17.991 UTC [grpc] switchBalancer -> DEBU 03d ClientConn switching balancer to "pick_first"
2019-08-15 16:10:17.991 UTC [grpc] HandleSubConnStateChange -> DEBU 03e pickfirstBalancer: HandleSubConnStateChange: 0xc00260b710, CONNECTING
2019-08-15 16:10:18.009 UTC [grpc] createTransport -> DEBU 03f grpc: addrConn.createTransport failed to connect to {orderer-2.hlf-orderers.svc.cluster.local:7050 0  <nil>}. Err :connection error: desc = "transport: authentication handshake failed: x509: certificate signed by unknown authority". Reconnecting...
2019-08-15 16:10:18.012 UTC [grpc] HandleSubConnStateChange -> DEBU 040 pickfirstBalancer: HandleSubConnStateChange: 0xc00260b710, TRANSIENT_FAILURE
2019-08-15 16:10:18.991 UTC [grpc] HandleSubConnStateChange -> DEBU 041 pickfirstBalancer: HandleSubConnStateChange: 0xc00260b710, CONNECTING
2019-08-15 16:10:19.003 UTC [grpc] createTransport -> DEBU 042 grpc: addrConn.createTransport failed to connect to {orderer-2.hlf-orderers.svc.cluster.local:7050 0  <nil>}. Err :connection error: desc = "transport: authentication handshake failed: x509: certificate signed by unknown authority". Reconnecting...
2019-08-15 16:10:19.003 UTC [grpc] HandleSubConnStateChange -> DEBU 043 pickfirstBalancer: HandleSubConnStateChange: 0xc00260b710, TRANSIENT_FAILURE
2019-08-15 16:10:20.719 UTC [grpc] HandleSubConnStateChange -> DEBU 044 pickfirstBalancer: HandleSubConnStateChange: 0xc00260b710, CONNECTING
2019-08-15 16:10:20.731 UTC [grpc] createTransport -> DEBU 045 grpc: addrConn.createTransport failed to connect to {orderer-2.hlf-orderers.svc.cluster.local:7050 0  <nil>}. Err :connection error: desc = "transport: authentication handshake failed: x509: certificate signed by unknown authority". Reconnecting...
2019-08-15 16:10:20.733 UTC [grpc] HandleSubConnStateChange -> DEBU 046 pickfirstBalancer: HandleSubConnStateChange: 0xc00260b710, TRANSIENT_FAILURE
2019-08-15 16:10:20.990 UTC [ConnProducer] NewConnection -> ERRO 047 Failed connecting to {orderer-2.hlf-orderers.svc.cluster.local:7050 [OrdererMSP]} , error: context deadline exceeded

我生成了如下使用的证书：

订购者管理员

fabric-ca-client enroll -u https://u:p@ca.example.com -M ./OrdererMSP

排序节点 X

因为我对 TLS 使用相同的证书，所以我在此处添加了用于 TLS 目的的主机

订购者-x.hlf-orderers.svc.cluster.local #kubernetes
排序者-x.hlf-排序者#kubernetes
orderer-x #kubernetes
localhost #本地调试

fabric-ca-client enroll -m orderer-x \
  -u https://ox:px@ca.example.com \
  --csr.hosts orderer-x.hlf-orderers.svc.cluster.local,orderer-x.hlf-orderers,orderer-x,localhost \
  -M orderer-x-MSP

同行管理员

fabric-ca-client enroll -u https://u:p@ca.example.com -M ./PeerMSP

对等节点 X

fabric-ca-client enroll -m peer-x \
  -u https://ox:px@ca.example.com \
  --csr.hosts peer-x.hlf-peers.svc.cluster.local,peer-x.hlf-peers,peer-x,localhost \
  -M peer-x-MSP

现在所有这些，都有相同的ca.crt (/cacerts/ca.example.com.pem)

configtx.yaml

        Orderer:
            <<: *OrdererDefaults
            OrdererType: etcdraft
            EtcdRaft:
                Consenters:
                    - Host: orderer-1.hlf-orderers.svc.cluster.local
                      Port: 7050
                      ClientTLSCert: orderer-1-MSP/signcerts/cert.pem
                      ServerTLSCert: orderer-1-MSP/signcerts/cert.pem
                    - Host: orderer-2.hlf-orderers.svc.cluster.local
                      Port: 7050
                      ClientTLSCert: orderer-2-MSP/signcerts/cert.pem
                      ServerTLSCert: orderer-2-MSP/signcerts/cert.pem
                    - Host: orderer-3.hlf-orderers.svc.cluster.local
                      Port: 7050
                      ClientTLSCert: orderer-3-MSP/signcerts/cert.pem
                      ServerTLSCert: orderer-3-MSP/signcerts/cert.pem
            Addresses:
                - orderer-1.hlf-orderers.svc.cluster.local:7050
                - orderer-2.hlf-orderers.svc.cluster.local:7050
                - orderer-3.hlf-orderers.svc.cluster.local:7050

我已经多次检查了正确的证书是否安装在正确的位置并进行了配置。

在同行方面，我确保：

CORE_PEER_TLS_CLIENTROOTCAS_FILES 设置正确并且（正确的）文件已挂载 (CORE_PEER_TLS_CLIENTROOTCAS_FILES: "/var/hyperledger/tls/client/cert/ca.crt")
同上 CORE_PEER_TLS_CLIENTKEY_FILE & CORE_PEER_TLS_CLIENTCERT_FILE
CORE_PEER_TLS_CLIENTAUTHREQUIRED 设置为真

在订购方方面，我确保：

ORDERER_GENERAL_TLS_CLIENTAUTHREQUIRED 设置为真
ORDERER_GENERAL_CLUSTER_CLIENTCERTIFICATE设置正确
ORDERER_GENERAL_CLUSTER_CLIENTPRIVATEKEY设置正确
ORDERER_GENERAL_TLS_CLIENTROOTCAS 设置正确

令我感到奇怪的是，排序者能够相互交谈（因为他们正在选举领导者），而对等点却不能这样做

Answer 1

您似乎遇到了以下错误

E0923 16:30:14.963567129 31166 ssl_transport_security.cc:989] Handshake failed with fatal error SSL_ERROR_SSL: error:14094412:SSL routines:ssl3_read_bytes:sslv3 alert bad certificate.
E0923 16:30:15.964456710 31166 ssl_transport_security.cc:188] ssl_info_callback: error occured.

根据您的详细信息，一切似乎都是正确的但是检查下面

certificate signed by unknown authority -> This makes me bit doubt on your certificate mapping

确认一下

同行：

CORE_PEER_TLS_ENABLED=真
CORE_PEER_TLS_CERT_FILE=/opt/gopath/src/github.com/hyperledger/fabric/peer/tls/server.crt
CORE_PEER_TLS_KEY_FILE=/opt/gopath/src/github.com/hyperledger/fabric/peer/tls/server.key
CORE_PEER_TLS_ROOTCERT_FILE=/data/maersksea-rca-maersksea-chain.pem
CORE_PEER_TLS_CLIENTCERT_FILE=/data/tls/maersksea-peer-maersksea-client.crt
CORE_PEER_TLS_CLIENTKEY_FILE=/data/tls/maersksea-peer-maersksea-client.key
CORE_PEER_TLS_CLIENTAUTHREQUIRED=真
CORE_PEER_TLS_CLIENTROOTCAS_FILES=/data/maersksea-rca-maersksea-chain.pem

订购者：

ORDERER_GENERAL_TLS_CLIENTAUTHREQUIRED=真
ORDERER_GENERAL_TLS_CLIENTROOTCAS=[/data/maersksea-rca-maersksea-chain.pem]

Answer 2

看来，tlscacerts 应该在 msp(s) 目录中 PRIOR 以创建创世/通道块。仅仅在运行时将它们安装在 pod 中是不够的

我的 msp 目录（在 configtx.yaml 中使用）如下所示：

管理员证书
tlscacerts
证书
...

在此之后一切开始工作

Hyperledger Fabric - Peer 无法使用 Mutual TLS 连接到 (raft) Orderer

Hyperledger Fabric - Peer unable to connect to (raft) Orderer with Mutual TLS

ssl

hyperledger

hyperledger-fabric