为什么 Ignite 在网格激活时为 JVM 分配 32GB 的额外内部内存？
Why is Ignite allocating 32GB of additional internal memory for the JVM upon grid activation?
您好，我们正在使用 Apache Ignite 2.7（8 个节点，每个 120GB）并配置 16GB 堆和 100GB 数据区域（启用持久性）。使用本机内存跟踪，我们看到通常预期的类别（如堆、线程等）符合预期，但 "Internal"（即堆外）高达 132GB。这是 JVM 需要 运行 的所有其他内容之上的。由于 JVM 提出了如此巨大的内存请求，系统正在进入内存不足的状态（OS 内存不足）。
作为一项实验，我们将数据区域减少到 1GB，并测量了网格激活前后的 JVM 内部内存使用情况（网格由我们附加的客户端节点激活）。我们看到内部（阅读：不安全的堆外）内存在网格激活时从 62,154 KB 跳到 32,897,187 KB。所以32GB的开销似乎与数据区域的大小无关。
这 32GB 的额外系统 RAM 使用对我们来说是一个真正的问题。为什么 Ignite 这样做以及我们如何控制它？
谢谢
这是我们看到的典型本机内存摘要。注意巨大的内部分配。
native memory Total: reserved=156688325KB, committed=156439245KB
- Java Heap (reserved=16777216KB, committed=16777216KB) (mmap: reserved=16777216KB, committed=16777216KB)
- Class (reserved=112257KB, committed=111489KB) (classes #17951) (malloc=1665KB #17624) (mmap: reserved=110592KB, committed=109824KB)
- Thread (reserved=229015KB, committed=229015KB) (thread #223) (stack: reserved=228032KB, committed=228032KB) (malloc=723KB #1128)
(arena=260KB #432)
- Code (reserved=255790KB, committed=40250KB) (malloc=6190KB #11547) (mmap: reserved=249600KB, committed=34060KB)
- GC (reserved=704014KB, committed=704014KB) (malloc=48654KB #22251) (mmap: reserved=655360KB, committed=655360KB)
- Compiler (reserved=420KB, committed=420KB) (malloc=289KB #1284) (arena=131KB #15)
- Internal (reserved=138544815KB, committed=138544811KB) (malloc=138544779KB #35177) (mmap: reserved=36KB, committed=32KB)
- Symbol (reserved=26536KB, committed=26536KB) (malloc=24002KB #216741) (arena=2533KB #1)
- Native Memory Tracking (reserved=4822KB, committed=4822KB) (malloc=30KB #346) (tracking overhead=4791KB)
- Arena Chunk (reserved=673KB, committed=673KB) (malloc=673KB)
- Unknown (reserved=32768KB, committed=0KB) (mmap: reserved=32768KB, committed=0KB)
PS
我们将默认数据区域设置为 128MB，将 systemRegionMaxSize 设置为 8GB，将 systemRegionInitialSize 设置为 512MB。
配置：
<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation=" http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd">
<bean id="ignite.cfg" class="org.apache.ignite.configuration.IgniteConfiguration">
<property name="gridLogger">
<bean class="org.apache.ignite.logger.log4j.Log4JLogger">
<constructor-arg type="java.lang.String" value="/opt/ignite/apache-ignite/config/log4j.xml"/>
</bean>
</property>
<property name="metricsLogFrequency" value="600000"/>
<property name="rebalanceThreadPoolSize" value="12"/>
<property name="peerClassLoadingEnabled" value="true"/>
<property name="publicThreadPoolSize" value="32"/>
<property name="systemThreadPoolSize" value="32"/>
<property name="workDirectory" value="/data/ignite/work"/>
<property name="segmentationPolicy" value="RESTART_JVM"/>
<property name="dataStorageConfiguration">
<bean class="org.apache.ignite.configuration.DataStorageConfiguration">
<property name="checkpointReadLockTimeout" value="0"/>
<property name="systemRegionInitialSize" value="#{512L * 1024 * 1024}"/>
<property name="systemRegionMaxSize" value="#{8L * 1024 * 1024 * 1024}"/>
<property name="storagePath" value="/data/ignite/persistentStore"/>
<property name="defaultDataRegionConfiguration">
<bean class="org.apache.ignite.configuration.DataRegionConfiguration">
<property name="name" value="Default_Region"/>
<property name="initialSize" value="67108864"/>
<property name="maxSize" value="134217728"/>
<property name="persistenceEnabled" value="false"/>
<property name="metricsEnabled" value="true"/>
</bean>
</property>
<property name="dataRegionConfigurations">
<list>
<bean class="org.apache.ignite.configuration.DataRegionConfiguration">
<property name="name" value="Tiered_Region"/>
<property name="initialSize" value="53687091200"/>
<property name="maxSize" value="53687091200"/>
<property name="persistenceEnabled" value="true"/>
<property name="pageEvictionMode" value="RANDOM_2_LRU"/>
<property name="evictionThreshold" value="0.75"/>
<property name="metricsEnabled" value="true"/>
</bean>
</list>
</property>
</bean>
</property>
<property name="cacheConfiguration">
<list>
<bean class="org.apache.ignite.configuration.CacheConfiguration">
<property name="name" value="default"/>
<property name="atomicityMode" value="ATOMIC"/>
<property name="backups" value="0"/>
</bean>
</list>
</property>
<property name="communicationSpi">
<bean class="org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi">
<property name="messageQueueLimit" value="#{1 * 1024}"/>
<property name="idleConnectionTimeout" value="30000"/>
</bean>
</property>
<property name="discoverySpi">
<bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
<property name="ipFinder">
<bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.s3.TcpDiscoveryS3IpFinder">
<property name="awsCredentials" ref="aws.creds"/>
<property name="bucketName" value="project-test-xyz"/>
</bean>
</property>
</bean>
</property>
</bean>
<bean id="aws.creds" class="com.amazonaws.auth.BasicAWSCredentials">
<constructor-arg value="foo"/>
<constructor-arg value="bar"/>
</bean>
</beans>
[在下面添加日志]
[2019-05-17 22:28:39,592][WARN ][main][IgniteKernal] Peer class
loading is enabled (disable it in production for performance and
deployment consistency reasons) [2019-05-17 22:28:39,593][WARN
][main][IgniteKernal] Please set system property
'-Djava.net.preferIPv4Stack=true' to avoid possible problems in mixed
environments. [2019-05-17 22:28:40,141][WARN
][main][NoopCheckpointSpi] Checkpoints are disabled (to enable
configure any GridCheckpointSpi implementation) [2019-05-17
22:28:40,214][WARN ][main][GridCollisionManager] Collision resolution
is disabled (all jobs will be activated upon arrival). [2019-05-17
22:28:41,690][WARN ][main][GridCacheDatabaseSharedManager]
DataRegionConfiguration.maxWalArchiveSize instead
DataRegionConfiguration.walHistorySize would be used for removing old
archive wal files [2019-05-17 22:28:41,826][WARN
][main][PartitionsEvictManager] Logging at INFO level without checking
if INFO level is enabled: Evict partition permits=4 [2019-05-17
22:28:46,291][WARN ][main][IgniteKernal] Nodes started on local
machine require more than 80% of physical RAM what can lead to
significant slowdown due to swapping (please decrease JVM heap size,
data region size or checkpoint buffer size) [required=12516MB,
available=14008MB] log4j: Finalizing appender named [null].
[2019-05-17 22:31:19,958][WARN
][disco-event-worker-#42][GridDiscoveryManager] Local node's value of
'java.net.preferIPv4Stack' system property differs from remote node's
(all nodes in topology should have identical value)
[locPreferIpV4=null, rmtPreferIpV4=true, locId8=f25228c0,
rmtId8=eac4211d, rmtAddrs=[192.168.1.5/127.0.0.1, /192.168.1.5],
rmtNode=ClusterNode [id=eac4211d-c272-4eb0-9bd5-f91dfa34a0e9, order=2,
addr=[127.0.0.1, 192.168.1.5], daemon=false]] [2019-05-17
22:32:24,265][WARN ][exchange-worker-#43][GridAffinityAssignmentCache]
Logging at INFO level without checking if INFO level is enabled: Local
node affinity assignment distribution is not ideal [cache=default,
expectedPrimary=1024.00, actualPrimary=1024, expectedBackups=1024.00,
actualBackups=0, warningThreshold=50.00%] [2019-05-17
22:32:24,269][WARN ][exchange-worker-#43][GridAffinityAssignmentCache]
Logging at INFO level without checking if INFO level is enabled: Local
node affinity assignment distribution is not ideal [cache=default,
expectedPrimary=1024.00, actualPrimary=1024, expectedBackups=1024.00,
actualBackups=0, warningThreshold=50.00%] [2019-05-17
22:32:24,850][WARN ][exchange-worker-#43][GridAffinityAssignmentCache]
Logging at INFO level without checking if INFO level is enabled: Local
node affinity assignment distribution is not ideal [cache=default,
expectedPrimary=1024.00, actualPrimary=1024, expectedBackups=1024.00,
actualBackups=0, warningThreshold=50.00%] [2019-05-17
22:32:24,911][WARN
][disco-notifier-worker-#41][GridClusterStateProcessor] Logging at
INFO level without checking if INFO level is enabled: Received state
change finish message: true 22:33:49.086 [exchange-worker-#43] INFO
c.b.aa.ceres.loader.S3CacheLoader - load
eb5445c7-d7fa-4018-95b6-63c4a0911eae received inject ignite instance
IgniteKernal [longJVMPauseDetector=LongJVMPauseDetector
[workerRef=Thread[jvm-pause-detector-worker,5,main], longPausesCnt=0,
longPausesTotalDuration=0, longPausesTimestamps=[0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], longPausesDurations=[0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]],
cfg=IgniteConfiguration [igniteInstanceName=null, pubPoolSize=32,
svcPoolSize=32, callbackPoolSize=8, stripedPoolSize=8, sysPoolSize=16,
mgmtPoolSize=4, igfsPoolSize=8, dataStreamerPoolSize=8,
utilityCachePoolSize=8, utilityCacheKeepAliveTime=60000,
p2pPoolSize=2, qryPoolSize=8, igniteHome=/opt/ignite/apache-ignite,
igniteWorkDir=/data/ignite/work,
mbeanSrv=com.sun.jmx.mbeanserver.JmxMBeanServer@6f94fa3e,
nodeId=f25228c0-afbc-4626-990a-68f97fd5b258, marsh=BinaryMarshaller
[], marshLocJobs=false, daemon=false, p2pEnabled=true,
netTimeout=5000, sndRetryDelay=1000, sndRetryCnt=3,
metricsHistSize=10000, metricsUpdateFreq=2000,
metricsExpTime=9223372036854775807, discoSpi=TcpDiscoverySpi
[addrRslvr=null, sockTimeout=5000, ackTimeout=5000,
marsh=JdkMarshaller
[clsFilter=org.apache.ignite.marshaller.MarshallerUtils@44a3f602],
reconCnt=10, reconDelay=2000, maxAckTimeout=600000,
forceSrvMode=false, clientReconnectDisabled=false, internalLsnr=null],
segPlc=NOOP, segResolveAttempts=2, waitForSegOnStart=true,
allResolversPassReq=true, segChkFreq=60000,
commSpi=TcpCommunicationSpi
[connectGate=org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi$ConnectGateway@6020964a,
connPlc=org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi$FirstConnectionPolicy@3f2874d5,
enableForcibleNodeKill=false, enableTroubleshootingLog=true,
locAddr=null, locHost=0.0.0.0/0.0.0.0, locPort=47100,
locPortRange=100, shmemPort=-1, directBuf=true, directSndBuf=false,
idleConnTimeout=600000, connTimeout=5000, maxConnTimeout=600000,
reconCnt=10, sockSndBuf=32768, sockRcvBuf=32768, msgQueueLimit=1024,
slowClientQueueLimit=0, nioSrvr=GridNioServer [selectorSpins=0,
filterChain=FilterChain[filters=[GridNioCodecFilter
[parser=org.apache.ignite.internal.util.nio.GridDirectParser@7873ad1,
directMode=true], GridConnectionBytesVerifyFilter], closed=false,
directBuf=true, tcpNoDelay=true, sockSndBuf=32768, sockRcvBuf=32768,
writeTimeout=2000, idleTimeout=600000, skipWrite=false,
skipRead=false, locAddr=0.0.0.0/0.0.0.0:47100, order=LITTLE_ENDIAN,
sndQueueLimit=1024, directMode=true, sslFilter=null,
msgQueueLsnr=null, readerMoveCnt=0, writerMoveCnt=0,
readWriteSelectorsAssign=false], shmemSrv=null,
usePairedConnections=false, connectionsPerNode=1, tcpNoDelay=true,
filterReachableAddresses=false, ackSndThreshold=32,
unackedMsgsBufSize=0, sockWriteTimeout=2000, boundTcpPort=47100,
boundTcpShmemPort=-1, selectorsCnt=4, selectorSpins=0, addrRslvr=null,
ctxInitLatch=java.util.concurrent.CountDownLatch@7b757828[Count = 0],
stopping=false],
evtSpi=org.apache.ignite.spi.eventstorage.NoopEventStorageSpi@282cb7c7,
colSpi=NoopCollisionSpi [], deploySpi=LocalDeploymentSpi [],
indexingSpi=org.apache.ignite.spi.indexing.noop.NoopIndexingSpi@50de186c,
addrRslvr=null,
encryptionSpi=org.apache.ignite.spi.encryption.noop.NoopEncryptionSpi@5a3bc7ed,
clientMode=false, rebalanceThreadPoolSize=1,
txCfg=TransactionConfiguration [txSerEnabled=false,
dfltIsolation=REPEATABLE_READ, dfltConcurrency=PESSIMISTIC,
dfltTxTimeout=0, txTimeoutOnPartitionMapExchange=0,
pessimisticTxLogSize=0, pessimisticTxLogLinger=10000,
tmLookupClsName=null, txManagerFactory=null, useJtaSync=false],
cacheSanityCheckEnabled=true, discoStartupDelay=60000,
deployMode=SHARED, p2pMissedCacheSize=100, locHost=null,
timeSrvPortBase=31100, timeSrvPortRange=100,
failureDetectionTimeout=60000, sysWorkerBlockedTimeout=null,
clientFailureDetectionTimeout=30000, metricsLogFreq=60000,
hadoopCfg=null, connectorCfg=ConnectorConfiguration [jettyPath=null,
host=null, port=11211, noDelay=true, directBuf=false,
sndBufSize=32768, rcvBufSize=32768, idleQryCurTimeout=600000,
idleQryCurCheckFreq=60000, sndQueueLimit=0, selectorCnt=4,
idleTimeout=7000, sslEnabled=false, sslClientAuth=false,
sslCtxFactory=null, sslFactory=null, portRange=100, threadPoolSize=8,
msgInterceptor=null], odbcCfg=null, warmupClos=null,
atomicCfg=AtomicConfiguration [seqReserveSize=1000,
cacheMode=PARTITIONED, backups=1, aff=null, grpName=null],
classLdr=null, sslCtxFactory=null, platformCfg=null, binaryCfg=null,
memCfg=null, pstCfg=null, dsCfg=DataStorageConfiguration
[sysRegionInitSize=41943040, sysRegionMaxSize=104857600,
pageSize=1024, concLvl=0, dfltDataRegConf=DataRegionConfiguration
[name=Default_Region, maxSize=134217728, initSize=67108864,
swapPath=null, pageEvictionMode=DISABLED, evictionThreshold=0.9,
emptyPagesPoolSize=100, metricsEnabled=true,
metricsSubIntervalCount=5, metricsRateTimeInterval=60000,
persistenceEnabled=false, checkpointPageBufSize=0],
dataRegions=[DataRegionConfiguration [name=Tiered_Region,
maxSize=8589934592, initSize=8589934592, swapPath=null,
pageEvictionMode=DISABLED, evictionThreshold=0.9,
emptyPagesPoolSize=100, metricsEnabled=true,
metricsSubIntervalCount=5, metricsRateTimeInterval=60000,
persistenceEnabled=true, checkpointPageBufSize=0]],
storagePath=/data/ignite/persistentStore, checkpointFreq=180000,
lockWaitTime=30000, checkpointThreads=8,
checkpointWriteOrder=SEQUENTIAL, walHistSize=20,
maxWalArchiveSize=1073741824, walSegments=10, walSegmentSize=67108864,
walPath=db/wal, walArchivePath=db/wal/archive, metricsEnabled=false,
walMode=LOG_ONLY, walTlbSize=131072, walBuffSize=0, walFlushFreq=2000,
walFsyncDelay=1000, walRecordIterBuffSize=67108864,
alwaysWriteFullPages=false,
fileIOFactory=org.apache.ignite.internal.processors.cache.persistence.file.AsyncFileIOFactory@2fb68ec6,
metricsSubIntervalCnt=5, metricsRateTimeInterval=60000,
walAutoArchiveAfterInactivity=-1, writeThrottlingEnabled=true,
walCompactionEnabled=false, walCompactionLevel=1,
checkpointReadLockTimeout=null], activeOnStart=true,
autoActivation=true, longQryWarnTimeout=3000, sqlConnCfg=null,
cliConnCfg=ClientConnectorConfiguration [host=null, port=10800,
portRange=100, sockSndBufSize=0, sockRcvBufSize=0, tcpNoDelay=true,
maxOpenCursorsPerConn=128, threadPoolSize=8, idleTimeout=0,
jdbcEnabled=true, odbcEnabled=true, thinCliEnabled=true,
sslEnabled=false, useIgniteSslCtxFactory=true, sslClientAuth=false,
sslCtxFactory=null], mvccVacuumThreadCnt=2, mvccVacuumFreq=5000,
authEnabled=false, failureHnd=null, commFailureRslvr=null],
igniteInstanceName=null, startTime=1558132126418,
rsrcCtx=org.apache.ignite.internal.processors.resource.GridSpringResourceContextImpl@556d0e12,
reconnectState=ReconnectState [firstReconnectFut=GridFutureAdapter
[ignoreInterrupts=false, state=INIT, res=null, hash=1426466647],
curReconnectFut=null, reconnectDone=null]]
我认为那是 Checkpoint Page Buffer，默认情况下它是数据区域大小的 20%。
您可以明确指定它以确保您不会忘记它，并相应地减小区域大小以确保您不会 运行 内存不足。
应该只适用于持久区域。
请注意，您还应该期望您的 OS 对其数据结构和块缓存占用几 GB，因此我认为您不应该将 120G 中的 116G 分配给 Ignite 的 Off-Heap。也不要忘记堆。
您好，我们正在使用 Apache Ignite 2.7（8 个节点，每个 120GB）并配置 16GB 堆和 100GB 数据区域（启用持久性）。使用本机内存跟踪，我们看到通常预期的类别（如堆、线程等）符合预期，但 "Internal"（即堆外）高达 132GB。这是 JVM 需要 运行 的所有其他内容之上的。由于 JVM 提出了如此巨大的内存请求，系统正在进入内存不足的状态（OS 内存不足）。
作为一项实验，我们将数据区域减少到 1GB，并测量了网格激活前后的 JVM 内部内存使用情况（网格由我们附加的客户端节点激活）。我们看到内部（阅读：不安全的堆外）内存在网格激活时从 62,154 KB 跳到 32,897,187 KB。所以32GB的开销似乎与数据区域的大小无关。
这 32GB 的额外系统 RAM 使用对我们来说是一个真正的问题。为什么 Ignite 这样做以及我们如何控制它？
谢谢
这是我们看到的典型本机内存摘要。注意巨大的内部分配。
native memory Total: reserved=156688325KB, committed=156439245KB - Java Heap (reserved=16777216KB, committed=16777216KB) (mmap: reserved=16777216KB, committed=16777216KB) - Class (reserved=112257KB, committed=111489KB) (classes #17951) (malloc=1665KB #17624) (mmap: reserved=110592KB, committed=109824KB) - Thread (reserved=229015KB, committed=229015KB) (thread #223) (stack: reserved=228032KB, committed=228032KB) (malloc=723KB #1128) (arena=260KB #432) - Code (reserved=255790KB, committed=40250KB) (malloc=6190KB #11547) (mmap: reserved=249600KB, committed=34060KB) - GC (reserved=704014KB, committed=704014KB) (malloc=48654KB #22251) (mmap: reserved=655360KB, committed=655360KB) - Compiler (reserved=420KB, committed=420KB) (malloc=289KB #1284) (arena=131KB #15) - Internal (reserved=138544815KB, committed=138544811KB) (malloc=138544779KB #35177) (mmap: reserved=36KB, committed=32KB) - Symbol (reserved=26536KB, committed=26536KB) (malloc=24002KB #216741) (arena=2533KB #1) - Native Memory Tracking (reserved=4822KB, committed=4822KB) (malloc=30KB #346) (tracking overhead=4791KB) - Arena Chunk (reserved=673KB, committed=673KB) (malloc=673KB) - Unknown (reserved=32768KB, committed=0KB) (mmap: reserved=32768KB, committed=0KB)
PS
我们将默认数据区域设置为 128MB，将 systemRegionMaxSize 设置为 8GB，将 systemRegionInitialSize 设置为 512MB。
配置：
<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation=" http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd">
<bean id="ignite.cfg" class="org.apache.ignite.configuration.IgniteConfiguration">
<property name="gridLogger">
<bean class="org.apache.ignite.logger.log4j.Log4JLogger">
<constructor-arg type="java.lang.String" value="/opt/ignite/apache-ignite/config/log4j.xml"/>
</bean>
</property>
<property name="metricsLogFrequency" value="600000"/>
<property name="rebalanceThreadPoolSize" value="12"/>
<property name="peerClassLoadingEnabled" value="true"/>
<property name="publicThreadPoolSize" value="32"/>
<property name="systemThreadPoolSize" value="32"/>
<property name="workDirectory" value="/data/ignite/work"/>
<property name="segmentationPolicy" value="RESTART_JVM"/>
<property name="dataStorageConfiguration">
<bean class="org.apache.ignite.configuration.DataStorageConfiguration">
<property name="checkpointReadLockTimeout" value="0"/>
<property name="systemRegionInitialSize" value="#{512L * 1024 * 1024}"/>
<property name="systemRegionMaxSize" value="#{8L * 1024 * 1024 * 1024}"/>
<property name="storagePath" value="/data/ignite/persistentStore"/>
<property name="defaultDataRegionConfiguration">
<bean class="org.apache.ignite.configuration.DataRegionConfiguration">
<property name="name" value="Default_Region"/>
<property name="initialSize" value="67108864"/>
<property name="maxSize" value="134217728"/>
<property name="persistenceEnabled" value="false"/>
<property name="metricsEnabled" value="true"/>
</bean>
</property>
<property name="dataRegionConfigurations">
<list>
<bean class="org.apache.ignite.configuration.DataRegionConfiguration">
<property name="name" value="Tiered_Region"/>
<property name="initialSize" value="53687091200"/>
<property name="maxSize" value="53687091200"/>
<property name="persistenceEnabled" value="true"/>
<property name="pageEvictionMode" value="RANDOM_2_LRU"/>
<property name="evictionThreshold" value="0.75"/>
<property name="metricsEnabled" value="true"/>
</bean>
</list>
</property>
</bean>
</property>
<property name="cacheConfiguration">
<list>
<bean class="org.apache.ignite.configuration.CacheConfiguration">
<property name="name" value="default"/>
<property name="atomicityMode" value="ATOMIC"/>
<property name="backups" value="0"/>
</bean>
</list>
</property>
<property name="communicationSpi">
<bean class="org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi">
<property name="messageQueueLimit" value="#{1 * 1024}"/>
<property name="idleConnectionTimeout" value="30000"/>
</bean>
</property>
<property name="discoverySpi">
<bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
<property name="ipFinder">
<bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.s3.TcpDiscoveryS3IpFinder">
<property name="awsCredentials" ref="aws.creds"/>
<property name="bucketName" value="project-test-xyz"/>
</bean>
</property>
</bean>
</property>
</bean>
<bean id="aws.creds" class="com.amazonaws.auth.BasicAWSCredentials">
<constructor-arg value="foo"/>
<constructor-arg value="bar"/>
</bean>
</beans>
[在下面添加日志]
[2019-05-17 22:28:39,592][WARN ][main][IgniteKernal] Peer class loading is enabled (disable it in production for performance and deployment consistency reasons) [2019-05-17 22:28:39,593][WARN ][main][IgniteKernal] Please set system property '-Djava.net.preferIPv4Stack=true' to avoid possible problems in mixed environments. [2019-05-17 22:28:40,141][WARN ][main][NoopCheckpointSpi] Checkpoints are disabled (to enable configure any GridCheckpointSpi implementation) [2019-05-17 22:28:40,214][WARN ][main][GridCollisionManager] Collision resolution is disabled (all jobs will be activated upon arrival). [2019-05-17 22:28:41,690][WARN ][main][GridCacheDatabaseSharedManager] DataRegionConfiguration.maxWalArchiveSize instead DataRegionConfiguration.walHistorySize would be used for removing old archive wal files [2019-05-17 22:28:41,826][WARN ][main][PartitionsEvictManager] Logging at INFO level without checking if INFO level is enabled: Evict partition permits=4 [2019-05-17 22:28:46,291][WARN ][main][IgniteKernal] Nodes started on local machine require more than 80% of physical RAM what can lead to significant slowdown due to swapping (please decrease JVM heap size, data region size or checkpoint buffer size) [required=12516MB, available=14008MB] log4j: Finalizing appender named [null]. [2019-05-17 22:31:19,958][WARN ][disco-event-worker-#42][GridDiscoveryManager] Local node's value of 'java.net.preferIPv4Stack' system property differs from remote node's (all nodes in topology should have identical value) [locPreferIpV4=null, rmtPreferIpV4=true, locId8=f25228c0, rmtId8=eac4211d, rmtAddrs=[192.168.1.5/127.0.0.1, /192.168.1.5], rmtNode=ClusterNode [id=eac4211d-c272-4eb0-9bd5-f91dfa34a0e9, order=2, addr=[127.0.0.1, 192.168.1.5], daemon=false]] [2019-05-17 22:32:24,265][WARN ][exchange-worker-#43][GridAffinityAssignmentCache] Logging at INFO level without checking if INFO level is enabled: Local node affinity assignment distribution is not ideal [cache=default, expectedPrimary=1024.00, actualPrimary=1024, expectedBackups=1024.00, actualBackups=0, warningThreshold=50.00%] [2019-05-17 22:32:24,269][WARN ][exchange-worker-#43][GridAffinityAssignmentCache] Logging at INFO level without checking if INFO level is enabled: Local node affinity assignment distribution is not ideal [cache=default, expectedPrimary=1024.00, actualPrimary=1024, expectedBackups=1024.00, actualBackups=0, warningThreshold=50.00%] [2019-05-17 22:32:24,850][WARN ][exchange-worker-#43][GridAffinityAssignmentCache] Logging at INFO level without checking if INFO level is enabled: Local node affinity assignment distribution is not ideal [cache=default, expectedPrimary=1024.00, actualPrimary=1024, expectedBackups=1024.00, actualBackups=0, warningThreshold=50.00%] [2019-05-17 22:32:24,911][WARN ][disco-notifier-worker-#41][GridClusterStateProcessor] Logging at INFO level without checking if INFO level is enabled: Received state change finish message: true 22:33:49.086 [exchange-worker-#43] INFO c.b.aa.ceres.loader.S3CacheLoader - load eb5445c7-d7fa-4018-95b6-63c4a0911eae received inject ignite instance IgniteKernal [longJVMPauseDetector=LongJVMPauseDetector [workerRef=Thread[jvm-pause-detector-worker,5,main], longPausesCnt=0, longPausesTotalDuration=0, longPausesTimestamps=[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], longPausesDurations=[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]], cfg=IgniteConfiguration [igniteInstanceName=null, pubPoolSize=32, svcPoolSize=32, callbackPoolSize=8, stripedPoolSize=8, sysPoolSize=16, mgmtPoolSize=4, igfsPoolSize=8, dataStreamerPoolSize=8, utilityCachePoolSize=8, utilityCacheKeepAliveTime=60000, p2pPoolSize=2, qryPoolSize=8, igniteHome=/opt/ignite/apache-ignite, igniteWorkDir=/data/ignite/work, mbeanSrv=com.sun.jmx.mbeanserver.JmxMBeanServer@6f94fa3e, nodeId=f25228c0-afbc-4626-990a-68f97fd5b258, marsh=BinaryMarshaller [], marshLocJobs=false, daemon=false, p2pEnabled=true, netTimeout=5000, sndRetryDelay=1000, sndRetryCnt=3, metricsHistSize=10000, metricsUpdateFreq=2000, metricsExpTime=9223372036854775807, discoSpi=TcpDiscoverySpi [addrRslvr=null, sockTimeout=5000, ackTimeout=5000, marsh=JdkMarshaller [clsFilter=org.apache.ignite.marshaller.MarshallerUtils@44a3f602], reconCnt=10, reconDelay=2000, maxAckTimeout=600000, forceSrvMode=false, clientReconnectDisabled=false, internalLsnr=null], segPlc=NOOP, segResolveAttempts=2, waitForSegOnStart=true, allResolversPassReq=true, segChkFreq=60000, commSpi=TcpCommunicationSpi [connectGate=org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi$ConnectGateway@6020964a, connPlc=org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi$FirstConnectionPolicy@3f2874d5, enableForcibleNodeKill=false, enableTroubleshootingLog=true, locAddr=null, locHost=0.0.0.0/0.0.0.0, locPort=47100, locPortRange=100, shmemPort=-1, directBuf=true, directSndBuf=false, idleConnTimeout=600000, connTimeout=5000, maxConnTimeout=600000, reconCnt=10, sockSndBuf=32768, sockRcvBuf=32768, msgQueueLimit=1024, slowClientQueueLimit=0, nioSrvr=GridNioServer [selectorSpins=0, filterChain=FilterChain[filters=[GridNioCodecFilter [parser=org.apache.ignite.internal.util.nio.GridDirectParser@7873ad1, directMode=true], GridConnectionBytesVerifyFilter], closed=false, directBuf=true, tcpNoDelay=true, sockSndBuf=32768, sockRcvBuf=32768, writeTimeout=2000, idleTimeout=600000, skipWrite=false, skipRead=false, locAddr=0.0.0.0/0.0.0.0:47100, order=LITTLE_ENDIAN, sndQueueLimit=1024, directMode=true, sslFilter=null, msgQueueLsnr=null, readerMoveCnt=0, writerMoveCnt=0, readWriteSelectorsAssign=false], shmemSrv=null, usePairedConnections=false, connectionsPerNode=1, tcpNoDelay=true, filterReachableAddresses=false, ackSndThreshold=32, unackedMsgsBufSize=0, sockWriteTimeout=2000, boundTcpPort=47100, boundTcpShmemPort=-1, selectorsCnt=4, selectorSpins=0, addrRslvr=null, ctxInitLatch=java.util.concurrent.CountDownLatch@7b757828[Count = 0], stopping=false], evtSpi=org.apache.ignite.spi.eventstorage.NoopEventStorageSpi@282cb7c7, colSpi=NoopCollisionSpi [], deploySpi=LocalDeploymentSpi [], indexingSpi=org.apache.ignite.spi.indexing.noop.NoopIndexingSpi@50de186c, addrRslvr=null, encryptionSpi=org.apache.ignite.spi.encryption.noop.NoopEncryptionSpi@5a3bc7ed, clientMode=false, rebalanceThreadPoolSize=1, txCfg=TransactionConfiguration [txSerEnabled=false, dfltIsolation=REPEATABLE_READ, dfltConcurrency=PESSIMISTIC, dfltTxTimeout=0, txTimeoutOnPartitionMapExchange=0, pessimisticTxLogSize=0, pessimisticTxLogLinger=10000, tmLookupClsName=null, txManagerFactory=null, useJtaSync=false], cacheSanityCheckEnabled=true, discoStartupDelay=60000, deployMode=SHARED, p2pMissedCacheSize=100, locHost=null, timeSrvPortBase=31100, timeSrvPortRange=100, failureDetectionTimeout=60000, sysWorkerBlockedTimeout=null, clientFailureDetectionTimeout=30000, metricsLogFreq=60000, hadoopCfg=null, connectorCfg=ConnectorConfiguration [jettyPath=null, host=null, port=11211, noDelay=true, directBuf=false, sndBufSize=32768, rcvBufSize=32768, idleQryCurTimeout=600000, idleQryCurCheckFreq=60000, sndQueueLimit=0, selectorCnt=4, idleTimeout=7000, sslEnabled=false, sslClientAuth=false, sslCtxFactory=null, sslFactory=null, portRange=100, threadPoolSize=8, msgInterceptor=null], odbcCfg=null, warmupClos=null, atomicCfg=AtomicConfiguration [seqReserveSize=1000, cacheMode=PARTITIONED, backups=1, aff=null, grpName=null], classLdr=null, sslCtxFactory=null, platformCfg=null, binaryCfg=null, memCfg=null, pstCfg=null, dsCfg=DataStorageConfiguration [sysRegionInitSize=41943040, sysRegionMaxSize=104857600, pageSize=1024, concLvl=0, dfltDataRegConf=DataRegionConfiguration [name=Default_Region, maxSize=134217728, initSize=67108864, swapPath=null, pageEvictionMode=DISABLED, evictionThreshold=0.9, emptyPagesPoolSize=100, metricsEnabled=true, metricsSubIntervalCount=5, metricsRateTimeInterval=60000, persistenceEnabled=false, checkpointPageBufSize=0], dataRegions=[DataRegionConfiguration [name=Tiered_Region, maxSize=8589934592, initSize=8589934592, swapPath=null, pageEvictionMode=DISABLED, evictionThreshold=0.9, emptyPagesPoolSize=100, metricsEnabled=true, metricsSubIntervalCount=5, metricsRateTimeInterval=60000, persistenceEnabled=true, checkpointPageBufSize=0]], storagePath=/data/ignite/persistentStore, checkpointFreq=180000, lockWaitTime=30000, checkpointThreads=8, checkpointWriteOrder=SEQUENTIAL, walHistSize=20, maxWalArchiveSize=1073741824, walSegments=10, walSegmentSize=67108864, walPath=db/wal, walArchivePath=db/wal/archive, metricsEnabled=false, walMode=LOG_ONLY, walTlbSize=131072, walBuffSize=0, walFlushFreq=2000, walFsyncDelay=1000, walRecordIterBuffSize=67108864, alwaysWriteFullPages=false, fileIOFactory=org.apache.ignite.internal.processors.cache.persistence.file.AsyncFileIOFactory@2fb68ec6, metricsSubIntervalCnt=5, metricsRateTimeInterval=60000, walAutoArchiveAfterInactivity=-1, writeThrottlingEnabled=true, walCompactionEnabled=false, walCompactionLevel=1, checkpointReadLockTimeout=null], activeOnStart=true, autoActivation=true, longQryWarnTimeout=3000, sqlConnCfg=null, cliConnCfg=ClientConnectorConfiguration [host=null, port=10800, portRange=100, sockSndBufSize=0, sockRcvBufSize=0, tcpNoDelay=true, maxOpenCursorsPerConn=128, threadPoolSize=8, idleTimeout=0, jdbcEnabled=true, odbcEnabled=true, thinCliEnabled=true, sslEnabled=false, useIgniteSslCtxFactory=true, sslClientAuth=false, sslCtxFactory=null], mvccVacuumThreadCnt=2, mvccVacuumFreq=5000, authEnabled=false, failureHnd=null, commFailureRslvr=null], igniteInstanceName=null, startTime=1558132126418, rsrcCtx=org.apache.ignite.internal.processors.resource.GridSpringResourceContextImpl@556d0e12, reconnectState=ReconnectState [firstReconnectFut=GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=1426466647], curReconnectFut=null, reconnectDone=null]]
我认为那是 Checkpoint Page Buffer，默认情况下它是数据区域大小的 20%。
您可以明确指定它以确保您不会忘记它，并相应地减小区域大小以确保您不会 运行 内存不足。
应该只适用于持久区域。
请注意，您还应该期望您的 OS 对其数据结构和块缓存占用几 GB，因此我认为您不应该将 120G 中的 116G 分配给 Ignite 的 Off-Heap。也不要忘记堆。