如何修复 sonarqube 服务器 6.7.2.37468 上的内存不足问题
How to fix Out Of Memory issue on sonarqube server 6.7.2.37468
当 Sonarqube 服务器版本 6.7.2 在 AWS EC2 实例上 运行ning 时，大约每月一次，出现内存不足错误并且服务器关闭，我们需要手动重启它，有OS/JRE 级别或应用程序服务器级别存在内存泄漏或错误的设置配置。
一开始，由于内存不足问题，sonarqube 服务器每月宕机两次，我们按照 PRD 实例的官方设置文档，尝试将容量从 t2.small 垂直扩展到 t2.medium 在 EC2 实例上 运行 3 java 进程具有更多内存：
For Web:
sonar.web.javaOpts=-Xms256m -Xmx1024m -XX:+HeapDumpOnOutOfMemoryError
For ElasticSearch:
sonar.search.javaOpts=-Xms1024m -Xmx1024m -XX:+HeapDumpOnOutOfMemoryError
For Compute Engine:
sonar.ce.javaOpts=-Xms256m -Xmx1024m -XX:+HeapDumpOnOutOfMemoryError
但问题依然存在
--> Wrapper Started as Daemon
Launching a JVM...
Wrapper (Version 3.2.3) http://wrapper.tanukisoftware.org
Copyright 1999-2006 Tanuki Software, Inc. All Rights Reserved.
2019.04.29 06:59:38 INFO app[][o.s.a.AppFileSystem] Cleaning or creating
temp directory /opt/sonarqube-6.7.2/temp
2019.04.29 06:59:38 INFO app[][o.s.a.es.EsSettings] Elasticsearch
listening on /127.0.0.1:9001
2019.04.29 06:59:38 INFO app[][o.s.a.p.ProcessLauncherImpl] Launch
process[[key='es', ipcIndex=1, logFilenamePrefix=es]] from [/opt/sonarqube-6.7.2/elasticsearch]: /opt/sonarqube- 6.7.2/elasticsearch/bin/elasticsearch -Epath.conf=/opt/sonarqube- 6.7.2/temp/conf/es
2019.04.29 06:59:38 INFO app[][o.s.a.SchedulerImpl] Waiting for Elasticsearch to be up and running
2019.04.29 06:59:39 INFO app[][o.e.p.PluginsService] no modules loaded
2019.04.29 06:59:39 INFO app[][o.e.p.PluginsService] loaded plugin [org.elasticsearch.transport.Netty4Plugin]
2019.04.29 07:00:00 INFO app[][o.s.a.SchedulerImpl] Process[es] is up
2019.04.29 07:00:00 INFO app[][o.s.a.p.ProcessLauncherImpl] Launch process[[key='web', ipcIndex=2, logFilenamePrefix=web]] from [/opt/sonarqube- 6.7.2]: /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.191.b12- 1.el7_6.x86_64/jre/bin/java -Djava.awt.headless=true -Dfile.encoding=UTF-8 - Djava.io.tmpdir=/opt/sonarqube-6.7.2/temp -Xmx1024m -Xms256m - XX:+HeapDumpOnOutOfMemoryError -Dhttp.proxyHost= -Dhttp.proxyPort= - Dhttp.auth.ntlm.domain= -DsocksProxyHost= -DsocksProxyPort= -Dhttps.proxyHost= -Dhttps.proxyPort= -cp ./lib/common/*:./lib/server/*:/opt/sonarqube- 6.7.2/lib/jdbc/postgresql/postgresql-42.2.1.jar org.sonar.server.app.WebServer /opt/sonarqube-6.7.2/temp/sq-process5028201971842772853properties
2019.04.29 07:00:25 INFO app[][o.s.a.SchedulerImpl] Process[web] is up
2019.04.29 07:00:25 INFO app[][o.s.a.p.ProcessLauncherImpl] Launch process[[key='ce', ipcIndex=3, logFilenamePrefix=ce]] from [/opt/sonarqube- 6.7.2]: /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.191.b12- 1.el7_6.x86_64/jre/bin/java -Djava.awt.headless=true -Dfile.encoding=UTF-8 - Djava.io.tmpdir=/opt/sonarqube-6.7.2/temp -Xms256m -Xmx1024m - XX:+HeapDumpOnOutOfMemoryError -Dhttp.proxyHost= -Dhttp.proxyPort= - Dhttp.auth.ntlm.domain= -DsocksProxyHost= -DsocksProxyPort= -Dhttps.proxyHost= -Dhttps.proxyPort= -cp ./lib/common/*:./lib/server/*:./lib/ce/*:/opt/sonarqube- 6.7.2/lib/jdbc/postgresql/postgresql-42.2.1.jar org.sonar.ce.app.CeServer /opt/sonarqube-6.7.2/temp/sq-process7912202285523412339properties
2019.04.29 07:00:30 INFO app[][o.s.a.SchedulerImpl] Process[ce] is up
2019.04.29 07:00:30 INFO app[][o.s.a.SchedulerImpl] SonarQube is up
2019.04.29 15:10:28 INFO app[][o.s.a.SchedulerImpl] Process [ce] is stopped
2019.04.29 15:10:31 INFO app[][o.s.a.SchedulerImpl] Process [web] is stopped
2019.04.29 15:10:31 WARN app[][o.s.a.p.AbstractProcessMonitor] Process exited with exit value [es]: 143
2019.04.29 15:10:31 INFO app[][o.s.a.SchedulerImpl] Process [es] is stopped
2019.04.29 15:10:31 INFO app[][o.s.a.SchedulerImpl] SonarQube is restarting
2019.04.29 15:10:31 WARN app[][o.s.a.p.AbstractProcessMonitor] Process exited with exit value [es]: 143
2019.04.29 15:10:31 INFO app[][o.s.a.AppFileSystem] Cleaning or creating temp directory /opt/sonarqube-6.7.2/temp
2019.04.29 15:10:31 INFO app[][o.s.a.es.EsSettings] Elasticsearch listening on /127.0.0.1:9001
2019.04.29 15:10:31 INFO app[][o.s.a.p.ProcessLauncherImpl] Launch process[[key='es', ipcIndex=1, logFilenamePrefix=es]] from [/opt/sonarqube- 6.7.2/elasticsearch]: /opt/sonarqube-6.7.2/elasticsearch/bin/elasticsearch - Epath.conf=/opt/sonarqube-6.7.2/temp/conf/es
2019.04.29 15:10:31 INFO app[][o.e.p.PluginsService] no modules loaded
2019.04.29 15:10:31 INFO app[][o.e.p.PluginsService] loaded plugin [org.elasticsearch.transport.Netty4Plugin]
2019.04.29 15:10:51 INFO app[][o.s.a.SchedulerImpl] Process[es] is up
2019.04.29 15:10:51 INFO app[][o.s.a.p.ProcessLauncherImpl] Launch process[[key='web', ipcIndex=2, logFilenamePrefix=web]] from [/opt/sonarqube- 6.7.2]: /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.191.b12- 1.el7_6.x86_64/jre/bin/java -Djava.awt.headless=true -Dfile.encoding=UTF-8 - Djava.io.tmpdir=/opt/sonarqube-6.7.2/temp -Xmx1024m -Xms256m - XX:+HeapDumpOnOutOfMemoryError -Dhttp.proxyHost= -Dhttp.proxyPort= - Dhttp.auth.ntlm.domain= -DsocksProxyHost= -DsocksProxyPort= -Dhttps.proxyHost= -Dhttps.proxyPort= -cp ./lib/common/*:./lib/server/*:/opt/sonarqube- 6.7.2/lib/jdbc/postgresql/postgresql-42.2.1.jar org.sonar.server.app.WebServer /opt/sonarqube-6.7.2/temp/sq-process2158511348006070798properties
2019.04.29 15:11:12 INFO app[][o.s.a.SchedulerImpl] Process[web] is up
2019.04.29 15:11:12 INFO app[][o.s.a.p.ProcessLauncherImpl] Launch process[[key='ce', ipcIndex=3, logFilenamePrefix=ce]] from [/opt/sonarqube- 6.7.2]: /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.191.b12- 1.el7_6.x86_64/jre/bin/java -Djava.awt.headless=true -Dfile.encoding=UTF-8 - Djava.io.tmpdir=/opt/sonarqube-6.7.2/temp -Xms256m -Xmx1024m - XX:+HeapDumpOnOutOfMemoryError -Dhttp.proxyHost= -Dhttp.proxyPort= - Dhttp.auth.ntlm.domain= -DsocksProxyHost= -DsocksProxyPort= -Dhttps.proxyHost= -Dhttps.proxyPort= -cp ./lib/common/*:./lib/server/*:./lib/ce/*:/opt/sonarqube- 6.7.2/lib/jdbc/postgresql/postgresql-42.2.1.jar org.sonar.ce.app.CeServer /opt/sonarqube-6.7.2/temp/sq-process1917579452884722345properties
2019.04.29 15:11:17 INFO app[][o.s.a.SchedulerImpl] Process[ce] is up
2019.04.29 15:11:17 INFO app[][o.s.a.SchedulerImpl] SonarQube is up
OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x00000000dbc80000, 43515904, 0) failed; error='Cannot allocate memory' (errno=12)
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 43515904 bytes for committing reserved memory.
# An error report file with more information is saved as:
# /opt/sonarqube-6.7.2/hs_err_pid10967.log
2019.05.14 09:56:08 WARN app[][o.s.a.p.AbstractProcessMonitor] Process exited with exit value [ce]: 1
2019.05.14 09:56:08 INFO app[][o.s.a.SchedulerImpl] Process [ce] is stopped
2019.05.14 09:56:11 INFO app[][o.s.a.SchedulerImpl] Process [web] is stopped
2019.05.14 09:56:11 INFO app[][o.s.a.SchedulerImpl] Process [es] is stopped
2019.05.14 09:56:11 WARN app[][o.s.a.p.AbstractProcessMonitor] Process exited with exit value [es]: 143
2019.05.14 09:56:11 INFO app[][o.s.a.SchedulerImpl] SonarQube is stopped
<-- Wrapper Stopped
我们希望 Sonarqube 服务器启动并 运行ning 没有足够的内存供 JRE 继续运行。如果你需要/opt/sonarqube-6.7.2/hs_err_pid10967.log中的JVM转储，如果这有助于调试内存问题，我也可以提供。
感谢您的帮助和支持。
看来为 Compute Engine 分配的内存不足。
为了修复它，您需要至少升级到 t2.large 并将分配给 Compute Engine 的内存大小增加到 2 或 4Gb。
当 Sonarqube 服务器版本 6.7.2 在 AWS EC2 实例上 运行ning 时，大约每月一次，出现内存不足错误并且服务器关闭，我们需要手动重启它，有OS/JRE 级别或应用程序服务器级别存在内存泄漏或错误的设置配置。
一开始，由于内存不足问题，sonarqube 服务器每月宕机两次，我们按照 PRD 实例的官方设置文档，尝试将容量从 t2.small 垂直扩展到 t2.medium 在 EC2 实例上 运行 3 java 进程具有更多内存：
For Web:
sonar.web.javaOpts=-Xms256m -Xmx1024m -XX:+HeapDumpOnOutOfMemoryError
For ElasticSearch:
sonar.search.javaOpts=-Xms1024m -Xmx1024m -XX:+HeapDumpOnOutOfMemoryError
For Compute Engine:
sonar.ce.javaOpts=-Xms256m -Xmx1024m -XX:+HeapDumpOnOutOfMemoryError
但问题依然存在
--> Wrapper Started as Daemon
Launching a JVM...
Wrapper (Version 3.2.3) http://wrapper.tanukisoftware.org
Copyright 1999-2006 Tanuki Software, Inc. All Rights Reserved.
2019.04.29 06:59:38 INFO app[][o.s.a.AppFileSystem] Cleaning or creating
temp directory /opt/sonarqube-6.7.2/temp
2019.04.29 06:59:38 INFO app[][o.s.a.es.EsSettings] Elasticsearch
listening on /127.0.0.1:9001
2019.04.29 06:59:38 INFO app[][o.s.a.p.ProcessLauncherImpl] Launch
process[[key='es', ipcIndex=1, logFilenamePrefix=es]] from [/opt/sonarqube-6.7.2/elasticsearch]: /opt/sonarqube- 6.7.2/elasticsearch/bin/elasticsearch -Epath.conf=/opt/sonarqube- 6.7.2/temp/conf/es
2019.04.29 06:59:38 INFO app[][o.s.a.SchedulerImpl] Waiting for Elasticsearch to be up and running
2019.04.29 06:59:39 INFO app[][o.e.p.PluginsService] no modules loaded
2019.04.29 06:59:39 INFO app[][o.e.p.PluginsService] loaded plugin [org.elasticsearch.transport.Netty4Plugin]
2019.04.29 07:00:00 INFO app[][o.s.a.SchedulerImpl] Process[es] is up
2019.04.29 07:00:00 INFO app[][o.s.a.p.ProcessLauncherImpl] Launch process[[key='web', ipcIndex=2, logFilenamePrefix=web]] from [/opt/sonarqube- 6.7.2]: /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.191.b12- 1.el7_6.x86_64/jre/bin/java -Djava.awt.headless=true -Dfile.encoding=UTF-8 - Djava.io.tmpdir=/opt/sonarqube-6.7.2/temp -Xmx1024m -Xms256m - XX:+HeapDumpOnOutOfMemoryError -Dhttp.proxyHost= -Dhttp.proxyPort= - Dhttp.auth.ntlm.domain= -DsocksProxyHost= -DsocksProxyPort= -Dhttps.proxyHost= -Dhttps.proxyPort= -cp ./lib/common/*:./lib/server/*:/opt/sonarqube- 6.7.2/lib/jdbc/postgresql/postgresql-42.2.1.jar org.sonar.server.app.WebServer /opt/sonarqube-6.7.2/temp/sq-process5028201971842772853properties
2019.04.29 07:00:25 INFO app[][o.s.a.SchedulerImpl] Process[web] is up
2019.04.29 07:00:25 INFO app[][o.s.a.p.ProcessLauncherImpl] Launch process[[key='ce', ipcIndex=3, logFilenamePrefix=ce]] from [/opt/sonarqube- 6.7.2]: /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.191.b12- 1.el7_6.x86_64/jre/bin/java -Djava.awt.headless=true -Dfile.encoding=UTF-8 - Djava.io.tmpdir=/opt/sonarqube-6.7.2/temp -Xms256m -Xmx1024m - XX:+HeapDumpOnOutOfMemoryError -Dhttp.proxyHost= -Dhttp.proxyPort= - Dhttp.auth.ntlm.domain= -DsocksProxyHost= -DsocksProxyPort= -Dhttps.proxyHost= -Dhttps.proxyPort= -cp ./lib/common/*:./lib/server/*:./lib/ce/*:/opt/sonarqube- 6.7.2/lib/jdbc/postgresql/postgresql-42.2.1.jar org.sonar.ce.app.CeServer /opt/sonarqube-6.7.2/temp/sq-process7912202285523412339properties
2019.04.29 07:00:30 INFO app[][o.s.a.SchedulerImpl] Process[ce] is up
2019.04.29 07:00:30 INFO app[][o.s.a.SchedulerImpl] SonarQube is up
2019.04.29 15:10:28 INFO app[][o.s.a.SchedulerImpl] Process [ce] is stopped
2019.04.29 15:10:31 INFO app[][o.s.a.SchedulerImpl] Process [web] is stopped
2019.04.29 15:10:31 WARN app[][o.s.a.p.AbstractProcessMonitor] Process exited with exit value [es]: 143
2019.04.29 15:10:31 INFO app[][o.s.a.SchedulerImpl] Process [es] is stopped
2019.04.29 15:10:31 INFO app[][o.s.a.SchedulerImpl] SonarQube is restarting
2019.04.29 15:10:31 WARN app[][o.s.a.p.AbstractProcessMonitor] Process exited with exit value [es]: 143
2019.04.29 15:10:31 INFO app[][o.s.a.AppFileSystem] Cleaning or creating temp directory /opt/sonarqube-6.7.2/temp
2019.04.29 15:10:31 INFO app[][o.s.a.es.EsSettings] Elasticsearch listening on /127.0.0.1:9001
2019.04.29 15:10:31 INFO app[][o.s.a.p.ProcessLauncherImpl] Launch process[[key='es', ipcIndex=1, logFilenamePrefix=es]] from [/opt/sonarqube- 6.7.2/elasticsearch]: /opt/sonarqube-6.7.2/elasticsearch/bin/elasticsearch - Epath.conf=/opt/sonarqube-6.7.2/temp/conf/es
2019.04.29 15:10:31 INFO app[][o.e.p.PluginsService] no modules loaded
2019.04.29 15:10:31 INFO app[][o.e.p.PluginsService] loaded plugin [org.elasticsearch.transport.Netty4Plugin]
2019.04.29 15:10:51 INFO app[][o.s.a.SchedulerImpl] Process[es] is up
2019.04.29 15:10:51 INFO app[][o.s.a.p.ProcessLauncherImpl] Launch process[[key='web', ipcIndex=2, logFilenamePrefix=web]] from [/opt/sonarqube- 6.7.2]: /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.191.b12- 1.el7_6.x86_64/jre/bin/java -Djava.awt.headless=true -Dfile.encoding=UTF-8 - Djava.io.tmpdir=/opt/sonarqube-6.7.2/temp -Xmx1024m -Xms256m - XX:+HeapDumpOnOutOfMemoryError -Dhttp.proxyHost= -Dhttp.proxyPort= - Dhttp.auth.ntlm.domain= -DsocksProxyHost= -DsocksProxyPort= -Dhttps.proxyHost= -Dhttps.proxyPort= -cp ./lib/common/*:./lib/server/*:/opt/sonarqube- 6.7.2/lib/jdbc/postgresql/postgresql-42.2.1.jar org.sonar.server.app.WebServer /opt/sonarqube-6.7.2/temp/sq-process2158511348006070798properties
2019.04.29 15:11:12 INFO app[][o.s.a.SchedulerImpl] Process[web] is up
2019.04.29 15:11:12 INFO app[][o.s.a.p.ProcessLauncherImpl] Launch process[[key='ce', ipcIndex=3, logFilenamePrefix=ce]] from [/opt/sonarqube- 6.7.2]: /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.191.b12- 1.el7_6.x86_64/jre/bin/java -Djava.awt.headless=true -Dfile.encoding=UTF-8 - Djava.io.tmpdir=/opt/sonarqube-6.7.2/temp -Xms256m -Xmx1024m - XX:+HeapDumpOnOutOfMemoryError -Dhttp.proxyHost= -Dhttp.proxyPort= - Dhttp.auth.ntlm.domain= -DsocksProxyHost= -DsocksProxyPort= -Dhttps.proxyHost= -Dhttps.proxyPort= -cp ./lib/common/*:./lib/server/*:./lib/ce/*:/opt/sonarqube- 6.7.2/lib/jdbc/postgresql/postgresql-42.2.1.jar org.sonar.ce.app.CeServer /opt/sonarqube-6.7.2/temp/sq-process1917579452884722345properties
2019.04.29 15:11:17 INFO app[][o.s.a.SchedulerImpl] Process[ce] is up
2019.04.29 15:11:17 INFO app[][o.s.a.SchedulerImpl] SonarQube is up
OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x00000000dbc80000, 43515904, 0) failed; error='Cannot allocate memory' (errno=12)
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 43515904 bytes for committing reserved memory.
# An error report file with more information is saved as:
# /opt/sonarqube-6.7.2/hs_err_pid10967.log
2019.05.14 09:56:08 WARN app[][o.s.a.p.AbstractProcessMonitor] Process exited with exit value [ce]: 1
2019.05.14 09:56:08 INFO app[][o.s.a.SchedulerImpl] Process [ce] is stopped
2019.05.14 09:56:11 INFO app[][o.s.a.SchedulerImpl] Process [web] is stopped
2019.05.14 09:56:11 INFO app[][o.s.a.SchedulerImpl] Process [es] is stopped
2019.05.14 09:56:11 WARN app[][o.s.a.p.AbstractProcessMonitor] Process exited with exit value [es]: 143
2019.05.14 09:56:11 INFO app[][o.s.a.SchedulerImpl] SonarQube is stopped
<-- Wrapper Stopped
我们希望 Sonarqube 服务器启动并 运行ning 没有足够的内存供 JRE 继续运行。如果你需要/opt/sonarqube-6.7.2/hs_err_pid10967.log中的JVM转储，如果这有助于调试内存问题，我也可以提供。
感谢您的帮助和支持。
看来为 Compute Engine 分配的内存不足。 为了修复它，您需要至少升级到 t2.large 并将分配给 Compute Engine 的内存大小增加到 2 或 4Gb。