NIFI：尽管增加了堆大小，但仍出现内存不足错误

Question

我遇到了 NIFI 服务拒绝启动的问题。监控了一下nifi-app.log，发现是OutOfMemory错误。我增加了 bootstap.conf 文件中的内存。但是错误仍然存在。

附上nifi-app.log

    2017-12-19 17:08:37,133 ERROR [ActiveMQ InactivityMonitor ReadCheckTimer] org.apache.nifi.NiFi An Unknown Error Occurred in Thread Thread[ActiveMQ InactivityMonitor ReadCheckTimer,5,main]: java.lang.OutOfMemoryError: Java heap space
2017-12-19 17:08:37,134 ERROR [Provenance Maintenance Thread-2] org.apache.nifi.NiFi An Unknown Error Occurred in Thread Thread[Provenance Maintenance Thread-2,5,main]: java.lang.OutOfMemoryError: Java heap space
2017-12-19 17:08:37,134 ERROR [logback-2] org.apache.nifi.NiFi An Unknown Error Occurred in Thread Thread[logback-2,5,main]: java.lang.OutOfMemoryError: Java heap space
2017-12-19 17:08:37,133 ERROR [FileSystemRepository Workers Thread-2] org.apache.nifi.NiFi An Unknown Error Occurred in Thread Thread[FileSystemRepository Workers Thread-2,5,main]: java.lang.OutOfMemoryError: Java heap space
2017-12-19 17:08:37,135 ERROR [pool-15-thread-1] org.apache.nifi.NiFi An Unknown Error Occurred in Thread Thread[pool-15-thread-1,5,main]: java.lang.OutOfMemoryError: Java heap space
2017-12-19 17:08:37,133 ERROR [Flow Service Tasks Thread-1] org.apache.nifi.NiFi An Unknown Error Occurred in Thread Thread[Flow Service Tasks Thread-1,5,main]: java.lang.OutOfMemoryError: Java heap space
2017-12-19 17:08:39,496 ERROR [Provenance Maintenance Thread-2] org.apache.nifi.NiFi 
java.lang.OutOfMemoryError: Java heap space
2017-12-19 17:08:39,496 ERROR [logback-2] org.apache.nifi.NiFi 
java.lang.OutOfMemoryError: Java heap space
2017-12-19 17:08:39,496 ERROR [FileSystemRepository Workers Thread-2] org.apache.nifi.NiFi 
java.lang.OutOfMemoryError: Java heap space
2017-12-19 17:08:38,482 ERROR [Flow Service Tasks Thread-1] org.apache.nifi.NiFi 
java.lang.OutOfMemoryError: Java heap space
2017-12-19 17:08:38,482 ERROR [Event-Driven Process Thread-1] org.apache.nifi.engine.FlowEngine A flow controller task execution stopped abnormally
java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java heap space
        at java.util.concurrent.FutureTask.report(FutureTask.java:122)
        at java.util.concurrent.FutureTask.get(FutureTask.java:192)
        at org.apache.nifi.engine.FlowEngine.afterExecute(FlowEngine.java:100)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1150)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.OutOfMemoryError: Java heap space
2017-12-19 17:08:38,483 ERROR [FileSystemRepository Workers Thread-4] org.apache.nifi.engine.FlowEngine A flow controller task execution stopped abnormally
java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java heap space
        at java.util.concurrent.FutureTask.report(FutureTask.java:122)
        at java.util.concurrent.FutureTask.get(FutureTask.java:192)
        at org.apache.nifi.engine.FlowEngine.afterExecute(FlowEngine.java:100)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1150)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.OutOfMemoryError: Java heap space
2017-12-19 17:08:38,482 ERROR [pool-15-thread-1] org.apache.nifi.NiFi 
java.lang.OutOfMemoryError: Java heap space
2017-12-19 17:08:39,497 ERROR [ActiveMQ InactivityMonitor ReadCheckTimer] org.apache.nifi.NiFi 
java.lang.OutOfMemoryError: Java heap space
2017-12-19 17:08:40,509 INFO [Write-Ahead Local State Provider Maintenance] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@5351d0ef checkpointed with 2358 Records and 0 Swap Files in 114969 milliseconds (Stop-the-world time = 2361 milliseconds, Clear Edit Logs time = 936 millis), max Transaction ID 7622
2017-12-19 17:10:41,433 INFO [Write-Ahead Local State Provider Maintenance] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@5351d0ef checkpointed with 2358 Records and 0 Swap Files in 922 milliseconds (Stop-the-world time = 2 milliseconds, Clear Edit Logs time = 2 millis), max Transaction ID 7622

我也曾尝试删除流文件和内容存储库，因为担心它可能已损坏。但是，还是没有变化。

我能理解的是这是一个线程错误，但我找不到它的解决方法。

Answer 1

尝试关机，删除存储库，并在 nifi.properties 中设置 nifi.flowcontroller.autoResumeState=false，然后重新启动。

这将确保当 NiFi 启动时，整个流程将停止，因此您的流程无法带入更多数据并导致内存不足错误，然后您可以启动部分流程以查看问题所在是。

OOM 的典型原因是获取一个大流文件并将其拆分为数百万个小流文件，比如一个包含 1000 万行的 CSV 进入行数为 1 的 SplitText。

另一个原因可能是使用将流文件的全部内容读入内存的处理器，并且有一个多 GB 的流文件命中该处理器，或者使用 ExtractText 将大量内容提取到属性中。

NIFI：尽管增加了堆大小，但仍出现内存不足错误

NIFI: out of memory error inspite of increasing the heap size

apache-nifi