Jmeter - Hive giving java.lang.NoClassDefFoundError: org/apache/hadoop/security/SaslRpcServer
Jmeter - Hive giving java.lang.NoClassDefFoundError: org/apache/hadoop/security/SaslRpcServer
我正在使用 Jmeter 在 Hive 上创建负载。
虽然 运行 Jmeter 我在文件中收到以下错误 jmeter.log :
2021-02-05 10:11:02,927 ERROR o.a.j.JMeter: Uncaught exception in thread Thread[HiveThreadGroupTWANG 1-1,5,main]
java.lang.NoClassDefFoundError: org/apache/hadoop/security/SaslRpcServer
at org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge23.<clinit>(HadoopThriftAuthBridge23.java:66) ~[hive-exec-3.1.2.jar:3.1.2]
at org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge.getBridge(HadoopThriftAuthBridge.java:82) ~[hive-exec-3.1.2.jar:3.1.2]
at org.apache.hive.service.auth.KerberosSaslHelper.getKerberosTransport(KerberosSaslHelper.java:55) ~[hive-service-3.1.2.jar:3.1.2]
at org.apache.hive.jdbc.HiveConnection.createBinaryTransport(HiveConnection.java:601) ~[hive-jdbc-3.1.2.jar:3.1.2]
at org.apache.hive.jdbc.HiveConnection.openTransport(HiveConnection.java:341) ~[hive-jdbc-3.1.2.jar:3.1.2]
at org.apache.hive.jdbc.HiveConnection.<init>(HiveConnection.java:228) ~[hive-jdbc-3.1.2.jar:3.1.2]
at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:107) ~[hive-jdbc-3.1.2.jar:3.1.2]
at org.apache.commons.dbcp2.DriverConnectionFactory.createConnection(DriverConnectionFactory.java:52) ~[commons-dbcp2-2.8.0.jar:2.8.0]
at org.apache.commons.dbcp2.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:357) ~[commons-dbcp2-2.8.0.jar:2.8.0]
at org.apache.commons.dbcp2.BasicDataSource.validateConnectionFactory(BasicDataSource.java:103) ~[commons-dbcp2-2.8.0.jar:2.8.0]
at org.apache.commons.dbcp2.BasicDataSource.createPoolableConnectionFactory(BasicDataSource.java:652) ~[commons-dbcp2-2.8.0.jar:2.8.0]
at org.apache.commons.dbcp2.BasicDataSource.createDataSource(BasicDataSource.java:534) ~[commons-dbcp2-2.8.0.jar:2.8.0]
at org.apache.commons.dbcp2.BasicDataSource.getConnection(BasicDataSource.java:734) ~[commons-dbcp2-2.8.0.jar:2.8.0]
at org.apache.jmeter.protocol.jdbc.config.DataSourceElement$DataSourceComponentImpl.getConnection(DataSourceElement.java:362) ~[ApacheJMeter_jdbc.jar:5.4]
at org.apache.jmeter.protocol.jdbc.config.DataSourceElement.getConnection(DataSourceElement.java:198) ~[ApacheJMeter_jdbc.jar:5.4]
at org.apache.jmeter.protocol.jdbc.sampler.JDBCSampler.sample(JDBCSampler.java:80) ~[ApacheJMeter_jdbc.jar:5.4]
at org.apache.jmeter.threads.JMeterThread.doSampling(JMeterThread.java:635) ~[ApacheJMeter_core.jar:5.4]
at org.apache.jmeter.threads.JMeterThread.executeSamplePackage(JMeterThread.java:558) ~[ApacheJMeter_core.jar:5.4]
at org.apache.jmeter.threads.JMeterThread.processSampler(JMeterThread.java:489) ~[ApacheJMeter_core.jar:5.4]
at org.apache.jmeter.threads.JMeterThread.run(JMeterThread.java:256) ~[ApacheJMeter_core.jar:5.4]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_281]
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.security.SaslRpcServer
at java.net.URLClassLoader.findClass(URLClassLoader.java:382) ~[?:1.8.0_281]
at java.lang.ClassLoader.loadClass(ClassLoader.java:418) ~[?:1.8.0_281]
at java.lang.ClassLoader.loadClass(ClassLoader.java:351) ~[?:1.8.0_281]
... 21 more
我在 lib/ext 目录中添加的额外 jar 是：
old-hive-exec-3.1.0.3.1.5.0-152.jar
hadoop-common-3.1.1.3.1.5.0-152.jar.bkp
slf4j-log4j12-1.7.25.jar
slf4j-api-1.7.25.jar
old-hive-service-3.1.0.3.1.5.0-152.jar
old-hive-metastore-3.1.0.3.1.5.0-152.jar
old-hive-jdbc-handler-3.1.0.3.1.5.0-152.jar
old-hive-jdbc-3.1.0.3.1.5.0-152-standalone.jar.bkp
old-hive-jdbc-3.1.0.3.1.5.0-152.jar
log4j-1.2.17.jar
libthrift-0.9.3-1.jar
libfb303-0.9.3.jar
hive-service-3.1.2.jar
hive-metastore-3.1.2.jar
hive-jdbc-3.1.2.jar
hive-exec-3.1.2.jar
curator-client-5.1.0.jar
curator-framework-5.1.0.jar
zookeeper-3.6.2.jar
zookeeper-3.3.3.jar
hadoop-common-3.1.1.3.1.0.0-78.jar.bkp
将 hadoop-common-3.1.1.3.1.5.0-152.jar 移动到 lib 目录后，出现以下错误。
java.lang.NoClassDefFoundError: org/apache/commons/configuration2/Configuration
at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.<init>(DefaultMetricsSystem.java:43) ~[hadoop-common-3.1.1.3.1.5.0-152.jar:?]
at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.<clinit>(DefaultMetricsSystem.java:41) ~[hadoop-common-3.1.1.3.1.5.0-152.jar:?]
at org.apache.hadoop.security.UserGroupInformation$UgiMetrics.create(UserGroupInformation.java:143) ~[hadoop-common-3.1.1.3.1.5.0-152.jar:?]
at org.apache.hadoop.security.UserGroupInformation.<clinit>(UserGroupInformation.java:276) ~[hadoop-common-3.1.1.3.1.5.0-152.jar:?]
at org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge.createClientWithConf(HadoopThriftAuthBridge.java:95) ~[hive-exec-3.1.2.jar:3.1.2]
at org.apache.hive.service.auth.KerberosSaslHelper.getKerberosTransport(KerberosSaslHelper.java:55) ~[hive-service-3.1.2.jar:3.1.2]
at org.apache.hive.jdbc.HiveConnection.createBinaryTransport(HiveConnection.java:601) ~[hive-jdbc-3.1.2.jar:3.1.2]
at org.apache.hive.jdbc.HiveConnection.openTransport(HiveConnection.java:341) ~[hive-jdbc-3.1.2.jar:3.1.2]
at org.apache.hive.jdbc.HiveConnection.<init>(HiveConnection.java:228) ~[hive-jdbc-3.1.2.jar:3.1.2]
at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:107) ~[hive-jdbc-3.1.2.jar:3.1.2]
at org.apache.commons.dbcp2.DriverConnectionFactory.createConnection(DriverConnectionFactory.java:52) ~[commons-dbcp2-2.8.0.jar:2.8.0]
at org.apache.commons.dbcp2.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:357) ~[commons-dbcp2-2.8.0.jar:2.8.0]
at org.apache.commons.dbcp2.BasicDataSource.validateConnectionFactory(BasicDataSource.java:103) ~[commons-dbcp2-2.8.0.jar:2.8.0]
at org.apache.commons.dbcp2.BasicDataSource.createPoolableConnectionFactory(BasicDataSource.java:652) ~[commons-dbcp2-2.8.0.jar:2.8.0]
at org.apache.commons.dbcp2.BasicDataSource.createDataSource(BasicDataSource.java:534) ~[commons-dbcp2-2.8.0.jar:2.8.0]
at org.apache.commons.dbcp2.BasicDataSource.getConnection(BasicDataSource.java:734) ~[commons-dbcp2-2.8.0.jar:2.8.0]
at org.apache.jmeter.protocol.jdbc.config.DataSourceElement$DataSourceComponentImpl.getConnection(DataSourceElement.java:362) ~[ApacheJMeter_jdbc.jar:5.4]
at org.apache.jmeter.protocol.jdbc.config.DataSourceElement.getConnection(DataSourceElement.java:198) ~[ApacheJMeter_jdbc.jar:5.4]
at org.apache.jmeter.protocol.jdbc.sampler.JDBCSampler.sample(JDBCSampler.java:80) ~[ApacheJMeter_jdbc.jar:5.4]
at org.apache.jmeter.threads.JMeterThread.doSampling(JMeterThread.java:635) ~[ApacheJMeter_core.jar:5.4]
at org.apache.jmeter.threads.JMeterThread.executeSamplePackage(JMeterThread.java:558) ~[ApacheJMeter_core.jar:5.4]
at org.apache.jmeter.threads.JMeterThread.processSampler(JMeterThread.java:489) ~[ApacheJMeter_core.jar:5.4]
at org.apache.jmeter.threads.JMeterThread.run(JMeterThread.java:256) ~[ApacheJMeter_core.jar:5.4]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_281]
Caused by: java.lang.ClassNotFoundException: org.apache.commons.configuration2.Configuration
at java.net.URLClassLoader.findClass(URLClassLoader.java:382) ~[?:1.8.0_281]
at java.lang.ClassLoader.loadClass(ClassLoader.java:418) ~[?:1.8.0_281]
at java.lang.ClassLoader.loadClass(ClassLoader.java:351) ~[?:1.8.0_281]
... 24 more
- 您需要将
hadoop-common .jar 文件添加到 JMeter Classpath
- 依赖库应该放在 lib 文件夹，而不是 lib/ext 文件夹，lib/ext 是 JMeter Plugins
- 你有
hadoop-common-3.1.1.3.1.5.0-152.jar.bkp，我不知道什么是 .bkp 文件，但我相信它应该有 .jar 扩展名，很可能你下载了其他东西
- 您是否知道 jmeter-hadoop plugin? It looks like it can be useful for your use case, at least you will get dependency jars managed. See How to Load Test HBase with JMeter 文章了解更多详情
我正在使用 Jmeter 在 Hive 上创建负载。
虽然 运行 Jmeter 我在文件中收到以下错误 jmeter.log :
2021-02-05 10:11:02,927 ERROR o.a.j.JMeter: Uncaught exception in thread Thread[HiveThreadGroupTWANG 1-1,5,main]
java.lang.NoClassDefFoundError: org/apache/hadoop/security/SaslRpcServer
at org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge23.<clinit>(HadoopThriftAuthBridge23.java:66) ~[hive-exec-3.1.2.jar:3.1.2]
at org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge.getBridge(HadoopThriftAuthBridge.java:82) ~[hive-exec-3.1.2.jar:3.1.2]
at org.apache.hive.service.auth.KerberosSaslHelper.getKerberosTransport(KerberosSaslHelper.java:55) ~[hive-service-3.1.2.jar:3.1.2]
at org.apache.hive.jdbc.HiveConnection.createBinaryTransport(HiveConnection.java:601) ~[hive-jdbc-3.1.2.jar:3.1.2]
at org.apache.hive.jdbc.HiveConnection.openTransport(HiveConnection.java:341) ~[hive-jdbc-3.1.2.jar:3.1.2]
at org.apache.hive.jdbc.HiveConnection.<init>(HiveConnection.java:228) ~[hive-jdbc-3.1.2.jar:3.1.2]
at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:107) ~[hive-jdbc-3.1.2.jar:3.1.2]
at org.apache.commons.dbcp2.DriverConnectionFactory.createConnection(DriverConnectionFactory.java:52) ~[commons-dbcp2-2.8.0.jar:2.8.0]
at org.apache.commons.dbcp2.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:357) ~[commons-dbcp2-2.8.0.jar:2.8.0]
at org.apache.commons.dbcp2.BasicDataSource.validateConnectionFactory(BasicDataSource.java:103) ~[commons-dbcp2-2.8.0.jar:2.8.0]
at org.apache.commons.dbcp2.BasicDataSource.createPoolableConnectionFactory(BasicDataSource.java:652) ~[commons-dbcp2-2.8.0.jar:2.8.0]
at org.apache.commons.dbcp2.BasicDataSource.createDataSource(BasicDataSource.java:534) ~[commons-dbcp2-2.8.0.jar:2.8.0]
at org.apache.commons.dbcp2.BasicDataSource.getConnection(BasicDataSource.java:734) ~[commons-dbcp2-2.8.0.jar:2.8.0]
at org.apache.jmeter.protocol.jdbc.config.DataSourceElement$DataSourceComponentImpl.getConnection(DataSourceElement.java:362) ~[ApacheJMeter_jdbc.jar:5.4]
at org.apache.jmeter.protocol.jdbc.config.DataSourceElement.getConnection(DataSourceElement.java:198) ~[ApacheJMeter_jdbc.jar:5.4]
at org.apache.jmeter.protocol.jdbc.sampler.JDBCSampler.sample(JDBCSampler.java:80) ~[ApacheJMeter_jdbc.jar:5.4]
at org.apache.jmeter.threads.JMeterThread.doSampling(JMeterThread.java:635) ~[ApacheJMeter_core.jar:5.4]
at org.apache.jmeter.threads.JMeterThread.executeSamplePackage(JMeterThread.java:558) ~[ApacheJMeter_core.jar:5.4]
at org.apache.jmeter.threads.JMeterThread.processSampler(JMeterThread.java:489) ~[ApacheJMeter_core.jar:5.4]
at org.apache.jmeter.threads.JMeterThread.run(JMeterThread.java:256) ~[ApacheJMeter_core.jar:5.4]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_281]
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.security.SaslRpcServer
at java.net.URLClassLoader.findClass(URLClassLoader.java:382) ~[?:1.8.0_281]
at java.lang.ClassLoader.loadClass(ClassLoader.java:418) ~[?:1.8.0_281]
at java.lang.ClassLoader.loadClass(ClassLoader.java:351) ~[?:1.8.0_281]
... 21 more
我在 lib/ext 目录中添加的额外 jar 是：
old-hive-exec-3.1.0.3.1.5.0-152.jar
hadoop-common-3.1.1.3.1.5.0-152.jar.bkp
slf4j-log4j12-1.7.25.jar
slf4j-api-1.7.25.jar
old-hive-service-3.1.0.3.1.5.0-152.jar
old-hive-metastore-3.1.0.3.1.5.0-152.jar
old-hive-jdbc-handler-3.1.0.3.1.5.0-152.jar
old-hive-jdbc-3.1.0.3.1.5.0-152-standalone.jar.bkp
old-hive-jdbc-3.1.0.3.1.5.0-152.jar
log4j-1.2.17.jar
libthrift-0.9.3-1.jar
libfb303-0.9.3.jar
hive-service-3.1.2.jar
hive-metastore-3.1.2.jar
hive-jdbc-3.1.2.jar
hive-exec-3.1.2.jar
curator-client-5.1.0.jar
curator-framework-5.1.0.jar
zookeeper-3.6.2.jar
zookeeper-3.3.3.jar
hadoop-common-3.1.1.3.1.0.0-78.jar.bkp
将 hadoop-common-3.1.1.3.1.5.0-152.jar 移动到 lib 目录后，出现以下错误。
java.lang.NoClassDefFoundError: org/apache/commons/configuration2/Configuration
at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.<init>(DefaultMetricsSystem.java:43) ~[hadoop-common-3.1.1.3.1.5.0-152.jar:?]
at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.<clinit>(DefaultMetricsSystem.java:41) ~[hadoop-common-3.1.1.3.1.5.0-152.jar:?]
at org.apache.hadoop.security.UserGroupInformation$UgiMetrics.create(UserGroupInformation.java:143) ~[hadoop-common-3.1.1.3.1.5.0-152.jar:?]
at org.apache.hadoop.security.UserGroupInformation.<clinit>(UserGroupInformation.java:276) ~[hadoop-common-3.1.1.3.1.5.0-152.jar:?]
at org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge.createClientWithConf(HadoopThriftAuthBridge.java:95) ~[hive-exec-3.1.2.jar:3.1.2]
at org.apache.hive.service.auth.KerberosSaslHelper.getKerberosTransport(KerberosSaslHelper.java:55) ~[hive-service-3.1.2.jar:3.1.2]
at org.apache.hive.jdbc.HiveConnection.createBinaryTransport(HiveConnection.java:601) ~[hive-jdbc-3.1.2.jar:3.1.2]
at org.apache.hive.jdbc.HiveConnection.openTransport(HiveConnection.java:341) ~[hive-jdbc-3.1.2.jar:3.1.2]
at org.apache.hive.jdbc.HiveConnection.<init>(HiveConnection.java:228) ~[hive-jdbc-3.1.2.jar:3.1.2]
at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:107) ~[hive-jdbc-3.1.2.jar:3.1.2]
at org.apache.commons.dbcp2.DriverConnectionFactory.createConnection(DriverConnectionFactory.java:52) ~[commons-dbcp2-2.8.0.jar:2.8.0]
at org.apache.commons.dbcp2.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:357) ~[commons-dbcp2-2.8.0.jar:2.8.0]
at org.apache.commons.dbcp2.BasicDataSource.validateConnectionFactory(BasicDataSource.java:103) ~[commons-dbcp2-2.8.0.jar:2.8.0]
at org.apache.commons.dbcp2.BasicDataSource.createPoolableConnectionFactory(BasicDataSource.java:652) ~[commons-dbcp2-2.8.0.jar:2.8.0]
at org.apache.commons.dbcp2.BasicDataSource.createDataSource(BasicDataSource.java:534) ~[commons-dbcp2-2.8.0.jar:2.8.0]
at org.apache.commons.dbcp2.BasicDataSource.getConnection(BasicDataSource.java:734) ~[commons-dbcp2-2.8.0.jar:2.8.0]
at org.apache.jmeter.protocol.jdbc.config.DataSourceElement$DataSourceComponentImpl.getConnection(DataSourceElement.java:362) ~[ApacheJMeter_jdbc.jar:5.4]
at org.apache.jmeter.protocol.jdbc.config.DataSourceElement.getConnection(DataSourceElement.java:198) ~[ApacheJMeter_jdbc.jar:5.4]
at org.apache.jmeter.protocol.jdbc.sampler.JDBCSampler.sample(JDBCSampler.java:80) ~[ApacheJMeter_jdbc.jar:5.4]
at org.apache.jmeter.threads.JMeterThread.doSampling(JMeterThread.java:635) ~[ApacheJMeter_core.jar:5.4]
at org.apache.jmeter.threads.JMeterThread.executeSamplePackage(JMeterThread.java:558) ~[ApacheJMeter_core.jar:5.4]
at org.apache.jmeter.threads.JMeterThread.processSampler(JMeterThread.java:489) ~[ApacheJMeter_core.jar:5.4]
at org.apache.jmeter.threads.JMeterThread.run(JMeterThread.java:256) ~[ApacheJMeter_core.jar:5.4]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_281]
Caused by: java.lang.ClassNotFoundException: org.apache.commons.configuration2.Configuration
at java.net.URLClassLoader.findClass(URLClassLoader.java:382) ~[?:1.8.0_281]
at java.lang.ClassLoader.loadClass(ClassLoader.java:418) ~[?:1.8.0_281]
at java.lang.ClassLoader.loadClass(ClassLoader.java:351) ~[?:1.8.0_281]
... 24 more
- 您需要将
hadoop-common.jar 文件添加到 JMeter Classpath - 依赖库应该放在 lib 文件夹，而不是 lib/ext 文件夹，lib/ext 是 JMeter Plugins
- 你有
hadoop-common-3.1.1.3.1.5.0-152.jar.bkp，我不知道什么是 .bkp 文件，但我相信它应该有 .jar 扩展名，很可能你下载了其他东西 - 您是否知道 jmeter-hadoop plugin? It looks like it can be useful for your use case, at least you will get dependency jars managed. See How to Load Test HBase with JMeter 文章了解更多详情