JSON路径2.3.0与hadoop 2.7环境冲突JSON-smart1.2.0.jar
JSON Path 2.3.0 conflicts with hadoop 2.7 Environment JSON-smart1.2.0.jar
- 在 HDFS 位置
中输入 JSON
- 需要解析 JSON 并聚合结果
- 我正在使用 PIG UDF，它使用 JSON-路径库
- 在hadoop2.7环境jar上：json-smart1.2,json-path1.2是hardbinded
每当我执行 PIG Mapreduce 时都会抛出异常
java.lang.NoSuchFieldError: defaultReader
at com.jayway.jsonpath.spi.json.JsonSmartJsonProvider.<init>(JsonSmartJsonProvider.java:39)
at com.jayway.jsonpath.internal.DefaultsImpl.jsonProvider(DefaultsImpl.java:21)
at com.jayway.jsonpath.Configuration.defaultConfiguration(Configuration.java:174)
at com.jayway.jsonpath.internal.JsonContext.<init>(JsonContext.java:52)
at com.jayway.jsonpath.JsonPath.parse(JsonPath.java:596)
为了解决问题尝试了以下选项
选项 1：
尝试设置注册 json-smart2.3.jar & json-path2.3.0.jar
但没有令人满意的结果（因为它引用的 Jar 是 json-path1.2.jar）
选项 2：
将我的模块依赖项降级为 json-path1.2.jar
没有结果
选项 3：
使用自定义 classLoaders 尝试加载 JSON-path2.3.0 jar 的 jar 吧
加载 class 进入 Org.slf4j 绑定问题
识别出多个绑定路径，但 sun.misc classloader
出现问题
Failed to instantiate SLF4J LoggerFactory
Reported exception:
java.lang.NullPointerException
at sun.net.util.URLUtil.urlNoFragString(URLUtil.java:50)
at sun.misc.URLClassPath.getLoader(URLClassPath.java:485)
at sun.misc.URLClassPath.getNextLoader(URLClassPath.java:457)
at sun.misc.URLClassPath.access0(URLClassPath.java:64)
at sun.misc.URLClassPath.next(URLClassPath.java:239)
at sun.misc.URLClassPath.hasMoreElements(URLClassPath.java:250)
at java.net.URLClassLoader.run(URLClassLoader.java:601)
at java.net.URLClassLoader.run(URLClassLoader.java:599)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.next(URLClassLoader.java:598)
at java.net.URLClassLoader.hasMoreElements(URLClassLoader.java:623)
at sun.misc.CompoundEnumeration.next(CompoundEnumeration.java:45)
at sun.misc.CompoundEnumeration.hasMoreElements(CompoundEnumeration.java:54)
at org.slf4j.LoggerFactory.findPossibleStaticLoggerBinderPathSet(LoggerFactory.java:238)
at org.slf4j.LoggerFactory.bind(LoggerFactory.java:138)
at org.slf4j.LoggerFactory.performInitialization(LoggerFactory.java:120)
at org.slf4j.LoggerFactory.getILoggerFactory(LoggerFactory.java:331)
at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:283)
at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:304)
at com.jayway.jsonpath.internal.JsonContext.<clinit>(JsonContext.java:41)
at com.jayway.jsonpath.internal.ParseContextImpl.parse(ParseContextImpl.java:38)
at com.jayway.jsonpath.JsonPath.parse(JsonPath.java:599)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.optum.pdm.ReferenceDataUpdate.addURL(ReferenceDataUpdate.java:112)
at com.optum.pdm.ReferenceDataUpdate.main(ReferenceDataUpdate.java:124)
Exception in thread "main" java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.optum.pdm.ReferenceDataUpdate.addURL(ReferenceDataUpdate.java:112)
at com.optum.pdm.ReferenceDataUpdate.main(ReferenceDataUpdate.java:124)
Caused by: java.lang.ExceptionInInitializerError
at com.jayway.jsonpath.internal.ParseContextImpl.parse(ParseContextImpl.java:38)
at com.jayway.jsonpath.JsonPath.parse(JsonPath.java:599)
... 6 more
Caused by: java.lang.IllegalStateException: Unexpected initialization failure
at org.slf4j.LoggerFactory.bind(LoggerFactory.java:167)
at org.slf4j.LoggerFactory.performInitialization(LoggerFactory.java:120)
at org.slf4j.LoggerFactory.getILoggerFactory(LoggerFactory.java:331)
at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:283)
at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:304)
at com.jayway.jsonpath.internal.JsonContext.<clinit>(JsonContext.java:41)
... 8 more
Caused by: java.lang.NullPointerException
at sun.net.util.URLUtil.urlNoFragString(URLUtil.java:50)
at sun.misc.URLClassPath.getLoader(URLClassPath.java:485)
at sun.misc.URLClassPath.getNextLoader(URLClassPath.java:457)
at sun.misc.URLClassPath.access0(URLClassPath.java:64)
at sun.misc.URLClassPath.next(URLClassPath.java:239)
at sun.misc.URLClassPath.hasMoreElements(URLClassPath.java:250)
at java.net.URLClassLoader.run(URLClassLoader.java:601)
at java.net.URLClassLoader.run(URLClassLoader.java:599)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.next(URLClassLoader.java:598)
at java.net.URLClassLoader.hasMoreElements(URLClassLoader.java:623)
at sun.misc.CompoundEnumeration.next(CompoundEnumeration.java:45)
at sun.misc.CompoundEnumeration.hasMoreElements(CompoundEnumeration.java:54)
at org.slf4j.LoggerFactory.findPossibleStaticLoggerBinderPathSet(LoggerFactory.java:238)
at org.slf4j.LoggerFactory.bind(LoggerFactory.java:138)
... 13 more
有人可以建议我解决这个问题吗，可以找到一个 Whosebug link，它讲述了 weblogic 而不是可以应用于 Hadoop2.7 的通用解决方案（ )
由于 hadoop 环境（Pig、hdfs、Hive 等）正在使用 json-path-2.3.0，其更好的用户 Mapper 逻辑应该使用另一个版本“jsonpath-1.0.jar”解决问题
为load/parse提供自定义实现JSON，这样我们就可以避免使用Json-smart-2.x/1.x of Hadoop/lib
public static void changeJsonPathConfig() {
if (!configChanged) {
Configuration.setDefaults(new Configuration.Defaults() {
private final JsonProvider jsonProvider = new GsonJsonProvider(
new GsonBuilder().serializeNulls().create());
private final MappingProvider mappingProvider = new GsonMappingProvider();
@Override
public JsonProvider jsonProvider() {
return jsonProvider;
}
@Override
public MappingProvider mappingProvider() {
return mappingProvider;
}
@Override
public Set<Option> options() {
return EnumSet.noneOf(Option.class);
}
});
configChanged = true;
}
}
- 在 HDFS 位置 中输入 JSON
- 需要解析 JSON 并聚合结果
- 我正在使用 PIG UDF，它使用 JSON-路径库
- 在hadoop2.7环境jar上：json-smart1.2,json-path1.2是hardbinded
每当我执行 PIG Mapreduce 时都会抛出异常
java.lang.NoSuchFieldError: defaultReader at com.jayway.jsonpath.spi.json.JsonSmartJsonProvider.<init>(JsonSmartJsonProvider.java:39) at com.jayway.jsonpath.internal.DefaultsImpl.jsonProvider(DefaultsImpl.java:21) at com.jayway.jsonpath.Configuration.defaultConfiguration(Configuration.java:174) at com.jayway.jsonpath.internal.JsonContext.<init>(JsonContext.java:52) at com.jayway.jsonpath.JsonPath.parse(JsonPath.java:596)
为了解决问题尝试了以下选项
选项 1： 尝试设置注册 json-smart2.3.jar & json-path2.3.0.jar 但没有令人满意的结果（因为它引用的 Jar 是 json-path1.2.jar）
选项 2： 将我的模块依赖项降级为 json-path1.2.jar 没有结果
选项 3： 使用自定义 classLoaders 尝试加载 JSON-path2.3.0 jar 的 jar 吧 加载 class 进入 Org.slf4j 绑定问题
识别出多个绑定路径，但 sun.misc classloader
出现问题Failed to instantiate SLF4J LoggerFactory
Reported exception:
java.lang.NullPointerException
at sun.net.util.URLUtil.urlNoFragString(URLUtil.java:50)
at sun.misc.URLClassPath.getLoader(URLClassPath.java:485)
at sun.misc.URLClassPath.getNextLoader(URLClassPath.java:457)
at sun.misc.URLClassPath.access0(URLClassPath.java:64)
at sun.misc.URLClassPath.next(URLClassPath.java:239)
at sun.misc.URLClassPath.hasMoreElements(URLClassPath.java:250)
at java.net.URLClassLoader.run(URLClassLoader.java:601)
at java.net.URLClassLoader.run(URLClassLoader.java:599)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.next(URLClassLoader.java:598)
at java.net.URLClassLoader.hasMoreElements(URLClassLoader.java:623)
at sun.misc.CompoundEnumeration.next(CompoundEnumeration.java:45)
at sun.misc.CompoundEnumeration.hasMoreElements(CompoundEnumeration.java:54)
at org.slf4j.LoggerFactory.findPossibleStaticLoggerBinderPathSet(LoggerFactory.java:238)
at org.slf4j.LoggerFactory.bind(LoggerFactory.java:138)
at org.slf4j.LoggerFactory.performInitialization(LoggerFactory.java:120)
at org.slf4j.LoggerFactory.getILoggerFactory(LoggerFactory.java:331)
at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:283)
at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:304)
at com.jayway.jsonpath.internal.JsonContext.<clinit>(JsonContext.java:41)
at com.jayway.jsonpath.internal.ParseContextImpl.parse(ParseContextImpl.java:38)
at com.jayway.jsonpath.JsonPath.parse(JsonPath.java:599)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.optum.pdm.ReferenceDataUpdate.addURL(ReferenceDataUpdate.java:112)
at com.optum.pdm.ReferenceDataUpdate.main(ReferenceDataUpdate.java:124)
Exception in thread "main" java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.optum.pdm.ReferenceDataUpdate.addURL(ReferenceDataUpdate.java:112)
at com.optum.pdm.ReferenceDataUpdate.main(ReferenceDataUpdate.java:124)
Caused by: java.lang.ExceptionInInitializerError
at com.jayway.jsonpath.internal.ParseContextImpl.parse(ParseContextImpl.java:38)
at com.jayway.jsonpath.JsonPath.parse(JsonPath.java:599)
... 6 more
Caused by: java.lang.IllegalStateException: Unexpected initialization failure
at org.slf4j.LoggerFactory.bind(LoggerFactory.java:167)
at org.slf4j.LoggerFactory.performInitialization(LoggerFactory.java:120)
at org.slf4j.LoggerFactory.getILoggerFactory(LoggerFactory.java:331)
at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:283)
at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:304)
at com.jayway.jsonpath.internal.JsonContext.<clinit>(JsonContext.java:41)
... 8 more
Caused by: java.lang.NullPointerException
at sun.net.util.URLUtil.urlNoFragString(URLUtil.java:50)
at sun.misc.URLClassPath.getLoader(URLClassPath.java:485)
at sun.misc.URLClassPath.getNextLoader(URLClassPath.java:457)
at sun.misc.URLClassPath.access0(URLClassPath.java:64)
at sun.misc.URLClassPath.next(URLClassPath.java:239)
at sun.misc.URLClassPath.hasMoreElements(URLClassPath.java:250)
at java.net.URLClassLoader.run(URLClassLoader.java:601)
at java.net.URLClassLoader.run(URLClassLoader.java:599)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.next(URLClassLoader.java:598)
at java.net.URLClassLoader.hasMoreElements(URLClassLoader.java:623)
at sun.misc.CompoundEnumeration.next(CompoundEnumeration.java:45)
at sun.misc.CompoundEnumeration.hasMoreElements(CompoundEnumeration.java:54)
at org.slf4j.LoggerFactory.findPossibleStaticLoggerBinderPathSet(LoggerFactory.java:238)
at org.slf4j.LoggerFactory.bind(LoggerFactory.java:138)
... 13 more
有人可以建议我解决这个问题吗，可以找到一个 Whosebug link，它讲述了 weblogic 而不是可以应用于 Hadoop2.7 的通用解决方案（
由于 hadoop 环境（Pig、hdfs、Hive 等）正在使用 json-path-2.3.0，其更好的用户 Mapper 逻辑应该使用另一个版本“jsonpath-1.0.jar”解决问题
为load/parse提供自定义实现JSON，这样我们就可以避免使用Json-smart-2.x/1.x of Hadoop/lib
public static void changeJsonPathConfig() {
if (!configChanged) {
Configuration.setDefaults(new Configuration.Defaults() {
private final JsonProvider jsonProvider = new GsonJsonProvider(
new GsonBuilder().serializeNulls().create());
private final MappingProvider mappingProvider = new GsonMappingProvider();
@Override
public JsonProvider jsonProvider() {
return jsonProvider;
}
@Override
public MappingProvider mappingProvider() {
return mappingProvider;
}
@Override
public Set<Option> options() {
return EnumSet.noneOf(Option.class);
}
});
configChanged = true;
}
}