Android 使用 androidx orchestrator 的仪器测试部分失败
Android Instrumentation Tests Paritally Failing with androidx orchestrator
我正在尝试 运行在 androidx orchestrator 下进行测试。
它似乎部分有效，但并未找到所有测试并且 运行。
我已将相关测试代码提取到我在项目 build.gradle 上调用的辅助方法中 build.gradle。
ext.enableTesting = { Project project ->
logger.lifecycle "Enable Testing: " + project.displayName
project.android {
defaultConfig {
testInstrumentationRunner "androidx.test.runner.AndroidJUnitRunner"
// runs pm clear after each test invocation
testInstrumentationRunnerArguments clearPackageData: 'true'
}
testOptions {
execution 'ANDROIDX_TEST_ORCHESTRATOR'
animationsDisabled true
unitTests {
includeAndroidResources = true
}
}
useLibrary 'android.test.runner'
useLibrary 'android.test.base'
useLibrary 'android.test.mock'
}
project.dependencies {
// Tests - Including NoOp
testImplementation 'junit:junit:4.12'
androidTestUtil 'androidx.test:orchestrator:1.2.0'
// Core library
androidTestImplementation 'androidx.test:core:1.2.0'
// AndroidJUnitRunner and JUnit Rules
androidTestImplementation 'androidx.test:runner:1.2.0'
androidTestImplementation 'androidx.test:rules:1.2.0'
// Assertions
androidTestImplementation 'androidx.test.ext:junit:1.1.1'
androidTestImplementation 'androidx.test.ext:truth:1.2.0'
// Mockito
testImplementation "org.mockito:mockito-core:${project.ext.mockitoVersion}"
testImplementation "org.mockito:mockito-android:${project.ext.mockitoVersion}"
androidTestImplementation "org.mockito:mockito-core:${project.ext.mockitoVersion}"
androidTestImplementation "org.mockito:mockito-android:${project.ext.mockitoVersion}"
}
// 'androidx.test.ext:truth:1.2.0' pulls in some conflicting
// versions with other things we need. In particular it has a more
// recent version of guava which then screws up grpc stuff.
project.configurations.all {
resolutionStrategy {
force 'com.google.errorprone:error_prone_annotations:2.1.2',
'com.google.code.findbugs:jsr305:3.0.2',
'com.google.guava:guava:26.0-android',
'org.checkerframework:checker-compat-qual:2.5.3'
}
}
}
18 个测试通过，logcat 这些测试具有以下内容：
06-14 03:13:10.129: D/TestExecutor(17920): Adding listener androidx.test.internal.runner.listener.LogRunListener
06-14 03:13:10.129: D/TestExecutor(17920): Adding listener androidx.test.orchestrator.instrumentationlistener.OrchestratedInstrumentationListener
06-14 03:13:10.129: D/TestExecutor(17920): Adding listener androidx.test.internal.runner.listener.ActivityFinisherRunListener
06-14 03:13:10.129: D/TestExecutor(17920): Adding listener androidx.test.internal.runner.listener.CoverageListener
06-14 03:13:10.135: I/TestRunner(17920): run started: 1 tests
06-14 03:13:10.139: I/TestRunner(17920): started: testSetAvailable(app.impl.AvailabilityManagerTest)
06-14 03:13:11.914: I/TestRunner(17920): finished: testSetAvailable(app.impl.AvailabilityManagerTest)
06-14 03:13:11.924: I/TestRunner(17920): run finished: 1 tests, 0 failed, 0 ignored
但是，另一个包中的测试失败：
06-14 03:15:07.671: D/TestExecutor(28814): Adding listener androidx.test.internal.runner.listener.LogRunListener
06-14 03:15:07.671: D/TestExecutor(28814): Adding listener androidx.test.orchestrator.instrumentationlistener.OrchestratedInstrumentationListener
06-14 03:15:07.671: D/TestExecutor(28814): Adding listener androidx.test.internal.runner.listener.ActivityFinisherRunListener
06-14 03:15:07.671: D/TestExecutor(28814): Adding listener androidx.test.internal.runner.listener.CoverageListener
06-14 03:15:07.676: I/TestRunner(28814): run started: 1 tests
06-14 03:15:07.676: E/TestRunner(28814): failed: Test mechanism
06-14 03:15:07.677: E/TestRunner(28814): ----- begin exception -----
06-14 03:15:07.677: E/TestRunner(28814): java.lang.IllegalStateException: Unable to send notification, callback is null
06-14 03:15:07.677: E/TestRunner(28814): at androidx.test.orchestrator.instrumentationlistener.OrchestratedInstrumentationListener.sendTestNotification(OrchestratedInstrumentationListener.java:163)
06-14 03:15:07.677: E/TestRunner(28814): at androidx.test.orchestrator.instrumentationlistener.OrchestratedInstrumentationListener.testRunStarted(OrchestratedInstrumentationListener.java:94)
06-14 03:15:07.677: E/TestRunner(28814): at org.junit.runner.notification.SynchronizedRunListener.testRunStarted(SynchronizedRunListener.java:35)
06-14 03:15:07.677: E/TestRunner(28814): at org.junit.runner.notification.RunNotifier.notifyListener(RunNotifier.java:91)
06-14 03:15:07.677: E/TestRunner(28814): at org.junit.runner.notification.RunNotifier$SafeNotifier.run(RunNotifier.java:72)
06-14 03:15:07.677: E/TestRunner(28814): at org.junit.runner.notification.RunNotifier.fireTestRunStarted(RunNotifier.java:88)
06-14 03:15:07.677: E/TestRunner(28814): at org.junit.runner.JUnitCore.run(JUnitCore.java:136)
06-14 03:15:07.677: E/TestRunner(28814): at org.junit.runner.JUnitCore.run(JUnitCore.java:115)
06-14 03:15:07.677: E/TestRunner(28814): at androidx.test.internal.runner.TestExecutor.execute(TestExecutor.java:56)
06-14 03:15:07.677: E/TestRunner(28814): at androidx.test.runner.AndroidJUnitRunner.onStart(AndroidJUnitRunner.java:392)
06-14 03:15:07.677: E/TestRunner(28814): at android.app.Instrumentation$InstrumentationThread.run(Instrumentation.java:2106)
06-14 03:15:07.677: E/TestRunner(28814): ----- end exception -----
06-14 03:15:07.680: I/TestRunner(28814): started: testMatcherWithMatchFound(app.impl.matcher.MatcherTest)
06-14 03:15:07.681: E/TestRunner(28814): failed: Test mechanism
06-14 03:15:07.681: E/TestRunner(28814): ----- begin exception -----
06-14 03:15:07.681: E/TestRunner(28814): java.lang.IllegalStateException: Unable to send notification, callback is null
06-14 03:15:07.681: E/TestRunner(28814): at androidx.test.orchestrator.instrumentationlistener.OrchestratedInstrumentationListener.sendTestNotification(OrchestratedInstrumentationListener.java:163)
06-14 03:15:07.681: E/TestRunner(28814): at androidx.test.orchestrator.instrumentationlistener.OrchestratedInstrumentationListener.testStarted(OrchestratedInstrumentationListener.java:114)
06-14 03:15:07.681: E/TestRunner(28814): at org.junit.runner.notification.SynchronizedRunListener.testStarted(SynchronizedRunListener.java:49)
06-14 03:15:07.681: E/TestRunner(28814): at org.junit.runner.notification.RunNotifier.notifyListener(RunNotifier.java:121)
06-14 03:15:07.681: E/TestRunner(28814): at org.junit.runner.notification.RunNotifier$SafeNotifier.run(RunNotifier.java:72)
06-14 03:15:07.681: E/TestRunner(28814): at org.junit.runner.notification.RunNotifier.fireTestStarted(RunNotifier.java:118)
06-14 03:15:07.681: E/TestRunner(28814): at org.junit.internal.runners.model.EachTestNotifier.fireTestStarted(EachTestNotifier.java:42)
06-14 03:15:07.681: E/TestRunner(28814): at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:323)
06-14 03:15:07.681: E/TestRunner(28814): at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
06-14 03:15:07.681: E/TestRunner(28814): at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
06-14 03:15:07.681: E/TestRunner(28814): at org.junit.runners.ParentRunner.run(ParentRunner.java:290)
06-14 03:15:07.681: E/TestRunner(28814): at org.junit.runners.ParentRunner.schedule(ParentRunner.java:71)
06-14 03:15:07.681: E/TestRunner(28814): at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
06-14 03:15:07.681: E/TestRunner(28814): at org.junit.runners.ParentRunner.access[=13=]0(ParentRunner.java:58)
06-14 03:15:07.681: E/TestRunner(28814): at org.junit.runners.ParentRunner.evaluate(ParentRunner.java:268)
06-14 03:15:07.681: E/TestRunner(28814): at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
06-14 03:15:07.681: E/TestRunner(28814): at org.junit.runners.Suite.runChild(Suite.java:128)
06-14 03:15:07.681: E/TestRunner(28814): at org.junit.runners.Suite.runChild(Suite.java:27)
06-14 03:15:07.681: E/TestRunner(28814): at org.junit.runners.ParentRunner.run(ParentRunner.java:290)
06-14 03:15:07.681: E/TestRunner(28814): at org.junit.runners.ParentRunner.schedule(ParentRunner.java:71)
06-14 03:15:07.681: E/TestRunner(28814): at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
06-14 03:15:07.681: E/TestRunner(28814): at org.junit.runners.ParentRunner.access[=13=]0(ParentRunner.java:58)
06-14 03:15:07.681: E/TestRunner(28814): at org.junit.runners.ParentRunner.evaluate(ParentRunner.java:268)
06-14 03:15:07.681: E/TestRunner(28814): at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
06-14 03:15:07.681: E/TestRunner(28814): at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
06-14 03:15:07.681: E/TestRunner(28814): at org.junit.runner.JUnitCore.run(JUnitCore.java:115)
06-14 03:15:07.681: E/TestRunner(28814): at androidx.test.internal.runner.TestExecutor.execute(TestExecutor.java:56)
06-14 03:15:07.681: E/TestRunner(28814): at androidx.test.runner.AndroidJUnitRunner.onStart(AndroidJUnitRunner.java:392)
06-14 03:15:07.681: E/TestRunner(28814): at android.app.Instrumentation$InstrumentationThread.run(Instrumentation.java:2106)
06-14 03:15:07.681: E/TestRunner(28814): ----- end exception -----
此后不再进行测试 运行 且 firebase 测试实验室超时构建。
关于更改什么以使我们的所有测试达到 运行 的任何建议？
提前致谢！
原来这是Orchestration框架的一个bug，目前最好的解决办法就是干脆不用Orchestration。我为此创建了一个问题：
我正在尝试 运行在 androidx orchestrator 下进行测试。
它似乎部分有效，但并未找到所有测试并且 运行。
我已将相关测试代码提取到我在项目 build.gradle 上调用的辅助方法中 build.gradle。
ext.enableTesting = { Project project ->
logger.lifecycle "Enable Testing: " + project.displayName
project.android {
defaultConfig {
testInstrumentationRunner "androidx.test.runner.AndroidJUnitRunner"
// runs pm clear after each test invocation
testInstrumentationRunnerArguments clearPackageData: 'true'
}
testOptions {
execution 'ANDROIDX_TEST_ORCHESTRATOR'
animationsDisabled true
unitTests {
includeAndroidResources = true
}
}
useLibrary 'android.test.runner'
useLibrary 'android.test.base'
useLibrary 'android.test.mock'
}
project.dependencies {
// Tests - Including NoOp
testImplementation 'junit:junit:4.12'
androidTestUtil 'androidx.test:orchestrator:1.2.0'
// Core library
androidTestImplementation 'androidx.test:core:1.2.0'
// AndroidJUnitRunner and JUnit Rules
androidTestImplementation 'androidx.test:runner:1.2.0'
androidTestImplementation 'androidx.test:rules:1.2.0'
// Assertions
androidTestImplementation 'androidx.test.ext:junit:1.1.1'
androidTestImplementation 'androidx.test.ext:truth:1.2.0'
// Mockito
testImplementation "org.mockito:mockito-core:${project.ext.mockitoVersion}"
testImplementation "org.mockito:mockito-android:${project.ext.mockitoVersion}"
androidTestImplementation "org.mockito:mockito-core:${project.ext.mockitoVersion}"
androidTestImplementation "org.mockito:mockito-android:${project.ext.mockitoVersion}"
}
// 'androidx.test.ext:truth:1.2.0' pulls in some conflicting
// versions with other things we need. In particular it has a more
// recent version of guava which then screws up grpc stuff.
project.configurations.all {
resolutionStrategy {
force 'com.google.errorprone:error_prone_annotations:2.1.2',
'com.google.code.findbugs:jsr305:3.0.2',
'com.google.guava:guava:26.0-android',
'org.checkerframework:checker-compat-qual:2.5.3'
}
}
}
18 个测试通过，logcat 这些测试具有以下内容：
06-14 03:13:10.129: D/TestExecutor(17920): Adding listener androidx.test.internal.runner.listener.LogRunListener
06-14 03:13:10.129: D/TestExecutor(17920): Adding listener androidx.test.orchestrator.instrumentationlistener.OrchestratedInstrumentationListener
06-14 03:13:10.129: D/TestExecutor(17920): Adding listener androidx.test.internal.runner.listener.ActivityFinisherRunListener
06-14 03:13:10.129: D/TestExecutor(17920): Adding listener androidx.test.internal.runner.listener.CoverageListener
06-14 03:13:10.135: I/TestRunner(17920): run started: 1 tests
06-14 03:13:10.139: I/TestRunner(17920): started: testSetAvailable(app.impl.AvailabilityManagerTest)
06-14 03:13:11.914: I/TestRunner(17920): finished: testSetAvailable(app.impl.AvailabilityManagerTest)
06-14 03:13:11.924: I/TestRunner(17920): run finished: 1 tests, 0 failed, 0 ignored
但是，另一个包中的测试失败：
06-14 03:15:07.671: D/TestExecutor(28814): Adding listener androidx.test.internal.runner.listener.LogRunListener
06-14 03:15:07.671: D/TestExecutor(28814): Adding listener androidx.test.orchestrator.instrumentationlistener.OrchestratedInstrumentationListener
06-14 03:15:07.671: D/TestExecutor(28814): Adding listener androidx.test.internal.runner.listener.ActivityFinisherRunListener
06-14 03:15:07.671: D/TestExecutor(28814): Adding listener androidx.test.internal.runner.listener.CoverageListener
06-14 03:15:07.676: I/TestRunner(28814): run started: 1 tests
06-14 03:15:07.676: E/TestRunner(28814): failed: Test mechanism
06-14 03:15:07.677: E/TestRunner(28814): ----- begin exception -----
06-14 03:15:07.677: E/TestRunner(28814): java.lang.IllegalStateException: Unable to send notification, callback is null
06-14 03:15:07.677: E/TestRunner(28814): at androidx.test.orchestrator.instrumentationlistener.OrchestratedInstrumentationListener.sendTestNotification(OrchestratedInstrumentationListener.java:163)
06-14 03:15:07.677: E/TestRunner(28814): at androidx.test.orchestrator.instrumentationlistener.OrchestratedInstrumentationListener.testRunStarted(OrchestratedInstrumentationListener.java:94)
06-14 03:15:07.677: E/TestRunner(28814): at org.junit.runner.notification.SynchronizedRunListener.testRunStarted(SynchronizedRunListener.java:35)
06-14 03:15:07.677: E/TestRunner(28814): at org.junit.runner.notification.RunNotifier.notifyListener(RunNotifier.java:91)
06-14 03:15:07.677: E/TestRunner(28814): at org.junit.runner.notification.RunNotifier$SafeNotifier.run(RunNotifier.java:72)
06-14 03:15:07.677: E/TestRunner(28814): at org.junit.runner.notification.RunNotifier.fireTestRunStarted(RunNotifier.java:88)
06-14 03:15:07.677: E/TestRunner(28814): at org.junit.runner.JUnitCore.run(JUnitCore.java:136)
06-14 03:15:07.677: E/TestRunner(28814): at org.junit.runner.JUnitCore.run(JUnitCore.java:115)
06-14 03:15:07.677: E/TestRunner(28814): at androidx.test.internal.runner.TestExecutor.execute(TestExecutor.java:56)
06-14 03:15:07.677: E/TestRunner(28814): at androidx.test.runner.AndroidJUnitRunner.onStart(AndroidJUnitRunner.java:392)
06-14 03:15:07.677: E/TestRunner(28814): at android.app.Instrumentation$InstrumentationThread.run(Instrumentation.java:2106)
06-14 03:15:07.677: E/TestRunner(28814): ----- end exception -----
06-14 03:15:07.680: I/TestRunner(28814): started: testMatcherWithMatchFound(app.impl.matcher.MatcherTest)
06-14 03:15:07.681: E/TestRunner(28814): failed: Test mechanism
06-14 03:15:07.681: E/TestRunner(28814): ----- begin exception -----
06-14 03:15:07.681: E/TestRunner(28814): java.lang.IllegalStateException: Unable to send notification, callback is null
06-14 03:15:07.681: E/TestRunner(28814): at androidx.test.orchestrator.instrumentationlistener.OrchestratedInstrumentationListener.sendTestNotification(OrchestratedInstrumentationListener.java:163)
06-14 03:15:07.681: E/TestRunner(28814): at androidx.test.orchestrator.instrumentationlistener.OrchestratedInstrumentationListener.testStarted(OrchestratedInstrumentationListener.java:114)
06-14 03:15:07.681: E/TestRunner(28814): at org.junit.runner.notification.SynchronizedRunListener.testStarted(SynchronizedRunListener.java:49)
06-14 03:15:07.681: E/TestRunner(28814): at org.junit.runner.notification.RunNotifier.notifyListener(RunNotifier.java:121)
06-14 03:15:07.681: E/TestRunner(28814): at org.junit.runner.notification.RunNotifier$SafeNotifier.run(RunNotifier.java:72)
06-14 03:15:07.681: E/TestRunner(28814): at org.junit.runner.notification.RunNotifier.fireTestStarted(RunNotifier.java:118)
06-14 03:15:07.681: E/TestRunner(28814): at org.junit.internal.runners.model.EachTestNotifier.fireTestStarted(EachTestNotifier.java:42)
06-14 03:15:07.681: E/TestRunner(28814): at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:323)
06-14 03:15:07.681: E/TestRunner(28814): at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
06-14 03:15:07.681: E/TestRunner(28814): at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
06-14 03:15:07.681: E/TestRunner(28814): at org.junit.runners.ParentRunner.run(ParentRunner.java:290)
06-14 03:15:07.681: E/TestRunner(28814): at org.junit.runners.ParentRunner.schedule(ParentRunner.java:71)
06-14 03:15:07.681: E/TestRunner(28814): at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
06-14 03:15:07.681: E/TestRunner(28814): at org.junit.runners.ParentRunner.access[=13=]0(ParentRunner.java:58)
06-14 03:15:07.681: E/TestRunner(28814): at org.junit.runners.ParentRunner.evaluate(ParentRunner.java:268)
06-14 03:15:07.681: E/TestRunner(28814): at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
06-14 03:15:07.681: E/TestRunner(28814): at org.junit.runners.Suite.runChild(Suite.java:128)
06-14 03:15:07.681: E/TestRunner(28814): at org.junit.runners.Suite.runChild(Suite.java:27)
06-14 03:15:07.681: E/TestRunner(28814): at org.junit.runners.ParentRunner.run(ParentRunner.java:290)
06-14 03:15:07.681: E/TestRunner(28814): at org.junit.runners.ParentRunner.schedule(ParentRunner.java:71)
06-14 03:15:07.681: E/TestRunner(28814): at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
06-14 03:15:07.681: E/TestRunner(28814): at org.junit.runners.ParentRunner.access[=13=]0(ParentRunner.java:58)
06-14 03:15:07.681: E/TestRunner(28814): at org.junit.runners.ParentRunner.evaluate(ParentRunner.java:268)
06-14 03:15:07.681: E/TestRunner(28814): at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
06-14 03:15:07.681: E/TestRunner(28814): at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
06-14 03:15:07.681: E/TestRunner(28814): at org.junit.runner.JUnitCore.run(JUnitCore.java:115)
06-14 03:15:07.681: E/TestRunner(28814): at androidx.test.internal.runner.TestExecutor.execute(TestExecutor.java:56)
06-14 03:15:07.681: E/TestRunner(28814): at androidx.test.runner.AndroidJUnitRunner.onStart(AndroidJUnitRunner.java:392)
06-14 03:15:07.681: E/TestRunner(28814): at android.app.Instrumentation$InstrumentationThread.run(Instrumentation.java:2106)
06-14 03:15:07.681: E/TestRunner(28814): ----- end exception -----
此后不再进行测试 运行 且 firebase 测试实验室超时构建。
关于更改什么以使我们的所有测试达到 运行 的任何建议？
提前致谢！
原来这是Orchestration框架的一个bug，目前最好的解决办法就是干脆不用Orchestration。我为此创建了一个问题：