python 单元测试的执行顺序由他们的声明决定

Question

我正在使用 python 单元测试和 selenium，在我的代码中我有一个测试 class 有很多测试用例：

class BasicRegression(unittest.TestCase):

    @classmethod
    def setUpClass(cls):
        cls.driver = webdriver.Chrome(executable_path=Data.driver)
        cls.driver.implicitly_wait(1)
        cls.driver.maximize_window()

    def testcase1_some_stuff(self):
        do_something()

    def testcase2_some_stuff(self):
        do_something()

    def testcase3_some_stuff(self):
        do_something()

    ...

    @classmethod
    def tearDownClass(cls):
        cls.driver.close()
        cls.driver.quit()

if __name__ == '__main__':
    unittest.main()

测试按字母顺序执行，即 testcase1、testcase2 和 testcase3，直到 testcase9。标准。问题出现在testcase10等先执行的

我的问题是如何设置它们的执行顺序？

Answer 1

首先，单元测试应该是独立的。所以必须 python-unittest. Tests executed through python-unittest should be designed in such a way that they should be able to be run independently. Pure unit tests offer a benefit that when they fail, they often depicts what exactly went wrong. Still we tend to write functional tests, integration tests, and system tests with the unittest framework and these tests won't be feasible to run without ordering them since 使 浏览上下文 自动化。要实现排序，您至少需要为测试名称使用更好的命名约定，例如：test_1、test_2、test_3 等，这是有效的，因为测试是根据字符串的内置顺序排序。

但是，根据您的观察，问题出现在 test_10 等排序顺序似乎中断的地方。例如，在名称为 test_1、test_2 和 test_10 的 3 个测试中，单元测试似乎在 test_2:

之前执行 test_10

代码：

import unittest

class Test(unittest.TestCase):

    @classmethod
    def setUp(self):
        print("I'm in setUp")

    def test_1(self):
        print("I'm in test 1")

    def test_2(self):
        print("I'm in test 2")

    def test_10(self):
        print("I'm in test 10")

    @classmethod
    def tearDown(self):
        print("I'm in tearDown")

if __name__ == "__main__":
    unittest.main()

控制台输出：

Finding files... done.
Importing test modules ... done.

I'm in setUp
I'm in test 1
I'm in tearDown
I'm in setUp
I'm in test 10
I'm in tearDown
I'm in setUp
I'm in test 2
I'm in tearDown
----------------------------------------------------------------------
Ran 3 tests in 0.001s

OK

解决方案

在不同的讨论中提出了不同的解决方案，其中一些如下：

@max在讨论Unittest tests order中建议将sortTestMethodsUsing设置为None如下：
```
import unittest
unittest.TestLoader.sortTestMethodsUsing = None
```
@atomocopter 在讨论 changing order of unit tests in Python 中建议将 sortTestMethodsUsing 设置为某个值，如下所示：
```
import unittest
unittest.TestLoader.sortTestMethodsUsing = lambda _, x, y: cmp(y, x)
```
@ElmarZander 在讨论 Unittest tests order 中建议使用 nose 并将测试用例编写为函数（而不是某些派生的测试用例的方法 class）nose 不使用 fiddle 顺序，而是使用文件中定义的函数顺序。
@Keiji 在讨论中Controlling the order of unittest.TestCases提到：

sortTestMethodsUsing expects a function like Python 2's cmp, which has no equivalent in Python 3 (I went to check if Python 3 had a <=> spaceship operator yet, but apparently not; they expect you to rely on separate comparisons for < and ==, which seems much a backwards step...). The function takes two arguments to compare, and must return a negative number if the first is smaller. Notably in this particular case, the function may assume that the arguments are never equal, as unittest will not put duplicates in its list of test names.

With this in mind, here's the simplest way I found to do it, assuming you only use one TestCase class:

def make_orderer():
    order = {}

    def ordered(f):
        order[f.__name__] = len(order)
        return f

    def compare(a, b):
        return [1, -1][order[a] < order[b]]

    return ordered, compare

ordered, compare = make_orderer()
unittest.defaultTestLoader.sortTestMethodsUsing = compare

Then, annotate each test method with @ordered:

class TestMyClass(unittest.TestCase):
    @ordered
    def test_run_me_first(self):
        pass

    @ordered
    def test_do_this_second(self):
        pass

    @ordered
    def test_the_final_bits(self):
        pass

if __name__ == '__main__':
    unittest.main()

This relies on Python calling annotations in the order the annotated functions appear in the file. As far as I know, this is intended, and I'd be surprised if it changed, but I don't actually know if it's guaranteed behavior. I think this solution will even work in Python 2 as well, for those who are unfortunately stuck with it, though I haven't had a chance to test this.

If you have multiple TestCase classes, you'll need to run ordered, compare = make_orderer() once per class before the class definition, though how this can be used with sortTestMethodsUsing will be more tricky and I haven't yet been able to test this either.

For the record, the code I am testing does not rely on the test order being fixed - and I fully understand that you shouldn't rely on test order, and this is the reason people use to avoid answering this question. The order of my tests could be randomised and it'd work just as well. However, there is one very good reason I'd like the order to be fixed to the order they're defined in the file: it makes it so much easier to see at a glance which tests failed.

python 单元测试的执行顺序由他们的声明决定

Execution order of python unitests by their declaration

python

selenium

python-unittest

解决方案