PHPUnit fails, Pipelines doesn't find error in test report

[Kim Metso]

I have a pipeline that runs PHPUnit. One of the tests fail, and PHPUnit writes a test report:

The test report contains a record of the failure:

Yet Pipelines doesn't find the error:

The tests are consequently considered a success, which means the Tests tab isn't visible. The pipeline itself correctly fails because of the exit code from phpunit.

The expected result is that I should see the Tests tab, and within the tab I should see the failed test.

Is this a bug in Pipelines or am I doing something wrong?

[Elisa de Llano]

I have the same issue, did someone ever find a culprit or a solution?

[Elisa de Llano]

Mixing tests with a data provider and tests without is what causes this.

[Oleksandr Kyrdan]

Hi @Kim Metso ,

Thank you for your question!

Let's try command:

vendor/bin/phpunit --log-junit=test-reports/report.xml

 more details provided in the Test reporting in Pipelines guide.

[Kim Metso]

Thank you for your suggestion @Oleksandr Kyrdan. I tried this, but the result was the same as in my original post. The test report parser only finds 129 tests.

I'm not sure your suggested command does anything differently from my original one? My original command was from the page you are linking to.

Looking at the issues for PHPUnit (the ones concerning junit specifically), it seems the report format isn't very standardized. I tried updating PHPUnit to 8.5.8 but the resulting report is identical.

[ktomk]

Can you clarify if the output xml contains 129 or 132 test results?

Can you identify the 3 tests that are not reported by the test-reporter on Bitbucket? (Maybe not being reported as part of a test-suite or whatever this could be).

Phpunit invocation looks good to me: xml log file is created and found by test-reporter on bitbucket already in the OP one.

[Kim Metso]

@ktomk The XML file does contain 132 test results.

Here is the basic structure of the file (with a lot of stuff omitted for brevity):

<?xml version="1.0" encoding="UTF-8"?>
<testsuites>
    <testsuite name="" tests="132" assertions="491" errors="0" warnings="0" failures="1" skipped="0" time="0.823959">
        <!-- ... -->
        <testsuite name="Feature" tests="109" assertions="468" errors="0" warnings="0" failures="1" skipped="0" time="0.645507">
            <!-- ... -->
            <testsuite name="Tests\Feature\ObjectPageFieldsTest" file="<...>/tests/Feature/ObjectPageFieldsTest.php" tests="71" assertions="304" errors="0" warnings="0" failures="1" skipped="0" time="0.403150">
                <testsuite name="Tests\Feature\ObjectPageFieldsTest::it_prints_the_correct_value_for_the_field" tests="35" assertions="268" errors="0" warnings="0" failures="0" skipped="0" time="0.267753">
                    <testcase name="it_prints_the_correct_value_for_the_field with data set #0" class="Tests\Feature\ObjectPageFieldsTest" classname="Tests.Feature.ObjectPageFieldsTest" file="<...>/tests/Feature/ObjectPageFieldsTest.php" line="79" assertions="7" time="0.008839"/>
                    <testcase name="it_prints_the_correct_value_for_the_field with data set #1" class="Tests\Feature\ObjectPageFieldsTest" classname="Tests.Feature.ObjectPageFieldsTest" file="<...>/tests/Feature/ObjectPageFieldsTest.php" line="79" assertions="8" time="0.007462"/>
                    <!-- ... -->
                </testsuite>
                <testsuite name="Tests\Feature\ObjectPageFieldsTest::it_has_a_label_for_the_field" tests="35" assertions="35" errors="0" warnings="0" failures="0" skipped="0" time="0.131400">
                    <testcase name="it_has_a_label_for_the_field with data set #0" class="Tests\Feature\ObjectPageFieldsTest" classname="Tests.Feature.ObjectPageFieldsTest" file="<...>/tests/Feature/ObjectPageFieldsTest.php" line="98" assertions="1" time="0.004041"/>
                    <testcase name="it_has_a_label_for_the_field with data set #1" class="Tests\Feature\ObjectPageFieldsTest" classname="Tests.Feature.ObjectPageFieldsTest" file="<...>/tests/Feature/ObjectPageFieldsTest.php" line="98" assertions="1" time="0.003846"/>
                    <!-- ... -->
                </testsuite>
                <testcase name="test_exists_for_all_fields" class="Tests\Feature\ObjectPageFieldsTest" classname="Tests.Feature.ObjectPageFieldsTest" file="<...>/tests/Feature/ObjectPageFieldsTest.php" line="109" assertions="1" time="0.003998">
                    <failure type="PHPUnit\Framework\ExpectationFailedException">
                        <!-- PHPUnit assertion failure output here -->
                    </failure>
                </testcase>
            </testsuite>
        </testsuite>
    </testsuite>
</testsuites>

I have no idea what the structure of the file should be like, so I can't really say what could be wrong. The nested <testsuite> elements seem kind off odd, but on the other hand the report parser seems to be able to find most tests so I assume that's valid.

Some of the tests use a PHPUnit dataProvider, which seems to make those tests a testsuite by themselves. It makes sense, since the functions are in fact run multiple times but with different data. Each invocation is being output as a <testcase>.

The failing test, "test_exists_for_all_fields", does not use a dataProvider and is being output as a <testcase>, as a sibling to the <testsuite>s using dataProviders.

[Kim Metso]

A clue might be in that the PHPUnit output in the first screenshot says 132 tests, while the total count in the last screenshot says 129 tests. So I guess that failure ins't found/recognized. But why?