I hear you. same issues, same solutions. but we have another tool up our sleeve, we have a section called Reservoir that runs all new tests added in a loop for a week to determine if there is any flakiness in them, in that time they are not yet part of the critical CI path. happy to hear we are not alone. good day.
Thanks for the great blog-post! It seems that you categorize flakiness as a test issue, but the cause of the flaky test result could be in the production-code and therefor be a real issue. Did you investigate how many of the flaky tests are due to a real issue? And do you give flaky test results lower prio than tests that fail every time?
We do not currently keep accurate count of the number of times that flaky tests are really masking bugs in the code. We see it as a testing issue mostly because it makes it more difficult to use the tests for their intended purpose - finding problems with the code. From the testing system point of view a test that fails reliably is far better than a test that is flaky! A persistently failing test is giving a clear signal about what to do - even it means fixing the test.
Agreed. UI tests are definitely flaky because of how test harnesses interact with the UI, timing issues, handshaking, and extraction of state. See more here: http://comet.unl.edu/tutorial.php
Thanks John, wildly enough these are pretty common issues in large Functional Automation implementations. I know you have a heavy vest in Selenium, do you use another tool for Service Virtualization ?
Today at Google test authors and test infrastructure developers throughout the organization are responsible for creating/using service virtualization in their tests. We do not have a central framework - other than providing generic Mocking frameworks like Mockito.
You touched very important and common problem here. Have you tried to track the wasted time caused by flaky tests (developers unable to submit the changes, CI that requires additional cycles)?
I hear you. same issues, same solutions. but we have another tool up our sleeve, we have a section called Reservoir that runs all new tests added in a loop for a week to determine if there is any flakiness in them, in that time they are not yet part of the critical CI path.
ReplyDeletehappy to hear we are not alone.
good day.
Thanks for the great blog-post!
ReplyDeleteIt seems that you categorize flakiness as a test issue, but the cause of the flaky test result could be in the production-code and therefor be a real issue.
Did you investigate how many of the flaky tests are due to a real issue? And do you give flaky test results lower prio than tests that fail every time?
We do not currently keep accurate count of the number of times that flaky tests are really masking bugs in the code. We see it as a testing issue mostly because it makes it more difficult to use the tests for their intended purpose - finding problems with the code. From the testing system point of view a test that fails reliably is far better than a test that is flaky! A persistently failing test is giving a clear signal about what to do - even it means fixing the test.
DeleteThanks John Micco for sharing your experience.
ReplyDeleteAre those GUI level tests you're struggling with? They're usually considered flaky.
Do you use rerun mechanism for tests that fail?
Regards,
Sławek
Flaky tests appear everywhere in our corpuse, but there is probably some skew toward UI testing that we observe - although I have not quantified this.
DeleteOur rerun mechanism is only used for tests that are marked as flaky or when users specifically request it.
Agreed. UI tests are definitely flaky because of how test harnesses interact with the UI, timing issues, handshaking, and extraction of state. See more here: http://comet.unl.edu/tutorial.php
DeleteThanks John, wildly enough these are pretty common issues in large Functional Automation implementations. I know you have a heavy vest in Selenium, do you use another tool for Service Virtualization ?
ReplyDeleteToday at Google test authors and test infrastructure developers throughout the organization are responsible for creating/using service virtualization in their tests. We do not have a central framework - other than providing generic Mocking frameworks like Mockito.
DeleteYou touched very important and common problem here.
ReplyDeleteHave you tried to track the wasted time caused by flaky tests (developers unable to submit the changes, CI that requires additional cycles)?