This week the A*Team came together as part of the larger Boot2Gecko work week in SF. We worked with a bunch of people across various teams and got a bunch of stuff landed.
TBPL
We’ve been working toward automation on TBPL for a while. This week we worked with Aki from releng to get mochitest, reftest, and emulator-based WebAPI tests running on Mozilla-Central, Try, Services-Central, Mozilla-Inbound, Fx-Team, Cedar, and Ash. We are set to add them to Mozilla-Aurora on Monday.
- Cedar is our staging area for mozilla-central. If you would like to turn on a new set of tests for mochitest or reftest, you can land that change here to see what happens. (If you break something, please back out)
- Ash is our staging area for mozilla-aurora. Here too, you can land things to help expand the automation destined for aurora. (If you break something please back out).
- Try support does exist, but the try chooser is not yet updated, so to run all B2G builds on try, use this try chooser syntax: try: -b o -p ics_armv7a_gecko -t none -u all
Gaia tests/QA Support
The Stephen Donner’s Web QA team is working on automating the Gaia smoketests. This week we pivoted hard to get those stood up in CI. We have them running in our Jenkins instance on an Unagi device and reporting to Autolog.
We met with Geo’s Web API team and made some fixes to the B2G mochitest suite so that they are unblocked and can continue creating webAPI mochitests.
Rob Wood and Dave Hunt continue working closely with the QA team to help them generate more tests. This week, Rob added ten more webSMS emulator tests alone.
PandaBoards
Pandaboards are our on-change solution for automation that needs to run on device, like the Gaia automated smoketests. There are many moving parts to stand these up at scale. Here is a run down of what happened this week:
- Thomas Zimmermann worked with us and we now have a more reliable and stable pandaboard kernel.
- Chris Atlee from releng got our buildbot-provided panda builds to include the proper version of Gaia as well as this updated kernel which is going to be critical to the testing we’ll do next week.
- We have rolled out this new build to our test boards in IT’s colo, and are troubleshooting some networking issues now. Once those are resolved, we will be clear to start testing our pandaboard automation.
Eideticker Performance System
This week, Will Lachance got all the existing Eideticker benchmarks running using a pandaboard hooked to an Eideticker system. He has uploaded a screencast of the tasks.js panning test being performed on B2G here.
Next Steps
We’re not done yet. Here are the things we’re working on next:
- Turn on B2G reftest and mochitest on Aurora (target: Monday, November 12)
- Perform flash testing to ensure we can flash pandaboards per push (target: November )
- Create B2G specific Eideticker tests to measure specific B2G performance point (target: next week, by Nov 16)
- Fix the QA Automated Smoketest reliability issues (target: next week, by Nov 16)
- Add xpcshell to our B2G automated test suite (target: November 23)
- Get pandaboards running Gaia Smoketest and Gaia Integration tests per checkin (target: optimistically, end of November)
- Continue expanding the set of mochitest, reftest, and xpcshell tests that we are running in the B2G test automation (ongoing)