In a first screen, a QA automation engineer should be able to show that they have written and maintained automated tests in a real codebase, and can explain why those tests fail. Fifteen to twenty minutes is enough if every question asks for a specific example from their own work.
What does a QA automation engineer do?
The Bureau of Labor Statistics covers this work on its software developers, quality assurance analysts and testers page, which says testers "identify problems with applications or programs and report defects," using manual or automated testing. An automation engineer does most of that by writing code: tests that click through a web app, call an API or check data, and run automatically whenever developers push new code. Much of the job is keeping those tests reliable as the product changes.
Which skills should you check at each level?
Ask the hiring manager which tool and language the team uses before you start. A strong Cypress engineer can learn Playwright, but some clients won't wait.
| Level | What they should be able to talk about |
|---|---|
| Junior (0 to 2 years) | Writing UI or API tests in one tool (for example Selenium, Playwright or Cypress) and one language, reading a failed test, writing a clear bug report, Git basics |
| Mid-level (2 to 5 years) | Building and maintaining a test framework, API testing, test data, running suites in a CI pipeline such as Jenkins or GitHub Actions, fixing flaky tests |
| Senior (5+ years) | Test strategy across unit, API and UI layers, choosing tools, cutting suite run time, parallel runs, coaching developers to write their own tests |
What screening questions should you ask a QA automation engineer?
Swap in the client's tools and test language.
Which tool and language did you use on your last project, and what did you test with it?
It tells you whether their stack matches the role.
Strong answer: names one tool, one language and the kind of app, for example Playwright with TypeScript against a web app and its API, with a rough count of tests and when they ran.
Weak answer: lists every tool on the resume and describes no single project in detail.
Tell me about a flaky test you fixed. What was causing it?
Flaky tests pass sometimes and fail other times. Anyone who has done this job for a year has fixed several.
Strong answer: a specific cause, such as the test clicking before the page finished loading or two tests sharing the same data, and a specific fix, such as waiting for a condition on the page or giving each test its own data.
Weak answer: "I added a sleep" or "I reran it until it passed." Fixed waits are the habit good engineers try to remove.
How do you decide what to automate and what to leave for manual testing?
Automating everything is slow and expensive to maintain, so you want someone who chooses.
Strong answer: automates stable checks that run often, such as login, checkout and core API calls, and leaves one-off, visual or fast-changing features to manual or exploratory testing. They mention the cost of maintaining each test.
Weak answer: "Everything should be automated," with no reasoning.
Your UI test suite takes two hours to run. How would you make it faster?
Slow suites get ignored, and senior engineers usually own suite speed.
Strong answer: runs tests in parallel, moves checks that don't need a browser down to API or unit tests, sets up test data through the API instead of clicking through screens, and runs a short smoke set on every change with the full suite nightly.
Weak answer: "Get faster machines," or only "run in parallel" with no thought about tests that share data.
How do you test an API, and what do you check besides the status code?
Many automation roles include API testing, and the answer shows whether they understand what the app does underneath.
Strong answer: names a tool (Postman, REST Assured, pytest with requests, or Playwright's request support) and checks the fields in the response, error cases, permissions and whether the data actually changed.
Weak answer: checks that the call returns 200 and stops there.
Where do your tests run, and what happens when one fails in the pipeline?
Tests that only run on one person's laptop do little for a team.
Strong answer: names the CI tool, when the suite runs (every pull request, nightly), how failures are reported (screenshots, videos or logs of the failed step) and who looks at them first.
Weak answer: "I run them locally before a release."
Tell me about a real bug your automation caught before release.
It shows whether their tests found anything that mattered.
Strong answer: the bug, the test that found it, what would have happened in production and what they changed afterward.
Weak answer: can't recall one, or describes a bug that a manual tester found.
What red flags should you note?
- Claims expert level in five or more automation tools but can't describe one framework they built or maintained.
- Fixes every timing problem with fixed waits.
- Can't say roughly how many tests were in the suite, how long it ran or how often.
- Describes only record-and-playback tools when the role asks for coded automation.
- Says "we" for everything and can't separate their own work from the team's.
- Leans on a certificate. The ISTQB Certified Tester Foundation Level is an entry-level credential on testing concepts. It doesn't show that someone can write automation code.
Probe each one in your own call. For remote roles, how to spot a proxy interview covers the signs that someone else is answering.
How do you run this screen with Simplehire?
Create the role with the skills the hiring manager confirmed as its required skills, for example Playwright, TypeScript, API testing and CI pipelines, and send the candidate the interview link. The candidate talks with our AI interviewer in the browser, and you get a screening report before you submit anyone.
| Tier | Price and length | QA roles it fits |
|---|---|---|
| Basic | $2 per screen, about 15 minutes | First screens for junior QA roles and high-volume reqs |
| Adaptive | $3 per screen, about 20 minutes | Most client submissions. Follow-up questions show whether flaky-test and pipeline stories hold up |
| Advanced | $5 per screen, about 30 minutes | Senior and lead roles, with deeper follow-ups and an optional coding round scored separately |
Skills demonstrated X/Y shows how many of the skills you set came through, and Areas to probe gives you a list for your own call. The verdict is a recommendation, and submission is your decision. Proctoring runs on every screen: enforced full-screen mode, copy and paste restricted or flagged, and tab switches recorded with timestamps. For remote contractors you can also require an ID check or a paired second device, and every alert is reviewed by a person.
Every account gets 25 free Basic screens a month with no credit card required, and the pricing page has the rest. For related roles, see the Java developer, Python developer and DevOps engineer questions.
Frequently asked questions
Can a non-technical recruiter screen a QA automation engineer?
Yes, for a first screen. Ask for one real project and keep asking for specifics: the tool, the language, how many tests, how long they took to run and what broke. Candidates who built the tests themselves answer in detail. Candidates who only watched them run stay general.
What is the difference between a QA automation engineer and a manual tester?
A manual tester checks the software by hand, following test cases or exploring. A QA automation engineer writes code that runs those checks automatically, usually on every code change, and spends much of the job keeping that code reliable. Many automation engineers started as manual testers.
Should a QA automation screen include a coding exercise?
For senior and lead roles, a short coding exercise helps, because the job is mostly writing and reviewing test code. For junior roles, a conversation about one test they wrote and one failure they fixed is usually enough at the first stage.
Which Simplehire tier fits a QA automation engineer?
Senior and lead automation engineers spend much of the job writing test code, so Advanced ($5 per screen, about 30 minutes), with its optional coding round scored separately, fits them well. For most other submissions, Adaptive ($3, about 20 minutes) uses follow-up questions to test whether flaky-test and pipeline examples hold up. Basic ($2, about 15 minutes) covers junior first screens.
