Firmware QA/Benchmarking Engineer

We are building FirmGen under CraftifAI Orbit , an AI-assisted development platform that helps embedded teams generate, build, flash, debug, and validate firmware across multiple MCU platforms.

FirmGen combines LLM workflows, firmware engineering knowledge, SDK understanding, hardware-aware reasoning, build/flash automation, and validation agents to generate reliable embedded firmware projects.

We are looking for a QA Engineer who can test the product deeply across AI output quality, firmware generation, build validation, hardware behavior, UI workflows, and regression scenarios .

This role is not limited to traditional manual testing.

You will help validate whether FirmGen can correctly understand user prompts, generate proper firmware architecture, create buildable projects, use the right SDK APIs, configure peripherals correctly, handle errors, and recover from failures.

You will work closely with firmware, AI, and product teams to improve the reliability of FirmGen’s AI-assisted firmware generation pipeline.

• Test FirmGen’s end-to-end flow from user prompt to firmware generation, build, flash, monitor, and validation.
• Validate AI-generated firmware projects for structure, completeness, buildability, and expected behavior.
• Create test prompts for different MCU use cases such as GPIO, UART, ADC, PWM, I2C, SPI, timers, RTOS, BLE/Wi-Fi, MQTT, Modbus, OTA, and bootloader flows.
• Verify generated HLD, LLD, architecture diagrams, board topology, memory mapping, firmware components, and documentation outputs.
• Test SDK discovery, project generation, build logs, flashing flows, serial monitor behavior, and diagnostic reports.
• Identify issues such as hallucinated APIs, wrong pin mapping, missing clock configuration, incorrect peripheral setup, broken build files, invalid dependencies, or incomplete generated code.
• Perform regression testing across supported boards, SDK versions, prompts, and generated projects.
• Validate UI workflows including project creation, file tree, chat flow, generated artifacts, build/flash buttons, logs, error states, and report views.
• Capture clear bug reports with steps, prompt used, expected output, actual output, logs, screenshots, videos, build errors, and hardware observations.
• Help define acceptance criteria, test cases, evaluation datasets, QA checklists, and release validation reports.
• Work with firmware and AI teams to improve FirmGen’s self-healing and validation feedback loops.

• 1–6 years of QA experience in software, embedded systems, developer tools, or technical products.
• Strong understanding of QA processes, test case design, regression testing, bug reporting, and release validation.
• Ability to test complex workflows involving UI, backend, generated files, logs, and hardware behavior.
• Basic understanding of embedded systems, microcontrollers, peripherals, firmware projects, and build/flash flows.
• Ability to read logs, error messages, build failures, serial monitor output, and generated project structure.
• Good analytical skills to compare expected vs actual AI-generated outputs.
• Strong written communication for documenting bugs, test results, and release reports.

• Experience testing embedded firmware, IoT products, hardware tools, SDKs, or developer platforms.
• Familiarity with STM32, ESP32, Arduino, Zephyr, FreeRTOS, or similar embedded ecosystems.
• Basic knowledge of C/C++, Python, shell scripts, build systems, Git, and command-line tools.
• Experience with AI tools, LLM-based products, prompt testing, RAG systems, or AI output evaluation.
• Experience with tools such as Jira, GitHub Issues, TestRail, Postman, CI/CD, or automation frameworks.
• Hands-on exposure to UART logs, flashing tools, serial terminals, logic analyzers, or hardware boards.

1–6 years of QA experience.

Strong freshers may also be considered if they have good testing fundamentals, technical curiosity, and interest in embedded systems or AI products.

You will help build the quality foundation of FirmGen , an AI-native firmware development platform.

This role is for QA engineers who want to move beyond simple UI testing and work on a deep technical product involving AI, firmware generation, hardware validation, developer workflows, and embedded systems.