Pragmatic Dave Thomas: Testing might not mean what you think it means
(pragdave.me)
I’d like you do stop and thing about the following question.
I’d like you do stop and thing about the following question.
AI Eats Software Testing
(sabrina.dev)
In the fast-paced world of software, squashing bugs is a never-ending battle…
In the fast-paced world of software, squashing bugs is a never-ending battle…
Why Property Testing Finds Bugs Unit Testing Does Not (2021)
(buttondown.com)
I intended this newsletter to be my thoughts without editing, and I have a new thought, so here goes. I want to respond to this discussion:
I intended this newsletter to be my thoughts without editing, and I have a new thought, so here goes. I want to respond to this discussion:
The State of Open-Source AI-Powered Test Automation
(alumnium.ai)
AI is quickly reshaping software test automation, with open-source solutions emerging alongside commercial tools. As a test engineer maintaining Selenium tests, I faced confusion about these AI options: what exists, how they fit my workflows, their capabilities, reliability, speed, and cost. Finding no clear answers, I researched hands-on.
AI is quickly reshaping software test automation, with open-source solutions emerging alongside commercial tools. As a test engineer maintaining Selenium tests, I faced confusion about these AI options: what exists, how they fit my workflows, their capabilities, reliability, speed, and cost. Finding no clear answers, I researched hands-on.
Production tests: a guidebook for better systems and more sleep
(martincapodici.com)
Your customers expect your site to be fully working whenever they need it. This means you need to aim for near-perfect uptime not just for the site itself, but for all features customers may use.
Your customers expect your site to be fully working whenever they need it. This means you need to aim for near-perfect uptime not just for the site itself, but for all features customers may use.
Semantic unit testing: test code without executing it
(alexmolas.com)
Left Wallapop a couple of weeks ago, heading to RevenueCat soon. In that classic ‘between jobs’ hacking window, I built suite: a Python library for semantic unit testing. What’s semantic unit testing? Think unit tests that understand context and meaning, not just assert obj == expected. Sound interesting? I’ll break down what semantic unit testing is, how suite works under the hood, and how you can integrate it.
Left Wallapop a couple of weeks ago, heading to RevenueCat soon. In that classic ‘between jobs’ hacking window, I built suite: a Python library for semantic unit testing. What’s semantic unit testing? Think unit tests that understand context and meaning, not just assert obj == expected. Sound interesting? I’ll break down what semantic unit testing is, how suite works under the hood, and how you can integrate it.
Falsify: Hypothesis-Inspired Shrinking for Haskell (2023)
(well-typed.com)
Consider this falsify property test that tries to verify the (obviously false) property that all elements of all lists of up to 10 binary digits are the same (we will explain the details below; hopefully the intent is clear):
Consider this falsify property test that tries to verify the (obviously false) property that all elements of all lists of up to 10 binary digits are the same (we will explain the details below; hopefully the intent is clear):
Functional Tests as a Tree of Continuations (2010)
(evanmiller.org)
One of the most essential practices for maintaining the long-term quality of computer code is to write automated tests that ensure the program continues to act as expected, even when other people (including your future self) muck with it.
One of the most essential practices for maintaining the long-term quality of computer code is to write automated tests that ensure the program continues to act as expected, even when other people (including your future self) muck with it.
Revolutionizing software testing: Introducing LLM-powered bug catchers
(engineering.fb.com)
Meta’s Automated Compliance Hardening (ACH) tool is a system for mutation-guided, LLM-based test generation. ACH hardens platforms against regressions by generating undetected faults (mutants) in source code that are specific to a given area of concern and using those same mutants to generate tests.
Meta’s Automated Compliance Hardening (ACH) tool is a system for mutation-guided, LLM-based test generation. ACH hardens platforms against regressions by generating undetected faults (mutants) in source code that are specific to a given area of concern and using those same mutants to generate tests.
Testtrim: A testing tool that couldn't test itself (until now)
(fenniak.net)
Today, we’re going to deep-dive into the kind of thing you can only “invest” time on if you’re a single engineer working on a project with no supervision. I just finished a crazy complicated development effort in my project, testtrim, and all I want to do is talk about how surprised I am that it actually worked.
Today, we’re going to deep-dive into the kind of thing you can only “invest” time on if you’re a single engineer working on a project with no supervision. I just finished a crazy complicated development effort in my project, testtrim, and all I want to do is talk about how surprised I am that it actually worked.
The Fuzzing Book
(fuzzingbook.org)
Welcome to "The Fuzzing Book"! Software has bugs, and catching bugs can involve lots of effort. This book addresses this problem by automating software testing, specifically by generating tests automatically. Recent years have seen the development of novel techniques that lead to dramatic improvements in test generation and software testing. They now are mature enough to be assembled in a book – even with executable code.
Welcome to "The Fuzzing Book"! Software has bugs, and catching bugs can involve lots of effort. This book addresses this problem by automating software testing, specifically by generating tests automatically. Recent years have seen the development of novel techniques that lead to dramatic improvements in test generation and software testing. They now are mature enough to be assembled in a book – even with executable code.
State Space Explosion: The Reason We Can Never Test Software to Perfection(2021)
(concerningquality.com)
Have you ever seen a test suite actually prevent 100% of bugs? With all of the time that we spend testing software, how do bugs still get through? Testing seems ostensibly simple – there are only so many branches in the code, only so many buttons in the UI, only so many edge cases to consider. So what is difficult about testing software?
Have you ever seen a test suite actually prevent 100% of bugs? With all of the time that we spend testing software, how do bugs still get through? Testing seems ostensibly simple – there are only so many branches in the code, only so many buttons in the UI, only so many edge cases to consider. So what is difficult about testing software?
Faster CI with Selective Testing
(mill-build.org)
Selective testing is a key technique necessary for working with any large codebase or monorepo: picking which tests to run to validate a change or pull-request, because running every test every time is costly and slow. This blog post will explore what selective testing is all about, the different approaches you can take with selective testing, based on my experience working on developer tooling and CI for the last decade at Dropbox and Databricks.
Selective testing is a key technique necessary for working with any large codebase or monorepo: picking which tests to run to validate a change or pull-request, because running every test every time is costly and slow. This blog post will explore what selective testing is all about, the different approaches you can take with selective testing, based on my experience working on developer tooling and CI for the last decade at Dropbox and Databricks.
Reliably Benchmarking Small Changes – Ankush Menat
(ankush.dev)
I often have to benchmark web services to see if some small change has a meaningful impact on performance. Typically, you spawn a few web service workers and use another program (ideally on another machine in the same network) to hammer that service. During this time, the test program will keep track of how many requests were processed and the latency for each of them. If throughput goes up and/or latency goes down, your change was effective.
I often have to benchmark web services to see if some small change has a meaningful impact on performance. Typically, you spawn a few web service workers and use another program (ideally on another machine in the same network) to hammer that service. During this time, the test program will keep track of how many requests were processed and the latency for each of them. If throughput goes up and/or latency goes down, your change was effective.
Show HN: Hercules – Open-Source software testing agent
(testzeus.com)
Effortless, Autonomous Test Automation—Zero Coding, Zero Maintenance, Infinite Possibilities.
Effortless, Autonomous Test Automation—Zero Coding, Zero Maintenance, Infinite Possibilities.
Using LLMs to enhance our testing practices
(assembled.com)
At Assembled, engineering velocity is our competitive edge. We pride ourselves on delivering new features at a fast pace. But how do we maintain quality without slowing down? The answer lies in robust testing.
At Assembled, engineering velocity is our competitive edge. We pride ourselves on delivering new features at a fast pace. But how do we maintain quality without slowing down? The answer lies in robust testing.
Launch HN: GPT Driver (YC S21) – End-to-end app testing in natural language
(ycombinator.com)
Hey HN, we are Chris and Chris from MobileBoost (https://mobileboost.io/). We’re building GPT Driver, an AI-native approach to create and execute end-to-end (E2E) tests on mobile applications.
Hey HN, we are Chris and Chris from MobileBoost (https://mobileboost.io/). We’re building GPT Driver, an AI-native approach to create and execute end-to-end (E2E) tests on mobile applications.
Webinar – How Generative AI Can Help Scale Your QA Processes
(testrigor.com)
Join us for an insightful webinar hosted by testRigor, where we’ll delve into the transformative potential of scaling QA processes using Generative AI.
Join us for an insightful webinar hosted by testRigor, where we’ll delve into the transformative potential of scaling QA processes using Generative AI.
Accumulated Test Vectors
(filippo.io)
I like tests. I especially like reusable test vector libraries. Sometimes test vectors are lovingly handcrafted to target obscure edge-cases. Those vectors belong in Wycheproof or with the upstream specification. Sometimes though vectors are produced by sheer brute force. Enumerate every possible input and check the output. Try a million random inputs and see what happens. Combine all possible input sizes for every parameter. Make one very, very large input.
I like tests. I especially like reusable test vector libraries. Sometimes test vectors are lovingly handcrafted to target obscure edge-cases. Those vectors belong in Wycheproof or with the upstream specification. Sometimes though vectors are produced by sheer brute force. Enumerate every possible input and check the output. Try a million random inputs and see what happens. Combine all possible input sizes for every parameter. Make one very, very large input.
I Am Tired of AI
(ontestautomation.com)
Unless you have been living under a rock for the last few years, you probably have seen the same massive surge I’ve seen in the application of artificial intelligence (AI) to pretty much every problem out there, in software testing, in software development, and in life in general.
Unless you have been living under a rock for the last few years, you probably have seen the same massive surge I’ve seen in the application of artificial intelligence (AI) to pretty much every problem out there, in software testing, in software development, and in life in general.
Show HN: Scandium: No-Code Test Automation Tool for Software Testing
(getscandium.com)
Create and maintain automated test cases without a line of code.
Create and maintain automated test cases without a line of code.
Earn $200K by Fuzzing for a Weekend
(secret.club)
By applying well-known fuzzing techniques to a popular target, I found several bugs that in total yielded over $200K in bounties. In this article I will demonstrate how powerful fuzzing can be when applied to software which has not yet faced sufficient testing.
By applying well-known fuzzing techniques to a popular target, I found several bugs that in total yielded over $200K in bounties. In this article I will demonstrate how powerful fuzzing can be when applied to software which has not yet faced sufficient testing.
Automated feature testing of Verilog parsers using fuzzing
(wordpress.com)
I’m delighted that Quentin Corradi, a PhD student I jointly supervise with George Constantinides, will be presenting his work to improve the reliability of hardware design tools next week at the FUZZING’24 workshop, a satellite event of the ISSTA conference.
I’m delighted that Quentin Corradi, a PhD student I jointly supervise with George Constantinides, will be presenting his work to improve the reliability of hardware design tools next week at the FUZZING’24 workshop, a satellite event of the ISSTA conference.