Hacker News with Generative AI: Software Testing

Falsify: Hypothesis-Inspired Shrinking for Haskell (2023) (well-typed.com)
Consider this falsify property test that tries to verify the (obviously false) property that all elements of all lists of up to 10 binary digits are the same (we will explain the details below; hopefully the intent is clear):

Programming Languages, Haskell, Software Testing, Functional Programming

89 points by birdculture 6 days ago | 17 comments

Functional Tests as a Tree of Continuations (2010) (evanmiller.org)
One of the most essential practices for maintaining the long-term quality of computer code is to write automated tests that ensure the program continues to act as expected, even when other people (including your future self) muck with it.

Software Testing, Programming, Functional Programming

69 points by amenghra 44 days ago | 32 comments

Revolutionizing software testing: Introducing LLM-powered bug catchers (engineering.fb.com)
Meta’s Automated Compliance Hardening (ACH) tool is a system for mutation-guided, LLM-based test generation. ACH hardens platforms against regressions by generating undetected faults (mutants) in source code that are specific to a given area of concern and using those same mutants to generate tests.

Software Testing, Artificial Intelligence, Machine Learning

7 points by nadis 80 days ago | 5 comments

Testtrim: A testing tool that couldn't test itself (until now) (fenniak.net)
Today, we’re going to deep-dive into the kind of thing you can only “invest” time on if you’re a single engineer working on a project with no supervision. I just finished a crazy complicated development effort in my project, testtrim, and all I want to do is talk about how surprised I am that it actually worked.

Software Testing, Software Development, Personal Projects

10 points by mfenniak 91 days ago | 0 comments

Compiler Fuzzing in Continuous Integration: A Case Study on Dafny [pdf] (ic.ac.uk)

Compiler Fuzzing, Continuous Integration, Software Testing, Programming Languages, Case Studies

42 points by matt_d 95 days ago | 2 comments

The Fuzzing Book (fuzzingbook.org)
Welcome to "The Fuzzing Book"! Software has bugs, and catching bugs can involve lots of effort. This book addresses this problem by automating software testing, specifically by generating tests automatically. Recent years have seen the development of novel techniques that lead to dramatic improvements in test generation and software testing. They now are mature enough to be assembled in a book – even with executable code.

Software Testing, Automation, Programming Books

250 points by chautumn 97 days ago | 10 comments

State Space Explosion: The Reason We Can Never Test Software to Perfection(2021) (concerningquality.com)
Have you ever seen a test suite actually prevent 100% of bugs? With all of the time that we spend testing software, how do bugs still get through? Testing seems ostensibly simple – there are only so many branches in the code, only so many buttons in the UI, only so many edge cases to consider. So what is difficult about testing software?

Software Testing, Software Development, Software Quality

10 points by thunderbong 101 days ago | 1 comments

Property-Based Testing for the People (repository.upenn.edu)

Property-Based Testing, Software Testing, Software Development

99 points by matt_d 110 days ago | 67 comments

Faster CI with Selective Testing (mill-build.org)
Selective testing is a key technique necessary for working with any large codebase or monorepo: picking which tests to run to validate a change or pull-request, because running every test every time is costly and slow. This blog post will explore what selective testing is all about, the different approaches you can take with selective testing, based on my experience working on developer tooling and CI for the last decade at Dropbox and Databricks.

Continuous Integration, Software Testing, Software Development, CI/CD, Large Codebases

45 points by lihaoyi 121 days ago | 49 comments

Why Playwright Is Dominating Browser Automation: A Deep Dive (leapcell.io)

Browser Automation, Software Testing, Web Development, JavaScript

7 points by issac_0v0 126 days ago | 5 comments

Reliably Benchmarking Small Changes – Ankush Menat (ankush.dev)
I often have to benchmark web services to see if some small change has a meaningful impact on performance. Typically, you spawn a few web service workers and use another program (ideally on another machine in the same network) to hammer that service. During this time, the test program will keep track of how many requests were processed and the latency for each of them. If throughput goes up and/or latency goes down, your change was effective.

Web Development, Performance Optimization, Benchmarking, Software Testing

32 points by rbanffy 155 days ago | 16 comments

Show HN: Hercules – Open-Source software testing agent (testzeus.com)
Effortless, Autonomous Test Automation—Zero Coding, Zero Maintenance, Infinite Possibilities.

Software Testing, Open Source, Automation, AI

10 points by Smilinrobin 159 days ago | 12 comments

Using LLMs to enhance our testing practices (assembled.com)
At Assembled, engineering velocity is our competitive edge. We pride ourselves on delivering new features at a fast pace. But how do we maintain quality without slowing down? The answer lies in robust testing.

Generative AI, Software Testing, Engineering

186 points by johnjwang 184 days ago | 77 comments

Launch HN: GPT Driver (YC S21) – End-to-end app testing in natural language (ycombinator.com)
Hey HN, we are Chris and Chris from MobileBoost (https://mobileboost.io/). We’re building GPT Driver, an AI-native approach to create and execute end-to-end (E2E) tests on mobile applications.

Software Testing, Artificial Intelligence, Mobile Development, Startups

129 points by cschiller 185 days ago | 82 comments

SMURF: Beyond the Test Pyramid (googleblog.com)
SMURF: Beyond the Test Pyramid

Software Testing, Software Development, Google, Testing Frameworks

72 points by BerislavLopac 189 days ago | 53 comments

Webinar – How Generative AI Can Help Scale Your QA Processes (testrigor.com)
Join us for an insightful webinar hosted by testRigor, where we’ll delve into the transformative potential of scaling QA processes using Generative AI.

Generative AI, QA, Software Testing, Webinars

10 points by Dmoerner1991 190 days ago | 0 comments

Accumulated Test Vectors (filippo.io)
I like tests. I especially like reusable test vector libraries. Sometimes test vectors are lovingly handcrafted to target obscure edge-cases. Those vectors belong in Wycheproof or with the upstream specification. Sometimes though vectors are produced by sheer brute force. Enumerate every possible input and check the output. Try a million random inputs and see what happens. Combine all possible input sizes for every parameter. Make one very, very large input.

Software Testing, Security, Code Quality

5 points by fanf2 199 days ago | 0 comments

I Am Tired of AI (ontestautomation.com)
Unless you have been living under a rock for the last few years, you probably have seen the same massive surge I’ve seen in the application of artificial intelligence (AI) to pretty much every problem out there, in software testing, in software development, and in life in general.

Artificial Intelligence, Software Development, Software Testing

1209 points by Liriel 211 days ago | 1120 comments

Show HN: Scandium: No-Code Test Automation Tool for Software Testing (getscandium.com)
Create and maintain automated test cases without a line of code.

Software Testing, Test Automation, No-Code Tools

7 points by IbraheemAdebisi 214 days ago | 2 comments

Earn $200K by Fuzzing for a Weekend (secret.club)
By applying well-known fuzzing techniques to a popular target, I found several bugs that in total yielded over $200K in bounties. In this article I will demonstrate how powerful fuzzing can be when applied to software which has not yet faced sufficient testing.

Security, Bug Bounties, Software Testing

11 points by udev4096 216 days ago | 0 comments

Automated feature testing of Verilog parsers using fuzzing (wordpress.com)
I’m delighted that Quentin Corradi, a PhD student I jointly supervise with George Constantinides, will be presenting his work to improve the reliability of hardware design tools next week at the FUZZING’24 workshop, a satellite event of the ISSTA conference.

Hardware Design, Software Testing, Fuzzing

15 points by matt_d 229 days ago | 2 comments

Avoiding Test-Case Permutation Blowout (stevenhicks.me)

Software Testing, Software Engineering, Programming, Test Automation

36 points by todsacerdoti 234 days ago | 14 comments

Rust Just Failed an Important Test (mccue.dev)

Programming Languages, Rust, Software Testing

62 points by lladnar 268 days ago | 52 comments

Show HN: A fake SMTP server for software integration testing (fakemail.stream)

Software Testing, Software Development, Tools, Email

127 points by aeaa3 279 days ago | 49 comments

Show HN: How we use LLMs to find testing gaps, vulnerabilities in codebases (github.com/codeintegrity-ai)