2570 Commits

Author SHA1 Message Date
Romain Malmain
44c841ffb1
WIP: QEMU exit handler (#1745)
* Added paging filtering.
Reworked address range filtering to fit with new generic code.

* Fix: renamed remaining QemuInstrumentationFilter instances.

* Renamed sync breakpoint to sync exit.

* Split emu in systemmode.rs / usermode.rs for specific code.
EmuExitHandler implementation.

* sync_backdoor.rs removal.
Formatting.

* Updated `bindgen` and `which`.
Adapting code to work with update.

* fix: reconfigure cleanly if prior configure was interrupted abruptly.

* Enable sanitizers in QEMU during debug.

* Added target-usable files.

* Added breakpoint structure.

* Adapted other files to work with ExitHandler.

* Adapted existing fuzzer to work with new exit handler.

* fix: use get to avoid crashes.

* Updated README to indicate cargo-make should be installed.

* Added QEMU internal exit handler.

* Adapted qemu_systemmode example with new exit handler.

* Fixed fuzzers to work with new exit handler.

* Trying to fix CI (#1739)

* test

* dummy

* dummy

* Added new examples.

* Forgot to add build scripts.

* format

* format

* clang-format

* python emulator adaptation.

* fixed python bindings.

* clippy fixes.

* python bindings.

* fix qemu_sugar.

* fix fuzzbench.

* fixed import issues.

* misc fixes.

* renamed crate.

* Updated x86_64 stub bindings.

* Fixed static naming.

* binding fmt

* clippy

* clippy

* Removed useless return statement.

* removed advice to install cargo-make in individual repositories.

* symcc_update (#1749)

* Remove unused create_anymap_for_trait macro (fixes #1719) (#1752)

* Fix `as_object` UB discussed in #1748 (#1751)

* Fix as_object UB discussed in #1748

* More cleanup, more less UB

* Fix fixes

* Added uninit_on_shmem api

* clippy

* fmt

* trying to fix fuzzers, libfuzzer wrapper

* Add OwnedRefMit::owned constructor, libfuzzer fix

* Some more fixes

* Add BacktaceObserver::owned fn

* fmt

* more fmt

* Ignore SigPipe by default (#1741)

* Ignore SigPipe by default

* Fix no_std

* fmt

* Fix incorrect imports (#1758)

* Fix incorrect imports

https://doc.rust-lang.org/core/simd/trait.SimdOrd.html

* Fix

* Try fix ci

* Documentation fixes (#1761)

* Documentation fixes

* Fix InProcessExecutor url

* Update all urls to latest

* Miri ignores for M1 regex (#1762)

* Enabling DrCov on Windows (#1765)

* Enabling DrCov for Windows

* Dedup common code in scheduler (#1702)

* dedup common code in scheduler

* del eco

* fixing

* fix

* replace `Emulator::new_empty` by `Emulator::get` calls outside of `emu.rs` for safety. (#1763)

* Add mute_inprocess_target fn, SimpleFdLogger::set_logger, and more (#1754)

* Add mute_inprocess_target fn, SimpleFdLogger::set_logger, set_error_print_panic_hook

* Trying to fix #1753

* typo

* More fix

* Fix test?

* more testcase fixes

* Fix: renamed remaining QemuInstrumentationFilter instances.

* Split emu in systemmode.rs / usermode.rs for specific code.
EmuExitHandler implementation.

* format

* format

* format

* Replace sync_exit with sync_backdoor.

* Rework command system.

* fix bad import.

* format.

* cargo fmt

* disable af-xdp as well to avoid linking errors.

* End of merging.

* format.

* Adaptation for usermode.

* format.

* injection support.

* usermode fixes.
format.

* clippy

* clippy + format

* Do not unwrap emu + format.

* fix: entry_point breakpoint

* inital commit.

* clippy

* tests

* clippy

* adapt example

* systemmode.

* renaming

* fmt

* fix lints.

* more lint fix.

* even more lint fixes.

* always more lint fixes.

* lint fix.

* allow unused qualifications for crate when it could be confusing.

* Still lint fixes.

* Lint fixes on generated code.

* Some lint fixes.

* merge continue.

* renamed modules as well.

* fixing merge.

* systemmode compiling.

* fmt

* fix early emulator drop.

* fmt

* fix cast to c_void of the wrong object.

* Added global enum for snapshot managers.
Some renaming.

* move things around.

* WIP: generic inclusion of QEMU Executor in exit handler.

* * Moved extern calls to `libafl_qemu_sys`
* Replaced old `Emulator` by `Qemu` and only kept C functions wrappers
* Now `Emulator` is for higher-level interactions with QEMU. Kept old functions for compatibility calling to `Qemu` functions
* A direct side effect of this slit is the removal of the `IsEmuExitHandler` trait dependency added in many parts of the code.
* Removed old dirty casting for `QemuExecutor` helpers and used the brand-new access to `QemuExecutorState` instead.
* Minor changes to `Qemu` and `Emulator` `get` methods for cleaner getters.

* Add missing `Qemu` function.

* Updated `qemu_systemmode` example.

* Adapted QEMU fuzzers + renaming.

* Fixed python.

* fix libafl_sugar with new implementation.

* fix dangling RefCell.
adapt new examples.
TODO: merge `libafl_systemmode.*` examples.

* clippy.

* fix more fuzzers.

* clippy.

* Implement `HasInstrumentationFilter` generically.
Updated `StdInstrumentationFilter` accordingly.

* Renamed breakpoint functions for QEMU.
`qemu.run()` handling.

* Removed OnceCell / RefCell in signature.
more explicit `MmapPerms` method names.

* minor code refactoring

* Emulator::run_handle refactoring

* deprecated Emulator functions calling directly to QEMU functions.

* IsSnapshotManager -> SnapshotManager

* IsEmuExitHandler -> EmuExitHandler + fmt

* Generic register when it makes sense.

* reverted IsSnapshotManager -> SnapshotManager because of a collision.

* fix syntax + clippy

* fmt

---------

Co-authored-by: Dongjia "toka" Zhang <tokazerkje@outlook.com>
Co-authored-by: Dominik Maier <domenukk@gmail.com>
Co-authored-by: lazymio <mio@lazym.io>
Co-authored-by: Bet4 <0xbet4@gmail.com>
Co-authored-by: mkravchik <mkravchik@hotmail.com>
2024-03-22 18:03:29 +01:00
Maurice
50843b19d1
libafl_nyx: Allow custom input buffer size to be passed to NyxHelper (#1960)
* add option to specify input buffer size

* fix typo

* use `libafl::Error` as default error type

* derive `TypedBuilder` for `NyxSettings`

* update nyx_libxml2_standalone

* update nyx_libxml2_parallel

* update nyx_libxml2_standalone

* update nyx_libxml2_standalone

* update nyx_libxml2_parallel
2024-03-21 22:53:01 +01:00
Dongjia "toka" Zhang
6b94db2260
Use OwnedPtr in ListObserver (#1961)
* aaa

* clp
2024-03-21 19:46:42 +01:00
Dongjia "toka" Zhang
15f4613894
More usable ListFeedback (#1959)
* real list feedback

* add

* fox

* obs

* fix

* FMT

* more

* fmt

* rev
2024-03-21 16:21:15 +01:00
Dominik Maier
7abc26ebc9
Change AnyMap API, add unsafe_ assert (#1958)
* Change AnyMap API, add unsafe_ assert

* clippy

* Add anymap serialize test

* Add test

* fmt
2024-03-20 14:45:23 +01:00
Dominik Maier
14fd3040bd
Remove hash for AnyMap (since it's a hashmap already) (#1951) 2024-03-19 20:41:15 +01:00
Dominik Maier
2efa747292
SerdeAnyMap: add unsafe_stable_anymap feature that uses type_name instead of TypeId::of (#1952)
* Test: Use type_name instead of type_id in AnyMap

* Hide behind 'unsafe_stable_anymap' feature

* nostd?

* rename fn

* cleanup a bit
2024-03-19 20:15:31 +01:00
am009
5eab4fb78b
Update tinyinst_simple to support Linux (#1316) (#1955)
* Make fuzzers/tinyinst_simple support Linux (#1316)

Fix a documentation error for `MmapShMemProvider`.

* Support shmem for `fuzzers/tinyinst_simple` on Linux.

Format code.

* Fix CI to install cxxbridge-cmd.

* Add `CARGO_TARGET_DIR` in makefile to fix CI.
2024-03-19 20:15:07 +01:00
Michael Chen
fde9cda5ee
Gramatron: Upgrade automaton files (#1956)
`auto.postcard` file is following `postcard@0.7` wire format, which is
not compatible with the `postcard@1.0` dependency. `auto.json` is out of
date with the latest `Trigger` struct definition. Regenerated both using
the data from `auto.postcard`
2024-03-19 20:09:47 +01:00
Romain Malmain
928cf80782
fix unwrap. (#1954) 2024-03-18 18:51:19 +01:00
Dongjia "toka" Zhang
0f26f6ea32
Remove useless feedback (#1953)
* stuff

* FMT

* remove stuff
2024-03-18 16:41:43 +01:00
Dominik Maier
a0143d02a6
Rename OSError -> OsError and merge with redundant Error::File (#1944)
* OSError -> OsError

* Move File errors to io Errors

* Removing File errors

* fixes :)

* More format

* fix libfuzzer runtime
2024-03-18 08:46:48 +01:00
Stefan Zabka
fbd0d497d4
Refactor(build.rs): extract common code between branches (#1950) 2024-03-17 21:31:30 +01:00
Dongjia "toka" Zhang
44a37da680
Use const_assert not assert (#1949)
* stuff

* FMT
2024-03-16 03:33:20 +01:00
Dongjia "toka" Zhang
c6420c0987
Separate fuzzbench fuzzers's test (#1947)
* separate fuzzbench test

* upd
2024-03-15 19:44:22 +01:00
Dongjia "toka" Zhang
34b4a6ac1d
Better error message instead of "No entries in corpus" 2024-03-15 19:19:55 +01:00
Dongjia "toka" Zhang
c6875b8cf6
Fix self.iterations underflow (#1942)
* fix

* fix

* FMT
2024-03-15 19:13:58 +01:00
Dongjia "toka" Zhang
e30ff57456
Update exec counts in objective (#1945)
* fix

* fix

* update exec count
2024-03-15 18:52:53 +01:00
Dongjia "toka" Zhang
5cc0180835
Fix self.restart_helper.execs_since_progress_start in calibration stage (#1943)
* fix

* fix
2024-03-15 18:31:44 +01:00
Dongjia "toka" Zhang
ee6385c25b
Fire events in append_metadata not in is_interesting (#1936)
* stuff

* ok

* Recalc filled slightly differently... (#1939)

* Recalc filled slightly differently...

* Make requested changes as per PR review

* unused

* fix

---------

Co-authored-by: Dan Blackwell <danblackwell95@yahoo.co.uk>
2024-03-15 13:24:26 +01:00
Romain Malmain
b3ddab3bce
fix libafl_qemu_sys unnecessarily rebuilding. (#1938) 2024-03-14 18:25:19 +01:00
Dominik Maier
04d87ccc89
Remove (almost) unused stage_idx (#1929)
* Remove (almost) unused stage_idx

* Fix text-based test

* fixed critical whitespace

* clippy

* more fmt

* fix push stage
2024-03-14 13:14:57 +01:00
am009
93f67aa405
Fix os error printing in unix_shmem (#1406) (#1935)
Use `std::io::Error::last_os_error` instead of `libc::perror` for error printing in unix_shmem.
Add a new enum item `OSError` to enum Error.
2024-03-14 13:07:04 +01:00
Dongjia "toka" Zhang
afa2965f3c
Remove some arguments from pre_exec/post_exec in ExecutorHook (#1933)
* refactor

* no std, fmt

* win mac

* state

* frida

* ctx

* trait bound

* clip
2024-03-13 18:49:09 +01:00
Dominik Maier
4f3d9d2e50
Tiny optimization for into_vec (#1931)
* Tiny optimization for into_vec

* fix bug
2024-03-13 16:52:15 +01:00
Dongjia "toka" Zhang
44aec56f7e
remove more args (#1934) 2024-03-13 16:17:46 +01:00
Dominik Maier
2763d945a3
Cleanup last redundant corpus_idx usages (#1930)
* Cleanup last redundant corpus_idx usages

* More cleanup
2024-03-13 01:24:11 +01:00
Romain Malmain
9b96149f6c
Do not build doc with all features. (#1928)
Add documentation about default features for documentation.
2024-03-12 23:08:37 +01:00
Dongjia "toka" Zhang
5f67b9fbc4
Event Manager handle_in_client hooks (#1916)
* event manager hooks

* tcp

* fix

* FMT

* fix

* fix
2024-03-12 14:39:16 +01:00
Stefan Zabka
fe1c7a34b7
Dockerfile: setup for building libafl_qemu in Docker (#1899)
* fix(Dockerfile): add build_linux.rs to libafl_qemu and libafl_qemu_sys

* fix(Dockerfile): workaround for llvm.sh failing on bookworm

Revert this once https://github.com/llvm/llvm-project/issues/62475 has been resolved

* fix(libafl_qemu): add unsafe to callbacks

* chore(devcontainer): update devcontainer to work with newer VSCode versions

* revert(libafl_qemu): revert unsafe keyword

* fix(devcontainer): move settings and extension under customizations.vscode

* fix(devcontainer): use postCreateCommand instead of initializeCommand
2024-03-12 01:00:35 +01:00
Dominik Maier
dd410c590a
Allow dyn in StagesTuple, add Current Testcase API, Untraitify Progress (#1915)
* Move into_vec to extra trait

* fix no_std

* First step towards stages tuples as vecs

* Allow dyn in StagesTuple, add Current Testcase API, un-traitify Progress, cleanups

* Move from generics to impl keyword, more replacements with better API

* rename fn

* Fix additional stages, more cleanup, rename progress to retries

* Fix more fixes

* Fixes

* Rename ProgressHelper -> RestartHelper

* Fix sugar, python, add perform_restartable

* fixes

* remove prelude bs

* rename to restart_progress_should_run

* more cleanup, remove tests I don't understand (sorry)

* fix docs

* more fix

* fix miri

* unsafe safety annotations

* more comments

* last docs

* Mixed_attributes only allowed for bindgen
2024-03-12 00:58:07 +01:00
Mohammad Omidvar
61046c4157
Fix reproducibility of the docker image (#1924)
Add `--locked` flag when installing `sccache` in the first stages to ensure the image gets built when the base image's Rust version satisfies `sccache`'s MSRV.

Fixes #1923 🐛

Co-authored-by: Dongjia "toka" Zhang <tokazerkje@outlook.com>
2024-03-11 19:15:54 +01:00
Dongjia "toka" Zhang
9b780cc0a7
Fix UB in anymap.rs and other minor warnings. (#1926)
* fix

* clippy

* fix
2024-03-11 18:15:56 +01:00
Romain Malmain
e745401a39
Update QEMU version. (#1922)
* update QEMU version.

* clippy
2024-03-08 13:15:00 +01:00
Aarnav
d6fe67c3c8
Introduce multicore loading for the initial seed corpus (#1905)
* introduce multicore load initial corpus

* update fuzzers/libfuzzer_libpng_norestart to use multicore corpus loading

* run clippy

* use CoreId and Cores in state if std

* misc. typos

* adapt multicore load initial inputs to allow resumable corpus loading
in case of crashes or timeouts during corpus loading.

* add std feature flag to multicore_inputs_processed

* fix doc comment

* run fmt for example fuzzer

---------

Co-authored-by: aarnav <aarnav@srlabs.de>
Co-authored-by: Romain Malmain <romain.malmain@pm.me>
Co-authored-by: Dongjia "toka" Zhang <tokazerkje@outlook.com>
2024-03-08 02:56:08 +01:00
Dongjia "toka" Zhang
1b9f4ea29c
Fix batched timeout (#1920)
* fix ptr

* fix ptr

* fix

* One less windows warning

* One less windows warning
2024-03-07 23:52:56 +01:00
Romain Malmain
781e830923
fix cast to c_void of the wrong object. (#1921) 2024-03-07 21:28:28 +01:00
Mrmaxmeier
6747a7dc2f
token mutations: avoid a few infallible unwraps (#1919) 2024-03-07 18:26:59 +01:00
Mrmaxmeier
e48c8c0e23
fuzzbench_ctx: fix duplicate package warning (#1918)
Cargo previously complained when loading libafl as a git dependency:

warning: skipping duplicate package `fuzzbench` found at
`~/.cargo/git/checkouts/libafl-c33dc6f5ec2f7a70/55a300d/fuzzers/fuzzbench`
2024-03-07 17:56:40 +01:00
Dongjia "toka" Zhang
9a2e7b0e64
Fix redundant xor in Ngram calculation (#1901)
* fix double xor

* tmp

* clp

* no notnightly

* fix

* plural
2024-03-07 14:49:01 +01:00
Stefan Zabka
2ac075024b
fix(Dockerfile): allow running commands with docker run (#1894)
* fix(Dockerfile): allow running commands with docker run

According to
<https://gitlab.com/gitlab-org/gitlab-runner/-/issues/2109#note_47480476>
adding the "-c" to the ENTRYPOINT is necessary to be able to run
commands like `docker run libafl "cargo build"`

* fix(Dockerfile): restore ability to docker run without command

As `bash -c` always requires a string, the previous commit
removed the ability to just `docker run -it libafl` and
get a shell.
This commit restores that ability
2024-03-06 19:14:18 +01:00
Dongjia "toka" Zhang
969aa12c7e
aa (#1913) 2024-03-06 19:05:32 +01:00
Romain Malmain
3b3e2f6efa
Improve readability of InProcessExecutor-related code (#1912)
* inital commit.

* clippy

* tests

* clippy

* adapt example

* systemmode.

* renaming

* fmt

* fix lints.

* more lint fix.

* even more lint fixes.

* always more lint fixes.

* lint fix.

* allow unused qualifications for crate when it could be confusing.

* Still lint fixes.

* Lint fixes on generated code.

* Some lint fixes.

* renamed modules as well.

* Separated inner from InProcessExecutor.

* fix

* unused import

* unused import

* fix import

* fix import
2024-03-06 00:30:35 +01:00
Romain Malmain
55a300d508
Make executor state available to the harness V2 (#1900)
* inital commit.

* clippy

* tests

* clippy

* adapt example

* systemmode.

* renaming

* fmt

* fix lints.

* more lint fix.

* even more lint fixes.

* always more lint fixes.

* lint fix.

* allow unused qualifications for crate when it could be confusing.

* Still lint fixes.

* Lint fixes on generated code.

* Some lint fixes.

* renamed modules as well.
2024-03-05 19:28:46 +01:00
Rayhan Faizel
44b0b0ad82
libafl_targets/src/cmplog.c: Fix warning for deprecated syscall() in macOS (#1911) 2024-03-05 19:04:20 +01:00
Dan Blackwell
cd8003ee69
Fix for #1881, ProbabilitySamplingScheduler seems to be inverted (#1884)
* Fix for #1881, ProbabilitySamplingScheduler seems to be inverted

* Run 'cargo +nightly fmt'

* Added debug_assert as negative and infinite probability values would break the current implementation of next()

---------

Co-authored-by: Dan Blackwell <danblackwell@eduroam-int-dhcp-97-192-218.ucl.ac.uk>
Co-authored-by: Dan Blackwell <danblackwell@ML-W0NYVW4XCK.local>
Co-authored-by: Dongjia "toka" Zhang <tokazerkje@outlook.com>
2024-03-05 19:04:13 +01:00
Dongjia "toka" Zhang
e3f837d712
Fix inconsistent settings of client_timeout (#1897)
* a

* fix client timeout

* revert

* more

* std

* import

* import

* sdt

* FMT

* backtick again
2024-03-05 17:58:44 +01:00
Dongjia "toka" Zhang
1a0e692f33
Print PID when you create a new Llmp Sender. (#1898)
* aaaa

* FMT

* adding std
2024-03-05 16:02:47 +01:00
Dongjia "toka" Zhang
95bed5b016
Fix the centralize disconnection bug. (#1896)
* fixing the first bug

* backtick
2024-03-05 15:49:35 +01:00
Romain Malmain
d96a1426d5
Fix lint errors (#1909)
* fix lints.

* more lint fix.

* even more lint fixes.

* always more lint fixes.

* lint fix.

* allow unused qualifications for crate when it could be confusing.

* Still lint fixes.

* Lint fixes on generated code.

* Some lint fixes.
2024-03-05 14:16:26 +01:00