1077 Commits

Author SHA1 Message Date
mkravchik
886519b10c
Pr/fixing forkserver libafl cc (#2066)
* Fixing forserver_libafl_cc

* Adding tests and showing user stats

* Restoring the map truncation

* Fmt

* small fix

* fix

* fix

* fix

---------

Co-authored-by: Dongjia "toka" Zhang <tokazerkje@outlook.com>
2024-04-17 18:28:34 +02:00
Dongjia "toka" Zhang
cbb323f26c
LibAFL 0.12 (#1823) 2024-04-12 18:42:14 +02:00
Addison Crump
3d702f403b
Implement CanTrack tracking enforcement through rust types (#1886)
* sample implementation of tracking enforcement (incomplete)

* helpful compiler output

* make it look like a real compiler output

* ensure that the macro may be used outside of libafl

* separate index/novelty tracking funcs

* default const generic values so that we don't need to change this everywhere

* fix tests

* rollback unnecessary specification of stdmapobserver

* register metadata in doc tests

* doc fixes

* doc cleanup

* doc cleanup 2

* reduce implementor overhead to zero

* renaming/docs fixes

* asref isn't reflexive??

* generalization stage updates

* add better documentation about require_{indices,novelties}_tracking

* remaining generic updates

* round one CI pass (knowingly introduces breaking changes)

* typo

* round 2 clippy

* rollback: libafl_frida changes

* fmt

* moar porting

* fix remaining fuzzers

* fix windows build, maybe

* fixup libafl_libfuzzer

* fmt nighlty all the things

* attempt to fix some broken additions

* fix fmt

* oops

* fix new invocation

* minimizer scheduler fixes

* fix accounting

* rename

* fix

* Fix build

* Sort generics

* Move more generics into the right place

* Rename A -> C

* Fix test

* Fix test some more

* Fix doc some more

* critical formatting

* More A->C

* CanTrack harder

---------

Co-authored-by: Dominik Maier <dmnk@google.com>
2024-04-12 18:33:52 +02:00
Maurice
8b9b5a8767
Introduce StdOutToMetadataFeedback and StdErrToMetadataFeddback (#2042)
* introduce `StdOutFeedback` and `StdErrFeddback`

* hide behind "std" feature flag

* update

* update
2024-04-12 14:26:47 +02:00
Dominik Maier
2f618b62d7
Rename HasAFLSchedulerMetadata to simply AflScheduler (it doesn't hold a Metadata at all) (#2043) 2024-04-12 13:24:00 +02:00
Dominik Maier
0f778b5892
Order differential generics alphabetically (#2044) 2024-04-12 12:14:49 +02:00
Dongjia "toka" Zhang
287d1ac7c7
Add HasAdaptiveSerializer trait. (#2040)
* fix

* revert test

* add

* a

* check
2024-04-11 15:36:08 +02:00
Dongjia "toka" Zhang
0d5c6219d8
Dedup CentralizedEventManager's serialize_observers() (#2034)
* why

* clp
2024-04-10 16:13:03 +02:00
Aarnav
47c41c2925
Introduce disabled testcases for splicing (#1932)
* introduce disabled field to Testcase

* separate executor's processing of execution (adding to corpus/solution/discarding) and execution of input

* introduce add_disabled_input function

* enable splicing mutators to fetch disabled inputs

* reset modified example

* clean up

* update docs

* update docs for count_with_disabled

* fix random_corpus_id for splicing mutator not considering disabled entries

* fmt

* update docs

* clippy

* fix corpus_btreemap not working

* fix clippy warnings

* fix python bindings

* typo in count_with_disabled implementations

* fix certain splicing mutators not considering disabled inputs

* rename count_with_disabled to count_all

* introduce count_disabled function

* update docs for count_all, count_disabled and count

* * introduce get_from_all and nth_from_all for corpus implementations so get() and nth() do not silently fetch disabled entries.
* remove boolean flag from random_corpus_id which allowed inclusion of disabled ids and make it into a new function random_corpus_id_with_disabled
* update docs

* remove boolean is_disabled flag from corpus::insert and make it into a separate function insert_disabled

* rename do_insert to _insert

* make get_from_all inline for cached and inmemory

* add missing functions implementation for PythonCorpus
prevent writing feedback when adding disabled testcases

* fix nth_from_all overfetching enabled corpus entries

* fix clippy & rename execute_with_res to execute_no_process

* refactor _insert for corpus_btreemap

* make LibfuzzerCorpus and ArtifactCorpus to accomodate disabled entries

* fix typo

* fix missing docs for map field

* fix clippy

* test

* (hopefully) fix CachedOnDiskCorpus using incorrect corpus when caching testcase

* fix typo in inmemory_ondisk leading to fetching disabled entry from enabled corpus

---------

Co-authored-by: aarnav <aarnav@srlabs.de>
Co-authored-by: Dominik Maier <domenukk@gmail.com>
2024-04-10 01:03:00 +02:00
Dongjia "toka" Zhang
c4f076356f
Fix the OOM exit code from 137 to 9 (#2029)
* i don't know if there really exit 137

* change message

* not 137 but 9
2024-04-09 17:16:25 +02:00
Dongjia "toka" Zhang
e8fe5bb614
Move HasMetadata/HasNamedMetadata trait to libafl_bolts (#2016)
* move around

* fmt

* rev

* fmt

* doc job

* name

* move to libafl

* separate even more

* miri
2024-04-08 19:24:56 +02:00
Dominik Maier
ff938261df
Fix Clippy on MacOS for qemu, accept some clippy::missing_transmute_annotations suggestions (#2009)
* Fix Clippy/CI

* Ignore transmute annotations in pybind

* fix

* more clippy

* more clippy

* fix fixes
2024-04-08 16:24:10 +02:00
Dongjia "toka" Zhang
159da0d311
Change when weighted scheduler make new table & don't update global metadata in on_remove/on_replace (#2011)
* fix

* backtick
2024-04-07 17:24:27 +02:00
Dongjia "toka" Zhang
527b892c1d
Separate fuzzer test in CI (#2010)
* ci

* ci

* ci

* aa

* aa

* a

* a

* a

* fix

* no fail fast

* system

* what is this lint...

* a

* clp

* disk

* ok no fastfail

* no qemu for now, reenable monday
2024-04-06 21:01:57 +02:00
Dongjia "toka" Zhang
4901846588
Remove python bindings (#2005)
* delete python bindings

* fix

* example

* fix
2024-04-06 15:16:46 +02:00
Dominik Maier
1c85c3af13
Add option to enabled/disable client stats and fix #1771 (#2001)
* Add option to enabled/disable client stats and fix #1771

* more fix

* fix map_density

* even more fix

* remove need for vec in Aggregator::aggregate

* fix json weirdness - remove individual clients (is that all right? )

* Make pretty
2024-04-05 14:23:56 +02:00
Dongjia "toka" Zhang
98d3dfe821
Replace LLMP client timeout with client deregistration (#1982)
* wip

* done

* fix

* fix

* some fixes

* stuff

* fmt

* stuff

* use

* not 1337

* comment

* move functions around

* fix

* fix

* doc

* mistake

* aa

* fixer

* wipe out restarter id

* denig

* fix

* fix

* include

* fix

* fix

* fix

* clp

* fix

* fix

* error log
2024-04-04 21:01:11 +02:00
Dongjia "toka" Zhang
2137ad0f8f
Don't resume in std mutational stage (#1946)
* fix

* fix

* fix

* fix

* FMT

* don't resume in mutational stage
2024-04-02 13:26:47 +02:00
ToSeven
10f373d587
fix AFL-Style UI errors when allowing the introspection feature (#1986) 2024-04-01 14:09:15 +02:00
Andrea Fioraldi
45d47214c2
Fix OOM restarts with LlmpShouldSaveState (#1974)
* LlmpSaveState and OOM restarts

* clippy

* clippy

* rename
2024-03-30 17:05:03 +01:00
Dongjia "toka" Zhang
0cafa5c72d
Don't use unwrap() in the crash handler (#1979)
* don't

* win

* fix

---------

Co-authored-by: Andrea Fioraldi <andreafioraldi@gmail.com>
2024-03-30 17:04:28 +01:00
s1341
1fc5ba63fa
batch_timeout: fix overflowing subtraction (#1976) 2024-03-28 16:50:12 +01:00
am009
c221108916
Add MutatedTransform to the input type in TMinMutationalStage (#1251) (#1971)
* Support `MutatedTransform` in `TMinMutationalStage`.

* Run `MutatedTransformPost` for the replaced testcase.

* Add clone trait bound for `MutatedTransformPost`.

* Return an error instead of using unwrap.
2024-03-27 18:59:45 +01:00
Dongjia "toka" Zhang
53bfae0a08
Increase llmp timeout & Print PID in logger (#1970)
* increase timeout & print pid for logger

* log

* aa

* rename stuff
2024-03-25 19:15:46 +01:00
Dongjia "toka" Zhang
7170b5391b
Fix UpdateExecStats not sent to the main broker in centralized 2024-03-25 16:39:30 +01:00
Dongjia "toka" Zhang
3cc8ea0336
Fix EventManagerHook (#1968) 2024-03-25 13:04:41 +01:00
Romain Malmain
44c841ffb1
WIP: QEMU exit handler (#1745)
* Added paging filtering.
Reworked address range filtering to fit with new generic code.

* Fix: renamed remaining QemuInstrumentationFilter instances.

* Renamed sync breakpoint to sync exit.

* Split emu in systemmode.rs / usermode.rs for specific code.
EmuExitHandler implementation.

* sync_backdoor.rs removal.
Formatting.

* Updated `bindgen` and `which`.
Adapting code to work with update.

* fix: reconfigure cleanly if prior configure was interrupted abruptly.

* Enable sanitizers in QEMU during debug.

* Added target-usable files.

* Added breakpoint structure.

* Adapted other files to work with ExitHandler.

* Adapted existing fuzzer to work with new exit handler.

* fix: use get to avoid crashes.

* Updated README to indicate cargo-make should be installed.

* Added QEMU internal exit handler.

* Adapted qemu_systemmode example with new exit handler.

* Fixed fuzzers to work with new exit handler.

* Trying to fix CI (#1739)

* test

* dummy

* dummy

* Added new examples.

* Forgot to add build scripts.

* format

* format

* clang-format

* python emulator adaptation.

* fixed python bindings.

* clippy fixes.

* python bindings.

* fix qemu_sugar.

* fix fuzzbench.

* fixed import issues.

* misc fixes.

* renamed crate.

* Updated x86_64 stub bindings.

* Fixed static naming.

* binding fmt

* clippy

* clippy

* Removed useless return statement.

* removed advice to install cargo-make in individual repositories.

* symcc_update (#1749)

* Remove unused create_anymap_for_trait macro (fixes #1719) (#1752)

* Fix `as_object` UB discussed in #1748 (#1751)

* Fix as_object UB discussed in #1748

* More cleanup, more less UB

* Fix fixes

* Added uninit_on_shmem api

* clippy

* fmt

* trying to fix fuzzers, libfuzzer wrapper

* Add OwnedRefMit::owned constructor, libfuzzer fix

* Some more fixes

* Add BacktaceObserver::owned fn

* fmt

* more fmt

* Ignore SigPipe by default (#1741)

* Ignore SigPipe by default

* Fix no_std

* fmt

* Fix incorrect imports (#1758)

* Fix incorrect imports

https://doc.rust-lang.org/core/simd/trait.SimdOrd.html

* Fix

* Try fix ci

* Documentation fixes (#1761)

* Documentation fixes

* Fix InProcessExecutor url

* Update all urls to latest

* Miri ignores for M1 regex (#1762)

* Enabling DrCov on Windows (#1765)

* Enabling DrCov for Windows

* Dedup common code in scheduler (#1702)

* dedup common code in scheduler

* del eco

* fixing

* fix

* replace `Emulator::new_empty` by `Emulator::get` calls outside of `emu.rs` for safety. (#1763)

* Add mute_inprocess_target fn, SimpleFdLogger::set_logger, and more (#1754)

* Add mute_inprocess_target fn, SimpleFdLogger::set_logger, set_error_print_panic_hook

* Trying to fix #1753

* typo

* More fix

* Fix test?

* more testcase fixes

* Fix: renamed remaining QemuInstrumentationFilter instances.

* Split emu in systemmode.rs / usermode.rs for specific code.
EmuExitHandler implementation.

* format

* format

* format

* Replace sync_exit with sync_backdoor.

* Rework command system.

* fix bad import.

* format.

* cargo fmt

* disable af-xdp as well to avoid linking errors.

* End of merging.

* format.

* Adaptation for usermode.

* format.

* injection support.

* usermode fixes.
format.

* clippy

* clippy + format

* Do not unwrap emu + format.

* fix: entry_point breakpoint

* inital commit.

* clippy

* tests

* clippy

* adapt example

* systemmode.

* renaming

* fmt

* fix lints.

* more lint fix.

* even more lint fixes.

* always more lint fixes.

* lint fix.

* allow unused qualifications for crate when it could be confusing.

* Still lint fixes.

* Lint fixes on generated code.

* Some lint fixes.

* merge continue.

* renamed modules as well.

* fixing merge.

* systemmode compiling.

* fmt

* fix early emulator drop.

* fmt

* fix cast to c_void of the wrong object.

* Added global enum for snapshot managers.
Some renaming.

* move things around.

* WIP: generic inclusion of QEMU Executor in exit handler.

* * Moved extern calls to `libafl_qemu_sys`
* Replaced old `Emulator` by `Qemu` and only kept C functions wrappers
* Now `Emulator` is for higher-level interactions with QEMU. Kept old functions for compatibility calling to `Qemu` functions
* A direct side effect of this slit is the removal of the `IsEmuExitHandler` trait dependency added in many parts of the code.
* Removed old dirty casting for `QemuExecutor` helpers and used the brand-new access to `QemuExecutorState` instead.
* Minor changes to `Qemu` and `Emulator` `get` methods for cleaner getters.

* Add missing `Qemu` function.

* Updated `qemu_systemmode` example.

* Adapted QEMU fuzzers + renaming.

* Fixed python.

* fix libafl_sugar with new implementation.

* fix dangling RefCell.
adapt new examples.
TODO: merge `libafl_systemmode.*` examples.

* clippy.

* fix more fuzzers.

* clippy.

* Implement `HasInstrumentationFilter` generically.
Updated `StdInstrumentationFilter` accordingly.

* Renamed breakpoint functions for QEMU.
`qemu.run()` handling.

* Removed OnceCell / RefCell in signature.
more explicit `MmapPerms` method names.

* minor code refactoring

* Emulator::run_handle refactoring

* deprecated Emulator functions calling directly to QEMU functions.

* IsSnapshotManager -> SnapshotManager

* IsEmuExitHandler -> EmuExitHandler + fmt

* Generic register when it makes sense.

* reverted IsSnapshotManager -> SnapshotManager because of a collision.

* fix syntax + clippy

* fmt

---------

Co-authored-by: Dongjia "toka" Zhang <tokazerkje@outlook.com>
Co-authored-by: Dominik Maier <domenukk@gmail.com>
Co-authored-by: lazymio <mio@lazym.io>
Co-authored-by: Bet4 <0xbet4@gmail.com>
Co-authored-by: mkravchik <mkravchik@hotmail.com>
2024-03-22 18:03:29 +01:00
Dongjia "toka" Zhang
6b94db2260
Use OwnedPtr in ListObserver (#1961)
* aaa

* clp
2024-03-21 19:46:42 +01:00
Dongjia "toka" Zhang
15f4613894
More usable ListFeedback (#1959)
* real list feedback

* add

* fox

* obs

* fix

* FMT

* more

* fmt

* rev
2024-03-21 16:21:15 +01:00
Dominik Maier
7abc26ebc9
Change AnyMap API, add unsafe_ assert (#1958)
* Change AnyMap API, add unsafe_ assert

* clippy

* Add anymap serialize test

* Add test

* fmt
2024-03-20 14:45:23 +01:00
Dongjia "toka" Zhang
0f26f6ea32
Remove useless feedback (#1953)
* stuff

* FMT

* remove stuff
2024-03-18 16:41:43 +01:00
Dominik Maier
a0143d02a6
Rename OSError -> OsError and merge with redundant Error::File (#1944)
* OSError -> OsError

* Move File errors to io Errors

* Removing File errors

* fixes :)

* More format

* fix libfuzzer runtime
2024-03-18 08:46:48 +01:00
Dongjia "toka" Zhang
44a37da680
Use const_assert not assert (#1949)
* stuff

* FMT
2024-03-16 03:33:20 +01:00
Dongjia "toka" Zhang
34b4a6ac1d
Better error message instead of "No entries in corpus" 2024-03-15 19:19:55 +01:00
Dongjia "toka" Zhang
c6875b8cf6
Fix self.iterations underflow (#1942)
* fix

* fix

* FMT
2024-03-15 19:13:58 +01:00
Dongjia "toka" Zhang
e30ff57456
Update exec counts in objective (#1945)
* fix

* fix

* update exec count
2024-03-15 18:52:53 +01:00
Dongjia "toka" Zhang
5cc0180835
Fix self.restart_helper.execs_since_progress_start in calibration stage (#1943)
* fix

* fix
2024-03-15 18:31:44 +01:00
Dongjia "toka" Zhang
ee6385c25b
Fire events in append_metadata not in is_interesting (#1936)
* stuff

* ok

* Recalc filled slightly differently... (#1939)

* Recalc filled slightly differently...

* Make requested changes as per PR review

* unused

* fix

---------

Co-authored-by: Dan Blackwell <danblackwell95@yahoo.co.uk>
2024-03-15 13:24:26 +01:00
Dominik Maier
04d87ccc89
Remove (almost) unused stage_idx (#1929)
* Remove (almost) unused stage_idx

* Fix text-based test

* fixed critical whitespace

* clippy

* more fmt

* fix push stage
2024-03-14 13:14:57 +01:00
Dongjia "toka" Zhang
afa2965f3c
Remove some arguments from pre_exec/post_exec in ExecutorHook (#1933)
* refactor

* no std, fmt

* win mac

* state

* frida

* ctx

* trait bound

* clip
2024-03-13 18:49:09 +01:00
Dominik Maier
4f3d9d2e50
Tiny optimization for into_vec (#1931)
* Tiny optimization for into_vec

* fix bug
2024-03-13 16:52:15 +01:00
Dongjia "toka" Zhang
44aec56f7e
remove more args (#1934) 2024-03-13 16:17:46 +01:00
Dominik Maier
2763d945a3
Cleanup last redundant corpus_idx usages (#1930)
* Cleanup last redundant corpus_idx usages

* More cleanup
2024-03-13 01:24:11 +01:00
Dongjia "toka" Zhang
5f67b9fbc4
Event Manager handle_in_client hooks (#1916)
* event manager hooks

* tcp

* fix

* FMT

* fix

* fix
2024-03-12 14:39:16 +01:00
Dominik Maier
dd410c590a
Allow dyn in StagesTuple, add Current Testcase API, Untraitify Progress (#1915)
* Move into_vec to extra trait

* fix no_std

* First step towards stages tuples as vecs

* Allow dyn in StagesTuple, add Current Testcase API, un-traitify Progress, cleanups

* Move from generics to impl keyword, more replacements with better API

* rename fn

* Fix additional stages, more cleanup, rename progress to retries

* Fix more fixes

* Fixes

* Rename ProgressHelper -> RestartHelper

* Fix sugar, python, add perform_restartable

* fixes

* remove prelude bs

* rename to restart_progress_should_run

* more cleanup, remove tests I don't understand (sorry)

* fix docs

* more fix

* fix miri

* unsafe safety annotations

* more comments

* last docs

* Mixed_attributes only allowed for bindgen
2024-03-12 00:58:07 +01:00
Aarnav
d6fe67c3c8
Introduce multicore loading for the initial seed corpus (#1905)
* introduce multicore load initial corpus

* update fuzzers/libfuzzer_libpng_norestart to use multicore corpus loading

* run clippy

* use CoreId and Cores in state if std

* misc. typos

* adapt multicore load initial inputs to allow resumable corpus loading
in case of crashes or timeouts during corpus loading.

* add std feature flag to multicore_inputs_processed

* fix doc comment

* run fmt for example fuzzer

---------

Co-authored-by: aarnav <aarnav@srlabs.de>
Co-authored-by: Romain Malmain <romain.malmain@pm.me>
Co-authored-by: Dongjia "toka" Zhang <tokazerkje@outlook.com>
2024-03-08 02:56:08 +01:00
Dongjia "toka" Zhang
1b9f4ea29c
Fix batched timeout (#1920)
* fix ptr

* fix ptr

* fix

* One less windows warning

* One less windows warning
2024-03-07 23:52:56 +01:00
Romain Malmain
781e830923
fix cast to c_void of the wrong object. (#1921) 2024-03-07 21:28:28 +01:00
Mrmaxmeier
6747a7dc2f
token mutations: avoid a few infallible unwraps (#1919) 2024-03-07 18:26:59 +01:00
Dongjia "toka" Zhang
969aa12c7e
aa (#1913) 2024-03-06 19:05:32 +01:00