A summer with QEMU

18 Aug, 2026

Day 1

7 March 2026 (~2hr)

Today's Target: My first target will not be diving into code but understanding migration

• Starting point: https://wiki.qemu.org/ToDo/LiveMigration#Fast_load_snapshot

• Keywords to work on:

  • QMP "snapshot-load" and QMP "migrate_incoming"
  • userfaultfd
  • loadvm(process)
  • postcopy live migration
  • mapped-ram feautre

• So basically Idea I got:

  • Usually on bootup the RAM for VM is created from scratch(initramfs), but VM gives us ability to use snapshots
  • Snapshot is when we save the entire state of VM and boot it from that exact state with no need to boot
  • Usually if the RAM of VM is $x$ GiB we need to load the entire $x$ GiB into RAM and then let the VM run
  • What this optimization wants to do is initiate VM with minimal possible state(load only device data like CPU regs)
  • Now whenever the VM accesses a memory that should have been there in memory in eager case a page fault will occur
  • We will use userfaultfd to handle this page fault and load the required page in memory and let VM continue

• Following step: read the man pages for userfaultfd

  • Hmm so we use int syscall(SYS_userfaultfd, int flags);, does this mean it is a syscall in linux?
  • Let me see what is the definition of SYS_userfaultfd in syscall.h, ... and it is 323
  • Reading from man 2 syscall it seems, linux has 470 syscalls 😮, xv6 had only 22 😔
  • Reading through actual linux/userfaultfd.h I didn't really understand a lot, I was looking for a proper function userfaultfd()
  • userfaultfd object in man pages is I think the uffd_msg, and is configured by ioctl which I dont know anything about
  • So all it gives is a file descriptor to space where all errors/page faults will be written
  • Important key words: ioctl, UFFDIO_API

• Basic idea:

  thread 1 causes page fault
  -> goes to sleep
  -> OS writes the error in `userfaultfd`
  -> OS wakes up thread 2(if sleeping)
  -> thread 2 reads error from `userfaultfd`
  -> handles error using `ioctl_userfaultfd`
  -> goes back to sleep
  -> OS marks thread 1 as ready
  -> thread 1 reissues error instruction and continue like nothing happened
  

• It would be better to learn more about its usage from the actual code that uses it in postcopy live migration process, we will get back to man pages later on

• Following target: understand the code behind postcopy live migration process

• I think this will be the starting resource, I have cloned the git repo but we will read code after reading this bit of documentation, https://www.qemu.org/docs/master/devel/migration/postcopy.html

• Now reading the documentation at Developer Information/Internal Subsystem/Information Migration/Migration features/Postcopy:

  • Reading first line I think I need to read about what exactly migration here is, going to its documentation:
    • So we are basically running the virtual guest on physical machine A but then we want to move to physical machine B and that is called migration
    • I get the gist, there is feature of live migration where the guest keeps running while migration till the last few data transfer is reached
    • This would likely require postcopy, but first need to find what exactly is convergence
  • So what postcopy guarantees is that amount of time and traffic for migration will be bound but a failure on either side will be catastrophic for guest
  • So this nearly does exactly what our project is meant to do, this does it on a running migration stream and we have sort of a file
  • Our job seems easier than already implemented logic, and seems essentially a subset, though deeper analysis might be required
  • Important state cycle: ADVISE->DISCARD->LISTEN->RUNNING->END
    • ADVISE: Checks if the OS is compatible(maybe like if the OS has stuff like userfaultfd), RAM setup might be allocating space so problem does not occur later on
    • DISCARD: This function requires all huge pages to be disabled, exact function ambiguous
    • LISTEN: Now the actual job starts, recieving end, actively starts listening for data on migration stream and prepares RAM to signal data absence(userfault)
    • RUNNING: Processing nears finish as data transfer is complete and the actual guest starts running
    • END: Listen thread quits and after some cleanup migration is complete
  • Following Data transfer says that the initial device data is sent in one packaged blob so that when destination begins running, then on the stream is sensatized to destination requests
  • [HOW] is the live migration implemented in "pre-copy" regime(the normal migration) as there we send all data and then run the program, here we first run the program(using basic device data) and copy rest over in the post fashion
  • It looks like dirty pages refer to the pages that have been written on but not read from, so the destination sends signals for pages it wants to read from but are not there, and the source tries to send those pages and keeps sending pages near it not sent yet, but how would it know about dirty ones(maybe the destination will signal it)
  • WAIT! what if after migration not only destination but the source is also required to keep continuing the guest process? That changes a lot of things! A quick google search and no it is a move
  • In case of network error the source data is safe, it is the entire data that was at beginning of transfer. If it is meant that the destination machine has made progress and is ahead of source they indeed are different but then we can restart process ie the failure is not catastrophic as source machine has data preserved. The destination can also progress until an absent page is required, after which it is stalled till recovery is complete.
  • But if you want performance it is catastrophic as you need to recopy the entire data sent which is the worst possible thing to do, I remember cursing google drive when it reset my backup upload to zero after any small network disturbance🥲. The resolution needs to pin point where the error occured and sync must restore, making it harder.
  • But my idea will not have a lot like this, this is extra feature maybe, but network errors are much more prone than file reading errors, and I think the first prototype can just start from scratch, if enough time is given maybe we can think of adding safegauard and resolution in case of disk failure
  • I think strategically I can skip large hugepages as logic would translate, implementation will be cumbersome, all this would be later stages of project
  • Shared memory might or might not affect us but again is something I think can be skipped for not. again later stages
  • Postcopy blocktime gives the statistics of how fast the operation occurs, this sort of a thing will be necessary in testing and benchmarking of code

• Understood much of Postcopy, next day will be based on implementation details and the mapped-ram migration feature

• So a crude idea forming:

  • Start of project will be like taking the Postcopy methodology and moulding it into our specific usecase
  • Implement the most basic and primitive version in say a month(preferably 3 weeks for me)
  • Add multiple practical features like huge pages support, better error handling etc then(preferabley 3 weeks)
  • Then the rest time will be given to working on documentation(qemu has good documentation and I must match the quality)
  • Also the last time might include adding multiple tests and potentially benchmarking

Day 2

8 March 2026 (~3hrs)

Today's Target: Today we will read documentation of mapped-ram migration and if time permits move to reading actual code

• Starting point for today: https://www.qemu.org/docs/master/devel/migration/mapped-ram.html

• This seems to work on file: migration? looks like it is migrating the guest into a file? so a snapshot?

• This essentially is sort of a new stream format for parallelized migration to files

• Wait, I think they it refers to the fact that everything in linux is a file, so this is just a copy paste which pre decides which part of the user data(RAM) goes to which part of the destination file

• Yeah the following note resolves lot of confusion: Mapped-ram migration is best done non-live, i.e. by stopping the VM on the source side before migrating

• What this specific method is meant for uploading the guest into a file in file system as sort of a snapshot!!

• Ohh so this specific process of migrating a paused VM using mapped ram is usually faster for large RAM VMs than even the usual snapshot

• Basic idea:

  Usual methodology:
  thread A says I am going to send following 1..n ramblocks, and then starts sending the streams in order
  
  Mapped RAM:
  thread A says I am going to send a ramblock that is mapped at some location and then sends
  Keep this going in sequence or parallel
  

• As quite understandable this does not allow sockets etc. for streaming data as we need offsets, but this does not really affect us as our aim is to load a snapshot from file to RAM, both of which have offsets defined

• Well that was fast, 20 mins, lets dive into the code now

• I think first let me see if I can build the system:

  • Just 4 commands! Build is extrememly easy and smooth(Just very long)
  • Oh wait I forgot -j8, now it's fast(er)
  • It build I have no idea how to use/test 🥲
  • Ok from https://wiki.qemu.org/Hosts/Linux#Simple_build_and_test I was able to get it running 🥳

• Now we dive into the C code!!

• I think most of required stuff will be in the migration directory, but no idea where to start?

• Postcopy grep shows a lot of outputs, lets first begin with migration.c as it looks like the central file

  • I will mostly look for postcopy as otherwise the codebase is too huge to get lost

  • Very nice I found file that mixes both things I need to find postcopy-ram.h 🫠

  • Interestingly this function

    bool migrate_postcopy_preempt(void);
    

    Is not defined anywhere(according to vscode reference finder), which is quite weird?

  • I think I can assume this returns true if postcopy will be used

  • The function migration_object_init is important as that is where it will start(thought not necessary to me as what I need is the process understanding)

  • What!! there is a struct called Object?? and everyone is initialized form that 😲. C programming practices never fail to surprise me

  • Really interesting that they are using the fact that first attribute of struct Object is ObjectClass and of that is Type so we can find the type using pointers directly, no arithmatic required

  • Again the declaration of function MIGRATION_OBJ is done using another macro DECLARE_OBJ_CHECKERS that itself is full of macros to simply make functions to inialize a type of object from Object pointer

  • still after all this toil the static MigrationState current_migration will be filled with a simple class?

  • Now its initalization is seemingly simple, with all postcopy related attributes being set to 0

  • I wonder why false and FALSE are two different macros begin 0 and (0), looks like an easily cleanable thing, might be in true too

  • Ah so this might be important, the PostcopyState in migration_incoming_state_destroy, oh but all it does is required cleanup

  • This function will be necessary to request for pages maybe migrate_send_rp_message_req_pages, but a point to think is what sort of system is better:

    • 2 threads, one working on sending/uploading the data onto the RAM like migration from file
    • 1 thread that just waits for faults and fills ram on demand

  • I think first one might be more generalized on say usecases where we want to run guest on another machine and is stored somewhere else, like using network I want to just instantly open the snapshot of a simple VM stored on other machine.

  • However for this summer project this is too ambitious and using the single thread is better, so all it would do is wait for page faults by guest and send data as per that and there is no destination that will request pages

  • What we are supposed to do is trivial in front of what is already here, surprising it has not been done before as postcopy was implemented 5 years ago(at least the line I am at now)

  • This comment seems outdated

    /* Request one page from the source VM at the given start address.
     *   rb: the RAMBlock to request the page in
     *   Start: Address offset within the RB
     *   Len: Length in bytes required - must be a multiple of pagesize
     */
    int migrate_send_rp_message_req_pages(MigrationIncomingState *mis,
                                          RAMBlock *rb, ram_addr_t start)
    

  • It looks like RAMBlock is representation of a chunk of memory(a page?) and start is just at what location in block is the data requested, but len is manually set to be one page

  • Just got to know India nearly won the T20 world cup, no way New Zealand can make 150 in 50 🥳

  • Considering my lack of expereince in networking and concurrent programming(I think networking is next sem but concorrency in OS might be this sem(Maybe!)), so not really understanding what locking mutex and all is, but assuming it is just a declaration to all threads not to change some variables while I read/write it I get some idea

  • Just going around, all this code is written by people with redhat email ids, looks like RHLE runs these parts of the forest

  • Stuff like channels and recovery is seemingly deep and not required yet, maybe skip it till necessary

  • Finally a real proper function postcopy_start:

    • Mostly not anything I see of grave importance expept this migrate_postcopy_ram function
    • Ohh but this again is just a boolean function, that is sad
    • Important terminology: sender is loadvm and destination is savevm?

  • I am done with this file, not any very useful thing I found here

• I am really surprised by the number of error handling and small pieces of code you never think of while writing, without all this I could have just(of course some simple definition functions will be rquired but the gist):

  // signal of userfault handler
  int x = GET_PAGE_NEEDED(sig_info_t);
  char* buffer[PGSIZE];
  read(save_file, x, PGSIZE, buffer);
  upload_to_vm(GET_LOCATION(sig_info_t), buffer, PGSIZE);
  

• Most other changes will be adding more features and modularizing, honestly I dont know why exaclty reading all this was required for doing this. This is a simple concept at heart

• Maybe error handling and edge cases will be handled properly if I properly read the original code but I seem to get the gist

• Did not read postcopy-ram.c, that will be priority for next day and also will actually just for userfaultfd as it was not even used in migration.c

Day 3

9 March 2026 (~45 mins)

Today's Target: First step will be reading the code actually using userfaultfd, then if time remains we shall see what happens

• Let's begin from postcopy-ram.c:
  • A thing that caught my attention was the headerfile included in postcopy-ram.c is the #include "qemu/userfaultfd.h"
    • Looking into the file it declares the basic userfault functions I think that will be used, but they are not defined, this is some problem
    • Some probing around using find I found a file userfaultfd.c exists in utils, it is that vscode intellisense is not intelligent enough to sense it 😏
    • We will look into these later on, first lets go for the postcopy-ram.c
  • I wonder what this Discard state does, it has a struct for it
  • Really good explanation on why at a point we can have multiple faults in or list
  • A function to make sure ufd works/has all required features, stuff is getting really very very modular here, so many functions just to make sure al required features are given by the UFFD_API
  • Nearly every use of uffd, is for an error handle, looks like I will have to add much more time in phase 2(polishing error handling and features) of project from phase 1(raw proof of concept)
  • Finally the function we have been waiting for postcopy_ram_fault_thread
  • All this will take too much time to actually understand as most of it is handling edge cases and stuff, getting to the core of logic is hard
  • I think I sould just look at the userfaultfd.c now as those functions are ones I will be using
  • I need to read a lot and lot of functions from migration.c infrastructure to actually understand and reconstruct the snapshot loading mechanism

Day 4

10 March 2026(~3 hrs)

Today's Target: Analyse the qemu thread API and look into use of concurrency

• As the last mail of mentor asks to explore this problem, today I will be focusing on getting into the depth of this problem:

  We need to do both eager and lazy loading
  eager: keep loading pages no matter if guest needs it
  lazy: upload only when guest is blocked and needs it
  We have two options: rely on one thread do both works, or having two
  threads doing different part.
  

• Mostly I will trying to explore multithreaded implementations of qemu and look into the positive and negative points of both

• Looking around postcopy-ram.c I found just the thing postcopy_thread_create, which uses qemu_thread_create to create a thread

• I love it when these large c codebases have functions for nearly everything, love the modularity

• We need a list of remaining pages to share, which I think can be done using a linked list:

  • The linked list will have this sort of nodes

    struct Node {
      int start;
      int end;
      struct Node* next;
    }
    

  • By default the linked list will have start = 0 and end = MAX_RAM

  • Eager uploader will pop the head of list, see the start value, increment it by one and push it back in and upload the page at index start

  • Now as we have lazy uploads/on demand upload, it can ask for any page at any given point of time, so in that case it will find which Node range it lies in

  • If a Node is found we split it and upload the page in between which was required

  • This goes on till the list is not empty

• Note for one thread the implementation will be simple:

  • When we have an interrupt/signal that the guest caused page fault we handle it by uploading the page required by finding it in linked list, if not found we are uploading it no matter what
  • Another I think better implementation might be achieved using a circular linked list, where we update only the head at signal(and maybe split the node) and not actually upload, then as the thread continues it will pop it and load in the required page
  • In the circular list case the head need not be reset so we continue popping from the point the demand came from

• Now if we consider this same thing using two threads, the complexity will rise:

  • We cant directly do, upload eager and on demand indepently as then our eager might over write pages already loaded in by on demand thread
  • So now the linked list must be updated by the on demand thread by locking it.
  • Then a problem can arise that if we have already popped the page demanded, then the on demand thread might load it in, guest may alter it and we may then overwrite it.
  • So in a naive implementation we might have to check the location we are uploading to before actually loading it, but such check might be non trivial(maybe the flags can be checked will have to look in, but even if possible will be inefficient as we might have to block guest to check its memory)
  • Having the on demand thread just updating the queue by locking the eager thread makes no sense, so the 2 thread model needs changes or it is vastly inferior to the single thread one

• So I think 2 threads will overly complicate a task that can be much more elegently solved using a single thread

• Also another thought came to my mind was if we use multiple threads to load memory it could be fast but thinking more it seems it wont as the process will be disk bound, the one thread will spend most of time waiting for disk to load memory in RAM, many threads will just increase traffic as disk output is bounded

• Let me once read on how snapshot is implemented right now, and WOW first time using find gave a very useful output, I always confuse on how to use find 😢, but let's read through include/migration/snapshot.h

  • Straight in we get save_snapshot funciton defined in savevm.c, and right next is the one I was looking for load_snapshot, defined in savevm.c.
  • Interestingly BlockDriverState is a very important object and I think is a central struct
  • Wait wait why am I focuesed on load, that I will do later, my idea was to see how save works and invert it
  • So after many many lines of checking states and handling errors/exceptions, finally the save vm, opens a file and writes in using qemu_savevm_state, which is just using migrate
  • All this elegent code, makes me question my abilities to write code that does not look hideous in this perfect haeven

• So any way the thing we take off is that using many threads would be bad, hey but wait, the load_snapshot uses migration too, so cant we just alter the options and all to use postcopy migration from file to machine?

• Problem is I only documentation and the code is very thick, some of these parts severely lack documentation, every function is just written with a decent name but nothing beats 4 lines explaining what it does

• There even is a comment on line 3082(savevm.c) that says Postcopy threads might be running, hence it is entirely possible that Postcopy Migration is used

• So the project is just implementing mapped ram feature here? This is getting weird and kind of awkward, so usually it does postcopy but without mapped ram feature? Does this mean it is a lazy policy? I might need to get this theory confirmed from mentor

• Oh the problem is solved, it was just waiting for the previous postcopy threads if existing to die and then do it entirely post copy, this is the function we might target

• So a point posed by AI against my thories really stand out, modern SSDs are highly parallel, I bought it first, but getting deeper I think it is hallucinating, this led me to a much deeper rabbit hole about SSDs and HDDs operation:

  • Hard disk drives have one reader and so can serve only one service at a time at very slow rate
  • The rate is so slow that getting a new request is insignificant part of load for small requests
  • On the other hand Soft state disks are much faster and thus this process of translating the request to disk is considered significant, from discussions here
  • Another conversation here again reinforces on the fact that SSDs are parallel in form that they serve multiple requests simultaniously but that queueing the requests gives less downtime to the SSD
  • This one clarifies it further and points to some wikipedia pages, which go deeper into NVMe, the new SSD standard
  • So the point is that modern SSDs(NVMe) have a lot more parallelism as they can support a lot of deep queues, and support higher number of data streams having more threads instantly makes much more sense
  • The original idea of using one thread assumed a premise that disk is sequential(which most are) but modern ones are not and in order to prevent the guest from freezing for long time waiting for page fault handling using 2 threads is more sensible
  • Using one thread would cause the guest to freeze, as if the guest needs a page and we have sent a request for uploading large amount of ram pages we must wait for it to end and so must guest
  • If we have a second thread it would just fetch it in parallel and get it loaded keeping everything smooth

• So we change the decision to working with 2 threads, and now we discuss about the linked list, again with some deeper reserach a point raised is that a bitmap would be better so I went to analyse that:

  • First I got to know major OS/systems books use bitmaps and linked lists to manage memory extensively, maybe it was my bad luck most of what I saw used linked lists 😅
  • Just raw analysis gives simple comparison that
    • Bitmaps are much more predictable and fast but less scalable as their size scales according to memory to manage and for a 4GB RAM with 4KB pages, we would have a 1MB bitmap, which is significant and might be larger but still easily managable
    • On other hand a linked list is unpredictable(you might find what you are looking for in O(1) or O(n)(please excuse the abuse of notation)), and can be quite slow in search owing to pointer chasing. However they allow flexibility and have insignificant memory footprint compared to bitmaps
  • This is a good source too
  • This brought back an idea I had at the back of my head but thought will look into when implementing but comes up here is, Just use what QEMU uses, which honestly would be the absolute best. Let me quicly check it out on savevm.c
  • The system goes very deep in load and I got to see that there is a point before actually loading that returns back the load command if migration was in postcopy state, hmm maybe I could piggy back this infrastructure, will have to see
  • Considering that I wanted to see if bitmap or linked list was used for memory management I got nothing here as precopy needs a stream of data, I need to look into postcopy-ram.c
  • This looks like a bitmap ramblock_recv_bitmap_test_byte_offset, so well a bitmap it is

• Now I have completely unraveled my thread and back from scratch use 2 threads and a bitmap

  • First I think the bitmap might need to be locked as the two threads access it or we can use atomic primitives(from standard libraries or qemu might have home brewed version) which will make it 100x easier 😁(finally reading about AtomicAdd in PMPP helped)
  • So the bitmap will be build with all the entries that need to be loaded marked 1 and others 0, signifying that these need to loaded in
  • Thread one has a counter that begins from start and iterates over loading in every page it sees and marking zero instantly(I think there is an atomic primitive for that otherwise what could happen is between a read 1 and write 0 the 2nd thread might do its nasty stuff)
  • Now simmilarily thread 2 will mostly sleep except when signalled that guest has page fault, go on and see if the place in disk needs to be uploaded or thread1 is taking care of it, ohh but what now
  • If thread 1 is not taking care we do it and life is good, but what if this does not happen?
  • If thread 1 is handling that page we can't upload it ourselves, if we do and guest writes it, thread 1 might overwrite it later, so we might go to sleep blocking guest for some time but what exact mechanism to employ needs to be considered(I have no idea yet)

• So we use a 2 thread system(modern systems can handle it, I believe in them) with bitmaps to maintain memory

Days in between

I spent a few days in between to write a formal proposal and sent it to mentor for review. Sorry I did not write anything in the log but my proposal more or less has it so I will just put that here

Abstract

The present implementation of snapshot load, loads all the guest data on QEMU instance before beginning the process. This is not necessary and can be improved on using the ideas from postcopy and mapped-ram migration. Loading the device data and letting the guest run instantly is a feasible option, given we keep supplying it with the pages it needs. This will be achieved using userfaultfd in Linux to detect page fault by the guest, which tells us what page to supply. Coupling it with another eager loading thread this project aims to substantially speed up the snapshot loading process in QEMU.

About me

I am a sophomore in the Computer Science and Engineering Department at IIT Bombay (GPA 9.47). This semester, I took the core Operating Systems course and found the concepts of emulation and virtualization quite interesting. I am interested in systems and low-level optimization, and find fun in navigating through massive C/C++ codebases. One of my recent experiences was with llama.cpp, a high-performance LLM inference engine, where I first got introduced to open source. I worked on optimizing CUDA kernels for SSM scan and cum sum, successfully getting a 2-3x speedup in each(If you want to read more about my wrestle with it do look up my blog where I posted my logs) On the crossection of my newfound urge to contribute to open source and curiosity to learn more about virtualization, exists this project. I would love the opportunity to learn more about virtualization hands on and work with the great community of QEMU, from where I can learn a lot as a student. GitHub github.com/Aadeshveer Blog aadeshveer.bearblog.dev

The problem

Current implementation of snapshot load, uses the simple method of loading in all VM data into the QEMU instance for it to start. Though it is usually acceptable considering the case when the user wants to quickly open multiple snapshots it can be quite slow. This load is not necessary and by loading in only the required data at t = 0 and loading in rest of data later we can improve on speed of loading. This essentially will require use of userfaultfd provided by linux to get to know when the guest hits in a page fault and needs some data with high priority.

Existing Infrastructure

Current infrastructure has post-copy and pre-copy migration implemented(along with mapped ram feature). The logic used in this process will be very useful for correctly implementing what can be said as the inverse of migration. Most of the implementation regarding mapped ram and postcopy exist in the migration directory of source code. There already exist wrappers on userfaultfd in the file include/qemu/userfaultfd.h which can be used readily. Initial loading verification is already done in the qemu_loadvm_state_main (in migration/savevm.c), there needs to be a code divergence from here if fast snapshot load is enabled. Atomic wrappers like qatomic_and exist in include/qemu/atomic.h to implement bit maps

Proposed solution

Basic Idea

The basic idea to solve the problem which has partly been directed to in the project idea is the following:

• Load the necessary device data on the QEMU instance to get the guest running the instance it is started
• As the guest runs and requires the memory stored in snapshot it will go into a pagefault, which we will handle using userfaultfd
• We load in the page required by the guest with priority so that the guest can be unblocked and reissue the instruction causing the fault

Eagerness and multithreading

This needs to be conducted eagerly, loading pages even when the guest doesn’t need them, to be finished with the load procedure as soon as possible. The loading will have to be done by two threads, one that will eagerly keep loading in pages and other loads in the on demand pages. This architectural decision is based on the following points:

• Using One thread though simpler can be much slower as in the case where, The eager loading thread is blocked by the drive read, the on demand load will have to wait for these pages to be loaded. So the guest will also be blocked for that time period. With the second thread we can instantly serve the fault not waiting for any other load in process.
• This choice might not make any big difference on older hard drive based systems as they are able to process only one request at a time so, The second thread will have to wait for first thread’s request to finish. Considering the speed of HDDs, the time boost we might get from queueing the request early will be negligible. On the other hand this might make huge difference on modern SSD systems as the process of getting a request on drive is significant owing to the speed of serving requests. Modern standards like NVMe also allow for much more parallel, deeper queues which can serve the request by thread 2 in parallel with thread 1

Bitmaps vs queues

Another design choice I spent time on was to use bitmap or linked list for managing memory.

• Linked list thought suffer from pointer chasing and O(n) search time are much more effecient memory wise(implemented using ranges)
• Bitmaps on the other hand are great for their O(1) lookups but need a lot more memory, considering that a 4GiB RAM with 4KiB pages needs 4×1024^3 / 4×1024 = 1024^2 bytes = 1 MiB, it is easily manageable on modern systems. Also bitmaps seem to be implemented in existing source code and so using bitmaps here would keep the consistency.

How many bitmaps

On further exploration about how to keep thread 2 waiting till thread 1 uploads the pages it claims, it turns out using one bitmap is not enough. We need three states of a page, WAITING, UPLOADING, and UPLOADED. So we need atleast 2 bits for each datapoint and so 2 bitmaps. One bitmap will be 1 if the page needs to be loaded and the second will have 1 for pages that have been loaded.

Pseudocode solution

Set up the userfaultfd fill in bitmap_loaded for data with all zeros fill in bitmap_waiting for data with all ones load in the device data of the guest make the thread for on demand loading let the guest run

thread 1: ctr = 0 while (ctr < maxRAM) { if (atomic read and clear at ctr on bitmap_waiting) { // we need to load this page load the page(ctr); add 1 at ctr on bitmap_loaded } else { // page was already loaded continue; } } cleanup kill thread 2 exit()

thread 2: while (1) { fault_info = read(uffd); if (atomic read and clear at fault_info.address) { // we need to load this page load the page(fault_info.address) add 1 at fault_info.address on bitmap_loaded } else { // page is being loaded by thread 1 while (bitmap_loaded 0 at fault_info.address) yield(); } wake up the guest }

Projected timeline

Considering that my university exams will be conducted by end of April, I will begin actively working on the project right in May. As the next semester will start in mid July, I plan to be done with core programming and debugging by end of July. Thus, I might start working earlier in May on the coding tasks and keep a buffer over the timeline. Time commitment: 40 hours per week for first 6 weeks followed by 20 hours per week in last 6 weeks.

Community engagement(3 weeks)

I will try to participate in the mailing list, and contribute a few patches regarding documentation. During this period I plan on studying the internals of postcopy-ram.c and finalize the design with mentor before coding begins.

Core programming(6 weeks)

This period will be marked by major code implementation and raw coding, architecting the bare bones of the project

Week 1 (25th May - 31st May)

Begin with changes in migration/savevm.c to direct the code when fast snapshot load is activated. Mostly skeletal changes with declaration of functions, followed by implementation of few.

Week 2 (1st June - 7th June)

Continue with the simplistic implementation to get a running prototype for a very simple case. Error handling and other features can be added iteratively.

Week 3 (8th June - 14th June)

Get a toy VM actually loaded in controlled environment, for proof of concept. Get started with error handling

Week 4 (15th June - 21st June)

Continue adding error handling code for various unexpected run time exceptions

Week 5 (22nd June - 28th June)

Work on adding support for other features like huge pages etc.

Week 6 (29th June - 5th July)

Get the loading working on most of the cases

Mid term evaluation

Midterm deliverable will be a semi functional fast snapshot load

Testing, benchmarking and documentation

This period involves, building on the surrounding necessities of a good project and confirms it’s readiness to be used by masses

Week 7 (6th July - 12th July)

Week for polishing the code and begin testing.

Week 8 (13th July - 19th July)

Continue with extensive testing and patching any exceptions left along the coding process. Testing will revolve around disk faults, corrupted data and multi signal handling

Week 9 (20th July - 26th July)

Begin with benchmarking the new implementation over the old one to test for substantiality of improvements.

Week 10 (27th July - 2nd August)

Write up proper documentation for official docs, and try to add more in code comments for functions

Week 11 (3rd August - 9th August)

Continue with adding documentation and engaging with community for more reviews

Week 12 (10th August - 16th August)

Buffer week

Day 5

15 March 2026(~1hr)

Today's Target: Work on the reply/feedback by mentor to improve the proposal

• Most of the proposal was good with a few hiccups, let's address them one by one

• First This project is not aimed to "speedup" the process, it might even be slower than insync loading but the ergonomics of instant GUI are good

• Updated to also attatch a link to the PRs in llama

• More than two threads can be used to load in data using multifd feature of qemu so that is something to explore in bonus

• There was a minor error iin calculation of the bitmap size required as each bit is 1 bit and not a byte. That was dumb 😅

• Now a point to think about is, are two bitmaps required? and I believe the answer lies in undrestanding userfaultfd and so what is better than the man pages

  • we will also need to read into ioctl as that is necessary for userfaultfd. I have enought idea about syscall from OS lab
  • So from what I understand reading some of userfaultfd and ioctl_userfaultfd(2) we need to signal a paused faulting thread to continue explicity
  • As thread 2 has the access to userfaultfd it is the one that will wake up guest thread
  • Hence if we use one bitmap there must be a contact between them using either a signal or a pipe
  • Also then there can be an active connection or a lazy connection
  • Possibilities for connection:
    • Active: Whenever thread 1 completes loading all the pages it had picked up it communicates to thread 2 that I am done with all pending uploads and you can go ahead ask guest to continue if it is waiting
    • Lazy: Whenever thread 2 knows thread 1 is loading a page guest is waiting on, it communicates to thread 1 about it and asks for reply when it is done. Now thread 1 checks on completion of current load checks if it is in queue to upload, if yes wait else communicate to 2 to go ahead
    • Signal: Not the most ideal thing to do, We will have to define custom signals and all. Too much for too little.
    • Pipe: Looks like a good point, meant for communication
  • Wait, we are communicating within threads of a process not processes! we can have a bit set when thread 2 begins to wait and it keeps polling it. Whenever thread 1 is done uploading the pages it commited to it resets the bit no matter what and goes on with its business. Thread 2 that was polling it sees the change and wakes the guest back up.
  • Though I might research for better methods this global bit idea looks feasible and simple for a single bitmap
  • Interestingly, we can directly upload from thread 1 using UFFD_COPY, so that it wakes up guest thread on copying data.
  • Picture looks good, but what if I load in a few pages, say 8 at once then each will cause guest to wake only to be hit with a page fault again till the correct page in 8 is not found
  • Finally another point is that I am a bit skecptical of allowing both threads the uffd abilities as it is better to keep stuff contained
  • Not sure again but if we use multifd, then this stuff goes foul again as I dont think multifd infrastructure would be happy working with UFFD

• Now about merging the error handling week into other ones, we'll try on that

• Also will add the digesting mapped-ram and postcopy blind spots to may period for playing with it

• Also have to add unit tests for migration-test.c

• To think on:

  One idea about split your real coding part is we can have some time
  refactoring postcopy to suite snapshot load's need.  It means you can
  have one patch prepare postcopy functions to be available for snapshot
  loads.  That should bring no functional change but paving way.
  

Day 6

16 March 2026(~30mins)

Today's Target: Work on getting a first patch in qemu mailing list

• I will look on into the outdated comment

• I think it is a trivial change and too small, lets keep it as a fallback and try looking for something genuine

• I saw this in migrate_send_rp_recv_bitmap, maybe I can try working on it

  /*
  * Next, we dump the received bitmap to the stream.
  *
  * TODO: currently we are safe since we are the only one that is
  * using the to_src_file handle (fault thread is still paused),
  * and it's ok even not taking the mutex. However the best way is
  * to take the lock before sending the message header, and release
  * the lock after sending the bitmap.
  */
  

• So the target is to lock a QEMUFile object mis->to_src_file Let us look at postcopy_incoming_setup now:

  • It starts with checking if postcopy ram feature is enabled, if yes calls the postcopy_ram_incoming_setup which we just understood
  • THen we create a listening thread, let's see what function it performs(postcopy_listen_thread):
    • It's purpose is to load in ram when main thread loads in data
    • sets the migration status from ACTIVE to DEVICE depending on the to_src_file but as a listner on recieving end why would there be a to_src_file?
    • Then it blocks on the from file and loads in entire stream, now this process I need to read about mapped ram feature to see its exact usability

• Interestingly it has no file descriptor inside the definition of struct

• Maybe the fd exists in the QIOChannel

• After going through files and files I think I have the point to lock mis->to_src_file->ioc->read_ctx->lock

• I think I need to look for its wrappers instead of straight away locking it or maybe add some wrappers

• I am quite sure channel.h has no locking on aio object and so there lacks proper locking for files

• Finally after 45 mins😩 of setting up clangd for intellisense, intellisense runs and found a bug straigt away(clangd shouted at me to fix it)

  • migration_update_counters function in migration.c this line exists:

    expected_bw_per_ms = switchover_bw / 1000;
    

  • Here switchover_bw is of uint64_t type but expected_bw_per_ms is of double type

  • Hence in C RHS type is uint64_t that then casts to a float

  • Somehow checking for usual values of swithcover_bw I found this interesting paper

  • I dont think this has been implemented into QEMU yet, seems interesting will read properly later on

  • I think usually switchover_bw is mch higher than 1000 and so the difference might be negligible that no one has noted

  • Still a bug at least

  • So it took some time to set up but I have finally submitted the little patch of adding explicit type casting

  • Now just need to update the proposal and we are done Update: The patch was review and accepted 🥳

Day 7

11 May 2026(~8 hrs)

Today's Target: Get qemu working and play with migration

• Rebuilding qemu from scratch, it was easy just use the README but instead of make used ninja cause I think it is faster(found this)
• Took around 5 mins on 22 processers 🙃, but now it is ready so lets test it on some VMs from the internet
• I do not have sphynx and so it does not compile documentation, I will have to look into that
• Running tests:

  • Using the documentation I did this simple test(the documentation seems to be old as it mentioned files not generated)

    # documentation
    # bin/debug/native/x86_64-softmmu/qemu-system-x86_64 -L pc-bios
    bin/build/native/qemu-system-x86_64 -L pc-bios
    

  • BIOS runs but could not read boot disk, all good

  • The KVM test is different and uses different file system from BIOS test, I believe they should all be unified, so one can test all in a go

  • KVM test requires a test.qcow2 file which I cannot find

  • Oh nevermind, qemu-img makes the file itself

  • Now I am downloading some ISOs to test, one fedora kde 44 for modern system and a windows XP old one

  • Finally after an hour of tweaking around with arguments windows xp runs, fedora kde also works but is much slower

  • Giving 16 GB ram instead of 4 seems to make fedora run smoother, though with kvm enabled it should have been smoother

  • I think it was just installation that took time, now same drive on 4GB RAM works good

  • I am installing a small debian too cause I think it will be faster to test, boot etc

  • Debain tty works too, now lets get in the source code

• Now lets try out migration starting from the documentation

  • First thing pointed to is include/migration/vmstate.h, which has lot of macros for using migration

  • Reading through the basic data structures I found this:

    struct VMStateField {
    const char *name;
    size_t offset;
    
    /*
     * @size or @size_offset specifies the size of the element embeded in
     * the field.  Only one of them should be present never both.  When
     * @size_offset is used together with VMS_VBUFFER, it means the size is
     * dynamic calculated instead of a constant.
     *
     * When the field is an array of any type, this stores the size of one
     * element of the array.
     *
     * NOTE: even if VMS_POINTER or VMS_ARRAY_OF_POINTER may be specified,
     * this parameter always reflects the real size of the objects that a
     * pointer point to.
     */
    size_t size;
    size_t size_offset;
    ...
    

    Here if only one of size and size_offset is used I think they can be wrapped in a union, lets test that

    • I think this is a valid change to

      union {
        size_t size;
        size_t size_offset;
      };
      

    • make check shows no error

      Ok:                897 
      Fail:              0   
      Skipped:           121 
      

    • I think I should verify it with more tests as if alignment is important here this might be breaking change in some cases

  • Documentation at https://www.qemu.org/docs/master/devel/migration/main.html refers to an example in hw/input/pckbd.c which I checked is out of date and should be updated.

  • I read most of migration framework in documentation and it mostly states what some structs are and what they do, but not how to use them, maybe I can get more in reading about the features

  • I read this blog too, thought it would explain the API but was more like an overview, but was quite good.

  • Let's read about snapshots, maybe that would teach how to use the API

  • Ok so using qemu monitor I am able to save and load VM

  • I need to find how to save it as a file so that I can load it later

  • After quite some time, trying stuff here and there I found about migrate command on qemu monitor and its use given here

  • Now I am able to save snapshots as files by doing migrate "exec:cat > statefile.img" and load using <qemu command> -incoming "exec:cat statefile.img"

  • I do not know how to migrate via networks as I have no experience of networking course, the file system is simple but I cant find how to migrate via file descriptors

  • Oh it is simple to use files, just use "file:<file name>" as the uri and we are done 😀

• I think I should set up gdb to trace the movement of code now
  • As always the set up is all weird.
  • It is throwing error in an assembly file with a huge name and I have no idea what it is for, I think KVM is the problem
  • Without KVM it seems to run but I can't migrate as my image is made with KVM enabled, so lets do it all again with KVM disabled this time
  • Finally after reinstalling debian and some more changes to launch.json, gdb works and I can use break points
  • Best thing here would be to look into normal snapshot loading, then maybe we can try to migrate between two qemu instances
• Our plan was to divert form qemu_loadvm_state_main so lets apply a break point in it and analyse

  • Finally I can see the entire stack and see where all the holes are filled

  • First lets see MigrationIncomingState struct passed to the function.

  • Most looks good, lets see the exact file from_src_file which is a QEMUFile object.

  • It has a channel to read from the ioc but declaration of its type is weird

    OBJECT_DECLARE_TYPE(QIOChannel, QIOChannelClass,
                        QIO_CHANNEL)
    

  • The #define are too deep and convoluted, I think it is better to start from who fills in the ioc

  • Let's keep going up the stack till we reach the point where mis is filled in

  • Looks like this function does stuff: qemu_loadvm_thread_pool_create() after migration_incoming_get_current() used for initialization

  • There is a global static MigrationIncomingState *current_incoming that stores the state, it should be the global state

  • migration_object_init initializes this global state using g_new0(assume that it is a safer version of malloc)

  • Status is intialized to MIGRATION_STATUS_NONE as expected, and it has postcopy_remote_fds as a Garray(C version of a std::vector) all set to zeros

  • All the semaphores and mutex are initialized, and page_requested seems to be a Gtree(C version of std::map) that takes an arbitrary ordering on value of pointers, so it can do updates etc in $O(\log n)$

  • That's it for initialization

  • qemu_loadvm_thread_pool_create only gets some new threads for migration, so the global state is being updated elsewhere

  • This coroutine function seems to be the one that sets stuff up process_incoming_migration_co as it is called by coroutine_trampoline

  • process_incoming_migration_co sets the state from MIGRATION_STATUS_SETUP to MIGRATION_STATUS_ACTIVE, so there must be something else that changes the state from MIGRATION_STATUS_NONE

  • Using regex search (migrate_set_state\(.*MIGRATION_STATUS_NONE.*\)) I was able to find that only function migrate_init changes the state from MIGRATION_STATUS_NONE to anything else.

  • Had a digression and set up the IRC weechat

  • continueing back, migrate_init is the one that starts everything, lets see who calls it:

    • migrate_prepare
    • qemu_savevm_state

  • We can be sure only migrate_prepare is called for loading, so we try to see who calls i, qmp_migrate only

  • Doubt is what is qmp and hmp, need to find that as hmp_migrate calls qmp_migrate with NULL as uri

  • But then how is the uri used?

  • Let's try reconstructing the stack

  • What is this hmp and qmp? hmp_migrate_status_cb function is never declared in any header file?

  • Oh they are monitor protocols, so I think they are called when an input is given to the qemu monitor

  • Let's try that on hmp_savevm using gdb, ... and it stops in it, nice!

  • qdict_get_try_str(qdict, "name") holds the name I entered

• Lets start from beginning and follow the code, we put a breakpoint at migrate_init but it never arrived, so lets have two breakpoints at migrate_get_current and migrate_incoming_get_current:
  • Ugh, vscode moves the breakpoints here and there so I think we need to enable debug(release compilations have that problem: source)
  • A long compilation later ...
  • We stop at migration_incoming_get_current first and it is called by migrate_caps_check to check compatibility, we will have to add the compatibility checks later on.
  • it was called by migration_object_check which was called by migration_object_init to initialize and check the initialization.
  • This is a bit weird as migration_object_check takes in current_migration but it does not pass it on to migrate_caps_check which has to use the function? Oh no, passed on is MigrationState and called function is MigrationIncomingState to verify stuff on both
  • Looks good, another point that came to my mind was, it would be better if current_migration is never used without the wrapper and I think that should be possible as the function migrate_get_current should serve everywhere well. It would simplify code and make it easy to follow as everytime we use the global variable we are forced to call the function making the tracking easy(though the overhead of function calling may increase). But, on the other hand calling function requires current_migration to be created. So some changes are valid:

    • In this function if object is not created, it will throw error, so we can use function

      bool migration_thread_is_self(void)
      {
          MigrationState *s = current_migration;
      
          return qemu_thread_is_self(&s->thread);
      }
      

    • This function too will have same error:

      void migration_file_set_error(int ret, Error *err)
      {
          MigrationState *s = current_migration;
      
          WITH_QEMU_LOCK_GUARD(&s->qemu_file_lock) {
              if (s->to_dst_file) {
                  qemu_file_set_error_obj(s->to_dst_file, ret, err);
              } else if (err) {
                  error_report_err(err);
              }
          }
      }
      

    • Same here:

      static bool migration_is_active(void)
      {
          MigrationState *s = current_migration;
      
          return (s->state == MIGRATION_STATUS_ACTIVE ||
                  s->state == MIGRATION_STATUS_POSTCOPY_DEVICE ||
                  s->state == MIGRATION_STATUS_POSTCOPY_ACTIVE);
      }
      

    • As migration_object_init runs once, it is ok to use current_migration as it will be NULL originally

    • Similarily for the migration_shutdown being called once can be exempted

    • migration_is_running can be called anywhere and needs to handle the inexistant object case, so needs to exempted too

  • current_incoming is good as it is always used wrapped in function or in initializer
  • These changes wont affect the flow so we can continue to track the changes to state
  • Going on, migration_object_init is called in qemu_init function and is ran always no matter if we use migration or not
  • On continue we reach break point to accesss current_incoming, and this is called by qmp_migrate_incoming, lets see what all it has:

    • uri given to qmp_migrate_incoming by qmp_x_exit_preconfig is NULL which is surprising given we have a uri as input "file:statefile.img" in debugger

    • Use of once here is ambiguous, is it used as a lock? Need to analyse this

      void qmp_migrate_incoming(const char *uri, bool has_channels,
                                MigrationChannelList *channels,
                                bool has_exit_on_error, bool exit_on_error,
                                Error **errp)
      {
          Error *local_err = NULL;
          static bool once = true;
          MigrationIncomingState *mis = migration_incoming_get_current();
      
          if (!once) {
              error_setg(errp, "The incoming migration has already been started");
              return;
          }
          if (!runstate_check(RUN_STATE_INMIGRATE)) {
              error_setg(errp, "'-incoming' was not specified on the command line");
              return;
          }
      
          if (!yank_register_instance(MIGRATION_YANK_INSTANCE, errp)) {
              return;
          }
      
          mis->exit_on_error =
              has_exit_on_error ? exit_on_error : INMIGRATE_DEFAULT_EXIT_ON_ERROR;
      
          qemu_setup_incoming_migration(uri, has_channels, channels, &local_err);
      
          if (local_err) {
              yank_unregister_instance(MIGRATION_YANK_INSTANCE);
              error_propagate(errp, local_err);
              return;
          }
      
          /*
          * Making sure MigrationState is available until incoming migration
          * completes.
          *
          * NOTE: QEMU _might_ leak this refcount in some failure paths, but
          * that's OK.  This is the minimum change we need to at least making
          * sure success case is clean on the refcount.  We can try harder to
          * make it accurate for any kind of failures, but it might be an
          * overkill and doesn't bring us much benefit.
          */
          migrate_incoming_ref_outgoing_state();
          once = false;
      }
      

    • All it does is set mis->exit_on_error which is true, where might it be false? Maybe on network where migration does repair too

    • So as it looks like parsing of input is already done so no uri is required and hence we pass on the exact channels list which here will of size 1 for the file

  • Next break is in qemu_setup_incoming_migration, which then tries to set the incoming state to MIGRATION_STATUS_SETUP
  • Worth noting is if the incoming migration status is postcopy paused, then it remains in paused state
  • Another coninue takes some time and is in qemu_main_loop, called by migration_channel_process_incoming which prepares the channel ioc to connect to followed by a check if multifd is possible and if yes life is simple but on no, life is complex
  • after we have the channel type, we setup the incoming file on the MigrationIncomingState and start the input
  • Then the start of input makes a Coroutine which I an not at all well versed in but need see what it is
  • My mind is getting weird😩 maybe enough for today

Day 8

12 May 2026(~6 hrs)

Today's Target: Get used to qemu tracepoints and set up patches

• Today's my birthday 🥳🥳

• First thing I was thinking was to set up the entire testing system for postcopy etc

• Using the command told by mentor in yesterday's meeting:

  echo 1 > /proc/sys/vm/unprivileged_userfaultfd
  

• It says permission denied, ... even sudo echo has permission denied 😠

• Researching around I found this, which means that the permissions are actually controlled by /dev/userfaultfd?

• Trying

  sudo chmod +666 /dev/userfaultfd
  

• It still does not have permissions

• Maybe I can chmod on /proc/sys/vm/unprivileged_userfaultfd, lets see its permissions ... yup it is 0644, lets do it 0666

• Operation not permitted, I need to go deeper into research about that

• Hmm documentation says that we can set it directly but I can't do so, maybe this is some fedora thing

• Thanks to discussion here, I got to know that redirection takes place in current shell and not with root privelages, so used this instead and it worked:

  echo 1 | sudo tee /proc/sys/vm/unprivileged_userfaultfd
  

• Running the tests now we see 120 SKIP 1 less than last time🥺

• Does look like all the postcopy tests passed

• I am skipping because of:

  • pci bus does not support hotplug
  • incomplete support for MSI
  • no support for user networking
  • need pylint
  • test required too much memory
  • unstable tests
  • no crypto backend
  • asan disabled
  • ECAP.PT not supported
  • tonnes of other unsupported stuff on aes, des etc

• So I think postcopy works.

• Now, lets test only migration code using

  export QTEST_QEMU_BINAYR=./qemu-system-x86_64
  ./tests/qtest/migration-test [--full]
  

• Tried both with and without --full both run with all tests ok 👍

• Now let us try putting up some patches

  • First is the union point.

    • I know it might be unsafe, but after testing it doesn't seem to be affecting anything

    • We need to check it with depth, first I saw this macro defines both when it shouldn't

      #define VMSTATE_VBUFFER_MULTIPLY(_field, _state, _version, _test,    \
                                       _field_size, _multiply) {           \
          .name         = (stringify(_field)),                             \
          .version_id   = (_version),                                      \
          .field_exists = (_test),                                         \
          .size_offset  = vmstate_offset_value(_state, _field_size, uint32_t),\
          .size         = (_multiply),                                      \
          .info         = &vmstate_info_buffer,                            \
          .flags        = VMS_VBUFFER|VMS_POINTER|VMS_MULTIPLY,            \
          .offset       = offsetof(_state, _field),                        \
      }
      

    • So this should change, let's see where all it is used:

    • Only in hw/net/virtio-net.c, what does this MULTIPLY mean?

    • It lies in inializer of vmstate_virtio_net_device, lets see who uses it and how:

    • virtio_net_class_init function uses it to init a device

    • Does look like this thing is out of hand for now, I am not sure about the usage of both size and size_offset, I need to ask someone about it

  • Let's look into the other point now, the wrapping of global variable one:

    • Replace the use of variable in 3 places with function, now testing after rebuild

    • Running individual migration test was kind of pain to read every number OK, so I wrote this script. (Just wish it saves more time than spent)

      # run the test and store output in a temp file
      ./tests/qtest/migration-test --full > temp.txt
      
      # grep all lines with ok and extract test number
      sum=`grep -r "ok" temp.txt | awk -F " " '{sum+=$2} END {print sum}'`
      
      # expected value
      expected=`echo '66*(66+1)/2' | bc`
      
      # verify
      if [ $sum -eq $expected ]; then
          echo "OK: Sum matches expected"
      else
          echo "FAIL: Sum does not match expected"
      fi
      
      # cleanup
      rm temp.txt
      

    • I forgot syntax for awk so used this

    • Added some more lines on top of the script:

      # rebuild to ensure changes are updated
      ninja
      
      # run the test and store output in a temp file
      export QTEST_QEMU_BINARY=./qemu-system-x86_64
      

    • Now as all test pass, let's make a patch

    • After quite some time finally I sent off the patch, I really need to set up git-publish, git send-email is long process

• Now last thing for today let's start understanding the tracepoint:

  • I just ran with trace and got output:

    migrate_set_state new state setup
    migrate_set_state new state active
    migrate_global_state_post_load loaded state: running
    migrate_set_state new state completed
    

  • I changed the string in migration/trace-events for migrate_state_state function, and the change clearly reflects in output

  • Let's read the output first and see what it holds:

    • Nice trace really looks like a really useful thing
    • Read documentation(skimmed over the backends, simple one is enough)

  • Nice tracing migration_file_outgoing I can see it output the file name

  • I thought it did something like if a function is called its trace counterpart if activated is called, but now I get that the trace_* functions are the ones that actually log the output

  • So we can put these functions at places and see the output

  • Let's start with a trace on initializing the migration

    • After long time of figuring rebuilding etc to prevent error, the problem was after my addition of mytest it did not end with a new line🥲

    • and it did not output anything?

    • 🤦 I put my trace in wrong function and so it never ran, now I put it in proper place

    • I tried putting in lot of traces and had tonnes of error, I thought for an hour I was doing something wrong but cland autocomplete had appended some header files on top that broke everything, I spent so much time on that

    • I think I will continue tomorrow, I have some tracing setup

      mytest_object_init      migration object initialized
      mytest_state_change     state changed form none to setup
      mytest_print_something2         MESSAGE: event generated for state  | setup
      mytest_state_change     state changed form setup to active
      mytest_print_something2         MESSAGE: event generated for state  | active
      mytest_state_change     state changed form active to completed
      mytest_print_something2         MESSAGE: event generated for state  | completed
      mytest_print_something  MESSAGE: incoming state destroy
      mytest_print_something  MESSAGE: shutdown started
      

Day 9

13 May 2026(~7hrs)

Today's Target: Continue toying with postcopy and mapped ram and look into postcopy-blocktime

• First I should work on the review recieved on the patch I sent yesterday.
• Done updating the patch, My original plan was right to change the 3 places, just got kind of over ambitious and changed some more places.
• First thing is running migration with postcopy on, so my attempt is to simply save a file.

  • Hmm, it seems I cant save snapshot to the file the current snapshot is loaded from, and due to some reason I can't save it with any file name.

  • Let me retry, the original snapshot is corrupted? Let's make a new one.

  • I should keep a backup, it is very slow to boot up

  • Due to some reason I am continuously getting qemu: Unable to read from file: Bad file descriptor, I cant open files no matter relative or absolute address

  • Am I out of disk space? ... Nope df says I am at 39% usage only

  • The problem is permissions, most of the generated images have 0600 and one that works has 0644

  • yup that is the problem again, I wonder why.

  • So even non post copy migrated files are 0600, and I can load them in even if they do not have user read permissions!

  • Oh nvm, I got confused, the permissions are all good, the 6 is for owner and here I am the owner.

  • But that does not explain why can't qemu write

  • Now I seem to be able to migrate all the files with no error

  • It seems like every time I try to activate postcopy, the file descriptors go bad

  • There is some problem with my postcopy as if I turn it off, no error, migration succeeds

  • The tests pass but if I use the monitor it fails 😭

  • AI recommended the use of strace tool to find which syscall failed so I ran this:

    strace -f -e trace=openat,ioctl,read,write,close -o temp.txt ./qemu-system-x86_64 -m 4G -smp 4 --incoming "file:statefile.img"
    

    and outputs are(I removed read/write to FD 6 as those might be logs or something, just filled the screen with no purpose):

    • Postcopy disabled

      350627 openat(AT_FDCWD, "statefile3.img", O_WRONLY|O_CREAT|O_CLOEXEC, 0600) = 18
      353505 +++ exited with 0 +++
      350627 close(18)                        = 0
      

    and

    • Postcopy enabled

      350627 openat(AT_FDCWD, "statefile3.img", O_WRONLY|O_CREAT|O_CLOEXEC, 0600) = 18
      355095 +++ exited with 0 +++
      350627 close(18 <unfinished ...>
      355096 +++ exited with 0 +++
      350627 <... close resumed>)             = 0
      350627 close(-1)                        = -1 EBADF (Bad file descriptor)
      350627 write(2, "qemu:", 5)             = 5
      350627 write(2, " ", 1)                 = 1
      350627 write(2, "Unable to read from file: Bad fi"..., 45) = 45
      350627 write(2, "\n", 1)                = 1
      350627 write(2, "qemu:", 5)             = 5
      350627 write(2, " ", 1)                 = 1
      350627 write(2, "Unable to read from file: Bad fi"..., 45) = 45
      350627 write(2, "\n", 1)                = 1
      

  • The -1 is suspicious, I went on to research what -1 as a file descriptor would mean but there is no specific meaning. I discussed it with AI and got a variety of answers, one was a double close other was a code error.

  • In any case I think this is unexpected I need to use GDB and tracing to find where is this close used

  • There are so many calls to close I can't find it, I need to use some other method

  • 😲 VS code has a show call heirarchy feature! That could be so useful, but I need find all refences in a file

  • No form of functional breakpoints work, I tried applying it on close(int) and fd == -1, none worked, I need to look into other methods to find this close(-1)

  • Using tracing, qemu_file_fclose is only called while tracing from qemu-file.c

  • I can see both the closing trace printed, lets use GDB here to on qemu_fclose

  • It's been an hour and I cant find where these files are closed, because there is an error it propagates back and forth and it does not seem to be very clear where the close for failure is.

  • I need some rest ...

  • Ok I am back, let's get back to finding the close

  • New strategy lets chase from where the error starts: break point at string: "Unable to read from file", it occurs at two places and placing a breakpoint at both does not stop code execution and printing happens

  • I have no idea how this thing is happening, if it outputs the string it was set somewhere but it wont stop at those places? Maybe try rebuilding(no change)

  • That's it I am stuck at this point, I used gdb to specifically stop at close syscall if $rdi == -1 but still it did not stop but strace is showing it happens?

  • FINALLY I FOUND IT !!, it is in qemu_close() in util/osdep.c I had to use fd == -1 it worked on GDB debug console 🥹

  • stepping out we should be able to see where this file comes from, it was called in qemu_fclose by qio_channel_close

  • This was a temp file !!

  • migration_cleanup is what we need to read now:

    • declare a NULL tmp file, and assign it the s->to_dst_file
    • So some how the MigrationState has file that has fd == -1
    • If only there was some way to get the fd or name of file that QEMUFile holds, I need better confirmations
    • As it seems QEMUFile has fds a circular linked list to hold all the file descriptors, and it is empty in this case, so that means cleanup wont get any FD
    • While clean up the ioc is unyanked?
    • But, fclose closes the channel, so the channel is problemetic
    • I was ignoring this for long time but I think I need to know what OBJECT_DECLARE_SIMPLE_TYPE is because I need to know how to extract fd from QIOChannel
    • I used AI to understand what it does and now that I understand the system I both amazed and furious at how C does things:
      • QIOChannel is smaller than QIOChannelFile and when it is allocated it is made sure that fd lies outside QIOChannel!
      • The macros feel simple but the metaphorical use of C never ceases to amaze

  • Now lets track who made this channel and if the fd was -1 all along:

    • Ok, we cant use GDB call stack as there are mutliple threads and presence of many libglib functions in call stack means use of coroutines, multithreading etc
• Let's leave it for tomorrow, I spent lot of time on it and my brain is mushy...

Day 10

14 May 2026(~6hrs)

Today's Target: Continue the postcopy exploration and look into postcopy-blocktime

• Reading through #qemu IRC, I found there is a scripts/qemu-gdb.py, that might be useful for exploration, will look into that
• There was an interesting conversation between kwolf and dupondje on IRC(though I didn't get technical parts of it), I lost it after I turned off laptop at ~19:30 IST
• Let us work with setup function postcopy_ram_incoming_setup it would setup the exact stuff that we target and might explain why save did not work

  • It begins with opeing the userfaultfd

  • (New VS code feature unlocked pinning a tab 😀)

  • The flags used to open the userfaultfd are O_CLOEXEC and O_NONBLOCK, reading from man (2) open: O_CLOEXEC flags the fd to close on fork and O_NONBLOCK, I don't quite get it, man pages say that it is meant to not block on reads/writes/open etc but it does and in future the sementics might be implemented

  • Reading from here and here, it looks like non block should not have any affect on normal files, however on userfaultfd objects the exact explanation is found in man userfaultfd:

    If the O_NONBLOCK flag is enabled in the associated open file description, the userfaultfd file descriptor can be monitored with  poll(2),  select(2),  and  epoll(7).
    When  events  are  available,  the file descriptor indicates as readable.  If the O_NONBLOCK flag is not enabled, then poll(2) (always) indicates the file as having a
    POLLERR condition, and select(2) indicates the file descriptor as both readable and writable.
    

  • So basic idea: nonblock helps us to wait and read from userfaultfd which is locked by the OS?

  • I found this in uffd_open:

    uffd_dev = open("/dev/userfaultfd", O_RDWR | O_CLOEXEC);
    

    it does use /dev/userfaultfd 🙃

  • Now lets see what this ufd_check_and_apply does:

    • It says it is not possible to request UFFD_API twice per one fd, I wonder why ... The ioctl man pages explain that using O_NONBLOCK is expected with ioctl so that explains the flag
    • It is better to read man (2) ioctl_userfaultfd, ... found it in man UFFDIO_API, as it makes a handshake of API's calling it once completes the handshake, and the features are supported, doing it twice gives EINVAL error.
    • So once it runs the features are filled in, then it tries to activate UFFD_FEATURE_THREAD_ID
    • Now we request_ufd_features on the ufd, which currently is only threads
    • But the point is that recieve_ufd_features opens a brand new file descriptor because UFFD_API is called on main one to request features. So we can call receive any number of times, it might be efficient to call it once but the comments seems misleading
    • The comment can be updated to something clearer, will look into that
    • then if not a hugepage we just return true(huge pages have their own worries)

  • Then we have blocktime, let's look into that later as it is not critical to path

  • then an eventfd is opened that too closes for fork, from man pages: what I understand is it is a signalling mechanism, it holds just a 64 bit uint and write adds a number and read reads it. Nice discussion here

  • Will have to see how it is used exactly

  • postcopy_thread_create would be important too ... not a big thing but this is important postcopy_ram_fault_thread the function it does:

    • Does look good, I think that no major change is required on this on fault, we might even be able to use it as is

    • Here the eventfd is useful, as we poll forever on userfaultfd, then eventfd actually makes it quit

    • Ahh so poll actually takes in a list of file descriptors to poll on(man poll), really nice, one is userfault, second is exit event but what are the extra fds?

    • Just reading through the man pages I think instead of this read on mis->userfault_event_fd it would be better to use eventfd_read, but I need a proper resource to know how they differ.

    • Found one, so it is just a wrapper, might be safer/future proof in case say the eventfd_t size changes(though unlikely), just a point mentioned is it blocks at ctr = 0 but read does that too. Maybe should try for a patch ... Done sent.

      diff --git a/migration/postcopy-ram.c b/migration/postcopy-ram.c
      index f5ef93f193..16113b166d 100644
      --- a/migration/postcopy-ram.c
      +++ b/migration/postcopy-ram.c
      @@ -1330,12 +1330,12 @@ static void *postcopy_ram_fault_thread(void *opaque)
              }
      
              if (pfd[1].revents) {
      -            uint64_t tmp64 = 0;
      +            eventfd_t tmp_event = 0;
      
                  /* Consume the signal */
      -            if (read(mis->userfault_event_fd, &tmp64, 8) != 8) {
      +            if (eventfd_read(mis->userfault_event_fd, &tmp_event)) {
                      /* Nothing obviously nicer than posting this error. */
      -                error_report("%s: read() failed", __func__);
      +                error_report("%s: eventfd_read() failed", __func__);
                  }
      
                  if (qatomic_read(&mis->fault_thread_quit)) {
      @@ -1773,13 +1773,11 @@ void postcopy_temp_page_reset(PostcopyTmpPage *tmp_page)
      
      void postcopy_fault_thread_notify(MigrationIncomingState *mis)
      {
      -    uint64_t tmp64 = 1;
      -
          /*
            * Wakeup the fault_thread.  It's an eventfd that should currently
            * be at 0, we're going to increment it to 1
            */
      -    if (write(mis->userfault_event_fd, &tmp64, 8) != 8) {
      +    if (eventfd_write(mis->userfault_event_fd, 1)) {
              /* Not much we can do here, but may as well report it */
              error_report("%s: incrementing failed: %s", __func__,
                            strerror(errno));
      

    • postcopy_ram_fault_thread is one of the most ready to use functions

  • Now we have a thread created, and we mark that mis has a fault thread, though I think it would be better to place it in function postcopy_thread_create itself as it already takes mis as input? Oh no, it just creates a thread it might create some other threads not realted to fault.

  • Now to understand this foreach_not_ignored_block(ram_block_enable_notify, mis), It takes a function and mis as a void*?

  • Then it applies the function on every ramblock that will not be ignored by migration. So as the iterating mechanism is used elsewhere the RAMBlockIterFunc takes void* as second argument to support other functions too

  • I found this directory of stubs, that is so sad 😔

  • So for each such ram block, a uffdio_register is filled in and ioctl tells userfaultfd to take care

  • Then temporary pages are setup, and it uses mmap never thought would see that(even OS lab never used it), let's undertand it's exact API first

  • It puts up temporary pages in mis, maybe there exact function will come up later, but this line feels off

    memset(mis->postcopy_tmp_zero_page, '\0', mis->largest_page_size);
    

  • Here I think '\0' is meant to set a byte to 0 but using normal zero could have worked to? Reading the commit message, It seems temp pages act like caches but does not explain the '\0'

  • I tried looking into the commit that added it initially but gitlens cant beyond one, but anyway this should be a type cast. Maybe the reason that only 1 byte is used is signified using the char, lets leave it as is

  • Then there is preemption ... Saw documentation for preemption, it is just allowing instant service of page faults, it is like in proposal the recieving thread was uploading the pages on requirement

  • Finally we return, all this should happen in ADVICE state?

• Let us look at postcopy_incoming_setup now:
  • It starts with checking if postcopy ram feature is enabled, if yes calls the postcopy_ram_incoming_setup which we just understood
  • Then we create a listening thread, let's see what function it performs(postcopy_listen_thread):
    • It's purpose is to load in ram when main thread loads in data
    • sets the migration status from ACTIVE to DEVICE depending on the to_src_file but as a listner on recieving end why would there be a to_src_file?
    • Then it blocks on the from file and loads in entire stream, now this process I need to read about mapped ram feature to see its exact usablility

Day 11

15 May 2026(~5hrs)

Today's Target: Look into postcopy block time and try to migrate in a file from file uri and see how postcopy behaves

• First let's read this file directed by mentor about explanation about each migration status
• We should begin from setup as that is likely where we will start blocktime setup
  • First we check if the migration has capability for block time and in that case initialize its context
  • I wonder how is it decided if it is capable of blocktime, lets see that first, it must be decided in where ever mis was initialized
    • Using call stack I can see that loadvm_process_command is the function that does not take mis input and uses migration_incoming_get_current() to get it
    • So it's caller should have filled in mis? Wait I think my logs had something about mis initialization, let me go back in time
    • Here it is migration_object_init, But it does not initialize the capabilities
  • Oh, I was looking into mis but it is actually decided from MigrationState not the incoming one
  • Let's look into it's initializer ... , this is where it should happen migration_object_check() called in initializer(migration_object_init)
  • This looks more important function for this specific job qmp_migrate_set_capabilities, so it does based on input, I think it is most of these macros that do the job, whenever input is given at hmp/qmp it sets the capabilities
  • Let's see what actually this context is:
    • For each cpu it stores faults count, current and total and some hash tables we can see in use
    • I think most of it is simple and can be implemented, we might not even need to do anything(big) as the recieving thread would mostly be same, let's continue on it after we start coding, I think it can be supported with not much difficulty
• Let us try to migrate from a file:

  • Ah I found this option -monitor stdio, this will save me so much time

  • Hmm it runs perfectly

    $ ./qemu-system-x86_64 -m 4G -smp 4 -incoming defer -monitor stdio
    QEMU 11.0.50 monitor - type 'help' for more information
    (qemu) migrate_set_capability postcopy-ram on 
    (qemu) migrate_incoming file:statefile2.img
    

  • I think I should run it through GDB to check out the process taken

  • VScode debugger messes up the paths, I have no idea where the images are stored relative to wherever vscode opens it

  • Qemu doesnt even have a pwd command in monitor, I think it really should have one, might look into that later on

  • For now I think I need to use absolute paths 😢

  • YESSS, I can set up the cwd in launch.json and now I can run and test it

  • Let's start with putting up a breakpoint in postcopy_ram_incoming_setup:

  • It ran without stopping on breakpoint, so it diverts from somewhere else

  • Let us breakpoint in postcopy_incoming_setup, it doesn't even stop there

  • I think loadvm_process_command is our target now, we need to see where it diverts ... It breaks here !

  • Looking around, this is the process, the first thread sends a listening signal(using qemu_savevm_send_postcopy_listen()) to reciever and so it starts listening, calls the setup etc

  • Even the postcopy_start function in migration.c is not called, need to keep getting higher up

  • Wow even migration_iteration_run() is not called

  • It must stop at migration_thread ... and it doesnt? How is that possible?

  • AI told to look into the incoming funtions, these are outgoing ones and are meant for process to process migration

  • Let us set a break point in postcopy_incoming_setup(), it doesnt stop there too

  • From what it looks is it doesnt use postcopy anywhere to load in a file

  • qemu_loadvm_state runs, now I can trace it from breakpoint ... No I cant the coroutines make it difficult for GDB to follow

  • There must be some other way, maybe I can try that later to look into flow of code using sockets

• Now let's divert focus to mapped ram too
  • It clearly says that mapped ram was desined for multifd support so that dafaults it to later implementation using multifd?
• My mech keyboard just arrived, signing off to assemble it

Day 12

16 May 2026(~4 hrs)

Today's Target: Decide point of divergence for fast snapshot load and again look into mapped ram support

• I believe after enough days of running postcopy, I should first decide where is the code diverting away from postcopy and where should the extra code be added

• Let's just continue with the GDB method of running and looking at path, trace back

• Call stack of postcopy_ram_incoming_setup shows call by postcopy_incoming_setup, which is called by loadvm_postcopy_handle_listen called by MIG_CMD_POSTCOPY_LISTEN

• We do reach qemu_lloadvm_state, after which the command is to call loadvm_postcopy_handle_switchover_start, which is same no matter postcopy has been activated or not

• Long time but GDB tracking is a failure

• Let's try running it with traces to see what happnes

  • After some tweaks on which trace to use found a useful trace(too big to paste here):
    • qemu_loadvm_state_setup is the entry function, which looks good
    • Then ran qemu_loadvm_state_main that loads everything simply
    • Not sure but function qemu_load_device_state should run first in our poscopy load to load in basic device data to let it start running
    • Ahh I think now I get the problem, ... as with normal postcopy just setting its capabilities is not enough we need to explicitly send migrate_start_postcopy and that command is read by loadvm_process_command which then initializes the postcopy environment
    • I need to verify this what happens when we given such a command, but the load is too fast for me to give in another command
    • I need to look in on the main thread who sends these signals to the listener, maybe I can play with hardcoding stuff to send a froced signal

• So first we need to find where the parent thread writes something into the QEMUFile stream

• Now f is just mis->from_src_file, a stream, lets see who fills in this entry

• Nice using the find all references features in VS code it might be easier

• Does look like migration_incoming_setup, but wait this function has switch case on channel being MAIN, MULTIFD, or POSTCOPY so the channel can be anything?

  switch (channel) {
  case CH_MAIN:
      f = qemu_file_new_input(ioc);
      assert(!mis->from_src_file);
      mis->from_src_file = f;
      qemu_file_set_blocking(f, false, &error_abort);
      break;
  
  case CH_MULTIFD:
      if (!multifd_recv_new_channel(ioc, errp)) {
          return false;
      }
      break;
  
  case CH_POSTCOPY:
      assert(!mis->postcopy_qemufile_dst);
      f = qemu_file_new_input(ioc);
      postcopy_preempt_new_channel(mis, f);
      return false;
  

• Let's try a breakpoint on the point of decision of channel in migration_channel_process_incoming:

  • migration_has_main_and_multifd_channels returned false beacause there is no from_src_file, and there is no to_src_file as per GDB
  • So no connection has begun yet, so CH_MAIN is what it returned
  • Then setup is as per the channels and in main it makes the file from the channel and puts it in from_src_file, but the same thing took place at CH_POSTCOPY case hence postcopy case had another from file to begin with

• Now lets force it to go with the postcopy method to load and see what errors does it throw around

  • I spent 10 mins wondering why it did not change, turns out I did not compile

  • Now it does go in migration_incoming_setup but it returns false for postcopy, because all necessary channels to proceed with incoming migration are established without error

  • Let's see where else is this function called for decision ... only migration_channel_process_incoming is the possible way to enter migratino_incoming_setup in CH_POSTCOPY

  • Maybe the function postcopy_preempt_new_channel blocks or returns in some other thread, but still why return false directly, but that is for preemtion, ie serving faulting pages with priority

  • I was doubtfus as migration_has_main_and_multifd_channels decies the channel so I thought let's turn on the multifd capability and got this error

    Error: Migration requires multi-channel URIs (e.g. tcp)
    

    But as multifd is optional target for project let's leave it for now

  • I have done enough exploration to decide this function is not the one responsible for the important job, it just sets the preempt file which we would love to add, but this is not critical place we are looking for

• Checking in on call stack of qemu_loadvm_section_start_full, there is snapshot_load which iirc is kind of different from migration but save_vm.c(include load_vm) is part of migration

• Leaving that let's see who sends the signal of MIG_CMD_POSTCOPY_LISTEN to listner thread, let's look into that

  • qemu_savevm_send_postcopy_listen does that, name is ambiguos maybe because it is in savevm.c
  • This lies in postcopy_start, which actually is called by migration_iteration_run, let's breakpoint this and see ... but it does not break 😕
  • Does this mean there is no migration thread? ... GDB confirms that, and it shhould be as migration_start_outgoing launches this thread and we are not exactly outgoing
  • Looking on call stack, found qmp_migrate lets proceed onward to see when can we say fast load conditions are possible
  • hmp takes in uri and find the channel and directly gives the channel to qmp
  • We should be able to parse input and looks like finds out the main_ch and cpr_ch(whatever that is)
  • As most likely CPR is turned off, we just have migration_connect_outgoing called on main address with current migration state
  • Looking into migration_connect_outgoing, ... This is where the point differs !! it checks for types and fills in the channel

• Wait there is something else, migrate incoming is not breaking on qmp_migrate ... It doesnt even break on hmp_migrate

• I need a fresh mind to continue, one idea I had to simply use most of present code is simply make a new thread to simulate a source but that will be quite bad as it will use a channel and without mapped RAM optimization it will be much more slow compared to block and load

Day 13

17 May 2026(~6hrs)

Today's Target: Finalise the divergence point and start thinking about week1

• We do understand that postcopy_ram_incoming_setup must be called nearly as is

• Similarily all other steps like loadvm_postcopy_handle_advice, loadvm_postcopy_handle_listen should be called

• However qemu_loadvm_state_main uses a file stream given from load_snapshot

• process_incoming_migration_co is the function that is actually the function called, let's see by whom?

• Finally I was able to piece together the entire tree:

  hmp_migrate_incoming ->
  qmp_migrate_incoming ->
  qemu_setup_incoming_migration ->
  migration_connect_incoming ->
  file_connect_incoming ->
  file_create_incoming_channels ->
  file_accept_incoming_migration ->
  migration_channel_process_incoming ->
  migration_start_incoming ->
  process_incoming_migration_co ->
  qemu_loadvm_state ->
  qemu_loadvm_state_main ->
  
  diverges to QEMU_VM_SECTION_FULL as there is no one to give command
  everything is simply loaded
  

• We need some time to understand the entire pipeline(I have read many functions individually but this is converging everything)

• as expected hmp_migrate_incoming extracts the uri and parses it, sending qmp_migrate_incoming just list of channels(only one as opened by parser here)

• MigrationChannel is just a struct of some things, like what transport it is (exec/file/socket etc) and channel type(MAIN, CPR), here it is main and file type

• qmp will now not get uri but channel and will not itself exit on error as it has exit on error functions inside?

• Here the once will force that migration can only occur once on a session, which is ok, It can be possible I want to migrate many times but makes sense to restart

• This is simply(for us) followed by a call to qemu_setup_incoming_migration

• migration_channel_parse_input is used to parse the input, only one of both channel and uri is taken and parsed

• After all the parsing(which I read but skipped here) we can assume main_ch has a copy of the main channel passed from parent.

• One doubt was it used QAPI_CLONE which says it deepcopies but in C it would mean it copies values of pointers and not the value pointed by pointer, so the deepcopy is only of the pointer to file? Otherwise we copied entire file, or maybe it copies all data of file as we are yet to read it

• It's compatibility is checked for migration and then migration_incoming_state_setup is called(which will be followed by migration_connect_incoming)

• migration_incoming_state_setup just does some checks and sets the state of mis to MIGRATION_STATUS_SETUP(note this is first point where mis is altered)

• Now migration_connect_incoming acts just like a junction for code splitting to various channel transport specific functions

• file_connect_incoming is directly given the file_args which is just the filename and offset

• So till now all we have done is put mis in setup and given the filename to file_connect_incoming(offset would be 0 by default)

• A new QIOChannelFile is declared that actually has the file now, all it has is a QIOChannel as parent and a file descriptor

  struct QIOChannel {
      Object parent;
      unsigned int features; /* bitmask of QIOChannelFeatures */
      char *name;
      AioContext *read_ctx;
      Coroutine *read_coroutine;
      AioContext *write_ctx;
      Coroutine *write_coroutine;
      bool follow_coroutine_ctx;
  };
  

• It was first all set to 0 and then fd and seekable was filled in, and seekable should be true

• Let us test it all once using GDB if it all is indeed 0, ... yes it is only fd and features is set

• Then if file is not seekable and the system tries to seek it returns which is fine

• Now it creates the channels from the QIOChannelFile using file_create_incoming_channels

• To this we just give filename and parent of fioc, so it not really has access to fd

• Then it checks for multifd, which I think we should see

• It looks good, increases the number of channels by number of multifd channels

• iocs is an array of all the channels and in our basic case is one as multifd would be false, the 0th channel is set and rest are made by calls to qio_channel_file_new_path to read that same file

• Then we just set their names and go in a step deeper into qio_channel_add_watch_full which is called for every channel

• It first creates a watch? then sets callback etc

• This looks like glibc calls that sets a callback function? means that function is run?

• The function is file_accept_incoming_migration, that further calls migration_channel_process_incoming, looks like that was somehow applying the function on all?

• If we go with that migration_channel_process_incoming is called with each ioc(one in our case) and it get mis(which should still have setup)

• We know we wont need TLS so it goes on and there is something "register yank"?

• That checks if the channel has feature to shutdown it will but in our case it is no because all we have set till now is SEEKABLE

• Let's veriify it again using GDB, the channel only has SEEKABLE(features == 32) and name set as "migration-file-incoming" ... Yup we are all good

• Now channel identification is to be done using migration_channel_identify, with channel magic🧙(no idea what that is)

• First check is if migration_has_main_and_multifd_channels, which should be false as mis has nothing set so it is not a CH_POSTCOPY

• Again as file is not peekable but set as seekable channel is multifd or main based on if multifd was active

• Now with this channel in we call migration_incoming_setup whose purpose it seems is to fill in mis and make a QEMUFile

• Cases:

  • CH_MAIN : set mis->from_src_file with a simple QEMUFile object that only has only ioc and false on is_writable, (can pass fds is false too). Set file to blocking and break. Mostlikely passes to return true
  • CH_MULTIFD : lets leave this for now but we can assume it sets up all the $n+1$ channels based on if it uses packets etc, returns true if multifd were setup correctly
  • CH_POSTCOPY : It just sets up a new preempt channel which is a reference to original file, always returns false

• Finally migration_start_incoming is called on return of true and process starts

• Mental note that this place looks really good to diverge in case we want to do postcopy, however it lacks many checks there might be a better place of divergence but still might work

• Now on migration_start_incoming it just calls process_incoming_migration_co and yeilds to the coroutine

• Now we move from SETUP to ACTIVE in migration

• mis is properly filled in here, with the largest_page_size and loadvm_co(coroutine that loads(itself))

• Now runs qemu_loadvm_state and it blocks savevm_state, not sure what that is

• Then it creates a thread pool for mis and sets it to mis->load_threads and inits mis->load_threads_abort to false

• Now the file is passed to qemu_loadvm_state_header and it loads in the head:

  • It starts by reading first few values like magic numbers, version copatibility checks
  • Then if migrate_get_current()->send_configuration is true(which it shoud as it is part of MigrationState which we did no follow) it loads in device data using vmstate_load_vmsd?

• As we continue we use qemu_loadvm_state_setup to setup the state and load device data

• Now migrate_switchover_ack is called, and if I am not wrong this is swtichover to postcopy(even if now we can add a condition to divert) and this might be the perfect place to divert code but again not sure

• All cpus are synchronized and we try to load the main chunk of vm(qemu_loadvm_state_main)

• This uses the channel as a stream not really looking if it is seekable

• This part should have been linear, now in our specific project here we should continue by running the guest VM and also use mapped RAM

• Finally we cleanup and return

• The best position to diverge off seems to be before calling qemu_load_vm_main although proper care will have to be taken about capabilities

• We now know the entire normal load pipeline, I think now should look into the loading and running of machine live and mapped RAM

• Now let's see how the guest runs if we have postcopy set, this looks like the function to go for vm_start(I arrived at by searching for liv in migration.c)

• Does look like a direct function to run the VM, so this part is quite simple, next we see more about mapped ram

• I thought postcopy-ram.c would have most of it but it doesn't, Then I think ram.c would have it. Let's begin the first time exploration:

  • First thing I read is that, 0x100000 offset is used because different systems may have different offset but for loading we may be able to use same most optimized offset for that machine

  • This looks important:

    /*
     * Below fields are only used by mapped-ram migration
     */
    /* bitmap of pages present in the migration file */
    unsigned long *file_bmap;
    ...
    

  • So there is a file_bmap of pages present, this would act as our counter I believe for thread 1 that loads all the pages actively

  • all this looks good, let's try to find the actual code that uses this in ram.c

  • Premtion is one of the optimization we planned to do but this method is slower, our plan would be to allow thread 2 to itself load in the file if it deems necessary without requesting thread 1

  • XBZRLE is something corporate specific/busniness related and I dont think specific cache for that is an immediate target, but could be an additional target

• I think that is enough for today, I will continue tomorrow with mapped ram and hopefully have a decent understandng of current system so I can come back and start programming after the 4 days.

Day 14

18 May 2026(~7hrs)

Today's Target: Look deeply into mapped RAM

• Continueing from yesterday on mapped ram lets look into this function mapped_ram_setup_ramblock:
  • First a MappedRamHeader struct is declared and its attributes version and page_size is filled in according to big endian system
  • Then we check if it is ignored by migration by checking its flags and migration mode
  • In case of ignoring the offsets are set to zero
  • Else, we fill it after some calculations of num_pages which is how big is the block divided by page size, size of bitmap required for this specific RAMBlock
  • then the bitmap_offset is set as file offset + header size, so the bitmap occurs after bitmap on the file
  • Similarily the pages_offset is roundup of bitmap offsert + bitmap size and the alignment
  • Converting them to be64 we put them in the header
  • Then the header is written into the file and if the block is not ignored, the file ptr is moved to pages offset to leave space for bitmap
• For this specific project mapped_ram_read_header would be more important(though it depends on if there is a way to save files in the specific format and I think there is)
• First we simply read the header from the file, read in the versions, pages sizes etc and return, very simple
• Let's look at the call stack of mapped_ram_read_header as that is what we will be working with to load in snapshots
  • It is called by parse_ramblock_mapped_ram which takes in the file, block that I believe must be filled in and address at which to parse(but that is set by ptr in file itself)
  • Ahh length is used later to read pages, it is how big the block is expected to be?
  • then we start setting some small things on block like page offset and after checking alignment move to reading the bitmap
  • Then it calls read_ramblock_mapped_ram that reads in num_pages as specified by the caller
  • It uses a simple for loop to check all the bits and then processes each one by one
  • First all pages between clear_bit_idx and set_bit_idx are set to be 0 and then the non zero pages are read
  • Then the read happens and we continue this until everything is loaded
  • trailing zero pages are handled and then it returns
  • Even upper level function is parse_ramblocks which is told just how many ram bytes to read and it declares a block, reads in from the file the id as a byte array and then reads in the length and id of block to read and passes on to parse_ramblock
• All this was called only and only by ram_load_precopy which proves that postcopy no matter how never uses mapped ram, however we can use the parse_ramblock but then we need to think about the granularilty of loading
• Does thread 1 load in one page at a time or block at a time, if thread 2 encounters a userfault does it load in entire block or just page.
• That depends on what exactly a RAMBlock is and how big is it
• Considering how it is composed of both zero and non zero pages, maybe for one simple VM there is only one RAMBlock, need to look into that
• Now that there is ram_load_precopy there must be a ram_load_postcopy too, let's find that
  • Found it, it takes a file and cahnnel used to load
  • but the comment says file is where to send the data? Isn't is supposed to load
  • Let's just continue and see what we get
  • As previous case we get the addr address from the file of where the page is to go
  • From that we extract the flags and get a ramblock from the stream by reading in its name
  • Then after a lot of doing stuff and putting the read page in temporary buffers there is one point where we simply do a postcopy_place_page
  • This uses qemu_ufd_copy_ioctl which is quite clean
• Another point I just thought of is multifd support is not really a big deal if we can copy in pages using multifd and then use copy from temporary buffers, it is kind of trivial
• Let us see this thread too postcopy_preempt_thread, it takes in the mis and seems to keep loading stuff we planned to do on thread 2 as per documentation but postcopy_ram_fault_thread is better for that job
• Now we have an idea about most of the system, from what I understand the duty of thread 2 that loads in at userfaults is all good, thread or the main thread on the other hand would need more work, it can be very much like ram_load_postcopy but would need changes to support mapped ram(similar to thread 2 too). Thread 2 will mostly be postcopy_ram_fault_thread called by postcopy_ram_incoming_setup
• We might also need changes to ram.c to make postcopy support mapped ram but we can have that later, right now the game plan is to finalize how the threads will operate
• Now we have a game plan, after my 4 days of travel I will come back and first off read the entire log (around 1.5k lines 🥲) and then start planning on exact divergence and update mechanism

WORK PLAN

• First the biggest thing to tackle is memory arrangement, mapped ram uses RAM blocks and postcopy uses pages
• One point on design of postcopy_ram_fault_thread is that instead of requesting main thread or source to send page it must load it itself
• ram_load_postcopy should be our plan for thread 1 that loads in all files but as it does not use mapped ram we need a lot of changes
• As basic purpose of mapped RAM is supporting multifd in file: URIs, it allows reading of ramblocks in parallel
• So the plan is that the main function for fast snapshot load first will call a setup function that reads the file and keeps a list of pointers to RAMBlock objects
• qemu_ram_block_from_host will tell us which RAMBlock can serve the page and then the thread 2 will load in and set file_bmap in ramblock atomically saying that it is loaded in
• From what I understand file_bmap is used by migration so will be populated by us in setup and used to serve everything, again thread 2 will see if main thread is already handling that page then it will back off
• Now how will main thread operate:
  • It will first set up the RAMBlocks and all so that qemu_ram_block_from_host returns correct values and also we need proper map for RAMBlocks to reading point in file
    • One option is to have two file channels one for each thread
    • Then the first one crawls loading everything and second one seeks and loads
  • Then it initializes the postcopy_ram_fault_thread which could serve any fault, note we start it before we let guest run as otherwise it may miss some uffds
  • The bare minimum device state is loaded in and the guest cpus are let go once some sync is done with thread 2
  • Now main thread has the duty to keep copying in anything that is left atomically, so it first loads stuff on temp pages and then sets them
  • I am not clear about how these make it atomic but we plan to change the bitmap as soon as we start transferring some of the pages
    • Answered by mentor: we first transfer data from dist to temp page and then make the page table point to it instead of first allocating the page and then filling it in
  • We wont transfer all at once as that would too long of a block with bitmap unmared in one go
  • We start from beginning and load in constant number of pages say C which we can fine tune later
    • We start with 1 page at a time as that would be safer and easier to do for first implementaiton
  • The bitmap file_bmap is currently only used while saving but not loading so we can use it freely as it has no purpose while loading and has good peripheral functions for atomic operations, initializatin etc
    • receivedmap can be used for this purpose too
• Once main thread has loaded everything it signals the fault thread to quit, cleans up bitmaps and some other things and returns to qemu main loop

Points I think I miss

• How exactly to begin execution of the guest and is it as easy as resuming it's cpus
• What exactly is device state and how is it detected?

postcopy handle run HMP "info ramblock" Guest RAM, a few ROMs, vGPU vRAM (sometimes)

Day 15

23 May 2026(~9 hrs)

Target: Start some coding, trying to add stubs that will be filled later

• Function migrate_caps_check in channel.c is the one that throws error when mapped ram and postcopy ram are both set together

• We need to analyse if removing this if else condition will break the system in some case

• But I think this would just be a warning and the code actaully never uses mapped RAM check on the postcopy path

• So we can remove this alltogether or bounce a warning to the monitor

• Now if we consider the divergence point it will be when qemu_loadvm_state calls, qemu_loadvm_state_main

• We will start our own function(the thread 1) and not go into qemu_loadvm_state_main

• It will be done by using if condition on postcopy, doing this:

  if (migrate_postcopy_ram()) {
      ret = qemu_loadvm_state_fast(f, mis, errp);
  }
  else {
      ret = qemu_loadvm_state_main(f, mis, errp);
  }
  

• It seems to silently fail, need GDB to confirm if that point is reached

• GDB worked and it went right where expected

• There was a segfault in my error report of no implemented

• I used error_report and it worked, Lets see if all tests run or there is some problem

• There was error in one test hence tests failed😢

• That means there was a test that we caused to imbalance, that should not have happened though?

• I did some extra things like qemu file set error and cancel incoming and now it ran, threw error and then loaded in?

• Due to some git problems I reset it all 💔, Build it better and from scratch again(the few lines I wrote)

• Let us make a branch to actually implement this ... operating in branch

• First we know we diverge at qemu_loadvm_state, here if postcopy is active we run the main function that we planned and not qemu_loadvm_state_main

• Finally I found out the option to disable all AI features, Idk why but even after unchecking all boxes AI pop ups appeared from nowhere. But now I have it disabled in all forms😌

• First change is this:

  if (migrate_postcopy_ram()) {
      ret = qemu_loadvm_state_postcopy(mis, errp);
  }
  else {
      ret = qemu_loadvm_state_main(f, mis, errp);
  }
  

• Now this new function should act as our main thread, qemu_loadvm_state_postcopy

• Let us first write a stub for it around qemu_loadvm_state_main

  int qemu_loadvm_state_postcopy(QEMUFile *f, MigrationIncomingState *mis,
                            Error **errp)
  {
      ERRP_GUARD();
      /*
      * 1 ) prepare RAM blocks
      * 2 ) Launch postcopy_ram_fault_thread
      * 3 ) If device state is loaded in caller then all good
      *     else do it now
      * 4 ) Let the user run ie resume all cpus
      * 5 ) start iterating from beginning for each page
      * 6 ) On reaching end, cleanup fault thread and return
      */
      error_prepend(errp, "qemu_loadvm_state_postcopy no implemented yet");
      return -1;
  }
  

• As we write everything we should do it in commits to prevent losing anything

• Currently it leads to a seg fault and no error printed, Let's boot up GDB again

• Looks like the error message is too long? trying to make it small ... Got it, error was that it should not prepend but set it

• Now it runs without exit(thought the error is not printed too)

• Let's try running tests now ... these failed and I have no idea why they should?

  • qemu-system-x86_64: Unable to shutdown socket: Transport endpoint is not connected
  • qemu-system-x86_64: Channel error: Input/output error

• My changes shouldn't even affect this piece of code, except if they have problem with capability setting, Let me ask AI about these errors

• The stub fails the code in other cases, let's do some different changes

• Let's see what qemu_loadvm_state_main does when it fails, ... It sets file error and cancels the dirty bitmap and some other postcopy conditions, Let's go ahead and put these error lines in our stub

• There is still some problem with the sockets, Let's try finding out something more, question is does it run?

• BOOM got it 💥, I have to check for both mapped ram and postcopy ram active at same time in that case it does not fail!

• Some other test fails now, ahh it is validate caps test, that is meant to as we remove the validation

• Done, removed that test, It was far too extensive than I had expected but maybe can be used in future

• Now all tests pass, Let's keep going now that we have the stub and are decently sure our redirection is correct

• First step is to find how to initialize RAM blocks so that we can load them easily

• First let me just write out all the steps in comments itself

• Done:

  1. prepare RAM blocks Migration is setup here and we prepare RAM to get easy access to blocks in drive. We also initizlize other datastructures like bitmaps we will be using
  2. Launch postcopy_ram_fault_thread This stage includes updates to thread so that it can also load in blocks
  3. Load device states We need to load in the minimum state for guest to start running 3.5) Let the user run: Unpause guest CPUs so that it can run
  4. start iterating from beginning for each page We actively load in pages as the guest operates
  5. On reaching end, cleanup Send signal to fault thread to kill and clean bitmaps etc used, then return

• Now as we begin with first one, we see a lot of functions in ram.c but I don't remember any of them being called in our exploration.

• This means it blocks were allocated while loading(unlikely) and more probable is as we give input ram is entirely allocated which eases our process significantly

• Going with the second assumption we dont really need a setup but we should really intialize the bitmaps

• Let us look at all the references to receivedmap to see who all have it

• It has surrounding structure for use but it is mostly atomic update and test separate, not test and set necessary for only one thread update

• I think it is better to currently using bmap is good enough, it is made for this stuff and seems to support atomic test and set

• Interestingly our implementation of moving constant number of pages is already implemented using clear_bitmap_shift, however the size is too large(default is 18), which makes it's use questionable

• I am postive it is not something that would make a lot of sense for use to use, our initial implementation is based on one page load per fault and that should be enough

• I used the most simple thing to have just file_bmap as its comments state it stores bmap pages present in migration file, where as receivedmap is for the ones received.

• We do not care about what all we have received but we need all pages that have not yet been loaded in so we use file_bmap for this purpose

  /* Skip setting bitmap if there is no RAM */
  if (ram_bytes_total()) {
      RAMBLOCK_FOREACH_NOT_IGNORED(block) {
          pages = block->max_length >> TARGET_PAGE_BITS;
          block->file_bmap = bitmap_new(pages);
          bitmap_set(block->file_bmap, 0, pages);
      }
  }
  

• After this all ramblocks should have the required bitmaps

• Oh wait 🤦, mapped RAM has it's own handling of bitmaps if I remember, nvm let's update it next time

• I believe this completes the progress to 0.5/5, let's continue tomorrow to get it to completion and potentially move on to step 2

Day 16

24 May 2026(~6hrs)

Target: Continue with filling in bmap and potentially move to step 2

• We know that zero pages are stored in a different way in mapped RAM, so the actual bitmap should be different

• Initially without the file we are sure that there is no zero page etc as entire RAM will page fault, what we do need to do is actually read in the bitmaps in files

• Let me revise the mapped RAM format again using it's parsing functions

• For the dirty bitmap we need to read in the headers followed by the bitmap

• Wait a minute I think there is a memory leak, bitmap from mapped RAM is allocated read but not freed, it doesnt use smart pointer etc, it is just a g_malloc0, that is a valid bug

• Running tests after freeing the bitmap ... Well I was wrong, tests gave a double free error

• The entire pipeline to read in ramblocks in both reusable and not reusable at same time, it does a good job reading in headers and bitmaps but also reads in the data

• We need to prevent that which should be fairly simple using the qemu_set_offset function

• Question is do we use the same function or a new one?

• I think we could use parse_ramblocks, or maybe we start working on cleanup patch to ram.c so that we can reuse ram_load_postcopy, I need to trace the ram loading using gdb too

• This is weird that it is actually code in ram.c that loads in the ram and not the pipeline we mapped

• What seems to happen in precopy case of load is first the entire RAM is loaded in memory(in a buffer in RAMBlock) and then it is loaded on the VM

• There should be a better way than this, let's try something else

• First and formost is loading in block header not everything, for that we can use parse_ramblock_mapped_ram but we need to update it

• Did the following update:

  /*
  In case postcopy ram is active, this function was called qemu_loadvm_state_postcopy and its purpose is to load in the bitmaps and headers only. The RAM will be loaded later in postcopy fashion
  */
  if (!migrate_postcopy_ram()) {
      if (!read_ramblock_mapped_ram(f, block, num_pages, bitmap, errp)) {
          return;
      }
  }
  

• All the tests passed so this looks good, also the later SEEK_SET would prepare it for the next block

• Considering all this we can also call parse_ramblocks directly, but I need to check I have a feeling that these ram loading functions are called by main qemu much before our code will run so we dont want to run them twice, let's check using GDB

• Ahh got it is vmstate_load that calls the ram_load function

• We need to do updates according to that, Best idea would be to reuse parse_ramblocks in ram_load_precopy

• However it would be better to have sort of a preload, That looks like a really good Idea, I'll write a stub for that, it will just read in the headers and fill in bitmaps

• It will be influenced by parse_ramblock_mapped_ram but would be a new function in it's own right

• We might even be able to clear up parse_ramblock_mapped_ram by adding the new say mapped_ram_read_bitmap, then we can use the mapped_ram_read_header function with this to read the bitmaps

• But the thing is only one line difference in this function and postcopy version might be very small of a factor?

• A valid difference is that postcopy version will store the bitmap but precopy one does not, so let's do it

• Writing the function something feels off about returning the bitmap pointer, I don't think there is any other such function in qemu and so metaphorically this is not the qemu way

• Considering that the ownership of the bitmap would become ambiguous this doesn't look like a good thing to do

• Let's just add a new function, but the eerie similarness is getting on my nerves

• There should be a better method ... This took me through the file bitmap.h which clearly has a new_bitmap and everyone should use that

• Hence we do have a function that returns unsigned long* but it is inline, however if my function is long enough it challenges consistancy

• How about this during the normal read too, it uses the bitmap in block and deletes it the moment the function returns

• Let's try this out ... All the test pass which is nice, just a doubt is does g_free sets pointer back to NULL cause that might be important, doesnt look like it so we should do it manually

• I tried doing the new function but the number of arguments is exceeding what I think should be good, Let's extend this function from mapped_ram_read_bitmap to mapped_ram_read_header_and_bmap

• The function is done, tests are running, ... 3 tests failed?

  292/1020 qemu:func-quick+func-riscv32 / func-riscv32-vnc                         ERROR            0.64s   exit status 1
  387/1020 qemu:func-quick+func-alpha / func-alpha-migration                       ERROR           41.36s   exit status 1
  545/1020 qemu:func-quick+func-ppc64 / func-ppc64-migration                       ERROR           34.04s   exit status 1
  

• I have no idea why these would fail, first one is not even related to migration? And am I using something ISA specific? x86, arm etc had no error. Is this some problem with endieness?

• But all I did was move code into a function, ... maybe just maybe there is some problem with returning long?

• I would say I certainly lack exact understanding of this error, maybe I should ask some AI about it

• Got the problem of memory leak in case of error, though it should not affect the tests but maybe it does, let's solve it first

• Interestingly I ran test again before solving the memory leak and it ran with no fails, I ran tests 3 times all passed, I might not have saved changes to file that time so the error came

• Now we need to write a function to use this to read in a all the bitmaps, and we should also follow it up by clearing all the bitmaps in step5 so we don't forget

• A new point to note is what is offset of current ram block and where is it stored ... got it, it should be fd_offset

• Another thing to take care I think now is to fill in fd_offset as I am not sure who fills it in but I dont think anyone does, if that is the case we will have to do that, let's note that in main funciton

• I saw that the goto fail statement was not really necessary in mapped_ram_read_header_and_bmap as the bmap was allocated later, it must be freed only in once case

• Now we need to fix the following for step 1:

  • Complete the function mapped_ram_preload_ramblocks which will read and load the bitmaps for each ramblock
  • We need to see if the fd_offset is being filled in(likely not) and size of bitmap in RAMBlock struct

• Thinking about step 2:

  • Add code for launching postcopy_ram_fault_thread in qemu_loadvm_state_postcopy(include eventfd for exit)
  • Open a new access to file for parallel reading of on demand pages
  • Add code to instead of requesting for pages load in pages itself
  • Idea would be to give either another input to the thread for QEMUFile* which is NULL in usual case and given in our case, or we can do the same by using attributes of MigrationIncomingState

• Step 3 and 3.5 should relatively be simple to just load device state and run, I just wish there is a specific function for it

• Step 4 also would have most things in place, first it will atoically test and set file bmap to say this page will be entertained by me and if successful to set it will go on to load it(or fill in case of zero page)

• Step 5 might be quite simple too, just free any extra bitmap/counter etc we added and return

• Back to step 1, Let's work with the preload function, before that I need to fully observe loading using mapped RAM

• Best method would be to follow precopy load and follow every read/write using hex editor on the file

• I followed some of it, need some more time mapping will continue later

Day 17

30 May 2026(~3hrs)

Target: Understand maped ram some more deeply starting trying to understand device states

• Last monday meeting with mentor resulted in a very important discovery for me, that device states are stored at last after RAM blocks and I had no Idea about that
• So today let's first focus on observing how this state data is loaded in
• It looks like the BH functions are the ones to look into for that
• This must be the line that loads in device state migration_bh_schedule(process_incoming_migration_bh, mis);
• However it never recieves the QEMUFile so it wont be able to read the state?
• Ahh this looks like it qemu_loadvm_section_part_end calling vmstate_load
• There are old and new styles, let's test the old one first as I am not fully clear on vmsd, looks like every hardware has it's own load_state defined
• So we just need to call these? from vmstate_load
• Looks like qemu_loadvm_section_part_end is the function to learn from, it is what loads these in
• So I think most of our changes need revision
• The main idea is that qemu_loadvm_state does a damn good job loading in everything, we just have to force it to skip loading RAM when it is called for fast snapshot load
• I believe it is the ram load function that is called when it sees a RAM
• This means most of to do is quite done and most of our last efforts were in vain, we just need to update ram_load_postcopy to load in only the header and bitmap in case of mapped RAM
• Maintaining the state of mis is the only challange then and it is quite weird, mis is very very cluttered with many attributes and their uses all around the place
• TODO is to revert the changes to precopy functions and update ram_load_postcopy to work with mapped ram to load only header and bitmap
• So in that case what we will do is instead of our current if else statement to call one of qemu_loadvm_state_postopcy and qemu_load_state_main we call the latter always as it takes care of our proposed steps 1 and 3(not 3.5)
• There should be no problem in lanuching postcopy_ram_fault_thread later after these
• Updated plan is to:
  • Hook up ram_load_postcopy to work with mapped RAM
  • Call qemu_loadvm_state_main always
  • Followed by call to qemu_loadvm_state_main call the new function qemu_loadvm_state_postcopy(name subject to change)
  • Function creates a uffd for fault thread to operate on
  • Function launches the postcopy_ram_fault_thread
  • postcopy_ram_fault_thread in case of mapped RAM instead of requesting pages loads pages itself
  • After the fault thread is launched call vm_start to let the guest begin(fault thread will start serving it right away and there wont be any dropped/missed userfaults)
  • Now main thread will keep loading pages one by one from RAM
  • On reaching end it will cleanup by clearing bitmaps, killing the fault thread, closing file etc
• I believe, vmsd would be there on modern systems, let's test using gdb ... and interestingly the old style is used
• That means we can happily look into load_state which is actually a function pointer most likely to change would be ram_load as others are hardware specific
• This tests if postcopy is running which in our case would and so ram_load_postcopy would be called with precopy channel
• It does not use the mapped ram functions and instead uses a ram_block_from_stream function to read from file like a stream
• Hmm but all it does is read in the id and not the actual block
• Then there is a whole switch case to read in flags and do everything
• We must bypass it, with otheer switch case that would read stuff differently and use our function to read both and continue over a lot
• Need quite some thought on this part
• So we are sure to change some parts of ram_load_postcopy to read in headers and bitmaps, preparing the blocks
• This would allow qemu_loadvm_state_main to read in device states and ram state for us
• I think it is better to begin with coding this next time after the travel, otherwise I will write code now and be disconnected when I return

Day 18

8 June 2026(~9hrs)

Target: Revise the logs to get back in field

• I spent some time testerday reading the logs and will contiue doing the same

  • A point came up in my mind while reading the notes was an optimization:
    • If the guest writes to a page not loaded yet we can let it resume and store that write.
    • However now I think of it again it is a really bad idea as then we need to know where on the page what write happened and the overhead of such mirco management of memory would exceed any benefit
  • Wait I had mention of BlockDriverState in my logs, what did it do, I might need to revise

• After having read most of logs, I think I am ready to go revise the changes done

  • Major change is the point that we are reusing the qemu_loadvm_state_main
  • I think most of our changes need revision
  • With a fresh mind I believe I should reset everything and then implement qemu_loadvm_state_postcopy as per new plan
  • Resetting back to master branch deleting the current changes
  • Done, now let's git pull, compile, make a new branch and start working on the new function

• Starting again with the function

  • Why does clangd break every time😢?

  • Trying to get clangd back on board ... Ah the problem was I had re enabled the C/C++ extension by microsoft and it was throwing the errors, clangd is a good guy

  • The idea is ready:

    int qemu_loadvm_state_postcopy(QEMUFile *f, MigrationIncomingState *mis,
                                  Error **errp) {
      ERRP_GUARD();
      /*
      * 1 ) prepare RAM blocks
      * Done by qemu_loadvm_state_main
      * Migration is setup here and we prepare RAM to get easy access to blocks in
      * drive. We also initizlize other datastructures like bitmaps we will be
      * using
      * TODO: Update qemu_loadvm_state_main path to initialize bitmaps and read in
      * headers but not load in entire ram blocks
      */
    
      /*
      * 2 ) Launch postcopy_ram_fault_thread
      * This stage includes updates to thread so that it can also load in blocks
      * TODO: call postcopy_ram_fault_thread here. Also update it to get ability to
      * load pages directly
      */
    
      /*
      * 3 ) Load device states and let guest run
      * We need to load in the minimum state for guest to start running and unpause
      * guest CPUs so that it can run Done by qemu_loadvm_state and covered in step
      * 1
      * TODO: Function to call is vm_start()
      */
    
      /*
      * 4 ) start iterating from beginning for each page
      * We actively load in pages as the guest operates
      * TODO: Use iteration to load in pages one by one for each ram block
      */
    
      /*
      * 5 ) On reaching end, cleanup
      * TODO: Send signal to fault thread to kill and clean bitmaps etc used, then
      * return
      */
    
      qemu_file_set_error(f, -1);
      /* Cancel bitmaps incoming regardless of recovery */
      dirty_bitmap_mig_cancel_incoming();
      error_setg(errp, "Progress 0/5");
      return -1;
    }
    

• Game plan:

  • Today our target is to get started with updates to qemu_loadvm_state_main and present the idea during meetin with mentor today
  • I believe giving 1-2 days to each step would be enough and would result in a raw implementation within a week or by next meeting
  • We can try to break up the updates and send RFC for first 2 steps as they mostly prepare qemu_loadvm_state_main and postcopy_ram_fault_thread for fast snapshot load
  • Then the following week we start working on elegent error and state handling
  • Target should be 2 weeks for implementation with first RFC being launched after meeting next week and improved one the following week each next day of meeting
  • Then on we focus on peripherals like testing framework, blocktime etc while working on feedback from community

• Step 0:

  • I need to just add a call to this function right after call to qemu_loadvm_state_main in qemu_loadvm_state with required checks

  • This should work now:

    ret = qemu_loadvm_state_main(f, mis, errp);
    qemu_event_set(&mis->main_thread_load_event);
    if (migrate_postcopy_ram() && migrate_mapped_ram()) {
        ret = qemu_loadvm_state_postcopy(f, mis, errp);
    }
    

  • Also we need to remove check in options.c to disallow postcopy-ram and mapped-ram being set simultaneously

  • Also removed the test for their compatibility check in misc-tests.c but that renders test_validate_caps_pair quite useless, I think there should be much more changes to disallow other pairs, maybe add this in future TODOs

  • Just commented out all the test for now, will discuss today in meeting about what to do with them

• Starting to work on step 1:

  • If I am correct qemu_loadvm_section_start_full is the function responsible for loading in major chunks of the RAM
  • Further in depth lies the vmstate_load function which calls the load_state defined as operation in SaveVMHandlers
  • We can be quite sure that it is the function ram_load that is actually called
  • We might need to check into the use of all these handlers in case we need any, adding to TODO
  • Now the function then called would be ram_load_postcopy with RAM_CHANNEL_PRECOPY
  • We need to understand what this function does completely before touching anything deeper
    • Comment states it loads a page in postcopy page but I am quite sure it wont load only one page
    • Let's just verify once by putting a breakpoint in this function if our manual trace of code flow is correct
    • I dont know why but GDB is not responding too, which is quite surprising as it worked last time I tried ... out of nowhere it works now?
    • Hmm it did not stop at ram_load_postcopy which is weird, let's go a few levels higher
    • Ahh so postcopy_is_running checks for current postcopy state and based on that decides but as we have not altered the postcopy state it goes straight to ram_load_precopy
    • It would be simpler to update ram_load_precopy and make it work but that would not be modular
  • Best would be to compare both ram_load_postcopy and ram_load_precopy, analyse the similarities and extract out a common function for our usecase
    • Hmm the differences are quite large, I think mergin them would be painful and might even be infeasible
    • Changing just the ram_load_precopy should work, let's add that as a cleaning TODO later and at that time think if we should move stuff out or rename function etc
  • Working on updating ram_load_precopy:

    • Looking in we get the parsing functions, we can use the method we devised last time to simply update parse_ramblocks and in that we can now work on making another function for parsing bitmap and header differently

    • We should be able to define similar function as we did last time for mapped_ram_read_header_and_bitmap

    • That should read in the header, followed by the bitmap and then based on if this is postcopy load in the block if required

    • Creating a new function might not even be necessary as we can simply define a new bitmap here

    • Wait I think only this should work:

      if (!migrate_postcopy_ram() && !read_ramblock_mapped_ram(f, block, num_pages, bitmap, errp)) {
          return;
      }
      
      /* Skip pages array */
      qemu_set_offset(f, block->pages_offset + length, SEEK_SET);
      

    • The skip works itself and the addition of migrate_postcopy_ram to if condition automatically takes care of not loading in ram blocks

    • However what it won't do is populate a bitmap, we need to do that too

    • Currently g_autofree is used to store bitmap pointer but I think we should be able to use an entry in the RAMBlock, the receivedmap for this purpose(just need to take care about memory leaks)

    • I think this would be correct, not sure how to verify it? Maybe should ask mentor about how can I test such things in today's meet

• Surprising enough I get to start on step 2, the real deal:

  • There are essentially 3 parts of this step:
    1. Initialization/setup
    2. Actual changes to fault thread
    3. synchronization with fault thread to begin guest VM only if fault thread is ready
  • We can move part 3 to step 2 for a more uniform distribution
  • Let's start thinking about setup:
    • It should be prepping all userfault fds and some other states/variables required(bitmaps are ready though)
    • postcopy_ram_incoming_setup should be the function that takes care of all this and it also starts the thread
    • However the order here is not something I feel is quite right
      • Frist the thread is created and then temp pages are setup and ram block enable notify is called later
      • Although assuming/considering the point that no fault will occur till we call vmstart all this should not really be an issue
    • Let us leave preemption and look into it later about what it wants
    • Bitmaps were already setup, so are uffds and event fd, we just need to specify the file where it can read from
    • So we can add the from_src_file before calling setup so that the thread can actually read it
    • A doubt might be can 2 threads read from same QEMUFile object, likely not as their offsets would be different
    • So we need a method to duplicate or reuse file objects(look into that later on)
  • Let's see into the major chunk, we need to actually load in pages:

    • postcopy_ram_fault_thread is the only function we need to alter

    • I thought we might need to signal the main thread from fault thread that it is ready but considering that the uffds are set up any userfault will be stored there waiting for fault thread

    • We have an if condition I need to verify the purpose of

      if (!mis->to_src_file) {
          /*
           * Possibly someone tells us that the return path is
           * broken already using the event. We should hold until
           * the channel is rebuilt.
           */
          postcopy_pause_fault_thread(mis);
      }
      

    • Considering that we wont have to_src_file initially we would try to do our logic here but this code means not having a to_src_file can be error

    • Maybe we could confirm it in today's meeting and better thing to do is check if(mapped ram)

    • Hmm, we dont actually have to diverge but just need to not pause here so let's update condition to !migrate_mapped_ram() && !mis->to_src_file

    • Now the point we actually need to change comes: after the label retry that requests page from source

    • Another TODO that comes to my mind for this thread is that if some read fails in the file case it should be persistant fault and the error handling is correct but mislabeled for this usecase, need to update comments etc

    • I was typing and somehow an arrow key came out of the PCB, I thougt I had pushed all in perfectly but maybe I was wrong

    • Instead of adding the code to the postcopy_ram_fault_thread it would be better to use postcopy_request_page(maybe look into renaming this function or making a wrapper)

    • Even better would be to use if else condition to call a completely new function to load in the actual file

    • I dont know why but this retry is very weird, it could be much better written in a while loop

    • So we can implement a new function with a stub like postcopy_mapped_ram_load_page whose name covers everything that it does

    • Note that the further loop is not really necessary at this point and we can look at it later(iirc it is for vhost)

    • so what we do is put the retry in an if else

    • Then the funciton internals would check if the page is being sent by main thread and loads in depending on that

    • I think this is enough for today

Day 19

9 June 2026(~8hrs)

Target: Get done with step 1 and move to step 2

• First we plan to get done with step 1, there is not a lot remaining,
  • Let me just go through the changes I did yesterday
  • I don't really think there is anything significant left to do, just replace the recievedmap with filebmap as recievedmap might be populated already(discussed in yesterday's meeting)
  • Maybe just start working on step 2 as step 1 mostly looks good, just added a TODO in step 5 to free bitmaps
  • Adding more comments can be done when putting up an RFC
• Working back on step 2:
  • Picking up where we left: function postcopy_mapped_ram_load_page, All that this function does is loads in one page from the file
  • Plan:
    • check the bitmap for if the eager thread is loading it in, if yes return else we do the actual stuff
    • First we might need to take the lock of the file and I am not quite sure if that is simple cause I remember a TODO for that and it becomes non trivial
    • If we have the file we can simply set the seek and read in the page and return then
  • Let's go:
    • Bitmap: We start by reading off from include/qemu/bitmap.h
      • Really good features already provided here
      • I should once look into use of bitmap in ram.c if these things can be of use for modularization
      • It does look better to use bitmap_new
      • Ther is no functionality to have a bitmap_free and I think it is better to make it modular by making a function for it so we can couple them up
      • Considering the fact that g_free_sized has been deprecated this would just be a simple g_free though safer I think this change is zero value and we can just free bitmaps manually
      • A thing to wonder is, what is the bitmap stored in mapped RAM, I think it signifies the zero page and non zero page, which complicates our task, and so we need another bit to say if it is written to or not
      • So in ram.c we make the new bitmap RAMBlock->recievedmap to store the recieved pages bit(not actually recieved but to be recieved soon)
      • I am surprised the test_and_set_bit is not atomic, we need to use atomic compare exchanges, is there some other method qemu uses?
      • Maybe locks are used more often, but an atomic compare and swap is arguably better as then no need to poll
      • Found the function bitmap_test_and_clear_atomic but it tests on a range, we can use nr as 1
      • Considering that it is test and clear, 0 value means it has been recieved and 1 means pending, we need to update that in original bitmap loading too to initialize at all 1
    • Read from file: We need some sort of a lock on file

      • There doesn't exist one currently, so either we add one or duplicate it

      • I think for multifd there are multiple channels, not sure exactly how they work but can be a good point

      • Let's leave the creation of new channels, first load assuming the given channel works

      • Based on wether the filebmap is 1 or 0 we fill in a zero page or load a page, let's read in that part of ram loading and about filling in zero pages

      • I think this should cover it:

        unsigned long page;
        void *host = (void *)haddr;
        size_t read;
        
        page = rb_offset >> TARGET_PAGE_BITS;
        
        if (bitmap_test_and_clear_atomic(rb->receivedmap, page, 1)) {
            if (test_bit(page, rb->file_bmap)) {
                read =
                    qemu_get_buffer_at(mis->from_src_file, host, TARGET_PAGE_SIZE,
                                      rb->fd_offset + (page << TARGET_PAGE_BITS));
                if (read != TARGET_PAGE_SIZE) {
                    error_report("%s: Read %zd bytes from ramblock expected %d",
                                __func__, read, TARGET_PAGE_SIZE);
                    return -1;
                }
            } else {
                // zero page
                ram_handle_zero(host, TARGET_PAGE_SIZE);
            }
        }
        return 0;
        

      • But what remains is the form_src_file must have different channels for both the fault thread and the eager loading thread, also atomicity of this read is ambiguous and I think I need to use temp pages for this

      • Handling atomicity would mean loading in a temp page and then replacing it, let's see how postcopy handles it somewhere so that we can do a similar thing

  • Now the two points of correctness for concurrency:
    1. Read from different file descriptor
    2. Read in ramblock in temp page an then update it
  • Let's keep going on this function and look into temp pages, we can look into different channels tomorrow
    • Interestingly I cam across this function postcopy_place_page_zero and now I wonder if this is better than ram_handle_zero
    • Ahh looks like it takes care of the atomicity itself, which is really cool
    • Replacing ram_handle_zero with postcopy_place_page_zero
    • Maybe there should other similar functions(I think postcopy_place_page is one) that can copy pages from temp to ram atomically
    • So we need to load it in some temporary place but considering that the temp pages in mis are for zero page and huge pages I think I am missing something
    • Let us g_malloc now and see later how it can be done more elegently ... added to TODO
    • It does look supporting hugepages might not be that big of a deal considering these functions but you never know
  • I think it is better to look into the copying of qemu file tomorrow, I'll spend some more time exploring this and see if it is correct or there is some minor bug

Day 20

10 June 2026(~5hrs)

Target: Complete step 2 basic implementation and try moving to step 3

• The major step today is looking into duplicating channels for same file so that there is no race condition
• If time permits we try fixing the workarounds of last day relating to allocating new page
• Looking into multifd channel creation
  • Hmm, couldn't find anything of much use in multifd.c
  • Need to scan through the entire pipeline of load to find that, I remember somewhere there was a code divergence for this purpose
  • Quite some exploration and back tracking later I think this is the perfect place: file_create_incoming_channels
  • In that we can have channels += 1 to signify additional channel for the fault thread
  • Now the question is how elegently does it need to be handled, does a simple ++ to channels work or we should do enums etc?
  • Let's say for now we just increament it and add to TODOs better methods
  • However tracing the code back this does not seem to be a viable option as then the code goes on to run a load on each thread
  • There should be some other option, maybe I should be asking this question on IRC?
  • I might disconnect from IRC today, let's ask that tomorrow
• Let's go back to not having to allocate a place_srouce for loading a page
  • I think I should go back to checking how functions are used normally and if I can use the tmp pages
  • So I think the point was that tmp_huge_page in PostcopyTmpPage is a large enough buffer to accomodate small pages so we juse that
  • This brings us to postcopy_temp_pages_setup where I find this interesting thing that it declares as many tmp page objects as channels
  • So that means, if we set this channel to 2 somewhere we should be able to
    1. Have enough temp pages that both loading threads do not overwrite each other
    2. Potentially make multiple from_src_file(I might be wrong here)
  • Oh, this happens in postcopy_ram_incoming_setup, which is nice as we call that no matter what
  • Thread is created before allocating temp pages which is has no problems as the thread wont require these pages till vm starts which after this function call
  • It takes me to the point of that maybe preempt is a feature that is very close and might be something we use
  • Let's analyse it's usability
• Looking into preemption:
  • It seems like we are doing preemption no matter what, so technically we should have it enabled?
  • This means there should be a non preemtion option too, hmm this needs to be discussed in next meeting
• Today was quite scattered, need to work harder tomorrow

Day 21

11 June 2026(~8hrs)

Target: Try completing step 2 and try having a definitive solution to the channel copy problem

• First I think continueing with preemption is best, It might have two implementations:
  1. It establishes two channels to file like we want
  2. Establish two network channels to contact with source
• If it is option 1 we should have a definitive answer to solve this problem, but the more likely case of option 2 would mean
• Starting form postcopy_preempt_setup for modern versions, it seems the channel creted is definatively a socket as it uses socket_send_channel_create
• So this doesn't work for sure, this is a problem I need some guidance upon form community
• Going back, I realized I was distracted from resolving the temp pages situation:
  • That too has this duplication problem, we need two temp pages or twice the buffer
  • A simpler solution to this would be to update PostcopyTmpPage struct to either
    1. have a new field for fault thread buffer
    2. have the buffer be twice the size
  • Technically the new field should be a better way to solve the problem, but we need to be sure we don't break any metaphorical abstraction
  • This brings to my mind a question, is multi socket migration possible? Let's ask in following meet
  • Ahh wait a sec, it actually is not a single page but an array of pages as many as channels
  • So we can set mis->postcopy_channels to 2, like the migrate_postcopy_preempt case does in postcopy_temp_pages_setup
  • The enum PostcopyChannels is quite weird and feels very out of order
  • I just added an or with the statement for 2 channels in case of mapped RAM too
  • Now we need our fault thread to store stuff in the second channel temp page, so this should be an easy resolution
  • Some cleaning is left relating to how the indexing can be improved for readability purpose
• I think all this can be isolated into another function for load page channel, so that both the threads call the same function with different channel, lets try doing that
  • Hmm it works just by adding a channel argument to the postcopy_mapped_ram_load_page
  • Noted in TODO is that we need to add this channel thing to the file reading mechanism too whenever we have that
• Now let us start working on step 3 as the file confusion is real and I need guidance
  • As discussed in last meeting we can't call vm_start directly, there are a few other things that need to be done
  • I think the function to use here is process_incoming_migration_bh as it has most stuff already implemented
  • Just need to make sure there is no problem
  • I am not sure if call to dirty_bitmap_mig_before_vm_start might affect some things as we do not set anything up related to that
  • We are certainly not using this function owing to it actually setting state of migration to completed and then destroying the state
  • We need a separate function, or if I am not wrong the only thing that actually needs to be done before vm_start is qemu_announce_self
  • Let's see what happens if we just do these two things
• Now technically it should run? though the extra threads and all will not be cleaned ...
  • Ok it failed because block->receivedmap was in use and my assert failed
  • Good thing I added an assert there, now we know the bug
  • Hmm now came another bug in ram_block_enable_notify, which says userfault register: Invalid argument
  • It is something about ioctl not understanding the device driver?
  • Handshake is all good, and the opening looks good too. So why is it throwing an error?
  • Hmm GDB says the length is 0 and start seems like an abnormally high value(140730305609728)
  • I think we need to setup rb-postcopy_length before doing anything here
  • init_range and cleanup_range need to checked. Otherwise I dont think any other block of code affects it
  • It is only called by loadvm_postcopy_handle_advice
  • I think function ram_postcopy_incoming_init should be good enough
  • Interestingly this time no error thrown but assertion failed that errp cannot be NULL on setting an error
  • I thought it was because we returned in main thread with error but even fixing that seems to throw same error, GDB it is then
  • Everything ran good enough but qemu_file_get_error returned error leading to errors the problem
  • This is the error in errp that gets overwritten: 0x5555590b7990 "error while loading state for instance 0x0 of device 'kvm-tpr-opt': post load hook failed for: kvm-tpr-opt, version_id: 1, minimum_version: 1, ret: -1"
  • This error can only be thrown by qemu_loadvm_section_start_full, so we must look for a call to it, although it is kvm things?
  • This is quite weird, GDB did not break at setting that error line, I think this is another thread causing problem thing, I need to learn more about multithread GDB
• That's all I could cover today, Today I can confirm on the channel handling, and then on need to start working with the errors here
• Debugging this might be complex beast but let's see what happen

Day 22

12 June 2026(~9hrs)

Target: Complete step 2 and step 3 so that we have a running VM

• Yesterday's meeting went quite well and got a lot things that resolve many issues
• First and formost mentor told that qemu_get_buffer_at uses preadv which should be thread safe and I checked it is(here)
• Another one is that for vm_start in place of process_incoming_migration_bh it is better to call/do exactly what loadvm_postcopy_handle_run_bh does
• So our fault thread logic should work as is and we just need a few changes to the loadvm_postcopy_handle_run_bh
  • So using that one assertion fails in bdrv_graph_rdlock_main_loop, that is !qemu_in_coroutine(), so as we were in coroutine it caused error
  • But why is it forced that only call this function when not in coroutine
  • Even comment says it asserts that not in coroutine but does not explain why?
  • Looking at all the functions taht call bdrv_graph_rdlock_main_loop changing it would have quite some repurcussions
  • Let us look back into how bottom half calls it as in meeting I got to know that bottom half runs in coroutines
  • Call stack using GDB:
    • migration_block_activate is the function that causes error
    • In that call of bdrv_activate_all, however before bdrv_activate can acutally be called, GRAPH_RDLOCK_GUARD_MAINLOOP has problem
    • That has graph_lockable_auto_lock_mainloop called that causes problems
  • These Macros are quite complex and I have no idea what most of them do
  • Best would be to go and look on how the bh function is called?

    • hmm it seems to be scheduled in a different form so that it is not a coroutine

      migration_bh_schedule(loadvm_postcopy_handle_run_bh, mis);
      

    • Maybe using this method will work, let's try it out

  • Now I think we have the same error again, that *errp == NULL assertion fails
• Going back to debugging what the problem is:

  • Hmm reading in the *errp value it is the same as last time

    0x5555578c93f0 "error while loading state for instance 0x0 of device 'kvm-tpr-opt': post load hook failed for: kvm-tpr-opt, version_id: 1, minimum_version: 1, ret: -1"
    

  • It should be set in function qemu_loadvm_section_start_full, however errp has the func set to vmstate_post_load

  • vmstate_load should have returned a negative number, let's see if GDB stops here and it does(did not stop yesterday?)

  • We can see that ret = -22 which is -EINVAL so there was some error, let's see what happens if we do normal load

  • Normal load does not have a negative return, let's see exactly what happens

  • What fails is vmstate_load_vmsd, and there are so many return false here, setting breakpoint on every one to test which one fails

  • GDB takes a lot of time to startup, maybe using traces would be better

  • As per trace of vmstate_load_state_fail, vmstate_post_load is the function that fails, let's go in deeper on that

  • Now I need to use GDB to see what fails inside that, interestingly it has no declaration in any .h file?

  • So vmsd->post_load failed with return value -1 and I thing it is different for each device and we certainly did not touch it for anything but ram

  • The device is "kvm-tpr-opt", we need to checkout what this device is?

  • Found the definition in i386/vapic.c, the post_load for this is vapic_post_load

    • As per this, it returns -1 if state is not inactive and preparing for vapic is failing

    • Looks like it some enlightenment flag 🤨 for windows VM, but why is it failing on my debian test?

    • I think really should be something that should come up in our case?

    • However the error throw here was not succesful, let's see why it allowed everything else to continue

    • Ahh found a bug, didn't think I would find the bug in this way, so what happens ins when we call qemu_loadvm_state_main currently we dont check if it failed and so we overwrite the return value with the one of qemu_loadvm_state_postcopy

    • Solution is simply running the new funciton if ret == 0

    • However the original problem exists and it seems to occur before we even start our new code?

    • Let's test with precopy again, and it still works, that exceeds my area of expertise?

    • Exact point of failure seems to be that vapic_map_rom_writable returns -ve

    • Ahhh found it 😭

      /* grab RAM memory region (region @rom_paddr may still be pc.rom) */
      section = memory_region_find(mr, 0, 1);
      
      /* read ROM size from RAM region */
      if (rom_paddr + 2 >= memory_region_size(section.mr)) {
          return -1;
      }
      ram = memory_region_get_ram_ptr(section.mr);
      rom_size = ram[rom_paddr + 2] * ROM_BLOCK_SIZE;
      if (rom_size == 0) {
          return -1;
      }
      

    • ROM size is stored in RAM that is not yet uploaded

  • I thought all this should be part of device state, I wonder how normal postcopy handles this

  • get_system_memory is the unholy function that allows access to memory, and there are too many devices that call it😔

  • I think initialization is where all this should be setup though again not really sure which part of RAM does what

• A solution could be that the fault thread is intialized before this so that these faults(but it may be possible that it doesn't even fault?)
• Let's try some of that out, and moving step 2 before runs everything but system was unresponsive

  • Hmm I used print to stderr to see if it goes correct and it seems like it does, but then I tested info migrate and we get the same error of kvm-tpr-opt

  • Added a few debug prints in vapic_map_rom_writable and got outputs for precopy as:

    >> trying to access RAM
    >> ram trying to access 0x7f7383ecb002
    >> reading values
    12 6 E 7 31 C0 B9 0
    

  • and postcopy(snapshot) as(here >> lines are from vapic and other from fault thread):

    >> trying to access RAM
    >> ram trying to access 0x7f5233ecb002
    need to load at 0x7f5233ecb000
    page was pending
    not zero page
    read 4096 bytes
    >> reading values
    0 0 0 0 0 0 0 0
    Value at 0x7f5233ecb000 is
    0 0 0 0 0 0 0 0
    return success
    

  • This looks like some problem, fault thread loads in the wrong value? This does verify that userfaults are raised

  • Oh a difference I dont know what rom_paddr is and also the addresses accessed are different, wait let me match just the ram pointer

  • Even after all this the ram region they are trying to access should be same

  • I think I might be doing something wrong then? but what, how can the memory region change for postcopy and precopy?

• Looking into how the memory regions are decided
  • I saw in MemoryRegion, there are a bunch of booleans, and requirement thta they fit in a cache line, I dont know why that is required but it could be better to use an int and use bitwise operations
  • I am not sure how this value is decided but it feels like it would be a hardcoded value
• This thing is way too deep, let's continue working on it tomorrow, I dont think there some bug introduced by me but need to check why are the locations accessed different by the driver itself for the same file/snapshot

Day 23

13 June 2026(~8hrs)

Target: Resolve the reading 0 value bug and get the basic VM running

• First lets get back to the bug

  • One thing I got confused on was why is the address read different and I came to realize that it is actually in host space ie pointers randomized by OS

  • Problem is read value is all zero there must be some place we miss loading something

  • AI suggested that from my logs, I noted at one place that the fd offset of blocks was not set and I needed to set that, let's look into that

  • I used fprintf to error stream and it clearly shows offset is zero so it was never set and we must resolve this issue

  • There are two places in code where this offset is set currently, one is somewhere in hardware and other in physmem.c

  • This is a completely new thing? Ahh I came to the realization that we set block->pages_offset for this purpose, so instead of the fd offset using this is better

  • It works !! 😆😆😆, the values match with precopy case

    >> trying to access RAM
    >> ram trying to access 0x7fa2f3ecb002
    need to load at 0x7fa2f3ecb000
    page was pending
    not zero page
    fd offset is 0
    read 4096 bytes
    Value at 0x7fa2f3ecb000 is 
    55 AA 12 6 E 7 31 C0 
    return success
    >> reading values
    12 6 E 7 31 C0 B9 0 
    

  • However now it throws segfault after some time, but we got something running, let's try using GDB now to find exception

• New segfault is in fault thread, where it tries to place a zero page

  • It occurs on bitmap_set_atomic where the map is 0x0 and so a NULL? But receivedmap was used somewhere?

  • Ther is some place where the receivedmap was added/removed and we need to know how it reacts, I think it is somewhere in the qemu_loadvm_state_main that the map is initialized and then deleted

  • We updated ram_load path so that it does not load in the blocks, ram_load_setup is what inits it, and from what I think would be nice is to initialize our pending map here to, let me do that quickly

  • I was adding function for initializing the filebmap too when I thought I dont understand the actual purpose of these bmaps and should not be toying with them, maybe declare a new one for now and confirm these later in meeting

  • I am quite sure that ram_load_cleanup is called after qemu_loadvm_state_main finishes execution and we certainly don't want that to happen, we could the best thing would be to prevent it in ram_load_cleanup itself as otherwise we have to take care of which device cleanups to do

  • hmm got error:

    qemu-system-x86_64: ../migration/ram.c:4219: parse_ramblock_mapped_ram: Assertion `!block->pendingmap' failed.
    

  • Ohk I had some old allocation code there, cleaned it up now

  • New error comes up now:

    qemu: GLib: g_tree_lookup_node: assertion 'tree != NULL' failed
    

  • This does not tell where the failure happened so let's go to our friend GDB

  • Hmm it stopped somewhere else in postcopy_notify_shared_wake and has mis->postcopy_remote_fds(whatever that is) as NULL

  • IMO this should be empty, I dont know where it is set initially I think it is part of initialization, and it is part of migration_object_init which we must have called

  • Found it, on calling process_incoming_migration_bh it starts the vm and destroys the migration state

  • So in process_incoming_migration_co after we run qemu_loadvm_state, process_incoming_migration_bh is launched that starts vm and destroys the state, in ideal world, this should not happen

  • I think we should bypass this scheduling entirely as we take care of this in qemu_loadvm_state itself(for now)

  • IT WORKS !! 🕺🕺🕺

• We have the system running, now refactoring, cleanup and eager loads
• Now we move on to step 4, to load in the pages eagerly
  • There should be another thread for that purpose say the eager_load_thread
  • Let's plan on postcopy_ram_eager_load_thread
    • It will fist iterate over all pages loading the ones left
    • After iteration we are guranteed all pages are in RAM and we can exit
    • Cleanup
  • Implementing iterative loading:
    • First basic doubt is in what order do we load in pages
    • I think we can iterate over all ramblocks and just load in all blocks, another method might be to iterate over all addresses
    • Refactoring I think I found a bug in postcopy_mapped_ram_load_page, currently, it places page form temp to address accessed, but it should be at the page address
    • After a lot of time trying to have a good implementation I think this is a beautiful enough plan:
      • we call postcopy_ram_eager_load_setup that launches the thread
      • postcopy_ram_eager_load_thread simply runs for each not ignored block ram_block_load_eager
      • ram_block_load_eager simply iterates over all positions and calls our postcopy_mapped_ram_load_page to load pages
      • This would be quite clean as we wont need to write code again for checking the bitmaps
  • Running it now hangs everything and error logs show a lot of calls to uffd_zero_page or uffd_copy_page failed
  • Ahh another bug, this error did not float to top because postcopy_mapped_ram_load_page does not check if functions it called throw error or not
  • Let's see why userfaults failed

    • Output now is definitive first error

      qemu: uffd_copy_page() failed: dst_addr=0x7f0293e01000 src_addr=0x7f0258004ed0 length=4096 mode=0 errno=17
      qemu: ram_block_load_eager failed
      

    • There is still some where the error is not propagated as system just hangs, need to see how to handle if a thread fails, ... it just returns

    • The errno is 17 which I searched online is EEXIST, ie file exist or in this case the page exists, so the atomic check is failing in some way

    • One theory I have is that postcopy_place_page places from at host and not the page of host, though I am not sure about it

    • Checking out online I found that in non aligned case it throws EINVAL so that might not be the problem

    • Let's try to log which pages are loaded ... clearly there is some mistake:

      request to have something at address 0x7fbaf7ecb000 through channel 1
      request to have something at address 0x7fbaf7ecc000 through channel 1
      request to have something at address 0x7fbaf7ecd000 through channel 1
      request to have something at address 0x7fbaf7e01000 through channel 0
      request to have something at address 0x7fbaf7e01000 through channel 0
      qemu: uffd_copy_page() failed: dst_addr=0x7fbaf7e01000 src_addr=0x7fbac0004930 length=4096 mode=0 errno=17
      qemu: ram_block_load_eager failed
      

    • 🤦, I had a dumb mistake on update, fixed it and now it should work

  • Still does not work but throws no error, just hangs even with GDB let's try resolving this out
    • I think I have the problem? It seems to load way too many pages
    • fault thread still works perfectly, there must be something new we are doing that causes problem
    • Maybe there is some problem with the thread not being joined? nah even detatched thread wont work
    • The thread does exit but it does not seem to return to main thread and other threads seem to pause, let's try using GDB
    • I think pausing it in GDB and looking at call stack confirms that it waits on the thread sync event, that is what we were missing, let me see how the fault thread manages that
    • I am back after killing hyprland instead of qemu process, nice thought displayed after hyprland crash: "A day without Hyprland is a day wasted"
    • Added rcu registration and set the thread sync event
    • It works now 🥳🥳🥳
• This should conclude step 4 basic implementation, assuming refactoring and trace points can be added later on
• Now for today we can move on to step 5 or refactor etc, I think I will add trace points all together for everything by verifying the other pathways
• Let's think what step 5 should entail, and who should cleanup, send kill signals etc, we can begin implementation tomorrow
• Oh no before that I tried running tests and some are failing, let's try resolving them
  • Uhh I have no idea what I would have touched that caused the error, it is in some test the status is failed when it should not have
  • The errpr is Error reading dirty bitmap.
  • Ahh I think the error is that what used to happen was we allocated a temp bitmap for mapped ram load but I moved that to a bitmap in RAMBlock, however it was only allocated in snapshot case so we need to allocate it always
  • Same error again 🥺, ... I didn't rebuild 😅, ... All pass 😁

Day 24

14 June 2026(~8hrs)

Target: Read more about how postcopy manages cleanup and go on with step 5 so we have a proper cleanup

• First our target is to read on how postcopy waits for entire RAM to load and then calls for cleanup
  • We are clear that the fault thread will exit by polling on eventfd, so we need to find where the eventfd is set to signal fault thread to quit
  • A point I thought of was when is the eager load thread launched, should it be done before launching the VM, it should not affect correctness but there might be some optimization?
  • postcopy_fault_thread_notify function asks fault thread to die and as expected it is called in postcopy_ram_incoming_cleanup
  • That is called in the migration_incoming_state_destroy, and the only call to it that makes sense for us is in postcopy_listen_thread_bh
  • I think I get the solution, after the loading is complete in the eager load thread, we shcedule a bh cleanup and exit, that will take care of cleaning up stuff
  • Looks like we can schedule postcopy_listen_thread_bh as it does the stuff we wanted it to do even thought the name here is not that great
  • I think it is better to have our own new function for this purpose, we can look into merging them later on
  • Looking into the function, it does everything we need
    • We dont need multifd cleanup but the qemu_loadvm_state_cleanup is great to clean all bitmaps in RAMBlocks
      • Ahh, we bypassed it here and I think we shouldn't do that as that complicates stuff, oh maybe bypass cleanup in original path itself
      • Hmm, I cant find where It caused problems, let me go back in logs
      • Couldn't find it, let me see why I bypassed that ... ok so at that time it was being called cause of some failure, it should not be actually, let's fix it up
      • Done should not free up all bitmaps that exist
    • It cleans up the from_src_file nicely
    • Looks like it cleans everything though I am not sure if it cleans the tmp pages, let's try running it now ... runs good
  • Now I think everything is cleaned up, though I would like to be more sure by using GDB to verify everything is set to null after we are done
  • Maybe will add tracepoints too to verify everything goes as planned
  • Found one thing, I did not join the eager load thread
  • Hmm because of some reason It doesnt look like the fault thread exited so we must signal that too
  • postcopy_incoming_cleanup first tries to join, listen thread and calls postcopy_ram_incoming_cleanup, which then calls and cleans the fault thread but GDB told the trhead still existed no idea why
• I think now is good time to start working on refactoring and adding tracepoints on the new code, took me very less time than expected for cleanup
• While writing comments found a bug:
  • Currently I use place_source = g_malloc(TARGET_PAGE_SIZE); which has no problem but if I remove it and use place_source = &mis->postcopy_tmp_pages[channel], it has some problem as screen shows static
  • There is definately some problem with use of tmp pages, malloc has everything working great
  • I am dumb, I set the pointer to postcopy_tmp_pages struct and not the page, should work now, ... It does
• Looking through all the diff, I wrote some comments to help anyone trying to follow the code
• I am confused about what to return etc, sometimes at some functions return value is returned on error, in some it is carried and in some error always returns -EINVAL
• Now I need add tracepoints and maybe look into why when where what error to return
• There is another concern that has come up, I got the code checked with AI if anything was wrong and it came up with this:

  • currently in function postcopy_mapped_ram_load_page, we read at offset rb->pages_offset + (page << TARGET_PAGE_BITS)

  • However if we consider that zero pages are not even stored in memory, these offsets would certainly break

  • The solutions are something that would be considerably ugly:

    1. Replace the bitmap with an array of offsets for every page, say what ever it is for non zero pages, -1 for zero pages and -2 for non pending ones, which would need us to do atomic writes
    2. Maintian something like a prefix sum, use binary search on the list of page set offset pairs
    3. Find offset when required by iterating of bitmap

  • Analysis:

    | Metric | Idea 1 | Idea 2 | Idea 3 | |:------------------------------:|---------|--------------|--------| |Extra Space(beyond the bit maps)|$O(n)$ |$O(k)$ |0 | |Time complexity(per page) |$O(1)$ |$O(\log (k))$ |$O(n)$ |

    • $n$: Number of pages, scales with size of RAM
    • $k$: Number of continuous chunks of non zero ram, worst case $O(n)$, however in usual case I think it should be small

  • Considering that qemu prefers bitmaps over queues I think idea 1 makes more sense but I think Idea 2 is performant too, Idea 3 is obviously bad

  • Doubt is, can the small memory footprint overpower the small lograithimic latency introduced for loading each page, this needs discussion

  • If we need better performance after discussing with mentor we might go on to use a radix tree as I hear it is best of both worlds

• Let's solve this bug:

  • First I thought I could get rid of other two bitmaps but I can't, address is 64 bit even if some bits are for flag purpose, we can't use any of that for our purpose

  • So we introduced the page_file_offset in all ramblocks, init it, free it, populate it and use it

    • Init and free is same as where other bitmaps are used

    • For population we had to do a few changes, firt of all we need to call read_ramblock_mapped_ram, and inside that instead of loading the data we loop over all non zero pages and set the offsets

      /* Store the offset */
      for (size_t page = set_bit_idx; page < clear_bit_idx; page++, offset+=TARGET_PAGE_SIZE) {
          block->page_file_offset[page] = offset;
      }
      

    • As for usage we directly use rb->page_file_offset[page] as offset for loading in

  • Implementing this, running qemu we get seg fault after a decently long time

    • Let's see using GDB where does it reads bad values
    • Ahh found the error, as we dont read we dont update read which causes error as reading nothing is bad
    • Updated it so that the read logic is not ran

  • Still didn't run but info migrate revealed something:

    failed (load of migration failed: Invalid argument: error while loading state for instance 0x0 of device 'kvm-tpr-opt': post load hook failed for: kvm-tpr-opt, version_id: 1, minimum_version: 1, ret: -1)
    

  • Most likely that it reads zero values, let's go put some debug code back in fault thread

  • Debug log of the error:

    load page reporting to load a page from pc.ram and channel is 1
    address 0x7f7be7ecb000, offset 0xcb000
    non zero page to load
    file offset is 0xcb000, note that page offset was 0x100000
    Read values are : 
    00 00 00 00 00 00 00 00
    --- Call ended ---
    

  • Problem is clear, I had to use rb->pages_offset + rb->page_file_offset[page] as the offset, but maybe I should change that during the population of the table

  • Done now during population we store the offset with the pages offset

  • Works now 🥳🥳🥳

• I spent some time toying with the VM and I think there is some error, it shows a bunch of logs that disk failed etc, let me see if I can pinpoint error
• I ran it with precopy and same errors, so most likely error with my boot

Day 25

15 June 2026(~7hrs)

Target: Add tracepoints and maybe look into how to separate the patches

• I think most trace points can be copied and slightly modified to match the existing logical paths
  • I could not really find where all to add tracepoints but thought this: currently I return NULL in eager load thread however it should also signal the fault thread to exit and cleanup no matter what
  • Added Two:
    1. Entry for eager load thread
    2. Exit for eager load thred
  • I thought about adding one to postcopy_mapped_ram_load_page but it is not very useful as postcopy_place_page already has tracepoints with required data
  • Most of the critical code is reused, so couldnt find a lot of place for placing trace points
• Now that we have time let's see how to separate patches
• I spent a lot of time formatting patches, rebasing

Day 26

16 June 2026(~8hrs)

Target: Update the page offset calculations and add proper state management

• First in yesterday's meeting one thing came up is that mapped RAM has zero pages empty space so the offsets are calculatable in $O(1)$
  • This means the page_file_offset thing I did is useless
  • First let us revert those things
• Today I need to get a new image, there is some problem with my debian that throws device error, need a new one for testing
• Let's go make a new image again
  • Making one debian 13 and one fedora 44 KDE
  • Ugg, they are so slow to boot without KVM 😭
  • That's it I am not using fedora, it is too slow wihout KVM, let's just have debian
• I spent hours trying to configure networking on the image but it seems I missed out installing libslirp
• It is so slow to boot these VMs 😭😭😭
• Found the problem! the error is there because when migrate_incoming is used I have not specified a disk, so the snapshot is stuck with data of a persistant storage that no longer exists
• Finally after hours I am able to run a fedora 44 KDE with disk consistancy so that no errors come up and it doesnt freeze
• All changes to the offset calculation have been rectified
• Let's think about making qemu_get_buffer_at safe:
  • One idea that was given in last meeting was to just giet rid of error handling
  • I am not really sure about if that would be best option
  • I think a better option would be to make the error handling atomic
  • But as the callers of qemu_get_buffer_at:
    • parse_ramblock_mapped_ram
    • postcopy_mapped_ram_load_page
    • read_ramblock_mapped_ram
  • All these already take care of error handling
• Now what remains is the state updates, I think I dont have time today, we can do that first thing tomorrow
• We have a good enough test and it seems to work really good, tomorrow we setup the states, test/play with VM for quite some time, write a good cover letter and send out the RFC

Day 27

17 June 2026(~8hrs)

Target: Add proper state management and work on RFC cover letter

• Let's first see who all set the states

  • Currently all that happens is:

    migrate_set_state new state setup
    migrate_set_state new state active
    migrate_global_state_post_load loaded state: paused
    

  • We need a much better state system than this

  • Let's keep it simple and think, we need NONE at start which is good, then it goes to SETUP during reading for RAMBlock, followed by ACTIVE when we start the VM and completed when eager thread exits

  • However what about postcopy status? I think it might be better to use MIGRATION_STATUS_* from MigrationStatus

  • Let's look into qapi/migration.json for what each of these states exactly mean

• I added something like NONE -> SETUP -> POSTCOPY_DEVICE -> POSTCOPY_ACTIVE -> COMPLETED

  • It wont print in trace no Idea why?
  • Ahh, I think the states I expect are not there someone activates the state early on
  • Added a skip to setting state to active in case of fast snapshot load
  • There is some error in loading and figured out that this introduced a new bug in ram_load

    • ram_load uses postcopy_is_running() to decide weather to call ram_load_precopy() or ram_load_postcopy()

    • As postcopy_is_running() uses states it returned true as per new state regime which causes code to use ram_load_postcopy which is wrong as it does not support mapped ram.

    • The solution I came up with is instead of directly deciding on postcopy_is_running(), check using:

      bool load_using_postcopy = postcopy_is_running() && !migrate_fast_snapshot_load();
      

    • It works now

• Now that the state updates are all consistant(info migrate on hmp also works stating correct state), let's start working on the cover letter

• I wrote the cover letter and AI brought to my notice that failure is not handled properly, and it isnt

  • If disk fails postcopy_mapped_ram_load_page returns -1, and the thread just exits not signaling anywhere that we reached a critical problem

  • I think I asked about this in last meeting and forgot, but mentor replied to use asserts because if disk read fails we cannot recover and crashing is valid move

  • Changed the postcopy_mapped_ram_load_page function to assert that read == TARGET_PAGE_SIZE

  • Hmm in case of error, the VM hangs for some time after showing message and the core is dumped, looks good

  • Just to note, I used this to simulate drive failure in qemu_get_buffer_at

    static int ctr = 0;
    if (ctr++ == 20000) {
        return 0; 
    }
    

• Let's send the RFC it to mentor now

Day 28

18 June 2026(~5hrs)

Target: Look into adding postcopy blocktime support

• Sent off the RFC patches to mailing list, now let's wait for community feedback

• Work on adding postcopy blocktime support can be looked into

  • It looks like the mark_postcopy_blocktime_begin function paired wit mark_postcopy_blocktime_end should do all the heavy lifting and we can actaully call all this pretty easily

  • Looking into mark_postcopy_blocktime_begin I kind of feel there is something off, it asserts that the received bitmap has ont received address but it might happen that it is loaded in meantime

  • Ahh, maybe the page_request_mutex is what saves it from such problems

  • Just looking end is called by place page so that is happy ending, only the fault thread is where we need to alter stuff

  • It does look like the page_request_mutex locks the recevied bmap andother blocktime infrastructure

  • I think the start should also be in a lock

  • I wonder the use of macro WITH_QEMU_LOCK_GUARD vs manual taking lock and releasing, which is better as migrate_send_rp_req_pages uses macro and postcopy_place_page uses manual acquire and release

  • Macro feels cleaner so let's use that

  • Should work, let's try

  • Hmm the output is no different and info migrate just shows Status: completed

  • There must be some problem in setup etc, let's go deeper

    • Using tracepoints on blocktime there is some problem as in all the calls the cpu is -1
    • As per mark_postcopy_blocktime_begin cpu == -1 is when there is some kworker hepling KVM maybe like the one we used to load in vapic devices
    • But then why are all of them cpu -1 and why does the entire RAM fault
    • There are some with different cpus too but are quite rare
    • Can't even use GDB as KVM causes it to just not run 🥲
    • I tried doing some printing using fprintf(stderr, "...") to see does it not fill in fields and prit shows clear output that it does

  • Found out it requires info migrate -a to get everything 😅

  • Yup it runs and output seems good

    Postcopy Blocktime (ms): 0
    Postcopy vCPU Blocktime (ms):
    [4, 3, 2, 4]
    Postcopy Latency (ns): 7643
    Postcopy non-vCPU Latencies (ns): 7574
    Postcopy vCPU Latencies (ns):
    [8660, 9364, 7463, 9116]
    Postcopy Latency Distribution:
      [     1 us -     2 us ]:         11
      [     2 us -     4 us ]:        269
      [     4 us -     8 us ]:      21015
      [     8 us -    16 us ]:       6802
      [    16 us -    32 us ]:        481
      [    32 us -    64 us ]:         99
      [    64 us -   128 us ]:         20
      [   128 us -   256 us ]:          6
      [   256 us -   512 us ]:          4
      [   512 us -     1 ms ]:          0
      [     1 ms -     2 ms ]:          1
      [     2 ms -     4 ms ]:          0
      [     4 ms -     8 ms ]:          0
      [     8 ms -    16 ms ]:          0
      [    16 ms -    32 ms ]:          0
      [    32 ms -    65 ms ]:          0
      [    65 ms -   131 ms ]:          0
      [   131 ms -   262 ms ]:          0
      [   262 ms -   524 ms ]:          0
      [   524 ms -    1 sec ]:          0
      [    1 sec -    2 sec ]:          0
      [    2 sec -    4 sec ]:          0
      [    4 sec -    8 sec ]:          0
      [    8 sec -   16 sec ]:          0
    

• I looked somewhat into hugepages support, doesn't look like a very big task but let's see

Day 29

19 June 2026(~4hrs)

Target: Verify blocktime support

• I sent off the output to mentor for verification and he put a quite valid point that there might be a lot of page cache hits

• So using sudo sync && echo 3 | sudo tee /proc/sys/vm/drop_caches I cleared the cache and reran the tests and there is a much different picture this time

  Postcopy Blocktime (ms): 0
  Postcopy vCPU Blocktime (ms):
  [128, 117, 111, 97]
  Postcopy Latency (ns): 153571
  Postcopy non-vCPU Latencies (ns): 149885
  Postcopy vCPU Latencies (ns):
  [234229, 225562, 195604, 209678]
  Postcopy Latency Distribution:
    [     1 us -     2 us ]:         25
    [     2 us -     4 us ]:        370
    [     4 us -     8 us ]:       3938
    [     8 us -    16 us ]:       3599
    [    16 us -    32 us ]:        520
    [    32 us -    64 us ]:        192
    [    64 us -   128 us ]:      18643
    [   128 us -   256 us ]:       6199
    [   256 us -   512 us ]:       2768
    [   512 us -     1 ms ]:       1096
    [     1 ms -     2 ms ]:        486
    [     2 ms -     4 ms ]:         78
    [     4 ms -     8 ms ]:          5
    [     8 ms -    16 ms ]:          0
    [    16 ms -    32 ms ]:          0
    [    32 ms -    65 ms ]:          0
    [    65 ms -   131 ms ]:          0
    [   131 ms -   262 ms ]:          0
    [   262 ms -   524 ms ]:          0
    [   524 ms -    1 sec ]:          0
    [    1 sec -    2 sec ]:          0
    [    2 sec -    4 sec ]:          0
    [    4 sec -    8 sec ]:          0
    [    8 sec -   16 sec ]:          0
  

• This does look more logical considering that it is a bi modal distribution, most likely the higher mean is for non zero pages and the lower mean is for zero pages loaded directly

• Nevertheless the performance is really good if we consider that precopy can take a few seconds to load

• Before sending this to mailing list let me just run make check once ... uhh why is taking so much time today?

• A few minutes later 🤦, I did not use -j, so it was running on one thread.

• Formatted the patch to it and sent off as reply to cover letter.

Day 30

22 June 2026(~2hrs)

Target: Look into huge pages

• First basic idea is to simply replace TARGET_PAGE_SIZE, with actual page size for that RAM Block
  • Replaced all occurances with qemu_ram_pagesize(rb) in postcopy_mapped_ram_load_page and changed the iteration step size in ram_block_load_eager
  • One ugly thing was calculating page using rb_offset, which was done using bitshift on rb_offset by TARGET_PAGE_BITS, but we dont have anything like qemu_ram_pagebits
  • Workaround used was rb_offset / qemu_ram_pagesize(rb) which is correct but division is way slower than bitshift(though all this can be excused considering bottleneck is serving of SSD)
  • That reminds me while reading about SSDs a point was that the action of sending request to SSD for reading was also a significant operation, could we improve on that(maybe increasing granularity)
  • Now comes the hard part to test it 😤
• Setting up environment to test, mostly using AI

  • First running this command to allocate the huge pages

    echo 8500 | sudo tee /proc/sys/vm/nr_hugepages
    

  • Uhh my mem usage is through the roof now

  • Expanding on the -m 16G to -m 16G -object memory-backend-file,id=mem,size=16G,mem-path=/dev/hugepages,share=on -machine memory-backend=mem

  • It says Permission denied

  • Ok did sudo chmod 1777 /dev/hugepages and now it should run

  • It does not, why?? ... Ahh I need to remove the flag -incoming defer

  • Works now!!

• Testing:
  • So I think it flashed the snapshot and instantly restarted GRUB, there must be some error as it loaded too fast too
  • Firefox stored the pages I opened so it was definitely a crash
  • There was error thrown etc, Let's test it using migrate tracepoints
  • Block time latency distribtion shows around 1117 faults were served
  • Trace point says, migration_cancel? Let's trace to depth what exactly happens

Day 31

23 June 2026(~5hrs)

Target: Work on RFC feedback

• Starting on reply to patch 0, squash this in patch 4, let's rebase stuff
  • As directed reading this, understood brace postions.
  • The clang format file I have needs updating, let's do that later
• Done with that, let's go on to patch 1:
  • First we replace the nonzeropages with file_bmap, I was not really sure about this so used a new field.
  • This cannot be done with received bmap because of the difference in begin received by the VM vs claimed by a thread to load.
  • I think using them for same purpose can lead to horrible race conditions, and a lot of code will be changed.
    • postcopy-blocktime will have lot more problems too as it can happen a fault occurs but the eager thread is loading it
    • if received map is false as it is not in vm, fault thread will try to load it in again
    • if the eager thread was successful the VM would start running and might write data which we overwrite - BAD
    • if received map is repurposed to the pending map, it was true hence postcopy-blocktime would not track it which is not breaking but bad as that is valid fault
  • This should prove that having any one map is not really good
  • This reminds me, if we use file_bmap, can that break other cases as we write in it? Need to test it
  • I could try clean up in qemu_ufd_copy_ioctl
    • Removing the lock for file case might not be a good idea cuz there can be many threads calling it
    • Page request can be skipped, wait I think I need to re-analyze the use of receivedmap
    • Hmm, I think there can actually be a race condition here:
      • Fault at page $x$ so fault thread catches it
      • At just the next moment the eager thread starts loading page $x$ loads it in and sets calls end marking the receivedmap
      • Now as fault thread continues to call mark_postcopy_blocktime_begin even with lock the assert would fail
    • I think that assert should just be a return to prevent race conditions
    • That might be better to have a stand alone patch or maybe integrated in prep
    • But we need to keep different maps as otherwise it may happen that page fault being served by eager thread is not registered
    • The cleanup I can think of is wrapping in macro for lock and skip the page_requested part, this needs discussion
  • We can surely change keep the line in qemu_get_buffer_at
  • Added the point to add length checks, and maybe make a different patch for that
  • Moving to g_clear_pointer, not sure what that does ... oh it just frees and sets pointer to NULL, good
  • Ugh some clang format changing stuff I never meant to
  • Followed the migrate_mapped_ram nitpick
  • Will have to work on the boolean logic for using postcopy methods for ram load, added to TODO
• Patch 2:
  • Will look into how to format auto doc, using this
  • Ahh I see clang format destroyed my newlines 😠
  • Dropped the void* host variable
  • What is reverse christmas tree 🥺 ... a quick google search later ... done!
  • The error handling needs time to think on, it goes to TODO, the stuff might be deeper than it looks let's look into floating error to top later on
  • Dropped the zero page comment
  • Will revert the comment in postcopy_ram_fault_thread
  • Changed the hardcoded channels to names
• Patch 3:
  • Huh nothing here, so my eager thread is good? Lemme just change the channel number from hardcoded value
  • Wait a sec, there is a reply on devel archives but none in my inbox, no idea what happened, let's read from archives then
  • Changed the name of eager thread to snapshot load
  • Moved state registration from postcopy_ram_eager_load_bh to end of eager thread
  • Oh I need to update stuff with postcopy_listen_thread_bh, let me write that in TODOs, cancel last thing
  • The point of squashing this patch with its user feels kind of weird as that might make it very very long but for ease of review let's merge patch 3 and 4
• Patch 4:
  • Allowing staying active status for postcopy
  • Following changes make sense too
  • Will have to do git gymnstics for moving the caps check etc
  • I knew there was something wrong with returning just -EINVAL lets add that to TODO(I already had it there)
  • Makes sense to move around logic used only once but need to make sure that the prep etc are all done properly
• Patch 5:
  • Just need to move it to first
• Tomorrow we move onto to resolving the TODO and reply to some points I doubt

Day 32

24 June 2026(~5hrs)

Target: Work on RFC feedback

• Let's start by moving stuff around, first the changes in options.c need to moved to last patch(which we will later move to first)
  • I think we can just copy paste stuff?
  • Well it turns out just copy paste in git is HELL but somehow managed it
• Let's go linearly rebasing the small changes over all patches one by one
  • Patch 1:
    • Getting rid of nonzeropages and while we are at it use g_clear_pointer
    • Making the different patch for qemu_get_buffer_at would need deeper git gymnastics, let's push that to TODO
    • Found a point for hugepages, we need to allocate the bitmaps for corresponding size, though should not affect correctness
    • Umm we need a separate patch for these filebmap changes too
    • Did the boolean changes
• I kind of locked in fighting with git over little stuff and did not write any notes 😔

Day 33

25 June 2026(~6hrs)

Target: Continue working on RFC feedback and move to the TODOs

• Continuing from yesterday, lemme do the minor git changes first, writing them all here is too tedious
• Now that all minor changes are done let's once test it out
• Tests were successful
• I tried running snapshot load and it just hangs, let's find the bug:

  • Oh it threw error

    qemu-system-x86_64: ../migration/postcopy-ram.c:1136: mark_postcopy_blocktime_begin: Assertion `!ramblock_recv_bitmap_test(rb, (void *)addr)' failed.
    

  • It never occured before? But it did now after I had that figured out

  • But if that error does not occur it runs fine

• Now let's work on the other points
  • First reordering last patch to first ... hmm that was fast
  • Now we make separate patch for file_bmap changes
  • Finally done all changes for patch 1
  • Pending points are length checks and some qemu_ufd_copy_ioctl clean up
• qemu_ufd_copy_ioctl needs some thinking

  • First janitorial change is using WITH_QEMU_LOCK_GUARD but not a big thing

  • one point is that we only need this in case of postcopy-blocktime and earlier it ran always which can be slow as everyone takes the lock and does the search for no absolute reason

  • In any case if blocktime is off we allocate and populate the recv map for no reason

  • Is it that only blocktime uses this map ... oh no a lot of functions use it

  • I dont think it can be repurposed let us think about it again:

    • If we shift it to be the pending bmap, then postcopy-blocktime wont be able to register faults for any pages that are being loaed in eager thread but fault occurs
    • In current semantics using it for communication between threads is useless

  • I think it is like we need both and it is not possible to support blocktime without both

  • Let's keep it for v2

  • I am thinking that do we need the page_requested ... ok we dont need it for our usecase other code still uses it

  • Hmm, doesnt look good I think this will silently break stuff we need to update this bitmap and requested pages etc in most cases

  • Found this commit(d2a81ca8c6fbe6ed691889d953d0b5fe2c7e4671) message explaining why the mutex is used

    migration/postcopy: Push blocktime start/end into page req mutex
    
    The postcopy blocktime feature was tricky that it used quite some atomic
    operations over quite a few arrays and vars, without explaining how that
    would be thread safe.  The thread safety here is about concurrency between
    the fault thread and the fault resolution threads, possible to access the
    same chunk of data.  All these atomic ops can be expensive too before
    knowing clearly how it works.
    
    OTOH, postcopy has one page_request_mutex used to serialize the received
    bitmap updates.  So far it's ok - we don't yet have a lot of threads
    contending the lock.  It might change after multifd will be supported, but
    that's a separate story.  What is important is, with that mutex, it's
    pretty lightweight to move all the blocktime maintenance into the mutex
    critical section.  It's because the blocktime layer is lightweighted:
    almost "remember which vcpu faulted on which address", and "ok we get some
    fault resolved, calculate how long it takes".  It's also an optional
    feature for now (but I have thought of changing that, maybe in the future).
    
    Let's push the blocktime layer into the mutex, so that it's always
    thread-safe even without any atomic ops.
    
    To achieve that, I'll need to add a tid parameter on fault path so that
    it'll start to pass the faulted thread ID into deeper the stack, but not
    too deep.  When at it, add a comment for the shared fault handler (for
    example, vhost-user devices running with postcopy), to mention a TODO.  One
    reason it might not be trivial is that vhost-user's userfaultfds should be
    opened by vhost-user process, so it's pretty hard to control making sure
    the TID feature will be around.  It wasn't supported before, so keep it
    like that for now.
    
    Now we should be as ease when everything is protected by a mutex that we
    always take anyway.
    
    One side effect: we can finally remove one ramblock_recv_bitmap_test() in
    mark_postcopy_blocktime_begin(), which was pretty weird and which also
    includes a weird (but maybe necessary.. but maybe not?) operation to inject
    a blocktime entry then quickly erase it..  When we're with the mutex, and
    when we make sure it's invoked after checking the receive bitmap, it's not
    needed anymore.  Instead, we assert.
    
    As another side effect, this paves way for removing all atomic ops in all
    the mem accesses in blocktime layer.
    
    Note that we need a stub for mark_postcopy_blocktime_begin() for Windows
    builds.
    
    Reviewed-by: Fabiano Rosas <farosas@suse.de>
    Link: https://lore.kernel.org/r/20250613141217.474825-3-peterx@redhat.com
    Signed-off-by: Peter Xu <peterx@redhat.com>
    Signed-off-by: Fabiano Rosas <farosas@suse.de>
    

  • Blocktime needs to stay in lock

  • A change required is instead of

    assert(!ramblock_recv_bitmap_test(rb, (void *)addr));
    

    We need to simply return as a race condition can occure

    • Fault occurs at page $x$ and uffd signals fault thread which goes on to try marking it
    • In mean time say eager thread loads the page and gets the page_request_lock first then it can update received map
    • Now as fault thread goes on it will see that page $x$ is actually present causing it to fail on assert

  • I am thinking why is it not a problem with normal postcopy?

  • Ahh so other caller is migrate_send_rp_req_pages which calls it if the bitmap shows page is missing and it is in lock so loader cant update it

  • Then it seems better to check that before calling the function

  • I hate the use of msg.arg.pagefault.address everywhere, I think we should be able to use local vars to make it readable(I think the compiler will optimize to same code)

  • All the test do pass but I am not sure how good the code quality is

  • Umm, testing myself I see all zeros ... oh my bad forgot !

  • Works great now

  • I doubt if I should wrap the old logic in !migrate_mapped_ram but if the g_tree is empty the search is $O(1)$, fast enough(maybe faster than a wrapper)

  • Yup used a temp assert for number of nodes being zero and no error occured in fast snapshot load

  • Ah, end checks for if postcopy blocktime context is NULL and exits. So our cleanup is not really something significant

  • Only change is that we need to check the recv map before actaully calling mark_postcopy_blocktime_begin()

  • Ok works now ...

• Let's continue tomorrow

Day 34

26 June 2026(~6hrs)

Target: Work on RFC, proper error handling

• First let's look into replied by Fabiano on the RFC
  • Reply to patch zero queries what happens if a savevm happens, I think postcopy migration should handle that part, I mean migration should have similar problem too
  • The point that we might not need the term fast snapshot at all is valid however it is readability that might be severely affected, or we might need to add comments explaining these parts
  • For third patch I think I need to explain the feature in more detail but I think the following points should answer the questions:
    • If there is no eager load thread and only fault thread we would load only faulting pages. So a page will be loaded if and only if the VM uses it. This causes the migration to wait for VM to use all pages which is indefinite.
    • This could lead to very long load times(even if blocktimes are low)
    • This feature can work without it however considering that migration would end only when all pages have been loaded and the guest may not access all it indefinitely locking us in migration
    • If someone say runs a guest for some time and now wants to save it, without eager thread it might still be waiting on loading all pages in migration which would be a problem
  • For patch 4 the bh causing bugs, I think I have that point to take care of. Will look into using bh functions. And as for the pair of mapped ram and postcopy ram enabling other migrations I dont think so because I analysed the code flow compeletely for that case before coding, it led to errors.
• Let's continue work on v2:
  • Updated and verified the formatting for comment on postcopy_mapped_ram_load_page, should be correct now
  • Now the error handling:
    • If there is an error, ie read != page_size ther was something wrong
    • I think considering the suggestion to use bool func(..., Error **errp), we should follow that atleast for postcopy_mapped_ram_load_page
    • Spent some time reading error.h
    • Ugh fault thread does not have an errp, So I declared one
    • I think it would better if postcopy_place_page and postcopy_place_page_zero have one too but they are called by ram_load, which just uses error_report
    • Maybe just set error in postcopy_mapped_ram_load_page
    • Now we need similar changes to eager thread path
    • Ahhh but a problem we cant pass errp to ram_block_load_eager as opaque is meant for MigrationIncomingState, or is it 🤔
    • We could have opaque as the Error**, but does not feel right
    • Thing is we can't really change the signature of ram_block_load_eager as we use the iteration function
    • Let's look more into that once we have more idea about bottom half handling
  • Bottom half is something I never fully explored but I need to, so let's do that first
    • Lets list out all the bh operations:
      1. postcopy_ram_eager_load_bh which we added
      2. postcopy_listen_thread_bh(only other one in postcopy-ram.c)
      3. loadvm_postcopy_handle_run_bh
      4. snapshot_load_job_bh(where did this come from?!)
      5. snapshot_save_job_bh
      6. snapshot_delete_job_bh
      7. process_incoming_migration_bh
      8. migration_cleanup_bh
      9. bg_migration_vm_start_bh
    • That covers all that shoud affect us, the snapshot_* seems to be for the hmp/qmp snapshot actions
    • Let's read first into migration_cleanup_bh as that seems useful
      • It seems to work on the outgoing migration state which I dont think we even work with it(it might just be NULL or empty)
    • postcopy_listen_thread_bh is one that really looks directly usable
    • We should be able to rename at as say incoming_migration_cleanup_bh like migratino_cleanup_bh or maybe postcopy_complete_bh as recommended on mailing list
    • It looks good in the form that it also would report error
    • Ok so I did some changes I think would be really good like updating state changes and much better error handling
    • It won't run now 🥺 ... trace shows as soon as eager thread entered it tried to load a page that was already loaded by fault thread?
    • Oh no it was not at all loaded
    • Got it I changed functions to return true on success but it expected 0 on success
    • Now it works, I think we can run the tests to verify it wont break anything else, It works 🥳

Day 35

27 June 2026(~2hrs)

Target: Keep working on RFC v2 and look into hugepages support

• First of all let's add length checks while parsing the RAMBlock headers

• After looking around and adding the length checks at bunch of places(and tearing it down because it did not look right) I think it should not be that complex

• Added check in parse_ramblock_mapped_ram:

  if (length > block->max_length) {
      error_setg(errp,
                  "mapped-ram header length %" PRIu64 " exceeds "
                  "RAMBlock(\"%s\") max_length %" PRIu64,
                  (uint64_t)length, block->idstr, (uint64_t)block->max_length);
      return;
  }
  

• Now we can think about qemu_loadvm_state error handling and logic

  • Wait a minute I just saw that postcopy_ram_incoming_setup has a local_err but it's caller has errp so we can just pass that on to it for a better error handling
  • Ok so I have tried cleaning up some of it so that everything sets the errp object and not do local error reports
  • Really nice all the tests run too 🥳
  • Let's put them up in git history

Day 36

28 June 2026(~6hrs)

Target: Keep working on RFC v2 and look into hugepages support

• For the error cleanup I think there is more things I can do better
  • First replace error_setg with error_setg_errno as that would be cleaner in postcopy_temp_pages_setup
  • It also does not need to return -errno as the called never checks the vaule, we have the errp set now it should be enough
  • After a bunch of testing and git actions I have it set
• Now let us look into moving around code from qemu_loadvm_state to process_incoming_migration_co
  • Honestly I dont think I understand correctly what should be done.
  • I think the point is that fast snapshot load will use the block in qemu_loadvm_state only in one case when coming through process_incoming_migration_co so code can be moved there
  • However there is a lot of common code, and so adding all that to process_incoming_migratino_co like this or as a function violates DRY
  • Maybe we can have a common setup function etc
  • Wait what is this mis->load_threads?
  • it just has a create function, a destroy function and a wait?
  • Ok it has a submit too but onl a hw, vfio uses it? ummm no there are other usecases too for switching precopy and postcopy, any case I dont think we need this in fas snapshot as we use just two threads
  • Wait a minute what is this loadvm_co, no one seems to use it?
  • I dont think removing it would ever cause a problem as no one actaully uses it, let's discuss that in a meeting
  • Maybe we can just sandwich the call to qemu_loadvm_state with other calls we have
  • It works but I dont like the name of function qemu_loadvm_state_postcopy, it just feels so ... generic when what it actually does is way different
  • I think loadvm_postcopy_handle_run does a similar job, we will just need to add the starting eager thread part
  • However loadvm_postcopy_handle_run is a static function and is one of the functions called for different listen thread commands
  • Maybe it is better to have them separate function and not disturb the consistancy
• Let's get back today to hugepages
  • I tried running it back and the same problem occurs, restarts right after boot
  • Again 1117 page faults
  • I dont seem to get anything useful from traces, I think it might be better to see why VM exits ... and it has nothing
  • Tried running with precopy that works, so there is some problem with postcopy, the disk image etc are good
  • I tried putting up some traces and wanted to compare the huge pages and normal image and suddenly my entire system crashed
  • Later I realized after giving around 16GB to hugepages I did not have enough memory to give to normal boot
  • Found a difference, the huge pages image was made when it was active, the normal one I think I paused before actaully saving it
  • But reading aroudn in global_state.c it should be suspended till start was called, I think we need some more detective gameplay
    • Let's use trace on runstate ... nah they are same
    • I see that resume_all_vcpus is called thrice when loading in, however it is called only once in precopy
    • So calls to main_loop_should_exit cause the restart, somehow reset was requested which would eventually cause the restart
    • Checking in depth 5 times is requests a reset, I wonder why
    • reason is SHUTDOWN_CAUSE_GUEST_RESET with reboot action reset, I need to go on to qemu documentation
    • Not sure what the error is?

Day 37

29 June 2026(~7hrs)

Target: Complete minor style points on the v2 and look into tests

• First let's start by renaming postcopy_listen_thread_bh to postcopy_incoming_complete_bh ... done
• Now I think we need to rewrite the commit messages, this might take some while
• Finally I think I am done with most of RFC v2, with changelog and all, now let's get into adding a test as that was part of last week plan
• Exploring Testing:
  • I think the first file that naturally comes up is tests/qtest/migration-test.c:
    • Smaller than expected?
    • hmm it uses g_test, now is that google test or gnu test or gnome test framework?
    • Ok it is GNOME, but why not google test, I heard that is really good too, maybe for C this is better or maybe that's just how QEMU rolls
    • Then tests are added and we run them, simple?
    • Let's go into migration_test_postcopy and see what that function does
      • First it add some smoke tests and then some seemingly complex ones, all using migration_test_add, after checking for uffd though
      • Let's dive into migration_test_add now
        • It takes in path and a void function that takes in a name and args
        • A new test object is allocated with its function and name specified
        • Then this function comes up qtest_add_data_func_full followed by pushing this test in a tests queue
      • Looking into qtest_add_data_func_full, it is just a wrapper to prepend architecture to path
      • So the major part should be the function passed to this
      • Let's cehck test_postcopy then
        • it just calls test_postcopy_common without caring for the name it is given
        • Then it just does a prepare, start and complete
        • Looking in it seems it uses qmp and asserts on the outputs matching which seems fine, I need to learn about qmp usage
      • Hmm in all the cases it just seems they add the capability in the args and run test_postcopy_common
    • Maybe I should look into testing of mapped RAM
      • These should be in precopy, ... oh it is in file tests
  • It seems like it would be very difficult or simply very easy, either just activate mapped RAM and let it go or build from scratch
  • After enough trials it seems it doesnt run because it uses unix ports, we need to go deeper than using the common postcopy as that seems to use only unix ports and not files
  • Maybe I should look into QMP, then I will have to write my own arguments for creating transport etc
  • Hmm so postcopy common no matter what declares a socket as transport we will have to look into using a file
  • there would only be a to state and on that we execute migrate incoming from a file, maybe there is precopy that uses file based testing

Day 38

30 June 2026(~6hrs)

Target: We can go deeper into tests

• As from the discussion in meeting yesterday I should look into tests like test_multifd_file_mapped_ram_live
  • So what it does is sets multifd and mapped ram to true and tests files
  • Let's first look into what test_file_common does
    • It begins with making two VMs from and to as per the args which we discussed is quite a problem as we need to set fast snapshot load only on one thread
    • One idea I had was to change the args so that first we have only_source and then only_target
    • uri if not already present will be set to whatever the temp file is
    • I dont know what hook is so need to look into that
  • Does seem rest of function does simple things we can reuse, let's try repurposing this function for our test
    • So we are just going to do for now is only run migration/precopy/file but update code in function for now
    • Ok so after adding the code to individually make the source and destination it now throws error for bad file descriptor
    • Looks like from side fails to open file?
    • File seems good, is it trying to load in from file? it should be saving right?
    • Ahh I think I see the problem, the capabilites of from are set no matter what if the args has only_source or only_target
    • I think this should be updated
    • After bunch of updates to core function to allow only a few changes i was able to get the test running with success
    • hmm the tests runs but integrating these changes into the codebase might be non trivial
  • Let's try working on adding the tests to the framework in one way or other

    • It seems there were some additions around this place

    • Ah ah ah, wait a minute I think this comment might just save everything:

      /*
       * Default number of channels should be fine for most
       * tests. Individual tests can override by calling
       * migrate_set_parameter() directly.
       */
      

    • I can manually set parameters and capabilities, nice let's try that out then, we declare both from and to together but after init change caps of to

    • Nice now this one liner needs to be integrated

    • Just thinking can we use start_hook it seems like it is that kind of thing we can

    • Damn it does look like we can use that

    • Now a simple problem is where should this go, in postcopy-tests or file-tests? I think adding to file tests would be simpler and make sense because if we consider precopy file tests they are here too

    • So it should be something like /migration/postcopy/file/mapped-ram

    • Also before we forget we skip if uffd is not available

    • Looks good now, it works and eager thread actaully ran(checked using printf)

• Maybe look somewhat into documentation, first I need to make qemu be able to use my downloaded sphinx package, currently it is not able to find it
  • Well a sudo dnf upgrade --refresh first and I have GBs of updates pending 😅
  • After agonizingly long time of updates, I can confirm I have python3-sphinx installed
  • Asking AI I got to know that I might need python3-sphinx_rtd_theme package which apprantly is different from the normal one
  • Yeah now it does find it, nice!
  • Just for a detour let's go into hugepages once
• Looking into hugepages
  • Post yesterday's I realise that qemu_ufd_copy_ioctl uses set range to set all the points on received bmap to 1
  • That means we should follow a similar thing on pending bmap
  • Here however we need to focus on what happens if some are set and unset in bitmap, though I think that might never happen if we are using ranges always
  • Hmm thinking about, not really, we wont need to set them all we just need to make eager thread requests and loading marks to be consistant
  • Let us keep going with the read and clear of one bit, that feels cleaner
  • I think one final thing to note would be if the rb_offset we are getting are aligned or not, let's see the trace output once
  • The granularity of calls to place page is 0x200000 which should be $2\times {16}^5=2^{1+4\times 5}=2^{21}$, hence 2 MiB, which is as expected
  • So I put a printf at reset and the problem is clear I believe, ram fault thread requests for an address 0x7f7cffc00000 and place zero page, it calls normal place page.
  • Problem seems to be either that the page was not meant to be a zero page or that page was not set properly to zero
  • Let's continue on this tomorrow

Day 39

1 July 2026(~4hrs)

Target: Work on hugepages support

• Continue working on hugepages

  • AI recommended me to follow up on the problem being page was non zero but file_bmap was read at one bit leading to some discrepency

  • Exploring file_bmap inconsistancies

    • Considering that point, for a zero huge page all bits must be zero in a range and any one set bit would mean that it must be loaded seemed plausable
    • I tried it out and it seems to fail in a different form, checking ctz64(qemu_ram_pagesize(rb)/qemu_target_page_size()) bits for it the load was success but the screen was blank
    • This means there is some sort of data corruption
    • I think it it should be better to see how it is actaully loaded, what size of bitmaps is used there

      • It seems mapped_ram_read_header is not called in this case as it throws error if header->page_size != TARGET_PAGE_SIZE

      • But it does as per printf? What even is the page size in header?

      • The header->page_size is 4096?? that was unexpected?

      • For normal case

        ********* BLOCK: pc.ram *********
        just len is  2^34
        max  len is  2^34
        used len is  2^34
        Page size is 2^12
        header page size is 2^12
        Number of pages is 2^22
        Bitmap size is 2^19
        *********************************
        

      • For huge pages it is

        ********* BLOCK: mem *********
        just len is  2^34
        max  len is  2^34
        used len is  2^34
        Page size is 2^21
        header page size is 2^12
        Number of pages is 2^22
        Bitmap size is 2^19
        ******************************
        

      • So everything is same except the page size? That means we should look for if any one of the bits is one in bitmap range

    • Problem is we need to check for 2MiB/4KiB bits, so need some modifications to find_first_bit(rb->file_bmap + BIT_WORD(page), sz) != sz though this needs page to be aligned with size of long which I think it is
    • Hmm now it wont reset or anything but it just hangs, migration
    • I tried and normal case also hanged, it means that there is some problem in this new code we wrote to checks many pages
    • Ahh, I think I found it, it is that if we say want to find first bit in say just 1 size it wont do that ... yup verified in some case for even normal page size test bit says 1 but first bit says 0
    • Oh come to think of it, problem is genuine as we use offset as BIT_WORD(page) it checks same bit for all bits in that word
    • Hmm find_next_bit seems useful, now the normal case runs successfully
    • We hit the same reset problem on , and I found the error, the offset cannot be page it will be page * page_size/target_page_szie
    • It works !! 🥳🥳🥳🥳🥳

  • Now we need to find a beautiful abstraction to make the code actually look good

    • I was thinking for hugepages maybe the pending_bmap should also use ranges but then it would actually be useless as we dont need all that space
    • Also considering that pending_bmap, is state maintained internally for each ramblock not to be saved I think we should keep it small
    • For that we just need to decrease it's allocation size
    • Maybe we can have a different function for bitmap operation on pending_bmap to check for non zero
    • Yeah that is cleaner with a ramblock_file_bitmap_page_is_nonzero
    • Yup everything runs great, though I kind of feel it is less responsive for some initial seconds, maybe I am overthinking it, it works fine a few moments later with migration going starts running smooth

  • Hugepages support done ✅

  • Some blocktime data

    Postcopy Blocktime (ms): 0
    Postcopy vCPU Blocktime (ms):
    [251, 192, 157, 301]
    Postcopy Latency (ns): 2270612
    Postcopy non-vCPU Latencies (ns): 1971159
    Postcopy vCPU Latencies (ns):
    [5238754, 5066972, 5636326, 5914961]
    Postcopy Latency Distribution:
      [     1 us -     2 us ]:          0
      [     2 us -     4 us ]:         29
      [     4 us -     8 us ]:        913
      [     8 us -    16 us ]:         47
      [    16 us -    32 us ]:          1
      [    32 us -    64 us ]:          2
      [    64 us -   128 us ]:          1
      [   128 us -   256 us ]:          1
      [   256 us -   512 us ]:          2
      [   512 us -     1 ms ]:         17
      [     1 ms -     2 ms ]:         11
      [     2 ms -     4 ms ]:        370
      [     4 ms -     8 ms ]:        507
      [     8 ms -    16 ms ]:         25
      [    16 ms -    32 ms ]:          4
      [    32 ms -    65 ms ]:          1
      [    65 ms -   131 ms ]:          0
      [   131 ms -   262 ms ]:          0
      [   262 ms -   524 ms ]:          0
      [   524 ms -    1 sec ]:          0
      [    1 sec -    2 sec ]:          0
      [    2 sec -    4 sec ]:          0
      [    4 sec -    8 sec ]:          0
      [    8 sec -   16 sec ]:          0
    

Day 40

5 July 2026(~3hrs)

Target: Working on documentation

• First we need to test the build documentation and it seems to work.
• Now just writing it out and getting spelling and grammer correct.
• Finally wrote it down as a new migration feature:

Fast Snapshot Load

Overview

Fast snapshot load is an extension of the postcopy migration feature to disk loads.

Unlike a usual snapshot load, which requires all VM data (RAM as well as device states) to be loaded into host RAM from the snapshot file for the guest to run, fast snapshot load uses postcopy infrastructure to load in only the required device states and load RAM pages on demand. The idea is to start the guest and serve its page faults on the go, reducing the perceived resume time for large snapshots.

Architecture

This feature combines postcopy migration and mapped-ram capabilities to load RAM pages on demand. It is done by catching guest faults using Linux userfaultfd and loading the page by calculating the offset of its location in the snapshot file using mapped-ram capabilities.

Fault Thread

The fault thread uses Linux userfaultfd to catch page faults caused by guest and directly load the page from the snapshot file. It is very similar to network postcopy fault thread, with primary difference being it loads pages directly by reading from the snapshot file.

Eager Thread

Eager thread iterates over all pages in RAM and loads each page not yet loaded by fault thread. It is required as unlike network postcopy where majority of RAM has already been loaded via precopy, here entire RAM is waiting to be loaded. If there is no eager loading thread each page will only be loaded when it is required by guest. In case there are some background pages that are never/rarely accessed by guest, the system will be locked in migration state indefinitely.

Synchronization

In order to make sure both of these threads do not load the same page twice potentially overwriting and corrupting user RAM, a bitmap is used (RAMBlock->pending_bmap) which tracks the pages claimed to be loaded by threads. This prevents race condition when one thread is loading the page and other one tries to do the same.

Usage

Simply enable mapped-ram and postcopy-ram capabilities on the destination:

migrate_set_capability mapped-ram on
migrate_set_capability postcopy-ram on

Use a file: URI for migration:

migrate_incoming file:/path/to/snapshot/file

Day 41

6 July 2026(~3hrs)

Target: Start looking into multifd support

• First our target is to enable multifd with postcopy over networks
• So looking into that, let's see what happens now if multifd was actaully activated
  • First we need to know how that even works?
  • I think it is something like one instance sends it and other receives it
  • ok so a very simple one worked and multifd seemed to work, most likely precopy supports it
  • Let's try running it with trace points to see what actually happens
  • Hmm that was quite short ... just like setups and cleanups, maybe should use an actual VM
    • Can't use the fedora one😔, two 16GB VMs would be too much for my laptop
    • Found the old debian boot, maybe this should work
    • Yup seems to work but how to maintain the file system consistancy otherwise stuff might get corrupted
    • AI told for my local case I can use same file, which is a nice idea as the sender will pause, but wait that is not the case for postcopy what happens then?
    • I just ran stuff and honestly the output is not something eyeopening, I think I need to use GDB for better understanding, specifically looking at the call stack
    • Now to rewrite launch.json
      • I think I should have two configs one for incoming and one for outgoing
      • Moment of truth let's see if it runs, ... why are there so many download infos for debugging!!
      • GDB is still downloading debug info for different libs, most likely because of my update to fedora 44, it will take quite some time

Day 42

7 July 2026(~5hrs)

Target: Work on reading about multifd and network postcopy

• Got it now GDB works let's look into some call stacks etc
• The most important thing I think is switchover process but before we look into that I would like my test instances to be slow so I can actually postcopy and not be greeted with instant precopy
• Ok setting the max bandwidth to 5 actaully allowed me to turn on postcopy, now we see how multifd works with all this
• Maybe first let's trace what happens on the source side
  • Wow there are a lot of things like migration iteration and migration thread etc I did not even look into
  • I think it should be where I should actually look around, the migration_thread
    • First it does multifd_send_setup which based on if multifd is present initializes a bunch of things in the multifd_send_state which sort of a global state thing in multifd.c
    • Then it goes on to call qemu_savevm_state_header and if I understand correctly s->to_dst_file should be absraction over the TCP connection here
    • It simply sends out a FILE_MAGIC number and version and in case send_configuration is true send that too, but I think we should always do that?
    • Again looks like no one sets it to true or false, it was initialized as true and remains so forever?
    • Any case the configuration seems to be a json object vmdesc
    • This is done using the big qemu lock I wonder why
    • Then it checks about the return path potentially for back communication
    • So first it sends a command MIG_CMD_OPEN_RETURN_PATH to the destination and then pings it? Let's look into all the commands later on
    • Then it checks if postcopy is on then we advice about stuff can complicate now
      • A thing to note is that it tests for postcopy as postcopy-ram or dirty-bitmaps, maybe we can leave the dirty-bitmaps case and assume it just postcopy-ram
      • Reading documentation it seems it refers to block devices hence the disk, maybe migrating the disk. We should be safely be able to ignore this then.
      • Continuing tomorrow cuz of blackout

Day 43

8 July 2026(~3hrs)

Target: Work comments about RFC v2

• First let me just go fill up the midterm evaluation form
• Nice done now let's get to the RFC comments
• Patch 0:
  • Yeah I will have to add the prevention of any other case that silently errors
• Patch 1:
  • Test removal needs to be moved to last? But iirc in v1 he told to move it from last to first?
• Patch 2:
  • Nothing
• Patch 3:
  • I can add the check for page_size being as expected, but wait isn't that supposed to be guaranteed?
  • However I think the check might be redundant as mapped_ram_read_header itself throws error in case header->page_size != TARGET_PAGE_SIZE
  • As length > block->max_length throws error too and header's page size is same as target page size we will never have illegal access
  • I did not modify it, it uses same, TARGET_PAGE_SIZE which I think is the host page size the smallest unit we can have, so maintain simplicity and consistancy across systems
• Patch 4:
  • We can do passing over the error
• Patch 5:
  • Wait I think my idea about host and guest pages is all messed up, lemme look around
  • Still I have this problem fixed in my update to support hugepages
• Patch 6:
  • Will add errp line
  • The testing of bitmap has been resolving using ranges, which I think might be better considering consistant system
  • Will have to work on this testing stuff, and also change that address to aligned one
• Patch 7:
  • Bunch of minor changes, nothing to comment on

Day 44

9 July 2026(~4hrs)

Target: Implement changes as suggested in RFC v2 comments

• Let's try to get all the things in one single git rebase
• First moving the test removal to last
• Now adding errp to qemu_get_buffer_at, and that works good, just a bug that could have happened was I did not change the error_setg in caller to error_prepend. I would not have been able to catch that later
• A few minor things like errp comment in postcopy_mapped_ram_load_page and removing that new line
• Now the bigger change for how to call blocktime_begin
  • I implemented the changes but I feel something is off
  • Semantics of this function are something that are either weird or dont match the name
  • What it does is checks if that page has been received, if yes, return true and if no insert it in page_requested gtree and call blocktime begin
  • It might just work though. How about renaming it to try_mark_blocktime_begin and returns true if mark successful and false if page already there
  • Looks like an elegent solution, but merging this in git might be pain, let's try out ... ok done!

Day 45

10 July 2026(~2hrs)

Target: Implement changes as suggested in RFC v2 comments

• Now we need to do the minor changes for review on patch 7
  • Nothing worth writing, all of it is menial changes, though removing the error handling seems to have greatly simplified some part of code
• I also think that the so called support for hugepages can actually be squashed in the previous commits
  • That would prevent any confusions and time waste for reviewer thinking this wont work with hugepages and then seeing a later commit makes it happen
  • Let's see if we can squash that with the fault thread patch
  • I did now let's verify everything is good
• Now looking into unexpected features that are not supported which needs some time

Day 46

11 July 2026(~4hrs)

Target: Decide on unexpected features

• First step should be to read through options.c
  • I think multifd is obviously one to be disabled
  • Then we need to look at each pof other capabilities and think
  • I remember in some of discussion with mentor it was recommended to go through qapi/migration.json
  • Yup found it, this seems really good for explanation

• Finally it seems multifd is one we need to disallow explicitly, other's either make no sense or are something not affecting/independently supported
• Also it reminds me I did not make a different patch for renaming the listen thrad bottom half to complete bh etc let me do that too ... and done

Day 47

13 July 2026(~4hrs)

Target: Continue on running network postcopy and use of multifd

• I forgot where did I leave out last time, let me read some previous logs and be back
  • Found it, I was looking into it on 7 July
  • Yeah I remember I was going through the migration_thread
    • So from what I remember the master migration thread on source first establishes connections and sends the header
    • Header should include the description of VM(vmdesc) which is a static const actaully?
      • It has versions, a bunch of errp, some vmstate fileds and VMStateDescriptions
      • That has the target page bits, capabilities and uuid
      • I am not sure what uuid is, lets use internet: https://unix.stackexchange.com/questions/284593/understanding-uuid-assignment is a good overview of why it is used and I tried reading the https://rfc-base.org/rfc-4122.html but that is out of my understanding right now.
      • I think this is to make sure that the system loads on the same persistant device(disk) on both sides
    • Now that the header and config is sent a return path is opened which is a thread, let's read about this thread first source_return_path_thread:

      • I thihnk this should just handle any signal that the destination sends back to the source

      • This is weird why would there me a return message, and if there is something like say error report etc or maybe preemption then that should not be optional

      • Looking into the setting of ms->rp_state.from_dst_file it seems to be a copy of ms->to_dst_file

      • The calls to qemu_get_be16 should be blocking as most likely there is not a continuous stream of data and communications is async

      • Yeah found the enum for message types(mig_rp_message_type), and it seems proper stuff

        |Message |Purpose| |:---------------------------:|:------| |MIG_RP_MSG_INVALID |Not realy set anywhere so might be to mitigate sending default values etc| |MIG_RP_MSG_SHUT |Destination says it is closing so the source knows not to waste resource| |MIG_RP_MSG_PONG |Reply to ping to verify connection| |MIG_RP_MSG_REQ_PAGES_ID |Request a page likely on fault during postcopy| |MIG_RP_MSG_REQ_PAGES |Same as last with less data?| |MIG_RP_MSG_RECV_BITMAP |bitmap of pages received, might be for maintaining consistance over lossy networks| |MIG_RP_MSG_RESUME_ACK |In case of network error both pause and source might ask destination if earthquake ended and everything is stable| |MIG_RP_MSG_SWITCHOVER_ACK |Source is asked to switch to postcopy so it conveys to destination and that preps for postcopy and acknowledges the switch|

      • Now it might be interesting to note how it tells the migration thread that there are requests etc

      • invalid just kills it so that is fine and also if the length of message != expected is killed

      • Then it reads the message and switch cases on that

        • ldl_be_p seems to be something like conversion from big endian to system
        • migrate_handle_rp_req_pages seems to be the one handling the request for pages

Day 48

14 July 2026(~5hrs)

Target: Send out formal patch set

• I just remembered I did not add restrictions/limitations to documentation so first do that
• Also I tested and if preempt is on snapshot load fails so I need to disallow that in options.c too
• After last meeting maybe I should look into RCUs and futex
  • Going in depth about RCU I found this really good talk
  • Saw half of it and seem to get a good understanding about it though there were mentions about perfbook, wonder what that is
  • Got perf book and it is really good, read 2 chapters, I might read more later
  • Reading somewhat about futex, It came back prof mentioning stuff about futex using hybrid kernal level and user level locking to make mutex really fast
  • This is a really good resource for that
• Let me get back to the patch set
• Wrote the change log and sent to the mailing list!

Day 49

15 July 2026(~6hrs)

Target: Back to reading about network postcopy

• I think I was working on the source_return_path_thread which got feedback from the destination about various stuff
  • Let's look into how the request for pages are handled which is done using migrate_handle_rp_req_pages
    • First thing I notice is use of qemu_real_host_page_size and instantly it feels weird, shouldn't it use general functions defined like qemu_target_page_size?
    • Let's try removing it, that also reminds me MigrationState->send_configuration can be removed and I can send a small patch for that, let me do that after testing this change
    • Wow just checking using clangd qemu_real_host_page_size is used by a huge number of devices, that makes me think why is there the target page size then
    • Ok so tests ran and chaning them made no difference, think I should save it for meeting why migration has the target page size
• Now First let's look into the send_configuration patch
  • We'll be starting with finding all it's references using clangd and deleting them one by one
    • First getting rid of definition so others actually throw error
    • Some changes in migration code need to assume it is true for if else
    • Now in migration_global_dump, it is printed and this is where question arises should we delete it or just print on
      • Point is that if someone has a script surrounding this output those might break for different systems
      • However it is in hmp and scripts should use qmp or libvirt etc, soooo let's get rid of this
    • Now remains one in options.c defining a property using a complex macro
      • It looks like it is defined as a Property in an array so we can delete it but we need to see if the array element is accessed then
      • Only one using is device_class_set_props_n adding properties to device
      • I figured most likely way to access it would be using the name and searching for the name came across xen_accel_class_init which hasa compat array seemingly setting compatibility for it to false
      • Ahh it looks like the xen accelarator has it false by default and so should error if we set it to true?
      • Looking around on internet about what xen is it is a type 1 hypervisor running directly over hardware!
      • But then what purpose is qemu? looks like for management etc
      • In such case I dont think we should be able to remove send_configuration as it might throw errors for that?
• Back to the network postcopy
  • We were looking at migrate_handle_rp_req_pages:
    • It makes sure that the start and length are both page size aligned which is a really good thing to assert/throw error on in our fast snapshot load feature, we can add that if there is another iteration
    • Then just a call to ram_save_queue_pages
    • ram_save_queue_pages first decides on which ramblock to work on and makes sure the page is in it using the offset
    • If preempt is active page is snet directly which is kind of a good optimization, just thinking can there a better method to have a common say send_page_over_network used by both
    • Kind of a NIT but instead of len % page_size == 0, QEMU_IS_ALIGNED is better but too small of a thing
    • Looks like the urgent save is why we cannot merge the paths
    • hmm ram_save_host_page and ram_save_host_page_urgent are different I wonder why
  • Comparing ram_save_host_page and ram_save_host_page_urgent
    • From comment on top the only difference is that urgent version needs bitmap_mutex heald
    • Going on looks quite similar thing, doing same kind of stuff
    • wait the non urgent one unlocks the mutex if preempt is active? Did it take it before the call?
    • Maybe it is the case that it always holds the lock?
    • Any case then it calls ram_save_target_page and its return value is checked to be positive but urgent had it to one?
    • What it looks like is that, normal save has lock usually and when it is about to save something it gives the preempt thread to work its own magic
    • This does feel like it could be solved using a pending bmap again?
    • It does occur to me that maybe ram_save_target_page might return non 1 value if a hugepage is sent but honestly doesnt make sense
    • Oh the comment on top also mentioned that the function needs to have the bitmap_mutex in both functions
    • I feel that one thing that could happen is that there be a function ram_save_target_page and it takes a boolean for urgent in which case it would do everything that the urgent function does
    • First I think instead of the sent boolean pages would work
    • First thing it does is check for overlap, that might better be solved using bitmap and atomic operations
    • Let's see how can we actually resolve this, what happens is that urgent gets lock only when normal path unlocks with a prepped pss and then checks with that
    • So on an urgent call we should check for overlap?
    • ugh doesnt seem to work, I think I need a better mapping of what each does and why they are different
• Let's continue tomorrow

Day 50

17 July 2026(~5hrs)

Target: Network postcopy and potentially think more about ram_save_host_page

• I think it would be much better to first just read more about what the paths do and get a general idea then move on to see if they are mergable
• Let's get back from the migration thread path and continue on handlers for the rp
  • Let's look into migrate_handle_rp_recv_bitmap
    • First it finds the block and calls ram_dirty_bitmap_reload on it
    • It works only on postcopy recovery so potentially we can skip it first we shuld get a better general idea
• Continuing back on the migration thread
  • An advice is sent to destination that postcopy may be used in which case a simple command is sent with ram page size summary sent for postcopy ram
  • Maybe the auto converge can be skipped entirely(for now)
  • Now a call to the setup function qemu_savevm_state_do_setup:
    • Then it looks like state is sent using vmdesc but I think send configuration had that done early on?
    • Will have to look on receiving side, if it expects it twice? Oh got it, the header case just sent configuration, though it ends the object, not sure if it should delete the write, maybe not
    • Then there is a wait for unplugging devices, Looks fine
    • Now just note the time of setup and go into migration
  • Then a rate limited iteration followed by error handling on it, let's look into the iteration run now:
    • There is something wrong with this function migration_iteration_run as the comment says it return true/false but it returns MigItaerateState
    • Both in the same commit though, ... iteresting
    • Let's first look into precopy case
      • Then there is a check on if next iteration can be moved towards and that checks the threshold which decides the final stage
      • In other cases of stop copy, I wonder what that is?
      • It moves to next iteration in that case seems like it first saves the pending data in stats
      • then a switchover query starts postcopy, we'll look into later
      • It does not make clear how load is done in precopy case, maybe I missed it
  • Ahh got it that is part of ram handlers
• I think a better strategy would be to understand the PageSearchStatus infrastructure
  • Looking into the ram.c infra we start from PageLocation
  • PageLocation has the pointer to block and offset so each page's position in block
  • Nice there is a PageLocationHint for using spatial locality, so wait does this mean otherwise getting the location of page is hard? might be let's see.
  • Now we have the bigger struct PageSearchStatus:
    • This should be like the manager for sending page?
    • First it has channel where the page should be sent
    • Then a pointer to last block, which I dont know the use of but should be some sort of optimization
    • Then a pointer to current block certainly
    • Page number
    • complete_round makes me think the search is linear ?!
    • Sending host page might be a vhost user thing?
    • then if host page its start and end is stored which is really good meaning usually a single page is used
    • Why is there start and end? maybe cuz the guest page size may be larger and so we send multiple pages
  • XBZRLE can be avoided as it looks some kind of a delta patching like optimization
• Now back to understanding ram_find_and_save_block
  • First the pss is extracted, which seems like there is a pss for each channel
  • Variables prep for next page and block
  • zero ram returns instatnly a 0
  • Yup found the line if host-page-size > target-page-size then it sends all the pages, I wonder if same is the case for the urgent case
  • I do wonder qemu rarely uses const(at least in migration, it could use more of it?) and no constexpr though given that it is like very very recent feature it is excusable
  • Not having last seen block is bad so it is reset in that case
  • Now if hint is valid which happens only in preempt case we collect it
  • The RAMState has a single hint attribute for this purpose
  • Collecting that makes stuff good else next vars are set to last and pss is init
  • The init sets the block page and complete_round false for that channel's status
  • So a wile loop starts up and first checks for a queued page using get_queued_page
    • Seems like it is queue of pages requested by postcopy and in case of preempt I think this queue should not even exist?
    • If queue is empty return NULL else take the src_page_req_mutex, which means checking for empty queue should not be atomic which might be fine
    • Hmm this seems like a race condition though ... if no request return NULL, take the lock and assert that it is not empty
    • if the queue was not empty and then we pass the check and return but then someone popped it we take the lock and then assert fails
    • It might happen that no one can pop from the list except this function but then the assert wont make sense?
    • Maybe a prep check, as checking the references to src_page_requests it seems it is popped either on freeing the page queue or in our current unqueue_page function
    • If length of entry is large then it pops just one host page(the TARGET_PAGE_SIZE), and small might not be possible as everything is actaully divisible by TARGET_PAGE_SIZE
    • Now the block has been received, page idx is calculated and dirty bit is checked, in case of non dirty popping continues till we hit no block or there is a dirty
    • If no block is found it goes on to poll on write faults which seems like another thing we would want to do if read blocked pages are not foundif block is found set stuff on block
    • on pss, complete_round is set to false and this comment is something I not really get
    • Finally we return !!block which is odd, maybe a type cast into bool?
• Let's continue later now from this complete round thing, I want to know why it is used

Day 51

22 July 2026(~2hrs)

Target: work on the v3 review

• This time around the number of review points seem quite small, good that stuff is falling in place

• Going one by one the patches:

  1. Adding tests makes sense, I removed that cuz after most changes I did not have the debian image, maybe should test on that ... I need to make a new one, maybe I can just report working on fedora. This should be good:

  This patch series was testing on following system using normal page
  size and with 2MB hugepages. Testing was limited to x86_64.
  guest OS  : Fedora 44
  guest RAM : 16GB
  guest CPU : 4 cores
  host OS   : Fedora 44
  host RAM  : 32GB DDR5
  host CPU  : Intel Ultra9 185H(22 cores)
  

  1. Adding the review tag
  2. Same here
  3. Again
  4. And again
  5. Once more
  6. Wait what!? I thought I had changed that in some earlier commit, I think it must have been lost in some commit refactorings
  7. Ok some more comments here
  • Reading the second one first, I had the change here, which I think I messed up
  • The first one seems quite a good idea to instead of what we do now loop over and load pages in buffer
  • Then the case of different psizes all cases depends on mapped ram handling them. I might need to go deeper into that as I read documentation but could not find anything
  • I think once I understand that I might add that to documentaion to make it easier for some new
  1. Ohh really nice suggestion, it is much more cleaner, I think I just copied what fault thread used for error report, any way easily solvable?
  2. Just a rename, does make sense for it to not have just test tag
  3. This is all good
  4. Changed the paragraph to something more generalized, I do need to add the protection against vhost user, it is good that I now know I need to check the postcopy_notifier_list

Day 52

24 July 2026(~2hrs)

Target: work on the v3 review

• First we need to resolve the mapped ram issue and I think the reply on patch will help a lot
  • First thing to note is that mapped ram and as a matter of fact received map both use a bit per target page ie the guest page
  • Wait now if the target page size is the guest page sise then the ram page size should be same?
  • Oh maybe it is something like for x86 4KiB is the target page size and if we use hugepages then ram page size increases not the targe page size
  • Honestly that makes sense, then the problem might be about the atomic swaps? as ramblock_file_bitmap_page_is_nonzero should work perfectly?
  • Oh no the problem is that it will still copy in zero ranges which we dont want to happen, so instead of this we iterate
  • This leads to a different question though, how does place zero handle that?
  • Reading about it, it just calls postcopy_place_page with a custom pre zeroed out page
  • Hence I think we need to loop over the bitmap for the size and atomically put stuff on the buffer and then place the buffer and this should turn out fine
  • I do wonder how the case of host page size affect it, I need to read the review once again
    • Oh ok reading thoughroughly the problem occurs here is that uffd is host page aware and just swaps host pages
    • This makes me think a lot deeper on how these interact cuz if they have equal sizes world is heaven but other cases hell breaks loose
    • If guest psize is greater than host psize
      • One userfault will tell about a fault on a host page
      • We any case find the address of the guest page and place that entire thing in buffer and place it
      • Hmm, I thought one ioctl would only fill in one page, but as man pages suggest that len can be a multiple of page size we should be able to place multiple pages which resolves this case
    • If host psize is greater than guest psize
      • As already pointed this case should be somewhat qirky as ioctl accepts len as multiple of psize and hence we cannot just place one page
      • The general idea that comes to my mind is that in such cases we need to load in multiple pages but let's not make a rash decision yet and first look deeper
  • I read the second comment and it covered the page differences really good, now that I have a much better idea let's first go into reading about the comments on pss_host_page_prepare
    • Oh this has all the cases covered
    • Byt wait guest_pfns = qemu_ram_pagesize(pss->block) >> TARGET_PAGE_BITS means that target page size is host ?!
    • I remeber reading this and that was when I made it in mind that target page is host page
    • I think the comment is misleading?
    • Well I asked AI and the pss->block holds data about the host but in that case it is host pages per guest page?
    • 🤯 I realized just now the comment says how many guest pages in host page not guest pages per host page!
    • That is where I was interpretting everything wrong
• That was a breakthrough in my understanding, let's continue tomorrow and actaully get stuff done

Day 53

25 July 2026(~2hrs)

Target: work on the v3 review

• So back on solving the problem and well the idea I have by default is to increase granularity of the load, let's see how pss handles it first
  • First the point is what is pss->block
    • It is the current block being searched which would mean that it has no data about page size of host?
    • Wait so is the page size in RAMBlock actually the host pages! That goes against a lot of my assumptions
    • I am so confused right now, but I think this clears a lot of things
    • I always assumed that guest hugepages are big deal but as it turns out hypervisor doesn't even care about that
    • Wow ... that was mind blowing after realizing how it made sense that hypervisor rarely cares about the size of guest pages
  • Ok now that we are back stuff seems to make a lot more sense
  • problem on small host psizes is that we can load one host page at a time so granularity must increase
  • Now let's look back at pss_host_page_prepare
    • If guest_pfns <= 1 it means that guest page size >= host page size which we covered as we can load a lot of pages
    • In other case however we need to load a page of higher granularity and makes complete sense
  • This should work quite well though, however does this mean if host uses huge pages then we load a huge page cuz that would not really be that helpful and not be very fast
• I think I can tackle it now by converting the if (ramblock_file_bitmap_page_is_nonzero(rb, page)) in a loop but I am kind of playing with the idea of using the PageSearchStatus architecture along with it
• One thing we should be able to assume is that page sizes are powers of 2, though I can't find an explicit statement it seems to be an unsaid rule considering the speed of operations etc
• Now let's think about using pss infrastructure here
  • So first we need to call pss_init to fill in required stuff, ie the block and page location
  • Then we call prepare to decide what actual range to load in
• Then we should do the loop over for all the bits and here I think stuff might start getting out of hand for pss infrastructure
• Let us plan it again once
  1. if the page sizes are same we just need a bit check
  2. if guest pages are large, we also check one bit and automcatically we load in a bunch of host pages using uffd
  3. now if host pages are larger, we need to first put stuff in a buffer and then replace one page only
• Hmm I think peter pointed on point 2 and that seems solved if guest page size is large, I would love testing these though
• Let's think how we should go about case 3
  • The loop honestly seems the best idea and I think reeling in the page search status might just make the code convoluted
  • Though from prespective of cleaner code I might be redoing that sort of logic again, not sure
  • I think the loop is good solution, I'll implement that tomorrow and try thinking if the pss can be integrated too that might be awesome

Day 54

26 July 2026(~3hrs)

• Ok let's get done with the loop first and see how that works
  • The if becomes a for loop, the place page occurs in any case and the zero case is changed
  • I think the current case should not have worked considering that it loads host pages only
  • This looks like going for a new function that loads in a guest page will make it cleaner
  • Made a new funtion and all it does is load a guest page in a buffer
  • This would use test_bit much simpler
  • Now for zero pages I cant use postcopy_place_zero so I went online and searched the most efficient way to zero out memory(I could copy from the temp zero page but maybe there is something faster)
  • Got memset and this makes me wonder why does place zero page not use it? maybe cuz it is not atomic
• Now that we have loading of a single page done, let's look into how we handle multiple ones
  • Wait but we need to know about pending bmap usage
    • We initialize it to host page sizes, making it kind of simpler but confusing at same time
    • If we consider guest pages are larger then we must test and flip all bits
    • If we consider the converse we must flip only one
    • In first case I think taking the assumption that all bits are same atomic operation on one bit would be enough but I think we do have access to atomic operations on a range of bits
  • I think I have a cheap solution that should work and has a much smaller footprint that I thought
  • I just need to be fully sure about this now
    • first step is to round down the haddr and rb_offset to host page sizes so that the case of host page greater than guest is covered(in others the ROUND_DOWN would do nothing)
    • Then page is decided as the host page to load using rb_offset/qemu_ram_pagesize(rb) and granularity is target by host size
    • Then if the granularity is zero ie host page is larger we set it to one meaning load one a page
    • Maybe better to have something like guest pages to load, that is one if sizes are same or if guest page is larger, else it is number of guest pages in a host page
    • Ahh but then pending bmap uses host pages so will the ranged test and clear work?
    • If the guest page size is larger then the guest_to_load is one, maybe we can do something like invert semantics?
    • Maybe I should get rid of this variable and use MAX(1, division) for both the purposes
    • Now the ranged check is MAX(1, qemu_target_page_size() / qemu_ram_pagesize(rb)) which is 1 if page sizes are same or that host is larger so a single is tested, in case of larger guest page we test many bits
    • Now we need to loop over all the guest pages in it
    • I think the code is correct but the for loop is ver very ugly, I think it can be made better using something like a while loop
    • The code looks clean but cluttered at same time, just too dense, I wonder on how to make it look better
    • Let me redesign the loop somewhat to make it not so idiomatic
    • Ok I think a for loop using offsets should work really well
• This solves the problem and should actually work with all the cases for difference in page sizes
• It seems I got another thing that for qemu_get_buffer_at if f->last_error is true it returns zero which is error with no errp set
• It would be better to set it ourselves, and should be added to the patch, it is worth noting that qemu_get_buffer_at returns buflen on success meaning it returns only 0 on error and cannot read less
• I think there is one optimization worth testing about if we should use postcopy_place_page_zero to utilize zero filling in uffd making it faster
• Now that a solution is here, I just need to integrate it and run a bunch of tests, let's do that tomorrow

Day 55

27 July 2026(~3hrs)

• Let me first start with prepping the changes to postcopy_mapped_ram_load_page so it actually reflects what it does in a better way
  • First I need to update the comment to reflect that it does not load a page but pages so that address can be accessed
  • Considering the very common case of both pages being of same size I believe I should add the optimization to check if page is zero use UFFD_ZERO as that is much faster
  • I came about this and I rember seeing use of '\0' somewhere to set bits and I think I am using memset too so for clarity maybe I should use that too
  • Now back to writing the description of postcopy_mapped_ram_load_page
    • What it does is load page(s) required to access the host address and that depends on variation of page sizes
    • I was looking at the callers and they all do align to host pages, hence we need to alter the offsets in case they are not guest aligned!
    • Wait pss assums that pages are guest page and that kind of makes sense, so stuff seems to be good
    • I do need to print the guest page in error
  • Ok written enough comments, now to test it
  • Interestingly the difference of adding the uffdzero case is not really that significant, maybe my testing was not objective
  • Ugh, I spent nearly hour and half on making a good enough python script using GMM to analyse the difference and it seems the difference is there but not really visible
  • Now we have this done, all that is left is adding this to a patch ... done commit message todo
• Updating patch for qemu_get_buffer_at to set errp in case of error ... done
• Wrapping up for today

Day 56

28 July 2026(~3hrs)

Target: Complete v3 review points and prep for v4

• Adding check against vhost-user in case of fast snapshot load
  • It seems to be a QLIST, which I think is just a linked list
  • There is no specific function for that purpose so I think I might need to add one as the list is static in postcopy-ram.c
  • There is a function notifier_list_empty but I doubt it as it takes in NotifierList not a NotifierWithReturnList
  • Oh it is sad that no one calls this function, who added it then?
  • After adding a lot of functions for a very tiny thing finally added that point too
• Now all that is left is to update the commit message for the updated page load
  • I think ramblock_file_bitmap_page_is_zero is no longer required as it just searches for a non zero bit, reason I made a new function was for calculating what chunk to search but as main code does that it is better to not have so many functions
  • Ok done, added a simple para explaining what should happen
• I also remeber I added reviewed tag on the capability check, I need to remove that as a lot of new changes were added
• Now all that remains before v4 is testing, I think this time I will try to have more thourough tests with different architectures
  • Same page size works now let's see if page size is larger
  • I am fed up spending hours trying to configure a debian cloud image to run with ppc64 architecture for 64KB page size
  • I have tried everything I could, I am just loading iso files tomorrow and testing with them, that would be faster than this

Day 57 & 58

29 & 30 July 2026(~3hrs)

Target: Testing v4

• Both these days were just spent testing possibilites of page sizes.
• Hours and hours of booting and creating a bootable image, nothing worth mentioning here

Day 59

8 August 2026(~1hr)

Target: Improve on v4

• After being sick I am back and as per last meeting with mentor there is a bug, whose solution has been discussed but needs implementing
  • currently postcopy_mapped_ram_load_page uses bit_test_and_clear_atomic on rb->pending_bmap on a range of host pages in guest pages
  • This should work fine if range is one but if no, the way bit_test_and_clear_atomic is implemented stuff goes wrong
  • Against all odds, and clear naming of bit_test_and_clear_atomic is not a completely global operation, it takes the range and does the atomic testing and clearing word by word(each word atomically)
  • This means it would atomically clear first word but second word might be cleared by others making the use of this bitmap highly questionable as if two threads are working on it, easily one can read first word and second can read second word causing wrong things to happen
  • Maybe let's see where else is it used
    • Once in physical_memory_test_and_clear_dirty, it uses size as one so it is atomic
    • Other time in vmbus_signal_event, which uses size 1 too
  • So this is just a bit_test_and_clear_atomic not a range test! why take nr and complicate stuff
  • Ah wait wait wait, ther is a smp_mb here, which I dont know what it is but might be something
    • This is some sort of a memory barrier
    • Well I checked using AI and it still does not make it useful
• Now for the solution
  • Cutting the chase the best one that can be done seems to be that the semantics of pending_bmap be changed so that it has bits for guest pages/host pages whichever is larger
  • In that case the test will always be a single bit
• Implementing:
  • I think it would be better to have more functions to implement this functionality
  • A function that takes in guest page and host page and decides
  • Wait, we can simply use MIN(host_page, page) as whichever is larger will have smaller indexing
  • That seems like a one liner here, also a one line change in init
• Implementing turned out simple but now it is requried to explain why

Day 60

12 August 2026(~2hrs)

Target: Improve on v4

• Damn a new problem exists again, loading using place page functions loads exactly one host page(I thought it worked on guest page granularity, from previous misunderstanding of qemu_ram_pagesize)
• A solution came to me but again is quite weird to do, pass the number of host pages to load in place page functions
  • Then it would call the ioctl with that many pages
  • This will increase some complexity but will solve the problem
  • Let me clean up the rest of stuff first and this can be a standalone patch
• Cleaning up on previous patches
  • It was somewhat weird but able tod apply the changes on corresponding patches
  • Also changed the page variable in postcopy_mapped_ram_load_page to guest_page for consistancy
• Now for our solution to the problem on guest page sizes being larger than host page size and place page not working
  • I think the basic thing to start working with is how does postcopy handle that?
  • It is called in ram_load_postcopy, and it seems that the place_source there was of guest size?
  • Wait if the page sizes not match then it uses a page_buffer
  • This is well complicated to place the data first in a tmp page and then place it. Makes sense but the implementation is convoluted
  • I wonder if there is some reason for it to be so complex, if I should follow this pattern or the adding of number of pages to load works
  • Umm I dont think the present code works with large guest pages? Need to see properly, but it depends on the sender so it is entirely possible that sender takes care of the order to make it good
  • A comment explains that source ensures that components of a host page are sent in one chunk, which might not mean vice versa
  • Seems like it won't work for this case
  • Ahh found this function postcopy_ram_supported_by_host which disallows qemu_target_page_size() > pagesize
  • Now we can either add this check or try solving the problem
• Let's say we do, all we need to do is add how many pages to place as ioctl can manage loading many host pages(I read in man pages)
  • So I added a num_pages argument to postcopy_place_page and postcopy_place_zero_page and in existing places gave it 1 and for our case used MAX(1, qemu_target_page_size() / qemu_ram_pagesize(rb))
  • After enough changes I think this wont work, because I am not sure how postcopy_notify_shared_wake works, need to verify that first.
    • Hmm it looks like it works on remote fds, used by vhost-user
    • Thing is that we dont support it but not doing anything will create a technical debt, it must be explained that this does not work
    • Maybe we can loop over number of pages and call the waker at all positions?
    • It looks like it checks if this page was somehow in the region of this vhost user?
    • If yes then it is woken up using uffd_wakeup, and because of the current code the wakeup range is kind of hardcoded to pagesize
• Let's continue tomorrow

Day 61

16 August 2026(~3hrs)

Target: Finalize a v5

• After discussion with mentor it has been decided to disallow large guest pages as it is disabled in normal postcopy too

• It can be further improved on later but for now this should be good as large guest page size is something not really common

  • There shouldnt be many changes but the most important part might be to explain this in comment, let's see how remote postcoy disables it and use similar things
  • So our target is to find where does postcopy_ram_supported_by_host stop code
    • First is in loadvm_postcopy_handle_advice, ie is the advice state and so it makes sense
    • Second is in migrate_caps_check, where it checks if postcopy ram is activated now and was off before and the runstate is INMIGRATE?
    • So I should not be able to set capability, but I can, most likely cuz the runstate is different
    • Hmm, I think we might need a different thing for this
    • After a lot of time trying to figure out why is this thing not blocking fast snapshot load when I try the ppc I got here:
      • Apparantly PPC allows 4KiB hardware pages🤦
      • So I had target page size as 4KiB and guest used 64KiB
    • However I spent some time trying out a gentoo on alpha architecture and it throws an error as alpha is inherently 8KiB page based
  • Well that is done, just a comment explaining it

• I added a note in postcopy_mapped_ram_load_page, this should explain the wishful thinking

• Now we go on to disable the migrate command after enabling fast snapshot load

• This should work

  if (migrate_mapped_ram()) {
      if (migrate_postcopy_ram()) {
          error_setg(errp, "Cannot migrate with fast snapshot load enabled(mapped-ram + postcopy-ram)");
          return false;
      }
  }
  

• Now one final thing, which commit to put this in?

  • I think this would be best placed in commit to update capability conflict tests

Running TODOS

• Prepare RFC
  • Decide on code divergence point
  • Prepare implementation plan for the feature
  • Implementation step 1 to change qemu_loadvm_state_main
  • Implementation step 2 to change postcopy_ram_fault_thread and call it
    • Initialize and setup uffd and other states
    • Impelement basic functionality
    • Use temp pages to make reads to RAM atomic in fault thread
    • Make a new fault channel so fault thread can load in data independently of eager thread
    • Update postcopy_mapped_ram_load_page to read form file according to channel
  • Implementation step 3 to start VM
    • Setup all the states the vm needs
    • Start the VM
  • Implementation step 4 to start eager loading of pages by main thread a new thread
  • Implementation step 5 to kill fault thread and cleanup after migration from new thread
    • Look into the usage of all the different SaveVMHandlers and analyse potential to use some for cleanup(step 5)
    • Implement step 5
    • Free bitmaps stored in RAMBlock
  • Add comments for any reader trying to map out the logic flow
  • Is it required/safe to call dirty_bitmap_mig_before_vm_start for snapshot load
  • Look into return values in qemu_loadvm_state
  • Add tracepoints in newly added code
  • Get an RFC ready for review
  • Instead of allocating page in postcopy_mapped_ram_load_page, preallocate somewhere else
  • Update comments and error handling to signify both network error and potential file fails
  • Think about using ram_load_postcopy
  • Use proper bitmap functions instead of bare alloc and free for recievedmap
• Review RFC comments
  • Patch 1
    • make separate patch for file_bmap
    • Add bitmap size length checks for file_bmap etc while reading mapped ram headers
    • Take care clang format does not change minor stuff
    • Look into the load_using_postcopy boolean logic by adding new function(follow from comment on rfc)
    • Length checks
    • qemu_ufd_copy_ioctl updates
  • Patch 2
    • Docs formatting for postcopy_mapped_ram_load_page(Only verification left)
    • Add proper error handling logic to postcopy_mapped_ram_load_page disk failure case(report in threads not here)
    • Revert/improve comment in postcopy_ram_fault_thread
  • Patch 3
    • Update postcopy_listen_thread_bh to something like postcopy_complete_bh so both listen thread and eager thread can use same thing(renaming left)
  • Patch 4
    • Update migration.c schedule of process_incoming_migration_bh as per comment
    • Move the removal of caps check to test removal patch
    • Error return in qemu_loadvm_state need to rethought using errp etc
    • Look into moving around logic to call qemu_loadvm_state
  • Move Patch 5 to first
  • Add qemu_get_buffer_at patch separatly
  • Rewrite all commit messages and cover letter
  • Remove migrate_fast_snapshot_load
• Add error handling support for some more postcopy functions, read comments at start of include/qapi/error.h
• Check if qemu_ram_pagebits would improve efficiency on the page calcualting in postcopy_mapped_ram_load_page
• Review RFC v2 comments:
  • Fail on migrating with unexpected features like multifd + fast snapshot load
  • Move removing test to last?
  • Pass errp to qemu_get_buffer_at
  • Add errp to comment of postcopy_mapped_ram_load_page()
  • Remove empty line in if else line 1000(😲)
  • Update infra for mark_postcopy_blocktime_begin to have new function for testing recv bitmap and send aligned address
  • Do the minor changes on patch 7 review
• Review v3 comments:
  • move the pending bmap size calculation to one commit
  • Look into the copying and loading of zero pages for the loop etc for hugepages
  • Update git commit message for this update
  • Add a check against use of vhost-user using the postcopy_notifier_list
  • make qemu_get_buffer_at set errp in case of f->last_error following that every function that returns error should set errp
  • Add a way to call postcopy_place_page_zero to make zero page case faster(or check if improvement is notable or overhead is high)
• Conduct proper testing with page sizes before v4
• Write v3-v4 changelog
• Work on v5
• We can use assert(QEMU_IS_ALIGNED(addr, page_size))
• Add tests to validate capability pair for other disallowed pairs
• Look into compatibility of new mapped ram + postcopy ram with other ones and how something might break with different capabilities
• Look into error handling on incoming side instead of using MigrationState->error

Questions

All resolved

Resolved questions

• MemoryRegion use of bools and not bits
  • Correct point, but till not required bools have lower probability of mistake
• Using a sorted list of offset for bin search vs random access array for storing file offsets for the pages
  • no need, random access are not hard to calculate
• How should eager thread exit in case of error
  • Use asserts and just crash
• channels for fault and eager thread
  • make qemu_get_buffer_at thread safe, by removing the error part
• What exactly does VMStateDescription do?
  • It acts as API for hardware to store the data/state like registers, queues, ctr etc
• What do these bh functions do
  • These are bottom half functions of migration postcopy
• How can I test internal memory state of the host
  • No proper tool exists
• What if mis->to_src_file is empty in fault thread(line 1323 postcopy-ram.c)
  • Might happen in failure case one thread closes it
• goto vs while line 1390 postcopy-ram.c
  • old code, leave it
• How to duplicate file channels
  • No need as pthreadv is thread safe
• Should preemption be enabled during fast snapshot load
  • Not requried
• Is there / Can there be multi socket migration
  • Ther can be but not allowed yet
• Calling process_incoming_migration_bh did not make sense, so need verification
  • Use loadvm_postcopy_handle_run_bh
• MigrationIncomingState->loadvm_co purpose(no reference to it as per clangd)
  • Used in RDMA, to be deprecated soon, series that plans to do so is here
• Confirm tests method
  • It uses QMP and asserts output
• Fast snapshot load is part of postcopy ram features or migration for documentation
  • Keep if new feature for now
• What if the guest was memory starved and was swapping out pages? Userfault might be activated in that case too
  • userfault has many modes, that is taken care by what mode we use
• How is file coherence maintiained during migration(especially for live case)
  • Mostly file is shared, if not there is something called block migration
• How to give instructions to HMP/QMP fast
  • Use scripts or dev feature -global migration.{stuff}=on
• Why would we not want ot send configuration during migration(MigrationState->send_configuration is init to true but never set to false or anything)
  • send_config is not useful anymore, can send a patch to clean it up
• What all does the big qemu lock protect
  • rcu does cleanup
• What purpose is MIGRATION_CAPABILITY_EVENTS
  • Should be on, good but not necessarily like tests have it off, mostly used by libvirt
• Why does MIG_RP_MSG_INVALID exist and who sends it?
  • Just a reservation for 0
• Use of qemu_target_page_size vs qemu_real_host_page_size
  • one is guest page size and other is host page size
• About MigrationState->send_configuration, seems like xen accelarator uses it
  • xen uses it potentially because xen does not have a machine type associated(at least not a versioned one)
• What is stopcopy?
  • It is the middle stage between precopy and postcopy when execution is transferred
• How to test everything is fine with loaded VM and there is no corruption
  • No exact thing exist, keep using and if no error yay
• why would anyone use bitmap_test_and_clear_atomic, when you have bitmap_test_and_clear
  • No serious usage, most likely an error
• What do the waker function do
  • Used in case many different processes fault on pages and uffd serve