配置mutt作为邮件客户端
我一直是使用Gmail的Web GUI来发邮件的，用起来倒也方便。但是现在要提交patch了，还是换一个常见点的客户端吧。在内核的Documentation/email-clients.txt里面列了一些常见的客户端，由于Gmail会自动把tab换成空格、默认format=flowed的设定以及based64编码的问题，它并不合适作为客户端使用。
上午折腾了一下Alpine和Mutt，最后还是选用Mutt了，觉得用起来还不错。把配置文件记一下。
set from = "YOUREMAIL" set realname = "YOURNAME" set imap_user = "YOUREMAIL" set imap_pass = "" set imap_keepalive = 900 set smtp_url = "smtp://YOUREMAIL@smtp.gmail.com:587/" set smtp_pass = "" set folder = "imaps://imap.gmail.com:993" set spoolfile = "+INBOX" set postponed = "+[Gmail]/Drafts" set trash = "imaps://imap.gmail.com/[Gmail]/Trash" set header_cache = ~/.mutt/cache/headers set message_cachedir = ~/.mutt/cache/bodies set certificate_file = ~/.mutt/certificates set move = no #set editor='vim + -c "set textwidth=80" -c "set wrap" -c "set nocp" -c "?^$"' set editor='emacs -nw -q' my_hdr Cc: YOUREMAIL set query_command = "goobook query '%s'" bind editor <Tab> complete-query
光有Mutt还不够，由于我的联系人都在Gmail上面，写邮件的没有没有address book也是很烦人的。我用的是goobook来同步Gmail联系人，在Mutt里面也能调用。配置文件的最后两行就是goobook在Mutt中的配置。
goobook的缺点就是在Mutt中password的提示不大好使，所以要在goobook配置文件中写入password。这不够安全。折中的办法就是，先同步一次，goobook会做好cache，然后再把配置文件中的password去掉。
实习啊实习
刚才接到TX工程师的电话，问起实习的事情，确实是对不起了。
原来863未验收的时候，老板是肯定不会放人的，所以就申请个GSoC玩玩。
后来863有确定的结果了，才考虑是不是去找个实习什么的。因为我知道TX有8月的实习时间，所以打算是先把GSoC做完，然后再去TX。
但是老板突然又说某个项目又来了，又要我带新人，下半年放人的可能性也很小了。
我觉得实习对于一个学习计算机的学生来说实在是太重要了。国内的研究生都是给导师打苦工，还是得毕业的。老板说一个“不”，谁又敢违逆呢。
发完牢骚，继续抠腚去……
开始VirtIO for pure PV
上周给Xen写了几个Patch。
1. 简单的typo fix，没什么好说的。
2. 为模拟设备注入MSIX而写的HVMOP，进入staging。QEMU部分也发到qemu-devel了，但是QEMU开发者说模拟MSIX的注入还要再讨论一下，那我就等等吧。
3. 给libxl加上VirtIO Disk的支持，这个还要再完善。
现在其实VirtIO for HVM已经差不多了。下面要开始最难的VirtIO for pure PV的工作了。
Stefano说we don’t exactly know how long it is going to take，要考虑的东西还是挺多的：
1. Xenstore里面要写什么东西。
2. VirtIO要怎么初始化。
3. 底层相关实现的替换（evtchn等）。
而这些工作，又要和现有的功能和平相处。所以还要再了解现有的设计，才能提出合理的设计。
这两天就先看看现有代码。一是Linux kernel里面VirtIO的代码，二是Xen PV初始化的代码，三是参考PV net等等的Xenstore参数形式。再和Stefano及Konrad再深入讨论一下。
VirtIO for HVM进展
虽然我说要尽快尽快开始，但是实际上进展不快。因为很多时候都是在调其他的bug。Xen unstable和SeaBIOS配合的时候，对IRQ配置有分歧，所以最后IRQ的注入有问题。
由于对IO-APIC和LAPIC不熟悉，在这里卡了很久。最后还是Stefano把这个bug解决了。惭愧。
这个bug解决之后，反而VirtIO for HVM的主要问题已经解决了。只要在guest command line加上”pci=nomsi”，其实VirtIO网卡已经可以使用了。
Stefano告诉我，下一步是把VirtIO disk for HVM搞好——这得等到Ian Jackson把libxenlight重构之后才可以进行，主要是一些配置的parse和driver的问题。
再下一步，就是把MSI for emulated devices做好。现在Xen只支持向passthrough的设备注入MSI，但对模拟设备没有接口——这也是为什么前面说要加nomsi的原因。
其实debug好像也没有什么好办法，都是加printk之类的。但是别人为什么debug这么快，我为什么这么慢，这真是个问题——我对底层还不是特别了解。以后还要再加油。
如何移植VirtIO到Xen的HVM上
这只是一篇分析文章，没有具体代码。分析也不全面，可能会有错。权当自己的笔记。
把Anthony的QEMU-dm看了一下。以前写proposal的时候，主要看的是KVM的处理代码（kvm-all.c），马上开始干了，要先了解一下Xen的处理代码（xen-all.c）。两者的基本原理是一样的，只是一些命名和代码逻辑上有所区别。初步看来，这个阶段的工作难度相对来说不大，目前已经有比较清晰的思路了。
我在Porting VirtIO to Xen里面提到过，KVM的dispatch函数是kvm_cpu_exec()，它按照VMEXIT的原因去dispatch这些请求。Xen也必然会有自己的dispatcher。在xen_init()的最后，xen_vm_change_state_handler()被注册为QEMU的change state handler。顺着xen_vm_change_state_handler()又会注册cpu_handle_ioreq()作为event channel的handler。
cpu_handle_ioreq()会调用handle_buffered_iopage()和handle_ioreq()两个函数来处理IO请求。在handle_ioreq()中，又分为了几个IO类型。Xen的对它们的命名和KVM有所区别。但是简单对比一下就知道对应关系了。
1. KVM_EXIT_IO对应IOREQ_TYPE_PIO
2. KVM_EXIT_MMIO对应IOREQ_TYPE_COPY
…
VirtIO在向KVM注册设备的时候，用的就是直接写Virtual PCI的方法，也就是说会引发KVM＿EXIT＿MMIO。然后cpu_physical_memory_rw()会被调用，在这个函数里面，对IO mem的处理是由一些由用户自己注册的函数来完成的。
VirtIO的handler注册到IO mem区域是一个比较复杂的过程。主要涉及的文件有pci.c和exec.c。在pci.c中，有一个pci_update_mappings()，它会调用cpu_register_physical_memory()；cpu_register_physical_memory()又调用cpu_register_physical_memory_offset()来更新IO mem的映射。这些handler在初始化的时候通过cpu_register_io_memory都注册到到对应的函数数组中，io_mem_write[]和io_mem_read[]。其他有关的数组还有io_mem_opaque[]。
io_mem_write[]的类型是CPUWriteMemoryFunc，io_mem_read[]的类型是CPUReadMemoryFunc，这两个类型和virtio_ioport_write()及virtio_ioport_read()的类型吻合。（有相应的typedef可查）
虽然我没有完全很详细了了解QEMU的设备注册机理，但是我初步猜测最后这些处理一定是会进入到virtio_ioport_wirte()，到实际写代码的时候会进行验证。
于是可以得出结论，注册到Xen的时候，控制逻辑应该会进入到IOREQ＿TYPE＿COPY中。同理，VirtIO在KVM中要触发事件（kick），也是直接写Virtual PCI的。最后的逻辑还是会进入同样的地方。现在如何注册、如何触发事件已经有思路了。实际上，这些控制逻辑要改动的地方不会太多。因为最后它们都会被dispatch到VirtIO的handler去处理。我们最切实的关注点应该是VirtIO底层的实现函数（比如说vp_notify()）。目前的情况是，这些handler直接使用了KVM的的功能。据Stefano的说法，最好把这些函数换成QEMU-generic的函数，相当于再加一层glue，然后再分别为KVM和Xen实现底层的处理逻辑。
Virtual PCI要改吗？不用。就算分析到这里，我也没有对QEMU到底是怎么注册设备完全了解。但是按照目前的情况来看，了解到这里也基本够用了。我要关注的，还是VirtIO用到了哪些KVM的接口，这些是要真正着手修改的地方。Xen tools改动的地方不大。
我的打算是尽快开始写代码，没想到开发环境却怎么也没有完全搞好。先是VGA console不工作，搞了几天。Upstream QEMU和Xen QEMU不完全相同，也调了两天，现在是可以用了，但是也还是有问题。这些天解决了不少问题，但都不是核心问题，所以多少有点烦燥。对于时间的估计，我不会太乐观。貌似搞底层开发的不可预料的情况比较多，很令人烦燥。只能说尽快开始搞，多争取点时间了。
Porting VirtIO to Xen
by Wei Liu <liuw #SPAMFREE# liuw #DOT# name>
Table of Contents
1 Overview
VirtIO is a unified paravirtualized IO framework created by Rusty Russell. It’s not hypervisor-specified, but mainly used in KVM. It is possible to port VirtIO to Xen without much effort.
This article is organized as serveral sections. Section 1 discovers how VirtIO is used in KVM. I will pay much attention to code analysis. Section 2 discusses how we can port VirtIO to Xen, both for normal PV and PV-on-HVM. Section 3 illustrates what performance tests will be done. Section 4 introduces porting plan for Spice (spice-space.org).
2 How VirtIO is used in KVM
This topic can be divided into three subtopics.
2.1 The role of KVM
KVM acts as hypervisor. It is responsible for capturing events then passing events to QEMU. I’m not going to illustrate how it works because this is out of our scope.
2.2 How VirtIO is used in Linux kernel
There are two key perspectives in creating a cross-domain communication channel:
- how to deliver events, e.g. Xen’s event channel, KVM’s handler to trap VM_EXIT and event dispatcher.
- how to share data, e.g. Xen’s ring buffer, VirtIO’s virtqueue.
It is obvious that Xen provide these two perspectives out of the box. However, VirtIO only provides mechanism to data-sharing, and notification is left for the user. VirtIO also registers as a bus inside kernel, which resembles XenBus.
The core structure in VirtIO is virtqueue. It contains a vring (just like ring buffer) and some other information. One thing worth mentioning is that virtqueue alse wraps up two important function pointers, one for notification, which is exactly what we need to replace, the other is for callback, which is somewhat irrelevant to porting.
Let’s take virtio network device as an example.
First thing first, virtio network in Linux kernel is implemented as a PCI device, so it is necessary to implement a virtio pci bus. See drivers/virtio/virtio_pci.c for details. When porting to Xen, it might be necessary to replace this PCI bus with XenBus. (However, it might be left unchanged in PV-on-HVM, we need to do a VM_EXIT and trap into hypervisor anyway.)
Source of virtio network lies in drivers/virtio_net.c . virtnet_probe() is responsible for probing. In this function, virtnet is setup with at least two vrings “input” and “output” and an optional vring for “control”. Callback for “input” is skb_recv_done() and callback for “output” is skb_xmit_done() .
After calling vdev->config->find_vqs(), these 2 or 3 vrings are setup. If we trace down this function – it lies in virtio_pci.c as vp_find_vqs() – we can find that it consequently calls vp_try_to_find_vqs(), setup_vq() and request_irq() .
In setup_vq(), the framework actually allocates the vring for data sharing. It is worth noting that the notify function is vp_notify(), which directly writes queue_index to (vp_dev->ioaddr+VIRTIO_PCI_QUEUE_NOTIFY) to generate a VM_EXIT. So that hypervisor can catch the event and dispatch it.
And the requested irqs are used to invoke the callback functions, i.e. skb_xmit_done() and skb_recv_done() .
2.3 How VirtIO is used in QEMU
QEMU runs on top of KVM and it interacts with KVM via /dev/kvm . KVM has to cooperate with QEMU. Actually, a virtual machine in KVM is merely a process.
VM instructions execute natively on CPU. However, when a VM executes some sensitive instruction, it will be trapped by KVM. Then KVM passes this instruction to QEMU to emulate / handle it.
KVM hands over the instruction to QEMU in kvm_cpu_exec(), which is in kvm-all.c . There is a `switch` on the exit_reason. Exit reasons include (port) IO, interrupt and MMIO, etc.
QEMU has full access of guest’s memory. But it has to grab the virtqueue inside VM first to communicate with vritio_net. When VM calls setup_vq(), it voluntarily writes virtqueue’s address to (vp_dev->ioaddr+VIRTIO_PCI_QUEUE_PFN), which will be trapped by KVM. KVM passes it to QEMU. QEMU then calls virtio_ioport_write() -> virtio_queue_set_addr() to set VRing, which is the control structure used in VirtIO in QEMU. This is how QEMU and VM create their data-sharing channel.
As for notification channel, it seems much easier. As mentioned above, VM writes index to VIRTIO_PCI_QUEUE_NOTIFY. It is trapped by virtio_ioport_write(), then passed to virtio_queue_notify(). In virtio net’s case, this request is finally handled by virtio_net_handle_rx() or virtio_net_handle_tx_{timer,bh}() .
There are many VirtIO-related files reside in the hw/ subdirectory. And also many other files in Linux kernel’s directory.
3 How to port VirtIO to Xen
Now that we’ve got our first impression on VirtIO. It time to discuss how we can port it to Xen.
3.1 PV-on-HVM
It is obvious that in the PV-on-HVM case, things are more or less the same as they are in KVM. Xen traps VM_EXIT and passes exception to QEMU. QEMU emulates. Then Xen sends result, VM resumes running.
3.1.1 How to grab virtqueue address
Xen utilizes QEMU as KVM does. So Linux kernel can stay untouched. Xen captures guest’s write to PCI configuration space, then QEMU will handle changes in VirtIO configuration.
3.1.2 How to deliver event
In VirtIO’s current virtual PCI implementation ($QEMU/hw/virtio-pic.c), it uses KVM’s event notification functions like kvm_set_ioeventfd_pio_word() and kvm_has_many_ioeventfds(). It is necessary to replace them with corresponding implementation in Xen. Anthony’s QEMU-dm ships with xen-all.c , it might give me some hints on implementation.
3.2 Normal PV
When it comes to normal PV case, we will have to do things in different way. I will try my best to detail what should be done and how it is done. However, this is just a rough design, things may change when implementing.
3.2.1 How to grab virtqueue address
In Xen’s covention, it is common for Dom0 / DomU to use Xenstore to expose their public information like netfront/netback, blkfront/blkback, etc. So it is a good idea to use Xenstore to expose VirtIO information. Xenstore’s well-defined API will greatly reduce work needed.
Down to implementation level, it is necessary to replace virtual PCI bus with XenBus. VirtIO utilizes virtual PCI to configure network device. However, in normal PV case, it is not necessary to expose a virtual PCI device to VM. We can follow the pattern how netfront and netback establish their channel.
QEMU-dm from Anthony has functions to manipulate Xenstore, that should help a lot. It also has xen_nic.c, which can greatly inspire how I can implement a VirtIO network for Xen.
3.2.2 How to deliver event
No doubt that event channel is the best choice. Anthony’s QEMU-dm contains a file named xen_backend.c, which is used for event handling. Linux kernel has event channel handling functions, too. (drivers/xen/{events,evtchn}.c)
So, just replace any notification-related function with Xen’s implementation. That’s the plan.
3.3 Other stuff
In PV-on-HVM case, QEMU needs to emulate. In normal PV case, no emulation is needed. Either case, QEMU works as backend dispatcher for VirtIO. Once the channel between two VMs are established, QEMU is supposed to work out of the box. However, I can’t be too optimistic here, it might require some work, such as rewriting and debugging some functions.
3.4 Knowledge required
To be honest, QEMU’s concept (like proxy / virtual device management) is somewhat strange to me. There are some high level design document, but they are just too high-level. A thorough understanding of QEMU’s internal is required.
Knowledge of hareware virtualization is also required. I need to understand how Xen implements HVM interface and choose the right function for certain functionality.
Knowledge of XenBus configuration is a must in normal PV porting. I’ve read about it before, so this is the easier part.
4 Performance tests
Performance tests will be run with industrial standard software like kernbench, ioperf and netperf. Testsuits will be run on several different configurations:
- Native Linux, CPU, disk and network.
- Xen with normal PV VirtIO support, CPU, disk and network.
- Xen with PV-on-HVM VirtIO support, CPU, disk and network.
- Xen with original PV support, CPU, disk and network.
- KVM with VirtIO support, CPU, disk and network.
And a short report will be written based on the result, which compares between outcoming data and analyzes advantages / disadvantages between configurations.
5 Porting of Spice
Spice will be ported to Xen’s HVM environment as a real-world testsuit. According to its design, Spice communicates with QEMU via Virtual Device Interface (VDI). Spice client and server run entirely in userland (correct me if I’m wrong, I’m not Spice expert). If we are able to run QEMU with QXL or any other VirtIO devices on Xen, it would not be so hard to get Spice running on Xen.
AFAIK, QXL in QEMU (hw/qxl.c) uses its own paravirtualized ring implementation. It also use qemu_set_irq() to deliver event. So the main idea is to replace this implementation with Xen’s ones, which is already done in porting VirtIO.
The plan is to run Spice with our modified QEMU and eliminate any bugs encountered.
6 Reference
- Linux kernel 2.6.38.2
- QEMU-dm, git://xenbits.xen.org/people/aperard/qemu-dm.git
- Xen-unstable, git://xenbits.xen.org/xen-unstable.git
- Spice project, spice-space.org
HTML generated by org-mode 6.21b in emacs 23