Thanks for testing my work
Firstly I would like to thank you who have tested my work. I want to thank especially to Roman Bogorodskiy and Arne Meyer who were the first to notice that my scheduler has problems with playing audio under high cpu load. There is a good chance that I would not have noticed it myself because when I do coding, I do not usually listen to music. And when I am runnning FreeBSD, I usually do the coding:-)
What was the problem
The problem with the scheduler was that if you were playing some audio and running some cpu intensive applications at the same time, the music would become highly distorted – kind of slowed down.
Let’s now have a look at an application that plays music from the scheduler’s point of view. For the scheduler it is just an application that sleeps most of the time. Sometimes it wakes up, probably sends some data to the audio card and then quickly goes to sleep again.
What I firstly did is that I added some logging to my scheduler code. I logged some scheduler relevant events like context switch, waking up of a thread and similar. The timestamp was also included in the log so that I could see where the delays come from. Here is a short excerpt of the logs I got:
493778161690 "mpg123" wakes up on cpu 0
493778162080 ipi send to cpu 1
493778163940 ast on cpu 1
493778165150 "skuska" switch off cpu 1
493778166760 "irq256: mskc0" switch on cpu 1 with prio: 16
493778246160 "irq256: mskc0" switch off cpu 1
493778246490 "swi4: clock" switch on cpu 1 with prio: 52
493778249780 "initial thread" wakes up on cpu 1 with deadline 188337
493778252570 "swi4: clock" switch off cpu 1
493778253890 "swi6: Giant taskq" switch on cpu 1 with prio: 60
493778267110 "initial thread" wakes up on cpu 1
493778271830 "swi6: Giant taskq" switch off cpu 1
493778274240 "mpg123" switch on cpu 1 with prio: 76
493778281920 "mpg123" sleep on cpu 1
493778283780 "mpg123" switch off cpu 1
The first column contains a nanosecond timestamp. skuska is the name of a cpu-intensive program I was running. And mpg123 was my music playing application.
This is what happened according to the log. Firstly we see that the mpg123 application wakes up on cpu 0. Because the scheduler knows that this application was running last time on cpu 1, and currently on cpu 1 there is running the skuska application which does no have high priority nor small virtual deadline, it sends an inter processor interrupt (IPI) to the cpu 1. The third line refers to that IPI being processed on cpu 1. The next line shows that the skuska application is descheduled – taken off the cpu. So far so good and there were not big delays between these events.
However, next we see that instead of the mpg123 application, the scheduler chooses firstly some high priority kernel threads (irq256, swi4 and swi6). Only after these threads are processed, the mpg123 application is run. And this is the cause of the delays in audio playing.
So this is the bug: Some high priority threads get queued up in the scheduler instead of being run immediately after they are set as schedulable. And then when our music application (or some other interactive process) wakes up and preempts some thread, it does not get scheduled because the high priority kernel threads take precedence.
How I fixed the bug
Once I knew what was the problem, it was easy to fix it. Now, whenever a thread is inserted into the scheduler, the scheduler checks the priority of the thread and the priority of the currently running threads. If the new thread has high priority, it is not just registered in the scheduler data structures, but it is sent directly to the cpu.
Feedback is welcome
Although I think the bug is fixed, it would be nice if somebody could confirm it in the discussion. You can download the patch from the latest patch available section. Also please note in the discussion of that section that fidaj has ported the scheduler to the CURRENT branch. I would like to thank him here. And if he is so kind to create a new patch with this bug fixed I will be happy to include it in the Latest patch available section so that people can find it easily.