Lore

[...]

This week people in our community confronted me about my lifetime of not understanding emotions. My flippant attacks in emails have been both unprofessional and uncalled for. Especially at times when I made it personal. In my quest for a better patch, this made sense to me. I know now this was not OK and I am truly sorry.

The above is basically a long-winded way to get to the somewhat painful personal admission that hey, I need to change some of my behavior, and I want to apologize to the people that my personal behavior hurt and possibly drove away from kernel development entirely.

I am going to take time off and get some assistance on how to understand people’s emotions and respond appropriately.

[...]

[...]

C++ is a horrible language. It's made more horrible by the fact that a lot of substandard programmers use it, to the point where it's much much easier to generate total and utter crap with it. Quite frankly, even if the choice of C were to do *nothing* but keep the C++ programmers out, that in itself would be a huge reason to use C.

[...]

In other words, the only way to do good, efficient, and system-level and portable C++ ends up to limit yourself to all the things that are basically available in C. And limiting your project to C means that people don't screw that up, and also means that you get a lot of programmers that do actually understand low-level issues and don't screw things up with any idiotic "object model" crap.

So I'm sorry, but for something like git, where efficiency was a primary objective, the "advantages" of C++ is just a huge mistake. The fact that we also piss off people who cannot see that is just a big additional advantage.

[...]

[...]

This project is a complete rewrite of the Linux task scheduler. My goal is to address various feature requests and to fix deficiencies in the vanilla scheduler that were suggested/found in the past few years, both for desktop scheduling and for server scheduling workloads.

[...]

i'd like to give credit to Con Kolivas for the general approach here: he has proven via RSDL/SD that 'fair scheduling' is possible and that it results in better desktop scheduling. Kudos Con!

[...]

CFS's design is quite radical: it does not use runqueues, it uses a time-ordered rbtree to build a 'timeline' of future task execution, and thus has no 'array switch' artifacts (by which both the vanilla scheduler and RSDL/SD are affected).

CFS uses nanosecond granularity accounting and does not rely on any jiffies or other HZ detail. Thus the CFS scheduler has no notion of 'timeslices' and has no heuristics whatsoever. There is only one central tunable:

/proc/sys/kernel/sched_granularity_ns

which can be used to tune the scheduler from 'desktop' (low latencies) to 'server' (good batching) workloads. It defaults to a setting suitable for desktop workloads. SCHED_BATCH is handled by the CFS scheduler module too.

[...]

[...]

I'd also like to point out that unlike every single horror I've ever witnessed when looking closer at SCM products, git actually has a simple design, with stable and reasonably well-documented data structures. In fact, I'm a huge proponent of designing your code around the data, rather than the other way around, and I think it's one of the reasons git has been fairly successful (*).

So it's easy enough to just write whatever Java code or something to just access the databases yourself. The object model of git may be smart, but it's neither proprietary nor patented. I suspect it's often a lot easier to integrate git into other projects _that_ way, rather than try to actually port the code itself.

[...]

(*) I will, in fact, claim that the difference between a bad programmer and a good one is whether he considers his code or his data structures more important. Bad programmers worry about the code. Good programmers worry about data structures and their relationships.

Technically, it's not data=writeback. It's more like XFS's delayed allocation; I've added workarounds so that files that which are replaced via truncate or rename get pushed out right away, which should solve most of the problems involved with files becoming zero-length after a system crash.

[...]

With ext2 after a system crash you need to run fsck. With ext4, fsck isn't an issue, but if the application doesn't use fsync(), yes, there's no guarantee (other than the workarounds for replace-via-truncate and replace-via-rename), but there's plenty of prior history that says that applications that care about data hitting the disk should use fsync(). Otherwise, it will get spread out over a few minutes; and for some files, that really won't make a difference.

For precious files, applications that use fsync() will be safe --- otherwise, even with ext3, you can end up losing the contents of the file if you crash right before 5 second commit window. At least back in the days when people were proud of their Linux systems having 2-3 year uptimes, and where jiffies could actually wrap from time to time, the difference between 5 seconds and 3 minutes really wasn't that big of a deal. People who really care about this can turn off delayed allocation with the nodelalloc mount option. Of course then they will have the ext3 slower fsync() problem.

You are right that data=writeback and delayed allocation do both mean that data can get pushed out much later than the metadata. But that's allowed by POSIX, and it does give some very nice performance benefits.

[...]

Just to clarify, I'm only on this list because I acked 3 smaller, independent memcg cleanup patches in this series. I have repeatedly expressed strong reservations over folios themselves.

The arguments for a better data interface between mm and filesystem in light of variable page sizes are plentiful and convincing. But from an MM point of view, it's all but clear where the delineation between the page and folio is, and what the endgame is supposed to look like.

One one hand, the ambition appears to substitute folio for everything that could be a base page or a compound page even inside core MM code. Since there are very few places in the MM code that expressly deal with tail pages in the first place, this amounts to a conversion of most MM code - including the LRU management, reclaim, rmap, migrate, swap, page fault code etc. - away from "the page".

However, this far exceeds the goal of a better mm-fs interface. And the value proposition of a full MM-internal conversion, including e.g. the less exposed anon page handling, is much more nebulous. It's been proposed to leave anon pages out, but IMO to keep that direction maintainable, the folio would have to be translated to a page quite early when entering MM code, rather than propagating it inward, in order to avoid huge, massively overlapping page and folio APIs.

It's also not clear to me that using the same abstraction for compound pages and the file cache object is future proof. It's evident from scalability issues in the allocator, reclaim, compaction, etc. that with current memory sizes and IO devices, we're hitting the limits of efficiently managing memory in 4k base pages per default. It's also clear that we'll continue to have a need for 4k cache granularity for quite a few workloads that work with large numbers of small files. I'm not sure how this could be resolved other than divorcing the idea of a (larger) base page from the idea of cache entries that can correspond, if necessary, to memory chunks smaller than a default page.

[...]

As an MM stakeholder, I don't think folios are the answer for MM code.

[...]

The Linux kernel has _always_ been under the GPL v2. Nothing else has ever been valid.

The "version 2 of the License, or (at your option) any later version" language in the GPL copying file is not - and has never been - part of the actual License itself. It's part of the _explanatory_ text that talks about how to apply the license to your program, and it says that _if_ you want to accept any later versions of the GPL, you can state so in your source code.

The Linux kernel has never stated that in general. Some authors have chosen to use the suggested FSF boilerplate (including the "any later version" language), but the kernel in general never has.

In other words: the _default_ license strategy is always just the particular version of the GPL that accompanies a project. If you want to license a program under _any_ later version of the GPL, you have to state so explicitly. Linux never did.

[...]

After some arm twisting from Christoph, I finally caved and divorced the aio-poll patches from aio/libaio itself. The io_uring interface itself is useful and efficient, and after rebasing all the new goodies on top of that, there was little reason to retail the aio connection.

Hence io_uring was born. This is what I previously called scqring for aio, but now as a standalone entity. Patch #5 adds the core of this interface, but in short, it has two main data structures:

[...]

The SQ ring is an array of indexes into an array of io_uring_iocbs, which describe the IO to be done.

[...]

Both of these rings are mapped into the application through mmap(2), at special magic offsets. The application manipulates the ring directly, and then communicates with the kernel through these two system calls:

[...]

In terms of features, this has everything that the prior aio-poll postings did. Later patches add support for polled IO, fixed buffers, kernel side submission and polling, buffered aio, etc. Also a number of bug fixes in here from previous postings.

[...]

Ok, as a number of people are already aware (and in some cases have been aware over the last several weeks), we've been trying to work out a conflict over BK usage over the last month or two (and it feels like longer ;). That hasn't been working out, and as a result, the kernel team is looking at alternatives.

[...]

NOTE! BitKeeper isn't going away per se. Right now, the only real thing that has happened is that I've decided to not use BK mainly because I need to figure out the alternatives, and rather than continuing "things as normal", I decided to bite the bullet and just see what life without BK looks like. So far it's a gray and bleak world ;)

So don't take this to mean anything more than it is. I'm going to be effectively off-line for a week (think of it as a normal "Linus went on a vacation" event) and I'm just asking that people who continue to maintain BK trees at least try to also make sure that they can send me the result as (individual) patches, since I'll eventually have to merge some other way.

[...]

[...]

I decided to just bite the bullet, and call the next version 3.0. It will get released close enough to the 20-year mark, which is excuse enough for me, although honestly, the real reason is just that I can no longe rcomfortably count as high as 40.

The whole renumbering was discussed at last years Kernel Summit, and there was a plan to take it up this year too. But let's face it - what's the point of being in charge if you can't pick the bike shed color without holding a referendum on it? So I'm just going all alpha-male, and just renumbering it. You'll like it.

[...]

NOTHING. Absolutely nothing. Sure, we have the usual two thirds driver changes, and a lot of random fixes, but the point is that 3.0 is *just* about renumbering, we are very much *not* doing a KDE-4 or a Gnome-3 here. No breakage, no special scary new features, nothing at all like that. We've been doing time-based releases for many years now, this is in no way about features. If you want an excuse for the renumbering, you really should look at the time-based one ("20 years") instead.

[...]

[...]

The point of all this churn is to allow filesystems and the page cache to manage memory in larger chunks than PAGE_SIZE. The original plan was to use compound pages like THP does, but I ran into problems with some functions that take a struct page expect only a head page while others expect the precise page containing a particular byte.

This pull request converts just parts of the core MM and the page cache. For 5.16, we intend to convert various filesystems (XFS and AFS are ready; other filesystems may make it) and also convert more of the MM and page cache to folios. For 5.17, multi-page folios should be ready.

The multi-page folios offer some improvement to some workloads. The 80% win is real, but appears to be an artificial benchmark (postgres startup, which isn't a serious workload). Real workloads (eg building the kernel, running postgres in a steady state, etc) seem to benefit between 0-10%. I haven't heard of any performance losses as a result of this series. Nobody has done any serious performance tuning; I imagine that tweaking the readahead algorithm could provide some more interesting wins. There are also other places where we could choose to create large folios and currently do not, such as writes that are larger than PAGE_SIZE.

[...]

Add memory folios, a new type to represent either order-0 pages or the head page of a compoud page. This should be enough infrastructure to support filesystems converting from pages to folios.

[...]

[...]

This pull request contains a single commit that adds a memory model to the tools directory. This memory model can (roughly speaking) be thought of as an automated version of memory-barriers.txt. It is written in the "cat" language, which is executable by the externally provided "herd7" simulator, which exhaustively explores the state space of small litmus tests.

[...]

This memory model has matured to the point where it would be good to include it in the Linux kernel, for example, to allow it to track changes as new hardware and use cases are added. We expect the rate of change to be similar to that of Documentation/memory-barriers.txt.

[...]

Hello everybody out there using minix -

I'm doing a (free) operating system (just a hobby, won't be big and professional like gnu) for 386(486) AT clones. This has been brewing since april, and is starting to get ready. I'd like any feedback on things people like/dislike in minix, as my OS resembles it somewhat (same physical layout of the file-system (due to practical reasons) among other things).

I've currently ported bash(1.08) and gcc(1.40), and things seem to work. This implies that I'll get something practical within a few months, and I'd like to know what features most people would want. Any suggestions are welcome, but I won't promise I'll implement them :-)

Linus ([email protected])

PS. Yes - it's free of any minix code, and it has a multi-threaded fs. It is NOT protable (uses 386 task switching etc), and it probably never will support anything other than AT-harddisks, as that's all I have :-(.

[...]

The problem is, it actually results in the wrong dynamics and psychology in the system. First off, it makes the implicit assumption that there is an "up" and "down" in the food-chain, and I think that's wrong. It's increasingly a "network" in the kernel. I'm less and less "the top", as much as a "fairly central" person. And that is how it should be. I used to think of kernel development as a hierarchy, but I long since switched to thinking about it as a fairly arbitrary network.

The other thing it does is that it implicitly puts the burden of quality control at the upper-level maintainer ("I'll pick the good things out of your tree"), while _not_ being able to cherry-pick means that there is pressure in both directions to keep the tree clean.

And that is IMPORTANT. I realize that not cherry-picking means that people who want to merge upstream (or sideways or anything) are now forced to do extra work in trying to keep their tree free of random crap. And that's a HUGELY IMPORTANT THING! It means that the pressure to keep the tree clean flows in all directions, and takes pressure off the "central" point. In onther words it distributes the pain of maintenance.

[...]

[...]

But the kind of people for whom the machine they are on IS THEIR ONLY MACHINE - whether it be a workstation, a laptop, or a cellphone - there is absolutely zero situation where "let's just kill the machine" is *EVER* approproate. Even a BUG_ON() will try to continue as well as it can after killing the current thread, but it's going to be iffy, because locking etc.

So WARN_ON_ONCE() is the thing to aim for. BUG_ON() is the thing for "oops, I really don't know what to do, and I physically *cannot* continue" (and that is *not* "I'm too lazy to do error handling").

There is no room for PANIC. None. Ever.

The only thing there is are "I don't care about this machine because I've got 999,999 other machines, so I'd rather take one machine offline for analysis".

Understand? The "should I panic and reboot" is fundamentally not about the code, and it's not a choice that the kernel code gets to make. It's purely about the choice of the person maintaining the machine.

As a kernel developer, you do not EVER get to say "panic" or "kill the machine".

End of story.

Yes.

And no, that's not the position of the Linux kernel.

I realize that some people really dislike AI, but this is an area where I'm willing to absolutely put my foot down as the top-level maintainer.

Linux is not one of those anti-AI projects, and if somebody has issues with that, they can do the open-source thing and fork it.

Or just walk away.

AI is a tool, just like other tools we use. And it's clearly a useful one.

It may not have been that "clearly" even just a year ago, but it's no longer in question today.

There are other questions around AI (like what the economy of it will actually look like in the end), but "is it useful" is no longer one of those questions. Anybody who doubts that clearly hasn't actually used it.

Yes, it can also be a somewhat painful tool, both for maintainer workloads and just from a "it keeps finding embarrassing bugs" standpoint.

But the solution is not to put your head in the sand and sing "La La La, I can't hear you" at the top of your voice like some people seem to do.

The solution is to make sure those LLM tools _help_ maintainers instead of just causing them pain. There's no question on that side.

We're not forcing anybody to use it, but I will very loudly ignore people who try to argue against other people from using it.

And no, AI isn't perfect. But Christ, anybody who points to the problems at AI had better be looking in the mirror and pointing at themselves at the same time.

Because it's not like natural intelligence is always all that great either.

The kernel project has been and will continue to be about the technology.

Sure, the social angle of working on open source is important and often a very motivating part of the project, but in the end that's a side benefit, not the _point_ of the project.

This is *NOT* some kind of "social warrior" project, never has been, and never will be.

In the kernel community we do open source because it results in better technology, not because of religious reasons.

And so we make decisions primarily based on technical merit. Not fear of new tools.

now that new-year's parties are over things are getting boring again. For those who want to see and perhaps even try something more complex, i'm announcing this patch that is a pretty radical rewrite of the Linux scheduler for 2.5.2-pre6:

[...]

- fully O(1) scheduling. Are you tired of the recalculation loop blowing the L1 cache away every now and then? Do you think the goodness loop is taking a bit too long to finish if there are lots of runnable processes? This new scheduler takes no prisoners: wakeup(), schedule(), the timer interrupt are all O(1) algorithms. There is no recalculation loop. There is no goodness loop either.

- 'perfect' SMP scalability. With the new scheduler there is no 'big' runqueue_lock anymore - it's all per-CPU runqueues and locks - two tasks on two separate CPUs can wake up, schedule and context-switch completely in parallel, without any interlocking. All scheduling-relevant data is structured for maximum scalability. (see the benchmark section later on.)

- better SMP affinity. The old scheduler has a particular weakness that causes the random bouncing of tasks between CPUs if/when higher priority/interactive tasks, this was observed and reported by many people. The reason is that the timeslice recalculation loop first needs every currently running task to consume its timeslice. But when this happens on eg. an 8-way system, then this property starves an increasing number of CPUs from executing any process. Once the last task that has a timeslice left has finished using up that timeslice, the recalculation loop is triggered and other CPUs can start executing tasks again - after having idled around for a number of timer ticks. The more CPUs, the worse this effect.

Furthermore, this same effect causes the bouncing effect as well: whenever there is such a 'timeslice squeeze' of the global runqueue, idle processors start executing tasks which are not affine to that CPU. (because the affine tasks have finished off their timeslices already.)

The new scheduler solves this problem by distributing timeslices on a per-CPU basis, without having any global synchronization or recalculation.

[...]

- *two*, priority-ordered 'priority arrays' per CPU. There is an 'active' array and an 'expired' array. The active array contains all tasks that are affine to this CPU and have timeslices left. The expired array contains all tasks which have used up their timeslices - but this array is kept sorted as well. The active and expired array is not accessed directly, it's accessed through two pointers in the per-CPU runqueue structure. If all active tasks are used up then we 'switch' the two pointers and from now on the ready-to-go (former-) expired array is the active array - and the empty active array serves as the new collector for expired tasks.

[...]

[...]

So this email is not about some "Rust policy". This email is about a much bigger issue: as a maintainer you are in charge of your code, sure - but you are not in charge of who uses the end result and how.

You don't have to like Rust. You don't have to care about it. That's been made clear pretty much from the very beginning, that nobody is forced to suddenly have to learn a new language, and that people who want to work purely on the C side can very much continue to do so.

[...]

But that "wall of protection" basically goes both ways. If you don't want to deal with the Rust code, you get no *say* on the Rust code.

Put another way: the "nobody is forced to deal with Rust" does not imply "everybody is allowed to veto any Rust code".

[...]

[...]

When submitting patches created by a tool, everyone who does so submits them with wording like "found by tool XXX, we are not sure if this is correct or not, please advise." which is NOT what you did here at all. You were not asking for help, you were claiming that these were legitimate fixes, which you KNEW to be incorrect.

[...]

Our community welcomes developers who wish to help and enhance Linux. That is NOT what you are attempting to do here, so please do not try to frame it that way.

Our community does not appreciate being experimented on, and being "tested" by submitting known patches that are either do nothing on purpose, or introduce bugs on purpose. If you wish to do work like this, I suggest you find a different community to run your experiments on, you are not welcome here.

Because of this, I will now have to ban all future contributions from your University and rip out your previous contributions, as they were obviously submitted in bad-faith with the intent to cause problems.

*plonk*

This topic has come up countless times, and usually doesn't go anywhere. This time I thought I'd bring it up with a slightly narrower focus, updated for some developments over the last three years: we finally can make /dev/urandom always secure, in light of the fact that our RNG is now always seeded.

Ever since Linus' 50ee7529ec45 ("random: try to actively add entropy rather than passively wait for it"), the RNG does a haveged-style jitter dance around the scheduler, in order to produce entropy (and credit it) for the case when we're stuck in wait_for_random_bytes(). How ever you feel about the Linus Jitter Dance is beside the point: it's been there for three years and usually gets the RNG initialized in a second or so.

As a matter of fact, this is what happens currently when people use getrandom(). It's already there and working, and most people have been using it for years without realizing.

So, given that the kernel has grown this mechanism for seeding itself from nothing, and that this procedure happens pretty fast, maybe there's no point any longer in having /dev/urandom give insecure bytes. In the past we didn't want the boot process to deadlock, which was understandable. But now, in the worst case, a second goes by, and the problem is resolved. It seems like maybe we're finally at a point when we can get rid of the infamous "urandom read hole".

[...]

On systems that really do want unseeded randomness, we already offer getrandom(GRND_INSECURE), which is in use by, e.g., systemd for seeding their hash tables at boot. Nothing in this commit would affect GRND_INSECURE, and it remains the means of getting those types of random numbers.

This patch goes a long way toward eliminating a long overdue userspace crypto footgun. After several decades of endless user confusion, we will finally be able to say, "use any single one of our random interfaces and you'll be fine. They're all the same. It doesn't matter." And that, I think, is really something. Finally all of those blog posts and disagreeing forums and contradictory articles will all become correct about whatever they happened to recommend, and along with it, a whole class of vulnerabilities eliminated.

With very minimal downside, we're finally in a position where we can make this change.

[...]

[...]

It's a bug alright - in the kernel. How long have you been a maintainer? And you *still* haven't learnt the first rule of kernel maintenance?

If a change results in user programs breaking, it's a bug in the kernel. We never EVER blame the user programs. How hard can this be to understand?

To make matters worse, commit f0ed2ce840b3 is clearly total and utter CRAP even if it didn't break applications. ENOENT is not a valid error return from an ioctl. Never has been, never will be. ENOENT means "No such file and directory", and is for path operations. ioctl's are done on files that have already been opened, there's no way in hell that ENOENT would ever be valid.

[...]