Reading/writing on sockets starts several iou-wrk "workers"

[armanbilge]

Consider the following fs2-io_uring program, which starts an echo server and connects a number of clients.

//> using lib "com.armanbilge::fs2-io_uring::0.1.0"
//> using platform "native"
// brew install liburing
//> using nativeCompile "-I/home/linuxbrew/.linuxbrew/include"
//> using nativeLinking "/home/linuxbrew/.linuxbrew/lib/liburing.a"

import cats.effect.*
import cats.syntax.all.*
import fs2.*
import fs2.io.uring.*
import fs2.io.uring.net.*

object App extends UringApp:
  def run(args: List[String]) =
    UringSocketGroup[IO].serverResource().use { (address, clients) =>
      val echoServer = clients.map { c =>
        Stream.exec(IO.println("client accepted")) ++
          c.reads.through(c.writes)
      }.parJoinUnbounded
      echoServer.compile.drain.background.surround {
        UringSocketGroup[IO]
          .client(address, Nil)
          .use { client =>
            IO.println("client connected") *>
              client
                .write(Chunk.constant(0.toByte, 1024))
                .foreverM
                .background
                .surround {
                  client.reads.compile.drain
                }
          }
          .parReplicateA_(args.head.toInt)
      }
    } as ExitCode.Success

If we start a 100 concurrent clients like so:

$ scala-cli --power package -f test.scala
$ ./App 100

We can observe a number of iou-wrk threads. I observed it to fluctuate between 10-50.

$ ps -T -p 29476 
    PID    SPID TTY          TIME CMD
  29476   29476 pts/1    00:06:12 App
  29476   32502 pts/1    00:00:00 iou-wrk-2458706
  29476   32513 pts/1    00:00:00 iou-wrk-2458706
  29476   32530 pts/1    00:00:00 iou-wrk-2458706
  29476   32531 pts/1    00:00:00 iou-wrk-2458706
  29476   32535 pts/1    00:00:00 iou-wrk-2458706
  29476   32536 pts/1    00:00:00 iou-wrk-2458706
  29476   32537 pts/1    00:00:00 iou-wrk-2458706
  29476   32538 pts/1    00:00:00 iou-wrk-2458706
  29476   32539 pts/1    00:00:00 iou-wrk-2458706
  29476   32540 pts/1    00:00:00 iou-wrk-2458706
  29476   32541 pts/1    00:00:00 iou-wrk-2458706
  29476   32546 pts/1    00:00:00 iou-wrk-2458706
  29476   32547 pts/1    00:00:00 iou-wrk-2458706
  29476   32548 pts/1    00:00:00 iou-wrk-2458706
  29476   32551 pts/1    00:00:00 iou-wrk-2458706

I'm not sure what to make of this. I'm pretty sure we don't want this, since more threads are bad for performance. But I also don't understand where these threads are coming from anyway! Reading / writing to sockets should be non-blocking ...

Best reading on this topic I'm aware of is:

• https://blog.cloudflare.com/missing-manuals-io_uring-worker-pool/

[armanbilge]

Hmmmm 🤔

IORING_FEAT_FAST_POLL
If this flag is set, then io_uring supports using an internal poll mechanism to drive data/space readiness. This means that requests that cannot read or write data to a file no longer need to be punted to an async thread for handling, instead they will begin operation when the file is ready. This is similar to doing poll + read/write in userspace, but eliminates the need to do so. If this flag is set, requests waiting on space/data consume a lot less resources doing so as they are not blocking a thread. Available since kernel 5.7.

[armanbilge]

See also netty/netty-incubator-transport-io_uring#156. Seems they are using a more "traditional" technique of polling before read/writing 🤔

[armanbilge]

A simpler reproducer, separated into an independent server and client. The server is an echo. The client writes but does not read. Both applications end up with an iou-wrk thread.

//> using lib "com.armanbilge::fs2-io_uring::0.1.0"
//> using platform "native"
// brew install liburing
//> using nativeCompile "-I/home/linuxbrew/.linuxbrew/include"
//> using nativeLinking "/home/linuxbrew/.linuxbrew/lib/liburing.a"

import cats.effect.*
import cats.syntax.all.*
import fs2.*
import fs2.io.uring.*
import fs2.io.uring.net.*
import com.comcast.ip4s.*

object Server extends UringApp.Simple:
  def run =
    UringSocketGroup[IO].serverResource(port = port"8080".some).use {
      (address, clients) =>
        val echoServer = clients.map { c =>
          Stream.exec(IO.println("client accepted")) ++
            c.reads.through(c.writes)
        }.parJoinUnbounded

        IO.println(address) *>
          echoServer.compile.drain
    }

//> using lib "com.armanbilge::fs2-io_uring::0.1.0"
//> using platform "native"
// brew install liburing
//> using nativeCompile "-I/home/linuxbrew/.linuxbrew/include"
//> using nativeLinking "/home/linuxbrew/.linuxbrew/lib/liburing.a"

import cats.effect.*
import cats.syntax.all.*
import fs2.*
import fs2.io.uring.*
import fs2.io.uring.net.*
import com.comcast.ip4s.*

object Client extends UringApp.Simple:
  def run =
    UringSocketGroup[IO].client(SocketAddress.fromString("localhost:8080").get).use { client =>
      IO.println("connected") *>
        client.write(Chunk.constant(0.toByte, 1024)).foreverM
    }

[armanbilge]

Linking to:

• When does io_uring create tasks or otherwise start iou-wrk threads? axboe/liburing#842

[armanbilge]

Seems the best answer we have right now is to just use a newer kernel 🙃

[armanbilge]

For the record I was observing issues with kernel 5.15.

Linux armanbilge-fs2iouring-v9qzh8jt6kk 5.15.0-47-generic #51-Ubuntu SMP Thu Aug 11 07:51:15 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux