Weblog
06/06/06
Strange Performance with sendmsg() and IPv6 Flow Labels Under Linux
Category: General
As most of you know, I have been playing around heavily with IPv6 flow labels under Linux lately due to my master thesis. My newest finding is that performance of sendmsg() is really bad when one specifies the sin6_flowinfo field of the sockaddr_in6 structure. My test application is a modified version of D-ITG from the University of Napels. The application normally uses a connect()/send() scheme, even when sending UDP packets. However, as I just wanted to hack in QoS-support, I modified the scheme to use sendmsg() rather than send() and specify the destination address on each call. This seemed to work fine up to the point where I enabled flow labels for the socket and filled in the sin6_flowinfo field. Suddenly I could send no more than 90 packets a second, as apposed to the 2000 packets I was toying around with.
I feared that I had some hidden overhead somewhere in my QoS system. Luckily I found out that the problem was with the sendmsg() call. It seems that specifying a destination address with flow labels triggers some awful slow operation in the kernel.
My, rather unqualified, guess is that the kernel copies hop-by-hop headers and destination headers from the flow label into the packet on every sendmsg() call. If one instead uses the connect()/send() scheme, this data copy is only performed once. Wether this is true or not I can't say, but using connect() first and specifying the flow label there and using send() to send the messages works beautifully. I can now send thousands of packets a second again :-)
Modified: Wed 19-Dec-07 15:33:50 PST