Ah, ok. I thought that DisplayLink is just an option and you could alternatively use HDMI.
Since the “udlfb” driver for DisplayLink uses framebuffer fb1 you would have to edit “/usr/share/X11/xorg.conf.d/99-fbturbo.conf”:
sudo nano /usr/share/X11/xorg.conf.d/99-fbturbo.conf
and change
Option "fbdev" "/dev/fb0"
to
Option "fbdev" "/dev/fb1"