Yeah, this is why you either need to disable the android usb driver and use default linux ones (enabling which, broke the compile for the vendor kernel on my device)

Or patch the android stuff to add HID support: https://github.com/pelya/android-keyboard-gadget