Hello Bendik,
Since the IE10 works fine and the issue only happens in IE11, and you have implemented all SAP notes, then most probably the backend logics are ok. But maybe you can make st12 traces and httpwatch traces to see if there are any differences on time consuming, which part takes time most.
Please check IE11 version, is it the latest?
Best regards, Corrine