I think I figured out some workaround. I made following style
<Style x:Key="{x:Type ListBoxItem}" TargetType="ListBoxItem">
<Setter Property="Focusable" Value="False"/>
</Style>
keyboard navigation seems to work now, only odd behaviour is now that when you set selected index / item from code behind listbox selecteditem highligths again something else than my button but I think I wont do codebehind selections.