WPF ListView with ComboBoxes in header: keyboard navigation

I've got a WPF ListView with ComboBoxes in the headers (meant for filtering in the list, see sample below). Keyboard navigation is awkward and I cannot get it right:
 
If the user presses cursor(arrow)-up when the first item in the list is selected, focus moves to the ComboBox in the header of the first column.
And then it's trapped there, pressing down walks the elements in the ComboBox; it does not moves focus back to the list.
Pressing TAB moves focus to the next control in the window. There doesn't seem to be a way to get focus to the list elements anymore by keyboard.
I guess what I want is that a) pressing 'arrow up' on the first element of the list does not transfer focus to the header, and b) that the list and header part can be separately reached by tabbing into them. I cannot seem to get this right.
 
Can this be done?
 
Sample code:

<ListView>
    <ListView.View>
        <GridView>
            <GridViewColumn>
                <GridViewColumn.Header>
                    <ComboBox Width="150">
                        <ComboBoxItem Content="Filter Item 1" />
                        <ComboBoxItem Content="Filter Item 2" />
                    </ComboBox>
                </GridViewColumn.Header>
            </GridViewColumn>
 
            <GridViewColumn>
                <GridViewColumn.Header>
                    <ComboBox Width="150">
                        <ComboBoxItem Content="Filter Item 1" />
                        <ComboBoxItem Content="Filter Item 2" />
                    </ComboBox>
                </GridViewColumn.Header>
            </GridViewColumn>
        </GridView>
    </ListView.View>
    <ListViewItem Content="Item 1" />
    <ListViewItem Content="Item 2" />
    <ListViewItem Content="Item 3" />
    <ListViewItem Content="Item 4" />
</ListView>
 
<TextBox or other control />
 
Thanks!