Generally VS Code uses ctrl+j to open Terminal so I created a keybinding to switch with ctrl+k combination, like below at keybindings.json:
[{"key": "ctrl+k","command": "workbench.action.terminal.focus"},{"key": "ctrl+k","command": "workbench.action.focusActiveEditorGroup","when": "terminalFocus"}]