We can use a space separator to group related apps together in macOS dock.

To add a separator to the dock, run the following command in the Terminal.

defaults write com.apple.dock persistent-apps -array-add '{"tile-type"="spacer-tile";}'; killall Dock

This will add a spacer tile to the dock which can then be positioned within the dock like any normal app icon.