Sending different requests to different servers based on protocal via dns
Is it possible to route dns to different servers based on the protocol of the request without using a proxy server?
For example wss://example.com goes to 1 server and https://example.com goes to a different one.
In principle no, this is not possible. Although there are exceptions.
When you an application and request a network address, e.g. example.com the application will (typically) pass on a request to the OS to open a connection to that address.
Because the OS can only make a connection to an IP Address, the first thing it does is to make a DNS request to find out the address it can connect to. There is no mechanism at all, to tell that DNS request what protocol is being requested. At the point the OS makes the DNS request it is simply, 'What is the address for example.com' there is no space in the body of the message for more information.
In that sense it is not possible.
However there are a few services that use SRV records to find the server they need to connect to.
In these cases the application will say to the OS 'get me the SRV record for _somefancyservice.example.com' When it has that record, it will then send another request to the OS to open a connection the whatever it found in the SRV record.
So you could, in theory, write an application that performed the SRV record query first, and then opened a connection to whatever was returned.
I wrote a more detailed answer specifically about wss, which you may find interesting.