In 2.7 we refactored several areas of communication between master daemon and remote agents. The changes are “under the hood”, tackling issues that take a toll in certain scenarios or high loaded setups using distributed indexes with remote nodes.
Asynchronous DNS
This has been a sort of a problem especially for users using cloud infrastructure. On Linux the daemon will use getaddrinfo_a() if available for asynchronous DNS. This means we don’t have to actively wait for an answer, instead we schedule the task and when (if) DNS answers we continue - or abort after a while by timeout. On systems where getaddrinfo_a is not available we issue a new thread dedicated to DNS resolving (which as well works asynchronous).
Remote agents network threads
When a query comes to a distributed index with remote agents, we fired many threads to connect and send query then wait all of them to finish in order to advance to next stages. Problem here is if one node responds slow it can slow down the entire process. Instead to send the query for each node that responded, they would had to wait until everyone was ready. Overall, this could mean precious time idling and increasing the whole query response time. Now, each connection works independent from start to termination and their part in execution is not influenced (stalled) by possible slower ones. Related to this, a first step in previous versions was to truly decouple the blackhole agents from the normal agents as, in the past, while master didn’t had to wait for response from a blackhole agent, it was still waiting for an initial response from it - and a problematic blackhole could slowed down the whole query.
API proto
A third improvement regarding networking is about the API proto. In previous versions, the connection between a master and a remote agent started with a ‘handshake’ - a small packet containing API version. The other end of line would answer with it’s version as well and if everything is ok, the query starts. This handshake was setup in the idea that address/port in the configuration of agent could be wrong and not to send a bigger packet in this case. However we saw it as a waste of time (consider network latencies) and bandwidth to just check if the user set up things correctly. This is something less probable to happen and even if is, it can be fixed by user. Instead, now, this ‘handshake’ is sent along with the query in a single packet, reducing the total time spent in querying a remote node. If indeed the remote host is wrong, we just sent a packet in vain, but this can be noticed by user and correct it - we don’t have to always check this. This proto change maintains compatibility with older proto, thus if the master is upgraded to 2.7+ it will still be able to communicate with remote agents on < 2.7.
TFO for master-nodes
A last improvement is about TCP Fast Open - in previous versions we added the possibility for clients to use tfo connections when connecting to a search daemon. Now TFO connections can be also used in a distributed index between master and remote agents. No change in configuration is required, the daemons will detect if TFO is available on the host system and just use it.