CAP, Consistency, Availability, Partition tolerance & ES, Elastic Search

 CAP, Consistency, Availability, Partition tolerance

Consistency

Consistency means that all clients see the same data at the same time, no matter which node they connect to. For this to happen, whenever data is written to one node, it must be instantly forwarded or replicated to all the other nodes in the system before the write is deemed ‘successful.’

Availability

Availability means that that any client making a request for data gets a response, even if one or more nodes are down. Another way to state this—all working nodes in the distributed system return a valid response for any request, without exception.

Partition tolerance

A partition is a communications break within a distributed system—a lost or temporarily delayed connection between two nodes. Partition tolerance means that the cluster must continue to work despite any number of communication breakdowns between nodes in the system.
From: https://www.ibm.com/cloud/learn/cap-theorem

CA： Oracle， My SQL;

AP：Coach DB, Cassandra, DynamoDB;  E.g. Search ticks, purchase ticks.The train tickets, showing   remaining tickets when grabbing tickets, but when the order is placed for payment, there are no remaining tickets. Although this will reduce the user experience, it will not block the server with a large number of user requests.

CP: MongoDB, HBase, Redis;  E.g.  All payment transaction data must be guaranteed, and a little usability must be sacrificed every time.

https://www.elastic.co/guide/en/elasticsearch/reference/2.4/cat-master.html

curl 'localhost:9200/_cat/master?v'

https://discuss.elastic.co/t/elasticsearch-and-the-cap-theorem/15102/7

Itamar Syn-Hershko Elasticsearch Partner
Founder, CTO BigData Boutique
http://bigdataboutique.com
http://code972.com | @synhershko
http://BigDataBoutique.co.il/
code972.com     ：

Every distributed system design starts with the assumption that you don't
want to compromise on Partition Tolerance, and therefore have to decide
whether you want full Availability or full Consistency. No distributed
system really gives up on P.

The very fact that you can have enough replicas of your index which will
make the cluster never get to a red state (e.g. the number of nodes you
have) proves this. An index (an eventually, all indexes on your cluster)
can survive a network split. It can also be always available, hence ES is
AP.

Elasticsearch's compromise is on C - consistency - like most NoSQL
databases. It uses Eventual Consistency to answer queries, not just because
of NRT search, but also because you may be querying a replica (a slave
node) which hasn't been brought up to speed yet.

jprante：

 Consistency is not given up by ES. First, on doc level, you have "write
your own read" consistency implemented as versioning - a doc read followed
by a doc write is guaranteed to be consistent if both read and write
versions match (MVCC). Write-your-read consistency is only eventually
consistent because other clients can not be sure when to read the old and
when to read the new value, and this is different from for example ACID
(causal consistency). But it's just another model of consistency.

And second, on index level, after a node crash, ES tries an index recovery
that should end up always in a consistent index state. This is possible
because of the WAL (translog) at each shard.

If ES gave up on consistency, there would be no doc versioning and no index
recovery.

Replica are not interfering with consistency, they are for availability.
The higher the replica level , the higher the probability that an index is
still available although a number of nodes are faulty. Replica level 0 (no
replica) is reducing availability, if just one node fails, availability has
gone.

If primary shard and replica shards differ in their responses to queries,
that should be considered as a bug. Maybe a recovery did not work out right.

ES gives up on partition tolerance. Just a few observations: split brains
can happen and ES can happily proceed reading and writing to the index in
such a case, but the result is not predictable - the usual case is that two
masters are going to control two divergent indices, and that is
catastrophic. This is not a fault but a (nasty) feature, and must be
controlled by extra safeguarding, by setting the minimum master value in
the configuration - for example, if this config is set over the quorum,
more than half of the nodes must be started before a master is elected and
the cluster is formed, so the probability is extremely low for a split
brain, but the probability is not 0 unless minimum master is equal to the
number of all master eligible nodes, which in other words disables
partition tolerance completely.

Also, another observation of mine, the discovery is detecting node
failures, but not network failures. The master node "pings" all other nodes
(under the assumption ES is always running on an always available network)
each 5 secs. If a node does not answer, the cluster state changes and marks
this node as not available. With the current algorithm, it is not possible
for ES to decide if it was the network or the node that failed. And this
makes sense if ES is already giving up on partition tolerance, so it simply
does not matter if it is a node or a network fault.
https://discuss.elastic.co/t/elasticsearch-and-the-cap-theorem/15102/9
http://elasticsearch-users.115913.n3.nabble.com/CAP-theorem-td891925.html#a894234
decPendingAndFinishIfNeeded
https://github.com/elastic/elasticsearch/blob/master/server/src/main/java/org/elasticsearch/action/support/replication/ReplicationOperation.java

BASE: Basically Available, Soft State, Eventual Consistency;