嘗試從密鑰空間讀取時Cassandra複製節點失敗

Question

我的Cassandra（3.0.2）複製集中有3個節點。我的一致性級別是「一個」。在開始時，我的所有密鑰空間的複製因子都等於1.我通過更改表進行更改，並在所有節點上運行「nodetool修復」。現在，當我試圖選擇一些數據（而不是在每一個密鑰空間），我得到這樣的事情（選擇keyspace.table *）：

Traceback (most recent call last): File "/usr/bin/cqlsh.py", line 1258, in perform_simple_statement result = future.result() File "cassandra/cluster.py", line 3781, in cassandra.cluster.ResponseFuture.result (cassandra/cluster.c:73073) raise self._final_exception ReadFailure: Error from server: code=1300 [Replica(s) failed to execute read] message="Operation failed - received 0 responses and 1 failures" info={'failures': 1, 'received_responses': 0, 'required_responses': 1, 'consistency': 'ONE'}

在「/var/log/cassandra/system.log 「我得到：

WARN [SharedPool-Worker-2] 2017-04-07 12:46:20,036 AbstractTracingAwareExecutorService.java:169 - Uncaught exception on thread Thread[SharedPool-Worker-2,5,main]: {} java.lang.AssertionError: null at org.apache.cassandra.db.columniterator.AbstractSSTableIterator$IndexState.updateBlock(AbstractSSTableIterator.java:463) ~[apache-cassandra-3.0.2.jar:3.0.2] at org.apache.cassandra.db.columniterator.SSTableIterator$ForwardIndexedReader.computeNext(SSTableIterator.java:268) ~[apache-cassandra-3.0.2.jar:3.0.2] at org.apache.cassandra.db.columniterator.SSTableIterator$ForwardReader.hasNextInternal(SSTableIterator.java:158) ~[apache-cassandra-3.0.2.jar:3.0.2] at org.apache.cassandra.db.columniterator.AbstractSSTableIterator$Reader.hasNext(AbstractSSTableIterator.java:352) ~[apache-cassandra-3.0.2.jar:3.0.2] at org.apache.cassandra.db.columniterator.AbstractSSTableIterator.hasNext(AbstractSSTableIterator.java:219) ~[apache-cassandra-3.0.2.jar:3.0.2] at org.apache.cassandra.db.columniterator.SSTableIterator.hasNext(SSTableIterator.java:32) ~[apache-cassandra-3.0.2.jar:3.0.2] at org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:95) ~[apache-cassandra-3.0.2.jar:3.0.2] at org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:32) ~[apache-cassandra-3.0.2.jar:3.0.2] at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) ~[apache-cassandra-3.0.2.jar:3.0.2] at org.apache.cassandra.utils.MergeIterator$Candidate.advance(MergeIterator.java:369) ~[apache-cassandra-3.0.2.jar:3.0.2] at org.apache.cassandra.utils.MergeIterator$ManyToOne.advance(MergeIterator.java:189) ~[apache-cassandra-3.0.2.jar:3.0.2] at org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:158) ~[apache-cassandra-3.0.2.jar:3.0.2] at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) ~[apache-cassandra-3.0.2.jar:3.0.2] at org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:426) ~[apache-cassandra-3.0.2.jar:3.0.2] at org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:286) ~[apache-cassandra-3.0.2.jar:3.0.2] at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) ~[apache-cassandra-3.0.2.jar:3.0.2] at org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:95) ~[apache-cassandra-3.0.2.jar:3.0.2] at org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:32) ~[apache-cassandra-3.0.2.jar:3.0.2] at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) ~[apache-cassandra-3.0.2.jar:3.0.2] at org.apache.cassandra.db.transform.BaseRows.hasNext(BaseRows.java:108) ~[apache-cassandra-3.0.2.jar:3.0.2] at org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:131) ~[apache-cassandra-3.0.2.jar:3.0.2] at org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:87) ~[apache-cassandra-3.0.2.jar:3.0.2] at org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:77) ~[apache-cassandra-3.0.2.jar:3.0.2] at org.apache.cassandra.db.partitions.UnfilteredPartitionIterators$Serializer.serialize(UnfilteredPartitionIterators.java:298) ~[apache-cassandra-3.0.2.jar:3.0.2] at org.apache.cassandra.db.ReadResponse$LocalDataResponse.build(ReadResponse.java:136) ~[apache-cassandra-3.0.2.jar:3.0.2] at org.apache.cassandra.db.ReadResponse$LocalDataResponse.(ReadResponse.java:128) ~[apache-cassandra-3.0.2.jar:3.0.2] at org.apache.cassandra.db.ReadResponse$LocalDataResponse.(ReadResponse.java:123) ~[apache-cassandra-3.0.2.jar:3.0.2] at org.apache.cassandra.db.ReadResponse.createDataResponse(ReadResponse.java:65) ~[apache-cassandra-3.0.2.jar:3.0.2] at org.apache.cassandra.db.ReadCommand.createResponse(ReadCommand.java:289) ~[apache-cassandra-3.0.2.jar:3.0.2] at org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:1721) ~[apache-cassandra-3.0.2.jar:3.0.2] at org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:2375) ~[apache-cassandra-3.0.2.jar:3.0.2] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_121] at org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164) ~[apache-cassandra-3.0.2.jar:3.0.2] at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) [apache-cassandra-3.0.2.jar:3.0.2] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_121] DEBUG [SharedPool-Worker-1] 2017-04-07 12:46:20,037 ReadCallback.java:126 - Failed; received 0 of 1 responses

而且我得到：

DEBUG [SharedPool-Worker-1] 2017-04-07 13:20:30,002 ReadCallback.java:126 - Timed out; received 0 of 1 responses

我檢查有沒有關於端口9042和7000。我改變選項中的節點之間的連接‘/etc/cassandra/cassandra.yml’像「read_request_timeout_in_ms」，「range_request_timeout_in_m s「，」write_request_timeout_in_ms「或」truncate_request_timeout_in_ms「。我更改了文件「〜/ .cassandra/cqlshrc」和選項「client_timeout = 3600」。 Additonally當我如做「從keyspace.table選擇*其中列1 =‘值’和列2 =價值」，我得到：

ReadTimeout: Error from server: code=1200 [Coordinator node timed out waiting for replica nodes' responses] message="Operation timed out - received only 0 responses." info={'received_responses': 0, 'required_responses': 1, 'consistency': 'ONE'}

任何想法？

Answer 1

這或多或少是一條評論，但因爲有這麼多話要說它不適合評論。

這將是非常好的說什麼複製因素你改變了價值。我只是假設它是3，因爲它非常標準。然後再次，因爲你只有一羣3人有時將RF設置爲2.你還提到你更新了表上的複製因子。據我所知，複製因子是在密鑰空間級別上設置的。

如果您發佈了發生錯誤的密鑰空間的描述，這將非常有幫助。

考慮到select * from something可能在您的集羣中非常密集，特別是如果您有大量數據。如果你要在cqlsh中執行這個查詢，你可能會得到10 000，然後你再次只提到了cqlsh，並且沒有應用程序代碼，所以我只是注意到這個問題。

能否請你提供nodetool status只是爲了確保你實際上沒有運行一些節點下的查詢。因爲第一個錯誤看起來像這樣。

第二個錯誤，你發佈了一個堆棧跟蹤，它看起來像你缺少磁盤上的一些sstables？是否有某種其他過程以某種方式操縱sstables的可能性？

你也改變了cassandra.yaml中的很多屬性，基本上你將預期的響應時間減少了近50％，我想這也難怪節點沒有時間來響應......在整個表上計數通常需要超過3.6秒

推理這些值改變的原因簡直就是缺失。

Answer 2

MarkoŠvaljek，是的，我將複製因子從1更改爲3（因爲我的複製中有3個節點）。你是對的;你改變了密鑰空間的複製因子，這就是我所做的。在這裏，你有我的密鑰空間的描述通常在那裏我得到了一些錯誤（當然它與其他keyspaces發生）：

soi@cqlsh> desc keyspace engine; 

CREATE KEYSPACE engine WITH replication = {'class': 'SimpleStrategy', 'replication_factor': '3'} AND durable_writes = true; 

CREATE TABLE engine.messages (
    persistence_id text, 
    partition_nr bigint, 
    sequence_nr bigint, 
    timestamp timeuuid, 
    timebucket text, 
    message blob, 
    tag1 text, 
    tag2 text, 
    tag3 text, 
    used boolean static, 
    PRIMARY KEY ((persistence_id, partition_nr), sequence_nr, timestamp, timebucket) 
) WITH CLUSTERING ORDER BY (sequence_nr ASC, timestamp ASC, timebucket ASC) 
    AND bloom_filter_fp_chance = 0.01 
    AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'} 
    AND comment = '' 
    AND compaction = {'bucket_high': '1.5', 'bucket_low': '0.5', 'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy', 'enabled': 'true', 'max_threshold': '32', 'min_sstable_size': '50', 'min_threshold': '4', 'tombstone_compaction_interval': '86400', 'tombstone_threshold': '0.2', 'unchecked_tombstone_compaction': 'false'} 
    AND compression = {'chunk_length_in_kb': '64', 'class': 'org.apache.cassandra.io.compress.LZ4Compressor'} 
    AND crc_check_chance = 1.0 
    AND dclocal_read_repair_chance = 0.1 
    AND default_time_to_live = 0 
    AND gc_grace_seconds = 864000 
    AND max_index_interval = 2048 
    AND memtable_flush_period_in_ms = 0 
    AND min_index_interval = 128 
    AND read_repair_chance = 0.0 
    AND speculative_retry = '99PERCENTILE'; 

CREATE MATERIALIZED VIEW engine.eventsbytag1 AS 
    SELECT tag1, timebucket, timestamp, persistence_id, partition_nr, sequence_nr, message 
    FROM engine.messages 
    WHERE persistence_id IS NOT NULL AND partition_nr IS NOT NULL AND sequence_nr IS NOT NULL AND tag1 IS NOT NULL AND timestamp IS NOT NULL AND timebucket IS NOT NULL 
    PRIMARY KEY ((tag1, timebucket), timestamp, persistence_id, partition_nr, sequence_nr) 
    WITH CLUSTERING ORDER BY (timestamp ASC, persistence_id ASC, partition_nr ASC, sequence_nr ASC) 
    AND bloom_filter_fp_chance = 0.01 
    AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'} 
    AND comment = '' 
    AND compaction = {'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy', 'max_threshold': '32', 'min_threshold': '4'} 
    AND compression = {'chunk_length_in_kb': '64', 'class': 'org.apache.cassandra.io.compress.LZ4Compressor'} 
    AND crc_check_chance = 1.0 
    AND dclocal_read_repair_chance = 0.1 
    AND default_time_to_live = 0 
    AND gc_grace_seconds = 864000 
    AND max_index_interval = 2048 
    AND memtable_flush_period_in_ms = 0 
    AND min_index_interval = 128 
    AND read_repair_chance = 0.0 
    AND speculative_retry = '99PERCENTILE'; 

CREATE TABLE engine.config (
    property text PRIMARY KEY, 
    value text 
) WITH bloom_filter_fp_chance = 0.01 
    AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'} 
    AND comment = '' 
    AND compaction = {'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy', 'max_threshold': '32', 'min_threshold': '4'} 
    AND compression = {'chunk_length_in_kb': '64', 'class': 'org.apache.cassandra.io.compress.LZ4Compressor'} 
    AND crc_check_chance = 1.0 
    AND dclocal_read_repair_chance = 0.1 
    AND default_time_to_live = 0 
    AND gc_grace_seconds = 864000 
    AND max_index_interval = 2048 
    AND memtable_flush_period_in_ms = 0 
    AND min_index_interval = 128 
    AND read_repair_chance = 0.0 
    AND speculative_retry = '99PERCENTILE'; 

CREATE TABLE engine.metadata (
    persistence_id text PRIMARY KEY, 
    deleted_to bigint, 
    properties map<text, text> 
) WITH bloom_filter_fp_chance = 0.01 
    AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'} 
    AND comment = '' 
    AND compaction = {'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy', 'max_threshold': '32', 'min_threshold': '4'} 
    AND compression = {'chunk_length_in_kb': '64', 'class': 'org.apache.cassandra.io.compress.LZ4Compressor'} 
    AND crc_check_chance = 1.0 
    AND dclocal_read_repair_chance = 0.1 
    AND default_time_to_live = 0 
    AND gc_grace_seconds = 864000 
    AND max_index_interval = 2048 
    AND memtable_flush_period_in_ms = 0 
    AND min_index_interval = 128 
    AND read_repair_chance = 0.0 
    AND speculative_retry = '99PERCENTILE'; 

我平時代碼錯誤沒有。 1200或1300，你可以在第一篇文章中看到。在這裏你有我的「節點工具狀態」：

ubuntu@cassandra-db1:~$ nodetool status 
Datacenter: datacenter1 
======================= 
Status=Up/Down 
|/ State=Normal/Leaving/Joining/Moving 
-- Address  Load  Tokens  Owns Host ID        Rack 
UN 192.168.1.13 3.94 MB 256   ?  8ebcc3fe-9869-44c5-b7a5-e4f0f5a0beb1 rack1 
UN 192.168.1.14 4.26 MB 256   ?  977831cb-98fe-4170-ab15-2b4447559003 rack1 
UN 192.168.1.15 4.94 MB 256   ?  7515a967-cbdc-4d89-989b-c0a2f124173f rack1 

Note: Non-system keyspaces don't have the same replication settings, effective ownership information is meaningless 

我不這麼認爲一些其他進程可以操縱磁盤上的一些數據。我會補充說，我有類似的複製，我有更多的數據，我沒有這樣的問題。

Answer 3

固定！我將Cassandra版本從3.0.2更改爲3.0.9，並解決了問題。