Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

A confusing warning for a keyspace with Network-Topology: "SimpleStrategy replication class is not recommended, but was used for keyspace scylla_bench" #21084

Open
2 tasks
yarongilor opened this issue Oct 13, 2024 · 13 comments
Labels
area/tablets area/topology changes P3 Medium Priority tests/longevity Issue detected during longevity

Comments

@yarongilor
Copy link

yarongilor commented Oct 13, 2024

Packages

Scylla version: 6.3.0~dev-1.tabletmerge.20241013.259dcf894703 with build-id #NO_BUILDID

Kernel Version: 6.8.0-1016-aws

Issue description

  • This issue is a regression.
  • It is unknown if this issue is a regression.

Describe your issue in detail and steps it took to produce it.

in a master latest based build ( reproduced in 6.3 master as well) a keyspace with a network-strategy is created, as reported in node-1,
yet later on, the nodes report a warning that it has a simple-strategy.
SCT issues a create-keyspace cql like:

< t:2024-10-13 13:24:27,997 f:common.py       l:1324 c:utils                p:DEBUG > Executing CQL 'CREATE KEYSPACE IF NOT EXISTS scylla_bench WITH replication = {'class': 'NetworkTopologyStrategy', 'replication_factor': 3} and tablets={'initial':1};' ...

[node-1] keyspace creation:

Oct 13 13:24:28 tablets-split-merge-master-db-node-ecba0625-1 scylla[5107]:  [shard 0:stmt] migration_manager - Create new Keyspace: KSMetaData{name=scylla_bench, strategyClass=org.apache.cassandra.locator.NetworkTopologyStrategy, strategyOptions={"us-east": "3"}, cfMetaData={}, durable_writes=true, tablets={"initial":1}, userTypes=org.apache.cassandra.config.UTMetaData@0x600008debdc0}

[node-3] warning messages:

UN 10.12.0.234 2.96 GB 256    ?    e4519345-f4c5-479c-9f9e-2e3703f0c01f 1a  
UN 10.12.2.173 9.70 GB 256    ?    7c7eb8fb-c283-4a74-84c7-c39a8c58508b 1a  
UN 10.12.2.65  9.48 GB 256    ?    b6a59734-481e-41d3-8a51-475be81ca5d6 1a  

Note: Non-system keyspaces don't have the same replication settings, effective ownership information is meaningless
This EC2 instance is optimized for Scylla.

scyllaadm@tablets-split-merge-master-db-node-ecba0625-3:~$ grep scylla_bench  /var/log/messages | grep -i simplestrategy
Oct 13 13:24:52 tablets-split-merge-master-db-node-ecba0625-3 scylla[4854]:  [shard 0:stmt] create_keyspace - SimpleStrategy replication class is not recommended, but was used for keyspace scylla_bench. You may suppress this warning by delisting SimpleStrategy from replication_strategy_warn_list configuration option, or make it into an error by listing this replication strategy on replication_strategy_fail_list.
Oct 13 13:25:09 tablets-split-merge-master-db-node-ecba0625-3 scylla[4854]:  [shard 0:stmt] create_keyspace - SimpleStrategy replication class is not recommended, but was used for keyspace scylla_bench. You may suppress this warning by delisting SimpleStrategy from replication_strategy_warn_list configuration option, or make it into an error by listing this replication strategy on replication_strategy_fail_list.
Oct 13 13:25:19 tablets-split-merge-master-db-node-ecba0625-3 scylla[4854]:  [shard 0:stmt] create_keyspace - SimpleStrategy replication class is not recommended, but was used for keyspace scylla_bench. You may suppress this warning by delisting SimpleStrategy from replication_strategy_warn_list configuration option, or make it into an error by listing this replication strategy on replication_strategy_fail_list.
scyllaadm@tablets-split-merge-master-db-node-ecba0625-3:~$ cqlsh 10.12.2.65
Connected to tablets-split-merge-master-db-cluster-ecba0625 at 10.12.2.65:9042
[cqlsh 6.0.23.dev9+gb09bc79 | Scylla 6.3.0~dev-1.tabletmerge.20241013.259dcf894703 | CQL spec 3.3.1 | Native protocol v4]
Use HELP for help.
cqlsh> DESC KEYSPACE scylla_bench ;

CREATE KEYSPACE scylla_bench WITH replication = {'class': 'org.apache.cassandra.locator.NetworkTopologyStrategy', 'us-east': '3'} AND durable_writes = true AND tablets = {'initial': 1};

CREATE TABLE scylla_bench.test (
    pk bigint,
    ck bigint,
    v blob,
    PRIMARY KEY (pk, ck)
) WITH CLUSTERING ORDER BY (ck ASC)
    AND bloom_filter_fp_chance = 0.01
    AND caching = {'keys': 'ALL', 'rows_per_partition': 'ALL'}
    AND comment = ''
    AND compaction = {'class': 'SizeTieredCompactionStrategy'}
    AND compression = {}
    AND crc_check_chance = 1
    AND default_time_to_live = 0
    AND gc_grace_seconds = 864000
    AND max_index_interval = 2048
    AND memtable_flush_period_in_ms = 0
    AND min_index_interval = 128
    AND speculative_retry = 'NONE'
    AND tombstone_gc = {'mode': 'repair', 'propagation_delay_in_seconds': '3600'};

cqlsh> 

then, 20 minutes later, node-1 also got these warning:

Oct 13 13:44:26 tablets-split-merge-master-db-node-ecba0625-1 scylla[5107]:  [shard 0:stmt] create_keyspace - SimpleStrategy replication class is not recommended, but was used for keyspace scylla_bench. You may suppress this warning by delisting SimpleStrategy from replication_strategy_warn_list configuration option, or make it into an error by listing this replication strategy on replication_strategy_fail_list.
Oct 13 14:03:22 tablets-split-merge-master-db-node-ecba0625-1 scylla[5107]:  [shard 0:stmt] create_keyspace - SimpleStrategy replication class is not recommended, but was used for keyspace scylla_bench. You may suppress this warning by delisting SimpleStrategy from replication_strategy_warn_list configuration option, or make it into an error by listing this replication strategy on replication_strategy_fail_list.

node-1 does show the keyspace is created with tablets like:

scyllaadm@tablets-split-merge-master-db-node-ecba0625-1:~$ cqlsh 10.12.0.234
Connected to tablets-split-merge-master-db-cluster-ecba0625 at 10.12.0.234:9042
[cqlsh 6.0.23.dev9+gb09bc79 | Scylla 6.3.0~dev-1.tabletmerge.20241013.259dcf894703 | CQL spec 3.3.1 | Native protocol v4]
Use HELP for help.
cqlsh> select * from system_schema.scylla_keyspaces;

 keyspace_name | initial_tablets | storage_options | storage_type
---------------+-----------------+-----------------+--------------
  scylla_bench |               1 |            null |         null

(1 rows)

The table of system_schema.scylla_keyspaces has:

scyllaadm@tablets-split-merge-master-db-node-ecba0625-1:~$ cqlsh 10.12.0.234
Connected to tablets-split-merge-master-db-cluster-ecba0625 at 10.12.0.234:9042
[cqlsh 6.0.23.dev9+gb09bc79 | Scylla 6.3.0~dev-1.tabletmerge.20241013.259dcf894703 | CQL spec 3.3.1 | Native protocol v4]
Use HELP for help.
cqlsh> select * from system_schema.scylla_keyspaces;

 keyspace_name | initial_tablets | storage_options | storage_type
---------------+-----------------+-----------------+--------------
  scylla_bench |               1 |            null |         null

(1 rows)
cqlsh> 

Some tablets activity in logs is:

< t:2024-10-13 10:27:15,649 f:db_log_reader.py l:125  c:sdcm.db_log_reader   p:DEBUG > 2024-10-13T10:27:15.624+00:00 tablets-split-merge-master-db-node-184b7ff0-1     !INFO | scylla[4912]:  [shard 0:main] init - loading tablet metadata
< t:2024-10-13 10:27:22,531 f:db_log_reader.py l:125  c:sdcm.db_log_reader   p:DEBUG > 2024-10-13T10:27:22.519+00:00 tablets-split-merge-master-db-node-184b7ff0-2     !INFO | scylla[4842]:  [shard 0:main] init - loading tablet metadata
< t:2024-10-13 10:27:34,331 f:db_log_reader.py l:125  c:sdcm.db_log_reader   p:DEBUG > 2024-10-13T10:27:34.316+00:00 tablets-split-merge-master-db-node-184b7ff0-3     !INFO | scylla[4849]:  [shard 0:main] init - loading tablet metadata
< t:2024-10-13 10:31:19,148 f:db_log_reader.py l:125  c:sdcm.db_log_reader   p:DEBUG > 2024-10-13T10:31:19.065+00:00 tablets-split-merge-master-db-node-184b7ff0-3     !INFO | scylla[4849]:  [shard 0: gms] schema_tables - Tablet metadata changed
< t:2024-10-13 10:31:19,240 f:db_log_reader.py l:125  c:sdcm.db_log_reader   p:DEBUG > 2024-10-13T10:31:19.151+00:00 tablets-split-merge-master-db-node-184b7ff0-2     !INFO | scylla[4842]:  [shard 0: gms] schema_tables - Tablet metadata changed
< t:2024-10-13 10:31:19,240 f:db_log_reader.py l:125  c:sdcm.db_log_reader   p:DEBUG > 2024-10-13T10:31:19.158+00:00 tablets-split-merge-master-db-node-184b7ff0-1     !INFO | scylla[4912]:  [shard 0: gms] schema_tables - Tablet metadata changed

Impact

Describe the impact this issue causes to the user.

How frequently does it reproduce?

Describe the frequency with how this issue can be reproduced.

Installation details

Cluster size: 3 nodes (i3.large)

Scylla Nodes used in this run:

  • tablets-split-merge-master-db-node-ecba0625-3 (18.207.111.166 | 10.12.2.65) (shards: 2)
  • tablets-split-merge-master-db-node-ecba0625-2 (44.203.32.197 | 10.12.2.173) (shards: 2)
  • tablets-split-merge-master-db-node-ecba0625-1 (44.202.212.93 | 10.12.0.234) (shards: 2)

OS / Image: ami-07af88cdc0b3d8af9 (aws: undefined_region)

Test: byo-longevity-test-yg
Test id: ecba0625-b97d-4d41-b708-7538ce55fcca
Test name: scylla-staging/yarongilor/byo-longevity-test-yg
Test method: tablets_split_merge_test.TabletsSplitMergeTest.test_tablets_split_merge
Test config file(s):

Logs and commands
  • Restore Monitor Stack command: $ hydra investigate show-monitor ecba0625-b97d-4d41-b708-7538ce55fcca
  • Restore monitor on AWS instance using Jenkins job
  • Show all stored logs command: $ hydra investigate show-logs ecba0625-b97d-4d41-b708-7538ce55fcca

Logs:

Jenkins job URL
Argus

@yarongilor
Copy link
Author

The issue is not reproduced in a standard master longevity (like 100gb-4h), but it is reproduced on master with this scenario of creating a new keyspace (scylla_bench) with initial-tablets(1):

< t:2024-10-10 19:35:23,682 f:db_log_reader.py l:125  c:sdcm.db_log_reader   p:DEBUG > 2024-10-10T19:35:23.661+00:00 tablets-split-merge-master-db-node-88967c6a-2     !INFO | scylla[4783]:  [shard 0:stmt] migration_manager - Create new Keyspace: KSMetaData{name=scylla_bench, strategyClass=org.apache.cassandra.locator.NetworkTopologyStrategy, strategyOptions={"eu-west": "3"}, cfMetaData={}, durable_writes=true, tablets={"initial":1}, userTypes=org.apache.cassandra.config.UTMetaData@0x60000909cdc0}
yarongilor@yarongilor:~/Downloads/logs/last$ grep create_keyspace csct
< t:2024-10-10 19:26:36,596 f:sct_config.py   l:2068 c:sdcm.sct_config      p:INFO  > pre_create_keyspace: 'CREATE KEYSPACE IF NOT EXISTS scylla_bench WITH replication
< t:2024-10-10 19:35:48,768 f:db_log_reader.py l:125  c:sdcm.db_log_reader   p:DEBUG > 2024-10-10T19:35:48.700+00:00 tablets-split-merge-master-db-node-88967c6a-1  !WARNING | scylla[4940]:  [shard 0:stmt] create_keyspace - SimpleStrategy replication class is not recommended, but was used for keyspace scylla_bench. You may suppress this warning by delisting SimpleStrategy from replication_strategy_warn_list configuration option, or make it into an error by listing this replication strategy on replication_strategy_fail_list.
< t:2024-10-10 19:35:55,513 f:db_log_reader.py l:125  c:sdcm.db_log_reader   p:DEBUG > 2024-10-10T19:35:55.449+00:00 tablets-split-merge-master-db-node-88967c6a-1  !WARNING | scylla[4940]:  [shard 0:stmt] create_keyspace - SimpleStrategy replication class is not recommended, but was used for keyspace scylla_bench. You may suppress this warning by delisting SimpleStrategy from replication_strategy_warn_list configuration option, or make it into an error by listing this replication strategy on replication_strategy_fail_list.
< t:2024-10-10 19:36:05,464 f:db_log_reader.py l:125  c:sdcm.db_log_reader   p:DEBUG > 2024-10-10T19:36:05.412+00:00 tablets-split-merge-master-db-node-88967c6a-2  !WARNING | scylla[4783]:  [shard 0:stmt] create_keyspace - SimpleStrategy replication class is not recommended, but was used for keyspace scylla_bench. You may suppress this warning by delisting SimpleStrategy from replication_strategy_warn_list configuration option, or make it into an error by listing this replication strategy on replication_strategy_fail_list.
< t:2024-10-10 20:07:32,335 f:db_log_reader.py l:125  c:sdcm.db_log_reader   p:DEBUG > 2024-10-10T20:07:32.279+00:00 tablets-split-merge-master-db-node-88967c6a-2  !WARNING | scylla[4783]:  [shard 0:stmt] create_keyspace - SimpleStrategy replication class is not recommended, but was used for keyspace scylla_bench. You may suppress this warning by delisting SimpleStrategy from replication_strategy_warn_list configuration option, or make it into an error by listing this replication strategy on replication_strategy_fail_list.
< t:2024-10-10 20:07:42,276 f:db_log_reader.py l:125  c:sdcm.db_log_reader   p:DEBUG > 2024-10-10T20:07:42.199+00:00 tablets-split-merge-master-db-node-88967c6a-1  !WARNING | scylla[4940]:  [shard 0:stmt] create_keyspace - SimpleStrategy replication class is not recommended, but was used for keyspace scylla_bench. You may suppress this warning by delisting SimpleStrategy from replication_strategy_warn_list configuration option, or make it into an error by listing this replication strategy on replication_strategy_fail_list.
< t:2024-10-10 20:38:55,714 f:db_log_reader.py l:125  c:sdcm.db_log_reader   p:DEBUG > 2024-10-10T20:38:55.700+00:00 tablets-split-merge-master-db-node-88967c6a-1  !WARNING | scylla[4940]:  [shard 0:stmt] create_keyspace - SimpleStrategy replication class is not recommended, but was used for keyspace scylla_bench. You may suppress this warning by delisting SimpleStrategy from replication_strategy_warn_list configuration option, or make it into an error by listing this replication strategy on replication_strategy_fail_list.
< t:2024-10-10 20:39:05,749 f:db_log_reader.py l:125  c:sdcm.db_log_reader   p:DEBUG > 2024-10-10T20:39:05.748+00:00 tablets-split-merge-master-db-node-88967c6a-3  !WARNING | scylla[4777]:  [shard 0:stmt] create_keyspace - SimpleStrategy replication class is not recommended, but was used for keyspace scylla_bench. You may suppress this warning by delisting SimpleStrategy from replication_strategy_warn_list configuration option, or make it into an error by listing this replication strategy on replication_strategy_fail_list.

Packages

Scylla version: 6.3.0~dev-20241008.b4df07df71fa with build-id 162d22a45a35a12aeee516d566ff2b66f7d37123

Kernel Version: 6.8.0-1016-aws

Issue description

  • This issue is a regression.
  • It is unknown if this issue is a regression.

Describe your issue in detail and steps it took to produce it.

Impact

Describe the impact this issue causes to the user.

How frequently does it reproduce?

Describe the frequency with how this issue can be reproduced.

Installation details

Cluster size: 3 nodes (i4i.large)

Scylla Nodes used in this run:

  • tablets-split-merge-master-db-node-88967c6a-3 (108.129.115.33 | 10.4.1.40) (shards: 2)
  • tablets-split-merge-master-db-node-88967c6a-2 (3.250.172.168 | 10.4.0.134) (shards: 2)
  • tablets-split-merge-master-db-node-88967c6a-1 (34.252.155.28 | 10.4.0.84) (shards: 2)

OS / Image: ami-0b1ef254157a14188 (aws: undefined_region)

Test: byo-longevity-test-yg
Test id: 88967c6a-5485-4a33-895c-807c52f0e7f6
Test name: scylla-staging/yarongilor/byo-longevity-test-yg
Test method: tablets_split_merge_test.TabletsSplitMergeTest.test_tablets_split_merge
Test config file(s):

Logs and commands
  • Restore Monitor Stack command: $ hydra investigate show-monitor 88967c6a-5485-4a33-895c-807c52f0e7f6
  • Restore monitor on AWS instance using Jenkins job
  • Show all stored logs command: $ hydra investigate show-logs 88967c6a-5485-4a33-895c-807c52f0e7f6

Logs:

Jenkins job URL
Argus

@bhalevy
Copy link
Member

bhalevy commented Oct 13, 2024

@ptrsmrn could this be related to the recent changes you introduced?

@ptrsmrn
Copy link
Contributor

ptrsmrn commented Oct 14, 2024

@ptrsmrn could this be related to the recent changes you introduced?

@bhalevy do you have any specific changes on your mind or you are asking in general?

@bhalevy
Copy link
Member

bhalevy commented Oct 14, 2024

@ptrsmrn could this be related to the recent changes you introduced?

@bhalevy do you have any specific changes on your mind or you are asking in general?

2d9b8f2 touched the validation code

@ptrsmrn
Copy link
Contributor

ptrsmrn commented Oct 14, 2024

I'm on master OSS on this commit 86fd9ce, which has 2d9b8f2, and if I create KS like that, it doesn't return this warning:

cqlsh> create keyspace ks with replication = { 'class': 'NetworkTopologyStrategy', 'replication_factor': 1 } and tablets = {'initial': 1 };

Warnings :
Tables in this keyspace will be replicated using Tablets and will not support CDC, LWT and counters features. To use CDC, LWT or counters, drop this keyspace and re-create it without tablets by adding AND TABLETS = {'enabled': false} to the CREATE KEYSPACE statement.

Using Replication Factor replication_factor=1 lower than the minimum_replication_factor_warn_threshold=3 is not recommended.

@ptrsmrn
Copy link
Contributor

ptrsmrn commented Oct 14, 2024

Perhaps scylla_bench is created with SimpleStrategy after all, why do we see so many warnings? By looking at the logs snippet attached to this issue's description, I'm not sure the Create new Keyspace: KSMetaData is creating the exact same keyspace we later get warnings about.

@ptrsmrn
Copy link
Contributor

ptrsmrn commented Oct 14, 2024

I see a couple of things happened just before this warning got logged, e.g.:

2024-10-13T14:02:37.136+00:00 tablets-split-merge-master-db-node-ecba0625-1     !INFO | scylla[5107]:  [shard 0:strm] api - perform_keyspace_flush: keyspace=scylla_bench tables=[]
2024-10-13T14:02:38.885+00:00 tablets-split-merge-master-db-node-ecba0625-1     !INFO | scylla[5107]:  [shard 0:strm] api - force_keyspace_compaction: keyspace=scylla_bench tables=[table{name=test, id=794be060-8966-11ef-b27d-468d4f8f9f12}], flush=true consider_only_existing_data=false
2024-10-13T14:02:38.885+00:00 tablets-split-merge-master-db-node-ecba0625-1     !INFO | scylla[5107]:  [shard 0:strm] compaction_manager - User initiated compaction started on behalf of scylla_bench.test compaction_group=0
2024-10-13T14:02:38.885+00:00 tablets-split-merge-master-db-node-ecba0625-1     !INFO | scylla[5107]:  [shard 0:strm] compaction - [Compact scylla_bench.test ce2b60b0-896b-11ef-b4be-7f87e95d107f] Compacting [/var/lib/scylla/data/scylla_bench/test-794be060896611efb27d468d4f8f9f12/me-3gkc_130c_5s4a82qy7e7up6aqv3-big-Data.db:level=0:origin=memtable,/var/lib/scylla/data/scylla_bench/test-794be060896611efb27d468d4f8f9f12/me-3gkc_12cp_3863l2qy7e7up6aqv3-big-Data.db:level=0:origin=compaction]
...

2024-10-13T14:03:13.637+00:00 tablets-split-merge-master-db-node-ecba0625-1     !INFO | scylla[5107]:  [shard 1:stmt] database - Truncating scylla_bench.test with auto-snapshot
2024-10-13T14:03:13.637+00:00 tablets-split-merge-master-db-node-ecba0625-1     !INFO | scylla[5107]:  [shard 1:stmt] database - Truncated scylla_bench.test

If I do truncate on a table belonging to this ks in cqlsh, I don't get this warning (neither in cqlsh nor in scylla server). I think something is internally recreating this ks and either sets an invalid replication strategy, or it is corrected only later.

@ptrsmrn ptrsmrn added the P3 Medium Priority label Oct 14, 2024
@ptrsmrn
Copy link
Contributor

ptrsmrn commented Oct 14, 2024

Anyways, I don't think it's a major thing.

@yarongilor
Copy link
Author

yarongilor commented Oct 14, 2024

Thanks @ptrsmrn , so according to this, a reproduction should be quite simple:

  1. create a keyspace with network strategy.
  2. create (adding "if not exist" or not) the same keyspace with simple-strategy.

by the way, the logs may indeed point the warnings come right after scylla-bench start running -

< t:2024-10-13 13:14:19,550 f:base.py         l:369  c:sdcm.sct_events.base p:INFO  >   * ScyllaBenchEvent
< t:2024-10-13 13:24:52,227 f:file_logger.py  l:101  c:sdcm.sct_events.file_logger p:INFO  > 2024-10-13 13:24:52.222: (ScyllaBenchEvent Severity.NORMAL) period_type=begin event_id=bc1380f5-2864-4d88-b845-f222cf4fccbe: node=Node tablets-split-merge-master-loader-node-ecba0625-1 [3.238.38.192 | 10.12.2.192]
< t:2024-10-13 13:24:52,771 f:db_log_reader.py l:125  c:sdcm.db_log_reader   p:DEBUG > 2024-10-13T13:24:52.720+00:00 tablets-split-merge-master-db-node-ecba0625-3  !WARNING | scylla[4854]:  [shard 0:stmt] create_keyspace - SimpleStrategy replication class is not recommended, but was used for keyspace scylla_bench. You may suppress this warning by delisting SimpleStrategy from replication_strategy_warn_list configuration option, or make it into an error by listing this replication strategy on replication_strategy_fail_list.
< t:2024-10-13 13:25:07,973 f:file_logger.py  l:101  c:sdcm.sct_events.file_logger p:INFO  > 2024-10-13 13:25:07.970: (ScyllaBenchEvent Severity.NORMAL) period_type=end event_id=bc1380f5-2864-4d88-b845-f222cf4fccbe duration=15s: node=Node tablets-split-merge-master-loader-node-ecba0625-1 [3.238.38.192 | 10.12.2.192]
< t:2024-10-13 13:25:09,432 f:file_logger.py  l:101  c:sdcm.sct_events.file_logger p:INFO  > 2024-10-13 13:25:09.430: (ScyllaBenchEvent Severity.NORMAL) period_type=begin event_id=30fc9344-57cc-4f9e-a9e2-a0022376284f: node=Node tablets-split-merge-master-loader-node-ecba0625-1 [3.238.38.192 | 10.12.2.192]
< t:2024-10-13 13:25:10,032 f:db_log_reader.py l:125  c:sdcm.db_log_reader   p:DEBUG > 2024-10-13T13:25:09.967+00:00 tablets-split-merge-master-db-node-ecba0625-3  !WARNING | scylla[4854]:  [shard 0:stmt] create_keyspace - SimpleStrategy replication class is not recommended, but was used for keyspace scylla_bench. You may suppress this warning by delisting SimpleStrategy from replication_strategy_warn_list configuration option, or make it into an error by listing this replication strategy on replication_strategy_fail_list.
< t:2024-10-13 13:25:19,539 f:file_logger.py  l:101  c:sdcm.sct_events.file_logger p:INFO  > 2024-10-13 13:25:19.536: (ScyllaBenchEvent Severity.NORMAL) period_type=begin event_id=7bde3ac7-83d5-4bb6-8d7e-f3b2bbf39113: node=Node tablets-split-merge-master-loader-node-ecba0625-1 [3.238.38.192 | 10.12.2.192]
< t:2024-10-13 13:25:20,061 f:db_log_reader.py l:125  c:sdcm.db_log_reader   p:DEBUG > 2024-10-13T13:25:19.968+00:00 tablets-split-merge-master-db-node-ecba0625-3  !WARNING | scylla[4854]:  [shard 0:stmt] create_keyspace - SimpleStrategy replication class is not recommended, but was used for keyspace scylla_bench. You may suppress this warning by delisting SimpleStrategy from replication_strategy_warn_list configuration option, or make it into an error by listing this replication strategy on replication_strategy_fail_list.
< t:2024-10-13 13:28:16,984 f:file_logger.py  l:101  c:sdcm.sct_events.file_logger p:INFO  > 2024-10-13 13:28:16.982: (ScyllaBenchEvent Severity.NORMAL) period_type=end event_id=7bde3ac7-83d5-4bb6-8d7e-f3b2bbf39113 duration=2m57s: node=Node tablets-split-merge-master-loader-node-ecba0625-1 [3.238.38.192 | 10.12.2.192]
< t:2024-10-13 13:44:26,825 f:file_logger.py  l:101  c:sdcm.sct_events.file_logger p:INFO  > 2024-10-13 13:44:26.824: (ScyllaBenchEvent Severity.NORMAL) period_type=begin event_id=174e2c97-2736-4ff6-bb6b-27491dde3ec3: node=Node tablets-split-merge-master-loader-node-ecba0625-1 [3.238.38.192 | 10.12.2.192]
< t:2024-10-13 13:44:27,417 f:db_log_reader.py l:125  c:sdcm.db_log_reader   p:DEBUG > 2024-10-13T13:44:27.386+00:00 tablets-split-merge-master-db-node-ecba0625-1  !WARNING | scylla[5107]:  [shard 0:stmt] create_keyspace - SimpleStrategy replication class is not recommended, but was used for keyspace scylla_bench. You may suppress this warning by delisting SimpleStrategy from replication_strategy_warn_list configuration option, or make it into an error by listing this replication strategy on replication_strategy_fail_list.
< t:2024-10-13 13:47:14,883 f:file_logger.py  l:101  c:sdcm.sct_events.file_logger p:INFO  > 2024-10-13 13:47:14.882: (ScyllaBenchEvent Severity.NORMAL) period_type=end event_id=174e2c97-2736-4ff6-bb6b-27491dde3ec3 duration=2m48s: node=Node tablets-split-merge-master-loader-node-ecba0625-1 [3.238.38.192 | 10.12.2.192]
< t:2024-10-13 13:55:10,037 f:file_logger.py  l:101  c:sdcm.sct_events.file_logger p:INFO  > 2024-10-13 13:55:10.036: (ScyllaBenchEvent Severity.NORMAL) period_type=end event_id=30fc9344-57cc-4f9e-a9e2-a0022376284f duration=30m0s: node=Node tablets-split-merge-master-loader-node-ecba0625-1 [3.238.38.192 | 10.12.2.192]
< t:2024-10-13 14:03:13,048 f:file_logger.py  l:101  c:sdcm.sct_events.file_logger p:INFO  > 2024-10-13 14:03:13.046: (ScyllaBenchEvent Severity.NORMAL) period_type=begin event_id=99ac12c7-5972-4f96-8c65-d56333d17985: node=Node tablets-split-merge-master-loader-node-ecba0625-1 [3.238.38.192 | 10.12.2.192]
< t:2024-10-13 14:03:13,493 f:db_log_reader.py l:125  c:sdcm.db_log_reader   p:DEBUG > 2024-10-13T14:03:13.419+00:00 tablets-split-merge-master-db-node-ecba0625-2  !WARNING | scylla[4859]:  [shard 0:stmt] create_keyspace - SimpleStrategy replication class is not recommended, but was used for keyspace scylla_bench. You may suppress this warning by delisting SimpleStrategy from replication_strategy_warn_list configuration option, or make it into an error by listing this replication strategy on replication_strategy_fail_list.
< t:2024-10-13 14:03:22,875 f:file_logger.py  l:101  c:sdcm.sct_events.file_logger p:INFO  > 2024-10-13 14:03:22.874: (ScyllaBenchEvent Severity.NORMAL) period_type=begin event_id=cbab4dbd-2a33-428c-b2f0-47235c13d63d: node=Node tablets-split-merge-master-loader-node-ecba0625-1 [3.238.38.192 | 10.12.2.192]
< t:2024-10-13 14:03:23,431 f:db_log_reader.py l:125  c:sdcm.db_log_reader   p:DEBUG > 2024-10-13T14:03:23.386+00:00 tablets-split-merge-master-db-node-ecba0625-1  !WARNING | scylla[5107]:  [shard 0:stmt] create_keyspace - SimpleStrategy replication class is not recommended, but was used for keyspace scylla_bench. You may suppress this warning by delisting SimpleStrategy from replication_strategy_warn_list configuration option, or make it into an error by listing this replication strategy on replication_strategy_fail_list.
< t:2024-10-13 14:05:59,773 f:file_logger.py  l:101  c:sdcm.sct_events.file_logger p:INFO  > 2024-10-13 14:05:59.772: (ScyllaBenchEvent Severity.NORMAL) period_type=end event_id=99ac12c7-5972-4f96-8c65-d56333d17985 duration=2m46s: node=Node tablets-split-merge-master-loader-node-ecba0625-1 [3.238.38.192 | 10.12.2.192]
< t:2024-10-13 14:33:23,546 f:file_logger.py  l:101  c:sdcm.sct_events.file_logger p:INFO  > 2024-10-13 14:33:23.544: (ScyllaBenchEvent Severity.NORMAL) period_type=end event_id=cbab4dbd-2a33-428c-b2f0-47235c13d63d duration=30m0s: node=Node tablets-split-merge-master-loader-node-ecba0625-1 [3.238.38.192 | 10.12.2.192]

So according to this, the warning message is wrong. It would better, of course, be something like "keyspace already exist with network-strategy" or no warning at all.

@bhalevy
Copy link
Member

bhalevy commented Oct 14, 2024

bhalevy@lt db-cluster-ecba0625$ grep -En '(Creat|SimpleStrategy).*scylla_bench' */system.log

tablets-split-merge-master-db-node-ecba0625-1/system.log:994:Oct 13 13:24:28.006881 tablets-split-merge-master-db-node-ecba0625-1 scylla[5107]:  [shard 0:stmt] migration_manager - Create new Keyspace: KSMetaData{name=scylla_bench, strategyClass=org.apache.cassandra.locator.NetworkTopologyStrategy, strategyOptions={"us-east": "3"}, cfMetaData={}, durable_writes=true, tablets={"initial":1}, userTypes=org.apache.cassandra.config.UTMetaData@0x600008debdc0}
tablets-split-merge-master-db-node-ecba0625-1/system.log:995:Oct 13 13:24:28.010203 tablets-split-merge-master-db-node-ecba0625-1 scylla[5107]:  [shard 0: gms] schema_tables - Creating keyspace scylla_bench
tablets-split-merge-master-db-node-ecba0625-1/system.log:998:Oct 13 13:24:28.651227 tablets-split-merge-master-db-node-ecba0625-1 scylla[5107]:  [shard 0: gms] schema_tables - Creating scylla_bench.test id=794be060-8966-11ef-b27d-468d4f8f9f12 version=794be061-8966-11ef-b27d-468d4f8f9f12
tablets-split-merge-master-db-node-ecba0625-1/system.log:1240:Oct 13 13:44:26.923815 tablets-split-merge-master-db-node-ecba0625-1 scylla[5107]:  [shard 0:stmt] create_keyspace - SimpleStrategy replication class is not recommended, but was used for keyspace scylla_bench. You may suppress this warning by delisting SimpleStrategy from replication_strategy_warn_list configuration option, or make it into an error by listing this replication strategy on replication_strategy_fail_list.
tablets-split-merge-master-db-node-ecba0625-1/system.log:1482:Oct 13 14:03:22.990331 tablets-split-merge-master-db-node-ecba0625-1 scylla[5107]:  [shard 0:stmt] create_keyspace - SimpleStrategy replication class is not recommended, but was used for keyspace scylla_bench. You may suppress this warning by delisting SimpleStrategy from replication_strategy_warn_list configuration option, or make it into an error by listing this replication strategy on replication_strategy_fail_list.

tablets-split-merge-master-db-node-ecba0625-2/system.log:957:Oct 13 13:24:28.040989 tablets-split-merge-master-db-node-ecba0625-2 scylla[4859]:  [shard 0: gms] schema_tables - Creating keyspace scylla_bench
tablets-split-merge-master-db-node-ecba0625-2/system.log:960:Oct 13 13:24:28.646355 tablets-split-merge-master-db-node-ecba0625-2 scylla[4859]:  [shard 0:stmt] migration_manager - Create new ColumnFamily: org.apache.cassandra.config.CFMetaData@0x600008963180[cfId=794be060-8966-11ef-b27d-468d4f8f9f12,ksName==scylla_bench,cfName=test,cfType=Standard,comparator=org.apache.cassandra.db.marshal.CompositeType(org.apache.cassandra.db.marshal.LongType,org.apache.cassandra.db.marshal.UTF8Type),comment=,tombstoneGcOptions={"mode":"repair","propagation_delay_in_seconds":"3600"},gcGraceSeconds=864000,minCompactionThreshold=4,maxCompactionThreshold=32,columnMetadata=[ColumnDefinition{name=pk, type=org.apache.cassandra.db.marshal.LongType, kind=PARTITION_KEY, componentIndex=0, droppedAt=-9223372036854775808}, ColumnDefinition{name=ck, type=org.apache.cassandra.db.marshal.LongType, kind=CLUSTERING_COLUMN, componentIndex=0, droppedAt=-9223372036854775808}, ColumnDefinition{name=v, type=org.apache.cassandra.db.marshal.BytesType, kind=REGULAR, componentIndex=null, droppedAt=-9223372036854775808}],compactionStrategyClass=class org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy,compactionStrategyOptions={enabled=true},compressionParameters={},bloomFilterFpChance=0.01,memtableFlushPeriod=0,caching={"keys":"ALL","rows_per_partition":"ALL"},cdc={},defaultTimeToLive=0,minIndexInterval=128,maxIndexInterval=2048,speculativeRetry=NONE,triggers=[],isDense=false,version=794be061-8966-11ef-b27d-468d4f8f9f12,droppedColumns={},collections={},indices={}]
tablets-split-merge-master-db-node-ecba0625-2/system.log:961:Oct 13 13:24:28.657200 tablets-split-merge-master-db-node-ecba0625-2 scylla[4859]:  [shard 0: gms] schema_tables - Creating scylla_bench.test id=794be060-8966-11ef-b27d-468d4f8f9f12 version=794be061-8966-11ef-b27d-468d4f8f9f12
tablets-split-merge-master-db-node-ecba0625-2/system.log:1030:Oct 13 14:03:13.136113 tablets-split-merge-master-db-node-ecba0625-2 scylla[4859]:  [shard 0:stmt] create_keyspace - SimpleStrategy replication class is not recommended, but was used for keyspace scylla_bench. You may suppress this warning by delisting SimpleStrategy from replication_strategy_warn_list configuration option, or make it into an error by listing this replication strategy on replication_strategy_fail_list.

tablets-split-merge-master-db-node-ecba0625-3/system.log:896:Oct 13 13:24:28.040915 tablets-split-merge-master-db-node-ecba0625-3 scylla[4854]:  [shard 0: gms] schema_tables - Creating keyspace scylla_bench
tablets-split-merge-master-db-node-ecba0625-3/system.log:899:Oct 13 13:24:28.657436 tablets-split-merge-master-db-node-ecba0625-3 scylla[4854]:  [shard 0: gms] schema_tables - Creating scylla_bench.test id=794be060-8966-11ef-b27d-468d4f8f9f12 version=794be061-8966-11ef-b27d-468d4f8f9f12
tablets-split-merge-master-db-node-ecba0625-3/system.log:912:Oct 13 13:24:52.327873 tablets-split-merge-master-db-node-ecba0625-3 scylla[4854]:  [shard 0:stmt] create_keyspace - SimpleStrategy replication class is not recommended, but was used for keyspace scylla_bench. You may suppress this warning by delisting SimpleStrategy from replication_strategy_warn_list configuration option, or make it into an error by listing this replication strategy on replication_strategy_fail_list.
tablets-split-merge-master-db-node-ecba0625-3/system.log:921:Oct 13 13:25:09.511674 tablets-split-merge-master-db-node-ecba0625-3 scylla[4854]:  [shard 0:stmt] create_keyspace - SimpleStrategy replication class is not recommended, but was used for keyspace scylla_bench. You may suppress this warning by delisting SimpleStrategy from replication_strategy_warn_list configuration option, or make it into an error by listing this replication strategy on replication_strategy_fail_list.
tablets-split-merge-master-db-node-ecba0625-3/system.log:922:Oct 13 13:25:19.634500 tablets-split-merge-master-db-node-ecba0625-3 scylla[4854]:  [shard 0:stmt] create_keyspace - SimpleStrategy replication class is not recommended, but was used for keyspace scylla_bench. You may suppress this warning by delisting SimpleStrategy from replication_strategy_warn_list configuration option, or make it into an error by listing this replication strategy on replication_strategy_fail_list.

@ptrsmrn
Copy link
Contributor

ptrsmrn commented Oct 14, 2024

Thanks @ptrsmrn , so according to this, a reproduction should be quite simple:

  1. create a keyspace with network strategy.
  2. create (adding "if not exist" or not) the same keyspace with simple-strategy.

Yes and no, because SimpleStrategy doesn't work with tablets, with which the orignal ks was created, so when I only change the replication strategies I get an error, not a warning:

cqlsh> create keyspace ks with replication = { 'class': 'NetworkTopologyStrategy', 'replication_factor': 1 } and tablets = {'initial': 1 };

Warnings :
Tables in this keyspace will be replicated using Tablets and will not support CDC, LWT and counters features. To use CDC, LWT or counters, drop this keyspace and re-create it without tablets by adding AND TABLETS = {'enabled': false} to the CREATE KEYSPACE statement.

Using Replication Factor replication_factor=1 lower than the minimum_replication_factor_warn_threshold=3 is not recommended.

cqlsh> create keyspace if not exists ks with replication = { 'class': 'SimpleStrategy', 'replication_factor': 1 } and tablets = {'initial': 1 };
ConfigurationException: SimpleStrategy doesn't support tablet replication

But, if instead one tries to create a ks with "if not exists", then all the validation is triggered and we may get these warnings, e.g.:

// keyspace `ks` exists
cqlsh> create keyspace if not exists ks with replication = { 'class': 'NetworkTopologyStrategy', 'replication_factor': 1 } and tablets = {'initial': 1 };

Warnings :
Using Replication Factor replication_factor=1 lower than the minimum_replication_factor_warn_threshold=3 is not recommended.

cqlsh> create keyspace if not exists ks with replication = { 'class': 'SimpleStrategy', 'replication_factor': 1 };

Warnings :
SimpleStrategy replication class is not recommended, but was used for keyspace ks. You may suppress this warning by delisting SimpleStrategy from replication_strategy_warn_list configuration option, or make it into an error by listing this replication strategy on replication_strategy_fail_list.

Using Replication Factor replication_factor=1 lower than the minimum_replication_factor_warn_threshold=3 is not recommended.

cqlsh> 

That being said, I'm not sure if these warnings are all that bad, after all someone wants to create a ks with not recommended options, which we are warning about. The query is succesful in the sense that it doesn't return an error (nor creates a ks).

@yarongilor
Copy link
Author

bhalevy@lt db-cluster-ecba0625$ grep -En '(Creat|SimpleStrategy).*scylla_bench' */system.log

tablets-split-merge-master-db-node-ecba0625-1/system.log:994:Oct 13 13:24:28.006881 tablets-split-merge-master-db-node-ecba0625-1 scylla[5107]:  [shard 0:stmt] migration_manager - Create new Keyspace: KSMetaData{name=scylla_bench, strategyClass=org.apache.cassandra.locator.NetworkTopologyStrategy, strategyOptions={"us-east": "3"}, cfMetaData={}, durable_writes=true, tablets={"initial":1}, userTypes=org.apache.cassandra.config.UTMetaData@0x600008debdc0}
tablets-split-merge-master-db-node-ecba0625-1/system.log:995:Oct 13 13:24:28.010203 tablets-split-merge-master-db-node-ecba0625-1 scylla[5107]:  [shard 0: gms] schema_tables - Creating keyspace scylla_bench
tablets-split-merge-master-db-node-ecba0625-1/system.log:998:Oct 13 13:24:28.651227 tablets-split-merge-master-db-node-ecba0625-1 scylla[5107]:  [shard 0: gms] schema_tables - Creating scylla_bench.test id=794be060-8966-11ef-b27d-468d4f8f9f12 version=794be061-8966-11ef-b27d-468d4f8f9f12
tablets-split-merge-master-db-node-ecba0625-1/system.log:1240:Oct 13 13:44:26.923815 tablets-split-merge-master-db-node-ecba0625-1 scylla[5107]:  [shard 0:stmt] create_keyspace - SimpleStrategy replication class is not recommended, but was used for keyspace scylla_bench. You may suppress this warning by delisting SimpleStrategy from replication_strategy_warn_list configuration option, or make it into an error by listing this replication strategy on replication_strategy_fail_list.
tablets-split-merge-master-db-node-ecba0625-1/system.log:1482:Oct 13 14:03:22.990331 tablets-split-merge-master-db-node-ecba0625-1 scylla[5107]:  [shard 0:stmt] create_keyspace - SimpleStrategy replication class is not recommended, but was used for keyspace scylla_bench. You may suppress this warning by delisting SimpleStrategy from replication_strategy_warn_list configuration option, or make it into an error by listing this replication strategy on replication_strategy_fail_list.

tablets-split-merge-master-db-node-ecba0625-2/system.log:957:Oct 13 13:24:28.040989 tablets-split-merge-master-db-node-ecba0625-2 scylla[4859]:  [shard 0: gms] schema_tables - Creating keyspace scylla_bench
tablets-split-merge-master-db-node-ecba0625-2/system.log:960:Oct 13 13:24:28.646355 tablets-split-merge-master-db-node-ecba0625-2 scylla[4859]:  [shard 0:stmt] migration_manager - Create new ColumnFamily: org.apache.cassandra.config.CFMetaData@0x600008963180[cfId=794be060-8966-11ef-b27d-468d4f8f9f12,ksName==scylla_bench,cfName=test,cfType=Standard,comparator=org.apache.cassandra.db.marshal.CompositeType(org.apache.cassandra.db.marshal.LongType,org.apache.cassandra.db.marshal.UTF8Type),comment=,tombstoneGcOptions={"mode":"repair","propagation_delay_in_seconds":"3600"},gcGraceSeconds=864000,minCompactionThreshold=4,maxCompactionThreshold=32,columnMetadata=[ColumnDefinition{name=pk, type=org.apache.cassandra.db.marshal.LongType, kind=PARTITION_KEY, componentIndex=0, droppedAt=-9223372036854775808}, ColumnDefinition{name=ck, type=org.apache.cassandra.db.marshal.LongType, kind=CLUSTERING_COLUMN, componentIndex=0, droppedAt=-9223372036854775808}, ColumnDefinition{name=v, type=org.apache.cassandra.db.marshal.BytesType, kind=REGULAR, componentIndex=null, droppedAt=-9223372036854775808}],compactionStrategyClass=class org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy,compactionStrategyOptions={enabled=true},compressionParameters={},bloomFilterFpChance=0.01,memtableFlushPeriod=0,caching={"keys":"ALL","rows_per_partition":"ALL"},cdc={},defaultTimeToLive=0,minIndexInterval=128,maxIndexInterval=2048,speculativeRetry=NONE,triggers=[],isDense=false,version=794be061-8966-11ef-b27d-468d4f8f9f12,droppedColumns={},collections={},indices={}]
tablets-split-merge-master-db-node-ecba0625-2/system.log:961:Oct 13 13:24:28.657200 tablets-split-merge-master-db-node-ecba0625-2 scylla[4859]:  [shard 0: gms] schema_tables - Creating scylla_bench.test id=794be060-8966-11ef-b27d-468d4f8f9f12 version=794be061-8966-11ef-b27d-468d4f8f9f12
tablets-split-merge-master-db-node-ecba0625-2/system.log:1030:Oct 13 14:03:13.136113 tablets-split-merge-master-db-node-ecba0625-2 scylla[4859]:  [shard 0:stmt] create_keyspace - SimpleStrategy replication class is not recommended, but was used for keyspace scylla_bench. You may suppress this warning by delisting SimpleStrategy from replication_strategy_warn_list configuration option, or make it into an error by listing this replication strategy on replication_strategy_fail_list.

tablets-split-merge-master-db-node-ecba0625-3/system.log:896:Oct 13 13:24:28.040915 tablets-split-merge-master-db-node-ecba0625-3 scylla[4854]:  [shard 0: gms] schema_tables - Creating keyspace scylla_bench
tablets-split-merge-master-db-node-ecba0625-3/system.log:899:Oct 13 13:24:28.657436 tablets-split-merge-master-db-node-ecba0625-3 scylla[4854]:  [shard 0: gms] schema_tables - Creating scylla_bench.test id=794be060-8966-11ef-b27d-468d4f8f9f12 version=794be061-8966-11ef-b27d-468d4f8f9f12
tablets-split-merge-master-db-node-ecba0625-3/system.log:912:Oct 13 13:24:52.327873 tablets-split-merge-master-db-node-ecba0625-3 scylla[4854]:  [shard 0:stmt] create_keyspace - SimpleStrategy replication class is not recommended, but was used for keyspace scylla_bench. You may suppress this warning by delisting SimpleStrategy from replication_strategy_warn_list configuration option, or make it into an error by listing this replication strategy on replication_strategy_fail_list.
tablets-split-merge-master-db-node-ecba0625-3/system.log:921:Oct 13 13:25:09.511674 tablets-split-merge-master-db-node-ecba0625-3 scylla[4854]:  [shard 0:stmt] create_keyspace - SimpleStrategy replication class is not recommended, but was used for keyspace scylla_bench. You may suppress this warning by delisting SimpleStrategy from replication_strategy_warn_list configuration option, or make it into an error by listing this replication strategy on replication_strategy_fail_list.
tablets-split-merge-master-db-node-ecba0625-3/system.log:922:Oct 13 13:25:19.634500 tablets-split-merge-master-db-node-ecba0625-3 scylla[4854]:  [shard 0:stmt] create_keyspace - SimpleStrategy replication class is not recommended, but was used for keyspace scylla_bench. You may suppress this warning by delisting SimpleStrategy from replication_strategy_warn_list configuration option, or make it into an error by listing this replication strategy on replication_strategy_fail_list.

@bhalevy , btw, it looks like the source of the issue comes from this line:
https://github.com/scylladb/scylla-bench/blob/6eca1c970a3a29c3fa15a2e5ca5f5cd468e89fce/main.go#L118
if it make sense, we can open an issue, not to use simple-strategy as default by scylla-bench

@bhalevy
Copy link
Member

bhalevy commented Oct 15, 2024

if it make sense, we can open an issue, not to use simple-strategy as default by scylla-bench

@yarongilor definitely. SimpleStrategy is deprecated and we should stop using it everywhere.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/tablets area/topology changes P3 Medium Priority tests/longevity Issue detected during longevity
Projects
None yet
Development

No branches or pull requests

3 participants