#nodetool

I used the #PenTool & #NodeTool mostly in #AffinityDesigner2. Then generated a #plasma#texture in the free programme #Krita. I 'married' both elements in #AffinityPhoto2 & added some Textures which I made from scratch by using 'foggy' Brushes.

I wanted to try my hands on a very³ quick #DigitalMandala style piece to calm down while gathering some new ideas for the future. Our Geography/History/Politics Teacher in Middle School, the lovely Mr. H., would give us a 'Mandala #pattern sheet' to colour the pattern as we wished, if we were done on a test while the rest of the class wasn't. I loved that as a kid, very calming.💖 I hope you like it.

(Of course this is not really a sacred Mandala or #Yantra per se. But for for the sake of convenience I'll use this word)

We're using Cassandra for a project at work. In one lower environment I've launched without problems a 6 nodes cluster at the beginning of the week. I'm saying without problems because all boxes came up ok and no clients had problems talking to the cluster.

But when I ran nodetool ring from one of the boxes it only showed 5 machines, one of which was owning 33% of the ring:

Address DC Rack Status State Load Owns Token 141784319550391026443072753098378663705 <IP1> us-east 1a Up Normal 271.02 KB 16.67% 1808575600 <IP2> us-east 1d Up Normal 417.66 KB 33.33% 56713727820156410577229101240436610842 <IP3> us-east 1a Up Normal 324.21 KB 16.67% 85070591730234615865843651859750628463 <IP4> us-east 1d Up Normal 125.87 KB 16.67% 113427455640312821154458202479064646084 <IP5> us-east 1e Up Normal 268.65 KB 16.67% 141784319550391026443072753098378663705

When ran from any of the 6 boxes, including the one that was missing from the list - let's call it IPX - nodetool ring showed everyone was up and normal & pretty much the same information. Running nodetool info on IPX it seemed to think it had the same token as IP5 (the other box in AZ 1e):

Token : 141784319550391026443072753098378663705 Gossip active : true Load : 322.3 KB Generation No : 1388719379 Uptime (seconds) : 254182 Heap Memory (MB) : 89.50 / 3753.88 Data Center : us-east Rack : 1e Exceptions : 0

To try and make things whole again I decided to move the token of IPX to a better place on the ring.

As a consequence running nodetool info from IP5 produced:

Token : 113427455640312821154458202479064646084 Gossip active : true Load : 317.96 KB Generation No : 1388719379 Uptime (seconds) : 257647 Heap Memory (MB) : 85.41 / 3753.88 Exception in thread "main" java.lang.AssertionError: Could not find myself in the endpoint list, something is very wrong! at org.apache.cassandra.tools.NodeProbe.getEndpoint(NodeProbe.java:557) at org.apache.cassandra.tools.NodeProbe.getDataCenter(NodeProbe.java:564) at org.apache.cassandra.tools.NodeCmd.printInfo(NodeCmd.java:313) at org.apache.cassandra.tools.NodeCmd.main(NodeCmd.java:651)

 And running nodetool ring from another box showed:

Address DC Rack Status State Load Owns Token 141784319550391026443072753098378663705 <IP1> us-east 1a Up Normal 275.44 KB 16.67% 1808575600 <IPX> us-east 1d Up Normal 273.17 KB 16.67% 28356863910078205288614550619314017621 <IP2> us-east 1d Up Normal 422.08 KB 16.67% 56713727820156410577229101240436610842 <IP3> us-east 1a Up Normal 315.42 KB 16.67% 85070591730234615865843651859750628463 <IP4> us-east 1e Up Normal 273.07 KB 33.33% 141784319550391026443072753098378663705

So it looks like IPX got in the ring and IP5 got kicked out (and it's not happy about it)...

Restarting Cassandra on IP5 seemed to make everyone happy, so I ran nodetool cleanup followed by nodetool repair to wrap things up.