Improved node setup and monitoring for the validator of the Agoric testnet
I have set up a node for the agoric testnet on my dedicated server with Ubuntu 20.04 according to the official validator guide, but with some improvements:
- I have created a separate user and installed GO into it’s home directory, to prevent interfiering with any other software installed on the server uses GO from the /usr/local:
curl https://dl.google.com/go/go1.15.7.linux-amd64.tar.gz | sudo tar -C/home/agoric -zxvf -
export GOROOT=$HOME/go
2. I have changed a default listen ports of validator and RPC to another. This will require to set up additional parameter for the ag-cosmos-helper to use non-default port:
-n tcp://HOST:PORT
--node tcp://HOST:PORT
3. I have increased a open files limit at the systemd unit, because default value of 4096 could be ok at the start of the network, but later it could be suddenly hit the limit:
LimitNOFILE=200000
4. Protect prometheus metrics ports by the firewall, left access only for the whitelisted IPs. There is an example for the 9464 port:
iptables -I INPUT -p tcp --dport 9464 <ALLOWED IP> -j ACCEPT
iptables -I INPUT -p tcp --dport 9464 -j DROP
5. And the most important part: I have set up a monitoring with telegram bot notifications. This shell script below will check status of the validator with selected moniker, and send a message throught the telegram bot if validator is jaled:
#!/bin/sh
time=`date “+%Y-%m-%d-%H:%M”`
check=`ag-cosmos-helper query — node tcp://localhost:28657 staking validators --output json | jq ‘.validators[] | select(.description.moniker==”johndoe”) | .jailed’ | grep -v false`
if [ -z “$check” ]
then
echo “$time: agoric node is OK” >> /home/scripts/agoric.log
else
/usr/bin/curl -s -X POST https://api.telegram.org/botXXXXXXXXXXX:TELEGRAM_API_KEY/sendMessage -d chat_id=CHAT_ID -d text=”agoric validator is jailed”;
echo “$time: agoric node is jailed” >> /home/scripts/agoric.log
fi
Where botXXXXXXXXXXX is the telegram bot name, TELEGRAM_API_KEY is a bot api key and CHAT_ID is the id of the chat beetween bot and you. More detailed info about creating a telegram bot is described at the telegram faq: https://core.telegram.org/bots
Shell script has been added to the cron with reasonable check period.
