Proxmox: How to resolve “service start-limit-hit”

Posted on by mario

The error message on the proxmox gui

()
Job for ceph-mgr@pve-03.service failed.
See "systemctl status ceph-mgr@pve-03.service" and "journalctl -xe" for details.
TASK ERROR: command '/bin/systemctl start ceph-mgr@pve-03' failed: exit code 1

The error message from systemctl

ceph-mgr@pve-03.service: Start request repeated too quickly.
ceph-mgr@pve-03.service: Failed with result 'start-limit-hit'.

Solve with

systemctl reset-failed ceph-mgr@pve-03
systemctl start ceph-mgr@pve-03

change pve-03 to your node name.

proxmox qmp command blockdev-snapshot-delete-internal-sync failed

Posted on by mario

While trying to move a VM from one node to another I got the error message:

VM 100 qmp command ‘blockdev-snapshot-delete-internal-sync’ failed – Failed to get a snapshot list: Operation not supported

One snapshot was stuck and the VM locked. How to solve this?

qm unlock 100
qm listsnapshot 100
qm delsnapshot 100 preFirstBoot --force

qm unlock <ID>
qm listsnapshot <ID>
qm delsnapshot <ID> <snapname> --force

it might be that the snapshot will remain on the hard disk.

Happy unlocking :)

zfs snapshot send resume

Posted on by mario 
zfs send ... |  ssh host2 zfs receive -s otherpool/new-fs

On the receiving side, get the opaque token with the DMU object #, offset stored in it

zfs send ... |  ssh host2 zfs receive -s otherpool/new-fs
zfs get receive_resume_token otherpool/new-fs
# 1-e604ea4bf-e0-789c63a2...

Re-start sending from the DMU object #, offset stored in the token

zfs send -t 1-e604ea4bf-e0-789c63a2... |  ssh host2 zfs receive -s otherpool/new-fs

If you don’t want to resume the send, abort to remove the partial state on the receiving system

zfs receive -A otherpool/new-fs

ZFS enable email notification

Posted on by mario

Edit 

/etc/zfs/zed.d/zed.rc

uncomment 

ZED_EMAIL_ADDR="mail@example.com"

and add a valid email address.

uncomment 

ZED_EMAIL_PROG="mail"

uncomment 

ZED_EMAIL_OPTS="-s '@SUBJECT@' @ADDRESS@"

uncomment 

ZED_NOTIFY_VERBOSE=0

if you want to get an email after every scrup set the value to 1

save the file and restart zed service

systemctl restart zed.service

esxi enable snmp

Posted on by mario

How to enable snmapd on your vmware esxi server

replace 10.10.0.0 with your IP-range
esxcli system snmp set --communities public
esxcli system snmp set --enable true
esxcli network firewall ruleset set --ruleset-id snmp --allowed-all true
esxcli network firewall ruleset set --ruleset-id snmp --enabled true
esxcli system snmp set --syslocation "My Location"
esxcli system snmp set --targets=10.10.0.0@161/public

Now you can start the service from the UI

failed Import ZFS pools by cache file

Posted on by mario

A single disk zpool “test” crashed on my server (the disk died). It was just for testing, so nothing dramatic. However, when I rebooted the server I got the error message “failed Import ZFS pools by cache file”.  A zpool destroy -f did not solve the problem. zpool status still showed the “test” pool. The other pool tank was still working.

What did help was

# disable the cache file for the existing pool(s)
zpool set cachefile=none tank
# delete the old pool file
rm -rf /etc/zfs/zpool.cache
# recreate if
touch /etc/zfs/zpool.cache
reboot
# re-enable the cache
zpool set cachefile=/etc/zfs/zpool.cache tank

Well, the cache file should be automatically updated when your pool configuration is changed, but with the crashed pool it did not.

resize2fs new size too large to be expressed in 32 bits

Posted on by mario

After virtualizing a real computer with an old Linux I wanted to increase the partition size of the data drive. But I got this warning: resize2fs new size too large to be expressed in 32 bits

How to solve this? I started the VM with gparted-live.iso

# check file system
e2fsck -f /dev/sdb1
# auf 64 bit ändern
resize2fs -b /dev/sdb1
# increase partition .... wait :D / optional coffee
resize2fs -p /dev/sdb1
# check file system
e2fsck -f /dev/sdb1

Done :)

ban ’em all

Posted on by mario

Ban all the attackers. Easier said than done. A website is constantly under attack as the whole server. One day I decided it was too difficult to maintain every single server and ban those attackers. Blocking IPs on the website level is too late. Also, it consumes a lot of resources. So I went for iptables. You can find it on github/JBlond/ban_em_all

DROP vs REJECT. Well, DROP is a bad option for debugging. Also, it is not the default behavior of the OS itself. Nothing is listing on a port? The OS sends a reject. Sadly I haven’t found a way to use REJECT when it comes to IPs. Using DROP on the other hand the automatic server/website scanners assume a firewall and it is more likely to continue the scan.

Add real emojis and icons to mintty bash on windows

Posted on by mario

Installing git for windows isn’t hard anymore. I recommend installing “Git for Windows SDK” so you can install extra packages via Pacman. I like fish shell and tmux, and those aren’t included in the default installer. Either way, you can run bash and or fish on windows. The emojis and icons however don’t look very pretty, more like DOS-style.

Open a shell and cd to

C:\Program Files\Git\usr\share\mintty\emojis or C:\git-sdk-64\usr\share\mintty\emojis depending what installation you chose.

cd C:\git-sdk-64\usr\share\mintty\
mkdir emojis
cd emojis
curl -LO https://raw.githubusercontent.com/wiki/mintty/mintty/getemojis
getemoji -d
getemoji .

In mintty:
Options -> Text -> Emojis -> style: google
Options -> Text -> Character set -> UTF-8