Zookeeper Issue : Last transaction was partial
Issue: ZooKeeper was continuously crashing with below
error
2018-03-23 12:20:53,374 ERROR
org.apache.zookeeper.server.persistence.Util: Last transaction was partial.
2018-03-23 12:20:53,375 ERROR
org.apache.zookeeper.server.ZooKeeperServerMain: Unexpected exception, exiting
abnormally
Brief: Like any other transnational system;
zookeeper works just like any other software where any transactions that are
related to state of zookeeper will be written first to the disk and then its
updates takes place. When the transaction log file reaches a certain size,
a new transaction log file gets created.
ZooKeeper stores its data in a data directory and its
transaction log in a transaction log directory.
Data Directory
- myid - contains a single integer in human readable ASCII text that represents the server id.
- snapshot.<zxid> - holds the fuzzy snapshot of a data tree.
Log Directory
The Log Directory contains the ZooKeeper transaction
logs. Before any update takes place, Zookeeper ensures that the transaction
that represents the update is written to non-volatile storage.
A new log file
is started each time a snapshot is begun. The log file's suffix is the first
zxid written to that log.
After talking to the team and getting info what all was
done prior this error; came to know Hadoop eco system was aborted once during
one transaction activity.
Since the termination was abnormal and issue was
related to transaction entry so I rush to LOG directory and saw
Zookeeper transactions log file was sized 0<junk file> ; which means that while
booting up ; zookeeper was trying apply logs for the consistent state but due to
0 sized it didn’t manage to replay the transactions and hence failed to
start.
-rw-r--r-- 1 zookeeper
zookeeper 0 Mar 23 03:17 log.da092
Solution: I removed the junk log file and started
Zookeeper again and succeeded. Simple enough!!!!
Writing to transaction log files is not efficient step
for a heavy loaded system, because let say on startup zookeeper would have to
replay every transaction it ever processed.
So periodically, zookeeper will write a snapshot of the
current state of its in memory database to file.
In short both snapshot and transactions logs are very important
to zookeeper.
Enjoy Learning!!!
