Bug 90815

Summary: [220] [journalctl] fails to open journal files
Product: systemd Reporter: Florian Pritz <bluewind>
Component: generalAssignee: systemd-bugs
Status: RESOLVED WORKSFORME QA Contact: systemd-bugs
Severity: normal    
Priority: medium CC: alexander.dunlap, goeran, onecoldworld, quae
Version: unspecified   
Hardware: x86-64 (AMD64)   
OS: Linux (All)   
URL: https://github.com/systemd/systemd/issues/487
Whiteboard:
i915 platform: i915 features:
Attachments: emerge info

Description Florian Pritz 2015-06-02 12:07:12 UTC
My computer crashed and afterwards I was unable to read my journal. journalctl returns the following error and no log entries:
Error was encountered while opening journal files: No data available

I was also unable to run other journalctl commands like --disk-usage --verify or --list-boots. I downgraded to 219 and was able to read my journal again and bisected the issue to 7b300be75e6d5755778dd7da63e7147866f21351 (sd-journal: return error when we cannot open a file).

journalctl --verify (from 219) tells me that I have some corruption. The 3 files mentioned are timestamped May 1 2015, Jul 14 2014, Jun 2 2015.
444188: invalid object
File corruption detected at /var/log/journal/889479b0ab6d255dc4b25c5448fb1d0f/system@00051500f404eedd-1ca9f54ada114353.journal~:444188 (of 8388608 bytes, 53%).
000098: entry number mismatch
File corruption detected at /var/log/journal/889479b0ab6d255dc4b25c5448fb1d0f/system@1f4ac564b53b45429a68bed9bd4d66b9-00000000000df3c5-0004fdff4b8778af.journal:000000 (of 8388608 bytes, 0%).
8c4000: invalid object
File corruption detected at /var/log/journal/889479b0ab6d255dc4b25c5448fb1d0f/system@00051784997e6452-430d86854332cfc4.journal~:8c4000 (of 16777216 bytes, 54%).

I have not tested if journalctl from 220 actually worked before the crash.

Is this a bug or is this an expected outcome of the mentioned patch?
Comment 1 Daniel Shafer 2015-07-06 03:37:20 UTC
I am getting the same problem as well, but I have no idea if it's related.

I had a huge system crash, and now I can't read journals.

daniel-gentoo ~ # journalctl --verify
Error was encountered while opening journal files: No data available
daniel-gentoo ~ # journalctl --disk-usage
Error was encountered while opening journal files: No data available


It's happening on everything now though

System Info:
Portage 2.2.20 (python 2.7.10-final-0, default/linux/amd64/13.0/desktop/plasma/systemd, gcc-4.9.3, glibc-2.20-r2, 4.1.1-gentoo-r1 x86_64)
=================================================================
                         System Settings
=================================================================
System uname: Linux-4.1.1-gentoo-r1-x86_64-Intel-R-_Core-TM-_i7-4790K_CPU_@_4.00GHz-with-gentoo-2.2
KiB Mem:    32857904 total,  23458340 free
KiB Swap:          0 total,         0 free
Timestamp of repository gentoo: Mon, 06 Jul 2015 03:30:02 +0000
sh bash 4.3_p39
ld GNU ld (Gentoo 2.25 p1.2) 2.25
app-shells/bash:          4.3_p39::gentoo
dev-java/java-config:     2.2.0::gentoo
dev-lang/perl:            5.20.2-r1::gentoo
dev-lang/python:          2.7.10::gentoo, 3.3.5-r1::gentoo
dev-util/cmake:           3.2.3::gentoo
dev-util/pkgconfig:       0.28-r3::gentoo
sys-apps/baselayout:      2.2::gentoo
sys-apps/openrc:          0.17::gentoo
sys-apps/sandbox:         2.6-r1::gentoo
sys-devel/autoconf:       2.13::gentoo, 2.69-r1::gentoo
sys-devel/automake:       1.13.4::gentoo, 1.14.1::gentoo, 1.15::gentoo
sys-devel/binutils:       2.25-r1::gentoo
sys-devel/gcc:            4.9.3::gentoo
sys-devel/gcc-config:     1.8::gentoo
sys-devel/libtool:        2.4.6-r1::gentoo
sys-devel/make:           4.1-r1::gentoo
sys-kernel/linux-headers: 4.1::gentoo (virtual/os-headers)
sys-libs/glibc:           2.20-r2::gentoo
Repositories:

gentoo
    location: /usr/portage
    sync-type: rsync
    sync-uri: rsync://rsync.gentoo.org/gentoo-portage
    priority: -1000

daniel
    location: /usr/local/portage
    masters: gentoo

dotnet
    location: /var/lib/layman/dotnet
    masters: gentoo
    priority: 50

kde
    location: /var/lib/layman/kde
    masters: gentoo
    priority: 50

soehest
    location: /var/lib/layman/soehest
    masters: gentoo
    priority: 50

tomahawk
    location: /var/lib/layman/tomahawk
    masters: gentoo
    priority: 50

vmware
    location: /var/lib/layman/vmware
    masters: gentoo
    priority: 50

Installed sets: @kde-frameworks
ACCEPT_KEYWORDS="amd64 ~amd64"
ACCEPT_LICENSE="* -@EULA"
CBUILD="x86_64-pc-linux-gnu"
CFLAGS="-march=native -O2 -pipe"
CHOST="x86_64-pc-linux-gnu"
CONFIG_PROTECT="/etc /usr/lib64/libreoffice/program/sofficerc /usr/share/config /usr/share/gnupg/qualified.txt /usr/share/themes/oxygen-gtk/gtk-2.0"
CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/dconf /etc/env.d /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/php/apache2-php5.6/ext-active/ /etc/php/cgi-php5.6/ext-active/ /etc/php/cli-php5.6/ext-active/ /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo"
CXXFLAGS="-march=native -O2 -pipe"
DISTDIR="/usr/portage/distfiles"
EMERGE_DEFAULT_OPTS="--quiet-build=y"
FCFLAGS="-O2 -pipe"
FEATURES="assume-digests binpkg-logs config-protect-if-modified distlocks ebuild-locks fixlafiles merge-sync news parallel-fetch preserve-libs protect-owned sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox usersync"
FFLAGS="-O2 -pipe"
GENTOO_MIRRORS="http://distfiles.gentoo.org"
LANG="en_US.utf8"
LDFLAGS="-Wl,-O1 -Wl,--as-needed"
MAKEOPTS="-j14"
PKGDIR="/usr/portage/packages"
PORTAGE_CONFIGROOT="/"
PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --omit-dir-times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages"
PORTAGE_TMPDIR="/var/tmp"
USE="X a52 aac acl acpi aim alsa amd64 berkdb binary-drivers bluetooth bluray branding btrfs bzip2 cairo cdda cdr chart cli corefonts cracklib crypt cups cxx dbus declarative dmraid dri dts dvd dvdr emboss encode exif facebook fam fat firefox flac fortran ftdi gdbm gif glamor google gpm gtk hddtemp hfs hpcups iconv infinality internal-jsoncpp ipv6 irc jpeg kde kipi konsole lcms ldap libnotify lm_sensors mad mdadm meanwhile mercurial minimal mmx mmxext mng modules mp3 mp4 mpeg msn mtp multilib multimedia ncurses networkmanager nls nptl nsplugin ntfs ode ogg oom opengl openmp oscar otr pam pango pcre pdf phonon plasma png policykit ppds pulseaudio qml qt3support qt5 quicktime readline samba sdl semantic-desktop sensord session sip sipe snmp spell sse sse2 ssl startup-notification steam svg systemd tcpd teamarena telepathy thumbnail tiff truetype udev udisks unicode upower usb v4l vaapi vdpau video vlc vnc voice vorbis wayland webp webstart widgets wma wma-fixed wxwidgets x264 x265 xcb xcomposite xfs xinerama xml xmpp xorg xscreensaver xv xvid yahoo zlib" ABI_X86="64" ALSA_CARDS="ali5451 als4000 atiixp atiixp-modem bt87x ca0106 cmipci emu10k1x ens1370 ens1371 es1938 es1968 fm801 hda-intel intel8x0 intel8x0m maestro3 trident usb-audio via82xx via82xx-modem ymfpci" APACHE2_MODULES="authn_core authz_core socache_shmcb unixd actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" CALLIGRA_FEATURES="kexi words flow plan sheets stage tables krita karbon braindump author" CAMERAS="ptp2" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" CPU_FLAGS_X86="aes avx avx2 fma3 mmx mmxext popcnt sse sse2 sse3 sse4_1 sse4_2 ssse3" DRACUT_MODULES="plymouth caps" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf superstar2 timing tsip tripmate tnt ublox ubx" GRUB_PLATFORMS="efi-64" INPUT_DEVICES="keyboard mouse evdev" KERNEL="linux" LCD_DEVICES="lis" LIBREOFFICE_EXTENSIONS="presenter-console presenter-minimizer" OFFICE_IMPLEMENTATION="libreoffice" PHP_TARGETS="php5-5" PYTHON_SINGLE_TARGET="python2_7" PYTHON_TARGETS="python2_7 python3_3" RUBY_TARGETS="ruby19 ruby20" USERLAND="GNU" VIDEO_CARDS="nvidia" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account"
Unset:  CPPFLAGS, CTARGET, INSTALL_MASK, LC_ALL, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS, USE_PYTHON

=================================================================
                        Package Settings
=================================================================

sys-apps/systemd-220-r2::gentoo was built with the following:
USE="acl gudev introspection kmod lz4 pam (policykit) seccomp ssl -apparmor -audit -cryptsetup -curl -doc -elfutils -gcrypt -gnuefi -http -idn -importd -kdbus -lzma -nat -python -qrcode (-selinux) -sysv-utils -terminal -test -vanilla -xkb" ABI_X86="32 64 -x32" PYTHON_SINGLE_TARGET="python2_7 -python3_3 -python3_4" PYTHON_TARGETS="python2_7 python3_3 -python3_4"
Comment 2 Lennart Poettering 2015-09-06 16:26:06 UTC
I am pretty sure this is fixed since PR #695 (v223). Closing hence.

If you manage to reproduce the issue with a newer systemd version please file a new issue on github, and attach the journal files that trigger the issue. Thanks.
Comment 3 rtgiskard 2016-10-24 02:14:34 UTC
I'm pretty sure that this problem appears again, at least with sys-apps/systemd-226-r2
Comment 4 rtgiskard 2016-10-24 02:16:00 UTC
Created attachment 127507 [details]
emerge info
Comment 5 Lennart Poettering 2017-10-27 18:02:37 UTC
Hmm, so in the meantime quite a number of additional fixes have been added to the journal file decoder, hence I figure this is fixed now. If this is still a problem in v235, please open a new issue on github, and attach the offending journal files (or send them to me personally, by email)

Use of freedesktop.org services, including Bugzilla, is subject to our Code of Conduct. How we collect and use information is described in our Privacy Policy.