GeeCON 2017: Piotr Przybył - Graylog: one tool to log 'em all

### Overview At GeeCON 2017, Piotr Przybył presented an overview and practical demonstration of Graylog, an open-source log aggregation and analysis platform. He distinguished debugging (evaluating running program state at breakpoints in an IDE) from logging, which is post-factum or post-mortem and relies exclusively on recorded logs. Przybył emphasized that in non-ideal real-world environments, centralized remote logging is critical for post-incident diagnostics, system visibility, and handling multi-instance or microservice architectures.

### Key Topics - **Architecture and Deployment**: Graylog is distributed as a single fat JAR embedding a Grizzly/Jersey REST server and a React.js web client. State and configuration metadata are stored in MongoDB, while log message indexing and searching are handled by Elasticsearch. Deployment options range from virtual machine appliances (such as Oracle VirtualBox) to production tarballs and orchestration scripts. - **Ingestion and Inputs**: Graylog utilizes inputs that collect and interpret specific log formats, such as standard syslog over UDP or the Graylog Extended Log Format (GELF). - **Application Integration**: Java applications can integrate with Graylog without code modifications using SLF4J, Logback, and appenders like logstash-gelf. Przybył demonstrated capturing per-thread contextual data via Mapped Diagnostic Context (MDC), including user identities and instance IDs. - **Log Organization and Processing**: Using features like Streams, Extractors (e.g., regex-based field cleanup), and Pipelines with staged rules, incoming logs can be filtered, mutated, and routed based on conditions like CIDR IP matching. - **Alerting and Dashboards**: Graylog allows defining stream-based alert conditions (such as specific field values like HTTP 500 status codes, or message counts drops to zero) to trigger notifications via email, HTTP requests, or external monitoring systems.

### Technical Details - **Timezone Pitfalls**: A critical troubleshooting point highlighted is mismatched timezones. Devices emitting logs without explicit timezone data (such as routers sending syslog via UDP) are interpreted by Graylog in UTC. If the sending device uses local time without an offset, logs appear as timestamps from the future and will not show in default relative searches. - **Processing Pipelines**: Przybył demonstrated solving timezone mismatches using a processing pipeline rule. Stages execute in ascending numerical order, where an earlier stage can halt subsequent processing. The rule parsed the router's raw timestamp, formatted it with the `Europe/Warsaw` (Europe/Krakow) timezone, and replaced the original timestamp field without modifying application code. - **Distributed Tracking via MDC**: In a simulated microservice setup running across multiple GlassFish instances behind a round-robin load balancer, MDC fields (`username`, `instance`, `response_code`) were exported into Graylog via GELF. This allowed tracing a single user's actions across multiple backend nodes, displaying chronological surrounding messages and diagnosing intermittent HTTP 500 errors. - **Internal Components**: Graylog utilizes Apache Shiro for role-based access control, Google Guice (Juice) for dependency injection and multi-bindings plugins, Airline for CLI command parsing, and Swagger for browsing the underlying REST API.

### Takeaways Centralized logging with Graylog provides unified search across distributed applications and hardware devices where manual log gathering via SSH/SCP is impractical. For sizing, performance testing in pre-production is required, and adding RAM typically provides more immediate performance gains than adding Graylog nodes. When compared to the ELK stack, Graylog was chosen by the speaker due to its GPL license and its purpose-built, log-focused web UI.