Browse Poster Wiki: 2022 Poster Award Winners, Awards Ceremony (Watch Recording from minute marker 1630), Posters by 2022 HPCC Systems Interns, Posters by Academic Partners, Poster Judges, About Virtual Judging, 2022 Poster Contest Home Page, Poster Contest Previous Years

Nathalia Ribas is a second-year Computer Engineering student at Federal University of Santa Catarina, Brazil.

Since August 2021, Nathalia was working on her undergraduate research with the help of Alysson Oliveira and Hugo Watanuki from LexisNexis Risk Solutions.

2022 Winner of the Community Choice Award

Poster Abstract

The importance of log monitoring and analysis during, and after, an application development is essential nowadays. Indeed, log files play an indispensable role during an in-depth analysis of an application’s data: in addition to containing in-depth data of each system’s general information, it is possible to find abnormal behaviors and detect errors and their reasons more easily, making the search for an optimized solution faster.

However, with the increased amount of data being generated by IT systems today and the adoption of a cloud computing paradigm, log analysis can be a daunting problem as a large amount of remote data must be processed every second. In this context, the analysis of log files becomes a complex challenge not only because there is too much data to be processed and something can go unnoticed, but also because the remote access to log files in a cloud infrastructure can be slow, thus making real time analysis almost impossible in some cases. This is the context where the log management tools can play an important role.

This study aims to explore how log analysis can be implemented effectively in a cloud containerized Big Data processing platform. For this purpose, an experimental design was built using the Big Data platform HPCC Systems, and the log filtering, management, and visualization tools Elasticsearch and Kibana.

As a result of this study a series of log analysis dashboards were proposed in an attempt to facilitate visualization of data such as status of work units and platform components. It is expected that the results of this study will contribute with ECL programmers and HPCC Systems administrators in better understanding the behavior of their work units and the platform components, respectively

Presentation

In this Video Recording, Nathalia provides a tour and explanation of her poster content.

HPCC Systems Log Monitoring in the Cloud

Click on the poster for a larger image.