Open-source News

Nitrux 2.5 Is Here as One of the First Distros Powered by Linux 6.0 and KDE Plasma 5.26  9to5Linux

Mozilla may extend Firefox on Windows 7 and 8.1 support  Ghacks

SuperTuxKart 1.4 is out today as the newest feature release to this open-source kart racing game originally inspired by Mario Kart...

SuperTuxKart 1.4 Released With Initial Vulkan Renderer  Phoronix

How to Remove Snap Packages from Ubuntu and Other Linux  It's FOSS

Clément Lefèbvre has published the latest monthly status report for the Linux Mint distribution that is the popular desktop Linux distribution based on Ubuntu Linux...

Linux Mint Makes Improvements Around Flatpaks With Update Manager Integration  Phoronix

Landing in "net-next" on Monday is wired networking support for the MotorComm YT8521 Ethernet Gigabit PHY. This network ASIC may not ring a bell for most folks, but is used so far by one notable RISC-V development board...

MotorComm YT8521 Gigabit Ethernet Support Coming For Linux 6.2  Phoronix

With the fifteen Linux hardware reviews and 245 original open-source/Linux news stories written by your's truly last month, here is a look back at what was exciting Phoronix readers the most from Google's new KataOS to the release of the speedy Python 3.11, Linux 6.1 taking shaping, and Intel releasing Arc Graphics A750 and A770 graphics cards...

Arc Graphics, Google KataOS, Python 3.11 & Linux 6.1 Excited Open-Source Enthusiasts  Phoronix

Adding to the long list of Zink OpenGL-on-Vulkan driver improvements coming with Mesa 22.3 this quarter is now a working Mesa front-end shader caching implementation...

OBS Studio as the leading open-source and cross-platform screencasting/streaming app that is popular with gamers and YouTubers is now out with version 28.1...

Linux world gains ability to repair exFAT drives  The Register

Linux Lite 6.2: Latest release from distro with a misleading name  The Register

Linux Fu: Easy VMs  Hackaday

Linux world gains ability to repair exFAT drives  The Register

How to Force Linux Users to Change Password at Next Login  MUO - MakeUseOf

First Arch Linux ISO Release Powered by Linux Kernel 6.0 Is Now Available for Download  9to5Linux

Get started with Parseable, an open source log storage and observability platform Nitish Tiwari Tue, 11/01/2022 - 03:00

Log data is one of the fastest-growing segments across data storage. It's also one of the most complicated spaces. There are several products and solutions with overlapping use cases and confusing marketing.

This article looks at Parseable, a log storage and observability platform. Parseable is geared towards a better user experience, with an easy-to-deploy and use interface and a simple, cloud-native architecture. I'll also show how to set up Parseable with FluentBit to store logs.

More on Microservices Microservices cheat sheet How to explain microservices to your CEO Free eBook: Microservices vs. service-oriented architecture Free online course: Developing cloud-native applications with microservices arc… Latest microservices articles What is Parseable?

Parseable is a free and open source log storage and observability platform. Written in Rust, Parseable leverages data compression, storage, and networking advances to bring a simple, efficient logging platform that just works.

Some core concepts for building Parseable are:

Indexing free

Traditionally, text search engines like Elastic have doubled as log storage platforms. This makes sense because log data must be searched to be really useful. But indexing comes at a high cost. It is CPU intensive and slows down ingestion. Also, the index data generated by these systems are of the same order of storage as the raw log data. This doubles the storage cost and increases complexity. Parseable changes this. With columnar data formats (parquet), it is possible to compress and query the log data efficiently without indexing it.

Ownership of both data and content

With parquet as the storage format and stored in standard object storage buckets, users own their log data and have complete access to the actual content. This means users can easily use analysis tools like Spark, Presto, or TensorFlow to extract more value from the data. This feature is extremely powerful, opening up new avenues of data analysis.

Fluid schema

Logs are generally semi-structured by nature, and they're ever-evolving. For example, a developer may start with a log schema like this:

{
  "Status": "Ready",
  "Application": "Example"
}

But as more information is collected, the log schema may evolve to:

{
  "Status": "Ready",
  "Application": {
    "UserID": "3187F492-8449-4486-A2A0-015AE34F1D09",
    "Name": "Example"
  }
}

Engineering and SRE teams regularly face schema-related issues. Parseable solves this with a fluid schema approach that lets users change the schema on the fly.

Simple ingestion

The current ingestion mechanism for logging platforms is quite convoluted, with several available protocols and connectors. Parseable aims to make log ingestion as easy as possible. The result is you can use HTTP POST calls to send logs to Parseable. No complicated SDKs are required.

What if you want to use a logging agent like FluentBit, Vector, LogStash, or others? Almost all the major log collectors support HTTP, so Parseable is already compatible with your favorite log collection agent.

Get started

You can use a Docker image to try out Parseable. This image shows Parseable in demo mode, using publicly-accessible object storage.

$ cat << EOF > parseable-env
P_S3_URL=https://minio.parseable.io:9000
P_S3_ACCESS_KEY=minioadmin
P_S3_SECRET_KEY=minioadmin
P_S3_REGION=us-east-1
P_S3_BUCKET=parseable
P_LOCAL_STORAGE=/data
P_USERNAME=parseable
P_PASSWORD=parseable
EOF

$ mkdir -p /tmp/data

$ docker run \
  -p 8000:8000 \
  --env-file parseable-env \
  -v /tmp/data:/data \
  parseable/parseable:latest

Log in to the Parseable UI using the credentials passed here (that's parseable and parseable.) The demo already contains some data because Parseable is pointing to the publicly-open bucket.

Make sure to change the bucket and credentials to your object storage instance before sending any data to Parseable.

Refer to the documentation to understand how Parseable works and how to ingest logs.

Set up FluentBit to send logs to Parseable

You can use a Docker compose file to configure both Parseable and FluentBit, making it easier to set up and tear down as needed.

First, save this file as fluent-bit.conf in a directory. The file is the configuration used to send data to Parseable.

[SERVICE]
  Flush 5
  Daemon Off
  Log_Level debug

[INPUT]
  Name dummy
  Tag dummy

[OUTPUT]
  Name http
  Match *
  Host parseable
  http_User parseable
  http_Passwd parseable
  format json
  Port 8000
  Header X-P-META-meta1 value1
  Header X-P-TAG-tag1 value1
  URI /api/v1/logstream/fluentbit1
  Json_date_key timestamp
  Json_date_format iso8601

Now save the following file as docker-compose.yaml in the same directory as above:

version: "3.7"

services:
  fluent-bit:
    image: fluent/fluent-bit
    volumes:
     - ./fluent-bit.conf:/fluent-bit/etc/fluent-bit.conf
    depends_on:
     - parseable

  parseable:
    image: parseable/parseable
    ports:
     - "8000:8000"
    environment:
     - P_S3_URL=https://minio.parseable.io:9000
      - P_S3_ACCESS_KEY=minioadmin
      - P_S3_SECRET_KEY=minioadmin
      - P_S3_REGION=us-east-1
      - P_S3_BUCKET=parseable
      - P_LOCAL_STORAGE=/tmp/data
      - P_USERNAME=parseable
      - P_PASSWORD=parseable

The docker-compose.yaml refers to the fluent-bit.conf file and passes it to the FluentBit container as the configuration file.

Parseable is deployed with the default configuration (as in the above Docker setup). You can observe the data FluentBit container sent to Parseable in the Parseable Console running at http://localhost:8000.

Wrap up

In this article, you've taken your first look at Parseable, the open source log storage and analysis platform built in Rust. A single Docker command gets you started with Parseable so you can experience the UI and establish FluentBit as a data source. If you think this looks too easy, then it's probably time to try Parseable!

Written in Rust, Parseable leverages data compression, storage, and networking advances to bring a simple, efficient logging platform that just works.

Image by:

Opensource.com

Rust Microservices Cloud What to read next This work is licensed under a Creative Commons Attribution-Share Alike 4.0 International License. Register or Login to post a comment.